Skip to content

[SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package #45053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 15 commits into from
Closed

[SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package #45053

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
3817a45
Decouple PySpark core API to pyspark.core package
HyukjinKwon Feb 7, 2024
f0a70c5
Separate packaging for Pure Python library
HyukjinKwon Feb 8, 2024
1da14d6
Fix test failure
HyukjinKwon Feb 8, 2024
ebb9ac9
fix pip installation
HyukjinKwon Feb 8, 2024
b85f7d3
Fix errors
HyukjinKwon Feb 9, 2024
b8e942d
Fix pip packaging
HyukjinKwon Feb 10, 2024
08ab336
fix packaging
HyukjinKwon Feb 10, 2024
3013053
Fix packaging
HyukjinKwon Feb 10, 2024
ea9e66c
fix import
HyukjinKwon Apr 2, 2024
88da76e
fixup linter
HyukjinKwon Apr 2, 2024
b186b67
simplify
HyukjinKwon Apr 2, 2024
74b9521
avoid circular import
HyukjinKwon Apr 2, 2024
66c6105
fix up
HyukjinKwon Apr 2, 2024
de8bb09
Update util.py
HyukjinKwon Apr 2, 2024
4919bea
Address compatibility concerns
HyukjinKwon Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix errors
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Apr 3, 2024
commit b85f7d3bcdced0b76f8833fbc1938c97ebdf9914
2 changes: 1 addition & 1 deletion dev/run-pip-tests
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ for python in "${PYTHON_EXECS[@]}"; do
echo "Run basic sanity check with import based"
python3 "$FWDIR"/dev/pip-sanity-check.py
echo "Run the tests for context.py"
python3 "$FWDIR"/python/pyspark/context.py
python3 "$FWDIR"/python/pyspark/core/context.py

cd "$FWDIR"

Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/sql/readwriter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -838,7 +838,7 @@ def csv(

has_core = True
try:
import pyspark.core # noqa: F401
from pyspark.core.rdd import RDD # noqa: F401
except ImportError:
has_core = False

Expand Down Expand Up @@ -968,7 +968,7 @@ def xml(

has_core = True
try:
import pyspark.core # noqa: F401
from pyspark.core.rdd import RDD # noqa: F401
except ImportError:
has_core = False

Expand Down