Skip to content

[SPARK-43082][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect #40725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xinrong-meng wants to merge 13 commits into from
Closed

[SPARK-43082][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect #40725

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
63bb36b
_create_arrow_py_udf
xinrong-meng Apr 10, 2023
d702b67
in Connect
xinrong-meng Apr 10, 2023
01f7190
tests
xinrong-meng Apr 10, 2023
0fb7712
- debug
xinrong-meng Apr 10, 2023
f46d006
docstrings
xinrong-meng Apr 10, 2023
3abeef4
TEST
xinrong-meng Apr 10, 2023
f6fc6e1
TEST
xinrong-meng Apr 12, 2023
63ef94e
rmv duplicate test
xinrong-meng Apr 19, 2023
86938d5
tearDownClass
xinrong-meng Apr 20, 2023
f5aef18
rmv f from _create_arrow_py_udf
xinrong-meng Apr 20, 2023
f313063
UserWarning
xinrong-meng Apr 20, 2023
5e78632
finally super tearDownClass
xinrong-meng Apr 20, 2023
ac86bf1
fallback to regular udf
xinrong-meng Apr 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
- debug
  • Loading branch information
@xinrong-meng
xinrong-meng committed Apr 17, 2023
commit 0fb7712ae93662172db19d403badb511ff8b6214
1 change: 0 additions & 1 deletion python/pyspark/sql/connect/udf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ def _create_py_udf(
and not isinstance(return_type, ArrayType)
)
if is_arrow_enabled and is_output_atomic_type and is_func_with_args:
print("entering _create_arrow_py_udf")
return _create_arrow_py_udf(f, regular_udf)
else:
return regular_udf
Expand Down
1 change: 0 additions & 1 deletion python/pyspark/sql/udf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,6 @@ def _create_py_udf(


def _create_arrow_py_udf(f, regular_udf): # type: ignore
Copy link
Member Author

@xinrong-meng xinrong-meng Apr 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignoring the type annotations of _create_arrow_py_udf because it is shared between vanilla PySpark and Spark Connect Python Client.

Copy link
Member Author

@xinrong-meng xinrong-meng Apr 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function is only an extraction of original code L142 - L179 for code reuse.

print("entered _create_arrow_py_udf")
require_minimum_pandas_version()
require_minimum_pyarrow_version()

Expand Down