Skip to content

[SPARK-43082][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect #40725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xinrong-meng wants to merge 13 commits into from
Closed

[SPARK-43082][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect #40725

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
63bb36b
_create_arrow_py_udf
xinrong-meng Apr 10, 2023
d702b67
in Connect
xinrong-meng Apr 10, 2023
01f7190
tests
xinrong-meng Apr 10, 2023
0fb7712
- debug
xinrong-meng Apr 10, 2023
f46d006
docstrings
xinrong-meng Apr 10, 2023
3abeef4
TEST
xinrong-meng Apr 10, 2023
f6fc6e1
TEST
xinrong-meng Apr 12, 2023
63ef94e
rmv duplicate test
xinrong-meng Apr 19, 2023
86938d5
tearDownClass
xinrong-meng Apr 20, 2023
f5aef18
rmv f from _create_arrow_py_udf
xinrong-meng Apr 20, 2023
f313063
UserWarning
xinrong-meng Apr 20, 2023
5e78632
finally super tearDownClass
xinrong-meng Apr 20, 2023
ac86bf1
fallback to regular udf
xinrong-meng Apr 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
tearDownClass
  • Loading branch information
@xinrong-meng
xinrong-meng committed Apr 20, 2023
commit 86938d56dc2493c366b1088b6a623e324c69ebfe
5 changes: 5 additions & 0 deletions python/pyspark/sql/tests/connect/test_parity_arrow_python_udf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ def setUpClass(cls):
super(ArrowPythonUDFParityTests, cls).setUpClass()
cls.spark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")

@classmethod
def tearDownClass(cls):
cls.spark.conf.unset("spark.sql.execution.pythonUDF.arrow.enabled")
super(ArrowPythonUDFParityTests, cls).tearDownClass()


if __name__ == "__main__":
import unittest
Expand Down
5 changes: 5 additions & 0 deletions python/pyspark/sql/tests/test_arrow_python_udf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,11 @@ def setUpClass(cls):
super(PythonUDFArrowTests, cls).setUpClass()
cls.spark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")

@classmethod
def tearDownClass(cls):
cls.spark.conf.unset("spark.sql.execution.pythonUDF.arrow.enabled")
super().tearDownClass()


if __name__ == "__main__":
from pyspark.sql.tests.test_arrow_python_udf import * # noqa: F401
Expand Down