Skip to content

[SPARK-48733][PYTHON][TESTS] Do not test SET command in Python UDTF test #47117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 1 commit into from
Closed

[SPARK-48733][PYTHON][TESTS] Do not test SET command in Python UDTF test #47117

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Do not test SET command in Python UDTF test
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Jun 27, 2024
commit 3626e8e2aef6a955ee9baa8f12884667c7fa72dc
20 changes: 11 additions & 9 deletions python/pyspark/sql/tests/test_udtf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2559,16 +2559,18 @@ class TestUDTF:
def eval(self):
yield 1,

# We do not use `self.sql_conf` here to test the SQL SET command
# instead of using PySpark's `spark.conf.set`.
old_value = self.spark.conf.get("spark.sql.execution.pythonUDTF.arrow.enabled")
self.spark.sql("SET spark.sql.execution.pythonUDTF.arrow.enabled=False")
self.assertEqual(udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_TABLE_UDF)
self.spark.sql("SET spark.sql.execution.pythonUDTF.arrow.enabled=True")
self.assertEqual(
udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_ARROW_TABLE_UDF
)
self.spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", old_value)
try:
self.spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", False)
self.assertEqual(
udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_TABLE_UDF
)
self.spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", True)
self.assertEqual(
udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_ARROW_TABLE_UDF
)
finally:
self.spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", old_value)

def test_udtf_eval_returning_non_tuple(self):
@udtf(returnType="a: int")
Expand Down