Skip to content

[SPARK-24721][SQL] Extract Python UDFs at the end of optimizer #22104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
icexelloss wants to merge 14 commits into from
Closed

[SPARK-24721][SQL] Extract Python UDFs at the end of optimizer #22104

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
27fd57d
Fix file strategy to exclude python UDF filters
icexelloss Aug 14, 2018
499acde
Remove white space
icexelloss Aug 14, 2018
595399e
Fix style
icexelloss Aug 14, 2018
470d63a
Fix python style
icexelloss Aug 14, 2018
cfd568e
Add test for data source and data source v2; Fix projection in EvalPy…
icexelloss Aug 15, 2018
24eebdc
Fix style
icexelloss Aug 15, 2018
96d2570
Split tests; Fix EvalPythonExec
icexelloss Aug 15, 2018
195ba5b
Fix python style
icexelloss Aug 16, 2018
f64854e
wip
icexelloss Aug 23, 2018
4e63704
Disable data source v2 test for now
icexelloss Aug 23, 2018
ca0195e
Add require_test_compiled
icexelloss Aug 23, 2018
7076820
Revert small changes
icexelloss Aug 23, 2018
d440cbf
Move ExtractPythonUDFs to end of optimize stage
icexelloss Aug 27, 2018
2325a4f
Add more tests; Fix lint; Remove hardcoded scala-2.11 from require_te…
icexelloss Aug 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix style
  • Loading branch information
@icexelloss
icexelloss committed Aug 27, 2018
commit 595399eca621d410ae3b0f0de83f94934aecdedb
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ object FileSourceStrategy extends Strategy with Logging {
val filterSet = ExpressionSet(filters)

// SPARK-24721: Filter out Python UDFs, otherwise ExtractPythonUDF rule will throw exception
val validFilters = filters.filter(_.collectFirst{case e: PythonUDF => e}.isEmpty)
val validFilters = filters.filter(_.collectFirst{ case e: PythonUDF => e }.isEmpty)

// The attribute name of predicate could be different than the one in schema in case of
// case insensitive, we should change them to match the one in schema, so we do not need to
Expand Down