Skip to content

[SPARK-42896][SQL][PYTHON] Make mapInPandas / mapInArrow support barrier mode execution #40520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WeichenXu123 wants to merge 8 commits into from
Closed

[SPARK-42896][SQL][PYTHON] Make mapInPandas / mapInArrow support barrier mode execution #40520

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4de9d07
init
WeichenXu123 Mar 22, 2023
f612bcb
update
WeichenXu123 Mar 22, 2023
e95fd95
update
WeichenXu123 Mar 22, 2023
aa7862f
update
WeichenXu123 Mar 23, 2023
9be4cad
update
WeichenXu123 Mar 24, 2023
9ebb3f2
Merge branch 'master' into barrier-udf
WeichenXu123 Mar 24, 2023
85b840b
fix test
WeichenXu123 Mar 24, 2023
66f13cb
update format
WeichenXu123 Mar 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update 
Signed-off-by: Weichen Xu <[email protected]>
  • Loading branch information
@WeichenXu123
WeichenXu123 committed Mar 22, 2023
commit f612bcb86d824e4ac13ef38ab18715ac975a22fd
4 changes: 2 additions & 2 deletions ...catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,13 +233,13 @@ object DeduplicateRelations extends Rule[LogicalPlan] {
newVersion.copyTagsFrom(oldVersion)
Seq((oldVersion, newVersion))

case oldVersion @ MapInPandas(_, output, _)
case oldVersion @ MapInPandas(_, output, _, _)
if oldVersion.outputSet.intersect(conflictingAttributes).nonEmpty =>
val newVersion = oldVersion.copy(output = output.map(_.newInstance()))
newVersion.copyTagsFrom(oldVersion)
Seq((oldVersion, newVersion))

case oldVersion @ PythonMapInArrow(_, output, _)
case oldVersion @ PythonMapInArrow(_, output, _, _)
if oldVersion.outputSet.intersect(conflictingAttributes).nonEmpty =>
val newVersion = oldVersion.copy(output = output.map(_.newInstance()))
newVersion.copyTagsFrom(oldVersion)
Expand Down