Skip to content

[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name #21427

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanCutler wants to merge 15 commits into from
Closed

[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name #21427

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
21e0c3e
added test for diff column order
BryanCutler May 24, 2018
bbe3587
needed to adjust expected values to compare results
BryanCutler May 24, 2018
a653e9b
for grouped map results, get columns based on name instead of position
BryanCutler May 24, 2018
88e2aa3
added test for positional columns
BryanCutler May 25, 2018
7cc0c49
adjust positional test
BryanCutler May 25, 2018
d4b5da1
add fallback to positional assignment
BryanCutler May 25, 2018
63c3963
added test for col name typo
BryanCutler May 25, 2018
9bbf014
fix test for py3
BryanCutler May 25, 2018
74c5d8e
added tests for integer index, incorrect field type with range index
BryanCutler May 29, 2018
b2d0966
documented behavior of column labels
BryanCutler May 29, 2018
5a7edb2
passing conf map to runner, tests pass
BryanCutler Jun 18, 2018
59972d6
added conf for assignment by position, passing general map of conf to…
BryanCutler Jun 18, 2018
27b4cad
fix some typos
BryanCutler Jun 18, 2018
c593650
address nits
BryanCutler Jun 22, 2018
2d2ced6
changed from catching KeyError to explicit check for string labels
BryanCutler Jun 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix some typos
  • Loading branch information
@BryanCutler
BryanCutler committed Jun 18, 2018
commit 27b4cad4ad1ff6aea77951878132db5fff80a907
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1166,7 +1166,7 @@ object SQLConf {
.internal()
.doc("When true, a grouped map Pandas UDF will assign columns from the returned " +
"Pandas DataFrame based on position, regardless of column label type. When false, " +
"columns will be looked up by name if labeled with a string and fallback to use" +
"columns will be looked up by name if labeled with a string and fallback to use " +
"position if not.")
.booleanConf
.createWithDefault(false)
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ class ArrowPythonRunner(
}

protected override def writeIteratorToStream(dataOut: DataOutputStream): Unit = {
val arrowSchema = ArrowUtils.toArrowSchema(schema, timeZoneId)
val allocator = ArrowUtils.rootAllocator.newChildAllocator(
s"stdout writer for $pythonExec", 0, Long.MaxValue)
val arrowSchema = ArrowUtils.toArrowSchema(schema, timeZoneId)
val root = VectorSchemaRoot.create(arrowSchema, allocator)

Utils.tryWithSafeFinally {
Expand Down