Skip to content

[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name #21427

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanCutler wants to merge 15 commits into from
Closed

[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name #21427

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
21e0c3e
added test for diff column order
BryanCutler May 24, 2018
bbe3587
needed to adjust expected values to compare results
BryanCutler May 24, 2018
a653e9b
for grouped map results, get columns based on name instead of position
BryanCutler May 24, 2018
88e2aa3
added test for positional columns
BryanCutler May 25, 2018
7cc0c49
adjust positional test
BryanCutler May 25, 2018
d4b5da1
add fallback to positional assignment
BryanCutler May 25, 2018
63c3963
added test for col name typo
BryanCutler May 25, 2018
9bbf014
fix test for py3
BryanCutler May 25, 2018
74c5d8e
added tests for integer index, incorrect field type with range index
BryanCutler May 29, 2018
b2d0966
documented behavior of column labels
BryanCutler May 29, 2018
5a7edb2
passing conf map to runner, tests pass
BryanCutler Jun 18, 2018
59972d6
added conf for assignment by position, passing general map of conf to…
BryanCutler Jun 18, 2018
27b4cad
fix some typos
BryanCutler Jun 18, 2018
c593650
address nits
BryanCutler Jun 22, 2018
2d2ced6
changed from catching KeyError to explicit check for string labels
BryanCutler Jun 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
for grouped map results, get columns based on name instead of position
  • Loading branch information
@BryanCutler
BryanCutler committed Jun 14, 2018
commit a653e9b5606c235746eeca7ed451fdbf1a90503d
4 changes: 1 addition & 3 deletions python/pyspark/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,7 @@ def wrapped(key_series, value_series):
"Number of columns of the returned pandas.DataFrame "
"doesn't match specified schema. "
"Expected: {} Actual: {}".format(len(return_type), len(result.columns)))
arrow_return_types = (to_arrow_type(field.dataType) for field in return_type)
return [(result[result.columns[i]], arrow_type)
for i, arrow_type in enumerate(arrow_return_types)]
return [(result[field.name], to_arrow_type(field.dataType)) for field in return_type]

return wrapped

Expand Down