[SPARK-36617][PYTHON] Fix type hints for approxQuantile to support multi-column version
#33880
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
ok to test |
Member
zero323
reviewed
Sep 1, 2021
Member
|
Could you also add |
Contributor
Author
|
I'm not sure what the problems is. My branch is in sync with upstream/master and Github actions are enabled (see below):
|
carylee
changed the title
approxQuantile to support multi-column versionapproxQuantile to support multi-column version
Member
|
Could you check your actions tab (https://github.com/carylee/spark/actions) directly (you might see something like described in this comment)? |
Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied.
Member
|
ok to test |
zero323
approved these changes
Sep 1, 2021
Member
zero323
left a comment
zero323
left a comment
There was a problem hiding this comment.
LGTM, subject to passing Jenkins tests.
Member
|
test this please |
Contributor
Author
|
@zero323 It looks like the tests passed? https://github.com/carylee/spark/actions/runs/1191206733 |
HyukjinKwon
approved these changes
Sep 2, 2021
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
LGTM2. I will leave it to @zero323.
dongjoon-hyun
approved these changes
Sep 2, 2021
Member
|
ok to test |
Member
|
I think it's good to go and merge since GitHub Actions passed. This change shouldn't break any of tests anyway. |
zero323
pushed a commit
that referenced
this pull request
Sep 2, 2021
…multi-column version ### What changes were proposed in this pull request? Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied. ### Why are the changes needed? The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see [the approxQuantile docs](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.approxQuantile.html).) This change was also introduced to pyspark-stubs (zero323/pyspark-stubs#552). zero323 asked me to open a PR for the upstream change. ### Does this PR introduce _any_ user-facing change? This change only affects type hints - it brings the `approxQuantile` type hints up to date with the actual code. ### How was this patch tested? Ran `./dev/lint-python`. Closes #33880 from carylee/master. Authored-by: Cary Lee <[email protected]> Signed-off-by: zero323 <[email protected]> (cherry picked from commit 37f5ab0) Signed-off-by: zero323 <[email protected]>
zero323
pushed a commit
that referenced
this pull request
Sep 2, 2021
…multi-column version ### What changes were proposed in this pull request? Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied. ### Why are the changes needed? The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see [the approxQuantile docs](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.approxQuantile.html).) This change was also introduced to pyspark-stubs (zero323/pyspark-stubs#552). zero323 asked me to open a PR for the upstream change. ### Does this PR introduce _any_ user-facing change? This change only affects type hints - it brings the `approxQuantile` type hints up to date with the actual code. ### How was this patch tested? Ran `./dev/lint-python`. Closes #33880 from carylee/master. Authored-by: Cary Lee <[email protected]> Signed-off-by: zero323 <[email protected]> (cherry picked from commit 37f5ab0) Signed-off-by: zero323 <[email protected]>
Member
|
Merged to master, branch-3.2 and branch-3.1. |
Member
|
Thanks everyone! |
flyrain
pushed a commit
to flyrain/spark
that referenced
this pull request
Sep 21, 2021
…multi-column version ### What changes were proposed in this pull request? Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied. ### Why are the changes needed? The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see [the approxQuantile docs](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.approxQuantile.html).) This change was also introduced to pyspark-stubs (zero323/pyspark-stubs#552). zero323 asked me to open a PR for the upstream change. ### Does this PR introduce _any_ user-facing change? This change only affects type hints - it brings the `approxQuantile` type hints up to date with the actual code. ### How was this patch tested? Ran `./dev/lint-python`. Closes apache#33880 from carylee/master. Authored-by: Cary Lee <[email protected]> Signed-off-by: zero323 <[email protected]> (cherry picked from commit 37f5ab0) Signed-off-by: zero323 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Update both
DataFrame.approxQuantileandDataFrameStatFunctions.approxQuantileto support overloaded definitions when multiple columns are supplied.Why are the changes needed?
The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see the approxQuantile docs.) This change was also introduced to pyspark-stubs (zero323/pyspark-stubs#552). @zero323 asked me to open a PR for the upstream change.
Does this PR introduce any user-facing change?
This change only affects type hints - it brings the
approxQuantiletype hints up to date with the actual code.How was this patch tested?
Ran
./dev/lint-python.