Commit 0e186e8
committed
[SPARK-37686][PYTHON][SQL] Use _invoke_function helpers for all pyspark.sql.functions
### What changes were proposed in this pull request?
This PR proposes conversion of functions not covered by SPARK-32084 to `_invoke_functions` style.
Two new `_invoke` functions where added:
- `_invoke_function_over_columns`
- `_invoke_function_over_seq_of_columns`
to address common examples.
### Why are the changes needed?
To reduce boilerplate (especially related to type checking) and improve manageability.
Additionally, it opens opportunity for reducing driver-side invocation latency.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes apache#34951 from zero323/SPARK-37686.
Authored-by: zero323 <mszymkiewicz@gmail.com>
Signed-off-by: zero323 <mszymkiewicz@gmail.com>1 parent 0cbc934 commit 0e186e8
1 file changed
+256
-591
lines changed
0 commit comments