Skip to content

Commit 0e186e8

Browse files
committed
[SPARK-37686][PYTHON][SQL] Use _invoke_function helpers for all pyspark.sql.functions
### What changes were proposed in this pull request? This PR proposes conversion of functions not covered by SPARK-32084 to `_invoke_functions` style. Two new `_invoke` functions where added: - `_invoke_function_over_columns` - `_invoke_function_over_seq_of_columns` to address common examples. ### Why are the changes needed? To reduce boilerplate (especially related to type checking) and improve manageability. Additionally, it opens opportunity for reducing driver-side invocation latency. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closes apache#34951 from zero323/SPARK-37686. Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: zero323 <mszymkiewicz@gmail.com>
1 parent 0cbc934 commit 0e186e8

File tree

1 file changed

+256
-591
lines changed

1 file changed

+256
-591
lines changed

0 commit comments

Comments
 (0)