Commit 0df4c01
committed
[SPARK-43502][PYTHON][CONNECT] DataFrame.drop` should accept empty column
### What changes were proposed in this pull request?
Make `DataFrame.drop` accept empty column
### Why are the changes needed?
to be consistent with vanilla PySpark
### Does this PR introduce _any_ user-facing change?
yes
```
In [1]: df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
In [2]: df.drop()
```
before:
```
In [2]: df.drop()
---------------------------------------------------------------------------
PySparkValueError Traceback (most recent call last)
Cell In[2], line 1
----> 1 df.drop()
File ~/Dev/spark/python/pyspark/sql/connect/dataframe.py:449, in DataFrame.drop(self, *cols)
444 raise PySparkTypeError(
445 error_class="NOT_COLUMN_OR_STR",
446 message_parameters={"arg_name": "cols", "arg_type": type(cols).__name__},
447 )
448 if len(_cols) == 0:
--> 449 raise PySparkValueError(
450 error_class="CANNOT_BE_EMPTY",
451 message_parameters={"item": "cols"},
452 )
454 return DataFrame.withPlan(
455 plan.Drop(
456 child=self._plan,
(...)
459 session=self._session,
460 )
PySparkValueError: [CANNOT_BE_EMPTY] At least one cols must be specified.
```
after
```
In [2]: df.drop()
Out[2]: DataFrame[id: bigint, age: bigint]
```
### How was this patch tested?
enabled UT
Closes #41180 from zhengruifeng/connect_drop_empty_col.
Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>1 parent 4bf979c commit 0df4c01
File tree
3 files changed
+2
-11
lines changed- python/pyspark/sql
- connect
- tests/connect
3 files changed
+2
-11
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
445 | 445 | | |
446 | 446 | | |
447 | 447 | | |
448 | | - | |
449 | | - | |
450 | | - | |
451 | | - | |
452 | | - | |
453 | 448 | | |
454 | 449 | | |
455 | 450 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
664 | 664 | | |
665 | 665 | | |
666 | 666 | | |
667 | | - | |
| 667 | + | |
| 668 | + | |
668 | 669 | | |
669 | 670 | | |
670 | 671 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | 87 | | |
93 | 88 | | |
94 | 89 | | |
| |||
0 commit comments