Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
SPARK-37981 Add note for deleting Null and NaN
### What changes were proposed in this pull request?

This is for SPARK-37981Deletes columns with all Null as default.
Do also see #26098
User HyukjinKwon did a reviewed on 21 Oct 2019 
"Hey, you should document this in DataFrameWrtier, DataStreamWrtier, readwriter.py"

### Why are the changes needed?
Users need to know why there column(s) with all NaN or Null are gone. 

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
No.
  • Loading branch information
bjornjorgensen authored Jan 23, 2022
commit 196d14fdce1cbea059a7d3e734d1d87c4c6db5d5
2 changes: 2 additions & 0 deletions python/pyspark/pandas/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,8 @@ def to_json(

.. note:: output JSON format is different from pandas'. It always use `orient='records'`
for its output. This behaviour might have to change in the near future.

.. note:: If column have only NaN or Null values. The column well be deleted.

Note NaN's and None will be converted to null and datetime objects
will be converted to UNIX timestamps.
Expand Down