Skip to content

[SPARK-20290][MINOR][PYTHON][SQL] Add PySpark wrapper for eqNullSafe #17605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zero323 wants to merge 5 commits into from
Closed

[SPARK-20290][MINOR][PYTHON][SQL] Add PySpark wrapper for eqNullSafe #17605

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6cd969f
Add PySpark wrapper for eqNullSafe
zero323 Apr 11, 2017
771cd07
Add detailed docstring
zero323 Apr 26, 2017
1e2f821
Add example with None literal
zero323 Apr 26, 2017
a658969
Clarify NaN behavior
zero323 Apr 29, 2017
99f836e
Link to NaN Semantics
zero323 May 1, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clarify NaN behavior
  • Loading branch information
@zero323
zero323 committed Apr 30, 2017
commit a658969f0acfa1e11d7dd29cf21727aef8c66657
20 changes: 19 additions & 1 deletion python/pyspark/sql/column.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,25 @@ def __init__(self, jc):
0
>>> df1.join(df2, df1["value"].eqNullSafe(df2["value"])).count()
1

>>> df2 = spark.createDataFrame([
... Row(id=1, value=float('NaN')),
... Row(id=2, value=42.0),
... Row(id=3, value=None)
... ])
>>> df2.select(
... df2['value'].eqNullSafe(None),
... df2['value'].eqNullSafe(float('NaN')),
... df2['value'].eqNullSafe(42.0)
... ).show()
+----------------+---------------+----------------+
|(value <=> NULL)|(value <=> NaN)|(value <=> 42.0)|
+----------------+---------------+----------------+
| false| true| false|
Copy link
Member

@viirya viirya May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Pandas/numpy, the nan's don’t compare equal, i.e., np.nan != np.nan, but in Spark we treat them as equal. Shall we document it too?

Copy link
Member Author

@zero323 zero323 May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is already covered by SQL guide (https://spark.apache.org/docs/latest/sql-programming-guide.html#nan-semantics). Maybe a link would be better?

Copy link
Member

@viirya viirya May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me.

| false| false| true|
| true| false| false|
+----------------+---------------+----------------+

.. note:: Unlike Pandas, PySpark doesn't consider NaN values to be NULL.
.. versionadded:: 2.3.0
"""
eqNullSafe = _bin_op("eqNullSafe", _eqNullSafe_doc)
Expand Down