[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN #33994

AngersZhuuuu · 2021-09-14T14:06:07Z

What changes were proposed in this pull request?

For query

select array_except(array(cast('nan' as double), 1d), array(cast('nan' as double)))

This returns [NaN, 1d], but it should return [1d].
This issue is caused by OpenHashSet can't handle Double.NaN and Float.NaN too.
In this pr fix this based on #33955

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

ArrayExcept won't show handle equal NaN value

How was this patch tested?

Added UT

….Nan

SparkQA · 2021-09-14T15:06:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47767/

SparkQA · 2021-09-14T15:16:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47767/

SparkQA · 2021-09-14T15:30:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47769/

SparkQA · 2021-09-14T15:39:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47769/

SparkQA · 2021-09-14T19:07:49Z

Test build #143264 has finished for PR 33994 at commit e9cb989.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-09-14T19:32:39Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

-              hs.add(elem)
+            if (isNaN(elem)) {
+              if (notFoundNaNElement) {
+                arrayBuffer += elem


For this, let's wait a little bit for the decision at the first PR.

https://github.com/apache/spark/pull/33955/files#r708570515

SparkQA · 2021-09-14T19:36:34Z

Test build #143266 has finished for PR 33994 at commit d73557c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-09-15T14:37:41Z

ping @cloud-fan

SparkQA · 2021-09-15T15:43:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47808/

SparkQA · 2021-09-15T15:52:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47808/

SparkQA · 2021-09-15T19:35:51Z

Test build #143306 has finished for PR 33994 at commit 061132f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-09-16T06:08:37Z

ping @cloud-fan

SparkQA · 2021-09-16T07:13:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47844/

SparkQA · 2021-09-16T07:26:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47844/

SparkQA · 2021-09-16T11:00:37Z

Test build #143338 has finished for PR 33994 at commit 662bf05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-17T05:45:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47897/

SparkQA · 2021-09-17T05:54:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47897/

SparkQA · 2021-09-17T06:52:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47905/

SparkQA · 2021-09-17T07:01:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47905/

SparkQA · 2021-09-17T13:44:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47920/

SparkQA · 2021-09-17T13:54:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47920/

SparkQA · 2021-09-17T15:49:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47923/

SparkQA · 2021-09-17T16:00:35Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47923/

SparkQA · 2021-09-17T17:34:46Z

Test build #143411 has finished for PR 33994 at commit 82640f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class WriterBucketSpec(

SparkQA · 2021-09-17T18:02:44Z

Test build #143413 has finished for PR 33994 at commit 77b1661.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-17T19:43:05Z

Test build #143416 has finished for PR 33994 at commit cd3641d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

SparkQA · 2021-09-20T12:06:13Z

Test build #143452 has finished for PR 33994 at commit 797089d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-20T12:09:30Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47961/

SparkQA · 2021-09-20T14:22:28Z

Test build #143455 has started for PR 33994 at commit 6eaa96a.

SparkQA · 2021-09-20T15:11:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47964/

SparkQA · 2021-09-20T15:20:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47964/

AngersZhuuuu · 2021-09-21T01:09:00Z

retest this please

SparkQA · 2021-09-21T02:01:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47980/

SparkQA · 2021-09-21T02:44:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47980/

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

SparkQA · 2021-09-21T05:52:17Z

Test build #143469 has finished for PR 33994 at commit 6eaa96a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-09-21T13:24:22Z

ping @cloud-fan

cloud-fan · 2021-09-22T15:51:40Z

thanks, merging to master/3.2/3.1/3.0!

….NaN ### What changes were proposed in this pull request? For query ``` select array_except(array(cast('nan' as double), 1d), array(cast('nan' as double))) ``` This returns [NaN, 1d], but it should return [1d]. This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too. In this pr fix this based on #33955 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? ArrayExcept won't show handle equal `NaN` value ### How was this patch tested? Added UT Closes #33994 from AngersZhuuuu/SPARK-36753. Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a7cbe69) Signed-off-by: Wenchen Fan <[email protected]>

dongjoon-hyun · 2021-09-22T19:39:31Z

Thank you, @AngersZhuuuu and @cloud-fan .

….NaN ### What changes were proposed in this pull request? For query ``` select array_except(array(cast('nan' as double), 1d), array(cast('nan' as double))) ``` This returns [NaN, 1d], but it should return [1d]. This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too. In this pr fix this based on apache#33955 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? ArrayExcept won't show handle equal `NaN` value ### How was this patch tested? Added UT Closes apache#33994 from AngersZhuuuu/SPARK-36753. Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a7cbe69) Signed-off-by: Wenchen Fan <[email protected]>

[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float…

e9cb989

….Nan

github-actions bot added the SQL label Sep 14, 2021

update

d73557c

dongjoon-hyun reviewed Sep 14, 2021

View reviewed changes

AngersZhuuuu added 3 commits September 15, 2021 22:27

Merge branch 'master' into SPARK-36753

2b607fa

Update collectionOperations.scala

bde7927

Merge branch 'master' into SPARK-36753

061132f

trigger

662bf05

AngersZhuuuu added 4 commits September 17, 2021 12:35

update

08f28c0

Update collectionOperations.scala

394df1d

Update collectionOperations.scala

5f4d1bb

Update collectionOperations.scala

3ef22d7

Update collectionOperations.scala

cd3641d

cloud-fan reviewed Sep 20, 2021

View reviewed changes

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated Show resolved Hide resolved

AngersZhuuuu added 2 commits September 20, 2021 19:47

Merge branch 'master' into SPARK-36753

d319d65

Update collectionOperations.scala

797089d

Update collectionOperations.scala

6eaa96a

dongjoon-hyun reviewed Sep 21, 2021

View reviewed changes

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Show resolved Hide resolved

cloud-fan closed this in a7cbe69 Sep 22, 2021

[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN #33994

[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN #33994

Uh oh!

Conversation

AngersZhuuuu commented Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

dongjoon-hyun Sep 14, 2021

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Sep 15, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

AngersZhuuuu commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

AngersZhuuuu commented Sep 16, 2021

Uh oh!

SparkQA commented Sep 16, 2021

Uh oh!

SparkQA commented Sep 16, 2021

Uh oh!

SparkQA commented Sep 16, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

Uh oh!

SparkQA commented Sep 20, 2021

Uh oh!

SparkQA commented Sep 20, 2021

Uh oh!

SparkQA commented Sep 20, 2021

Uh oh!

SparkQA commented Sep 20, 2021

Uh oh!

SparkQA commented Sep 20, 2021

Uh oh!

AngersZhuuuu commented Sep 21, 2021

Uh oh!

AngersZhuuuu commented Sep 14, 2021 •

edited

Loading