[SPARK-36702][SQL] ArrayUnion handle duplicated Double.NaN and Float.Nan

[AngersZhuuuu]

What changes were proposed in this pull request?

For query

select array_union(array(cast('nan' as double), cast('nan' as double)), array())

This returns [NaN, NaN], but it should return [NaN].
This issue is caused by OpenHashSet can't handle Double.NaN and Float.NaN too.
In this pr we add a wrap for OpenHashSet that can handle null, Double.NaN, Float.NaN together

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

ArrayUnion won't show duplicated NaN value

How was this patch tested?

Added UT

[AngersZhuuuu]

ping @cloud-fan

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47646/

[cloud-fan]

The data added to this set will always be the same data type. I think we can just have a single containsNaN flag.

[AngersZhuuuu]

The data added to this set will always be the same data type. I think we can just have a single containsNaN flag.

I have thought about this too, but since it can support any type, so keep this may be better?

[cloud-fan]

Maybe we should do the null/nan check at the caller side

class SQLOpenHashSet ... {
  def add(k: T)
  def addNull()
  def addNaN()
}

// caller side
if (row.isNullAt...) {
  set.addNull()
} else {
  ...
  if (java.lang.Double.isNaN(value)) {
    set.addNaN()
  }
}

[AngersZhuuuu]

How about current

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47656/

[SparkQA]

Test build #143142 has finished for PR 33955 at commit 8579c97.

• This patch fails from timeout after a configured wait of 500m.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](

[SparkQA]

Test build #143152 has finished for PR 33955 at commit 1857988.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47669/

[SparkQA]

Test build #143165 has finished for PR 33955 at commit 4e533fd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AngersZhuuuu]

ping @cloud-fan

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47685/

[SparkQA]

Test build #143182 has finished for PR 33955 at commit 119679c.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #143186 has finished for PR 33955 at commit 45d1fee.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47688/

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47690/

[HyukjinKwon]

Thanks guys.This is possibly flaky. I'll keep my eyes on the build.

[HyukjinKwon]

I still see this test failure, see https://github.com/apache/spark/runs/3628995384. Shall we revert this PR?

[HyukjinKwon]

Actually there are some more: https://github.com/apache/spark/runs/3619357249

[cloud-fan]

This is so weird. There is no randomness in the test. How frequently do we see the test failure?