Skip to content

[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN and Float.NaN #34003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
AngersZhuuuu wants to merge 4 commits into from
Closed

[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN and Float.NaN #34003

AngersZhuuuu wants to merge 4 commits into from

Conversation

@AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented Sep 15, 2021

What changes were proposed in this pull request?

According to #33955 (comment) use normalized NaN

Why are the changes needed?

Use normalized NaN for duplicated NaN value

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exiting UT

@github-actions github-actions bot added the SQL label Sep 15, 2021
@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Sep 15, 2021

ping @dongjoon-hyun @cloud-fan

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47797/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47797/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143296 has finished for PR 34003 at commit 13deceb.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47799/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47800/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47801/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47800/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47801/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143294 has finished for PR 34003 at commit 1c7ad8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Sep 15, 2021

ping @cloud-fan

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143297 has finished for PR 34003 at commit e467ff3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143298 has finished for PR 34003 at commit 73e3145.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented Sep 15, 2021

thanks, merging to master/3.2/3.1!

@cloud-fan cloud-fan closed this in 6380859 Sep 15, 2021
cloud-fan pushed a commit that referenced this pull request Sep 15, 2021
@AngersZhuuuu @cloud-fan
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
e641556 
…and Float.NaN

### What changes were proposed in this pull request?
According to #33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes #34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <[email protected]>
cloud-fan pushed a commit that referenced this pull request Sep 15, 2021
@AngersZhuuuu @cloud-fan
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
4a266f0 
…and Float.NaN

### What changes were proposed in this pull request?
According to #33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes #34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <[email protected]>
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Sep 15, 2021

+1, LGTM. Thank you!

fishcus pushed a commit to fishcus/spark that referenced this pull request Jan 12, 2022
@AngersZhuuuu @fishcus
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
29b38d3 
…and Float.NaN

### What changes were proposed in this pull request?
According to apache#33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes apache#34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AngersZhuuuu @SparkQA @cloud-fan @dongjoon-hyun