[SPARK-24079][SQL] Update the nullability of Join output based on inferred predicates

[maropu]

What changes were proposed in this pull request?

This pr added code to update the nullability of Join.output based on inferred predicates in InferFiltersFromConstraints. In the master, a logical Join node does not respect the nullability that the optimizer rule InferFiltersFromConstraints might change when inferred predicates have IsNotNull,
e.g.,

scala> val df1 = Seq((Some(1), Some(2))).toDF("k", "v0")
scala> val df2 = Seq((Some(1), Some(3))).toDF("k", "v1")
scala> val joinedDf = df1.join(df2, df1("k") === df2("k"), "inner")
scala> joinedDf.explain
== Physical Plan ==
*(2) BroadcastHashJoin [k#83], [k#92], Inner, BuildRight
:- *(2) Project [_1#80 AS k#83, _2#81 AS v0#84]
:  +- *(2) Filter isnotnull(_1#80)
:     +- LocalTableScan [_1#80, _2#81]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)))
   +- *(1) Project [_1#89 AS k#92, _2#90 AS v1#93]
      +- *(1) Filter isnotnull(_1#89)
         +- LocalTableScan [_1#89, _2#90]

scala> joinedDf.queryExecution.optimizedPlan.output.map(_.nullable)
res15: Seq[Boolean] = List(true, true, true, true)

But, these nullable values should be:

scala> joinedDf.queryExecution.optimizedPlan.output.map(_.nullable)
res15: Seq[Boolean] = List(false, true, false, true)

This ticket comes from the previous discussion: #18576 (review)

How was this patch tested?

Added tests in UpdateNullabilityInAttributeReferencesSuite.

[SparkQA]

Test build #89818 has finished for PR 21148 at commit 240ae7a.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #89833 has finished for PR 21148 at commit 240ae7a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #97705 has finished for PR 21148 at commit 240ae7a.

• This patch fails due to an unknown error code, -9.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #97737 has finished for PR 21148 at commit 240ae7a.

• This patch fails due to an unknown error code, -9.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #97773 has finished for PR 21148 at commit 240ae7a.

• This patch fails build dependency tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #105091 has finished for PR 21148 at commit 240ae7a.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!