[SPARK-21272] SortMergeJoin LeftAnti does not update numOutputRows

[juliuszsompolski]

What changes were proposed in this pull request?

Updating numOutputRows metric was missing from one return path of LeftAnti SortMergeJoin.

How was this patch tested?

Non-zero output rows manually seen in metrics.

[kiszk]

Would it be possible to add a test suite into SQLMetricsSuite?

[rxin]

cc @hvanhovell

[viirya]

Yeah, it's better to add a test to SQLMetricsSuite for this.

[hvanhovell]

ok to test

[SparkQA]

Test build #79047 has finished for PR 18494 at commit 580dc46.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

add to whitelist

[SparkQA]

Test build #79069 has finished for PR 18494 at commit 580dc46.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Unfortunately, the comments in the test cases of SQLMetricsSuite are out-of-dated. Anyway, could you add a test case to SQLMetricsSuite for verifying the logics. Below is an example.

  test("SortMergeJoin(left-anti) metrics when right table is empty") {
    val testDataForJoin = testData2.filter("a < 0") // Empty but not optimized to EmptyRDD
    withTempView("testDataForJoin") {
      testDataForJoin.createOrReplaceTempView("testDataForJoin")
      val df = spark.sql(
        "SELECT * FROM testData2 ANTI JOIN testDataForJoin ON testData2.a = testDataForJoin.a")
      testSparkPlanMetrics(df, 1, Map(
        0L -> ("SortMergeJoin", Map("number of output rows" -> 6L)))
      )
    }
  }

[SparkQA]

Test build #79467 has finished for PR 18494 at commit 93f7b88.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

[gatorsmile]

Thanks! Merging to master/2.2.