[SPARK-16613] [CORE] RDD.pipe returns values for empty partitions

[srowen]

What changes were proposed in this pull request?

Document RDD.pipe semantics; don't execute process for empty input partitions.

Note this includes the fix in #14256 because it's necessary to even test this. One or the other will merge the fix.

How was this patch tested?

Jenkins tests including new test.

[SparkQA]

Test build #62521 has finished for PR 14260 at commit 62692af.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Is it possible that the underlying command always return something even for 0 rows? e.g. if it is counting the number of elements?

[srowen]

Yeah that's the 'problem' -- consider wc -l which returns 0 for no input at all. That explains the behavior now, where an empty partition results in a partition of one element: "0". That seems a bit odd. It's not that odd because pipe is actually quite partition-oriented. I don't feel strongly about changing the behavior of an empty partition though, if anyone thinks it actual has the right semantics. But in any event the current behavior could be documented better.

[SparkQA]

Test build #62595 has finished for PR 14260 at commit 4f9d4d8.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Jenkins retest this please

[SparkQA]

Test build #62601 has finished for PR 14260 at commit 4f9d4d8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

LGTM

[rxin]

Merging in master/2.0.