[SPARK-10073] [SQL] Python withColumn should replace the old column

[davies]

DataFrame.withColumn in Python should be consistent with the Scala one (replacing the existing column that has the same name).

cc @marmbrus

[SparkQA]

Test build #41205 has finished for PR 8300 at commit 9eb857b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

This is not consistent with scala version, see the doc, we should not report error here.

Actually I'm wondering why we need to do checking at python side(not only this one)? Can we just call the scala API and catch the java exception?

[davies]

Good catch, will remove it. I'm really surprised by the behavior in Scala.

The reason we want to have some check on Python side is that the Java exception is not easy to understand for Python programmer (nested inside a Py4j exception). The Python exception or messages does improve the experience for Python programmer, especially beginners.

[marmbrus]

Forgive my lack of python skills... could have have a generic wrapper that we use whenever we call back into scala that catches and unwraps certain expressions nicely?

[davies]

We already did this for some of them, for example, AnalysisException and IllegalArgumentException, these could help.

[marmbrus]

Oh cool, ideally we will pretty much only throw those two unless there is a bug. If there are cases where that is not true we should consider fixing on the scala side.

[SparkQA]

Test build #41260 has finished for PR 8300 at commit 68ff0a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks! Merging to master and 1.5.