[SPARK-3013] [SQL] [PySpark] convert array into list

[davies]

because Pyrolite does not support array from Python 2.6

[SparkQA]

QA tests have started for PR 1928. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18455/consoleFull

[SparkQA]

QA results for PR 1928:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18455/consoleFull

[marmbrus]

test this please

[davies]

jenkins, test this please

[SparkQA]

QA tests have started for PR 1928. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18463/consoleFull

[SparkQA]

QA results for PR 1928:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18463/consoleFull

[marmbrus]

Thanks! I've merge this into master and 1.1

[kanzhang]

Now that I think about it, our WritableToJavaConverter should probably convert array to java collection (which gets pickled to Python list), instead of Object[] (which gets picked to Python tuple), as it is now.

[davies]

@kanzhang I think either way is fine, tuple is like read-only list. RDD is read only, so there is no difference between having list or tuple in RDD.

[kanzhang]

@davies that's true. One thing I like about Python list is that it gets mapped to Java List (symmetry). However, it's not array any more.