[SPARK-4113] [SQL] [PySpark] type conversionfor python udf

[davies]

Call Python UDF on ArrayType/MapType/PrimitiveType, the returnType can also be ArrayType/MapType/PrimitiveType.

For StructType, it will act as tuple (without attributes). If returnType is StructType, it also should be tuple.

[SparkQA]

Test build #22341 has started for PR 2973 at commit 35caa4f.

• This patch merges cleanly.

[SparkQA]

Test build #22341 has finished for PR 2973 at commit 35caa4f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22341/
Test FAILed.

[SparkQA]

Test build #485 has started for PR 2973 at commit 35caa4f.

• This patch merges cleanly.

[SparkQA]

Test build #22352 has started for PR 2973 at commit 79afb4e.

• This patch merges cleanly.

[SparkQA]

Test build #485 has finished for PR 2973 at commit 35caa4f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #22352 has finished for PR 2973 at commit 79afb4e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22352/
Test PASSed.

[marmbrus]

Nit/existing: I'd avoid creating attributes which is actually pretty expensive due to an AtomicLong and use the public API for schema. I think instead of these two lines you can just do the following here and above.

val fieldTypes = schema.fields.map(_.dataType)

[marmbrus]

Thanks for the quick fix! A few minor comments.

[SparkQA]

Test build #22377 has started for PR 2973 at commit 11395fa.

• This patch merges cleanly.

[SparkQA]

Test build #22378 has started for PR 2973 at commit 2c00e43.

• This patch merges cleanly.

[SparkQA]

Test build #489 has started for PR 2973 at commit 2c00e43.

• This patch does not merge cleanly.

[SparkQA]

Test build #22379 has started for PR 2973 at commit 306956e.

• This patch merges cleanly.

[SparkQA]

Test build #22377 has finished for PR 2973 at commit 11395fa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DateType(PrimitiveType):
  • case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseNot(child: Expression) extends UnaryExpression

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22377/
Test PASSed.

[SparkQA]

Test build #22378 has finished for PR 2973 at commit 2c00e43.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):
  • class DateType(PrimitiveType):
  • case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseNot(child: Expression) extends UnaryExpression

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22378/
Test PASSed.

[SparkQA]

Test build #22379 has finished for PR 2973 at commit 306956e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DateType(PrimitiveType):
  • case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseNot(child: Expression) extends UnaryExpression
  • case class CreateTableAsSelect[T](
  • logDebug("Found class for $serdeName")

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22379/
Test PASSed.

[SparkQA]

Test build #489 has finished for PR 2973 at commit 2c00e43.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):
  • abstract class Broadcast[T: ClassTag](val id: Long) extends Serializable with Logging
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):
  • class DateType(PrimitiveType):
  • case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic
  • case class BitwiseNot(child: Expression) extends UnaryExpression

[marmbrus]

Thanks! Merged to master.

[marmbrus]

FYI this is actually: SPARK-4113

[davies]

thanks, update the link