Skip to content

[SPARK-11810][SQL] Java-based encoder for opaque types in Datasets. #9802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rxin wants to merge 3 commits into from
Closed

[SPARK-11810][SQL] Java-based encoder for opaque types in Datasets. #9802

rxin wants to merge 3 commits into from

Conversation

@rxin
Copy link
Contributor

@rxin rxin commented Nov 18, 2015

This patch refactors the existing Kryo encoder expressions and adds support for Java serialization.

rxin
rxin reviewed Nov 18, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/Encoder.scala Outdated
Copy link
Contributor Author

@rxin rxin Nov 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @marmbrus for the name here ...

Copy link
Contributor

@marmbrus marmbrus Nov 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

javaSerializer javaSerialization? generic doesn't make it clear to me what this is doing.

@SparkQA
Copy link

SparkQA commented Nov 18, 2015

Test build #2086 has finished for PR 9802 at commit 0a85d0c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public class JavaGradientBoostedTreeClassifierExample\n * public class JavaGradientBoostedTreeRegressorExample\n * public class JavaRandomForestClassifierExample\n * public class JavaRandomForestRegressorExample\n * case class EncodeUsingSerializer(child: Expression, kryo: Boolean) extends UnaryExpression\n * case class DecodeUsingSerializer[T](child: Expression, tag: ClassTag[T], kryo: Boolean)\n

@rxin
Copy link
Contributor Author

rxin commented Nov 18, 2015

Alright I'm going to merge this. Thanks for looking at it.

asfgit pushed a commit that referenced this pull request Nov 18, 2015
@rxin
[SPARK-11810][SQL] Java-based encoder for opaque types in Datasets.
d9945bc 
This patch refactors the existing Kryo encoder expressions and adds support for Java serialization.

Author: Reynold Xin <[email protected]>

Closes #9802 from rxin/SPARK-11810.

(cherry picked from commit 5df0894)
Signed-off-by: Reynold Xin <[email protected]>
@asfgit asfgit closed this in 5df0894 Nov 18, 2015
@SparkQA
Copy link

SparkQA commented Nov 19, 2015

Test build #46247 has finished for PR 9802 at commit 19e34be.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rxin @SparkQA @marmbrus @cloud-fan