Skip to content

[SPARK-746][CORE] Added Avro Serialization to Kryo #7004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JDrit wants to merge 13 commits into from
Closed

[SPARK-746][CORE] Added Avro Serialization to Kryo #7004

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
97fba62
Added a custom Kryo serializer for generic Avro records to reduce the…
Jun 23, 2015
2b545cc
started working on fixes for pr
Jun 25, 2015
f4ae251
fixed serialization error in that SparkConf cannot be serialized
Jun 26, 2015
ab46d10
Changed Avro dependency to be similar to parent
Jun 29, 2015
d421bf5
updated pom to removed versions
Jul 1, 2015
6d1925c
fixed to changes suggested by @squito
Jul 14, 2015
0f5471a
implemented @squito suggestion to use a codec that is already in spark
Jul 14, 2015
c5fe794
implemented @squito suggestion
Jul 15, 2015
fa9298b
forgot a couple of fixes
Jul 15, 2015
1183a48
updated codec settings
Jul 15, 2015
dd71efe
fixed bug with serializing
Jul 15, 2015
c0cf329
implemented @squito suggestion for SparkEnv
Jul 16, 2015
8158d51
updated per feedback
JDrit Jul 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Changed Avro dependency to be similar to parent
  • Loading branch information
Joseph Batchik committed Jun 29, 2015
commit ab46d10a22b4d9d81d93111cf8a71abe387cf415
4 changes: 2 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,13 +402,13 @@
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>${avro.version}</version>
<scope>compile</scope>
<scope>${hadoop.deps.scope}</scope>
Copy link
Contributor

@squito squito Jun 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this included already?

Copy link
Contributor

@markhamstra markhamstra Jun 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avro and avro-mapred are already in the dependencyMangement of the parent POM, so nothing more than groupId, artifactId and perhaps scope should be included here. In particular, version should not generally be specified when it is already in the dependencyManagement. This is a frequent error in the Spark POMs.

Copy link
Contributor

@vanzin vanzin Jul 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to Mark's comments.

</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>${avro.version}</version>
Copy link
Contributor

@vanzin vanzin Jul 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly to the previous one, this one should only contain groupId / artifactId (everything else is inherited).

Copy link
Contributor

@vanzin vanzin Jul 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one thing you have to add here, though, is:

  <classifier>${avro.mapred.classifier}</classifier>

Copy link
Contributor Author

@JDrit JDrit Jul 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out, just fixed that.

<scope>compile</scope>
<scope>${hadoop.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
Expand Down