Skip to content

[SPARK-7314][SPARK-3524][PySpark] upgrade Pyrolite to 4.4 #5850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mengxr wants to merge 7 commits into from
Closed

[SPARK-7314][SPARK-3524][PySpark] upgrade Pyrolite to 4.4 #5850

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc3903a
upgrade Pyrolite to 4.4-0 for testing
mengxr May 1, 2015
7824a9c
use Pyrolite 3.1
mengxr May 3, 2015
d2d5b5b
change back to 4.4
mengxr May 4, 2015
6ddac0e
reverse the machine code for float/double
mengxr May 4, 2015
fe7e29b
switch to maven central
mengxr May 4, 2015
da3c2dd
remove my repo
mengxr May 4, 2015
2ed4a95
Merge remote-tracking branch 'apache/master' into SPARK-7314
mengxr May 5, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@
<dependency>
<groupId>org.spark-project</groupId>
<artifactId>pyrolite</artifactId>
<version>2.0.1</version>
<version>4.4</version>
</dependency>
<dependency>
<groupId>net.sf.py4j</groupId>
Expand Down
7 changes: 2 additions & 5 deletions core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,13 @@ private[spark] object SerDeUtil extends Logging {
// {'\0', 0, 0, 0} /* Sentinel */
// };
// TODO: support Py_UNICODE with 2 bytes
// FIXME: unpickle array of float is wrong in Pyrolite, so we reverse the
// machine code for float/double here to workaround it.
// we should fix this after Pyrolite fix them
val machineCodes: Map[Char, Int] = if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)) {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 3, 'h' -> 5, 'I' -> 7, 'i' -> 9,
'L' -> 11, 'l' -> 13, 'f' -> 14, 'd' -> 16, 'u' -> 21
'L' -> 11, 'l' -> 13, 'f' -> 15, 'd' -> 17, 'u' -> 21
)
} else {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 2, 'h' -> 4, 'I' -> 6, 'i' -> 8,
'L' -> 10, 'l' -> 12, 'f' -> 15, 'd' -> 17, 'u' -> 20
'L' -> 10, 'l' -> 12, 'f' -> 14, 'd' -> 16, 'u' -> 20
)
}
override def construct(args: Array[Object]): Object = {
Expand Down