Skip to content

[SPARK-7314][SPARK-3524][PySpark] upgrade Pyrolite to 4.4 #5850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mengxr wants to merge 7 commits into from
Closed

[SPARK-7314][SPARK-3524][PySpark] upgrade Pyrolite to 4.4 #5850

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc3903a
upgrade Pyrolite to 4.4-0 for testing
mengxr May 1, 2015
7824a9c
use Pyrolite 3.1
mengxr May 3, 2015
d2d5b5b
change back to 4.4
mengxr May 4, 2015
6ddac0e
reverse the machine code for float/double
mengxr May 4, 2015
fe7e29b
switch to maven central
mengxr May 4, 2015
da3c2dd
remove my repo
mengxr May 4, 2015
2ed4a95
Merge remote-tracking branch 'apache/master' into SPARK-7314
mengxr May 5, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reverse the machine code for float/double
  • Loading branch information
@mengxr
mengxr committed May 4, 2015
commit 6ddac0ec05d4eeb78f90c5669c377f14e3f3809a
7 changes: 2 additions & 5 deletions core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,13 @@ private[spark] object SerDeUtil extends Logging {
// {'\0', 0, 0, 0} /* Sentinel */
// };
// TODO: support Py_UNICODE with 2 bytes
// FIXME: unpickle array of float is wrong in Pyrolite, so we reverse the
// machine code for float/double here to workaround it.
// we should fix this after Pyrolite fix them
val machineCodes: Map[Char, Int] = if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)) {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 3, 'h' -> 5, 'I' -> 7, 'i' -> 9,
'L' -> 11, 'l' -> 13, 'f' -> 14, 'd' -> 16, 'u' -> 21
'L' -> 11, 'l' -> 13, 'f' -> 15, 'd' -> 17, 'u' -> 21
)
} else {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 2, 'h' -> 4, 'I' -> 6, 'i' -> 8,
'L' -> 10, 'l' -> 12, 'f' -> 15, 'd' -> 17, 'u' -> 20
'L' -> 10, 'l' -> 12, 'f' -> 14, 'd' -> 16, 'u' -> 20
)
}
override def construct(args: Array[Object]): Object = {
Expand Down