Skip to content

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 11 commits into from
Closed

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b84222b
[SPARK-21070][PYSPARK] Upgrade cloudpickle
rgbkrk Jun 12, 2017
6d5e5cf
Merge branch 'master' into holden-rgbkrk-cloudpickle-upgrades
holdenk Jul 24, 2017
1cfd38f
Copy over support work weakset, dynamic classess, and remove __transi…
holdenk Jul 24, 2017
f8ff2da
Test fix
holdenk Jul 25, 2017
cff6bfb
Revert "Copy over support work weakset, dynamic classess, and remove …
holdenk Jul 25, 2017
195cd21
Fixed named tuple issue
holdenk Jul 25, 2017
74880ea
Try and move the fix for namedtuple and re-enable the rest of the use…
holdenk Jul 25, 2017
9a0f9b4
Re-enable our custom exception message
holdenk Jul 25, 2017
09cf41e
Save and restore the module info functions
holdenk Jul 25, 2017
7ad278a
six importing work around restore
holdenk Aug 18, 2017
f986c25
Restore the six work around and update the hasattry check for package.
holdenk Aug 18, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Re-enable our custom exception message
  • Loading branch information
@holdenk
holdenk committed Jul 25, 2017
commit 9a0f9b4b9958c5fed6d2c84d725cb03a7be7d41e
12 changes: 12 additions & 0 deletions python/pyspark/cloudpickle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
import types
import weakref

from pyspark.util import _exception_message

if sys.version < '3':
from pickle import Pickler
Expand Down Expand Up @@ -236,6 +237,17 @@ def dump(self, obj):
if 'recursion' in e.args[0]:
msg = """Could not pickle object as excessively deep recursion required."""
raise pickle.PicklingError(msg)
except pickle.PickleError:
raise
except Exception as e:
emsg = _exception_message(e)
if "'i' format requires" in emsg:
msg = "Object too large to serialize: %s" % emsg
else:
msg = "Could not serialize object: %s: %s" % (e.__class__.__name__, emsg)
print_exec(sys.stderr)
raise pickle.PicklingError(msg)


def save_memoryview(self, obj):
"""Fallback to save_string"""
Expand Down