Skip to content

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 11 commits into from
Closed

[SPARK-21070][PYSPARK] Attempt to update cloudpickle again #18734

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b84222b
[SPARK-21070][PYSPARK] Upgrade cloudpickle
rgbkrk Jun 12, 2017
6d5e5cf
Merge branch 'master' into holden-rgbkrk-cloudpickle-upgrades
holdenk Jul 24, 2017
1cfd38f
Copy over support work weakset, dynamic classess, and remove __transi…
holdenk Jul 24, 2017
f8ff2da
Test fix
holdenk Jul 25, 2017
cff6bfb
Revert "Copy over support work weakset, dynamic classess, and remove …
holdenk Jul 25, 2017
195cd21
Fixed named tuple issue
holdenk Jul 25, 2017
74880ea
Try and move the fix for namedtuple and re-enable the rest of the use…
holdenk Jul 25, 2017
9a0f9b4
Re-enable our custom exception message
holdenk Jul 25, 2017
09cf41e
Save and restore the module info functions
holdenk Jul 25, 2017
7ad278a
six importing work around restore
holdenk Aug 18, 2017
f986c25
Restore the six work around and update the hasattry check for package.
holdenk Aug 18, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Save and restore the module info functions
  • Loading branch information
@holdenk
holdenk committed Jul 25, 2017
commit 09cf41eb3e75e92cc9914e675fb2cb2f99290d38
4 changes: 3 additions & 1 deletion python/pyspark/cloudpickle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,7 @@ def save_function_tuple(self, func):
save(f_globals)
save(defaults)
save(dct)
save(func.__module__)
save(closure_values)
write(pickle.TUPLE)
write(pickle.REDUCE) # applies _fill_function on the tuple
Expand Down Expand Up @@ -1007,13 +1008,14 @@ def __reduce__(cls):
return cls.__name__


def _fill_function(func, globals, defaults, dict, closure_values):
def _fill_function(func, globals, defaults, dict, module, closure_values):
""" Fills in the rest of function data into the skeleton function object
that were created via _make_skel_func().
"""
func.__globals__.update(globals)
func.__defaults__ = defaults
func.__dict__ = dict
func.__module__ = module

cells = func.__closure__
if cells is not None:
Expand Down