Skip to content

[SPARK-5929][PYSPARK] Context addPyPackage and addPyRequirements #12398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
buckhx wants to merge 24 commits into from
Closed

[SPARK-5929][PYSPARK] Context addPyPackage and addPyRequirements #12398

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
0ed060d
added requirements file to pyspark
Feb 18, 2015
6b8bcde
tarfile has no contextmanager in python2.
Feb 18, 2015
2773483
reqs fix
Feb 19, 2015
0371ad9
temp tar file
Feb 19, 2015
f2a46e5
bubbled up try finally
Mar 5, 2015
fca4be6
forgot to remove
Mar 5, 2015
d287522
added requirementsFile tests and switch to __import__
Apr 3, 2015
76ff637
merged tests
Apr 3, 2015
565bf7f
pep8 styling
Jul 14, 2015
23771fd
support namespace packages and extract addModule logic
Oct 5, 2015
cd21c5c
tmp_dir to mod
Oct 5, 2015
39f26d9
remove reqs from context constructor
Oct 21, 2015
49a4ed0
upstream merge
Oct 21, 2015
1501d0f
upstream merge
buckhx Mar 31, 2016
3af35bb
Merge remote-tracking branch 'upstream/master'
Apr 13, 2016
88a1d6c
add_py_package test
Apr 13, 2016
93b9e9f
uncommented pip_requirements test
Apr 14, 2016
82476a6
removed todo
Apr 14, 2016
ce9966e
spacing
Apr 14, 2016
82534d0
formatting and addPyPackage example
May 9, 2016
ea6b89f
add py-requirements submit option
May 10, 2016
1d5d25f
Merge pull request #1 from robert3005/fork/master
buckhx May 11, 2016
f4af842
addRequirementsFile -> addPyRequirements
May 13, 2016
9c37e06
pep8ing
May 13, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
pep8ing
  • Loading branch information
buck heroux committed May 13, 2016
commit 9c37e0649a75d22d5712b08dcd6c3acee412f719
2 changes: 1 addition & 1 deletion python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -834,7 +834,7 @@ def addPyPackage(self, pkg):

import pyspark
import foolib

sc = pyspark.SparkContext()
sc.addPyPackage(foolib)
# foolib now in workers PYTHONPATH
Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1955,8 +1955,8 @@ def test_add_py_package(self):
temp.write("triple = lambda x: 3*x")
pkg = __import__(name)
with SparkContext() as sc:
#trips = sc.parallelize([0, 1, 2, 3]).map(pkg.triple)
#sc.addPyPackage(pkg)
# trips = sc.parallelize([0, 1, 2, 3]).map(pkg.triple)
# sc.addPyPackage(pkg)
trips = sc.parallelize([0, 1, 2, 3]).map(lambda x: pkg.triple(x))
self.assertSequenceEqual([0, 3, 6, 9], trips.collect())
finally:
Expand Down