Skip to content

[SPARK-1065] [PySpark] improve supporting for large broadcast #1912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 7 commits into from
Closed

[SPARK-1065] [PySpark] improve supporting for large broadcast #1912

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6226189
improve large broadcast
davies Aug 13, 2014
e93cf4b
address comments: add test
davies Aug 13, 2014
9a7161f
fix doc tests
davies Aug 13, 2014
c7baa8c
compress serrialized broadcast and command
davies Aug 13, 2014
631a827
Merge branch 'master' into broadcast
davies Aug 13, 2014
db3f232
fix serialization of accumulator
davies Aug 13, 2014
e06df4a
load broadcast from disk in driver automatically
davies Aug 15, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments: add test
  • Loading branch information
@davies
davies committed Aug 13, 2014
commit e93cf4b660d8612a4c34b05b630b507d2d89325d
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,8 +315,7 @@ private[spark] object PythonRDD extends Logging {
JavaRDD.fromRDD(sc.sc.parallelize(objs, parallelism))
}

def readBroadcastFromFile(sc: JavaSparkContext, filename: String):
Broadcast[Array[Byte]] = {
def readBroadcastFromFile(sc: JavaSparkContext, filename: String): Broadcast[Array[Byte]] = {
val file = new DataInputStream(new FileInputStream(filename))
val length = file.readInt()
val obj = new Array[Byte](length)
Expand Down
7 changes: 7 additions & 0 deletions python/pyspark/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,13 @@ def test_namedtuple_in_rdd(self):
theDoes = self.sc.parallelize([jon, jane])
self.assertEquals([jon, jane], theDoes.collect())

def test_large_broadcast(self):
N = 100000
data = [[float(i) for i in range(300)] for i in range(N)]
bdata = self.sc.broadcast(data) # 270MB
m = self.sc.parallelize(range(1), 1).map(lambda x: len(bdata.value)).sum()
self.assertEquals(N, m)


class TestIO(PySparkTestCase):

Expand Down