Skip to content

[SPARK-11271][SPARK-11016][Core] Use Spark BitSet instead of RoaringBitmap to reduce memory usage #9243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 7 commits into from
Closed

[SPARK-11271][SPARK-11016][Core] Use Spark BitSet instead of RoaringBitmap to reduce memory usage #9243

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
392975d
Use Spark BitSet instead of RoaringBitmap to reduce memory usage.
viirya Oct 23, 2015
d30ec97
Remove RoaringBitmap from KryoSerializer and pom.
viirya Oct 23, 2015
e981fc9
Add BitSet to KryoSerializer.
viirya Oct 24, 2015
0ac1fae
Merge remote-tracking branch 'upstream/master' into mapstatus-bitset
viirya Oct 31, 2015
37f85ce
Fix the test getting stuck by adding default constructor.
viirya Oct 31, 2015
95495f8
Merge remote-tracking branch 'upstream/master' into mapstatus-bitset
viirya Nov 2, 2015
a6ea5fc
Use default constructor.
viirya Nov 2, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove RoaringBitmap from KryoSerializer and pom.
  • Loading branch information
@viirya
viirya committed Oct 23, 2015
commit d30ec974401bbf09ecea3f825faa5f55b3ccfcf3
4 changes: 0 additions & 4 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,6 @@
<groupId>net.jpountz.lz4</groupId>
<artifactId>lz4</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
Expand Down
7 changes: 0 additions & 7 deletions core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ import com.esotericsoftware.kryo.io.{Input => KryoInput, Output => KryoOutput}
import com.esotericsoftware.kryo.serializers.{JavaSerializer => KryoJavaSerializer}
import com.twitter.chill.{AllScalaRegistrar, EmptyScalaKryoInstantiator}
import org.apache.avro.generic.{GenericData, GenericRecord}
import org.roaringbitmap.{ArrayContainer, BitmapContainer, RoaringArray, RoaringBitmap}

import org.apache.spark._
import org.apache.spark.api.python.PythonBroadcast
Expand Down Expand Up @@ -363,12 +362,6 @@ private[serializer] object KryoSerializer {
classOf[StorageLevel],
classOf[CompressedMapStatus],
classOf[HighlyCompressedMapStatus],
classOf[RoaringBitmap],
classOf[RoaringArray],
classOf[RoaringArray.Element],
classOf[Array[RoaringArray.Element]],
classOf[ArrayContainer],
classOf[BitmapContainer],
classOf[CompactBuffer[_]],
classOf[BlockManagerId],
classOf[Array[Byte]],
Expand Down
5 changes: 0 additions & 5 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,11 +623,6 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<version>0.4.5</version>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
Expand Down