Skip to content

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 6 commits into from
Closed

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fcf64b2
Use Spark's CompressionCodec and optimize deserializeMapStatuses
dbtsai Oct 23, 2019
da276aa
fix import
dbtsai Oct 23, 2019
d0b1f4c
add comment
dbtsai Oct 23, 2019
013e7b2
Add JDK11
dongjoon-hyun Oct 23, 2019
6212855
Add JDK8
dongjoon-hyun Oct 23, 2019
e5acdbf
Merge pull request #8 from dongjoon-hyun/PR-26235
dbtsai Oct 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add JDK8
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Oct 23, 2019
commit 6212855b8bf707ab6fcc9b696a3192ecf92934fb
38 changes: 19 additions & 19 deletions core/benchmarks/MapStatusesSerDeserBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@ OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 236 245 18 0.8 1179.1 1.0X
Deserialization 842 885 37 0.2 4211.4 0.3X
Serialization 178 187 15 1.1 887.5 1.0X
Deserialization 530 558 32 0.4 2647.5 0.3X

Compressed Serialized MapStatus sizes: 400 bytes
Compressed Serialized MapStatus sizes: 411 bytes
Compressed Serialized Broadcast MapStatus sizes: 2 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 213 219 8 0.9 1065.1 1.0X
Deserialization 846 870 33 0.2 4228.6 0.3X
Serialization 167 175 7 1.2 835.7 1.0X
Deserialization 523 537 22 0.4 2616.2 0.3X

Compressed Serialized MapStatus sizes: 2 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
Expand All @@ -24,43 +24,43 @@ OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 624 709 167 0.3 3121.1 1.0X
Deserialization 885 908 22 0.2 4427.0 0.7X
Serialization 351 416 147 0.6 1754.4 1.0X
Deserialization 546 551 8 0.4 2727.6 0.6X

Compressed Serialized MapStatus sizes: 418 bytes
Compressed Serialized Broadcast MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 427 bytes
Compressed Serialized Broadcast MapStatus sizes: 13 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 603 604 2 0.3 3014.9 1.0X
Deserialization 892 895 5 0.2 4458.7 0.7X
Serialization 320 321 1 0.6 1598.0 1.0X
Deserialization 542 549 7 0.4 2709.0 0.6X

Compressed Serialized MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 13 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4612 4945 471 0.0 23061.0 1.0X
Deserialization 1493 1495 2 0.1 7466.3 3.1X
Serialization 1671 1877 290 0.1 8357.3 1.0X
Deserialization 943 970 32 0.2 4715.8 1.8X

Compressed Serialized MapStatus sizes: 546 bytes
Compressed Serialized Broadcast MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 556 bytes
Compressed Serialized Broadcast MapStatus sizes: 121 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4452 4595 202 0.0 22261.4 1.0X
Deserialization 1464 1477 18 0.1 7321.4 3.0X
Serialization 1373 1436 89 0.1 6865.0 1.0X
Deserialization 940 970 37 0.2 4699.1 1.5X

Compressed Serialized MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 121 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes