Skip to content

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 6 commits into from
Closed

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fcf64b2
Use Spark's CompressionCodec and optimize deserializeMapStatuses
dbtsai Oct 23, 2019
da276aa
fix import
dbtsai Oct 23, 2019
d0b1f4c
add comment
dbtsai Oct 23, 2019
013e7b2
Add JDK11
dongjoon-hyun Oct 23, 2019
6212855
Add JDK8
dongjoon-hyun Oct 23, 2019
e5acdbf
Merge pull request #8 from dongjoon-hyun/PR-26235
dbtsai Oct 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add JDK11
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Oct 23, 2019
commit 013e7b2bdda24e81b1c69486f5fe71e8e520ebc4
38 changes: 19 additions & 19 deletions core/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@ OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 205 213 13 1.0 1023.6 1.0X
Deserialization 908 939 27 0.2 4540.2 0.2X
Serialization 170 178 9 1.2 849.7 1.0X
Deserialization 530 535 9 0.4 2651.1 0.3X

Compressed Serialized MapStatus sizes: 400 bytes
Compressed Serialized MapStatus sizes: 411 bytes
Compressed Serialized Broadcast MapStatus sizes: 2 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 195 204 24 1.0 976.9 1.0X
Deserialization 913 940 33 0.2 4566.7 0.2X
Serialization 157 165 7 1.3 785.4 1.0X
Deserialization 495 588 79 0.4 2476.7 0.3X

Compressed Serialized MapStatus sizes: 2 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
Expand All @@ -24,43 +24,43 @@ OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 616 619 3 0.3 3079.1 1.0X
Deserialization 936 954 22 0.2 4680.5 0.7X
Serialization 344 351 4 0.6 1720.4 1.0X
Deserialization 527 579 99 0.4 2635.9 0.7X

Compressed Serialized MapStatus sizes: 418 bytes
Compressed Serialized Broadcast MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 427 bytes
Compressed Serialized Broadcast MapStatus sizes: 13 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 586 588 3 0.3 2928.8 1.0X
Deserialization 929 933 4 0.2 4647.0 0.6X
Serialization 317 321 4 0.6 1583.8 1.0X
Deserialization 530 540 15 0.4 2648.3 0.6X

Compressed Serialized MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 13 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4740 4916 249 0.0 23698.5 1.0X
Deserialization 1578 1597 27 0.1 7890.6 3.0X
Serialization 1738 1849 156 0.1 8692.0 1.0X
Deserialization 946 977 33 0.2 4730.2 1.8X

Compressed Serialized MapStatus sizes: 546 bytes
Compressed Serialized Broadcast MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 556 bytes
Compressed Serialized Broadcast MapStatus sizes: 121 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4492 4573 115 0.0 22458.3 1.0X
Deserialization 1533 1547 20 0.1 7664.8 2.9X
Serialization 1379 1432 76 0.1 6892.6 1.0X
Deserialization 929 941 19 0.2 4645.5 1.5X

Compressed Serialized MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 121 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes