Skip to content

[SPARK-29434][Core] Improve the MapStatuses Serialization Performance #26085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 18 commits into from
+87 −55
Closed

[SPARK-29434][Core] Improve the MapStatuses Serialization Performance #26085

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
78bbcf2
temp
dbtsai Sep 25, 2019
9714f39
tempMerge branch 'master' into mapStatus
dbtsai Oct 8, 2019
18e5bda
finish
dbtsai Oct 10, 2019
39502ed
tune test
dbtsai Oct 10, 2019
3958c01
ready to submit PR
dbtsai Oct 10, 2019
a4a807e
removed unused import
dbtsai Oct 10, 2019
d0c3532
revert
dbtsai Oct 10, 2019
ed08f2e
fix
dbtsai Oct 10, 2019
a601356
address feedback
dbtsai Oct 11, 2019
8dc8fad
update
dbtsai Oct 16, 2019
0bf182a
Fix styling and typo
dbtsai Oct 17, 2019
bd88abd
temp
dbtsai Oct 18, 2019
f184c4c
remove MapStatusesSerDeserBenchmark.scala
dbtsai Oct 18, 2019
bc6a14c
Merge branch 'master' into mapStatus
dbtsai Oct 18, 2019
5aadd8f
revert unneeded change
dbtsai Oct 18, 2019
d7fce82
Add JDK11
dongjoon-hyun Oct 18, 2019
7095b60
Add JDK8 result
dongjoon-hyun Oct 18, 2019
08c8fb2
Merge pull request #7 from dongjoon-hyun/PR-26085
dbtsai Oct 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add JDK11
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Oct 18, 2019
commit d7fce8202a48ecdfa32da08e5df1447482c36a0a
42 changes: 21 additions & 21 deletions core/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,65 +2,65 @@ OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 609 631 22 0.3 3043.8 1.0X
Deserialization 840 897 67 0.2 4201.2 0.7X
Serialization 205 213 13 1.0 1023.6 1.0X
Deserialization 908 939 27 0.2 4540.2 0.2X

Compressed Serialized MapStatus sizes: 393 bytes
Compressed Serialized Broadcast MapStatus sizes: 3 MB
Compressed Serialized MapStatus sizes: 400 bytes
Compressed Serialized Broadcast MapStatus sizes: 2 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 591 599 8 0.3 2955.3 1.0X
Deserialization 878 913 31 0.2 4392.2 0.7X
Serialization 195 204 24 1.0 976.9 1.0X
Deserialization 913 940 33 0.2 4566.7 0.2X

Compressed Serialized MapStatus sizes: 3 MB
Compressed Serialized MapStatus sizes: 2 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 1776 1778 2 0.1 8880.5 1.0X
Deserialization 1086 1086 0 0.2 5427.9 1.6X
Serialization 616 619 3 0.3 3079.1 1.0X
Deserialization 936 954 22 0.2 4680.5 0.7X

Compressed Serialized MapStatus sizes: 411 bytes
Compressed Serialized Broadcast MapStatus sizes: 15 MB
Compressed Serialized MapStatus sizes: 418 bytes
Compressed Serialized Broadcast MapStatus sizes: 14 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 1725 1726 1 0.1 8624.9 1.0X
Deserialization 1093 1094 2 0.2 5463.6 1.6X
Serialization 586 588 3 0.3 2928.8 1.0X
Deserialization 929 933 4 0.2 4647.0 0.6X

Compressed Serialized MapStatus sizes: 15 MB
Compressed Serialized MapStatus sizes: 14 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 12421 12522 142 0.0 62104.4 1.0X
Deserialization 3020 3043 32 0.1 15102.0 4.1X
Serialization 4740 4916 249 0.0 23698.5 1.0X
Deserialization 1578 1597 27 0.1 7890.6 3.0X

Compressed Serialized MapStatus sizes: 544 bytes
Compressed Serialized Broadcast MapStatus sizes: 131 MB
Compressed Serialized MapStatus sizes: 546 bytes
Compressed Serialized Broadcast MapStatus sizes: 123 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 11719 11737 26 0.0 58595.3 1.0X
Deserialization 3018 3051 46 0.1 15091.7 3.9X
Serialization 4492 4573 115 0.0 22458.3 1.0X
Deserialization 1533 1547 20 0.1 7664.8 2.9X

Compressed Serialized MapStatus sizes: 131 MB
Compressed Serialized MapStatus sizes: 123 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes