Skip to content

[SPARK-34815][SQL] Update CSVBenchmark #31917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 2 commits into from
Closed

[SPARK-34815][SQL] Update CSVBenchmark #31917

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
750f92b
Update CSVBenchmark
HyukjinKwon Mar 22, 2021
3575e48
Update JDK 11 results
HyukjinKwon Mar 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 44 additions & 44 deletions sql/core/benchmarks/CSVBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,66 +2,66 @@
Benchmark to measure CSV read/write performance
================================================================================================

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Parsing quoted values: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
One quoted string 53332 53484 194 0.0 1066633.5 1.0X
One quoted string 21212 21537 327 0.0 424244.5 1.0X

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Wide rows with 1000 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Select 1000 columns 127472 128337 1185 0.0 127472.4 1.0X
Select 100 columns 43731 43856 130 0.0 43730.7 2.9X
Select one column 37347 37401 47 0.0 37347.4 3.4X
count() 8014 8028 25 0.1 8013.8 15.9X
Select 100 columns, one bad input field 95603 95726 106 0.0 95603.0 1.3X
Select 100 columns, corrupt record field 111851 111969 171 0.0 111851.4 1.1X
Select 1000 columns 73744 74898 1930 0.0 73743.8 1.0X
Select 100 columns 22704 22860 236 0.0 22704.4 3.2X
Select one column 17837 17977 121 0.1 17837.2 4.1X
count() 4304 4320 27 0.2 4304.0 17.1X
Select 100 columns, one bad input field 42060 42280 378 0.0 42059.8 1.8X
Select 100 columns, corrupt record field 46633 47520 773 0.0 46632.5 1.6X

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Count a dataset with 10 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Select 10 columns + count() 20364 20481 110 0.5 2036.4 1.0X
Select 1 column + count() 14706 14803 153 0.7 1470.6 1.4X
count() 3855 3880 32 2.6 385.5 5.3X
Select 10 columns + count() 9906 10132 246 1.0 990.6 1.0X
Select 1 column + count() 6497 6616 104 1.5 649.7 1.5X
count() 2285 2322 32 4.4 228.5 4.3X

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Write dates and timestamps: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Create a dataset of timestamps 2191 2205 14 4.6 219.1 1.0X
to_csv(timestamp) 13209 13253 43 0.8 1320.9 0.2X
write timestamps to files 12300 12380 71 0.8 1230.0 0.2X
Create a dataset of dates 2254 2269 14 4.4 225.4 1.0X
to_csv(date) 7980 8006 22 1.3 798.0 0.3X
write dates to files 7076 7100 26 1.4 707.6 0.3X
Create a dataset of timestamps 902 932 30 11.1 90.2 1.0X
to_csv(timestamp) 8537 8851 274 1.2 853.7 0.1X
write timestamps to files 7810 8000 238 1.3 781.0 0.1X
Create a dataset of dates 929 931 2 10.8 92.9 1.0X
to_csv(date) 5170 5237 62 1.9 517.0 0.2X
write dates to files 4163 4220 49 2.4 416.3 0.2X

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read dates and timestamps: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
read timestamp text from files 2405 2408 5 4.2 240.5 1.0X
read timestamps from files 54502 54624 207 0.2 5450.2 0.0X
infer timestamps from files 112896 113040 135 0.1 11289.6 0.0X
read date text from files 2127 2141 23 4.7 212.7 1.1X
read date from files 30229 30257 29 0.3 3022.9 0.1X
infer date from files 28156 28621 409 0.4 2815.6 0.1X
timestamp strings 3096 3097 1 3.2 309.6 0.8X
parse timestamps from Dataset[String] 63096 63751 571 0.2 6309.6 0.0X
infer timestamps from Dataset[String] 120916 121262 556 0.1 12091.6 0.0X
date strings 3445 3457 13 2.9 344.5 0.7X
parse dates from Dataset[String] 37481 37585 91 0.3 3748.1 0.1X
from_csv(timestamp) 57933 57996 69 0.2 5793.3 0.0X
from_csv(date) 35312 35469 164 0.3 3531.2 0.1X
read timestamp text from files 1475 1497 33 6.8 147.5 1.0X
read timestamps from files 18596 18811 343 0.5 1859.6 0.1X
infer timestamps from files 37182 37511 342 0.3 3718.2 0.0X
read date text from files 1183 1210 31 8.5 118.3 1.2X
read date from files 8797 9099 283 1.1 879.7 0.2X
infer date from files 11296 11427 218 0.9 1129.6 0.1X
timestamp strings 1379 1382 4 7.3 137.9 1.1X
parse timestamps from Dataset[String] 18243 19000 721 0.5 1824.3 0.1X
infer timestamps from Dataset[String] 38253 39096 731 0.3 3825.3 0.0X
date strings 1686 1721 35 5.9 168.6 0.9X
parse dates from Dataset[String] 10474 10680 184 1.0 1047.4 0.1X
from_csv(timestamp) 18643 18965 350 0.5 1864.3 0.1X
from_csv(date) 9814 10018 188 1.0 981.4 0.2X

OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Java HotSpot(TM) 64-Bit Server VM 11.0.3+12-LTS on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Filters pushdown: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
w/o filters 24751 24829 67 0.0 247510.6 1.0X
pushdown disabled 24856 24889 29 0.0 248558.7 1.0X
w/ filters 1881 1892 11 0.1 18814.4 13.2X
w/o filters 11243 11535 287 0.0 112433.9 1.0X
pushdown disabled 11093 11117 34 0.0 110931.9 1.0X
w/ filters 794 800 5 0.1 7942.1 14.2X


Loading