Skip to content

[SPARK-31296][SQL][TESTS] Benchmark date-time rebasing in Parquet datasource #28057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 14 commits into from
Closed

[SPARK-31296][SQL][TESTS] Benchmark date-time rebasing in Parquet datasource #28057

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e217139
Add benchmarks for rebasing date-time in parquet
MaxGekk Mar 27, 2020
24476a3
Minor change: `after` -> `before`
MaxGekk Mar 27, 2020
f3c30b7
Benchmarks for dates
MaxGekk Mar 28, 2020
0734e46
Remove LocalDate
MaxGekk Mar 28, 2020
417001b
Re-gen results of DateTimeRebaseBenchmark on JDK 8
MaxGekk Mar 28, 2020
50247e5
Re-gen results of DateTimeRebaseBenchmark on JDK 11
MaxGekk Mar 28, 2020
8703579
Merge branch 'rebase-bechmark-run' into rebase-bechmark
MaxGekk Mar 28, 2020
27a75a6
Minor change of load bench name
MaxGekk Mar 28, 2020
69bb4ed
Fix date generation
MaxGekk Mar 28, 2020
10382e2
Re-gen results of DateTimeRebaseBenchmark on JDK 11
MaxGekk Mar 28, 2020
912dee4
Re-gen results of DateTimeRebaseBenchmark on JDK 8
MaxGekk Mar 28, 2020
c2cc385
Regen benchmark results on Linux JDK 8
MaxGekk Mar 29, 2020
c89f2c9
Regen benchmark results on Linux JDK 11
MaxGekk Mar 29, 2020
e0aedf5
Rename benchmark
MaxGekk Mar 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Regen benchmark results on Linux JDK 8
  • Loading branch information
@MaxGekk
MaxGekk committed Mar 29, 2020
commit c2cc3859db4d2e149378dca97258428adbfb4d47
72 changes: 36 additions & 36 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,52 +2,52 @@
Parquet read/write
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1058-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 19105 19105 0 5.2 191.0 1.0X
before 1582, noop 8289 8289 0 12.1 82.9 2.3X
after 1582, rebase off 29998 29998 0 3.3 300.0 0.6X
after 1582, rebase on 61779 61779 0 1.6 617.8 0.3X
before 1582, rebase off 18777 18777 0 5.3 187.8 1.0X
before 1582, rebase on 58391 58391 0 1.7 583.9 0.3X
after 1582, noop 9472 9472 0 10.6 94.7 1.0X
before 1582, noop 9226 9226 0 10.8 92.3 1.0X
after 1582, rebase off 21201 21201 0 4.7 212.0 0.4X
after 1582, rebase on 56471 56471 0 1.8 564.7 0.2X
before 1582, rebase off 20179 20179 0 5.0 201.8 0.5X
before 1582, rebase on 65717 65717 0 1.5 657.2 0.1X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1058-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 11302 11569 328 8.8 113.0 1.0X
after 1582, vec off, rebase on 31014 31140 198 3.2 310.1 0.4X
after 1582, vec on, rebase off 3187 3357 261 31.4 31.9 3.5X
after 1582, vec on, rebase on 22647 22781 115 4.4 226.5 0.5X
before 1582, vec off, rebase off 11099 11209 96 9.0 111.0 1.0X
before 1582, vec off, rebase on 33300 33428 119 3.0 333.0 0.3X
before 1582, vec on, rebase off 3255 3366 131 30.7 32.6 3.5X
before 1582, vec on, rebase on 24548 24993 386 4.1 245.5 0.5X
after 1582, vec off, rebase off 12294 12434 205 8.1 122.9 1.0X
after 1582, vec off, rebase on 36959 36967 12 2.7 369.6 0.3X
after 1582, vec on, rebase off 3644 3691 49 27.4 36.4 3.4X
after 1582, vec on, rebase on 26764 26852 92 3.7 267.6 0.5X
before 1582, vec off, rebase off 12830 12917 85 7.8 128.3 1.0X
before 1582, vec off, rebase on 38897 39053 229 2.6 389.0 0.3X
before 1582, vec on, rebase off 3638 3693 85 27.5 36.4 3.4X
before 1582, vec on, rebase on 28956 29007 44 3.5 289.6 0.4X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1058-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2094 2094 0 47.8 20.9 1.0X
before 1582, noop 2073 2073 0 48.2 20.7 1.0X
after 1582, rebase off 13400 13400 0 7.5 134.0 0.2X
after 1582, rebase on 77886 77886 0 1.3 778.9 0.0X
before 1582, rebase off 13378 13378 0 7.5 133.8 0.2X
before 1582, rebase on 90746 90746 0 1.1 907.5 0.0X
after 1582, noop 2952 2952 0 33.9 29.5 1.0X
before 1582, noop 2880 2880 0 34.7 28.8 1.0X
after 1582, rebase off 15928 15928 0 6.3 159.3 0.2X
after 1582, rebase on 82816 82816 0 1.2 828.2 0.0X
before 1582, rebase off 15988 15988 0 6.3 159.9 0.2X
before 1582, rebase on 92636 92636 0 1.1 926.4 0.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1058-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 13037 13180 186 7.7 130.4 1.0X
after 1582, vec off, rebase on 53351 53398 41 1.9 533.5 0.2X
after 1582, vec on, rebase off 4027 4076 49 24.8 40.3 3.2X
after 1582, vec on, rebase on 46889 47083 326 2.1 468.9 0.3X
before 1582, vec off, rebase off 13720 13913 219 7.3 137.2 1.0X
before 1582, vec off, rebase on 57410 57779 320 1.7 574.1 0.2X
before 1582, vec on, rebase off 4045 4183 146 24.7 40.4 3.2X
before 1582, vec on, rebase on 49008 49458 397 2.0 490.1 0.3X
after 1582, vec off, rebase off 14863 14917 78 6.7 148.6 1.0X
after 1582, vec off, rebase on 54819 54939 140 1.8 548.2 0.3X
after 1582, vec on, rebase off 4905 4941 32 20.4 49.0 3.0X
after 1582, vec on, rebase on 44914 45008 124 2.2 449.1 0.3X
before 1582, vec off, rebase off 14928 14970 48 6.7 149.3 1.0X
before 1582, vec off, rebase on 59752 59996 245 1.7 597.5 0.2X
before 1582, vec on, rebase off 4892 4916 33 20.4 48.9 3.0X
before 1582, vec on, rebase on 46854 46977 198 2.1 468.5 0.3X