Skip to content

[SPARK-31296][SQL][TESTS] Benchmark date-time rebasing in Parquet datasource #28057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 14 commits into from
Closed

[SPARK-31296][SQL][TESTS] Benchmark date-time rebasing in Parquet datasource #28057

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e217139
Add benchmarks for rebasing date-time in parquet
MaxGekk Mar 27, 2020
24476a3
Minor change: `after` -> `before`
MaxGekk Mar 27, 2020
f3c30b7
Benchmarks for dates
MaxGekk Mar 28, 2020
0734e46
Remove LocalDate
MaxGekk Mar 28, 2020
417001b
Re-gen results of DateTimeRebaseBenchmark on JDK 8
MaxGekk Mar 28, 2020
50247e5
Re-gen results of DateTimeRebaseBenchmark on JDK 11
MaxGekk Mar 28, 2020
8703579
Merge branch 'rebase-bechmark-run' into rebase-bechmark
MaxGekk Mar 28, 2020
27a75a6
Minor change of load bench name
MaxGekk Mar 28, 2020
69bb4ed
Fix date generation
MaxGekk Mar 28, 2020
10382e2
Re-gen results of DateTimeRebaseBenchmark on JDK 11
MaxGekk Mar 28, 2020
912dee4
Re-gen results of DateTimeRebaseBenchmark on JDK 8
MaxGekk Mar 28, 2020
c2cc385
Regen benchmark results on Linux JDK 8
MaxGekk Mar 29, 2020
c89f2c9
Regen benchmark results on Linux JDK 11
MaxGekk Mar 29, 2020
e0aedf5
Rename benchmark
MaxGekk Mar 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Re-gen results of DateTimeRebaseBenchmark on JDK 8
  • Loading branch information
@MaxGekk
MaxGekk committed Mar 28, 2020
commit 417001b0e372f86c4ac74574eecb9d585e53ecbb
53 changes: 53 additions & 0 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
================================================================================================
Parquet read/write
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 8873 8873 0 11.3 88.7 1.0X
before 1582, noop 8543 8543 0 11.7 85.4 1.0X
after 1582, rebase off 20145 20145 0 5.0 201.4 0.4X
after 1582, rebase on 51934 51934 0 1.9 519.3 0.2X
before 1582, rebase off 20253 20253 0 4.9 202.5 0.4X
before 1582, rebase on 58564 58564 0 1.7 585.6 0.2X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Load date from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 9260 9308 48 10.8 92.6 1.0X
after 1582, vec off, rebase on 27658 28050 348 3.6 276.6 0.3X
after 1582, vec on, rebase off 2689 2799 115 37.2 26.9 3.4X
after 1582, vec on, rebase on 2558 2564 6 39.1 25.6 3.6X
before 1582, vec off, rebase off 8990 9009 20 11.1 89.9 1.0X
before 1582, vec off, rebase on 29806 30053 268 3.4 298.1 0.3X
before 1582, vec on, rebase off 2550 2567 15 39.2 25.5 3.6X
before 1582, vec on, rebase on 2578 2586 13 38.8 25.8 3.6X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2099 2099 0 47.6 21.0 1.0X
before 1582, noop 2080 2080 0 48.1 20.8 1.0X
after 1582, rebase off 12626 12626 0 7.9 126.3 0.2X
after 1582, rebase on 76490 76490 0 1.3 764.9 0.0X
before 1582, rebase off 12548 12548 0 8.0 125.5 0.2X
before 1582, rebase on 81903 81903 0 1.2 819.0 0.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Load timestamp from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 12780 12890 126 7.8 127.8 1.0X
after 1582, vec off, rebase on 53386 53418 35 1.9 533.9 0.2X
after 1582, vec on, rebase off 4000 4055 84 25.0 40.0 3.2X
after 1582, vec on, rebase on 45193 45277 125 2.2 451.9 0.3X
before 1582, vec off, rebase off 12806 12913 101 7.8 128.1 1.0X
before 1582, vec off, rebase on 54768 54976 355 1.8 547.7 0.2X
before 1582, vec on, rebase off 4017 4300 294 24.9 40.2 3.2X
before 1582, vec on, rebase on 46892 47118 200 2.1 468.9 0.3X