Skip to content

[SPARK-31359][SQL] Speed up timestamps rebasing #28119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 42 commits into from
Closed

[SPARK-31359][SQL] Speed up timestamps rebasing #28119

Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
354965f
Optimize rebaseGregorianToJulianMicros
MaxGekk Apr 4, 2020
c6a70b2
Optimize rebaseJulianToGregorianMicros
MaxGekk Apr 4, 2020
52bf1d4
Set LA as the default time zone
MaxGekk Apr 4, 2020
bf08bd2
Re-gen DateTimeBenchmark results on JDK 8
MaxGekk Apr 4, 2020
4b89d12
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Apr 4, 2020
ac2beef
Optimize rebaseMicros
MaxGekk Apr 5, 2020
66a8af4
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Apr 5, 2020
2b21ad7
Re-gen DateTimeBenchmark results on JDK 8
MaxGekk Apr 5, 2020
17f2684
Move days rebase functions to RebaseDateTime
MaxGekk Apr 5, 2020
ddd1794
Move micros rebase functions to RebaseDateTime
MaxGekk Apr 5, 2020
743d111
Load rebase records from resource files
MaxGekk Apr 5, 2020
a52175d
Move rebasing tests to RebaseDateTimeSuite
MaxGekk Apr 5, 2020
2b4a437
Add comments
MaxGekk Apr 5, 2020
6202535
Add comments to julian-gregorian rebase functions
MaxGekk Apr 6, 2020
d39c835
Merge remote-tracking branch 'remotes/origin/master' into optimize-re…
MaxGekk Apr 6, 2020
248b9a5
Fix DaysWritable v1.2
MaxGekk Apr 6, 2020
d5800f9
Fix OrcColumnVector v1.2
MaxGekk Apr 6, 2020
d87ebec
Validate JSON files
MaxGekk Apr 6, 2020
5d417b7
Add tests for rebasing micros
MaxGekk Apr 7, 2020
ba96e3c
Re-gen JSON files
MaxGekk Apr 7, 2020
f9d0faa
Assume seconds in JSON files
MaxGekk Apr 7, 2020
377ef8a
Add JSON generate function
MaxGekk Apr 7, 2020
44df0dc
Don't check that zone id is resolvable
MaxGekk Apr 7, 2020
be672b6
Try AnyRefMap
MaxGekk Apr 7, 2020
8cd02d9
Try Java HashMap
MaxGekk Apr 7, 2020
be37b16
Revert "Try Java HashMap"
MaxGekk Apr 7, 2020
f6cf352
Put switches and diff to one array
MaxGekk Apr 7, 2020
88ff829
Revert "Put switches and diff to one array"
MaxGekk Apr 7, 2020
0ffd2a6
Fix comments
MaxGekk Apr 7, 2020
9a38c62
Random step in tests
MaxGekk Apr 7, 2020
03e2ed3
Intro JsonRebaseRecord
MaxGekk Apr 7, 2020
5204265
Add note about the system JVM time zone
MaxGekk Apr 7, 2020
b9951b9
Add a comment for RebaseDateTime
MaxGekk Apr 7, 2020
8c3570d
Re-gen DateTimeRebaseBenchmark results on JDK 8
MaxGekk Apr 8, 2020
9402336
Blacklist some time zones
MaxGekk Apr 8, 2020
90dc0bd
Re-gen DateTimeRebaseBenchmark results on JDK 11
MaxGekk Apr 8, 2020
25f0996
Merge branch 'optimize-rebase-micros' of github.com:MaxGekk/spark int…
MaxGekk Apr 8, 2020
864734d
Disable JSON generation
MaxGekk Apr 8, 2020
a84d40e
Add comments for refRebaseGregorianToJulianDays() and refRebaseJulian…
MaxGekk Apr 8, 2020
bb12a81
Fix the comment for RebaseDateTime
MaxGekk Apr 8, 2020
d2d03b2
Fix indentation and rename params
MaxGekk Apr 8, 2020
4281d4b
Address Wenchen's review comments
MaxGekk Apr 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Re-gen DateTimeRebaseBenchmark results on JDK 8
  • Loading branch information
@MaxGekk
MaxGekk committed Apr 8, 2020
commit 8c3570d4933f14ffcc907bfb6060394071a209ca
88 changes: 44 additions & 44 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,49 @@ OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 23027 23027 0 4.3 230.3 1.0X
before 1582, noop 10049 10049 0 10.0 100.5 2.3X
after 1582, rebase off 36528 36528 0 2.7 365.3 0.6X
after 1582, rebase on 35754 35754 0 2.8 357.5 0.6X
before 1582, rebase off 21920 21920 0 4.6 219.2 1.1X
before 1582, rebase on 22700 22700 0 4.4 227.0 1.0X
after 1582, noop 24030 24030 0 4.2 240.3 1.0X
before 1582, noop 10900 10900 0 9.2 109.0 2.2X
after 1582, rebase off 35666 35666 0 2.8 356.7 0.7X
after 1582, rebase on 35872 35872 0 2.8 358.7 0.7X
before 1582, rebase off 22131 22131 0 4.5 221.3 1.1X
before 1582, rebase on 22967 22967 0 4.4 229.7 1.0X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 12996 13142 163 7.7 130.0 1.0X
after 1582, vec off, rebase on 13371 13397 37 7.5 133.7 1.0X
after 1582, vec on, rebase off 3689 3703 14 27.1 36.9 3.5X
after 1582, vec on, rebase on 5243 5288 46 19.1 52.4 2.5X
before 1582, vec off, rebase off 13064 13159 88 7.7 130.6 1.0X
before 1582, vec off, rebase on 13822 13915 92 7.2 138.2 0.9X
before 1582, vec on, rebase off 3673 3690 19 27.2 36.7 3.5X
before 1582, vec on, rebase on 5540 5569 42 18.1 55.4 2.3X
after 1582, vec off, rebase off 12523 12589 58 8.0 125.2 1.0X
after 1582, vec off, rebase on 13387 13466 86 7.5 133.9 0.9X
after 1582, vec on, rebase off 3646 3678 41 27.4 36.5 3.4X
after 1582, vec on, rebase on 5104 5198 123 19.6 51.0 2.5X
before 1582, vec off, rebase off 13041 13101 98 7.7 130.4 1.0X
before 1582, vec off, rebase on 14077 14104 23 7.1 140.8 0.9X
before 1582, vec on, rebase off 3667 3726 87 27.3 36.7 3.4X
before 1582, vec on, rebase on 5504 5521 21 18.2 55.0 2.3X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2821 2821 0 35.5 28.2 1.0X
before 1582, noop 2699 2699 0 37.1 27.0 1.0X
after 1582, rebase off 16095 16095 0 6.2 161.0 0.2X
after 1582, rebase on 17704 17704 0 5.6 177.0 0.2X
before 1582, rebase off 16328 16328 0 6.1 163.3 0.2X
before 1582, rebase on 18303 18303 0 5.5 183.0 0.2X
after 1582, noop 2782 2782 0 36.0 27.8 1.0X
before 1582, noop 2778 2778 0 36.0 27.8 1.0X
after 1582, rebase off 16980 16980 0 5.9 169.8 0.2X
after 1582, rebase on 20023 20023 0 5.0 200.2 0.1X
before 1582, rebase off 17618 17618 0 5.7 176.2 0.2X
before 1582, rebase on 20416 20416 0 4.9 204.2 0.1X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 15397 15490 102 6.5 154.0 1.0X
after 1582, vec off, rebase on 16950 17039 81 5.9 169.5 0.9X
after 1582, vec on, rebase off 5067 5292 203 19.7 50.7 3.0X
after 1582, vec on, rebase on 8691 8751 53 11.5 86.9 1.8X
before 1582, vec off, rebase off 15107 15215 156 6.6 151.1 1.0X
before 1582, vec off, rebase on 17565 17569 4 5.7 175.6 0.9X
before 1582, vec on, rebase off 5104 5226 165 19.6 51.0 3.0X
before 1582, vec on, rebase on 9200 9230 29 10.9 92.0 1.7X
after 1582, vec off, rebase off 15098 15146 55 6.6 151.0 1.0X
after 1582, vec off, rebase on 18127 18214 86 5.5 181.3 0.8X
after 1582, vec on, rebase off 4936 4984 42 20.3 49.4 3.1X
after 1582, vec on, rebase on 9685 9745 58 10.3 96.8 1.6X
before 1582, vec off, rebase off 15233 15259 25 6.6 152.3 1.0X
before 1582, vec off, rebase on 18710 18727 24 5.3 187.1 0.8X
before 1582, vec on, rebase off 4954 4980 24 20.2 49.5 3.0X
before 1582, vec on, rebase on 10224 10240 22 9.8 102.2 1.5X


================================================================================================
Expand All @@ -59,36 +59,36 @@ OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 22958 22958 0 4.4 229.6 1.0X
before 1582, noop 10209 10209 0 9.8 102.1 2.2X
after 1582 31392 31392 0 3.2 313.9 0.7X
before 1582 18976 18976 0 5.3 189.8 1.2X
after 1582, noop 23770 23770 0 4.2 237.7 1.0X
before 1582, noop 10939 10939 0 9.1 109.4 2.2X
after 1582 32879 32879 0 3.0 328.8 0.7X
before 1582 20267 20267 0 4.9 202.7 1.2X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 41856 41916 52 2.4 418.6 1.0X
after 1582, vec on 3673 3692 17 27.2 36.7 11.4X
before 1582, vec off 38616 38693 111 2.6 386.2 1.1X
before 1582, vec on 4059 4082 22 24.6 40.6 10.3X
after 1582, vec off 39651 39686 31 2.5 396.5 1.0X
after 1582, vec on 3647 3660 13 27.4 36.5 10.9X
before 1582, vec off 38155 38219 61 2.6 381.6 1.0X
before 1582, vec on 4041 4046 6 24.7 40.4 9.8X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2725 2725 0 36.7 27.2 1.0X
before 1582, noop 2704 2704 0 37.0 27.0 1.0X
after 1582 64771 64771 0 1.5 647.7 0.0X
before 1582 70222 70222 0 1.4 702.2 0.0X
after 1582, noop 2802 2802 0 35.7 28.0 1.0X
before 1582, noop 2797 2797 0 35.8 28.0 1.0X
after 1582 59877 59877 0 1.7 598.8 0.0X
before 1582 61361 61361 0 1.6 613.6 0.0X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 54826 54904 69 1.8 548.3 1.0X
after 1582, vec on 47165 47260 135 2.1 471.7 1.2X
before 1582, vec off 60245 60394 193 1.7 602.5 0.9X
before 1582, vec on 53305 53367 76 1.9 533.0 1.0X
after 1582, vec off 48197 48288 118 2.1 482.0 1.0X
after 1582, vec on 38247 38351 128 2.6 382.5 1.3X
before 1582, vec off 53179 53359 249 1.9 531.8 0.9X
before 1582, vec on 44076 44268 269 2.3 440.8 1.1X