Skip to content

[SPARK-31359][SQL] Speed up timestamps rebasing #28119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 42 commits into from
Closed

[SPARK-31359][SQL] Speed up timestamps rebasing #28119

Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
354965f
Optimize rebaseGregorianToJulianMicros
MaxGekk Apr 4, 2020
c6a70b2
Optimize rebaseJulianToGregorianMicros
MaxGekk Apr 4, 2020
52bf1d4
Set LA as the default time zone
MaxGekk Apr 4, 2020
bf08bd2
Re-gen DateTimeBenchmark results on JDK 8
MaxGekk Apr 4, 2020
4b89d12
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Apr 4, 2020
ac2beef
Optimize rebaseMicros
MaxGekk Apr 5, 2020
66a8af4
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Apr 5, 2020
2b21ad7
Re-gen DateTimeBenchmark results on JDK 8
MaxGekk Apr 5, 2020
17f2684
Move days rebase functions to RebaseDateTime
MaxGekk Apr 5, 2020
ddd1794
Move micros rebase functions to RebaseDateTime
MaxGekk Apr 5, 2020
743d111
Load rebase records from resource files
MaxGekk Apr 5, 2020
a52175d
Move rebasing tests to RebaseDateTimeSuite
MaxGekk Apr 5, 2020
2b4a437
Add comments
MaxGekk Apr 5, 2020
6202535
Add comments to julian-gregorian rebase functions
MaxGekk Apr 6, 2020
d39c835
Merge remote-tracking branch 'remotes/origin/master' into optimize-re…
MaxGekk Apr 6, 2020
248b9a5
Fix DaysWritable v1.2
MaxGekk Apr 6, 2020
d5800f9
Fix OrcColumnVector v1.2
MaxGekk Apr 6, 2020
d87ebec
Validate JSON files
MaxGekk Apr 6, 2020
5d417b7
Add tests for rebasing micros
MaxGekk Apr 7, 2020
ba96e3c
Re-gen JSON files
MaxGekk Apr 7, 2020
f9d0faa
Assume seconds in JSON files
MaxGekk Apr 7, 2020
377ef8a
Add JSON generate function
MaxGekk Apr 7, 2020
44df0dc
Don't check that zone id is resolvable
MaxGekk Apr 7, 2020
be672b6
Try AnyRefMap
MaxGekk Apr 7, 2020
8cd02d9
Try Java HashMap
MaxGekk Apr 7, 2020
be37b16
Revert "Try Java HashMap"
MaxGekk Apr 7, 2020
f6cf352
Put switches and diff to one array
MaxGekk Apr 7, 2020
88ff829
Revert "Put switches and diff to one array"
MaxGekk Apr 7, 2020
0ffd2a6
Fix comments
MaxGekk Apr 7, 2020
9a38c62
Random step in tests
MaxGekk Apr 7, 2020
03e2ed3
Intro JsonRebaseRecord
MaxGekk Apr 7, 2020
5204265
Add note about the system JVM time zone
MaxGekk Apr 7, 2020
b9951b9
Add a comment for RebaseDateTime
MaxGekk Apr 7, 2020
8c3570d
Re-gen DateTimeRebaseBenchmark results on JDK 8
MaxGekk Apr 8, 2020
9402336
Blacklist some time zones
MaxGekk Apr 8, 2020
90dc0bd
Re-gen DateTimeRebaseBenchmark results on JDK 11
MaxGekk Apr 8, 2020
25f0996
Merge branch 'optimize-rebase-micros' of github.com:MaxGekk/spark int…
MaxGekk Apr 8, 2020
864734d
Disable JSON generation
MaxGekk Apr 8, 2020
a84d40e
Add comments for refRebaseGregorianToJulianDays() and refRebaseJulian…
MaxGekk Apr 8, 2020
bb12a81
Fix the comment for RebaseDateTime
MaxGekk Apr 8, 2020
d2d03b2
Fix indentation and rename params
MaxGekk Apr 8, 2020
4281d4b
Address Wenchen's review comments
MaxGekk Apr 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Re-gen DateTimeBenchmark results on JDK 8
  • Loading branch information
@MaxGekk
MaxGekk committed Apr 4, 2020
commit bf08bd2ee77b0c74a8e6e47c244b3d771b213dd9
88 changes: 44 additions & 44 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,49 @@ OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 9488 9488 0 10.5 94.9 1.0X
before 1582, noop 9301 9301 0 10.8 93.0 1.0X
after 1582, rebase off 20109 20109 0 5.0 201.1 0.5X
after 1582, rebase on 20004 20004 0 5.0 200.0 0.5X
before 1582, rebase off 19906 19906 0 5.0 199.1 0.5X
before 1582, rebase on 20466 20466 0 4.9 204.7 0.5X
after 1582, noop 22579 22579 0 4.4 225.8 1.0X
before 1582, noop 10092 10092 0 9.9 100.9 2.2X
after 1582, rebase off 34532 34532 0 2.9 345.3 0.7X
after 1582, rebase on 37458 37458 0 2.7 374.6 0.6X
before 1582, rebase off 22423 22423 0 4.5 224.2 1.0X
before 1582, rebase on 30290 30290 0 3.3 302.9 0.7X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 12593 12653 52 7.9 125.9 1.0X
after 1582, vec off, rebase on 13350 13489 121 7.5 133.5 0.9X
after 1582, vec on, rebase off 3665 3681 25 27.3 36.6 3.4X
after 1582, vec on, rebase on 5193 5210 16 19.3 51.9 2.4X
before 1582, vec off, rebase off 13023 13059 32 7.7 130.2 1.0X
before 1582, vec off, rebase on 13855 13937 115 7.2 138.6 0.9X
before 1582, vec on, rebase off 3651 3665 12 27.4 36.5 3.4X
before 1582, vec on, rebase on 5623 5671 45 17.8 56.2 2.2X
after 1582, vec off, rebase off 12547 12664 103 8.0 125.5 1.0X
after 1582, vec off, rebase on 15870 15948 124 6.3 158.7 0.8X
after 1582, vec on, rebase off 3666 3712 40 27.3 36.7 3.4X
after 1582, vec on, rebase on 7014 7061 44 14.3 70.1 1.8X
before 1582, vec off, rebase off 13188 13240 67 7.6 131.9 1.0X
before 1582, vec off, rebase on 24141 24369 215 4.1 241.4 0.5X
before 1582, vec on, rebase off 3635 3652 16 27.5 36.3 3.5X
before 1582, vec on, rebase on 12307 12329 25 8.1 123.1 1.0X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2798 2798 0 35.7 28.0 1.0X
before 1582, noop 2955 2955 0 33.8 29.6 0.9X
after 1582, rebase off 15889 15889 0 6.3 158.9 0.2X
after 1582, rebase on 84247 84247 0 1.2 842.5 0.0X
before 1582, rebase off 16134 16134 0 6.2 161.3 0.2X
before 1582, rebase on 100006 100006 0 1.0 1000.1 0.0X
after 1582, noop 2704 2704 0 37.0 27.0 1.0X
before 1582, noop 2817 2817 0 35.5 28.2 1.0X
after 1582, rebase off 17018 17018 0 5.9 170.2 0.2X
after 1582, rebase on 20671 20671 0 4.8 206.7 0.1X
before 1582, rebase off 17308 17308 0 5.8 173.1 0.2X
before 1582, rebase on 30178 30178 0 3.3 301.8 0.1X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 14920 15045 116 6.7 149.2 1.0X
after 1582, vec off, rebase on 55062 55171 140 1.8 550.6 0.3X
after 1582, vec on, rebase off 4871 4952 72 20.5 48.7 3.1X
after 1582, vec on, rebase on 44955 44981 23 2.2 449.5 0.3X
before 1582, vec off, rebase off 15236 15386 142 6.6 152.4 1.0X
before 1582, vec off, rebase on 57290 57368 79 1.7 572.9 0.3X
before 1582, vec on, rebase off 4919 4930 15 20.3 49.2 3.0X
before 1582, vec on, rebase on 47351 47713 400 2.1 473.5 0.3X
after 1582, vec off, rebase off 15070 15152 140 6.6 150.7 1.0X
after 1582, vec off, rebase on 19237 19345 93 5.2 192.4 0.8X
after 1582, vec on, rebase off 4891 4947 52 20.4 48.9 3.1X
after 1582, vec on, rebase on 10663 10699 32 9.4 106.6 1.4X
before 1582, vec off, rebase off 15128 15223 88 6.6 151.3 1.0X
before 1582, vec off, rebase on 28453 28509 49 3.5 284.5 0.5X
before 1582, vec on, rebase off 4872 4897 25 20.5 48.7 3.1X
before 1582, vec on, rebase on 20223 20249 31 4.9 202.2 0.7X


================================================================================================
Expand All @@ -59,36 +59,36 @@ OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 9451 9451 0 10.6 94.5 1.0X
before 1582, noop 9765 9765 0 10.2 97.7 1.0X
after 1582 18722 18722 0 5.3 187.2 0.5X
before 1582 18864 18864 0 5.3 188.6 0.5X
after 1582, noop 22417 22417 0 4.5 224.2 1.0X
before 1582, noop 10126 10126 0 9.9 101.3 2.2X
after 1582 37292 37292 0 2.7 372.9 0.6X
before 1582 41610 41610 0 2.4 416.1 0.5X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 24897 25095 247 4.0 249.0 1.0X
after 1582, vec on 3719 3780 84 26.9 37.2 6.7X
before 1582, vec off 31290 31347 50 3.2 312.9 0.8X
before 1582, vec on 4166 4188 25 24.0 41.7 6.0X
after 1582, vec off 41899 41909 14 2.4 419.0 1.0X
after 1582, vec on 5894 5976 121 17.0 58.9 7.1X
before 1582, vec off 39192 39652 687 2.6 391.9 1.1X
before 1582, vec on 11579 11728 143 8.6 115.8 3.6X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2882 2882 0 34.7 28.8 1.0X
before 1582, noop 2991 2991 0 33.4 29.9 1.0X
after 1582 53951 53951 0 1.9 539.5 0.1X
before 1582 54276 54276 0 1.8 542.8 0.1X
after 1582, noop 2804 2804 0 35.7 28.0 1.0X
before 1582, noop 2806 2806 0 35.6 28.1 1.0X
after 1582 66893 66893 0 1.5 668.9 0.0X
before 1582 65435 65435 0 1.5 654.3 0.0X

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 41411 41514 97 2.4 414.1 1.0X
after 1582, vec on 32163 32201 36 3.1 321.6 1.3X
before 1582, vec off 43013 43111 131 2.3 430.1 1.0X
before 1582, vec on 34114 34152 45 2.9 341.1 1.2X
after 1582, vec off 50935 50994 89 2.0 509.4 1.0X
after 1582, vec on 41139 41325 185 2.4 411.4 1.2X
before 1582, vec off 53840 53990 168 1.9 538.4 0.9X
before 1582, vec on 44715 44778 61 2.2 447.1 1.1X