Skip to content

[SPARK-48573][SQL] Upgrade ICU version #47011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mihailomilosevic2001 wants to merge 18 commits into from
Closed

[SPARK-48573][SQL] Upgrade ICU version #47011

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d54453e
Fix OracleIntegrationSuite for JDK8
mihailomilosevic2001 May 30, 2024
7d00764
Revert "Fix OracleIntegrationSuite for JDK8"
mihailomilosevic2001 May 30, 2024
ad01ff9
Merge remote-tracking branch 'upstream/master'
mihailomilosevic2001 Jun 4, 2024
1592c78
Merge remote-tracking branch 'upstream/master'
mihailomilosevic2001 Jun 11, 2024
7fe2a4d
Merge remote-tracking branch 'upstream/master'
mihailomilosevic2001 Jun 14, 2024
8e811d4
Merge remote-tracking branch 'upstream/master'
mihailomilosevic2001 Jun 15, 2024
7586b5f
Merge remote-tracking branch 'upstream/master'
mihailomilosevic2001 Jun 18, 2024
fe94569
Upgrade icu version
mihailomilosevic2001 Jun 18, 2024
e042f7f
Fix deps file
mihailomilosevic2001 Jun 18, 2024
92b4d26
Regenerate golden file
mihailomilosevic2001 Jun 19, 2024
2de46e1
Merge branch 'apache:master' into upgrade-icu
mihailomilosevic2001 Jun 24, 2024
005d6db
Merge remote-tracking branch 'upstream/master' into upgrade-icu
mihailomilosevic2001 Jun 24, 2024
84ba74b
Add benchmarks
mihailomilosevic2001 Jun 24, 2024
9a115b3
Merge branch 'upgrade-icu' of github.com:mihailom-db/spark into upgra…
mihailomilosevic2001 Jun 24, 2024
7d8876e
Reverse benchmark order
mihailomilosevic2001 Jun 24, 2024
5a13a1f
Merge branch 'apache:master' into upgrade-icu
mihailomilosevic2001 Jun 24, 2024
5587aeb
Revert order change
mihailomilosevic2001 Jun 25, 2024
3d9505d
Merge branch 'upgrade-icu' of github.com:mihailom-db/spark into upgra…
mihailomilosevic2001 Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ hk2-locator/3.0.3//hk2-locator-3.0.3.jar
hk2-utils/3.0.3//hk2-utils-3.0.3.jar
httpclient/4.5.14//httpclient-4.5.14.jar
httpcore/4.4.16//httpcore-4.4.16.jar
icu4j/72.1//icu4j-72.1.jar
icu4j/75.1//icu4j-75.1.jar
ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.2//ivy-2.5.2.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@
<datasketches.version>6.0.0</datasketches.version>
<netty.version>4.1.110.Final</netty.version>
<netty-tcnative.version>2.0.65.Final</netty-tcnative.version>
<icu4j.version>72.1</icu4j.version>
<icu4j.version>75.1</icu4j.version>
<junit-jupiter.version>5.9.3</junit-jupiter.version>
<junit-platform.version>1.9.3</junit-platform.version>
<sbt-jupiter-interface.version>0.11.1</sbt-jupiter-interface.version>
Expand Down
48 changes: 24 additions & 24 deletions sql/core/benchmarks/CollationBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,53 @@ OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1355 1358 4 0.1 13551.1 1.0X
UTF8_LCASE 4983 4984 3 0.0 49826.4 0.3X
UNICODE 18212 18220 12 0.0 182120.9 0.1X
UNICODE_CI 17568 17577 14 0.0 175677.2 0.1X
UTF8_BINARY 1352 1352 1 0.1 13516.4 1.0X
UTF8_LCASE 4678 4693 21 0.0 46778.6 0.3X
UNICODE 17213 17223 13 0.0 172131.7 0.1X
UNICODE_CI 17101 17133 46 0.0 171009.6 0.1X

OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1772 1774 3 0.1 17722.3 1.0X
UTF8_LCASE 4365 4365 0 0.0 43649.6 0.4X
UNICODE 16538 16544 9 0.0 165375.5 0.1X
UNICODE_CI 16296 16305 12 0.0 162961.9 0.1X
UTF8_BINARY 1775 1775 0 0.1 17749.8 1.0X
UTF8_LCASE 5342 5367 35 0.0 53423.0 0.3X
UNICODE 17011 17020 12 0.0 170110.1 0.1X
UNICODE_CI 16734 16760 37 0.0 167338.2 0.1X

OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 7279 7280 1 0.0 72791.2 1.0X
UTF8_LCASE 18538 18543 6 0.0 185381.0 0.4X
UNICODE 71514 71520 8 0.0 715144.6 0.1X
UNICODE_CI 60488 60488 0 0.0 604880.9 0.1X
UTF8_BINARY 7253 7256 4 0.0 72529.6 1.0X
UTF8_LCASE 16634 16676 59 0.0 166342.5 0.4X
UNICODE 66146 66163 23 0.0 661461.1 0.1X
UNICODE_CI 54563 54606 62 0.0 545625.5 0.1X

OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 7516 7519 4 0.0 75162.9 1.0X
UTF8_LCASE 120330 120338 12 0.0 1203299.2 0.1X
UNICODE 371784 371946 228 0.0 3717840.7 0.0X
UNICODE_CI 427401 427547 207 0.0 4274009.0 0.0X
UTF8_BINARY 6570 6577 10 0.0 65696.6 1.0X
UTF8_LCASE 120073 120137 91 0.0 1200726.4 0.1X
UNICODE 364027 364291 374 0.0 3640267.9 0.0X
UNICODE_CI 421444 422138 981 0.0 4214438.7 0.0X

OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6504 6507 3 0.0 65044.6 1.0X
UTF8_LCASE 60331 60359 40 0.0 603313.9 0.1X
UNICODE 369394 369404 13 0.0 3693943.0 0.0X
UNICODE_CI 427382 427421 55 0.0 4273819.7 0.0X
UTF8_BINARY 6573 6596 32 0.0 65733.4 1.0X
UTF8_LCASE 60284 60293 12 0.0 602844.4 0.1X
UNICODE 363685 364220 757 0.0 3636848.4 0.0X
UNICODE_CI 422761 423000 337 0.0 4227611.0 0.0X

OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6600 6601 1 0.0 66002.7 1.0X
UTF8_LCASE 58723 58751 39 0.0 587230.1 0.1X
UNICODE 379668 379789 172 0.0 3796677.7 0.0X
UNICODE_CI 437119 437194 106 0.0 4371189.5 0.0X
UTF8_BINARY 6507 6508 1 0.0 65068.3 1.0X
UTF8_LCASE 59098 59118 28 0.0 590983.6 0.1X
UNICODE 378437 378790 499 0.0 3784367.4 0.0X
UNICODE_CI 433987 434294 435 0.0 4339869.2 0.0X

48 changes: 24 additions & 24 deletions sql/core/benchmarks/CollationBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,53 @@ OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1370 1370 1 0.1 13698.4 1.0X
UTF8_LCASE 4836 4836 0 0.0 48359.5 0.3X
UNICODE 19239 19271 45 0.0 192391.8 0.1X
UNICODE_CI 18936 18954 25 0.0 189362.4 0.1X
UTF8_BINARY 1372 1374 3 0.1 13718.1 1.0X
UTF8_LCASE 6311 6311 0 0.0 63106.7 0.2X
UNICODE 19273 19300 37 0.0 192731.3 0.1X
UNICODE_CI 18991 18998 10 0.0 189906.3 0.1X

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1726 1727 1 0.1 17260.4 1.0X
UTF8_LCASE 6293 6304 16 0.0 62927.1 0.3X
UNICODE 18677 18679 4 0.0 186768.3 0.1X
UNICODE_CI 18488 18504 23 0.0 184879.6 0.1X
UTF8_BINARY 1725 1726 2 0.1 17249.0 1.0X
UTF8_LCASE 5806 5828 31 0.0 58061.5 0.3X
UNICODE 19105 19111 8 0.0 191051.5 0.1X
UNICODE_CI 18991 18996 7 0.0 189913.3 0.1X

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 3028 3029 1 0.0 30283.4 1.0X
UTF8_LCASE 19773 19830 81 0.0 197726.4 0.2X
UNICODE 68565 68594 41 0.0 685646.9 0.0X
UNICODE_CI 53100 53101 2 0.0 530996.0 0.1X
UTF8_BINARY 3019 3021 3 0.0 30194.7 1.0X
UTF8_LCASE 19437 19439 3 0.0 194372.6 0.2X
UNICODE 63550 63568 25 0.0 635504.3 0.0X
UNICODE_CI 57839 57866 39 0.0 578385.0 0.1X

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 7024 7026 3 0.0 70244.6 1.0X
UTF8_LCASE 118693 118703 15 0.0 1186926.5 0.1X
UNICODE 385409 386299 1257 0.0 3854093.7 0.0X
UNICODE_CI 434618 435527 1285 0.0 4346181.0 0.0X
UTF8_BINARY 7022 7023 2 0.0 70216.8 1.0X
UTF8_LCASE 118836 118887 72 0.0 1188364.9 0.1X
UNICODE 376381 376546 234 0.0 3763807.3 0.0X
UNICODE_CI 427858 427981 174 0.0 4278584.6 0.0X

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6069 6090 29 0.0 60691.9 1.0X
UTF8_LCASE 61809 61828 27 0.0 618094.5 0.1X
UNICODE 370523 371729 1705 0.0 3705229.7 0.0X
UNICODE_CI 435805 436945 1612 0.0 4358051.5 0.0X
UTF8_BINARY 6720 6723 5 0.0 67197.9 1.0X
UTF8_LCASE 67132 67177 63 0.0 671324.6 0.1X
UNICODE 368690 369292 852 0.0 3686899.6 0.0X
UNICODE_CI 431481 431583 144 0.0 4314814.9 0.0X

OpenJDK 64-Bit Server VM 17.0.11+9-LTS on Linux 6.5.0-1022-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6725 6732 10 0.0 67247.9 1.0X
UTF8_LCASE 54990 55010 28 0.0 549896.0 0.1X
UNICODE 380872 383258 3375 0.0 3808722.0 0.0X
UNICODE_CI 443911 444111 283 0.0 4439112.3 0.0X
UTF8_BINARY 7097 7099 3 0.0 70970.8 1.0X
UTF8_LCASE 57327 57351 35 0.0 573265.4 0.1X
UNICODE 375819 376473 924 0.0 3758191.8 0.0X
UNICODE_CI 445167 445212 64 0.0 4451666.7 0.0X

Loading