Skip to content

[SPARK-26427][BUILD] Upgrade Apache ORC to 1.5.4 #23364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dongjoon-hyun wants to merge 2 commits into from
Closed

[SPARK-26427][BUILD] Upgrade Apache ORC to 1.5.4 #23364

dongjoon-hyun wants to merge 2 commits into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Dec 21, 2018 •
edited
Loading

What changes were proposed in this pull request?

This PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. (Release Notes)

[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script

How was this patch tested?

Build and pass Jenkins.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 21, 2018

cc @srowen , @HyukjinKwon , @gatorsmile , @dbtsai

@dbtsai
Copy link
Member

dbtsai commented Dec 21, 2018

LGTM.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 21, 2018

Thank you for review, @dbtsai .

@SparkQA
Copy link

SparkQA commented Dec 21, 2018

Test build #100365 has finished for PR 23364 at commit 363d28c.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 21, 2018

Retest this please.

MaxGekk
MaxGekk reviewed Dec 21, 2018
View reviewed changes
@gatorsmile
Copy link
Member

gatorsmile commented Dec 21, 2018

Any ORC JIRA mentioned in the release impacts us?

@gatorsmile
Copy link
Member

gatorsmile commented Dec 21, 2018

Nice! https://issues.apache.org/jira/browse/SPARK-23458

gatorsmile
gatorsmile reviewed Dec 21, 2018
View reviewed changes
Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thanks! Hopefully, https://issues.apache.org/jira/browse/SPARK-23458 can be closed!

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 21, 2018

Thank you for reviews, @gatorsmile ! Only our resource leakage patches impact us. The other stuffs are related OpenJDK and related stuff.

@SparkQA
Copy link

SparkQA commented Dec 22, 2018

Test build #100375 has finished for PR 23364 at commit 363d28c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 22, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented Dec 22, 2018

Test build #100377 has finished for PR 23364 at commit 0b71045.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 22, 2018

So far, one python failure, HiveClientSuite failure, and the last one is irrelevant TimSort OOM failure.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.OutOfMemoryError: Java heap space
	at org.apache.spark.util.collection.TestTimSort.createArray(TestTimSort.java:56)
	at org.apache.spark.util.collection.TestTimSort.getTimSortBugTestSet(TestTimSort.java:43)

@SparkQA
Copy link

SparkQA commented Dec 22, 2018

Test build #100380 has finished for PR 23364 at commit 0b71045.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 22, 2018

Merged to master.

@asfgit asfgit closed this in 81addaa Dec 22, 2018
@dongjoon-hyun dongjoon-hyun deleted the SPARK-26427 branch December 22, 2018 08:47
@HyukjinKwon
Copy link
Member

HyukjinKwon commented Dec 22, 2018

LGTM as well

holdenk pushed a commit to holdenk/spark that referenced this pull request Jan 5, 2019
@dongjoon-hyun @holdenk
[SPARK-26427][BUILD] Upgrade Apache ORC to 1.5.4
ffd2ef6 
## What changes were proposed in this pull request?

This PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

## How was this patch tested?

Build and pass Jenkins.

Closes apache#23364 from dongjoon-hyun/SPARK-26427.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@dongjoon-hyun dongjoon-hyun mentioned this pull request Jan 25, 2019
Closed
dongjoon-hyun added a commit that referenced this pull request Jan 25, 2019
@dongjoon-hyun
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4
08b6379 
## What changes were proposed in this pull request?

This is a backport of #23364.

To make Apache Spark 2.4.1 more robust, this PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

## How was this patch tested?

Build and pass Jenkins.

Closes #23646 from dongjoon-hyun/SPARK-26427-2.4.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@dongjoon-hyun
[SPARK-26427][BUILD] Upgrade Apache ORC to 1.5.4
42f6596 
## What changes were proposed in this pull request?

This PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

## How was this patch tested?

Build and pass Jenkins.

Closes apache#23364 from dongjoon-hyun/SPARK-26427.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 23, 2019
@dongjoon-hyun @kai-chi
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4
d69b43b 
This is a backport of apache#23364.

To make Apache Spark 2.4.1 more robust, this PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

Build and pass Jenkins.

Closes apache#23646 from dongjoon-hyun/SPARK-26427-2.4.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Conflicts:
	pom.xml
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 25, 2019
@dongjoon-hyun @kai-chi
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4
8e648e3 
This is a backport of apache#23364.

To make Apache Spark 2.4.1 more robust, this PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

Build and pass Jenkins.

Closes apache#23646 from dongjoon-hyun/SPARK-26427-2.4.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Conflicts:
	pom.xml
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Aug 1, 2019
@dongjoon-hyun @kai-chi
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4
5d90583 
This is a backport of apache#23364.

To make Apache Spark 2.4.1 more robust, this PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

Build and pass Jenkins.

Closes apache#23646 from dongjoon-hyun/SPARK-26427-2.4.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MaxGekk MaxGekk MaxGekk left review comments

@gatorsmile gatorsmile gatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dongjoon-hyun @dbtsai @SparkQA @gatorsmile @HyukjinKwon @MaxGekk