Skip to content

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 17 commits into from
Closed

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9a43e72
Add the support to specify compression codec for Parquet and Orc
HyukjinKwon Mar 2, 2016
9ce0ab7
Add some tests for testing compressions
HyukjinKwon Mar 2, 2016
59a77d4
Merge upstream
HyukjinKwon Mar 2, 2016
04e4a51
Add tests and some comments
HyukjinKwon Mar 2, 2016
a96e510
Correct comments
HyukjinKwon Mar 2, 2016
d107d50
Remove the duplicated test
HyukjinKwon Mar 2, 2016
9c53d10
Remove unused imports
HyukjinKwon Mar 2, 2016
12e7275
Add compression as an argument and make shorten names as lower cases
HyukjinKwon Mar 2, 2016
2304bfb
Resolve conflicts
HyukjinKwon Mar 3, 2016
baf7a63
Add some comments for consistent compression names and add none/uncom…
HyukjinKwon Mar 3, 2016
99c39c6
Use lower-cases for compression codec names
HyukjinKwon Mar 3, 2016
4e212b1
Update error meesages for TextSuite
HyukjinKwon Mar 3, 2016
4c1ffc5
Add some tests and functionality for expliclty setting no compression
HyukjinKwon Mar 3, 2016
29a8d61
Remove unused variable
HyukjinKwon Mar 3, 2016
2817c9a
Add removed test
HyukjinKwon Mar 3, 2016
d2ace37
Update test names
HyukjinKwon Mar 3, 2016
e4d4cfb
Make the options function
HyukjinKwon Mar 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Make the options function
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Mar 3, 2016
commit e4d4cfbc5a65ad95c4eee31ea5a2f09de7e5934a
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
}

test("SPARK-13543 Write the output as uncompressed via option") {
test("SPARK-13543 Write the output as uncompressed via option()") {
val clonedConf = new Configuration(hadoopConfiguration)
hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress", "true")
hadoopConfiguration
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1525,7 +1525,7 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}

test("SPARK-13543 Write the output as uncompressed via option") {
test("SPARK-13543 Write the output as uncompressed via option()") {
val clonedConf = new Configuration(hadoopConfiguration)
hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress", "true")
hadoopConfiguration
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ class TextSuite extends QueryTest with SharedSQLContext {
"Known codecs are"))
}

test("SPARK-13543 Write the output as uncompressed via option") {
test("SPARK-13543 Write the output as uncompressed via option()") {
val clonedConf = new Configuration(hadoopConfiguration)
hadoopConfiguration.set("mapreduce.output.fileoutputformat.compress", "true")
hadoopConfiguration
Expand Down