Skip to content

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 17 commits into from
Closed

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9a43e72
Add the support to specify compression codec for Parquet and Orc
HyukjinKwon Mar 2, 2016
9ce0ab7
Add some tests for testing compressions
HyukjinKwon Mar 2, 2016
59a77d4
Merge upstream
HyukjinKwon Mar 2, 2016
04e4a51
Add tests and some comments
HyukjinKwon Mar 2, 2016
a96e510
Correct comments
HyukjinKwon Mar 2, 2016
d107d50
Remove the duplicated test
HyukjinKwon Mar 2, 2016
9c53d10
Remove unused imports
HyukjinKwon Mar 2, 2016
12e7275
Add compression as an argument and make shorten names as lower cases
HyukjinKwon Mar 2, 2016
2304bfb
Resolve conflicts
HyukjinKwon Mar 3, 2016
baf7a63
Add some comments for consistent compression names and add none/uncom…
HyukjinKwon Mar 3, 2016
99c39c6
Use lower-cases for compression codec names
HyukjinKwon Mar 3, 2016
4e212b1
Update error meesages for TextSuite
HyukjinKwon Mar 3, 2016
4c1ffc5
Add some tests and functionality for expliclty setting no compression
HyukjinKwon Mar 3, 2016
29a8d61
Remove unused variable
HyukjinKwon Mar 3, 2016
2817c9a
Add removed test
HyukjinKwon Mar 3, 2016
d2ace37
Update test names
HyukjinKwon Mar 3, 2016
e4d4cfb
Make the options function
HyukjinKwon Mar 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove unused variable
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Mar 3, 2016
commit 29a8d6168f41bb52c8b5206c67174537a81e311c
3 changes: 1 addition & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ private[sql] class JSONOptions(
parameters.get("allowNonNumericNumbers").map(_.toBoolean).getOrElse(true)
val allowBackslashEscapingAnyCharacter =
parameters.get("allowBackslashEscapingAnyCharacter").map(_.toBoolean).getOrElse(false)
private val compressionName = parameters.get("compression")
val compressionCodec = compressionName.map(CompressionCodecs.getCodecClassName)
val compressionCodec = parameters.get("compression").map(CompressionCodecs.getCodecClassName)

/** Sets config options on a Jackson [[JsonFactory]]. */
def setJacksonOptions(factory: JsonFactory): Unit = {
Expand Down