Skip to content

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 17 commits into from
Closed

[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() #11464

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9a43e72
Add the support to specify compression codec for Parquet and Orc
HyukjinKwon Mar 2, 2016
9ce0ab7
Add some tests for testing compressions
HyukjinKwon Mar 2, 2016
59a77d4
Merge upstream
HyukjinKwon Mar 2, 2016
04e4a51
Add tests and some comments
HyukjinKwon Mar 2, 2016
a96e510
Correct comments
HyukjinKwon Mar 2, 2016
d107d50
Remove the duplicated test
HyukjinKwon Mar 2, 2016
9c53d10
Remove unused imports
HyukjinKwon Mar 2, 2016
12e7275
Add compression as an argument and make shorten names as lower cases
HyukjinKwon Mar 2, 2016
2304bfb
Resolve conflicts
HyukjinKwon Mar 3, 2016
baf7a63
Add some comments for consistent compression names and add none/uncom…
HyukjinKwon Mar 3, 2016
99c39c6
Use lower-cases for compression codec names
HyukjinKwon Mar 3, 2016
4e212b1
Update error meesages for TextSuite
HyukjinKwon Mar 3, 2016
4c1ffc5
Add some tests and functionality for expliclty setting no compression
HyukjinKwon Mar 3, 2016
29a8d61
Remove unused variable
HyukjinKwon Mar 3, 2016
2817c9a
Add removed test
HyukjinKwon Mar 3, 2016
d2ace37
Update test names
HyukjinKwon Mar 3, 2016
e4d4cfb
Make the options function
HyukjinKwon Mar 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Correct comments
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Mar 2, 2016
commit a96e510295af1ed2bae25a05ee36d8f97b3e6b41
2 changes: 1 addition & 1 deletion python/pyspark/sql/readwriter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -491,7 +491,7 @@ def parquet(self, path, mode=None, partitionBy=None):
* ``compression`` (default ``None``): compression codec to use when saving to file.
Copy link
Contributor

@rxin rxin Mar 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just put this as an argument?

This can be one of the known case-insensitive shorten names
(``uncompressed``, ``snappy``,``gzip``, and ``lzo``).
This will overwrite ``orc.compress``.
This will overwrite ``spark.sql.parquet.compression.codec``.

>>> df.write.parquet(os.path.join(tempfile.mkdtemp(), 'data'))
"""
Expand Down