[SPARK-13792][SQL] Addendum: Fix Python API #13800
Conversation
|
cc @felixcheung |
| allowComments, allowUnquotedFieldNames, allowSingleQuotes, | ||
| allowNumericLeadingZero, allowBackslashEscapingAnyCharacter, | ||
| mode, columnNameOfCorruptRecord) | ||
| self._set_json_opts( |
There was a problem hiding this comment.
cc @tdas
previously these options were too susceptible to positional change in the arg list.
|
Test build #60907 has finished for PR 13800 at commit
|
|
Test build #3120 has finished for PR 13800 at commit
|
|
cc @davies any idea why this would fail python tests? |
|
Maybe #13793 broke master. It was sent to branch-1.6 but merged into master and branch-2.0. |
|
test this please |
|
Test build #60914 has finished for PR 13800 at commit
|
| if columnNameOfCorruptRecord is not None: | ||
| self.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord) | ||
|
|
||
| def _set_csv_opts(self, schema, sep, encoding, quote, escape, |
There was a problem hiding this comment.
This function could be:
def _set_csv_opts(self, schema, **options):
if schema is not None:
self.schema(schema)
for k in options:
if options[k] is not None:
self.option(k, options[k])
There was a problem hiding this comment.
That's a good idea. There are a bunch of things I want to do to the readwrite.py (mainly break it apart). I will do it there and merge this to unblock the rc.
## What changes were proposed in this pull request? This is a follow-up to #13795 to properly set CSV options in Python API. As part of this, I also make the Python option setting for both CSV and JSON more robust against positional errors. ## How was this patch tested? N/A Author: Reynold Xin <[email protected]> Closes #13800 from rxin/SPARK-13792-2. (cherry picked from commit 9333880) Signed-off-by: Reynold Xin <[email protected]>
4 participants
What changes were proposed in this pull request?
This is a follow-up to #13795 to properly set CSV options in Python API. As part of this, I also make the Python option setting for both CSV and JSON more robust against positional errors.
How was this patch tested?
N/A