Fix extra quotation marks issue #582
Conversation
|
@srujanjoshi @donggrant The issue I was trying to solve with the PR is that unwanted quotation marks will be added to the output csv file if there’re double quotation marks within a sentence. It comes in 2 folds: suppose we have the below csv file：
it will return something like:
because double quotation mark is the escape character for csv files, a double quotation mark needs a double quotation mark when writen to the csv file.
it will return something like:
which is due to how the csv sniffer handles the input file. It has this minor glitch when double quotation marks are found inside a sentence already wrapped in double quotation marks. When it reads I wasn’t able to find a way to fix the csv sniffer, so I thought we could just preprocess the input file by replacing the double quotation marks with single quotation marks. But this turned out to be a messy solution because we need to create a temp file to store the processed input for the csv sniffer, overwriting the original input file may be risky. |
|
Some more cases with the bug:
|
|
see #623 for a newer solution |
What does this PR do?
Summary
When augmenting from CSV files, we use the CSV sniffer to try automatically inferring the correct CSV format. This works most of the time but does not function properly when there're more than 2 double quotation marks within a line, due to the fact that double quotation marks are recognized as escape characters by CSV sniffer.
The workaround first goes through the input CSV file, replaces double quotation marks with single quotation marks, and then creates a temporary CSV file to store the modified inputs. We need to create the temp file because we couldn't directly provide the modified inputs (list of strings) to the CSV sniffer, so the modified lines need to be saved to a CSV file and that file will be read by the sniffer.
Users will be warned on which lines of the original input file are modified.
Additions