Quote escape configured to double quote (quote value) character if escape not detected

environment:
univocity-parsers 2.8.3
AdoptOpenJDK 11.0.8+10

Hi,

I have a simple configuration that reads a few lines from a csv file. The content in the file does not have enough information to detect and set the quote character and the quote escape character. However, after parsing, the detected csv format for quote escape is incorrect after I have explicitly set the value and the data has not driven any change through detection.
When quote escape is defaulted to backslash, the result csv format is always double quote.

CsvParserSettings settings = new CsvParserSettings();
settings.getFormat().setQuoteEscape('\\');

CsvParser parser = new CsvParser(settings);
parseAll(...)
...
CsvFormat csvFormat = parser.getDetectedFormat();

I poked around a bit and I think there is an issue in the CsvFormatDetector at line 246,
where
char quoteEscape = max(escape, totals, quote);
should be
char quoteEscape = max(escape, totals, quoteEscape);

If I understand the logic correctly, it is wanting to use the escape most used or default to a value if no detection. The value should be the quote escape, and not the quote.

Thanks,
James

[jbax]

Can you try version 2.9.0? A few changes were made there in regards to format detection. If it doesn't help can you please submit a reproducible unit test? Can't do much based on the description alone. Thanks!

…

On Mon, Sep 7, 2020, 10:12 PM James McClure ***@***.***> wrote: environment: univocity-parsers 2.8.3 AdoptOpenJDK 11.0.8+10 Hi, I have a simple configuration that reads a few lines from a csv file. The content in the file does not have enough information to detect and set the quote character and the quote escape character. However, after parsing, the detected csv format for quote escape is incorrect after I have explicitly set the value and the data has not driven any change through detection. When quote escape is defaulted to backslash, the result csv format is always double quote. CsvParserSettings settings = new CsvParserSettings(); settings.getFormat().setQuoteEscape('\\'); CsvParser parser = new CsvParser(settings); parseAll(...) ... CsvFormat csvFormat = parser.getDetectedFormat(); I poked around a bit and I think there is an issue in the CsvFormatDetector at line 246, where char quoteEscape = max(escape, totals, quote); should be char quoteEscape = max(escape, totals, quoteEscape); If I understand the logic correctly, it is wanting to use the escape most used or default to a value if no detection. The value should be the quote escape, and not the quote. Thanks, James — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#414>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWFQPQGESJOVUXRXF55BW3SETIJ3ANCNFSM4Q6D5OCQ> .

I am getting the same result with version 2.9.0 as with 2.8.3. Maybe my use of the library is not correct? JUnit 5 test attached.

CsvParseTest.java.zip

[jbax]

Thanks for the example. I've fixed and released a 2.9.1-SNAPSHOT build you can use to test.
Thank you for using our parsers!

Excellent csv parser! Thanks for the quick turnaround.

[maayan-samari]

hi @jbax line 263 in 2.9.1 is still buggy for my use case
char quoteEscape = doubleQuoteCount == 0 && singleQuoteCount == 0 ? suggestedQuoteEscape : max(escape, totals, quote);

for simple CSV with this kind of column value: "hi.", where I do have double quote, and no setting for escape quote is done, I get . (dot) identified as quote escape due to line 134 that in the first place adds any non letter no digit to the escape list:

if (!Character.isLetterOrDigit(prev) && prev != '\n' && prev != '\r') { increment(escape, prev);

Therefore it parsed the file wrongly.

I think that in case of any quote, still need to take the user's setting, I also tried settings.getFormat().setQuoteEscape('\\'); but it is not taken.

should I open a different issue? any workaround for it?
Thank you!