[Feature][connector-file-local]localfile supports splitting and parallel reading of large files of csv, text, and json

[chl-wxp]

Issue

#10129

Check list

• Accuracy of file segmentation logic
• Unit test coverage of file splitting
• The rationality of the code architecture of the file splitting strategy

[LiJie20190102]

LGTM

[chl-wxp]

@zhangshenghang @corgy-w

[corgy-w]

First, you should update the documentation.
After enabling enable_split_file, the original skip_header behavior for the file no longer works as before. Is there any way to keep it compatible?

[chl-wxp]

First, you should update the documentation. After enabling enable_split_file, the original skip_header behavior for the file no longer works as before. Is there any way to keep it compatible?

1. Both Chinese and English documents have been added;
2. Regarding the previous logic of skipping a few lines in csv and text, I made compatibility in the code. When single file sharding is not enabled, the original skip logic will still be used. Although somewhat redundant, this is the best solution currently.

[hailin0]

add line count check

[hailin0]

add line count check

[hailin0]

Add escape characters to test: when the field content contains a newline character, the line is not truncated.

e.g(csv):

1,t1,"a
b"
2,t2,"c
d"

[chl-wxp]

Add escape characters to test: when the field content contains a newline character, the line is not truncated.

e.g(csv):

1,t1,"a
b"
2,t2,"c
d"

please see: #10185

[corgy-w]

Your change not only involves localfile, but also other files related documents need to be changed.

[chl-wxp]

Your change not only involves localfile, but also other files related documents need to be changed.

Connectors of different file types have different logic. Currently, only local implements AccordingToSplitSizeSplitStrategy, so only the localfile document is modified.