Skip to content

[SPARK-39469][SQL] Infer date type for CSV schema inference #36871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Jonathancui123 wants to merge 21 commits into from
Closed

[SPARK-39469][SQL] Infer date type for CSV schema inference #36871

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5048253
infer date type in csv
Jonathancui123 Jun 15, 2022
5058d92
fix
Jonathancui123 Jun 15, 2022
f16e5e1
parse dates from timestamp column
Jonathancui123 Jun 15, 2022
c2b5fdc
cleanup
Jonathancui123 Jun 15, 2022
9514c2c
test date and timestamp same format
Jonathancui123 Jun 15, 2022
be7aabd
e2e test WIP
Jonathancui123 Jun 16, 2022
db3b442
e2e test data
Jonathancui123 Jun 16, 2022
7d98686
fix e2e date test
Jonathancui123 Jun 17, 2022
966bdb6
added inferDate flag
Jonathancui123 Jun 21, 2022
638064b
Updated documentation and fixed tests
Jonathancui123 Jun 22, 2022
50a91a6
added new error for inferDate with legacy parser and fixed feedback
Jonathancui123 Jun 22, 2022
d71558d
Update CSVInferSchema.scala
Jonathancui123 Jun 23, 2022
601dfc8
Update CSVOptions.scala
Jonathancui123 Jun 23, 2022
5aa4ab6
change inferDate error type and improve docs
Jonathancui123 Jun 23, 2022
2282c59
fix error class
Jonathancui123 Jun 23, 2022
2484b77
fix sqlState code and typo
Jonathancui123 Jun 24, 2022
762e0d8
Update CSVInferSchema.scala
Jonathancui123 Jun 24, 2022
2c93af5
updated docs and test parsing dates in TimestampNTZ column
Jonathancui123 Jun 28, 2022
41fa8eb
Fix case where dateFormat is not specified
Jonathancui123 Jun 30, 2022
e1170d0
allow legacy parser with inferDate
Jonathancui123 Jul 12, 2022
1e8f938
Revert "allow legacy parser with inferDate"
Jonathancui123 Jul 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix sqlState code and typo
  • Loading branch information
@Jonathancui123
Jonathancui123 committed Jul 12, 2022
commit 2484b772efcaa63297d37c95b88c9528f23f55f2
2 changes: 1 addition & 1 deletion core/src/main/resources/error/error-classes.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"message" : [
"Cannot infer date in schema inference when LegacyTimeParserPolicy is \"LEGACY\". Legacy Date formatter does not support strict date format matching which is required to avoid inferring timestamps and other non-date entries to date."
],
"sqlState" : "22005"
"sqlState" : "22007"
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.analysis.TypeCoercion
import org.apache.spark.sql.catalyst.expressions.ExprUtils
import org.apache.spark.sql.catalyst.util.{DateFormatter, TimestampFormatter}
import org.apache.spark.sql.catalyst.util.LegacyDateFormats.{FAST_DATE_FORMAT, LENIENT_SIMPLE_DATE_FORMAT}
import org.apache.spark.sql.catalyst.util.LegacyDateFormats.{FAST_DATE_FORMAT}
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
Expand All @@ -49,7 +49,7 @@ class CSVInferSchema(val options: CSVOptions) extends Serializable {
private lazy val dateFormatter = DateFormatter(
options.dateFormatInRead,
options.locale,
legacyFormat = LENIENT_SIMPLE_DATE_FORMAT,
legacyFormat = FAST_DATE_FORMAT,
isParsing = true)

private val decimalParser = if (options.locale == Locale.US) {
Expand Down