Skip to content

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 20 commits into from
Closed

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
388be6d
[SPARK-27442][SQL] Remove check filename when reading data
AngersZhuuuu Jan 17, 2022
4bf6a19
Add UT
AngersZhuuuu Jan 18, 2022
f3f4e4c
Update FileFormat.scala
AngersZhuuuu Jan 18, 2022
3d7114a
follow comment
AngersZhuuuu Jan 18, 2022
59e9030
update
AngersZhuuuu Jan 18, 2022
d09f83d
Update DataSource.scala
AngersZhuuuu Jan 18, 2022
5a9d993
update
AngersZhuuuu Jan 19, 2022
5dc6741
Update FileFormat.scala
AngersZhuuuu Jan 19, 2022
b2d64af
follow comment
AngersZhuuuu Jan 19, 2022
b3f0f09
Update ParquetFileFormat.scala
AngersZhuuuu Jan 19, 2022
fe3aeb2
Follow comment
AngersZhuuuu Jan 19, 2022
e11bcdb
update
AngersZhuuuu Jan 19, 2022
5f3430f
Delete field_with_invalid_char.snappy.parquet
AngersZhuuuu Jan 19, 2022
bd5540e
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
310b00c
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e6253a2
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e3efd3f
remove related UT
AngersZhuuuu Jan 19, 2022
8a1dc91
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
8883634
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
3d4adf9
Update HiveDDLSuite.scala
AngersZhuuuu Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
follow comment
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Jan 18, 2022
commit 3d7114a267759f8696082e5ba9373919285e273f
Binary file modified sql/core/src/test/resources/test-data/field_with_invalid_char.snappy.parquet
View file
Open in desktop
Binary file not shown.
5 changes: 3 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4247,8 +4247,9 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
test("SPARK-27442: Spark support read parquet file with invalid char in field name") {
withResourceTempPath("test-data/field_with_invalid_char.snappy.parquet") { dir =>
Copy link
Contributor

@cloud-fan cloud-fan Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can write the parquet file in the test, instead of generating it ahead.

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can write the parquet file in the test, instead of generating it ahead.

Updated

val df = spark.read.parquet(dir.getAbsolutePath)
checkAnswer(df, Row(1, 2, 3) :: Nil)
assert(df.schema.names.sameElements(Array("max(t)", "a b", "{")))
checkAnswer(df, Row(1, 2, 3, 4, 5) :: Nil)
assert(df.schema.names.sameElements(Array("max(t)", "a b", "{", ".", "a.b")))
checkAnswer(df.select("`max(t)`", "`a b`", "`{`", "`.`", "`a.b`"), Row(1, 2, 3, 4, 5) :: Nil)
}
}
}
Expand Down