Skip to content

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 20 commits into from
Closed

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
388be6d
[SPARK-27442][SQL] Remove check filename when reading data
AngersZhuuuu Jan 17, 2022
4bf6a19
Add UT
AngersZhuuuu Jan 18, 2022
f3f4e4c
Update FileFormat.scala
AngersZhuuuu Jan 18, 2022
3d7114a
follow comment
AngersZhuuuu Jan 18, 2022
59e9030
update
AngersZhuuuu Jan 18, 2022
d09f83d
Update DataSource.scala
AngersZhuuuu Jan 18, 2022
5a9d993
update
AngersZhuuuu Jan 19, 2022
5dc6741
Update FileFormat.scala
AngersZhuuuu Jan 19, 2022
b2d64af
follow comment
AngersZhuuuu Jan 19, 2022
b3f0f09
Update ParquetFileFormat.scala
AngersZhuuuu Jan 19, 2022
fe3aeb2
Follow comment
AngersZhuuuu Jan 19, 2022
e11bcdb
update
AngersZhuuuu Jan 19, 2022
5f3430f
Delete field_with_invalid_char.snappy.parquet
AngersZhuuuu Jan 19, 2022
bd5540e
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
310b00c
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e6253a2
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e3efd3f
remove related UT
AngersZhuuuu Jan 19, 2022
8a1dc91
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
8883634
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
3d4adf9
Update HiveDDLSuite.scala
AngersZhuuuu Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update HiveDDLSuite.scala
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Jan 20, 2022
commit 8a1dc9148817511553185d423f944d8ce21d42d5
38 changes: 0 additions & 38 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2926,44 +2926,6 @@ class HiveDDLSuite
}
}

test("SPARK-36201: Add check for inner field of parquet/orc schema") {
withView("v") {
spark.range(1).createTempView("v")
withTempPath { path =>
val e = intercept[AnalysisException] {
spark.sql(
s"""
|INSERT OVERWRITE LOCAL DIRECTORY '${path.getCanonicalPath}'
|STORED AS PARQUET
|SELECT
|NAMED_STRUCT('ID', ID, 'IF(ID=1,ID,0)', IF(ID=1,ID,0), 'B', ABS(ID)) AS col1
Copy link
Contributor

@cloud-fan cloud-fan Jan 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Hive Serde fail for this case as well?

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Hive Serde fail for this case as well?

Yea, updated

|FROM v
""".stripMargin)
}.getMessage
assert(e.contains("Column name \"IF(ID=1,ID,0)\" contains" +
" invalid character(s). Please use alias to rename it."))
}
}
}

test("SPARK-36312: ParquetWriteSupport should check inner field") {
Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Jan 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No exception thrown by hive serde, seems hive serder won't check inner fields. So just remove this . cc @cloud-fan

withView("v") {
spark.range(1).createTempView("v")
withTempPath { path =>
val e = intercept[AnalysisException] {
spark.sql(
"""
|SELECT
|NAMED_STRUCT('ID', ID, 'IF(ID=1,ID,0)', IF(ID=1,ID,0), 'B', ABS(ID)) AS col1
|FROM v
|""".stripMargin).write.mode(SaveMode.Overwrite).parquet(path.toString)
}.getMessage
assert(e.contains("Column name \"IF(ID=1,ID,0)\" contains" +
" invalid character(s). Please use alias to rename it."))
}
}
}

test("SPARK-34261: Avoid side effect if create exists temporary function") {
withUserDefinedFunction("f1" -> true) {
sql("CREATE TEMPORARY FUNCTION f1 AS 'org.apache.hadoop.hive.ql.udf.UDFUUID'")
Expand Down