Skip to content

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 20 commits into from
Closed

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
388be6d
[SPARK-27442][SQL] Remove check filename when reading data
AngersZhuuuu Jan 17, 2022
4bf6a19
Add UT
AngersZhuuuu Jan 18, 2022
f3f4e4c
Update FileFormat.scala
AngersZhuuuu Jan 18, 2022
3d7114a
follow comment
AngersZhuuuu Jan 18, 2022
59e9030
update
AngersZhuuuu Jan 18, 2022
d09f83d
Update DataSource.scala
AngersZhuuuu Jan 18, 2022
5a9d993
update
AngersZhuuuu Jan 19, 2022
5dc6741
Update FileFormat.scala
AngersZhuuuu Jan 19, 2022
b2d64af
follow comment
AngersZhuuuu Jan 19, 2022
b3f0f09
Update ParquetFileFormat.scala
AngersZhuuuu Jan 19, 2022
fe3aeb2
Follow comment
AngersZhuuuu Jan 19, 2022
e11bcdb
update
AngersZhuuuu Jan 19, 2022
5f3430f
Delete field_with_invalid_char.snappy.parquet
AngersZhuuuu Jan 19, 2022
bd5540e
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
310b00c
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e6253a2
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e3efd3f
remove related UT
AngersZhuuuu Jan 19, 2022
8a1dc91
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
8883634
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
3d4adf9
Update HiveDDLSuite.scala
AngersZhuuuu Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update HiveDDLSuite.scala
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Jan 20, 2022
commit 88836342bf366f901c36462da9308c364f61552b
14 changes: 14 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2926,6 +2926,20 @@ class HiveDDLSuite
}
}

test("SPARK-33844: Insert overwrite directory should check schema too") {
withView("v") {
spark.range(1).createTempView("v")
withTempPath { path =>
val e = intercept[SparkException] {
spark.sql(s"INSERT OVERWRITE LOCAL DIRECTORY '${path.getCanonicalPath}' " +
s"STORED AS PARQUET SELECT ID, if(1=1, 1, 0), abs(id), '^-' FROM v")
}
assert(e.getCause.getCause.getMessage.contains(
"field ended by ';': expected ';' but got 'IF' at line 2: optional int32 (IF"))
}
}
}

test("SPARK-34261: Avoid side effect if create exists temporary function") {
withUserDefinedFunction("f1" -> true) {
sql("CREATE TEMPORARY FUNCTION f1 AS 'org.apache.hadoop.hive.ql.udf.UDFUUID'")
Expand Down