Skip to content

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 20 commits into from
Closed

[SPARK-27442][SQL] Remove check field name when reading/writing data in parquet #35229

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
388be6d
[SPARK-27442][SQL] Remove check filename when reading data
AngersZhuuuu Jan 17, 2022
4bf6a19
Add UT
AngersZhuuuu Jan 18, 2022
f3f4e4c
Update FileFormat.scala
AngersZhuuuu Jan 18, 2022
3d7114a
follow comment
AngersZhuuuu Jan 18, 2022
59e9030
update
AngersZhuuuu Jan 18, 2022
d09f83d
Update DataSource.scala
AngersZhuuuu Jan 18, 2022
5a9d993
update
AngersZhuuuu Jan 19, 2022
5dc6741
Update FileFormat.scala
AngersZhuuuu Jan 19, 2022
b2d64af
follow comment
AngersZhuuuu Jan 19, 2022
b3f0f09
Update ParquetFileFormat.scala
AngersZhuuuu Jan 19, 2022
fe3aeb2
Follow comment
AngersZhuuuu Jan 19, 2022
e11bcdb
update
AngersZhuuuu Jan 19, 2022
5f3430f
Delete field_with_invalid_char.snappy.parquet
AngersZhuuuu Jan 19, 2022
bd5540e
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
310b00c
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e6253a2
Update SQLQuerySuite.scala
AngersZhuuuu Jan 19, 2022
e3efd3f
remove related UT
AngersZhuuuu Jan 19, 2022
8a1dc91
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
8883634
Update HiveDDLSuite.scala
AngersZhuuuu Jan 20, 2022
3d4adf9
Update HiveDDLSuite.scala
AngersZhuuuu Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update FileFormat.scala
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Jan 19, 2022
commit 5dc67416303c40ce70f55441c5d76dc7a4b0ffda
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ trait FileFormat {
def supportDataType(dataType: DataType): Boolean = true

/**
* Returns whether this format supports the given filed name in write path.
* Returns whether this format supports the given filed name in read/write path.
* By default all field name is supported.
*/
def supportFieldName(name: String): Boolean = true
Expand Down