Skip to content

[SPARK-16628][SQL] Translate file-based relation schema when file schema is inconsistent with catalog schema #14365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 8 commits into from
Closed

[SPARK-16628][SQL] Translate file-based relation schema when file schema is inconsistent with catalog schema #14365

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f9fab59
Fix the possible inconsistency between file schema and catalog schema…
viirya Jul 25, 2016
cc953a3
Fix test with correct require condition.
viirya Jul 26, 2016
0c17748
Add document.
viirya Jul 29, 2016
5cfdff1
Merge remote-tracking branch 'upstream/master' into fix-file-ds-incon…
viirya Aug 4, 2016
4125e6f
Do schema translation in OrcFileFormat.
viirya Dec 8, 2016
9cbb406
Merge remote-tracking branch 'upstream/master' into fix-file-ds-incon…
viirya Dec 8, 2016
2bb8368
Refine code comment.
viirya Dec 8, 2016
141cb1d
Add more test.
viirya Dec 19, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix test with correct require condition.
  • Loading branch information
@viirya
viirya committed Jul 26, 2016
commit cc953a3075675ecc97cf5342c6e29d940b1dfe3e
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/SchemaMapping.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ private[sql] trait SchemaMapping {
val catalogSchema: StructType

require(catalogSchema.length == 0 ||
dataSchema.length + partitionSchema.length == catalogSchema.length,
dataSchema.merge(partitionSchema).length == catalogSchema.length,
s"The data schema in files: $dataSchema plus the partition schema: $partitionSchema " +
s"should have the same number of fields with the schema in catalog: $catalogSchema.")

Expand Down