Skip to content

[SPARK-11103][SQL] Filter applied on Merged Parquet shema with new column fail #9327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 2 commits into from
Closed

[SPARK-11103][SQL] Filter applied on Merged Parquet shema with new column fail #9327

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
85dadbc
[SPARK-11103][SQL] Disable predicate push down when using merged sche…
HyukjinKwon Oct 28, 2015
7007c21
[SPARK-11103][SQL] Add test code and update comments
HyukjinKwon Oct 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-11103][SQL] Disable predicate push down when using merged sche… 
…ma for Parquet.
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Oct 28, 2015
commit 85dadbcb458eec1067b6bae838ca930b8d958f4a
6 changes: 5 additions & 1 deletion ...e/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,10 @@ private[sql] class ParquetRelation(
val assumeBinaryIsString = sqlContext.conf.isParquetBinaryAsString
val assumeInt96IsTimestamp = sqlContext.conf.isParquetINT96AsTimestamp

// When using merged schema and the column of the given filter does not exist, Parquet emits
// an error which is an issue of Parquet (PARQUET-389).
val safeParquetFilterPushDown = !shouldMergeSchemas && parquetFilterPushDown

// Parquet row group size. We will use this value as the value for
// mapreduce.input.fileinputformat.split.minsize and mapred.min.split.size if the value
// of these flags are smaller than the parquet row group size.
Expand All @@ -305,7 +309,7 @@ private[sql] class ParquetRelation(
dataSchema,
parquetBlockSize,
useMetadataCache,
parquetFilterPushDown,
safeParquetFilterPushDown,
assumeBinaryIsString,
assumeInt96IsTimestamp) _

Expand Down