Skip to content

Commit d3944b6

Browse files
lianchengrxin
authored andcommitted
[Minor] [SQL] Follow-up of PR #5210
This PR addresses rxin's comments in PR #5210. <!-- Reviewable:start --> [<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5219) <!-- Reviewable:end --> Author: Cheng Lian <lian@databricks.com> Closes #5219 from liancheng/spark-6554-followup and squashes the following commits: 41f3a09 [Cheng Lian] Addresses comments in #5210
1 parent 251698f commit d3944b6

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -434,17 +434,18 @@ private[sql] case class ParquetRelation2(
434434
FileInputFormat.setInputPaths(job, selectedFiles.map(_.getPath): _*)
435435
}
436436

437-
// Push down filters when possible. Notice that not all filters can be converted to Parquet
438-
// filter predicate. Here we try to convert each individual predicate and only collect those
439-
// convertible ones.
437+
// Try to push down filters when filter push-down is enabled.
440438
if (sqlContext.conf.parquetFilterPushDown) {
439+
val partitionColNames = partitionColumns.map(_.name).toSet
441440
predicates
442441
// Don't push down predicates which reference partition columns
443442
.filter { pred =>
444-
val partitionColNames = partitionColumns.map(_.name).toSet
445443
val referencedColNames = pred.references.map(_.name).toSet
446444
referencedColNames.intersect(partitionColNames).isEmpty
447445
}
446+
// Collects all converted Parquet filter predicates. Notice that not all predicates can be
447+
// converted (`ParquetFilters.createFilter` returns an `Option`). That's why a `flatMap`
448+
// is used here.
448449
.flatMap(ParquetFilters.createFilter)
449450
.reduceOption(FilterApi.and)
450451
.foreach(ParquetInputFormat.setFilterPredicate(jobConf, _))

0 commit comments

Comments
 (0)