Skip to content

[SPARK-4502][SQL] Parquet nested column pruning - foundation #21320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mallman wants to merge 24 commits into from
Closed

[SPARK-4502][SQL] Parquet nested column pruning - foundation #21320

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6e41081
[SPARK-4502][SQL] Parquet nested column pruning
Jun 24, 2016
0d0e8a0
Refactor SelectedFieldSuite to make its tests simpler and more
mallman Jun 4, 2018
44e78cb
Remove test "select function over nested data" of unknown origin and
mallman Jun 4, 2018
9488cb5
Improve readability of ParquetSchemaPruning and
mallman Jun 4, 2018
f3735b1
Don't handle non-data-field partition column names specially when
mallman Jun 4, 2018
2120ab5
Add test coverage for ParquetSchemaPruning for partitioned tables whose
mallman Jun 4, 2018
8d53bbd
Remove the ColumnarFileFormat type to put it in another PR
mallman Jun 12, 2018
e213471
Add test coverage for the enhancements to "is not null" constraint
mallman Jun 12, 2018
9e6ef5f
Revert changes to QueryPlanConstraints.scala and basicPhysicalOperato…
mallman Jun 24, 2018
e6ea9c2
Revert a whitespace change in DataSourceScanExec.scala
mallman Jun 24, 2018
2d02ab3
Remove modifications to ParquetFileFormat.scala and
mallman Jul 21, 2018
cfffc95
PR review: simplify some syntax and add a code doc
mallman Jul 21, 2018
2779351
When creating a pruned schema by merging an array of root structs, sort
mallman Jul 28, 2018
9329f77
Re-enable ignored test in ParquetSchemaPruningSuite.scala that is
mallman Aug 4, 2018
ec313c1
Enable schema pruning by default
mallman Aug 4, 2018
42aff39
Revert "Enable schema pruning by default"
mallman Aug 5, 2018
71f4c7b
Add a method to not only check a query's scan schemata, but verify th…
mallman Aug 5, 2018
0e5594b
Revert "Revert "Enable schema pruning by default""
mallman Aug 9, 2018
1573ae8
Revert changes to ParquetTest.scala. I'm sure they were useful at some
mallman Aug 17, 2018
09dd655
Refactor code based on code review comments
ajacques Aug 2, 2018
61c7937
Update terminology for parquet-mr reader in
mallman Aug 20, 2018
97b3a51
Handle differences in letter case in columns and fields between query
mallman Aug 21, 2018
2711746
Add test permutations to the "testMixedCasePruning" method to test
mallman Aug 21, 2018
e6baf68
Disable SQL schema pruning by default and revert changes to
mallman Aug 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove the ColumnarFileFormat type to put it in another PR
  • Loading branch information
@mallman
mallman committed Aug 9, 2018
commit 8d53bbde33fd25adc6f74318c017c034c2cc6f66
15 changes: 1 addition & 14 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,20 +303,7 @@ case class FileSourceScanExec(
} getOrElse {
withOptPartitionCount
}

Copy link
Member

@gatorsmile gatorsmile Jun 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line?

val withOptColumnCount = relation.fileFormat match {
case columnar: ColumnarFileFormat =>
SparkSession
.getActiveSession
.map { sparkSession =>
val columnCount = columnar.columnCountForSchema(sparkSession, requiredSchema)
withSelectedBucketsCount + ("ColumnCount" -> columnCount.toString)
} getOrElse {
withSelectedBucketsCount
}
case _ => withSelectedBucketsCount
}
withOptColumnCount
withSelectedBucketsCount
}

private lazy val inputRDD: RDD[InternalRow] = {
Expand Down
32 changes: 0 additions & 32 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ColumnarFileFormat.scala
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions ...src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ import org.apache.spark.util.{SerializableConfiguration, ThreadUtils}

class ParquetFileFormat
extends FileFormat
with ColumnarFileFormat
with DataSourceRegister
with Logging
with Serializable {
Expand All @@ -73,14 +72,6 @@ class ParquetFileFormat

override def equals(other: Any): Boolean = other.isInstanceOf[ParquetFileFormat]

override def columnCountForSchema(sparkSession: SparkSession, readSchema: StructType): Int = {
val converter = new SparkToParquetSchemaConverter(
sparkSession.sessionState.conf.writeLegacyParquetFormat,
sparkSession.sessionState.conf.parquetOutputTimestampType)
val parquetSchema = converter.convert(readSchema)
parquetSchema.getPaths.size
}

override def prepareWrite(
sparkSession: SparkSession,
job: Job,
Expand Down