[SPARK-4502][SQL] Parquet nested column pruning - foundation#21320

Closed

mallman wants to merge 24 commits intoapache:masterfrom

VideoAmp:spark-4502-parquet_column_pruning-foundation

Commits on Aug 9, 2018

[SPARK-4502][SQL] Parquet nested column pruning

Michael Allman
authored and
mallman
committed
Refactor SelectedFieldSuite to make its tests simpler and more
mallman
committed
Remove test "select function over nested data" of unknown origin and
mallman
committed
Improve readability of ParquetSchemaPruning and
mallman
committed
Don't handle non-data-field partition column names specially when
mallman
committed
Add test coverage for ParquetSchemaPruning for partitioned tables whose
mallman
committed
Remove the ColumnarFileFormat type to put it in another PR
mallman
committed
Add test coverage for the enhancements to "is not null" constraint
mallman
committed
Revert changes to QueryPlanConstraints.scala and basicPhysicalOperators.scala. Remove related unit tests. This change removes support for schema pruning on filters involving attributes of complex type
mallman
committed
Revert a whitespace change in DataSourceScanExec.scala
mallman
committed
Remove modifications to ParquetFileFormat.scala and
mallman
committed
PR review: simplify some syntax and add a code doc
mallman
committed
When creating a pruned schema by merging an array of root structs, sort
mallman
committed
Re-enable ignored test in ParquetSchemaPruningSuite.scala that is
mallman
committed
Enable schema pruning by default
mallman
committed
Revert "Enable schema pruning by default"
mallman
committed
Add a method to not only check a query's scan schemata, but verify that it can
mallman
committed
Revert "Revert "Enable schema pruning by default""
mallman
committed

Commits on Aug 17, 2018

Revert changes to ParquetTest.scala. I'm sure they were useful at some
mallman
committed

Commits on Aug 20, 2018

Refactor code based on code review comments

ajacques
authored and
mallman
committed
Update terminology for parquet-mr reader in
mallman
committed

Commits on Aug 21, 2018

Commits on Aug 23, 2018

Disable SQL schema pruning by default and revert changes to
mallman
committed