Skip to content

[SPARK-4502][SQL] Parquet nested column pruning - foundation #21320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mallman wants to merge 24 commits into from
Closed

[SPARK-4502][SQL] Parquet nested column pruning - foundation #21320

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6e41081
[SPARK-4502][SQL] Parquet nested column pruning
Jun 24, 2016
0d0e8a0
Refactor SelectedFieldSuite to make its tests simpler and more
mallman Jun 4, 2018
44e78cb
Remove test "select function over nested data" of unknown origin and
mallman Jun 4, 2018
9488cb5
Improve readability of ParquetSchemaPruning and
mallman Jun 4, 2018
f3735b1
Don't handle non-data-field partition column names specially when
mallman Jun 4, 2018
2120ab5
Add test coverage for ParquetSchemaPruning for partitioned tables whose
mallman Jun 4, 2018
8d53bbd
Remove the ColumnarFileFormat type to put it in another PR
mallman Jun 12, 2018
e213471
Add test coverage for the enhancements to "is not null" constraint
mallman Jun 12, 2018
9e6ef5f
Revert changes to QueryPlanConstraints.scala and basicPhysicalOperato…
mallman Jun 24, 2018
e6ea9c2
Revert a whitespace change in DataSourceScanExec.scala
mallman Jun 24, 2018
2d02ab3
Remove modifications to ParquetFileFormat.scala and
mallman Jul 21, 2018
cfffc95
PR review: simplify some syntax and add a code doc
mallman Jul 21, 2018
2779351
When creating a pruned schema by merging an array of root structs, sort
mallman Jul 28, 2018
9329f77
Re-enable ignored test in ParquetSchemaPruningSuite.scala that is
mallman Aug 4, 2018
ec313c1
Enable schema pruning by default
mallman Aug 4, 2018
42aff39
Revert "Enable schema pruning by default"
mallman Aug 5, 2018
71f4c7b
Add a method to not only check a query's scan schemata, but verify th…
mallman Aug 5, 2018
0e5594b
Revert "Revert "Enable schema pruning by default""
mallman Aug 9, 2018
1573ae8
Revert changes to ParquetTest.scala. I'm sure they were useful at some
mallman Aug 17, 2018
09dd655
Refactor code based on code review comments
ajacques Aug 2, 2018
61c7937
Update terminology for parquet-mr reader in
mallman Aug 20, 2018
97b3a51
Handle differences in letter case in columns and fields between query
mallman Aug 21, 2018
2711746
Add test permutations to the "testMixedCasePruning" method to test
mallman Aug 21, 2018
e6baf68
Disable SQL schema pruning by default and revert changes to
mallman Aug 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert a whitespace change in DataSourceScanExec.scala
  • Loading branch information
@mallman
mallman committed Aug 9, 2018
commit e6ea9c22bd788cce2b3d828e5b56c0b73662ff54
1 change: 1 addition & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,7 @@ case class FileSourceScanExec(
} getOrElse {
withOptPartitionCount
}

withSelectedBucketsCount
}

Expand Down