Skip to content

[SPARK-39819][SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions) #37320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
beliefer wants to merge 11 commits into from
Closed

[SPARK-39819][SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions) #37320

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9351d1d
DS V2 aggregate push down can work with Top N or Paging (Sort with gr…
beliefer Jul 27, 2022
a920f45
Update code
beliefer Jul 27, 2022
b417cfc
Update code
beliefer Jul 29, 2022
3653fa1
Update code
beliefer Jul 29, 2022
c509e85
Update code
beliefer Jul 29, 2022
e3cd5cb
Update cod
beliefer Aug 2, 2022
e4d4eb5
Update code
beliefer Aug 2, 2022
515e95d
Update code
beliefer Aug 2, 2022
ecd7dcc
Update code
beliefer Aug 2, 2022
c6b20b0
Update code
beliefer Aug 6, 2022
293590b
Update code
beliefer Aug 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update code
  • Loading branch information
@beliefer
beliefer committed Aug 8, 2022
commit a920f4518971293188929b74c0b5dd9d5ddb6b2f
4 changes: 2 additions & 2 deletions ...src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,8 +413,8 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
case s @ Sort(order, _, operation @ PhysicalOperation(project, Nil, sHolder: ScanBuilderHolder))
// Without building the Scan, we do not know the resulting column names after aggregate
// push-down, and thus can't push down Top-N which needs to know the ordering column names.
// TODO: we can support simple cases like GROUP BY columns directly and ORDER BY the same
// columns, which we know the resulting column names: the original table columns.
// In particular, we push down the simple cases like GROUP BY expressions directly and
// ORDER BY the same expressions, which we know the original table columns.
if CollapseProject.canCollapseExpressions(order, project, alwaysInline = true) =>
val aliasMap = getAliasMap(project)
def findGroupExprForSortOrder(sortOrder: SortOrder): SortOrder = sortOrder match {
Expand Down