Skip to content

[SPARK-25497][SQL] Limit operation within whole stage codegen should not consume all the inputs #22630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 10 commits into from
Closed

[SPARK-25497][SQL] Limit operation within whole stage codegen should not consume all the inputs #22630

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
13d882a
Limit operation within whole stage codegen should not consume all the…
cloud-fan Oct 3, 2018
0a6c79a
improve doc
cloud-fan Oct 4, 2018
51ce7be
address comment
cloud-fan Oct 5, 2018
2188b27
small update
cloud-fan Oct 5, 2018
e0bc621
address comments
cloud-fan Oct 5, 2018
dc2dfa5
add comment
cloud-fan Oct 5, 2018
e61078b
create a trait for blocking operators
cloud-fan Oct 8, 2018
9114107
address comment
cloud-fan Oct 8, 2018
eac31b2
address comments
cloud-fan Oct 8, 2018
4fc4301
address comment
cloud-fan Oct 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add comment
  • Loading branch information
@cloud-fan
cloud-fan committed Oct 5, 2018
commit dc2dfa5b42bcd597a0929aca16152e260c7f5122
3 changes: 3 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ trait BaseLimitExec extends UnaryExecNode with CodegenSupport {
}

override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
// The counter name is already obtained by the upstream operators via `limitNotReachedChecks`.
// Here we have to inline it to not change its name. This is fine as we won't have many limit
// operators in one query.
ctx.addMutableState(CodeGenerator.JAVA_INT, countTerm, forceInline = true, useFreshName = false)
Copy link
Contributor

@mgaido91 mgaido91 Oct 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to forceInline?

Copy link
Contributor Author

@cloud-fan cloud-fan Oct 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the counter variable name is decided before we obtain the CodegenContext. If we don't inline here, we need a way to notify the upstream operators about the counter name, which is hard to do.

s"""
| if ($countTerm < $limit) {
Expand Down