Skip to content

[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative #14034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gatorsmile wants to merge 16 commits into from
Closed

[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative #14034

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
1255968
fix
gatorsmile Jul 3, 2016
bdf4e56
reverted
gatorsmile Jul 3, 2016
3c402d3
address comments
gatorsmile Jul 4, 2016
5b36fbc
Detected foldable expression in limit
gatorsmile Jul 5, 2016
a2a828f
address comments.
gatorsmile Jul 5, 2016
f600ba4
address comments.
gatorsmile Jul 8, 2016
8fd72f6
address comments.
gatorsmile Jul 8, 2016
1abdbb9
address comments.
gatorsmile Jul 8, 2016
3036847
address comments.
gatorsmile Jul 8, 2016
d135b77
change another way
gatorsmile Jul 9, 2016
028aa79
revert it back
gatorsmile Jul 9, 2016
01137dc
address comments
gatorsmile Jul 10, 2016
0ebbdfe
Merge remote-tracking branch 'upstream/master' into limit
gatorsmile Jul 11, 2016
dec5ad9
address comments
gatorsmile Jul 11, 2016
2e6f8d8
address comments
gatorsmile Jul 11, 2016
d66870b
fix test case
gatorsmile Jul 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments.
  • Loading branch information
@gatorsmile
gatorsmile committed Jul 8, 2016
commit f600ba4f4455890b17080bc1cd3f647ec9023799
14 changes: 8 additions & 6 deletions ...st/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -660,11 +660,12 @@ case class GlobalLimit(limitExpr: Expression, child: LogicalPlan) extends UnaryN
}
override lazy val statistics: Statistics = {
val limit = limitExpr.eval().asInstanceOf[Int]
var sizeInBytes = (limit: Long) * output.map(a => a.dataType.defaultSize).sum
if (sizeInBytes == 0) {
val sizeInBytes = if (limit == 0) {
// sizeInBytes can't be zero, or sizeInBytes of BinaryNode will also be zero
// (product of children).
sizeInBytes = 1
1
} else {
(limit: Long) * output.map(a => a.dataType.defaultSize).sum
}
child.statistics.copy(sizeInBytes = sizeInBytes)
}
Expand All @@ -680,11 +681,12 @@ case class LocalLimit(limitExpr: Expression, child: LogicalPlan) extends UnaryNo
}
override lazy val statistics: Statistics = {
val limit = limitExpr.eval().asInstanceOf[Int]
var sizeInBytes = (limit: Long) * output.map(a => a.dataType.defaultSize).sum
if (sizeInBytes == 0) {
val sizeInBytes = if (limit == 0) {
// sizeInBytes can't be zero, or sizeInBytes of BinaryNode will also be zero
// (product of children).
sizeInBytes = 1
1
} else {
(limit: Long) * output.map(a => a.dataType.defaultSize).sum
}
child.statistics.copy(sizeInBytes = sizeInBytes)
}
Expand Down