Skip to content

[SPARK-13306] [SQL] uncorrelated scalar subquery #11190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 9 commits into from
Closed

[SPARK-13306] [SQL] uncorrelated scalar subquery #11190

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0665a69
uncorrelated scalar subquery
Feb 12, 2016
236ac88
add missing file
Feb 12, 2016
016c36c
use broadcastTimeout
Feb 12, 2016
a4bae33
address comments
Feb 13, 2016
d0974cf
improve explain on subquery
Feb 20, 2016
3a8f08d
address comments
Feb 20, 2016
7596173
add tests and codegen
Feb 20, 2016
0034172
move wait subqueries into execute()/produce()
Feb 20, 2016
e082845
address comments
Feb 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments
  • Loading branch information
Davies Liu committed Feb 13, 2016
commit a4bae33c7f42660d13f5f8cf98d34bae0325f6ad
2 changes: 2 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ case class ScalarSubquery(query: LogicalPlan) extends SubqueryExpression with Co

override def withNewPlan(plan: LogicalPlan): ScalarSubquery = ScalarSubquery(plan)

// TODO: support sql()

// the first column in first row from `query`.
private var result: Any = null

Expand Down
48 changes: 43 additions & 5 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystQlSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.types.BooleanType
import org.apache.spark.unsafe.types.CalendarInterval

class CatalystQlSuite extends PlanTest {
Expand Down Expand Up @@ -203,10 +204,47 @@ class CatalystQlSuite extends PlanTest {
}

test("subquery") {
parser.parsePlan("select (select max(b) from s) ss from t")
parser.parsePlan("select * from t where a = (select b from s)")
parser.parsePlan("select * from t where a > (select b from s)")
parser.parsePlan("select * from t group by g having a = (select b from s)")
parser.parsePlan("select * from t group by g having a > (select b from s)")
comparePlans(
parser.parsePlan("select (select max(b) from s) ss from t"),
Project(
Copy link
Contributor Author

@davies davies Feb 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hvanhovell I'm going to remove these plan checking, it's very easy to break. The details of plan does not mean much, we will have other tests to verify the correctness.

UnresolvedAlias(
Alias(
ScalarSubquery(
Project(
UnresolvedAlias(
UnresolvedFunction("max", UnresolvedAttribute("b") :: Nil, false)) :: Nil,
UnresolvedRelation(TableIdentifier("s")))),
"ss")(ExprId(0))) :: Nil,
UnresolvedRelation(TableIdentifier("t"))))
comparePlans(
parser.parsePlan("select * from t where a = (select b from s)"),
Project(
UnresolvedAlias(
UnresolvedStar(None)) :: Nil,
Filter(
EqualTo(
UnresolvedAttribute("a"),
ScalarSubquery(
Project(
UnresolvedAlias(
UnresolvedAttribute("b")) :: Nil,
UnresolvedRelation(TableIdentifier("s"))))),
UnresolvedRelation(TableIdentifier("t")))))
comparePlans(
parser.parsePlan("select * from t group by g having a > (select b from s)"),
Filter(
Cast(
GreaterThan(
UnresolvedAttribute("a"),
ScalarSubquery(
Project(
UnresolvedAlias(
UnresolvedAttribute("b")) :: Nil,
UnresolvedRelation(TableIdentifier("s"))))),
BooleanType),
Aggregate(
UnresolvedAttribute("g") :: Nil,
UnresolvedAlias(UnresolvedStar(None)) :: Nil,
UnresolvedRelation(TableIdentifier("t")))))
}
}
4 changes: 4 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2115,6 +2115,10 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
"select a from (select 1 as a union all select 2 as a) t " +
"where a = (select max(b) from t2) ").collect()
Copy link
Contributor

@rxin rxin Feb 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we support nested subqueries, can we add a test case

Copy link
Contributor Author

@davies davies Feb 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

}

assertResult(Array(Row(3))) {
sql("select (select (select 1) + 1) + 1").collect()
}
}

test("SPARK-13056: Null in map value causes NPE") {
Expand Down