Skip to content

[SQL] SPARK-6981: Factor out SparkPlanner and QueryExecution from SQLContext #6122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
evacchi wants to merge 12 commits into from
Closed

[SQL] SPARK-6981: Factor out SparkPlanner and QueryExecution from SQLContext #6122

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
83668c1
Re-defines abstract class QueryPlanner as trait
evacchi May 13, 2015
45d6c9d
Make abstract class SparkStrategies a trait; remove self-type declara…
evacchi May 13, 2015
f622267
Remove 'this' from 'QueryExecution' type declaration
evacchi May 13, 2015
acf7e20
Refactor out QueryExecution from SQLContext and HiveContext
evacchi May 13, 2015
718ef71
Cast sqlContext to HiveContext in HiveContext.hivePlanner
evacchi May 13, 2015
fcb9ee4
Re-align QueryExecution name in PruningSuite
evacchi May 13, 2015
9038248
Fix build errors
evacchi May 13, 2015
c530eb7
Fix merge conflict with master
evacchi May 14, 2015
443d42f
add `lazy val = sqlContext.planner`
evacchi May 14, 2015
095355d
Merge branch 'master' into sqlctx-refactoring
evacchi May 15, 2015
ac03efe
Merge branch 'master' of https://github.com/apache/spark into sqlctx-…
evacchi May 18, 2015
a2ef558
Merge branch 'master' of https://github.com/apache/spark into sqlctx-…
evacchi May 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add lazy val = sqlContext.planner
  • Loading branch information
@evacchi
evacchi committed May 14, 2015
commit 443d42f44bd849228dbdeb70722983104107a49e
11 changes: 6 additions & 5 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
def unsafeEnabled: Boolean
def numPartitions: Int


lazy val planner = sqlContext.planner

/**
* Used to build table scan operators where complex projection and filtering are done using
* separate physical operators. This function returns the given scan operator with Project and
Expand Down Expand Up @@ -181,7 +182,7 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
if canBeCodeGened(
allAggregates(partialComputation) ++
allAggregates(rewrittenAggregateExpressions)) &&
sqlContext.planner.codegenEnabled =>
planner.codegenEnabled =>
execution.GeneratedAggregate(
partial = false,
namedGroupingAttributes,
Expand Down Expand Up @@ -300,7 +301,7 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
} else {
identity[Seq[Expression]] _
}
sqlContext.planner.pruneFilterProject(
planner.pruneFilterProject(
projectList,
filters,
prunePushedDownFilters,
Expand All @@ -316,7 +317,7 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
object InMemoryScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
case PhysicalOperation(projectList, filters, mem: InMemoryRelation) =>
sqlContext.planner.pruneFilterProject(
planner.pruneFilterProject(
projectList,
filters,
identity[Seq[Expression]], // All filters still need to be evaluated.
Expand All @@ -327,7 +328,7 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {

// Can we automate these 'pass through' operations?
object BasicOperators extends Strategy {
def numPartitions: Int = sqlContext.planner.numPartitions
def numPartitions: Int = planner.numPartitions

def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
case r: RunnableCommand => ExecutedCommand(r) :: Nil
Expand Down