Skip to content

[SPARK-33463][SQL] Keep Job Id during incremental collect in Spark Thrift Server #30390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gumartinm wants to merge 4 commits into from
Closed

[SPARK-33463][SQL] Keep Job Id during incremental collect in Spark Thrift Server #30390

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64dbb73
[SPARK-33463][SQL] Spark Thrift Server, keep Job Id when using increm…
gumartinm Nov 16, 2020
dde6d91
[SPARK-33463][SQL] getNextRowSetInternal: fixing private function scope
gumartinm Nov 16, 2020
ac2791a
[SPARK-33463][SQL] getNextRowSetInternal: fixing private function scope
gumartinm Nov 16, 2020
327fdf8
Using substitutorStatement as private class field
gumartinm Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions ...rc/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ private[hive] class SparkExecuteStatementOperation(
}
}

private val substitutorStatement = SQLConf.withExistingConf(sqlContext.conf) {
new VariableSubstitution().substitute(statement)
}

private var result: DataFrame = _

// We cache the returned rows to get iterators again in case the user wants to use FETCH_FIRST.
Expand Down Expand Up @@ -126,6 +130,17 @@ private[hive] class SparkExecuteStatementOperation(
}

def getNextRowSet(order: FetchOrientation, maxRowsL: Long): RowSet = withLocalProperties {
try {
sqlContext.sparkContext.setJobGroup(statementId, substitutorStatement)
getNextRowSetInternal(order, maxRowsL)
} finally {
sqlContext.sparkContext.clearJobGroup()
}
}

private def getNextRowSetInternal(
order: FetchOrientation,
maxRowsL: Long): RowSet = withLocalProperties {
log.info(s"Received getNextRowSet request order=${order} and maxRowsL=${maxRowsL} " +
s"with ${statementId}")
validateDefaultFetchOrientation(order)
Expand Down Expand Up @@ -306,9 +321,6 @@ private[hive] class SparkExecuteStatementOperation(
parentSession.getSessionState.getConf.setClassLoader(executionHiveClassLoader)
}

val substitutorStatement = SQLConf.withExistingConf(sqlContext.conf) {
new VariableSubstitution().substitute(statement)
}
sqlContext.sparkContext.setJobGroup(statementId, substitutorStatement)
result = sqlContext.sql(statement)
logDebug(result.queryExecution.toString())
Expand Down