Skip to content

[SPARK-33463][SQL] Keep Job Id during incremental collect in Spark Thrift Server #30390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gumartinm wants to merge 4 commits into from
Closed

[SPARK-33463][SQL] Keep Job Id during incremental collect in Spark Thrift Server #30390

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64dbb73
[SPARK-33463][SQL] Spark Thrift Server, keep Job Id when using increm…
gumartinm Nov 16, 2020
dde6d91
[SPARK-33463][SQL] getNextRowSetInternal: fixing private function scope
gumartinm Nov 16, 2020
ac2791a
[SPARK-33463][SQL] getNextRowSetInternal: fixing private function scope
gumartinm Nov 16, 2020
327fdf8
Using substitutorStatement as private class field
gumartinm Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-33463][SQL] Spark Thrift Server, keep Job Id when using increm… 
…ental collect
  • Loading branch information
@gumartinm
gumartinm committed Nov 16, 2020
commit 64dbb73b63fea54d8bf20a242d490f3f8fcc1204
9 changes: 9 additions & 0 deletions ...rc/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,15 @@ private[hive] class SparkExecuteStatementOperation(
}

def getNextRowSet(order: FetchOrientation, maxRowsL: Long): RowSet = withLocalProperties {
try {
sqlContext.sparkContext.setJobGroup(statementId, statement)
Copy link
Contributor

@juliuszsompolski juliuszsompolski Nov 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: in execute substitutorStatement is used as a description of the jobgroup. Maybe pull out substitutorStatement to be a class field and use it here as well?

Copy link
Contributor Author

@gumartinm gumartinm Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right!!! Thanks a lot for taking the time of revisioning my code.

I implemented the change, hopefully now it looks better. Looking forward to hearing your thoughts.

getNextRowSetInternal(order, maxRowsL)
} finally {
sqlContext.sparkContext.clearJobGroup()
}
}

def getNextRowSetInternal(order: FetchOrientation, maxRowsL: Long): RowSet = withLocalProperties {
log.info(s"Received getNextRowSet request order=${order} and maxRowsL=${maxRowsL} " +
s"with ${statementId}")
validateDefaultFetchOrientation(order)
Expand Down