Skip to content

[SPARK-43755][CONNECT] Move execution out of SparkExecutePlanStreamHandler and to a different thread #41315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
juliuszsompolski wants to merge 19 commits into from
Closed

[SPARK-43755][CONNECT] Move execution out of SparkExecutePlanStreamHandler and to a different thread #41315

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
3d5ee30
move execution
juliuszsompolski May 23, 2023
2ba6d26
fix compile
juliuszsompolski Jul 6, 2023
bbf8761
renames
juliuszsompolski Jul 6, 2023
1a8db60
dump
juliuszsompolski Jul 7, 2023
bbba4ca
todo - deadlocks and needs debug
juliuszsompolski Jul 10, 2023
e4e3615
passes tests, needs cleanup, needs a bit of thought about error handling
juliuszsompolski Jul 11, 2023
6f39e7b
separate ExecuteHolder / ExecuteRunner; todo comments and docs
juliuszsompolski Jul 11, 2023
20cb3dd
fix hang and cancellation
juliuszsompolski Jul 12, 2023
bcea9d4
docs and comments
juliuszsompolski Jul 12, 2023
0f88963
nits
juliuszsompolski Jul 12, 2023
2e6effa
scalastyle
juliuszsompolski Jul 13, 2023
e98db60
change job tag pattern
juliuszsompolski Jul 13, 2023
fee8838
self review nits and cleanup
juliuszsompolski Jul 13, 2023
0757c1c
add error class to documentation
juliuszsompolski Jul 13, 2023
a0d0792
typo
juliuszsompolski Jul 13, 2023
428dd77
Merge branch 'master' into sc-execute-thread
juliuszsompolski Jul 13, 2023
9104557
fix OPERATION_CANCELED error class
juliuszsompolski Jul 13, 2023
aa4093c
Merge branch 'master' into sc-execute-thread
juliuszsompolski Jul 14, 2023
c5008b5
Update connector/connect/server/src/main/scala/org/apache/spark/sql/c…
HyukjinKwon Jul 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into sc-execute-thread
  • Loading branch information
@juliuszsompolski
juliuszsompolski committed Jul 14, 2023
commit aa4093ceee3ccc7289208b99e9899e9003a0d96f
5 changes: 5 additions & 0 deletions ...or/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
// foreachBatch() in Streaming. Lazy since most sessions don't need it.
private lazy val dataFrameCache: ConcurrentMap[String, DataFrame] = new ConcurrentHashMap()

// Mapping from id to StreamingQueryListener. Used for methods like removeListener() in
// StreamingQueryManager.
private lazy val listenerCache: ConcurrentMap[String, StreamingQueryListener] =
new ConcurrentHashMap()

private[connect] def createExecuteHolder(request: proto.ExecutePlanRequest): ExecuteHolder = {
val operationId = UUID.randomUUID().toString
val executePlanHolder = new ExecuteHolder(request, operationId, this)
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.