[SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server #46221
Conversation
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
seems fine from a cursory look ..... but would be great if @juliuszsompolski has a chance to take a quick look ...
|
The SESSION_CHANGED error comes from #45499, cc @nemanja-boric-databricks and @hvanhovell . This seems like a followup-fix to that change and it looks good to me, but it would be best if Nemanja took a look. |
nemanja-boric-databricks
left a comment
nemanja-boric-databricks
left a comment
There was a problem hiding this comment.
LGTM, yes we most likely need the same for Scala.
|
thanks, merged to master |
@nemanja-boric-databricks @nija-at would one of you have time to followup and check and fix for scala if needed? |
|
Yeah, I'll followup with the fix PR for scala client |
ghost
mentioned this pull request
|
@juliuszsompolski I just ported the fix + the follow-up to Scala - #47008, could you review this? |
… the default session is closed by the server ### What changes were proposed in this pull request? This is a Scala port of #46221 and #46435. A client is unaware of a server restart or the server having closed the client until it receives an error. However, at this point, the client in unable to create a new session to the same connect endpoint, since the stale session is still recorded as the active and default session. With this change, when the server communicates that the session has changed via a GRPC error, the session and the respective client are marked as stale, thereby allowing a new default connection can be created via the session builder. In some cases, particularly when running older versions of the Spark cluster (3.5), the error actually manifests as a mismatch in the observed server-side session id between calls. With this fix, we also capture this case and ensure that this case is also handled. ### Why are the changes needed? Being unable to use getOrCreate() after an error is unacceptable and should be fixed. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ./build/sbt testOnly *SparkSessionE2ESuite ### Was this patch authored or co-authored using generative AI tooling? No Closes #47008 from changgyoopark-db/SPARK-47986. Authored-by: Changgyoo Park <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
6 participants
What changes were proposed in this pull request?
When the server closes a session, usually after a cluster restart,
the client is unaware of this until it receives an error.
At this point, the client in unable to create a new session to the
same connect endpoint, since the stale session is still recorded
as the active and default session.
With this change, when the server communicates that the session
has changed via a GRPC error, the session and the respective client
are marked as stale. A new default connection can be created
via the session builder.
Why are the changes needed?
See section above.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Attached unit tests
Was this patch authored or co-authored using generative AI tooling?
No.