feat(csharp/test/Drivers/Databricks): Enable RunAsync option in TExecuteStatementReq #3171
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
@CurtHagenlocher can you add @jadewang-db as the reviewer before merging this. |
Contributor
Only official Arrow committers can be added as reviewers, I'm afraid, but I can certainly wait until @jadewang-db signs off. |
CurtHagenlocher
approved these changes
Jul 16, 2025
Contributor
birschick-bq
left a comment
birschick-bq
left a comment
There was a problem hiding this comment.
Looks goods. A few small suggestions.
jadewang-db
approved these changes
Jul 16, 2025
| this.isLz4Compressed = isLz4Compressed; | ||
| this.statement = statement; | ||
| if (statement.DirectResults != null && !statement.DirectResults.ResultSet.HasMoreRows) | ||
| if (statement.DirectResults?.ResultSet? != null && !statement.DirectResults.ResultSet.HasMoreRows) |
Contributor
There was a problem hiding this comment.
Suggested change
| if (statement.DirectResults?.ResultSet? != null && !statement.DirectResults.ResultSet.HasMoreRows) | |
| if (statement.DirectResults?.ResultSet != null && !statement.DirectResults.ResultSet.HasMoreRows) |
CurtHagenlocher
approved these changes
Jul 16, 2025
CurtHagenlocher
pushed a commit
that referenced
this pull request
Aug 5, 2025
…lt true (#3232) ## Motivation In PR #3171, `RunAsync` option in `TExecuteStatementReq` was added and exposed via connection parameter `adbc.databricks.enable_run_async_thrift`, but it is not enabled by default. This is turned on by default in other Databricks drivers, we should turn in on by default in ADBC as well. ## Change - Set `DatabricksConnection:_runAsyncInThrift` default value to `true` ## Test - PBI PQTest with all the test cases
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Databricks adds
RunAsync(default is false) option for all the thrift operations, when it is set astrue, the operation runs async at the backend and the status can be polled by calling getStatus, it helps Databricks backend to better manage capacity and load on client requests. Furthermore, it can avoid the unnecessary retry on thrift operation (e.g. TExecuteStatementReq) when the warehouse is stopped or unavailable, the backend will returns 503 error whenRunAsyncis false and client has to retry on this thrift operation till the warehouse is up, this generates lots of queries with 503 errors in the Databricks query history and consume more resources. WhenRunAsync=true, server will return 200 with query statePENDING, the client will poll the status till the warehouse is up, this only generates one query in the query history.Change
adbc.databricks.enable_run_async_thrift, default isfalse(it will be changed totruelater)RunAsyncofTExecuteStatementReqwith above connection parameter (RunAsyncInThrift) inDatabricksStatement:SetStatementPropertiesBaseDatabricksReaderby addingnullcheck onstatement.DirectResults.ResultSetTestVarcharExceptionDataDatabricksinStringValuesTest: when error is in theDirectResult.status(case forRunAsync=true), it throw the error withDisplayMessage(see here) instead of theMessageandDisplayMessagedoes not include some internal error info such as error exception class at the backend.Test
csharp/test/Drivers/Databricks/E2EwhenConnection.RunAsyncInThriftis both on and off