Skip to content

[SPARK-46308] Forbid recursive error handling by adding recursion guards #44210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cdkrot wants to merge 10 commits into from
Closed

[SPARK-46308] Forbid recursive error handling by adding recursion guards #44210

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
285b85c
test
cdkrot Dec 4, 2023
eaa16e6
minor
cdkrot Dec 4, 2023
8b707fd
test coverage
cdkrot Dec 6, 2023
3565443
Merge branch 'master' into forbid_recursive_error_handling_2
cdkrot Dec 6, 2023
8ed9134
Revert "[SPARK-46241][PYTHON][CONNECT] Fix error handling routine so …
cdkrot Dec 6, 2023
a9433bd
slight rewrite
cdkrot Dec 6, 2023
c7f2130
changed to be martin way
cdkrot Dec 6, 2023
e4f9280
remove
cdkrot Dec 7, 2023
8220c46
thread locals
cdkrot Dec 7, 2023
e0dfa86
whitespace
cdkrot Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test coverage
  • Loading branch information
@cdkrot
cdkrot committed Dec 6, 2023
commit 8b707fd736cd09906ecefa806c6f1e55c9c6c027
21 changes: 21 additions & 0 deletions python/pyspark/sql/tests/connect/client/test_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from collections.abc import Generator
from typing import Optional, Any

from pyspark.sql.connect.client.core import ForbidRecursion
from pyspark.testing.connectutils import should_test_connect, connect_requirement_message
from pyspark.testing.utils import eventually

Expand Down Expand Up @@ -133,6 +134,26 @@ def test_channel_builder_with_session(self):
client = SparkConnectClient(chan)
self.assertEqual(client._session_id, chan.session_id)

def test_forbid_recursion(self):
Copy link
Contributor

@grundprinzip grundprinzip Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test does not test directly the scenario, we're talking about. Ideally you can just use the mock tests we have to fail any query and see that the recursion guard works.

Copy link
Contributor Author

@cdkrot cdkrot Dec 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually tried, but it seems hard to make the mock test for this because it needs to pass through this pieces of code:

spark/python/pyspark/sql/connect/client/core.py

Line 1545 in 348d881

status = rpc_status.from_call(cast(grpc.Call, rpc_error))

This seems hard to create a mock exception which would pass this without poking grpc's internals significantly. Alternatively we could introduce some testing clutches here, i.e. check if exception is from testing code, but that's not great either.

@grundprinzip

guard = ForbidRecursion()
max_depth = 0

def g(n):
nonlocal max_depth
with guard:
max_depth = n
g(n + 1)

with self.assertRaises(RecursionError):
g(1)
self.assertEqual(max_depth, 1)

# Do the same test again to check that guard resets.
max_depth = 0
with self.assertRaises(RecursionError):
g(1)
self.assertEqual(max_depth, 1)


class TestPolicy(DefaultPolicy):
def __init__(self):
Expand Down