Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
124ffc0
Introducing contributed code for isoscope scheduling.
vkruglik-aka Sep 6, 2024
ed63fe4
Integrate isoscope scheduling and distributed sccope isolation into x…
vkruglik-aka Sep 6, 2024
329f218
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 6, 2024
17ad842
Addressed pre-commit findings.
vkruglik-aka Sep 9, 2024
be0a561
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
3689139
Addressed additional pre-commit findings.
vkruglik-aka Sep 9, 2024
2671eec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
879ff2c
Addressed additional pre-commit findings.
vkruglik-aka Sep 9, 2024
94ae58c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
20caaeb
Addressed additional pre-coimmit finding.
vkruglik-aka Sep 9, 2024
77b4157
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
4486a9f
Addressed pre-coimmit finding.
vkruglik-aka Sep 9, 2024
1b9f92d
Add isoscope parametrization to generic acceptance tests test_single_…
vkruglik-aka Oct 22, 2024
c5d49b1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
9597f6f
Use List and Tuple from typing when specifying these types with subsc…
vkruglik-aka Oct 22, 2024
359938d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
7a29564
Address pre-commit findings concerning the use of tuple vs Tuple and …
vkruglik-aka Oct 22, 2024
017c1f6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
d8d97a1
Implemented acceptance test class TestIsoScope baesd on TestLoadScope.
vkruglik-aka Oct 23, 2024
95a16a6
Fix expected counts in TestIsoScope test_by_module, test_by_class, an…
vkruglik-aka Oct 23, 2024
61f5fad
Implemented acceptance test TestIsoScope.test_single_scope_all_worker…
vkruglik-aka Oct 23, 2024
edc3bd3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
db311ed
Implemented acceptance test TestIsoScope.test_single_scope_subset_of_…
vkruglik-aka Oct 23, 2024
6bb3ef2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
7f35d75
Fixed worker name key error for unutilized worker in acceptance test …
vkruglik-aka Oct 23, 2024
eebc3f4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
4695e43
Fixed default test counts in acceptance test TestIsoScope.test_single…
vkruglik-aka Oct 23, 2024
a4398b2
Removed extraneous setdefault in acceptance test TestIsoScope.test_si…
vkruglik-aka Oct 23, 2024
47fcf7a
Implemented acceptance test TestIsoScope.test_multi_scope_with_insuff…
vkruglik-aka Oct 24, 2024
680d855
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 24, 2024
d2bff97
Fixed test discoverability and pre-commit findings in acceptance test…
vkruglik-aka Oct 24, 2024
0e00ed9
Fixed TestFenceA/B search patterin in acceptance test TestIsoScope.te…
vkruglik-aka Oct 24, 2024
fa97762
Fix test_fence_scopes pytest filename in acceptance test TestIsoScope…
vkruglik-aka Oct 24, 2024
3b7f2de
Implemented acceptance test TestIsoScope.test_two_tests_min_per_worke…
vkruglik-aka Oct 24, 2024
da65e86
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 24, 2024
ad0f4af
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4658a1c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
97fe2e6
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
6aa321d
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
d110668
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
46f33a6
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4711108
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
5f988d9
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
c649eb5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
b865e9d
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4292294
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
f1bf9aa
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
b71c911
Removed whitespace from blank lines.
vkruglik-aka Oct 25, 2024
06fe668
Initialize the status file used by underlying test in TestIsoScope.te…
vkruglik-aka Oct 25, 2024
b1f032d
Create parent directories for the status file used by underlying test…
vkruglik-aka Oct 25, 2024
52c0815
Fixed SyntaxError: (unicode error) 'unicodeescape' codec can't decode…
vkruglik-aka Oct 25, 2024
95b91c8
Fix node_path.relative_to(...) in _DistributedSetupCoordinatorImpl.ma…
vkruglik-aka Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Addressed pre-commit findings.
  • Loading branch information
vkruglik-aka committed Oct 25, 2024
commit 17ad842c50a66a1463b994126f7a8d36ba36311b
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@ repos:
- execnet>=2.1.0
- types-psutil
- setproctitle
- filelock>=3.13.1
5 changes: 2 additions & 3 deletions src/xdist/iso_scheduling_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Generator
from typing import Optional


_LOGGER = logging.getLogger(__name__)
Expand Down Expand Up @@ -255,8 +254,8 @@ def __init__(

self._worker_id: str = worker_id

self._setup_context: Optional[DistributedSetupContext] = None
self._teardown_context: Optional[DistributedTeardownContext] = None
self._setup_context: DistributedSetupContext | None = None
self._teardown_context: DistributedTeardownContext | None = None

def maybe_call_setup(
self,
Expand Down
3 changes: 2 additions & 1 deletion src/xdist/iso_scheduling_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Generator
from contextlib import AbstractContextManager

import pytest

Expand All @@ -61,7 +62,7 @@ class IsoSchedulingFixture(abc.ABC):
@abc.abstractmethod
def coordinate_setup_teardown(
self, setup_request: pytest.FixtureRequest
) -> Generator[DistributedSetupCoordinator, None, None]:
) -> AbstractContextManager[DistributedSetupCoordinator]:
"""Context manager that yields an instance of
`DistributedSetupCoordinator` for distributed coordination of Setup
and Teardown.
Expand Down
42 changes: 15 additions & 27 deletions src/xdist/scheduler/isoscope.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@
from collections.abc import Iterable
from collections.abc import ValuesView
from typing import NoReturn
from typing import Optional
from typing import Sequence

import xdist.remote
Expand Down Expand Up @@ -129,7 +128,7 @@ def __init__(self, config: pytest.Config, log: xdist.remote.Producer):

# Scope ID of tests that are currently executing; `None` prior to the
# initial distribution
self._active_scope_id: Optional[str] = None
self._active_scope_id: str | None = None

# The initial expected number of remote workers taking part.
# The actual number of workers will vary during the scheduler's
Expand All @@ -143,10 +142,10 @@ def __init__(self, config: pytest.Config, log: xdist.remote.Producer):
# is performed once the number of registered node collections reaches
# `_expected_num_workers`. It is initialized to None and then updated
# after validation succeeds.
self._official_test_collection: Optional[tuple[str]] = None
self._official_test_collection: tuple[str] | None = None
# Remote worker node having `_official_test_collection` as its test
# collection (for reporting failed collection validations)
self._official_test_collection_node: Optional[WorkerController] = None
self._official_test_collection_node: WorkerController | None = None

# Ordered collection of Scope Worksets. Each Scope Workset is an ordered
# collection of tests belonging to the given scope. Initially empty,
Expand Down Expand Up @@ -257,7 +256,7 @@ def add_node(self, node: WorkerController) -> None:

self._pending_worker_by_node[node] = _WorkerProxy(node)

def remove_node(self, node: WorkerController) -> Optional[str]:
def remove_node(self, node: WorkerController) -> str | None:
"""Remove a Remote Worker node from the scheduler.

This should be called either when the node crashed or at node shutdown
Expand Down Expand Up @@ -386,7 +385,7 @@ def add_node_collection(
# Check if we now have enough collections to establish a final one

# Get all pending workers with registered test collection
w: _WorkerProxy
# ZZZ remove line w: _WorkerProxy
workers_with_collection = [
w for w in self._pending_worker_by_node.values() if w.collection is not None
]
Expand Down Expand Up @@ -472,6 +471,7 @@ def remove_pending_tests_from_node(
node: WorkerController,
indices: Sequence[int],
) -> NoReturn:
"""Not supported"""
raise NotImplementedError()

def schedule(self) -> None:
Expand Down Expand Up @@ -868,7 +868,7 @@ def _distribute_workset(
@classmethod
def _generate_fence_items(
cls, source_worksets: Iterable[_ScopeWorkset]
) -> Generator[Optional[_TestProxy], None, None]:
) -> Generator[_TestProxy | None, None, None]:
"""Generator that withdraws (i.e., dequeues) Fence test items from the
given ordered Scope Worksets and yields them until it runs out of the
fence items per limits described below, and will thereafter yield
Expand Down Expand Up @@ -1065,16 +1065,14 @@ class _WorkerProxy:
"""

def __init__(self, node: WorkerController):
"""
:param node: The corresponding xdist worker node.
"""
""":param node: The corresponding xdist worker node."""
# node: node instance for communication with remote worker,
# provided by pytest-xdist controller
self._node: WorkerController = node

# An ordered collection of test IDs collected by the remote worker.
# Initially None, until assigned by the Scheduler
self._collection: Optional[tuple[str]] = None
self._collection: tuple[str] | None = None

self._pending_test_by_index: OrderedDict[int, _TestProxy] = OrderedDict()

Expand All @@ -1083,13 +1081,11 @@ def __repr__(self):

@property
def node(self) -> WorkerController:
"""
:return: The corresponding xdist worker node.
"""
""":return: The corresponding xdist worker node."""
return self._node

@property
def collection(self) -> Optional[tuple[str]]:
def collection(self) -> tuple[str] | None:
"""
:return: An ordered collection of test IDs collected by the remote
worker; `None` if the collection is not available yet.
Expand Down Expand Up @@ -1260,9 +1256,7 @@ def scope_id(self) -> str:


class _ScopeWorkset:
"""
Ordered collection of Tests for the given scope
"""
"""Ordered collection of Tests for the given scope."""

__slots__ = (
"scope_id",
Expand All @@ -1271,9 +1265,7 @@ class _ScopeWorkset:
)

def __init__(self, scope_id: str):
"""
:param scope_id: Test Scope to which the tests in this workset belong;
"""
""":param scope_id: Test Scope to which the tests in this workset belong;"""
self.scope_id = scope_id

# High watermark for number of tests in the workset
Expand All @@ -1294,9 +1286,7 @@ def empty(self) -> bool:

@property
def high_water(self) -> int:
"""
:return: High Watermark of the number of tests in the workset.
"""
""":return: High Watermark of the number of tests in the workset."""
return self._high_water

@property
Expand All @@ -1315,9 +1305,7 @@ def enqueue_test(self, test: _TestProxy) -> None:
self._test_by_index[test.test_index] = test

# Update high watermark
new_num_tests = len(self._test_by_index)
if new_num_tests > self._high_water:
self._high_water = new_num_tests
self._high_water = max(self._high_water, len(self._test_by_index))

def dequeue_tests(self, num_tests: int) -> list[_TestProxy]:
"""
Expand Down