Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
124ffc0
Introducing contributed code for isoscope scheduling.
vkruglik-aka Sep 6, 2024
ed63fe4
Integrate isoscope scheduling and distributed sccope isolation into x…
vkruglik-aka Sep 6, 2024
329f218
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 6, 2024
17ad842
Addressed pre-commit findings.
vkruglik-aka Sep 9, 2024
be0a561
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
3689139
Addressed additional pre-commit findings.
vkruglik-aka Sep 9, 2024
2671eec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
879ff2c
Addressed additional pre-commit findings.
vkruglik-aka Sep 9, 2024
94ae58c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
20caaeb
Addressed additional pre-coimmit finding.
vkruglik-aka Sep 9, 2024
77b4157
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
4486a9f
Addressed pre-coimmit finding.
vkruglik-aka Sep 9, 2024
1b9f92d
Add isoscope parametrization to generic acceptance tests test_single_…
vkruglik-aka Oct 22, 2024
c5d49b1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
9597f6f
Use List and Tuple from typing when specifying these types with subsc…
vkruglik-aka Oct 22, 2024
359938d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
7a29564
Address pre-commit findings concerning the use of tuple vs Tuple and …
vkruglik-aka Oct 22, 2024
017c1f6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 22, 2024
d8d97a1
Implemented acceptance test class TestIsoScope baesd on TestLoadScope.
vkruglik-aka Oct 23, 2024
95a16a6
Fix expected counts in TestIsoScope test_by_module, test_by_class, an…
vkruglik-aka Oct 23, 2024
61f5fad
Implemented acceptance test TestIsoScope.test_single_scope_all_worker…
vkruglik-aka Oct 23, 2024
edc3bd3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
db311ed
Implemented acceptance test TestIsoScope.test_single_scope_subset_of_…
vkruglik-aka Oct 23, 2024
6bb3ef2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
7f35d75
Fixed worker name key error for unutilized worker in acceptance test …
vkruglik-aka Oct 23, 2024
eebc3f4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 23, 2024
4695e43
Fixed default test counts in acceptance test TestIsoScope.test_single…
vkruglik-aka Oct 23, 2024
a4398b2
Removed extraneous setdefault in acceptance test TestIsoScope.test_si…
vkruglik-aka Oct 23, 2024
47fcf7a
Implemented acceptance test TestIsoScope.test_multi_scope_with_insuff…
vkruglik-aka Oct 24, 2024
680d855
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 24, 2024
d2bff97
Fixed test discoverability and pre-commit findings in acceptance test…
vkruglik-aka Oct 24, 2024
0e00ed9
Fixed TestFenceA/B search patterin in acceptance test TestIsoScope.te…
vkruglik-aka Oct 24, 2024
fa97762
Fix test_fence_scopes pytest filename in acceptance test TestIsoScope…
vkruglik-aka Oct 24, 2024
3b7f2de
Implemented acceptance test TestIsoScope.test_two_tests_min_per_worke…
vkruglik-aka Oct 24, 2024
da65e86
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 24, 2024
ad0f4af
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4658a1c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
97fe2e6
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
6aa321d
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
d110668
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
46f33a6
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4711108
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
5f988d9
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
c649eb5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
b865e9d
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
4292294
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 25, 2024
f1bf9aa
Work in progress on acceptance test TestIsoScope.test_distributed_set…
vkruglik-aka Oct 25, 2024
b71c911
Removed whitespace from blank lines.
vkruglik-aka Oct 25, 2024
06fe668
Initialize the status file used by underlying test in TestIsoScope.te…
vkruglik-aka Oct 25, 2024
b1f032d
Create parent directories for the status file used by underlying test…
vkruglik-aka Oct 25, 2024
52c0815
Fixed SyntaxError: (unicode error) 'unicodeescape' codec can't decode…
vkruglik-aka Oct 25, 2024
95b91c8
Fix node_path.relative_to(...) in _DistributedSetupCoordinatorImpl.ma…
vkruglik-aka Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Addressed additional pre-commit findings.
  • Loading branch information
vkruglik-aka committed Oct 25, 2024
commit 3689139d3b3dab2bc9b2cae93a71bd1ea4bb73e8
10 changes: 5 additions & 5 deletions src/xdist/iso_scheduling_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -353,11 +353,11 @@ def maybe_call_teardown(
teardown_callback(self._teardown_context)


def _map_file_lock_exception(f: Callable):
def _map_file_lock_exception(f: Callable): # type: ignore[no-untyped-def, type-arg]
"""Decorator: map `FileLock` exceptions of interest to our own exceptions."""

@functools.wraps(f)
def wrapper(*args, **kwargs):
def wrapper(*args, **kwargs): # type: ignore[no-untyped-def]
try:
return f(*args, **kwargs)
except filelock.Timeout as err:
Expand All @@ -380,11 +380,11 @@ class _DistributedSetupCoordinationImpl:
class DistributedState:
"""State of the Distributed Setup-Teardown Coordination."""

def __init__(self, setup_count, teardown_count):
def __init__(self, setup_count: int, teardown_count: int) -> None:
self.setup_count = setup_count
self.teardown_count = teardown_count

def __repr__(self):
def __repr__(self) -> str:
return (
f"<{self.__class__.__qualname__}: "
f"setup_count={self.setup_count}; "
Expand All @@ -404,7 +404,7 @@ def load_from_file_path(
return cls(**json.loads(state_file_path.read_text()))

@property
def as_json_kwargs_dict(self) -> dict:
def as_json_kwargs_dict(self) -> dict[str, int]:
"""
:return: JSON-compatible representation of the instance that is also
suitable for constructing the instance after fetching from file.
Expand Down
32 changes: 15 additions & 17 deletions src/xdist/scheduler/isoscope.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@
import random
from typing import TYPE_CHECKING

from _pytest.runner import CollectReport
import pytest

from xdist.report import report_collection_diff
Expand Down Expand Up @@ -142,7 +141,7 @@ def __init__(self, config: pytest.Config, log: xdist.remote.Producer):
# is performed once the number of registered node collections reaches
# `_expected_num_workers`. It is initialized to None and then updated
# after validation succeeds.
self._official_test_collection: tuple[str] | None = None
self._official_test_collection: tuple[str, ...] | None = None
# Remote worker node having `_official_test_collection` as its test
# collection (for reporting failed collection validations)
self._official_test_collection_node: WorkerController | None = None
Expand Down Expand Up @@ -367,7 +366,7 @@ def add_node_collection(

# Check that the new collection matches the official collection
if self._do_two_nodes_have_same_collection(
reference_node=self._official_test_collection_node,
reference_node=self._official_test_collection_node, # type: ignore[arg-type]
reference_collection=self._official_test_collection,
node=node,
collection=collection,
Expand Down Expand Up @@ -528,7 +527,7 @@ def _reschedule_workers(self) -> None:
"""Distribute work to workers if needed at this time."""
assert self._state is not None

traversed_states = []
traversed_states: list[IsoScopeScheduling._State] = []
previous_state = None
while self._state != previous_state:
# NOTE: This loop will terminate because completion of tests and
Expand Down Expand Up @@ -815,12 +814,11 @@ def _distribute_workset(
)

num_tests_remaining = workset.high_water
worker: _WorkerProxy
num_available_workers: int
for worker, num_available_workers in zip(
workers, range(num_workers_to_use, 0, -1)
):
worker: _WorkerProxy
num_available_workers: int

# Workers ready for distribution must have no more than one pending
# test
assert (
Expand Down Expand Up @@ -1021,7 +1019,7 @@ def _get_workers_ready_for_fencing(self, scope_id: str) -> list[_WorkerProxy]:
def _do_two_nodes_have_same_collection(
self,
reference_node: WorkerController,
reference_collection: tuple[str],
reference_collection: tuple[str, ...],
node: WorkerController,
collection: tuple[str, ...],
) -> bool:
Expand Down Expand Up @@ -1050,7 +1048,7 @@ def _do_two_nodes_have_same_collection(
# NOTE: Not sure why/when `_config` would be `None`. Copied check
# from the `loadscope` scheduler.

report = CollectReport(node.gateway.id, "failed", longrepr=msg, result=[])
report = pytest.CollectReport(node.gateway.id, "failed", longrepr=msg, result=[])
self._config.hook.pytest_collectreport(report=report)

return False
Expand All @@ -1072,11 +1070,11 @@ def __init__(self, node: WorkerController):

# An ordered collection of test IDs collected by the remote worker.
# Initially None, until assigned by the Scheduler
self._collection: tuple[str] | None = None
self._collection: tuple[str, ...] | None = None

self._pending_test_by_index: OrderedDict[int, _TestProxy] = OrderedDict()

def __repr__(self):
def __repr__(self) -> str:
return self.verbose_repr(verbose=False)

@property
Expand All @@ -1085,15 +1083,15 @@ def node(self) -> WorkerController:
return self._node

@property
def collection(self) -> tuple[str] | None:
def collection(self) -> tuple[str, ...] | None:
"""
:return: An ordered collection of test IDs collected by the remote
worker; `None` if the collection is not available yet.
"""
return self._collection

@collection.setter
def collection(self, collection: tuple[str]):
def collection(self, collection: tuple[str, ...]):
"""
:param collection: An ordered collection of test IDs collected by the
remote worker. Must not be `None`. Also, MUST NOT be set already.
Expand Down Expand Up @@ -1243,7 +1241,7 @@ def __init__(self, test_id: str, test_index: int):
self.test_id: str = test_id
self.test_index: int = test_index

def __repr__(self):
def __repr__(self) -> str:
return (
f"<{self.__class__.__name__}: test_index={self.test_index} "
f"scope_id={self.scope_id} test_id={self.test_id}>"
Expand Down Expand Up @@ -1273,7 +1271,7 @@ def __init__(self, scope_id: str):

self._test_by_index: OrderedDict[int, _TestProxy] = OrderedDict()

def __repr__(self):
def __repr__(self) -> str:
return (
f"<{self.__class__.__name__}: scope_id={self.scope_id} "
f"num_tests={self.num_tests} high_water={self.high_water}>"
Expand Down Expand Up @@ -1333,10 +1331,10 @@ def dequeue_tests(self, num_tests: int) -> list[_TestProxy]:
class _WorksetQueue:
"""Ordered collection of Scope Worksets grouped by scope id."""

def __init__(self):
def __init__(self) -> None:
self._workset_by_scope: OrderedDict[str, _ScopeWorkset] = OrderedDict()

def __repr__(self):
def __repr__(self) -> str:
return (
f"<{self.__class__.__name__}: "
f"num_worksets={len(self._workset_by_scope)}>"
Expand Down