Skip to content

feat: add support for parallel kind processing with threads #840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
bhearsum wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat: add support for parallel kind processing with threads #840

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a308f9
fix: use multiprocessing whenever `fork` context is available
bhearsum Oct 24, 2025
d2d2941
fix: make schema code thread-safe with locking
bhearsum Oct 24, 2025
05e295a
feat: add support for parallel kind processing with threads
bhearsum Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
fix: use multiprocessing whenever fork context is available
  • Loading branch information
@bhearsum
bhearsum committed Oct 24, 2025
commit 6a308f9344cd46d0816263d141029aff5997b9d9
5 changes: 3 additions & 2 deletions src/taskgraph/generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import logging
import multiprocessing
import os
import platform
from concurrent.futures import (
FIRST_COMPLETED,
ProcessPoolExecutor,
Expand Down Expand Up @@ -441,7 +440,9 @@ def _run(self):
# redone in the new processes. Ideally this would be fixed, or we
# would take another approach to parallel kind generation. In the
# meantime, it's not supported outside of Linux.
if platform.system() != "Linux" or os.environ.get("TASKGRAPH_SERIAL"):
if "fork" not in multiprocessing.get_all_start_methods() or os.environ.get(
"TASKGRAPH_SERIAL"
):
all_tasks = self._load_tasks_serial(kinds, kind_graph, parameters)
else:
all_tasks = self._load_tasks_parallel(kinds, kind_graph, parameters)
Expand Down
7 changes: 7 additions & 0 deletions test/test_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
# file, You can obtain one at http://mozilla.org/MPL/2.0/.


import multiprocessing
from concurrent.futures import ProcessPoolExecutor

import pytest
Expand All @@ -12,6 +13,11 @@
from taskgraph.generator import Kind, load_tasks_for_kind, load_tasks_for_kinds
from taskgraph.loader.default import loader as default_loader

forkonly = pytest.mark.skipif(
"fork" not in multiprocessing.get_all_start_methods(),
reason="requires 'fork' multiprocessing support",
)


class FakePPE(ProcessPoolExecutor):
loaded_kinds = []
Expand All @@ -21,6 +27,7 @@ def submit(self, kind_load_tasks, *args):
return super().submit(kind_load_tasks, *args)


@forkonly
def test_kind_ordering(mocker, maketgg):
"When task kinds depend on each other, they are loaded in postorder"
mocked_ppe = mocker.patch.object(generator, "ProcessPoolExecutor", new=FakePPE)
Expand Down
Loading