Skip to content

fix(native): retain failed pwntools process and runtime cleanup - #1708

Merged
morluto merged 6 commits into
morluto:mainfrom
1NormalGuy:fix/pwntools-cleanup-ownership
Oct 12, 2026
Merged

morluto merged 6 commits into
morluto:mainfrom
1NormalGuy:fix/pwntools-cleanup-ownership

Conversation

@1NormalGuy

@1NormalGuy 1NormalGuy commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Retain owned decoder/debugger processes and runtime files until cleanup is confirmed. Finalize one-shot CLI inspections before publishing their original outcome, so a successful cleanup retry returns the decoded report and a genuine decoder failure retains its original diagnostics. These defects were reproduced during repository review; there is no related issue.

Problem and expected behavior

Previously, failed cleanup could discard its supervisor and delete files still belonging to a worker. A later CLI cleanup retry could also succeed while the command still reported stale cleanup_incomplete resources and hid its decoded report. Ownership must survive failed cleanup, and published outcomes must reflect the final cleanup attempt.

Change and scope

  • Retain worker/debugger supervisors and runtime roots, serialize admission, preserve prompt queued cancellation, block acquisition while cleanup remains unconfirmed, and allow close() to retry.
  • Execute both decoder stages through one owner-bound command runner. Retire each process group before returning its result or original failure, reading replies or failure markers, or acquiring the next stage. Pwndbg receives the executor instead of launcher/cleanup-transfer callbacks.
  • Add an explicit terminal inspection operation at the provider/service boundary and remove CLI-side result reconstruction. Successful retries restore the original success or typed primary failure.
  • Report only resources still owned after partial cleanup. Preserve validated reports through the existing partial_observation error field when cleanup remains incomplete, including artifact identity and diagnostics. Failed debugger enrichment retains validated basic core evidence with explicit requested: true and status: unavailable.
  • Close shared diagnostic services during MCP shutdown and register failure-test teardown before assertions.

Contract and boundary impact

  • Semantic owner and earliest changed stage: owned process execution and decoder lifecycle.
  • CLI and MCP/tool-catalog contract: advertised schemas unchanged; existing typed cleanup and partial-observation fields carry failures and retained reports.
  • Provider/platform scope: pwntools binary layout, recorded-crash decoding, and optional Pwndbg enrichment. No new target execution authority.
  • Evidence/provenance: caller-selected artifact paths, snapshot digests, byte counts, validated reports, primary failures, output, and cleanup reasons survive finalization.
  • Internal terminal-operation contract: an admitted inspection finalizes before publication; input validation or cancelled admission acquires nothing and does not close an existing provider. Reusable callers retain explicit close() responsibility.
  • Generated metadata, installation, package, and release impact: none; no public migration required.

Evidence and regression coverage

Actual owned Node workers and filesystem boundaries cover retained live children, retryable shutdown, admission blocking, queued cancellation, primary-decoder retirement before debugger acquisition, and current resource lists after partial cleanup. Public CLI regressions cover restored success, preserved command failure, recovered process cleanup, persistent runtime-removal failure with a retained report, and cancellation during cleanup. CLI/MCP service checks cover shared workflow projection and shutdown.

Representative final outcomes:

  • A successful runtime-removal retry returns normal Evidence with exit status 0.
  • A real decoder exit failure remains execution_failure after cleanup recovers.
  • Persistent cleanup returns cleanup_incomplete with current resources and a validated partial_observation when available.
  • Combined analysis/cleanup failures preserve both primary_error and cleanup_error.

Validation performed

On fix commit a357aa41a, with Node 24.18.0 and npm 11.16.0:

  • Focused provider/process/service/public CLI/MCP suite: 95 passed, 68 Linux-only cases skipped on macOS, across 15 files. Required runtime artifacts built successfully.
  • npm run check:ci: passed, including typecheck, lint, formatting, unused-code/metadata checks, duplication checks, module boundaries, and architecture guards.
  • npm run docs:check: passed, including README consistency.
  • npm run verify:test-discovery: all 949 test files discovered exactly once.
  • git diff --check and the pre-push check: passed.

After merging current upstream ad08510a1 into 433a744d1, the affected EVM/layout/recorded-crash MCP and retirement suite passed all 15 tests across four files, and npm run check:ci passed again. The server conflict preserves upstream static EVM binding and this PR's shared diagnostic cleanup.

The local Node-worker regressions establish actual POSIX ownership/filesystem behavior, with only the decoder availability gate simulated. They do not establish real pwntools/GDB behavior. Previous reviewed commit 66a32a734 passed all CI, including real Linux pwntools and recorded-crash workflows. CI on final merge head 433a744d1 is pending; real-provider verification will be recorded separately when it completes.

Review follow-up

Merged current upstream main cleanly. Addressed the cleanup-retry/result-reconciliation thread at the owning process/provider boundary. Separate standards and specification reviews also found and fixed stale resource lists after partial cleanup, lost decoded observations on persistent cleanup failure, and missing failure-test teardown. Follow-up review also fixed both worker-file races: failure markers and debugger replies are consumed only after confirmed retirement. Eight real-worker regressions cover surviving descendants rewriting files, unresolved ownership in both stages, and debugger output overflow. Optional debugger validation retains truthful partial evidence. No remaining findings from those reviews.

Review checklist

  • Concrete problem and focused Conventional Commit title.
  • Meaningful cleanup, cancellation, malformed-output, and ownership regressions.
  • Caller identity, provenance, failed attempts, and explicit uncertainty preserved.
  • Contracts and generated documentation checked.
  • Final diff checked for unrelated changes and unsupported provider claims.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T17:37:56.645642Z 433a744 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6aca51bec4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/binaryDiagnosticsCommands.ts Resolved
@CassieFox23

CassieFox23 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: On Linux with real pwntools 4.15.0, the PR works as described. When a worker's stop is unconfirmed, head keeps its runtime folder, and base deletes it. One gap remains: the CLI can retry cleanup at exit, succeed, and still report failure.

The pwntools version is 4.15.0 (PyPI license field: "Mostly MIT, some GPL/BSD"). I set it with REA_PWNTOOLS_PYTHON and installed nothing. Base is the merge-base 13e03e9 (#1694). Head is 6aca51b.

Results

Check (Node 24) Base 13e03e9 Head 6aca51b
Normal CLI run, 3 ELF files exit 0, nothing left behind exit 0, output byte-identical to base, nothing left behind
Normal MCP run, 3 ELF files ok, nothing left behind ok, output byte-identical to base, nothing left behind
Worker cleanup fails (CLI) exit 1. It deletes the runtime folder. The unverified child keeps running. The error names a folder that is gone. exit 1. It keeps the runtime folder and its snapshot. The unverified child also keeps running. The error shows the first failure and the cleanup failure.
Same failure, 3 MCP calls in one server Call 2 starts a second worker. Both leave a child behind. Call 2 is refused in 26 to 36 ms and starts nothing. Call 3 works after I killed the child, and the folder is gone.
MCP shutdown, cleanup still failing Exit 0 in 31 to 33 ms. No message. Exit 0 in 54 to 66 ms. It logs that provider cleanup failed. The folder stays on disk.
MCP shutdown, normal run 30 to 36 ms 27 to 38 ms
Real worker frozen with SIGSTOP Timeout error at 31 s. Worker killed. Folder deleted. Same as base
CLI retry at exit succeeds exit 1 with cleanup_incomplete exit 1 with the same error, though the folder is gone. Seen in 5 of 6 timed runs.
Changed tests (8 files) Node 22.23.1 Node 24.18.0 Node 26.11.1
Head 64 passed, 1 skipped 64 passed, 1 skipped 64 passed, 1 skipped

The one skip is a test that only runs on non-Linux hosts. The 34 Linux-gated tests the PR body lists as skipped did run here. The same tests against base code gave 9 failed and 55 passed on Node 24. The PR applies cleanly to main 2165045 (#1733), and the tests pass there on Node 24.

What to know

  • Forcing the failure. SIGSTOP alone does not work, because REA cleans up with SIGKILL, and SIGKILL ends a stopped process. I added a child that REA cannot verify to the worker's process group. Details are in the Commands fold.
  • Retained folder. When cleanup fails, the CLI or server process exits and the runtime folder stays on disk. It holds a copy of the input file. Nothing removes it later. This fits the PR's design, but it is a visible change.
  • The retry gap. I used a child that exits on its own about 0.2 s after start. The CLI's first cleanup fails, and its retry at exit then works. The error shape shows this: it has no cleanup_error field, which the CLI adds only when the retry also fails. The folder is gone too. The exit code and error are still the same as on base. The decode itself had worked. This matches the P2 review comment on the PR.
  • MCP shutdown. Shutdown waits no longer on a normal run. With cleanup stuck it takes about 25 to 30 ms more, because it retries once. The box was shared (load average about 8 to 10 on 24 threads), so these are rough.

Not run

  • Recorded crash tools and the GDB or pwndbg path, beyond the PR's unit test.
  • macOS and Windows.
  • The full test suite, and npm run check.
  • A worker stuck in uninterruptible sleep (the D state).
  • MCP shutdown with a frozen worker.

Thanks, 1NormalGuy, for the clear PR body and for tests that kill real processes. They ran on Linux and passed.

Commands
export REA_PWNTOOLS_PYTHON=~/.local/opt/pwntools-4.15.0/bin/python
# normal runs, base and head, per ELF
TMPDIR=<fresh dir> node scripts/rea.mjs inspect-binary-layout <elf> --format json
# MCP over stdio: initialize, tools/call inspect_binary_layout, then close stdin and time the exit
# forced failure: REA_PWNTOOLS_PYTHON=<wrapper that starts "env -i sleep 600 &" then execs the real python>
# SIGSTOP run: a watcher sends SIGSTOP to the worker whose argv names my own TMPDIR, then SIGCONT and SIGKILL by PID
# the unverified child has no REA run token, so REA refuses to signal the group:
# that is the "unowned or PID-reused process" case. The worker itself still runs the real pwntools Python.
# changed tests, one Node at a time
npm run test:focused -- src/application/binaryDiagnostics/BinaryLayoutService.test.ts \
  src/application/binaryDiagnostics/RecordedCrashService.test.ts \
  tests/boundary/binaryDiagnostics/pwntoolsLayoutProvider.test.ts \
  tests/boundary/mcp/binaryDiagnosticsMcp.test.ts tests/boundary/mcp/recordedCrashMcp.test.ts \
  tests/boundary/native/PwndbgCoreContext.test.ts \
  tests/process-global/binaryDiagnosticsCliCleanup.test.ts tests/process-global/pwntoolsCleanup.test.ts
# applies to main
git apply --check <merge-base..head diff>   # in a copy of main 21650455a

The ELF files were /usr/bin/true from Fedora's coreutils 9.10 (GPL-3.0-or-later). The other two were a small C program built with GCC 16.2.1, once as a PIE file and once as a non-PIE file. Every stray child I started was killed by its PID. Every run used its own TMPDIR, so I could count leftover folders and processes.

Test setup
  • Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
  • CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
  • RAM: 32 GB DDR4-3600
  • Node: 22.23.1, 24.18.0, 26.11.1 (builds and failure runs on 24.18.0)
  • Head: 6aca51b
  • Base: 13e03e9 (perf(evidence): fuse streamed JSON token handling #1694), the merge-base with main
  • Other: pwntools 4.15.0 ("Mostly MIT, some GPL/BSD" on PyPI), GCC 16.2.1. The box was shared with other test jobs.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@dandgabr dandgabr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consolidated review

  • P2 — Reconcile cleanup retries before returning the CLI result. In src/cli/binaryDiagnosticsCommands.ts:64, an initial cleanup_incomplete result remains projected even if the CLI's exit cleanup retry succeeds and removes the retained pwntools runtime. This reports failure after the cleanup has completed. Reconcile the final cleanup outcome before serializing the result. This consolidates the still-applicable inline finding; no additional actionable findings were confirmed.

Keep failed worker/debugger cleanup owners and their private files until
verified process shutdown and successful directory removal. Retry cleanup
before another inspection and expose shutdown through the shared services,
CLI and MCP server. Serialize acquisition without delaying cancellation of
queued callers, and preserve primary diagnostics when CLI cleanup fails.

The original decoder reproduction left a live owned Node child with its
runtime deleted and no retained cleanup owner; the regression now verifies
retention, admission blocking, retries and eventual process/file cleanup.

Validation: 10 lifecycle/public CLI/MCP tests, 20 service tests, test
discovery 814/814, build:cached and full check including architecture guards.
Actual POSIX process/filesystem boundaries used injected Node workers on
macOS; real Linux pwntools/GDB provider parsing was not exercised.
@1NormalGuy
1NormalGuy force-pushed the fix/pwntools-cleanup-ownership branch from 6aca51b to 1b565a4 Compare October 12, 2026 14:58

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 66a32a7348

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/native/pwntools/PwntoolsDecoder.ts Resolved
Comment thread src/native/pwndbg/PwndbgCoreContext.ts Resolved
@morluto
morluto merged commit 11fe5a1 into morluto:main Oct 12, 2026
33 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 433a744d15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

this.profile.operation,
);
}
if (!outcome.ok) throw outcome.error;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Refresh diagnostics after retrying process cleanup

When a timeout or cancellation occurs before output closes and the first stop is incomplete, the retry at line 466 can collect additional stdout/stderr before it successfully stops and disposes the supervisor. This line nevertheless rethrows the original OwnedCommandFailure, whose snapshot was taken in collectOwnedCommand immediately after the first stop, so those final bytes are lost even though cleanup is confirmed; rebuild the failure from the retained supervisor's final snapshot before clearing ownership.

AGENTS.md reference: AGENTS.md:L13-L13

Useful? React with 👍 / 👎.

Comment on lines +518 to +521
[
pending.process.launch.ownership?.runId ?? "owned-decoder",
...(pending.root === undefined ? [] : [pending.root.path]),
],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the retained process group in cleanup resources

When process cleanup remains incomplete, this array lists only the run ID and runtime directory even though the retained supervisor still has leaderPid and processGroupId. OwnedCommand's non-transfer cleanup error reports pid: and process-group: coordinates, and omitting them here means the projected cleanup result does not identify the potentially live group the caller must inspect; include both while this owner is retained.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

@morluto

morluto commented Oct 12, 2026

Copy link
Copy Markdown
Owner

@1NormalGuy, thanks for tackling the cleanup ownership edge cases across the pwntools and pwndbg flows. Retaining workers and runtime files until cleanup is confirmed, then reporting the final outcome while preserving validated crash evidence, makes failure handling much more trustworthy. Welcome aboard REA!

@morluto morluto added bug Something isn't working area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior labels Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants