Skip to content

fix(harness-manager): a run that opened a pull request waits on it, whatever the judge reads (FIX-2579) - #4990

Merged
jhoffner merged 1 commit into
mainfrom
claude/project-thread-csw1hs
Nov 10, 2026
Merged

jhoffner merged 1 commit into
mainfrom
claude/project-thread-csw1hs

Conversation

@jhoffner

@jhoffner jhoffner commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Requested by Jake · project thread

Problem

FIX-2579, problem 1 (wrong outcome). A coding task that pushed and opened a PR (#4798) was marked Failed. With awaitPullRequests on, settle checked the goal judge before the task's pull requests. So a confident not-done (≥ floor) threw the judge found the task not done: … even though the work had landed as a PR. That's likely in this scene: after a revise turn, the final message is about the push and PR, not the change.

Fix

settle now reads the task's pull requests first. If the run opened one, the task waits on that PR as FIX-2578 intended, and the judge's reason goes in a status note. A run that opened no PR still fails on a confident not-done. With the option off, nothing changes.

Evidence

  • New tests in test/await-pull-requests.spec.ts → "a judged run that opened a pull request". The run records its own PR from a renamed branch, and the judge says not-done at 0.8.
    • Red on main: the row went back to pending with feedback the judge found the task not done: ….
    • Green: the task waits on PR 9 (in_progress, one system/external hold).
    • Controls: no PR → still fails on the judge; option off → still fails.
  • @flow-state-dev/harness-manager: 545/545 tests pass. Typecheck passes through turbo.

Not in this PR

  • The exact error on Jake's row is still unconfirmed. If it reads the run stopped without finishing, the cause is in decide, not the judge, and this fix doesn't cover it.
  • Problems 2 and 3 (grouping and dismiss) stay on feat(claude-code-style): dismiss a failed thread so it stops waiting on you (FIX-2579) #4820. That PR is on hold for Jake's call on how to build dismiss.
  • The test drives the settle directly. It doesn't replay the review/revise round first. The settle code is the same either way.

Changeset: judge-defers-to-pull-request.md (harness-manager patch).


Generated by Claude Code

…hatever the judge reads (FIX-2579)

With awaitPullRequests on, settle read the judge's verdict before the
task's pull requests, so a confident not-done failed a run whose work had
landed as a pull request. The pull requests are now read first: a run
that opened one settles on it, and the judge's reason goes in a status
note. A run with no pull request, or with the option off, still fails on
a confident not-done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NczDiDusxC39h1dkbTPtPC
@vercel

vercel Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
flow-state-dev-build-site Ready Ready Preview Nov 9, 2026 11:24pm UTC
flow-state-dev-docs Ready Ready Preview Nov 9, 2026 11:24pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
flow-state-dev-ui Ignored Ignored Preview Nov 9, 2026 11:24pm UTC

Request Review

@jhoffner jhoffner assigned jhoffner and Copilot and unassigned jhoffner Nov 9, 2026
@linear-code

linear-code Bot commented Nov 9, 2026

Copy link
Copy Markdown

FIX-2579

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T23:27:29.303710Z 1d23553 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

jhoffner commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Second look + architecture notes

Small, targeted diff. The approach (let the PR reviewer be the check when a PR exists) is reasonable and the leaner shape is basically what's here. A few things for the implementing agent to weigh:

1. ask is still handled before the PR check (inconsistent). In settle (manager.ts ~L3686), a confident ask verdict still goes to holdOnQuestion even when prs.length > 0, while a confident not-done now defers to the PR. If the rationale is "the person reviewing the PR is the check," an ask on a run that opened a PR arguably deserves the same treatment (or an explicit comment/test saying why it's intentionally different: ask is a request for a human, not-done is a verdict). Right now neither the code nor the changeset states this, and there's no test pinning the ask + PR behavior either way.

2. The not-done branch rebuilds output differently from the main path. It hardcodes review: null, whereas the fall-through path computes review from decision.reviewed ("accepted" | "deadline" | null). If a run reached settle with reviewed === "accepted", the not-done+PR path drops it. Probably harmless (a not-done judge reading and an accepted review shouldn't co-occur), but it's a second construction of ManagerOutput to keep in sync. Suggest hoisting a single output build above the judge block and reusing it in both PR paths, which also removes the duplicated { ...decision.output, judged, review, landed } literal.

3. taskPullRequests now runs on every settle, before the judge. Previously it ran only when the run would otherwise complete. With awaitPullRequests on, a done/no-judge run pays the same single read as before, but a judge-ask or no-PR-not-done run now pays an extra readRunRecord + PR lookup it didn't need. Fine functionally; if the read is non-trivial, gate it on read?.verdict === "not-done" || read === null || read.verdict === "done" or simply accept the cost and note it.

4. Tests: confirm the red state. The new tests in await-pull-requests.spec.ts should fail on main for the right reason (run record reports failed, not a setup error). Worth adding: awaitPullRequests off + not-done + PR still fails (the changeset claims "nothing changes", but I'd want that asserted rather than inferred), and the below-floor / ask cases.

5. Housekeeping (minor).

  • The new // comment mixes two tracker IDs on one line; fine under BP-006 as a // pointer, but the JSDoc additions are clean — keep IDs out of anything that becomes a string or JSDoc.
  • Changeset body has (FIX-2579) in user-visible changelog prose; the docs-writer/outsider rule says to describe behavior, not tracker IDs. Consider dropping it from the sentence.

Architecture (improve-codebase-architecture lens): settle is already a very large handler inside a ~3.6k-line manager.ts, and the verdict→outcome decision (done / not-done / ask × PR present × floor) is now a 3×2 matrix expressed as nested ifs. This PR is a reasonable moment to extract that decision into a small pure function (judgeOutcome(read, floor, prs) → "complete" | "fail" | "hold" | "wait-on-pr") that's unit-testable without the handler context — it would also resolve point 1 by making the ask + PR cell an explicit, tested choice. I would not split manager.ts itself in this PR; that's unrelated to the bug fix.

No blocking correctness bugs found; points 1 and 4 are the ones I'd want addressed before merge.


Generated by Claude Code

@gitar-bot

gitar-bot Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Code Review ✅ Approved

🟡 Medium risk · With awaitPullRequests, deferring a confident not-done judgment when a run opened a PR could leave ineffective work waiting for review instead of retrying.

Reorders settle in the harness-manager so a run that opened a pull request waits on that PR before the goal judge is consulted, so a confident not-done no longer fails work that already landed as a PR. Runs without a PR still fail on a confident not-done, and behavior with awaitPullRequests off is unchanged. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@jhoffner
jhoffner added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit d8e983b Nov 10, 2026
23 checks passed
@jhoffner
jhoffner deleted the claude/project-thread-csw1hs branch November 10, 2026 00:46

This branch was successfully deployed

1 active deployment
Preview – flow-state-dev-docs — 1d235539 Deployed Nov 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants