Skip to content

fix(FIX-2137): a resume never re-runs completed steps when trace observability is off - #3749

Merged
jhoffner merged 12 commits into
mainfrom
fix/FIX-2137
Oct 23, 2026
Merged

jhoffner merged 12 commits into
mainfrom
fix/FIX-2137

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

In production, when a request resumes after someone answers a stop, the steps that had already finished run again. Their writes, sends and model calls all happen a second time.

Resume skips a finished step by reading the output recorded on its block_trace. That record was only written when trace observability was on, and observability is off by default in production.

What this does

  • Lean replay record. With observability off, every completed block still records a lean block_trace holding only its identity and output, marked lean: true. Both records are built by one shared baseBlockTraceFromIdentity. Full capture (inputs, prompts, resource loads, failed traces) stays behind FSDEV_TRACE_OBSERVABILITY.
  • Where skipping applies. It applies when a durability provider is configured: when a request resumes after a stop, and on /continue. Without a provider, /continue skips a completed block only from a full trace and ignores lean ones (buildReplayLog(…, { skipLeanTraces })). So it behaves as before this PR, and its behaviour follows how the request was recorded, not the trace setting at recovery.
  • One index allocator. Every item index in createExecutionContext now comes from reserveItemIndex(), which reconciles with the response cursor (getItemCount, otherwise the highest getItems() index + 1). This also fixes an existing collision with observability on: block traces and items a block emitted directly could share an index.
  • wasRescued survives a resume. A completed block_trace carries rescued: true when .rescue() produced the output. Records also gain an optional lean: true marker. Both are new optional fields in contracts, plus ReplayLog.completedRescued in core. Older records read as full and not rescued.
  • Folds the trims from Cursor's POC POC: shorter traces-off replay notes #3757.

Behaviour changes worth knowing

  • An output over the recorded-value limit is stored as an omitted placeholder. A resume that reaches it fails with RECORDED_VALUE_OMITTED and names the block. It no longer re-runs silently.
  • With a provider, crash recovery is at-least-once for the block whose completion had not been persisted when the process died. Before this PR it re-ran every completed block.

What's asked of you

Nothing to decide. The calls I made:

  • The replay record is kept for every request.
  • The lean record is not awaited before the next block, because that would put a store write on the hot path.
  • Provider-free /continue is left as it was. Its existing state-loss gap is FIX-2148.

Verification

Each new test fails with the fix reverted and passes with it:

  • Resume after two answers.
  • A generator called once across a resume.
  • /continue in 7 cases, with and without a provider, including flipping observability between the run and the recovery (off→on, on→off).
  • Index uniqueness, including seeded and partial responses.
  • wasRescued across a resume.
  • A core unit test for skipLeanTraces.

Suites, all passing on the latest head:

Suite Passed
engine 4027 (8 skipped)
core 2521
store-sqlite 404
integration-tests 154

The other packages (claude-code, codex, cursor, orchestration, harness-manager, workforce, patterns) passed on the earlier head, and this push doesn't touch them. Typecheck, the vocabulary sweep and the retired-terms check are clean. Codex is clean on 9a77116.

Follow-ups: FIX-2146 (route direct emissions through the shared allocator) and FIX-2148 (dev Continue loses sequencer state). Close POC #3757 unmerged once this lands.

Fixes FIX-2137 · Epic FIX-2058 · Unblocks FIX-2063, FIX-2129


🤖 Generated with Claude Code

https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

…traces off

Resume replay skips a completed block by reading its recorded block_trace
output, but block_trace capture was installed only when trace observability
is on, and it is off by default in production. A resumed request therefore
ran every completed step again in production.

With observability off, the trace hook now writes one lean block_trace per
completed, non-transient block (identity + output only). Full capture
(input, generator prompt, resource loads, failed traces) stays gated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6
@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 23, 2026 7:48pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 23, 2026 7:48pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 23, 2026 7:48pm UTC

Request Review

@linear-code

linear-code Bot commented Oct 23, 2026

Copy link
Copy Markdown

FIX-2137

Copy link
Copy Markdown
Collaborator Author

@codex review: please pay special attention to these areas at f6355014:

  1. Replay correctness with traces off. Does the lean block_trace (identity plus output, written at the output phase) carry everything resume needs to skip a finished step? Check nested sequencer steps, router branches, .map children, and blocks inside a resumed loop iteration. Is any of those keyed by something the lean record leaves out?
  2. Item ordering. itemIndex is assigned when a block finishes, not when it starts. Can a parent and its children, or concurrent .map branches, end up with indices that break resume's lookup or the persisted log's sequence-number resume for SSE clients?
  3. Leaks. Can any input, prompt, resource-load or tool-argument data still reach the lean record, for example through output for blocks whose output echoes their input? Do live streams without includeTrace still never see these items, across the SQLite, Postgres and filesystem stores?
  4. Non-completed outputs. Skipping suspended and failed blocks: could a block that suspended, then completed after its answer, end up with no record and re-run on a second resume?

Direction-level findings matter most. Skip style nits.


Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T19:51:42.127515Z 9a77116 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jhoffner jhoffner left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second look + architecture pass

Approach: right one. The replay path already reads block_trace via buildReplayLog, so recording a lean completed trace is the smallest fix; gating on a durability provider would leave /continue crash recovery broken. I agree with the "every request" recommendation. Alternatives considered (a separate replay_record item type, or checkpointing outputs into durable state) are a bigger surface for no gain here.

Worth acting on before merge

  1. Extract the shared block_trace item construction so the lean record is a subset of the full one by construction (inline comment).
  2. Add a generator (ref-shaped output) replay test and a /continue test with traces off. The current tests only prove inline handlers (inline comment).
  3. Docs: state that outputs over the recorded-value limit still re-run on resume. "Never re-runs a finished step" is stated unconditionally in the changeset/PR.

Architecture opportunity, only as it relates to this change: the ternary that returned undefined was the zero-overhead switch; now that the hook is always defined, collapse to one hook with an early return. I would not widen into a broader trace-subsystem refactor.

Noted, not blocking: the two store-adapter test headers the PR mentions are still stale; fine as a follow-up. Changeset and env docs look good; the changeset embeds (FIX-2137), which is allowed there.

No blockers. Not approving or requesting changes — leaving that to the author agent.


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem

With FSDEV_TRACE_OBSERVABILITY off (production default), onBlockTraceCapture was undefined, so completed blocks left no persisted block_trace. buildReplayLog / ReplayLog.getCompletedOutput then had nothing to inject on resume, and durable flows re-ran finished steps (duplicate side effects, double state patches).

Verdict

Approve. The fix matches how replay is already designed: the items log is the replay ledger, and observability was incorrectly acting as the on/off switch for that ledger. Emitting a lean completed block_trace (identity + output only) when observability is off is the smallest correct change. Docs and test updates line up with the new contract.

This is not an over-built solution relative to the bug. A separate durability channel or new item type would be cleaner conceptually but would touch buildReplayLog, persistence, and DevTool assumptions for a wider refactor than FIX-2137 needs.

/simplify — where complexity crept in (non-blocking)

Area Finding Recommendation
Dual callbacks isTraceObservabilityEnabled() ? fullHook : leanHook duplicates two ~100-line behaviors and two BlockTraceItem literals that share the same provenance/identity stamping. Prefer one onBlockTraceCapture with early returns: when observability is off, ignore non-output phases; at output + completed, emit the lean row; otherwise keep today’s FIX-573 lifecycle.
Identity stamping item_block_trace_* id, provenance, blockKind cast, etc. appear in both arms (~3167 and ~3259). Module-private helper, e.g. baseBlockTraceFromIdentity(identity, requestId, itemIndex, ts), with each arm adding only mode-specific fields. Matches the post–FIX-573 pattern (engine materializes rows; core sends partial payloads) without pulling factories into @flow-state-dev/core.
Terminal emit Lean path inlines emitItemAdded → emitItemDone; buildTraceEmitters already documents that pair (~400+). Reuse emitPair / existing emitter helper if it is in closure scope — avoids a third copy of the same fire-and-forget shape.
Comments FIX-2137 rationale is repeated in the block header and the lean branch. One canonical comment at the hook; keep the branch to a single line.
Tests resume-replay-traces-off.test.ts is the right integration tier for durable resume; ~164 lines is justified. Optional nit: the “lean shape” case could assert after the first continuation instead of running the full two-answer flow again (same invariant, less test time).

Alternatives considered (not recommending for this PR)

  1. Dedicated replay completion item (or internal journal) — clearest separation of “DevTool trace” vs “replay skip token”; higher blast radius.
  2. Strip observability fields at persist time while always running the full lifecycle — more item.updated traffic and persisted churn in prod than lean terminal emit.
  3. Third-party package — nothing obvious; this is engine-specific replay semantics.

Performance (acceptable tradeoff)

Prod now retains one small block_trace per completed non-transient block and one persistItems on done (vs zero rows before and vs many item.updated rows with observability on). That is the intended cost of correct resume. Transient blocks stay excluded. Pre-existing dead work in core still building full generator/added payloads when the hook only needs output is not introduced here; could be a separate optimization if profiling says so.

Skipped (would need product/architecture call)

  • Splitting replay ledger from block_trace in the type system / DevTool — worthwhile epic, not a merge blocker for this fix.

No changeset concern from this review; the included engine patch changeset looks appropriate.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: f6355014b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob. Outcome 1: a shorter shape of the same contract. POC (draft, does not merge): #3757 (poc/shorter-traces-off-replay @ 49933dc04).

Fold 49933dc04 onto f6355014b. Do not reset onto the POC. Close #3757 unmerged after this PR lands.

Net −35 (+18/−53). The lean onBlockTraceCapture write is unchanged. What goes:

  • Comments that restated the next line. Two lines stay on the hook: resume reads the record, and itemIndex is taken at completion so the record follows the block's items.
  • The second two-answer drive. One run checks both the pass list and the stored traces.
  • The report step. It only returned sequencer state. ask returns that state when it does not suspend.
  • The resource-load test name still said "records nothing". The expects are the record. The comment that repeated them is gone.

Still passes: resume-replay-traces-off, block-trace-lifecycle, resource-load-trace — 22. eslint exit 0.

Red: an immediate return in the lean branch re-ran pass 1. Expected ["pass 1: brief", "pass 2: answer: A1"]. Received an extra "pass 1: brief".

Left as recorded: the lean item (identity, output, completedAt, duration), skip transient and non-completed, both doc sentences, the changeset, the background-work suite still forcing observability on, every request rather than durability-only.

Tried and left: a shared item constructor (same length). requestTraceEmitters.blockTrace (also writes the trace store). Folding the lean branch into the full hook (index moves to block start, and input is recorded).

View PR

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob. Outcome 1: a shorter shape of the same contract. POC (draft, does not merge): #3757 (poc/shorter-traces-off-replay @ 01bda809e).

Fold 01bda809e onto b91f14990. Do not reset onto the POC. Do not cherry-pick 49933dc04 (cut from f6355014b; it conflicts on the comment b91f14990 reworded, and this branch deletes that comment). Close #3757 unmerged after this PR lands. The POC body still names 49933dc04; this note is the one that counts.

Net −35 (+18/−53) against b91f14990. The lean onBlockTraceCapture write is unchanged. What goes:

  • Comments that restated the next line. Two lines stay on the hook: resume reads the record, and itemIndex is taken at completion so the record follows the block's items.
  • The second two-answer drive. One run checks both the pass list and the stored traces.
  • The report step. It only returned sequencer state. ask returns that state when it does not suspend.
  • The resource-load test name still said "records nothing". The expects are the record. The comment that repeated them is gone.

Still passes: resume-replay-traces-off, block-trace-lifecycle, resource-load-trace — 22 before the rebase. After it, resume-replay-traces-off passed again. eslint exit 0.

Red: an immediate return in the lean branch re-ran pass 1. Expected ["pass 1: brief", "pass 2: answer: A1"]. Received an extra "pass 1: brief".

Left as recorded: the lean item (identity, output, completedAt, duration), skip transient and non-completed, both doc sentences, the changeset, the background-work suite still forcing observability on, every request rather than durability-only.

Tried and left: a shared item constructor (same length). requestTraceEmitters.blockTrace (also writes the trace store). Folding the lean branch into the full hook (index moves to block start, and input is recorded).

View PR

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

cursoragent and others added 2 commits October 23, 2026 18:33
Drop the second drive, the report step that only returned sequencer state,
and comments that restated the next line; rename the stale resource-load
test. Cherry-picked from poc/shorter-traces-off-replay (01bda80). The lean
capture hook is unchanged.

Co-authored-by: Jake Hoffner <jhoffner@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6
…r and /continue replay

- baseBlockTraceFromIdentity builds the identity half of both the full
  `added` item and the lean completion record, so the lean record stays a
  subset of the full one when an identity field is added.
- The lean branch stays a separate branch of the capture hook: the full
  branch opens its row at `added` with the block's input, so an early return
  in it cannot produce the lean record. One comment at the hook states why,
  and that failed, suspended and transient blocks get no lean record.
- Tests: a durable sequencer with a stub-model generator (ref-shaped output)
  calls the model once across suspend and resume; /continue crash recovery
  with no durability provider skips the finished step; the lean trace set
  must equal the completed set full capture records.
- Docs and changeset: an output over the recorded-value limit is kept as an
  omitted placeholder, so a resume that reaches it fails with
  RECORDED_VALUE_OMITTED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: please focus on these areas at 629b4e54b:

  1. Shared builder keeps the lean record a subset. baseBlockTraceFromIdentity in createExecutionContext.ts now builds the identity half of both the full added item and the lean completion record. Check the two things this introduces. First, itemIndex is now taken inside the builder: for the full branch it should still be taken at added, and for the lean branch at output. Second, the full item's field set should be unchanged, with transient, input and declaredResources still added after the spread. Is any identity field left that only one branch sets?
  2. The capture hook's early returns. The lean branch returns early on non-output phases, non-completed statuses, a missing identity and transient blocks. Can a block that replay needs to skip reach output with a status other than completed, for example a step that suspended and then completed after its answer, or a router branch? If so, it would get no record and run again on resume.
  3. Replaying a generator's ref-shaped output. With observability off, a generator's lean trace records output: { kind: "ref", sourceItemId } pointing at its message item. buildReplayLog resolves that against the persisted log. Is the referenced message always persisted before, or with, the lean trace on every store, including a crash between the two writes? Does /continue crash recovery without a durability provider resolve it the same way?

Direction-level findings matter most. Skip style nits.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 629b4e54b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Blocks that emit straight through ctx.response (a generator's message, a
tool output) take the emitter's cursor without advancing the context's
counter, so a later block_trace could reuse or undercut their index. Stores
that order history by itemIndex then reorder the log. Every reservation in
createExecutionContext now goes through reserveItemIndex, which starts at
whichever of the counter and the cursor is further along. The collision also
existed with full observability; the lean record made it reach production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

jhoffner commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: please focus on item-index allocation at 8113e9eac:

  1. Lean vs. full path. reserveItemIndex() in createExecutionContext.ts returns max(emittedItemCount, response.getItemCount()) and advances the counter. It now backs every reservation: the full added trace, the lean completion trace, router decisions, _reserveItemIndex and the emission-context nextItemIndex. Is there a path where an index is reserved and then held across an await before its item is inserted? A block's direct ctx.response emit could then take the same cursor value. Also check the fallback response with no getItemCount, and a continuation seeded with startItemIndex.
  2. Concurrent .map children and parallel branches. Generators and tool outputs read the cursor with getEmitterItemCount() and insert in the same synchronous step. Can two concurrent children interleave a cursor read and an insert so that two items get one index, or so that a lean trace reserved at one child's output lands below an item a sibling already emitted?

Direction-level findings matter most. Skip style nits.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

from: epic-coordinator
session: FIX-2058
kind: reply

PR description addendum for 8113e9ea. This PR also fixes an older bug in item ordering. Block traces and items that a block emits directly could share an index, even with observability on (dev). Every index in createExecutionContext now comes from one reserveItemIndex(). A new test asserts that indexes are unique and strictly increasing in both modes. It failed in both modes before the fix.

Verification at this head: the engine suite passes (4017 passed, 8 skipped), and so does @flow-state-dev/store-sqlite (404 passed). Typecheck is clean.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8113e9eac1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…() is absent

A response with emit()/getItems() but no getItemCount() gave the context a
cursor of 0, so a lean completion record could reuse the index of an item a
block had emitted straight through it. Mirror core's getEmitterItemCount
fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: one narrow area at fab91aad7. responseItemCursor() in createExecutionContext.ts now falls back to getItems().length when the response has no getItemCount(), mirroring core's getEmitterItemCount. Is that length a correct cursor for every partial response we accept? Consider a response whose getItems() drops transient items, or one seeded with a prior log, so its length is not the next free index. And with only emit(), does the counter alone still give unique indexes?


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fab91aad73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A getItems()-only response can hold items that don't start at 0, such as a
seeded log whose one item has index 5; its length (1) is not the next free
index. Use the highest itemIndex plus 1, or the length when no item carries
one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: one narrow area at a4a6dabe9. When a response has no getItemCount(), responseItemCursor() in createExecutionContext.ts now returns the highest itemIndex in getItems() plus 1, or the length when no item carries one. Is that the next free index for every partial response we accept, including one whose getItems() omits transient items? Core's getEmitterItemCount still uses the length for a block's direct emits on such a response. Can that still put a generator message below a seeded index?


Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

@codex review: final pass at a4a6dabe9. Please re-check the whole PR for direction-level problems only. Pay particular attention to these two areas:

  1. Replay correctness with traces off. The lean record is built from baseBlockTraceFromIdentity. Check whether it carries everything resume needs for handlers, generators with ref-shaped outputs, and /continue.
  2. Item-index allocation. This covers reserveItemIndex() and the cursor fallbacks (getItemCount, then the highest getItems() index + 1, then length). Check whether indexes stay unique and monotonic across the full and lean paths and across concurrent .map children.

Skip style nits.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a4a6dabe98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ability provider

Crash recovery replays completed blocks from the log whether or not a
durability provider is configured, but restored the root sequencer's
checkpoint only when one was. A skipped step's state write was then lost.
The checkpoint is read from stores.checkpoints, which the item hook writes
either way, so restore it on every crash-recovery continuation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6
… restore

- A completed block_trace carries `rescued: true` when a .rescue() handler
  produced the output (full and lean records). buildReplayLog exposes it as
  completedRescued, and the sequencer hands it to _registerReplayedChild, so
  ctx.wasRescued() reads the same after a resume.
- The provider-free /continue test now uses a durable stateful sequencer
  whose completed step patches ctx.sequencer.state.
- Changeset and docs: crash recovery is at-least-once for a block whose
  completion had not been persisted when the process died; the rescued mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: please focus on these two areas at 5e231433d:

  1. Checkpoint restore and replay on provider-free /continue. runAction.ts now restores the root sequencer checkpoint from stores.checkpoints on every crash-recovery continuation, not only when a durability provider is set. Can a stale or foreign root checkpoint be restored this way? Examples: one left from an earlier attempt of the same request id, or a non-durable root that never wrote one. Does replay plus the restored state now match what the same flow gets with a provider?
  2. The rescued mark. A completed block_trace carries rescued: true, set from the sibling entry at the output phase on both the full and lean paths. buildReplayLog.completedRescued reads it, and the sequencer passes it to _registerReplayedChild. Are there replay paths that register a replayed child without it, such as a router branch, a .map/forEach child or a parallel branch, so ctx.wasRescued() could still change after a resume?

Direction-level findings matter most. Skip style nits.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e231433d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Revert the provider-free root-checkpoint restore. Crash recovery with no
durability provider now reads a replay log with no completed outputs, so it
runs every block again from the top; router decisions, resolved gates and the
prior items stay readable. The /continue test covers both: with a provider
the finished step replays with its restored state; without one it re-runs
once on a fresh state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6
Gate the no-completed-outputs replay log on trace observability being off
as well, so provider-free crash recovery is unchanged in both modes: with
observability on it still skips completed blocks from their full traces
(the documented DevTool Continue), and with it off it still re-runs them.
The /continue test runs over provider and observability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: one narrow area at a837e980e, the replay gate in runAction.ts. On crash recovery (no resumeContext) with no durability provider and trace observability off, the replay log is replaced by withoutCompletedOutputs(replayLog). That copy keeps router decisions, resolved gates and items(), and blanks getCompletedOutput, recordedBlockTraceId and completedRescued. The aim is that this path behaves exactly as before this PR.

  • Does anything still skip a block on this path?
  • Could a router, a generator resume or the continuation item read the blanked methods and behave differently from before?
  • Is a suspension resume (with a resumeContext) or any provider-backed path ever caught by the gate?

Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a837e980e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Mark the lean completion record `lean: true` (optional in contracts, absent
on full traces and old records). Provider-free crash recovery builds its
replay log with skipLeanTraces, so it skips a completed block only from a
full trace, exactly as before lean records existed, whatever
FSDEV_TRACE_OBSERVABILITY says at recovery. Drops the env check and the
withoutCompletedOutputs copy. The /continue test now flips the setting
between the run and the recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

Copy link
Copy Markdown
Collaborator Author
from: issue-worker
session: FIX-2058
kind: reply

@codex review: one narrow area at 9a7711652, the provider-free /continue gate.

runAction.ts builds the replay log with skipLeanTraces when there is no resumeContext and no durability provider. buildReplayLog then ignores completed traces marked lean: true when it selects completed outputs. Ref lookup still sees every item.

  • Is lean: true set on every lean record and never on a full trace?
  • Can a full trace's output ref resolve through a lean trace, so that skipping lean traces changes what a full trace replays?
  • Does any provider-backed or suspension-resume path reach this option?

Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 9a7711652a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jhoffner pushed a commit that referenced this pull request Oct 23, 2026
…e runs the resume path's ownership checks; a cancelled task refuses new filings

- waitForUser has three kinds: approval, review, input. A question with a
  task row holds its task (FIX-2133, ER-8); with none it is input.
- D2: the answer route runs ownsRecord + callerReachesRequest (flow
  instance, user, tenant, org) before it loads or resolves the stop.
- A filing under an already-cancelled parent is refused, since the
  cascade covers only work filed up to the cancel's time.
- Goal leg e: a crash between the cancel's row write and the resume.
  Goal check runs with FSDEV_TRACE_OBSERVABILITY=0; a stored-user-only
  control.
- Depends on FIX-2137 (#3749, unmerged) for approval resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6
@jhoffner
jhoffner merged commit 3b0161c into main Oct 23, 2026
19 checks passed
@jhoffner
jhoffner deleted the fix/FIX-2137 branch October 23, 2026 20:10
jhoffner added a commit that referenced this pull request Oct 23, 2026
#3735)

* spec(FIX-2129): a stop for a person holds its task

waitForUser on ctx.suspend, the person mark keyed on one predicate with
the stop recorded on the row, four endings through one fence, and one
trusted answer route. FIX-2130 folded in, as the epic decided.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): review round 1 — the deadline error is SuspensionTimeoutError, a stop holds back no notice

D1 names the error core already exports and nothing throws, and says
plainly that an approval keeps FIX-1846's no. The deadline is one clock
(expiresAt); the 7-day default is waitForUser's only. A person's stop
never holds back a notice (ER-2). A cancelled block's effects outside
the task land; the 'filed after cancel never starts' guarantee is cut.
Docs line gains the concept amendment and a changeset. Review notes
recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): review round 2 — the answer route finds a stop by request and id; a written cancel beats a late answer

Folds Codex's two P1 findings on PR #3735: a stop's id is unique only
within its request, so D2's route takes the pair; and an answer landing
between the cancel's row write and its debt must lose. Adds the
architecture execution-and-errors page to the docs line (P2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): to the epic amendment — no question kind; answer route runs the resume path's ownership checks; a cancelled task refuses new filings

- waitForUser has three kinds: approval, review, input. A question with a
  task row holds its task (FIX-2133, ER-8); with none it is input.
- D2: the answer route runs ownsRecord + callerReachesRequest (flow
  instance, user, tenant, org) before it loads or resolves the stop.
- A filing under an already-cancelled parent is refused, since the
  cascade covers only work filed up to the cancel's time.
- Goal leg e: a crash between the cancel's row write and the resume.
  Goal check runs with FSDEV_TRACE_OBSERVABILITY=0; a stored-user-only
  control.
- Depends on FIX-2137 (#3749, unmerged) for approval resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): a cross-board child of a cancelled task is never claimable; racing leg c' and a cancel-after-add control

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): a cross-board child lands cancelled in its own write, read against the committed parent; leg c' latches the add window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): cross-spec alignment — the person-stop host setting replaces FIX-2133's question-deadline constant; FIX-2137 merged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): waitingOn "person" becomes "user" (owner), the old value dual-read under BP-030

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): waitingOn rename is plain; no persisted-data dual-read

Owner, 2026-10-23: stores are reset, so no row or suspension from before
this work is read. Drops the "person" dual-read, goal leg a's seeded
legacy row and the changeset's stored-rows note; records the waiver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

* spec(FIX-2129): goal leg f proves the person-stop host default

FIX-2129 swaps FIX-2133's 7-day constant for the host setting, so it owns
the proof: a non-7-day default persists on a defaulted waitForUser and a
held question, while a raw ctx.suspend keeps its own timeoutMs (Codex P2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vdxnv28JumHyq2za2ybqz6

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants