Skip to content

Add execution trace system for block lifecycle events - #80

Merged
jhoffner merged 4 commits into
mainfrom
claude/implement-fix-135-QZFUS
Mar 16, 2026
Merged

jhoffner merged 4 commits into
mainfrom
claude/implement-fix-135-QZFUS

Conversation

@jhoffner

@jhoffner jhoffner commented Mar 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements a comprehensive execution trace system that persists block lifecycle events (start, completion, failure) and router decisions as structural items in the response stream. These trace items are marked with trace: true to exclude them from LLM context while maintaining visibility for debugging and monitoring.

Key Changes

  • New item types: Added BlockToolOutputItem and RouterDecisionItem types to represent tool calls and routing decisions as first-class items
  • Lifecycle tracking: Blocks now emit block_output items with timing metadata (startedAt, completedAt, duration) and blockKind for nested block execution
  • Router decision tracing: Router blocks emit router_decision items when a route is selected, including the router name and selected route
  • Trace flag filtering: Added trace boolean field to OutputItemBase to mark structural items; these are excluded from LLM context via itemToLLMMessage()
  • Nested block tracing: Modified _withExecutionScope in createExecutionContext.ts to emit lifecycle items for nested blocks (sequencer steps, etc.) while avoiding duplicates for root blocks
  • Root block tracing: Updated executeBlock.ts to emit trace items for root-level block execution with complete timing information
  • Trace tree visualization: Enhanced trace-tree.ts to use precise timing metadata from lifecycle items and support new item types (router decisions, tool outputs)
  • Block identity threading: Added _blockIdentity internal property to BlockContext so nested code (generator tool outputs, router decisions) can construct proper provenance with parentBlockInstanceId
  • Tool output emission: Generators now emit block_tool_output items when tools execute, with proper provenance for trace tree nesting
  • Documentation: Updated streaming architecture docs with new item types and trace flag semantics

Implementation Details

  • Trace items are emitted via the existing emitItemAdded/emitItemDone response emitter interface
  • Nested blocks only emit trace items when parentChain !== undefined to prevent duplication with root block tracing
  • Router decision items are emitted immediately after route selection in the execution context
  • The itemToLLMMessage() function filters out all items with trace: true before converting to LLM messages
  • block_tool_output items are NOT marked as trace — they enter LLM context for multi-turn tool calling
  • Timing metadata is preserved in trace items for accurate performance analysis
  • _blockIdentity is set by _withExecutionScope and used by generator tool output emission and router decision emission for correct provenance
  • Comprehensive test suite validates trace emission, LLM filtering, provenance correctness, and router decision identity
Open in Web Open in Cursor 

@linear

linear Bot commented Mar 13, 2026

Copy link
Copy Markdown

@vercel

vercel Bot commented Mar 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
flow-state-dev-docs Ready Ready Preview, Comment Mar 15, 2026 9:57pm
flow-state-dev-kitchen-sink Ready Ready Preview, Comment Mar 15, 2026 9:57pm

@jhoffner

Copy link
Copy Markdown
Collaborator Author

@claude can you please review this code

@jhoffner

Copy link
Copy Markdown
Collaborator Author

@cursoragent please review this code

@cursor

cursor Bot commented Mar 13, 2026 •

Copy link
Copy Markdown
Contributor

The GitHub CLI doesn't have write access to post comments. Here's the full review:


Code Review: Execution Trace System (PR #80)

All 140 server tests pass. The overall design is solid — using trace: true to separate structural/lifecycle items from LLM context is clean, and the nested vs. root block tracing split is well-reasoned. Here are the findings:

1. Fire-and-forget promise for router decision emission (Bug)

File: packages/server/src/context/createExecutionContext.ts:1679

void typed.emitItemAdded(decisionItem).then(() => typed.emitItemDone!(decisionItem));

This silently swallows errors and does not await completion. Compare with the nested block tracing ~70 lines later, which properly awaits both emitItemAdded and emitItemDone. If the emitter throws (e.g. store write failure), the error vanishes. The void pattern also means the router decision item might not be fully persisted before downstream blocks start executing.

Suggestion: Await both calls, matching the nested block tracing pattern.

2. Synthetic blockInstanceId for router decisions creates orphan trace nodes (Design)

File: createExecutionContext.ts:1671

blockInstanceId: `${routerName}_decision`,

This doesn't match the router block's actual blockInstanceId. In trace-tree.ts, block nodes are grouped by provenance.blockInstanceId and parented via parentBlockInstanceId. This synthetic ID will create an orphan block node in the trace tree instead of appearing under the router block.

3. BlockToolOutputItem type is defined but never emitted (Dead code)

The type, LLM conversion, and trace-tree handling are all wired up, but nothing in the server code actually creates BlockToolOutputItem items. The comment labels block_output with toolCall as "legacy," but the migration isn't in this PR. Fine as groundwork for a follow-up, but it's currently dead code with no test coverage for the emission path or the itemToLLMMessage branch.

4. trace: true early-return in itemToLLMMessage is fragile (Design)

The trace check runs before all type-specific logic. If a future code path ever sets trace: true on a block_output with toolCall (or a block_tool_output), the tool output would silently vanish from LLM context, breaking multi-turn tool calling. Consider either adding a comment making the invariant explicit or moving the trace check after tool-output handling.

5. Duplicated response casting pattern (Code quality)

The responseRef.current as { emitItemAdded?, emitItemDone? } cast with capability check appears in two places. Could be extracted into a small helper.

6. Inconsistent item ID generation (Minor)

Root uses Date.now() as the discriminator; nested uses traceItemIndex. Both have random suffixes so they're unique, but the inconsistency is surprising.

7. Tests don't verify actual LLM filtering (Test gap)

The test "marks trace items with trace: true so they are excluded from LLM context" only checks the flag is set — it doesn't verify itemToLLMMessage() actually returns null. Testing indirectly via a multi-turn generator that builds LLM history would provide stronger confidence. Tests also use toBeGreaterThanOrEqual(1) everywhere, which won't catch duplicate or missing emissions.

8. Missing documentation update

The streaming docs item type table (docs/architecture/streaming.md:32-44) needs block_tool_output and router_decision. The trace boolean on OutputItemBase should also be documented as a cross-cutting filtering mechanism.

Summary

Priority Issue
Fix #1 — Await router decision emission (silent error swallowing)
Fix #2 — Router decision blockInstanceId creates orphan trace nodes
Address #4 — Document or restructure the trace vs toolCall ordering invariant
Address #8 — Update streaming docs for new item types
Note #3, #5, #6, #7 — Clean up or track for follow-up
Open in Web Open in Cursor 

jhoffner and others added 2 commits March 14, 2026 10:43
…ock outputs

- Introduced new BlockOutputItem and RouterDecisionItem types to capture detailed lifecycle events.
- Updated buildTraceTree function to utilize block timing and status from lifecycle metadata.
- Implemented emitNestedBlockTrace function for efficient trace emission during block execution.
- Added tests to validate the correct emission of trace items with timing and status information.

This update improves the observability of block execution flows and enhances debugging capabilities.
When a generator invokes a tool block, the framework now emits a
block_tool_output item capturing the tool result, call ID, and
generator block name. This item enters LLM context (not marked as
trace) so the model can see tool results in conversation history.

Changes:
- Add BlockToolOutputItem type to core/items
- Update GeneratorModelTool.execute signature to accept toolCallId
- Emit block_tool_output in compileToolsWithExecute after tool runs
- Handle block_tool_output in itemToLLMMessage (role: "tool")
- Add block_tool_output to LLM_AUDIENCE_TYPES
- Update devtool trace-tree to infer generator kind from tool outputs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jhoffner

Copy link
Copy Markdown
Collaborator Author

@cursoragent review the changes, make sure everything was covered from your past review, and determine if there is anything new we should be paying attention to.

@cursor

cursor Bot commented Mar 15, 2026 •

Copy link
Copy Markdown
Contributor

The GitHub CLI still can't post comments. Here's the complete follow-up review:


Follow-Up Review: Execution Trace System (PR #80)

All tests pass (226 core, 143 server). Two new commits since my last review: 21588d2 (lifecycle metadata refactoring) and b4be364 (BlockToolOutputItem emission). Plus merged PRs bringing performance and resource snapshot changes.


Previous Review — Status Check

# Finding Status Notes
1 Fire-and-forget router emission Addressed Now uses .catch(() => {}) with "best-effort" comment. Pattern applied consistently to emitNestedBlockTrace too. No unhandled rejections.
2 Synthetic blockInstanceId for router decisions Partially addressed Changed from ${routerName}_decision to ${routerName}_${requestId}, which groups per-request. Still orphan in trace tree (see N2 below).
3 BlockToolOutputItem never emitted Fully addressed compileToolsWithExecute in generator.ts now emits block_tool_output items with test coverage.
4 trace: true early-return fragility Addressed JSDoc on OutputItemBase.trace and inline comments on emitBlockOutputItem document the invariant.
5 Duplicated response casting Fully addressed Extracted to shared emissionResponse with duck-type fallback.
6 Inconsistent item ID generation Partially addressed Different prefixes (item_trace_, item_router_, item_block_output_) are useful for debugging. Minor discriminator inconsistency remains.
7 Tests don't verify LLM filtering Not addressed Tests still only check trace: true flag, not that itemToLLMMessage() returns null.
8 Missing streaming docs update Not addressed docs/architecture/streaming.md item type table still missing block_tool_output, router_decision, and trace flag docs.

New Issues

N1. block_tool_output provenance lacks parentBlockInstanceId — orphan in trace tree

In packages/core/src/blocks/generator.ts:354-358:

provenance: {
  blockName: tool.name,
  blockInstanceId: tool.name,
  phase: "main" as const
},

Two problems:

  • No parentBlockInstanceId — the trace tree builds parent-child relationships via provenance.parentBlockInstanceId. Without it, tool outputs appear as root-level blocks instead of nesting under their generator.
  • blockInstanceId: tool.name is static — if the same tool executes 3 times in one request, all 3 outputs share the same blockInstanceId and merge into one block node. This differs from every other block where instance IDs are unique.

N2. Router decision blockInstanceId still creates orphan trace node

The onRouteSelected callback only receives (routerName, selectedBlockName), not the router's actual blockInstanceId. The synthetic ${routerName}_${requestId} won't group with the router's other items. This is an architectural limitation — fixing it requires threading the instance ID through the runtime hooks.

N3. block_tool_output emitted via ctx.response.emit(), not emitItemAdded/emitItemDone

In generator.ts:369-370, the generator uses the generic emit() method, while trace items in createExecutionContext.ts use the specialized emitItemAdded/emitItemDone. Both work, but if the direct methods ever do anything beyond event dispatch, this path would bypass it. Worth noting for consistency.

N4. Early user message emission could leave orphan items on context creation failure

In runAction.ts, user messages are now emitted before createExecutionContext. If context creation throws (not a parse error), the response has a user message but no execution context was established. The deferred parseError pattern is clean, but the context-failure edge case is worth verifying.


Good changes since last review

  • emitNestedBlockTrace extracted as a clean helper with single-phase emission (no two-phase added/done split)
  • RouterDecisionItem.reason field removed — it was unused and speculative
  • JSDoc comments added to OutputItemBase.trace, BlockOutputItem timing fields, and BlockToolOutputItem
  • Router tests properly await fire-and-forget emissions with setTimeout(50)
  • GeneratorModelTool.execute signature updated to accept options?: { toolCallId?: string }

Updated Summary

Priority Issue
Address N1 — block_tool_output provenance lacks parentBlockInstanceId, creates orphan trace nodes
Address #8 (still open) — Update streaming docs for new item types and trace flag
Consider N2 — Router decision orphan (architectural, may warrant follow-up)
Note N3, N4 — Consistency and edge case items
Note #7 (still open) — Test gap for LLM filtering behavior
Open in Web Open in Cursor 

cursoragent and others added 2 commits March 15, 2026 21:51
…and router decisions

- Add _blockIdentity internal property to BlockContext exposing the
  current block's name, instanceId, and parentBlockInstanceId
- Set _blockIdentity in _withExecutionScope so nested blocks (tool
  executions, router routes) inherit proper provenance
- Use _blockIdentity in generator.ts for block_tool_output provenance,
  fixing orphan trace nodes (parentBlockInstanceId was missing)
- Thread blockInstanceId through onRouteSelected callback so router
  decision items use the router's actual instance ID instead of a
  synthetic one
- Fixes review items N1 and N2: trace tree now correctly parents tool
  outputs under their generator and router decisions under their router

Co-authored-by: Jake Hoffner <jhoffner@users.noreply.github.com>
…ag to item type docs

test(trace): add LLM filtering and _blockIdentity provenance tests

- Document block_tool_output and router_decision in the item type table
- Add Trace Flag section explaining the trace: true filtering mechanism
- Update emission rules to mention block_tool_output and router_decision
- Add test verifying trace items are excluded from LLM history while
  block_tool_output items pass through correctly
- Add test verifying nested block trace items have parentBlockInstanceId
- Add test verifying router decisions use the actual router blockInstanceId

Co-authored-by: Jake Hoffner <jhoffner@users.noreply.github.com>
@jhoffner
jhoffner merged commit 371a1b4 into main Mar 16, 2026
4 checks passed
@jhoffner
jhoffner deleted the claude/implement-fix-135-QZFUS branch March 16, 2026 21:50

This branch was successfully deployed

2 active deployments
Preview – flow-state-dev-docs — 585b014c Deployed Mar 15, 2026 by vercel[bot]
Preview – flow-state-dev-kitchen-sink — 585b014c Deployed Mar 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants