Repository navigation
Add execution trace system for block lifecycle events - #80
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@claude can you please review this code |
|
@cursoragent please review this code |
|
The GitHub CLI doesn't have write access to post comments. Here's the full review: Code Review: Execution Trace System (PR #80)All 140 server tests pass. The overall design is solid — using 1. Fire-and-forget promise for router decision emission (Bug)File: void typed.emitItemAdded(decisionItem).then(() => typed.emitItemDone!(decisionItem));This silently swallows errors and does not await completion. Compare with the nested block tracing ~70 lines later, which properly Suggestion: Await both calls, matching the nested block tracing pattern. 2. Synthetic
|
| Priority | Issue |
|---|---|
| Fix | #1 — Await router decision emission (silent error swallowing) |
| Fix | #2 — Router decision blockInstanceId creates orphan trace nodes |
| Address | #4 — Document or restructure the trace vs toolCall ordering invariant |
| Address | #8 — Update streaming docs for new item types |
| Note | #3, #5, #6, #7 — Clean up or track for follow-up |
…ock outputs - Introduced new BlockOutputItem and RouterDecisionItem types to capture detailed lifecycle events. - Updated buildTraceTree function to utilize block timing and status from lifecycle metadata. - Implemented emitNestedBlockTrace function for efficient trace emission during block execution. - Added tests to validate the correct emission of trace items with timing and status information. This update improves the observability of block execution flows and enhances debugging capabilities.
When a generator invokes a tool block, the framework now emits a block_tool_output item capturing the tool result, call ID, and generator block name. This item enters LLM context (not marked as trace) so the model can see tool results in conversation history. Changes: - Add BlockToolOutputItem type to core/items - Update GeneratorModelTool.execute signature to accept toolCallId - Emit block_tool_output in compileToolsWithExecute after tool runs - Handle block_tool_output in itemToLLMMessage (role: "tool") - Add block_tool_output to LLM_AUDIENCE_TYPES - Update devtool trace-tree to infer generator kind from tool outputs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5e341bd to
b4be364
Compare
|
@cursoragent review the changes, make sure everything was covered from your past review, and determine if there is anything new we should be paying attention to. |
|
The GitHub CLI still can't post comments. Here's the complete follow-up review: Follow-Up Review: Execution Trace System (PR #80)All tests pass (226 core, 143 server). Two new commits since my last review: Previous Review — Status Check
New IssuesN1. In provenance: {
blockName: tool.name,
blockInstanceId: tool.name,
phase: "main" as const
},Two problems:
N2. Router decision The N3. In N4. Early user message emission could leave orphan items on context creation failure In Good changes since last review
Updated Summary
|
…and router decisions - Add _blockIdentity internal property to BlockContext exposing the current block's name, instanceId, and parentBlockInstanceId - Set _blockIdentity in _withExecutionScope so nested blocks (tool executions, router routes) inherit proper provenance - Use _blockIdentity in generator.ts for block_tool_output provenance, fixing orphan trace nodes (parentBlockInstanceId was missing) - Thread blockInstanceId through onRouteSelected callback so router decision items use the router's actual instance ID instead of a synthetic one - Fixes review items N1 and N2: trace tree now correctly parents tool outputs under their generator and router decisions under their router Co-authored-by: Jake Hoffner <jhoffner@users.noreply.github.com>
…ag to item type docs test(trace): add LLM filtering and _blockIdentity provenance tests - Document block_tool_output and router_decision in the item type table - Add Trace Flag section explaining the trace: true filtering mechanism - Update emission rules to mention block_tool_output and router_decision - Add test verifying trace items are excluded from LLM history while block_tool_output items pass through correctly - Add test verifying nested block trace items have parentBlockInstanceId - Add test verifying router decisions use the actual router blockInstanceId Co-authored-by: Jake Hoffner <jhoffner@users.noreply.github.com>


Summary
Implements a comprehensive execution trace system that persists block lifecycle events (start, completion, failure) and router decisions as structural items in the response stream. These trace items are marked with
trace: trueto exclude them from LLM context while maintaining visibility for debugging and monitoring.Key Changes
BlockToolOutputItemandRouterDecisionItemtypes to represent tool calls and routing decisions as first-class itemsblock_outputitems with timing metadata (startedAt,completedAt,duration) andblockKindfor nested block executionrouter_decisionitems when a route is selected, including the router name and selected routetraceboolean field toOutputItemBaseto mark structural items; these are excluded from LLM context viaitemToLLMMessage()_withExecutionScopeincreateExecutionContext.tsto emit lifecycle items for nested blocks (sequencer steps, etc.) while avoiding duplicates for root blocksexecuteBlock.tsto emit trace items for root-level block execution with complete timing informationtrace-tree.tsto use precise timing metadata from lifecycle items and support new item types (router decisions, tool outputs)_blockIdentityinternal property toBlockContextso nested code (generator tool outputs, router decisions) can construct proper provenance withparentBlockInstanceIdblock_tool_outputitems when tools execute, with proper provenance for trace tree nestingImplementation Details
emitItemAdded/emitItemDoneresponse emitter interfaceparentChain !== undefinedto prevent duplication with root block tracingitemToLLMMessage()function filters out all items withtrace: truebefore converting to LLM messagesblock_tool_outputitems are NOT marked as trace — they enter LLM context for multi-turn tool calling_blockIdentityis set by_withExecutionScopeand used by generator tool output emission and router decision emission for correct provenance