Skip to content

feat(mcp): derive an API surface view from retained web network captures - #1891

Merged
morluto merged 20 commits into
morluto:mainfrom
mvanhorn:feat/web-capture-api-surface-view
Oct 12, 2026
Merged

morluto merged 20 commits into
morluto:mainfrom
mvanhorn:feat/web-capture-api-surface-view

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Agents can now turn a retained HAR capture into its API surface: inspect_analysis_view with view: {"kind": "api-surface"} groups recorded requests by method, origin and path template, and each group cites the parent record ordinals it came from. It builds on the retained-view workflow from #1180 and adds no new tool.

No issue is linked. This is a new view kind on an existing tool, proposed here with the implementation.

Demo

API surface view demo

Full-quality MP4

The sample is a generated 22-request HAR for a made-up shop.example.test, not third-party traffic. Every number in the video is copied from the CLI output on this branch.

Problem and expected behavior

inspect_web_network_capture returns one record per request in producer order. On a single-page app that means many near-identical records (GET /v1/users/1042, /v1/users/2210, ...), and an agent has to read all of them to answer "which endpoints does this app call, with what statuses". Nothing in REA groups recorded requests today.

With this change, inspect_analysis_view reads a retained (or inline) inspect_web_network_capture HAR Evidence and returns groups (method, origin, path_template, templated_segments, record_ordinals, statuses, content_types and coarse flags such as single-sample, mixed-statuses and mixed-content-types), a records summary of examined, total and excluded counts that stays separate from group paging coverage, and excluded_records with a reason for each record whose URL is redacted (null) or unparseable, or whose request or method is missing.

Change and scope

Path segments are templated only when they match one of four exact formats: numeric (ASCII digits), uuid (RFC 9562 text form), hex (32+ hex digits) and iso-date (a valid calendar YYYY-MM-DD). Placeholders are named after the rule ({numeric}), never after a neighbouring segment, and there is no fuzzy or cross-sample matching. So /api/v1 vs /api/v2, PascalCase routes like /api/CreateDocument vs /api/ListDocuments, the invalid date 2026-13-45 and short ids like abc123 stay literal. I kept the rule set this narrow on purpose: anything looser starts inventing structure the capture doesn't show.

A format match is still a pattern match, so each templated segment carries basis: "inferred" with every observed literal, and the view Evidence uses confidence: "inferred" when any segment on the page was templated. Pages with only literal paths stay derived, and the existing view kinds are unchanged. No header, body, query or fragment values are copied into groups, and query strings are not part of the group key.

The api-surface result variant has its own artifact schema with a nullable digest, because a capture writes artifact.sha256: null when the digest matched a sensitive_values declaration. The parent-digest integrity check runs only when both digests exist, so the shared result schema used by the other views is not relaxed.

The view reads the recorded path of canonical scheme://authority/path URLs before WHATWG dot-segment removal, so /a/%2e%2e/admin and /admin stay apart; other spellings (https:host/x, https:///user@host/x) are excluded as url-unparseable rather than guessed. Parents inspected with sensitive_values are refused with a typed unsupported error, because the view writes templates and limitation text it cannot check against declarations the parent does not retain. Each path is bounded before splitting, and only newly retained templated segments and distinct observed values count against the capture's 96 MiB reply budget.

Out of scope: native mitmproxy parents (typed unsupported error), body or schema inference (#882), handler identity or causality claims (#774), and any fetching or replay.

Contract and boundary impact

  • Semantic owner and earliest changed stage: domain src/domain/analysisView/webNetworkApiSurfaceView.ts, dispatched from projectAnalysisView; AnalysisViewService picks the Evidence confidence.
  • CLI and MCP/tool-catalog contract: inspect_analysis_view / inspect-analysis-view accept view.kind: "api-surface" with optional offset/limit; parent_operation adds inspect_web_network_capture. Contract example in src/contracts/analysisViewApiSurfaceExample.ts comes from a fixture run.
  • Provider, bridge, target-format, or platform compatibility: none. HAR parents only.
  • Evidence, artifact, provenance, or reconstruction contract: derived view Evidence links to the parent capture; confidence is inferred when templating fired; record ordinals, exclusions and parent limitations are preserved.
  • Process execution, authorization, cleanup, or containment impact: none beyond recording the derived Evidence record.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: authored skill text in .agents/skills/reverse-engineer-anything/references/evidence-workflows.md; no version bump needed after rebasing on main (already 39), and metadata:check passes. Build outputs not committed.

Evidence and regression coverage

  • Tests added or updated: webNetworkApiSurfaceView.test.ts (sensitive parents refused; HAR status: 0 and empty MIME treated as unknown; per-path and retained-value budgets; slashless and authority-less URLs excluded; encoded dot segments kept; literal {numeric} paths kept apart from templated ones; each rule fires and nothing else does; v1/v2, Report2/Report3 and PascalCase pairs stay separate; deterministic result for the same parent; group paging exhausts; ordinal-subset parent; null/unparseable URLs excluded; mitmproxy parent unsupported); AnalysisViewService.test.ts (inferred vs derived, existing views unchanged, null-digest parent); MCP and CLI boundary tests; scripts/verify-web-network-captures.mjs gains an api-surface step.
  • Base reproduction or other evidence: source-derived. On bcd350c the view request union has no api-surface kind and parent_operation has no inspect_web_network_capture, so a capture parent cannot be projected.
  • User-visible CLI/MCP output: the 22-request sample HAR gives 12 groups with records: {examined: 22, total: 22, excluded: 0} and confidence: "inferred"; capture output 41,717 bytes, view output 6,673 bytes. Example group: GET /v1/users/{numeric}, record_ordinals: [1, 2, 3, 4, 5], statuses: [200, 404], flags: ["mixed-statuses"], observed values 1042, 2210, 3377, 4120, 9999.
  • Remaining proof gaps: verify:web:network-captures needs Linux and mitmdump 12.2.3, so the new step did not run on this macOS host. The api-surface path is covered by the MCP and CLI boundary tests.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

All on macOS with Node 26 and the pinned npm 11.16.0.

npm run test:focused -- tests/boundary/cli/analysisViewCli.test.ts tests/boundary/mcp/analysisViewMcp.test.ts \
  src/domain/analysisView/webNetworkApiSurfaceView.test.ts src/application/analysisView/AnalysisViewService.test.ts \
  src/domain/analysisView/analysisView.test.ts          # pass: 5 files, 60 tests
npm run check:fast                                      # pass
npm run check                                           # pass
npm run docs:check                                      # pass
npm run verify:test-discovery                           # pass
rea inspect-web-network-capture shop.har har --json     # sample HAR, then:
rea inspect-analysis-view view.json --json              # 12 groups, output as above

npm run check:pr had six failures on the first run. Five (process and Hopper timing) passed on rerun. The remaining one, machoSliceReader.test.ts ("checks real /bin/ls slice ranges against its FAT table"), fails the same way on bcd350c without this change on this host.

Compatibility, safety, and release

  • Breaking changes or migration steps: none. Additive view kind.
  • Real Hopper/Ghidra, browser, or OS coverage: macOS only; Linux mitmdump lane not run.
  • Package or release metadata impact: skill reference text only.
  • Security, privacy, process, or containment review: the view reads caller-supplied or same-session Evidence only, copies no header, body or query values, and makes no network requests.

Review checklist

  • This PR addresses a concrete problem or an agreed enhancement.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T02:44:38.965628Z 98dac58 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fcadbb06c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/domain/analysisView/analysisView.ts Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d1ea92458

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/verify-web-network-captures.mjs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64dcd2412a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a45482093

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mvanhorn
mvanhorn force-pushed the feat/web-capture-api-surface-view branch from 0a45482 to 0ee5971 Compare October 11, 2026 18:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ee597123c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5c3b9a966

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/domain/analysisView/analysisView.ts Resolved
Comment thread docs/mcp-contracts.md Resolved
Charge retained deduplication sets before allocation, advertise only
each analysis operation's summary from its detail output, and document
that any templated group makes every API-surface page inferred.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e848938506

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Charge retained literal path copies before allocation, publish only
valid media-type essences, and deduplicate examined record limitations
onto the view.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e0a725f15

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…budget

Sensitive-parent refusals now use the token sensitive_values_unretained so declared literals are not echoed. A later path split is charged against bytes already retained, and the published example includes the HAR record limitation with its matching digest.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef7b437e9d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/web-network-captures.md Resolved
Charge method and origin text against the reply budget before the group key is serialized. Document that samples merge only when method, origin, and exact segment rules match.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d926092969

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Reject authorities containing a tab or newline so WHATWG cannot move userinfo into the recorded path. Charge four path copies, two method and origin copies, and each distinct content type against bytes already retained.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6050bed14

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Total path, method, origin, and segment charges before retaining a new group. Track content types in a set so distinct MIME essences are not compared quadratically.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e74e96fe7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Create the per-group content-type set when the first essence is retained, and charge it before allocation. Groups with no content type no longer keep an empty set.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05e58952c4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Own grouping, deduplication, and combined retained/temporary resource admission in one accumulator. Charge fixed group storage before allocation and materialize only the requested page.

Share recorded URL parsing between capture redaction and projection, preserving source paths and excluding ambiguous credential fields. Cover resource admission invariants and the compiled HAR-to-view workflow.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a52a7deac4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56e1dea0c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69d3d69fb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@morluto
morluto merged commit 880a603 into morluto:main Oct 12, 2026
31 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98dac582dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

parent_evidence_id: parent.evidenceId,
parent_operation: "inspect_web_network_capture",
parent_digest: parent.evidenceId.slice(3),
artifact: { path: capture.artifact.path, sha256: capture.artifact.sha256 },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Charge artifact path text before sealing retained views

For a session record imported from a portable Evidence bundle, artifact.path is an unconstrained string and admitCaptureClone deliberately charges no scalar text. Placing a multi-megabyte path in the result here therefore leaves the accumulator counter nearly unchanged, while sealing the view and then creating the child Evidence each encode that path with the retained parent still live; the projection can bypass its 96 MiB representation budget and block or exhaust the synchronous process before MCP transport admission rejects the response. Charge projected identity text against the accumulator before returning it.

AGENTS.md reference: AGENTS.md:L44-L44

Useful? React with 👍 / 👎.

method,
parsed.origin,
route.value,
typeof status === "number" && status !== 0 ? status : null,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat non-integer HAR statuses as unavailable

For portable inline capture Evidence with response.status: 200.5, the capture schema accepts the reported value as generic JSON and this condition publishes 200.5 in statuses (and can use it to set mixed-statuses). HAR response statuses are integers, so malformed numeric producer values should remain unknown just like status 0 and malformed MIME values rather than becoming response facts; require an integer before retaining the status.

AGENTS.md reference: AGENTS.md:L23-L23

Useful? React with 👍 / 👎.

@github-actions github-actions Bot mentioned this pull request Oct 12, 2026
3 tasks done
@mvanhorn

Copy link
Copy Markdown
Contributor Author

@morluto thanks for merging the api-surface view from HAR captures, plus raw binary admission in #1868 and package import aliases in #1862. good few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: mcp MCP server, transport, and tool registration enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants