Skip to content

fix(web-capture): bound the mitmproxy headline facet index - #2876

Merged
morluto merged 40 commits into
morluto:mainfrom
Dragolone:fix/web-capture-preview-mitmproxy-headline
Nov 6, 2026
Merged

morluto merged 40 commits into
morluto:mainfrom
Dragolone:fix/web-capture-preview-mitmproxy-headline

Conversation

@Dragolone

@Dragolone Dragolone commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The mitmproxy headline preview from the earlier revisions of this PR landed on main in f690571. This PR now carries only the two follow-up review fixes that came after that revision. Both bound the per-record facet index that webNetworkCaptureView.ts builds for mitmproxy HTTP records.

As landed on main, indexFacets has two problems:

  1. Unbounded ancestor expansion. It adds every strict ancestor of every facet pointer to containers. pointerSchema puts no length or depth bound on facet pointers, so a single 1,000,000-character pointer such as "/".repeat(1_000_000) produces a million set entries.
  2. Indexing facets nobody queries. It groups every numeric sidecar and byte descriptor by pointer and copies all their keys into scalars. The mitmproxy bridge emits one sidecar per number, so a record with a large integer array under an unrelated extension allocates an entry for each element. Only /type, the six target/status coordinates and their ancestors are ever queried.

Problem and expected behavior

Expected: the index size depends neither on how many facets a record has nor on how long or deep their pointers are, and every headline decision is unchanged.

Change and scope

indexFacets now keeps a facet only when its pointer is a headline coordinate or one of their ancestors: "", /type, /request, /response and the six target/status fields. It keeps at most two facets per pointer, because a second facet already fails every representation check.

Every other facet only marks the headline coordinates it lies beneath. It does this with startsWith against seven fixed "<coordinate>/" prefixes, so no strings are derived from its pointer.

Not included: moving headline normalization into the mitmproxy producer. That would interpret native state in the bridge and add a field to the shared record schema; it is under discussion in the review thread.

Contribution scope: a resource-bound fix in existing code; a maintainer scope decision is not applicable.

Contract and boundary impact

  • Semantic owner and earliest changed stage: indexFacets in src/domain/analysisView/webNetworkCaptureView.ts.
  • Contribution scope fit and linked maintainer decision: in scope; not applicable.
  • CLI and MCP/tool-catalog contract: none.
  • Provider, bridge, target-format, or platform compatibility: none.
  • Evidence, artifact, provenance, or reconstruction contract: none; the same headlines and sidecars are selected.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: src/domain/analysisView/webNetworkCaptureView.test.ts. The new tests are "indexes an unrelated / a path-descendant million-character facet pointer by its headline prefix only". They check that a 1,000,000-character pointer is still classified correctly: unrelated, or beneath /request/path. They also count the strings longer than 1,000 characters that the preview call adds to any Set and expect none. Only the pointer and strings cut from it are that long, so the count stands in for the index size without timing or heap checks. "keeps headline omissions and controls beside many unrelated facets" places 20,000 unrelated sidecars beside a redacted host and a path with a facet beneath it: both stay withheld, while the scheme, port and status remain with only their own sidecars. It passes against both the current main projection and this change, so it pins the omission/uncertainty behavior rather than the bound.
  • Base reproduction or other evidence: executed.
    • With main's webNetworkCaptureView.ts (47cedcc) the two million-character tests fail on Node 24.22.1: the old index adds 999,000 and 499,507 such strings (expected 999000 to be +0, expected 499507 to be +0). With this change it adds none.
    • Before 67c3550 these tests checked only classification. On Node 22 and 24 they passed on the base file, and only Node 26 caught the old index, by running out of heap (about 4 GB), as @CassieFox23 reproduced twice in this comment. The set-addition count follows the nit in the latest review from @CassieFox23.
    • Same comment, one run each on a shared box: 200,000 sidecars plus a 1,000,000-character pointer took 450 ms (Node 22) / 296 ms (Node 24) / heap out of memory (Node 26) on the base file, and 86 / 62 / 91 ms with this change. 6,000 random records produced identical previews on the base file and this change.
    • My own local benchmark (not committed) on Node 24: about 510 ms on main vs about 295 ms with this change, including fixture building and schema parsing.
  • User-visible CLI/MCP output (if applicable): unchanged.
  • Remaining proof gaps: no committed test asserts the index size directly, because the index is internal; the tests count long-string set additions over the whole preview call instead. I ran that count on Node 24.22.1 only.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npx vitest run src/domain/analysisView/ src/browser/history/ on this branch merged with main 47cedcc (Node 24.22.1) — 76 files, 1,733 tests passed.
  • npx vitest run src/domain/analysisView/webNetworkCaptureView.test.ts — 55 passed. With webNetworkCaptureView.ts replaced by main's, the two million-character tests fail as shown above, and the boundary-control test passes.
  • npx tsc --noEmit — 0 errors.
  • npx oxlint on the changed files — 0 errors. The three warnings on webNetworkCaptureView.ts (max-lines, max-params, max-lines-per-function) are also present on main's file.
  • npx oxfmt --check on the changed files — clean.
  • npm run verify:module-boundaries — passed.
  • npm run check:fast (typecheck, lint) — passed.
  • node scripts/check-doc-facts.mjs after npm run compile — passed.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: reduces transient allocation on the synchronous inspect_analysis_view path for admitted mitmproxy Evidence.

Author readiness

Human owner: @Dragolone

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

…previews

Capture previews selected HAR/SAZ headline names (url, status) only, so a
native mitmproxy record lost its request target (scheme, host, port,
authority, path) and its status_code while keeping the reason phrase, and
the matching numeric sidecars were filtered out. Select those mitmproxy
headline fields too; headers and payloads stay omitted.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-04T13:21:52.475535Z
We couldn’t complete this review. Comment @codex review to try again.
67c3550 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issue was found.

Summary

Bounds the per-record facet index used for mitmproxy headlines without changing which fields qualify.

  • Mitmproxy previews keep facet indexing bounded.

No actionable issues found. The earlier, unnumbered OPTIONS * finding is fixed: headlineShape accepts "*", and a regression test covers it.

Review used the full PR changes because the previous review and current head have different merge bases with the supplied target commit. Runtime checks were not run. Live supersession status could not be checked because GitHub authentication was unavailable.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Record facets] --> B{Headline coordinate or ancestor?}
  B -->|Yes| C[Keep at most two per group]
  B -->|No| D[Do not retain facet]
  A --> E[Check seven fixed descendant prefixes]
  C --> F[Existing headline checks]
  E --> F
  F --> G[Same preview choices]
Loading

Reviews (34) · Last reviewed commit: "test(web-capture): count long set additi..." · Reviewed by Greptile

dakjdakd and others added 5 commits October 14, 2026 01:00
* fix(comparison): preserve incomplete function call evidence

Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty.

Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls.

* fix(comparison): recheck historical calls-only changes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfb36b5655

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Select the mitmproxy target and status fields only for mitmproxy
records, only when they are scalars, and the path only in origin form.
An absolute, network-path or otherwise ambiguous path spelling can carry
userinfo that the native bridge's redaction does not recognise, and a
nested value could copy an arbitrary record subtree into the preview;
both stay in complete record pages. Other formats keep their existing
headline fields.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0ab028759

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

morluto and others added 2 commits October 14, 2026 02:33
…rom previews

Origin-form checks alone accepted slash-backslash and slash-control
spellings such as /\user:password@host/a, which WHATWG resolves as an
authority with credentials and the recorded transport-URL interpretation
already treats as ambiguous credentials. Preview a mitmproxy path only
when parseRecordedTransportUrl finds no credentials, and omit mitmproxy
headline strings longer than 8 KiB so one record cannot exceed the reply
budget; both stay in complete record pages.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6fa5bbe84

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…f previews

Native redaction covers only request path and authority, so a malformed
state such as host "user:password@example.test" passed through the
preview. Omit every non-path mitmproxy headline string that contains a
userinfo marker (@); valid schemes, hosts and ports never do, and the
value stays in complete record pages.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81b10c30f9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@CassieFox23

CassieFox23 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: The fix works on a real mitmproxy capture. Without it, the preview drops the request path and the status code. With it, both show, and headers and bodies stay out.

Edit: I retested a newer head, 77b4933. The new results are here. All six items in the Not run list below are still not run, so none is struck through. The new comment also says why the real mitmdump capture could not be rerun.

I tested 81b10c3 against base 2895a18 (#2943). The head moved twice while I tested, so I also ran a8df0da and e6fa5bb (see the first fold).

Results

Check Node 22.23.1 Node 24.18.0 Node 26.11.1
PR test file on 81b10c3 16 pass 16 pass 16 pass
PR test file, source file from 2895a18 6 fail, 10 pass 6 fail, 10 pass 6 fail, 10 pass

Old source means the file from 2895a18. The six failures are the headline test, the OPTIONS test and the four new userinfo tests. The seven "keeps ... out" tests pass on the old code too, because it adds no fields. The three older tests pass as well.

Real capture: mitmdump 12.2.3 (MIT) as a reverse proxy on loopback, six requests, status 200, 404 and 501, Node 24.18.0.

Check With the PR Old source
REA decodes the file 6 records 6 records, identical to the PR column
Preview shows the request target scheme, host, port, path (query string kept) none of them
Preview shows the status code 200, 404, 501 no, only the reason text
Port and status as numeric literals yes no
Header or body text in the preview none none

Two credential cases came up in review: an odd path and a host value. Both are withheld on 81b10c3. I checked them with a real request and with a flow I built with the mitmproxy library. The credentials fold has the details.

Not run

  • macOS and Windows.
  • HTTPS, HTTP/2 and WebSocket captures.
  • The MCP tool. I used the CLI form of inspect_analysis_view.
  • A real client that sends a host with credentials.
  • Node 22 and 26 for the real capture.
  • The full test suite, type check and lint.

Thanks, Dragolone, for the fix and for the quick follow-ups on the review notes.

Results on two earlier heads
  • Head a8df0da, PR test file: 9 pass on Node 22, 24 and 26. With the old source: 2 fail, 7 pass on all three.
  • Head e6fa5bb, PR test file: 12 pass on Node 22, 24 and 26. With the old source: 2 fail, 10 pass on all three.
  • Real capture on both: the preview showed scheme, host, port, path and status code. The old source showed none of them.
  • Two related boundary files on Node 24 (23 tests): all pass on a8df0da, e6fa5bb and 81b10c3. The PR text reports 21 pass and 2 skipped on macOS. The file has two Linux-only tests that only check the mitmdump path rule. They run no mitmproxy.
  • The source file is identical on d0ab028 and a8df0da, so those two heads give the same results.
More checks on credentials and the fixture script
  • On a8df0da, one odd path showed its user:password text in the preview. It began with a slash and a backslash. I sent it as a real request. The bridge did not redact it. e6fa5bb and 81b10c3 withhold it.
  • On 81b10c3, a slash-tab path and a path of 8,193 characters are withheld. A path of 8,001 characters still shows.
  • A host value that holds user:password before the name showed in the preview on e6fa5bb. 81b10c3 omits it. I built that flow with the mitmproxy library. I did not get it from network traffic.
  • The complete view of the same file does hold the header and body text, so the preview leaves it out on purpose.
  • The repo's fixture script runs with mitmdump 12.2.3. Its records have the same keys as my capture at record, request and response level.
  • Its path is in absolute form, so the preview withholds the path there and shows the rest.
  • A regular-mode proxy capture (extra) stored the path in origin form. The preview showed it.
Commands
node scripts/run-development-tests.mjs focused src/domain/analysisView/webNetworkCaptureView.test.ts
REA_MITMDUMP_COMMAND=/tmp/pr2876/venv/bin/mitmdump node scripts/rea.mjs inspect-web-network-capture /tmp/pr2876/flows.mitm mitmproxy --json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "capture", detail: "preview", offset: 0, limit: 6}}' capture.json > view.json
node scripts/rea.mjs inspect-analysis-view view.json --json
  • Each test run used one Node binary, called by its full path.
  • The old-source runs swap that one file into the head tree, then rebuild with npm run build:cached.
  • mitmdump ran with --mode reverse:URL, --listen-host 127.0.0.1 and -w for the flows file. I stopped it by its process id so the file flushed.
  • The setup used a throwaway Python virtual environment under /tmp. I deleted it afterwards.
Test setup

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

…formed shapes

A per-string length cap still let a malformed capture fill every target
slot with escape-heavy text, so one default preview page could expand to
about 18 MiB of JSON. Preview each mitmproxy field only in its own shape:
a short scheme token, host/authority without controls, whitespace or
userinfo markers, integer port and status code, and a path of at most
2 KiB without control characters. That bounds every record's headline to
about 4 KiB serialized, even for an escape-heavy path.
@Dragolone

Copy link
Copy Markdown
Contributor Author

Thanks @CassieFox23 — the real mitmdump capture across Node 22/24/26 is great confirmation. On the 8,001-character path you noted: 5149edf now admits each mitmproxy field only in its well-formed shape, including a path of at most 2 KiB without control characters, so that case is withheld from the preview as well (it stays on complete pages).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5149edfac7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…integer type

Select mitmproxy target fields only from the request and status_code
only from the response, so an unknown scalar extension such as
response.path or request.status_code is not promoted to a headline.
Admit port and status_code only when the record's numeric sidecar
reports the producer's integer type; a native float such as 443.0 is
otherwise indistinguishable from 443 after JSON parsing.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f069727642

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

… numeric headlines

An inline parent could pair request.port 443 with an integer sidecar
whose literal is unbounded or does not represent the value, or with
duplicate sidecars, and the selected-pointer filter would copy them into
the preview. Admit port and status_code only with exactly one integer
sidecar whose literal is the value's canonical decimal, which also
bounds the copied literal to a few characters.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8bc0b745ae

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

The mitmproxy container format does not make every retained flow HTTP; the bridge keeps non-HTTP and unknown flow states with their extensions. Add the mitmproxy target and status candidates only when the record's own type is "http".
@greptile-apps

greptile-apps Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

# Conflicts:
#	src/domain/analysisView/webNetworkCaptureView.test.ts
#	src/domain/analysisView/webNetworkCaptureView.ts
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@Dragolone Dragolone changed the title fix(web-capture): keep mitmproxy request targets and status codes in previews fix(web-capture): bound the mitmproxy headline facet index Oct 14, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@CassieFox23

CassieFox23 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: On head 77b4933, the PR's test file passes on Node 22, 24 and 26 (52 tests). The 8,001-character path I flagged is now withheld, and so are values just past the field limits. Main already holds those limits. This PR adds a smaller facet index. Without it, the test file runs out of memory on Node 26.

Edit: I tested the open review threads on this same head. Results are here.

I retested head 77b4933. The base was 7375ab6. My last results were on 81b10c3 (see my earlier comment). The PR is now small: source +49/-35 and test +26 on top of main.

Results

Check 81b10c3 (Node 22 / 24 / 26) 77b4933 (Node 22 / 24 / 26)
PR test file 16 pass on all three 52 pass on all three
Same file, base source 6 fail, 10 pass on all three (base 2895a18) 52 pass, 52 pass, crash on Node 26 (base 7375ab6)
Two boundary files (Node 24 only) 23 pass 26 pass, none skipped
Real mitmdump capture 6 records, Node 24 not run: mitmdump is not on this box

The crash is on Node 26. With the base source, the two "million-character facet pointer" tests fail. They fail with "JavaScript heap out of memory" at about 4 GB. I ran it twice, then again on those two tests alone. Node 22 and 24 pass them on the base source. I do not know why Node 26 differs.

Field limits, shown with a test I wrote. The test builds one mitmproxy record and checks what reaches the preview. I ran it on Node 24. Head is the same on Node 22 and 26:

Input 81b10c3 base 7375ab6 77b4933
path of 8,001 characters shown withheld withheld
path of 2,048 / 2,049 characters shown / shown shown / withheld shown / withheld
scheme of 32 / 33 characters shown / shown shown / withheld shown / withheld
host of 255 / 256 characters shown / shown shown / withheld shown / withheld
authority of 262 / 263 characters shown / shown shown / withheld shown / withheld
tab in the path, scheme, host or authority shown withheld withheld

So the 8,001-character path is gone on both base and head. The 2 KiB path limit and the other shape rules already sit on main. This PR changes none of these rows compared with base.

Not run

  • macOS and Windows.
  • HTTPS, HTTP/2 and WebSocket captures.
  • The MCP tool. I used the test runner and the CLI form only.
  • A real client that sends a host with credentials.
  • A real mitmdump capture on any Node. mitmdump is not installed here and I did not install it. My saved capture files could not be decoded without it.
  • The full test suite, type check and lint.
  • The old base 2895a18 against the new test file. Main now holds the feature, so I used 7375ab6.
  • Main's own source file. It needs a file the head tree lacks.

Thanks, Dragolone, for the quick fixes and for the plain answer on the 8,001-character path.

Commands
npm ci && npm run prepare && npm run build:cached
node scripts/run-development-tests.mjs focused src/domain/analysisView/webNetworkCaptureView.test.ts
node scripts/run-development-tests.mjs focused src/domain/analysisView/edgeProbe.test.ts
node scripts/run-development-tests.mjs focused tests/boundary/browser/historicalWebCapture.test.ts tests/boundary/browser/historicalCaptureDecoder.test.ts
git show 7375ab6e3:src/domain/analysisView/webNetworkCaptureView.ts > src/domain/analysisView/webNetworkCaptureView.ts
node node_modules/vitest/vitest.mjs run src/domain/analysisView/webNetworkCaptureView.test.ts -t "million" --pool=forks
node scripts/rea.mjs inspect-web-network-capture flows-reverse-real.mitm mitmproxy --json
  • Each test run used one Node binary, with its folder first on PATH.
  • edgeProbe.test.ts is a throwaway test of mine. It copies the PR test's own record helper, feeds it the inputs in the table, and prints which fields reach the preview.
  • For the base and 81b10c3 rows, I copied that one source file into the head tree from git, ran the probe, then put the head file back. Vitest reads the TypeScript directly, so I did not rebuild.
  • The decode of my saved capture failed with exit 1: REA needs an absolute REA_MITMDUMP_COMMAND path to mitmdump 12.2.3.
  • GitHub reported this PR as mergeable but blocked when I checked. I did not look into what blocks it.
Test setup
  • Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
  • CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
  • RAM: 32 GB DDR4-3600
  • Node: 22.23.1, 24.18.0, 26.11.1
  • Head / base: 77b4933 / 7375ab6
  • Other: mitmdump not installed; other test jobs shared the box

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@CassieFox23

CassieFox23 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Same head as my last comment, 77b4933. 29 of the 30 open Codex threads describe a problem that does not reproduce on it. The 30th asks to move the work into the bridge. I did not test that. The PR's own change holds. The old code crashes Node 26 on a large probe, and the new code does not.

Edit (2 Nov): I retested this PR on its new head, a049415. Results are here.

I tested head 77b4933 again, this time against every open review thread. All 30 open threads are from Codex and all are marked outdated. Outdated only means the line moved, so I fed each thread's input to the current code. The base was 7375ab6. Main has moved to 92dadf9. Its copy of this file differs from the base only in imports and one message, so main still has the old index function.

Verdicts

Every row's input was a fixture I wrote. "Not real" means the bad input is kept out of the preview. A control record in the same run does get its headline fields, so the checks are not empty.

Claim Threads Verdict What I saw
Credentials in the path ambiguous credentials, slash-backslash, unparseable port Not real Six spellings, including https:user:password@example.test/path and /\user:password@example.test/path: path withheld, no "password" in the output
Credentials in host or authority credential-bearing hosts Not real user:password@example.test as host, authority or scheme: withheld
Size bounds scalar fields, serialized page, nested object under path Not real A 6 MiB path, host, authority, scheme or nested object: withheld. 64 records with the longest valid values: 349,367 JSON characters in total
Integer slots and sidecars on target fields floats, sidecar literal, sidecar on a string, non-number port Not real Float port: withheld. Mismatched or 6 MiB literal: withheld and not copied. {toString: null} port: withheld, no throw
Message side and flow type message side, HTTP gate Not real response.path and request.status_code: withheld. Flow types tcp, udp, dns, websocket, null, missing, and a HAR record: no headlines
Redactions exact, ancestor, byte descriptor, flow type Not real Redaction at the field, /request, the root, a redacted descriptor, or /type: the covered fields are withheld
Facets on the flow type, ancestors, children sidecar on type, sidecar on ancestor, bytes as type, bytes on ancestor, child facet Not real Facet at /type, root, /request, /type/child, or bytes at /type, /request: no headlines
Byte descriptors and surrogates provenance, descriptor facts, non-scalar strings, descriptor-free strings Not real Wrong representation, bytes, sha256 or content: path withheld. Unpaired surrogate in path, host, authority or scheme: withheld
Index facets once index once Not real The old file already builds one index per record (indexFacets is in the base). I read the head code: one call per record. I did not time the pass count
Facet index cost pointer expansion, relevant facets only Not real at head. Fixed by this PR On the old file, a 1,000,000-character pointer and 200,000 sidecars crash Node 26 and run 3 to 5 times slower on Node 22 and 24. See the second table
Move the work into the bridge producer boundary Not run A design call. The PR description says it is not included

The nested-object thread in the size row also asks for the redesign in the last row. I tested only its example.

The same 22 checks also pass on the old file (7375ab6, Node 24). So the first eight rows were already fine before this PR. Only the facet-index cost row is about what this PR changes.

Results by Node

Check Node 22 Node 24 Node 26
PR test file, 52 tests, head pass pass pass
My 22 checks, head pass pass pass
6,000 random records, old file against head 0 differences 0 differences 0 differences
200,000 sidecars plus a 1,000,000-character pointer, old file 450 ms 296 ms heap out of memory, twice
Same input, head 86 ms 62 ms 91 ms

The timings are one run each on a shared box, so treat them as rough. I did not measure memory in a way I trust, and the memory use of the whole test run was similar on Node 22 and 24.

Not run

  • The MCP tool and its response budget. I ran the preview function only.
  • The mitmproxy bridge. Codex's claims about what it emits are taken from the threads, not checked.
  • A real mitmdump capture. It is not installed here.
  • qlog records. I used a HAR record as the non-mitmproxy case.
  • Normalization in the bridge. I did not test moving it there.
  • The full test suite, type check and lint.
  • macOS and Windows.

Thanks, Dragolone, for the work on the facet index. In 6,000 random records, the preview matched the old file's exactly.

Test setup
  • Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
  • CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
  • RAM: 32 GB DDR4-3600
  • Node: 22.23.1, 24.18.0, 26.11.1
  • Head / base: 77b4933 / 7375ab6 (main was 92dadf9 at start and end)
  • Other: other test jobs shared the box, so timings are rough
Commands
npm ci --ignore-scripts; npm run prepare; npm run build:cached; npm run test:prepare   (Node 24, tree at 77b493358)
npm run test:focused -- src/domain/analysisView/webNetworkCaptureView.test.ts   (once each on Node 22, 24, 26)
npm run test:focused -- src/domain/analysisView/probe2876.test.ts   (my probe file; once each on Node 22, 24, 26)
PROBE_ALL=base npm run test:focused -- src/domain/analysisView/probe2876.test.ts   (Node 24; behaviour checks on the old file)
PROBE_IMPL=base npm run test:focused -- src/domain/analysisView/probe2876.test.ts   (Node 22, 24, 26; old file in the scale test)
PROBE_IMPL=base node node_modules/vitest/vitest.mjs run src/domain/analysisView/probe2876.test.ts -t scale --pool=forks   (Node 26, twice, then once with head)
  • probe2876.test.ts builds mitmproxy records with my own fixtures and calls the view function. For the old file I copied 7375ab6's webNetworkCaptureView.ts next to the head file and imported both.
  • The 6,000-record check uses a fixed seed and compares the whole preview of old file and head as JSON.
  • The test file does not mention doctor; I checked with grep first.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@Dragolone

Copy link
Copy Markdown
Contributor Author

Thank you @CassieFox23, this is very useful. The Node 26 heap exhaustion on the base file is a stronger signal than my Node 24 benchmark, which only showed the slowdown. I've updated the PR description's evidence section to cite your results, including the 6,000-record equivalence check.

Following your thread-by-thread retest, I'm resolving the Codex threads you found do not reproduce on 77b4933. I'm leaving the two threads that ask to move headline normalization into the mitmproxy bridge open, since that is a design decision for the maintainers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaoru0822-kitauji

Copy link
Copy Markdown
Collaborator

Merged latest main into this branch (60ecf21) for a fresh CI run. No conflicts, no force-push.

Why

  • The last test run (Oct 19) failed only in tests/boundary/process/processCli.test.ts (JSON depth via the comparison CLI) and tests/boundary/evm/evmoleInterfaceProvider.test.ts (process-cleanup retention). This PR doesn't touch either; its diff is webNetworkCaptureView.ts and its test.
  • Validate author declaration still needs the Human owner line and the readiness boxes in the PR body, which only you can fill in.
  • The open review question about moving the work into the bridge is a design decision for you and the maintainers; I didn't touch it.

Validation (local macOS, merged head)

  • tsc --noEmit: clean
  • npm run test:focused on webNetworkCaptureView and analysisViewMcp: 74 passed
  • check:fast, format:check: pass

@CassieFox23

Copy link
Copy Markdown
Contributor

Thanks, Dragolone. I checked the thread list on GitHub, and the resolved set matches my retest. Of the 30 threads I tested, 28 are resolved and 2 are open: the producer boundary thread and the nested object thread. The nested object example did not reproduce, but that thread also asks for the bridge change, so it stays open. Both are for the maintainers.

🤖 Written with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@morluto

morluto commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Thanks for isolating the facet-index follow-up. The headline projection has landed, but current main still expands ancestors and indexes record facets here. This remains a distinct resource-bound proposal; please retain the existing omission/uncertainty behavior and representative boundary controls while updating it.

…related facets

The bounded facet index must not change what a mitmproxy preview withholds or keeps. Beside 20,000 unrelated sidecars, a redacted host and a path with a facet beneath it stay withheld while the scheme, port and status remain, with only their own sidecars retained. The test passes against both the current main projection and this change.
@Dragolone

Copy link
Copy Markdown
Contributor Author

Thanks @morluto. I've updated the branch to current main (6a226bd) in a049415. The merge was clean: main's recent changes to this file (parseAnalysisViewResult, the subject check and the z.int() schemas) are untouched, and the diff still only changes how indexFacets builds its index. It keeps facets only at the headline coordinates and their ancestors (at most two per pointer), and other facets only mark the coordinates they lie beneath.

Omission and uncertainty behavior is retained:

  • All existing omission, redaction, sidecar and byte-descriptor tests in webNetworkCaptureView.test.ts are unchanged and pass (54 tests).
  • New boundary control, "keeps headline omissions and controls beside many unrelated facets": 20,000 unrelated sidecars sit beside a redacted host and a path with a facet beneath it. Both stay withheld; scheme, port and status remain with only their own sidecars. It passes against both main's current projection and this change, so it pins behavior rather than the bound.
  • @CassieFox23's earlier run found identical previews for 6,000 random records between the two implementations.

Validation on Node 24.22.1: src/domain/analysisView/ + src/browser/history/ — 73 files, 1,668 tests passed. tsc, lint, format and module boundaries are clean.

@CassieFox23 CassieFox23 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR: Head a049415 passes all 54 tests in the PR's test file on Node 22, 24 and 26. The source change is the same one I tested on 77b4933, and the one new commit only adds a test. The bound works. On Node 26 the old index runs out of memory on the PR's two tests with huge pointers. The new code takes a few ms.

Edit (5 Nov): Dragolone took the nit in 67c3550. The two big pointer tests now count the long strings the preview adds to sets, and expect none. That should catch the old index on every Node, not only by a crash on Node 26. I read the change and the index code it checks. I didn't run it.

I retested head a049415. The base was 6a226bd. Main is now 2d11fd9. It still has the old index (addAncestors and groupByPointer), as morluto said on 26 Oct. The PR's change applies to main's file with only line offsets. Below, "old file" means the base's copy of webNetworkCaptureView.ts. Its index code is identical to main's.

Results

Check Node 22.23.4 Node 24.22.1 Node 26.12.1
PR test file at head, 54 tests pass pass pass
My 22 earlier checks at head pass pass pass
6,000 random records, old file against head no differences no differences no differences
PR's two million-character pointer tests, on the old file pass, 330 and 181 ms pass, 291 and 145 ms out of memory, each test alone
Same two tests at head 6 and 5 ms 5 and 4 ms 20 and 5 ms
200,000 sidecars plus a 1,000,000-character pointer, old file 507 ms 393 ms out of memory
Same input at head 84 ms 69 ms 96 ms

The timings are single runs in a VM, so treat them as rough. The test file had 52 tests last time. It has 54 now: one came from main, and one is the new control.

Findings

Should fix: nothing.

Nit

  • The two million-character tests have no time or size limit. On Node 22 and 24 they pass on the old index too, so only Node 26 catches a regression, and it does so by crashing. A timing or heap check would catch it on every Node. Timing checks can be flaky, so this is optional.

Looks good

  • The source change is the same as at 77b4933. All four heads I compared have the same patch id (d90ccc99e8), each against its own base. The new commit a049415 adds 50 test lines and nothing else.
  • The bound holds for long and deep pointers. In my own timing test, 2,000 facets with 500-level pointers took 16 to 25 ms at head and 854 to 1,056 ms on the old file (Node 22, 24 and 26). For 200,000 facets with short pointers, head took 715 to 771 ms and the old file 804 to 844 ms. That is about the same. I timed the whole preview call, so I can't say how much of it is the index.
  • The new control test does its job. It passes on the old index too, as the PR says, so it pins what is withheld and kept, not the bound. In a scratch copy with container marking switched off, 5 tests fail, the control among them. With redactions left out of the index, 6 fail, the control among them.
  • The two threads that were open on 21 Oct are still the only ones open: the producer boundary and the nested object. Both are design asks for the maintainers. I did not retest them.

Not run

  • The MCP tool and its response budget. I ran the preview function only.
  • The mitmproxy bridge. I did not read or run it.
  • A real mitmdump capture. It is not installed.
  • qlog records.
  • Moving the work into the bridge.
  • Main's current tree. Its file needs two files this PR's tree lacks. So I used the base's file, which has the same index code. I also checked that the PR applies to main's file.
  • test:fast, test:built, the full suite, type check and lint. CI already passed them on this head.
  • A clean memory figure that leaves out the test fixture.
  • macOS and Windows.
Test setup
  • Box: Fedora 44 VM (4 vCPUs, 8 GB) on my Nobara 44 machine, guest kernel 7.2.11
  • Node: 22.23.4, 24.22.1, 26.12.1
  • Head / base: a049415 / 6a226bd (main was 9bea563 at the start and 2d11fd9 at the end; neither touches the PR's two files)
  • Other: rea-agents 6.8.0, g++ 16.2.1. The timings are single runs, and I did not check whether other jobs shared the VM.
Commands
npm --prefix <tree> ci --offline                 (Node 24, tree at a04941520)
npm --prefix <tree> run test:prepare             (Node 24)
npm --prefix <tree> run test:focused -- src/domain/analysisView/webNetworkCaptureView.test.ts   (Node 22, 24, 26)
node node_modules/vitest/vitest.mjs run src/domain/analysisView/webNetworkCaptureView.test.ts --reporter=verbose   (old file in a copy: Node 22, 24, 26; head: Node 22, 24)
node node_modules/vitest/vitest.mjs run <same file> -t million-character --pool=forks   (old file Node 26; head Node 26)
node node_modules/vitest/vitest.mjs run src/domain/analysisView/probe2876.test.ts --pool=forks   (my earlier probe file, head: Node 22, 24, 26)
PROBE_IMPL=base ... probe2876.test.ts -t scale --pool=forks   (old file: Node 22, 24, 26)
  • The old file is the base's webNetworkCaptureView.ts (6a226bd) copied into a hard-linked copy of the tree, next to the PR's test file.
  • The timing test and the two broken copies are my own files in the guest tree. They are not PR code. The timing test is the PR's test file plus two tests that call its nativePreview helper with many facets.
  • The test files do not mention doctor; I checked with grep first.

🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

…pointer tests

The two million-character pointer tests had no time or size bound, so on
Node 22 and 24 they passed against the old ancestor index too, and only
Node 26 caught a regression by running out of memory. Count the strings
longer than 1,000 characters the preview adds to sets instead: only the
pointer and strings cut from it are that long, the old index added
999,000 and 499,507 of them, and the new index adds none.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@Dragolone

Copy link
Copy Markdown
Contributor Author

Thanks @CassieFox23 for the retest. I took the nit in 67c3550.

The two million-character tests now count the strings longer than 1,000 characters that the preview call adds to any Set, and expect none. Only the pointer and strings cut from it are that long. It counts calls, like the existing Set.prototype.add spy in javascriptSemanticValues.memo.test.ts, so there is no timing to flake.

Source Unrelated pointer Path-descendant pointer
main's webNetworkCaptureView.ts fails: expected 999000 to be +0 fails: expected 499507 to be +0
this change 0 0

So a regression now fails on every Node version instead of only crashing on Node 26. I ran this on Node 24.22.1 only.

b4b9962 merges current main (47cedcc) cleanly. That brings in #7832 and #8061, which both touched this file. Main still has addAncestors and groupByPointer. The source diff against main keeps patch id d90ccc99e8.

On the merged tree:

  • src/domain/analysisView/ and src/browser/history/: 76 files, 1,733 tests passed.
  • check:fast and the doc facts both passed.

@morluto

morluto commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Hey @Dragolone, thanks for this! The fix looks good. What's left: there are 2 unresolved review threads from Codex, all on outdated lines, so if they're already addressed, resolving them should be all that's needed. Happy to get this in once that's sorted out.

@Dragolone

Dragolone commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @morluto, both Codex threads are resolved now.

  • The first was fixed in 7eab486, and the fix is still in place at the current head. mitmproxy target fields are previewed only for mitmproxy records, and only scalar values are copied. The "a nested path value" and "a path in a non-mitmproxy record" tests cover both cases.
  • The second asked whether to move normalization to the producer. I've taken your comment to mean this PR keeps the consumer-side version. If you want the producer-side design, it can go in a separate issue.

The branch still merges into current main without conflicts, and CI is green.

@morluto
morluto merged commit e69f27f into morluto:main Nov 6, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: runtime Process capture and runtime observation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants