Skip to content

fix(capture): end special-scheme URL authorities at a backslash - #2767

Merged
kaoru0822-kitauji merged 25 commits into
mainfrom
fix/issue-2671
Oct 14, 2026
Merged

kaoru0822-kitauji merged 25 commits into
mainfrom
fix/issue-2671

Conversation

@kaoru0822-kitauji

Copy link
Copy Markdown
Collaborator

Fixes #2671.

Problem

The native mitmproxy adapter (bridge/mitmproxy/capture.py) removes transport userinfo from Location/Referer/Origin and native request.path/authority with this regex:

^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@

The authority therefore ran to the last @ before /, ? or #. For Location: https://example.test\part@tail, the adapter treated example.test\part as userinfo:

  • the reported Location became https://tail, a different destination;
  • the original bytes were withdrawn as a transport-credential redaction (state: "redacted", no content, size or digest).

The shared TypeScript owner, src/domain/recordedTransportUrl.ts, follows WHATWG URL parsing, as HAR captures do. There, a backslash ends the authority of a special scheme (http, https, ws, wss, ftp, file) or a scheme-relative // reference. That @ is ordinary path text, and the HAR projector preserves the value.

Fix

without_userinfo locates the authority with the same boundary rule as recordedTransportUrl.ts:

  • It ends at /, ? or #, and also at \ for special or scheme-relative references.
  • It removes userinfo only up to the last @ inside that authority.
  • Byte values are processed as Latin-1, so every other byte stays identical.

Userinfo before a backslash, as in https://user:pw@example.test\path, is still removed and redacted, and non-special schemes keep their previous behavior. The /authority path handling is unchanged.

Validation

  • New bridge/mitmproxy/test_capture.py, added to npm run test:python, which CI's static job runs. It stubs the mitmproxy imports and calls the real project_record / without_userinfo:
    • https://example.test\part@tail is reported unchanged, with its byte sidecar retained and no redaction. This fails on main, which returns https://tail.
    • https://user:pw@example.test\path becomes https://example.test\path, with the transport-credential redaction.
    • https://user:pw@example.test/a@b becomes https://example.test/a@b.
    • The non-special and scheme-relative boundary cases are covered too.
  • npm run test:python passes (13 + 11 tests), and so does tests/boundary/filesystem/ciTestReports.test.ts, which checks the test:python wiring. npm run check:fast, metadata:check and format:check pass.
  • Not run: the real mitmdump 12.2.3 native workflow, which needs Linux and REA_MITMDUMP_COMMAND.

🤖 Generated with Claude Code

The native mitmproxy adapter removed transport userinfo with
`^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?"
or "#". For `https://example.test\part@tail` it treated
`example.test\part` as userinfo, reported `https://tail` as the
Location, and withdrew the original bytes as a transport credential.
WHATWG URL parsing, used by the shared TypeScript owner
(recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or
scheme-relative authority at a backslash, so that "@" is ordinary path
text.

Locate the authority with the same boundary rule before removing
userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay
identical. Userinfo before the backslash is still removed and redacted.
Add source unit tests for the adapter projection to test:python.

Fixes #2671

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T00:33:40.386970Z 110151e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 553cd6e358

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 3/5

[Medium impact] Fixes URL authority parsing for special schemes.

The PR should not merge while credential-bearing malformed URLs remain visible in captured output.

Findings

  1. P1 Security Malformed URLs expose passwords ▶
Fix with agent prompt
### Issue 1
bridge/mitmproxy/capture.py:191-192
`without_userinfo` now keeps passwords when a special-scheme host or port is rejected. For example, a captured `Location: https://user:pw@example.test:99999/x` keeps `user:pw` in both the reported text and the original bytes, with no redaction. A bad port does not make the supplied password safe to expose. The shared `parseRecordedTransportUrl` still removes this raw userinfo span.

Keep credential removal separate from host acceptance. Apply the same protection to the new `file` early return.

**How this was verified:** Projecting this rejected-port header kept its password in the reported text and original bytes with no redactions.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The current changes add host and port checks to the native capture adapter and expand its tests.

  • Special URLs only lose credentials when their authority parses.

Reviews (16) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

Comment thread bridge/mitmproxy/test_capture.py Resolved
Ending the authority at a backslash exposed a credential: for
`https://\user:pass@host/path`, WHATWG parsing skips every leading "/"
or "\" after a special scheme (and in scheme-relative references), so
`user:pass` is userinfo, but the new boundary stopped at the first
backslash and found no "@". The same applied to `https:\\user:pass@host`
and `https:user:pass@host`.

Start the authority after that whole slash run for non-file special
schemes and scheme-relative references. File URLs keep their "//"
boundary because they carry no userinfo. Load the adapter in tests with
fresh mitmproxy stand-ins scoped to its import instead of leaving them
in sys.modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread bridge/mitmproxy/capture.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d972a7e85

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72e931f3a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8361b7715

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved
Comment thread bridge/mitmproxy/capture.py Resolved
@morluto

morluto commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open.

  1. thread 1

The first boundary stopped before a special-scheme authority had consumed its leading slash/backslash run. The existing fix now consumes that run before locating userinfo, and the retained regression source covers the leading-backslash spelling and byte-sidecar redaction. The later redirect fix also uses the owning recorded request's scheme so relative Location paths retain their text.

Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection.

  1. thread 1

The authority parser lacked the recorded request's URL context. Projection now derives the scheme once at each flow/backup boundary and supplies it specifically to Location handling. Same-scheme references without two slash-equivalent separators keep their relative path text and original bytes; absolute and cross-scheme authorities still remove userinfo. Regression source includes ordinary, slash/backslash-relative, absolute, and backup-flow cases.

Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection.

  1. thread 1

Authority lookup had interpreted raw controls that WHATWG removes first. The adapter now derives a tab/LF/CR-normalized parse view, locates authority/userinfo there, and maps only the credential span back to the original spelling in a linear scan without an index array. Original noncredential text/bytes remain unchanged; any removed userinfo also redacts the raw byte sidecar. Regression source covers controls inside the scheme, leading separators, and credentials.

Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection.

  1. thread 1

The no-scheme prefix recognized only literal //. It now recognizes every pair of slash-equivalent separators for network references, then consumes their leading run before locating the authority. Explicit non-special schemes still require their literal // syntax. The regression source includes backslash/backslash and both mixed pairs, preserving the original prefix while removing userinfo and redacting its byte sidecar.

Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0aff997b31

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved
@morluto

morluto commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open.

  1. thread 1

The authority parser treated every scheme-relative header as special, independently of the recorded request scheme. It now derives the separator/terminator rule from the parsed base scheme: a non-special base recognizes // and keeps backslashes within authority data, so userinfo and its byte sidecar are redacted together. The producer regression includes foo plus //user\name:pass@host/x and a non-special mixed-separator path. The existing source-spelling/byte projection stays owned by the same redactor; no downstream repair or related cleanup remains.

Source validation: inspected the producing boundary, affected consumers, authored regressions and final diff. Tests, checks, builds, lint, typechecks, provider workloads and CI reruns were not run at the user's explicit request; no runtime/provider verification is claimed.

Source fix: 0aff997.

  1. thread 1

The projection applied header URL-reference semantics to native request.path, conflating origin-form path evidence with network authorities. The producer now distinguishes header URL references, native authority, and native paths: origin-form separator runs are retained exactly, while explicitly schemed absolute-form URLs still remove userinfo. Authored regressions cover /\invoice@customer, //invoice@customer, backslash-prefixed paths, retained byte sidecars, and absolute-form credential redaction. No related cleanup remains.

Source validation: inspected the producing boundary, affected consumers, authored regressions and final diff. Tests, checks, builds, lint, typechecks, provider workloads and CI reruns were not run at the user's explicit request; no runtime/provider verification is claimed.

Source fix: 0aff997.

@morluto morluto added bug Something isn't working area: runtime Process capture and runtime observation area: bridge Hopper bridge (Python) related changes P2: medium Medium priority issue labels Oct 13, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d5d1f5daa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Resolved
@morluto

morluto commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Source fixes and publication evidence for the review findings below. Tests and executable checks were skipped at the maintainer's request. Findings that still require a decision or execution evidence remain open.

review thread

The relative-Location parser treated all leading slash and backslash separators as an authority boundary even when the base was a file: URL, so ///u@h could reinterpret path bytes as credentials and discard the original byte evidence. The parser now keeps the fixed // boundary for file bases, matching the explicit-scheme path and preserving the remaining bytes as path data. Published at 4d5d1f5daa332cbffc9e3bcdc6a159d2ac8d0ec2. Validation was source inspection and git diff --check; tests, builds, checks, and CI reruns were skipped under the explicit task constraints. No directly related source cleanup remains.

dakjdakd and others added 3 commits October 14, 2026 01:23
* fix(comparison): preserve incomplete function call evidence

Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty.

Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls.

* fix(comparison): recheck historical calls-only changes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Comment thread bridge/mitmproxy/capture.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 281c4b8ec5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 557148ed3f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bridge/mitmproxy/capture.py Outdated
Comment on lines +61 to +65
if name is not None and name in SPECIAL_SCHEMES and name != "file":
# A same-scheme Location without two slash-equivalent separators is
# relative to the recorded request, so its @ belongs to the path.
following = text[scheme.end():scheme.end() + 2]
if name == base_scheme and (len(following) < 2 or any(char not in "/\\" for char in following)):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve invalid special-scheme path spellings

When an Origin or Referer contains a malformed absolute spelling such as https:user@%, base_scheme is None; WHATWG rejects the absolute form, and the shared owner in src/domain/recordedTransportUrl.ts:22-35 reparses it as a relative path with no credentials. This branch nevertheless treats everything after https: as an authority, so the adapter reports https:% and withdraws the original byte sidecar as a credential redaction. Require a successfully parsed authority before rewriting this form so malformed URL evidence is retained.

AGENTS.md reference: AGENTS.md:L13-L13

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 110151e. Root cause: the bridge rewrote any @ inside a special authority without checking that WHATWG would accept the host, so a URL that fails to parse (and therefore has no credentials) still got redacted. without_userinfo now leaves the value alone when special_host_parses rejects the host or port, which covers empty hosts, forbidden domain code points after percent-decoding (%, ^, space and so on), ports that aren't decimal or exceed 65535, and an unclosed [. https:user@% is now kept as is, while https:user@%41 is still redacted, matching Node. I checked this against Node's URL over 30,000 generated spellings × base schemes: no under-redaction, and over-redaction went from 20,540 to 5,865. What's left is the deliberate empty-userinfo case pinned by test_empty_userinfo_marker_is_removed, IPv4 range and IDNA failures (not modeled, so they fail safe toward redaction), and non-special or baseless references, which are outside this finding.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 929aa67336

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES:
prefix = re.match(r"^[\x00-\x20]*//", text)
return None if prefix is None else (prefix.end(), False)
prefix = re.match(NETWORK_PREFIX, text)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve invalid file-scheme network references

When the captured request scheme is file, a Location such as \\user:pass@host/x or /\user:pass@host/x is accepted by this broadened prefix and the later file branch treats user:pass@ as userinfo. WHATWG file-URL resolution rejects these references because file URLs cannot contain credentials, but the adapter rewrites them to \\host/x and withdraws the byte sidecar as redacted. Restrict slash-equivalent network prefixes for a file base so these malformed raw values remain intact.

AGENTS.md reference: AGENTS.md:L13-L13

Useful? React with 👍 / 👎.

@kaoru0822-kitauji kaoru0822-kitauji Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 110151e. A file URL can never carry userinfo: an @ in a file host makes WHATWG reject the whole URL (Node: URL.parse('\\\\user:pass@host/x', 'file:///dir/'), '/\\user:pass@host/x', '//user:pass@host/x' and 'file://user:pass@host/x' all return null). authority_start now returns None for an explicit file: scheme and for scheme-relative references against a file base, so the source text and byte sidecar are kept. New tests: test_file_urls_reject_userinfo_and_keep_their_source, test_rejected_special_hosts_keep_their_source and test_accepted_special_hosts_still_remove_userinfo. All three fail on the previous head (9 failures) and pass now, together with npm run test:python. Remaining note: the TS owner parseRecordedTransportUrl still reports a credential span for file://user:pass@host (it uses at >= 0 when the parse fails), so the native and HAR adapters can still disagree there. Changing that is a separate owner-side decision.

kaoru0822-kitauji and others added 2 commits October 14, 2026 00:22
…info

File URLs cannot carry userinfo (an "@" in a file host fails the parse),
and a special authority whose host or port WHATWG rejects fails the whole
URL. Neither has credentials, so the bridge now keeps their source text
and byte sidecar. Checked against Node's URL over 30,000 spellings: no
under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty
userinfo, IPv4 range, IDNA, non-special and baseless references).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread bridge/mitmproxy/capture.py Resolved
Comment thread bridge/mitmproxy/capture.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 110151e523

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +106 to +108
host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace")
if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host):
return False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve malformed special-host evidence

For a captured Location such as https://user:pass@256.256.256.256/path, WHATWG rejects the out-of-range IPv4 host, so it has no parsed credentials; however, this code only screens domain code points, removes user:pass@, and withdraws the original byte sidecar. The same gap applies to malformed bracketed IPv6 literals. Retain these rejected URLs (or validate special hosts completely) rather than reporting a destination that was never captured.

AGENTS.md reference: AGENTS.md:L13-L13

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 64fb36d. Special hosts now go through WHATWG's ends-in-a-number check and IPv4 parser: decimal, 0x hex and 0-prefixed octal parts, at most 4 parts, every part but the last ≤ 255, and the last < 256^(5−n). Bracketed literals go through ipaddress.IPv6Address, with zone IDs (%) rejected. 256.256.256.256, 1.2.3.4.5, 1.2.3.09, 4294967296, a.0x, 1..2, [1::2::3] and [::1%25eth0] are now kept unchanged. 0x7f.1, 1.2.3.4., 4294967295, 1.2.3.08a and [::ffff:1.2.3.4] are still redacted, matching Node. Across the 126,000-case differential, the only special-scheme over-redactions left are IDNA failures (xn--, not modeled, so they fail safe) and the empty-userinfo case pinned by test_empty_userinfo_marker_is_removed.

Comment thread bridge/mitmproxy/capture.py Outdated
Comment on lines +112 to +113
digits = port[1:]
return port[0] == ":" and (digits == "" or (digits.isascii() and digits.isdigit() and int(digits) <= 65535))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject overlong ports without numeric conversion

For a Location with a 4,301-digit decimal port, int(digits) raises Python's integer-string limit ValueError before this predicate can return false. That error escapes project_record, causing OfflineCapture.running to return a generic decoder failure for the entire snapshot instead of retaining the malformed URL; compare the digit length/value lexically before converting it.

AGENTS.md reference: AGENTS.md:L75-L75

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same fix as the thread above (64fb36d): ports are compared lexically before int(), so a 4,301-digit port no longer raises.

… ports

The previous host check counted trailing C0 controls and spaces, which
WHATWG strips, so `https://user:pw@host ` kept its password. Trailing
C0/space is now removed from the parse view (raw offsets are unchanged).

Special hosts now also follow the WHATWG IPv4 parser (ends-in-a-number,
hex/octal parts, range) and IPv6 literal syntax (zone IDs rejected), and
ports are bounded lexically before int(), which raised ValueError past
4300 digits on Python 3.11+ and failed the whole capture.

Node URL differential over 126,000 spellings: under-redaction 8,084 -> 0,
ValueError 2,550 -> 0; remaining special-scheme over-redaction is IDNA
and the pinned empty-userinfo case. Cost: +0.6 us per URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

Comment thread bridge/mitmproxy/capture.py Resolved
morluto added a commit that referenced this pull request Oct 14, 2026
Own service-worker CDP registrations and retry failed teardown. Admit redaction and network metadata against remaining capture capacity, retain explicit omission reasons, and preserve source URL, redirect ancestry, caller-selected content, and historical capture values. Forward CLI cancellation through provider cleanup.

Reviewed fixes: #2263, #2637, #2767, #2845, #2859, #2863, #2872, #2874, #2876, #2879, #2955, #2957, #2961, #3033.

Validation: 25 current event/network source tests passed; earlier focused browser lanes passed 201 source tests and 176 boundary tests with two skips, plus 35 Python tests. Real browser/Windows lanes were not rerun. Playwright allocation before admission remains unbounded by these retention limits.
# Conflicts:
#	bridge/mitmproxy/capture.py
#	bridge/mitmproxy/test_capture.py
#	scripts/lib/clean-linux-hopper-runtime.mjs
@kaoru0822-kitauji kaoru0822-kitauji added the track:bugs Small-bug loop track PRs label Oct 14, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

A non-ASCII special host may map under UTS #46 to an accepted host
(fullwidth digits, ideographic full stops, ignored code points), so
treat it as accepted instead of running the ASCII IPv4 check on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

Comment on lines +191 to 192
if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])):
return value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Malformed URLs expose passwords

without_userinfo now keeps passwords when a special-scheme host or port is rejected. For example, a captured Location: https://user:pw@example.test:99999/x keeps user:pw in both the reported text and the original bytes, with no redaction. A bad port does not make the supplied password safe to expose. The shared parseRecordedTransportUrl still removes this raw userinfo span.

Keep credential removal separate from host acceptance. Apply the same protection to the new file early return.

How this was verified: Projecting this rejected-port header kept its password in the reported text and original bytes with no redactions.

Prompt To Fix With AI
This is a comment left during a code review.
Path: bridge/mitmproxy/capture.py
Line: 191-192

Comment:
**Malformed URLs expose passwords**

`without_userinfo` now keeps passwords when a special-scheme host or port is rejected. For example, a captured `Location: https://user:pw@example.test:99999/x` keeps `user:pw` in both the reported text and the original bytes, with no redaction. A bad port does not make the supplied password safe to expose. The shared `parseRecordedTransportUrl` still removes this raw userinfo span.

Keep credential removal separate from host acceptance. Apply the same protection to the new `file` early return.

**How this was verified:** Projecting this rejected-port header kept its password in the reported text and original bytes with no redactions.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@kaoru0822-kitauji
kaoru0822-kitauji merged commit 77be0f6 into main Oct 14, 2026
32 checks passed
@kaoru0822-kitauji
kaoru0822-kitauji deleted the fix/issue-2671 branch October 14, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: bridge Hopper bridge (Python) related changes area: runtime Process capture and runtime observation bug Something isn't working P2: medium Medium priority issue track:bugs Small-bug loop track PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(capture): native capture loses ordinary URL text and original bytes

3 participants