Repository navigation
fix(capture): end special-scheme URL authorities at a backslash - #2767
Conversation
The native mitmproxy adapter removed transport userinfo with `^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?" or "#". For `https://example.test\part@tail` it treated `example.test\part` as userinfo, reported `https://tail` as the Location, and withdrew the original bytes as a transport credential. WHATWG URL parsing, used by the shared TypeScript owner (recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or scheme-relative authority at a backslash, so that "@" is ordinary path text. Locate the authority with the same boundary rule before removing userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay identical. Userinfo before the backslash is still removed and redacted. Add source unit tests for the adapter projection to test:python. Fixes #2671 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 553cd6e358
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Ending the authority at a backslash exposed a credential: for `https://\user:pass@host/path`, WHATWG parsing skips every leading "/" or "\" after a special scheme (and in scheme-relative references), so `user:pass` is userinfo, but the new boundary stopped at the first backslash and found no "@". The same applied to `https:\\user:pass@host` and `https:user:pass@host`. Start the authority after that whole slash run for non-file special schemes and scheme-relative references. File URLs keep their "//" boundary because they carry no userinfo. Load the adapter in tests with fresh mitmproxy stand-ins scoped to its import instead of leaving them in sys.modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d972a7e85
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 72e931f3a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8361b7715
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open. The first boundary stopped before a special-scheme authority had consumed its leading slash/backslash run. The existing fix now consumes that run before locating userinfo, and the retained regression source covers the leading-backslash spelling and byte-sidecar redaction. The later redirect fix also uses the owning recorded request's scheme so relative Location paths retain their text. Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection. The authority parser lacked the recorded request's URL context. Projection now derives the scheme once at each flow/backup boundary and supplies it specifically to Location handling. Same-scheme references without two slash-equivalent separators keep their relative path text and original bytes; absolute and cross-scheme authorities still remove userinfo. Regression source includes ordinary, slash/backslash-relative, absolute, and backup-flow cases. Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection. Authority lookup had interpreted raw controls that WHATWG removes first. The adapter now derives a tab/LF/CR-normalized parse view, locates authority/userinfo there, and maps only the credential span back to the original spelling in a linear scan without an index array. Original noncredential text/bytes remain unchanged; any removed userinfo also redacts the raw byte sidecar. Regression source covers controls inside the scheme, leading separators, and credentials. Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection. The no-scheme prefix recognized only literal //. It now recognizes every pair of slash-equivalent separators for network references, then consumes their leading run before locating the authority. Explicit non-special schemes still require their literal // syntax. The regression source includes backslash/backslash and both mixed pairs, preserving the original prefix while removing userinfo and redacting its byte sidecar. Source fix: e8361b7. Tests and checks were skipped at the user's explicit request; this assessment is based on source inspection. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0aff997b31
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open. The authority parser treated every scheme-relative header as special, independently of the recorded request scheme. It now derives the separator/terminator rule from the parsed base scheme: a non-special base recognizes // and keeps backslashes within authority data, so userinfo and its byte sidecar are redacted together. The producer regression includes foo plus //user\name:pass@host/x and a non-special mixed-separator path. The existing source-spelling/byte projection stays owned by the same redactor; no downstream repair or related cleanup remains. Source validation: inspected the producing boundary, affected consumers, authored regressions and final diff. Tests, checks, builds, lint, typechecks, provider workloads and CI reruns were not run at the user's explicit request; no runtime/provider verification is claimed. Source fix: 0aff997. The projection applied header URL-reference semantics to native request.path, conflating origin-form path evidence with network authorities. The producer now distinguishes header URL references, native authority, and native paths: origin-form separator runs are retained exactly, while explicitly schemed absolute-form URLs still remove userinfo. Authored regressions cover /\invoice@customer, //invoice@customer, backslash-prefixed paths, retained byte sidecars, and absolute-form credential redaction. No related cleanup remains. Source validation: inspected the producing boundary, affected consumers, authored regressions and final diff. Tests, checks, builds, lint, typechecks, provider workloads and CI reruns were not run at the user's explicit request; no runtime/provider verification is claimed. Source fix: 0aff997. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d5d1f5daa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Source fixes and publication evidence for the review findings below. Tests and executable checks were skipped at the maintainer's request. Findings that still require a decision or execution evidence remain open. The relative-Location parser treated all leading slash and backslash separators as an authority boundary even when the base was a |
* fix(comparison): preserve incomplete function call evidence Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty. Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls. * fix(comparison): recheck historical calls-only changes --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 281c4b8ec5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 557148ed3f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if name is not None and name in SPECIAL_SCHEMES and name != "file": | ||
| # A same-scheme Location without two slash-equivalent separators is | ||
| # relative to the recorded request, so its @ belongs to the path. | ||
| following = text[scheme.end():scheme.end() + 2] | ||
| if name == base_scheme and (len(following) < 2 or any(char not in "/\\" for char in following)): |
There was a problem hiding this comment.
Preserve invalid special-scheme path spellings
When an Origin or Referer contains a malformed absolute spelling such as https:user@%, base_scheme is None; WHATWG rejects the absolute form, and the shared owner in src/domain/recordedTransportUrl.ts:22-35 reparses it as a relative path with no credentials. This branch nevertheless treats everything after https: as an authority, so the adapter reports https:% and withdraws the original byte sidecar as a credential redaction. Require a successfully parsed authority before rewriting this form so malformed URL evidence is retained.
AGENTS.md reference: AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 110151e. Root cause: the bridge rewrote any @ inside a special authority without checking that WHATWG would accept the host, so a URL that fails to parse (and therefore has no credentials) still got redacted. without_userinfo now leaves the value alone when special_host_parses rejects the host or port, which covers empty hosts, forbidden domain code points after percent-decoding (%, ^, space and so on), ports that aren't decimal or exceed 65535, and an unclosed [. https:user@% is now kept as is, while https:user@%41 is still redacted, matching Node. I checked this against Node's URL over 30,000 generated spellings × base schemes: no under-redaction, and over-redaction went from 20,540 to 5,865. What's left is the deliberate empty-userinfo case pinned by test_empty_userinfo_marker_is_removed, IPv4 range and IDNA failures (not modeled, so they fail safe toward redaction), and non-special or baseless references, which are outside this finding.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 929aa67336
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES: | ||
| prefix = re.match(r"^[\x00-\x20]*//", text) | ||
| return None if prefix is None else (prefix.end(), False) | ||
| prefix = re.match(NETWORK_PREFIX, text) |
There was a problem hiding this comment.
Preserve invalid file-scheme network references
When the captured request scheme is file, a Location such as \\user:pass@host/x or /\user:pass@host/x is accepted by this broadened prefix and the later file branch treats user:pass@ as userinfo. WHATWG file-URL resolution rejects these references because file URLs cannot contain credentials, but the adapter rewrites them to \\host/x and withdraws the byte sidecar as redacted. Restrict slash-equivalent network prefixes for a file base so these malformed raw values remain intact.
AGENTS.md reference: AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 110151e. A file URL can never carry userinfo: an @ in a file host makes WHATWG reject the whole URL (Node: URL.parse('\\\\user:pass@host/x', 'file:///dir/'), '/\\user:pass@host/x', '//user:pass@host/x' and 'file://user:pass@host/x' all return null). authority_start now returns None for an explicit file: scheme and for scheme-relative references against a file base, so the source text and byte sidecar are kept. New tests: test_file_urls_reject_userinfo_and_keep_their_source, test_rejected_special_hosts_keep_their_source and test_accepted_special_hosts_still_remove_userinfo. All three fail on the previous head (9 failures) and pass now, together with npm run test:python. Remaining note: the TS owner parseRecordedTransportUrl still reports a credential span for file://user:pass@host (it uses at >= 0 when the parse fails), so the native and HAR adapters can still disagree there. Changing that is a separate owner-side decision.
…info File URLs cannot carry userinfo (an "@" in a file host fails the parse), and a special authority whose host or port WHATWG rejects fails the whole URL. Neither has credentials, so the bridge now keeps their source text and byte sidecar. Checked against Node's URL over 30,000 spellings: no under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty userinfo, IPv4 range, IDNA, non-special and baseless references). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 110151e523
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace") | ||
| if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host): | ||
| return False |
There was a problem hiding this comment.
Preserve malformed special-host evidence
For a captured Location such as https://user:pass@256.256.256.256/path, WHATWG rejects the out-of-range IPv4 host, so it has no parsed credentials; however, this code only screens domain code points, removes user:pass@, and withdraws the original byte sidecar. The same gap applies to malformed bracketed IPv6 literals. Retain these rejected URLs (or validate special hosts completely) rather than reporting a destination that was never captured.
AGENTS.md reference: AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 64fb36d. Special hosts now go through WHATWG's ends-in-a-number check and IPv4 parser: decimal, 0x hex and 0-prefixed octal parts, at most 4 parts, every part but the last ≤ 255, and the last < 256^(5−n). Bracketed literals go through ipaddress.IPv6Address, with zone IDs (%) rejected. 256.256.256.256, 1.2.3.4.5, 1.2.3.09, 4294967296, a.0x, 1..2, [1::2::3] and [::1%25eth0] are now kept unchanged. 0x7f.1, 1.2.3.4., 4294967295, 1.2.3.08a and [::ffff:1.2.3.4] are still redacted, matching Node. Across the 126,000-case differential, the only special-scheme over-redactions left are IDNA failures (xn--, not modeled, so they fail safe) and the empty-userinfo case pinned by test_empty_userinfo_marker_is_removed.
| digits = port[1:] | ||
| return port[0] == ":" and (digits == "" or (digits.isascii() and digits.isdigit() and int(digits) <= 65535)) |
There was a problem hiding this comment.
Reject overlong ports without numeric conversion
For a Location with a 4,301-digit decimal port, int(digits) raises Python's integer-string limit ValueError before this predicate can return false. That error escapes project_record, causing OfflineCapture.running to return a generic decoder failure for the entire snapshot instead of retaining the malformed URL; compare the digit length/value lexically before converting it.
AGENTS.md reference: AGENTS.md:L75-L75
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same fix as the thread above (64fb36d): ports are compared lexically before int(), so a 4,301-digit port no longer raises.
… ports The previous host check counted trailing C0 controls and spaces, which WHATWG strips, so `https://user:pw@host ` kept its password. Trailing C0/space is now removed from the parse view (raw offsets are unchanged). Special hosts now also follow the WHATWG IPv4 parser (ends-in-a-number, hex/octal parts, range) and IPv6 literal syntax (zone IDs rejected), and ports are bounded lexically before int(), which raised ValueError past 4300 digits on Python 3.11+ and failed the whole capture. Node URL differential over 126,000 spellings: under-redaction 8,084 -> 0, ValueError 2,550 -> 0; remaining special-scheme over-redaction is IDNA and the pinned empty-userinfo case. Cost: +0.6 us per URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Own service-worker CDP registrations and retry failed teardown. Admit redaction and network metadata against remaining capture capacity, retain explicit omission reasons, and preserve source URL, redirect ancestry, caller-selected content, and historical capture values. Forward CLI cancellation through provider cleanup. Reviewed fixes: #2263, #2637, #2767, #2845, #2859, #2863, #2872, #2874, #2876, #2879, #2955, #2957, #2961, #3033. Validation: 25 current event/network source tests passed; earlier focused browser lanes passed 201 source tests and 176 boundary tests with two skips, plus 35 Python tests. Real browser/Windows lanes were not rerun. Playwright allocation before admission remains unbounded by these retention limits.
# Conflicts: # bridge/mitmproxy/capture.py # bridge/mitmproxy/test_capture.py # scripts/lib/clean-linux-hopper-runtime.mjs
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
A non-ASCII special host may map under UTS #46 to an accepted host (fullwidth digits, ideographic full stops, ignored code points), so treat it as accepted instead of running the ASCII IPv4 check on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
| if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])): | ||
| return value |
There was a problem hiding this comment.
Malformed URLs expose passwords
without_userinfo now keeps passwords when a special-scheme host or port is rejected. For example, a captured Location: https://user:pw@example.test:99999/x keeps user:pw in both the reported text and the original bytes, with no redaction. A bad port does not make the supplied password safe to expose. The shared parseRecordedTransportUrl still removes this raw userinfo span.
Keep credential removal separate from host acceptance. Apply the same protection to the new file early return.
How this was verified: Projecting this rejected-port header kept its password in the reported text and original bytes with no redactions.
Prompt To Fix With AI
This is a comment left during a code review.
Path: bridge/mitmproxy/capture.py
Line: 191-192
Comment:
**Malformed URLs expose passwords**
`without_userinfo` now keeps passwords when a special-scheme host or port is rejected. For example, a captured `Location: https://user:pw@example.test:99999/x` keeps `user:pw` in both the reported text and the original bytes, with no redaction. A bad port does not make the supplied password safe to expose. The shared `parseRecordedTransportUrl` still removes this raw userinfo span.
Keep credential removal separate from host acceptance. Apply the same protection to the new `file` early return.
**How this was verified:** Projecting this rejected-port header kept its password in the reported text and original bytes with no redactions.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Fixes #2671.
Problem
The native mitmproxy adapter (
bridge/mitmproxy/capture.py) removes transport userinfo fromLocation/Referer/Originand nativerequest.path/authoritywith this regex:The authority therefore ran to the last
@before/,?or#. ForLocation: https://example.test\part@tail, the adapter treatedexample.test\partas userinfo:https://tail, a different destination;transport-credentialredaction (state: "redacted", no content, size or digest).The shared TypeScript owner,
src/domain/recordedTransportUrl.ts, follows WHATWG URL parsing, as HAR captures do. There, a backslash ends the authority of a special scheme (http,https,ws,wss,ftp,file) or a scheme-relative//reference. That@is ordinary path text, and the HAR projector preserves the value.Fix
without_userinfolocates the authority with the same boundary rule asrecordedTransportUrl.ts:/,?or#, and also at\for special or scheme-relative references.@inside that authority.Userinfo before a backslash, as in
https://user:pw@example.test\path, is still removed and redacted, and non-special schemes keep their previous behavior. The/authoritypath handling is unchanged.Validation
bridge/mitmproxy/test_capture.py, added tonpm run test:python, which CI's static job runs. It stubs themitmproxyimports and calls the realproject_record/without_userinfo:https://example.test\part@tailis reported unchanged, with its byte sidecar retained and no redaction. This fails on main, which returnshttps://tail.https://user:pw@example.test\pathbecomeshttps://example.test\path, with thetransport-credentialredaction.https://user:pw@example.test/a@bbecomeshttps://example.test/a@b.npm run test:pythonpasses (13 + 11 tests), and so doestests/boundary/filesystem/ciTestReports.test.ts, which checks thetest:pythonwiring.npm run check:fast,metadata:checkandformat:checkpass.REA_MITMDUMP_COMMAND.🤖 Generated with Claude Code