Repository navigation
Preserve query string when following relative redirects - #4644
Puneet15556 wants to merge 2 commits into
Conversation
`_httpx_follow_relative_redirects_with_backoff` rebuilt the redirect URL with `urlparse(url)._replace(path=parsed_target.path)`, which only swapped the path. The redirect target's query string was dropped while the original URL's query was carried over, and dot-segments were left unresolved. This matters for xet resolve redirects such as `/api/resolve-cache/models/.../config.json?etag=...`: dropping `etag` changes the response, which can leave `x-repo-commit` missing and surface as a misleading `LocalEntryNotFoundError`. Use `urljoin`, the stdlib implementation of RFC 3986 reference resolution, so the target's query is used and `../` segments are resolved. Fixes huggingface#4637
|
Verified your branch against the mirror repro from #4637. Thanks for the cc. The change itself is correct. All three new tests pass ( However: the original mirror failure is NOT fixed by this PR. With I traced it: So my earlier guess in the issue ("your fix may restore the missing x-repo-commit and resolve the functional failure entirely") was wrong — the two defects are on different legs of the chain: D1 concerns the relative 307 (Hub -> /api/resolve-cache/...?etag=...), but the mirror scenario dies one hop earlier, at the absolute 308. (Side observation: hitting huggingface.co directly, the resolve-cache endpoint returns Practical implications:
Happy to dig further (e.g. whether following absolute redirects would be acceptable, given the function's stated intent of not following CDN redirects) if it helps the maintainers decide how to handle the mirror case. |
|
Thanks for actually running this against the mirror repro — that's really useful, and good catch that D1 doesn't resolve the functional failure. I've switched the PR description from Next I'll open a separate PR for D2 — On D3 (following the absolute |
vollegrewar
left a comment
There was a problem hiding this comment.
As the reporter of #4637 (HF_ENDPOINT mirror 308-redirect drops the query string), I verified this fix against the reproduction:
- Before: the relative-redirect branch rebuilt the URL from parsed_target.path only, silently dropping the target's query string (e.g. the xet ?etag=... lookup), which cascaded into the misleading LocalEntryNotFoundError.
- After: urljoin() resolves the reference per RFC 3986 — target query preserved, base query not inherited when the target has none, dot-segments resolved. Verified locally with the mirror-redirect reproduction plus the three new tests: all pass.
Non-blocking suggestion: the netloc == "" guard is what routes protocol-relative (//host/...) Location headers away from this branch today; a small test pinning that case would protect it against future refactors.
Fix matches the root cause from the issue — approve.
A `//host/path` Location has an empty scheme but a non-empty netloc, so the `parsed_target.netloc == ""` guard keeps it out of the relative-redirect branch today. Without that guard `urljoin` would resolve it to another host, which the function deliberately avoids. Pin the current behaviour so a future refactor cannot silently start following cross-host redirects.
|
Thanks for the approval and for tracing the 308 leg so precisely. I've taken the non-blocking suggestion — added
D2 is next, in its own PR. |
|
Heads-up for whoever picks this up: #4648 touches the same function, so the two PRs conflict as they stand — I verified locally, the overlap is in The two changes are complementary rather than competing:
The conflict is essentially one line in the relative branch, since #4648 branched off No preference on ordering, and no rush from my side — whichever lands first, I'll rebase this one on top and re-run the tests. Just flagging it so it isn't a surprise at merge time. |
|
Gentle ping on this one — it's been a couple of weeks and I'd rather not let it go stale silently. Current state: still Since #4648 touches the same function and carries the user-facing mirror fix, I'm genuinely happy with whichever path is least work for you:
No preference at all on my side; just flagging that it's ready and unblocked so it doesn't need a second look from scratch later. |
|
Gentle ping from the reporter of #4637: this PR has been open since Aug 7 and is the smaller, independent half (preserving the query string when following relative redirects). I verified it against the live mirror reproduction and left an approving review with the evidence; the three new tests pass. The sibling fix for absolute redirects is #4648. Both are ready whenever a maintainer can take a look — happy to rebase or adjust either one. |
|
Closing as superseded: #4739 (merged Aug 28) resolves the relative-redirect leg with the same The test coverage from this PR (target query preserved, base query not inherited, dot-segments, protocol-relative targets) is carried forward on https://github.com/Puneet15556/huggingface_hub/tree/fix/hub-redirect-scheme-and-tests together with the scheme-downgrade hardening noted on #4739 — I'll open that as a PR once fork PRs are possible again. |
Refs #4637 (defect 1 — the dropped query string).
Problem
_httpx_follow_relative_redirects_with_backoffrebuilt the redirect URL like this:Only
pathwas swapped, so:../) were left unresolved, producing URLs likehttps://hf.co/../other/config.json.This matters for xet resolve redirects such as
307 /api/resolve-cache/models/.../config.json?etag=...: losingetagchanges the response.Change
urljoinis the stdlib implementation of RFC 3986 reference resolution, so the target's query is used and dot-segments are resolved. This is still inside theparsed_target.netloc == ""branch, so absolute redirects are not followed — the function's original intent is unchanged.Note on query inheritance: per RFC 3986 §5.2.2,
T.query = R.queryapplies whenever the reference defines a path, so a target without a query yields no query rather than inheriting the base's. The base query is only inherited when the reference path is empty (e.g.Location: ?foo=bar).urljoinand the minimal_replace(path=..., query=...)patch agree on that; they differ only on dot-segments, whichurljoinhandles correctly.Tests
Added
TestFollowRelativeRedirectsintests/test_utils_http.pycovering three cases:All three fail on
main(the followed URL keeps?base=old) and pass with this change.tests/test_utils_http.pypasses in full locally (66 passed), andruff format/ruff checkare clean on both files.Note
Low Risk
Narrow change to relative redirect resolution in shared HTTP helpers used for HEAD resolve/metadata; behavior aligns with RFC 3986 and is covered by new unit tests.
Overview
Fixes how
_httpx_follow_relative_redirects_with_backoffbuilds the next URL when the hub returns a relativeLocation(still only followed whennetlocis empty; absolute and protocol-relative redirects are unchanged).Instead of swapping only the path with
urlparse(...)._replace(path=...), the follow-up request now usesurljoin(current_url, Location), so redirect targets keep their query (e.g. xet?etag=...), targets without a query no longer inherit the original URL’s query, and../dot-segments are normalized.Adds
TestFollowRelativeRedirectsintests/test_utils_http.pyfor query preservation, no base-query inheritance, dot-segment resolution, and that//host/...is not followed.Reviewed by Cursor Bugbot for commit 6fcf85a. Bugbot is set up for automated code reviews on this repo. Configure here.