Skip to content

Preserve query string when following relative redirects - #4644

Closed
Puneet15556 wants to merge 2 commits into
huggingface:mainfrom
Puneet15556:fix/relative-redirect-query
Closed

Puneet15556 wants to merge 2 commits into
huggingface:mainfrom
Puneet15556:fix/relative-redirect-query

Conversation

@Puneet15556

@Puneet15556 Puneet15556 commented Aug 7, 2026 •

Copy link
Copy Markdown

Refs #4637 (defect 1 — the dropped query string).

Note: this does not close #4637. As verified by @vollegrewar against the mirror repro, the reported failure has a separate root cause (the absolute 308 from the mirror is never followed, so the response carries no x-repo-commit). This PR is a correctness fix on its own; the issue should stay open until the 308 leg and/or the error misclassification is addressed.

Problem

_httpx_follow_relative_redirects_with_backoff rebuilt the redirect URL like this:

url = urlparse(url)._replace(path=parsed_target.path).geturl()

Only path was swapped, so:

  • the redirect target's query string was dropped,
  • the original URL's query was carried over onto the new path,
  • dot-segments (../) were left unresolved, producing URLs like https://hf.co/../other/config.json.

This matters for xet resolve redirects such as 307 /api/resolve-cache/models/.../config.json?etag=...: losing etag changes the response.

Change

url = urljoin(url, response.headers["Location"])

urljoin is the stdlib implementation of RFC 3986 reference resolution, so the target's query is used and dot-segments are resolved. This is still inside the parsed_target.netloc == "" branch, so absolute redirects are not followed — the function's original intent is unchanged.

Note on query inheritance: per RFC 3986 §5.2.2, T.query = R.query applies whenever the reference defines a path, so a target without a query yields no query rather than inheriting the base's. The base query is only inherited when the reference path is empty (e.g. Location: ?foo=bar). urljoin and the minimal _replace(path=..., query=...) patch agree on that; they differ only on dot-segments, which urljoin handles correctly.

Tests

Added TestFollowRelativeRedirects in tests/test_utils_http.py covering three cases:

  • relative redirect with a query → target query preserved,
  • relative redirect without a query → base query not inherited,
  • relative redirect with dot-segments → resolved.

All three fail on main (the followed URL keeps ?base=old) and pass with this change. tests/test_utils_http.py passes in full locally (66 passed), and ruff format/ruff check are clean on both files.


Note

Low Risk
Narrow change to relative redirect resolution in shared HTTP helpers used for HEAD resolve/metadata; behavior aligns with RFC 3986 and is covered by new unit tests.

Overview
Fixes how _httpx_follow_relative_redirects_with_backoff builds the next URL when the hub returns a relative Location (still only followed when netloc is empty; absolute and protocol-relative redirects are unchanged).

Instead of swapping only the path with urlparse(...)._replace(path=...), the follow-up request now uses urljoin(current_url, Location), so redirect targets keep their query (e.g. xet ?etag=...), targets without a query no longer inherit the original URL’s query, and ../ dot-segments are normalized.

Adds TestFollowRelativeRedirects in tests/test_utils_http.py for query preservation, no base-query inheritance, dot-segment resolution, and that //host/... is not followed.

Reviewed by Cursor Bugbot for commit 6fcf85a. Bugbot is set up for automated code reviews on this repo. Configure here.

`_httpx_follow_relative_redirects_with_backoff` rebuilt the redirect URL with
`urlparse(url)._replace(path=parsed_target.path)`, which only swapped the path.
The redirect target's query string was dropped while the original URL's query
was carried over, and dot-segments were left unresolved.

This matters for xet resolve redirects such as
`/api/resolve-cache/models/.../config.json?etag=...`: dropping `etag` changes
the response, which can leave `x-repo-commit` missing and surface as a
misleading `LocalEntryNotFoundError`.

Use `urljoin`, the stdlib implementation of RFC 3986 reference resolution, so
the target's query is used and `../` segments are resolved.

Fixes huggingface#4637
@vollegrewar

Copy link
Copy Markdown

Verified your branch against the mirror repro from #4637. Thanks for the cc.

The change itself is correct. All three new tests pass (TestFollowRelativeRedirects, 3/3), and tests/test_utils_http.py passes in full here: 64 passed, 2 failed — both failures are pre-existing environment issues on my machine (test_proxy_env_is_used is affected by local proxy env vars, test_client_get_request needs external connectivity), and they fail identically on main. The urljoin behavior matches your RFC 3986 analysis, including the query-dropping-when-target-has-path semantics and dot-segment resolution.

However: the original mirror failure is NOT fixed by this PR. With HF_ENDPOINT=https://hf-mirror.com (which still 308s to huggingface.co as of today), both main and pr-4644 behave identically:

main:    META commit_hash = None  -> snapshot_download: LocalEntryNotFoundError
pr-4644: META commit_hash = None  -> snapshot_download: LocalEntryNotFoundError

I traced it: _httpx_follow_relative_redirects_with_backoff never follows the mirror's absolute 308 (netloc != "" -> break), so the "final" response it returns is the 308 itself, which carries no x-repo-commit header:

final status : 308
final url    : https://hf-mirror.com/Systran/faster-whisper-base/resolve/main/config.json
location hdr : https://huggingface.co/Systran/faster-whisper-base/resolve/main/config.json
x-repo-commit: None

So my earlier guess in the issue ("your fix may restore the missing x-repo-commit and resolve the functional failure entirely") was wrong — the two defects are on different legs of the chain: D1 concerns the relative 307 (Hub -> /api/resolve-cache/...?etag=...), but the mirror scenario dies one hop earlier, at the absolute 308. (Side observation: hitting huggingface.co directly, the resolve-cache endpoint returns X-Repo-Commit even without the ?etag= query — curl-verified — so D1 has no functional impact on the default endpoint either; it's a correctness/RFC-compliance fix.)

Practical implications:

  1. This PR is a solid, well-tested RFC-compliance fix and worth merging on its own. I'd keep it independent as you planned.
  2. It does not close snapshot_download fails with misleading LocalEntryNotFoundError when HF_ENDPOINT mirror 308-redirects to the Hub (x-repo-commit missing); relative-redirect follower drops query string #4637's functional failure. D2 (_raise_on_head_call_error misclassifying FileMetadataError as "check your connection") is therefore not merely a diagnostics nicety — mirror users still can't download at all, and the message actively points them the wrong way. I'd argue D2 remains a genuine functional fix.

Happy to dig further (e.g. whether following absolute redirects would be acceptable, given the function's stated intent of not following CDN redirects) if it helps the maintainers decide how to handle the mirror case.

@Puneet15556

Copy link
Copy Markdown
Author

Thanks for actually running this against the mirror repro — that's really useful, and good catch that D1 doesn't resolve the functional failure.

I've switched the PR description from Fixes #4637 to Refs #4637 so that merging this doesn't auto-close the issue while the mirror case is still broken. This PR stands on its own as a correctness fix (target query preserved, base query not inherited, dot-segments resolved), but it is not the mirror fix.

Next I'll open a separate PR for D2 — _raise_on_head_call_error classifying FileMetadataError as a connectivity problem, which is what produces the misleading "check your connection" message.

On D3 (following the absolute 308): I'd rather leave that call to the maintainers, since the function deliberately avoids following absolute redirects to skip CDN hops, so widening it has implications beyond this issue. If they're open to it, one narrow option would be to follow absolute redirects only when they stay on the same endpoint/host family. Happy to test whichever direction they prefer against your repro.

@vollegrewar vollegrewar left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the reporter of #4637 (HF_ENDPOINT mirror 308-redirect drops the query string), I verified this fix against the reproduction:

  • Before: the relative-redirect branch rebuilt the URL from parsed_target.path only, silently dropping the target's query string (e.g. the xet ?etag=... lookup), which cascaded into the misleading LocalEntryNotFoundError.
  • After: urljoin() resolves the reference per RFC 3986 — target query preserved, base query not inherited when the target has none, dot-segments resolved. Verified locally with the mirror-redirect reproduction plus the three new tests: all pass.

Non-blocking suggestion: the netloc == "" guard is what routes protocol-relative (//host/...) Location headers away from this branch today; a small test pinning that case would protect it against future refactors.

Fix matches the root cause from the issue — approve.

A `//host/path` Location has an empty scheme but a non-empty netloc, so the
`parsed_target.netloc == ""` guard keeps it out of the relative-redirect branch
today. Without that guard `urljoin` would resolve it to another host, which the
function deliberately avoids. Pin the current behaviour so a future refactor
cannot silently start following cross-host redirects.
@Puneet15556

Copy link
Copy Markdown
Author

Thanks for the approval and for tracing the 308 leg so precisely.

I've taken the non-blocking suggestion — added test_protocol_relative_redirect_is_not_followed, which asserts that a //host/... Location leaves the redirect unfollowed (single request, the 3xx response returned as-is). I checked it actually pins the behaviour: temporarily removing the netloc == "" guard makes it fail, since urljoin would otherwise resolve //huggingface.co/... onto another host.

tests/test_utils_http.py is at 67 passed locally, and ruff format/ruff check are clean.

D2 is next, in its own PR.

@Puneet15556

Copy link
Copy Markdown
Author

Heads-up for whoever picks this up: #4648 touches the same function, so the two PRs conflict as they stand — I verified locally, the overlap is in src/huggingface_hub/utils/_http.py and tests/test_utils_http.py.

The two changes are complementary rather than competing:

The conflict is essentially one line in the relative branch, since #4648 branched off main and still carries the old _replace(path=...) there. Worth noting that urljoin also covers the absolute case (urljoin(base, "https://host/x") returns the absolute URL unchanged), so the merged form can use it in both branches if that reads better.

No preference on ordering, and no rush from my side — whichever lands first, I'll rebase this one on top and re-run the tests. Just flagging it so it isn't a surprise at merge time.

@Puneet15556

Copy link
Copy Markdown
Author

Gentle ping on this one — it's been a couple of weeks and I'd rather not let it go stale silently.

Current state: still MERGEABLE against main (no conflicts despite the intervening commits), checks green, and independently verified against the mirror repro by @vollegrewar. The change itself is small and self-contained — urljoin instead of hand-rolled URL rebuilding in _httpx_follow_relative_redirects_with_backoff, plus four tests covering target query preserved, base query not inherited, dot-segments resolved, and protocol-relative targets left unfollowed.

Since #4648 touches the same function and carries the user-facing mirror fix, I'm genuinely happy with whichever path is least work for you:

  1. merge this first and I'll help rebase Fix metadata fetch on endpoints that return absolute redirects #4648, or
  2. merge Fix metadata fetch on endpoints that return absolute redirects #4648 first and I'll rebase this on top, or
  3. close this and let Fix metadata fetch on endpoints that return absolute redirects #4648 absorb the urljoin change — the tests here would still be worth keeping.

No preference at all on my side; just flagging that it's ready and unblocked so it doesn't need a second look from scratch later.

@vollegrewar

Copy link
Copy Markdown

Gentle ping from the reporter of #4637: this PR has been open since Aug 7 and is the smaller, independent half (preserving the query string when following relative redirects). I verified it against the live mirror reproduction and left an approving review with the evidence; the three new tests pass. The sibling fix for absolute redirects is #4648. Both are ready whenever a maintainer can take a look — happy to rebase or adjust either one.

@Puneet15556

Copy link
Copy Markdown
Author

Closing as superseded: #4739 (merged Aug 28) resolves the relative-redirect leg with the same urljoin approach, so this change is now on main. Glad the fix landed.

The test coverage from this PR (target query preserved, base query not inherited, dot-segments, protocol-relative targets) is carried forward on https://github.com/Puneet15556/huggingface_hub/tree/fix/hub-redirect-scheme-and-tests together with the scheme-downgrade hardening noted on #4739 — I'll open that as a PR once fork PRs are possible again.

@Puneet15556 Puneet15556 closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants