Repository navigation
fix(router): stop a per-checkpoint pin from silently disabling the ca… - #690
Conversation
bde5ba6 to
5871cb1
Compare
…ller's digest and revision
`Router.load` dropped two of the integrity controls it was meant to forward.
`agent_kwargs["expected_sha256"]` is a documented option and deliberately not router-owned, but
any `sha256_digests` entry for the checkpoint being loaded replaced it outright -- including the
`{}` placeholder `_digests_from_env` leaves for every checkpoint a nested `LAYA_SHA256_DIGESTS`
does not name. Pinning one checkpoint through the environment therefore turned verification off
for the others, and `{}` means "verify nothing" because `verify_digests` falls back to the
environment only when the argument is None. The two maps are now merged file by file, and a
placeholder never cancels a pin.
For a file both channels name, the per-checkpoint entry wins. They are not equally specific:
`agent_kwargs` reaches every checkpoint the Router builds, and `model.safetensors` is the one name
every checkpoint uses for a *different* file, so a shared entry for it cannot be a correct claim
about all of them at once. Refusing the overlap as a contradiction -- which an earlier draft of this
fix did -- turned a configuration that worked before the fix into a startup ValueError: verified
against upstream, a shared `agent_kwargs` pin plus `sha256_digests={"multilingual": {...}}` loaded
english with the shared digest and multilingual with its own. Through LAYA_SHA256_DIGESTS it was
worse, because `load("english")` succeeded while `load("multilingual")` raised and `laya/serve.py`
turns that into an HTTP 422 on every request routed there. Nothing raises now.
The distinction that is load-bearing is None versus `{}`, not who wrote the entry. An absent entry
passes no kwarg, leaving `verify_digests` its own fallback; an empty one passes `{}`, which masks it
-- necessary in both directions, because a flat variable is applied by `verify_digests` for every
checkpoint, and a nested one, if left to it, is read as an artifact map and raises `cannot verify
'english': no such file` on a checkpoint it never named. Both halves are tested; I asserted the
opposite of the second first and a run corrected me.
A `None` or blank entry in `revisions` was read as a pin of nothing: it dropped the Router-wide
`revision` and let `$LAYA_REVISION` decide, so a caller's explicit pin lost to the environment on
exactly the checkpoint it named -- while docs/docker.md promises the opposite. Such an entry now
means "no override for this model" and inherits `revision`. An unconfigured Router still passes
no revision and no digest kwarg, so NandhaKishorM#332's opt-in guarantee is unchanged.
Verified against the real multilingual checkpoint, offline: a digest that cannot match was
accepted in two of three configurations before and is refused in all three now (peak RSS 2.14 GB
-> 0.23 GB, the load no longer reaching the weights), a correct one still loads and answers, and
an unconfigured Router still loads an HF_HUB_OFFLINE=1 cache. 24 checks added to
tests/test_revision_pinning.py (50 -> 74), 14 of which go red with laya/router.py reverted; no
existing check needed changing.
End to end on the real 322M multilingual checkpoint, one subprocess per case so peak RSS means what
it says: a digest that cannot match, delivered through each of the three channels that document
themselves as pinning, is refused by all three (0.19-0.20 GB peak, the load never reaching the
weights) where upstream accepted it through one of them at 2.74 GB. An unconfigured Router still
loads and answers, which is the control that stops "refuses everything" looking like a fix.
Both distinctions are read from `sha256_digests` at load time rather than decided when the Router is
built, because the attribute is public and mutable: a per-checkpoint pin assigned after `Router(...)`,
or added to an existing entry in place, has to count, and deciding from a key set frozen at
construction dropped exactly that.
… untested guards From an adversarial review, which called the fix itself "correct and genuinely well tested" -- 120 configuration combinations and 27 mutants, 18 killed, and the diff strictly increases what gets verified relative to 9d95567. Two docstring claims were wrong, and three guards had no test. THE CLAIMS. The class docstring said an `agent_kwargs["expected_sha256"]` map "is never discarded wholesale: where a checkpoint also has digests of its own, from `sha256_digests` or from the environment, the two maps are merged file by file". That is true of a per-checkpoint `LAYA_SHA256_DIGESTS`, and false of a FLAT one: `verify_digests` reads a flat variable only when nothing else pins (`if expected is None`), so any `expected_sha256` reaching `Agent` masks it entirely. Verified on real files -- a flat variable pinning `model.safetensors` plus an `agent_kwargs` map pinning `tokenizer.json` loads a tampered `model.safetensors`. I tried to fix the behaviour first, by merging the flat shape in as a lowest-precedence layer. Four existing checks went red, and they were right: `test_flat_map_is_left_to_revisions`, `test_unparseable_environment_is_not_this_layers_error`, and two pinning that an explicit `None` entry MASKS a flat environment map -- which is what "load this one unverified" has to mean. The behaviour is deliberate and tested; the docstring was what overclaimed. It now names both exceptions and says the flat case is unchanged from `main`. Second claim: the blank-`revision` case was called "the one place a weaker source ends up ahead of an explicit argument". It is one of two. A per-checkpoint entry synthesised from `LAYA_SHA256_DIGESTS` also beats an `expected_sha256` passed in code -- which `test_an_environment_pin_overrides_the_shared_one_per_checkpoint` enshrines as intended, so it is now stated plainly rather than contradicted. An environment variable beating an explicit argument deserves a line of its own on a supply-chain control. THREE MUTANTS THAT SURVIVED EVERY LANE, now killed: - dropping the `per_model` non-dict guard: a list of pairs would be `dict.update`-ed in and silently accepted as a digest map. Both shapes still fail closed, but only one fails in `verify_digests`' own words, which is what the guard is for. - dropping the `shared` non-dict guard: the same gap on the other half. - `resolve_revision`'s blank-argument behaviour. `(revision or env).strip()` means a truthy-but-blank argument wins the `or` and then strips to empty, so `$LAYA_REVISION` does not apply. The Router drops a blank before it gets there and that is tested; `Agent` and `laya.load` do not, so a config line that ends up whitespace silently loses a deployment's pin. Undocumented and untested -- the mutant that makes a blank behave like an absent value survived every lane. Now pinned as the behaviour that exists, alongside the documented gap already in this PR's description. A fourth check pins the flat-environment exception itself, so the docstring cannot drift from it again. Verified: tests/test_revision_pinning.py 78 tests, OK (was 74). test_hooks_api and test_download green. All 60 ci.yml script suites pass. ruff and compileall clean. Real 322M checkpoint, wrong digest through all three channels: 3 of 3 refused at 0.19-0.20 GB peak RSS, control loads. Mutation sweep after the change: 5 of 5 killed, 0 survivors.
efe1d97 to
489fb8e
Compare
Bruce-Yii
left a comment
There was a problem hiding this comment.
Both bugs reproduce on current main and both are fixed here. I re-derived them from the source and then measured the resolution on both trees, weight-free, so neither claim rests on the description.
Bug 1 — a per-checkpoint entry silently discards the caller's pin
laya/router.py:451 merges the caller's map in, and :452-455 then overwrites it:
kwargs.update(self.agent_kwargs) # :451 caller pins tokenizer.json
if key in self.sha256_digests:
kwargs["expected_sha256"] = self.sha256_digests[key] or {} # :455 replacedA stubbed Agent recording what the Router hands it, same probe both trees:
| configuration | main |
this branch |
|---|---|---|
caller pins tokenizer.json, per-checkpoint entry is {} |
{} — caller's pin gone |
{'tokenizer.json': …} |
caller pins tokenizer.json, per-checkpoint pins model.safetensors |
{'model.safetensors': …} — tokenizer pin dropped |
both, merged |
caller pins tokenizer.json, nested LAYA_SHA256_DIGESTS also names it |
{} |
{'tokenizer.json': …} |
The or {} is the sharp edge. laya/revisions.py:89 reads the environment only if expected is None, so a non-None empty map does not just mean "verify nothing" — it also masks a LAYA_SHA256_DIGESTS that was set correctly. An unfilled per-checkpoint config line therefore switches off a digest control the operator asked for. Merging file-by-file is the right shape, because a digest is a claim about one file and dropping either half verifies less than was requested.
Bug 2 — a blank per-checkpoint revision discards the router-wide pin
laya/router.py:447 is self.revisions.get(key, self.revision), and .get falls back only on an absent key. :448 then omits the kwarg entirely when the value is None. Measured, with Router(revision="REVIEWED_AAA"):
| configuration | main reaches the Agent with |
this branch |
|---|---|---|
revisions={"english": None} |
absent | REVIEWED_AAA |
revisions={"english": ""} |
"" |
REVIEWED_AAA |
the documented revisions={"english": os.environ.get("EN_SHA")} with EN_SHA unset |
absent | REVIEWED_AAA |
control: revisions={"english": "MODEL_CCC"} |
MODEL_CCC |
MODEL_CCC |
Absent is the outcome that matters: the checkpoint then resolves to the hub's default branch head, which is an unreviewed commit. So the documented pattern in the class's own docstring loses the pin whenever the environment variable is unset — the most likely state, not an edge case. Treating unset and blank as the same answer, because both are an unfilled configuration line, is the correct reading.
On the precedence table, since a wrong table is where this kind of change does damage
I checked the rows that could bite. Per-checkpoint winning over a shared entry for the same file is right, and the stated reason holds: shared reaches every checkpoint, and model.safetensors names a different file in each, so a shared entry for it cannot be a correct claim about all of them. Keeping {} reaching Agent rather than "simplifying" it away is also right, for the reason in finding 1 — it is what keeps a nested variable from being read as an artifact map and raising no such file on a checkpoint it never named. A non-mapping is passed through untouched, which keeps verify_digests owning that message.
One behaviour change beyond the two reported bugs, disclosed
Router(revision=" ") used to reach resolve_revision, where (revision or env).strip() meant a truthy-but-blank string suppressed LAYA_REVISION. Here it is dropped, so $LAYA_REVISION now applies. That is in the docstring and pinned by a test, and it is the safer direction — but it is a third behaviour change riding along with two, so I am naming it rather than letting it pass as part of the fix.
The red on this PR is not yours
TypeScript (node22) fails, and it is worth not spending review time on it. The failing step is Unit tests, and the single failure is laya-ts/tests/identifier-complexity.test.ts > "4x the input costs under 8x the time", a wall-clock ratio assertion that missed by 0.7% (expected 8.059531496767748 to be less than 8) on a shared runner. This PR changes Python files and no TypeScript. Across the open pool the pattern is 20 PRs with both node20 and node22 green, 3 with node20 green and node22 red, 1 with both red — a flake signature, not a deterministic break, on a file main has already tried to de-flake twice.
Approving.
|
Merged for 0.3.23. Verified on the integrated branch, not on the PR in isolation: every suite in the Thank you. |
fix(router): stop a per-checkpoint pin from silently disabling the caller's digest and revision
laya/router.py+154/-16 ·tests/test_revision_pinning.py+376/-1 ·tests/test_hooks_api.py+14Rebased onto
upstream/main@6d942c9. Both bugs are still present there — verified below.Two bugs, both in
Router.loadBug 1 — a digest pin can turn verification OFF.
laya/router.py:455onmain:That replaces whatever the caller passed in
agent_kwargs["expected_sha256"]. So a Routerconfigured like this verifies nothing for
multilingual:…the moment anything puts an entry for
multilingualintosha256_digests— including the{}placeholder
_digests_from_envwrites for every model a nestedLAYA_SHA256_DIGESTSdoes not name.{}means "verify nothing". A digest is a security control, and this is it silently switching itselfoff because an unrelated checkpoint was pinned through the environment.
Bug 2 — a blank revision is a pin.
laya/router.py:447onmain:.get(key, default)only falls back when the key is absent. A key present with""orNone—what a config line that never got filled in leaves behind — wins over the router-wide
revision, and""is not a commit anything can be pinned to.The fix
Three small helpers, each with a reason:
_revision_pin— unset and blank are the same answer, because both are an unfilledconfiguration line and neither is a commit.
_digest_entry— one checkpoint's entry assha256_digestsholds it right now. Read at loadtime rather than decided at construction, because
sha256_digestsis a public mutable attribute: apin assigned to it after
Router(...), or added to an existing entry in place, has to count. Decidingfrom a key set frozen at construction dropped exactly that.
_merge_expected_digests— the two channels merged file by file, because each names files anda digest is a claim about one file, so dropping either half verifies less than the caller asked for.
Precedence, and the one case where a weaker source wins
expected_sha256passed —verify_digestskeeps its own env fallback, so an unconfigured load stays byte-for-byte what it was (#332's opt-in guarantee)agent_kwargs{}orNone{}still reachesAgent, deliberatelyThe same-file rule is not arbitrary:
sharedreaches every checkpoint, andmodel.safetensorsis theone name every checkpoint uses for a different file, so a shared entry for it cannot be a correct
claim about all of them at once. Refusing the overlap as a contradiction broke the ordinary shape it
appears in — a shared pin plus a per-checkpoint override — which loaded correctly before
_merge_expected_digestsexisted.The
{}case is the subtle one and it must not be "simplified" away: an empty map masksverify_digests' environment fallback, and for a nested variable that fallback would otherwise beread as an artifact map and raise
cannot verify 'english': no such fileon a checkpoint the variablenever named.
A value that is not a mapping is handed on untouched, so
verify_digestskeeps ownership of the"expected_sha256 must be a mapping" message instead of this layer growing a second copy.
Reproduction — real checkpoint, each channel that documents itself as pinning
The real 322M multilingual checkpoint, a digest that cannot match, delivered three ways. Each case
runs in its own process, because
ru_maxrssis a peak over the whole process: once any case has pagedin the checkpoint, every later case reports that same peak and the number stops meaning anything.
Channel A is the one
mainlets through. The RSS figures are the difference between "refused" and"refused before reading the weights" — 0.19–0.20 GB is the load never reaching the 614 MB
model.safetensors, and it reproduces on every run because the refusal precedes any weight read.Two controls, without which this would prove nothing:
E is new in this revision, and it matters. An earlier revision of this description listed a
correct-digest control as evidence; the harness documented it and never ran it, so "3 of 3 refused"
rested on nothing ruling out "refuses everything". It runs now — the real SHA-256 of
model.safetensors, computed from the file, passed throughagent_kwargs— and the load completes andanswers. The two controls' peak RSS is reported without a figure on purpose: it has ranged 1.6–2.2 GB
across runs, because it depends on when the allocator releases, so it is not a measurement.
Verification
Mutation table: 13 of 14 killed. Both bugs, both directions of the precedence table, and every
guard: restoring the replace-the-caller's-map behaviour, reversing which side wins for a shared file,
passing
{}when nothing is configured, dropping the{}placeholder,_digest_entryturning amissing key into
{}or an explicitNoneintoNone, reading a key set frozen at construction,moving the non-mapping rejection out of
verify_digests, restoring.get(key, default), and_revision_pintreating a blank or whitespace string as a real pin.The one survivor is equivalent, not unpinned:
dict(v)in_digests_from_envis a defensive copyof a value already checked to be a
dict, and the JSON it would alias is local and discarded. Checkedrather than asserted — across 7 environment shapes, including deliberately mutating the returned maps
and re-reading, the copy and the alias produce identical results.
A pre-existing fragility, noted rather than fixed
tests/test_revision_pinning.pyinheritsLAYA_SHA256_DIGESTSfrom the environment it runs in. With anested variable exported — the shape this feature exists to support —
_digests_from_envseeds a{}placeholder for every model, soexpected_sha256={}reachesAgentand any check asserting thatno map is passed fails with no code change:
The five are upstream's own and are left alone — they are not what this PR is about, and touching them
would put unrelated churn in the diff. Three of the checks added here had the same flaw and are fixed,
so this branch adds none. Worth a separate one-line
conftest.pyif you want it.