Skip to content

fix(router): stop a per-checkpoint pin from silently disabling the ca… - #690

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
keemsisi:fix/router-pinning-controls
Oct 1, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
keemsisi:fix/router-pinning-controls

Conversation

@keemsisi

@keemsisi keemsisi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

fix(router): stop a per-checkpoint pin from silently disabling the caller's digest and revision

laya/router.py +154/-16 · tests/test_revision_pinning.py +376/-1 ·
tests/test_hooks_api.py +14
Rebased onto upstream/main @ 6d942c9. Both bugs are still present there — verified below.

Two bugs, both in Router.load

Bug 1 — a digest pin can turn verification OFF. laya/router.py:455 on main:

kwargs["expected_sha256"] = self.sha256_digests[key] or {}

That replaces whatever the caller passed in agent_kwargs["expected_sha256"]. So a Router
configured like this verifies nothing for multilingual:

Router(
    models={"multilingual": ..., "english": ...},
    agent_kwargs={"expected_sha256": {"model.safetensors": "<the digest I want enforced>"}},
)

…the moment anything puts an entry for multilingual into sha256_digests — including the {}
placeholder _digests_from_env writes for every model a nested LAYA_SHA256_DIGESTS does not name.

{} means "verify nothing". A digest is a security control, and this is it silently switching itself
off because an unrelated checkpoint was pinned through the environment.

Bug 2 — a blank revision is a pin. laya/router.py:447 on main:

model_revision = self.revisions.get(key, self.revision)

.get(key, default) only falls back when the key is absent. A key present with "" or None —
what a config line that never got filled in leaves behind — wins over the router-wide revision, and
"" is not a commit anything can be pinned to.

The fix

model_revision = _revision_pin(self.revisions.get(key)) or _revision_pin(self.revision)
expected = _merge_expected_digests(kwargs.pop("expected_sha256", None),
                                   _digest_entry(self.sha256_digests, key))
if expected is not None:
    kwargs["expected_sha256"] = expected

Three small helpers, each with a reason:

  • _revision_pin — unset and blank are the same answer, because both are an unfilled
    configuration line and neither is a commit.
  • _digest_entry — one checkpoint's entry as sha256_digests holds it right now. Read at load
    time rather than decided at construction, because sha256_digests is a public mutable attribute: a
    pin assigned to it after Router(...), or added to an existing entry in place, has to count. Deciding
    from a key set frozen at construction dropped exactly that.
  • _merge_expected_digests — the two channels merged file by file, because each names files and
    a digest is a claim about one file, so dropping either half verifies less than the caller asked for.

Precedence, and the one case where a weaker source wins

situation result
neither channel set no expected_sha256 passed — verify_digests keeps its own env fallback, so an unconfigured load stays byte-for-byte what it was (#332's opt-in guarantee)
only agent_kwargs the caller's map, unchanged
only the per-checkpoint entry that entry
both name different files the union — both are enforced
both name the same file the per-checkpoint entry wins
entry is {} or None {} still reaches Agent, deliberately

The same-file rule is not arbitrary: shared reaches every checkpoint, and model.safetensors is the
one name every checkpoint uses for a different file, so a shared entry for it cannot be a correct
claim about all of them at once. Refusing the overlap as a contradiction broke the ordinary shape it
appears in — a shared pin plus a per-checkpoint override — which loaded correctly before
_merge_expected_digests existed.

The {} case is the subtle one and it must not be "simplified" away: an empty map masks
verify_digests' environment fallback, and for a nested variable that fallback would otherwise be
read as an artifact map and raise cannot verify 'english': no such file on a checkpoint the variable
never named.

A value that is not a mapping is handed on untouched, so verify_digests keeps ownership of the
"expected_sha256 must be a mapping" message instead of this layer growing a second copy.

Reproduction — real checkpoint, each channel that documents itself as pinning

The real 322M multilingual checkpoint, a digest that cannot match, delivered three ways. Each case
runs in its own process, because ru_maxrss is a peak over the whole process: once any case has paged
in the checkpoint, every later case reports that same peak and the number stops meaning anything.

A  agent_kwargs + env names a different model   REFUSED   peak RSS 0.20 GB
B  Router(sha256_digests=...)                   REFUSED   peak RSS 0.19 GB
C  LAYA_SHA256_DIGESTS names this model         REFUSED   peak RSS 0.20 GB

3 of 3 wrong-digest channels refused

Channel A is the one main lets through. The RSS figures are the difference between "refused" and
"refused before reading the weights" — 0.19–0.20 GB is the load never reaching the 614 MB
model.safetensors, and it reproduces on every run because the refusal precedes any weight read.

Two controls, without which this would prove nothing:

D  unconfigured Router (#332 opt-in guarantee)  LOADED    answers normally
E  the CORRECT digest, through agent_kwargs     LOADED    answers normally

E is new in this revision, and it matters. An earlier revision of this description listed a
correct-digest control as evidence; the harness documented it and never ran it, so "3 of 3 refused"
rested on nothing ruling out "refuses everything". It runs now — the real SHA-256 of
model.safetensors, computed from the file, passed through agent_kwargs — and the load completes and
answers. The two controls' peak RSS is reported without a figure on purpose: it has ranged 1.6–2.2 GB
across runs, because it depends on when the allocator releases, so it is not a measurement.

Verification

tests/test_revision_pinning.py      80 tests, OK        (52 on main — this adds 28)
tests/test_hooks_api.py            394 passed           (393 on main)
the ci.yml pytest lane             208 passed, 2 skipped (unchanged from main)
script-style suites from ci.yml      67 ok, 0 failed
ruff + compileall                    clean

Mutation table: 13 of 14 killed. Both bugs, both directions of the precedence table, and every
guard: restoring the replace-the-caller's-map behaviour, reversing which side wins for a shared file,
passing {} when nothing is configured, dropping the {} placeholder, _digest_entry turning a
missing key into {} or an explicit None into None, reading a key set frozen at construction,
moving the non-mapping rejection out of verify_digests, restoring .get(key, default), and
_revision_pin treating a blank or whitespace string as a real pin.

The one survivor is equivalent, not unpinned: dict(v) in _digests_from_env is a defensive copy
of a value already checked to be a dict, and the JSON it would alias is local and discarded. Checked
rather than asserted — across 7 environment shapes, including deliberately mutating the returned maps
and re-reading, the copy and the alias produce identical results.

A pre-existing fragility, noted rather than fixed

tests/test_revision_pinning.py inherits LAYA_SHA256_DIGESTS from the environment it runs in. With a
nested variable exported — the shape this feature exists to support — _digests_from_env seeds a
{} placeholder for every model, so expected_sha256={} reaches Agent and any check asserting that
no map is passed fails with no code change:

upstream/main, nested LAYA_SHA256_DIGESTS exported:   4 failures, 1 error of 52
this branch,   same environment:                      4 failures, 1 error of 80   (the same five)

The five are upstream's own and are left alone — they are not what this PR is about, and touching them
would put unrelated churn in the diff. Three of the checks added here had the same flaw and are fixed,
so this branch adds none. Worth a separate one-line conftest.py if you want it.

…ller's digest and revision

`Router.load` dropped two of the integrity controls it was meant to forward.

`agent_kwargs["expected_sha256"]` is a documented option and deliberately not router-owned, but
any `sha256_digests` entry for the checkpoint being loaded replaced it outright -- including the
`{}` placeholder `_digests_from_env` leaves for every checkpoint a nested `LAYA_SHA256_DIGESTS`
does not name. Pinning one checkpoint through the environment therefore turned verification off
for the others, and `{}` means "verify nothing" because `verify_digests` falls back to the
environment only when the argument is None. The two maps are now merged file by file, and a
placeholder never cancels a pin.

For a file both channels name, the per-checkpoint entry wins. They are not equally specific:
`agent_kwargs` reaches every checkpoint the Router builds, and `model.safetensors` is the one name
every checkpoint uses for a *different* file, so a shared entry for it cannot be a correct claim
about all of them at once. Refusing the overlap as a contradiction -- which an earlier draft of this
fix did -- turned a configuration that worked before the fix into a startup ValueError: verified
against upstream, a shared `agent_kwargs` pin plus `sha256_digests={"multilingual": {...}}` loaded
english with the shared digest and multilingual with its own. Through LAYA_SHA256_DIGESTS it was
worse, because `load("english")` succeeded while `load("multilingual")` raised and `laya/serve.py`
turns that into an HTTP 422 on every request routed there. Nothing raises now.

The distinction that is load-bearing is None versus `{}`, not who wrote the entry. An absent entry
passes no kwarg, leaving `verify_digests` its own fallback; an empty one passes `{}`, which masks it
-- necessary in both directions, because a flat variable is applied by `verify_digests` for every
checkpoint, and a nested one, if left to it, is read as an artifact map and raises `cannot verify
'english': no such file` on a checkpoint it never named. Both halves are tested; I asserted the
opposite of the second first and a run corrected me.

A `None` or blank entry in `revisions` was read as a pin of nothing: it dropped the Router-wide
`revision` and let `$LAYA_REVISION` decide, so a caller's explicit pin lost to the environment on
exactly the checkpoint it named -- while docs/docker.md promises the opposite. Such an entry now
means "no override for this model" and inherits `revision`. An unconfigured Router still passes
no revision and no digest kwarg, so NandhaKishorM#332's opt-in guarantee is unchanged.

Verified against the real multilingual checkpoint, offline: a digest that cannot match was
accepted in two of three configurations before and is refused in all three now (peak RSS 2.14 GB
-> 0.23 GB, the load no longer reaching the weights), a correct one still loads and answers, and
an unconfigured Router still loads an HF_HUB_OFFLINE=1 cache. 24 checks added to
tests/test_revision_pinning.py (50 -> 74), 14 of which go red with laya/router.py reverted; no
existing check needed changing.

End to end on the real 322M multilingual checkpoint, one subprocess per case so peak RSS means what
it says: a digest that cannot match, delivered through each of the three channels that document
themselves as pinning, is refused by all three (0.19-0.20 GB peak, the load never reaching the
weights) where upstream accepted it through one of them at 2.74 GB. An unconfigured Router still
loads and answers, which is the control that stops "refuses everything" looking like a fix.

Both distinctions are read from `sha256_digests` at load time rather than decided when the Router is
built, because the attribute is public and mutable: a per-checkpoint pin assigned after `Router(...)`,
or added to an existing entry in place, has to count, and deciding from a key set frozen at
construction dropped exactly that.
… untested guards

From an adversarial review, which called the fix itself "correct and genuinely well tested" -- 120
configuration combinations and 27 mutants, 18 killed, and the diff strictly increases what gets
verified relative to 9d95567. Two docstring claims were wrong, and three guards had no test.

THE CLAIMS. The class docstring said an `agent_kwargs["expected_sha256"]` map "is never discarded
wholesale: where a checkpoint also has digests of its own, from `sha256_digests` or from the
environment, the two maps are merged file by file". That is true of a per-checkpoint
`LAYA_SHA256_DIGESTS`, and false of a FLAT one: `verify_digests` reads a flat variable only when
nothing else pins (`if expected is None`), so any `expected_sha256` reaching `Agent` masks it
entirely. Verified on real files -- a flat variable pinning `model.safetensors` plus an
`agent_kwargs` map pinning `tokenizer.json` loads a tampered `model.safetensors`.

I tried to fix the behaviour first, by merging the flat shape in as a lowest-precedence layer. Four
existing checks went red, and they were right: `test_flat_map_is_left_to_revisions`,
`test_unparseable_environment_is_not_this_layers_error`, and two pinning that an explicit `None`
entry MASKS a flat environment map -- which is what "load this one unverified" has to mean. The
behaviour is deliberate and tested; the docstring was what overclaimed. It now names both exceptions
and says the flat case is unchanged from `main`.

Second claim: the blank-`revision` case was called "the one place a weaker source ends up ahead of
an explicit argument". It is one of two. A per-checkpoint entry synthesised from
`LAYA_SHA256_DIGESTS` also beats an `expected_sha256` passed in code -- which
`test_an_environment_pin_overrides_the_shared_one_per_checkpoint` enshrines as intended, so it is now
stated plainly rather than contradicted. An environment variable beating an explicit argument
deserves a line of its own on a supply-chain control.

THREE MUTANTS THAT SURVIVED EVERY LANE, now killed:
- dropping the `per_model` non-dict guard: a list of pairs would be `dict.update`-ed in and silently
  accepted as a digest map. Both shapes still fail closed, but only one fails in `verify_digests`'
  own words, which is what the guard is for.
- dropping the `shared` non-dict guard: the same gap on the other half.
- `resolve_revision`'s blank-argument behaviour. `(revision or env).strip()` means a truthy-but-blank
  argument wins the `or` and then strips to empty, so `$LAYA_REVISION` does not apply. The Router
  drops a blank before it gets there and that is tested; `Agent` and `laya.load` do not, so a config
  line that ends up whitespace silently loses a deployment's pin. Undocumented and untested -- the
  mutant that makes a blank behave like an absent value survived every lane. Now pinned as the
  behaviour that exists, alongside the documented gap already in this PR's description.

A fourth check pins the flat-environment exception itself, so the docstring cannot drift from it
again.

Verified: tests/test_revision_pinning.py 78 tests, OK (was 74). test_hooks_api and test_download
green. All 60 ci.yml script suites pass. ruff and compileall clean. Real 322M checkpoint, wrong
digest through all three channels: 3 of 3 refused at 0.19-0.20 GB peak RSS, control loads. Mutation
sweep after the change: 5 of 5 killed, 0 survivors.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both bugs reproduce on current main and both are fixed here. I re-derived them from the source and then measured the resolution on both trees, weight-free, so neither claim rests on the description.

Bug 1 — a per-checkpoint entry silently discards the caller's pin

laya/router.py:451 merges the caller's map in, and :452-455 then overwrites it:

kwargs.update(self.agent_kwargs)          # :451  caller pins tokenizer.json
if key in self.sha256_digests:
    kwargs["expected_sha256"] = self.sha256_digests[key] or {}   # :455  replaced

A stubbed Agent recording what the Router hands it, same probe both trees:

configuration main this branch
caller pins tokenizer.json, per-checkpoint entry is {} {} — caller's pin gone {'tokenizer.json': …}
caller pins tokenizer.json, per-checkpoint pins model.safetensors {'model.safetensors': …} — tokenizer pin dropped both, merged
caller pins tokenizer.json, nested LAYA_SHA256_DIGESTS also names it {} {'tokenizer.json': …}

The or {} is the sharp edge. laya/revisions.py:89 reads the environment only if expected is None, so a non-None empty map does not just mean "verify nothing" — it also masks a LAYA_SHA256_DIGESTS that was set correctly. An unfilled per-checkpoint config line therefore switches off a digest control the operator asked for. Merging file-by-file is the right shape, because a digest is a claim about one file and dropping either half verifies less than was requested.

Bug 2 — a blank per-checkpoint revision discards the router-wide pin

laya/router.py:447 is self.revisions.get(key, self.revision), and .get falls back only on an absent key. :448 then omits the kwarg entirely when the value is None. Measured, with Router(revision="REVIEWED_AAA"):

configuration main reaches the Agent with this branch
revisions={"english": None} absent REVIEWED_AAA
revisions={"english": ""} "" REVIEWED_AAA
the documented revisions={"english": os.environ.get("EN_SHA")} with EN_SHA unset absent REVIEWED_AAA
control: revisions={"english": "MODEL_CCC"} MODEL_CCC MODEL_CCC

Absent is the outcome that matters: the checkpoint then resolves to the hub's default branch head, which is an unreviewed commit. So the documented pattern in the class's own docstring loses the pin whenever the environment variable is unset — the most likely state, not an edge case. Treating unset and blank as the same answer, because both are an unfilled configuration line, is the correct reading.

On the precedence table, since a wrong table is where this kind of change does damage

I checked the rows that could bite. Per-checkpoint winning over a shared entry for the same file is right, and the stated reason holds: shared reaches every checkpoint, and model.safetensors names a different file in each, so a shared entry for it cannot be a correct claim about all of them. Keeping {} reaching Agent rather than "simplifying" it away is also right, for the reason in finding 1 — it is what keeps a nested variable from being read as an artifact map and raising no such file on a checkpoint it never named. A non-mapping is passed through untouched, which keeps verify_digests owning that message.

One behaviour change beyond the two reported bugs, disclosed

Router(revision=" ") used to reach resolve_revision, where (revision or env).strip() meant a truthy-but-blank string suppressed LAYA_REVISION. Here it is dropped, so $LAYA_REVISION now applies. That is in the docstring and pinned by a test, and it is the safer direction — but it is a third behaviour change riding along with two, so I am naming it rather than letting it pass as part of the fix.

The red on this PR is not yours

TypeScript (node22) fails, and it is worth not spending review time on it. The failing step is Unit tests, and the single failure is laya-ts/tests/identifier-complexity.test.ts > "4x the input costs under 8x the time", a wall-clock ratio assertion that missed by 0.7% (expected 8.059531496767748 to be less than 8) on a shared runner. This PR changes Python files and no TypeScript. Across the open pool the pattern is 20 PRs with both node20 and node22 green, 3 with node20 green and node22 red, 1 with both red — a flake signature, not a deterministic break, on a file main has already tried to de-flake twice.

Approving.

@NandhaKishorM
NandhaKishorM merged commit b8208ac into NandhaKishorM:main Oct 1, 2026
23 of 25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23.

Verified on the integrated branch, not on the PR in isolation: every suite in the ci.yml test lane plus the pytest lane green, compileall clean, and a decision diff against main on the real english and multilingual checkpoints (72 answers over choice, score and noul, short and long states, budgeted and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants