Repository navigation
feat(router): verify each checkpoint with its own SHA-256 map - #572
Conversation
`Router` forwards `revision` per checkpoint but has no way to reach the
other integrity option every loader takes, `Agent(expected_sha256=...)`.
The only lever a multi-checkpoint deployment has is `LAYA_SHA256_DIGESTS`,
which is one flat map, and the bundled repository ships a separate
`model.safetensors`, `rl_agent_config.json` and tokenizer under each
subfolder, so a map that matches `english` mismatches every sibling.
`sha256_digests={"english": {...}}` gives each checkpoint its own map,
keyed and normalised exactly like `revisions` so a misspelled model name
fails at construction instead of quietly leaving that checkpoint
unverified. Listing a model with `None` loads it unverified and masks the
environment default for that checkpoint only.
|
Thanks for the PR. Per-checkpoint SHA-256 verification is a solid security addition, and the A few points:
Otherwise the design is clean and the tests cover the key normalization behavior. |
A process configured only by environment could not pin a set of resident
checkpoints: the bundled repository ships its own `model.safetensors` per
checkpoint, so one flat {artifact: digest} map matches one of them and
refuses the rest at startup. Accept the model-keyed shape that
`Router(sha256_digests=...)` already takes, leave the flat map to
`laya.revisions` unchanged, and let an argument entry win for the model it
names.
#568 and #572 added agent_kwargs and sha256_digests in the middle of the signature, ahead of max_loaded, so Router(models, device, token, revision, revisions, 2) would have bound 2 to agent_kwargs. Both are new, so they go last instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks @aashish254, merged. The new parameter moved to the end of |
NandhaKishorM#347 shipped checkpoint-integrity controls and nothing documented them: no hit for `expected_sha256`, `PINNED_REVISIONS` or `LAYA_SHA256_DIGESTS` in README.md or docs/. That matters more than a usual docs gap because NandhaKishorM#332 kept pinning opt-in so older offline snapshots keep loading, and because not using it is not an error -- a load with no revision and no digest map succeeds silently, reports `agent.revision is None`, and hashes nothing. The only thing that ever switches an inert control on is knowing it exists. docs/security.md, under Deployment: pinning a revision, verifying digests, generating the map, the environment variable, what a mismatch looks like in the server, reading back what loaded, the laya-ts equivalents, and the limits. Rebased onto main, and the page now describes what landed while this sat in draft: - `LAYA_REVISION` (NandhaKishorM#605), including `reviewed`, which resolves each repository's own SHA and raises for one the table has no entry for rather than loading it unpinned. This replaced the section saying revision pinning was unreachable from the server. - `Router(sha256_digests=...)` and the model-keyed `LAYA_SHA256_DIGESTS` (NandhaKishorM#572). The page now documents both shapes of the variable -- flat `{artifact: digest}` for every checkpoint, nested `{model: {artifact: digest}}` per checkpoint -- instead of the one-map-per-process limitation that shape removed, and says a checkpoint the nested map does not name is deliberately unpinned while an unknown model name raises. - `tests/test_env_docs.py`'s DEFERRED row for `LAYA_SHA256_DIGESTS` is retired, since this page is what documented it. What executing the examples still turns up, and the page says rather than tidies: - `PINNED_REVISIONS` is keyed by the three standalone repositories while a default `Router` loads all three from the one bundle repository, so a per-model pin needs `standalone_repos=True`; without it the SHA goes to a repository that does not contain it and the load fails. The example pins all three, because a `Router` offers every checkpoint it knows about regardless of what is preloaded. - A digest mismatch in the server is a **422**, not a 500: `laya/serve.py` maps the `ValueError` and returns the digest text to the caller. A listed but absent file raises `FileNotFoundError` and becomes a generic 500 with the reason only in the log. The page says to plan for the 422, because it sorts as a client error and so will not appear where an operator looks for a broken deployment. - Only listed files are checked, and `tokenizer/tokenizer_config.json` is parsed but left out of the recommended map because Laya may normalise and rewrite it just after the digest check -- conditionally, so the page says it depends on the checkpoint rather than stating it as universal. `compose.http.yaml` passes `LAYA_SHA256_DIGESTS` through, defaulting to empty. The row added to docs/docker.md would otherwise be false for the deployment that table describes: the variable reached neither the container nor docker/entrypoint.py, so an operator setting it and running compose got no verification at all, silently. The page carries no literal digest and no literal commit SHA outside an elided illustration. Digests change with a checkpoint's files, so it ships a generator and says to run that -- including rather than copying from the page. There is deliberately no command that produces the map: one computed from the copy Laya just downloaded would hash those bytes and verify them against themselves. A "Do you need both?" section says what each control trusts -- a revision asks the Hub for a commit and believes the answer, a digest is a record the operator keeps and is the only check that does not ask the thing under test to vouch for itself. `PINNED_REVISIONS` is exported from the package root for parity with laya-ts's index.ts, through the `_LAZY_ATTRS` table rather than an eager import, so `import laya` pulls in no new module and torch stays absent. Because that is a public API change, tests/test_hooks_api.py gains the export: `__all__` membership plus a `hasattr` check, which is what proves the lazy `__getattr__` resolves it. Removing the export fails exactly those two. tests/test_lazy_import.py asserts the pins resolve with torch blocked, and tests/test_revision_pinning.py adds PublicApiTests for the root export, identity with laya.revisions, and that each pin is a usable revision string -- not a 40-hex shape, which would invent a policy the library does not state.
NandhaKishorM#347 shipped checkpoint-integrity controls and nothing documented them: no hit for `expected_sha256`, `PINNED_REVISIONS` or `LAYA_SHA256_DIGESTS` in README.md or docs/. That matters more than a usual docs gap because NandhaKishorM#332 kept pinning opt-in so older offline snapshots keep loading, and because not using it is not an error -- a load with no revision and no digest map succeeds silently, reports `agent.revision is None`, and hashes nothing. The only thing that ever switches an inert control on is knowing it exists. docs/security.md, under Deployment: pinning a revision, verifying digests, generating the map, the environment variable, what a mismatch looks like in the server, reading back what loaded, the laya-ts equivalents, and the limits. Rebased onto main, and the page now describes what landed while this sat in draft: - `LAYA_REVISION` (NandhaKishorM#605), including `reviewed`, which resolves each repository's own SHA and raises for one the table has no entry for rather than loading it unpinned. This replaced the section saying revision pinning was unreachable from the server. - `Router(sha256_digests=...)` and the model-keyed `LAYA_SHA256_DIGESTS` (NandhaKishorM#572). The page now documents both shapes of the variable -- flat `{artifact: digest}` for every checkpoint, nested `{model: {artifact: digest}}` per checkpoint -- instead of the one-map-per-process limitation that shape removed, and says a checkpoint the nested map does not name is deliberately unpinned while an unknown model name raises. - `tests/test_env_docs.py`'s DEFERRED row for `LAYA_SHA256_DIGESTS` is retired, since this page is what documented it. What executing the examples still turns up, and the page says rather than tidies: - `PINNED_REVISIONS` is keyed by the three standalone repositories while a default `Router` loads all three from the one bundle repository, so a per-model pin needs `standalone_repos=True`; without it the SHA goes to a repository that does not contain it and the load fails. The example pins all three, because a `Router` offers every checkpoint it knows about regardless of what is preloaded. - A digest mismatch in the server is a **422**, not a 500: `laya/serve.py` maps the `ValueError` and returns the digest text to the caller. A listed but absent file raises `FileNotFoundError` and becomes a generic 500 with the reason only in the log. The page says to plan for the 422, because it sorts as a client error and so will not appear where an operator looks for a broken deployment. - Only listed files are checked, and `tokenizer/tokenizer_config.json` is parsed but left out of the recommended map because Laya may normalise and rewrite it just after the digest check -- conditionally, so the page says it depends on the checkpoint rather than stating it as universal. `compose.http.yaml` passes `LAYA_SHA256_DIGESTS` through, defaulting to empty. The row added to docs/docker.md would otherwise be false for the deployment that table describes: the variable reached neither the container nor docker/entrypoint.py, so an operator setting it and running compose got no verification at all, silently. The page carries no literal digest and no literal commit SHA outside an elided illustration. Digests change with a checkpoint's files, so it ships a generator and says to run that -- including rather than copying from the page. There is deliberately no command that produces the map: one computed from the copy Laya just downloaded would hash those bytes and verify them against themselves. A "Do you need both?" section says what each control trusts -- a revision asks the Hub for a commit and believes the answer, a digest is a record the operator keeps and is the only check that does not ask the thing under test to vouch for itself. `PINNED_REVISIONS` is exported from the package root for parity with laya-ts's index.ts, through the `_LAZY_ATTRS` table rather than an eager import, so `import laya` pulls in no new module and torch stays absent. Because that is a public API change, tests/test_hooks_api.py gains the export: `__all__` membership plus a `hasattr` check, which is what proves the lazy `__getattr__` resolves it. Removing the export fails exactly those two. tests/test_lazy_import.py asserts the pins resolve with torch blocked, and tests/test_revision_pinning.py adds PublicApiTests for the root export, identity with laya.revisions, and that each pin is a usable revision string -- not a 40-hex shape, which would invent a policy the library does not state.
The gap
laya.Agentandlaya.ONNXAgentboth takeexpected_sha256, andRouterforwards the othersupply-chain option they share —
revision, plusrevisionsper checkpoint — but there is noRouter-level entry point for digests. That leaves
LAYA_SHA256_DIGESTSas the only lever amulti-checkpoint deployment has, and it is one flat map.
The default
Routerloads all three checkpoints out of one bundle repository, and each of themships its own
model.safetensors,rl_agent_config.jsonand tokenizer at the same relativepath. So one map can match at most one checkpoint, and matching means the siblings refuse to
load. Measured on the real cached weights, with the digest for
englishset correctly:LAYA_SHA256_DIGESTS={"model.safetensors": <english digest>},Router(preload=["english","multilingual"])SHA-256 mismatchRouter(sha256_digests=...)onmainTypeError: unexpected keyword argumentThe three digests really are three different values behind that one filename:
891102d37268…(english),9d628fd971b7…(multilingual),4fa56de72383…(typed-decisions).#528 documents this collision and concludes that a deployment that serves more than one
checkpoint should "otherwise pass
expected_sha256per load in code, where each checkpoint getsits own map". This PR adds the entry point that recommendation needs: on a
Routerthere is noper-load call site to pass it to, because the Router builds the agents.
The change
One argument, keyed and normalised exactly like
revisions:normalise_name, so aliases work and a misspelled model raises at construction.That matters more here than it does for
revisions: a typo in a revision pin leaves acheckpoint unpinned, while a typo in a digest map silently leaves it unverified, which is the
control that was supposed to be on.
expected_sha256shorthand. One map for three checkpointsis the thing that cannot be right, and a global alias would invite exactly the config this
measures as broken.
None(or{}) loads unverified and forwards{}, not an absent argument,so it also masks
LAYA_SHA256_DIGESTSfor that one checkpoint — the way to serve a mirrored orself-trained checkpoint next to two reviewed ones.
Measured after
Same script, same machine, real checkpoints,
LAYA_DEVICE=cpu, offline cache.preload=True['english', 'multilingual', 'typed-decisions']loadedmultilingualentry replaced with0*64SHA-256 mismatch for model.safetensorsenglishlistedplay_music/play_music, field-for-field identicalanswersfor an English and a German stateLAYA_SHA256_DIGESTSon this branch still refuses the second checkpoint — the flat map is stillflat, and this PR does not pretend otherwise. It adds the per-checkpoint path. Reaching it from
laya-serveneeds the server to build its Router from config, which #528 already flags as aseparate limitation ("
revisionandrevisionsare not reachable from the server"); the same istrue of
sha256_digests, and fixing it there is a bigger change in a file several PRs are in.Tests
tests/test_revision_pinning.py: 20 → 30, no weights and no network. Five local checkpointdirectories built in
setUp, wheremodel.safetensorsexists butrl_agent_config.jsondoesnot: a matching digest gets past the digest gate to the config check, a mismatching one never
leaves it, so "reached the config error" is the witness that verification passed rather than an
absence of failure.
One of those tests is the regression itself, weight-free: the same flat map for two checkpoints
that share a filename →
SHA-256 mismatchon the second; per-checkpoint maps → both pass thegate.
Mutation checks against the suite (each mutation reverted afterwards):
or {}dropped (soNonereachesverify_digestsasNone)normalise_nameif key in self.sha256_digestsguard droppedtests/test_hooks_api.py131 → 132 (the signature check next torevisions).Gates
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120— all checks passedpython -m compileall -q laya/ tests/— cleantests/test_hooks_api.py132 passed ·tests/test_revision_pinning.py30 OK ·tests/test_router.py546 passed ·tests/test_mcp.py123 passedzensical build --strict --clean— no issues; the Router reference page renders the new paragraphOverlap
laya/__init__.py,compose.http.yaml). Its "passexpected_sha256per load in code" sentence is the one this PRmakes possible for a
Router, so a follow-up sentence there may be worth having — left a noteon that PR rather than restating it here.
Router.__init__argument and atest_hooks_api.pycheck, so the twowill need a one-line rebase whichever lands second. They are different mechanisms:
agent_kwargsis one map applied to every checkpoint, which for digests is exactly the broken case above.
Second commit (
9655978):LAYA_SHA256_DIGESTScan name each checkpointThe same split, reachable without writing Python.
The gap
Commit 1 gave
Routera per-checkpoint digest map, but a process configured only byenvironment still had one flat
LAYA_SHA256_DIGESTS, whichlaya.revisions.verify_digestsreads as
{artifact: digest}— the same files checked against every checkpoint the processloads. Since
convaiinnovations/layaships a separatemodel.safetensorsper checkpoint,that shape can match at most one of them. Measured on
main, real cached weights, loadingenglishthenmultilingualfrom oneRouter— the same pairLAYA_MODELS=english,multilingualpreloads in
laya-serve:LAYA_SHA256_DIGESTSenglishmultilingualSHA-256 mismatch for model.safetensors: expected 891102d3…, got 9d628fd9…{english: {…}, multilingual: {…}}cannot verify 'english': no such file …So the caller who sets the variable for supply-chain verification gets the opposite: the
first checkpoint loads and the rest are refused as mismatched. And the value with the shape
you would reach for does not parse either — it is read as an artifact map, so
englishislooked for as a filename. There is no value of this variable that verifies a
Routerholdingtwo checkpoints, which is the configuration the docs point at for language routing.
The change
_digests_from_env()inlaya/router.py, ~30 lines; the value types decide the reading:{}solaya.revisionskeeps applyingit exactly as before. Row 1 above is unchanged behaviour, not something this commit fixes —
it is the flat shape's limitation, and it stays.
name becomes
{}: deliberately unpinned, rather than falling back to the flat environmentread that produced
cannot verify 'english'above.meant would silently drop verification for the other half.
Router(sha256_digests=...)overrides per checkpoint, so one model can be pinned in code andthe rest left to the environment. Environment keys go through
normalise_name, soennamesenglishand a name core does not know fails at construction instead of quietly leaving thatcheckpoint unverified.
laya/revisions.py,laya/serve.pyand the flat shape are untouched.After, same probe and machine
LAYA_SHA256_DIGESTSenglishmultilingualSHA-256 mismatch … expected 000000…, got 9d628fd9…Router(sha256_digests={"english": correct})Rows 3 and 4 are the point: one bad digest refuses exactly one checkpoint instead of the
process, and an explicit argument still wins for the model it names.
Tests
10 new weight-free tests in
tests/test_revision_pinning.py(30 → 40): the kwarg-levelclaims go through a patched
Agent, the end-to-end ones through commit 1's five fakecheckpoint dirs, where reaching the config error is the witness that the digest gate passed.
Eight of the ten fail at
4b4dbddby name — six assertion failures, and twoKeyErrors fromreading
router.sha256_digests, where the nested map simply produced no entry — and passhere:
test_nested_environment_pins_both_checkpoints·test_nested_environment_fails_only_the_checkpoint_it_calls_out·test_nested_environment_leaves_an_unnamed_checkpoint_unpinned·test_nested_map_is_split_per_checkpoint·test_nested_map_keys_are_normalised·test_argument_wins_for_the_checkpoint_it_names·test_misspelled_key_in_the_environment_fails_at_construction·test_mixed_artifact_and_model_keys_fail_rather_than_guessThe other two (
test_flat_map_is_left_to_revisions,test_unparseable_environment_is_not_this_layers_error) pass before and after on purpose:they are the no-change guards for the flat shape and for who owns a parse error.
Mutation check against the subject, 7 mutants of
laya/router.py, each reverted and the fileverified byte-identical afterwards. All 7 caught by a named test:
test_flat_map_is_left_to_revisionstest_argument_wins_for_the_checkpoint_it_namestest_mixed_artifact_and_model_keys_fail_rather_than_guesstest_unparseable_environment_is_not_this_layers_errorGates
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120— all checks passedpython -m compileall -q laya/ tests/— clean ·git diff --check— cleantests/test_revision_pinning.py40 OK ·tests/test_router.py546 passed ·tests/test_hooks_api.py132 passed, unchanged from commit 1 (no signature moves, so theAPI contract suite has nothing new to assert)
Limits, and what this deliberately does not do
laya.revisions' ownrather than a second opinion on the same string.
ONNXAgentstill reads only the flat shape: it is loaded outside theRouterpath thisseeds, so the model-keyed map would need its own call site to say which model it is. Left
alone rather than half-wired.
have two PRs editing the same table. Whoever lands first leaves the nested shape to add as
a follow-up on the other side.
Overlap
The change is one module-level helper plus the seeding line inside
Router.__init__, i.e.inside commit 1's own hunk, so it conflicts with nothing that does not already conflict with
commit 1.
tests/test_revision_pinning.pyis shared with #528 and #568; both add their ownclasses, and my additions are methods in the class commit 1 introduced plus one new class, so
no line collides with either.