Skip to content

feat(router): verify each checkpoint with its own SHA-256 map - #572

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:feat-router-digests
Sep 27, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:feat-router-digests

Conversation

@aashish254

@aashish254 aashish254 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

The gap

laya.Agent and laya.ONNXAgent both take expected_sha256, and Router forwards the other
supply-chain option they share — revision, plus revisions per checkpoint — but there is no
Router-level entry point for digests. That leaves LAYA_SHA256_DIGESTS as the only lever a
multi-checkpoint deployment has, and it is one flat map.

The default Router loads all three checkpoints out of one bundle repository, and each of them
ships its own model.safetensors, rl_agent_config.json and tokenizer at the same relative
path. So one map can match at most one checkpoint, and matching means the siblings refuse to
load. Measured on the real cached weights, with the digest for english set correctly:

configuration english multilingual typed-decisions
LAYA_SHA256_DIGESTS={"model.safetensors": <english digest>}, Router(preload=["english","multilingual"]) loaded refused: SHA-256 mismatch —
Router(sha256_digests=...) on main — — TypeError: unexpected keyword argument

The three digests really are three different values behind that one filename:
891102d37268… (english), 9d628fd971b7… (multilingual), 4fa56de72383… (typed-decisions).

#528 documents this collision and concludes that a deployment that serves more than one
checkpoint should "otherwise pass expected_sha256 per load in code, where each checkpoint gets
its own map". This PR adds the entry point that recommendation needs: on a Router there is no
per-load call site to pass it to, because the Router builds the agents.

The change

One argument, keyed and normalised exactly like revisions:

router = laya.Router(
    preload=True,
    sha256_digests={
        "english":         {"model.safetensors": "891102…", "rl_agent_config.json": "…"},
        "multilingual":    {"model.safetensors": "9d628f…", "rl_agent_config.json": "…"},
        "typed-decisions": {"model.safetensors": "4fa56d…", "rl_agent_config.json": "…"},
    },
)
  • Keys go through normalise_name, so aliases work and a misspelled model raises at construction.
    That matters more here than it does for revisions: a typo in a revision pin leaves a
    checkpoint unpinned, while a typo in a digest map silently leaves it unverified, which is the
    control that was supposed to be on.
  • There is deliberately no Router-wide expected_sha256 shorthand. One map for three checkpoints
    is the thing that cannot be right, and a global alias would invite exactly the config this
    measures as broken.
  • A model listed with None (or {}) loads unverified and forwards {}, not an absent argument,
    so it also masks LAYA_SHA256_DIGESTS for that one checkpoint — the way to serve a mirrored or
    self-trained checkpoint next to two reviewed ones.
  • Models you do not list are untouched: the environment default still applies to them.

Measured after

Same script, same machine, real checkpoints, LAYA_DEVICE=cpu, offline cache.

case result
all three correct digests, preload=True ['english', 'multilingual', 'typed-decisions'] loaded
multilingual entry replaced with 0*64 english loaded; multilingual refused: SHA-256 mismatch for model.safetensors
only english listed both load; multilingual never checked (unchanged behaviour)
decisions, verified vs unverified Router play_music / play_music, field-for-field identical answers for an English and a German state
cost, preload english+multilingual 0.4 s unverified → 1.1 s verified (2.3 GB of three checkpoints hashes in 0.8 s, warm cache)

LAYA_SHA256_DIGESTS on this branch still refuses the second checkpoint — the flat map is still
flat, and this PR does not pretend otherwise. It adds the per-checkpoint path. Reaching it from
laya-serve needs the server to build its Router from config, which #528 already flags as a
separate limitation ("revision and revisions are not reachable from the server"); the same is
true of sha256_digests, and fixing it there is a bigger change in a file several PRs are in.

Tests

tests/test_revision_pinning.py: 20 → 30, no weights and no network. Five local checkpoint
directories built in setUp, where model.safetensors exists but rl_agent_config.json does
not: a matching digest gets past the digest gate to the config check, a mismatching one never
leaves it, so "reached the config error" is the witness that verification passed rather than an
absence of failure.

One of those tests is the regression itself, weight-free: the same flat map for two checkpoints
that share a filename → SHA-256 mismatch on the second; per-checkpoint maps → both pass the
gate.

Mutation checks against the suite (each mutation reverted afterwards):

mutation caught by
forwarding removed entirely 4 tests
or {} dropped (so None reaches verify_digests as None) 2 tests
keys stored without normalise_name 2 tests
whole map forwarded instead of the per-checkpoint entry 6 tests
if key in self.sha256_digests guard dropped 2 new + 2 existing revision tests

tests/test_hooks_api.py 131 → 132 (the signature check next to revisions).

Gates

  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 — all checks passed
  • python -m compileall -q laya/ tests/ — clean
  • tests/test_hooks_api.py 132 passed · tests/test_revision_pinning.py 30 OK ·
    tests/test_router.py 546 passed · tests/test_mcp.py 123 passed
  • zensical build --strict --clean — no issues; the Router reference page renders the new paragraph

Overlap

  • docs(security): document opt-in revision pinning and digest verification #528 documents the collision this changes; no shared lines (it is docs, laya/__init__.py,
    compose.http.yaml). Its "pass expected_sha256 per load in code" sentence is the one this PR
    makes possible for a Router, so a follow-up sentence there may be worth having — left a note
    on that PR rather than restating it here.
  • feat(router): reach the Agent options a Router never forwards #568 (mine) also adds a Router.__init__ argument and a test_hooks_api.py check, so the two
    will need a one-line rebase whichever lands second. They are different mechanisms: agent_kwargs
    is one map applied to every checkpoint, which for digests is exactly the broken case above.

Second commit (9655978): LAYA_SHA256_DIGESTS can name each checkpoint

The same split, reachable without writing Python.

The gap

Commit 1 gave Router a per-checkpoint digest map, but a process configured only by
environment still had one flat LAYA_SHA256_DIGESTS, which laya.revisions.verify_digests
reads as {artifact: digest} — the same files checked against every checkpoint the process
loads. Since convaiinnovations/laya ships a separate model.safetensors per checkpoint,
that shape can match at most one of them. Measured on main, real cached weights, loading
english then multilingual from one Router — the same pair LAYA_MODELS=english,multilingual
preloads in laya-serve:

LAYA_SHA256_DIGESTS english multilingual
flat map of english's two files loaded SHA-256 mismatch for model.safetensors: expected 891102d3…, got 9d628fd9…
{english: {…}, multilingual: {…}} cannot verify 'english': no such file … the same error, about english

So the caller who sets the variable for supply-chain verification gets the opposite: the
first checkpoint loads and the rest are refused as mismatched. And the value with the shape
you would reach for does not parse either — it is read as an artifact map, so english is
looked for as a filename. There is no value of this variable that verifies a Router holding
two checkpoints, which is the configuration the docs point at for language routing.

The change

_digests_from_env() in laya/router.py, ~30 lines; the value types decide the reading:

  • every value a string → flat, and this layer returns {} so laya.revisions keeps applying
    it exactly as before. Row 1 above is unchanged behaviour, not something this commit fixes —
    it is the flat shape's limitation, and it stays.
  • every value a dict → model-keyed, split per checkpoint, and a checkpoint the map does not
    name becomes {}: deliberately unpinned, rather than falling back to the flat environment
    read that produced cannot verify 'english' above.
  • mixed → refused with a message naming the variable, because guessing which half the author
    meant would silently drop verification for the other half.

Router(sha256_digests=...) overrides per checkpoint, so one model can be pinned in code and
the rest left to the environment. Environment keys go through normalise_name, so en names
english and a name core does not know fails at construction instead of quietly leaving that
checkpoint unverified. laya/revisions.py, laya/serve.py and the flat shape are untouched.

After, same probe and machine

LAYA_SHA256_DIGESTS english multilingual
flat, english's files loaded mismatch (unchanged, see above)
nested, both correct loaded loaded
nested, multilingual's digest zeroed loaded SHA-256 mismatch … expected 000000…, got 9d628fd9…
nested, english zeroed, plus Router(sha256_digests={"english": correct}) loaded loaded

Rows 3 and 4 are the point: one bad digest refuses exactly one checkpoint instead of the
process, and an explicit argument still wins for the model it names.

Tests

10 new weight-free tests in tests/test_revision_pinning.py (30 → 40): the kwarg-level
claims go through a patched Agent, the end-to-end ones through commit 1's five fake
checkpoint dirs, where reaching the config error is the witness that the digest gate passed.
Eight of the ten fail at 4b4dbdd by name — six assertion failures, and two KeyErrors from
reading router.sha256_digests, where the nested map simply produced no entry — and pass
here:

test_nested_environment_pins_both_checkpoints ·
test_nested_environment_fails_only_the_checkpoint_it_calls_out ·
test_nested_environment_leaves_an_unnamed_checkpoint_unpinned ·
test_nested_map_is_split_per_checkpoint · test_nested_map_keys_are_normalised ·
test_argument_wins_for_the_checkpoint_it_names ·
test_misspelled_key_in_the_environment_fails_at_construction ·
test_mixed_artifact_and_model_keys_fail_rather_than_guess

The other two (test_flat_map_is_left_to_revisions,
test_unparseable_environment_is_not_this_layers_error) pass before and after on purpose:
they are the no-change guards for the flat shape and for who owns a parse error.

Mutation check against the subject, 7 mutants of laya/router.py, each reverted and the file
verified byte-identical afterwards. All 7 caught by a named test:

mutant caught by
nested shape never taken over 7 tests
seeding dropped, so an unnamed checkpoint falls back to the flat env read 2
flat shape taken over too test_flat_map_is_left_to_revisions
the argument loses to the environment test_argument_wins_for_the_checkpoint_it_names
environment keys not normalised 2
mixed shape silently unpinned instead of refused test_mixed_artifact_and_model_keys_fail_rather_than_guess
a malformed value re-raised by this layer test_unparseable_environment_is_not_this_layers_error

Gates

  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 — all checks passed
  • python -m compileall -q laya/ tests/ — clean · git diff --check — clean
  • tests/test_revision_pinning.py 40 OK · tests/test_router.py 546 passed ·
    tests/test_hooks_api.py 132 passed, unchanged from commit 1 (no signature moves, so the
    API contract suite has nothing new to assert)

Limits, and what this deliberately does not do

  • A malformed JSON value is left unparsed here, so the message stays laya.revisions' own
    rather than a second opinion on the same string.
  • ONNXAgent still reads only the flat shape: it is loaded outside the Router path this
    seeds, so the model-keyed map would need its own call site to say which model it is. Left
    alone rather than half-wired.
  • No docs touched. The README row for this variable is docs(env): write down the two LAYA_* names no page carried, and sweep for the rest #575's file, and I would rather not
    have two PRs editing the same table. Whoever lands first leaves the nested shape to add as
    a follow-up on the other side.

Overlap

The change is one module-level helper plus the seeding line inside Router.__init__, i.e.
inside commit 1's own hunk, so it conflicts with nothing that does not already conflict with
commit 1. tests/test_revision_pinning.py is shared with #528 and #568; both add their own
classes, and my additions are methods in the class commit 1 introduced plus one new class, so
no line collides with either.

`Router` forwards `revision` per checkpoint but has no way to reach the
other integrity option every loader takes, `Agent(expected_sha256=...)`.
The only lever a multi-checkpoint deployment has is `LAYA_SHA256_DIGESTS`,
which is one flat map, and the bundled repository ships a separate
`model.safetensors`, `rl_agent_config.json` and tokenizer under each
subfolder, so a map that matches `english` mismatches every sibling.

`sha256_digests={"english": {...}}` gives each checkpoint its own map,
keyed and normalised exactly like `revisions` so a misspelled model name
fails at construction instead of quietly leaving that checkpoint
unverified. Listing a model with `None` loads it unverified and masks the
environment default for that checkpoint only.
@rahul05ranjan

Copy link
Copy Markdown
Contributor

Thanks for the PR. Per-checkpoint SHA-256 verification is a solid security addition, and the revisions-style normalization is the right design.

A few points:

  1. or {} masking semantics: the comment explains that an explicit None must reach Agent as {} to mask the LAYA_SHA256_DIGESTS env default. That's subtle — a reader could easily misread sha256_digests={"typed": None} as "don't verify" when it actually means "verify nothing (mask the env default)". Worth making the docstring even more explicit, or considering a sentinel that's harder to confuse with "unset".

  2. expected_sha256 naming: the Router param is sha256_digests but the Agent kwarg is expected_sha256. The mapping is clear in load(), but the two names for the same concept could trip up someone grepping. Minor.

  3. Normalization asymmetry: revisions normalizes keys via normalise_name, and sha256_digests does the same — good. But note the test asserts Router().sha256_digests == {} while revisions presumably also defaults to {}. Consistent.

  4. Digest format validation: the map values are {path: hexdigest} but there's no validation that the hexdigest is a 64-char hex string. A typo'd digest would fail at load time (good) but with a potentially confusing error. Worth a quick format check at construction for a clearer message.

Otherwise the design is clean and the tests cover the key normalization behavior.

A process configured only by environment could not pin a set of resident
checkpoints: the bundled repository ships its own `model.safetensors` per
checkpoint, so one flat {artifact: digest} map matches one of them and
refuses the rest at startup. Accept the model-keyed shape that
`Router(sha256_digests=...)` already takes, leave the flat map to
`laya.revisions` unchanged, and let an argument entry win for the model it
names.
@NandhaKishorM
NandhaKishorM merged commit 7febda4 into NandhaKishorM:main Sep 27, 2026
22 checks passed
NandhaKishorM added a commit that referenced this pull request Sep 27, 2026
#568 and #572 added agent_kwargs and sha256_digests in the middle of the signature,
ahead of max_loaded, so Router(models, device, token, revision, revisions, 2) would
have bound 2 to agent_kwargs. Both are new, so they go last instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. The new parameter moved to the end of Router.__init__ (with #568's agent_kwargs) so positional callers are unaffected, and a per-model digest wins over an expected_sha256 passed through agent_kwargs. It ships in 0.3.21.

keemsisi added a commit to keemsisi/laya that referenced this pull request Sep 28, 2026
NandhaKishorM#347 shipped checkpoint-integrity controls and nothing documented them: no hit for
`expected_sha256`, `PINNED_REVISIONS` or `LAYA_SHA256_DIGESTS` in README.md or
docs/. That matters more than a usual docs gap because NandhaKishorM#332 kept pinning opt-in so
older offline snapshots keep loading, and because not using it is not an error --
a load with no revision and no digest map succeeds silently, reports
`agent.revision is None`, and hashes nothing. The only thing that ever switches an
inert control on is knowing it exists.

docs/security.md, under Deployment: pinning a revision, verifying digests,
generating the map, the environment variable, what a mismatch looks like in the
server, reading back what loaded, the laya-ts equivalents, and the limits.

Rebased onto main, and the page now describes what landed while this sat in draft:

- `LAYA_REVISION` (NandhaKishorM#605), including `reviewed`, which resolves each repository's own
  SHA and raises for one the table has no entry for rather than loading it unpinned.
  This replaced the section saying revision pinning was unreachable from the server.
- `Router(sha256_digests=...)` and the model-keyed `LAYA_SHA256_DIGESTS` (NandhaKishorM#572). The
  page now documents both shapes of the variable -- flat `{artifact: digest}` for
  every checkpoint, nested `{model: {artifact: digest}}` per checkpoint -- instead of
  the one-map-per-process limitation that shape removed, and says a checkpoint the
  nested map does not name is deliberately unpinned while an unknown model name
  raises.
- `tests/test_env_docs.py`'s DEFERRED row for `LAYA_SHA256_DIGESTS` is retired, since
  this page is what documented it.

What executing the examples still turns up, and the page says rather than tidies:

- `PINNED_REVISIONS` is keyed by the three standalone repositories while a default
  `Router` loads all three from the one bundle repository, so a per-model pin needs
  `standalone_repos=True`; without it the SHA goes to a repository that does not
  contain it and the load fails. The example pins all three, because a `Router`
  offers every checkpoint it knows about regardless of what is preloaded.
- A digest mismatch in the server is a **422**, not a 500: `laya/serve.py` maps the
  `ValueError` and returns the digest text to the caller. A listed but absent file
  raises `FileNotFoundError` and becomes a generic 500 with the reason only in the
  log. The page says to plan for the 422, because it sorts as a client error and so
  will not appear where an operator looks for a broken deployment.
- Only listed files are checked, and `tokenizer/tokenizer_config.json` is parsed but
  left out of the recommended map because Laya may normalise and rewrite it just
  after the digest check -- conditionally, so the page says it depends on the
  checkpoint rather than stating it as universal.

`compose.http.yaml` passes `LAYA_SHA256_DIGESTS` through, defaulting to empty. The
row added to docs/docker.md would otherwise be false for the deployment that table
describes: the variable reached neither the container nor docker/entrypoint.py, so
an operator setting it and running compose got no verification at all, silently.

The page carries no literal digest and no literal commit SHA outside an elided
illustration. Digests change with a checkpoint's files, so it ships a generator and
says to run that -- including rather than copying from the page. There is
deliberately no command that produces the map: one computed from the copy Laya just
downloaded would hash those bytes and verify them against themselves. A "Do you need
both?" section says what each control trusts -- a revision asks the Hub for a commit
and believes the answer, a digest is a record the operator keeps and is the only
check that does not ask the thing under test to vouch for itself.

`PINNED_REVISIONS` is exported from the package root for parity with laya-ts's
index.ts, through the `_LAZY_ATTRS` table rather than an eager import, so `import
laya` pulls in no new module and torch stays absent. Because that is a public API
change, tests/test_hooks_api.py gains the export: `__all__` membership plus a
`hasattr` check, which is what proves the lazy `__getattr__` resolves it. Removing
the export fails exactly those two. tests/test_lazy_import.py asserts the pins
resolve with torch blocked, and tests/test_revision_pinning.py adds PublicApiTests
for the root export, identity with laya.revisions, and that each pin is a usable
revision string -- not a 40-hex shape, which would invent a policy the library does
not state.
keemsisi added a commit to keemsisi/laya that referenced this pull request Sep 28, 2026
NandhaKishorM#347 shipped checkpoint-integrity controls and nothing documented them: no hit for
`expected_sha256`, `PINNED_REVISIONS` or `LAYA_SHA256_DIGESTS` in README.md or
docs/. That matters more than a usual docs gap because NandhaKishorM#332 kept pinning opt-in so
older offline snapshots keep loading, and because not using it is not an error --
a load with no revision and no digest map succeeds silently, reports
`agent.revision is None`, and hashes nothing. The only thing that ever switches an
inert control on is knowing it exists.

docs/security.md, under Deployment: pinning a revision, verifying digests,
generating the map, the environment variable, what a mismatch looks like in the
server, reading back what loaded, the laya-ts equivalents, and the limits.

Rebased onto main, and the page now describes what landed while this sat in draft:

- `LAYA_REVISION` (NandhaKishorM#605), including `reviewed`, which resolves each repository's own
  SHA and raises for one the table has no entry for rather than loading it unpinned.
  This replaced the section saying revision pinning was unreachable from the server.
- `Router(sha256_digests=...)` and the model-keyed `LAYA_SHA256_DIGESTS` (NandhaKishorM#572). The
  page now documents both shapes of the variable -- flat `{artifact: digest}` for
  every checkpoint, nested `{model: {artifact: digest}}` per checkpoint -- instead of
  the one-map-per-process limitation that shape removed, and says a checkpoint the
  nested map does not name is deliberately unpinned while an unknown model name
  raises.
- `tests/test_env_docs.py`'s DEFERRED row for `LAYA_SHA256_DIGESTS` is retired, since
  this page is what documented it.

What executing the examples still turns up, and the page says rather than tidies:

- `PINNED_REVISIONS` is keyed by the three standalone repositories while a default
  `Router` loads all three from the one bundle repository, so a per-model pin needs
  `standalone_repos=True`; without it the SHA goes to a repository that does not
  contain it and the load fails. The example pins all three, because a `Router`
  offers every checkpoint it knows about regardless of what is preloaded.
- A digest mismatch in the server is a **422**, not a 500: `laya/serve.py` maps the
  `ValueError` and returns the digest text to the caller. A listed but absent file
  raises `FileNotFoundError` and becomes a generic 500 with the reason only in the
  log. The page says to plan for the 422, because it sorts as a client error and so
  will not appear where an operator looks for a broken deployment.
- Only listed files are checked, and `tokenizer/tokenizer_config.json` is parsed but
  left out of the recommended map because Laya may normalise and rewrite it just
  after the digest check -- conditionally, so the page says it depends on the
  checkpoint rather than stating it as universal.

`compose.http.yaml` passes `LAYA_SHA256_DIGESTS` through, defaulting to empty. The
row added to docs/docker.md would otherwise be false for the deployment that table
describes: the variable reached neither the container nor docker/entrypoint.py, so
an operator setting it and running compose got no verification at all, silently.

The page carries no literal digest and no literal commit SHA outside an elided
illustration. Digests change with a checkpoint's files, so it ships a generator and
says to run that -- including rather than copying from the page. There is
deliberately no command that produces the map: one computed from the copy Laya just
downloaded would hash those bytes and verify them against themselves. A "Do you need
both?" section says what each control trusts -- a revision asks the Hub for a commit
and believes the answer, a digest is a record the operator keeps and is the only
check that does not ask the thing under test to vouch for itself.

`PINNED_REVISIONS` is exported from the package root for parity with laya-ts's
index.ts, through the `_LAZY_ATTRS` table rather than an eager import, so `import
laya` pulls in no new module and torch stays absent. Because that is a public API
change, tests/test_hooks_api.py gains the export: `__all__` membership plus a
`hasattr` check, which is what proves the lazy `__getattr__` resolves it. Removing
the export fails exactly those two. tests/test_lazy_import.py asserts the pins
resolve with torch blocked, and tests/test_revision_pinning.py adds PublicApiTests
for the root export, identity with laya.revisions, and that each pin is a usable
revision string -- not a 40-hex shape, which would invent a policy the library does
not state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants