Skip to content

docs(guides): add production use cases and architectural patterns (#675) - #684

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Swatantra-66:docs/production-use-cases
Oct 1, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Swatantra-66:docs/production-use-cases

Conversation

@Swatantra-66

Copy link
Copy Markdown
Contributor

What

Introduces an architectural patterns and production use cases guide in docs/use-cases.md and indexes it in docs/.nav.yml under Guides:.

Key architectural patterns covered:

  • Decision Engine vs. LLM vs. Vector Similarity: Trade-off analysis covering latency budgets (sub-30ms vs. multi-second), token unit economics, and output reliability.
  • Smart Ingress LLM Gateway: Cost-slashing pattern offloading deterministic FAQs and simple routing to local CPU before dispatching complex queries to frontier APIs.
  • Sub-200ms Voice Turn-Taking: Low-latency interruption detection and conversational fillers between STT and TTS engines.
  • Pre-LLM Security Firewall: Non-autoregressive input guardrails intercepting prompt injections and system prompt extraction via prediction hooks.
  • Air-Gapped Edge RAG Router: Local domain routing to segregated vector stores in compliance-constrained environments.
  • Hierarchical Multi-Agent Supervisor: Sub-30ms agent delegation eliminating multi-second LLM orchestration hops.
  • High-Throughput Ticket Triage: Batch classification pipeline handling 200+ events/sec per GPU using decide_batch().

All architectures are illustrated with interactive GitHub-native Mermaid (graph TD) diagrams.

Why

Fixes #675. Provides production engineers and architects with turnkey blueprints, reference implementations, and concrete criteria schemas for integrating Laya into production workflows.

How it was verified

  • Validated documentation compilation:
    zensical build --strict --clean (exited with code 0 in 3.34s)
  • Verified repository CI gates:
    ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
    python -m compileall -q laya/ tests/

Checklist

  • Focused on one change
  • Strict docs build passes (zensical build --strict --clean)
  • Conventional commit format (docs(guides): ...)
  • Rebased onto latest upstream/main

@LEVELING2108

Copy link
Copy Markdown
Contributor

Thanks for picking up #675 and putting this guide together, @Swatantra-66! The structure, matrix comparison, and Mermaid diagrams look great.

A couple of quick review notes to ensure the snippets run cleanly for users:

  1. Hook Import in Section 3:
    In docs/use-cases.md (line 173), rom laya.hooks import BasePredictionHook will raise an ImportError because the class in laya/hooks.py is named BaseHook:
    `python
    from laya.hooks import BaseHook, PredictContext

    class SecurityFirewallHook(BaseHook):
    def init(self, guard_agent):
    self.guard = guard_agent

    def on_predict_start(self, ctx: PredictContext):
        ...
    

    `
    (Alternatively, you can also mention or showcase the native LayaTaskGuard from laya.integrations.crewai which ships with Laya for this exact pattern).

  2. Cross-Link Native Framework Integrations:
    In Section 5 (Hierarchical Multi-Agent Task Delegation), it would be very helpful to cross-link the existing guides for LayaCrewRouter (docs/crewai.md) and LayaSingleSelector (docs/llamaindex.md) alongside the generic Router.predict snippet, so users know there are already first-party integrations available.

With the BaseHook import fixed, this will be a solid addition to the Guides!

@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the review and great catches, @LEVELING2108!

I have pushed an update addressing both points:

  1. Hook Import: Updated Section 3 to from laya.hooks import BaseHook, PredictContext with SecurityFirewallHook(BaseHook), and added a tip highlighting the pre-built LayaTaskGuard integration for CrewAI.
  2. Framework Cross-Links: Added explicit cross-links in Section 5 to LayaCrewRouter (docs/crewai.md), LayaSingleSelector (docs/llamaindex.md), and laya.integrations.langchain (docs/langchain.md).

Verified with zensical build --strict --clean with zero warnings/errors.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding: The use-cases guide includes unmeasured latency/cost/throughput numbers and vendor price comparisons. Please verify BaseHook existence and hook raising semantics against the current core API.

@Swatantra-66
Swatantra-66 force-pushed the docs/production-use-cases branch from f9e5774 to 2102c57 Compare September 29, 2026 18:21
@Swatantra-66

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review @NandhaKishorM! I have updated the PR to address all feedback:

  1. Unmeasured numbers & vendor comparisons removed: Stripped out all speculative latency numbers (e.g. 15-40ms, 3-8ms), dollar cost estimates, and vendor mentions (OpenAI, Anthropic, Deepgram, Whisper) across the guide. The document now focuses purely on architectural topology and structural profiles (non-autoregressive forward passes vs sequential token generation).
  2. Hook contract & BaseHook: Verified against laya.hooks — updated snippets to subclass BaseHook and noted that Hook is a Protocol supporting duck typing.
  3. Hook error semantics clarified: Added explicit documentation explaining that when hooks_raise=True, an exception raised inside on_predict_start aborts execution prior to tokenization or inference and propagates to the caller (with on_error and on_predict_end still firing for auditability).
  4. CI & docs build: Rebased onto latest main and fixed the relative link in typescript-sdk.md that was failing the strict docs check. zensical build --strict --clean, ruff check laya/, and python -m compileall are all passing cleanly.

…tops failing

`docs (strict build)` is red on main, and has been since before this branch: `zensical build
--strict` aborts on one warning.

    Warning: page does not exist
        ╭─[ typescript-sdk.md:67:21 ]
     67 │ See the [SDK guide](../sdk/typescript/README.md) for setup, examples, and npm
        │                     ─────────────┬─────────────
        │                                  ╰─────────────── page does not exist

`sdk/typescript/README.md` exists in the repository but not in the built site, because the site is
built from `docs/` and a `.md` link is resolved as a page. The workflow already states the convention
for this case -- "Use absolute GitHub URLs for files outside docs/ so links work when pages in docs/
are read on GitHub or on the site" -- and six links in `docs/` already follow it (BENCHMARKS.md,
research/README.md, the Kaggle notebook, compose.example.yml and others). This one did not.

Why only this link, when `docs/hooks/*.md` carries a dozen `../../examples/...` links that also point
outside `docs/`: those target `.py` files, which are never resolved as pages. Swept the whole tree for
the case that breaks -- a `.md` link resolving outside `docs/` -- and this is the only one.

Verified: `zensical build --strict --clean` goes from `1 issue found`, exit 1, to `No issues found`,
exit 0. The job's other gate, `grep '^griffe:'`, was never the problem -- the build emits no griffe
lines at all, before or after -- so the "Fix the docstring warnings above" error text on the failing
job was misleading; the failure was this link. Revert check: restoring the relative link fails the
build again. The link target returns HTTP 200. tests/test_doc_tables.py 114 passed,
tests/test_env_docs.py 9 passed, tests/test_structured_docs.py 87 passed, tests/test_evals.py 83
passed, tests/test_evals_api.py green.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both things you were asked to check have been done, and I verified them rather than taking the description's word for it. This is close to merge-ready — two one-line items.

Holding item 1: the unmeasured numbers are gone

Grepping the current docs/use-cases.md for the categories that were objected to:

$            0        cost per 1k      0        OpenAI / Anthropic / GPT-4   0
Whisper      0        Deepgram         0        "per second"                 0

The comparison table has been rebuilt around structural properties (computation model, execution profile, hosting and topology) rather than numbers, which is the right shape for a page like this. The three remaining latency mentions are all qualitative — "depends on latency constraints", a section heading, "adds redundant latency" — with no figure attached.

Holding item 2: BaseHook exists, and the raising semantics are described correctly

BaseHook exists: <class 'laya.hooks.BaseHook'>
lifecycle methods: ['on_error', 'on_evict', 'on_load', 'on_predict_end',
                    'on_predict_start', 'on_route']

A real no-op base class with all six lifecycle methods, and laya/hooks.py does not import torch — which is what that module's own contract claims, so the import stays cheap.

I also traced the three claims about raising, since "verify against the current core API" is easy to nod at. In laya/agent.py:

  • on_predict_start dispatches at :1143
  • tokenization happens at :1207 (_encode_state)
  • so a raising start hook aborts before any tokenization or inference — the guide's first bullet is true
  • ctx.error = exc at :1259, on_error at :1261, on_predict_end at :1274, and :1277-1278 re-raises with the original preserved as __context__

So all three bullets hold as written. This also resolves the earlier "BasePredictionHook does not exist" report from another contributor — the name in the guide now matches the code.

The one number left, and the duplicate hunk

docs/use-cases.md:407 — "ModernBERT-large requires ~1.6 GB of memory." This is the only unsourced figure remaining, and it is defensible but not stated anywhere: README.md:118 publishes 421M parameters, and 421M × 4 bytes is 1.57 GB of fp32 weights. So the figure is derivable — it just is not derived, and it omits that calibration or fine-tuning needs considerably more. Since this page is otherwise clean, either add the derivation in a clause or drop the number.

docs/typescript-sdk.md — your second commit carries the strict-build link fix, and that patch is byte-identical to #742, which is the focused one-line PR:

684: +See the [SDK guide](https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md) ...
742: +See the [SDK guide](https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md) ...

Worth knowing in connection: #742 is the fix that clears the docs (strict build) red currently attached to 11 open PRs, and I have reproduced both sides of the build — main aborts with exit 1, #742's head exits 0. Dropping your copy and rebasing onto it keeps one copy of that change in the tree and gets your docs (strict build) green from the same place.

Neither item touches the guide itself, so this is a merge from here once they are dealt with.

@Swatantra-66
Swatantra-66 force-pushed the docs/production-use-cases branch from 2102c57 to e159ecc Compare September 30, 2026 13:00
@Swatantra-66
Swatantra-66 force-pushed the docs/production-use-cases branch from e159ecc to 7c62c33 Compare September 30, 2026 13:12
@Swatantra-66

Swatantra-66 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough verification and guidance, @Bruce-Yii! Both items have been addressed:

  1. Hardware sizing figure: Dropped the ~1.6 GB memory figure at line 407, keeping the checklist guidance focused on host memory sizing and thread pool configuration without unsourced metrics.
  2. Rebased onto fix(docs): link the TypeScript SDK guide by URL so the strict build stops failing #742: Rebased directly on top of commit d2736a4 and dropped our duplicate copy of typescript-sdk.md. This PR now cleanly introduces only docs/use-cases.md and docs/.nav.yml on top of that fix.

All 24 GitHub Actions CI checks (including docs (strict build) and tests (windows)) are passing 100% green. Ready for merge!

@NandhaKishorM
NandhaKishorM merged commit 5f07114 into NandhaKishorM:main Oct 1, 2026
25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23.

Both of the things I held this for are gone: no unmeasured latency or cost figures, no vendor price comparisons, and the hook example uses the class that exists (BaseHook) with the contract core actually has. Rebasing onto #742 also means that fix came in with your branch, so #742 is merged too.

Verified on the integrated branch: full test lane green, docs tree builds, zero decision flips against main on the real checkpoints.

Thank you, and thanks to @LEVELING2108 for catching the import.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature]: Add production use cases and architectural patterns guide to docs

5 participants