Skip to content

Optional TileLang GPU fast path: 4-5x lower latency per call, identical answers - #25

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
cklxx:feat/tilelang-fast-path
Sep 23, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
cklxx:feat/tilelang-fast-path

Conversation

@cklxx

@cklxx cklxx commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

What

An opt-in forward for CUDA that replaces the stock ModernBERT + decision-head path with fused TileLang kernels:

  • laya/tl_kernels.py: GEMM with bias/activation epilogues, GEMM+GEGLU (the MLP up-projection and gate in one pass), residual-add+LayerNorm, in-place RoPE on the packed QKV buffer, and a flash attention that reads QKV directly, honours the key-padding mask and ModernBERT's bidirectional sliding window (|i−j| ≤ 65). Row count is a runtime symbol so the GEMM/LN/RoPE kernels compile once; attention uses dynamic B/L kernels for inputs ≤256 tokens and static per-bucket kernels beyond.
  • laya/fast.py: rebuilds DecisionModel.forward from those kernels with bf16-resident weights (no per-call autocast weight casts) and captures one CUDA graph per (batch, length) bucket, which removes the ~200 Python-side kernel launches that dominate a 1-question call.
  • Agent.accelerate() / Agent.deaccelerate(), laya.load(..., fast=True), and a laya[fast] extra. Nothing changes unless you ask for it; on CPU/MPS or without tilelang it falls back to the stock forward with a warning (or raises with strict=True).

Numbers (RTX 4070 Ti SUPER, torch 2.11 / CUDA 13, tilelang 0.1.14)

benchmarks/bench_fast.py --eval 1000, raw JSON in benchmarks/results/. Full tables in BENCHMARKS.md → GPU fast path.

checkpoint 1 question e2e 3 questions 30 q × 512-966 tok AG News acc (stock → fast) emotion acc argmax agreement
laya 17.7 → 4.6 ms (3.8×) 18.9 → 6.6 ms 328 → 232 ms 0.922 → 0.925 0.588 → 0.588 99.5 %
laya-multilingual 14.1 → 2.8 ms (5.1×) 15.0 → 3.9 ms 321 → 188 ms 0.923 → 0.923 0.538 → 0.540 99.6 %

ECE is unchanged to the third decimal on both datasets. The max change in any reported probability against an fp32 forward is the same order as the stock bf16 autocast path (≤0.004 fast vs ≤0.013 stock on the English checkpoint).

Tests

tests/test_fast.py checks every kernel against a torch reference (including non-tile-aligned shapes, padding rows and both window modes) and, with LAYA_TEST_MODEL set, the full forward against the stock model. It skips without CUDA/tilelang, so CI is unaffected; ruff (the CI selection) and compileall pass.

Notes

  • First use of a new length bucket compiles kernels (a few seconds, cached on disk by tilelang). Short inputs share one dynamic-shape kernel.
  • Large batches are GEMM bound and the fused kernels sit at ~80 TFLOPS there, so the remaining headroom at 30+ questions is small; the win is per-call latency and the sliding-window attention on long inputs (16× over SDPA with a dense mask at L=1024).

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for contributing the TileLang acceleration path. Since it introduces a compiler dependency, we are considering keeping this as an optional extra.

@cklxx
cklxx force-pushed the feat/tilelang-fast-path branch from 26cdedb to 8cb68d6 Compare September 22, 2026 07:05
@cklxx

cklxx commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — keeping it as an optional extra is exactly how it's built, so no objection at all. To make the boundary explicit:

  • pip install laya is unchanged: tilelang lives only in the laya[fast] extra, and laya/fast.py / laya/tl_kernels.py are imported lazily inside Agent.accelerate(). import laya never imports tilelang (checked: 'tilelang' in sys.modules is False after import laya).
  • Nothing changes unless a user asks for it (laya.load(..., fast=True) or agent.accelerate()); on CPU/MPS or without tilelang the stock forward is used with a warning (strict=True raises instead).
  • tests/test_fast.py skips at module level without CUDA + tilelang, so the CPU CI matrix is unaffected. The five test files your CI runs (test_router, test_criteria, test_download, test_shortlist, test_decision_model) pass on this branch locally.
  • Rebased onto 0.3.5 (pyproject.toml conflict with the new [tool.setuptools] block resolved, stray .log files dropped). The new embedding shortlist (feat: opt-in embedding shortlist for high-cardinality choice #106) is upstream of the model call, so the two compose without changes.

If you'd rather isolate it further I'm happy to do either: move the two files under laya/contrib/tilelang/ (or ship them as a separate laya-fast package that registers itself), and/or require an explicit LAYA_FAST=1 in addition to the flag. Just say which shape you want and I'll adjust.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and for keeping it strictly opt-in. Identical answers at 4 to 5x lower latency is a strong result. This needs a rebase onto today's main, which changed the model build path (#195 now builds under no_init_weights). After that, a run showing identical probabilities on a fixed set of states would be the evidence I'd want, since I can't test the CUDA kernels locally.

Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm,
in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py,
which rebuilds the DecisionModel forward from those kernels with bf16-resident
weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() /
deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls
tilelang. Nothing is imported or changed unless fast=True is requested; without
CUDA or tilelang the stock forward is used (warning, or raise with strict=True).

Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical
within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go
from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x.

benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset
comparison); tests/test_fast.py checks each kernel against torch and the full
forward against the stock model (skips without CUDA/tilelang).
…in fp32; add parity evidence

Two deviations from the stock forward surfaced on a fixed-state parity run
(benchmarks/parity_fast.py, new):

- The local-attention kernel used ModernBertAttention.sliding_window (config
  value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps
  |i - j| <= config.sliding_window, so every local layer saw one extra key per
  side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some
  inputs (mmBERT-base was barely affected). The kernel now takes
  config.sliding_window.
- The residual stream was stored in bf16 between layers; the stock path keeps
  it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16
  loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits
  the bf16 normalised input; embeddings are gathered from an exact fp16 copy.

Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path
is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32
<= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type,
and on every disagreement the fast path sides with fp32. Per-option
probabilities for all three paths are in benchmarks/results/parity_*.json.
Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).
@cklxx
cklxx force-pushed the feat/tilelang-fast-path branch from 8cb68d6 to 2e54f87 Compare September 23, 2026 09:05
@cklxx

cklxx commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto today's main (0.3.7, so on top of #195's no_init_weights build path — the fast path only reads weights after load_state_dict, so nothing needed to change there) and added the evidence you asked for.

benchmarks/parity_fast.py answers a fixed, deterministic set of 60 states × up to 8 questions (the five presets over 12 texts in six languages, short and long; 288 questions per checkpoint) three ways: stock bf16-autocast forward, fast path, and an fp32 forward as the reference. Every per-option probability from all three is committed in benchmarks/results/parity_{english,multilingual}_rtx4070.json, so you can re-check the comparison without a GPU.

checkpoint type n max |p_fast − p_stock| max |p_fast − p_fp32| max |p_stock − p_fp32| argmax fast = stock fast = fp32
laya choice 48 0.031 0.022 0.024 47/48 47/48
laya noul 180 0.076 0.044 0.058 180/180 180/180
laya score 60 0.015 0.011 0.017 59/60 60/60
laya-multilingual choice 48 0.049 0.015 0.039 47/48 47/48
laya-multilingual noul 180 0.037 0.046 0.045 180/180 179/180
laya-multilingual score 60 0.010 0.009 0.009 59/60 59/60

So: the fast path is at least as close to fp32 as the stock bf16 path is, the two bf16 paths differ only by accumulation order, and on every argmax disagreement (near-tie options) the fast path sides with fp32.

Being honest about what the run found: the first pass of this parity set exposed two deviations that my earlier 3–4-option checks had not caught, both fixed in the second commit (2e54f87):

  1. the local-attention kernel used ModernBertAttention.sliding_window (config value + 1, flash-attn's inclusive convention) whereas HF's bidirectional mask keeps |i − j| <= config.sliding_window, so each local layer saw one extra key per side — up to 0.6 on some ModernBERT-large noul outputs;
  2. the residual stream was carried in bf16 between layers, while the stock path keeps it in fp32 — ModernBERT-large's residual activations reach ~3·10⁴, where bf16 loses ~100 units per add. It is fp32 now (same as stock), which is why the fast path now lands closer to fp32 than the stock bf16 path.

Latency is unchanged: 1-question predict() 3.2 ms (multilingual) / 5.3 ms (English) vs 16–20 ms stock, benchmarks/results/fast_*_rtx4070.json regenerated. Locally: all 16 test files in your CI pass on this branch, import laya still never imports tilelang, tests/test_fast.py skips without CUDA + tilelang.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for the parity evidence. It shows exactly what I needed: the fast path agrees with the stock bf16 path on 286 of 288 questions per checkpoint, and it is as close to the fp32 reference as stock bf16 is, so the differences are bf16 rounding rather than the kernels. It is opt-in and falls back to the stock forward when TileLang or CUDA is missing. Merging. One small follow-up for the README: "identical answers" would read more accurately as "matches stock bf16 within rounding", with the parity numbers linked.

@NandhaKishorM
NandhaKishorM merged commit 2a2a2b1 into NandhaKishorM:main Sep 23, 2026
13 checks passed
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…al answers (NandhaKishorM#25)

* feat: optional TileLang GPU fast path (laya.load(..., fast=True))

Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm,
in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py,
which rebuilds the DecisionModel forward from those kernels with bf16-resident
weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() /
deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls
tilelang. Nothing is imported or changed unless fast=True is requested; without
CUDA or tilelang the stock forward is used (warning, or raise with strict=True).

Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical
within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go
from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x.

benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset
comparison); tests/test_fast.py checks each kernel against torch and the full
forward against the stock model (skips without CUDA/tilelang).

* fix(fast): match HF's sliding window bound, keep the residual stream in fp32; add parity evidence

Two deviations from the stock forward surfaced on a fixed-state parity run
(benchmarks/parity_fast.py, new):

- The local-attention kernel used ModernBertAttention.sliding_window (config
  value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps
  |i - j| <= config.sliding_window, so every local layer saw one extra key per
  side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some
  inputs (mmBERT-base was barely affected). The kernel now takes
  config.sliding_window.
- The residual stream was stored in bf16 between layers; the stock path keeps
  it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16
  loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits
  the bf16 normalised input; embeddings are gathered from an exact fp16 copy.

Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path
is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32
<= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type,
and on every disagreement the fast path sides with fp32. Per-option
probabilities for all three paths are in benchmarks/results/parity_*.json.
Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants