Repository navigation
Optional TileLang GPU fast path: 4-5x lower latency per call, identical answers - #25
Conversation
|
Thanks for contributing the TileLang acceleration path. Since it introduces a compiler dependency, we are considering keeping this as an optional extra. |
26cdedb to
8cb68d6
Compare
|
Thanks — keeping it as an optional extra is exactly how it's built, so no objection at all. To make the boundary explicit:
If you'd rather isolate it further I'm happy to do either: move the two files under |
|
Thank you, and for keeping it strictly opt-in. Identical answers at 4 to 5x lower latency is a strong result. This needs a rebase onto today's |
Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm, in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py, which rebuilds the DecisionModel forward from those kernels with bf16-resident weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() / deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls tilelang. Nothing is imported or changed unless fast=True is requested; without CUDA or tilelang the stock forward is used (warning, or raise with strict=True). Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x. benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset comparison); tests/test_fast.py checks each kernel against torch and the full forward against the stock model (skips without CUDA/tilelang).
…in fp32; add parity evidence Two deviations from the stock forward surfaced on a fixed-state parity run (benchmarks/parity_fast.py, new): - The local-attention kernel used ModernBertAttention.sliding_window (config value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps |i - j| <= config.sliding_window, so every local layer saw one extra key per side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some inputs (mmBERT-base was barely affected). The kernel now takes config.sliding_window. - The residual stream was stored in bf16 between layers; the stock path keeps it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16 loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits the bf16 normalised input; embeddings are gathered from an exact fp16 copy. Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32 <= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type, and on every disagreement the fast path sides with fp32. Per-option probabilities for all three paths are in benchmarks/results/parity_*.json. Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).
8cb68d6 to
2e54f87
Compare
|
Rebased onto today's
So: the fast path is at least as close to fp32 as the stock bf16 path is, the two bf16 paths differ only by accumulation order, and on every argmax disagreement (near-tie options) the fast path sides with fp32. Being honest about what the run found: the first pass of this parity set exposed two deviations that my earlier 3–4-option checks had not caught, both fixed in the second commit (
Latency is unchanged: 1-question |
|
Thank you for the parity evidence. It shows exactly what I needed: the fast path agrees with the stock bf16 path on 286 of 288 questions per checkpoint, and it is as close to the fp32 reference as stock bf16 is, so the differences are bf16 rounding rather than the kernels. It is opt-in and falls back to the stock forward when TileLang or CUDA is missing. Merging. One small follow-up for the README: "identical answers" would read more accurately as "matches stock bf16 within rounding", with the parity numbers linked. |
…al answers (NandhaKishorM#25) * feat: optional TileLang GPU fast path (laya.load(..., fast=True)) Adds laya/tl_kernels.py (fused GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm, in-place RoPE, padding-aware sliding-window flash attention) and laya/fast.py, which rebuilds the DecisionModel forward from those kernels with bf16-resident weights and a CUDA graph per (batch, length) bucket. Agent.accelerate() / deaccelerate() and the `fast=` flag on load(); the `laya[fast]` extra pulls tilelang. Nothing is imported or changed unless fast=True is requested; without CUDA or tilelang the stock forward is used (warning, or raise with strict=True). Numerics match the stock bf16 path (AG News / emotion accuracy and ECE identical within noise, 99.5% argmax agreement, same max |dp| vs fp32). 1-question calls go from 14-18 ms to 3-5 ms on an RTX 4070 Ti SUPER; 30-question batches 1.2-1.9x. benchmarks/bench_fast.py reproduces the numbers (--eval N adds the dataset comparison); tests/test_fast.py checks each kernel against torch and the full forward against the stock model (skips without CUDA/tilelang). * fix(fast): match HF's sliding window bound, keep the residual stream in fp32; add parity evidence Two deviations from the stock forward surfaced on a fixed-state parity run (benchmarks/parity_fast.py, new): - The local-attention kernel used ModernBertAttention.sliding_window (config value + 1, flash-attn's inclusive convention). HF's bidirectional mask keeps |i - j| <= config.sliding_window, so every local layer saw one extra key per side. On ModernBERT-large this moved noul probabilities by up to 0.6 on some inputs (mmBERT-base was barely affected). The kernel now takes config.sliding_window. - The residual stream was stored in bf16 between layers; the stock path keeps it in fp32 and ModernBERT-large's residual activations reach ~3e4, where bf16 loses ~100 units per add. add_ln_kernel now updates an fp32 stream and emits the bf16 normalised input; embeddings are gathered from an exact fp16 copy. Result on 60 fixed states x up to 8 questions (both checkpoints): the fast path is at least as close to an fp32 forward as the stock bf16 path (max |dp| vs fp32 <= 0.046 fast vs <= 0.058 stock), argmax agreement >= 47/48 per question type, and on every disagreement the fast path sides with fp32. Per-option probabilities for all three paths are in benchmarks/results/parity_*.json. Latency is unchanged (1-question e2e 3.2 ms multilingual / 5.3 ms English).
What
An opt-in forward for CUDA that replaces the stock ModernBERT + decision-head path with fused TileLang kernels:
laya/tl_kernels.py: GEMM with bias/activation epilogues, GEMM+GEGLU (the MLP up-projection and gate in one pass), residual-add+LayerNorm, in-place RoPE on the packed QKV buffer, and a flash attention that reads QKV directly, honours the key-padding mask and ModernBERT's bidirectional sliding window (|i−j| ≤ 65). Row count is a runtime symbol so the GEMM/LN/RoPE kernels compile once; attention uses dynamic B/L kernels for inputs ≤256 tokens and static per-bucket kernels beyond.laya/fast.py: rebuildsDecisionModel.forwardfrom those kernels with bf16-resident weights (no per-call autocast weight casts) and captures one CUDA graph per (batch, length) bucket, which removes the ~200 Python-side kernel launches that dominate a 1-question call.Agent.accelerate()/Agent.deaccelerate(),laya.load(..., fast=True), and alaya[fast]extra. Nothing changes unless you ask for it; on CPU/MPS or without tilelang it falls back to the stock forward with a warning (or raises withstrict=True).Numbers (RTX 4070 Ti SUPER, torch 2.11 / CUDA 13, tilelang 0.1.14)
benchmarks/bench_fast.py --eval 1000, raw JSON inbenchmarks/results/. Full tables in BENCHMARKS.md → GPU fast path.ECE is unchanged to the third decimal on both datasets. The max change in any reported probability against an fp32 forward is the same order as the stock bf16 autocast path (≤0.004 fast vs ≤0.013 stock on the English checkpoint).
Tests
tests/test_fast.pychecks every kernel against a torch reference (including non-tile-aligned shapes, padding rows and both window modes) and, withLAYA_TEST_MODELset, the full forward against the stock model. It skips without CUDA/tilelang, so CI is unaffected;ruff(the CI selection) andcompileallpass.Notes