Repository navigation
perf(lang): lazy state text traversal and string fast-path - #1448
Conversation
jev_confidence no longer returns a value outside [0, 1] when finite weights sum to infinity (#1442), language detection flattens large states lazily (#1448), laya-ts counts and cuts code points like Python (#1445, #1446), and Router.predictBatch honours per-request token budgets (#1441). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged into 0.4.22. The correctness case is solid, and I want to be specific about what I checked, because converting The two risks, both clear. My own differential, beyond your 588 cases: 6232 compared outputs across The performance table needs correcting, though. The direction is right and the big win is real, but the numbers as published are not reproducible here, and one row points the wrong way. Measured on this machine, median of 5 repeats at 2000 loops each, per call:
Two things to take from that. First, your table reports 155.70 ms and 330.56 ms for single That also means the framing "speeds up plain string detection by 4.5x, the majority of incoming traffic" does not describe the router's own path.
None of that changes the merge. Trading about 0.2 us on a small state for 27 us on a large one is a good trade, the regression is invisible beside a forward pass measured in milliseconds, and the pathological case this fixes is the one that actually hurt. I am recording it so the measured claim in the history is the honest one, and so nobody later reads "1.2x faster on 50-turn dicts" as something the repo verified. If you want the small-state cost back, the cheap route is bypassing the generator when the state is a flat one-or-two-leaf mapping, the same shape of shortcut as your |
|
yeah the table numbers were the total sum across the 1000 iterations loop rather than divided per call... and also makes total sense on the yield from overhead on small dicts that dont hit the budget early too. thanks for the review and merge! |
What
in
laya/lang.py:_iter_textfrom an eager list materializer into a lazy generator (yield/yield from), halting traversal as soon asmax_charsis satisfied instate_text.strandbytesinstate_text, bypassing_iter_text, list allocations, and" ".join().leaves = _iter_text(state)multi-leaf scan inanalysewith a 2-leaf lookahead on the iterator.tests/test_router.pyverifying plain string fast-paths, bytes decoding, and early budget halting.Why
every call to
router.route()androuter.predict()flattens states for language detection viastate_text(state, max_chars=4000).previously,
_iter_texteagerly recursed through the entire state tree into a Python list. on large nested payloads (like multi-turn chat histories or ~500 KB documents), it spent hundreds of milliseconds walking and materializing thousands of leaves even thoughstate_textonly ever reads the first 4,000 characters.additionally, plain strings (the majority of incoming traffic) still paid for generator/list wrapping and
" ".join().making traversal lazy and fast-pathing plain strings cuts large state latency from 330 ms down to 7.2 ms (45x faster) and speeds up plain string detection by 4.5x, with 0 byte-level differences across 588 tested shapes and budgets.
tested on
state_textbefore and after (1,000 iterations):How it was verified
all passed: