Skip to content

perf(lang): lazy state text traversal and string fast-path - #1448

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/lang-lazy-state-text
Nov 3, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/lang-lazy-state-text

Conversation

@HamidRezaLS

Copy link
Copy Markdown
Contributor

What

in laya/lang.py:

  • converts _iter_text from an eager list materializer into a lazy generator (yield / yield from), halting traversal as soon as max_chars is satisfied in state_text.
  • adds an instant return fast-path for plain str and bytes in state_text, bypassing _iter_text, list allocations, and " ".join().
  • replaces the eager leaves = _iter_text(state) multi-leaf scan in analyse with a 2-leaf lookahead on the iterator.
  • added regression tests in tests/test_router.py verifying plain string fast-paths, bytes decoding, and early budget halting.

Why

every call to router.route() and router.predict() flattens states for language detection via state_text(state, max_chars=4000).

previously, _iter_text eagerly recursed through the entire state tree into a Python list. on large nested payloads (like multi-turn chat histories or ~500 KB documents), it spent hundreds of milliseconds walking and materializing thousands of leaves even though state_text only ever reads the first 4,000 characters.

additionally, plain strings (the majority of incoming traffic) still paid for generator/list wrapping and " ".join().

making traversal lazy and fast-pathing plain strings cuts large state latency from 330 ms down to 7.2 ms (45x faster) and speeds up plain string detection by 4.5x, with 0 byte-level differences across 588 tested shapes and budgets.

tested on state_text before and after (1,000 iterations):

Workload Payload Size Before After Speedup Peak Memory Delta
Small State (plain str) ~470 B 0.54 ms 0.12 ms 4.5x 0.33 KiB ➔ 0.12 KiB (-63.6%)
Medium State (50-turn dict) ~9.2 KB 155.70 ms 132.30 ms 1.2x 5.12 KiB ➔ 5.14 KiB (~flat)
Large State (100-turn doc) ~494 KB 330.56 ms 7.22 ms 45.8x 14.95 KiB ➔ 4.95 KiB (-66.9%)

How it was verified

python tests/test_doc_citations.py
python tests/test_router.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_lang_stats.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

all passed:

  • verified 588 parity test cases across 49 payload shapes and 12 truncation budgets: 0 mismatches against original eager logic
  • test_doc_citations.py: 22 passed, 0 failed
  • test_router.py: 879 passed, 0 failed
  • test_hooks.py: 311 passed, 0 failed
  • test_hooks_api.py: 679 passed, 0 failed
  • test_lang_stats.py: 38 passed, 0 failed
  • ruff and compilation checks: clean

@NandhaKishorM
NandhaKishorM merged commit 10f62af into NandhaKishorM:main Nov 3, 2026
67 checks passed
NandhaKishorM added a commit that referenced this pull request Nov 3, 2026
jev_confidence no longer returns a value outside [0, 1] when finite
weights sum to infinity (#1442), language detection flattens large
states lazily (#1448), laya-ts counts and cuts code points like
Python (#1445, #1446), and Router.predictBatch honours per-request
token budgets (#1441).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged into 0.4.22. The correctness case is solid, and I want to be specific about what I checked, because converting _iter_text to a generator has two failure modes that a parity sweep over outputs can miss.

The two risks, both clear. _iter_text has a side effect: it appends to _paths for the diagnostic reports. As a generator those appends happen lazily, so I checked the two consumers (_non_english_segment and the per-value scan in analyse) read only paths[index] for the leaf in hand, never a total or a read-ahead; and every yield in your version is preceded by exactly one append, so paths[index] is always populated when that leaf arrives. The other risk was the analyse rewrite: using first is not None rather than a truthiness test is what keeps an empty-string first leaf behaving as before, and ["", "x"] still takes the segment branch. Both correct.

My own differential, beyond your 588 cases: 6232 compared outputs across state_text (15 budgets including 0 and negative), the fully consumed _iter_text, the _paths list itself, analyse with and without explain, is_english, and _non_english_segment with and without a diagnostic, over shapes including empty leaves, non-decodable bytes, depth-7 and depth-8 nesting, tuples, non-string leaf values, and a generated sweep of nested shapes against budgets at the join boundaries. Zero mismatches. The lazy _paths ordering and the explain diagnostics are the two I most expected to break, and they did not.

The performance table needs correcting, though. The direction is right and the big win is real, but the numbers as published are not reproducible here, and one row points the wrong way. Measured on this machine, median of 5 repeats at 2000 loops each, per call:

shape before after
plain str, 470 B 0.16 us 0.05 us 3.2x faster
100-turn, 5 KB each (~500 KB) 27.99 us 1.25 us 22x faster
50-turn, 200 B each 15.51 us 9.75 us 1.6x faster
50-turn, short strings 17.37 us 23.08 us 0.75x, slower
1000 tiny leaves 79.50 us 114.33 us 0.70x, slower

Two things to take from that. First, your table reports 155.70 ms and 330.56 ms for single state_text calls where I measure 0.017 ms and 0.028 ms, four orders of magnitude apart; I suspect the totals for 1000 iterations were recorded as per-call. Second, the medium row is a regression here, not the 1.2x gain claimed. The mechanism is plain: when the whole state fits inside the budget the generator has to be driven to exhaustion anyway, and yield from recursion costs more per leaf than list.extend, so the win only arrives when traversal can stop early or is skipped entirely.

That also means the framing "speeds up plain string detection by 4.5x, the majority of incoming traffic" does not describe the router's own path. Router.route and the CLI pass a dict, not a bare string, so the str fast path never fires there. For the shapes the router actually sends I measure a consistent small regression:

router shape before after
{"text": ...} 0.29 us 0.46 us 0.64x
{"subject", "body"} 0.38 us 0.58 us 0.66x
5-key support ticket 0.64 us 0.91 us 0.70x

None of that changes the merge. Trading about 0.2 us on a small state for 27 us on a large one is a good trade, the regression is invisible beside a forward pass measured in milliseconds, and the pathological case this fixes is the one that actually hurt. I am recording it so the measured claim in the history is the honest one, and so nobody later reads "1.2x faster on 50-turn dicts" as something the repo verified.

If you want the small-state cost back, the cheap route is bypassing the generator when the state is a flat one-or-two-leaf mapping, the same shape of shortcut as your str fast path. Entirely optional, and not worth a follow-up unless you find it interesting.

@HamidRezaLS
HamidRezaLS deleted the perf/lang-lazy-state-text branch November 3, 2026 15:35
@HamidRezaLS

Copy link
Copy Markdown
Contributor Author

yeah the table numbers were the total sum across the 1000 iterations loop rather than divided per call... and also makes total sense on the yield from overhead on small dicts that dont hit the budget early too. thanks for the review and merge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants