Skip to content

perf(lang): optimize unicode script detection with binary search - #1349

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/lang-bisect-script-detection
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HamidRezaLS:perf/lang-bisect-script-detection

Conversation

@HamidRezaLS

Copy link
Copy Markdown
Contributor

What

in laya/lang.py:

  • flattens _SCRIPT_RANGES into sorted interval arrays at module import (_STARTS, _ENDS, _SCRIPTS).
  • replaces the linear generator scan in _script_counts and _script_of with $O(\log K)$ binary search via bisect.bisect_right.
  • keeps the ASCII fast-path (cp < 0x0250) untouched.

Why

in _script_counts and _script_of, detecting non-ASCII scripts previously iterated through 25 script definitions across 39 disjoint intervals, evaluating any(lo <= cp <= hi for lo, hi in ranges) per character. for scripts defined near the end of the list (like Han, Hiragana, Katakana, Hangul), this evaluated up to 25 generator expressions per character.

replacing linear scans with binary search over sorted static intervals finds the matching script in at most 5 comparisons in C ($O(\log K)$ vs $O(K)$ ), with 0 mismatches verified across all 1,114,112 Unicode codepoints (0 to 0x10FFFF).

tested on _script_counts (isolated 3,000 iterations):

  • Chinese (Han): 731.9 ms → 24.4 ms (30.0x)
  • Japanese (Kana+Han): 977.4 ms → 32.0 ms (30.6x)
  • Arabic: 334.3 ms → 43.2 ms (7.7x)
  • English (ASCII): 11.9 ms → 11.6 ms (zero regression)

tested on Router.route_batch before and after:

Workload Batch Size Before After Speedup Latency Delta
Multilingual / CJK 16 19.74 ms 2.52 ms 7.8x -17.22 ms
64 73.57 ms 8.80 ms 8.4x -64.77 ms
128 135.87 ms 18.25 ms 7.4x -117.62 ms
Mixed Real-World 16 11.57 ms 3.64 ms 3.2x -7.93 ms
64 46.87 ms 14.95 ms 3.1x -31.92 ms
128 90.65 ms 29.83 ms 3.0x -60.81 ms
Pure ASCII / English 128 21.38 ms 19.92 ms 1.1x -1.46 ms

How it was verified

python tests/test_doc_citations.py
python tests/test_lang_stats.py
python tests/test_router.py
python tests/test_hooks.py
python tests/test_hooks_api.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

all passed:

  • exhaustive test across all 1,114,112 Unicode codepoints (0 to 0x10FFFF): 0 mismatches against original linear scan
  • test_doc_citations.py: 22 passed, 0 failed
  • test_lang_stats.py: 38 passed, 0 failed
  • test_router.py: 837 passed, 0 failed
  • test_hooks.py: 308 passed, 0 failed
  • test_hooks_api.py: 645 passed, 0 failed
  • ruff and compilation checks: clean

@NandhaKishorM
NandhaKishorM merged commit a5ddd97 into NandhaKishorM:main Oct 24, 2026
48 of 52 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants