Skip to content

docs(benchmarks): guide to results and known limits - #480

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:docs/benchmarks-guide
Sep 25, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
rahul05ranjan:docs/benchmarks-guide

Conversation

@rahul05ranjan

Copy link
Copy Markdown
Contributor

Summary

 docs/
+  benchmarks.md   # how to choose, read and reproduce published results
   index.md        # link the new guide from the docs home page

Adds the “Benchmarks and known limits” guide requested in #390. It points readers to the existing tables, scripts and raw results; distinguishes fine-tuned from base-checkpoint results and third-party Jev comparisons; and gives practical checks for language, option count, calibration, wording, long inputs and latency. The guide uses the repository's published evidence and introduces no new benchmark claim.

Evidence

  • Before: The docs home page sent readers to the README for benchmarks and known limits; there was no dedicated guide.
  • After: uvx --from zensical==0.0.64 --with mkdocstrings-python==2.0.9 --with ruff==0.16.8 zensical build --strict --clean — passed, “No issues found”.
  • python tests/test_doc_tables.py — 54 passed, 0 failed.
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 and python -m compileall -q laya/ tests/ — passed.
  • git diff --cached --check — passed.

Merge Danger

Door: two-way

Blast Radius: docs

Only the guide and its home-page link change. Benchmark refresh work remains tracked in #285.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, this is a really useful page. I checked each limit and figure against BENCHMARKS.md, the README and research/, and they all match, including the note about the calibration columns that predate the clamp. It gives readers one place to judge a result before they rely on it, without adding any new claims. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants