Skip to content

feat: add torch.compile support and ONNX Runtime inference engine - #215

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Swish78:feat/perf-optimizations
Sep 23, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Swish78:feat/perf-optimizations

Conversation

@Swish78

@Swish78 Swish78 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

This PR implements two major performance enhancements for the DecisionModel runtime: optional torch.compile support for GPU inference, and a completely new ONNX export/inference pipeline for fast CPU/edge deployments.

1. Optional torch.compile Support

The Agent class now accepts a compile: bool = False argument. While eager mode remains the default to prevent graph-compilation hangs on unsupported platforms or small batch sizes, advanced users can now explicitly enable torch.compile for a massive latency reduction on GPUs.

Benchmarks (NVIDIA Tesla T4 x2)

Benchmarked using 100 iterations (after 10 warmup runs).

  • Eager Mode: 45.31 ms / req
  • Compiled Mode: 33.75 ms / req
  • Speedup: 25.5%

2. ONNX Export & Runtime Pipeline

Since Laya is a non-autoregressive BERT-based model, it is highly amenable to ONNX export. We've added a full pipeline to strip away the PyTorch overhead entirely for CPU deployments.

  • scripts/export_onnx.py: A new CLI tool to trace the model and export it to a dynamic .onnx graph (opset 18).
  • laya/onnx_agent.py (ONNXAgent): A drop-in replacement for Agent that uses onnxruntime.InferenceSession. It perfectly mirrors the predict and system_one API.
  • tests/test_onnx.py: A comprehensive parity test verifying that the PyTorch and ONNX models output numerically identical probabilities (down to 1e-3 tolerance).
  • pyproject.toml: Added an optional [onnx] dependency group.

Verification

  • Verified torch.compile speedups on T4 architecture.
  • Verified PyTorch vs ONNX mathematical parity (pytest tests/test_onnx.py).
  • Tested fallback logic for OOMs and unsupported devices.

@PerryLink

Copy link
Copy Markdown
Contributor

Flagging this before it merges, because it looks green and would break main.

main gained its own [project.optional-dependencies] table today (pyproject.toml:39, the serve extra from #31). This PR's base predates that, so its hunk anchors on the end of the dependencies array and inserts a second table with the same name:

]                                          # end of dependencies

[project.optional-dependencies]            # <- this PR
onnx = ["onnx", "onnxruntime"]

[tool.setuptools]
...
[project.optional-dependencies]            # <- already on main, from #31
serve = [
    "fastapi>=0.110.0",
    "uvicorn>=0.27.0",
]

Declaring a table twice is a TOML error, not a merge conflict, so the two hunks do not touch and GitHub reports this PR as MERGEABLE with no conflict to resolve:

$ git diff -> clean merge, no conflict
$ python -c "import tomllib; tomllib.load(open('pyproject.toml','rb'))"
tomllib.TOMLDecodeError: Cannot declare ('project', 'optional-dependencies') twice (at line 42, column 31)

That parse happens before anything is installed, so if this lands the test job fails at its pip install -e . step and the package job fails at python -m build — both required checks — on a tree that currently passes everything.

The fix is one line: put onnx inside the existing table rather than opening a new one.

[project.optional-dependencies]
serve = [
    "fastapi>=0.110.0",
    "uvicorn>=0.27.0",
]
onnx = ["onnx", "onnxruntime"]

Worth doing at the same time, since it is the same rebase: the PR's compile parameter reaches Agent(...) as a new keyword, and Agent.__init__ is positional-or-keyword in a documented API — compile=False is fine as a default, just noting it is a public signature change for the changelog. Rebased onto today's main, self.model.encoder.config.reference_compile = compile also interacts with #195, which landed no_init_weights on this same code path and is what makes loading fast now.

I have not run tests/test_onnx.py (the onnx extra is not installed here), so I am not commenting on the ONNX path itself — only on the packaging state of the diff.

@Swish78
Swish78 force-pushed the feat/perf-optimizations branch from 0927df6 to 1f4b28d Compare September 23, 2026 11:18
@Swish78

Swish78 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Good catch @PerryLink, thanks! Rebased onto latest main and merged the onnx extra into the existing [project.optional-dependencies] table. Should be clean now.

# Conflicts:
#	laya/agent.py
#	pyproject.toml
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and thanks for rebasing after @PerryLink's catch on the duplicate optional-dependencies table. I exported the English checkpoint with your script and compared ONNXAgent with Agent on short and long inputs (up to 973 tokens across two questions). The answers and confidences are identical. Merging. One follow-up: ONNX Runtime prints shape warnings because the trace uses batch size 1, so act_logits gets a fixed batch dimension. Tracing with a batch of 2 would silence them. There are now two exporters (yours and #243's split one), so it would be nice to converge on one later. #25 landed a fast=True option on the same constructor today, so I resolved the conflict on your branch by keeping both, and made fast take precedence when both are set, since the TileLang path replaces the forward itself. I also checked compile=True on CPU: identical answers to eager on six states.

@NandhaKishorM
NandhaKishorM merged commit 6096540 into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…ndhaKishorM#215)

Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants