Repository navigation
feat: add torch.compile support and ONNX Runtime inference engine - #215
Conversation
|
Flagging this before it merges, because it looks green and would break
] # end of dependencies
[project.optional-dependencies] # <- this PR
onnx = ["onnx", "onnxruntime"]
[tool.setuptools]
...
[project.optional-dependencies] # <- already on main, from #31
serve = [
"fastapi>=0.110.0",
"uvicorn>=0.27.0",
]Declaring a table twice is a TOML error, not a merge conflict, so the two hunks do not touch and GitHub reports this PR as MERGEABLE with no conflict to resolve: That parse happens before anything is installed, so if this lands the The fix is one line: put [project.optional-dependencies]
serve = [
"fastapi>=0.110.0",
"uvicorn>=0.27.0",
]
onnx = ["onnx", "onnxruntime"]Worth doing at the same time, since it is the same rebase: the PR's I have not run |
0927df6 to
1f4b28d
Compare
|
Good catch @PerryLink, thanks! Rebased onto latest main and merged the onnx extra into the existing [project.optional-dependencies] table. Should be clean now. |
# Conflicts: # laya/agent.py # pyproject.toml
|
Thank you, and thanks for rebasing after @PerryLink's catch on the duplicate optional-dependencies table. I exported the English checkpoint with your script and compared |
…ndhaKishorM#215) Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
This PR implements two major performance enhancements for the
DecisionModelruntime: optionaltorch.compilesupport for GPU inference, and a completely new ONNX export/inference pipeline for fast CPU/edge deployments.1. Optional
torch.compileSupportThe
Agentclass now accepts acompile: bool = Falseargument. While eager mode remains the default to prevent graph-compilation hangs on unsupported platforms or small batch sizes, advanced users can now explicitly enabletorch.compilefor a massive latency reduction on GPUs.Benchmarks (NVIDIA Tesla T4 x2)
Benchmarked using 100 iterations (after 10 warmup runs).
2. ONNX Export & Runtime Pipeline
Since Laya is a non-autoregressive BERT-based model, it is highly amenable to ONNX export. We've added a full pipeline to strip away the PyTorch overhead entirely for CPU deployments.
scripts/export_onnx.py: A new CLI tool to trace the model and export it to a dynamic.onnxgraph (opset 18).laya/onnx_agent.py(ONNXAgent): A drop-in replacement forAgentthat usesonnxruntime.InferenceSession. It perfectly mirrors thepredictandsystem_oneAPI.tests/test_onnx.py: A comprehensive parity test verifying that the PyTorch and ONNX models output numerically identical probabilities (down to 1e-3 tolerance).pyproject.toml: Added an optional[onnx]dependency group.Verification
torch.compilespeedups on T4 architecture.pytest tests/test_onnx.py).