Repository navigation
fix(calibrate): run records_from_labeled with gradients off - #832
Merged
NandhaKishorM merged 2 commits intoOct 2, 2026
Merged
NandhaKishorM merged 2 commits into
NandhaKishorM merged 2 commits into
Conversation
Agent._forward converts the logits to numpy and expects its caller to have turned gradients off. predict_batch does that, records_from_labeled did not, so it raised RuntimeError: Can't call numpy() on Tensor that requires grad on any loaded Agent.
The stand-in agents in test_calibrate.py return numpy from _forward, so they never reach the failing line. This one loads the tiny local checkpoint and runs the model's own forward pass.
4 tasks done
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
This was referenced Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
records_from_labelednow runs under@torch.no_grad(). This is a one line change inlaya/calibrate.py, plus a test intests/test_calibration_persistence.pythat calls it on a loaded agent.Why
records_from_labeledraisedRuntimeError: Can't call numpy() on Tensor that requires gradon any loadedAgent, so the path theAgent.fit_temperaturesdocstring points to for building calibration records did not work.Agent._forwardconverts the logits to numpy and expects its caller to have turned gradients off.predict_batchdoes that.records_from_labeleddid not.The script from #826 on the
convaiinnovations/layacheckpoint:How it was verified
tests/test_calibration_persistence.py: 7 tests, OK (6 on main). With thecalibrate.pychange removed: the new test fails with theRuntimeErrorabove.tests/test_calibrate.pyreturn numpy from_forward, so they could not catch this.tests/test_calibrate.py: 132 passed, 0 failed.testsjob runs, in a fresh Python 3.12 environment installed withpip install -e ".[mcp,serve,langchain]" httpx pytest: all passed.Checklist
mainFixes #826