Repository navigation
fix(reviewer): success=False when CharacterAnimationReviewer finds QA issues - #227
Conversation
…sues The contract test in test_character_animation_pipeline.py asserts result.success after the QA run, implying success should reflect whether QA passed. But CharacterAnimationReviewer always returned success=True even when issues were found and status was 'revise'. This creates a silent failure path: callers that gate on result.success (the standard ToolResult contract) would treat a broken rig as a clean pass without ever reading report['status']. Fix: return success=not issues so result.success is False when QA finds problems, consistent with the test contract and with how execution errors are already handled (success=False on exception paths). Also adds test_character_reviewer_success_false_when_qa_finds_issues to explicitly assert this contract — a broken rig with missing joints must produce status='revise', non-empty issues[], and success=False.
0xDevNinja
left a comment
There was a problem hiding this comment.
Thanks for picking this back up and engaging carefully with the #166 discussion — the underlying concern is real: the happy-path contract test asserts success and then status == "pass", and a dedicated negative-path test for the reviewer was genuinely missing. The new test_character_reviewer_success_false_when_qa_finds_issues covers an important gap.
But I don't think flipping success is the right fix, and I'd ask to redirect it. Three pieces of evidence from the current tree:
1. The cited precedent points the other way. tools/analysis/visual_qa.py is the closest sibling QA tool, and its _probe keeps success=True while surfacing the verdict in the data:
# tools/analysis/visual_qa.py:277-281
info["validation_issues"] = issues
info["validation_passed"] = len(issues) == 0
return ToolResult(
success=True,So "how visual_qa surfaces validation failures" is actually: success=True, failure in validation_passed. Adopting success=not issues here would make character_animation the only QA tool that diverges from that pattern.
2. The real consumer already gates on report.status, not success. The compose-director skill — the actual caller of this report — is explicit:
# skills/pipelines/character-animation/compose-director.md:48-49
Do not present the output as complete when `character_qa_report.status` is
`revise` or `fail`.
There is no Python caller that gates on this reviewer's .success (grep across lib/ and tools/ finds none). The documented contract is status-based.
3. Risk of the flip. Any caller using the common if result.success: <use data> else: <treat as failure> pattern would now discard a perfectly valid revise report — losing the recommended_action: fix_rig signal that is the entire value of a reviewer. That's the inverse of the bug we want to avoid.
Suggested redirect (keeps the good test, fixes the semantics):
- Leave
ToolResult(success=True, ...)— a successful review that finds issues is still a successful execution. - Change the new test to assert the status-based contract instead:
assert result.success is True # tool ran assert qa_report["status"] == "revise" # QA verdict assert len(qa_report["issues"]) > 0
That tests the real contract (broken rig => status revise + issues) without making success mean two different things across tools. Happy to look again once it's adjusted.
Per maintainer feedback on PR OpenMontageInc#227: - Revert success=not issues back to success=True — tool execution succeeded even when QA finds issues; verdict lives in status/issues - Update test to assert the real contract: success=True + status='revise' + issues non-empty, matching how compose-director actually gates - Consistent with visual_qa.py: success=True, verdict in validation_passed
|
Redirected as suggested — all three points addressed:
|
calesthio
left a comment
There was a problem hiding this comment.
Thanks for adjusting this after the earlier review. I checked the current branch and the semantics now match the status-based contract: CharacterAnimationReviewer.execute() still returns success=True for a completed review, while the QA verdict lives in character_qa_report.status / issues.
One cleanup before merge: the new test is still named test_character_reviewer_success_false_when_qa_finds_issues, and the PR title still says success=False, but the current test correctly asserts result.success is True. Please rename the test, and ideally update the PR title, so future readers do not take away the old contract by mistake.
Verification I ran:
python -m pytest tests/contracts/test_character_animation_pipeline.py -q-> 7 passedpython -m py_compile tools\character\character_animation.py-> passed
I did not find an actionable security issue in the reviewed diff.
calesthio
left a comment
There was a problem hiding this comment.
Approving from my side. The remaining rename/title mismatch is only a non-blocking cleanup note; the current code and test now assert the intended status-based contract.
|
Thanks @calesthio |
Per maintainer feedback on PR OpenMontageInc#227: - Revert success=not issues back to success=True — tool execution succeeded even when QA finds issues; verdict lives in status/issues - Update test to assert the real contract: success=True + status='revise' + issues non-empty, matching how compose-director actually gates - Consistent with visual_qa.py: success=True, verdict in validation_passed
…reviewer-success-contract fix(reviewer): success=False when CharacterAnimationReviewer finds QA issues
What
CharacterAnimationReviewer.execute()always returnedsuccess=Trueeven when the QA check found issues and setstatus: revise. This was flagged in #166 and discussed with @0xDevNinja.Why the previous fix was in the wrong place — and where the real contract is
@0xDevNinja correctly noted that the existing callers in
lib/source_media_review.pyuseresult.successto mean "safe to read the data", not "QA passed". That was right for those callers. But the contract test in this very repo tells a different story:The test asserts
result.successand then checksstatus == "pass"— which only makes sense ifsuccesstracks QA outcome. With the old code,successwas alwaysTrue, so the first assert was vacuous and the contract was untested.Fix
Two changes:
tools/character/character_animation.pytests/contracts/test_character_animation_pipeline.py— new testtest_character_reviewer_success_false_when_qa_finds_issuesthat explicitly asserts the contract: a broken rig (missing joints) must producestatus='revise', non-emptyissues[], andsuccess=False.Reproduction (before fix)
After fix:
result.successisFalsewhen issues are found.