Skip to content

fix(reviewer): success=False when CharacterAnimationReviewer finds QA issues - #227

Merged
calesthio merged 2 commits into
OpenMontageInc:mainfrom
kapil971390:fix/character-reviewer-success-contract
Jul 1, 2026
Merged

calesthio merged 2 commits into
OpenMontageInc:mainfrom
kapil971390:fix/character-reviewer-success-contract

Conversation

@kapil971390

Copy link
Copy Markdown

What

CharacterAnimationReviewer.execute() always returned success=True even when the QA check found issues and set status: revise. This was flagged in #166 and discussed with @0xDevNinja.

Why the previous fix was in the wrong place — and where the real contract is

@0xDevNinja correctly noted that the existing callers in lib/source_media_review.py use result.success to mean "safe to read the data", not "QA passed". That was right for those callers. But the contract test in this very repo tells a different story:

# tests/contracts/test_character_animation_pipeline.py
qa_result = CharacterAnimationReviewer().execute(...)
assert qa_result.success          # ← gates on success
qa_report = qa_result.data["character_qa_report"]
assert qa_report["status"] == "pass"  # ← then checks status

The test asserts result.success and then checks status == "pass" — which only makes sense if success tracks QA outcome. With the old code, success was always True, so the first assert was vacuous and the contract was untested.

Fix

Two changes:

tools/character/character_animation.py

# before
return ToolResult(success=True, ...)

# after
return ToolResult(success=not issues, ...)

tests/contracts/test_character_animation_pipeline.py — new test test_character_reviewer_success_false_when_qa_finds_issues that explicitly asserts the contract: a broken rig (missing joints) must produce status='revise', non-empty issues[], and success=False.

Reproduction (before fix)

from tools.character.character_animation import CharacterAnimationReviewer

result = CharacterAnimationReviewer().execute({
    "rig_plan": {"characters": [{"id": "char1", "role": "lead"}]},
    "pose_library": {},
    "action_timeline": {},
    "review_level": "static",
})

print(result.success)                               # True  ← wrong
print(result.data["character_qa_report"]["status"]) # revise
print(len(result.data["character_qa_report"]["issues"]))  # > 0

After fix: result.success is False when issues are found.

…sues

The contract test in test_character_animation_pipeline.py asserts
result.success after the QA run, implying success should reflect
whether QA passed. But CharacterAnimationReviewer always returned
success=True even when issues were found and status was 'revise'.

This creates a silent failure path: callers that gate on result.success
(the standard ToolResult contract) would treat a broken rig as a clean
pass without ever reading report['status'].

Fix: return success=not issues so result.success is False when QA finds
problems, consistent with the test contract and with how execution errors
are already handled (success=False on exception paths).

Also adds test_character_reviewer_success_false_when_qa_finds_issues to
explicitly assert this contract — a broken rig with missing joints must
produce status='revise', non-empty issues[], and success=False.
@kapil971390
kapil971390 requested a review from calesthio as a code owner June 29, 2026 13:45

@0xDevNinja 0xDevNinja left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this back up and engaging carefully with the #166 discussion — the underlying concern is real: the happy-path contract test asserts success and then status == "pass", and a dedicated negative-path test for the reviewer was genuinely missing. The new test_character_reviewer_success_false_when_qa_finds_issues covers an important gap.

But I don't think flipping success is the right fix, and I'd ask to redirect it. Three pieces of evidence from the current tree:

1. The cited precedent points the other way. tools/analysis/visual_qa.py is the closest sibling QA tool, and its _probe keeps success=True while surfacing the verdict in the data:

# tools/analysis/visual_qa.py:277-281
info["validation_issues"] = issues
info["validation_passed"] = len(issues) == 0
return ToolResult(
    success=True,

So "how visual_qa surfaces validation failures" is actually: success=True, failure in validation_passed. Adopting success=not issues here would make character_animation the only QA tool that diverges from that pattern.

2. The real consumer already gates on report.status, not success. The compose-director skill — the actual caller of this report — is explicit:

# skills/pipelines/character-animation/compose-director.md:48-49
Do not present the output as complete when `character_qa_report.status` is
`revise` or `fail`.

There is no Python caller that gates on this reviewer's .success (grep across lib/ and tools/ finds none). The documented contract is status-based.

3. Risk of the flip. Any caller using the common if result.success: <use data> else: <treat as failure> pattern would now discard a perfectly valid revise report — losing the recommended_action: fix_rig signal that is the entire value of a reviewer. That's the inverse of the bug we want to avoid.

Suggested redirect (keeps the good test, fixes the semantics):

  • Leave ToolResult(success=True, ...) — a successful review that finds issues is still a successful execution.
  • Change the new test to assert the status-based contract instead:
    assert result.success is True            # tool ran
    assert qa_report["status"] == "revise"   # QA verdict
    assert len(qa_report["issues"]) > 0

That tests the real contract (broken rig => status revise + issues) without making success mean two different things across tools. Happy to look again once it's adjusted.

Per maintainer feedback on PR OpenMontageInc#227:
- Revert success=not issues back to success=True — tool execution
  succeeded even when QA finds issues; verdict lives in status/issues
- Update test to assert the real contract: success=True + status='revise'
  + issues non-empty, matching how compose-director actually gates
- Consistent with visual_qa.py: success=True, verdict in validation_passed
@kapil971390

Copy link
Copy Markdown
Author

Redirected as suggested — all three points addressed:

  • character_animation.py: reverted to success=True; verdict stays in character_qa_report.status and issues, matching the visual_qa.py pattern
  • Test updated to assert the real contract: result.success is True, qa_report["status"] == "revise", len(qa_report["issues"]) > 0
  • All 7 contract tests pass locally

@calesthio calesthio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adjusting this after the earlier review. I checked the current branch and the semantics now match the status-based contract: CharacterAnimationReviewer.execute() still returns success=True for a completed review, while the QA verdict lives in character_qa_report.status / issues.

One cleanup before merge: the new test is still named test_character_reviewer_success_false_when_qa_finds_issues, and the PR title still says success=False, but the current test correctly asserts result.success is True. Please rename the test, and ideally update the PR title, so future readers do not take away the old contract by mistake.

Verification I ran:

  • python -m pytest tests/contracts/test_character_animation_pipeline.py -q -> 7 passed
  • python -m py_compile tools\character\character_animation.py -> passed

I did not find an actionable security issue in the reviewed diff.

@calesthio calesthio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving from my side. The remaining rename/title mismatch is only a non-blocking cleanup note; the current code and test now assert the intended status-based contract.

@calesthio
calesthio merged commit 1242f84 into OpenMontageInc:main Jul 1, 2026
1 check passed
@kapil971390

Copy link
Copy Markdown
Author

Thanks @calesthio

TonkaToyXL pushed a commit to TonkaToyXL/OpenMontage that referenced this pull request Jul 8, 2026
Per maintainer feedback on PR OpenMontageInc#227:
- Revert success=not issues back to success=True — tool execution
  succeeded even when QA finds issues; verdict lives in status/issues
- Update test to assert the real contract: success=True + status='revise'
  + issues non-empty, matching how compose-director actually gates
- Consistent with visual_qa.py: success=True, verdict in validation_passed
TonkaToyXL pushed a commit to TonkaToyXL/OpenMontage that referenced this pull request Jul 8, 2026
…reviewer-success-contract

fix(reviewer): success=False when CharacterAnimationReviewer finds QA issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants