Skip to content

fix(reviewer): return success=False when QA finds issues - #166

Closed
kapil971390 wants to merge 1 commit into
OpenMontageInc:mainfrom
kapil971390:fix/reviewer-success-on-qa-failure
Closed

kapil971390 wants to merge 1 commit into
OpenMontageInc:mainfrom
kapil971390:fix/reviewer-success-on-qa-failure

Conversation

@kapil971390

Copy link
Copy Markdown

CharacterAnimationReviewer.execute() always returned success=True even when the QA check found issues — so the compose-director had no way to know the rig was broken.

Reproduction

from tools.character.character_animation import CharacterAnimationReviewer

result = CharacterAnimationReviewer().execute({
    "rig_plan": {"characters": [{"id": "char1", "role": "lead"}]},  # missing joints
    "pose_library": {},
    "action_timeline": {},
    "review_level": "static",
})

print(result.success)                               # True  ← wrong
print(result.data["character_qa_report"]["status"]) # revise

Output before fix:

result.success          : True
report['status']        : revise
report['issues'] count  : 5
issues[0]               : Preview path is required for character animation QA.

Expected : result.success == False  (QA found issues)
Actual   : result.success == True   <- BUG

Fix

One-line change in CharacterAnimationReviewer.execute():

# before
return ToolResult(success=True, ...)

# after
return ToolResult(success=not issues, ...)

After fix:

With issues → success: False | status: revise

The success field now correctly reflects whether QA passed, so downstream agents and compose-director can gate on it.

@kapil971390
kapil971390 requested a review from calesthio as a code owner June 24, 2026 14:39

@0xDevNinja 0xDevNinja left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the underlying concern is valid (a QA run that finds issues shouldn't look like a clean pass). But I think flipping success is the wrong lever, and it diverges from how the rest of the codebase treats QA results.

ToolResult.success means "the tool executed correctly", not "QA passed."

  • The sibling QA tool follows this: tools/analysis/visual_qa.py builds validation_issues and still returns success=True (around L277-281), reserving success=False for actual execution errors (L126, L138, L140).
  • The QA verdict is already surfaced in the report itself: status is set to "pass"/"revise" at tools/character/character_animation.py:868, and the findings are in issues[] (L883).
  • The pipeline already gates on that status, not on success: skills/meta/reviewer.md L291-306 makes its decisions off final_review.status (revise/pass), not ToolResult.success.

Regression risk: several callers use if result.success: to mean "it ran, safe to read the data" (e.g. lib/source_media_review.py:50, 96, 201). With this change, a normal revise outcome returns success=False, so that pattern would treat a perfectly valid QA report as a failed run and skip it.

Suggested change: revert to success=True. If the problem is that the pipeline isn't stopping when QA returns revise, the fix belongs in the consumer/skill that should be honoring report["status"] — not in overloading success.

Happy to be corrected if there's a specific consumer that reads success as the QA verdict here — if so, can you point to it?

@kapil971390

Copy link
Copy Markdown
Author

Thanks for the detailed review @0xDevNinja — you're right on all counts.

I traced through the call chain more carefully:

  • skills/meta/reviewer.md L291-306 gates on final_review.status, not success
  • visual_qa.py consistently uses success=True for clean runs regardless of validation outcome
  • The if result.success: pattern in lib/source_media_review.py means "safe to read data", not "QA passed"

Flipping success here would break that contract and risk silently dropping valid QA reports in those callers. The real fix, if anything goes wrong downstream, belongs in the consumer that isn't honoring status: revise — not here.

Closing this PR. Thanks for taking the time to walk through the codebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants