Skip to content

fix: Arabic-aware routing and cross-workflow regression benchmarks - #175

Closed
jaganthoutam wants to merge 1 commit into
NandhaKishorM:mainfrom
jaganthoutam:feature/arabic-support-benchmarks
Closed

jaganthoutam wants to merge 1 commit into
NandhaKishorM:mainfrom
jaganthoutam:feature/arabic-support-benchmarks

Conversation

@jaganthoutam

Copy link
Copy Markdown

Arabic text can reach the English checkpoint when it is a minority of an otherwise English message, or when only the question instructions/choice labels are Arabic. This change routes those requests to multilingual while preserving explicit model/task/language overrides and opt-in workflow precedence. Structured instructions now retain Unicode instead of passing escaped Arabic text to the tokenizer.

Adds an Arabic triage preset with stable output keys, plus evaluation beyond refunds: intent, sentiment, moderation, relevance, and urgency. Regression tests cover Arabic script variants, English controls, question labels, output contracts, metric calculations, and routing-only execution. CI and release checks run the new tests; source distributions include their fixtures.

Validation:

  • All eight local test suites passed: 327 checks, including 12 Arabic tests.
  • Ruff and diff checks passed; wheel/sdist build succeeded; Arabic tests passed from the extracted sdist.
  • Real CPU inference: 72 triage evaluations (216 typed decisions), plus 160 cross-workflow decisions under each of the original and new routing policies. Both paths use identical checkpoints and the current inference runtime; predictions are reused when routing agrees.
  • Verified the original-policy reconstruction against upstream 573e5b6 on all 80 workflow fixtures. Saved reports include raw answers, labels, configuration, checkpoint revisions, and source hashes.

Scope and measured limits:

  • Cross-workflow expected routing is 100% after the fix, versus 50% with Arabic questions and 75% with English questions before it.
  • With Arabic questions, sentiment accuracy improves from 87.5% to 100%, moderation from 81.3% to 87.5%, and urgency MAE from 0.640 to 0.553. Intent stays at 100%.
  • Quality does not improve uniformly: relevance drops from 93.8% to 87.5% under both question languages, and English-question urgency MAE increases from 0.580 to 0.641. Mixed-text refund prediction remains weak in the separate triage evaluation.
  • These are small AI-authored synthetic corpora, not native-speaker-reviewed or representative Arabic qualification sets. Parallel language variants are related samples. No weights or temperatures were trained/fitted, and no task-specific routing was selected from these results. Overall Arabic model quality still needs separate evaluation and model work.

See BENCHMARKS.md and research/README.md for per-task results, baselines, limitations, and reproduction commands. This PR fixes shared Arabic handling and makes remaining model gaps measurable; it does not claim that all Arabic quality issues are resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant