Repository navigation
fix: Arabic-aware routing and cross-workflow regression benchmarks - #175
Closed
jaganthoutam wants to merge 1 commit into
Closed
jaganthoutam wants to merge 1 commit into
jaganthoutam wants to merge 1 commit into
Conversation
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arabic text can reach the English checkpoint when it is a minority of an otherwise English message, or when only the question instructions/choice labels are Arabic. This change routes those requests to multilingual while preserving explicit model/task/language overrides and opt-in workflow precedence. Structured instructions now retain Unicode instead of passing escaped Arabic text to the tokenizer.
Adds an Arabic triage preset with stable output keys, plus evaluation beyond refunds: intent, sentiment, moderation, relevance, and urgency. Regression tests cover Arabic script variants, English controls, question labels, output contracts, metric calculations, and routing-only execution. CI and release checks run the new tests; source distributions include their fixtures.
Validation:
573e5b6on all 80 workflow fixtures. Saved reports include raw answers, labels, configuration, checkpoint revisions, and source hashes.Scope and measured limits:
See
BENCHMARKS.mdandresearch/README.mdfor per-task results, baselines, limitations, and reproduction commands. This PR fixes shared Arabic handling and makes remaining model gaps measurable; it does not claim that all Arabic quality issues are resolved.