Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
fix: route Arabic inputs and questions correctly; add cross-workflow …
…benchmarks
  • Loading branch information
jaganthoutam committed Sep 22, 2026
commit 101f14be31907c157063fcc953840e068b80389e
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ jobs:
- name: Routing and language-detection tests
run: |
python tests/test_router.py
python tests/test_arabic.py
python tests/test_criteria.py
python tests/test_download.py
python tests/test_shortlist.py
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ jobs:
- name: Tests must pass before anything is published
run: |
python tests/test_router.py
python tests/test_arabic.py
python tests/test_criteria.py
python tests/test_email.py
python tests/test_download.py
Expand Down
73 changes: 73 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,79 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).

## Languages

### Arabic regression suite (local CPU)

The Arabic suite uses 36 AI-authored synthetic messages: 12 parallel scenarios in
standard Arabic, Emirati Arabic, and mixed Arabic-English. Each message is tested
with Arabic and English question wording (72 evaluations, 216 typed decisions).
The examples and labels have **not been native-speaker reviewed**. These small,
related samples measure regression behavior, not general Arabic proficiency.

Measured on 2026-09-22 using the multilingual checkpoint at Hugging Face revision
`1c5edc17a7acd8701df6fc341c0d179f1c62c982`, on macOS arm64 CPU:

| Question language | Message group | Category accuracy | Refund accuracy | Urgency MAE (0–2, lower better) |
|---|---|---:|---:|---:|
| Arabic | Standard Arabic | 91.7% | 91.7% | 0.586 |
| Arabic | Emirati Arabic | 83.3% | 83.3% | 0.618 |
| Arabic | Mixed | 83.3% | 58.3% | 0.727 |
| English | Standard Arabic | 75.0% | 75.0% | 0.700 |
| English | Emirati Arabic | 75.0% | 75.0% | 0.607 |
| English | Mixed | 83.3% | 75.0% | 0.618 |

All 72 evaluations routed to multilingual without a locale override. Routing
success is separate from prediction accuracy. The category majority baseline is
25%; the refund majority baseline is 75%; always predicting urgency 1 gives MAE
0.667. **Arabic-question refund detection on mixed text falls below baseline.**
Urgency prediction is also weak. Arabic wording improves category accuracy on
this corpus, but does not consistently improve every decision type.

The [raw report](research/results/arabic_triage_cpu.json) contains all predictions,
labels, calibration metrics, warm latency, configuration, and source hashes.
See the [runner and reproduction commands](research/README.md#arabic-regression-benchmark).
No weights or temperatures were fitted to these cases.

### Cross-workflow Arabic routing comparison

80 synthetic messages cover intent, sentiment, moderation, relevance, and urgency
in standard Arabic, Emirati, mixed text, and English controls. Each is evaluated
with Arabic and English question wording under two routing policies. There are
160 decisions per policy; the predictions are reused where both policies select
the same checkpoint. This is a routing comparison using identical weights and
the current inference runtime, not a retrained model comparison.

| Question language | Task | Original routing | Arabic-aware routing |
|---|---|---:|---:|
| Arabic | Intent accuracy | 100% | 100% |
| Arabic | Sentiment accuracy | 87.5% | 100% |
| Arabic | Moderation accuracy | 81.3% | 87.5% |
| Arabic | Relevance accuracy | 93.8% | 87.5% |
| Arabic | Urgency MAE (lower better) | 0.640 | 0.553 |
| English | Intent accuracy | 93.8% | 93.8% |
| English | Sentiment accuracy | 93.8% | 100% |
| English | Moderation accuracy | 93.8% | 93.8% |
| English | Relevance accuracy | 93.8% | 87.5% |
| English | Urgency MAE (lower better) | 0.580 | 0.641 |

Each row aggregates 16 cases, including English controls. The expected routing
policy matches 100% of cases after the fix versus 50% with Arabic questions and
75% with English questions before it. **Prediction quality does not improve
uniformly:** relevance regresses under both question languages, and urgency error
increases with English questions. The routing fix ensures Arabic reaches the
multilingual model, but does not resolve all model-quality gaps. Do not infer a
general quality gain from routing accuracy. No workflow-specific checkpoint
selection was fitted to these results.

This corpus was authored after a small triage prompt exploration; it was not
used for prompt selection or training. It is still synthetic, unreviewed, and
too small to qualify production behavior (four cases per task/message group).
Native-speaker-reviewed, representative evaluation and separate training data
are needed for broader model improvement.

[Raw paired results](research/results/arabic_workflows_cpu.json) include every
answer, per-group metrics and baselines, checkpoint revisions, and source hashes.
[Reproduction commands](research/README.md#cross-workflow-arabic-comparison).

### All 51 MASSIVE languages — intent, 20 options (random = 0.050)

| | laya | laya-multilingual |
Expand Down
5 changes: 5 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Keep the Arabic tests and their evaluation fixtures runnable from an sdist.
include research/data/arabic_triage.json
include research/data/arabic_workflows.json
include research/scripts/bench_arabic.py
include research/scripts/bench_arabic_workflows.py
54 changes: 54 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,60 @@ router.route({"body": "Der Kunde wurde zweimal belastet"}, questions).reason
# "Latin script but language looks like 'de', not English"
```

### UAE Arabic and mixed Arabic–English

Emirati Arabic written in Arabic script uses the existing multilingual checkpoint.
Mixed Arabic–English messages also route there when Arabic-script letters appear
in the inspected state, even if most of the message is English.

```python
from laya import Router, arabic_triage_questions

router = Router()
questions = arabic_triage_questions()

result = router.predict(
{"message": "أبا أسترجع المبلغ، انخصم مني مرتين"},
questions,
lang="ar-AE", # Use the known locale, or omit it for script-based routing.
)

decision = router.route(
{"message": "Please check invoice 4411 and refund the duplicate payment, أباه اليوم"},
questions,
)
assert decision.model == "multilingual"
```

Detection inspects up to 4,000 characters of string values in the state; JSON keys
are ignored. Arabic-script letters trigger multilingual routing, while Arabic
digits and punctuation alone do not. If the state would use the English model,
the router also inspects up to 4,000 characters of serialized question instructions
and criteria, including choice labels. Arabic question text selects multilingual
even for English input; question IDs do not affect detection.
Explicit `model`, `task`, or `lang` settings
and opt-in workflow detection retain their existing routing precedence.

This is routing support using existing weights, not a newly trained Emirati
checkpoint or a claim of validated dialect accuracy. The detector identifies
script, not UAE dialect. Evaluate predictions and confidence on representative
Emirati data before relying on them. Automatic Arabizi detection is not included.

`arabic_triage_questions()` provides Arabic instructions and rubrics for `category`,
`refund_requested`, and `urgency`. Category values remain stable English keys
(`billing`, `technical`, `delivery`, `general`), and urgency is an expected score
from 0 to 2. The existing English presets are unchanged.

Run the [Arabic regression benchmark](research/README.md#arabic-regression-benchmark)
to evaluate standard Arabic, Emirati Arabic, and mixed messages with both Arabic
and English question wording. Its synthetic examples are not a substitute for
native-speaker-reviewed evaluation data.

The broader [workflow comparison](research/README.md#cross-workflow-arabic-comparison)
covers intent, sentiment, moderation, relevance, and urgency, comparing the
original state-only routing policy with Arabic-aware routing. Model quality still
varies by task; correct routing does not guarantee a correct decision.

### Why Route: The Evidence

On a shared benchmark (17,416 questions, one T4 GPU, identical questions per model):
Expand Down
2 changes: 2 additions & 0 deletions laya/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from .lang import analyse as detect_language
from .lang import detect_script, is_english
from .presets import (
arabic_triage_questions,
email_questions,
guard_questions,
moderation_questions,
Expand All @@ -40,6 +41,7 @@
"clean_email_body",
"email_questions",
"email_state",
"arabic_triage_questions",
"guard_questions",
"moderation_questions",
"router_questions",
Expand Down
2 changes: 1 addition & 1 deletion laya/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ def _to_internal(qdef: Dict) -> Dict:
crit = {c: None for c in crit}
ins = qdef["instructions"]
if not isinstance(ins, str):
ins = json.dumps(ins)
ins = json.dumps(ins, ensure_ascii=False)
return {"t": t, "ins": ins, "crit": crit}

@torch.no_grad()
Expand Down
5 changes: 4 additions & 1 deletion laya/lang.py
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,10 @@ def analyse(state: Union[str, dict, list, None]) -> Dict[str, object]:
return {"script": "unknown", "script_profile": prof, "language": None,
"is_english": True, "language_undecided": True, "diacritic_rate": 0.0,
"non_latin_fraction": 0.0}
if script != "latin":
# UAE messages often mix English with short Arabic requests. Even a minority
# of Arabic-script letters needs the multilingual encoder. This detects a
# script, not an Arabic dialect (other languages also use Arabic script).
if script != "latin" or prof.get("arabic", 0.0) > 0:
return {"script": script, "script_profile": prof, "language": None,
"is_english": False, "language_undecided": True, "diacritic_rate": 0.0,
"non_latin_fraction": non_latin}
Expand Down
34 changes: 34 additions & 0 deletions laya/presets.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,40 @@
from typing import Dict, Optional


def arabic_triage_questions() -> Dict:
"""Arabic support-triage rubric with stable English output keys.

Supports Arabic-script input, including Emirati and mixed Arabic-English,
through the multilingual checkpoint. This preset does not tune model weights.
"""
return {
"category": {
"type": "choice",
"instructions": "ما القسم الأنسب للتعامل مع طلب العميل في message؟",
"criteria": {
"billing": "الفواتير والدفعات واسترداد المبالغ",
"technical": "الأعطال التقنية ومشاكل تسجيل الدخول والتطبيق",
"delivery": "توصيل الطلبات والشحن وتتبع الشحنات",
"general": "الاستفسارات العامة وساعات العمل ومعلومات الخدمات",
},
},
"refund_requested": {
"type": "noul",
"instructions": "هل يطلب العميل صراحة استرداد مبلغ مالي في message؟",
"criteria": {"false": "لا يطلب استرداد مبلغ", "true": "يطلب استرداد مبلغ"},
},
"urgency": {
"type": "score",
"instructions": "ما مدى استعجال الطلب في message؟",
"criteria": [
"غير مستعجل ويمكن الانتظار",
"يحتاج إلى متابعة قريبة أو خلال اليوم دون موعد نهائي محدد",
"مشكلة تمنع العمل أو موعد نهائي صريح لا يمكن تأجيله",
],
},
}


def triage_questions() -> Dict:
"""Preset questions for customer support ticket triage."""
return {
Expand Down
20 changes: 20 additions & 0 deletions laya/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
`auto_task_detection=True` or pass `task="typed_decisions"`: it is fine-tuned on four specific
synthetic workflows and should not be a silent default.
"""
import json
import os
import threading
from typing import Any, Dict, List, Optional, Union
Expand Down Expand Up @@ -296,6 +297,9 @@ def route(
key = "multilingual"
reason = "non-Latin script (%s, %.0f%% of letters); the English checkpoint cannot read it" % (
det["script"], 100 * float(det["non_latin_fraction"]))
elif det["script_profile"].get("arabic", 0.0) > 0:
key = "multilingual"
reason = "mixed Latin and Arabic script; Arabic-script content needs the multilingual checkpoint"
elif not det["is_english"]:
key = "multilingual"
if det["language"]:
Expand All @@ -308,6 +312,22 @@ def route(
else:
key = "english"
reason = "English Latin text"
# Questions are part of the encoder input too. English state text does
# not make Arabic instructions or choice labels readable by that model.
# Ignore question IDs, but include criteria keys because they are labels.
if key == "english" and questions:
question_text = json.dumps(
[{"instructions": q.get("instructions", ""), "criteria": q.get("criteria")}
for q in questions.values() if isinstance(q, dict)],
ensure_ascii=False, default=str,
)
question_detection = analyse(question_text)
if question_detection["script_profile"].get("arabic", 0.0) > 0:
return RouteDecision(
model="multilingual", repo=_repo_str(self.models["multilingual"]),
reason="Arabic-script question text needs the multilingual checkpoint",
detection=det, question_detection=question_detection, workflow=workflow,
)
return RouteDecision(model=key, repo=_repo_str(self.models[key]), reason=reason,
detection=det, workflow=workflow)

Expand Down
85 changes: 85 additions & 0 deletions research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,91 @@ checkpoint fine-tuned on that benchmark's own training split.
**Speed.** 32.8 ms for one question and 7.2 ms/question at batch 10 on a T4; 103–332 questions/s
batched.

## Arabic regression benchmark

`scripts/bench_arabic.py` evaluates the public `Router.predict` API on
[`data/arabic_triage.json`](data/arabic_triage.json): 12 parallel support scenarios
in each of standard Arabic (`msa`), Emirati Arabic (`emirati`), and mixed
Arabic-English (`mixed`). Each message has category, refund-request, and urgency
labels. Both Arabic and English question wording run by default, giving 72
evaluations and 216 typed decisions. No checkpoint is fine-tuned by this script.

The corpus is AI-authored, synthetic, and not native-speaker reviewed. Scenarios
across groups are related, not independent samples. Results are smoke/regression
measurements, not production qualification or a representative Arabic benchmark.
Do not use these cases for training or temperature fitting. Add a separate,
reviewed held-out dataset before making dialect-quality claims.

From the repository root, after installing Laya:

```bash
# Routing only: no model downloads or prediction-accuracy claims.
python research/scripts/bench_arabic.py --routing-only \
--output research/results/arabic_routing.json

# Real inference: downloads the multilingual checkpoint on first use.
python research/scripts/bench_arabic.py --device cpu \
--output research/results/arabic_triage_cpu.json

# Or supply an existing local multilingual checkpoint and a single question language.
python research/scripts/bench_arabic.py --model-path /path/to/multilingual \
--question-language ar --output /tmp/arabic.json

# Offline regressions, also included in CI.
python tests/test_arabic.py
```

Reports contain raw predictions and expected labels, routing accuracy, category
accuracy, refund accuracy and binary Brier score, urgency mean absolute error
and nearest-level accuracy, warm per-message p50/p95 latency, and majority/constant
baselines. Every metric is split by message group and question language. ECE uses
the predicted-label probability, **not** Laya's entropy-based confidence; with
only 12 cases per group it is descriptive and unstable. Urgency levels round
half-up. Latency includes all three questions per message and excludes model
loading and one warm-up per question language. Failures abort without writing a
partial success report. Metadata records the corpus hash, code revision and dirty
state, package versions, checkpoint configuration, and actual device.

`lang="ar-AE"` is available for applications with a known locale. The benchmark
deliberately omits that hint so it also checks automatic Arabic/mixed routing.

## Cross-workflow Arabic comparison

`scripts/bench_arabic_workflows.py` extends evaluation beyond refunds and triage.
[`data/arabic_workflows.json`](data/arabic_workflows.json) contains 20 scenarios
across intent, sentiment, moderation, relevance, and urgency. Each has standard
Arabic, Emirati, mixed Arabic-English, and English-control versions: 80 messages,
tested with Arabic and English questions (160 decisions per routing policy).

```bash
python research/scripts/bench_arabic_workflows.py --routing-only \
--output /tmp/arabic_workflow_routing.json
python research/scripts/bench_arabic_workflows.py --device cpu \
--output research/results/arabic_workflows_cpu.json
```

Optional `--english-path` and `--multilingual-path` select local checkpoint
directories. Inference holds both checkpoints in memory. Model loading is
excluded from the recorded evaluation duration, which is not a latency benchmark.

The original policy is the upstream default behavior at `573e5b6`: classify the
state's dominant script, use the Latin-language heuristic if needed, and ignore
question text. Its reconstruction was checked against that upstream router on
all 80 fixtures. The current policy additionally handles minority Arabic in the
state and Arabic question text. Explicit overrides and workflow-detection modes
are outside this comparison. Both paths use the same current inference runtime;
predictions are reused where the selected checkpoint is identical.

The report includes every answer and compares accuracy, Brier score, ECE, ordinal
error, and simple baselines per workflow, message group, and question language.
No overall quality score combines these different tasks. Routing-only mode emits
no prediction-quality metrics. This corpus was created after exploratory prompt
tests on the separate triage corpus; its question sets were not selected by
testing variants on these cases. **Neither corpus is native-speaker-reviewed or
representative**, and related language versions are not independent samples.
Per-group cells have only four cases. This is a regression comparison, not proof
of general Arabic capability. It does not train weights or fit temperatures.

## On comparisons with Jev

There is no TypeSafe API credential in this project, so **Jev was never run here**. Every Jev
Expand Down
Loading