Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions .github/workflows/evals.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Evals

# The real-checkpoint quality gate. It downloads weights and the MASSIVE dataset, so it is kept
# out of PR CI: it runs weekly, before a release, and on demand, and compares the fresh numbers
# out of PR CI: it runs weekly, after publication, and on demand, and compares the fresh numbers
# to the committed baseline in research/results/. A failure is a regression in the published
# numbers, not a blocked PR.
on:
Expand All @@ -10,6 +10,19 @@ on:
release:
types: [published]
workflow_dispatch:
inputs:
release_candidate:
description: "Require a pinned, complete MASSIVE English report"
type: boolean
default: false
model_revision:
description: "Full checkpoint commit SHA for a release candidate"
type: string
required: false
dataset_revision:
description: "Full MASSIVE dataset commit SHA for a release candidate"
type: string
required: false

permissions:
contents: read
Expand Down Expand Up @@ -52,9 +65,20 @@ jobs:
pip install -e ".[onnx,serve]" datasets httpx pytest

- name: Evaluate the English checkpoint on MASSIVE English
env:
RELEASE_CANDIDATE: ${{ inputs.release_candidate }}
MODEL_REVISION: ${{ inputs.model_revision }}
DATASET_REVISION: ${{ inputs.dataset_revision }}
run: |
python research/eval/laya_eval.py \
--model convaiinnovations/laya --langs en --out report.json
if [ "$RELEASE_CANDIDATE" = "true" ]; then
python research/eval/laya_eval.py \
--model convaiinnovations/laya --langs en --device cpu --out report.json \
--model-revision "$MODEL_REVISION" --dataset-revision "$DATASET_REVISION" \
--require-complete
else
python research/eval/laya_eval.py \
--model convaiinnovations/laya --langs en --out report.json
fi

- name: Compare to the committed baseline
run: |
Expand Down
15 changes: 15 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).
|---|---|---|
| T4 Colab | typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair | `research/results/t4_colab_benchmark.json` |
| CPU sweep | MASSIVE intent across **all 51 languages**; its typed-decisions part (`part_b`) covers the English checkpoint only | `research/results/cpu_51_language_sweep.json` |
| English CPU release check | MASSIVE `en`, 100 test rows, 20 options, pinned checkpoint and dataset, per-case predictions | `research/results/massive_en_cpu_release_20261001.json` |
| Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |
| NVIDIA capacity | `laya` served with dynamic batching under p99 SLOs on RTX PRO 5000, RTX PRO 6000 and H100 NVL (eager FP16, TensorRT FP16 via ONNX Runtime), backend parity, 24-hour replay | `research/results/nvidia_capacity_20260925.json` |

Expand Down Expand Up @@ -52,6 +53,20 @@ Headline Laya cells for AG News and DAIR Emotion are the Applications-run number

## Languages

### English CPU release check (2026-10-01 UTC)

The [pinned report](research/results/massive_en_cpu_release_20261001.json) measures the English checkpoint on the first 100 MASSIVE `en` test rows, with 20 options and seed 13, on a Windows CPU using laya 0.3.23. It records all 100 decisions and the ordered input fingerprint (`a6363b6c…`). The code revision is `9bf2b51`, checkpoint revision `55cf4c4`, and dataset revision `940fd47`; the full commits and runtime versions are in the JSON.

| metric | committed English baseline | pinned CPU run |
|---|---:|---:|
| accuracy | 0.8200 | 0.8200 |
| macro F1 | 0.7876 | 0.7876 |
| ECE | 0.1382 | 0.1382 |
| mean confidence | 0.9582 | 0.9582 |
| accuracy at 50% coverage | 0.9800 | 0.9800 |

This is the English CPU regression protocol used by the Evals workflow. The historical baseline has no input fingerprint, so matching numbers alone cannot prove that both runs scored identical inputs. This run does not re-measure the T4 headline or the 51-language sweep below.

### All 51 MASSIVE languages — intent, 20 options (random = 0.050)

| | laya | laya-multilingual |
Expand Down
59 changes: 54 additions & 5 deletions research/eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,17 @@ and a JSON document with four parts:

| key | contents |
|---|---|
| `config` | checkpoint, device, `max_len`, `head_max_len`, dataset, `per_lang`, `n_opts`, seed, the fixed instructions, the temperatures in force, laya version |
| `report` | per language: `n`, `accuracy`, `macro_f1`, `ece`, `mean_confidence`, `acc_at_50_coverage`, `temperature` |
| `config` | checkpoint, resolved model revision, requested dataset revision, code revision, UTC run time, device, library versions, `max_len`, `head_max_len`, dataset, `per_lang`, `n_opts`, seed, fixed instructions, temperatures, laya version |
| `report` | per language: `n`, `accuracy`, `macro_f1`, `ece`, `mean_confidence`, `acc_at_50_coverage`, `temperature`, input SHA-256 |
| `summary` | macro accuracy / ECE / macro-F1 over the languages that ran |
| `cases` | every individual decision |

Each case carries `state`, `instructions`, `options`, `gold_index`, `gold_label`,
Each case carries `state`, `instructions`, ordered option keys (`options`) and
their criteria descriptions (`option_texts`), `gold_index`, `gold_label`,
`pred_index`, `pred_label`, `probability`, `p_gold`, `confidence`, `correct` and the
`temperature` used. That is enough to re-derive every number in `report` from the
`temperature` used. Probabilities are saved at full precision, so rounding does
not change ECE bin membership or the confidence ordering used for coverage.
That is enough to re-derive every number in `report` from the
file alone, with no model and no network:

```python
Expand All @@ -78,6 +81,52 @@ acc = sum(c["correct"] for c in d["cases"]) / n
assert abs(acc - d["report"]["en"]["accuracy"]) < 5e-5
```

## English release-candidate run

The [Evals workflow](../../.github/workflows/evals.yml) still runs the English CPU
check weekly and after a release is published. Before publication, use **Run workflow**
on the candidate code ref, select `release_candidate`, and supply full 40-character
commits for `convaiinnovations/laya` and `mteb/amazon_massive_intent`. The manual
candidate path requires no tracked checkout edits, pinned inputs, exactly 100 scored English
cases and all 100 per-case records. A partial run fails, but its JSON is uploaded
for diagnosis when one was written.

Candidate validation checks case fields, label/index consistency and finite
probabilities, then recomputes all five metrics from the saved cases with the
same functions used by the harness. Missing, non-finite or inconsistent metrics
fail the candidate rather than entering the numeric comparison as release evidence.

The equivalent local command, from a checkout without tracked edits and with
`datasets` installed, is:

```bash
python research/eval/laya_eval.py --model convaiinnovations/laya \
--langs en --device cpu --per-lang 100 --n-opts 20 \
--model-revision <full-checkpoint-sha> \
--dataset-revision <full-dataset-sha> \
--require-complete --out report.json
```

`report.en.input_sha256` identifies the ordered states, instructions, option keys,
criteria descriptions and gold positions actually scored. The head renders each
choice from its key and description; `option_texts` stores the description, not
that combined text or its tokenized, potentially truncated form. The hash excludes predictions
and timing, so two checkpoint revisions can be compared on the same inputs. The
resolved checkpoint revision and exact runtime versions remain in `config`.
With a dataset revision supplied, the harness downloads that revision's
`test/<language>.json.gz` directly through `huggingface_hub`. This avoids the
`datasets` offline fallback selecting a different cached revision. Unpinned
runs keep using the existing dataset loader.
The existing `check_regression.py` compares numbers with the historical baseline;
that baseline has no input fingerprint, so a pass there does **not** establish
identical inputs. Review the new raw JSON before committing it under
`research/results/` and citing its measured scope from `BENCHMARKS.md`.

The [2026-10-01 English CPU run](../results/massive_en_cpu_release_20261001.json)
is a committed example with pinned inputs and all 100 decisions. Its accuracy,
ECE and mean confidence match the existing numeric baseline; the report records
the exact environment and input fingerprint for future comparisons.

## Method

Chosen so results are comparable with the published tables, which is the point of a
Expand Down Expand Up @@ -166,7 +215,7 @@ forcing 256 or 512 drops it to 0.79.
checkpoint, no network:

```bash
python research/eval/test_laya_eval.py # 64 passed, 0 failed
python research/eval/test_laya_eval.py # 158 passed, 0 failed
```

It pins the upstream constants (seed 13, 20 options, the exact instruction string),
Expand Down
Loading
Loading