Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Record documented GPT-6.2 API reasoning levels
Use the OpenAI model index for exact Sol and Astra efforts. Keep the model-specific default unresolved and validate both registered OpenAI protocols. Require model-level reasoning evidence and consumer checks in the metadata maintenance workflow.

Signed-off-by: 澄潭 <zty98751@alibaba-inc.com>
  • Loading branch information
johnlanni committed Nov 20, 2026
commit 80903f857ec71c3d0906479becac915df5b2cbe9
3 changes: 3 additions & 0 deletions .agents/skills/model-metadata-maintainer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Maintain `assets/model-data/current/inputs/metadata-catalog.json` as HiRoute's s
- product/model/upstream/protocol facts on `endpoint_bindings`;
- non-stable routers on `dynamic_routes`.
5. Preserve uncertainty explicitly. Do not replace `unknown`, `conflict`, `conditional`, or `runtime-required` with a guessed boolean or number. A provider-scoped field may only stop being `unknown` through step 8's reviewed, labeled closure — never by writing an unlabeled value into the record.
For new models or reasoning updates, check the official model index or comparison table, the individual model page, and the protocol documentation. Verify reasoning support, the full set of supported efforts, the model-specific default, and the parameter for each affected product and protocol as separate facts. Record the sources checked and any unresolved field in the change discussion. A rejected effort constrains the supported set.
6. Add or update normalized `evidence_sources`, then reference them by `source_key`. When authenticated bytes are captured, store a SHA-256 digest; never store credentials, tokens, headers, account IDs, or response bodies containing secrets.
7. Update `as_of` only for facts actually reviewed. Do not imply that untouched providers were refreshed.
8. Close the catalog: `tools/model-metadata/close_model_metadata.py` rewrites every provider-scoped record so no closable field stays `unknown` — it either keeps an observed assertion, assigns an inferred value from `tools/model-metadata/inference-rules.json` (recording `{basis: inferred, rule_key}`), or leaves an explicit `unsupported`/`not-applicable`. When a new or changed fact needs a rule, add or amend the rule there instead of hand-writing a value; a rule that stops matching is discarded and re-closed, never left stale.
Expand Down Expand Up @@ -109,6 +110,8 @@ node apps/website/scripts/model-feed.mjs check

`close_model_metadata.py` owns the closure rules and writes the source catalog in place; `--check` fails when the catalog is not a fixed point. Run it before validation, and run it twice when it changed a record: the second run must produce no diff. Then run the current generators and feed builder twice; their second run must produce no diff. The feed check uses shared Rust parsing and admission against recorded supported release bundles; Node checks only complement it. Pass the intended PR/push baseline to `feed build --check --base BASE` so committed semantic changes are checked as well as local regeneration. First publication does not require a stable release or verified development package; HTTPS and the independent production model-feed pipeline establish trust without a signing-key lifecycle. The validator checks entity keys, references, state vocabularies, source authorization, token-null semantics, endpoint relationships, field-derived completeness and `usable_for`, provenance, the inference-rule table and per-field provenance, the closure fixed point, dynamic-route attribution, and a deterministic canonical digest. Structural failures are red; stale sources are reported separately and require refresh or an explicit runtime/excluded disposition.

For reasoning changes that affect API configuration, verify the emitted bundle and feed, then exercise the production connection prefill for each affected product and protocol. Assert the exact supported levels, parameter mapping, and default state against the evidence. [Registered API prefill tests](../../../apps/desktop/tests/registered-metadata.test.mjs) exercise this path with fixture data.

For a changed discovery profile or scanner, scan its target commit twice and compare the outputs byte-for-byte. Also run the scanner's self-tests:

```bash
Expand Down
6 changes: 5 additions & 1 deletion apps/desktop/tests/registered-metadata.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ function ids(id, modifiedOption = null) {

for (const modelId of ['gpt-6.2-astra', 'gpt-6.2-sol']) {
for (const protocol of ['chat_completions', 'responses']) {
test(`GPT-6.2 ${modelId}/${protocol} prefills the input cap and retains manual edits on refresh`, async () => {
test(`GPT-6.2 ${modelId}/${protocol} prefills limits and reasoning profiles and keeps manual edits`, async () => {
const { withRegisteredModels, buildCheckDraft } = await import('../src/features/model-connections/state.ts');
const openai = option('openai.platform.global.v1');
const endpoint = openai.endpoints.find(value => value.protocol === protocol);
Expand All @@ -35,12 +35,16 @@ for (const modelId of ['gpt-6.2-astra', 'gpt-6.2-sol']) {
assert.ok(candidate);
assert.deepEqual(candidate.capability_prefill.context_tokens, { value: 922_000, basis: 'user_declared' });
assert.deepEqual(candidate.capability_prefill.max_output_tokens, { value: 128_000, basis: 'user_declared' });
const reasoning = { kind: 'discrete', parameter: 'reasoning_effort', profiles: ['low', 'medium', 'high', 'xhigh', 'max'] };
assert.deepEqual(candidate.capability_prefill.native_reasoning, { value: reasoning, basis: 'user_declared' });
const models = withRegisteredModels([], [candidate]);
const automatic = buildCheckDraft({ provenance: { kind: 'user_configured', configuration_revision: 1 }, models });
assert.equal(automatic.models[0].capabilities.context_tokens.value, 922_000);
assert.deepEqual(automatic.models[0].capabilities.native_reasoning, candidate.capability_prefill.native_reasoning);
const edited = { ...models[0], capabilities: { ...models[0].capabilities,
context_tokens: { value: 800_000, basis: 'user_declared' },
max_output_tokens: { value: 64_000, basis: 'user_declared' },
native_reasoning: { value: { ...reasoning, default_profile: 'low' }, basis: 'user_declared' },
} };
const refreshed = withRegisteredModels([edited], [candidate]);
const manual = buildCheckDraft({ provenance: { kind: 'user_configured', configuration_revision: 1 }, models: refreshed });
Expand Down
4 changes: 2 additions & 2 deletions assets/model-data/current/inputs/client-discovery-runs.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"artifact_kind": "hiroute.model-metadata-client-discovery-runs",
"as_of": "2026-11-18",
"catalog_digest_compared": "sha256:dbeac5f9ce25d32e0b8e891478d70397e2e0bf29a1690b654f03d0a6db05ded6",
"as_of": "2026-11-20",
"catalog_digest_compared": "sha256:87236224902dbdef108a22511b238f4449b948e65a7415bb36d1e24755426e1d",
"policy": {
"candidate_outputs_materialized": true,
"metadata_records_materialized": true,
Expand Down
37 changes: 31 additions & 6 deletions assets/model-data/current/inputs/metadata-catalog.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"artifact_kind": "hiroute.model-metadata-current-source",
"as_of": "2026-11-18",
"as_of": "2026-11-20",
"scope": {
"catalog_generation": "current",
"delivery": "client-bundled",
Expand Down Expand Up @@ -2538,6 +2538,15 @@
"bytes_digest": "sha256:dae79e0da343370b181a1662b01da4017413b82df5702da4b30f3b86b147d40c",
"digest_state": "captured"
},
{
"source_key": "source-openai-gpt-6-2-model-index-20261120",
"authority": "developers.openai.com",
"source_kind": "provider-documentation",
"locator": "https://developers.openai.com/api/docs/models",
"collected_on": "2026-11-20",
"bytes_digest": "sha256:ccb0e7a0ecedefb58ff5c525c84b970b8af388aaea4ddc34d3f55b0a14afe8fb",
"digest_state": "captured"
},
{
"source_key": "source-openai-gpt-6-2-sol-20261118",
"authority": "developers.openai.com",
Expand Down Expand Up @@ -5693,15 +5702,23 @@
"cache": "supported"
},
"reasoning": {
"kind": "model-documented",
"note": "Reasoning is supported and none is rejected on both APIs. The migration guide recommends low or medium; it does not establish the complete accepted effort set or provider default. Leave render profiles and default unresolved."
"kind": "discrete",
"profiles": [
"low",
"medium",
"high",
"xhigh",
"max"
],
"note": "The OpenAI model index lists these efforts. Both APIs reject none. Responses uses reasoning.effort and Chat Completions uses reasoning_effort. The model-specific default remains unconfirmed."
},
"identity_stability": "unknown",
"lifecycle": "active",
"source_stability_note": "direct-provider-model-documentation; snapshot permanence not established",
"evidence_refs": [
"source-openai-gpt-6-2-astra-20261118",
"source-openai-gpt-6-2-migration-20261118"
"source-openai-gpt-6-2-migration-20261118",
"source-openai-gpt-6-2-model-index-20261120"
],
"data_note": "Total context and independent input/output limits are distinct. Tools require Responses; Chat Completions supports text generation without tools. API documentation does not establish Codex or ChatGPT subscription access. Prices retain Standard, Batch and Fast conditions; Ultrafast, background completion windows, regional processing and tool fees must not inherit these rates."
},
Expand Down Expand Up @@ -5734,14 +5751,22 @@
"cache": "supported"
},
"reasoning": {
"kind": "model-documented",
"note": "Reasoning is supported and none is rejected on both APIs. The migration guide recommends low or medium; it does not establish the complete accepted effort set or provider default. Leave render profiles and default unresolved."
"kind": "discrete",
"profiles": [
"low",
"medium",
"high",
"xhigh",
"max"
],
"note": "The OpenAI model index lists these efforts. Both APIs reject none. Responses uses reasoning.effort and Chat Completions uses reasoning_effort. The model-specific default remains unconfirmed."
},
"identity_stability": "unknown",
"lifecycle": "active",
"source_stability_note": "direct-provider-model-documentation; snapshot permanence not established",
"evidence_refs": [
"source-openai-gpt-6-2-migration-20261118",
"source-openai-gpt-6-2-model-index-20261120",
"source-openai-gpt-6-2-sol-20261118"
],
"data_note": "Total context and independent input/output limits are distinct. Tools require Responses; Chat Completions supports text generation without tools. API documentation does not establish Codex or ChatGPT subscription access. Prices retain Standard, Batch and Fast conditions; Ultrafast, background completion windows, regional processing and tool fees must not inherit these rates."
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"schema": "hiroute.official-model-data-revision/v1",
"data_revision": 20
"data_revision": 21
}
4 changes: 2 additions & 2 deletions assets/model-data/current/maintenance.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"source_bundle": "assets/model-data/current/rating-seed.json",
"source_bytes_digest": "sha256:2f3b02526a9263e756c3a94e4d74336aca36724abd984fbec4a0a89833619b9f",
"metadata_catalog": "assets/model-data/current/inputs/metadata-catalog.json",
"metadata_catalog_bytes_digest": "sha256:616fcd30b9e1420c393672dcf9d4eddad0d9fa7b128110d1420f8909ad087b6f",
"metadata_catalog_bytes_digest": "sha256:f303998a7a6ba18a2335cf9bc656228cad6c42e84312e561c99317b0221b455b",
"client_discovery_runs": "assets/model-data/current/inputs/client-discovery-runs.json",
"client_discovery_runs_bytes_digest": "sha256:9e82b5708b48111d0af9072e027899f5d94448304acd976269e9a550a7672921",
"client_discovery_runs_bytes_digest": "sha256:997dfcfd068b8003ae0c78f2a96138d732e4915b39b4368a80f8aa94e7cd8f96",
"snapshot_version": "hiroute-20260913/current-1",
"method_revision": "preserved-explicit-native/v1",
"evidence": {
Expand Down
66 changes: 33 additions & 33 deletions assets/model-data/current/model-data.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion assets/model-data/current/runtime-projection.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion assets/release-facts/current/bundle/manifest.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"schema":"hiroute.release-facts/v2","tool_version":"hiroute-release-facts/2","catalog_id":"client-bundled/current","product_release":"mvp-current","sequence":1,"connector_registry_digest":"sha256:9195b73a8b648feb8bbd6397723809f002a4bed7f663ea991d317727bf002a71","model_data_digest":"sha256:fcc4c633d398ece56b0eb5b79ce0bedae621bde448ec2841c60ff784d71a0296","cross_reference_digest":"sha256:b145e53329747cfff81ac61c24e8f381598a972e38cbfce7625f6cf29728f184"}
{"schema":"hiroute.release-facts/v2","tool_version":"hiroute-release-facts/2","catalog_id":"client-bundled/current","product_release":"mvp-current","sequence":1,"connector_registry_digest":"sha256:9195b73a8b648feb8bbd6397723809f002a4bed7f663ea991d317727bf002a71","model_data_digest":"sha256:bb1ed299d8b93c7ab0120a9aff007f2cbb457aee985fb3df30ceb0fabc3f7372","cross_reference_digest":"sha256:5e2717db3fcbfbaf575271552b18d77e74fc3429c8e6ffe601f8a3e63bff5b36"}
66 changes: 33 additions & 33 deletions assets/release-facts/current/bundle/model-data.json

Large diffs are not rendered by default.

68 changes: 34 additions & 34 deletions assets/release-facts/current/bundle/official-model-data.json

Large diffs are not rendered by default.

43 changes: 33 additions & 10 deletions assets/release-facts/current/compiler-input.json
Original file line number Diff line number Diff line change
Expand Up @@ -4231,7 +4231,7 @@
"restriction": "General platform credentials"
}
],
"as_of": "2026-11-18",
"as_of": "2026-11-20",
"canonical_models": [
{
"canonical_identity": "claude-fable-5-1",
Expand Down Expand Up @@ -6214,7 +6214,8 @@
"display_name": "GPT-6.2 Astra",
"evidence_refs": [
"source-openai-gpt-6-2-astra-20261118",
"source-openai-gpt-6-2-migration-20261118"
"source-openai-gpt-6-2-migration-20261118",
"source-openai-gpt-6-2-model-index-20261120"
],
"identity_stability": "unknown",
"lifecycle": "active",
Expand All @@ -6237,9 +6238,15 @@
"publisher": "OpenAI",
"reasoning": {
"default": null,
"kind": "model-documented",
"note": "Reasoning is supported and none is rejected on both APIs. The migration guide recommends low or medium; it does not establish the complete accepted effort set or provider default. Leave render profiles and default unresolved.",
"profiles": []
"kind": "discrete",
"note": "The OpenAI model index lists these efforts. Both APIs reject none. Responses uses reasoning.effort and Chat Completions uses reasoning_effort. The model-specific default remains unconfirmed.",
"profiles": [
"low",
"medium",
"high",
"xhigh",
"max"
]
},
"source_stability_note": "direct-provider-model-documentation; snapshot permanence not established",
"upstream_id_state": null,
Expand Down Expand Up @@ -6267,6 +6274,7 @@
"display_name": "GPT-6.2 Sol",
"evidence_refs": [
"source-openai-gpt-6-2-migration-20261118",
"source-openai-gpt-6-2-model-index-20261120",
"source-openai-gpt-6-2-sol-20261118"
],
"identity_stability": "unknown",
Expand All @@ -6290,9 +6298,15 @@
"publisher": "OpenAI",
"reasoning": {
"default": null,
"kind": "model-documented",
"note": "Reasoning is supported and none is rejected on both APIs. The migration guide recommends low or medium; it does not establish the complete accepted effort set or provider default. Leave render profiles and default unresolved.",
"profiles": []
"kind": "discrete",
"note": "The OpenAI model index lists these efforts. Both APIs reject none. Responses uses reasoning.effort and Chat Completions uses reasoning_effort. The model-specific default remains unconfirmed.",
"profiles": [
"low",
"medium",
"high",
"xhigh",
"max"
]
},
"source_stability_note": "direct-provider-model-documentation; snapshot permanence not established",
"upstream_id_state": null,
Expand Down Expand Up @@ -14440,6 +14454,15 @@
"source_key": "source-openai-gpt-6-2-migration-20261118",
"source_kind": "provider-documentation"
},
{
"authority": "developers.openai.com",
"bytes_digest": "sha256:ccb0e7a0ecedefb58ff5c525c84b970b8af388aaea4ddc34d3f55b0a14afe8fb",
"collected_on": "2026-11-20",
"digest_state": "captured",
"locator": "https://developers.openai.com/api/docs/models",
"source_key": "source-openai-gpt-6-2-model-index-20261120",
"source_kind": "provider-documentation"
},
{
"authority": "developers.openai.com",
"bytes_digest": "sha256:3c1aa626387fc854be2560f16620f0c913f4a2aa97e8208d4239efd3279fe913",
Expand Down Expand Up @@ -107708,7 +107731,7 @@
}
],
"schema": "hiroute.model-metadata-catalog/v1",
"source_catalog_digest": "sha256:dbeac5f9ce25d32e0b8e891478d70397e2e0bf29a1690b654f03d0a6db05ded6"
"source_catalog_digest": "sha256:87236224902dbdef108a22511b238f4449b948e65a7415bb36d1e24755426e1d"
},
"rating_snapshot": {
"digest": "sha256:3903b9b5f5b0c6111d7dd2e752d9538390d32e72950c3a0edee8d025449f4919",
Expand Down Expand Up @@ -109843,7 +109866,7 @@
"schema": "hiroute.release-model-data/v2"
},
"official_model_data": {
"data_revision": 20,
"data_revision": 21,
"pricing_facts": [
{
"pricing_key": "anthropic-platform/claude-fable-5-2/claude-fable-5-2",
Expand Down
10 changes: 9 additions & 1 deletion crates/daemon/tests/support/model_input_limits_product.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Real form prefill -> protected check/save -> published Gateway input/output budgets."""
"""Verify form prefill, protected check/save and published Gateway budgets."""
import json
import os
from pathlib import Path
Expand All @@ -9,6 +9,12 @@
from recommended_model_product import Provider, TrialProduct
from standalone_headless_product import apply_from_preview, apply_plan, plan_change, wire

EXPECTED_REASONING = {
'value': {'kind': 'discrete', 'parameter': 'reasoning_effort',
'profiles': ['low', 'medium', 'high', 'xhigh', 'max']},
'basis': 'user_declared',
}


def frontend(repository, value):
bridge = repository / 'apps/desktop/tests/registered-input-limits-fixture.mjs'
Expand Down Expand Up @@ -51,6 +57,7 @@ def run(repository):
for case in cases:
stage = case['key']
draft = case['draft']
assert draft['models'][0]['capabilities']['native_reasoning'] == EXPECTED_REASONING, stage
protected = {'candidate_ref': draft['candidate_ref'], 'candidate_revision': 1}
product.register_secret(draft['candidate_ref'], 'isolated-input-limits-key')
product.secrets.add('isolated-input-limits-key')
Expand All @@ -74,6 +81,7 @@ def run(repository):
expected_output = 64_000 if case['manual'] else 128_000
assert saved['capabilities']['context_tokens'] == {'value': expected_input, 'basis': 'user_declared'}, saved
assert saved['capabilities']['max_output_tokens'] == {'value': expected_output, 'basis': 'user_declared'}, saved
assert saved['capabilities']['native_reasoning'] == EXPECTED_REASONING, saved
# Use the same authoring policy as the form/CLI, including its reasoning defaults.
authoring = product.cli('routing', 'options', '--request-stdin', payload={'authoring': {
'display_name': stage, 'purpose': 'Verify exact input and output budgets',
Expand Down
Loading