Skip to content

Preserve capability intent when checking and saving a model connection - #433

Open
nillikechatchat wants to merge 4 commits into
higress-group:mainfrom
nillikechatchat:feat/403-capability-intent
Open

nillikechatchat wants to merge 4 commits into
higress-group:mainfrom
nillikechatchat:feat/403-capability-intent

Conversation

@nillikechatchat

@nillikechatchat nillikechatchat commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A newly observed upstream model ID can be selected, but editing and rechecking it replaced unknown optional capabilities with generic values labeled as user declarations. A same-ID catalog candidate could also replace a saved local row, and selection rows did not distinguish official catalog membership from a return by the current directory check. The result was a saved model that claimed facts the user never declared.

Result

Phase 1 of Design #411 keeps capability intent consistent across Check, Save, restart and saved-source recheck.

  • Saved local rows rebuild from their stored values and bases. A label-only edit keeps unknown optional capabilities, an explicit false stays known, and an exact same-ID catalog candidate no longer replaces the saved local row. Qualification and denial still apply by ID during candidate generation.
  • Registered prefill offers observed-only IDs with unknown optional capabilities instead of minting declarations.
  • Desktop keeps template and metadata prefill as editable suggestions in editor context, separate from the facts serialized by Check and Save. Only an explicit field edit or Apply suggestions writes UserDeclared; clearing a field yields unknown and is remembered so a later adoption does not refill it.
  • Selection rows report catalog membership separately from this check's directory observation: the exact upstream_model_id in observed_model_ids marks a return from this check, a completed reachable directory distinguishes Not returned by this check, and partial, unavailable, invalid or omitted directories use Not confirmed by this check.
  • The production model-connection journey now covers an observed-only eligible ID, label-only and explicit field edits including false, explicit adoption, partial directory and a per-model tool test.

Phase 1 keeps the released ComputeCandidateViewV2 and existing Check request/response shapes. Its deny_unknown_fields contract is preserved, so field-level projection, price and free qualification stay in TASK-411001.

Compatibility

  • Public candidate rows keep their compact row-level fact_basis summary; the public view projects one basis per model, so mixed local facts surface as unknown there while durable recheck compares decoded per-field values and bases.
  • Saved capability persistence uses the current representation. No new persisted field was added.

Validation

Executed in this checkout:

On a860a286:

  • scripts/test-contract-convergence.py: 23 tests passed
  • scripts/check-contract-convergence.py: green, 42 subjects and 37 legacy entries
  • python3 -m py_compile and import of the extended product script
  • cd apps/desktop && node --experimental-strip-types --test tests/*.test.mjs: 465 passed, 0 failed
  • cd apps/desktop && ./node_modules/.bin/tsc --noEmit: passed

On c87c03a7 and 94c657c8:

  • Local Rust toolchain 1.97.1: cargo check -p hiroute-daemon --lib --tests (0 errors) and cargo fmt --all --check pass.
  • CI on head 94c657c8: backend-shard (integration-daemon) and (integration-product), plus quality, gates, integration-foundation, integration-smoke, integration-gateway, frontend, DCO, scope, selection and check all pass.

Known gaps:

  • backend-shard (unit) fails on control::agent_removal::tests::shared_mutation_excludes_teardown_and_teardown_refuses_both_apply_entries (crates/daemon/src/control/agent_removal.rs:96, DaemonUnavailable vs InvalidArguments). This is a pre-existing failure on canonical main at bf99f0f6 with the same assertion and values, in a file absent from this branch and outside the changed paths. A maintainer rerun of backend-shard (unit) is requested to confirm the flake.
  • real_process_native_model_save_publishes_and_reaches_gateway was not executed locally because building target/debug/hirouted exhausted the available disk; the integration shards cover this path in CI.
  • Desktop browser/visual acceptance needs Chrome, which is unavailable here.

Docs

  • issue-spec/specs/model-connections/spec.md: added the suggestion/declaration requirement and extended the saved-source recheck requirement.
  • apps/desktop/src/features/model-connections/README.md: updated for the suggestion and observation semantics.

Closes #432
Refs #411
Refs #403

… save

Reconstruct saved local capability facts from stored values and bases so a
label-only edit keeps unknown optional capabilities, explicit values survive
restart, and an exact same-ID catalog candidate no longer replaces the saved
local row.

Separate editor suggestions from the facts serialized by Check and Save in the
Desktop model-connection flow. Suggestions stay in editor context, explicit
field edits and Apply suggestions declare UserDeclared values, clearing yields
unknown, and selection rows report catalog membership separately from this
check's directory observation.

Extend the production model-connection product journey with a controlled
directory covering an observed-only eligible ID, label-only and explicit field
edits including false, explicit adoption, partial directory and a per-model
tool test.

Signed-off-by: nillikechatchat <yuanhenglizhen2050@163.com>
Co-authored-by: monkeycode-ai <monkeycode-ai@chaitin.com>
nillikechatchat and others added 3 commits November 9, 2026 03:18
Replace the closure that used an inferred generic placeholder with a generic
function so one helper checks bool, u64 and native reasoning facts without a
type mismatch in the daemon test build.

Signed-off-by: nillikechatchat <yuanhenglizhen2050@163.com>
Co-authored-by: monkeycode-ai <monkeycode-ai@chaitin.com>
Wrap the multi-ID test loop and the membership comparison so `cargo fmt
--check` passes on the daemon crate.

Signed-off-by: nillikechatchat <yuanhenglizhen2050@163.com>
Co-authored-by: monkeycode-ai <monkeycode-ai@chaitin.com>
The capability-intent product scenario asserted that an inference-only
Check reports an available directory and observed model IDs. That is not
what the product does: an inference-only Check (inference_model_id set)
short-circuits before the directory probe, records a missing-credential
observation with no issues, and therefore reports directory `not_run`
with no observed IDs. The assertion encoded an incorrect expectation
rather than a product defect.

Assert the actual contract instead: directory stays `not_run`, no model
ID is observed, no directory GET is issued, and the single POST probe
carries the selected model plus tools. This also proves the directory
result is independent of the inference evidence, which is the property
the scenario is meant to demonstrate.

Signed-off-by: nillikechatchat <yuanhenglizhen2050@163.com>
Co-authored-by: monkeycode-ai <monkeycode-ai@chaitin.com>

@johnlanni johnlanni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 94c657c8 against Proposal #403 and Phase 1 of Design #411.


/** Suggestions stay in the editor; the wire model carries only the draft's declared facts. */
export function toWireModel(model: ModelDeclaration): Omit<ModelDeclaration, 'client_id' | 'suggested_capabilities'> {
const { client_id: _clientId, suggested_capabilities: _suggestions, ...wire } = model;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep editor-only state out of Check requests.

toWireModel forwards cleared_capabilities, including [] after a regular field edit. Both Check paths use this serializer, and the shared NativeUserModelDeclarationV1 rejects unknown fields. I captured requests from the production form after editing a field, clearing one, and applying suggestions. The v0.6.0 decoder returned INVALID_ARGUMENTS for all three. Removing only that field let the requests reach transport.

Serialize the wire fields explicitly and exercise these three actions through the strict decoder for both registered and custom Check requests.

suggestions: ModelCapabilities | undefined,
): ModelDeclaration {
if (!suggestions) return model;
return model.suggested_capabilities ? model : { ...model, suggested_capabilities: suggestions };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the selected template's suggestions.

blankModel already sets generic suggestions, so withSuggestions discards the supplied capability_prefill. With the bundled Bailian Token Plan data, the qwen3.8-flash template offers 1,000,000 context tokens and 131,072 output tokens, while the editor displays 200,000 and 32,768. Applying those values would record the generic defaults as user declarations.

Use the applicable template suggestions when constructing the draft, and verify what the editor displays and adopts with a bundled template.

const canSelect = modelCanBeSelected(result, model, true);
return <div className="model-result-row" key={model.model_ref}><label className="check-row"><input type="checkbox" data-autofocus={index === 0 || undefined} disabled={!canSelect} checked={canSelect && selected.has(model.model_ref)} onChange={event => { const next = new Set(selected); if (event.target.checked) next.add(model.model_ref); else next.delete(model.model_ref); selectionRef.current = { result, refs: next }; setSelected(next); setError(''); }} /><div><strong>{model.display_name}</strong><span>{draft.display_name}{model.fact_basis === 'runtime_fallback' ? ` · ${zh ? '运行时保守兜底' : 'Conservative runtime fallback'}` : ''}{!canSelect ? ` · ${modelAvailabilityMessage(model.reason, zh)}` : ''}</span></div></label>{custom && !canSelect && <button className="btn btn-quiet" type="button" onClick={() => {
const observation = modelObservationMessage(result, model, zh);
return <div className="model-result-row" key={model.model_ref}><label className="check-row"><input type="checkbox" data-autofocus={index === 0 || undefined} disabled={!canSelect} checked={canSelect && selected.has(model.model_ref)} onChange={event => { const next = new Set(selected); if (event.target.checked) next.add(model.model_ref); else next.delete(model.model_ref); selectionRef.current = { result, refs: next }; setSelected(next); setError(''); }} /><div><strong>{model.display_name}</strong><span>{observation ? `${observation} · ` : ''}{draft.display_name}{model.fact_basis === 'runtime_fallback' ? ` · ${zh ? '运行时保守兜底' : 'Conservative runtime fallback'}` : ''}{!canSelect ? ` · ${modelAvailabilityMessage(model.reason, zh)}` : ''}</span></div></label>{custom && !canSelect && <button className="btn btn-quiet" type="button" onClick={() => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Show official catalog match alongside directory observation.

This row renders the current directory observation without using membership. In a partial-directory browser fixture, the catalog-matched row and the observed-only row both showed “Returned by this check” with no indication of their different origins. A row with a local display name also hid its exact upstream ID.

Render the catalog match from membership independently of the observation label, and show the upstream ID when the label differs. This is part of the Phase 1 selection contract in #411.

] as const).map(([key, label]) => <label className="field" key={key}><span className="field-label">{label}</span><input className="input" type="number" min="1" value={reasoning[key]} onChange={event => updateCapability('native_reasoning', { value: { ...reasoning, [key]: Number(event.target.value) }, basis: 'user_declared' })} /></label>)}
</div>}
</Disclosure>
{hasSuggestions && <div className="manual-capability-suggestions"><button className="btn btn-quiet" type="button" onClick={() => onChange(applySuggestedCapabilities(model))}>{zh ? '采纳建议值' : 'Apply suggested values'}</button><span className="field-help">{zh ? '建议值仅填充仍为“未知”的能力,不会覆盖你的填写。' : 'Suggestions fill only values still marked Unknown and never overwrite your edits.'}</span></div>}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Show the suggestions before they can be adopted.

I could click “Apply suggested values” with “Capabilities and limits” collapsed, adopting values I had not seen. Phase 1 calls for explicit adoption of the displayed suggestions. Put the action alongside the visible values and indicate which fields still contain pending suggestions. The regression should also confirm that adoption preserves fields the user edited or cleared.

model: Pick<CandidateModelView, 'upstream_model_id'>,
zh: boolean,
): string | null {
const observed = new Set(result.observed_model_ids ?? []);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Build the observed ID set once per Check result.

The list calls this helper for each candidate on each render, including when the user toggles a selection. At 1,000 returned IDs and 1,000 candidate rows, that is about one million set insertions per render. In a local measurement, label computation alone took about 19 ms per pass at that size. Reuse the set across rows and selection changes.

local_model('field-edit-model', 'Field Edit Model', edited),
local_model('adopted-model', 'Adopted Model', adopted_caps),
]
draft = native_check_draft(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exercise the same observed-only ID through registered Save and recheck.

The directory step proves that gpt-5.4-mini is selectable. The save step then uses three different, manually declared IDs on a custom connection. TASK-411004 calls for the registered production path. Use one eligible observed-only ID throughout Check, label edit, Save, restart, and saved-source recheck. Verify the stored and reconstructed values and bases, including unknown fields and an explicit false.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Implement] Preserve capability intent when checking and saving a model connection

2 participants