Skip to content

fix(ida): preserve native coverage and verify returned values - #2324

Merged
N0zoM1z0 merged 3 commits into
mainfrom
fix/ida-native-verification-oracles
Oct 13, 2026
Merged

N0zoM1z0 merged 3 commits into
mainfrom
fix/ida-native-verification-oracles

Conversation

@N0zoM1z0

@N0zoM1z0 N0zoM1z0 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Follow up the two late review findings on #1544: reject unknown native callee truncation and verify returned values in the real-provider lane.

  • Native callee successes require an explicit more marker; missing markers remain protocol failures. Upstream failures retain their original reason and observations.
  • Require the resolved procedure in inventory/search. Compare address, assembly, pseudocode, instruction windows and relationship results with the CLI/MCP dossier.
  • Add caller-selected --expected-string and --expected-xref positive oracles. Without independent oracles, report string/xref correctness as unverified. Callee parity does not establish semantic completeness.
  • Remove the unsupported raw-Evidence-parity claim and record the verified scope accurately.
  • Repair the CI-blocking Ghidra test fixtures: derive complete-file probe sizes from their bytes, avoiding stale ELF32 size constants in ELF64 cases. Runtime parser and installation behavior are unchanged.

Scope and contracts

Fixes within the existing attached native IDA profile and verification harness, plus the diagnosed CI fixture correction; a new-provider or mutation scope decision is not applicable. Existing CLI/MCP tool schemas, caller choices and legacy/modern behavior are preserved. The parser owns native coverage interpretation. No database save/close, target execution, installation, GUI startup or new analysis cache is introduced. Validation adds constant work to a native callee page; verification reuses the responses already requested.

Validation

  • 207 focused IDA/provider/public-entry tests passed; public-entry cases use an explicitly simulated native provider and owned process supervision, not a real engine.
  • Serial typecheck/lint/format/knip/metadata checks, architecture guards, test discovery and docs checks passed. Local builds/tests used one logical CPU and bounded Node heaps.
  • Real Windows IDA Pro 9.3 with native MCP upstream c9d15d98e8a143fc74f6e14037c48f63889b6d6c, including its navigation correction and a local loopback listener: eight production CLI/MCP checks passed on the unchanged public fixture. The expected CRT literal was confirmed independently in fixture bytes; the expected reference source address was identified from main assembly matching the C source.
  • A response-only loopback proxy in front of the actual IDA Pro provider reproduced the old verifier passing empty inventories and xrefs. The corrected verifier rejected those replies with positive oracles, empty string results, and omitted callee markers (six live before/after cases). The comparison used verifier source d1aaeee40 with the same current adapter runtime; it is not an engine-performance comparison.
  • Windows IDA Free, native macOS and stock release behavior remain unverified by these runs. Personal paths, license registration information, binaries and raw private reports are excluded. Test inputs were unchanged and never executed; attached GUI databases remained open.

See the verification and historical performance note. Its before/after measurements retain their historical source pins; this parser/verification correction adds no performance improvement claim.

Review findings addressed

CI and review must complete on this PR's current head before merge.

CI baseline diagnosis

The initial run on 85c36abe3 failed referenceSourceImportStatus.test.ts in test shard 2 and the Windows lane. Independent main CI on f611d8ce7 fails the exact same cases: two Linux full-output preflight cases and those two plus the Windows readable-root case. Main commit 570c40cde changed full-output failures to ok: false / error, while those assertions still expect ok: true / data. This PR does not touch that CLI output implementation or those tests. This is a confirmed main baseline failure; unrelated code is kept out of this fix.

The refreshed run on d8baca68f encountered three Ghidra decompiler-header fixture failures in src/ghidra/GhidraInstallation.test.ts. Independent main CI on the exact integrated base c462a4ebd has the same three failures in both Windows and shard 4. The Ghidra test, implementation, and ELF-header producer are identical to main. The initial diagnosis kept unrelated code out. Following the maintainer request to fix CI, commit d1e0b1095 repairs only the inconsistent fixture metadata: ELF64 bytes are 64 bytes, while the old probes declared a 52-byte file. A shared test helper derives size from the actual buffer for complete-file probes; the expected compatible/incompatible/unknown outcomes remain unchanged. All three failures reproduced locally before the correction, and all 115 relevant Ghidra installation/doctor and ELF-header/target tests pass afterward. This does not add real-Ghidra provider verification.

Current integration status

Manual Codex review completed on 85c36abe3 without findings. Its initial CI failures were independently reproduced on main, as documented above. Main subsequently fixed the stale assertions in 8224aeba9. This branch now incorporates main c462a4ebd through merge d8baca68f; no separate unrelated CLI edits were added to this PR. On integrated head d8baca68f, all 207 focused tests and serial typecheck/lint/format/knip/metadata checks passed, including the pre-push gate. Automated Codex review completed on d8baca68f without findings. That run completed with 32 successful jobs; only the Ghidra cases failed in Windows and shard 5, causing the two aggregate failures. Commit d1e0b1095 fixes those fixtures; 115 related tests, serial typecheck/lint/format and commit hooks pass. On corrected head d1e0b1095, automated review completed without findings or unresolved threads. CI run 38327862093 completed successfully: all 36 jobs passed, including all six test shards, coverage/timing, the Windows capability/package lane, real Apple CLI/MCP workflows, and CI required.

Merged as 37b7550ecc916acc8619fa72e5c05af45702733f after exact-head review and all 36 CI jobs passed. The two late #1544 review threads have been explained and resolved.

Test cleanup removed the portable Windows runtime, Node and transient reports and probes. Attached test GUI inputs/databases and compact real-provider receipts are retained; no GUI database was saved or closed.

Reject native callee successes without an explicit truncation marker while preserving producer error reasons. Require positive procedure inventory/search results and compare normalized function results with the CLI/MCP dossier. Add caller-selected string and xref oracles; explicitly report those behaviors as unverified when no independent oracle is supplied.

Verified with Windows IDA Pro 9.3 and the native MCP plugin on the unchanged public fixture. Eight production checks passed with a literal independently confirmed in fixture bytes and an xref source address identified from main assembly matching the C source. A response-only loopback proxy reproduced the earlier verifier passing empty inventories/xrefs; the corrected verifier rejected those results, empty strings and omitted callee markers. This used IDA Pro, not Windows IDA Free. No personal paths or license registration information are included.

Validation: 207 focused tests, serial static checks, architecture guards, discovery and docs checks passed. Public-entry regressions use an explicitly simulated native provider and owned process supervision; they do not establish real-engine behavior.

Follow-up to #1544.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@N0zoM1z0

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T00:36:30.413679Z d1e0b10 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 85c36abe3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@nikolay-dementiev

Copy link
Copy Markdown
Contributor

Checked this branch merged into main 91b8f58 on macOS arm64 with real IDA Professional 9.1 through the headless profile (ida-pro-mcp c133c38), which the runs in the description do not cover. The updated scripts/verify-real-ida.mjs passes (npm run verify:ida -- --target <inventory fixture> --procedure _rea_fixture_add, 5 checks), and a fixed 37-step real-IDA sweep gives results identical to main. The native profile was not exercised here: this host has no native IDA MCP server.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@N0zoM1z0

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: d8baca68ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The shared ELF parser validates the complete header extent. Three ELF64
installation fixtures still declared a 52-byte file despite containing
64-byte headers, turning positive format evidence into unknown evidence.
Construct complete-file probes from their bytes in one place, preserving
incompatible, truncated and unreadable-header assertions.

Reproduced all three CI failures before the correction. All 115 related
Ghidra installation/doctor and ELF target/header tests pass afterward.
The IDA changes in this PR were verified with Windows IDA Pro 9.3, not
IDA Free; this fixture-only correction adds no real-Ghidra verification.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@N0zoM1z0

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: d1e0b1095f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@N0zoM1z0
N0zoM1z0 merged commit 37b7550 into main Oct 13, 2026
36 checks passed
@N0zoM1z0
N0zoM1z0 deleted the fix/ida-native-verification-oracles branch October 13, 2026 00:44
@morluto morluto mentioned this pull request Oct 13, 2026
3 tasks done
@morluto morluto added bug Something isn't working area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior labels Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants