Skip to content

perf(snapshot): restore large legacy caches within live memory bounds - #1716

Merged
N0zoM1z0 merged 5 commits into
mainfrom
fix/evidence-snapshot-import-capacity
Oct 11, 2026
Merged

N0zoM1z0 merged 5 commits into
mainfrom
fix/evidence-snapshot-import-capacity

Conversation

@N0zoM1z0

@N0zoM1z0 N0zoM1z0 commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

An 83,049,642-byte snapshot from a real Ghidra analysis still fails open_binary(snapshot_path) on main with the existing 768 MiB Node old-space limit. Legacy files duplicate complete execution payloads in cache entries and authenticated Evidence. The reader currently assembles those copies together before validation can start.

Read large legacy snapshots in two scans on one admitted descriptor. Parse cache rows independently, retain private metadata/content-digest handles, then authenticate the complete Evidence bundle and restore cache payloads from exact matching records. Check that both scans saw identical bytes. Keep the existing JSON format, all content and caller metadata, strict JSON/UTF-8 semantics, live resource admission, and canonical query/profile/target checks. Small files retain the bounded native parser path.

The cache now owns immutable authenticated snapshots and reuses their payloads. Exported snapshots remain detached and mutable. CLI and MCP snapshot reads receive cancellation; queued cancellation is checked before publishing the restored session.

Validation:

  • 70 affected snapshot/cache/reconstruction-readiness/filesystem/CLI/MCP composition tests pass after integrating current main. The unchanged 24-case strict CLI JSON input lane also passed. Final typecheck, lint, Knip and architecture guards pass.
  • Actual SDK 2.3.1 stdio MCP, Node 24.18, 768 MiB old-space, two CPUs: cancelling the same large import leaves the session closed and healthy, then a fresh import succeeds. Original parent metadata, procedure, and the first/last two rows of 44,869 assembly entries and 22,505 outgoing references are accessible.
  • The complete 41,546,956-byte exported bundle has exactly the original SHA-256. Target and snapshot digests remain unchanged; replay launches no provider runtime and leaves its scratch directory empty. Guarded process-family peak is 1,095,970,816 bytes; one successful import took 14.54 seconds. These are single-workflow observations, not a general throughput benchmark or exhaustive page coverage.
  • Windows x64, Node 22.19, existing npm 10.9.3: freshly installed the complete source-built package from integrity-verified locked offline archives. Real Windows Ghidra PE snapshot recovery through public MCP passes with a 9 MiB whitespace prefix forcing streaming; the complete original bundle hash and procedure-view Evidence ID are exact, CLI/MCP summary records agree, close invalidates retained references, and replay leaves no provider process or scratch files. The same original 83 MB Linux-produced file also passes the installed Windows snapshot file boundary with its complete 41 MB bundle hash preserved; this additional lane does not claim Windows provider analysis of that ELF. Sampled process-family peak across the combined Windows workflow is 428,752,896 bytes.
  • The packaged Windows native helper and Ghidra bridge are source-equivalent to the already verified 6.4.0 artifact; no new native compilation claim. Real Hopper and IDA recovery remain unverified.

Fixes #1680. Refs #1065.

@N0zoM1z0 N0zoM1z0 added bug Something isn't working area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior performance Performance regressions, scaling limits, and bounded-resource work labels Oct 11, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@N0zoM1z0
N0zoM1z0 merged commit 092de8f into main Oct 11, 2026
31 checks passed
@N0zoM1z0
N0zoM1z0 deleted the fix/evidence-snapshot-import-capacity branch October 11, 2026 05:59
@morluto morluto mentioned this pull request Oct 13, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior bug Something isn't working performance Performance regressions, scaling limits, and bounded-resource work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(evidence): reduce heap expansion when importing complete large snapshots

1 participant