Skip to content

fix(hopper): prepare verified thin FAT32 slices for arm64 analysis - #3092

Closed
akushonkamen wants to merge 2 commits into
morluto:mainfrom
akushonkamen:fix/hopper-fat32-arm64e-slice
Closed

akushonkamen wants to merge 2 commits into
morluto:mainfrom
akushonkamen:fix/hopper-fat32-arm64e-slice

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

Summary

On Apple-silicon macOS, system binaries (/bin/*, /usr/bin/*) are FAT32 Mach-O containers whose arm64-family slices include Apple's arm64e.x1 variant (cpusubtype 12). REA loaded such containers through Hopper's implicit -l FAT slice choice, which selects arm64e.x1 for an --aarch64 target. That slice's function entries encode pacibsppc (0xdac1a7fe), which plain arm64 cannot decode, so Hopper's recursive descent aborts at every function entry and system binaries produce almost no code procedures — /bin/echo yields 5 import stubs instead of its 3 real functions, /bin/pwd yields a mis-aligned sub_10000058c at entry +4.

Fixes #3072

Problem and expected behavior

Smallest trigger: open a stripped Apple-system FAT Mach-O (lipo -archs /bin/echo → x86_64 arm64e arm64e.x1) with the Hopper provider on arm64 macOS and request a function.

Violated invariant: an arm64 target must be analyzed from a slice whose entry encodings decode under Hopper's --aarch64. The arm64e slice (cpusubtype 2) holds pacibsp (0xd503237f, a decodable HINT-class NOP) at the same offsets where arm64e.x1 holds pacibsppc; REA's read_bytes confirmed the loaded bytes matched the .x1 slice exactly. The session then failed closed with No procedure exists at the requested address — correct REA behavior on top of an unusable slice choice.

Expected behavior: for arm64 targets on FAT containers, REA prepares a verified thin image of a decodable slice (preferring cpusubtype 0, then 2; arm64e.x1 only when it is the arm64 family's only candidate) and loads it through the already-working thin Mach-O route, the same one the issue reporter validated with the lipo -thin arm64e workaround against Hopper 6.6.0-demo (/bin/echo: 5 stubs → 11 procedures, analyze_function 0x1000004f0 → EntryPoint with pseudocode; /bin/pwd boundary corrected to 0x588).

Change and scope

  • In src/hopper/HopperMachOImage.ts: resolveHopperMachOImage now accepts all four FAT magic values (FAT32 big/little-endian, FAT64 big/little-endian), sizes architecture-table entries at 20 bytes for FAT32 and 32 for FAT64, and applies the existing per-slice validation (range, alignment, embedded Mach-O identity, overlap; the reserved field check stays FAT64-only) to both container widths. New selectHopperSlice replaces "single match only": for arm64 targets it prefers cpusubtype 0 then 2 within the matched family (each preference applies only when exactly one slice carries it; more than one stays fail-closed), and arm64e.x1 is selected only when it is the family's sole candidate. Every non-arm64 ambiguity, and the no-match case, keep the previous fail-closed error. Committed method records fat32_thin_slice or fat64_thin_slice.
  • In prepareHopperMachOImage: the table-identity hash window is sized by container width (20- vs 32-byte entries) so the copy-time verification covers the same table it validated; streaming copy, whole-source SHA-256, and identity checks are unchanged.
  • In src/hopper/HopperAnalysisProfile.ts: the prepared-image path triggers for machoContainer === "fat32" || "fat64" (previously FAT64 only) when no loader-arguments override is configured. The profile keeps macho_container and now surfaces the selected slice's cpu_type/cpu_subtype via prepared_image, making the hidden slice choice from the issue visible (issue suggestion 3).
  • Loader arguments for prepared images are the verified thin form (-l Mach-O --aarch64) instead of [-l, FAT, --aarch64, -l, Mach-O]; HopperFileOffset needs no change because it already re-maps offsets by matching Hopper's loaded header on cpu type/subtype, and HopperProvider/BridgeLauncher consume the chain method-agnostically. HopperProvider.ts wording updates "FAT64" to "FAT" in the prepared-image notice.
  • docs/installation.md: documents FAT32+FAT64 preparation, the arm64 cpusubtype preference order, and that other ambiguous subtypes still require an explicitly extracted thin image.
  • Explicitly not included: changes to Hopper itself, new user-facing slice-selector flags, Ghidra or other-provider behavior, and any loosening of the FAT64 ambiguity/fail-closed policy beyond reusing it through the shared selector. Configured loader arguments remain explicit overrides.

Existing pattern harvested: this fix deliberately reuses the repo's established FAT64 thin-slice preparation pattern (validate table + selected header → commit source coordinates → stream a private thin copy with whole-source SHA-256 verification), introduced in #2201, and only generalizes it to FAT32 containers plus the cpusubtype preference the issue requests; no new mechanism is introduced.

Follow the contribution scope:

  • Contribution scope fit: ordinary fix within existing Hopper-provider boundaries; no new provider or capability involving target execution, injection, mutation, or protection recovery, so no maintainer scope decision is required ("not applicable").

Contract and boundary impact

  • Semantic owner and earliest changed stage: src/hopper/HopperMachOImage.ts (FAT image resolution/preparation) consumed by src/hopper/HopperAnalysisProfile.ts profile construction before Hopper launch.
  • Contribution scope fit and linked maintainer decision: not applicable; fix within the existing Hopper provider and the pattern already shipped in fix(ida): thin the selected FAT slice into the private headless input #2201.
  • CLI and MCP/tool-catalog contract: no tool schemas change. prepared_image.method widens from the literal "fat64_thin_slice" to "fat32_thin_slice" | "fat64_thin_slice"; binary_overview/inspect profiles for FAT32 targets now carry prepared_image with the selected slice's cpu_type/cpu_subtype where none appeared before.
  • Provider, bridge, target-format, or platform compatibility: Hopper provider, Mach-O FAT32/FAT64 containers, arm64/arm64e targets on Apple-silicon macOS. Thin Mach-O and non-FAT targets are untouched; configured loader arguments remain explicit overrides.
  • Evidence, artifact, provenance, or reconstruction contract: unchanged in kind — source identity and reported file offsets still refer to the original container; the owned temporary image and its Hopper document still close together including on MCP shutdown; the container-table SHA-256 identity hash now covers FAT32-width tables.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): reading is limited to the already-opened target file plus a private temp copy; no new authority, network, or injection. Temp-image cleanup path is the existing prepared-image lifecycle.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none (no tool catalog or package changes; docs/installation.md is authored documentation).

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

(Prepared images write a private temp copy of the target slice — file creation only, within the existing prepared-image lifecycle.)

Evidence and regression coverage

  • Tests added or updated:
    • src/hopper/HopperMachOImage.test.ts: 10 new FAT32 regression tests — table parsing in both byte orders, per-slice range/alignment/embedded-Mach-O validation, fat32_thin_slice method stamping, arm64 cpusubtype preference (0 > 2 > x1-only), fail-closed on ambiguity/missing slices, and prepareHopperMachOImage sizing the table-identity hash by container width.
    • src/hopper/HopperAnalysisProfile.test.ts: 1 new FAT32 profile regression — a FAT32 executable target now resolves prepared_image with cpu_type/cpu_subtype and thin loader arguments instead of [-l, FAT, --aarch64, -l, Mach-O] with no prepared_image.
  • Base reproduction or other evidence: on the pre-fix code the same two test files fail 12 of 14 — all 11 new FAT32 regressions red, including the profile layer still emitting [-l, FAT, --aarch64, -l, Mach-O] with no prepared_image for a fat32 target (assertion diff at HopperAnalysisProfile.test.ts:79). Restoring the fix turns the same command green. The real-provider failure mode itself (implicit -l FAT selecting arm64e.x1, pacibsppc entries, 0/mis-aligned procedures) is established by the issue reporter's byte-level evidence and the verified lipo -thin arm64e workaround on Hopper 6.6.0-demo.
  • User-visible CLI/MCP output (if applicable): profiles for FAT32 arm64 targets now report prepared_image (method fat32_thin_slice, source coordinates, cpu_type/cpu_subtype) and the prepared-image provenance notice replaces the implicit FAT-load path; analyze_function on Apple system binaries returns real procedures instead of No procedure exists at the requested address.
  • Remaining proof gaps: real-Hopper end-to-end behavior was not executed locally (see Validation performed); the FAT32 real lane needs a maintainer or the reporter to run verify:hopper:fat with an installed Hopper.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:local -- src/hopper/HopperMachOImage.test.ts src/hopper/HopperAnalysisProfile.test.ts — on pre-fix sources (checked out from the parent commit on the rebased branch): 2 files failed, 12 failed | 2 passed (14), all 11 new FAT32 regressions red; with the fix: 14 passed (14). Both runs on the rebased base.
  • npm run test:local -- src/hopper/HopperAnalysisProfile.test.ts src/hopper/HopperCleanup.test.ts src/hopper/HopperMachOImage.test.ts src/hopper/HopperProcessDiagnostic.test.ts src/hopper/HopperProvider.test.ts src/hopper/HopperProviderDiscovery.test.ts src/hopper/HopperRequestQueue.test.ts src/hopper/HopperTargetLease.test.ts src/domain/binaryTarget.test.ts src/native/AppleDispatchMetadata.test.ts — 10 files passed, 143 passed (143), after the rebase.
  • npm run check:fast — 2 successful, 2 total (turbo typecheck + lint), after the rebase. First attempt failed in the precheck:fast dependency check (saxes: missing from the installed dependency lockfile) because the branch moved onto a base that adds saxes; resolved by running npm ci as instructed, then re-running green.
  • npm run docs:check — exit 0 (includes check-doc-facts and the readmeConsistency suite, 5 passed), after the rebase.
  • npx oxfmt --check on the six changed files (docs/installation.md, src/hopper/HopperAnalysisProfile.ts, src/hopper/HopperAnalysisProfile.test.ts, src/hopper/HopperMachOImage.ts, src/hopper/HopperMachOImage.test.ts, src/hopper/HopperProvider.ts) — all matched files use the correct format.
  • Not run — npm run check:pr: the full local gate is optional for changes of this width per AGENTS.md ("optional for broad changes or CI diagnosis"); CI owns the full deterministic suite.
  • Not run — real Hopper lane verify:hopper:fat: no Hopper installation on this machine (nothing matching Hopper in /Applications, ~/Applications, or on PATH). Real-provider FAT32 behavior is anchored to the issue reporter's verified lipo -thin arm64e workaround against Hopper 6.6.0-demo; a maintainer or the reporter should run the real lane before merge.
  • Note: two checks named in the contributor brief, scripts/check_comment_history.py and scripts/check_brand_name.py, do not exist in this repository (no matches in the tree); the repository's own gates above were run instead.

Compatibility, safety, and release

  • Breaking changes or migration steps: none. Existing valid behavior is preserved: thin Mach-O targets, FAT64 targets, non-arm64 FAT targets, ambiguous non-arm64 selections (still fail-closed with the same guidance), and configured loader-arguments overrides all behave as before. The only behavior change is the intended one: FAT32 arm64 targets move from the implicit FAT load to a verified thin slice.
  • Real Hopper/Ghidra, browser, or OS coverage: real Hopper lane not runnable locally (no installation); source-level coverage is the 14-test red→green pair plus the 143-test related subset. Ghidra/browser untouched.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: no added execution authority; the fix narrows what gets loaded (a validated thin slice with committed source coordinates and whole-source SHA-256 verification) and keeps every ambiguity fail-closed. The temp image stays within the existing prepared-image cleanup lifecycle.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

(One follow-up for review: the real-Hopper FAT32 lane verify:hopper:fat still needs to run where Hopper is installed — flagged above rather than claimed as done.)

FAT32 containers previously loaded through Hopper's implicit `-l FAT`
slice choice, which selects Apple's arm64e.x1 (cpusubtype 12) slice on
arm64 targets. That slice's function entries encode `pacibsppc`, which
plain arm64 cannot decode, so recursive descent aborts at every function
entry and system binaries (`/bin/*`, `/usr/bin/*`) produce no code
procedures (fixes morluto#3072).

Extend the existing FAT64 thin-slice preparation to FAT32 containers:
resolveHopperMachOImage now parses 20-entry FAT32 tables (big- and
little-endian), validates every slice's range, alignment, and embedded
Mach-O identity, and commits verified source coordinates. For arm64
targets the selector prefers plain arm64 (cpusubtype 0) then arm64e
(cpusubtype 2), whose entry encodings decode under --aarch64; an
arm64e.x1 slice is selected only when it is the family's only candidate,
and every other ambiguity stays fail-closed. prepareHopperMachOImage
sizes the table-identity hash by container width. The profile keeps
macho_container and the selected cpu_type/cpu_subtype, loader arguments
become the verified thin form, and file-offset mapping is unchanged (it
already matches Hopper's loaded header by cpu type and subtype).
Configured loader arguments remain explicit overrides.

Validation: npm run test:local on the hopper, apple-domain, and
binary-target subsets (143 tests, all passing), including 11 new FAT32
regressions that fail on the pre-fix code and pass after it; npm run
check:fast and npm run docs:check pass. Real-Hopper lanes
(verify:hopper:fat) require an installed Hopper and were not run here;
the arm64e thin-image analysis behavior is established by the issue
reporter's lipo -thin arm64e workaround against Hopper 6.6.0-demo.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@greptile-apps

greptile-apps Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium impact] Extends FAT slice preparation to support FAT32 containers.

The PR should not merge until the remaining subtype comparison is fixed, because affected Apple system binaries still fail to open.

Findings

  1. P1 System binaries fail to open ▶
Fix with agent prompt
### Issue 1
src/hopper/HopperMachOImage.ts:188-191
`selectHopperSlice` compares the full `cpuSubtype` value with `0` and `2`, but Apple slices can include capability bits. Existing fixtures represent `arm64e` as `0x80000002` and `arm64e.x1` as `0x8000000c`. With both slices present, neither matches, so opening the FAT32 binary fails as ambiguous before Hopper starts. This misses the system-binary case this PR is meant to fix.

Compare with `machoCpuSubtypeBase(slice.cpuSubtype)` while keeping the full value for header checks and provenance. Add a regression using these capability-bearing values.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Extends Hopper's verified thin-image preparation from FAT64 to FAT32 and adds an arm64 slice preference.

  • Hopper gets a verified thin slice for arm64 FAT analysis.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Mach-O target] --> B{Explicit loader arguments?}
  B -->|Yes| C[Keep configured arguments]
  B -->|No| D{FAT32 or FAT64?}
  D -->|No| E[Use existing loader route]
  D -->|Yes| F[Validate table and embedded headers]
  F --> G[Select target slice]
  G --> H[Commit source coordinates and table hash]
  H --> I[Copy slice and verify whole-source hash]
  I --> J[Load verified thin image]
Loading

Reviews (2) · Last reviewed commit: "merge: restore JavaScript analysis regre..." · Reviewed by Greptile

Comment on lines +188 to +191
for (const cpuSubtype of [0, 2]) {
const preferred = matches.filter(
(slice) => slice.cpuSubtype === cpuSubtype,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 System binaries fail to open

selectHopperSlice compares the full cpuSubtype value with 0 and 2, but Apple slices can include capability bits. Existing fixtures represent arm64e as 0x80000002 and arm64e.x1 as 0x8000000c. With both slices present, neither matches, so opening the FAT32 binary fails as ambiguous before Hopper starts. This misses the system-binary case this PR is meant to fix.

Compare with machoCpuSubtypeBase(slice.cpuSubtype) while keeping the full value for header checks and provenance. Add a regression using these capability-bearing values.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/hopper/HopperMachOImage.ts
Line: 188-191

Comment:
**System binaries fail to open**

`selectHopperSlice` compares the full `cpuSubtype` value with `0` and `2`, but Apple slices can include capability bits. Existing fixtures represent `arm64e` as `0x80000002` and `arm64e.x1` as `0x8000000c`. With both slices present, neither matches, so opening the FAT32 binary fails as ambiguous before Hopper starts. This misses the system-binary case this PR is meant to fix.

Compare with `machoCpuSubtypeBase(slice.cpuSubtype)` while keeping the full value for header checks and provenance. Add a regression using these capability-bearing values.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

morluto added a commit that referenced this pull request Oct 14, 2026
Use the validated thin-image preparation path for FAT32 and FAT64. Prefer ordinary arm64 then arm64e, masking capability bits only for ranking while preserving raw slice identity; reject unresolved ambiguity. Bump the analysis profile epoch.

Validation: slice/profile fixtures cover raw subtypes, ambiguity, malformed tables and source integrity. Real Hopper execution was not run.

Related: #3072, #3092

Co-authored-by: akushonkamen <akushonkamen@163.com>
@morluto

morluto commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Superseded by published main commit cf806ea. FAT32 and FAT64 targets use verified thin-image preparation; arm64 slice ranking masks capability bits while preserving raw cpu_subtype for header identity and provenance, and ambiguity remains fail-closed. The committed fixture tests cover FAT32 byte order/range/header/table integrity, capability-bearing subtypes and ambiguity, and profile loader arguments. Limitation: no real Hopper installation was available, so verify:hopper:fat and the observed macOS/Hopper behavior were not rerun here. This closure records the implementation and source-level verification, not a new real-provider claim.

@morluto morluto closed this Oct 14, 2026
@morluto morluto added bug Something isn't working area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior labels Oct 14, 2026
@morluto

morluto commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Thanks @akushonkamen for working through the arm64e.x1 slice-selection failure and preserving source identity while selecting a verified thin image. Your FAT32/FAT64 slice fix is on main with ambiguity and capability-bit regressions (main).

@akushonkamen
akushonkamen deleted the fix/hopper-fat32-arm64e-slice branch October 25, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Hopper provider on macOS: FAT Mach-O slice selection loads arm64e.x1, collapsing analysis of system binaries

2 participants