Repository navigation
fix(hopper): prepare verified thin FAT32 slices for arm64 analysis - #3092
akushonkamen wants to merge 2 commits into
Conversation
FAT32 containers previously loaded through Hopper's implicit `-l FAT` slice choice, which selects Apple's arm64e.x1 (cpusubtype 12) slice on arm64 targets. That slice's function entries encode `pacibsppc`, which plain arm64 cannot decode, so recursive descent aborts at every function entry and system binaries (`/bin/*`, `/usr/bin/*`) produce no code procedures (fixes morluto#3072). Extend the existing FAT64 thin-slice preparation to FAT32 containers: resolveHopperMachOImage now parses 20-entry FAT32 tables (big- and little-endian), validates every slice's range, alignment, and embedded Mach-O identity, and commits verified source coordinates. For arm64 targets the selector prefers plain arm64 (cpusubtype 0) then arm64e (cpusubtype 2), whose entry encodings decode under --aarch64; an arm64e.x1 slice is selected only when it is the family's only candidate, and every other ambiguity stays fail-closed. prepareHopperMachOImage sizes the table-identity hash by container width. The profile keeps macho_container and the selected cpu_type/cpu_subtype, loader arguments become the verified thin form, and file-offset mapping is unchanged (it already matches Hopper's loaded header by cpu type and subtype). Configured loader arguments remain explicit overrides. Validation: npm run test:local on the hopper, apple-domain, and binary-target subsets (143 tests, all passing), including 11 new FAT32 regressions that fail on the pre-fix code and pass after it; npm run check:fast and npm run docs:check pass. Real-Hopper lanes (verify:hopper:fat) require an installed Hopper and were not run here; the arm64e thin-image analysis behavior is established by the issue reporter's lipo -thin arm64e workaround against Hopper 6.6.0-demo.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
| for (const cpuSubtype of [0, 2]) { | ||
| const preferred = matches.filter( | ||
| (slice) => slice.cpuSubtype === cpuSubtype, | ||
| ); |
There was a problem hiding this comment.
selectHopperSlice compares the full cpuSubtype value with 0 and 2, but Apple slices can include capability bits. Existing fixtures represent arm64e as 0x80000002 and arm64e.x1 as 0x8000000c. With both slices present, neither matches, so opening the FAT32 binary fails as ambiguous before Hopper starts. This misses the system-binary case this PR is meant to fix.
Compare with machoCpuSubtypeBase(slice.cpuSubtype) while keeping the full value for header checks and provenance. Add a regression using these capability-bearing values.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/hopper/HopperMachOImage.ts
Line: 188-191
Comment:
**System binaries fail to open**
`selectHopperSlice` compares the full `cpuSubtype` value with `0` and `2`, but Apple slices can include capability bits. Existing fixtures represent `arm64e` as `0x80000002` and `arm64e.x1` as `0x8000000c`. With both slices present, neither matches, so opening the FAT32 binary fails as ambiguous before Hopper starts. This misses the system-binary case this PR is meant to fix.
Compare with `machoCpuSubtypeBase(slice.cpuSubtype)` while keeping the full value for header checks and provenance. Add a regression using these capability-bearing values.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Use the validated thin-image preparation path for FAT32 and FAT64. Prefer ordinary arm64 then arm64e, masking capability bits only for ranking while preserving raw slice identity; reject unresolved ambiguity. Bump the analysis profile epoch. Validation: slice/profile fixtures cover raw subtypes, ambiguity, malformed tables and source integrity. Real Hopper execution was not run. Related: #3072, #3092 Co-authored-by: akushonkamen <akushonkamen@163.com>
|
Superseded by published main commit cf806ea. FAT32 and FAT64 targets use verified thin-image preparation; arm64 slice ranking masks capability bits while preserving raw cpu_subtype for header identity and provenance, and ambiguity remains fail-closed. The committed fixture tests cover FAT32 byte order/range/header/table integrity, capability-bearing subtypes and ambiguity, and profile loader arguments. Limitation: no real Hopper installation was available, so |
|
Thanks @akushonkamen for working through the arm64e.x1 slice-selection failure and preserving source identity while selecting a verified thin image. Your FAT32/FAT64 slice fix is on main with ambiguity and capability-bit regressions (main). |
Summary
On Apple-silicon macOS, system binaries (
/bin/*,/usr/bin/*) are FAT32 Mach-O containers whose arm64-family slices include Apple'sarm64e.x1variant (cpusubtype 12). REA loaded such containers through Hopper's implicit-l FATslice choice, which selectsarm64e.x1for an--aarch64target. That slice's function entries encodepacibsppc(0xdac1a7fe), which plain arm64 cannot decode, so Hopper's recursive descent aborts at every function entry and system binaries produce almost no code procedures —/bin/echoyields 5 import stubs instead of its 3 real functions,/bin/pwdyields a mis-alignedsub_10000058cat entry +4.Fixes #3072
Problem and expected behavior
Smallest trigger: open a stripped Apple-system FAT Mach-O (
lipo -archs /bin/echo→x86_64 arm64e arm64e.x1) with the Hopper provider on arm64 macOS and request a function.Violated invariant: an arm64 target must be analyzed from a slice whose entry encodings decode under Hopper's
--aarch64. Thearm64eslice (cpusubtype 2) holdspacibsp(0xd503237f, a decodable HINT-class NOP) at the same offsets wherearm64e.x1holdspacibsppc; REA'sread_bytesconfirmed the loaded bytes matched the.x1slice exactly. The session then failed closed withNo procedure exists at the requested address— correct REA behavior on top of an unusable slice choice.Expected behavior: for arm64 targets on FAT containers, REA prepares a verified thin image of a decodable slice (preferring cpusubtype 0, then 2;
arm64e.x1only when it is the arm64 family's only candidate) and loads it through the already-working thin Mach-O route, the same one the issue reporter validated with thelipo -thin arm64eworkaround against Hopper 6.6.0-demo (/bin/echo: 5 stubs → 11 procedures,analyze_function 0x1000004f0→EntryPointwith pseudocode;/bin/pwdboundary corrected to0x588).Change and scope
src/hopper/HopperMachOImage.ts:resolveHopperMachOImagenow accepts all four FAT magic values (FAT32 big/little-endian, FAT64 big/little-endian), sizes architecture-table entries at 20 bytes for FAT32 and 32 for FAT64, and applies the existing per-slice validation (range, alignment, embedded Mach-O identity, overlap; the reserved field check stays FAT64-only) to both container widths. NewselectHopperSlicereplaces "single match only": forarm64targets it prefers cpusubtype 0 then 2 within the matched family (each preference applies only when exactly one slice carries it; more than one stays fail-closed), andarm64e.x1is selected only when it is the family's sole candidate. Every non-arm64 ambiguity, and the no-match case, keep the previous fail-closed error. Committed method recordsfat32_thin_sliceorfat64_thin_slice.prepareHopperMachOImage: the table-identity hash window is sized by container width (20- vs 32-byte entries) so the copy-time verification covers the same table it validated; streaming copy, whole-source SHA-256, and identity checks are unchanged.src/hopper/HopperAnalysisProfile.ts: the prepared-image path triggers formachoContainer === "fat32" || "fat64"(previously FAT64 only) when no loader-arguments override is configured. The profile keepsmacho_containerand now surfaces the selected slice'scpu_type/cpu_subtypeviaprepared_image, making the hidden slice choice from the issue visible (issue suggestion 3).-l Mach-O --aarch64) instead of[-l, FAT, --aarch64, -l, Mach-O];HopperFileOffsetneeds no change because it already re-maps offsets by matching Hopper's loaded header on cpu type/subtype, andHopperProvider/BridgeLauncherconsume the chain method-agnostically.HopperProvider.tswording updates "FAT64" to "FAT" in the prepared-image notice.docs/installation.md: documents FAT32+FAT64 preparation, the arm64 cpusubtype preference order, and that other ambiguous subtypes still require an explicitly extracted thin image.Existing pattern harvested: this fix deliberately reuses the repo's established FAT64 thin-slice preparation pattern (validate table + selected header → commit source coordinates → stream a private thin copy with whole-source SHA-256 verification), introduced in #2201, and only generalizes it to FAT32 containers plus the cpusubtype preference the issue requests; no new mechanism is introduced.
Follow the contribution scope:
Contract and boundary impact
src/hopper/HopperMachOImage.ts(FAT image resolution/preparation) consumed bysrc/hopper/HopperAnalysisProfile.tsprofile construction before Hopper launch.prepared_image.methodwidens from the literal"fat64_thin_slice"to"fat32_thin_slice" | "fat64_thin_slice";binary_overview/inspectprofiles for FAT32 targets now carryprepared_imagewith the selected slice'scpu_type/cpu_subtypewhere none appeared before.docs/public/product-catalog.json), package, or installation impact: none (no tool catalog or package changes;docs/installation.mdis authored documentation).Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.
(Prepared images write a private temp copy of the target slice — file creation only, within the existing prepared-image lifecycle.)
Evidence and regression coverage
src/hopper/HopperMachOImage.test.ts: 10 new FAT32 regression tests — table parsing in both byte orders, per-slice range/alignment/embedded-Mach-O validation,fat32_thin_slicemethod stamping, arm64 cpusubtype preference (0 > 2 > x1-only), fail-closed on ambiguity/missing slices, andprepareHopperMachOImagesizing the table-identity hash by container width.src/hopper/HopperAnalysisProfile.test.ts: 1 new FAT32 profile regression — a FAT32 executable target now resolvesprepared_imagewithcpu_type/cpu_subtypeand thin loader arguments instead of[-l, FAT, --aarch64, -l, Mach-O]with noprepared_image.[-l, FAT, --aarch64, -l, Mach-O]with noprepared_imagefor a fat32 target (assertion diff atHopperAnalysisProfile.test.ts:79). Restoring the fix turns the same command green. The real-provider failure mode itself (implicit-l FATselectingarm64e.x1,pacibsppcentries, 0/mis-aligned procedures) is established by the issue reporter's byte-level evidence and the verifiedlipo -thin arm64eworkaround on Hopper 6.6.0-demo.prepared_image(methodfat32_thin_slice, source coordinates,cpu_type/cpu_subtype) and the prepared-image provenance notice replaces the implicit FAT-load path;analyze_functionon Apple system binaries returns real procedures instead ofNo procedure exists at the requested address.verify:hopper:fatwith an installed Hopper.For evidence-bearing changes:
Validation performed
npm run test:local -- src/hopper/HopperMachOImage.test.ts src/hopper/HopperAnalysisProfile.test.ts— on pre-fix sources (checked out from the parent commit on the rebased branch): 2 files failed, 12 failed | 2 passed (14), all 11 new FAT32 regressions red; with the fix: 14 passed (14). Both runs on the rebased base.npm run test:local -- src/hopper/HopperAnalysisProfile.test.ts src/hopper/HopperCleanup.test.ts src/hopper/HopperMachOImage.test.ts src/hopper/HopperProcessDiagnostic.test.ts src/hopper/HopperProvider.test.ts src/hopper/HopperProviderDiscovery.test.ts src/hopper/HopperRequestQueue.test.ts src/hopper/HopperTargetLease.test.ts src/domain/binaryTarget.test.ts src/native/AppleDispatchMetadata.test.ts— 10 files passed, 143 passed (143), after the rebase.npm run check:fast— 2 successful, 2 total (turbo typecheck + lint), after the rebase. First attempt failed in theprecheck:fastdependency check (saxes: missing from the installed dependency lockfile) because the branch moved onto a base that addssaxes; resolved by runningnpm cias instructed, then re-running green.npm run docs:check— exit 0 (includes check-doc-facts and the readmeConsistency suite, 5 passed), after the rebase.npx oxfmt --checkon the six changed files (docs/installation.md,src/hopper/HopperAnalysisProfile.ts,src/hopper/HopperAnalysisProfile.test.ts,src/hopper/HopperMachOImage.ts,src/hopper/HopperMachOImage.test.ts,src/hopper/HopperProvider.ts) — all matched files use the correct format.npm run check:pr: the full local gate is optional for changes of this width perAGENTS.md("optional for broad changes or CI diagnosis"); CI owns the full deterministic suite.verify:hopper:fat: no Hopper installation on this machine (nothing matching Hopper in/Applications,~/Applications, or onPATH). Real-provider FAT32 behavior is anchored to the issue reporter's verifiedlipo -thin arm64eworkaround against Hopper 6.6.0-demo; a maintainer or the reporter should run the real lane before merge.scripts/check_comment_history.pyandscripts/check_brand_name.py, do not exist in this repository (no matches in the tree); the repository's own gates above were run instead.Compatibility, safety, and release
Before merge
type(scope): outcome.(One follow-up for review: the real-Hopper FAT32 lane
verify:hopper:fatstill needs to run where Hopper is installed — flagged above rather than claimed as done.)