Repository navigation
feat(native): admit bounded MIPS ELF analysis through Ghidra - #1362
Conversation
Retain MIPS ELF identity and interpretation through Evidence and snapshots. Keep provider admission bounded to ELF32 ET_EXEC o32 MIPS32r2, without implicitly widening Hopper, IDA or Windows P0 coverage. Add source-owned fixtures and an optional real-provider verifier; real Ghidra verification remains pending in this contribution environment. Refs morluto#718
Reject explicit EF_MIPS_FP64 and EF_MIPS_NAN2008 declarations until their semantics have a real-provider verification lane. Add PSP PRX/Allegrex, o64/EABI and floating-point-mode refusal regressions. Document that the header gate is not a full .MIPS.abiflags check and advance the profile lane identity to invalidate older interpretation bindings. Validation: 24 standalone checks against the transpiled production profile module passed; the two new flag cases failed before this change. Clang 17 and LLD compiled the source-owned fixture for both endian targets. The complete Vitest/typecheck suite and real Ghidra verification remain pending. Refs morluto#718; PSP support remains follow-up morluto#1330.
Read bounded ELF32 PT_MIPS_ABIFLAGS and SHT_MIPS_ABIFLAGS records through the already-open target handle. Preserve declared ABI fields, reject conflicting/duplicate/truncated records, and keep missing records unknown. Wire inspection into target resolution before provider selection. Require compatible declarations for the deliberately narrow Ghidra lane and bind all ABI fields into profile identity (v3) for snapshot invalidation. Clang 17 produced eight source-owned fixtures spanning both byte orders and FPXX/FP32/FP64/soft-float modes with identical ELF header flags. The new reader and policy distinguish them. 24 focused regression test bodies passed after TypeScript transpilation using node:test in place of Vitest; strict isolated TypeScript checking of the reader and target/result types passed. These are limited checks, not full Vitest/project typechecking or real-Ghidra verification, which remain pending. Advances morluto#718. PSP/Allegrex remains morluto#1330 follow-up.
…avior Keep ELF targets out of Hopper Mach-O preparation and scope existing PE fixture types correctly. Move ABI tests into the filesystem boundary lane, use shared cleanup, add resolver regressions, apply the pinned formatter, and register the real MIPS verifier through the existing package-command pattern. Validated with repository-pinned Node/npm, check:ci, actual focused Vitest, discovery and docs:check. Real Ghidra conformance remains pending.
Cross-check the source-owned fixture with bounded GNU readelf inspection, compare production ELF/ABI interpretation and independent symbol addresses, and verify effective Ghidra language/compiler identity through existing public operations. Add a source-owned fixed-encoding probe for immediate and conditional branch checks plus global/string byte assertions. Add 16 assertion/producer-boundary regression cases without widening production admission, changing provider contracts, or introducing proprietary dependencies. Local focused suite: 99 passed, one existing root-permission test skipped. Typecheck, lint/module boundaries, formatting and 700-file discovery pass; full hosted checks and real Ghidra verification remain pending for this checkpoint.
|
TL;DR: The MIPS lane passed here on newer tools. Both byte orders worked. I ran this draft on Fedora 44. The tools were newer than your CI. What ran:
What is new: your CI used clang 18 and readelf 2.42. Clang 22 still gives the same flags, Edit: I ran these on main 9d6651e (#4131) after the merge. Results are in my new comment. Still not run: the full test suite, and any host but Linux x64. Not run: Test setup
🤖 Generated with Claude Code (Opus 5.5) 🦊 |
|
Thanks for running this on Fedora and documenting the exact tool versions and verification scope. The Clang 22 and readelf 2.46 results are useful corroboration beyond our original CI environment. |
…gration The conflict-free upstream merge failed TypeScript compilation because includes on the union of MIPS-only and other architecture arrays requires never. Use equality-based membership without casts or changing accepted native hosts. Extend the existing native-decompiler rejection table with a MIPS executable at a Linux x64 tool path; retain upstream FAT and unknown-header behavior. Hosted verification follows this commit; no success is claimed yet.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
The real Ghidra checks on both byte orders are particularly helpful here, @jaelliot. Thank you for adding MIPS32r2/o32 ELF support and being clear about which binaries the new path admits. Welcome to REA! |
|
TL;DR: This PR merged, so I reran my Not run items on current main. The real-Ghidra MIPS check passes for all six fixtures, and This comment is a record. The PR merged as bddddcc (#1362). My first comment tested the draft at adf1cda. It listed two Not run items: the full test suite with
Main has six fixtures. My first run reported two byte orders. Clang 22.1.8 built them. After the run, Still not run
Thanks, jaelliot, for the MIPS support. Commandsgit clone --no-checkout --no-hardlinks <local mirror> clone
git -C clone checkout --detach 9d6651e5b5296da31a88042a061bdc6f4e769f10
npm ci && npm run build && npm run test:prepare
GHIDRA_INSTALL_DIR=~/.local/opt/ghidra_12.1.4_PUBLIC JAVA_HOME=~/.local/opt/jdk-21.0.12.1+1 \
npm run verify:ghidra:mips
npm run test:focused -- src/domain/binaryTarget.mips.test.ts \
src/ghidra/GhidraMipsProfile.test.ts src/ghidra/GhidraProvider.behavior.test.ts \
src/hopper/HopperMachOImage.test.ts tests/boundary/filesystem/mipsAbiFlags.test.ts \
tests/composition/analysis-sessions/mips-identity.test.ts \
tests/conformance/ghidra/mips-checks.test.ts
npm run check:ci
npm run checkEach run used Test setup
🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
Scope
Advances the generic MIPS ELF follow-up in #718, not the whole firmware roadmap. PSP/Allegrex remains separate in #1330. This PR remains draft and unmerged; maintainer review of the bounded admission policy is still requested.
Current source head:
eeeada5663fcb58e21f9a642bbac50ef70d16310.Upstream revision incorporated and tested:
81cdafc132c7ba3bd6e12b1169f9e3687025d07b.Contribution
EM_MIPSasmips, preserving class, byte order, type, flags and ABI records separately from architecture-family identity.PT_MIPS_ABIFLAGS/SHT_MIPS_ABIFLAGSthrough the existing target handle, including beyond the initial probe; preserve cancellation and reject malformed, duplicate or conflicting records.npm run verify:ghidra:mips. The verifier independently compares ELF/ABI/symbol observations with GNU readelf and checks observed Ghidra configuration, selected instruction/data facts, CLI/MCP consistency, snapshot reopening, input preservation and cleanup.No emulator, commercial fixture, new MCP tool, automatic product installer or Allegrex implementation is added. PSP/PRX, MIPS64, n32/n64, MIPS16/microMIPS, vendor variants, relocatable/shared objects and unverified floating-point modes remain outside this verified profile. See
docs/ghidra-mips.mdfor exact prerequisites and limitations.Upstream synchronization and observed repair
2b7f9586d4f5aad0b4220de2f8971aac5d1c00aeis a real merge with parentsadf1cda29859ffcc063f40418b8ba39a0eac4faband the pinned upstream81cdafc.... There were no textual merge conflicts, but initial integration run 38022127678 failed TypeScript compilation in the upstream-added native decompiler check.The shared architecture union made
availableArchitectures.includes(expectedArchitecture)require aneverargument on a union of array types.eeeada5uses equality-based.some(...)membership without casts or changing accepted hosts. It adds one case to the existing native-decompiler rejection table: a MIPS executable at the Linux x64 decompiler path must be rejected.Upstream regular-file acquisition/stability checks, target-scoped snapshot validation, package/lockfile changes and architecture guards are retained. No production gate, threshold or ignore rule was relaxed.
The integrated commit was tested on temporary branches before a non-force, expected-head-checked update of the existing feature branch. Neither main branch nor either PR's draft/merge state was changed. Fork-only orchestration remains outside this contribution diff.
Verification on the integrated source
Integration verification run 38022307570, verification job 114125761674, completed successfully and explicitly checked out
eeeada5663fcb58e21f9a642bbac50ef70d16310. Its orchestration commitc36f1e9c8b5b0a0fbdbf845784f86f37f2d0568eis not the tested source SHA.npm run check:cinpm run docs:checkThe focused total includes the existing 100 affected tests plus 30 installation tests; only one new rejection test was added during synchronization. Upstream's current test inventory differs from the earlier 700-file checkout; no suite was removed by this integration fix.
Real-provider environment: Linux x64; checksum-verified Ghidra 12.1.4, Temurin 21.0.12.1+1, Clang/LLD 18.1.3, GNU readelf 2.42. Both byte orders passed independent ELF/ABI/symbol checks, effective language/compiler and input-hash observations, fixed instruction bytes/immediate/branch destination, global value and marker, known direct call, Evidence/profile binding, snapshot reopening and equivalent CLI analysis. Neither MIPS executable was run.
New ordinary PR CI: run 38022753836 was triggered by the source-branch update and is pending at this report checkpoint. The full deterministic/coverage/platform matrix is not yet claimed as passing on the integrated source. Earlier green counts are not carried forward.
Historical evidence
At
adf1cda..., ordinary CI 38002231653 passed all 19 jobs. Its merged log reports 6,254 passed and 89 skipped tests, 689 passed and 11 skipped test files (700 total), not the previously misreported 4,162/2. That run used fork test-mergee9b8bfabfdcb2c48057dad1d0299ed3fd779aa4cwith the older fork base. Real-provider run 38002495944 explicitly testedadf1cda....Another contributor's Fedora/Clang 22/readelf 2.46 report also concerns
adf1cda..., not this integration. It is retained as third-party corroboration rather than a substitute for current-source validation.Reader agreement and selected decoded properties do not prove full execution, delay-slot or floating-point equivalence. Pseudocode presence is a liveness check. Additional hosts and PSP remain separate verification claims.
Remaining gates
Fork development PR #1 tracks the same source branch, but its deliberately unchanged fork-main base also displays incorporated upstream commits. This upstream PR is the focused contribution diff. No request to merge automatically, close #718 or implement #1330 is implied.