Repository navigation
feat(native): search analyzed memory for masked byte patterns - #2341
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
TL;DR: I ran this PR on real Ghidra 12.1.4. Its own check passes, and the results match a Python search of the raw bytes. The red CI is not from this PR: main at 1c37914 failed the same tests. ResultsHead c6380a9, Ghidra 12.1.4, Node 24.18.0 unless noted.
The PR's lane timed its first search at 85 ms. On libxml2 the first search took about 100 to 170 ms, after a 34 s import. CIThe red checks are the #2343 high-bit test and the Two questions
Not run
Thanks, mvanhorn, for the clear PR and for writing the real-Ghidra check. The check ran as written, first time. Test setup
CommandsRun with a private My own harness opened one target over MCP and ran many 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
Adversarial Review Findings:
|
The diff adds search_bytes to the CLI, the shared MCP contract, Ghidra inventory, and the headless bridge, and advertises the same operation on Hopper as unavailable until that provider is verified. Patterns are hexadecimal byte pairs with optional ?? wildcards and an optional equal-length bit mask; the TypeScript boundary and the bridge both require a nonempty pattern, equal lengths, and at least one constrained bit. Ghidra walks initialized blocks in address order within inclusive bounds, keeps overlapping matches inside a single block, and returns each hit with its segment, unique source file offset, and containing procedure, continuing through a cursor bound to the target, profile, pattern, mask, and bounds. Each page reports the inclusive ranges examined and the skipped uninitialized blocks; cancellation returns a typed error and discards the active Ghidra database. address_to_file_offset now uses the same unique-offset helper, which rejects a mapping that is missing, ambiguous, or larger than a safe integer.
Without start or end, search_bytes walked every Ghidra block, including non-loaded file parts such as .symtab and overlay blocks, which the docs call out of scope. Unbounded searches now skip those blocks and list them in searched.skipped_not_loaded; explicit bounds in their address space still search them. Document that a match must end at or before end.
c6380a9 to
742a02b
Compare
|
TL;DR: I retested the new head, 742a02b, on real Ghidra 12.1.4. Matches are never repeated or skipped, and the new loaded-memory rule works. One small thing is still true: each page boundary lists a few bytes as searched twice. ResultsThe old head is c6380a9. The new head is 742a02b. Both ran the same fixtures on Ghidra 12.1.4. The Python search is an independent
The lower counts are the overlay blocks that the old head also searched. The other matches keep their order and file offsets. Skipped blocks now show in Review finding 2: reproduces, in a small wayThanks, matbrgz, for the careful review. Page 1 reports bytes up to the end of the next match. Page 2 starts at that match. So that many bytes, as many as the pattern is long, are listed twice. The matches are still right. Smallest repro: a 32-byte file opened with
Bytes 0x1008 and 0x1009 are on both pages. Should page 1 stop one byte before its cursor address? Not run
Thanks, mvanhorn, for the fast follow-up. This answers both of my earlier questions. Overlay blocks are skipped now. The docs also say how Review finding 1: not reproducedThanks, matbrgz, for the close read. The cursor holds the first match the page did not return. The loop checks What 742a02b changed, and one odd caseWith no Odd case: I edited a cursor to point into Paging and coverage, every runRepeats counts repeated matches, repeated cursors and a last match that came back on the next page. "Bytes listed twice" is the sum of page coverage minus the covered bytes.
Test setup
CommandsRun with a private My own harness opened one target over MCP and ran many 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
|
Thanks for running this on real Ghidra 12.1.4 and checking it against a raw-byte search. That is more than I could do here.
I also rebased onto current main, which should clear the stale-base CI failures you traced. Locally, |
|
Thanks! My retest just above ran that: 🤖 Generated with Claude Code (Opus 5.5) 🦊 |
|
@greptileai review |
|
This PR is set for a Plus review. TREX isn't supported with Plus or Apex reviews yet, so Greptile reviews this PR at Base while TREX is on. |
|
|
Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open. Fixed in 82cc51f. searchMemoryBytes now distinguishes omitted optional state from a present value: cursor uses requireString only when its key is present, so null, non-string and empty continuation values fail invalid_request instead of restarting page one. The same owning boundary rejects present malformed mask/start/end values while preserving genuine omissions. Source inspection and git diff --check passed; tests and real Ghidra verification were not run at explicit user request. No related cleanup remains. |
Summary
Agents can now find where a byte signature occurs in an opened native target:
search_bytes(CLIsearch-bytes) takes hexadecimal byte pairs with??wildcards and an optional bit mask, searches Ghidra's initialized analyzed memory, and returns ordered matches with segment, source file offset and containing procedure, plus exact page coverage and a bound continuation cursor. Hopper advertises the operation as unavailable with a typed reason.No issue is linked. This is a new native tool, proposed here with the implementation.
Demo
Full-quality MP4
The video shows real output from this branch's MCP server and Ghidra provider code, driven against the repository's fake Ghidra fixture (
tests/fixtures/fakeGhidra.mjs), since Ghidra is not installed on this host. The match addresses come from the fixture. It shows48 8b ?? ?? 89matching at 0x1000 and 0x1005 with.bsslisted as skipped, the same search paged withlimit: 1andnext_cursor, and??alone rejected as invalid.Problem and expected behavior
read_bytesreads bytes at an address the caller already knows, andsearch_stringsand procedure search cover names and strings. Onupstream/main(2e45b4d) nothing answers "where does48 8B ?? ?? 89occur", so locating a constant table, a prologue, a magic header or an opcode sequence with wildcarded operands means reading memory page by page and matching client side.With this change, one call such as
{"pattern": "48 8B ?? ?? 89", "limit": 100}on an open Ghidra session returns observed Evidence withmatches(address,segment,file_offsetornull,procedure{address, name}ornull),searched(blocksandbytesexamined on this page,skipped_uninitialized),completeandnext_cursor. Repeating the same query withcursor: next_cursorcontinues untilcompleteis true.Change and scope
Pattern grammar is owned by
src/domain/native/bytePattern.tsand shared by the MCP contract, the Ghidra inventory schema and the CLI. A pattern is hexadecimal byte pairs (either case) or??, with optional whitespace between pairs. Empty patterns, odd nibbles (4,4?), non-hex characters and patterns where every bit is unconstrained (??, orffwith mask00) are rejected as typed input errors before dispatch. The optionalmaskis hexadecimal pairs of the same length; each mask byte selects which bits of the pattern byte must match (pattern: "a0", mask: "f0"matchesa0throughaf).??stays fully unconstrained even when a mask is given, and an omitted mask meansfffor every non-wildcard byte.Optional
startandendare inclusive and must lie in one address space withstart <= end. Without bounds every memory block is eligible, in Ghidra address order.limitaccepts 1 to 1000 and defaults to 100.The Ghidra bridge (
searchMemoryBytesinbridge/ghidra/ReaGhidraBridge.java) walks blocks sorted by start address and callsMemory.findBytesper initialized block with the end bound pulled back bylength - 1, so a match never crosses a block boundary. After each hit it resumes at the next address, so overlapping matches are kept (00 00over00 00 00gives two hits). Uninitialized blocks inside the bounds are never searched and are listed insearched.skipped_uninitializedon every page.searched.blocksreports the inclusive ranges examined for this page, including the lookahead that found the next unreturned match, so coverage can overlap between pages and should not be summed as unique memory.The cursor is
<binding>:<address>. The binding is SHA-256 over a JSON array of the target SHA-256, profile digest, normalized pattern bytes, masks, canonical start and end, and the document. The address is the first match not yet returned, so resuming neither skips nor repeats a match. The bridge rejects a cursor whose binding differs from the current target and query, or whose address is outside initialized memory or the query bounds. Cancellation is checked between blocks and around eachfindBytescall; it returns a typedcancellederror, discards the active Ghidra database and never produces a successful partial page.The bridge re-validates its request at the socket boundary: required keys
document,pattern,limit, allowed extrasmask,start,end,cursor, anything else isinvalid_request. It also re-applies the pattern, mask, constraint and bounds rules instead of trusting the TypeScript layer.file_offsetcomes from a new shareduniqueFileOffsethelper that returns an offset only when exactly one file-backed source mapping covers the address and the offset fits a JavaScript safe integer.address_to_file_offsetnow uses the same helper. That is a small visible change: its failure text changes fromAddress has N authoritative source mappings: <addr>toAddress lacks a unique safe file offset: <addr>, and an offset aboveNumber.MAX_SAFE_INTEGERis now rejected instead of returned.Hopper lists
search_bytesin its capabilities withavailable: falseand the reason "Masked byte search has not been verified for Hopper.", so the call fails ascapability_unavailablewithout launching Hopper.Out of scope: a Hopper implementation, searching uninitialized or overlay memory, matches spanning block boundaries, string or regex search over bytes, and any write, patch or execution.
Contract and boundary impact
src/domain/native/bytePattern.ts(bytePatternFields,parseBytePattern,validateBytePattern,byteSearchResultSchema); Ghidra provider input/result schemas insrc/ghidra/GhidraInventoryValues.ts; search inbridge/ghidra/ReaGhidraBridge.java.search_bytes(src/contracts/hopperMemoryContracts.ts, strict input withpattern,mask?,document,start?,end?,limit1 to 1000 default 100,cursor?; output schema insrc/contracts/toolOutputSchemaGroups.ts; example{pattern: "48 8B ?? ?? 89"}insrc/contracts/toolContractExamples.ts; effectevidenceinsrc/contracts/toolEffects.ts). New CLI commandsearch-bytes <path> <pattern> --mask --start --end --cursor --limitplus the existing native open options (src/cli/coreNativeCommands.ts,src/cliCommandNames.ts,src/application/DirectAnalysis.ts). The result schema rejects pages wherecompletedisagrees withnext_cursororsearched.bytesdisagrees with the block sum, and the Ghidra schema also requires canonical addresses.search_bytesin its inventory, bridge dispatch and capability limitations (src/ghidra/GhidraProviderCapabilities.ts); same effects as other Ghidra reads, snapshot cache policy. Hopper reports the operation unavailable (src/hopper/HopperProviderCapabilities.ts).address_to_file_offseterror text changes as described above.docs/public/product-catalog.json), package, or installation impact: docs indocs/ghidra-provider.md(Masked byte search) anddocs/native-investigation.md(Locate a byte signature); three rows indocs/user-visible-errors.csv(unconstrained pattern, unequal mask length, cursor reused with another query). The new tool and command appear in the generated catalog; no authored skill text changed.metadata:checkanddocs:checkpass; no build outputs committed.Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.
The only process involved is the headless Ghidra analyzer that every Ghidra operation already starts; its capability effects are unchanged and the target itself is never launched or attached.
Evidence and regression coverage
src/domain/native/bytePattern.test.ts(whitespace, tab and no-break-space separators;??and bit masks combined; empty, odd-nibble,4?, non-hex, all-wildcard, all-zero-mask, unequal-length and non-hex masks rejected with typed issues on the right field);tests/boundary/mcp/ghidraInventoryMcp.test.ts(a page and its continuation retained as Evidence; a page whosecompletecontradictsnext_cursorrejected asunreadable_output; invalid patterns rejected before any provider call; cancellation projected as typedcancelledwith no structured content; an end-to-end run throughGhidraProviderand the socket bridge usingtests/fixtures/fakeGhidra.mjs, covering pagination withlimit: 1, cursor continuation, skipped.bsscoverage, file offsets and procedure);tests/boundary/mcp/officialArguments.test.ts(CLI and MCP dispatch the same arguments withlimit: 100defaulted;??, unequal mask,4?,limit: 1001andcursor: nullrejected before dispatch);tests/boundary/mcp/officialTools.test.ts(Hopper returnscapability_unavailableforsearch_byteswithout launching; strict schema rejects unknown keys and defaultslimitto 100).scripts/verify-real-ghidra-raw.mjsgains real-Ghidra assertions (see gaps).2e45b4d, give 7 failed and 9 passed: every new byte-search test fails (Tool search_bytes not found, or the contract lookup returnsundefined), and the pre-existing tests in those files still pass.bytePattern.test.tswas not run on the base because its module does not exist there.{"pattern": "48 8b ?? ?? 89", "limit": 1}ismatches: [{address: "0x1000", segment: ".text", file_offset: 0, procedure: {address: "0x1000", name: "fixture_main"}}],searched: {blocks: [{segment: ".text", start: "0x1000", end: "0x1009", bytes: 10}], bytes: 10, skipped_uninitialized: [{segment: ".bss", start: "0x2000", end: "0x200f", bytes: 16}]},complete: false; the continuation returns0x1005withfile_offset: 5andcomplete: true.{"pattern": "??"}fails withPattern must constrain at least one bit. On Hopper the call fails withcapability_unavailable,provider_id: "hopper",operation: "search_bytes". These values are fixture output, not a real Ghidra run.npm run verify:ghidra:raw(needsGHIDRA_INSTALL_DIR) was not run, and the new assertions inscripts/verify-real-ghidra-raw.mjswere not executed. Those assertions cover a single-match signature withfile_offset: 0and procedure, a 1005-match file paged as 1000 then 5 with the continuation starting at0x2770(no skip or repeat), abf/f0nibble mask, overlapping00 00matches at0x1002and0x1003, a cursor rejected for a different pattern, the target file unchanged after close, and first-search timing. Until that lane runs, the Java search loop, cursor binding and resume, block clamping and theaddress_to_file_offsetrefactor are source-derived; the TypeScript side is covered by the fixture boundary tests and unit tests above.For evidence-bearing changes:
Validation performed
All on macOS arm64 with Node 26.5.0 and npm 12.2.0.
On a clone at
2e45b4dwith only the updated boundary tests and fixture applied, the same focused run over the three boundary files fails 7 of 16 tests, all of them the new byte-search cases.npm run verify:ghidra:rawwas not run (no Ghidra install on this host).npm run check:prwas not run for this change.Compatibility, safety, and release
address_to_file_offsetfailure text, which now readsAddress lacks a unique safe file offset: <addr>, and its rejection of offsets aboveNumber.MAX_SAFE_INTEGER, which previously could not be represented exactly as a JavaScript number anyway. The new tool and command are additive.Before merge
type(scope): outcome.