Skip to content

feat(native): search analyzed memory for masked byte patterns - #2341

Merged
morluto merged 5 commits into
morluto:mainfrom
mvanhorn:fix/masked-byte-pattern-search
Oct 13, 2026
Merged

morluto merged 5 commits into
morluto:mainfrom
mvanhorn:fix/masked-byte-pattern-search

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

Summary

Agents can now find where a byte signature occurs in an opened native target: search_bytes (CLI search-bytes) takes hexadecimal byte pairs with ?? wildcards and an optional bit mask, searches Ghidra's initialized analyzed memory, and returns ordered matches with segment, source file offset and containing procedure, plus exact page coverage and a bound continuation cursor. Hopper advertises the operation as unavailable with a typed reason.

No issue is linked. This is a new native tool, proposed here with the implementation.

Demo

Masked byte search demo

Full-quality MP4

The video shows real output from this branch's MCP server and Ghidra provider code, driven against the repository's fake Ghidra fixture (tests/fixtures/fakeGhidra.mjs), since Ghidra is not installed on this host. The match addresses come from the fixture. It shows 48 8b ?? ?? 89 matching at 0x1000 and 0x1005 with .bss listed as skipped, the same search paged with limit: 1 and next_cursor, and ?? alone rejected as invalid.

Problem and expected behavior

read_bytes reads bytes at an address the caller already knows, and search_strings and procedure search cover names and strings. On upstream/main (2e45b4d) nothing answers "where does 48 8B ?? ?? 89 occur", so locating a constant table, a prologue, a magic header or an opcode sequence with wildcarded operands means reading memory page by page and matching client side.

With this change, one call such as {"pattern": "48 8B ?? ?? 89", "limit": 100} on an open Ghidra session returns observed Evidence with matches (address, segment, file_offset or null, procedure {address, name} or null), searched (blocks and bytes examined on this page, skipped_uninitialized), complete and next_cursor. Repeating the same query with cursor: next_cursor continues until complete is true.

Change and scope

Pattern grammar is owned by src/domain/native/bytePattern.ts and shared by the MCP contract, the Ghidra inventory schema and the CLI. A pattern is hexadecimal byte pairs (either case) or ??, with optional whitespace between pairs. Empty patterns, odd nibbles (4, 4?), non-hex characters and patterns where every bit is unconstrained (??, or ff with mask 00) are rejected as typed input errors before dispatch. The optional mask is hexadecimal pairs of the same length; each mask byte selects which bits of the pattern byte must match (pattern: "a0", mask: "f0" matches a0 through af). ?? stays fully unconstrained even when a mask is given, and an omitted mask means ff for every non-wildcard byte.

Optional start and end are inclusive and must lie in one address space with start <= end. Without bounds every memory block is eligible, in Ghidra address order. limit accepts 1 to 1000 and defaults to 100.

The Ghidra bridge (searchMemoryBytes in bridge/ghidra/ReaGhidraBridge.java) walks blocks sorted by start address and calls Memory.findBytes per initialized block with the end bound pulled back by length - 1, so a match never crosses a block boundary. After each hit it resumes at the next address, so overlapping matches are kept (00 00 over 00 00 00 gives two hits). Uninitialized blocks inside the bounds are never searched and are listed in searched.skipped_uninitialized on every page. searched.blocks reports the inclusive ranges examined for this page, including the lookahead that found the next unreturned match, so coverage can overlap between pages and should not be summed as unique memory.

The cursor is <binding>:<address>. The binding is SHA-256 over a JSON array of the target SHA-256, profile digest, normalized pattern bytes, masks, canonical start and end, and the document. The address is the first match not yet returned, so resuming neither skips nor repeats a match. The bridge rejects a cursor whose binding differs from the current target and query, or whose address is outside initialized memory or the query bounds. Cancellation is checked between blocks and around each findBytes call; it returns a typed cancelled error, discards the active Ghidra database and never produces a successful partial page.

The bridge re-validates its request at the socket boundary: required keys document, pattern, limit, allowed extras mask, start, end, cursor, anything else is invalid_request. It also re-applies the pattern, mask, constraint and bounds rules instead of trusting the TypeScript layer.

file_offset comes from a new shared uniqueFileOffset helper that returns an offset only when exactly one file-backed source mapping covers the address and the offset fits a JavaScript safe integer. address_to_file_offset now uses the same helper. That is a small visible change: its failure text changes from Address has N authoritative source mappings: <addr> to Address lacks a unique safe file offset: <addr>, and an offset above Number.MAX_SAFE_INTEGER is now rejected instead of returned.

Hopper lists search_bytes in its capabilities with available: false and the reason "Masked byte search has not been verified for Hopper.", so the call fails as capability_unavailable without launching Hopper.

Out of scope: a Hopper implementation, searching uninitialized or overlay memory, matches spanning block boundaries, string or regex search over bytes, and any write, patch or execution.

Contract and boundary impact

  • Semantic owner and earliest changed stage: domain grammar, parser and result schema in src/domain/native/bytePattern.ts (bytePatternFields, parseBytePattern, validateBytePattern, byteSearchResultSchema); Ghidra provider input/result schemas in src/ghidra/GhidraInventoryValues.ts; search in bridge/ghidra/ReaGhidraBridge.java.
  • Contribution scope fit and linked maintainer decision: a static query over memory Ghidra has already loaded for the caller's target; no target execution, injection, mutation or protection recovery, so no maintainer scope decision is required. Not applicable.
  • CLI and MCP/tool-catalog contract: new MCP tool search_bytes (src/contracts/hopperMemoryContracts.ts, strict input with pattern, mask?, document, start?, end?, limit 1 to 1000 default 100, cursor?; output schema in src/contracts/toolOutputSchemaGroups.ts; example {pattern: "48 8B ?? ?? 89"} in src/contracts/toolContractExamples.ts; effect evidence in src/contracts/toolEffects.ts). New CLI command search-bytes <path> <pattern> --mask --start --end --cursor --limit plus the existing native open options (src/cli/coreNativeCommands.ts, src/cliCommandNames.ts, src/application/DirectAnalysis.ts). The result schema rejects pages where complete disagrees with next_cursor or searched.bytes disagrees with the block sum, and the Ghidra schema also requires canonical addresses.
  • Provider, bridge, target-format, or platform compatibility: Ghidra gains search_bytes in its inventory, bridge dispatch and capability limitations (src/ghidra/GhidraProviderCapabilities.ts); same effects as other Ghidra reads, snapshot cache policy. Hopper reports the operation unavailable (src/hopper/HopperProviderCapabilities.ts). address_to_file_offset error text changes as described above.
  • Evidence, artifact, provenance, or reconstruction contract: observed Evidence bound to the session's target and profile. Each page carries its own coverage and skipped uninitialized blocks; the capability limitations state that coverage is per page, that uninitialized memory and cross-block spans are not searched, and that cancellation yields no partial page.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): uses the target the caller opened (or the CLI path) through the existing Ghidra provider path; reads loader-adjusted memory only. No new processes, writes or temp files beyond what every Ghidra operation already uses. Cancellation discards the active Ghidra database through the existing provider path.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: docs in docs/ghidra-provider.md (Masked byte search) and docs/native-investigation.md (Locate a byte signature); three rows in docs/user-visible-errors.csv (unconstrained pattern, unequal mask length, cursor reused with another query). The new tool and command appear in the generated catalog; no authored skill text changed. metadata:check and docs:check pass; no build outputs committed.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

The only process involved is the headless Ghidra analyzer that every Ghidra operation already starts; its capability effects are unchanged and the target itself is never launched or attached.

Evidence and regression coverage

  • Tests added or updated: src/domain/native/bytePattern.test.ts (whitespace, tab and no-break-space separators; ?? and bit masks combined; empty, odd-nibble, 4?, non-hex, all-wildcard, all-zero-mask, unequal-length and non-hex masks rejected with typed issues on the right field); tests/boundary/mcp/ghidraInventoryMcp.test.ts (a page and its continuation retained as Evidence; a page whose complete contradicts next_cursor rejected as unreadable_output; invalid patterns rejected before any provider call; cancellation projected as typed cancelled with no structured content; an end-to-end run through GhidraProvider and the socket bridge using tests/fixtures/fakeGhidra.mjs, covering pagination with limit: 1, cursor continuation, skipped .bss coverage, file offsets and procedure); tests/boundary/mcp/officialArguments.test.ts (CLI and MCP dispatch the same arguments with limit: 100 defaulted; ??, unequal mask, 4?, limit: 1001 and cursor: null rejected before dispatch); tests/boundary/mcp/officialTools.test.ts (Hopper returns capability_unavailable for search_bytes without launching; strict schema rejects unknown keys and defaults limit to 100). scripts/verify-real-ghidra-raw.mjs gains real-Ghidra assertions (see gaps).
  • Base reproduction or other evidence: executed. The three updated boundary test files plus the updated fake Ghidra fixture, copied onto a clone at 2e45b4d, give 7 failed and 9 passed: every new byte-search test fails (Tool search_bytes not found, or the contract lookup returns undefined), and the pre-existing tests in those files still pass. bytePattern.test.ts was not run on the base because its module does not exist there.
  • User-visible CLI/MCP output (if applicable): from the fake Ghidra fixture, the first page of {"pattern": "48 8b ?? ?? 89", "limit": 1} is matches: [{address: "0x1000", segment: ".text", file_offset: 0, procedure: {address: "0x1000", name: "fixture_main"}}], searched: {blocks: [{segment: ".text", start: "0x1000", end: "0x1009", bytes: 10}], bytes: 10, skipped_uninitialized: [{segment: ".bss", start: "0x2000", end: "0x200f", bytes: 16}]}, complete: false; the continuation returns 0x1005 with file_offset: 5 and complete: true. {"pattern": "??"} fails with Pattern must constrain at least one bit. On Hopper the call fails with capability_unavailable, provider_id: "hopper", operation: "search_bytes". These values are fixture output, not a real Ghidra run.
  • Remaining proof gaps: real Ghidra is not installed on this host, so npm run verify:ghidra:raw (needs GHIDRA_INSTALL_DIR) was not run, and the new assertions in scripts/verify-real-ghidra-raw.mjs were not executed. Those assertions cover a single-match signature with file_offset: 0 and procedure, a 1005-match file paged as 1000 then 5 with the continuation starting at 0x2770 (no skip or repeat), a bf/f0 nibble mask, overlapping 00 00 matches at 0x1002 and 0x1003, a cursor rejected for a different pattern, the target file unchanged after close, and first-search timing. Until that lane runs, the Java search loop, cursor binding and resume, block clamping and the address_to_file_offset refactor are source-derived; the TypeScript side is covered by the fixture boundary tests and unit tests above.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

All on macOS arm64 with Node 26.5.0 and npm 12.2.0.

npm run test:focused -- src/domain/native/bytePattern.test.ts \
  tests/boundary/mcp/ghidraInventoryMcp.test.ts \
  tests/boundary/mcp/officialArguments.test.ts \
  tests/boundary/mcp/officialTools.test.ts     # pass: 4 files, 25 tests
npm run check:fast                             # pass (typecheck, lint)
npm run check                                  # pass (typecheck, lint:built 0 errors, format:check, knip, metadata:check, architecture guards across 1213 files)
npm run docs:check                             # pass
npm run verify:test-discovery                  # pass: 951 test files discovered once
npm run metadata:check                         # pass

On a clone at 2e45b4d with only the updated boundary tests and fixture applied, the same focused run over the three boundary files fails 7 of 16 tests, all of them the new byte-search cases.

npm run verify:ghidra:raw was not run (no Ghidra install on this host). npm run check:pr was not run for this change.

Compatibility, safety, and release

  • Breaking changes or migration steps: none for callers of existing tools beyond the address_to_file_offset failure text, which now reads Address lacks a unique safe file offset: <addr>, and its rejection of offsets above Number.MAX_SAFE_INTEGER, which previously could not be represented exactly as a JavaScript number anyway. The new tool and command are additive.
  • Real Hopper/Ghidra, browser, or OS coverage: no real Ghidra run; fixture and unit coverage only, on macOS. Hopper reports the operation unavailable by design and is not exercised beyond that.
  • Package or release metadata impact: new tool and command appear in the generated catalog; no dependency changes.
  • Security, privacy, process, or containment review: read-only query over memory Ghidra has already loaded; no target execution, writes to the target, or network. Results are bounded at 1000 matches per page, cursors are bound to target, profile and query, and cancellation is checked inside the search loop.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: I ran this PR on real Ghidra 12.1.4. Its own check passes, and the results match a Python search of the raw bytes. The red CI is not from this PR: main at 1c37914 failed the same tests.

Results

Head c6380a9, Ghidra 12.1.4, Node 24.18.0 unless noted.

Check Result
npm run verify:ghidra:raw (the PR's lane) pass. The old copy of that script also passes on the same tree.
Block edges, start, end, limits, masks, bad patterns, cursor edits (106 cases, 32-byte raw image) all as the PR says
ELF with .bss, 16 queries same as the Python search in matches, order and offsets. Two bad bounds were rejected. .bss shows as skipped on every page.
libxml2 shared library, 1.5 MB (MIT AND ISC-Veillard AND W3C), 11 queries, up to 25,725 matches same as the Python search. Differences sit only in blocks the loader changed, and match what read_bytes returns.
CLI search-bytes, cursor from an earlier run works. No match skipped or repeated.
address_to_file_offset, 32 inputs, head against the base tree all 32 identical
The PR's 4 test files, Node 22, 24 and 26 25 of 25 pass on each

The PR's lane timed its first search at 85 ms. On libxml2 the first search took about 100 to 170 ms, after a 34 s import.

CI

The red checks are the #2343 high-bit test and the referenceSourceImportStatus tests. Main's own CI at 1c37914 failed the same tests. Main has changed both tests since, and its CI at 3b0ea59 passed.

Two questions

  1. The PR calls overlay memory out of scope. An open-ended search does cover it. On a real ELF it walks file parts that are not loaded, such as .symtab, and reports them as .symtab:0x0. Is that intended?
  2. A match must end at or before end. So end set to the first byte of a 4-byte match finds nothing. This is fine, but one docs sentence would help.

Not run

  • The new failure text of address_to_file_offset. I could not build a block that reaches it.
  • Cancelling a search part way.
  • Hopper, and PE, Mach-O or MIPS targets.
  • Any whole test suite.

Thanks, mvanhorn, for the clear PR and for writing the real-Ghidra check. The check ran as written, first time.

Test setup
  • Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
  • CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
  • RAM: 32 GB DDR4-3600
  • Node: 22.23.1, 24.18.0, 26.11.1
  • Head: c6380a9
  • Base: 2e45b4d
  • Ghidra: 12.1.4 (Apache-2.0)
  • JDK: Eclipse Temurin 21.0.12.1
  • JDK license is GPL-2.0-only plus the Classpath exception.
  • clang: 22.1.8 (Apache-2.0 WITH LLVM-exception OR NCSA)
  • The box was shared with another test agent. Load was not measured.
Commands

Run with a private HOME and TMPDIR, under systemd-run --user --scope -p MemoryMax=8G.

npm ci && npm run build && npm run test:prepare
node scripts/verify-real-ghidra-raw.base.mjs     # control: old script, same tree
npm run verify:ghidra:raw
npm run test:focused -- src/domain/native/bytePattern.test.ts \
  tests/boundary/mcp/ghidraInventoryMcp.test.ts \
  tests/boundary/mcp/officialArguments.test.ts \
  tests/boundary/mcp/officialTools.test.ts
node scripts/rea.mjs search-bytes libxml2.so.2.12.10 "48 8b ?? ?? 89" --limit 5 --format json --provider ghidra

My own harness opened one target over MCP and ran many search_bytes calls. A Python script searched the raw file bytes as the check.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@matbrgz

matbrgz commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Adversarial Review Findings: search_bytes Pagination & Coverage Accounting

While scrutinizing ReaGhidraBridge.java and bytePattern.ts for boundary and lifecycle edge cases, there are critical edge cases in cursor continuation and page coverage accounting:


1. Page Boundary Match Duplication / Infinite Pagination Loop (ReaGhidraBridge.java:110-114)

When matches.size() == limit, the search stops and constructs next_cursor:

if (matches.size() == limit) {
    next = binding + ":" + canonicalAddress(found);
    examinedEnd = found.addNoWrap(bytes.length - 1L);
    break;
}

Notice that:

  1. found is the address of the $(limit)$-th match (e.g. if limit == 100, found is the 100th matching pattern start).
  2. It sets next = binding + ":" + canonicalAddress(found).
  3. In the subsequent page request where cursor is passed, resume = parseReaAddress(...) is initialized to that exact found address:
    if (resume != null && low.compareTo(resume) < 0) low = resume;
    ...
    Address at = low; // at = found!
    while (at != null && at.compareTo(high) <= 0 && high.subtract(at) >= bytes.length - 1L) {
        Address found = currentProgram.getMemory().findBytes(at, ...);
  4. findBytes(at, ...) searches starting at at inclusively. Because at == found, findBytes immediately finds the exact same match at found again!
  5. Consequently, the $(limit)$-th match from page $N$ is re-returned as the 1st match of page $N+1$.
  6. Furthermore, if limit == 1, page 1 returns match $M_1$ and next_cursor pointing to $M_1$. Calling page 2 with next_cursor immediately hits matches.size() == 1 at $M_1$, returning match $M_1$ again with the identical next_cursor, causing an infinite pagination loop.

Fix:
The cursor should point to found.next() (or resume from resume.next()), or next_cursor must encode the next candidate address to search rather than re-emitting the final accepted match. Note that care must be taken if found is at the very end of the block (found.next() wraps or transitions across blocks).


2. Duplicate Blocks and Overlapping Coverage Reporting Across Cursor Pages (ReaGhidraBridge.java:99-130)

In page 1:

  • Block $B$ starts at 0x1000 and ends at 0x2000.
  • The search stops at match 100 at 0x1500.
  • coverage = byteSearchCoverage(block, low, examinedEnd) emits start: 0x1000, end: 0x1500 + pattern.length - 1, and blocks records this chunk.

In page 2 (with cursor resuming at 0x1500):

  • low = resume (i.e. 0x1500).
  • The remaining part of Block $B$ is searched from 0x1500 to 0x2000.
  • coverage = byteSearchCoverage(block, low, examinedEnd) emits start: 0x1500, end: 0x2000.
  • Notice that [0x1500, 0x1500 + pattern.length - 1] is reported as searched in both page 1 and page 2!
  • More subtly, byteSearchResultSchema in bytePattern.ts states:

    "Coverage describes this page; reuse the same query with next_cursor to continue."
    Because examinedEnd overlaps across pages (or if next resumes at found.next(), the bytes from found + 1 to found + pattern.length - 1 have already been partially examined), client consumers aggregating searched.bytes will double-count covered bytes across pagination boundaries.


3. Unchecked Overflow on found.addNoWrap(bytes.length - 1L) (ReaGhidraBridge.java:112)

When found is near the top of a 32-bit or 64-bit address space segment (e.g. 0xFFFFFFFF in a 32-bit segment), found.addNoWrap(...) throws AddressOverflowException:

Address examinedEnd = high;
...
if (matches.size() == limit) {
    next = binding + ":" + canonicalAddress(found);
    examinedEnd = found.addNoWrap(bytes.length - 1L);
    break;
}

If found.addNoWrap(...) throws an unhandled AddressOverflowException, the entire search request crashes with an internal RequestFailure or socket disconnection rather than returning the page cleanly.
examinedEnd should simply be bounded by high (found.add(bytes.length - 1L) or checking against high).

The diff adds search_bytes to the CLI, the shared MCP contract, Ghidra
inventory, and the headless bridge, and advertises the same operation on
Hopper as unavailable until that provider is verified. Patterns are
hexadecimal byte pairs with optional ?? wildcards and an optional
equal-length bit mask; the TypeScript boundary and the bridge both
require a nonempty pattern, equal lengths, and at least one constrained
bit. Ghidra walks initialized blocks in address order within inclusive
bounds, keeps overlapping matches inside a single block, and returns
each hit with its segment, unique source file offset, and containing
procedure, continuing through a cursor bound to the target, profile,
pattern, mask, and bounds. Each page reports the inclusive ranges
examined and the skipped uninitialized blocks; cancellation returns a
typed error and discards the active Ghidra database.
address_to_file_offset now uses the same unique-offset helper, which
rejects a mapping that is missing, ambiguous, or larger than a safe
integer.
Without start or end, search_bytes walked every Ghidra block, including
non-loaded file parts such as .symtab and overlay blocks, which the docs
call out of scope. Unbounded searches now skip those blocks and list them
in searched.skipped_not_loaded; explicit bounds in their address space
still search them. Document that a match must end at or before end.
@mvanhorn
mvanhorn force-pushed the fix/masked-byte-pattern-search branch from c6380a9 to 742a02b Compare October 13, 2026 04:06
@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: I retested the new head, 742a02b, on real Ghidra 12.1.4. Matches are never repeated or skipped, and the new loaded-memory rule works. One small thing is still true: each page boundary lists a few bytes as searched twice.

Results

The old head is c6380a9. The new head is 742a02b. Both ran the same fixtures on Ghidra 12.1.4. The Python search is an independent python3 -I search of the raw file bytes.

Check Old head New head
npm run verify:ghidra:raw pass (my earlier comment) pass
Review finding 1: a full page repeats its last match (72 paged runs) never never
limit 1 loops forever (stopped at 20 pages if so) no no
All pages joined vs the Python search same same
Review finding 2: bytes listed as searched on two pages yes yes, see below
ELF (1,960 bytes), unbounded 00 00 1,239 matches 593 matches
libxml2, unbounded: a call, 8 zero bytes, bf mask f0 19,987 / 12,516 / 25,725 19,869 / 12,108 / 23,962
Earlier checks: 109 (106 before, plus 3 complete flag checks) all pass all pass
address_to_file_offset, 32 inputs same answers same answers
The PR's 4 test files on Node 22, 24 and 26 25 of 25 each 25 of 25 each
CI red, from main 39 of 39 pass

The lower counts are the overlay blocks that the old head also searched. The other matches keep their order and file offsets. Skipped blocks now show in searched.skipped_not_loaded.

Review finding 2: reproduces, in a small way

Thanks, matbrgz, for the careful review. Page 1 reports bytes up to the end of the next match. Page 2 starts at that match. So that many bytes, as many as the pattern is long, are listed twice. The matches are still right.

Smallest repro: a 32-byte file opened with --target-format raw-binary --architecture x86_64 --entry-point 0x1000, with the bytes 90909090deadbeef909090909090cafe00000041bfb8a5af90909090909090c3. Search 90 90 with limit 3.

Page Matches Searched
1 0x1000, 0x1001, 0x1002 0x1000 to 0x1009
2 0x1008, 0x1009, 0x100a 0x1008 to 0x100c

Bytes 0x1008 and 0x1009 are on both pages. Should page 1 stop one byte before its cursor address?

Not run

  • The new failure text of address_to_file_offset. I could not build a block that reaches it.
  • A rebuild of the base 2e45b4d. Its answers come from my earlier run.
  • A block that is not loaded but is not an overlay. Every such block I could make is an overlay.
  • Cancelling a search part way, Hopper, and PE, Mach-O or MIPS targets.
  • Any whole test suite.

Thanks, mvanhorn, for the fast follow-up. This answers both of my earlier questions. Overlay blocks are skipped now. The docs also say how end works.

Review finding 1: not reproduced

Thanks, matbrgz, for the close read. The cursor holds the first match the page did not return. The loop checks limit before it adds a match. So the next page starts on a new match. I tried limit 1, 2, 3, 16 and 17 with patterns of 1 to 4 bytes. I also used wildcards, a mask and libxml2. No page repeated a match, and no cursor repeated.

What 742a02b changed, and one odd case

With no start or end, blocks that are not loaded, or that are overlays, are skipped. They show in searched.skipped_not_loaded. A bound inside an overlay still searches it. On the ELF the skipped blocks are .comment, .shstrtab, .strtab, .symtab and the section headers. On libxml2 they are four overlay blocks. The 646 matches dropped on the ELF are exactly the matches that sat in those blocks.

Odd case: I edited a cursor to point into .symtab:0x10 on an unbounded 00 00 search with limit 3. The new head returned 0 matches, complete true and no searched blocks. The old head resumed inside .symtab. I did not trace why. Is an error better here?

Paging and coverage, every run

Repeats counts repeated matches, repeated cursors and a last match that came back on the next page. "Bytes listed twice" is the sum of page coverage minus the covered bytes. ep-0000-limit1 stopped at 20 pages on purpose. The 00 00 rows differ between heads because the old head also searched the overlay blocks.

Run Pattern limit Pages old / new Matches old / new Repeats Bytes listed twice old / new
p-90-limit1 90 1 17 / 17 17 / 17 0 16 / 16
p-90-limit2 90 2 9 / 9 17 / 17 0 8 / 8
p-90-limit3 90 3 6 / 6 17 / 17 0 5 / 5
p-90-limit16 90 16 2 / 2 17 / 17 0 1 / 1
p-90-limit17 90 17 1 / 1 17 / 17 0 0 / 0
p-9090-limit1 90 90 1 14 / 14 14 / 14 0 26 / 26
p-909090-limit2 90 90 90 2 6 / 6 11 / 11 0 15 / 15
p-0000-limit1 00 00 1 2 / 2 2 / 2 0 2 / 2
p-wild-limit1 90 ?? 90 1 11 / 11 11 / 11 0 30 / 30
p-mask-limit1 bf mask f0 1 3 / 3 3 / 3 0 2 / 2
p-bounded-limit1 90 1 13 / 13 13 / 13 0 12 / 12
p-4byte-end-limit1 90 90 90 90 1 8 / 8 8 / 8 0 28 / 28
ep-dead-limit1 de ad be ef 1 5 / 5 5 / 5 0 16 / 16
ep-dead-limit2 de ad be ef 2 3 / 3 5 / 5 0 8 / 8
ep-dead-limit4 de ad be ef 4 2 / 2 5 / 5 0 4 / 4
ep-dead-limit5 de ad be ef 5 1 / 1 5 / 5 0 0 / 0
ep-0000-limit100 00 00 100 13 / 6 1,239 / 593 0 24 / 10
ep-0000-limit1000 00 00 1,000 2 / 1 1,239 / 593 0 2 / 0
ep-0000-limit1 00 00 1 20 / 20 20 / 20 0 38 / 38
ep-movabs-limit1 48 b8 and 8 wildcards 1 1 / 1 1 / 1 0 0 / 0
Test setup
  • Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
  • CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
  • RAM: 32 GB DDR4-3600
  • Node: 22.23.1, 24.18.0, 26.11.1
  • Head: 742a02b
  • Base: a6efa4f (test(mcp): reuse advertised schema in inventory check #2469)
  • Old head: c6380a9
  • Base I built earlier: 2e45b4d
  • Ghidra: 12.1.4 (Apache-2.0)
  • JDK: Eclipse Temurin 21.0.12.1
  • JDK license is GPL-2.0-only plus the Classpath exception.
  • libxml2: 2.12.10-6.fc44 (MIT AND ISC-Veillard AND W3C)
  • libxml2 is a 1.5 MB system library. I copied it to a private folder.
  • The fixtures are a 32-byte raw image and a 1,960-byte ELF with a .bss section. I built the ELF with clang in my earlier run.
  • Timing: the first search on libxml2, after a 33 s import, took 106 ms on 742a02b and 121 ms on c6380a9. The PR's own check timed its first search at 77 ms. The box was shared with another test agent, and I did not measure load.
Commands

Run with a private HOME and TMPDIR, under systemd-run --user --scope -p MemoryMax=8G.

npm ci && npm run build && npm run test:prepare
npm run verify:ghidra:raw
npm run test:focused -- src/domain/native/bytePattern.test.ts \
  tests/boundary/mcp/ghidraInventoryMcp.test.ts \
  tests/boundary/mcp/officialArguments.test.ts \
  tests/boundary/mcp/officialTools.test.ts
node scripts/rea.mjs search-bytes bssfix.elf "00 00" --limit 3 --format json --provider ghidra
node scripts/rea.mjs search-bytes bssfix.elf "00 00" --limit 3 --cursor <next_cursor> --format json --provider ghidra

My own harness opened one target over MCP and ran many search_bytes calls. A Python script searched the raw file bytes as the check.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@mvanhorn

Copy link
Copy Markdown
Contributor Author

Thanks for running this on real Ghidra 12.1.4 and checking it against a raw-byte search. That is more than I could do here.

  1. No, that was not intended. Without start or end, the search now covers loaded memory only. Non-loaded blocks such as .symtab, and overlay blocks, are listed in a new searched.skipped_not_loaded and are not searched. Passing start or end in their address space still searches them.
  2. Agreed. docs/ghidra-provider.md now says a match must end at or before end, so an end inside a match excludes it.

I also rebased onto current main, which should clear the stale-base CI failures you traced. Locally, npm run check, docs:check, test discovery and the four focused test files pass (25 tests). I have no Java or Ghidra on this machine, so the Java change has not run against real Ghidra yet. If you rerun your .symtab case, .symtab should now appear under skipped_not_loaded instead of as a match block.

@CassieFox23

Copy link
Copy Markdown
Contributor

Thanks! My retest just above ran that: .symtab now shows under searched.skipped_not_loaded, as you said. Two small open questions are in it, if useful: page boundaries list a few bytes as searched twice, and a cursor edited to point into a skipped block returns no matches.

🤖 Generated with Claude Code (Opus 5.5) 🦊

@morluto

morluto commented Oct 13, 2026

Copy link
Copy Markdown
Owner

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

This PR is set for a Plus review. TREX isn't supported with Plus or Apex reviews yet, so Greptile reviews this PR at Base while TREX is on.

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The changes appear safe to merge; the earlier null-cursor issue is fixed and no new blocking issue was found.

Summary

Adds search_bytes and CLI search-bytes for masked searches over Ghidra memory.

  • Ghidra searches analyzed memory for masked byte patterns.
  • File-offset lookups now return only unique, safely representable offsets.

Reviews (3) · Last reviewed commit: "fix(native): reject present malformed by..." · Reviewed by Greptile

Comment thread src/domain/native/bytePattern.ts Resolved
Comment thread bridge/ghidra/ReaGhidraBridge.java Resolved
@morluto

morluto commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Review findings addressed at their producing boundaries and consumers. Latest changes were reviewed from source; tests and provider runs were skipped at the maintainer's request. Findings requiring separate decisions or verification remain open.

  1. thread 1

Fixed in 82cc51f. searchMemoryBytes now distinguishes omitted optional state from a present value: cursor uses requireString only when its key is present, so null, non-string and empty continuation values fail invalid_request instead of restarting page one. The same owning boundary rejects present malformed mask/start/end values while preserving genuine omissions. Source inspection and git diff --check passed; tests and real Ghidra verification were not run at explicit user request. No related cleanup remains.

@morluto morluto added enhancement New feature or request area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior area: bridge Hopper bridge (Python) related changes area: cli CLI adapter and one-shot commands P2: medium Medium priority issue labels Oct 13, 2026
@morluto
morluto merged commit 2a78763 into morluto:main Oct 13, 2026
44 checks passed
@morluto morluto mentioned this pull request Oct 14, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: bridge Hopper bridge (Python) related changes area: cli CLI adapter and one-shot commands area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior enhancement New feature or request P2: medium Medium priority issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants