Skip to content

fix(javascript): preserve package and source-map boundary identity - #1939

Merged
morluto merged 12 commits into
mainfrom
codex/fix-javascript-package-boundaries
Oct 11, 2026
Merged

morluto merged 12 commits into
mainfrom
codex/fix-javascript-package-boundaries

Conversation

@morluto

@morluto morluto commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes #1881, fixes #1882, fixes #1883.

Bare ESM specifiers preserve query/fragment identity through exports lookup, package self-references use the correct loader rules, and retained prefixed source maps recover original-source evidence while preserving their bytes and digests.

The branch integrates @xu91102's implementation commits from #1888, #1884 and #1887, retaining their original authorship. Those PRs are superseded by this integration.

Behavior and ownership

  • Classify bare specifiers before URL suffix stripping. Exact exports keys retain their complete source spelling. Legacy ESM subpaths without exports decode only the derived filesystem path and select exact files; directory and extension/main/index fallback are refused. Exact misses explain what lookup was attempted.
  • Parse each bare package request once into its name, dependency subpath and exports eligibility. Self-resolution and ancestor lookup share that representation, replacing the separate name and eligibility helpers. ESM compares the parsed name exactly with metadata and reuses its subpath; CommonJS retains Node's raw-request equality/prefix rule and metadata-relative self subpath. A metadata name such as @scope therefore cannot steal an ESM import of the installed @scope/pkg dependency. Loader conditions and exports encapsulation remain intact.
  • After self-resolution, derive ESM dependency lookup from a package metadata URL once. Root suffixes such as legacy?x cannot select a literal suffixed directory or silently fall back to legacy. Exact suffixed self names and CommonJS literal dependency names retain their distinct behavior.
  • Reuse the existing exports-target selector directly for package imports, removing its redundant status wrapper. Self-reference and suffix cases share a Node control and CLI/MCP matrix.
  • Share the source-map parsing view across application analysis, browser fetching and point tracing. Original retained bytes, digests and source locations remain the evidence authority.

Contract and boundary impact

Artifact path resolution owns module identity and lookup; the domain source-map envelope owns the parsing view. CLI/MCP input schemas and tool catalog are unchanged. Module edges preserve original specifiers, source digests, locations and explicit unknowns. Process authority, package installation and generated metadata are unchanged.

Validation

Executed on macOS with pinned Node 24.18.0:

  • Latest resolver, package-boundary and public CLI/stdio MCP checks: 262 tests passed across nine files. Real Node controls cover malformed self names, valid scoped names, exact suffixed self names, literal root-suffix directory traps, ESM versus CommonJS, exact suffix keys, legacy directory/extension refusal, package import aliases, conditional exports, target validation and package precedence. Each test was bounded to five seconds.
  • Both the self-name and root-suffix regressions failed against their previous compiled runtimes with false resolved edges, then passed after their fixes. They verify source identity and digest as well as selected paths. The literal root-suffix filesystem fixture is explicitly skipped on Windows because it requires question-mark directory names; the self-name cases remain cross-platform.
  • The broader package integration run passed 166 tests across six files; source-map integration checks covered retained-prefix parsing and original-source nodes.
  • Runtime build, the complete static CI command, docs checks and pre-push checks pass on the branch merged with current main. Shared public CLI/MCP invocation and result validation remove repeated package-test setup. Duplication is 2.59%, below the unchanged 2.6% threshold.

No breaking input change or migration. These workflows exercise REA CLI/MCP and real Node resolution; they do not establish browser, Electron, Windows or Bun runtime behavior. Heavy resource-exhaustion tests and real Hopper/Ghidra/IDA providers were not run for this change.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T20:07:53.881360Z 84697a4 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a77c43b5a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5caa68f56f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d9efffbd7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: All three fixes work at 0d548f3, and the source-map fix is the same patch as xu91102's. Commit 3a77c43 broke imports with a suffix in old packages, and two later commits fixed that. Then 0d548f3 makes old packages refuse some imports, as Node does.

This follows my earlier #1887 comment. The link is below.

#1887 (comment)

The PR head moved while I tested. I ran 3a77c43, then 5caa68f, then 0d548f3. The newest head, 6d9efff, came after my runs. The base is d802304 (#1902), the merge base. It was main when I started, so this also covers a Not run item in my #1887 comment. Old packages here means packages with no exports field.

Source maps

Commit 6277401 holds the same patch as xu91102's 851cd98. The two diffs differ only in one git blob-hash line, which comes from the rebase. The author is kept. The later commits do not touch the source-map files.

I ran the same 17 fixture folders as before, through the CLI and the MCP server, on the base and on all three heads. The two agreed on every folder. The three heads gave identical results.

Map starts with Base d802304 All three heads
nothing (clean map) complete, 1 original complete, 1 original
six )]}' forms partial, 0 originals complete, 1 original
BOM only complete, 1 original complete, 1 original
six bad prefixes partial, 0 originals partial, 0 originals (rejected)
two BOMs, then )]}' partial, 0 originals complete, 1 original

This matches my #1887 results row for row. The six bad prefixes still say "Source map is not valid JSON". Inline data: folders still give 0 originals, as before.

The PR says it keeps original bytes, digests and source provenance. For every recovered folder, on the CLI and on MCP, I checked this:

  • source_map_sha256 equals the SHA-256 of the .map file on disk, prefix included.
  • The Evidence artifact digest on the recovered node equals that same hash.
  • source_sha256 equals the SHA-256 of the embedded source text, as UTF-8.
  • content_available is true and the map path is bundle.js.map.
  • Every file in each folder has the same hash before and after the run, so REA did not rewrite the map.

I added three folders with hard source text: accents, a fox emoji, CJK, CRLF, a tab, trailing spaces and U+2028. Two of them have a prefix. All three pass the checks above. Base recovers only the clean one.

The result holds a digest of the recovered source, not the text. So this is a digest match, not a byte-for-byte diff of the text.

The other two fixes

I used the files from issues #1881 and #1883, the shape from the PR test, and a legacy package. I also ran plain Node on every specifier, on Node 22.23.1, 24.18.0 and 26.11.1. All three agreed. The CLI and MCP agreed too.

Import Node loads it? Base 3a77c43 0d548f3
example/feature?cache=1 no linked not linked not linked
example/feature#entry no linked not linked not linked
example/literal?cache=1 yes not linked linked linked
./local.mjs?v=1 yes linked linked linked
self import, #1881 files yes not linked linked linked
self import, root and two keys yes not linked linked linked
self import, two missing keys no not linked not linked not linked
require of the package name yes linked linked linked
legacy/feature.js?cache=1 yes linked not found linked
legacy/feature.js#entry yes linked not found linked
legacy/with%20space.js?cache=1 yes not found not run linked
legacy/feature (no extension) no linked not run not found
legacy/?cache=1 (a directory) no linked not run rejected
legacy/nested/#entry (a directory) no linked not run rejected
legacy (the main field) yes linked not run linked
require of legacy/feature yes linked not run linked

"Linked" means REA resolved the import to the right file. "Not linked" means REA marked it external. Node refuses with ERR_PACKAGE_PATH_NOT_EXPORTED where it does not load the example and self rows. The example package exports ./feature and the exact key ./literal?cache=1. Self imports use the PR test shape. The legacy package has no exports field, a main field, and a nested folder with its own main.

Commit 3a77c43 broke the first two legacy rows. The base resolved them, and Node loads them. Commit 5caa68f fixed that. I ran 5caa68f on every row above the %20 row, and it matched 0d548f3.

What commit 0d548f3 changes, from its diff and my runs: an ESM import into an old package must now name a real file. A directory import is rejected, and the note says package main and index lookup were not tried. An import with no extension is not found. The ? and # suffix is cut off, and one percent-decoding pass runs, so %20 finds with space.js. CommonJS require keeps its old fallback. The base resolved all the directory and extension rows, and Node refuses them. The new rows now match Node.

Tests

Check Node 22.23.1 Node 24.18.0 Node 26.11.1
12 focused files, 384 tests, on 0d548f3 383 pass, 1 fails pass pass
npm run test:fast (399 files, 4,872 passed, 3 skipped) on 0d548f3 not run pass not run

The 12 files are the 3 new ones, the 4 related source-map files from my #1887 run, and 5 existing path-resolution files. The one Node 22 failure is in packageImportAliases.test.ts, which this PR does not touch. It fails the same way on the base: ERR_INVALID_MODULE_SPECIFIER. I did not look into why. The test:fast count is higher than in my #1887 run because main has moved on since then. Earlier heads 3a77c43 and 5caa68f gave the same counts and the same single Node 22 failure.

The three new test files fail on the base. The suffix test, in its 0d548f3 form, fails on the base with 2 failures out of 2.

Not run

  • The live browser path (analyze_web_bundle needs a CDP page). The PR's boundary test for the moved function passed.
  • recover_javascript_sources. It writes source maps and does not read retained ones.
  • A real-world open-source bundle with a prefixed map. I had none at hand.
  • npm run check, the full suite, and test:fast on Node 22 and 26.
  • Maps inside archives such as zip or asar.
  • macOS. The PR checks ran on macOS and mine ran on Linux.
  • Browser, Electron, Windows and Bun. The PR does not claim them, and I did not test them.
  • The last five legacy rows on 3a77c43. That build was gone by then.
  • Head 6d9efff. It came after my runs. Its commit is a refactor that merges the tests. I did not test it.

Thanks, xu91102, for the source-map fix and the clear issue. Thanks, morluto, for folding it in and for the follow-up commits.

All 17 fixtures and 3 byte checks
Map starts with Base d802304 All three heads
nothing (clean map) complete, 1 original complete, 1 original
)]}' then LF partial, 0 originals complete, 1 original
)]}' then CRLF partial, 0 originals complete, 1 original
)]}' plus text, then LF partial, 0 originals complete, 1 original
BOM, )]}' plus text, then CRLF (the PR test input) partial, 0 originals complete, 1 original
BOM, )]}', then LF partial, 0 originals complete, 1 original
BOM only complete, 1 original complete, 1 original
)]}' then CR only partial, 0 originals complete, 1 original
two BOMs, )]}', then LF partial, 0 originals complete, 1 original
)]}' with no newline partial, 0 originals partial, 0 originals (rejected)
foo then LF partial, 0 originals partial, 0 originals (rejected)
)]}' then half a map partial, 0 originals partial, 0 originals (rejected)
)]}' line only partial, 0 originals partial, 0 originals (rejected)
a space before )]}' partial, 0 originals partial, 0 originals (rejected)
)]}' twice partial, 0 originals partial, 0 originals (rejected)
inline data: map, clean complete, 0 originals complete, 0 originals
inline data: map, )]}' then LF complete, 0 originals complete, 0 originals
byte check: hard text, clean map complete, 1 original complete, 1 original
byte check: hard text, BOM, )]}' plus text, CRLF partial, 0 originals complete, 1 original
byte check: hard text, )]}' then LF partial, 0 originals complete, 1 original
Commands
# build each tree (Node 24.18.0); the lock file is identical, so one node_modules served all trees
npm ci && npm run build && npm run test:prepare

# source-map fixtures: 17 folders plus 3 byte-check folders, CLI and stdio MCP from each tree
node scripts/rea.mjs analyze-javascript-application <folder> --format json
# MCP: tool analyze_javascript_application, input_path=<folder>, format=directory,
# over stdio against node scripts/rea.mjs mcp from the same tree

# package fixtures (5 folders): same CLI and MCP calls, then plain Node on each specifier
node --input-type=module -e 'await import("<specifier>")'

# tests
npm run test:focused -- <12 files>      # Node 22 and 26: the same, with that node first on PATH
npm run test:fast

Timing is from a shared box. Other test jobs were running. The load average was 2.5 to 7 on 24 threads.

  • test:fast on Node 24: 90 s on 3a77c43, 92 s on 5caa68f, 86 s on 0d548f3.
  • 12 focused files on 0d548f3: 25.7 s on Node 24, 30.4 s on Node 22, 19.8 s on Node 26.
  • Builds: 13 s for the base (with npm ci), 6 to 8 s for each head.
  • The 34 source-map fixture runs for one tree: about 29 s.
Test setup

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9db68c0d81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95ec1e4220

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@morluto morluto added bug Something isn't working area: build Build, CI, packaging, and publishing labels Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: build Build, CI, packaging, and publishing bug Something isn't working

Projects

None yet

3 participants