Skip to content

feat(unreal): inspect Unreal pak footers and indexes - #4362

Open
8rulerstar wants to merge 99 commits into
morluto:mainfrom
8rulerstar:feat/unreal-pak
Open

8rulerstar wants to merge 99 commits into
morluto:mainfrom
8rulerstar:feat/unreal-pak

Conversation

@8rulerstar

@8rulerstar 8rulerstar commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Thank you for merging main into this branch to refresh CI. I merged the latest main again and fixed five reader boundaries found in a self-review; details are under Executed validation.

Summary

Adds inspect_unreal_pak (CLI: inspect-unreal-pak): static inspection of one local Unreal .pak (versions 1 to 12). It reads the footer and the index regions the footer names and returns each entry's path, offset, sizes, compression method, block facts, encryption and deletion flags and stored SHA-1. File data is never decoded. Refs #3625, narrower: no extraction, decompression, decryption or keys, no IoStore or .uasset.

Behavior and boundaries

  • src/unreal/UnrealPakFile.ts owns format interpretation; src/unreal/UnrealPakPathHash.ts owns the path hash rule. The service projects the decoded regions into Evidence locations, and the analysis view only projects retained Evidence (kind unreal-pak, collections entries by indices/paths and issues by indices/kinds, with fields; every row keeps its source index).
  • MCP defaults to a summary; the CLI exports complete Evidence with --detail summary --evidence_output FILE, matching inspect_il2cpp_metadata.
  • Redundant facts that disagree are recorded as offset-backed issues and decoding continues: region SHA-1, repeated path or path hash, entry counts between listings, an empty compression slot, a directory name without its closing /, and (version 11+, both indexes present) a directory index path whose FPakFile::HashPath value the path hash index does not map to the same record. An entry-count mismatch nulls index.entry_count; an empty slot nulls that entry's compression_name; the last two null the affected path, so no path is joined from a malformed directory name. A SHA-1 mismatch or repeated path is an issue only. Compression slot issues come only from records a listing references. Untrustworthy structure (out-of-file offsets or sizes, truncated records, dangling references) is refused with one of 12 named reasons.
  • Path hashes follow UE 5.8 PakFile.cpp and Misc/Fnv.cpp: ASCII lowercase, UTF-16LE code units, FNV-64 with the seed added to the offset basis. Version 10 is not checked: UE uses legacy constants there, repak 0.2.3 writes the version 11 hash, and no UE-written version 10 pak was available.
  • An encrypted index is not read: index.state: "encrypted", and empty entries/issues there mean "not read", not "none".
  • Admission before expansion: the index, path hash index and full directory index share one 256 MiB budget, entries are limited to 1,000,000 and the report to 64 MiB, refused from the declared entry count before entries are built; path text is counted in UTF-16 code units against the same 64 Mi budget before each join. paths-too-large reports resource transport with maximum_output_bytes and maximum_path_utf16_units. Size uses the shared jsonValueByteLength. Long decodes yield every 4,096 records, directories included. Peak memory is not measured.
  • Failure keeps facts that were complete: once the whole file was read and hashed, cancellation, close failure, schema failure or a finished report above the output budget keep partial Evidence with the hashed source identity only. The early output refusal from the declared entry count comes before hashing and keeps no partial Evidence. A cancellation observed after the reader returned keeps the decoded report.
  • Reads go through one owned descriptor (explicit absolute path, symlinks rejected, identity rechecked after hashing); src/artifacts/readStableArtifact.ts gains positioned reads with existing tests unchanged.
  • Scope: read-only inspection of a caller-selected file. No execution, injection, mutation or protection recovery, no new external toolchain, dependency or installer change. A new provider still needs the maintainer scope decision in [Feature] Unreal Engine reverse engineering: Pak/IoStore, UAsset reflection, and Blueprint bytecode reconstruction #3625.

Executed validation

At 7f194e500 (Greptile fixes on merge 1c4870b94 of main 8413ec08f; reader fixes in b7486a740):

  • Regressions written before the fix: 5 of 68 Unreal boundary tests failed on the previous reader (a directory name without / produced Game/ConfigDefaultGame.ini with no issue; a wrong path hash went unnoticed; 50,000 unreferenced records produced 50,000 issues; three regions each under the limit were accepted together; paths-too-large reported file-size without the path limit). After the fix, 68/68.
  • Greptile review of 1c4870b94, regressions first: a version 1 timestamp of real FDateTime ticks (about 6.4e17) failed region-out-of-bounds (58 passed / 1 failed in unrealPakFile.test.ts) and now reads (69/69 Unreal boundary tests). A 9,600,347-byte pak of 600,000 unreferenced records naming an empty slot ran out of a 128 MiB heap and now finishes (0 entries, 0 issues); structure-only walks keep no issues.
  • npm run check (typecheck, lint with 0 errors, format, knip, metadata, architecture guards), docs:check.
  • Focused: tests/boundary/unreal 69, unrealPakCli acceptance 4, stableArtifactConsumers 39.
  • Full suite at 7f194e500: 24,535 passed, 13 failed, 402 skipped; all 13 are outside Unreal (inspector, process, pcap, gdb, two acceptance files).
  • verify:unreal:pak on the UE 5.8.3 Blank58-Mac.pak: CLI/MCP parity, layout 12, 1,722 entries, 0 issues; all 1,722 directory index paths match their path hashes and the 1,722 paths repak list 0.2.3 prints. Replacing the seed by seed + 1 in a local experiment produced a path-hash-mismatch for the paths, so the comparison runs.
  • Not rerun at this head, 0 issues at b7486a740: 13 UE 5.8 paks from the CUE4Parse test fixtures (11 with both indexes, 2 with encrypted indexes) and 24 repak 0.2.3 paks (versions 1 to 11, stored and Zlib).

Contributor-reported at 162a90e8f (not rerun at the head above): 47 self-built paks matched repak 0.2.3 and CUE4Parse entry for entry, except CUE4Parse's single-block size convention; ten UE 5.8.3 UnrealPak variants also read correctly, including path-hash-only paks that both readers list as empty.

Tests: a byte-level fixture builder drives one boundary test per failure reason and issue kind and layouts v1, v3, v4 and v7 to v12, plus CLI/MCP parity. No threshold or exclusion changes.

Coverage limits

Not covered: Windows or console paks, real version 9 frozen indexes (refused), version 10 path hashes (unchecked), and layouts v2, v5 and v6 in the test matrix. Compression names identify methods only. Entry and hash agreement is structural evidence, not proof of a correct cook. Cost (time, memory) on large paks is unmeasured. No game was executed and no file was modified.

Contract and boundary impact

  • Semantic owner and earliest changed stage: src/unreal/UnrealPakFile.ts.
  • Contribution scope fit and linked maintainer decision: static format inspection; new provider, decision pending in [Feature] Unreal Engine reverse engineering: Pak/IoStore, UAsset reflection, and Blueprint bytecode reconstruction #3625.
  • CLI and MCP/tool-catalog contract: one new tool and CLI command; inspect_analysis_view gains kind unreal-pak, an Unreal pak summary and the inspect_unreal_pak parent operation, additively.
  • Provider, bridge, target-format, or platform compatibility: bundled, no external toolchain; pak versions 1 to 12.
  • Evidence, artifact, provenance, or reconstruction contract: observed Evidence with artifact SHA-256 and file-offset ranges of the footer and index regions read.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): see Behavior and boundaries.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: catalog entries regenerate; authored skill references updated; one verify:unreal:pak script; no package or installation change.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Author readiness

Human owner: @8rulerstar

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-16T04:23:11.588667Z 737479a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@greptile-apps

greptile-apps Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5 Tier: plus

[Medium impact] The documentation update appears safe; merging the feature still needs the acknowledged maintainer scope decision.

Summary

Adds read-only Unreal .pak inspection through CLI and MCP, with retained Evidence and paged entries. This update corrects the guide’s limit-error fields.

  • Unreal pak indexes now show footer facts and listed file entries.
  • CLI and MCP callers can inspect Unreal pak files and page their entries.

Reviews (52) · Last reviewed commit: "docs(unreal): show the boundary/maximum ..." · Reviewed by Greptile

Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread scripts/verify-unreal-pak.mjs Resolved
Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread src/artifacts/readStableArtifact.ts Resolved
Comment thread src/unreal/UnrealPakProvider.ts Resolved
Comment thread src/unreal/UnrealPakFile.ts Resolved
Comment thread src/unreal/UnrealPakProvider.ts Resolved
…nknown compression slots

Review found that a version 10+ index whose header count, path hash index and directory index disagreed, or whose listing regions had bytes left over, decoded as a partial success. The path hash region also holds a pruned directory index after the hashes, which UE 5.8.3 fills and repak writes empty; it is now read and its references checked. Version 12 file names reject a NUL, and a version 8+ entry naming an empty compression slot is corrupt.
…file and ranged stable reads

readStableArtifactRanges lived in its own module with a copy of readStableArtifact's symlink, identity, change and release checks. Both now run through one helper in readStableArtifact.ts, the ranged mode accepts the same filesystem injection, and the stable-artifact tests cover range decoding, out-of-range reads, symlinks and a change after decoding.
Index regions must match their recorded SHA-1, every entry must end before the index, and directory and file names may not build more path text than the report limit. Without these, a flipped index byte returned offsets past the end of the file as success, and a long directory name repeated across files could exhaust memory.
…crypted index bounds

Encoded references must name a record start found by walking the encoded entries, including per-block sizes, so a reference into the middle of a record fails instead of inventing entry facts. The encrypted index region is bounds- and limit-checked without being read. Directory counts no longer reach the entry-row hook, large path-hash-only indexes are appended without spreading, and tests now cover block counts, flags and record hashes. The verifier help names its own command.
Unreferenced encoded records could fill the walk's start set past the entry limit inside an allowed index region. The fixture now tracks the encoded length instead of rescanning earlier records.
The ranged reader hashed until EOF, so a file that kept growing after its ranges were decoded was followed past the snapshot before the change check ran. Growth now fails as an integrity error at the first extra byte.
Differential fuzzing against repak found that a path with an inner NUL was returned whole, while UE and repak stop at the first NUL. Such paths now fail as invalid-string.
@morluto

morluto commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Hey @8rulerstar, thanks for this! The addition looks good. What's left: it has merge conflicts with main in src/artifacts/readStableArtifact.ts and src/contracts/analysisViewToolContracts.ts (merging main into the branch is fine). Happy to get this in once that's sorted out.

@8rulerstar

8rulerstar commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Merged latest main (71f680d) into this branch in 0dc90bb. The two conflicts were the ones you named: in readStableArtifact.ts I kept this PR's refactor and carried over main's resolvedPath recovery hint for symlink rejection; in analysisViewToolContracts.ts I kept main's description and re-added the two Unreal pak phrases (the summary source list and the Unreal pak paging sentence). Typecheck, lint and the Unreal, artifact and contract tests pass locally (102 files). CI will cover docs:check, which I could not run on this machine.

# Conflicts:
#	src/artifacts/readStableArtifact.ts
#	src/contracts/analysisViewToolContracts.ts
@8rulerstar

Copy link
Copy Markdown
Contributor Author

The red checks are one test, tests/process-global/cutterDescriptorLifetime.test.ts ("does not assign a peer's later cancellation to a descriptor that already failed close", 2 errors instead of 1). This PR does not touch Cutter or the descriptor code, and the file passes locally (9/9), so it looks like a timing flake. I can't rerun upstream jobs from a fork; could you rerun the failed shard when you have a moment?

# Conflicts:
#	src/contracts/analysisViewToolContracts.ts
# Conflicts:
#	README.md
#	docs/.vitepress/config.ts
#	src/contracts/analysisViewToolContracts.ts
Comment thread docs/.vitepress/config.ts Resolved
Comment thread docs/unreal-pak.md Resolved
…e in the documentation directory

Imported Evidence can carry a different subject.local_path for the same pak,
so the view now checks the subject's format and digest, as the IL2CPP view
does. The guide moves from a hand-written sidebar entry to documentation.ts
and docs/README.md, which own navigation since morluto#2791.
@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: I ran the 7 changed test files on Node 22.23.4, 24.22.1 and 26.12.1, and all 299 tests passed on each. The CI workflow run on 42f9453 ended completed/failure. Its log names one failing test in tests/boundary/mcp/sourcePublicationComposed.test.ts, a file outside the 7 I ran.

Check Node 22.23.4 Node 24.22.1 Node 26.12.1
7 changed test files pass: 7/7 files, 299/299 tests pass: 7/7 files, 299/299 tests pass: 7/7 files, 299/299 tests

CI run 42545370971 on 42f9453 failed. Its log shows one failing test, in both mcp-boundary and mcp-boundary built: preserves the source when an inline result with unknown retention exceeds delivery. The AssertionError says an object with code resource_constraint did not match the expected object, which has code resource_constraint too. The log also says the lanes test and test-shard did not pass.

Check runs in failure: CI required, Portable tests and Test shard (4/8). Source tests, Static checks and seven other Test shard runs are success. Twelve check runs are skipped.

Workflow runs: CI is completed/failure, CodeQL production is completed/skipped, and PR readiness and PR title are completed/success. GitHub shows mergeable true and state blocked. The head is 2 commits behind main b46f4fc.

Not run

  • A run of a 600,000-record unreferenced-record pak under node --max-old-space-size=128
  • A merge with current main, b46f4fc
Test setup
  • Box: Fedora 44 VM (4 vCPUs, 8 GB) on my Nobara 44 PC, Linux 7.2.11
  • Node: 22.23.4, 24.22.1, 26.12.1
  • Head / base: 42f9453 / c78c8ae (merge base with main)
  • Install: npm ci --offline and npm run test:prepare on Node 24.22.1
Commands
npm ci --offline
npm run test:prepare
xvfb-run -a npm run test:focused -- tests/acceptance/unrealPakCli.test.ts tests/boundary/filesystem/stableArtifact.test.ts tests/boundary/mcp/unrealPakMcp.test.ts tests/boundary/unreal/unrealPakFile.test.ts tests/boundary/unreal/unrealPakProvider.test.ts tests/boundary/unreal/unrealPakView.test.ts tests/process-global/stableArtifactConsumers.test.ts

🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

Comment thread docs/specs/unreal/pak-index/spec.md Resolved
# Conflicts:
#	docs/specs/README.md
#	src/application/CapabilityInventory.ts
#	src/cli.ts
#	src/cliCommandNames.ts
#	src/composition/capabilityDeclarations.ts
#	src/composition/capabilityServices.ts
#	src/contracts/analysisViewToolContracts.ts
#	src/contracts/toolContracts.ts
#	src/contracts/toolFamilies.ts
#	src/server/createServer.ts
# Conflicts:
#	src/contracts/analysisViewToolContracts.ts
# Conflicts:
#	docs/specs/README.md
#	src/artifacts/readStableArtifact.ts
# Conflicts:
#	src/application/CapabilityInventory.ts
#	src/application/desktop/DesktopBuildExport.ts
#	src/cli.ts
#	src/cliCommandNames.ts
#	src/contracts/toolContracts.ts
#	src/contracts/toolEffects.ts
#	src/contracts/toolFamilies.ts
#	src/domain/analysisView/analysisView.ts
#	src/process/RegexMatcher.test.ts
#	src/resources/readStableArtifact.ts
#	src/server/createServer.ts
#	tests/boundary/cli/overnightBatch1CliSchemas.test.ts
#	tests/boundary/filesystem/stableArtifact.test.ts
… main

Classify pak failures through classifyProviderFailure, throw ParseFailureError
from the pak reader, read numbers through BinaryReader, register the CLI
through registerContractCommand and move the port under src/ports.
Comment thread tests/acceptance/unrealPakCli.test.ts Resolved
…11775

Main now reports parse-limit refusals as { boundary, maximum }, so the path budget test checks that shape.
Comment thread docs/unreal-pak.md Resolved
Limit refusals now report { boundary, maximum } after morluto#11775, and paths-too-large never emitted maximum_output_bytes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants