Repository navigation
feat(unreal): inspect Unreal pak footers and indexes - #4362
8rulerstar wants to merge 99 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
[Medium impact] The documentation update appears safe; merging the feature still needs the acknowledged maintainer scope decision. SummaryAdds read-only Unreal
Reviews (52) · Last reviewed commit: "docs(unreal): show the boundary/maximum ..." · Reviewed by Greptile |
862dc21 to
bd0eade
Compare
ccef175 to
472a8d2
Compare
472a8d2 to
0749991
Compare
f1232a0 to
d9dd6c8
Compare
… a real-file verifier
…nknown compression slots Review found that a version 10+ index whose header count, path hash index and directory index disagreed, or whose listing regions had bytes left over, decoded as a partial success. The path hash region also holds a pruned directory index after the hashes, which UE 5.8.3 fills and repak writes empty; it is now read and its references checked. Version 12 file names reject a NUL, and a version 8+ entry naming an empty compression slot is corrupt.
…file and ranged stable reads readStableArtifactRanges lived in its own module with a copy of readStableArtifact's symlink, identity, change and release checks. Both now run through one helper in readStableArtifact.ts, the ranged mode accepts the same filesystem injection, and the stable-artifact tests cover range decoding, out-of-range reads, symlinks and a change after decoding.
Index regions must match their recorded SHA-1, every entry must end before the index, and directory and file names may not build more path text than the report limit. Without these, a flipped index byte returned offsets past the end of the file as success, and a long directory name repeated across files could exhaust memory.
…crypted index bounds Encoded references must name a record start found by walking the encoded entries, including per-block sizes, so a reference into the middle of a record fails instead of inventing entry facts. The encrypted index region is bounds- and limit-checked without being read. Directory counts no longer reach the entry-row hook, large path-hash-only indexes are appended without spreading, and tests now cover block counts, flags and record hashes. The verifier help names its own command.
Unreferenced encoded records could fill the walk's start set past the entry limit inside an allowed index region. The fixture now tracks the encoded length instead of rescanning earlier records.
The ranged reader hashed until EOF, so a file that kept growing after its ranges were decoded was followed past the snapshot before the change check ran. Growth now fails as an integrity error at the first extra byte.
Differential fuzzing against repak found that a path with an inner NUL was returned whole, while UE and repak stop at the first NUL. Such paths now fail as invalid-string.
# Conflicts: # src/contracts/analysisViewToolContracts.ts
|
Hey @8rulerstar, thanks for this! The addition looks good. What's left: it has merge conflicts with main in |
|
Merged latest main (71f680d) into this branch in 0dc90bb. The two conflicts were the ones you named: in |
# Conflicts: # src/artifacts/readStableArtifact.ts # src/contracts/analysisViewToolContracts.ts
|
The red checks are one test, |
# Conflicts: # src/contracts/analysisViewToolContracts.ts
# Conflicts: # README.md # docs/.vitepress/config.ts # src/contracts/analysisViewToolContracts.ts
…e in the documentation directory Imported Evidence can carry a different subject.local_path for the same pak, so the view now checks the subject's format and digest, as the IL2CPP view does. The guide moves from a hand-written sidebar entry to documentation.ts and docs/README.md, which own navigation since morluto#2791.
|
TL;DR: I ran the 7 changed test files on Node 22.23.4, 24.22.1 and 26.12.1, and all 299 tests passed on each. The
CI run 42545370971 on 42f9453 failed. Its log shows one failing test, in both Check runs in Workflow runs: Not run
Test setupCommandsnpm ci --offline
npm run test:prepare
xvfb-run -a npm run test:focused -- tests/acceptance/unrealPakCli.test.ts tests/boundary/filesystem/stableArtifact.test.ts tests/boundary/mcp/unrealPakMcp.test.ts tests/boundary/unreal/unrealPakFile.test.ts tests/boundary/unreal/unrealPakProvider.test.ts tests/boundary/unreal/unrealPakView.test.ts tests/process-global/stableArtifactConsumers.test.ts🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊 |
# Conflicts: # docs/specs/README.md # src/application/CapabilityInventory.ts # src/cli.ts # src/cliCommandNames.ts # src/composition/capabilityDeclarations.ts # src/composition/capabilityServices.ts # src/contracts/analysisViewToolContracts.ts # src/contracts/toolContracts.ts # src/contracts/toolFamilies.ts # src/server/createServer.ts
# Conflicts: # src/contracts/analysisViewToolContracts.ts
# Conflicts: # docs/specs/README.md # src/artifacts/readStableArtifact.ts
# Conflicts: # src/application/CapabilityInventory.ts # src/application/desktop/DesktopBuildExport.ts # src/cli.ts # src/cliCommandNames.ts # src/contracts/toolContracts.ts # src/contracts/toolEffects.ts # src/contracts/toolFamilies.ts # src/domain/analysisView/analysisView.ts # src/process/RegexMatcher.test.ts # src/resources/readStableArtifact.ts # src/server/createServer.ts # tests/boundary/cli/overnightBatch1CliSchemas.test.ts # tests/boundary/filesystem/stableArtifact.test.ts
… main Classify pak failures through classifyProviderFailure, throw ParseFailureError from the pak reader, read numbers through BinaryReader, register the CLI through registerContractCommand and move the port under src/ports.
# Conflicts: # src/resources/index.ts
…11775 Main now reports parse-limit refusals as { boundary, maximum }, so the path budget test checks that shape.
Limit refusals now report { boundary, maximum } after morluto#11775, and paths-too-large never emitted maximum_output_bytes.
Thank you for merging main into this branch to refresh CI. I merged the latest main again and fixed five reader boundaries found in a self-review; details are under Executed validation.
Summary
Adds
inspect_unreal_pak(CLI:inspect-unreal-pak): static inspection of one local Unreal.pak(versions 1 to 12). It reads the footer and the index regions the footer names and returns each entry's path, offset, sizes, compression method, block facts, encryption and deletion flags and stored SHA-1. File data is never decoded. Refs #3625, narrower: no extraction, decompression, decryption or keys, no IoStore or.uasset.Behavior and boundaries
src/unreal/UnrealPakFile.tsowns format interpretation;src/unreal/UnrealPakPathHash.tsowns the path hash rule. The service projects the decoded regions into Evidence locations, and the analysis view only projects retained Evidence (kindunreal-pak, collectionsentriesby indices/paths andissuesby indices/kinds, withfields; every row keeps its sourceindex).--detail summary --evidence_output FILE, matchinginspect_il2cpp_metadata.issuesand decoding continues: region SHA-1, repeated path or path hash, entry counts between listings, an empty compression slot, a directory name without its closing/, and (version 11+, both indexes present) a directory index path whoseFPakFile::HashPathvalue the path hash index does not map to the same record. An entry-count mismatch nullsindex.entry_count; an empty slot nulls that entry'scompression_name; the last two null the affectedpath, so no path is joined from a malformed directory name. A SHA-1 mismatch or repeated path is an issue only. Compression slot issues come only from records a listing references. Untrustworthy structure (out-of-file offsets or sizes, truncated records, dangling references) is refused with one of 12 named reasons.PakFile.cppandMisc/Fnv.cpp: ASCII lowercase, UTF-16LE code units, FNV-64 with the seed added to the offset basis. Version 10 is not checked: UE uses legacy constants there, repak 0.2.3 writes the version 11 hash, and no UE-written version 10 pak was available.index.state: "encrypted", and emptyentries/issuesthere mean "not read", not "none".paths-too-largereports resourcetransportwithmaximum_output_bytesandmaximum_path_utf16_units. Size uses the sharedjsonValueByteLength. Long decodes yield every 4,096 records, directories included. Peak memory is not measured.src/artifacts/readStableArtifact.tsgains positioned reads with existing tests unchanged.Executed validation
At
7f194e500(Greptile fixes on merge1c4870b94of main8413ec08f; reader fixes inb7486a740):/producedGame/ConfigDefaultGame.iniwith no issue; a wrong path hash went unnoticed; 50,000 unreferenced records produced 50,000 issues; three regions each under the limit were accepted together;paths-too-largereportedfile-sizewithout the path limit). After the fix, 68/68.1c4870b94, regressions first: a version 1 timestamp of realFDateTimeticks (about 6.4e17) failedregion-out-of-bounds(58 passed / 1 failed inunrealPakFile.test.ts) and now reads (69/69 Unreal boundary tests). A 9,600,347-byte pak of 600,000 unreferenced records naming an empty slot ran out of a 128 MiB heap and now finishes (0 entries, 0 issues); structure-only walks keep no issues.npm run check(typecheck, lint with 0 errors, format, knip, metadata, architecture guards),docs:check.tests/boundary/unreal69,unrealPakCliacceptance 4,stableArtifactConsumers39.7f194e500: 24,535 passed, 13 failed, 402 skipped; all 13 are outside Unreal (inspector, process, pcap, gdb, two acceptance files).verify:unreal:pakon the UE 5.8.3Blank58-Mac.pak: CLI/MCP parity, layout 12, 1,722 entries, 0 issues; all 1,722 directory index paths match their path hashes and the 1,722 pathsrepak list0.2.3 prints. Replacing the seed by seed + 1 in a local experiment produced apath-hash-mismatchfor the paths, so the comparison runs.b7486a740: 13 UE 5.8 paks from the CUE4Parse test fixtures (11 with both indexes, 2 with encrypted indexes) and 24 repak 0.2.3 paks (versions 1 to 11, stored and Zlib).Contributor-reported at
162a90e8f(not rerun at the head above): 47 self-built paks matched repak 0.2.3 and CUE4Parse entry for entry, except CUE4Parse's single-block size convention; ten UE 5.8.3 UnrealPak variants also read correctly, including path-hash-only paks that both readers list as empty.Tests: a byte-level fixture builder drives one boundary test per failure reason and issue kind and layouts v1, v3, v4 and v7 to v12, plus CLI/MCP parity. No threshold or exclusion changes.
Coverage limits
Not covered: Windows or console paks, real version 9 frozen indexes (refused), version 10 path hashes (unchecked), and layouts v2, v5 and v6 in the test matrix. Compression names identify methods only. Entry and hash agreement is structural evidence, not proof of a correct cook. Cost (time, memory) on large paks is unmeasured. No game was executed and no file was modified.
Contract and boundary impact
src/unreal/UnrealPakFile.ts.inspect_analysis_viewgains kindunreal-pak, an Unreal pak summary and theinspect_unreal_pakparent operation, additively.observedEvidence with artifact SHA-256 and file-offset ranges of the footer and index regions read.docs/public/product-catalog.json), package, or installation impact: catalog entries regenerate; authored skill references updated; oneverify:unreal:pakscript; no package or installation change.Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.
Author readiness
Human owner: @8rulerstar
Before merge
type(scope): outcome.