Repository navigation
Filesystem stores: nested resource file layout behind one factory - #758
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
…e factory Map a resource key to a nested on-disk path with a leaf extension (.md content, .json state) instead of one flat encodeURIComponent(wholeKey) file, so the store root is a browsable, diffable file tree. - New resource-path.ts: injective key<->path mapping (per-segment escaping, extension as the sole literal dot, traversal/reserved-name validation, symlink-safe collectRecords with defensive canonicalization + BP-033 prefix narrowing). - New createFilesystemResourceStore<T> generic factory: both stores unified behind one CRUD impl (scope-id validation + containment, per-op ancestor and leaf symlink guards, durable per-subtree .fsdev-store-layout clean-break guard, collision surfacing, shared ENOENT-retry atomic write). content-store and resource-state-store are now thin config; content gains the state store's concurrent-scope resilience. - Wire the shared conformance suite against the filesystem adapter in both engine test files; add the files/src/utils.ts dotted-extension case; add on-disk-layout and legacy-guard tests + resource-path unit tests. - Docs: nested-tree layout in the source headers, persistence overview, durable-execution table, and engine README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
Review found getByPrefix with a >=2-complete-leading-segment prefix (e.g. "a/b/x") descended through prefix-narrowed intermediate dirs but only lstat'd the final start dir, so a symlinked intermediate (scopeDir/a -> /outside) could enumerate out-of-scope files. collectRecords now lstat's scopeDir AND every intermediate narrowed segment, rejecting a symlink anywhere on the path (removing the now-dead narrowStartDir helper and its duplicate validation). Also: add store-level symlink-safety tests (leaf, ancestor, deleteAll-unlink, the intermediate-symlink regression) and a corrupt (non-JSON) marker test; unexport decodeSegment (no external consumer, BP-038). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
There was a problem hiding this comment.
Simplify review: the nested layout and createFilesystemResourceStore factory are proportionate for FIX-890. Flat-only or no factory would recreate duplication; third-party path libs would still need custom encoding and legacy guard logic. Inherent complexity lives in resource-path.ts and symlink or marker policy; optional follow-ups are deduping KeyedResourceStore, caching marker after first set, and routing atomic writes through shared.ts. Performance cost is O(depth) lstat per op versus flat layout, acceptable for local dev. Approving; see inline notes for non-blocking cleanup ideas.
Sent by Cursor Automation: Review For Best Approach
| }; | ||
|
|
||
| /** The six-method keyed-resource-store contract shared by both public stores. */ | ||
| export interface KeyedResourceStore<T> { |
There was a problem hiding this comment.
Type duplication: KeyedResourceStore is also declared privately in resource-store-conformance.ts with the same six methods. Consider one internal definition and keep ContentStore / ResourceStateStore as the only public registry types. If this export is unused outside engine, dropping the public KeyedResourceStore export would match BP-038.
| value: T | ||
| ): Promise<void> { | ||
| await this.ensureLayout(); | ||
| await this.ensureMarker(); |
There was a problem hiding this comment.
ensureMarker on every set: after the first stamp, each write still mkdirs the subtree root and attempts exclusive create on the marker, relying on EEXIST. Correct, but extra syscalls on the hot path. A private markerEnsured flag (or trusting a successful marker read in resolveLayout) would simplify steady-state writes.
| * not-yet-created segment (a fresh scope has none). Guards against a recursive | ||
| * `mkdir`/`rm` walking through a symlinked ancestor to escape the store. | ||
| */ | ||
| private async assertAncestorsSafe(target: string): Promise<void> { |
There was a problem hiding this comment.
Symlink guard cost: assertAncestorsSafe runs on every get/set/delete/prefix scan with O(depth) sequential lstats. Appropriate for an editable vault, but the main perf delta vs the old flat store. Caching a verified scope dir chain per process would be a small follow-up if large trees hurt; not a merge blocker for me.
| } catch (error) { | ||
| throw this.collisionError(error, scopeType, scopeId, resourceKey, target); | ||
| } | ||
| const tempPath = `${target}.tmp-${process.pid}-${Date.now()}-${Math.random() |
There was a problem hiding this comment.
Reuse shared.ts atomicWrite: temp path plus writeFile plus rename matches atomicWrite in filesystem/shared.ts. Here you also need parent mkdir, ENOENT retry, collision mapping, and temp cleanup. A tiny atomicWriteWithParentRetry in shared.ts would centralize temp-file naming.
| * dot-hide. "*" is the one Windows-invalid char `encodeURIComponent` leaves | ||
| * through. | ||
| */ | ||
| export function encodeSegment(segment: string): string { |
There was a problem hiding this comment.
Keeping encoding in this module is the right boundary; I would not fold it back into filesystem-resource-store.ts unless file count is the only goal. resource-path.test.ts as the oracle is the simplification win.
| @@ -124,6 +124,19 @@ function runKeyedResourceStoreConformance<V>(options: ConformanceOptions<V>): vo | |||
| }); | |||
There was a problem hiding this comment.
Conformance already defines KeyedResourceStore locally. Export it once from here (or import the factory return type against this definition) to avoid drift with filesystem-resource-store.ts.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a1eaded. Configure here.
| // Fresh (empty / only empty dirs / only denylisted metadata): proceed | ||
| // WITHOUT writing anything, so read-only deployments and fresh-root probes | ||
| // stay non-mutating. The marker is a `set` responsibility. | ||
| } |
There was a problem hiding this comment.
Stale layout cache bypasses guard
Medium Severity
After the first successful ensureLayout on an empty unmarked subtree, later operations reuse a memoized promise and never re-run resolveLayout. On-disk changes such as adding nested resource files or removing .fsdev-store-layout are therefore not re-checked, so the legacy guard can be skipped for the lifetime of that store instance.
Reviewed by Cursor Bugbot for commit a1eaded. Configure here.
jhoffner
left a comment
There was a problem hiding this comment.
second-look + improve-codebase-architecture pass on this PR — see inline comment on filesystem-resource-store.ts for the full writeup.
Verdict: appropriately scoped. Checked each safety mechanism (symlink guards, Windows reserved-name rejection, collision guard, legacy-layout marker) against whether it has a real, present trigger — all either trace to a review-caught bug (the getByPrefix symlink case), an existing sibling-file precedent (request-store.ts's NTFS : hardening), or an explicit owner-signed-off decision documented in the PR body. No YAGNI/speculative-surface findings clear the bar.
One non-blocking architecture note: this PR's set() adds a third near-identical copy of the tmp-file+rename atomic-write pattern already duplicated twice in shared.ts. Not asking for a change in this PR (consolidating would mean touching pre-existing, out-of-scope code), but flagging as a candidate for a small follow-up — same for the encodeSegment-per-store fragmentation across this directory (request-store.ts, resource-path.ts, shared.ts each define their own).
Generated by Claude Code
There was a problem hiding this comment.
/second-look: appropriately scoped, one architecture note (non-blocking)
Went through this with a "was this the right shape / is any of it speculative" pass, cross-checked against docs/specs intent (the spec PR #741 body — the doc itself wasn't merged) and the sibling filesystem stores in this same directory. Verdict: appropriately scoped, not overbuilt. Net production code is ~+400 lines (415 new in resource-path.ts + filesystem-resource-store.ts, offset by ~-250 gutted from the two old classes), which lines up with a from-scratch path-mapping + one real, actively-exercised safety surface (symlink guards, legacy-layout guard, scope-id containment) for a store whose whole point is now to be a user-editable/git-diffable vault that outside tools (editors, git, symlinks) touch directly. I checked each guard against a "does this have a present, non-hypothetical trigger":
- Symlink safety (ancestors + leaf + intermediate narrowed dirs in
collectRecords) — not speculative; the PR body's own decision Add CI workflow for type checking and testing #4 says review caught a realgetByPrefixsymlink-escape case on a narrowed intermediate dir, now regression-tested. Kept. - Windows reserved-device-name rejection (
isWindowsReservedName,resource-path.ts:22-34) — no CI runs on Windows and no doc claims Windows support, so I first flagged this as speculative. Butrequest-store.ts:51in this same directory already hardens:for "Windows/NTFS" portability as a pre-existing, deliberate convention — this PR is consistent with an established sibling pattern, not inventing a new one. Not a finding. - Collision guard (
collisionError, ~16 lines) — the PR body itself calls this "effectively dormant" (decision Rewrite framework docs to showcase power and developer experience #5). True, but it's cheap defense-in-depth for a hand-editable tree and the removal is too small to be worth the row on its own. Not a finding. - Legacy clean-break guard + marker — this is decision Improve server runtime logs for action/block execution #2 in the PR body, explicitly owner-signed-off after "four spec-review rounds" against the dual-read alternative. Deliberate, not incidental. Not a finding.
One thing worth a note, not a blocker: this file's set() (lines ~303-321) hand-rolls a third copy of the tmp-file-plus-rename pattern that already exists twice in ../shared.ts (atomicWrite and, separately, the inline version inside writeRecord) — same ${target}.tmp-${pid}-${Date.now()}-${random} shape each time. It's not pure duplication (this version adds ENOENT-retry-on-mkdir and always cleans up the tmp file on failure, which the existing atomicWrite helper doesn't do), so I'm not asking for a change here — consolidating would mean also touching the two pre-existing copies, which is out of scope for a PR that's deliberately scoped to the two resource-keyed stores (decision #5). Flagging per CLAUDE.md's "common helpers belong in a shared utils file" only as a candidate for a small follow-up: a shared atomicWriteWithCleanup(target, content) in shared.ts that both filesystem-resource-store.ts and writeRecord call, so there's one canonical crash-safe-write implementation instead of three near-identical ones in this directory.
Also checked: resource-path.ts's new local encodeSegment (escapes ./*) is yet another sibling to request-store.ts's local encodeSegment (escapes :) and shared.ts's/suspension-store.ts's inline encodeURIComponent(id). Same observation as above — real fragmentation, but pre-existing and out of scope for this PR's intentionally-narrow blast radius (opaque-id stores are explicitly left alone per decision #5). Worth a standalone follow-up ticket to unify path-segment encoding across packages/engine/src/stores/filesystem/*, not a change here.
No YAGNI/speculative-surface findings that clear the bar (net removal + articulable loss + not a documented/precedented decision). Tests are well-proportioned too — the full legacy-guard/symlink matrix lives once in content-store.test.ts, with only a light 3-test smoke check in resource-state-store.test.ts since the guard logic is centralized in this factory; not duplicated across both.
Generated by Claude Code
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1eaded096
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (entry.name === LAYOUT_MARKER_NAME) continue; | ||
| if (METADATA_FILE_DENYLIST.has(entry.name)) continue; |
There was a problem hiding this comment.
Count metadata-named legacy resource files
When an old flat store contains a valid resource key named .DS_Store, Thumbs.db, .gitignore, or .gitkeep and no other data file, this denylist makes the legacy guard classify the subtree as fresh. Reads then silently return undefined from the new nested path, and the next write stamps the new-layout marker over the legacy data. These names were accepted by the previous store, so the upgrade guard must distinguish root metadata from files inside scope directories or reject these keys explicitly.
Useful? React with 👍 / 👎.
| if (entry.isDirectory()) { | ||
| if (METADATA_DIRS.has(entry.name)) continue; | ||
| if (await this.subtreeHasDataFile(full)) return true; |
There was a problem hiding this comment.
Scan metadata-named scope directories
When a legacy scope ID is exactly .git, .obsidian, .svn, or .hg, this recursive skip hides every resource in that scope from the clean-break scan. Such scope IDs were valid in the previous store; with no other legacy files present, the new adapter reports their resources as missing and can stamp a nested-layout marker alongside the unread flat files. Metadata directories should only be ignored where they cannot represent an encoded scope ID.
Useful? React with 👍 / 👎.
| await this.ensureLayout(); | ||
| await this.ensureMarker(); |
There was a problem hiding this comment.
Check ancestors before writing the layout marker
When <root>/content or <root>/state is a symlink to an otherwise empty directory, ensureLayout() succeeds and this call writes .fsdev-store-layout through the symlink before assertAncestorsSafe() rejects the operation below. Thus a failed set() still mutates a directory outside the configured store root, contradicting the per-operation symlink guard; build the target and validate its ancestors before stamping the marker.
Useful? React with 👍 / 👎.
| if (stat.isSymbolicLink()) { | ||
| throw new Error(`Refusing to read symlinked resource file: ${target}`); | ||
| } | ||
| return this.deserialize(await readFile(target, "utf8")); |
There was a problem hiding this comment.
Read through the checked file descriptor
When another local process can modify the store directory concurrently, it can replace the leaf with a symlink after lstat(target) succeeds but before this path-based readFile() opens it, causing get() to return data from outside the store root. getAll() has the same race between directory enumeration and its later readFile(record.absolutePath). To make the advertised symlink guard effective under concurrent mutation, open without following symlinks and read through that checked file descriptor rather than checking and reopening by path.
Useful? React with 👍 / 👎.
| async deleteAll(scopeType: ContentScopeType, scopeId: string): Promise<void> { | ||
| // Skips the legacy guard: an upgraded install with legacy data must be able | ||
| // to tear a scope down without a read first. | ||
| const dir = this.scopeDir(scopeType, scopeId); |
There was a problem hiding this comment.
Refuse deleteAll for an incompatible layout marker
When an older process opens a store carrying a future or otherwise unexpected layout marker, reads and writes correctly refuse it, but deleteAll() bypasses marker validation and recursively removes the v1 scope path anyway. This is reachable through the normal session deletion route (session-routes.ts calls both resource stores' deleteAll methods), so a rollback or mixed-version deployment can delete data owned by a layout it cannot interpret. Bypass the guard only for an absent legacy marker; an existing corrupt or unsupported marker must still block deletion.
Useful? React with 👍 / 👎.
…eAll marker guard - Run assertAncestorsSafe before ensureMarker in set(): a symlinked <root>/content could have .fsdev-store-layout written through it by a set() that the ancestor check was about to reject (Codex P2). - deleteAll now refuses a PRESENT-but-incompatible marker instead of bypassing the guard entirely — never rm -rf data owned by a layout this build can't interpret (a version-rollback path); an absent marker (legacy/fresh) still deletes so upgraded installs can tear old scopes down (Codex P2). Shared the marker-validation out of resolveLayout into markerValidOrAbsent(). - Regression tests for both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
Thanks all — triaged the review round. Two real bugs fixed in Fixed (
Documented, accepted tradeoff — no change (spec §11 #4):
By-design / non-blocking — no change:
Follow-up (per @jhoffner's architecture note): the tmp-file+rename atomic-write and per-store
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 172ebba5c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| private async markerValidOrAbsent(): Promise<boolean> { | ||
| let raw: string; | ||
| try { | ||
| raw = await readFile(this.markerPath, "utf8"); |
There was a problem hiding this comment.
Reject symlinked layout markers
When .fsdev-store-layout is a symlink, this path-based read follows it and trusts JSON outside the configured subtree. A symlink to an external nested-v1 marker therefore bypasses the legacy-data scan, so existing flat resources are silently treated as missing; a dangling marker is likewise treated as absent and later ensureMarker() ignores its EEXIST, allowing data to be written without a readable marker. Explicitly lstat and reject a symlinked marker before reading or creating it.
Useful? React with 👍 / 👎.
A path-based read of .fsdev-store-layout followed a symlink, so a planted symlink to an external nested-v1 marker would bypass the legacy-data scan (existing flat resources silently read as missing); ensureMarker's wx write would EEXIST on a symlink and leave it standing. Both paths now lstat the marker and reject a symlink, closing the last unguarded path in the store's symlink-safety invariant (Codex P2). Regression-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
Fixed in Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77583eb62c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| throw error; | ||
| } | ||
| for (const entry of entries) { | ||
| if (entry.isSymbolicLink()) continue; // never follow a symlinked dir/file |
There was a problem hiding this comment.
Treat legacy symlinks as data during the layout scan
When an old flat store uses a symlink as a resource file, the previous implementation follows it on get(), but this scan skips it and can classify the subtree as fresh. A subsequent write then stamps the nested-layout marker, after which the legacy resource silently reads as missing; count symlink entries as legacy data without following them so the clean-break guard refuses this store.
Useful? React with 👍 / 👎.
| } catch (error) { | ||
| if (errno(error) !== "EEXIST") throw error; |
There was a problem hiding this comment.
Validate an existing marker before writing v1 data
When this store instance has already cached a successful layout check and another process or migration tool replaces the marker with nested-v2, the next set() reaches this EEXIST branch and ignores the incompatible marker before writing a v1-formatted file. This defeats the mixed-version protection already applied to reads and deleteAll(); on EEXIST, re-read and validate the marker before proceeding.
Useful? React with 👍 / 👎.
jhoffner
left a comment
There was a problem hiding this comment.
second-look review posted above (verdict + architecture opportunity). Two inline comments point at the specific test-consolidation finding.
Generated by Claude Code
| }); | ||
| }); | ||
|
|
||
| describe("FilesystemContentStore legacy clean-break guard", () => { |
There was a problem hiding this comment.
This block plus the "FilesystemContentStore symlink safety" block below (through line ~470) exercise FilesystemResourceStore guard behavior that's identical for the .json state store — but resource-state-store.test.ts only gets 3 of these ~15 cases and none of the symlink-safety ones (see comment there). Since this PR already introduced the shared createFilesystemResourceStore factory and the shared resource-store-conformance.ts pattern, consider extracting these two describe blocks into a parameterized createFilesystemStoreGuardConformanceTests({name, createStore, ext, seedLegacyValue})-style helper that both test files call, instead of this file carrying the full matrix alone.
Generated by Claude Code
| }); | ||
| }); | ||
|
|
||
| describe("FilesystemResourceStateStore legacy clean-break guard", () => { |
There was a problem hiding this comment.
This only covers 3 of the ~15 legacy-guard cases in content-store.test.ts's equivalent block, and has no symlink-safety coverage at all (corrupt marker, wrong-version marker, symlinked marker, symlinked leaf/ancestor/scope-dir, marker-not-stamped-through-symlink — all untested here). Both stores run the identical FilesystemResourceStore class from filesystem-resource-store.ts, so a regression in e.g. the symlinked-leaf guard would only be caught on the .md content path, not .json state. Suggest pulling both stores' legacy-guard + symlink-safety tests into one shared parameterized suite (see companion comment in content-store.test.ts) so coverage stays symmetric by construction.
Generated by Claude Code
…eMarker EEXIST - subtreeHasDataFile now counts a symlink ENTRY as legacy data (without following it) instead of skipping it — an old flat store using a symlink as a resource file was being misclassified fresh, silently dropping it and stamping a new-layout marker (Codex P2). - ensureMarker validates the existing marker on EEXIST instead of ignoring it: if another process swapped it to an incompatible version after this instance cached its layout check, set() no longer writes v1 data under it — extends the mixed-version protection already on reads and deleteAll (Codex P2). - Tests: legacy-symlinked-resource refused; marker-swap-to-v2 blocks set. The symlinked-ancestor test now seeds a valid marker so it isolates the ancestor guard (the scan itself now counts symlinks as data). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb3b470328
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| private ensureLayout(): Promise<void> { | ||
| return (this.layoutPromise ??= this.resolveLayout().catch((error) => { |
There was a problem hiding this comment.
Revalidate the marker after caching layout success
After any operation successfully resolves nested-v1, this cached promise prevents get, getAll, and delete from checking the marker again. If a migration or mixed-version process subsequently replaces it with nested-v2 or corrupt data, the same store instance keeps interpreting—and, through delete, mutating—the subtree as v1, even though fresh instances, set(), and deleteAll() refuse that layout. Revalidate the marker for subsequent data operations, at least before destructive ones, rather than permanently bypassing it after the first success.
Useful? React with 👍 / 👎.
| await writeFile(this.markerPath, JSON.stringify({ layout: LAYOUT_VERSION }), { | ||
| flag: "wx" | ||
| }); |
There was a problem hiding this comment.
Publish the layout marker atomically
If the process is terminated or the machine loses power after writeFile(..., { flag: "wx" }) creates the marker but before its contents are fully written, an empty or partial marker remains. Every later operation then treats the marker as corrupt and refuses the otherwise valid subtree until an operator manually removes it. Write the completed marker through an atomic exclusive-publication path, analogous to the store's temp-file-and-rename payload writes, so a crash cannot publish a torn marker.
Useful? React with 👍 / 👎.
…idates; atomic marker Test consolidation (per review): extract the legacy-guard + symlink-safety + on-disk-layout cases into createFilesystemStoreGuardConformanceTests, run against BOTH the content (.md) and state (.json) stores. Both run the identical factory, so coverage is now symmetric by construction — the state store gains the full ~16-case matrix it was missing (it had 3, no symlink coverage). Two more mixed-version/crash-safety fixes (Codex P2): - delete() now re-validates the marker before its destructive rm, like set() and deleteAll() — a marker swapped to an incompatible version by another process no longer lets a cached-v1 instance mutate a v2 subtree. - ensureMarker publishes the marker atomically (temp-in-parent + rename) instead of writeFile wx, so a crash mid-write can't leave a torn/empty marker that later reads as corrupt. This also subsumes the prior symlink/EEXIST handling via markerValidOrAbsent (validates on every set). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
Done in @jhoffner — test consolidation (your review): extracted the legacy-guard + symlink-safety + on-disk-layout cases into Two more Codex P2s fixed in the same commit:
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b93bc6717
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // and stop if it's already a valid v1 marker. Re-checking on every set means | ||
| // a marker another process swapped to an incompatible layout blocks the | ||
| // write instead of letting v1 data land under it. | ||
| if (await this.markerValidOrAbsent()) return; |
There was a problem hiding this comment.
Re-scan the subtree before publishing an absent marker
When a cold read has cached an empty, unmarked subtree and an older process or migration tool subsequently writes flat-layout files, the next set() reaches this absent-marker result without repeating the legacy-data scan and then stamps nested-v1. Those legacy resources thereafter silently read as missing, defeating the clean-break guard; an absent marker on the write path must trigger a fresh data scan rather than trusting the cached read.
Useful? React with 👍 / 👎.
| ); | ||
| try { | ||
| await writeFile(tmp, JSON.stringify({ layout: LAYOUT_VERSION }), "utf8"); | ||
| await rename(tmp, this.markerPath); |
There was a problem hiding this comment.
Avoid replacing a concurrently published marker
When another process publishes an incompatible marker after markerValidOrAbsent() reports absent but before this rename, POSIX rename replaces that destination with nested-v1, allowing this process to write v1 data into a subtree another version owns. The fresh evidence after the earlier marker-validation finding is that atomic publication now uses a replacing rename rather than an exclusive operation whose collision can be re-read and validated.
Useful? React with 👍 / 👎.
| // `this.root`) never counts it as data; rename within the same filesystem is | ||
| // atomic. Concurrent first-sets publish identical bytes, so the race is safe. | ||
| const tmp = path.join( | ||
| path.dirname(this.root), |
There was a problem hiding this comment.
Keep marker temp files inside the owned subtree
When rootDir/content or rootDir/state is pre-created writable but rootDir itself is not writable, the first set() fails with EACCES because this temp file is created in the parent even though the store is documented to own only its subtree. A separately mounted subtree also makes the subsequent rename fail with EXDEV; publish the temp file on the destination filesystem inside the owned subtree and exclude that exact temp pattern from the legacy scan.
Useful? React with 👍 / 👎.
…marker publish Three follow-ups to the atomic-marker change (Codex): - ensureMarker re-scans the subtree for legacy data before publishing an absent marker, instead of trusting the cached ensureLayout result. A cold read that cached "fresh" could otherwise let a set() stamp nested-v1 over flat files another process wrote since, silently hiding them (P1). - The publish temp now lives INSIDE the owned subtree (not the parent), so a read-only or separately-mounted parent can't cause EACCES/EXDEV; the legacy scan skips `<marker>.tmp-*` (startsWith the marker name). - Publish via hard-link (atomic + EEXIST-exclusive) instead of a replacing rename, so a marker another version published concurrently is re-validated rather than clobbered with v1. Test: cached-fresh-read then legacy-appears then set -> throws (both stores). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
All three addressed in
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d94a2bf68b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| if (!entry.isFile()) continue; | ||
| // Skip the marker AND its in-flight publish temps (`<marker>.tmp-…`). | ||
| if (entry.name.startsWith(LAYOUT_MARKER_NAME)) continue; |
There was a problem hiding this comment.
Match only actual marker temporary files
When an old flat store contains a valid resource whose key starts with .fsdev-store-layout (for example .fsdev-store-layout-notes) and no other data, this broad startsWith check treats it as an internal marker temporary file. The subtree is then classified as fresh, a marker can be published, and the legacy resource becomes silently inaccessible; skip only the exact temporary-file naming pattern rather than every file sharing the marker prefix.
Useful? React with 👍 / 👎.
| // can be stale if another process swapped the marker to an incompatible | ||
| // layout since — delete must never mutate a subtree this build can't | ||
| // interpret (matches deleteAll / set). | ||
| await this.markerValidOrAbsent(); |
There was a problem hiding this comment.
Rescan before deleting under an absent marker
When this instance has cached an empty unmarked subtree and an older process subsequently writes a flat resource whose key includes the new extension (for example legacy key foo.md, stored at the same path as new key foo), markerValidOrAbsent() returns false but delete() continues and removes that legacy file. The fresh evidence after the earlier marker-revalidation fix is that an absent result here is ignored; rescan for legacy data before performing the destructive operation.
Useful? React with 👍 / 👎.
…legacy data - subtreeHasDataFile skips only the marker and its exact temp pattern (`<marker>.tmp-`), not every file sharing the prefix — a legacy resource keyed `.fsdev-store-layout-notes` was being treated as an internal temp and hidden (P1, a regression from the previous broad startsWith). - Factored the re-validate + re-scan guard into assertNoUnmarkedLegacyData, used by both ensureMarker (publish) and delete. delete now re-scans on an absent marker instead of proceeding on the cached layout, so it can't rm a flat legacy file a cold-cached-fresh instance never re-scanned for (P2). Tests: delete-after-cached-fresh-then-legacy throws; marker-lookalike legacy key trips the guard (both stores). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
|
Both fixed in
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af4042ca0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (await this.subtreeHasDataFile(full)) return true; | ||
| continue; | ||
| } | ||
| if (!entry.isFile()) continue; |
There was a problem hiding this comment.
Lstat entries whose directory type is unknown
On filesystems where readdir(..., { withFileTypes: true }) returns UV_DIRENT_UNKNOWN entries (including some NFS/FUSE mounts), both isDirectory() and isFile() are false, so this branch silently skips the entry. An existing legacy subtree can therefore be classified as empty and receive a nested-layout marker, making all of its flat resources inaccessible; the same assumption in resource-path.ts also makes getAll() omit nested-layout records. Fall back to lstat(full) for unclassified entries instead of continuing.
Useful? React with 👍 / 👎.
| if (entry.name === LAYOUT_MARKER_NAME || entry.name.startsWith(`${LAYOUT_MARKER_NAME}.tmp-`)) { | ||
| continue; |
There was a problem hiding this comment.
Count marker-temp-shaped files below the subtree root
When a legacy scope contains a valid resource key such as .fsdev-store-layout.tmp-123-456, this recursive skip treats its flat file as an internal marker temp, classifies the subtree as fresh, and allows a marker to be stamped over data that then becomes inaccessible. The fresh evidence after the earlier broad-prefix comment is that the narrowed .tmp- condition is still applied at every recursion depth even though ensureMarker() creates temporary markers only directly under this.root; only root-level generated marker temps should be ignored.
Useful? React with 👍 / 👎.
| try { | ||
| await writeFile(tmp, JSON.stringify({ layout: LAYOUT_VERSION }), "utf8"); | ||
| try { | ||
| await link(tmp, this.markerPath); |
There was a problem hiding this comment.
Avoid requiring hard-link support for first writes
When the configured store root is on a filesystem without hard-link support, such as FAT/exFAT or some FUSE and network mounts, the first set() always fails at this link() call even though ordinary file creation and atomic rename—the operations required by the previous adapter—work there. The filesystem adapter does not document a hard-link-capable-volume requirement, so marker publication needs a supported fallback or the adapter unexpectedly becomes unusable on these roots.
Useful? React with 👍 / 👎.
| const encoded = segments.map(encodeSegment); | ||
| return path.join(...encoded) + ext; |
There was a problem hiding this comment.
Reserve space for the structural leaf extension
On filesystems with the common 255-byte component limit, a single-segment key whose encoded name previously fit near that limit now fails with ENAMETOOLONG because this mapping appends .md or .json after encoding. For example, a 253-character ASCII content key or a 251-character state key was a valid filename in the flat store but exceeds the limit in the new layout, despite the release note listing only empty segments and reserved device names as newly rejected keys. Preserve these keys with an alternate mapping or validate and document the reduced leaf limit explicitly.
Useful? React with 👍 / 👎.




Summary
The filesystem content and state stores wrote every resource key as one flat
encodeURIComponent(wholeKey)file with no extension — a nested key likeconcepts/flow-state-dev/overviewbecameconcepts%2Fflow-state-dev%2Foverview, not browsable, not diffable, no editor language mode. This reworks both stores so a key maps to a real nested on-disk path with a leaf extension (.mdcontent,.jsonstate):set("session","s1","concepts/flow-state-dev/overview", …)writes…/content/session/s1/concepts/flow-state-dev/overview.md. The store root becomes a browsable, git-diffable file tree; the store API/behavior is unchanged and re-verified against the shared conformance suite.Implements the FIX-890 spec.
Fixes FIX-890.Changes
resource-path.ts— injective key↔path mapping: per-segment escaping (every.→%2Eso the leaf extension is the only literal dot, closing theavsa.md/bcollision), empty-segment / Windows reserved-device validation, and a symlink-safecollectRecordswalk with defensive canonicalization + BP-033 prefix narrowing.createFilesystemResourceStore<T>— one generic CRUD implementation both stores back onto;content-store.ts/resource-state-store.tsare now thin config (.md/identity,.json/JSON). The content store inherits the state store's concurrent-scope ENOENT-retry..fsdev-store-layoutmarker; a populated pre-nested store is refused with a clear error (lazy, per-op, never at construction;deleteAllstays usable so an upgraded install can tear old scopes down).deleteAll, which unlinks a symlinked scope dir rather thanrm -rfthrough it), scope-id validation + containment, collision surfacing (EEXIST/ENOTDIR/EISDIR→ clear error).resource-pathunit tests.minor,@flow-state-dev/engine+@flow-state-dev/cli, sincefsdev runusescreateFilesystemStores).Key decisions & ramifications
FilesystemContentStore/FilesystemResourceStateStoreclasses were dropped (nothing imports them; only the factory functions are referenced). One place for every guard; a hypothetical externalnew FilesystemContentStorewould break (none exist).migrateFilesystemStoretool stays a documented follow-up. Chosen over dual-read/lazy migration, which proved bug-prone across four spec-review rounds.encodeURIComponent(not the per-segment encoder) + reject./../""/device — keeps a dotteduserId(email) byte-identical to the legacy dir sodeleteAlland the guard stay consistent across upgrade; traversal scope ids now throw loudly instead of silently escaping.getByPrefixsymlink guard covers intermediate narrowed dirs — review caught that a deep prefix (a/b/x) could enumerate through a symlinked intermediate directory;collectRecordsnowlstats every intermediate segment (regression-tested).avsa/bcollision by construction, so it only fires on hand-edited on-disk shapes; kept as defense-in-depth for the user-editable vault.Minor behavior change: an empty resource key now throws early (
validateSegment) instead of an EISDIR-on-read.Test plan
pnpm --filter @flow-state-dev/engine typecheck— clean.pnpm --filter @flow-state-dev/engine test— 1691 passed, 1 skipped (143 files).mkdtemp), which pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01WdNcANsuBvoc3iw6JXNKyX
Generated by Claude Code
Note
Medium Risk
On-disk format breaks without migration—existing flat store roots error until moved or deleted—and filesystem key validation is stricter; persistence and enumeration logic is large but heavily tested.
Overview
Filesystem content and resource state persistence no longer use one flat
encodeURIComponent(wholeKey)file per resource. Resource keys map to a nested on-disk tree with a leaf extension (.mdfor content,.jsonfor state), so keys likeconcepts/flow-state-dev/overviewbecome browsable paths under the scope directory. The public store API is unchanged.Implementation is centralized in
createFilesystemResourceStoreplusresource-path.ts(segment encoding, enumeration, prefix narrowing). The formerFilesystemContentStore/FilesystemResourceStateStoreclasses are replaced by thin factory wrappers.Upgrade behavior: unmarked subtrees that still contain old flat data are rejected with an explicit error (no silent misread). Fresh trees get a
.fsdev-store-layoutmarker on first write;deleteAllcan clear legacy scopes without tripping the guard. Added checks include symlink-safe I/O, scope-id validation, collision errors, and rejection of empty segments / Windows reserved names.Docs and a minor changeset (
engine+cli) describe the layout and breaking on-disk format. Tests add conformance on the filesystem adapter,resource-pathunit tests, and coverage for legacy guard, symlinks, and nested layout.Reviewed by Cursor Bugbot for commit a1eaded. Configure here.
Greptile Summary
This PR changes filesystem-backed resources to use a nested, browsable file layout. The main changes are:
.mdand.jsonresource files with reversible key encoding.Confidence Score: 5/5
This looks safe to merge.
No blocking issues found in the changed code.
What T-Rex did
Important Files Changed
.mdfiles and string values..jsonfiles and structured state.Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Content or state store API] --> B[Shared filesystem resource store] B --> C[Validate scope and resource key] C --> D[Check layout marker] D --> E[Check ancestors and leaf] E --> F[Encode key segments] F --> G[Read or write nested .md or .json file] B --> H[Prefix enumeration] H --> I[Canonical symlink-safe tree walk]%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% flowchart TD A[Content or state store API] --> B[Shared filesystem resource store] B --> C[Validate scope and resource key] C --> D[Check layout marker] D --> E[Check ancestors and leaf] E --> F[Encode key segments] F --> G[Read or write nested .md or .json file] B --> H[Prefix enumeration] H --> I[Canonical symlink-safe tree walk]Reviews (5): Last reviewed commit: "fix(FIX-890): precise marker-temp skip; ..." | Re-trigger Greptile