Repository navigation
feat(knowledge-base): incubate OKF interchange adapter + capability (FIX-813) - #650
Conversation
…e capability (FIX-813) PR 2 of FIX-813. PR 1 (the core glob/grep/search nav tools in @flow-state-dev/core) already shipped; this adds the incubation lab that validates an OKF (Open Knowledge Format) v0.1 interchange boundary against a first consumer, before any public-package graduation. New private lab labs/knowledge-base/: - OKF concept collection: defineResourceCollection whose state mirrors OKF frontmatter (required `type`; nullable recommended fields; unknown keys preserved in a strict-compatible `extra` key/value list per BP-016; an `edges` slot; llmReadable/llmWritable so the core nav + content tools see it). - OKF import: parseOkfBundle (gray-matter on the eemeli `yaml` engine for YAML 1.2 — sidesteps the Norway problem; non-mapping frontmatter is handled best-effort, not corrupted) + importOkf (frontmatter -> state, body -> content, untyped markdown links -> typed `references` edges via a path index; dangling links and missing `type` become warnings, never fatal — SPEC §9). - OKF export: exportOkf (canonical-ordered frontmatter, verbatim body, programmatic edges materialized once into `# Related`, generated root index.md with okf_version "0.1"). Gated by an export -> import -> export byte-identical idempotency test (the load-bearing correctness check). - createKnowledgeBaseCapability: a DefinedCapability (memory-factory shape) — concept collection as its resource, the core nav tools in a `search` preset, an index.md-style progressive-disclosure context formatter, and fns helpers (listConcepts / readConcept / relate / importBundle / exportBundle). Nav tools are installed as-is rather than re-aliased, to avoid a second divergent tool surface (decision #4). - Runnable example flow (`knowledge`): a model-free `explore` action and a self-seeding `research` sequencer that lets a generator navigate the mounted concepts. Lab README + an OKF conformance smoke (`pnpm okf:smoke`). Decisions #4 (OKF stays an interchange boundary, not a new authoring surface) and #5 (curated KB navigation complements rather than replaces vector RAG) are recorded in the spec and the lab README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6WKjfdWTpSLmGBLaDYxYS
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fe3d5fb. Configure here.
| } | ||
| } | ||
|
|
||
| return { imported: concepts.length, warnings }; |
There was a problem hiding this comment.
Import leaves stale concepts
Medium Severity
importOkf upserts only concepts present in the bundle and never removes collection instances whose keys are absent. A later importBundle or exportBundle can leave or emit concepts from a previous mount, so the on-disk bundle and the collection diverge.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fe3d5fb. Configure here.
There was a problem hiding this comment.
Fixed in 22472bd: importOkf now syncs the collection to the bundle — before pass 1 it lists existing instances and deletes any whose concept ID isn't in the incoming bundle (labs/knowledge-base/src/okf/import.ts:34-42). After a mount the collection mirrors the bundle exactly, so a re-mount of a different bundle (and a subsequent export) no longer carries stragglers. Added a regression test (test/okf-import.spec.ts — "syncs the collection to the bundle") and documented the sync-not-merge semantic in the README.
Generated by Claude Code
| .map((target) => resolveLinkToConceptId(target, id)) | ||
| .filter((t): t is string => t !== null), | ||
| ); | ||
| concepts.push({ id, frontmatter: data, body, links }); |
There was a problem hiding this comment.
Parse errors abort bundle
Medium Severity
Per-concept parsing calls splitFrontmatter without a try/catch, so invalid YAML or gray-matter failures reject the whole parseOkfBundle/importOkf call instead of recording a warning and skipping that file as OKF §9 best-effort behavior describes.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fe3d5fb. Configure here.
There was a problem hiding this comment.
Fixed in 22472bd: per-concept frontmatter parsing in parseOkfBundle is now wrapped in try/catch (labs/knowledge-base/src/okf/parse.ts:41-54) — unparseable YAML in one file records a failed to parse warning and skips that concept rather than rejecting the whole bundle, matching OKF §9 best-effort consumption. Regression test added (test/okf-import.spec.ts — "skips a concept with unparseable YAML and warns").
Generated by Claude Code
| for (const match of body.matchAll(MARKDOWN_LINK)) { | ||
| targets.push(match[1]!); | ||
| } | ||
| return targets; |
There was a problem hiding this comment.
Image links become edges
Low Severity
extractLinkTargets uses a regex that matches any [text](url) substring, including markdown image syntax . In-bundle .md image targets are projected as references edges during import even though they are not prose links.
Reviewed by Cursor Bugbot for commit fe3d5fb. Configure here.
There was a problem hiding this comment.
Fixed in 22472bd: the link regex now carries a (?<!!) negative lookbehind (labs/knowledge-base/src/okf/links.ts:16), so image syntax [alt](x.md) is no longer matched and in-bundle image targets aren't projected as references edges. Regression test added (test/okf-import.spec.ts — "does not turn a markdown image into a concept edge").
Generated by Claude Code
…e, image links (FIX-813) Cursor Bugbot review on #650: - importOkf now SYNCS the collection to the bundle: concepts absent from the incoming bundle are pruned, so a re-mount (or a later export) mirrors the mounted bundle instead of leaving stragglers from a prior mount. - parseOkfBundle wraps per-concept frontmatter parsing in try/catch: malformed YAML in one file becomes a warning + skip, not a rejection of the whole bundle (OKF §9 best-effort consumption). - the markdown-link regex skips image syntax `` via a negative lookbehind, so an in-bundle image target is no longer projected as a `references` edge. Tests: +3 (sync/prune re-mount, unparseable-YAML skip, image-not-an-edge); 22/22. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6WKjfdWTpSLmGBLaDYxYS
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe3d5fb47b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| continue; | ||
| } | ||
|
|
||
| const { data, body } = splitFrontmatter(raw); |
There was a problem hiding this comment.
Keep malformed YAML from aborting bundle import
When an external bundle has one concept with syntactically invalid YAML frontmatter, such as an unterminated list, splitFrontmatter throws here because the only try/catch in the loop wraps fs.readFile. That makes parseOkfBundle/importOkf reject the entire bundle and skip otherwise valid concepts instead of collecting a per-concept warning for the best-effort OKF import path. Catch parse errors around this call and continue with a warning/default state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Already addressed in 22472bd (your review ran against the earlier fe3d5fb): per-concept parsing in parseOkfBundle is wrapped in try/catch — unparseable YAML in one file becomes a failed to parse warning and skips that concept rather than rejecting the bundle (OKF §9). Covered by test/okf-import.spec.ts — "skips a concept with unparseable YAML and warns".
Generated by Claude Code
| const instances = await collection.list(); | ||
| instances.sort((a, b) => a.path.localeCompare(b.path)); | ||
|
|
||
| await fs.mkdir(dir, { recursive: true }); |
There was a problem hiding this comment.
Clear stale concept files before exporting
When exportOkf is pointed at an existing bundle directory, concept files from a previous export that are no longer present in the collection are left on disk because this path only creates the directory and overwrites current files. A later parseOkfBundle/import of that export will include those stale concepts, so the emitted bundle no longer represents the collection. Clean the managed markdown files first or reject non-empty output directories.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0df66bd: exportOkf now clears existing .md files (and prunes emptied dirs) in the target directory before writing (labs/knowledge-base/src/okf/export.ts:42-43,59-79), so re-exporting into a directory that held a prior bundle reflects only the current collection — no stale concept files. Regression test added (test/okf-export.spec.ts — "clears stale concept files when re-exporting into an existing directory").
Generated by Claude Code
| }, | ||
| /** Install the glob/grep/search nav tools. Default-on. */ | ||
| search: { | ||
| tools: () => [nav.globResources, nav.grepResourceContent, nav.searchResources], |
There was a problem hiding this comment.
Add full content reads to the search preset
When a generator uses this capability and finds a concept with globResources, grepResourceContent, or searchResources, the installed tools only provide URIs or short snippets; the model has no readResourceContent tool to fetch the full concept body for answers that are not contained in the returned snippet. ctx.cap.knowledgeBase.readConcept is a host-side helper, not an LLM tool, so include readResourceContentTool() in this preset or the example generator's tools.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good call — fixed in 0df66bd: the search preset now also installs readResourceContentTool() (labs/knowledge-base/src/okf/../capability.ts), so a generator can fetch a full concept body by uri when a glob/grep/search snippet isn't enough. The collection is llmReadable, so the tool resolves the nav tools' uris directly. (readConcept stays a host-side fns helper as you noted.) Capability test updated to assert the four tools.
Generated by Claude Code
| function joinPosix(dir: string, rel: string): string { | ||
| const base = dir.length > 0 ? dir.split("/") : []; | ||
| for (const seg of rel.split("/")) { | ||
| if (seg === "." || seg === "") continue; | ||
| if (seg === "..") base.pop(); | ||
| else base.push(seg); | ||
| } | ||
| return base.join("/"); |
There was a problem hiding this comment.
Reject links that traverse above the bundle root
When a concept links above the bundle root, e.g. ../../foo.md from sub/note.md, this resolver silently pops past an empty base and normalizes the target to foo. If a root concept named foo.md exists, import records an in-bundle edge even though the markdown link actually escaped the OKF bundle. Track attempts to pop beyond the root and return null for those targets instead of clamping them to the root.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0df66bd: joinPosix now returns null when a .. pops above the base, and resolveLinkToConceptId drops the target (also rejecting a surviving .. from an absolute /../x.md) instead of clamping to a root concept (labs/knowledge-base/src/okf/links.ts:38-49,63-78). Regression test added (test/okf-import.spec.ts — "drops a link that traverses above the bundle root").
Generated by Claude Code
…, content tool (FIX-813) Codex review on #650: - exportOkf clears existing `.md` files in the target directory before writing, so exporting a collection into a directory that held a prior bundle no longer leaves stale concept files (the export-side analog of import's sync semantic). - resolveLinkToConceptId returns null when a relative link pops above the bundle root (`../../foo.md`) instead of clamping it to a spurious in-bundle edge; a surviving `..` from an absolute link is likewise rejected. - the knowledgeBase `search` preset now also installs `readResourceContentTool`, so a generator can fetch a full concept body by uri (the nav tools only return uris + snippets; `readConcept` is a host-side helper, not an LLM tool). The fourth Codex item (malformed YAML aborting import) was already fixed in 22472bd. Tests: +2 (export prune, link escape) and the search-preset assertion updated; 24/24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6WKjfdWTpSLmGBLaDYxYS
…fe export prune (FIX-813) Two P1s from the Greptile review on #650: - resolveLinkToConceptId routes ABSOLUTE link paths through the same segment resolver as relative ones, so a valid internal `..` (e.g. `/a/../b.md` -> `b`) resolves to its in-bundle concept instead of being dropped; only a `..` that pops above the bundle root returns null. Removes the now-dead normalizeSegments. - export no longer deletes markdown it does not own. `pruneStaleConcepts` removes only `.md` files that are NOT being rewritten AND parse as OKF concepts (non-empty `type` frontmatter); reserved `log.md` and non-concept user files (e.g. a hand-authored `README.md`) are preserved. Tests: +2 (absolute-link internal `..`, export preserves reserved/user markdown); 26/26. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6WKjfdWTpSLmGBLaDYxYS


Summary
PR 2 of FIX-813. PR 1 — the agent-facing
globResources/grepResourceContent/searchResourcesnav tools in@flow-state-dev/core— already shipped. This PR adds the incubation lab that validates an OKF (Open Knowledge Format) v0.1 interchange boundary over the resource graph against a first consumer, before any public-package graduation.Everything lands in a new private lab,
labs/knowledge-base/— no public API surface. OKF v0.1 is a one-day-old proof-of-concept and would be only the second consumer of the typed-edge primitive, so the adapter + capability are incubated rather than hardened (the project's "two consumers before a primitive hardens" trigger).Changes
src/concepts.ts) —defineResourceCollectionwhose state mirrors OKF frontmatter: requiredtype; nullable recommended fields; unknown keys preserved in a strict-compatibleextrakey/value list (notz.record, per BP-016); anedgesslot;llmReadable/llmWritableso the core nav + content tools see the concepts.src/okf/parse.ts,import.ts) — walk a bundle (skip reservedindex.md/log.mdat any level), split frontmatter via gray-matter on the eemeliyamlengine (YAML 1.2 — sidesteps the "Norway problem"), project untyped markdown links into typedreferencesedges through a path index. Best-effort per OKF §9: missingtype→ default + warning, dangling links → warning, non-mapping frontmatter → handled, never fatal.src/okf/export.ts) — canonical-ordered frontmatter, verbatim body, programmatic edges materialized once into a# Relatedsection, generated rootindex.mdwithokf_version: "0.1". Gated by anexport → import → exportbyte-identical idempotency test (the load-bearing correctness check).createKnowledgeBaseCapability(src/capability.ts) — aDefinedCapabilityof the memory-factory shape: concept collection as its resource, the core nav tools in asearchpreset, anindex.md-style progressive-disclosure context formatter, andfnshelpers (listConcepts/readConcept/relate/importBundle/exportBundle).src/flow.ts) + sample bundle + lab README + an OKF conformance smoke (pnpm okf:smoke).Key decisions & ramifications
searchKnowledge→searchResourcesaliases; instead the KB vocabulary lives infns+ the context formatter. Rules out a second divergent tool surface (decision Add CI workflow for type checking and testing #4) and shallow wrapper blocks (BP-013), at the cost of tool names that aren't KB-branded.# Related. Locks in a clean, byte-stable round trip; rules out lossless edge-metadata round-tripping (confidence/temporality are dropped — lossy by design, asserted on content + frontmatter + link-structure only).yamlengine, with a non-mapping-frontmatter guard. KeepsNO/onas strings and won't corrupt state on a malformed external bundle; carries a small extra dependency over gray-matter's default js-yaml.log.mdreserved but not round-tripped in v0. Excluded as a concept; not persisted or synthesized. Deviates from the spec mapping-table's "verbatim" wording in favor of §8's "not synthesized" — the dominant statement.Goal verdict
export → import → exportbyte-identical — PASS (test/okf-roundtrip.spec.ts).pnpm fsdev run knowledge research -i '{"question":"What on-disk format does this knowledge base use, and what does the welcome concept say?"}'→ the generator navigated the mounted concepts and answered with cited concept URIs (session/concepts/okf,session/concepts/welcome) — PASS.pnpm fsdev run knowledge explore -i '{}'→ imported 2 concepts, created instances, listed them — PASS.pnpm okf:smoke→ OKF v0.1 conformance OK — PASS.Test plan
pnpm --filter @flow-state-dev/knowledge-base test→ 19 passing (import, export, round-trip idempotency, capability; incl. malformed-frontmatter and nested-reserved edge cases).pnpm --filter @flow-state-dev/knowledge-base typecheck→ clean.Change set is isolated to the new private lab (+
labs/README.mdindex row + lockfile). No existing package was modified, so no changeset (private package).Fixes FIX-813.
🤖 Generated with Claude Code
Generated by Claude Code
Note
Low Risk
Isolated to a new private lab plus lockfile/README index; existing
@flow-state-dev/*packages are unchanged.Overview
Adds a private incubation lab at
labs/knowledge-base/(FIX-813) — no published package changes.labs/README.mdgains a row for the new lab.The lab introduces an OKF v0.1 interchange layer:
parseOkfBundle/importOkf/exportOkfmap bundle directories to a session-scoped concept resource collection (frontmatter → Zod state, bodies → content, markdown links → typedreferencesedges). Frontmatter uses YAML 1.2 via gray-matter +yaml; import is best-effort (warnings for missingtype, dangling links, bad frontmatter). Export emits canonical frontmatter, verbatim bodies, generated rootindex.md, and materializes programmatic-only edges under# Related.createKnowledgeBaseCapabilitycomposes that collection with coreglobResources/grepResourceContent/searchResources(no KB-specific tool aliases), a<knowledge>index preset, andfnsfor list/read/relate/import/export. A runnableknowledgeflow (exploremodel-free,researchseeds + generator) plusfsdev.config, sample bundle,pnpm okf:smoke, and vitest coverage including export → import → export byte-idempotency.Reviewed by Cursor Bugbot for commit fe3d5fb. Configure here.
Greptile Summary
This PR incubates a private
labs/knowledge-base/lab that validates an OKF (Open Knowledge Format) v0.1 interchange boundary over the resource graph before any public-package graduation. No published packages are modified — the change set is fully isolated to the new lab and alabs/README.mdrow, making it low-risk by default.src/okf/): parse → import → export pipeline; YAML 1.2 via eemeliyaml(Norway-problem fix); best-effort per OKF §9 (missingtype, dangling links, malformed frontmatter all warn rather than abort); idempotent round-trip gated byexport → import → exportbyte-equality test.createKnowledgeBaseCapability(src/capability.ts): thin composition over the concept collection —indexpreset (progressive-disclosure context),searchpreset (nav tools), andfnshelpers (importBundle,exportBundle,listConcepts,readConcept,relate); runnable example flow insrc/flow.ts..., and stale-concept pruning that preserves non-concept markdown.Confidence Score: 5/5
Safe to merge — the change is fully isolated to a new private lab with no modifications to any published package.
All logic is contained in the new
labs/knowledge-base/package. The two previously identified issues (absolute-path..resolution and stale-concept over-deletion) are addressed in the head commit with regression tests. The parse/import/export pipeline, stale-pruning guard, and byte-identical round-trip are all correct and well-covered. The one mildly confusing pattern — env-var cleanup in the smoke script executing after ESM imports in practice — is harmless because model resolution is lazy and the cleanup completes beforemain()runs.No files require special attention.
scripts/visualize-smoke.tshas a cosmetic ordering oddity (env-var cleanup loop placed before imports in source, though imports are hoisted in ESM) but no functional consequence.Important Files Changed
.mdpreserved;# Relatedsection for programmatic-only edges is correctly idempotent...traversal correctly returns null; image links correctly excluded.normalizeBodyhandles CRLF, multi-blank-line preambles, and empty bodies;serializeExtraValueguardsYAML.stringify(undefined)returning undefined at runtime.z.nullable()+z.default()on resource state (not generator output) — BP-016 correctly scoped;extrak/v list avoidsz.record; frontmatter round-trip stable via fixed key order and sorted extras.createMemoryCapabilityshape;ctx: anyintentional for incubation;index/searchpresets composable; nav tools installed as-is per decision #1.seedSamplehas no outputSchema and noreturn input, attached via.tap();explorereturns a fresh output object;researchsequencer wiring is clean.importdeclarations in source; in native ESM all imports are hoisted and evaluated first, so the cleanup runs after module init. Safe in practice because model resolution is lazy, but the code placement is misleading.Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant FS as "File System (OKF Bundle)" participant P as parseOkfBundle participant I as importOkf participant C as ConceptCollection participant E as exportOkf participant FS2 as "File System (Output)" FS->>P: "walk .md files (skip reserved)" P->>P: "splitFrontmatter (YAML 1.2)" P->>P: "extractLinkTargets → resolveLinkToConceptId" P-->>I: "ParsedOkfBundle {concepts, warnings}" I->>C: "list() — prune stale concepts" loop Pass 1 state+content I->>C: "create(id, state, replace)" I->>C: "ref.writeContent(body)" end loop Pass 2 edges I->>C: "ref.edges.add({from, to, references})" end C->>E: "list() — sorted instances" E->>E: "pruneStaleConcepts (reads type)" loop per concept E->>C: "ref.readContentRaw()" E->>E: "composeBody: verbatim + Related section" E->>FS2: "writeFile(id.md)" end E->>FS2: "writeFile(index.md, okf_version + listing)"%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant FS as "File System (OKF Bundle)" participant P as parseOkfBundle participant I as importOkf participant C as ConceptCollection participant E as exportOkf participant FS2 as "File System (Output)" FS->>P: "walk .md files (skip reserved)" P->>P: "splitFrontmatter (YAML 1.2)" P->>P: "extractLinkTargets → resolveLinkToConceptId" P-->>I: "ParsedOkfBundle {concepts, warnings}" I->>C: "list() — prune stale concepts" loop Pass 1 state+content I->>C: "create(id, state, replace)" I->>C: "ref.writeContent(body)" end loop Pass 2 edges I->>C: "ref.edges.add({from, to, references})" end C->>E: "list() — sorted instances" E->>E: "pruneStaleConcepts (reads type)" loop per concept E->>C: "ref.readContentRaw()" E->>E: "composeBody: verbatim + Related section" E->>FS2: "writeFile(id.md)" end E->>FS2: "writeFile(index.md, okf_version + listing)"Reviews (2): Last reviewed commit: "fix(knowledge-base): address Greptile re..." | Re-trigger Greptile