Skip to content

feat: build an extensible document knowledge base #2440

Open
0 of 4 issues completed
Open

Description

@Zxf-xufeng

Feature description

Add a document knowledge base to PowerContext for importing, maintaining, and retrieving business documentation and task SOPs.

Use a core knowledge-base module with replaceable components. The first release should provide a working Markdown ingestion and hybrid retrieval pipeline, with extension points for additional storage providers, external parsers, chunking strategies, rerankers, and bounded agentic retrieval.

Problem and proposed solution

Users need to manage tens of thousands of evolving documents while preserving original content, document structure, revisions, and citations.

Memory and Topic Memory can complement this workflow, but extracting or summarizing documents into memory does not replace retrieval over original document passages.

Reuse existing Source, Connector, Artifact, background processing, authorization, and search infrastructure where appropriate. The existing OpenDAL connector uses opendalfs, but document identity, lifecycle management, and integration with searchable artifacts require additional work.

Initial scope

  • Markdown batch ingestion and explicitly triggered updates and deletions.
  • One storage/ingestion adapter, one Markdown parser, and one default chunking strategy.
  • Document, section, chunk, and provenance management.
  • Full-text and vector hybrid retrieval with basic metadata filtering.
  • Reliable asynchronous processing, revision publication, recovery, and diagnostics.
  • Public APIs, Python client support, and necessary Agent-facing integration.
  • Existing Scope authorization boundaries, with an explicit knowledge-base-to-Scope mapping.

Architecture requirements

  • The core owns document identity, revision visibility, authorization, task state, and publication.
  • Components use explicit registration and startup configuration.
  • Storage, parsing, chunking, indexing, and retrieval have capability-specific interfaces.
  • Track source revisions, processing configuration versions, and index versions.
  • Validate component output before publication. Stale tasks must not overwrite newer revisions.
  • Keep credentials out of document metadata, Source payloads, and ordinary diagnostics.
  • Document supported database backends and capability differences.

Out of scope for the first release

Multiple parser integrations, advanced chunking algorithms, production reranking and agentic retrieval strategies, scheduled synchronization, a complete management UI, plugin marketplaces, and hot loading.

Implementation tracking

Acceptance criteria

  • Complete the ingestion → processing → retrieval → update → deletion → rebuild workflow.
  • Retrieval results identify the exact document revision and passage.
  • Default retrieval excludes superseded passages after publication and deleted documents after deletion.
  • Retries and process restarts do not cause duplicate publication or revision rollback.
  • Components can be replaced without changing public citation or lifecycle semantics.
  • Validate an agreed corpus containing tens of thousands of documents and report quality, throughput, latency, resource use, and model costs.

Alternatives considered

Using extracted memories as the only document representation would not preserve complete source details. Building a general workflow or plugin platform would unnecessarily expand the initial scope.

Activity

  1. AlexStocks commented on Nov 3, 2026

    @AlexStocks
    Contributor

    按当前 master(729dc4c5)核对后的分析,供实现前确认,不改变本 issue 的状态。

    1. 与现有代码的关系(无重复实现)
    src/powercontext/ 下没有 knowledge-base 目录,方向不构成重复实现。可复用的现有边界是:Source/Connector(sources/、source_management/、integrations/opendal)、后台处理与触发器(builtin/triggers、builtin/processing_diagnostics.py)、向量与全文能力(sqlite-vec、pyobvector、MemorySearchMode 的 fts|vector|hybrid|auto)。建议在 issue 内点名每个扩展点复用哪个现有接口,而不是新起并行实现。

    2. 需要先走 RFC
    docs/en/rfcs/README.md 把「新增公开 API、集成边界或扩展机制」和「持久化格式变更」列为应走 RFC 的情形。本 issue 三项同时命中:新公开 API、parser/chunker/reranker 扩展点、document/section/chunk 的持久化格式。建议先出 RFC(号取 PR 号,docs/{en,zh}/rfcs/ 同步,头部字段照 0000_example.md),把身份/修订语义、chunk 表结构与索引 profile 定下来,再进入分阶段实现。

    3. 规模与成本未量化,验收无法证伪
    当前验收只有功能条目,没有规模数字与延迟上界。按 issue 里「数万文档」估算:每文档 10–50 chunk ⇒ 1e5–1e6 行 chunk 与对应向量,这会决定嵌入调用量、索引构建时间与增量更新成本。建议补:单批导入上限与吞吐、检索 p50/p95 目标、单文档重建耗时上界、以及「tens of thousands」的确切量级。否则后续「性能可接受」既不能验证也不能复现。

    4. embedding profile 的归属要先定
    现有 EmbeddingProfile(profile_id, model, dimension, distance, normalization)(src/powercontext/builtin/artifacts/memory/models.py)是 Memory 索引的契约。KB 复用同一 profile 还是独立 profile,直接决定「换模型/改维度时重建哪些索引」以及是否与 Memory 的检索空间互相污染。建议在 issue 内写明选择及其迁移路径。

    5. 迁移与契约门禁是隐性成本
    实现阶段至少要覆盖:迁移链只能往 head 之后追加(不改已登记 revision 的 down_revision)、release/upgrade-support.json 同步、sqlite/mysql 两套 layout 与 DDL 预算登记;新增 OpenAPI 操作要同步五类消费方,并重算 Desktop 契约摘要与资格记录。建议把这些写进分期计划,避免在最后一个阶段集中返工。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions