Skip to content

feat: add rag artifact management and targeted document ingestion #2441

Open

Description

@Zxf-xufeng

Feature description

Implement Scheme B: a Scope represents a project or workspace and can contain multiple rag Artifacts. Users create a rag before importing documents into it. Imported Sources remain immutable; document Artifacts own their processed content.

Implementation scope

  • Register rag and document through existing Artifact management and reuse configuration revisions, tags, provenance, and Scope authorization.
  • Allow multiple rag Artifacts in one Scope. Creating or renaming a knowledge base must not create or rename its Scope.
  • Define explicit document ownership and logical document identity within a Scope and target rag. Importing the same path into two knowledge bases creates separate documents.
  • Extend the OpenDAL/opendalfs integration for Markdown and independently imported images. Record the original path, immutable managed snapshot, digest, media type, and captured metadata.
  • Define a dedicated document Source type directly consumable only by the document pipeline. Enforce this policy during input selection, execution, and Artifact write validation.
  • Persist target-specific import requests containing the rag reference, exact SourceRef, logical document identity, configuration snapshot, and request/input generations.
  • Atomically record accepted target work and request background processing in the database transaction. Reusing an existing Source for another rag must still schedule that target's work.
  • Follow existing Source/Artifact API and client conventions; define explicit update, deactivation, removal, and downstream invalidation contracts.

Acceptance criteria

  • Multiple rag Artifacts can be created in the same Scope without a singleton constraint.
  • An empty rag can be created without imported document Sources or document-processing work.
  • Imports reject missing, inaccessible, inactive, or mismatched targets.
  • Accepted snapshots remain readable after original files are overwritten or moved.
  • Identical repeated imports are idempotent; importing an existing Source into another rag creates the required target work.
  • Interrupted ingestion cannot leave an accepted target import without recoverable processing work.
  • Other Artifact families do not automatically consume dedicated document Sources.
  • Include migrations, API contracts, authorization and behavior tests, and configuration documentation.

Dependencies and boundaries

Parent tracking issue: #2440.

Design: Scheme B RFC.

The knowledge-base family is rag; the content family is document. Parsing, OCR, chunking, and index execution belong to #2442.

Phase one supports Markdown and independently imported images only. Plain TXT, PDF, Office, and automatic OCR of images referenced by Markdown are excluded.

Activity

  1. changed the title [-]feat: add knowledge-base foundations and extensible document ingestion[/-] [+]feat: add rag artifact management and targeted document ingestion[/+] on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions