Add RAG modes and strengthen strict mode

[csoriano2718]

This PR adds RAG mode control via the RAG_MODE environment variable, giving users clear control over how the RAG proxy balances document retrieval with general AI knowledge.

RAG Modes

Two operational modes are provided:

1. strict: Document-only responses, refuses general knowledge queries

   • Use case: Compliance, legal, private/sensitive data
   • Behavior: Answers ONLY from retrieved documents, says "I don't know" for anything else

2. augment (default): Freely combines documents with general AI knowledge

   • Use case: General assistant with access to local documents
   • Behavior: Uses documents when relevant, supplements with general knowledge when helpful

Usage

# Strict mode (documents only)
ramalama serve --env RAG_MODE=strict --rag /path/to/db model

# Augment mode (documents + general knowledge, default)
ramalama serve --env RAG_MODE=augment --rag /path/to/db model

Implementation

• Simple if/else logic for mode-specific system prompts
• Each mode has distinct instructions controlling RAG behavior
• Default mode is augment if RAG_MODE is not set

Testing

E2E tests included for both modes with positive/negative test cases:

• Strict mode: Correctly refuses general knowledge, answers from documents
• Augment mode: Answers both document and general knowledge queries

Tests are designed for models ≥7B parameters (e.g., deepseek-r1:14b, mistral:7b) which provide reliable retrieval and extraction.

Container Changes Required

This PR requires the RAG container to include the updated rag_framework script. The container image needs to be rebuilt with the changes from this branch.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Implements configurable RAG operation modes (strict, hybrid, augment) in the rag_framework script and strengthens the strict mode prompt to rely only on retrieved document content while preserving existing augment behavior as the default.

File-Level Changes

Change	Details	Files
Add configurable RAG modes (strict, hybrid, augment) to control how document retrieval is balanced with general model knowledge.	Introduce an environment-driven RAG_MODE setting with supported values strict, hybrid, and augment. Wire the selected RAG mode into the RAG pipeline so that query handling behavior changes based on the mode. Set augment as the default behavior to preserve existing semantics when RAG_MODE is not specified.	container-images/scripts/rag_framework
Strengthen strict mode behavior and prompt to enforce document-only answers and reduce hallucinations.	Update the strict mode system prompt/instructions so the model refuses to answer questions not covered by documents and returns an explicit "I don't know" when needed. Ensure strict mode prevents fallback to general knowledge when document context is insufficient.	container-images/scripts/rag_framework
Add hybrid mode behavior that prefers document answers but can fall back to general knowledge with attribution of the knowledge source.	Implement logic that first attempts to answer from retrieved documents and only falls back to general knowledge when no sufficient document context exists. Add response annotations or patterns so the model can indicate whether an answer is derived from documents or general knowledge in hybrid mode.	container-images/scripts/rag_framework

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[gemini-code-assist]

Summary of Changes

Hello @csoriano2718, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the RAG (Retrieval Augmented Generation) framework by introducing configurable operational modes. These modes allow users to precisely control the balance between relying solely on retrieved documents and leveraging the AI's general knowledge, addressing previous limitations where users had to choose between document-only or general knowledge responses. The update also fortifies the "strict" mode to prevent AI hallucinations and ensure responses are strictly grounded in provided data.

Highlights

• Introduction of RAG Modes: Three new operational modes (strict, hybrid, augment) have been added to provide granular control over how the system balances document retrieval with general AI knowledge.
• Strengthened Strict Mode: The strict RAG mode has been significantly enhanced with a more robust prompt, explicitly forbidding the use of general knowledge and requiring an "I don't know" response if information is not explicitly in the provided documents.
• Dynamic System Prompt Generation: The system now dynamically generates the system prompt based on the chosen RAG_MODE environment variable, tailoring the AI's behavior to the desired operational style.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sourcery-ai]

Hey there - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[gemini-code-assist]

Code Review

This pull request introduces configurable RAG modes (strict, hybrid, augment), which is a valuable enhancement for controlling the RAG system's behavior. The implementation correctly uses the RAG_MODE environment variable to select different system prompts, and the new strict mode prompt is well-crafted to prevent responses outside the provided context.

My main feedback is on code structure. The current implementation for selecting prompts has a fair amount of code duplication. I've provided a suggestion to refactor this using a data-driven approach, which should improve maintainability and make it easier to add more modes in the future.

Additionally, I recommend adding tests for the new RAG modes to verify their behavior and prevent regressions.

[bmahabirbu]

I really like the idea!

I have one comment in the near future, this rag pipeline can be added as an MCP server tool where the model can decide if it needs to use it or not! So we wouldn't need the strict vs augment functionality as we will follow an agentic workflow at that point

But until then, this should work!

[csoriano2718]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature for controlling RAG behavior through strict and augment modes. The implementation in rag_framework is clear and effectively uses different system prompts for each mode. The addition of E2E tests is also a great step towards ensuring the reliability of this feature. My review focuses on improving the thoroughness and correctness of these new tests. I've identified a gap in the augment mode test and a flawed implementation in the environment variable propagation test. Addressing these points will make the test suite more robust and reliable.

[csoriano2718]

@bmahabirbu ah that's a great idea, doing RAG as an MCP server.

I wonder, should we not implement this PR to avoid exposing new features/APIs in Ramalama that in the future Ramalama plans to recommend using MCP servers for? Or what's the approach of breaking past functionality to allow doing better in the future? I didn't intend to implement something that will slow down Ramalama development.

[csoriano2718]

/gemini review

[gemini-code-assist]

Code Review

This pull request effectively introduces RAG modes (strict and augment) to control document retrieval versus general AI knowledge, along with corresponding end-to-end tests. The implementation is straightforward, using environment variables to switch between system prompts.

My review focuses on enhancing the robustness and maintainability of the new code. I've identified a potential issue where an invalid RAG_MODE could silently fall back to the default, which might have security implications. Additionally, I've found a weakness in the test logic for strict mode that could allow incorrect behavior to pass, and I've suggested a way to improve the test suite's maintainability by reducing code duplication. Overall, these are solid changes that will be even better with a few adjustments.

[csoriano2718]

/gemini review

[gemini-code-assist]

Code Review

This pull request effectively introduces RAG modes (strict and augment) to control how the RAG proxy responds to queries, which is a great feature for providing more control to the user. The implementation is clear, using an environment variable to switch between system prompts. The addition of end-to-end tests for both modes is excellent and ensures the new functionality works as expected. My review includes a few suggestions to enhance maintainability by using constants for validation and refactoring large string prompts, and to improve the precision of the test assertions for the new strict mode. Overall, this is a solid contribution.

[csoriano2718]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a RAG_MODE environment variable to switch between strict (document-only) and augment (documents + general knowledge) modes for the RAG proxy. The implementation is clean, using distinct system prompts for each mode, and is accompanied by a comprehensive set of new E2E tests that validate the behavior of both modes. The changes are well-executed and add valuable control over the RAG's output. My review includes a couple of suggestions in the new test file to enhance robustness by using more specific exception handling.

[csoriano2718]

ok, I think Gemini and Cursor have reach an agreement now :-)

[rhatdan]

If we are going to add this feature it needs to be documented in a man page.

[rhatdan]

Might also want to have a setting in ramalama.conf

[rhatdan]

Failing Lint and you should squash and sign your commits.