Prompt Gaurd/Moderation

[everpeace]

Description: Users often would like to enforce some prompt guard or moderation policies to protect applications which uses the LLMs. The feature would fit for some enterprise users who build internal LLM/GenAI gateway by Envoy AI Gateway.

Describe the desired behavior, what scenario it enables and how it
would be used.

[optional Relevant Links:]

• Concept:
  • Navigating the LLM Landscape: Uber’s Innovation with GenAI Gateway
• Gateway Implementations
  • kgateway - Prompt Guards
• Other Guard/Moderation Implementations
  • LiteLLM -> https://docs.litellm.ai/docs/proxy/guardrails/quick_start
  • Bifrost -> https://docs.getbifrost.ai/enterprise/guardrails
• Guard/Moderation API
  • Open AI - Moderation
  • Anthoropic - Content moderation
  • Gemini for safety filtering and content moderation

Any extra documentation required to understand the issue.

From the API level perspective, I just imagined that the feature might be integrated into AIGatewayRoute or would introduce new API(CRD) to configure the guard/moderation policy because such policies are complex.

Perhaps, BackendTrafficPolicy or SecurityPolicy could be a candidate for integration target. But, these APIs are not Envoy AI Gateway's one. So, I think we should design more generic apis so that it can call external apis in this case.

Note: although Envoy AI Gateway does not support this feature natively, users could achieve this by implementing External Processing or External Authrization

[missBerg]

Thanks for raising!

[seluard]

Hello there !

It's something planned to work into this topic? I see this feature as high importance to increase eaigw adoption at enterprise level.

I saw this issue -> #1415 and get into this -> https://vllm-semantic-router.com/docs/installation/k8s/ai-gateway . Seems like vllm router could add some of this on top, but I think for eaigw should be providing this feature itself.

Other AI gateways guardrails docs refs:
LiteLLM -> https://docs.litellm.ai/docs/proxy/guardrails/quick_start
Bifrost -> https://docs.getbifrost.ai/enterprise/guardrails

[everpeace]

I see this feature as high importance to increase eaigw adoption at enterprise level.
Seems like vllm router could add some of this on top, but I think for eaigw should be providing this feature itself.

Totally agreed.

Other AI gateways guardrails docs refs:

Thanks! I added the references to the issue discription.

[Xunzhuo]

can you tell what/how to implement prompt guard in eaigw? By an external calling to external provider?

[everpeace]

By an external calling to external provider?

Something like that. Some enterprise users might have an internal guard/moderation API endpoint internally.