Governance policy hook in LLM ExtProc pipeline

[imran-siddique]

Problem

AI Gateway already has strong routing, translation, rate limiting, and upstream auth, plus general Envoy security integrations (JWT, ext_authz) on the data plane. The MCP side goes further with scoped authorization, CEL expressions, and tool selectors.

What's missing is a first-class, body-aware governance hook in the LLM ExtProc pipeline where translated requests and responses can be evaluated by an external policy engine before dispatch and after completion.

A few existing issues point to this recurring need:

• Adding support for external security provider integration #1919 raises external security provider integration with request/response scanning
• Prompt Gaurd/Moderation #1409 covers prompt guard/moderation
• Human-in-the-Loop for MCP Tool Calls #2073 discusses human-in-the-loop for MCP tool calls

These aren't identical problems, but they all need a governance extension point that doesn't exist in the LLM path today.

Proposal

Add an optional, phased governance hook to the LLM ExtProc pipeline. Starting narrow:

Phase 1: Pre-request governance + decision metadata

A route-level config field pointing to an external governance service (gRPC). The hook runs in upstreamProcessor.ProcessRequestHeaders() (in internal/extproc/processor_impl.go) after translation and header/body mutation are finalized, before upstream auth and dispatch.

The governance service receives:

• Translated request headers and body (or a configurable subset/redaction)
• Backend and route name
• Caller identity from already-authenticated upstream filters, validated JWT claims, or trusted forwarded headers

It returns allow/deny with a reason and optional metadata (matched policy rule, trust level, request classification).

On deny, the ExtProc returns an ImmediateResponse with the policy engine's reason. On allow, governance decision metadata is written to Envoy dynamic metadata so it flows into access logs, OTel traces, and rate limit descriptors.

Phase 2: Async response audit

After the response completes, send response metadata to the governance service asynchronously for audit logging and compliance tracking. Non-blocking; the response is not held.

Phase 3: Buffered response enforcement (later)

For non-streaming responses only, optionally gate the response through the governance service before returning to the client. Streaming responses stay audit-only since tokens already left the proxy.

What this enables

• Model access control: "team-X can only use gpt-4o-mini" enforced at the gateway, not in application code
• Content governance: request scanning delegated to an external engine (prompt injection detection, PII checks)
• Audit trail: every LLM call gets a structured governance decision alongside existing token/latency metrics
• Compliance: operators in regulated industries can prove every AI call went through policy evaluation

Implementation

We maintain Agent Governance Toolkit, an open-source governance framework with a policy engine, agent identity, trust scoring, and compliance auditing. We'd be willing to contribute a reference governance processor and documentation if there's interest.

The integration pattern is an open question. Options:

1. ExtProc-native hook in the existing processor pipeline (full body access, most capable)
2. ext_authz filter (simpler, headers-only, well-understood pattern)
3. Combination: ext_authz for fast allow/deny, ExtProc hook for body-level inspection

I'd lean toward option 1 since body access matters for content governance, but happy to hear what maintainers think fits best with the architecture direction, especially given the dynamic modules work tracked in #90.

[Rul1an]

This is a clean phasing. The pre-request decision plus async response audit split across Phases 1 and 2 feels right, and writing the decision into dynamic metadata so it can flow into access logs, traces, and rate-limit descriptors is probably the part most worth getting stable early.

Three design points seem worth settling before that metadata shape spreads downstream.

1. Treat the decision metadata as a contract.

Once the same record lands in logs, traces, and rate-limit descriptors, downstream consumers will parse it in different places. A small stable core would help: decision_id, policy_ref, policy_version, decision, reason, and the route or backend it applied to.

The ruleset version is where the shoe pinches. Without a stable version identifier, a later consumer cannot tell what a matched rule id meant at decision time.

2. Persist digests and redaction state, not bodies.

It makes sense for the governance service to inspect a translated body, or a configurable subset or redaction. But the durable record is a different question.

For logs, traces, and metadata, request_digest, response_digest, redaction_applied, and optional redaction_reason would keep the audit boundary reviewable without turning those sinks into payload stores.

3. An allow decision should not read like a general safety statement.

It is really: this policy engine, this ruleset version, at this time, evaluated this request subset and returned allow. The ruleset may not cover the relevant threat class, or may have evaluated a redacted subset.

If the decision record carries that scope explicitly, downstream consumers are less likely to confuse policy coverage with a broader safety guarantee.

None of that changes the phasing. It just makes the Phase 1 decision record more durable for the Phase 2 audit trail and any later downstream consumers.