litellm-operator

A Kubernetes operator for managing litellm resources.

Description

The operator is used to manage REST API operations on litellm resources:

• Virtual Keys
• Users
• Teams

Getting Started

It is expected that the operator will be deployed in the same namespace as the litellm service.

Prerequisites

• go version v1.22.0+
• docker version 17.03+.
• kubectl version v1.11.3+.
• Access to a Kubernetes v1.11.3+ cluster.
• Helm v3.8+ (for Helm installation method)

To Deploy on the cluster

Option 1: Using Helm (Recommended)

Authenticate with GitHub Container Registry:

helm registry login ghcr.io -u YOUR_GITHUB_USERNAME -p YOUR_GITHUB_TOKEN

Install the operator using Helm:

helm install litellm-operator oci://ghcr.io/bbdsoftware/charts/litellm-operator --version <VERSION>

NOTE: Replace <VERSION> with the desired version (e.g., 0.0.1). You can find available versions in the releases page.

Option 2: Manual Deployment

Build and push your image to the location specified by IMG:

make docker-build docker-push IMG=<some-registry>/litellm-operator:tag

NOTE: This image ought to be published in the personal registry you specified. And it is required to have access to pull the image from the working environment. Make sure you have the proper permission to the registry if the above commands don't work.

Install the CRDs into the cluster:

make install

Deploy the Manager to the cluster with the image specified by IMG:

make deploy IMG=<some-registry>/litellm-operator:tag

NOTE: If you encounter RBAC errors, you may need to grant yourself cluster-admin privileges or be logged in as admin.

Create instances of your solution You can apply the samples (examples) from the config/sample:

kubectl apply -k config/samples/

NOTE: Ensure that the samples has default values to test it out.

To Uninstall

If installed with Helm:

helm uninstall litellm-operator

If installed manually:

Delete the instances (CRs) from the cluster:

kubectl delete -k config/samples/

Delete the APIs(CRDs) from the cluster:

make uninstall

UnDeploy the controller from the cluster:

make undeploy

Project Distribution

Following are the steps to build the installer and distribute this project to users.

1. Build the installer for the image built and published in the registry:

make build-installer IMG=<some-registry>/litellm-operator:tag

NOTE: The makefile target mentioned above generates an 'install.yaml' file in the dist directory. This file contains all the resources built with Kustomize, which are necessary to install this project without its dependencies.

2. Using the installer

Users can just run kubectl apply -f to install the project, i.e.:

kubectl apply -f https://raw.githubusercontent.com/<org>/litellm-operator/<tag or branch>/dist/install.yaml

Monitoring and Observability

The LiteLLM Operator exposes comprehensive Prometheus metrics for monitoring controller health, performance, and resource management.

Metrics Available

• Controller Metrics: Reconciliation loops, error rates, and latency per controller
• Resource Metrics: Status and health of managed LiteLLM resources
• Performance Metrics: Latency histograms and throughput measurements

Accessing Metrics

Metrics are exposed on the controller manager's metrics endpoint (default: :8443/metrics) and can be scraped by Prometheus or any compatible monitoring system.

Documentation

For detailed metrics documentation, monitoring setup, and alerting guidelines, see:

• Complete Metrics Documentation - Comprehensive guide with all metrics, queries, and alerts
• Controller Metrics Package - Technical documentation for developers

Quick Start Monitoring

# Controller reconciliation rate
rate(litellm_reconcile_loops_total[5m])

# Error percentage by controller  
rate(litellm_reconcile_errors_total[5m]) / rate(litellm_reconcile_loops_total[5m]) * 100

# 95th percentile reconciliation latency
histogram_quantile(0.95, rate(litellm_reconcile_latency_seconds_bucket[5m]))

Contributing

See CONTRIBUTING.

License

The LiteLLM Operator is released under the Apache 2.0 license. See the LICENSE file for details.