Intelligent Router for Mixture-of-Models

Deploy a Production Ready Kubernetes Cluster

🌱 Kubernetes lifecycle management operator based on kubespray.

Gateway API Inference Extension

JobSet: a k8s native API for distributed ML training and HPC workloads

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

Distributed AI Model Training and LLM Fine-Tuning on Kubernetes

Kubernetes-native Job Queueing

LeaderWorkerSet: An API for deploying a group of pods as a unit of replication

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Universal Python SDK to run AI workloads on Kubernetes

A Datacenter Scale Distributed Inference Serving Framework

llm-d enables high-performance distributed LLM inference on Kubernetes

Production-Grade Container Scheduling and Management

helm charts for deploying models with llm-d

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

An open and reliable container runtime

OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)

Inference scheduler for llm-d

A toolkit to run Ray applications on Kubernetes

Node Resource Interface

Helm charts for llm-d

Autoscaling components for Kubernetes

Lightweight Kubernetes

eBPF-based Networking, Security, and Observability

NVIDIA GPU Operator creates, configures, and manages GPUs in Kubernetes

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...