A high-throughput and memory-efficient inference and serving engine for LLMs

FlashInfer: Kernel Library for LLM Serving

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

llm-d is a Kubernetes-native high-performance distributed LLM inference framework

DeepEP: an efficient expert-parallel communication library

Inference scheduler for llm-d

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

llm-d helm charts and deployment examples

Pod for benchmarking interactive in llm-d

A simple GPU reservation tool for single host shared development systems

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.