vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs
External link for vLLM
vLLM reposted this
Let me re-introduce vLLM Semantic Router to you, know our mission and what we have built and achieved with the wide and magic open source ecosystem 🌏 Come and join us 👋 Andy Luo Huamin Chen Steve Liu Tom Aarsen Lysandre Debut Ádám Kovács Bowei He Haolun Wu Yankai Chen Aayush Saini KunTai ,Wu Mahdi G. Eda Z.. Eddie Zhou Haichen Zhang Anup Sharma Alex Jia Wilson Wu Theo Hsiung Soobin Park Yifan Wang David Shrader Abhinav Mahajan Yincheng Ren Guan-Ming Chiu
We're excited to have a strong vLLM presence at PyTorch Conference North America this year. 🔷 Simon Mo, vLLM core maintainer and Co-founder / CEO of Inferact, will give the keynote on scaling open frontier inference infrastructure 🔷 vLLM core maintainer Nick Hill and George Novack will be at the "Meet the Developers of vLLM" 🔷 vLLM maintainers Tyler Michael Smith, Lucas Wilkinson, and Joseph Groenenboom from Red Hat will be giving talks on Agentic Inference and Attention in vLLM. Thank you to PyTorch for highlighting the community's work!
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs. At #PyTorchCon North America, you’ll find vLLM-related work throughout the program, from Simon Mo’s keynote to technical sessions and posters covering LLM inference and serving. Simon will present the keynote “vLLM Update: Scaling Open Frontier Inference Infrastructure,” covering improvements to vLLM’s core architecture and major optimizations in KV cache management and GPU kernels. Across #PyTorchCon, technical sessions dig into vLLM-related inference, including attention, KV cache management and transfer, disaggregated serving, elastic expert parallelism, scaling across hardware without forks, and more. "Meet the Developers of vLLM" will feature George Novack and Nick Hill. Posters cover additional vLLM-related work on custom accelerators, Trainium, expert parallelism and RDMA KV-cache transfer, multi-chip KV-cache transfer, tier-aware routing, and more. Explore the vLLM sessions: https://lnkd.in/eeD4c_xh PyTorch conferences are the open source AI community’s town square, where what’s next gets decided. Register for PyTorch Conference North America, October 20–21 in San Jose: https://hubs.la/Q04vK-VQ0
vLLM reposted this
v0.4 Hermes is here. Over 100 contributors, nearly 1,000 commits in the making. 🚀 vLLM Semantic Router v0.4.0 is our biggest release yet. More than 100 people contributed to the project for the first time on the way here. Hermes brings open Vela and Decision models, Mixture-of-Models recipes, a stronger runtime, and sr-bench together around one vision: many models working as one system. What makes me proudest is the community behind it. Thank you to everyone who contributed, reviewed, tested, documented, and helped others get involved. I can’t wait to see what you build with Hermes. 🤗 Read the release story: https://lnkd.in/gH_qTpW3 Website: https://vllm-sr.ai GitHub: https://lnkd.in/gDnEJjvi HuggingFace: https://lnkd.in/g852dtM6 To everyone who contributed to Hermes—through code, models, reviews, testing, documentation, or ideas—thank you. This release belongs to all of you. And congratulations to our new committers and maintainers! 🙌 Ramine Roane Andy Luo Huamin Chen Steve Liu Tom Aarsen Lysandre Debut Ádám Kovács Bowei He Aayush Saini KunTai ,Wu Mahdi G. Eda Z. Eddie Zhou Haichen Zhang Anup Sharma Alex Jia Wilson Wu Theo Hsiung Soobin Park Yifan Wang David Shrader Abhinav Mahajan Yincheng Ren Guan-Ming Chiu #vLLM #vLLMSemanticRouter #OpenSource #OpenSourceAI #GenerativeAI #LLM #AIInfrastructure #ModelServing #ModelRouting #MixtureOfModels #Inference #MLOps #AICommunity
Open-source AI is moving fast. There’s a lot to unpack as workloads become increasingly agentic. Great to have Kaichao You share his perspective at #ApsaraConference2026. Thanks Alibaba Cloud for having us. Excited to keep pushing open-source inference forward together. 🚀
Building for the next era of open-source AI. At #ApsaraConference2026, Kaichao You, vLLM Core Maintainer, Inferact Co-Founder and Chief Scientist, shared his perspective on the continued progress of open-source models and their growing role in the AI market. He also pointed to Qwen3.8-Max as giving him confidence that open-source models can stay at the frontier for complex tasks and agentic use cases. Stay tuned for more updates! Explore more: https://lnkd.in/emEvJkue #AgenticEra #AgentNative #AIAgentsAtApsara #BringYourAgent #AlibabaCloud
The Inferact team open sourced a TPU megakernel for Kimi K3 achieving 709 tokens/s, against 450 tokens/s on GB200. All 92 of K3's MoE layers run in a single Pallas kernel, with weight prefetching that reaches across layer boundaries so transfers for one layer overlap with computation in the previous one. Shoutout to the team! Writeup and repo link in the thread.
Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. Write-up and megakernel in the comments below.
DiffusionGemma-Jev now runs on vLLM 🚀 Ask yes/no, multiple-choice, or scored questions and get confidence with every answer. vLLM seeds a canvas with the response template, leaves only the answer slots noisy, then reads a probability distribution from every slot in a single denoising step. Huge thanks to mmastrac for driving this upstream! 🙏 Try it from the vLLM repo with vllm/vllm-openai:nightly. Start vLLM, launch the example server, then call the Jev-compatible endpoint: Pull Request: https://lnkd.in/gs9J8p3h Recipe: https://lnkd.in/gteugQCz
762 commits. 315 contributors. 104 first-timers. vLLM v0.30.0 is live. 🎉 Highlights: 🤖 Hybrid-attention hot paths: Kimi K3 streamlines KDA, AttnRes, and MLA; DeepSeek-V4.1-Flash adds MXFP8 KV and async Engram; Qwen3.8-Flash-Next fuses QSA/PLE and cuts sparse-GQA overhead 🗄️ HiSparse adds a host tier beneath sparse-MLA decode; under GPU pressure, only top-k misses return to a per-request hot buffer 🛠️ Model Runner V2 brings EAGLE3-style drafts to pipeline parallelism and extends adaptive verification to every draft-model speculator through online acceptance estimation (#50514, #52228) 🖋️ Dual-key Gumbel-max watermark generation and detection, with per-request opt-out and speculative-decoding support 🆕 New models include GLM-5.3-Flash, K2-Horizon, Cohere Compass, and Bailing V3 VL ⚡ Fast Start keeps post-quantized, TP-sharded weights in a per-GPU daemon; restarts map them over CUDA IPC with --load-format ipc_cache --- Performance work: 🎓 RL sampling masks: GPU compaction restores throughput to within 2–3% of no-mask at 512 requests, fixing an about 2x step-time regression (#54901) 🌙 Kimi K3 mixed batches: 5.2–7.7% higher E2E throughput (#56159) 🖼️ Gemma 4 multimodal prefixes: 3.23x E2E speedup on RTX PRO 6000 (#53147) 🔴 AMD ROCm: packed W4A16 zero-points cut median TPOT 26.4% on Gemma 4 AWQ (#54965) 💻 Arm CPU: about 6% higher throughput on Llama 3.1 8B W8A8 (#55352) --- ⚠️ Before you upgrade: • Scale-out endpoints are now opt-in with --enable-scale-out; the old env var is gone • GPTQ g_idx and items deprecated for 0.29 were removed • The all Mamba cache mode is deprecated; start gRPC with vllm serve --grpc • YaRN may derive a shorter max_model_len; DCP now rejects backends that do not declare support • The default audio resampler is now torchaudio 🙏 Thanks to all 315 contributors this cycle (104 first-timers). 📖 Full release notes → https://lnkd.in/gCRD85J4
vLLM reposted this
🎲 Introducing Decision 1.0: Open Decision Foundation Models. Jev’s launch has put decision models in the spotlight. The excitement points to a powerful opportunity: AI that makes fast, structured decisions inside the software we use every day. Today, we’re introducing six open models—Kai, Lex, Eos, Sol, Nox, and Lux—that turn context into choices, judgments, and scores your software can use directly. Route a request, choose an agent’s next action, check a policy, or evaluate whole batches of records. Open weights and inference code put that decision layer in your hands. ✨ For vLLM Semantic Router, this is a natural next step. We brought signal decision architecture last year, and a decision engine for model selection. Decision gives us a foundation for making those choices with generalization capability. 🚀 Next: native integration with vLLM-SR, making it as a runtime for decision models with an OpenDecision API, while retaining the System One-compatible interface for general decision workflows beyond model routing. We’re excited to put these models in your hands—and see what you build with them. 📖 Read the launch story: https://lnkd.in/gZ7_JrET 🤗 Try Decision Studio: https://lnkd.in/g5yXVkaZ 🎲 Explore the model family: https://lnkd.in/g5yXVkaZ A big thank you to everyone who contributed —and to the open-source community whose work we build on. More to come! 🙌 Andy Luo Huamin Chen Steve Liu Bowei He Haichen Zhang Ádám Kovács Lysandre Debut Daniel van Strien Tom Aarsen #OpenSource #DecisionModels #IntelligentRouting #AI #vLLM #Jev #OpenJev
vLLM reposted this
Vela 1.0 is here! ⛵️ The model family for intelligent routing — across text, images, and audio. Every request finds its way. 🌊 As AI systems bring more models together, understanding what each request needs becomes essential. That’s what we’re building Vela for ✨ This release brings together 14 open models for request understanding, safety and privacy detection, retrieval, and multimodal matching. Built for vLLM Semantic Router, and open to anyone building systems where models with different strengths work together. We’re excited to put these models in your hands—and see what you build with them. 📖 Read the launch story: https://lnkd.in/gy9mGDm4 🤗 Try Vela Studio: https://lnkd.in/gi6vAMBM ⛵️ Explore the model family: https://lnkd.in/gZKBDQxu This is just the beginning of our series of models specialized for intelligent routing. More to come! 🧙🪄 A big thank you to everyone who contributed, tested, shared feedback, and supported this release—and to the open-source community whose work we build on. Excited to keep building together! 🙌 Andy Luo Guruprasad MP Akshay Kharbanda Steve Liu Bowei He Huamin Chen Ádám Kovács Tom Aarsen #OpenSource #MultimodalAI #IntelligentRouting #vLLM
Thank you to everyone who showed up, spoke, and stayed throughout the vLLM Conference. Lines out the door for talks say a lot about this community. Thank you to Inferact for hosting! Session recordings are in the thread.
That's a wrap on the vLLM Conference at Ray Summit. Lines out the door for talks about an inference engine—that says everything about how much this community shows up. Thank you to Anyscale for co-hosting with us, and we're already looking forward to the next event. Every session recording is in the comments below: