About CoreWeave Inference - CoreWeave Docs

CoreWeave offers three ways to serve AI models: Serverless Inference, Dedicated Inference, and Inference on CoreWeave Kubernetes Service (CKS), which you operate yourself. Use this guide to choose a starting point based on your model, traffic pattern, performance targets, and the serving infrastructure your team wants to operate.

Choose a deployment option

Start with the model and runtime your application needs, then compare the operational responsibilities of each option.

Serverless Inference

Call open-weight models from a managed catalog, or serve your own LoRA adapters on supported base models. CoreWeave manages provisioning, routing, and scaling.

Dedicated Inference

Bring your own weights (BYOW), choose GPU resources and a supported runtime, and let CoreWeave operate the serving platform.

Inference on CKS

Operate your serving stack on Kubernetes when you need control over runtimes, networking, and orchestration.

The following table compares what you configure and operate in each option:

Consideration	Serverless Inference	Dedicated Inference	Inference on CKS
Model selection	Choose a catalog model, or serve a LoRA adapter on a supported base model.	Supply weights compatible with a supported runtime.	Deploy models and runtimes that you configure and operate.
Serving operations	CoreWeave operates the serving infrastructure.	CoreWeave operates the platform. You configure deployments, gateways, and scaling.	You operate the serving software, scaling, and application reliability on CKS.
Infrastructure control	Use the service through its API or the Playground.	Select supported GPU resources and Availability Zones.	Configure Kubernetes resources and workload networking.
Billing basis	Pay per token. Storage for LoRA adapters can incur separate charges.	GPU-hour or node-based billing, depending on your contract.	CKS compute billing, including applicable reservations.
Main tradeoff	Your workload must fit the catalog and service limits.	Your deployment must fit supported runtimes and platform configuration.	Your team takes responsibility for the serving stack.

Match the option to your workload

These recommendations are starting points for evaluation. A workload can fit more than one option. Model compatibility and measured performance determine the final choice. The following table maps common workloads to a starting option and what to evaluate before you commit:

Workload	Starting point	What to evaluate
Prototype or application with variable traffic and a catalog model	Serverless Inference	Check model availability, concurrency limits, spending caps, and latency under expected demand.
Fully fine-tuned or custom model with recurring production traffic	Dedicated Inference	Confirm runtime compatibility, model memory requirements, and utilization across quiet and busy periods. For a LoRA adapter on a supported base model, also evaluate Serverless Inference.
Interactive application with a defined latency target	Dedicated Inference	Use it when you need configurable replicas and capacity claims. Benchmark response latency during traffic bursts and deployment changes.
Coding assistant, agent, or retrieval-augmented generation (RAG) application with a catalog model	Serverless Inference	Measure cache reuse, prompt lengths, and response latency. Serverless Inference applies prefix caching to its hosted models. For custom weights, evaluate Dedicated Inference. Choose Inference on CKS when required runtime or routing changes exceed managed configuration.
Batch generation or evaluation with a catalog model	Serverless Inference	Separate batch completion deadlines from interactive response targets, and account for concurrency limits. Neither Serverless Inference nor Dedicated Inference provides a batch ingestion endpoint.
Custom image, audio, or video pipeline	Inference on CKS	Use it when the pipeline requires custom serving software or orchestration. Verify protocols, preprocessing, storage access, and scaling requirements. Evaluate Dedicated Inference only if its supported runtime and API fit the workload.
Distributed serving or infrastructure-level tuning	Inference on CKS	Use it when you need to operate the serving architecture. Evaluate runtime support, communication between GPUs, and the engineering effort to operate the deployment.

If your workload has data residency, isolation, encryption, or private connectivity requirements, verify each required control before choosing an option. Reserved GPU capacity or an Availability Zone selection doesn’t, by itself, establish that all application requirements are met.

Example deployment choices

The following hypothetical examples show how requirements affect the choice:

A team testing a document assistant starts with Serverless Inference because a catalog model meets its needs. It measures quality and latency before deciding whether a LoRA adapter on Serverless Inference or full custom weights on Dedicated Inference is justified.
A team serving a fine-tuned text model evaluates Dedicated Inference to keep control of weights and scaling without operating the serving platform. It stages the artifacts in CoreWeave AI Object Storage and benchmarks the deployment against production traffic.
A media application uses Inference on CKS to operate a custom model server alongside preprocessing workers and a queue. The team configures workload networking, scaling, and monitoring.
An application with overnight generation and interactive requests evaluates the two traffic patterns separately. It can use different deployments or options when their latency, model, or orchestration requirements differ.

For runnable examples, see Deploy Kimi K3 on Dedicated Inference, Deploy NVIDIA Dynamo on CKS, and Deploy vLLM for inference.

Compare performance, capacity, and cost

Define success using traffic representative of your application. Include typical and peak request rates, input and output lengths, and repeated context. Measure time to first token, response latency (including slow requests), throughput, and cost per completed request or token. For Dedicated Inference, tune concurrency against your latency and throughput targets. A setting that works for batch generation might not meet an interactive application’s response target. Follow the concurrency benchmarking guidance and repeat measurements when the model or traffic changes. Autoscaling adjusts replica counts within available capacity, but it doesn’t reserve GPUs. If you need reserved hardware, evaluate capacity claims. Managed claims incur charges while they exist, including when their capacity is idle. Dedicated Inference doesn’t support scale-to-zero, so account for its minimum running replicas when comparing costs.

Pricing

Compare Serverless Inference pricing with Dedicated Inference billing and CoreWeave compute pricing. Include expected utilization and the effort to operate a self-managed serving stack. Steady traffic is a reason to evaluate Dedicated Inference, but it doesn’t establish a universal cost advantage.

Key capabilities

Serverless Inference and Dedicated Inference expose OpenAI-compatible endpoints, so existing OpenAI client libraries, agents, and tooling can connect with minimal changes. On CKS, the serving software you deploy determines the API your application uses. Check the selected product’s documentation for model support, endpoint behavior, resource availability, and platform integrations. For Dedicated Inference, consult supported engines before choosing a runtime.

Access the API

You can configure Dedicated Inference resources through the Dedicated Inference API over REST, gRPC, or Connect, through the CoreWeave Intelligent CLI, or through Terraform. For more information, see the Dedicated Inference API reference and Deploy Dedicated Inference with Terraform. For Serverless Inference setup and request examples, see Serverless Inference prerequisites and usage examples.

Get started

Choose the guide for your deployment option:

​Choose a deployment option

Serverless Inference

Dedicated Inference

Inference on CKS

​Match the option to your workload

​Example deployment choices

​Compare performance, capacity, and cost

​Pricing

​Key capabilities

​Access the API

​Get started