[Feat][Executor] Introduce RayExecutorV2

[jeffreywang88]

Purpose

• Implement RayExecutorV2, a new Ray-based distributed executor that uses MessageQueue (shared memory + TCP fallback) for the control plane instead of Ray compiled graphs. It reuses MultiprocExecutor's MQ-based RPC and NCCL data plane while spawning workers as Ray actors into placement group bundles.
• Workers on the same node as the driver communicate via shared memory; cross-node workers automatically fall back to ZMQ TCP transport. Bundle assignments are sorted driver-node-first to ensure rank 0 is co-located with the executor.
• Add VLLM_USE_RAY_V2_EXECUTOR_BACKEND env var feature flag (default off) to opt into the new executor when distributed_executor_backend="ray". Enable async scheduling support for the new backend.

For more details, please refer to RFC: #35848.

EEP support is out-of-scope for this PR and is tracked here: #38164.

Test Plan

Unit tests

• pytest tests/distributed/test_ray_v2_executor.py: executor init, TP/PP combos, placement groups, RPC, worker death, shutdown
• pytest tests/utils_/test_ray_utils.py: bundle sorting logic
• Validate cross-node TCP path for MessageQueue with test_mq_tcp_multinode.py

Integration tests

• pytest tests/distributed/test_ray_v2_executor_e2e.py: Creates Ray actors which initialize AsyncLLMEngine internally and verify that they can serve requests.
• pytest tests/distributed/test_pipeline_parallel.py -k "ray": PP correctness with the new backend
• pytest tests/basic_correctness/test_basic_correctness.py -k "ray": basic correctness

Test Result

Benchmark results (Qwen/Qwen3-8B on L4)

Server:

# MP backend
vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 --distributed-executor-backend mp --port 8000

# Existing Ray backend
VLLM_USE_RAY_V2_EXECUTOR_BACKEND=0 vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 --distributed-executor-backend ray --port 8000

# Ray V2 backend
VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1 vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 --distributed-executor-backend ray --port 8000

Client

vllm bench serve --model Qwen/Qwen3-8B --dataset-name random --input-len 512 --output-len 128 --num-prompts 500 --request-rate 10 --port 8000

• TP=4; MP backend (async scheduling is on by default)

============ Serving Benchmark Result ============
Successful requests:                     500       
Failed requests:                         0         
Request rate configured (RPS):           10.00     
Benchmark duration (s):                  53.64     
Total input tokens:                      256000    
Total generated tokens:                  64000     
Request throughput (req/s):              9.32      
Output token throughput (tok/s):         1193.20   
Peak output token throughput (tok/s):    1475.00   
Peak concurrent requests:                82.00     
Total token throughput (tok/s):          5965.99   
---------------Time to First Token----------------
Mean TTFT (ms):                          117.12    
Median TTFT (ms):                        117.26    
P99 TTFT (ms):                           156.28    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          40.95     
Median TPOT (ms):                        41.81     
P99 TPOT (ms):                           46.68     
---------------Inter-token Latency----------------
Mean ITL (ms):                           40.95     
Median ITL (ms):                         40.80     
P99 ITL (ms):                            54.51     

• TP=4; Ray backend

============ Serving Benchmark Result ============
Successful requests:                     500       
Failed requests:                         0         
Request rate configured (RPS):           10.00     
Benchmark duration (s):                  53.93     
Total input tokens:                      256000    
Total generated tokens:                  64000     
Request throughput (req/s):              9.27      
Output token throughput (tok/s):         1186.80   
Peak output token throughput (tok/s):    1464.00   
Peak concurrent requests:                84.00     
Total token throughput (tok/s):          5934.02   
---------------Time to First Token----------------
Mean TTFT (ms):                          86.00     
Median TTFT (ms):                        86.32     
P99 TTFT (ms):                           120.62    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          45.88     
Median TPOT (ms):                        47.14     
P99 TPOT (ms):                           51.94     
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.88     
Median ITL (ms):                         47.21     
P99 ITL (ms):                            58.59     

• TP=4; Ray V2 backend w/ async scheduling

============ Serving Benchmark Result ============
Successful requests:                     500       
Failed requests:                         0         
Request rate configured (RPS):           10.00     
Benchmark duration (s):                  53.67     
Total input tokens:                      256000    
Total generated tokens:                  64000     
Request throughput (req/s):              9.32      
Output token throughput (tok/s):         1192.53   
Peak output token throughput (tok/s):    1442.00   
Peak concurrent requests:                82.00     
Total token throughput (tok/s):          5962.65   
---------------Time to First Token----------------
Mean TTFT (ms):                          119.11    
Median TTFT (ms):                        120.43    
P99 TTFT (ms):                           154.20    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          41.11     
Median TPOT (ms):                        42.06     
P99 TPOT (ms):                           46.64     
---------------Inter-token Latency----------------
Mean ITL (ms):                           41.11     
Median ITL (ms):                         40.82     
P99 ITL (ms):                            54.10 

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Hi @jeffreywang-anyscale, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[jeffreywang88]

@njhill FYI this PR is not ready for review yet as I'm iterating on the CI. Will let you know once it's in a good shape for review!

[mergify]

Hi @jeffreywang-anyscale, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @jeffreywang-anyscale, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[kouroshHakha]

ok beautiful. some broad comments after the first pass.