What happened + What you expected to happen
I am using 3 servers to deploy Qwen3-VL-235B, with the Vllm version being 0.12.0.
The following error occurred during inference, suspending all requests. The error log is as follows:
12-05 22:34:04 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.12.0) with config: model='/root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/', speculative_config=None, tokenizer='/root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=10240, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=3, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=Qwen3-VL-235B-A22B-Instruct-QX, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None},
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-89e746732c291d2d'], resumed_req_ids=[], new_token_ids=[[9707]], all_token_ids={chatcmpl-89e746732c291d2d: [151644, 872, 198, 14990, 151645, 198, 151644, 77091, 198, 9707]}, new_block_ids=[null], num_computed_tokens=[9], num_output_tokens=[1]), num_scheduled_tokens={chatcmpl-89e746732c291d2d: 1}, total_num_scheduled_tokens=1, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], pending_structured_output_tokens=false, kv_connector_metadata=null, ec_connector_metadata=null)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.0003549875754348397, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={})
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] Traceback (most recent call last):
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2525, in _execute_until
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] result = self._dag_output_fetcher.read(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 312, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs = self._read_list(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 403, in _read_list
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] raise e
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 385, in _read_list
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] result = c.read(min(remaining_timeout, iteration_timeout))
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 776, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return self._channel_dict[self._resolve_actor_id()].read(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 480, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ret = self._worker.get_objects(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1001, in get_objects
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ] = self.core_worker.get_objects(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "python/ray/_raylet.pyx", line 2873, in ray._raylet.CoreWorker.get_objects
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "python/ray/includes/common.pxi", line 123, in ray._raylet.check_status
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ray.exceptions.RayChannelTimeoutError: System error: Timed out waiting for object available to read. ObjectID: 007e768e7da845eb498bddc26311bb55310405340100000002e1f505
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845]
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845]
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] Traceback (most recent call last):
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 836, in run_engine_core
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] engine_core.run_busy_loop()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 863, in run_busy_loop
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] self._process_engine_step()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 892, in _process_engine_step
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in step_with_batch_queue
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] model_output = future.result()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_utils.py", line 150, in result
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs = ray.get(self.ref_or_refs, timeout=timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return fn(*args, **kwargs)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return func(*args, **kwargs)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2945, in get
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return object_refs.get(timeout=timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/compiled_dag_ref.py", line 115, in get
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] self._dag._execute_until(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2535, in _execute_until
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] raise RayChannelTimeoutError(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ray.exceptions.RayChannelTimeoutError: System error: If the execution is expected to take a long time, increase RAY_CGRAPH_get_timeout which is currently 600 seconds. Otherwise, this may indicate that the execution is hanging.
(EngineCore_DP0 pid=579) INFO 12-05 22:43:45 [ray_executor.py:121] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2167 -- Tearing down compiled DAG
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, ebea74494add595681fa8eb901000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a08ade3040230e9440e8389d01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, f3b8be490d685a5a72a186ad01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 8207c5c28dd1bbec13f979eb01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 3ca183b46a32cd32c1d0cc8101000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 547d703b3b2c832906844d3301000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a78ea2bbcc62bff8fd3b026e01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 9e4ee0292c71cbb3818ba1d701000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, b11de28c8cf83ee57386a2dc01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, f2ce2ed881ae26c225f7e89d01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 09cee016abcbb572a5c0e78801000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 6ba6dd69827d314c949f927001000000)
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] AsyncLLM output_handler failed.
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Error in chat completion stream generator.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Error in chat completion stream generator.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
....
.....
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self.format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) *** StackTrace Information ***
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] __libc_start_main
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7)
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=723) [2025-12-05 22:43:45,428 E 723 992] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=723) [2025-12-05 22:43:45,428 E 723 992] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14
node
node
(EngineCore_DP0 pid=579) 2025-12-05 22:43:47,909 INFO compiled_dag_node.py:2197 -- Teardown complete
(EngineCore_DP0 pid=579) Process EngineCore_DP0:
(EngineCore_DP0 pid=579) Traceback (most recent call last):
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2525, in _execute_until
(EngineCore_DP0 pid=579) result = self._dag_output_fetcher.read(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 312, in read
(EngineCore_DP0 pid=579) outputs = self._read_list(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 403, in _read_list
(EngineCore_DP0 pid=579) raise e
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 385, in _read_list
(EngineCore_DP0 pid=579) result = c.read(min(remaining_timeout, iteration_timeout))
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 776, in read
(EngineCore_DP0 pid=579) return self._channel_dict[self._resolve_actor_id()].read(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 480, in read
(EngineCore_DP0 pid=579) ret = self._worker.get_objects(
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1001, in get_objects
(EngineCore_DP0 pid=579) ] = self.core_worker.get_objects(
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "python/ray/_raylet.pyx", line 2873, in ray._raylet.CoreWorker.get_objects
(EngineCore_DP0 pid=579) File "python/ray/includes/common.pxi", line 123, in ray._raylet.check_status
(EngineCore_DP0 pid=579) ray.exceptions.RayChannelTimeoutError: System error: Timed out waiting for object available to read. ObjectID: 007e768e7da845eb498bddc26311bb55310405340100000002e1f505
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) Traceback (most recent call last):
(EngineCore_DP0 pid=579) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=579) self.run()
(EngineCore_DP0 pid=579) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=579) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 847, in run_engine_core
(EngineCore_DP0 pid=579) raise e
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 836, in run_engine_core
(EngineCore_DP0 pid=579) engine_core.run_busy_loop()
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 863, in run_busy_loop
(EngineCore_DP0 pid=579) self._process_engine_step()
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 892, in _process_engine_step
(EngineCore_DP0 pid=579) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in step_with_batch_queue
(EngineCore_DP0 pid=579) model_output = future.result()
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_utils.py", line 150, in result
(EngineCore_DP0 pid=579) outputs = ray.get(self.ref_or_refs, timeout=timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(EngineCore_DP0 pid=579) return fn(*args, **kwargs)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(EngineCore_DP0 pid=579) return func(*args, **kwargs)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2945, in get
(EngineCore_DP0 pid=579) return object_refs.get(timeout=timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/compiled_dag_ref.py", line 115, in get
(EngineCore_DP0 pid=579) self._dag._execute_until(
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2535, in execute_until
(EngineCore_DP0 pid=579) raise RayChannelTimeoutError(
(EngineCore_DP0 pid=579) ray.exceptions.RayChannelTimeoutError: System error: If the execution is expected to take a long time, increase RAY_CGRAPH_get_timeout which is currently 600 seconds. Otherwise, this may indicate that the execution is hanging.
(EngineCore_DP0 pid=579) (raylet) Raylet is terminated. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
(EngineCore_DP0 pid=579) - bytes errored: 0
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) *** StackTrace Information ***
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] _libc_start_main
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=219, ip=192.168.101.9) WARNING 12-05 22:32:34 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_RTX_5880_Ada_Generation.json'] [repeated 11x across cluster]
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=217, ip=192.168.101.9) INFO 12-05 22:32:36 [gpu_worker.py:359] Available KV cache memory: 2.52 GiB [repeated 2x across cluster]
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=718) [2025-12-05 22:43:45,428 E 718 1039] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14 [repeated 6x across cluster]
(EngineCore_DP0 pid=579) (raylet) Raylet is terminated. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
(EngineCore_DP0 pid=579) - bytes errored: 0
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) *** StackTrace Information ***
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] __libc_start_main
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579)
(APIServer pid=378) INFO: Shutting down
(APIServer pid=378) INFO: Waiting for application shutdown.
(APIServer pid=378) INFO: Application shutdown complete.
(APIServer pid=378) INFO: Finished server process [378]
Versions / Dependencies
vllm：0.12.0
ray：2.52.1
python：3.12
Reproduction script
vllm serve /root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/
--tensor-parallel-size 4
--pipeline-parallel-size 3
--max-model-len 10240
--max_num_seqs 20
--gpu_memory_utilization 0.95
--max_num_batched_tokens 2048
--trust-remote-code
--block-size 32
--enable-chunked-prefill
--enable-prefix-caching
--distributed-executor-backend ray
--enforce-eager
--served-model-name Qwen3-VL-235B-A22B-Instruct-QX
Issue Severity
None
What happened + What you expected to happen
I am using 3 servers to deploy Qwen3-VL-235B, with the Vllm version being 0.12.0.
The following error occurred during inference, suspending all requests. The error log is as follows:
12-05 22:34:04 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.12.0) with config: model='/root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/', speculative_config=None, tokenizer='/root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=10240, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=3, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=Qwen3-VL-235B-A22B-Instruct-QX, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None},
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-89e746732c291d2d'], resumed_req_ids=[], new_token_ids=[[9707]], all_token_ids={chatcmpl-89e746732c291d2d: [151644, 872, 198, 14990, 151645, 198, 151644, 77091, 198, 9707]}, new_block_ids=[null], num_computed_tokens=[9], num_output_tokens=[1]), num_scheduled_tokens={chatcmpl-89e746732c291d2d: 1}, total_num_scheduled_tokens=1, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], pending_structured_output_tokens=false, kv_connector_metadata=null, ec_connector_metadata=null)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.0003549875754348397, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={})
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] Traceback (most recent call last):
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2525, in _execute_until
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] result = self._dag_output_fetcher.read(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 312, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs = self._read_list(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 403, in _read_list
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] raise e
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 385, in _read_list
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] result = c.read(min(remaining_timeout, iteration_timeout))
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 776, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return self._channel_dict[self._resolve_actor_id()].read(timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 480, in read
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ret = self._worker.get_objects(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1001, in get_objects
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ] = self.core_worker.get_objects(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "python/ray/_raylet.pyx", line 2873, in ray._raylet.CoreWorker.get_objects
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "python/ray/includes/common.pxi", line 123, in ray._raylet.check_status
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ray.exceptions.RayChannelTimeoutError: System error: Timed out waiting for object available to read. ObjectID: 007e768e7da845eb498bddc26311bb55310405340100000002e1f505
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845]
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845]
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] Traceback (most recent call last):
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 836, in run_engine_core
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] engine_core.run_busy_loop()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 863, in run_busy_loop
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] self._process_engine_step()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 892, in _process_engine_step
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in step_with_batch_queue
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] model_output = future.result()
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_utils.py", line 150, in result
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] outputs = ray.get(self.ref_or_refs, timeout=timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return fn(*args, **kwargs)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return func(*args, **kwargs)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2945, in get
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] return object_refs.get(timeout=timeout)
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/compiled_dag_ref.py", line 115, in get
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] self._dag._execute_until(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2535, in _execute_until
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] raise RayChannelTimeoutError(
(EngineCore_DP0 pid=579) ERROR 12-05 22:43:45 [core.py:845] ray.exceptions.RayChannelTimeoutError: System error: If the execution is expected to take a long time, increase RAY_CGRAPH_get_timeout which is currently 600 seconds. Otherwise, this may indicate that the execution is hanging.
(EngineCore_DP0 pid=579) INFO 12-05 22:43:45 [ray_executor.py:121] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2167 -- Tearing down compiled DAG
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, ebea74494add595681fa8eb901000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a08ade3040230e9440e8389d01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, f3b8be490d685a5a72a186ad01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 8207c5c28dd1bbec13f979eb01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 3ca183b46a32cd32c1d0cc8101000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 547d703b3b2c832906844d3301000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a78ea2bbcc62bff8fd3b026e01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 9e4ee0292c71cbb3818ba1d701000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,339 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, b11de28c8cf83ee57386a2dc01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, f2ce2ed881ae26c225f7e89d01000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 09cee016abcbb572a5c0e78801000000)
(EngineCore_DP0 pid=579) 2025-12-05 22:43:45,340 INFO compiled_dag_node.py:2172 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 6ba6dd69827d314c949f927001000000)
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] AsyncLLM output_handler failed.
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [async_llm.py:546] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Error in chat completion stream generator.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Error in chat completion stream generator.
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] Traceback (most recent call last):
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self._format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
....
.....
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 619, in chat_completion_stream_generator
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] async for res in result_generator:
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in generate
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] out = q.get_nowait() or await q.get()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise output
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 498, in output_handler
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] outputs = await engine_core.get_output_async()
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 885, in get_output_async
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] raise self.format_exception(outputs) from None
(APIServer pid=378) ERROR 12-05 22:43:45 [serving_chat.py:1287] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) *** StackTrace Information ***
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] __libc_start_main
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579) (raylet, ip=192.168.101.7)
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=723) [2025-12-05 22:43:45,428 E 723 992] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=723) [2025-12-05 22:43:45,428 E 723 992] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14
node
node
(EngineCore_DP0 pid=579) 2025-12-05 22:43:47,909 INFO compiled_dag_node.py:2197 -- Teardown complete
(EngineCore_DP0 pid=579) Process EngineCore_DP0:
(EngineCore_DP0 pid=579) Traceback (most recent call last):
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2525, in _execute_until
(EngineCore_DP0 pid=579) result = self._dag_output_fetcher.read(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 312, in read
(EngineCore_DP0 pid=579) outputs = self._read_list(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 403, in _read_list
(EngineCore_DP0 pid=579) raise e
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 385, in _read_list
(EngineCore_DP0 pid=579) result = c.read(min(remaining_timeout, iteration_timeout))
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 776, in read
(EngineCore_DP0 pid=579) return self._channel_dict[self._resolve_actor_id()].read(timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 480, in read
(EngineCore_DP0 pid=579) ret = self._worker.get_objects(
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1001, in get_objects
(EngineCore_DP0 pid=579) ] = self.core_worker.get_objects(
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "python/ray/_raylet.pyx", line 2873, in ray._raylet.CoreWorker.get_objects
(EngineCore_DP0 pid=579) File "python/ray/includes/common.pxi", line 123, in ray._raylet.check_status
(EngineCore_DP0 pid=579) ray.exceptions.RayChannelTimeoutError: System error: Timed out waiting for object available to read. ObjectID: 007e768e7da845eb498bddc26311bb55310405340100000002e1f505
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) Traceback (most recent call last):
(EngineCore_DP0 pid=579) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=579) self.run()
(EngineCore_DP0 pid=579) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=579) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 847, in run_engine_core
(EngineCore_DP0 pid=579) raise e
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 836, in run_engine_core
(EngineCore_DP0 pid=579) engine_core.run_busy_loop()
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 863, in run_busy_loop
(EngineCore_DP0 pid=579) self._process_engine_step()
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 892, in _process_engine_step
(EngineCore_DP0 pid=579) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in step_with_batch_queue
(EngineCore_DP0 pid=579) model_output = future.result()
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_utils.py", line 150, in result
(EngineCore_DP0 pid=579) outputs = ray.get(self.ref_or_refs, timeout=timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(EngineCore_DP0 pid=579) return fn(*args, **kwargs)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(EngineCore_DP0 pid=579) return func(*args, **kwargs)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2945, in get
(EngineCore_DP0 pid=579) return object_refs.get(timeout=timeout)
(EngineCore_DP0 pid=579) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/experimental/compiled_dag_ref.py", line 115, in get
(EngineCore_DP0 pid=579) self._dag._execute_until(
(EngineCore_DP0 pid=579) File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2535, in execute_until
(EngineCore_DP0 pid=579) raise RayChannelTimeoutError(
(EngineCore_DP0 pid=579) ray.exceptions.RayChannelTimeoutError: System error: If the execution is expected to take a long time, increase RAY_CGRAPH_get_timeout which is currently 600 seconds. Otherwise, this may indicate that the execution is hanging.
(EngineCore_DP0 pid=579) (raylet) Raylet is terminated. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
(EngineCore_DP0 pid=579) - bytes errored: 0
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) *** StackTrace Information ***
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] _libc_start_main
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=219, ip=192.168.101.9) WARNING 12-05 22:32:34 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_RTX_5880_Ada_Generation.json'] [repeated 11x across cluster]
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=217, ip=192.168.101.9) INFO 12-05 22:32:36 [gpu_worker.py:359] Available KV cache memory: 2.52 GiB [repeated 2x across cluster]
(EngineCore_DP0 pid=579) (RayWorkerWrapper pid=718) [2025-12-05 22:43:45,428 E 718 1039] raylet_client.cc:202: Error pushing mutable object: RpcError: RPC error: Socket closed rpc_code: 14 [repeated 6x across cluster]
(EngineCore_DP0 pid=579) (raylet) Raylet is terminated. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
(EngineCore_DP0 pid=579) - bytes errored: 0
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579) [2025-12-05 22:44:44,904 C 36 36] (raylet) experimental_mutable_object_provider.cc:154: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: object_manager->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed.
(EngineCore_DP0 pid=579) *** StackTrace Information ***
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf4569a) [0x61fd065f069a] ray::operator<<()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf47cb9) [0x61fd065f2cb9] ray::RayLog::~RayLog()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x3ecf0a) [0x61fd05a97f0a] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x2a9e1f) [0x61fd05954e1f] ray::raylet::NodeManager::HandlePushMutableObject()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x31eb96) [0x61fd059c9b96] ray::rpc::ServerCallImpl<>::HandleRequestImpl()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x844438) [0x61fd05eef438] EventTracker::RecordExecution()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83c147) [0x61fd05ee7147] std::_Function_handler<>::_M_invoke()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x83d0eb) [0x61fd05ee80eb] boost::asio::detail::executor_op<>::do_complete()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf10aeb) [0x61fd065bbaeb] boost::asio::detail::scheduler::do_run_one()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf13099) [0x61fd065be099] boost::asio::detail::scheduler::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0xf135a2) [0x61fd065be5a2] boost::asio::io_context::run()
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x22031f) [0x61fd058cb31f] main
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7a87161abd90]
(EngineCore_DP0 pid=579) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7a87161abe40] __libc_start_main
(EngineCore_DP0 pid=579) /usr/local/lib/python3.12/dist-packages/ray/core/src/ray/raylet/raylet(+0x243007) [0x61fd058ee007]
(EngineCore_DP0 pid=579)
(EngineCore_DP0 pid=579)
(APIServer pid=378) INFO: Shutting down
(APIServer pid=378) INFO: Waiting for application shutdown.
(APIServer pid=378) INFO: Application shutdown complete.
(APIServer pid=378) INFO: Finished server process [378]
Versions / Dependencies
vllm：0.12.0
ray：2.52.1
python：3.12
Reproduction script
vllm serve /root/.cache/huggingface/Qwen3-VL-235B-A22B-Instruct/
--tensor-parallel-size 4
--pipeline-parallel-size 3
--max-model-len 10240
--max_num_seqs 20
--gpu_memory_utilization 0.95
--max_num_batched_tokens 2048
--trust-remote-code
--block-size 32
--enable-chunked-prefill
--enable-prefix-caching
--distributed-executor-backend ray
--enforce-eager
--served-model-name Qwen3-VL-235B-A22B-Instruct-QX
Issue Severity
None