[Docs] Document max_tasks_in_flight_per_actor vs max_concurrent_batches

[Partth101]

Adds documentation explaining the interaction between max_concurrent_batches and max_tasks_in_flight_per_actor parameters for vLLM batch inference. This addresses user confusion from a Slack thread where setting max_concurrent_batches=8 only resulted in 4 tasks running (because max_tasks_in_flight_per_actor defaults to 4).

The new section includes:

Parameter descriptions with verified default values from source code
Explanation of how they work together
Troubleshooting guidance for the autoscaling warning

Related Issue 60421

[gemini-code-assist]

Code Review

This pull request adds valuable documentation explaining the interaction between max_concurrent_batches and max_tasks_in_flight_per_actor in vLLM batch inference, which will be very helpful for users. The explanation of the parameters, how they work together, and the troubleshooting guide for the autoscaling warning are clear and well-structured. I've added a couple of minor suggestions to improve consistency within the documentation. Overall, this is a great addition to the Ray Data documentation.

[gemini-code-assist]

The recommendation for batch_size here is _batch_size >= 32_, but the docstring for vLLMEngineProcessorConfig in python/ray/data/llm.py suggests _batch_size >= 64_ for max_concurrent_batches to be effective. To maintain consistency between the documentation and the code's docstrings, it would be better to align this recommendation.

Suggested change

	**max_concurrent_batches** (default: 8)
	The number of batches that can execute concurrently within a single vLLM engine actor. This overlaps batch processing to hide tail latency. Works well for ``batch_size >= 32``.
	**max_concurrent_batches** (default: 8)
	The number of batches that can execute concurrently within a single vLLM engine actor. This overlaps batch processing to hide tail latency. Works well for ``batch_size >= 64``.

[Partth101]

Updated to batch_size >= 64 to match the public API docstring in llm.py:127.

[gemini-code-assist]

For consistency with other examples in this document, could you please use unsloth/Llama-3.1-8B-Instruct as the model_source? The literalinclude just above this snippet also uses the unsloth model, so this change would make the new section more consistent.

Suggested change

	model_source="meta-llama/Llama-3.1-8B-Instruct",
	model_source="unsloth/Llama-3.1-8B-Instruct",

[Partth101]

Changed to unsloth/Llama-3.1-8B-Instruct for consistency with the rest of the document.

[jeffreywang88]

Thanks for the contribution!

[jeffreywang88]

Suggested change

	How many tasks Ray Data can queue per actor before waiting for results. This enables task prefetching so there are always tasks ready when the actor finishes one. Access through the ``experimental`` dict.
	The number of tasks Ray Data can queue per actor before waiting for results. This enables task prefetching so there are always tasks ready when the actor finishes one. Access through the ``experimental`` dict.

[jeffreywang88]

Suggested change

	The number of batches that can execute concurrently within a single vLLM engine actor. This overlaps batch processing to hide tail latency. Works well for ``batch_size >= 64``.
	The number of batches that can execute concurrently within a single vLLM engine actor. This overlaps batch processing to hide tail latency.

The optimal batch size depends specifically on the workload.

[jeffreywang88]

Suggested change

	If ``max_tasks_in_flight_per_actor``, which defaults to 4, is less than ``max_concurrent_batches``, which defaults to 8, the actor can't reach full concurrency because there aren't enough queued tasks to fill all concurrent slots.
	To maximize throughput, increase ``max_tasks_in_flight_per_actor`` to keep the actor's task queue saturated.
	With ``max_tasks_in_flight_per_actor < max_concurrent_batches``, Ray Data actors are undersaturated. To maximize throughput, increase ``max_tasks_in_flight_per_actor`` to keep the actor's task queue saturated.

[jeffreywang88]

Great insight! Here's the full warning message:

026-01-28 10:04:06,208ksWARNING default_actor_autoscaler.py:241 -- ⚠️  Actor Pool configuration of the ActorPoolMapOperator[MapBatches(vLLMEngineStageUDF)] will not allow it to scale up: configured utilization threshold (175.0%) couldn't be reached with configured max_concurrency=8 and max_tasks_in_flight_per_actor=4 (max utilization will be max_tasks_in_flight_per_actor / max_concurrency = 50%) 

However, this doesn't matter for fixed size pools which vLLMEngineStageUDF uses. Autoscaling will never kick in because we have min_size == max_size == initial_size, but it's still a good practice to have higher max_tasks_in_flight_per_actor. We should adjust its default.

ray/python/ray/llm/_internal/batch/processor/vllm_engine_proc.py

Line 287 in 221a193

**config.get_concurrency(autoscaling_enabled=False),

ray/python/ray/llm/_internal/batch/processor/base.py

Line 127 in 221a193

return {"size": self.concurrency}

ray/python/ray/data/_internal/compute.py

Lines 144 to 146 in 221a193

	min_size = size
	max_size = size
	initial_size = size

We shouldn't raise this warning if fixed size pools are used in Ray Data. I'm working on a fix at the moment.

[jeffreywang88]

Let's do these:

1. Adjust the default value

   ray/python/ray/llm/_internal/batch/processor/base.py

   Line 22 in 221a193

   DEFAULT_MAX_TASKS_IN_FLIGHT = 4

   from 4 to 16.
2. Remove this block of comment.
3. Adjust the default values in the "Understanding the parameters" section above.

[jeffreywang88]

Here's the fix to bypass the warnings: #60569.

[jeffreywang88]

Could you please fix microcheck as well?

[jeffreywang88]

LGTM, thank you again for the contribution! Kicking off release tests.

[Partth101]

Thank you very much @jeffreywang-anyscale

[Partth101]

@jeffreywang-anyscale Just checking if there’s anything else I can contribute to this PR, or if it’s ready to be merged

[jeffreywang88]

Retrying premerge tests

[bveeramani]

Stamp