[Serve] Make batching work with multiplexing

[abrarsheikh]

fixes #56633

• Add documentation
• update get_multiplexed_model_id to see if we are batch context first
• update logic
• add tests
• does not introduce any backwards incompatibility, previously the system did not provide any guarantee about contents of a batch and now we are add a constraint that guarantees each batch contains requests for same model.
• execute sub batches concurrently

The thing I dislike about this implementation is that it does not fill the batch in the case where the replica is responsible for > 2 models and incoming traffic is equally distributed between those models. Becasue the current implementation fills the batch first, then divides them.

Metric	Baseline (42905 reqs)	Master (27526 reqs)	Δ Change (Master − Baseline)
Requests	42,905	27,526	−15,379
Fails	0	0	0
Median (ms)	290	300	+10 ms
95%ile (ms)	560	570	+10 ms
99%ile (ms)	620	640	+20 ms
Average (ms)	327.41	332.96	+5.55 ms
Min (ms)	61	80	+19 ms
Max (ms)	764	802	+38 ms
Avg Size (bytes)	13	13	0
Current RPS	299	293	−6
Current Failures/s	0	0	0

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[harshit-anyscale]

lgtm, except model_1.pt file is added but has no changes

[akyang-anyscale]

The way I understand this description is that Serve will treat each model's batch independently, i.e. waiting to reach the max_batch_size or the timeout before firing for each model, but in reality, it waits for the max_batch_size or timeout across all models. For example if our max_batch_size=8, Serve will process sub batches of size [1, 4, 3] instead of waiting for each model to have 8 request.

[abrarsheikh]

you are right.