Using realtime causal video pipelines with Miles Diffusion rollouts

[BitCalSaul]

Hi, we’re working on integrating MiniWorld, an action-conditioned block-causal video world model, into SGLang Diffusion.

The existing LingBot-World realtime pipeline looks like a good starting point. We plan to implement the MiniWorld model, action conditioning, KV-cache logic and pipeline ourselves.

Our main question is whether the realtime causal pipeline can currently be used with the post-training /rollout/generate API and Miles Diffusion, including stochastic rollout log-probs and weight updates.

Is this integration path supported, or would the realtime and rollout paths need to be connected first?

[flymysql]

Short answer: no — they're separate paths today, and the realtime session API is a different contract from a rollout API

I checked the current tree for you. There is no rollout/post-training entry point anywhere in the diffusion side:

$ git grep -rln "rollout" -- python/sglang/multimodal_gen/
# (no results)

So the integration you're proposing isn't a matter of turning on a flag — the realtime and rollout paths would need to be connected first. Here's what exists, so you can judge how much work that is.

What the realtime path actually is

docs/docs/sglang-diffusion/realtime_models.mdx defines two distinct modes, and the distinction matters for you:

Mode	State lifetime	API
Realtime session	persists until the client disconnects / session ends	/v1/realtime_video/generate WebSocket
Request-based causal generation	reused across chunks within one request, then released	standard video generation API

Current supported realtime pipelines are only:

Model family	Pipeline	Live controls
LingBot World	LingBotWorldCausalDMDPipeline	Camera actions and prompt updates
SANA-WM	SanaWMRealtimePipeline	Camera actions

Two constraints in that doc that bear directly on MiniWorld:

1. "A causal DiT is not automatically a realtime session model. The pipeline must also register a realtime adapter and implement the WebSocket session lifecycle." — So even though MiniWorld is block-causal, it will not work in the realtime path until you write that adapter. This is likely the bulk of your integration work, ahead of any rollout question.
2. "For supported LingBot World deployments, the server-level …" plus "QVG KV-cache quantization currently supports only the LingBot realtime …" — the KV-cache features are currently pinned to LingBot, not generic.

Your plan ("implement the MiniWorld model, action conditioning, KV-cache logic and pipeline ourselves", using LingBot-World as the starting point) matches the doc's own framing: LingBot-World is the reference because it is the one pipeline that implements both the causal-DiT internals and the session adapter. That's a reasonable base, but note you'd be reproducing both halves.

Why rollout is a genuinely separate mechanism

The rollout/post-training contract has requirements this session API doesn't express:

• stochastic rollout log-probs — the session path returns decoded video to a client; it has no notion of accumulating per-step log-probs for a policy-gradient objective,
• weight updates / weight-version bookkeeping — the AR side carries weight-version events for exactly this (truncate_weight_version_events appears in the request-retraction path), and I see no equivalent in multimodal_gen,
• a /rollout/generate-style endpoint — not present in the diffusion server.

So the two paths differ in what is emitted (frames vs. differentiable/scored trajectories), not merely in transport. A WebSocket that streams latents can't be wired to Miles Diffusion without defining what a rollout step is for your model — that's a design question, not a plumbing question.

What I'd do, concretely

1. Split the problem in two and settle the realtime half first. Get MiniWorld running as a request-based causal pipeline before touching realtime sessions — that path needs no WebSocket adapter and gives you the model, action conditioning, and KV-cache logic in a testable form. Verify against the causal cache controls (realtime_causal_sink_size, realtime_causal_kv_cache_num_frames) since your rolling-window design maps onto those directly.
2. Then ask whether you actually need sessions or rollouts. If the goal is training-time rollouts, you may not need the realtime WebSocket path at all — you need a generate-and-score path, which is closer to the request-based API plus explicit log-prob capture.
3. Raise the rollout question upstream as a design discussion, not a bug. Since multimodal_gen has no rollout surface, the useful thing to ask maintainers is whether the post-training entry point is intended to live in the diffusion server or in the trainer (calling the server); that determines whether your work is "implement a pipeline" or "implement a pipeline and a new API".

Caveat

I verified the absence of a rollout path and the contents of the realtime docs against main (2026-09-25), and the two-mode table plus the causal-DiT-vs-realtime note are quoted directly. What I cannot see from here is whether Miles Diffusion has an out-of-tree client that drives the standard video API and computes log-probs itself — if so, that changes the answer materially, and it's worth checking on the Miles side before you commit to implementing a rollout endpoint. If you can say which repo is expected to own the rollout contract, I can look at the specific seam.