fix(rnnt): prevent deadlock in multi-GPU validation when validation batch is split

[mayuriphad]

Fixes #16003

What & Why
When \use_loss_wer=True\ and \�alidation_ds.batch_size > joint.fused_batch_size, \RNNTJoint.forward\ splits the validation batch into micro-batches and accumulates WER using \self.wer.update()\ and \self.wer.compute().

Previously, the _to_sync\ flag was only overridden to \False\ during training (\if self.training:). During validation, this meant \self.wer.compute()\ attempted to synchronize across GPUs for every sub-batch. Since different ranks can receive different global batch portions and split them into a different number of sub-batches, the synchronization counts would mismatch and cause NCCL watchdogs to hang in a deadlock.

How it was fixed
This patch unconditionally un-sets the _to_sync\ flag before \update()\ and \compute()\ inside the \orward()\ sub-batch loop (not just during training), and restores it after. The sub-batch loop calculates local \wer_num\ and \wer_denom\ and simply returns them to the caller. The global sync across ranks will then be safely deferred to the outer Lightning validation epoch end loop where batch splits don't affect sync counts.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.