Skip to content

fix(asr): fall back to "text" in cal_write_wer when gt_text_attr_name is missing - #16327

Open
MohammadHijjawi97 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
MohammadHijjawi97:fix-cal-write-wer-gt-fallback
Open

MohammadHijjawi97 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
MohammadHijjawi97:fix-cal-write-wer-gt-fallback

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Make cal_write_wer actually use its fallback to the text field when gt_text_attr_name is missing from the manifest, instead of silently returning None.

Collection: ASR

Changelog

  • In cal_write_wer, return None, None, eval_metric was indented at the level of the outer if, so it ran even after the code had switched gt_text_attr_name to "text". WER was then skipped with no log message whenever the configured attribute was missing, even when text was present. The early return now happens only when neither field exists, and a warning is logged when falling back to text.
  • Added tests/collections/asr/utils/test_eval_utils.py, covering normal WER, the fallback, and the no-ground-truth case.

Usage

from nemo.collections.asr.parts.utils.eval_utils import cal_write_wer

# manifest lines: {"text": "hello world", "pred_text": "hello word"}
out, res, _ = cal_write_wer(pred_manifest="pred.json", gt_text_attr_name="normalized_text")
# before: (None, None, 'wer'), no log message
# after:  res["wer"] == 0.5, with a warning that it fell back to "text"

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

Trusted PRs run automatically through copy-pr-bot. For an untrusted PR, a maintainer can trigger CI by commenting
/ok to test <head-sha>; repeat this after a new push if the PR remains untrusted.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Checks run (CPU, Python 3.13): pytest --noconftest tests/collections/asr/utils/test_eval_utils.py tests/collections/asr/inference/test_pipeline_eval.py gives 8 passed, and the fallback test fails on main. pre-commit run is clean.
  • cal_write_text_metric has the same indentation, but I left it unchanged on purpose. The streaming pipeline calls it for BLEU with gt_text_attr_name: answer and expects it to skip when the translation reference is missing. Falling back to text there would score translations against the source transcript.
  • No documentation change is needed: this restores the fallback the code already intends.
… is missing

cal_write_wer switches to the "text" field when the configured
gt_text_attr_name is absent from a manifest line, but the following
`return None, None, eval_metric` sat outside the else branch, so it
returned in both cases. As a result WER was silently skipped (without
the "Cannot calculate WER" log line) whenever the configured attribute
was missing, even if "text" was present.

Only return when neither field is present, and log a warning when
falling back to "text".

Signed-off-by: Mohammad Hijjawi <mohammad.hijjawi1997@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

1 participant