Skip to content

chore(test): fix CI test selection for adversarial and recipe suites#957

Merged
ajcasagrande merged 8 commits into
mainfrom
ajc/fix-adversarial
May 19, 2026
Merged

chore(test): fix CI test selection for adversarial and recipe suites#957
ajcasagrande merged 8 commits into
mainfrom
ajc/fix-adversarial

Conversation

@ajcasagrande

@ajcasagrande ajcasagrande commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Mark component-integration adversarial DAG suites as stress while leaving unit adversarial tests in the default unit run.
  • Expand make test-stress so unit, component-integration, and integration stress tests are all reachable via separate pytest invocations.
  • Mark long-running search-recipe integration tests as slow so normal integration CI skips them.
  • Add a mock-server install verification step before integration CI runs.

Test plan

  • uv run pytest tests/unit/ --collect-only -m 'stress' -q
  • uv run pytest tests/component_integration/ --collect-only -m 'stress' -q
  • uv run pytest tests/integration/ --collect-only -m 'integration and stress' -q
  • uv run pytest tests/integration/test_search_recipes.py --collect-only -m 'integration and not performance and not ffmpeg and not stress and not slow' -q confirms search-recipe slow tests are deselected
  • Pre-commit hooks passed on committed changes
  • Let GitHub CI run the full unit and integration workflows

🤖 Generated with Claude Code

ajcasagrande and others added 3 commits May 18, 2026 21:49
Keep adversarial coverage available on demand while excluding these high-load edge-case suites from default pytest runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use the production AIPerf builder runner for the integration-test matrix instead of GitHub-hosted Ubuntu.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run integration CI on the production builder with the same setup, uv cache, and xdist worker cap used by unit tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@github-actions

github-actions Bot commented May 19, 2026

Copy link
Copy Markdown

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@b9063ad46d61dc025b1fd457c505bb00dfae26f1

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@b9063ad46d61dc025b1fd457c505bb00dfae26f1

Last updated for commit: b9063adBrowse code

@ajcasagrande ajcasagrande changed the title Align adversarial tests and integration CI May 19, 2026
@github-actions github-actions Bot added the chore label May 19, 2026
@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Integration test CI workflow updated to modernize tooling and add xdist worker capping; nineteen adversarial test modules marked with pytest.mark.stress to enable test categorization and selective execution.

Changes

CI Infrastructure and Test Stress Marking

Layer / File(s) Summary
Integration test workflow modernization
.github/workflows/run-integration-tests.yml
Workflow target OS changed to prod-aiperf-builder-amd-v1, GitHub Actions updated to v6 (checkout and setup-python), uv dependency manager added with caching via uv.lock, install command switched from make first-time-setup to make ci-install, and pytest-xdist worker count capped to 4 via PYTEST_XDIST_AUTO_NUM_WORKERS environment variable to prevent OOM-kills.
Adversarial test stress marking
tests/component_integration/timing/*, tests/unit/common/*, tests/unit/config/*, tests/unit/dataset/loader/*, tests/unit/endpoints/*, tests/unit/search_recipes/*, tests/unit/timing/*, tests/unit/transports/*
Module-level pytestmark = pytest.mark.stress added to 19 adversarial and stress test modules, enabling pytest test categorization and selective execution via the stress marker.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 The workflow hops with newer tools,
And pytest marks the stress-test schools,
With workers four and caching bright,
Our CI runs the tests just right! 🚀

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: marking adversarial and recipe test suites with a stress marker and updating CI test selection.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented May 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

ajcasagrande and others added 4 commits May 18, 2026 22:01
Restore unit adversarial tests to the default test selection while leaving component-integration adversarial stress markers intact.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Fail early when the mock server package or CLI entry point is missing, and tighten logger capture tests to target the modules that emit their records.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Undo the branch-specific integration workflow runner and setup changes while keeping the mock-server install guard in Makefile.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run unit, component-integration, and integration stress tests as separate pytest invocations so stress-marked tests remain reachable by Makefile targets.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande ajcasagrande changed the title chore(test): align adversarial tests and integration CI May 19, 2026
@ajcasagrande ajcasagrande changed the title Fix CI test selection for adversarial and recipe suites May 19, 2026
@ajcasagrande ajcasagrande enabled auto-merge (squash) May 19, 2026 05:29
@ajcasagrande ajcasagrande merged commit bb6f421 into main May 19, 2026
23 of 25 checks passed
@ajcasagrande ajcasagrande deleted the ajc/fix-adversarial branch May 19, 2026 05:30
ajcasagrande added a commit that referenced this pull request May 19, 2026
…957)

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 participants