feat: YAML-native v2 config + adaptive sweep orchestrator with BO & search recipes

[ajcasagrande]

Summary

• Sweep orchestrator port with a new pluggable executor (orchestrator/executor.py, local_executor.py) and a search_planner/ package: Bayesian (BoTorch GP/DSP kernel), Optuna (multi-objective), Monotonic, and Smooth-Isotonic planners, with shared helpers for cliff detection, margin normalization, replicate budget, pooled percentile, and SLA constraints. Aggregation gains SLA filtering and multi-objective search-history export.
• YAML-driven AIPerf config (src/aiperf/config/) replaces the old common/config/ package. Layered design: typed models, loader (Jinja2 templating, env-var interpolation, dotted-path overrides, duration parsing, strict-undefined plan), flags converter (CLI ↔ YAML), resolution layer, sweep DSL (grid, QMC/Sobol, adaptive, multi-run, distributions), public JSON schema, plus a bundled library of 20+ ready-to-run templates and reference trace data.
• Search recipes (src/aiperf/search_recipes/) with built-ins: max_concurrency_under_sla, max_goodput_under_slo, sla_breach_knee, itl_surface_fit, ttft_curve_fit, and Pareto sweep (axes, dominance, export, parser) plus post-process hooks.
• CLI runner refactored into _cli_runner_{helpers,sweep_helpers,post_process}.py + _sweep_table_logger.py; new aiperf config command for template discovery and validation.
• Auto-plot envelope (plot/auto_plot.py) materializes the resolved plot config into the artifact dir so aiperf plot <dir> reproduces. PlotEnvelopeConfig lets one YAML own its visualization.
• Finite/NaN invariants: new common/finite.py (FiniteFloat, scrub_non_finite, nan_safe_mean/std, is_finite_value); property-test corpus with ratcheted baselines in tests/unit/property/.
• Mock-server scheduler for deterministic latency simulation plus robustness/scheduler test suites.
• Plugin schema extended with orchestrator categories (executor, planner, recipe).
• Docs: new tutorials (sweeps, adaptive search, auto-plot, inline datasets, YAML config, YAML distributions), dev docs (sweep orchestrator design, global invariants, YAML config roadmap), sweeping reference (Bayesian optimization, search recipes, space-filling), troubleshooting/sweeps, and API reference for search history. Regenerated CLI and env-vars docs.
• Tests: extensive new unit/component/integration coverage under tests/unit/{config,orchestrator,search_recipes,search_planner,cli_runner,property}/ and adversarial chaos scripts under tests/scripts/chaos/.

766 files changed (+111246, -24748). Full design write-up at docs/dev/sweep-orchestrator.md.

Architecture

One pipeline at every cardinality

A single benchmark, a multi-run for confidence intervals, a grid/scenarios sweep, a Sobol/LHS characterization, an adaptive BO search, and (coming soon) a cluster-distributed BO search are seven cardinalities of one pipeline. BenchmarkPlan describes what to run, MultiRunOrchestrator decides when and in what order, an optional SearchPlanner decides what to try next, and a RunExecutor decides how to actually run one cell.

%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'htmlLabels': true}, 'themeVariables': {'fontSize': '14px'}}}%%
flowchart LR
    cfg["**config**<br/>AIPerfConfig"]
    exp["**expand**<br/>into N variations<br/>(BenchmarkPlan)"]
    run["**run**<br/>each variation<br/>M trials<br/>(via RunExecutor)"]
    agg["**aggregate**<br/>SweepAnalyzer<br/>-> sweep_aggregate/"]

    cfg --> exp --> run --> agg

    subgraph BACKENDS["RunExecutor backend (swap point)"]
        local["LocalSubprocessExecutor<br/>(today)"]
        k8s["K8sChildJobExecutor<br/><i>coming soon</i>"]
    end
    run -. selects .-> local
    run -. selects .-> k8s

    classDef stage fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#0d47a1
    classDef shipping fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px,color:#1b5e20
    classDef coming fill:#fafafa,stroke:#9e9e9e,stroke-width:1.5px,stroke-dasharray:5 3,color:#616161

    class cfg,exp,run,agg stage
    class local shipping
    class k8s coming

    style BACKENDS fill:transparent,stroke:#78909c,stroke-width:2px,stroke-dasharray:2 2

Loading

Search recipes → AdaptiveSearchSweep → planner

A user can author an AdaptiveSearchSweep directly under sweep: (low level) or pick a search_recipe plugin (high level) that builds one from a recipe + the user's existing benchmark config.

%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'htmlLabels': true}, 'themeVariables': {'fontSize': '14px'}}}%%
flowchart TB
    subgraph IN["user inputs"]
        cli["**--search-recipe NAME --param k=v**<br/>or<br/>**sweep: { type: adaptive_search, … }** in YAML<br/>or<br/>--search-space PATH:LO,HI[:KIND]<br/>--search-metric METRIC<br/>--search-stat STAT<br/>--search-direction DIRECTION<br/>--search-sla metric:stat:op:threshold (×N)"]
        uc["**AIPerfConfig.benchmark**<br/><i>(models, endpoint, phases, …)</i>"]
    end

    subgraph RECIPE["recipe layer (optional)"]
        ctx["**SearchRecipeContext**<br/><i>(benchmark_config, sla_targets,<br/>sweep_overrides)</i>"]
        rc["**SearchRecipe** plugin (Protocol)<br/><i>built-ins:</i><br/>max-throughput-ttft-sla<br/>max-throughput-itl-sla<br/>concurrency-ramp<br/>prefill-ttft-curve / decode-itl-curve<br/>max-goodput-under-slo<br/>max-concurrency-under-sla<br/>pareto-sweep"]
        out["**SearchRecipeOutput**<br/><i>(exactly one of:<br/>adaptive_search | sweep_parameters | scenarios)</i><br/>+ sla_filters, slos, post_process"]
    end

    subgraph CFG["adaptive sweep variant"]
        asc["**AdaptiveSearchSweep**<br/><i>(SweepConfig variant,<br/>type=adaptive_search)</i><br/>search_space, objectives,<br/>max_iterations, sla_filters,<br/>post_process, planner, …"]
    end

    subgraph DRIVE["runtime drivers"]
        plan["**AIPerfConfig.sweep**<br/>= AdaptiveSearchSweep"]
        plan2["**BenchmarkPlan.sweep**<br/><i>(is_adaptive_search is true)</i>"]
        sp["**SearchPlanner** plugin<br/><i>(BayesianSearchPlanner |<br/>MonotonicSLASearchPlanner |<br/>SmoothIsotonicSLAPlanner |<br/>OptunaSearchPlanner)</i>"]
        pph["**search_recipe_post_process** plugin<br/><i>(degradation_knee_detect, ttft_curve_fit,<br/>itl_surface_fit, sla_breach_knee,<br/>pareto_sweep_export)</i>"]
    end

    cli --> rc
    uc --> ctx --> rc --> out --> asc
    cli -.direct path.-> asc

    asc --> plan
    plan --> plan2
    plan2 --> sp
    plan2 --> pph

    classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#0d47a1
    classDef proc fill:#fff3e0,stroke:#e65100,stroke-width:1.5px,color:#bf360c
    classDef decision fill:#f3e5f5,stroke:#6a1b9a,stroke-width:1.5px,color:#4a148c
    classDef art fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px,color:#1b5e20

    class cli,uc data
    class ctx,rc proc
    class out,asc art
    class plan,plan2,sp,pph decision

Loading

Adaptive search — propose → execute → record loop

The BO outer loop is a propose -> execute -> record cycle inside MultiRunOrchestrator.execute_adaptive_search. BenchmarkRun and RunExecutor are unchanged from the grid path; the difference is that BenchmarkPlan.configs starts with one seed config and grows by one per iteration as the planner asks for the next point.

%%{init: {'sequence': {'mirrorActors': false}, 'themeVariables': {'fontSize': '14px'}}}%%
sequenceDiagram
    autonumber
    participant Plan as BenchmarkPlan<br/>(sweep is AdaptiveSearchSweep)
    participant Orch as MultiRunOrchestrator
    participant Pl as SearchPlanner<br/>(Bayesian / Monotonic / Optuna)
    participant Run as BenchmarkRun
    participant Exec as RunExecutor
    participant Res as RunResult
    participant PP as PostProcessHandler
    participant Out as search_history.json /<br/>sweep_aggregate

    Orch->>Pl: planner instantiated upstream<br/>(via _build_search_planner)<br/>and passed into execute

    loop until converged or max_iterations
        Orch->>Pl: ask
        Pl-->>Orch: (BenchmarkConfig_k, SweepVariation_k)<br/>or None (converged -> convergence_reason)
        alt got proposal
            Orch->>Orch: _run_independent_cell<br/>(fresh ExecutionStrategy per cell)
            loop trials inner (until strategy says stop)
                Orch->>Run: BenchmarkRun for cfg_k, variation_k, trial t, …
                Orch->>Exec: run the cell
                Exec-->>Res: RunResult
            end
            Orch->>Pl: tell with variation_k, cell_results
            Pl->>Pl: filter by SLAFilter,<br/>compute objective scalar,<br/>plateau / patience / max-iter check
            Orch-->>Out: write_search_history<br/>(incremental, includes<br/>boundary_summary if planner has it)
        end
    end

    Orch->>PP: process the sweep_aggregate with params<br/>(per PostProcessSpec on sweep)
    PP-->>Out: knees, curve fits, …
    Orch-->>Out: profile_export_aiperf_sweep.{json,csv}

Loading

Test plan

• uv run pytest -n auto tests/unit/
• uv run pytest -n auto tests/unit/property/ (finite/numeric invariants)
• uv run pytest -n auto -m component_integration
• uv run pytest -n auto -m integration
• make validate-plugin-schemas
• make generate-all-docs is idempotent (pre-commit confirms)
• Smoke a few bundled templates: aiperf run -f src/aiperf/config/templates/minimal.yaml, latency_test.yaml, sweep_with_plot.yaml
• Confirm aiperf plot <artifact_dir> reproduces the auto-plot envelope for a sweep run
• Run a search recipe end-to-end (e.g. max_concurrency_under_sla) and verify search_history.jsonl export
• Verify Bayesian and Smooth-Isotonic planners against a known SLA-cliff workload on the mock server

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@c357b7c48b1ec429317782209820491f88ecf396

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@c357b7c48b1ec429317782209820491f88ecf396

---

Last updated for commit: c357b7c • Browse code

[github-actions]

Fern Docs Preview: https://nvidia-preview-fa7f7392-df72-4046-97b6-fdca492f9fef.docs.buildwithfern.com/aiperf/dev

[debermudez]

This is a thorough, well-structured architectural overhaul. The new MultiRunOrchestrator, v2 config system, and adaptive search planner are solid. The issues below are mostly low-severity cleanup items, with one medium-priority data-loss risk in the adaptive cancel path.

13 findings: 0 critical, 0 high, 1 medium, 9 low, 3 nit.

[debermudez]

Good to go once its rebased on main. Very nice work!

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.