sidebar-title	Monitoring and Troubleshooting

Monitoring and Troubleshooting

AIPerf provides several tools for monitoring running benchmarks, diagnosing problems, and retrieving logs. This guide covers how to use each one.

Attaching to a Benchmark

The attach command connects to a running benchmark via port-forward and streams live progress updates via WebSocket:

aiperf kube attach

aiperf kube profile follows the run automatically after deploying unless you pass --detach, but in operator mode (the default) it polls the AIPerfJob CR rather than port-forwarding. Use attach to get the WebSocket progress stream, to reconnect after detaching, or to watch from a different terminal.

Attach to a specific job:

aiperf kube attach my-benchmark

Press Ctrl+C to detach. This only closes your local stream; the benchmark keeps running in the cluster. To stop the benchmark, run aiperf kube cancel <job>, which patches spec.cancel: true so the operator tears down the JobSet and stamps status.phase=Cancelled.

Listing Jobs

See all benchmark jobs and their status:

aiperf kube list

NAME               NAMESPACE          OWNER  PHASE      WORKERS  PROGRESS  THROUGHPUT  LATENCY   AGE
qwen3-benchmark    my-benchmarks      -      Running      10/10       67%    142.3 rps  318.7 ms  3m
llama-throughput   my-benchmarks      -      Completed    10/10      100%    137.9 rps  330.1 ms  15m
mistral-test       my-benchmarks      -      Failed         0/4         -            -         -  20m

WORKERS is ready/total, THROUGHPUT is requests per second, and LATENCY is the p99 request latency. A dash means the CR has not published that value. OWNER names the scoped operator holding that namespace's claim; - means the cluster-wide operator reconciles it and ? means the claim was unreadable.

Filter by Status

aiperf kube list --running
aiperf kube list --completed
aiperf kube list --failed

Wide Output

Show additional columns like model name, endpoint, and error messages:

aiperf kube list --wide

Live Refresh

Watch the list with automatic updates:

aiperf kube list --watch
aiperf kube list --watch --interval 10

Reading Logs

Get logs from all pods associated with a benchmark:

aiperf kube logs

Specific Job

aiperf kube logs my-benchmark

Specific Container

The controller pod runs control-plane (the SystemController) alongside per-service containers (dataset-manager, timing-manager, records-manager, api, event-bus-proxy, results-sidecar, and optionally gpu-telemetry-manager / server-metrics-manager). Worker pods run worker-group-manager. To see logs from a specific container:

aiperf kube logs --container control-plane

Follow Logs in Real-Time

aiperf kube logs -f

Last N Lines

aiperf kube logs --tail 100

Save Logs to Files

Save all pod logs to a directory (one file per pod):

aiperf kube logs --output ./my-logs

The closing line counts what actually reached disk (Saved logs for 7 of 8 pod(s) to ./my-logs/logs/), and any pod whose kubectl logs call failed gets its own warning with the exit code and stderr. When nothing was written the closing line is a warning, not a success.

Exit Codes in Scripts

attach and logs exit 1 when the benchmark you named does not exist, so they work as CI existence checks. A benchmark that exists but has no pods left (TTL-collected) still exits 0. Pass --ignore-not-found — same spelling and meaning as kubectl's — to force exit 0 for a missing benchmark in teardown scripts. The other addressing commands (cancel, delete, shutdown, debug, list) always exit 0; see the exit-code convention.

Debugging Failed Benchmarks

The debug command runs a one-shot diagnostic analysis of your deployment:

aiperf kube debug -n my-benchmarks

It inspects:

Pod states -- Identifies CrashLoopBackOff, ImagePullBackOff, ErrImagePull, OOMKilled, CreateContainerConfigError, RunContainerError, and Unschedulable, each with a suggested fix
Kubernetes events -- Shows the most recent warning events (or all recent events with --verbose)
Node resources -- Reports CPU, memory, and GPU allocatable vs. capacity for each node
Benchmark diagnostics -- When --job-id targets a specific job, runs the metric detectors in aiperf.kubernetes.benchmark_diagnosis over status.liveMetrics: high error rate, high tail latency (p99 above a multiple of the average), and a stalled job (Pending too long, or Running with neither throughput nor completed requests). Thresholds are the AIPERF_K8S_DIAGNOSIS_* environment variables. The section is omitted when nothing tripped.
Container logs -- With --verbose, fetches recent logs from problem pods

Debug a Specific Job

aiperf kube debug --job-id my-benchmark

Verbose Mode

Includes container logs from pods with problems:

aiperf kube debug --job-id my-benchmark --verbose

All Namespaces

Scan every namespace that has AIPerf deployments:

aiperf kube debug --all-namespaces

Sample Output

Diagnostic Report: my-benchmarks

POD                              STATUS    RESTARTS  NODE       ISSUES
aiperf-bench-controller-0-0      Running   0         node-1     0
aiperf-bench-workers-0-0         Running   3         node-2     1

Problems Found
[aiperf-bench-workers-0-0] OOMKilled (previous) (container: worker-0)
  Suggestion: Container was killed due to out-of-memory. Increase memory limits.

Node Resources
NODE      READY  CPU      MEMORY      GPU  PRESSURE
node-1    Yes    8/16     32Gi/64Gi   2/4  -
node-2    Yes    4/8      8Gi/16Gi    1/2  MemoryPressure

Summary
Pods: 2 total, 2 running, 1 with issues
Warning events: 3
Nodes under pressure: node-2

Pre-Flight Checks

Before deploying, validate that the cluster is ready:

aiperf kube preflight

It runs these checks in order, and stops early only if cluster connectivity fails:

Cluster connectivity and Kubernetes version
Namespace exists (AIPerf never creates it)
RBAC permissions in the target namespace
JobSet CRD installed and JobSet controller running
Resource quotas and node resources
Referenced secrets and image-pull access
Network policies and DNS resolution
Endpoint connectivity (when --endpoint-url is given)

With Specific Parameters

aiperf kube preflight \
  --image nvcr.io/nvidia/aiperf:latest \
  --endpoint-url http://my-server:8000 \
  --workers 20 \
  --namespace my-benchmarks

JSON Output

For CI/CD pipelines:

aiperf kube preflight -o json

Returns a structured JSON object with pass/fail/warn status for each check, suitable for automated gating.

Retrieving Results

After a benchmark completes, get the results:

aiperf kube results

By default this downloads the full results package from the operator's PVC storage (use --from-pods to pull from the benchmark pods instead). The results include:

profile_export_aiperf.json -- Summary metrics
profile_export.jsonl -- Per-request timing data
Server metrics and other exported files

From the Operator Storage

Even after pods are deleted, results are stored on the operator's PVC. This is the default:

aiperf kube results my-benchmark

Summary Only

Download only the summary results (faster):

aiperf kube results --summary-only

Direct From Pods

If you want to fetch results directly from the running benchmark pods (via the controller API):

aiperf kube results --from-pods

The default --all path uses the controller API and nothing else — if that call fails, the download fails. The kubectl cp fallback applies only to --summary-only, which tries the API first and then copies from the control-plane container.

Shut Down After Download

Free up cluster resources by shutting down the API service after downloading (only takes effect with --from-pods):

aiperf kube results --from-pods --shutdown

Save to Custom Directory

aiperf kube results --output ./my-results

Common Issues and Solutions

Expired Local Kubernetes Login

When an interactive aiperf kube command receives HTTP 401 or a recognized kubectl logged-out response, it pauses and reloads the selected kubeconfig until your normal credential provider works again. Complete the usual login (for example, your OIDC, cloud, or access-proxy login) in another terminal. AIPerf does not launch the provider or request credentials itself. Press Ctrl+C to stop waiting.

This behavior is limited to terminals with interactive input and output. Operator pods, in-cluster service accounts, redirected output, and CI fail immediately so unattended work cannot hang. HTTP 403 is also returned immediately because it means the authenticated identity lacks RBAC permission, not that the login expired. Missing credential-provider executables, malformed plugins, TLS failures, and unreachable API servers remain ordinary errors.

Pods Stuck in Pending

Symptom: aiperf kube list shows Pending and pods never start.

Diagnosis:

aiperf kube debug --job-id my-benchmark

Common causes:

Insufficient resources (CPU, memory, GPU) -- check node capacity in the debug output
Missing node selectors or tolerations -- pods may be targeting nodes that don't exist
Kueue quota exhausted -- check your ClusterQueue capacity

In operator mode, inspect the durable startup diagnosis directly:

kubectl get aiperfjob my-benchmark \
  -o jsonpath='{.status.startupIssue}{"\n"}{.status.conditions[?(@.type=="WorkersReady")]}{"\n"}'

Temporary capacity shortages, pending PVC binding, Kueue admission, and unknown scheduler reasons remain retryable even when timeoutSeconds: 0. Stable image, container-configuration, crash-loop, node-selector, untolerated-taint, and volume-affinity blockers fail only after the critical startup grace period. Tune warning and critical thresholds with AIPERF_K8S_WATCHDOG_PENDING_THRESHOLD_SECONDS and AIPERF_K8S_WATCHDOG_PENDING_CRITICAL_THRESHOLD_SECONDS on the operator container.

ImagePullBackOff

Symptom: Pods fail with ImagePullBackOff.

Fix: Verify the image exists and pull secrets are configured:

# Check preflight
aiperf kube preflight --image your-image:tag

# Add pull secrets
aiperf kube profile ... --image-pull-secrets my-registry-secret

OOMKilled

Symptom: Worker pods restart with OOMKilled status.

Fix: Reduce concurrency per worker pod so each pod uses less memory:

spec:
  connectionsPerWorker: 50    # reduce from default 100

spec.connectionsPerWorker is immutable after creation, so changing it means creating a new AIPerfJob.

The per-pod memory budget is resolved by the process that renders the JobSet, so it is set on the operator container rather than in the CR. The chart has no values key for it; patch the Deployment directly:

kubectl set env -n aiperf-system deploy/aiperf-operator \
  AIPERF_K8S_WORKER_POD_MEMORY=8Gi

Per job, spec.resourceMode selects the QoS shape: burstable (the default) sets requests without limits so containers are not cgroup-OOM-killed for exceeding their request, guaranteed sets requests == limits, and none omits both. It is also immutable after creation.

Benchmark Timeout

Symptom: Job transitions to Failed with timeout error.

Fix: Increase or disable the timeout:

spec:
  timeoutSeconds: 3600    # 1 hour
  # or
  timeoutSeconds: 0       # no timeout

Endpoint Unreachable

Symptom: Operator reports endpoint health check failure.

Diagnosis: Verify the Dynamo frontend is reachable from inside the cluster:

kubectl run curl-test --rm -it --image=curlimages/curl -- \
  curl -s http://dynamo-agg-frontend.dynamo-server.svc:8000/v1/models

Fix: Ensure the Dynamo deployment is healthy (kubectl get pods -n dynamo-server), and the URL in your config uses the correct frontend service DNS name: http://{deploy-name}-frontend.{namespace}.svc:8000/v1.

Stale Namespaces

The CLI-side watchdog that runs while aiperf kube profile follows a benchmark warns when it finds more than two leftover aiperf-* namespaces. Remove the finished benchmarks inside one with aiperf kube cleanup, or delete the namespace outright:

aiperf kube cleanup --namespace aiperf-bench-old --force
kubectl delete namespace aiperf-bench-old

Web Dashboard

The AIPerf operator includes a built-in web dashboard for comprehensive monitoring and analysis of your benchmarks.

Access it by port-forwarding to the operator:

kubectl port-forward -n aiperf-system deploy/aiperf-operator 8081:8081

Then open http://localhost:8081 in your browser.

Dashboard Features

Dashboard Tab -- Overview with KPI cards, active jobs count, and throughput trends across all jobs
Jobs Tab -- Sortable table of all benchmark jobs with phase filters (Running, Completed, Failed)
Job Detail Page -- Live metrics, charts, phase progress bar, and pod status for a single job
Sweeps Tab -- Table of AIPerfSweeps with per-sweep drill-down (variation curves, Pareto, children)
Leaderboard Tab -- Rank benchmark runs by any metric (throughput, latency percentiles, etc.)
Compare Tab -- Side-by-side comparison of multiple jobs to identify performance differences
History Tab -- Time-series charts showing how metrics evolve across all your benchmark runs

See Web Dashboard for the full page-by-page reference.

Quick Navigation

Use Ctrl+K to open the command palette and quickly jump to any job or page. Search by job name or view recent benchmarks.

Operator Prometheus Metrics

The kopf operator container exposes a Prometheus /metrics endpoint from an in-process daemon thread (src/aiperf/operator/metrics.py). The port is AIPERF_METRICS_PORT (OperatorEnvironment.METRICS_PORT, default 9090; set to 0 to disable). This is separate from the results-server on 8081.

kubectl port-forward -n aiperf-system deploy/aiperf-operator 9090:9090
curl http://localhost:9090/metrics

Exposed series:

Metric	Type	Labels	Meaning
`aiperf_operator_handler_duration_seconds`	Histogram	`handler`	Wall-clock duration of each instrumented kopf reconcile handler.
`aiperf_operator_handler_total`	Counter	`handler`, `outcome`	Reconcile-handler invocations by outcome.
`aiperf_operator_completion_claim_races_total`	Counter	—	Lost `try_claim_completion` races (concurrent ticks contending for the completion claim).

outcome is one of four values: success (returned normally), retry (raised kopf.TemporaryError, so kopf will re-dispatch), fatal (raised kopf.PermanentError, so kopf stops retrying and the CR is stuck), and error (anything else, including CancelledError, KeyboardInterrupt, and SystemExit). retry and fatal are separated so you can alert on stuck CRs without false positives from transient apiserver hiccups.

Only kopf reconcile handlers are instrumented (via the @track_handler("name") decorator); helper functions are not.

Benchmark Metrics from the Controller Pod

Separately from the operator's own reconcile metrics, each benchmark's controller pod serves its live benchmark metrics in Prometheus exposition format from the api container at /metrics on the API port (default 9090). The controller pod is annotated with prometheus.io/scrape: "true", prometheus.io/port, and prometheus.io/path: /metrics, so an annotation-based Prometheus scrape config picks it up without extra configuration.

Set serviceMonitor.enabled=true in the chart to have a Prometheus Operator ServiceMonitor scrape the Service's metrics port at /metrics. It is off by default, and is skipped when operator.metrics.port is 0 or when the monitoring.coreos.com/v1 CRDs are absent. The repository ships no Grafana dashboards or PrometheusRule alerts.

FilesExpand file tree

monitoring.md

Latest commit

History