Locust-based load tests for the trustify REST API, mirroring the endpoint coverage from trustify-scale-testing (Goose/Rust).

• Python 3.13+
• uv
• A running trustify instance
• OIDC credentials (unless running with AUTH_DISABLED=true)

A Makefile provides shortcuts for common test runs. All commands assume you are in the tools/perf/ directory.

make help                    # list all targets
make test                    # daily dashboard endpoint set, 10 users, 5 min
make test-v3                 # all v3 endpoint groups
make test-readonly           # read-only v3 tests (safe for prod)
make test-sbom               # v3 SBOM endpoint only
make ui                      # launch interactive web UI

Reports are written to reports/ with auto-generated filenames that encode what was run:

reports/v3-readonly_u10_5m_20260729T143022.html
reports/v3-sbom_u20_10m_20260729T150000.html
reports/v3-all_u10_5m_20260729T160000.html

Format: reports/<target>_u<users>_<duration>_<timestamp>.html

make test selects the endpoint groups represented in the daily dashboard: advisory, analysis, group, importer, advisory-label reads, license, pURL, SBOM, and vulnerability. It uses the daily workflow's pinned scenario file so scenario-based detail and recommendation tasks are enabled. Use make test-v3 for the additional product, organization, weakness, miscellaneous, and website workloads. Label mutation tasks remain excluded because the perf client lacks update permission.

make test HOST=https://my-server:8443 USERS=20 DURATION=10m
make test-readonly SCENARIO_FILE=etc/scenarios/main/full-20260412.json5

Run make clean to remove all generated reports and cached files.

CI runs use the repeatable stack from trustify-scale-test-runs: the PostgreSQL/storage snapshot 20260317T023702Z is restored, migrations are applied, the local OIDC client is configured, and this directory's Locust scenarios are run with uv.

The combined dashboard is at trustify-scale-test-runs/perf. The runner and Pages deployment are handled by .github/workflows/trustify-perf.yaml in the scale-test-runs repository. It stores raw runs and generated reports under publish/perf/pr/ and publish/perf/daily/, commits them to that repository's main branch, and deploys the publish/ site to GitHub Pages.

To trigger /perf-test, the trustify repository needs a GH_PAT Actions secret that can dispatch workflows in trustify-scale-test-runs. The command is restricted to commenters with write, maintain, or admin permission.

Each run includes a metadata.json file with the snapshot, scenario, branch, commit, workflow URL, and (for PR runs) pull request URL. Individual reports show this metadata and link back to the source run.

locust_report.py turns Locust CSV output into self-contained HTML without external dependencies. It supports both individual reports and a multi-run dashboard:

python3 locust_report.py report results -o report.html \
  --title "Trustify load test" --target http://localhost:8080

python3 locust_report.py index runs -o site \
  --title "Trustify load test dashboard" --prune

The index command recursively discovers *_stats.csv files, reads optional metadata.json files beside each run, compares each run with its predecessor, and caches parsed summaries. Use --force to rebuild all reports or --index-only to update only the dashboard page.

For full control, invoke uv run locust directly. uv run handles the virtual environment and dependency installation automatically.

uv run locust --host http://localhost:8080

Open http://localhost:8089 in your browser, set user count and spawn rate, then start the test.

uv run locust --host http://localhost:8080 -u 10 -r 2 -t 5m --headless

By default each user waits 1-3 seconds between requests. Override with environment variables:

# Max throughput (no delay)
TOOLS_PERF_WAIT_TIME_FROM=0 TOOLS_PERF_WAIT_TIME_TO=0 uv run locust --host http://localhost:8080 -u 10 -t 1m --headless

# Simulate slower users
TOOLS_PERF_WAIT_TIME_FROM=5 TOOLS_PERF_WAIT_TIME_TO=15 uv run locust --host http://localhost:8080 -u 10

uv run locust --host http://localhost:8080 -u 10 -t 5m --headless --html=report.html

uv run locust --host http://localhost:8080 -u 10 -t 5m --headless --csv=results

This produces results_stats.csv, results_stats_history.csv, results_failures.csv, and results_exceptions.csv.

Every Makefile target generates both an HTML report and matching CSV files in reports/. Use the compare targets to diff two runs:

# Aggregate summary (default: compares the two most recent runs)
make compare

# Per-endpoint breakdown
make compare-detail

# CSV output (for spreadsheets)
make compare-csv > comparison.csv

# Compare specific runs
make compare \
  A=reports/v3-readonly_u10_5m_20260729T071219_stats.csv \
  B=reports/v3-readonly_u10_5m_20260729T071311_stats.csv

# Custom regression threshold (default: 20%)
make compare THRESHOLD=10

When run without A and B, all compare targets auto-detect the two most recent _stats.csv files in reports/ — the second-newest as baseline, the newest as the current run.

make compare shows an aggregate summary table plus the top regressions and improvements:

| Metric               | Baseline | Current | Delta  |
|----------------------|----------|---------|--------|
| Total requests       | 43,849   | 31,661  | -27.8% |
| Requests/s           | 1093.0   | 1019.1  | -6.8%  |
| Avg response (ms)    | 5.6      | 6.0     | +7.7%  |
| p95 response (ms)    | 1        | 1       | 0.0%   |
| p99 response (ms)    | 2        | 2       | 0.0%   |
| Unique endpoints     | 31       | 34      | 15 new, 12 removed |

9 regressions, 3 improvements across 19 common endpoints (p95 threshold: 20%)

Top regressions (by p95 delta):
| Endpoint                                  | p95 old | p95 new | Delta   |
|-------------------------------------------|---------|---------|---------|
| GET /api/v3/group/sbom?parents=...        | 1000    | 2700    | +170.0% |
| GET /api/v3/license?total=true            | 1200    | 2000    | +66.7%  |

make compare-detail shows the full per-endpoint table with every endpoint's old/new values and REGRESSION/IMPROVED/NEW/REMOVED flags.

The comparison script can also be invoked directly:

uv run python compare.py reports/old_stats.csv reports/new_stats.csv
uv run python compare.py reports/old_stats.csv reports/new_stats.csv --detail
uv run python compare.py reports/old_stats.csv reports/new_stats.csv --format csv
uv run python compare.py reports/old_stats.csv reports/new_stats.csv --threshold 10

By default, all requests send Accept-Encoding: gzip, deflate, br to match real browser/client behaviour. Responses are decompressed transparently by the HTTP client, so reported response times include server-side compression cost and reflect realistic end-to-end latency.

To disable compression and measure raw uncompressed transfer cost:

TOOLS_PERF_NO_COMPRESSION=1 make test-readonly

This is useful for isolating query/serialization time from gzip overhead, or for comparing compressed vs uncompressed response sizes.

Every task is tagged with either readonly or mutate to indicate whether it modifies server state.

Read-only tasks (85 of 94) use only GET requests and are safe to run against production instances or read-only database replicas.

Mutate tasks (9 of 94) use POST, PUT, or PATCH and modify data (create analysis jobs, update labels, extract PURLs).

# Run only read-only tests
uv run locust --host http://localhost:8080 -u 10 --tags readonly

# Run everything except mutations
uv run locust --host http://localhost:8080 -u 10 --exclude-tags mutate

# Or use the Makefile shortcut
make test-readonly

By default, only v3 endpoints are tested. Use the TOOLS_PERF_API_VERSION environment variable to switch:

# v3 only (default)
uv run locust --host http://localhost:8080 -u 10

# v2 only
TOOLS_PERF_API_VERSION=v2 uv run locust --host http://localhost:8080 -u 10

# Both v2 and v3
TOOLS_PERF_API_VERSION=all uv run locust --host http://localhost:8080 -u 10

When TOOLS_PERF_API_VERSION=all, you can also filter by version tag:

TOOLS_PERF_API_VERSION=all uv run locust --host http://localhost:8080 -u 10 --tags v3
TOOLS_PERF_API_VERSION=all uv run locust --host http://localhost:8080 -u 10 --tags v2 advisory

Tests are organized into user classes with weights that control how frequently Locust assigns simulated users to each class. v3 tests are split one file per endpoint group:

Pass class names as positional arguments:

uv run locust AdvisoryUserV3 --host http://localhost:8080 -u 10
uv run locust AnalysisUserV2 --host http://localhost:8080 -u 10
uv run locust --host http://localhost:8080 -u 10 --class-picker   # choose in web UI

Every task is tagged with its API version (v2 or v3), endpoint group (e.g. advisory, sbom), access pattern (list, detail, slow), and access type (readonly or mutate).

uv run locust --host http://localhost:8080 -u 10 --tags advisory
uv run locust --host http://localhost:8080 -u 10 --tags sbom detail
uv run locust --host http://localhost:8080 -u 10 --exclude-tags slow mutate

By default, the perf tool authenticates against the trustify instance using OIDC client credentials. Set the following environment variables:

The tool performs OIDC discovery (/.well-known/openid-configuration), acquires a token via the client_credentials grant, and injects it as Authorization: Bearer <token> on every request. Tokens are cached and refreshed automatically before expiry.

export ISSUER_URL=https://sso.example.com/realms/trustify
export CLIENT_ID=testing
export CLIENT_SECRET=s3cret
make test HOST=https://trustify.example.com

AUTH_DISABLED=true make test

Set both wait time variables to 0 for max throughput (no delay between requests).

Some tests need pre-computed IDs (specific SBOMs, advisories, PURLs, etc.) to hit detail endpoints. These come from a JSON5 scenario file, set via the TOOLS_PERF_SCENARIO_FILE environment variable:

TOOLS_PERF_SCENARIO_FILE=etc/scenarios/empty.json5 uv run locust --host http://localhost:8080 -u 10

Without a scenario file, only the list/static endpoints run. The included etc/scenarios/empty.json5 has all fields set to null (everything disabled). To enable detail tests, copy it and fill in real IDs from your database:

{
  "get_sbom": "sha256:abc123...",
  "get_vulnerability": "CVE-2024-1234",
  "get_advisory": "f1e5eb17-2f31-4...",
  // ... see etc/scenarios/empty.json5 for all fields
}

Pre-built scenario files are organized under etc/scenarios/:

# Main branch scenarios (for use with the DS3/DS4 datasets)
TOOLS_PERF_SCENARIO_FILE=etc/scenarios/main/full-20260412.json5 make test

# Release-specific scenarios
TOOLS_PERF_SCENARIO_FILE=etc/scenarios/releases/0.4.z/full-20260412_atlas.json5 make test-analysis-v2

Open the relevant file in users/v3/ and add a new method with the @task decorator. Always include the readonly or mutate tag:

# users/v3/advisory.py

@tag("v3", "advisory", "list", "readonly")
@task
def list_advisory_by_severity(self) -> None:
    self.client.get(
        "/api/v3/advisory?q=severity=critical",
        name="/api/v3/advisory?q=severity=critical",
    )

The name parameter controls how the endpoint appears in reports. Use the stable route path for each request: retain query strings when they distinguish benchmark cases, and use {parameter} placeholders for dynamic path or query values. Locust reports the HTTP method separately, so don't include task names, IDs, or batch sizes in name. For example, name a component lookup /api/v3/analysis/component/{component} rather than get_analysis_component[...].

If the test needs a pre-computed ID, read it from SCENARIO and return early when it is None:

from scenario import SCENARIO

@tag("v3", "sbom", "detail", "readonly")
@task
def get_sbom_something(self) -> None:
    if not SCENARIO.get_sbom:
        return
    key = SCENARIO.get_sbom
    self.client.get(
        f"/api/v3/sbom/{quote(key, safe='')}/something",
        name=f"get_sbom_something[{key[:16]}...]",
    )

If the test needs a new scenario field, add it to the Scenario dataclass in scenario.py:

@dataclass(frozen=True)
class Scenario:
    # ... existing fields ...
    my_new_field: str | None = None

Then add the field to etc/empty.json5 (and any other scenario files).

Create a new file in users/v3/:

# users/v3/my_feature.py

from locust import tag, task
from config import WAIT_TIME
from users.base import AuthenticatedHttpUser

class MyFeatureUserV3(AuthenticatedHttpUser):
    weight = 2
    wait_time = WAIT_TIME

    @tag("v3", "my_feature", "readonly")
    @task
    def do_something(self) -> None:
        self.client.get("/api/v3/something", name="/api/v3/something")

Then import it in locustfile.py under the appropriate TOOLS_PERF_API_VERSION block:

if TOOLS_PERF_API_VERSION in ("v3", "all"):
    from users.v3.my_feature import MyFeatureUserV3  # noqa: F401

If the class has only readonly tasks, add it to V3_READONLY in the Makefile so make test-readonly picks it up. If it has only mutate tasks, do not add it (Locust errors on user classes with zero runnable tasks after tag filtering).

Use @task(N) to make a task run N times more often than @task (which defaults to 1):

@task(3)   # runs 3x as often as @task(1) tasks in the same class
def hot_endpoint(self) -> None:
    self.client.get("/api/v3/sbom", name="/api/v3/sbom")

@tag("v3", "sbom", "mutate")
@task
def create_something(self) -> None:
    self.client.post(
        "/api/v3/something",
        json={"key": "value"},
        name="create_something",
    )

Use catch_response=True to mark requests as pass/fail based on content:

@task
def validated_get(self) -> None:
    with self.client.get(
        "/api/v3/advisory",
        name="/api/v3/advisory",
        catch_response=True,
    ) as resp:
        if resp.status_code != 200:
            resp.failure(f"status {resp.status_code}")
        elif not resp.json().get("items"):
            resp.failure("empty result set")

tools/perf/
├── Makefile             # Test targets with auto-named reports
├── pyproject.toml       # Dependencies (locust, json5)
├── locustfile.py        # Entry point -- API version dispatch
├── compare.py           # Compare two _stats.csv runs (markdown/csv)
├── auth.py              # OIDC token provider (client_credentials)
├── config.py            # Shared wait time configuration
├── scenario.py          # Scenario data loader (JSON5)
├── reports/             # Auto-generated HTML + CSV reports (gitignored)
├── etc/
│   └── scenarios/
│       ├── empty.json5                          # Empty scenario (all fields null)
│       ├── main/                                # Main-branch scenarios
│       │   ├── full-20250323.json5
│       │   ├── full-20250604.json5
│       │   ├── full-20260317T023702Z.json5
│       │   └── full-20260412.json5              # Latest full scenario
│       └── releases/
│           └── 0.4.z/
│               ├── full-20260412_atlas.json5    # Atlas analysis-only
│               └── full-20260412_qe_atlas.json5 # QE Atlas analysis-only
└── users/
    ├── __init__.py
    ├── base.py              # AuthenticatedHttpUser (OIDC + compression)
    ├── website.py           # WebsiteUser (version-agnostic)
    ├── v3/
    │   ├── __init__.py
    │   ├── advisory.py      # AdvisoryUserV3 (weight 2)
    │   ├── analysis.py      # AnalysisUserV3 (weight 3)
    │   ├── group.py         # GroupUserV3 (weight 1)
    │   ├── importer.py      # ImporterUserV3 (weight 1)
    │   ├── labels.py        # AdvisoryLabelUserV3 (2) + SBOMLabelUserV3 (2)
    │   ├── license.py       # LicenseUserV3 (weight 1, incl. slow)
    │   ├── misc.py          # MiscUserV3 (weight 1)
    │   ├── organization.py  # OrganizationUserV3 (weight 1)
    │   ├── product.py       # ProductUserV3 (weight 1)
    │   ├── purl.py          # PurlUserV3 (weight 2, incl. slow)
    │   ├── sbom.py          # SBOMUserV3 (weight 2, incl. slow)
    │   ├── vulnerability.py # VulnerabilityUserV3 (weight 2)
    │   └── weakness.py      # WeaknessUserV3 (weight 1)
    └── v2/
        ├── __init__.py
        ├── rest_api.py      # RestAPIUserV2 (weight 10)
        ├── rest_api_slow.py # RestAPIUserSlowV2 (weight 1)
        └── analysis.py      # AnalysisUserV2 (weight 1)