Skip to content

aws-sigv4-auth.md

Latest commit

 

History

History
321 lines (241 loc) · 11.3 KB

aws-sigv4-auth.md

File metadata and controls

321 lines (241 loc) · 11.3 KB
Raw
sidebar-title AWS SigV4 Authentication

Benchmarking AWS Endpoints

This guide walks you through benchmarking inference endpoints protected by AWS IAM authentication. AIPerf signs every request with your AWS credentials automatically -- you just need to tell it your AWS region and service name.

--auth-type sigv4 signs every HTTP request AIPerf sends and works for any SigV4-protected endpoint.

What's Supported

Scenario Non-Streaming Streaming Notes
API Gateway + vLLM/TGI/NIM Yes Yes Standard HTTP + SSE
SageMaker + vLLM/LMI container Yes No Non-streaming only. SageMaker uses proprietary event framing instead of SSE.
Bedrock Converse / InvokeModel No No Different request/response schema -- not OpenAI-compatible

Endpoints that send multipart/form-data (image_edit, video_generation) are not supported: signing a multipart body is not implemented, so AIPerf rejects --auth-type sigv4 for them at startup rather than sending unauthenticated requests. --aws-region and --aws-service are both required with --auth-type sigv4, and the --aws-* flags are rejected if --auth-type sigv4 is not set, since they would otherwise be silently ignored.

Before You Start

  1. Install the AWS extra (this pulls in botocore for credential handling):
uv pip install 'aiperf[aws]'
  1. Make sure your AWS credentials are working:
aws sts get-caller-identity

If that prints your account and role info, you're good to go. If not, see Setting Up Credentials.

Quick Start

The key flags are --auth-type sigv4, --aws-region, and --aws-service. Add these to any aiperf profile command and AIPerf will sign every request automatically.

API Gateway with IAM Auth

Your API Gateway fronts an OpenAI-compatible server and has IAM authorization enabled. Both streaming and non-streaming work:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

If your API Gateway maps a custom path to the backend, use --endpoint to set it:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com \
    --endpoint /prod/inference/v1/chat/completions \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

SageMaker with vLLM or LMI (Non-Streaming)

SageMaker endpoints running vLLM or DJL LMI containers accept OpenAI-format request bodies through the /invocations path. The response body is passed through unchanged, so non-streaming works. Use --endpoint to set the SageMaker invocation path:

aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-east-1.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service sagemaker \
    --request-count 100

Streaming is not supported for SageMaker endpoints because SageMaker uses a proprietary event stream format instead of SSE. Do not pass --streaming with SageMaker.

Figuring Out Your Region and Service Name

The --aws-region should match the region in your endpoint URL:

https://abc123.execute-api.us-east-1.amazonaws.com/...
                           ^^^^^^^^^
                           this is your --aws-region

The --aws-service depends on which AWS service handles your traffic:

If your traffic goes through... Use --aws-service
API Gateway execute-api
SageMaker Runtime sagemaker

A common gotcha: the service name isn't always what you'd guess. For example, it's sagemaker not sagemaker-runtime. If you get a "SignatureDoesNotMatch" error, the service name is the first thing to double-check.

Setting Up Credentials

If aws sts get-caller-identity already works, you can skip this section -- AIPerf will pick up the same credentials automatically.

Environment Variables (simplest)

Good for quick local testing:

export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="wJal..."
export AWS_SESSION_TOKEN="FwoG..."  # only if using temporary credentials

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 100

Named Profiles (multiple accounts)

If you work with more than one AWS account, you probably already have profiles set up in ~/.aws/credentials. Point AIPerf at the right one with --aws-profile:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-west-2.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service execute-api \
    --aws-profile staging \
    --request-count 100

Without --aws-profile, AIPerf uses whichever credentials the AWS CLI would use by default (environment variables first, then [default] profile, then IAM roles).

SSO

If your team uses AWS IAM Identity Center (SSO), log in first, then pass the profile:

aws sso login --profile my-sso-profile

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --aws-profile my-sso-profile \
    --request-count 100

Kubernetes (EKS)

On EKS, credentials are typically injected into your pod automatically via IRSA or Pod Identity. You don't need --aws-profile -- just make sure your pod's service account has the right IAM role attached:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 1000

One thing to watch: if your pod has AWS_ACCESS_KEY_ID set as an environment variable (e.g., from a Kubernetes Secret), that takes priority over IRSA/Pod Identity. If you're hitting the wrong account, check for stale env vars.

Long-Running Benchmarks

AIPerf refreshes AWS credentials automatically before each request. This means temporary credentials (from SSO, assumed roles, or IRSA) won't expire mid-benchmark. If you're running a long benchmark with thousands of requests, you don't need to do anything special.

The one exception: if your SSO session itself expires (they typically last 8-12 hours), you'll need to re-run aws sso login and restart the benchmark.

Requests Other Than the Benchmark Itself

A benchmark run makes more than just inference requests, and all of them are signed with the same credential chain, region, and service:

Request When it fires Flags
Readiness probe (/v1/models, then a minimal inference request) Before the run starts --wait-for-model-mode, --wait-for-model-timeout, --wait-for-model-interval
KV-cache reset hook Before each run --reset-kv-cache, --reset-kv-cache-path
Server profiler start/stop hooks At profiling phase start and end --server-profiler, --server-profiler-start-path, --server-profiler-stop-path

Two details matter behind an IAM-protected endpoint:

  • Each attempt is signed fresh. A SigV4 signature embeds x-amz-date and AWS rejects it outside a five-minute skew window. Readiness polling and control-hook retry backoff both routinely run longer than that, so signing once per URL would start failing partway through the wait.
  • --api-key is suppressed on these paths when --auth-type sigv4 is set, exactly as it is on the request path. The signer owns the Authorization header; a stray Authorization: Bearer ... or x-api-key would overwrite or conflict with it.

If the probe were unsigned, API Gateway would answer 403, and the readiness rule treats any status below 500 as "the server is up" -- so the run would start against an endpoint that rejects every request. Signing the probe closes that gap: when a signer is configured, a 401/403 response is never treated as "ready". Preflight stops there and reports the likely causes, because none of them are fixed by retrying.

A rejection is not proof that the signature is wrong. An IAM policy denial, a WAF rule, or a path the endpoint does not route all return 401/403 on a perfectly valid signature -- API Gateway in particular answers an unrouted path with 403 Missing Authentication Token, which AIPerf names explicitly so you check the URL rather than the credentials.

Examples

High-Throughput API Gateway with Warmup

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-rate 50 \
    --request-count 1000 \
    --warmup-request-count 20

Multiple API Gateway Endpoints

Distribute load across two endpoints in the same region:

aiperf profile \
    --model my-model \
    --url https://abc123.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --url https://def456.execute-api.us-east-1.amazonaws.com/prod/v1 \
    --endpoint-type chat \
    --streaming \
    --auth-type sigv4 \
    --aws-region us-east-1 \
    --aws-service execute-api \
    --request-count 500

SageMaker with Custom Dataset

aiperf profile \
    --model my-model \
    --url https://runtime.sagemaker.us-west-2.amazonaws.com \
    --endpoint /endpoints/my-endpoint/invocations \
    --endpoint-type chat \
    --auth-type sigv4 \
    --aws-region us-west-2 \
    --aws-service sagemaker \
    --dataset prompts.jsonl \
    --dataset-type single_turn

Troubleshooting

"SignatureDoesNotMatch"

This is the most common error. Check these in order:

  1. Is --aws-region correct? It must match the region in the URL.
  2. Is --aws-service correct? See the service name table above. The names aren't always obvious.
  3. Is your system clock accurate? AWS rejects signatures that are more than 5 minutes off. Docker containers and VMs are especially prone to clock drift. Run date -u and compare to actual UTC.

"The security token included in the request is expired"

Your temporary credentials have expired. Re-authenticate:

# For SSO
aws sso login --profile my-profile

# For assumed roles, this usually resolves itself --
# botocore refreshes automatically if the source credentials are still valid

"No AWS credentials found"

AIPerf can't find any credentials. Verify with:

aws sts get-caller-identity

If that also fails, you need to set up credentials -- see Setting Up Credentials.

"SigV4 auth requires botocore"

Install the AWS extra:

uv pip install 'aiperf[aws]'