fix(serve): normalize multiplexed model ID header to support proxy-transformed names by Yunnglin · Pull Request #61869 · ray-project/ray

Yunnglin · 2026-03-19T10:21:14Z

Why are these changes needed?

HTTP proxies such as nginx and AWS API Gateway convert underscores to hyphens in header names by default. This means a client sending:

serve_multiplexed_model_id: my-model

arrives at Ray Serve as:

serve-multiplexed-model-id: my-model

The previous exact-match comparison in setup_request_context_and_handle against the constant "serve_multiplexed_model_id" silently failed in this case. As a result:

serve.get_multiplexed_model_id() returned "" (empty string)
load_model("") raised ValueError: The model ID cannot be empty.

What changes were made?

In proxy.py, normalize the incoming header key before comparing:

normalized_key = key.decode().lower().replace("-", "_")
if normalized_key == SERVE_MULTIPLEXED_MODEL_ID:

This makes both serve_multiplexed_model_id and serve-multiplexed-model-id (and any case variant like Serve-Multiplexed-Model-Id) equivalent — fully backward-compatible with no changes to the constant, docs, or tests.

Related issues / discussions

Proxies that strip or transform underscores: nginx (underscores_in_headers off is the default), AWS API Gateway, Cloudflare, etc.

gemini-code-assist

Code Review

The pull request effectively addresses the issue where HTTP proxies transform header names by converting underscores to hyphens. By normalizing the serve_multiplexed_model_id header key to lowercase and replacing hyphens with underscores, the system can now correctly recognize the header regardless of proxy transformations. This ensures that serve.get_multiplexed_model_id() and load_model() function as expected, preventing ValueError due to empty model IDs. The change is well-documented with clear comments explaining the normalization logic.

Copilot

Pull request overview

This PR improves Ray Serve’s HTTP proxy request context setup by normalizing the multiplexed model ID header name so requests continue to work when intermediaries (e.g., nginx, AWS API Gateway) transform underscores to hyphens.

Changes:

Normalize incoming header keys (lowercase + - → _) before comparing against SERVE_MULTIPLEXED_MODEL_ID.
Update inline comments to document why this normalization is required for common proxy behavior.

Comments suppressed due to low confidence (1)

python/ray/serve/_private/proxy.py:1243

key.decode() is still being computed multiple times per header (once for normalized_key and again for the request-id comparison). Consider decoding/lowercasing once per loop iteration and reusing the decoded value for both comparisons to reduce duplication and keep the matching logic consistent.

            normalized_key = key.decode().lower().replace("-", "_")
            if normalized_key == SERVE_MULTIPLEXED_MODEL_ID:
                multiplexed_model_id = value.decode()
                handle = handle.options(multiplexed_model_id=multiplexed_model_id)
                request_context_info["multiplexed_model_id"] = multiplexed_model_id
            if key.decode() == SERVE_HTTP_REQUEST_ID_HEADER:
                request_context_info["request_id"] = value.decode()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

abrarsheikh · 2026-03-19T17:30:38Z

+            normalized_key = key.decode().lower().replace("-", "_")
+            if normalized_key == SERVE_MULTIPLEXED_MODEL_ID:
                multiplexed_model_id = value.decode()


themavik

Solid fix for a subtle proxy compatibility issue. The lower().replace("-", "_") normalization elegantly handles nginx, API Gateway, and Cloudflare without changing the public API. One minor thought: have you considered adding a unit test with serve-multiplexed-model-id and Serve-Multiplexed-Model-Id to lock in this behavior?

themavik

Reviewed the changes — the implementation is clean and addresses the reported issue correctly.

themavik

Reviewed the changes — the fix is minimal and targeted. Good contribution.

abrarsheikh · 2026-03-19T17:30:12Z

please fix DCO

…ansformed names HTTP proxies (nginx, AWS API Gateway, etc.) convert underscores to hyphens in header names, so a client sending serve_multiplexed_model_id arrives at Ray Serve as serve-multiplexed-model-id. The previous exact-match comparison silently failed, causing get_multiplexed_model_id() to return '' and load_model() to raise: ValueError: The model ID cannot be empty. Fix: normalize the incoming header key by lowercasing and replacing hyphens with underscores before comparing, so both forms are accepted without any breaking change to existing users or documentation. Signed-off-by: Yunlin Mao <maoyl@smail.nju.edu.cn>

…zation

Yunnglin · 2026-03-20T02:41:11Z

I have fixed the DCO issue and added unit tests, please review the code.

harshit-anyscale · 2026-03-20T05:42:33Z

@Yunnglin there are some lint failures, can you please fix them?

…tion Add parametrized tests covering all three header forms that should be recognised by setup_request_context_and_handle: - serve_multiplexed_model_id (original underscore form, direct connection) - serve-multiplexed-model-id (hyphen form, produced by nginx / API Gateway) - Serve-Multiplexed-Model-Id (title-case hyphen form, some proxies) Signed-off-by: Yunlin Mao <maoyl@smail.nju.edu.cn>

…zation

Yunnglin · 2026-03-20T06:45:55Z

@Yunnglin there are some lint failures, can you please fix them?

@harshit-anyscale fixed

…zation

Yunnglin · 2026-03-24T06:55:05Z

@harshit-anyscale Can this PR be merged yet?

harshit-anyscale · 2026-03-24T08:36:14Z

cc: @abrarsheikh for approval and merge

Yunnglin · 2026-03-30T09:07:19Z

@abrarsheikh Can this PR be merged yet?

…ansformed names (ray-project#61869) ## Why are these changes needed? HTTP proxies such as nginx and AWS API Gateway convert underscores to hyphens in header names by default. This means a client sending: ``` serve_multiplexed_model_id: my-model ``` arrives at Ray Serve as: ``` serve-multiplexed-model-id: my-model ``` The previous exact-match comparison in `setup_request_context_and_handle` against the constant `"serve_multiplexed_model_id"` silently failed in this case. As a result: 1. `serve.get_multiplexed_model_id()` returned `""` (empty string) 2. `load_model("")` raised `ValueError: The model ID cannot be empty.` ## What changes were made? In `proxy.py`, normalize the incoming header key before comparing: ```python normalized_key = key.decode().lower().replace("-", "_") if normalized_key == SERVE_MULTIPLEXED_MODEL_ID: ``` This makes both `serve_multiplexed_model_id` and `serve-multiplexed-model-id` (and any case variant like `Serve-Multiplexed-Model-Id`) equivalent — fully backward-compatible with no changes to the constant, docs, or tests. ## Related issues / discussions - Proxies that strip or transform underscores: nginx (`underscores_in_headers off` is the default), AWS API Gateway, Cloudflare, etc. --------- Signed-off-by: Yunlin Mao <maoyl@smail.nju.edu.cn>

Yunnglin requested a review from a team as a code owner March 19, 2026 10:21

Copilot AI review requested due to automatic review settings March 19, 2026 10:21

gemini-code-assist Bot reviewed Mar 19, 2026

View reviewed changes

Copilot AI reviewed Mar 19, 2026

View reviewed changes

themavik reviewed Mar 19, 2026

View reviewed changes

ray-gardener Bot added serve Ray Serve Related Issue community-contribution Contributed by the community labels Mar 19, 2026

Yunnglin force-pushed the fix/serve-multiplexed-header-proxy-normalization branch from 979bd5e to da03a1a Compare March 20, 2026 01:53

Merge branch 'master' into fix/serve-multiplexed-header-proxy-normali…

78ea4ae

…zation

harshit-anyscale added the go add ONLY when ready to merge, run all tests label Mar 20, 2026

Yunnglin force-pushed the fix/serve-multiplexed-header-proxy-normalization branch from 6f76439 to 4b4d300 Compare March 20, 2026 06:31

Merge branch 'master' into fix/serve-multiplexed-header-proxy-normali…

586133d

…zation

Merge branch 'master' into fix/serve-multiplexed-header-proxy-normali…

de168e2

…zation

harshit-anyscale approved these changes Mar 23, 2026

View reviewed changes

abrarsheikh approved these changes Apr 11, 2026

View reviewed changes

abrarsheikh merged commit 94eb59b into ray-project:master Apr 11, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(serve): normalize multiplexed model ID header to support proxy-transformed names#61869

fix(serve): normalize multiplexed model ID header to support proxy-transformed names#61869
abrarsheikh merged 5 commits into
ray-project:masterfrom
Yunnglin:fix/serve-multiplexed-header-proxy-normalization

Yunnglin commented Mar 19, 2026

gemini-code-assist Bot left a comment

Copilot AI left a comment

abrarsheikh Mar 19, 2026

themavik left a comment

themavik left a comment

themavik left a comment

abrarsheikh commented Mar 19, 2026

Yunnglin commented Mar 20, 2026

harshit-anyscale commented Mar 20, 2026

Yunnglin commented Mar 20, 2026 •

edited

Loading

Yunnglin commented Mar 24, 2026

harshit-anyscale commented Mar 24, 2026

Yunnglin commented Mar 30, 2026

Uh oh!

Labels

5 participants

Uh oh!

Conversation

Yunnglin commented Mar 19, 2026

Why are these changes needed?

What changes were made?

Related issues / discussions

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

abrarsheikh Mar 19, 2026

Choose a reason for hiding this comment

themavik left a comment

Choose a reason for hiding this comment

themavik left a comment

Choose a reason for hiding this comment

themavik left a comment

Choose a reason for hiding this comment

abrarsheikh commented Mar 19, 2026

Yunnglin commented Mar 20, 2026

harshit-anyscale commented Mar 20, 2026

Yunnglin commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Yunnglin commented Mar 24, 2026

harshit-anyscale commented Mar 24, 2026

Yunnglin commented Mar 30, 2026

Uh oh!

Labels

5 participants

Yunnglin commented Mar 20, 2026 •

edited

Loading