[serve] Add env var overrides when throughput opt is enabled

[akyang-anyscale]

Thank you for contributing to Ray! 🚀
Please review the Ray Contribution Guide before opening a pull request.

⚠️ Remove these instructions before submitting your PR.

💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete.

Description

Briefly describe what this PR accomplishes and why it's needed.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

^{Cursor Bugbot reviewed your changes and found no issues for commit 719363a}

[gemini-code-assist]

Code Review

This PR adds an override for RAY_SERVE_ENABLE_DIRECT_INGRESS when throughput optimization is on, enabling it by default in that mode. The change is correct. I've added one comment regarding the pattern of re-defining configuration variables, suggesting a refactor for improved maintainability, though it can be addressed in a follow-up.

[gemini-code-assist]

This change follows the existing pattern of overriding constants when RAY_SERVE_THROUGHPUT_OPTIMIZED is enabled. While this is consistent with the surrounding code, this pattern of re-defining variables can be confusing and error-prone, as one has to read the entire file to determine the final value of a constant.

A more robust approach would be to define these constants only once, with their default value determined conditionally based on RAY_SERVE_THROUGHPUT_OPTIMIZED.

For example:

# This would replace the definition at line 574 and this override
_direct_ingress_default = "1" if RAY_SERVE_THROUGHPUT_OPTIMIZED else "0"
RAY_SERVE_ENABLE_DIRECT_INGRESS = get_env_bool(
    "RAY_SERVE_ENABLE_DIRECT_INGRESS", _direct_ingress_default
)

This would require refactoring the other overridden variables for consistency. While this is a larger change that could be done in a follow-up, it would significantly improve the code's clarity and maintainability.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Derived variable not updated after throughput optimization reassignment

Medium Severity

When RAY_SERVE_THROUGHPUT_OPTIMIZED is enabled, reassigning RAY_SERVE_USE_GRPC_BY_DEFAULT at line 617 doesn't update RAY_SERVE_PROXY_USE_GRPC, which was already computed earlier at lines 552-555 using the original (False) value. This means enabling throughput optimization won't properly enable gRPC-by-default behavior for the proxy since RAY_SERVE_PROXY_USE_GRPC remains False.

Additional Locations (1)

• python/ray/serve/_private/constants.py#L551-L555