[dashboard] mask password arguments in get_entrypoint_name() to prevent password exposure#61995
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a security enhancement to mask password arguments in get_entrypoint_name(), preventing them from being exposed in the Ray dashboard. The implementation correctly handles arguments in the format --key=value. I've suggested an improvement to also handle the --key value format for more comprehensive protection. Additionally, I've recommended adding a corresponding test case to ensure both formats are properly masked.
| sanitized = [ | ||
| re.sub(r"(--[\w-]*password=).*", r"\1****", arg) for arg in curr.cmdline() | ||
| ] | ||
| return prefix + list2cmdline(sanitized) |
There was a problem hiding this comment.
The current implementation only masks passwords provided in the --key=value format. It would be more robust to also handle passwords passed in the --key value format (e.g., --password mysecret). This change enhances security by covering another common way of passing sensitive information via command-line arguments.
| sanitized = [ | |
| re.sub(r"(--[\w-]*password=).*", r"\1****", arg) for arg in curr.cmdline() | |
| ] | |
| return prefix + list2cmdline(sanitized) | |
| cmd_line = curr.cmdline() | |
| sanitized = [] | |
| i = 0 | |
| while i < len(cmd_line): | |
| arg = cmd_line[i] | |
| # Mask arguments in the form --key=value | |
| new_arg = re.sub(r"(--[\w-]*password=).*", r"\1****", arg) | |
| if new_arg != arg: | |
| sanitized.append(new_arg) | |
| i += 1 | |
| # Mask arguments in the form --key value | |
| elif re.match(r"--[\w-]*password$", arg) and i + 1 < len(cmd_line) and not cmd_line[i + 1].startswith("-"): | |
| sanitized.append(arg) | |
| sanitized.append("****") | |
| i += 2 | |
| else: | |
| sanitized.append(arg) | |
| i += 1 | |
| return prefix + list2cmdline(sanitized) |
| assert line_exists( | ||
| outputs, f"result: {sys.executable} {fp} --redis-password=\\*\\*\\*\\*" | ||
| ) | ||
| assert not any("secret123" in line for line in outputs) |
There was a problem hiding this comment.
To ensure the password masking is robust, it's a good idea to add a test case for passwords passed in the --key value format. This will verify that both common argument styles are handled correctly.
| assert not any("secret123" in line for line in outputs) | |
| assert not any("secret123" in line for line in outputs) | |
| # Test that password arguments with a space are also masked. | |
| outputs = execute_driver([sys.executable, str(fp), "--another-password", "secret456"]) | |
| assert line_exists( | |
| outputs, f"result: {sys.executable} {fp} --another-password \*\*\*\*" | |
| ) | |
| assert not any("secret456" in line for line in outputs) |
f0bb557 to
eff6631
Compare
password exposure Mask any process argument matching --[\w-]*password=<value> in get_entrypoint_name() to prevent passwords from being exposed in the Ray dashboard. The entrypoint was being sent to the GCS with sensitive arguments exposed. This was then reflected in the Ray dashboard, leaking password values. Fixes ray-project#56927 Signed-off-by: Pedro Jeronimo <pedro.jeronimo@tecnico.ulisboa.pt>
eff6631 to
74eae17
Compare
|
This solution looks quite brittle. Could we instead just avoid passing the password as a CLI argument? It could be passed as an environment variable instead, for example. |
Thanks for the feedback, @edoakes ! You're right, passing the Redis password via an environment variable is a cleaner and more robust solution. I'll update the implementation accordingly and drop the masking approach. I'll keep the CLI argument option for retro-compatibility, marking it as deprecated. If provided, it will set the environment variable early (if not already set), so it doesn't get propagated to subprocesses as an argument, showing up in places like |
Directly removing the argument should be fine as this is an internal API (I don't expect anyone to be starting this server on their own manually). Here is the entrypoint: ray/python/ray/util/client/server/server.py Line 876 in 3ee7c9d |
Ah got it, thanks for the clarification @edoakes ! I was initially thinking about a more general change at the I'll go ahead and remove the argument from the client server entrypoint and rely on the environment variable there instead. |
…iable Stop passing the redis password to the ray client server as a CLI argument. It was being exposed in the job entrypoint shown in the dashboard. Use an environment variable instead to prevent this exposure. Fixes ray-project#56927 Signed-off-by: Pedro Jeronimo <pedro.jeronimo@tecnico.ulisboa.pt>
|
Hey @edoakes ! I just committed the discussed changes. Let me know what you think! Now the redis password no longer appears in the job entrypoint. |
edoakes
left a comment
There was a problem hiding this comment.
Looks good, minor comments
Signed-off-by: Pedro Jeronimo <pedro.jeronimo@tecnico.ulisboa.pt>
|
Hey @edoakes ! I saw that the buildkite/premerge is currently failing due to a dependency resolution error. I looked into it briefly, and it doesn’t seem related to my changes. Just wanted to check if there’s anything I should do on my side. Thanks! |
|
@pedrojeronim0 just merged it. Thanks for the contribution and the reminder! |
…nt password exposure (ray-project#61995) Mask any process argument matching `--[\w-]*password=<value>` in get_entrypoint_name() to prevent passwords from being exposed in the Ray dashboard. - [x] Verified with ruff linter. - [x] Added test to verify password values are not exposed in the output of get_entrypoint_name(). - [x] Verified that test_job.py which contains test_get_entrypoint passes. ## Related issues Fixes ray-project#56927 ## Additional information The entrypoint was being sent to the GCS with sensitive arguments exposed. This was then reflected in the Ray dashboard, leaking password values.   --------- Signed-off-by: Pedro Jeronimo <pedro.jeronimo@tecnico.ulisboa.pt>
…nt password exposure (ray-project#61995) Mask any process argument matching `--[\w-]*password=<value>` in get_entrypoint_name() to prevent passwords from being exposed in the Ray dashboard. - [x] Verified with ruff linter. - [x] Added test to verify password values are not exposed in the output of get_entrypoint_name(). - [x] Verified that test_job.py which contains test_get_entrypoint passes. ## Related issues Fixes ray-project#56927 ## Additional information The entrypoint was being sent to the GCS with sensitive arguments exposed. This was then reflected in the Ray dashboard, leaking password values.   --------- Signed-off-by: Pedro Jeronimo <pedro.jeronimo@tecnico.ulisboa.pt>
|
Hi @edoakes , I saw the PR is already merged. Are you planning to release this soon in the next version? |
|
@anhpham1509 we are cutting a release branch imminently. The release should go out next week. |
|
@edoakes Hello, just want to catch up on the release progress. It's few week drifted and I haven't seen a new release. |


Description
Mask any process argument matching
--[\w-]*password=<value>in get_entrypoint_name()to prevent passwords from being exposed in the Ray dashboard.
Related issues
Fixes #56927
Additional information
The entrypoint was being sent to the GCS with sensitive arguments exposed.
This was then reflected in the Ray dashboard, leaking password values.