[Runtime Env] Fix ray job submit --working-dir for local zip files and http:// URLs#62843
Merged
edoakes merged 3 commits intoMay 8, 2026
Merged
Conversation
![gemini-code-assist[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request adds support for the http protocol in Ray's runtime environment, allowing remote HTTP paths for zip files, and updates upload_working_dir_if_needed to utilize an optional upload_fn for local zip files. The review feedback suggests consolidating the http and https protocol handling blocks to reduce duplication, updating the type hint for upload_fn to include the new is_file parameter, and removing a redundant test case.
wenhaozhao011-cmd
force-pushed
the
fix-job-submit-working-dir
branch
from
aea1a79 to
4e96f0b
Compare
ray-gardener
Bot
added
core
edoakes
approved these changes
Apr 22, 2026
![cursor[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Reviewed by Cursor Bugbot for commit b14f5778843d08749cb4baa3ec8a8fbb113d9f96. Configure here.
…and http:// URLs Fix two bugs in `ray job submit --working-dir`: 1. **Local zip files (ray-project#34605)**: When a local `.zip` file is specified as `working_dir`, the upload path bypassed the HTTP-based `upload_fn` and called `upload_package_to_gcs()` directly, which requires `ray.init()` (never called by the CLI client). Now respects `upload_fn` when provided, following the same pattern used in `py_modules.py` for `.whl` files. 2. **HTTP URLs (ray-project#45954)**: `http://` URLs were not recognized as remote URIs because `http` was missing from the Protocol enum and remote protocols set. Added `http` protocol support, reusing the existing HTTPS handler since `smart_open` supports both. 🤖 Generated with [Qoder][https://qoder.com] Signed-off-by: leqiong <leqiong.zwh@alibaba-inc.com>
wenhaozhao011-cmd
force-pushed
the
fix-job-submit-working-dir
branch
from
b14f577 to
52de6bb
Compare
Contributor
Author
|
This CI failure appears to be unrelated to this PR. |
Contributor
Author
|
Hi @edoakes , I've updated the branch and all checks are passing. |
Collaborator
|
@wenhaozhao011-cmd merged -- thanks for the contribution! |
Lucas61000
pushed a commit
to Lucas61000/ray
that referenced
this pull request
May 15, 2026
…and http:// URLs (ray-project#62843) ## Why are these changes needed? When using the `ray job submit` command with `--working-dir` pointing to a local zip file or an HTTP URL of a zip file, we encountered some bugs. For details, please refer to the two issues below: ### 1. Local zip files fail to upload (ray-project#34605) When a local `.zip` file is specified as `working_dir`, the code in `upload_working_dir_if_needed()` bypassed the HTTP-based `upload_fn` callback and called `upload_package_to_gcs()` directly. This requires an active GCS connection via `ray.init()`, which is never called by the CLI client, causing a `RuntimeError`. **Fix:** Check `upload_fn` before choosing the upload mechanism, following the same pattern used in `py_modules.py` for `.whl` files (lines 140-154). ### 2. HTTP URLs not recognized as remote URIs (ray-project#45954) `http://` URLs were not recognized as valid remote URIs because the `http` protocol was missing from the `Protocol` enum and `get_remote_protocols()`. The URL was incorrectly treated as a local path. **Fix:** Add `http` to `get_protocols()`, `get_remote_protocols()`, and `download_remote_uri()`, reusing the existing HTTPS handler since `smart_open` already supports both HTTP and HTTPS. ## Related issue number Closes ray-project#34605 Closes ray-project#45954 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [x] I've added any new tests if needed. Signed-off-by: leqiong <leqiong.zwh@alibaba-inc.com> Co-authored-by: leqiong <leqiong.zwh@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
When using the
ray job submitcommand with--working-dirpointing to a local zip file or an HTTP URL of a zip file, we encountered some bugs. For details, please refer to the two issues below:1. Local zip files fail to upload (#34605)
When a local
.zipfile is specified asworking_dir, the code inupload_working_dir_if_needed()bypassed the HTTP-basedupload_fncallback and called
upload_package_to_gcs()directly. This requiresan active GCS connection via
ray.init(), which is never called bythe CLI client, causing a
RuntimeError.Fix: Check
upload_fnbefore choosing the upload mechanism, followingthe same pattern used in
py_modules.pyfor.whlfiles (lines 140-154).2. HTTP URLs not recognized as remote URIs (#45954)
http://URLs were not recognized as valid remote URIs because thehttpprotocol was missing from the
Protocolenum andget_remote_protocols().The URL was incorrectly treated as a local path.
Fix: Add
httptoget_protocols(),get_remote_protocols(), anddownload_remote_uri(), reusing the existing HTTPS handler sincesmart_openalready supports both HTTP and HTTPS.Related issue number
Closes #34605
Closes #45954
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.https://docs.ray.io/en/master/.