[train][Docs] Document S3-compatible storage by goanpeca · Pull Request #63103 · ray-project/ray

goanpeca · 2026-05-04T15:43:27Z

Why are these changes needed?

Ray Train already works with any S3-compatible object store through pyarrow's S3FileSystem (via endpoint_override in the storage_path URI, or the standard AWS_* environment variables). This PR documents that path in the Train persistent-storage guide and adds the Backblaze B2 specifics.

Docs-only, no code changes. (An earlier revision added an env-var aliasing helper; per review feedback it was removed in favor of documenting the setup users perform themselves.)

Changes to doc/source/train/user-guides/persistent-storage.rst:

Retitles the section to "S3-compatible storage (Backblaze B2, MinIO, etc.)".
Shows the endpoint_override query-parameter form for Backblaze B2 and MinIO (local).
Notes that the standard AWS environment variables (AWS_ENDPOINT_URL_S3, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) work with a plain s3://bucket/path.
Documents that Backblaze B2 publishes credentials as B2_APPLICATION_KEY_ID / B2_APPLICATION_KEY; users set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY to those values, since pyarrow reads only the AWS-named variables.
Links a complete end-to-end Backblaze B2 notebook example.

Related issue number

Related to #63104

Checks

Change is contained to doc/source/train/user-guides/persistent-storage.rst.
No code paths changed; existing tests unaffected.

gemini-code-assist · 2026-05-04T15:46:09Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

Copilot

Pull request overview

This PR improves Ray Train’s S3-compatible storage experience (notably Backblaze B2) by ensuring credentials provided via Backblaze’s CLI env var names are made visible to pyarrow’s S3 resolver, and by expanding the docs with a B2-focused example.

Changes:

Add _alias_s3_compatible_credentials_to_aws_env_vars() and call it from get_fs_and_path() to map B2 env vars onto AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY when appropriate.
Add unit tests covering aliasing behavior, no-op behavior, and warnings.
Update Train persistent-storage docs with a Backblaze B2 example, endpoint override guidance, and a link to an end-to-end notebook.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
python/ray/train/_internal/storage.py	Adds credential env var aliasing logic and invokes it during filesystem resolution.
python/ray/train/tests/test_storage.py	Adds tests validating the new env var aliasing behavior and logging.
doc/source/train/user-guides/persistent-storage.rst	Updates S3-compatible storage docs to include Backblaze B2 guidance and a runnable example link.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-05-29T01:19:49Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

pseudo-rnd-thoughts

Thanks for the PR @goanpeca, I've used claude to improve the documentation so don't worry about where that random commit came from. I just have a single question about the implementation

pseudo-rnd-thoughts · 2026-05-29T16:42:26Z

@@ -294,6 +297,35 @@ def _create_directory(fs: pyarrow.fs.FileSystem, fs_path: str) -> None:
        )


+def _alias_s3_compatible_credentials_to_aws_env_vars() -> None:


Could you provide more information about why this function is necessary? Shouldn't this be done on the user side rather than behind the scenes

Hi @pseudo-rnd-thoughts!

Good question! It is not strictly necessary: the functional path is just AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY plus an endpoint override, which the docs now cover.

The helper is really just a convenience for people already on Backblaze B2. B2's docs and CLI use B2_APPLICATION_KEY_ID / B2_APPLICATION_KEY, so if a user has those exported and points Ray at an s3:// path, pyarrow silently ignores them (it only reads the AWS_ names) and they hit a confusing auth error. The alias saves them re-exporting the same secret, and it is structured so other providers can be added later.

That said, I am happy to drop it and keep this docs-only if you would rather credentials stay explicit on the user side. Just let me know! 😄

Thanks for the answer @goanpeca, yes, could we remove the function from the code and add a note to the documentation for the changes that users will need to implement for access

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 5c29b68791fd328f75db0ae016a4063ccffa38cd. Configure here.}

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>

pseudo-rnd-thoughts

Thanks for making the changes

## Why are these changes needed? Ray Train already works with any S3-compatible object store through pyarrow's `S3FileSystem` (via `endpoint_override` in the `storage_path` URI, or the standard `AWS_*` environment variables). This PR documents that path in the Train persistent-storage guide and adds the Backblaze B2 specifics. **Docs-only, no code changes.** (An earlier revision added an env-var aliasing helper; per review feedback it was removed in favor of documenting the setup users perform themselves.) Changes to `doc/source/train/user-guides/persistent-storage.rst`: - Retitles the section to "S3-compatible storage (Backblaze B2, MinIO, etc.)". - Shows the `endpoint_override` query-parameter form for Backblaze B2 and MinIO (local). - Notes that the standard AWS environment variables (`AWS_ENDPOINT_URL_S3`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) work with a plain `s3://bucket/path`. - Documents that Backblaze B2 publishes credentials as `B2_APPLICATION_KEY_ID` / `B2_APPLICATION_KEY`; users set `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` to those values, since pyarrow reads only the AWS-named variables. - Links a complete end-to-end Backblaze B2 notebook example. ## Related issue number Related to ray-project#63104 ## Checks - [x] Change is contained to `doc/source/train/user-guides/persistent-storage.rst`. - [x] No code paths changed; existing tests unaffected. Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>

goanpeca requested a review from a team as a code owner May 4, 2026 15:43

Copilot AI review requested due to automatic review settings May 4, 2026 15:43

goanpeca mentioned this pull request May 4, 2026

[Train] Document Backblaze B2 (S3-compatible) storage path + alias B2_APPLICATION_KEY env vars onto AWS-named equivalents #63104

Open

Copilot started reviewing on behalf of goanpeca May 4, 2026 15:45 View session

goanpeca force-pushed the add-b2-integration branch 2 times, most recently from 1546e25 to c117eb5 Compare May 4, 2026 15:48

Copilot AI reviewed May 4, 2026

View reviewed changes

Comment thread python/ray/train/_internal/storage.py Outdated

Comment thread python/ray/train/_internal/storage.py Outdated

Comment thread doc/source/train/user-guides/persistent-storage.rst Outdated

cursor Bot reviewed May 4, 2026

View reviewed changes

Comment thread python/ray/train/_internal/storage.py Outdated

Comment thread python/ray/train/_internal/storage.py Outdated

goanpeca force-pushed the add-b2-integration branch from c117eb5 to 74a69f2 Compare May 4, 2026 16:36

cursor Bot reviewed May 4, 2026

View reviewed changes

Comment thread python/ray/train/_internal/storage.py Outdated

goanpeca force-pushed the add-b2-integration branch 2 times, most recently from 96cf884 to 8e00f5b Compare May 4, 2026 16:55

ray-gardener Bot added docs An issue or change related to documentation train Ray Train Related Issue community-contribution Contributed by the community labels May 4, 2026

goanpeca force-pushed the add-b2-integration branch from 8e00f5b to 5c29b68 Compare May 14, 2026 23:41

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label May 29, 2026

pseudo-rnd-thoughts reviewed May 29, 2026

View reviewed changes

github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels May 30, 2026

goanpeca changed the title [train][Docs] Add Backblaze B2 example + alias B2_APPLICATION_KEY env vars onto AWS_* Jun 2, 2026

goanpeca force-pushed the add-b2-integration branch from ed9c5f6 to 5c29b68 Compare June 2, 2026 14:31

cursor Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread python/ray/train/_internal/storage.py Outdated

goanpeca force-pushed the add-b2-integration branch from 5c29b68 to 6b3049f Compare June 2, 2026 14:43

goanpeca changed the title ~~[train][Docs] Document S3-compatible storage (MinIO, Backblaze B2)~~ Jun 2, 2026

goanpeca force-pushed the add-b2-integration branch from 6442fa2 to bb3f8de Compare June 2, 2026 14:53

goanpeca requested a review from Copilot June 2, 2026 14:54

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Comment thread doc/source/train/user-guides/persistent-storage.rst

Comment thread doc/source/train/user-guides/persistent-storage.rst Outdated

goanpeca force-pushed the add-b2-integration branch from bb3f8de to c8e4c18 Compare June 2, 2026 14:56

goanpeca changed the title ~~[train][Docs] Document S3-compatible storage (Backblaze B2, MinIO)~~ Jun 2, 2026

goanpeca force-pushed the add-b2-integration branch from c8e4c18 to bd029db Compare June 2, 2026 15:00

goanpeca requested a review from Copilot June 2, 2026 15:00

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Comment thread doc/source/train/user-guides/persistent-storage.rst Outdated

Comment thread doc/source/train/user-guides/persistent-storage.rst Outdated

goanpeca force-pushed the add-b2-integration branch from bd029db to 8ddc19b Compare June 2, 2026 15:24

goanpeca requested a review from Copilot June 2, 2026 15:24

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Comment thread doc/source/train/user-guides/persistent-storage.rst Outdated

goanpeca force-pushed the add-b2-integration branch from 8ddc19b to a0524bd Compare June 2, 2026 15:41

goanpeca requested a review from Copilot June 2, 2026 15:41

Copilot AI reviewed Jun 2, 2026

View reviewed changes

goanpeca requested a review from pseudo-rnd-thoughts June 2, 2026 15:48

Add S3 compatible storage documentation, include B2

98083b8

Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>

goanpeca force-pushed the add-b2-integration branch from a0524bd to 98083b8 Compare June 2, 2026 15:48

pseudo-rnd-thoughts approved these changes Jun 4, 2026

View reviewed changes

pseudo-rnd-thoughts added the go add ONLY when ready to merge, run all tests label Jun 4, 2026

matthewdeng merged commit a2f222d into ray-project:master Jun 8, 2026
8 checks passed

goanpeca deleted the add-b2-integration branch June 9, 2026 03:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[train][Docs] Document S3-compatible storage#63103

[train][Docs] Document S3-compatible storage#63103
matthewdeng merged 1 commit into
ray-project:masterfrom
goanpeca:add-b2-integration

goanpeca commented May 4, 2026 •

edited

Loading

gemini-code-assist Bot commented May 4, 2026

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026

pseudo-rnd-thoughts left a comment •

edited

Loading

pseudo-rnd-thoughts May 29, 2026

goanpeca Jun 2, 2026 •

edited

Loading

pseudo-rnd-thoughts Jun 2, 2026

goanpeca Jun 2, 2026

cursor Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

pseudo-rnd-thoughts left a comment

Uh oh!

Labels

4 participants

		@@ -294,6 +297,35 @@ def _create_directory(fs: pyarrow.fs.FileSystem, fs_path: str) -> None:
		)


		def _alias_s3_compatible_credentials_to_aws_env_vars() -> None:

Uh oh!

Conversation

goanpeca commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

gemini-code-assist Bot commented May 4, 2026

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026

pseudo-rnd-thoughts left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

pseudo-rnd-thoughts May 29, 2026

Choose a reason for hiding this comment

goanpeca Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

pseudo-rnd-thoughts Jun 2, 2026

Choose a reason for hiding this comment

goanpeca Jun 2, 2026

Choose a reason for hiding this comment

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

pseudo-rnd-thoughts left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

4 participants

goanpeca commented May 4, 2026 •

edited

Loading

pseudo-rnd-thoughts left a comment •

edited

Loading

goanpeca Jun 2, 2026 •

edited

Loading