[Data] Gate unsafe deserialization in WebDataset default decoder#63469
Merged
Conversation
…nd env var The default decoder in `read_webdataset` runs `pickle.loads` on `.pkl/.pickle` files and `torch.load(weights_only=False)` on `.pt/.pth` files from attacker-controlled TAR archives, enabling arbitrary code execution. Gate these branches behind `RAY_DATA_WEBDATASET_ALLOW_UNSAFE_DESERIALIZATION=1` and point users to the existing `decoder` parameter as the safe escape hatch. Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
![cursor[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cc6a31b. Configure here.
iamjustinhsu
approved these changes
May 18, 2026
_allow_unsafe_deserialization() checked os.environ on workers, which never saw monkeypatch.setenv from the driver. Instead, read the env var in WebDatasetDatasource.__init__ (driver-side), store the result as self._allow_unsafe_deserialization, and pass it to _default_decoder via functools.partial. The datasource is pickled to workers, so the flag propagates automatically. Also fix custom decoder test key match (pkl, not .pkl). Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
… messages Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
TruongQuangPhat
pushed a commit
to cyhapun/ray-fix-issue
that referenced
this pull request
May 27, 2026
…-project#63469) ## Description The default decoder in `read_webdataset` runs `pickle.loads` on `.pkl/.pickle` files and `torch.load(weights_only=False)` on `.pt/.pth` files from attacker-controlled TAR archives, enabling arbitrary code execution with no opt-in. This is the same class of bug as GHSA-mw35-8rx3-xf9r (patched in 2.55.0 for Parquet), but in a different code path that was not addressed. This PR gates the unsafe `.pkl/.pickle` and `.pt/.pth` branches in `_default_decoder` behind the `RAY_DATA_WEBDATASET_ALLOW_UNSAFE_DESERIALIZATION=1` environment variable. The error message points users to the existing `decoder` parameter on `ray.data.read_webdataset()` as the safe escape hatch for custom deserialization. ## Related issues Fixes GHSA-hhrp-gw25-jr43 ## Additional information - Existing tests that rely on `.pt` round-trip encoding/decoding now use a `monkeypatch.setenv` fixture to opt in - New tests verify: rejection of all four unsafe extensions by default, opt-in via env var, and bypass via custom `decoder` callable --------- Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: phattruong <23120318@student.hcmus.edu.vn>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The default decoder in
read_webdatasetrunspickle.loadson.pkl/.picklefiles andtorch.load(weights_only=False)on.pt/.pthfiles from attacker-controlled TAR archives, enabling arbitrary code execution with no opt-in. This is the same class of bug as GHSA-mw35-8rx3-xf9r (patched in 2.55.0 for Parquet), but in a different code path that was not addressed.This PR gates the unsafe
.pkl/.pickleand.pt/.pthbranches in_default_decoderbehind theRAY_DATA_WEBDATASET_ALLOW_UNSAFE_DESERIALIZATION=1environment variable. The error message points users to the existingdecoderparameter onray.data.read_webdataset()as the safe escape hatch for custom deserialization.Related issues
Fixes GHSA-hhrp-gw25-jr43
Additional information
.ptround-trip encoding/decoding now use amonkeypatch.setenvfixture to opt indecodercallable