[Data] Add polars usage instruction to docs by peterxcli · Pull Request #60029 · ray-project/ray

peterxcli · 2026-01-10T16:52:12Z

Description

We can use polars to make operations more efficient by

Use polars in map_batches UDF (this is already covered at: https://docs.ray.io/en/master/data/transforming-data.html#choosing-the-right-batch-format)
Set use_polars or use_polars_sort in DataContext to enable built-in polars ops (use_polars flag is deprecated, so I only add use_polars_sort at this time)

Related issues

Closes: #59224

Additional information

No

… section Signed-off-by: peterxcli <peterxcli@gmail.com>

peterxcli · 2026-01-10T16:53:04Z

cc @owenowenisme

owenowenisme · 2026-01-10T16:59:32Z

/gemini review

gemini-code-assist

Code Review

This pull request adds documentation on how to enable Polars operations in Ray Data via DataContext. The added section is clear and helpful. I've found a minor issue in one sentence which contains a typo and a grammatical error. I've provided a suggestion to fix it for better clarity.

gemini-code-assist

Code Review

This pull request adds documentation on how to enable Polars-based operations in Ray Data, specifically for sorting, by setting use_polars_sort in the DataContext. The change is clear and useful. I've found a minor grammatical issue in the new documentation and suggested a correction to improve clarity.

owenowenisme · 2026-01-10T17:20:03Z

@peterxcli please fix the suggestion from gemini, thanks

owenowenisme · 2026-01-10T17:22:38Z

And also, if we want to use glossary like "Polars", please use it with backtick like Polars or vale will raise error in CI.

github-actions · 2026-01-25T12:25:46Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Peter Lee <peterxcli@gmail.com>

Signed-off-by: peterxcli <peterxcli@gmail.com>

peterxcli · 2026-01-25T16:46:17Z

@owenowenisme please take another look. thanks!

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

peterxcli · 2026-01-26T13:38:45Z

@owenowenisme Thanks for the review🙏

bveeramani · 2026-01-28T22:45:20Z


 If you encounter an OOM errors, try decreasing your ``batch_size``.

+Enabling ``Polars`` operations


Nit: Here and elsewhere -- I think it makes more sense for Polars to not be code text, especially since it's not part of the glossary or anything.

bveeramani · 2026-01-28T22:48:26Z

+    ctx = ray.data.DataContext.get_current()
+    ctx.use_polars_sort = True
+
+When you enable this flag, Ray Data automatically uses ``Polars`` for tabular dataset sorting operations, which can significantly improve performance for certain workloads. This doesn't affect your UDF code, you can still use any batch format in :meth:`~ray.data.Dataset.map_batches`.


What're the user-facing Ray Data APIs that benefit from the polars feature?

IIUC it doesn't improve performance for most UDFs except for map_groups, and that's because of an implementation detail where we perform a sort.

Would this information be more appropriate in a different user guide(s)?

I moved it to performance-tips, WDYT?

iamjustinhsu · 2026-02-04T18:29:11Z

Hi @peterxcli, are you still working on this?

Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>

Copilot

Pull request overview

Adds documentation to the Ray Data “Transforming data” guide describing how to enable polars-backed optimizations via DataContext, in response to #59224.

Changes:

Add a new “Enabling Polars operations” subsection explaining DataContext.use_polars_sort.
Minor whitespace cleanup in existing .. testcode:: blocks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

## Description We can use polars to make operations more efficient by - Use polars in `map_batches` UDF (this is already covered at: https://docs.ray.io/en/master/data/transforming-data.html#choosing-the-right-batch-format) - Set `use_polars` or `use_polars_sort` in `DataContext` to enable built-in polars ops (`use_polars` flag is deprecated, so I only add `use_polars_sort` at this time) ## Related issues Closes: ray-project#59224 ## Additional information No --------- Signed-off-by: peterxcli <peterxcli@gmail.com> Signed-off-by: Peter Lee <peterxcli@gmail.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: Adel Nour <ans9868@nyu.edu>

## Description We can use polars to make operations more efficient by - Use polars in `map_batches` UDF (this is already covered at: https://docs.ray.io/en/master/data/transforming-data.html#choosing-the-right-batch-format) - Set `use_polars` or `use_polars_sort` in `DataContext` to enable built-in polars ops (`use_polars` flag is deprecated, so I only add `use_polars_sort` at this time) ## Related issues Closes: ray-project#59224 ## Additional information No --------- Signed-off-by: peterxcli <peterxcli@gmail.com> Signed-off-by: Peter Lee <peterxcli@gmail.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu>

peterxcli requested a review from a team as a code owner January 10, 2026 16:52

[Docs] Enhance transforming-data documentation with Polars operations…

5cbd0ef

… section Signed-off-by: peterxcli <peterxcli@gmail.com>

peterxcli force-pushed the docs/add-polars-detail-in-transforming-data branch from f3b3875 to 5cbd0ef Compare January 10, 2026 16:52

owenowenisme added data Ray Data-related issues community-contribution Contributed by the community labels Jan 10, 2026

gemini-code-assist Bot reviewed Jan 10, 2026

View reviewed changes

Comment thread doc/source/data/transforming-data.rst Outdated

gemini-code-assist Bot reviewed Jan 10, 2026

View reviewed changes

Comment thread doc/source/data/transforming-data.rst Outdated

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 25, 2026

peterxcli and others added 3 commits January 26, 2026 00:38

gemini suggestion

1b7503d

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Peter Lee <peterxcli@gmail.com>

add backtick around Polars

81ae02c

Signed-off-by: peterxcli <peterxcli@gmail.com>

Merge branch 'master' into docs/add-polars-detail-in-transforming-data

09d992f

ryankert01 removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 25, 2026

owenowenisme and others added 3 commits January 26, 2026 14:12

Merge branch 'master' into docs/add-polars-detail-in-transforming-data

a578b1c

update

078e2fc

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

update

0daf051

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

owenowenisme added the go add ONLY when ready to merge, run all tests label Jan 26, 2026

update

6de8f94

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

owenowenisme approved these changes Jan 26, 2026

View reviewed changes

bveeramani reviewed Jan 28, 2026

View reviewed changes

Update doc/source/data/transforming-data.rst

2c4e464

Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 9, 2026 03:31

Merge branch 'master' into docs/add-polars-detail-in-transforming-data

222d468

Copilot started reviewing on behalf of owenowenisme February 9, 2026 03:32 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Comment thread doc/source/data/transforming-data.rst Outdated

Comment thread doc/source/data/transforming-data.rst Outdated

Comment thread doc/source/data/transforming-data.rst Outdated

Comment thread doc/source/data/transforming-data.rst Outdated

update

86082dc

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

owenowenisme force-pushed the docs/add-polars-detail-in-transforming-data branch from 2a239fa to 86082dc Compare February 9, 2026 04:29

owenowenisme added 2 commits February 9, 2026 12:35

move tips to performance tips

21da8d0

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

update accept words list

390f150

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

bveeramani approved these changes Feb 10, 2026

View reviewed changes

bveeramani merged commit fa31667 into ray-project:master Feb 10, 2026
5 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Data] Add polars usage instruction to docs#60029

[Data] Add polars usage instruction to docs#60029
bveeramani merged 13 commits into
ray-project:masterfrom
peterxcli:docs/add-polars-detail-in-transforming-data

peterxcli commented Jan 10, 2026

peterxcli commented Jan 10, 2026

owenowenisme commented Jan 10, 2026

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

owenowenisme commented Jan 10, 2026

owenowenisme commented Jan 10, 2026 •

edited

Loading

github-actions Bot commented Jan 25, 2026

peterxcli commented Jan 25, 2026

peterxcli commented Jan 26, 2026

Uh oh!

bveeramani Jan 28, 2026

bveeramani Jan 28, 2026

owenowenisme Feb 9, 2026

iamjustinhsu commented Feb 4, 2026

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

6 participants


		If you encounter an OOM errors, try decreasing your ``batch_size``.

		Enabling ``Polars`` operations

Uh oh!

Conversation

peterxcli commented Jan 10, 2026

Description

Related issues

Additional information

peterxcli commented Jan 10, 2026

owenowenisme commented Jan 10, 2026

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

owenowenisme commented Jan 10, 2026

owenowenisme commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions Bot commented Jan 25, 2026

peterxcli commented Jan 25, 2026

peterxcli commented Jan 26, 2026

Uh oh!

bveeramani Jan 28, 2026

Choose a reason for hiding this comment

bveeramani Jan 28, 2026

Choose a reason for hiding this comment

owenowenisme Feb 9, 2026

Choose a reason for hiding this comment

iamjustinhsu commented Feb 4, 2026

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

6 participants

owenowenisme commented Jan 10, 2026 •

edited

Loading