[data] fix bug that _align_struct_fields fails if there are unaligned scalar… by 947132885 · Pull Request #58364 · ray-project/ray

947132885 · 2025-11-03T02:51:42Z

Description

ray fails to merge blocks when there are both unaligned scalar and struct fields.

Minimal code to reproduce:

import pyarrow as pa
from ray.data._internal.arrow_ops.transform_pyarrow import concat

table1 = pa.table({
    "a": [1, 2, 3],
    "s": [
        {"x": 7,},
        {"x": 8,},
        {"x": 9,},
    ],
})

table2 = pa.table({
    "b": [4, 5, 6],
})
concat([table1, table2])

Fix:

create null array in _align_struct_fields instead of assertion

gemini-code-assist

Code Review

This pull request addresses a bug in _align_struct_fields that caused a failure when processing blocks with missing scalar fields alongside struct fields. The fix correctly replaces an overly strict assertion with logic to pad missing columns with nulls. This change is correct, well-targeted, and resolves the issue described. The implementation is clean and follows standard practices for handling missing data in pyarrow tables.

goutamvenkat-anyscale · 2025-11-03T07:22:07Z

Thanks @947132885 for your contribution. Can you please add some tests for this change?

947132885 · 2025-11-03T12:02:52Z

ok, I added a simple test

goutamvenkat-anyscale · 2025-11-20T18:26:40Z

+                null_array = pa.nulls(len(block), type=schema.field(column_name).type)
+                new_columns.append(null_array)


Can you please use BlockAccessor abstraction for this operation?

Actually can we reuse _backfill_missing_fields for this?

goutamvenkat-anyscale · 2025-11-20T18:30:54Z

@@ -552,5 +552,36 @@ def test_arrow_block_max_chunk_size(table_data, max_chunk_size_bytes, expected):
    assert _get_max_chunk_size(table, max_chunk_size_bytes) == expected


+def test_arrow_block_concat():
+    def gen_block(table):


Instead of an E2E test, can we create a unit test for _align_struct_fields for this case?

947132885 · 2025-11-22T13:12:57Z

The actual issue is that the non-struct columns were not aligned before entering _align_struct_fields, and _align_struct_fields only aligns struct columns. So a better fix is to perform the alignment before calling _align_struct_fields, rather than doing it inside _align_struct_fields itself. I’ve updated this logic in the latest commit, and also replaced the e2e test with a unit test for concat.

bveeramani · 2025-11-24T20:39:41Z

@goutamvenkat-anyscale could you take another pass on this?

goutamvenkat-anyscale

Please address the remaining comment. Thanks

…end nulls for missing fields, then reorder before struct alignment Signed-off-by: Wang Jingxin <947132885@qq.com>

…ch is much faster Signed-off-by: Wang Jingxin <947132885@qq.com>

…elds Signed-off-by: Wang Jingxin <947132885@qq.com>

…column alignment Signed-off-by: Wang Jingxin <947132885@qq.com>

Signed-off-by: Wang Jingxin <947132885@qq.com>

alexeykudinkin · 2026-01-07T18:47:18Z

+
+        # Append missing columns with null values
+        # This avoids rebuilding the whole table and only adds what's needed
+        if missing_fields:


No need for this check, just do the loop

Also can you help me understand what you're trying to achieve moving adding missing columns in here?

You're right, I'll update it.

For the missing columns, the _align_struct_fields function is used to add missing fields to struct columns. Since it also takes tables as parameters, it makes sense for it to add missing columns as well. The function name might not be very clear, though.

Also, the logic for filling back missing columns in the concat function is not correct, because it requires a chunked array instead of a plain array.

alexeykudinkin · 2026-01-07T18:52:36Z

-            else:
-                col_chunked_arrays.append(pa.nulls(block.num_rows, type=col_type))
+        # _align_struct_fields guarantees the existence of the column
+        col_chunked_arrays = [block.column(col_name) for block in blocks]


@947132885 that's a good catch!

github-actions · 2026-01-22T00:42:52Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

Signed-off-by: Wang Jingxin <947132885@qq.com>

Signed-off-by: 947132885 <947132885@qq.com>

Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com> Signed-off-by: 947132885 <947132885@qq.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Signed-off-by: Wang Jingxin <947132885@qq.com>

… scalar… (ray-project#58364) ## Description ray fails to merge blocks when there are both unaligned scalar and struct fields. ## Minimal code to reproduce: ``` import pyarrow as pa from ray.data._internal.arrow_ops.transform_pyarrow import concat table1 = pa.table({ "a": [1, 2, 3], "s": [ {"x": 7,}, {"x": 8,}, {"x": 9,}, ], }) table2 = pa.table({ "b": [4, 5, 6], }) concat([table1, table2]) ``` ## Fix: create null array in _align_struct_fields instead of assertion --------- Signed-off-by: Wang Jingxin <947132885@qq.com> Signed-off-by: 947132885 <947132885@qq.com> Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com> Signed-off-by: Adel Nour <ans9868@nyu.edu>

… scalar… (ray-project#58364) ## Description ray fails to merge blocks when there are both unaligned scalar and struct fields. ## Minimal code to reproduce: ``` import pyarrow as pa from ray.data._internal.arrow_ops.transform_pyarrow import concat table1 = pa.table({ "a": [1, 2, 3], "s": [ {"x": 7,}, {"x": 8,}, {"x": 9,}, ], }) table2 = pa.table({ "b": [4, 5, 6], }) concat([table1, table2]) ``` ## Fix: create null array in _align_struct_fields instead of assertion --------- Signed-off-by: Wang Jingxin <947132885@qq.com> Signed-off-by: 947132885 <947132885@qq.com> Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>

947132885 requested a review from a team as a code owner November 3, 2025 02:51

gemini-code-assist Bot reviewed Nov 3, 2025

View reviewed changes

947132885 force-pushed the data_concat_bugfix branch from 4857c23 to cf1d6b3 Compare November 3, 2025 02:53

ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Nov 3, 2025

947132885 force-pushed the data_concat_bugfix branch 2 times, most recently from 8375796 to c332403 Compare November 3, 2025 11:58

bveeramani requested a review from goutamvenkat-anyscale November 13, 2025 00:09

bveeramani self-assigned this Nov 13, 2025

goutamvenkat-anyscale reviewed Nov 20, 2025

View reviewed changes

gvspraveen added the @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. label Nov 21, 2025

cursor Bot reviewed Nov 22, 2025

View reviewed changes

Comment thread python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated

947132885 force-pushed the data_concat_bugfix branch from d8f2a26 to eb0f84f Compare November 22, 2025 13:05

cursor Bot reviewed Nov 22, 2025

View reviewed changes

Comment thread python/ray/data/tests/test_arrow_block.py Outdated

947132885 force-pushed the data_concat_bugfix branch from 8c2f23c to 072fb22 Compare November 22, 2025 13:21

cursor Bot reviewed Nov 22, 2025

View reviewed changes

Comment thread python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated

947132885 force-pushed the data_concat_bugfix branch 3 times, most recently from b4c3bc0 to 7370b11 Compare November 22, 2025 17:09

goutamvenkat-anyscale reviewed Nov 24, 2025

View reviewed changes

Comment thread python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated

bveeramani removed their assignment Nov 27, 2025

947132885 force-pushed the data_concat_bugfix branch from 7370b11 to a26392b Compare November 27, 2025 07:46

goutamvenkat-anyscale reviewed Nov 30, 2025

View reviewed changes

Comment thread python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated

goutamvenkat-anyscale requested changes Nov 30, 2025

View reviewed changes

947132885 force-pushed the data_concat_bugfix branch from 4d9f7e8 to e9a8e80 Compare December 1, 2025 11:24

947132885 added 5 commits December 18, 2025 23:38

[data] optimize backfilling missing columns: skip aligned blocks, app…

3d52673

…end nulls for missing fields, then reorder before struct alignment Signed-off-by: Wang Jingxin <947132885@qq.com>

[data] substitute python list comparison with block.schema.equals whi…

954626a

…ch is much faster Signed-off-by: Wang Jingxin <947132885@qq.com>

[data] incorporate table level column alignment into _align_struct_fi…

670a7ce

…elds Signed-off-by: Wang Jingxin <947132885@qq.com>

[data] drop null-array fallback and rely on _align_struct_fields for …

881c4ef

…column alignment Signed-off-by: Wang Jingxin <947132885@qq.com>

[data] assert full block concat result with expected table

7b03095

Signed-off-by: Wang Jingxin <947132885@qq.com>

947132885 force-pushed the data_concat_bugfix branch from 04df156 to 7b03095 Compare December 18, 2025 15:38

github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Dec 19, 2025

gvspraveen removed @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. labels Jan 7, 2026

gvspraveen requested a review from alexeykudinkin January 7, 2026 18:26

alexeykudinkin reviewed Jan 7, 2026

View reviewed changes

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 22, 2026

947132885 and others added 3 commits February 3, 2026 12:13

[data] no redundant if

a224e4d

Signed-off-by: Wang Jingxin <947132885@qq.com>

Merge branch 'master' into data_concat_bugfix

59f7d1e

Signed-off-by: 947132885 <947132885@qq.com>

use block_schema_field_names

4ef0314

Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com> Signed-off-by: 947132885 <947132885@qq.com>

cursor Bot reviewed Feb 3, 2026

View reviewed changes

Comment thread python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated

[data] use block_schema_field_names

9519cee

Signed-off-by: Wang Jingxin <947132885@qq.com>

github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Feb 3, 2026

iamjustinhsu assigned alexeykudinkin Feb 4, 2026

alexeykudinkin approved these changes Feb 18, 2026

View reviewed changes

alexeykudinkin merged commit 1bd59fe into ray-project:master Feb 18, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[data] fix bug that _align_struct_fields fails if there are unaligned scalar…#58364

[data] fix bug that _align_struct_fields fails if there are unaligned scalar…#58364
alexeykudinkin merged 13 commits into
ray-project:masterfrom
947132885:data_concat_bugfix

947132885 commented Nov 3, 2025 •

edited by alexeykudinkin

Loading

gemini-code-assist Bot left a comment

goutamvenkat-anyscale commented Nov 3, 2025

947132885 commented Nov 3, 2025

goutamvenkat-anyscale Nov 20, 2025

goutamvenkat-anyscale Nov 20, 2025

goutamvenkat-anyscale Nov 20, 2025

Uh oh!

947132885 commented Nov 22, 2025

Uh oh!

Uh oh!

bveeramani commented Nov 24, 2025

Uh oh!

Uh oh!

goutamvenkat-anyscale left a comment

Uh oh!

alexeykudinkin Jan 7, 2026

alexeykudinkin Jan 7, 2026

947132885 Feb 3, 2026

alexeykudinkin Jan 7, 2026

github-actions Bot commented Jan 22, 2026

cursor Bot left a comment

Uh oh!

Uh oh!

Labels

6 participants

		null_array = pa.nulls(len(block), type=schema.field(column_name).type)
		new_columns.append(null_array)

Uh oh!

Conversation

947132885 commented Nov 3, 2025 • edited by alexeykudinkin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Minimal code to reproduce:

Fix:

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

goutamvenkat-anyscale commented Nov 3, 2025

947132885 commented Nov 3, 2025

goutamvenkat-anyscale Nov 20, 2025

Choose a reason for hiding this comment

goutamvenkat-anyscale Nov 20, 2025

Choose a reason for hiding this comment

goutamvenkat-anyscale Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

947132885 commented Nov 22, 2025

Uh oh!

Uh oh!

bveeramani commented Nov 24, 2025

Uh oh!

Uh oh!

goutamvenkat-anyscale left a comment

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 7, 2026

Choose a reason for hiding this comment

alexeykudinkin Jan 7, 2026

Choose a reason for hiding this comment

947132885 Feb 3, 2026

Choose a reason for hiding this comment

alexeykudinkin Jan 7, 2026

Choose a reason for hiding this comment

github-actions Bot commented Jan 22, 2026

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Labels

6 participants

947132885 commented Nov 3, 2025 •

edited by alexeykudinkin

Loading