[Data] Support appending a subset of columns to a Lance dataset.

[xuxiaoqiang666]

Description

Dataset.write_lance(..., mode="append") fails when the data being appended
contains only a subset of the target dataset's columns. On append, the sink
sets the write schema to the full existing dataset schema, and the positional
RecordBatchReader.from_batches(schema, ...) path then errors on the columns
the incoming blocks don't provide (reorder_columns_by_schema → Table.select
on a missing field), surfacing as a confusing LanceError / OSError instead
of anything actionable.

This PR makes APPEND align each block to the dataset schema and null-fill
the columns the block omits — the missing columns become null for the newly
appended rows, analogous to how Iceberg/Delta handle schema evolution on
append. If a block contains a column that is not in the target dataset, a
clear ValueError is raised (adding brand-new columns is out of scope; select
the existing columns first via ds.select_columns(...)).

Scope / compatibility:

• Only SaveMode.APPEND changes. CREATE / OVERWRITE keep their existing
  strict behavior (still use reorder_columns_by_schema).
• No public API change — composes with ds.select_columns(...).
• Extension types (e.g. arrow.json) and nested types are handled when
  building the null-filled columns.

Before (fails):

import ray, lance, pyarrow as pa

uri = "/tmp/lance_subset"
lance.write_dataset(
    pa.table({"a": [1, 2, 3], "b": [4, 5, 6], "c": [7, 8, 9]}), uri, mode="overwrite"
)
# Data provides only (a, b); the table is (a, b, c)
ray.data.from_items([{"a": 10, "b": 40}]).write_lance(uri, mode="append")
# -> OSError: LanceError(Arrow): ... KeyError: 'Field "c" does not exist in schema'
After (this PR): the append succeeds and the new rows have c = null.

Related issues
N/A

Additional information
Tests added in python/ray/data/tests/datasource/test_lance.py:

test_align_block_to_schema_reorders_fills_and_validates — unit test for the
alignment helper: reorders columns, null-fills missing columns, and raises on
columns not in the target schema.
test_null_column_supports_extension_and_nested_types — null-column
construction for extension (arrow.json-like) and nested types.
test_lance_write_append_fills_missing_columns_with_null — end-to-end:
appending (a, b) rows to an (a, b, c) table writes c as null.

[gemini-code-assist]

Code Review

This pull request introduces support for appending data with a subset of the target schema's columns in LanceDatasink by filling missing columns with nulls. It implements helper functions _null_column and _align_block_to_schema to align PyArrow tables to the target schema, along with comprehensive unit tests. The review feedback suggests an improvement opportunity in _align_block_to_schema to optimize column lookup complexity from O(M * N) to O(M + N) by using a set, and to pass the field object instead of field.name to tbl.append_column to preserve field metadata.

[gemini-code-assist]

Improvement Opportunity: Optimize Lookups and Preserve Field Metadata

1. Performance: tbl.schema.names is a list. Checking field.name not in tbl.schema.names inside a loop results in an $O(M \times N)$ complexity (where $M$ is the number of fields in schema and $N$ is the number of fields in tbl). Converting tbl.schema.names to a set before the loop reduces this to $O(M + N)$.
2. Metadata Preservation: Passing field (a pa.Field object) instead of field.name (a string) to tbl.append_column ensures that the exact field metadata, nullability, and other properties from the target schema are preserved rather than using default values.

Suggested change

	for field in schema:
	if field.name not in tbl.schema.names:
	tbl = tbl.append_column(
	field.name, _null_column(field.type, tbl.num_rows)
	)
	existing_names = set(tbl.schema.names)
	for field in schema:
	if field.name not in existing_names:
	tbl = tbl.append_column(
	field, _null_column(field.type, tbl.num_rows)
	)

[xuxiaoqiang666]

@abrarsheikh can you please review it?

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[github-actions]

This pull request has been automatically closed because there has been no more activity in the 14 days
since being marked stale.

Please feel free to reopen or open a new pull request if you'd still like this to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for your contribution!

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.