[Train] Handle Arrow-backed pandas dtypes in LightGBM examples

[pseudo-rnd-thoughts]

Description

#63017 updated Ray Data's Arrow-to-pandas conversion to preserve Arrow-backed pandas dtypes, such as int64[pyarrow], so dtypes can roundtrip more faithfully.

This exposed an incompatibility with LightGBM's pandas input path. Ray Train's LightGBM examples and legacy trainer code convert Ray Data shards to pandas before constructing lightgbm.Dataset. With Arrow-backed Ray Data inputs, those pandas DataFrames can now contain Arrow-backed dtypes, and LightGBM rejects them during pandas dtype validation even when the logical column type is numeric.

This PR updates the LightGBM paths to normalize pandas DataFrames to NumPy-nullable pandas dtypes before passing them to LightGBM. It also updates documentation and examples to show the same conversion for user-authored LightGBM training loops.

Changes

• Normalize pandas DataFrames in the legacy LightGBMTrainer path before constructing lightgbm.Dataset.
• Update LightGBM docs and docstring examples to call convert_dtypes(dtype_backend="numpy_nullable") after to_pandas().
• Add a regression test covering Arrow-backed Ray Data input from ray.data.from_items(...).
• Keep the restore test focused on trainer restore behavior by using a pandas-backed test dataset.
• Update the LightGBM release benchmark to avoid passing Arrow-backed pandas dtypes to LightGBM.

[gemini-code-assist]

Code Review

This pull request introduces the use of convert_dtypes() across LightGBM trainer implementations, documentation examples, and tests to ensure compatibility with Arrow-backed pandas DataFrames from Ray Data. It also re-enables the test_trainer_restore test. The review feedback recommends simplifying the code by removing redundant materialize() calls and omitting the explicit dtype_backend argument to maintain compatibility with pandas versions prior to 2.0.0. Additionally, the reviewer suggests applying these fixes to the legacy LightGBM trainer and warns about potential performance overhead when using convert_dtypes() on very large datasets.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 17abd91. Configure here.}

[goutamvenkat-anyscale]

do we have benchmarks on this function comparison?

[pseudo-rnd-thoughts]

Using lightgbm_train_batch_inference_benchmark_100G's release tests then I've tested both option which shows almost no difference so I would suspect that we don't have a good test to check this currently

normalize_pandas_for_lightgbm
real    4m35.544s
user    0m22.508s
sys     0m4.320s

df.convert_dtypes(dtype_backend="numpy_nullable")
real    4m35.533s
user    0m22.264s
sys     0m4.343s

[TimothySeah]

If that's the case should we either:

1. Just tell users to use df.convert_dtypes(dtype_backend="numpy_nullable") instead of adding a new API? That seems more stable than this API e.g. what if lightgbm starts accepting timestamp in the future?
2. If we still want this API, should we still call out that it's faster than df.convert_dtypes(dtype_backend="numpy_nullable") when the benchmark shows a negligible difference?

[TimothySeah]

Approving to unblock for now - main question is why not just use df.convert_dtypes(dtype_backend="numpy_nullable")?

[TimothySeah]

If that's the case should we either:

1. Just tell users to use df.convert_dtypes(dtype_backend="numpy_nullable") instead of adding a new API? That seems more stable than this API e.g. what if lightgbm starts accepting timestamp in the future?
2. If we still want this API, should we still call out that it's faster than df.convert_dtypes(dtype_backend="numpy_nullable") when the benchmark shows a negligible difference?

[TimothySeah]

nit: does https://arrow.apache.org/docs/python/generated/pyarrow.types.is_integer.html work or do we need to explicitly check both is_signed and is_unsigned?