What happened + What you expected to happen
ds.summary() crashes with ArrowNotImplementedError when the dataset contains boolean columns.
Expected: summary() should handle boolean columns without crashing.
How to reproduce
import pyarrow as pa
import pyarrow.compute as pac
bool_array = pa.array([True, False, True])
mean = pac.mean(bool_array).as_py()
pac.subtract(bool_array, mean) # ArrowNotImplementedError: no kernel for subtract(bool, double)
pac.quantile(bool_array, q=0.5) # ArrowNotImplementedError: no kernel for quantile(bool)
pac.equal(bool_array, 0) # ArrowNotImplementedError: no kernel for equal(bool, int64)
Root cause
In stats.py, DataType.bool() is mapped to _numerical_aggregators, which includes all 8 numeric aggregators. Three of them crash on boolean columns because PyArrow has no kernels for the underlying operations:
| Aggregator |
Operation |
Error |
Std |
pac.subtract(bool, double) in arrow_block.py:sum_of_squared_diffs_from_mean |
no kernel for subtract(bool, double) |
ApproximateQuantile |
pac.quantile(bool) in arrow_block.py:quantile |
no kernel for quantile(bool) |
ZeroPercentage |
pc.equal(bool, int64) in aggregate.py:aggregate_block |
no kernel for equal(bool, int64) |
Only Count, Mean, Min, Max, and MissingValuePercentage work on booleans.
Suggested fix
Give DataType.bool() its own aggregator list that excludes Std, ApproximateQuantile, and ZeroPercentage.
Versions / Dependencies
Ray 2.46+
What happened + What you expected to happen
ds.summary()crashes withArrowNotImplementedErrorwhen the dataset contains boolean columns.Expected:
summary()should handle boolean columns without crashing.How to reproduce
Root cause
In
stats.py,DataType.bool()is mapped to_numerical_aggregators, which includes all 8 numeric aggregators. Three of them crash on boolean columns because PyArrow has no kernels for the underlying operations:Stdpac.subtract(bool, double)inarrow_block.py:sum_of_squared_diffs_from_meanno kernel for subtract(bool, double)ApproximateQuantilepac.quantile(bool)inarrow_block.py:quantileno kernel for quantile(bool)ZeroPercentagepc.equal(bool, int64)inaggregate.py:aggregate_blockno kernel for equal(bool, int64)Only
Count,Mean,Min,Max, andMissingValuePercentagework on booleans.Suggested fix
Give
DataType.bool()its own aggregator list that excludesStd,ApproximateQuantile, andZeroPercentage.Versions / Dependencies
Ray 2.46+