rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-26 21:46:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(python): honor bad-vector policies for pandas NaNs (#4503)

Browse Source 
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN
mask and Rust's default append validator ignored null children, so
`add()` silently stored incomplete vectors despite
`on_bad_vectors="error"`.

Reject null components in the existing Rust ingestion validator, and
include null components in Python's bad-vector detection and filling.
`drop` removes affected rows, `fill` repairs missing components while
preserving valid ones, and `null` replaces affected vectors. Explicit
null components in Arrow and list inputs follow the same policies. Rust
allows whole null vectors and retains the explicit
`NaNVectorBehavior::Keep` opt-out.

The ingestion guide and synchronous/asynchronous `create_table()` option
docs describe null components, all four policies, and filling that
preserves valid components.

Validation:

- Reproduced the default pandas append failure with the regression test
before changing the Rust validator.
- Rebuilt the native Python extension from the merged sources. The
Python table and utility suites passed: 481 passed, 1 skipped. Coverage
includes create/append/overwrite, pandas/rows/Arrow inputs, every
bad-vector policy, float16/32/64, sliced and chunked arrays, all-null
components, and hidden children of null lists.
- All 43 focused Rust tests passed: 7 validator tests and 36 ingestion
tests.
- Rust Clippy with `remote`, tests, and examples passed. Ruff lint
passes for the Python package, formatting checks pass for modified
Python files, and `cargo fmt --all` is clean.

- Documentation follow-up: all 56 focused vector-policy tests passed;
Python package lint and creation-docstring formatting checks passed.

Repository-wide Ruff reports 49 existing errors in untouched
documentation examples. Create/overwrite retain their existing
`RuntimeError` wrapping for errors raised while consuming lazy Arrow
readers.

Fixes #4435

<!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
This commit is contained in:
lancedb-gatefixer[bot]
2026-10-26 17:46:35 +08:00
committed by GitHub
co-authored by Xuanwo
parent 256db55a51
commit 7ed03e1c40
7 changed files with 214 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/ingestion/create-table.mdx
+5 -3
View File
@@ -148,9 +148,11 @@ non-nullable column, or provide actual nulls for it, ingestion fails; nullable c
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.
For Python ingestion, malformed vectors fail by default (`on_bad_vectors="error"`). This includes
vectors with incorrect sizes, NaNs, or null components, including pandas NaNs converted to Arrow
nulls. Use `"drop"` to remove affected rows, or `"null"` to replace affected vectors with nulls.
Use `"fill"` to replace NaN or null components with `fill_value` while preserving valid components.
This also pads short vectors with `fill_value` and truncates long vectors to the expected dimension.
### From an Arrow Table {#from-an-arrow-table}
You can also create LanceDB tables directly from Arrow tables.
python/python/lancedb/db.py
+10 -4
View File
@@ -428,10 +428,13 @@ class DBConnection(EnforceOverrides):
it will not add the provided data but will validate against any
schema that's specified.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
One of "error", "drop", "fill".
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float
The value to use when filling vectors. Only used if on_bad_vectors="fill".
NaN or null elements and missing trailing elements are replaced with this
value. Valid elements are preserved, and vectors longer than the expected
dimension are truncated.
storage_options: dict, optional
Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here.
@@ -2550,10 +2553,13 @@ class AsyncConnection(object):
it will not add the provided data but will validate against any
schema that's specified.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
One of "error", "drop", "fill".
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float
The value to use when filling vectors. Only used if on_bad_vectors="fill".
NaN or null elements and missing trailing elements are replaced with this
value. Valid elements are preserved, and vectors longer than the expected
dimension are truncated.
storage_options: dict, optional
Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here.
python/python/lancedb/table.py
+18 -30
View File
@@ -547,10 +547,11 @@ def _sanitize_data(
metadata : Optional[dict], default None
The embedding metadata to add to the schema.
on_bad_vectors : Literal["error", "drop", "fill", "null"], default "error"
What to do if any of the vectors are not the same size or contains NaNs.
How to handle vectors with incorrect sizes, NaNs, or null elements.
fill_value : float, default 0.0
The value to use when filling vectors. Only used if on_bad_vectors="fill".
All entries in the vector will be set to this value.
NaN or null elements and missing trailing elements are replaced with this
value. Valid elements are preserved.
"""
# At this point, the table might not match the schema we are targeting:
# 1. There might be embedding columns missing that will be added
@@ -1898,8 +1899,8 @@ class Table(ABC):
"append" and "overwrite". Overwrite adopts explicitly declared blob
layouts, while inferred bytes retain the existing blob v2 fields.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
One of "error", "drop", "fill".
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill".
progress: bool, callable, or tqdm-like, optional
@@ -4254,7 +4255,7 @@ class LanceTable(Table):
"append" and "overwrite". Overwrite adopts explicitly declared blob
layouts, while inferred bytes retain the existing blob v2 fields.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill".
@@ -4571,7 +4572,7 @@ class LanceTable(Table):
otherwise just open the table, it will not add the provided
data but will validate against any schema that's specified.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill".
@@ -5260,7 +5261,7 @@ def _handle_bad_vector_column(
vector_column_name: str
The name of the vector column.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float, default 0.0
The value to use when filling vectors. Only used if on_bad_vectors="fill".
@@ -5279,7 +5280,8 @@ def _handle_bad_vector_column(
data = data.set_column(position, vector_column_name, vec_arr)
if expected_value_type is not None and (
pa.types.is_integer(vec_arr.type.value_type)
pa.types.is_null(vec_arr.type.value_type)
or pa.types.is_integer(vec_arr.type.value_type)
or pa.types.is_unsigned_integer(vec_arr.type.value_type)
):
vec_arr = pa.array(vec_arr.to_pylist(), type=pa.list_(expected_value_type))
@@ -5323,7 +5325,7 @@ def _handle_bad_vector_column(
)
else:
raise ValueError(
f"Vector column '{vector_column_name}' has NaNs. "
f"Vector column '{vector_column_name}' has NaNs or null values. "
"Set on_bad_vectors='drop' to remove them, "
"set on_bad_vectors='fill' and fill_value=<value> to replace them, "
"or set on_bad_vectors='null' to replace them with null."
@@ -5359,29 +5361,11 @@ def _fill_bad_vector_values(
arr = arr.combine_chunks()
# A fixed-size slice truncates long vectors and pads short vectors with nulls.
# Slice an array marking the original child nulls in parallel so padding nulls
# can be distinguished from null values that were already present.
# Both padding and existing null elements need to be filled.
sliced = pc.list_slice(arr, 0, dim, return_fixed_size_list=True)
child_nulls = pc.is_null(arr.values)
parent_nulls = pc.is_null(arr)
if pa.types.is_list(arr.type):
original_child_nulls = pa.ListArray.from_arrays(
arr.offsets, child_nulls, mask=parent_nulls
)
elif pa.types.is_large_list(arr.type):
original_child_nulls = pa.LargeListArray.from_arrays(
arr.offsets, child_nulls, mask=parent_nulls
)
else:
original_child_nulls = pa.FixedSizeListArray.from_arrays(
child_nulls, arr.type.list_size, mask=parent_nulls
)
sliced_child_nulls = pc.list_slice(
original_child_nulls, 0, dim, return_fixed_size_list=True
)
needs_fill = pc.is_null(sliced_child_nulls.values)
values = sliced.values
needs_fill = pc.is_null(values)
if pa.types.is_floating(values.type):
values_for_nan_check = (
values.cast(pa.float32()) if pa.types.is_float16(values.type) else values
@@ -5395,6 +5379,7 @@ def _fill_bad_vector_values(
def has_nan_values(arr: Union[pa.ListArray, pa.ChunkedArray]) -> pa.BooleanArray:
"""Find vectors with NaN or null components, including pandas-converted NaNs."""
if isinstance(arr, pa.ChunkedArray):
values = pa.chunked_array([chunk.flatten() for chunk in arr.chunks])
else:
@@ -5405,6 +5390,9 @@ def has_nan_values(arr: Union[pa.ListArray, pa.ChunkedArray]) -> pa.BooleanArray
values_has_nan = pc.is_nan(values.cast(pa.float32()))
else:
values_has_nan = pc.is_nan(values)
# Pandas represents NaN components as Arrow nulls. is_nan returns null
# for these values, so count them as invalid before filtering parent indices.
values_has_nan = pc.fill_null(values_has_nan, True)
values_indices = pc.list_parent_indices(arr)
if arr.null_count:
# Match flatten(), which excludes child spans under null parent lists.
@@ -6318,7 +6306,7 @@ class AsyncTable:
"append" and "overwrite". Overwrite adopts explicitly declared blob
layouts, while inferred bytes retain the existing blob v2 fields.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill".
python/python/tests/test_table.py
+65
View File
@@ -2170,6 +2170,71 @@ def test_add_with_nans(mem_db: DBConnection):
assert np.allclose(filled_vectors[22.0], np.array([5.0, 0.0]))
@pytest.mark.parametrize("operation", ["create", "append", "overwrite"])
@pytest.mark.parametrize("input_type", ["pandas", "rows", "arrow"])
@pytest.mark.parametrize("on_bad_vectors", [None, "error", "drop", "fill", "null"])
def test_write_vectors_with_null_elements(
mem_db: DBConnection, operation, input_type, on_bad_vectors
):
schema = pa.schema({"vector": pa.list_(pa.float32(), 2), "id": pa.int64()})
vectors = [[3.0, 4.0], [np.nan, 1.0], [np.nan, np.nan]]
if input_type == "pandas":
pd = pytest.importorskip("pandas")
data = pd.DataFrame(
{
"vector": [np.array(v, dtype=np.float32) for v in vectors],
"id": [0, 1, 2],
}
)
else:
# Pandas turns the NaNs above into null child elements. Explicit nulls
# in other input formats must follow the same bad-vector policy.
data = [
{"vector": [3.0, 4.0], "id": 0},
{"vector": [None, 1.0], "id": 1},
{"vector": [None, None], "id": 2},
]
if input_type == "arrow":
data = pa.Table.from_pylist(data, schema=schema)
options = {} if on_bad_vectors is None else {"on_bad_vectors": on_bad_vectors}
if on_bad_vectors == "fill":
options["fill_value"] = 42.0
table = None
if operation != "create":
table = mem_db.create_table("test", schema=schema)
def write():
if operation == "create":
return mem_db.create_table("test", data=data, schema=schema, **options)
table.add(data, mode=operation, **options)
return table
if on_bad_vectors in (None, "error"):
# Create/overwrite can wrap errors from lazy Arrow readers in RuntimeError.
expected_error = (
ValueError if operation == "append" else (ValueError, RuntimeError)
)
with pytest.raises(expected_error, match="NaN|null"):
write()
if table is not None:
assert table.count_rows() == 0
return
table = write()
if on_bad_vectors == "drop":
expected = [{"vector": [3.0, 4.0], "id": 0}]
else:
expected = [
{"vector": [3.0, 4.0], "id": 0},
{"vector": [42.0, 1.0] if on_bad_vectors == "fill" else None, "id": 1},
{"vector": [42.0, 42.0] if on_bad_vectors == "fill" else None, "id": 2},
]
assert table.to_arrow().to_pylist() == expected
def test_add_with_empty_fixed_size_list_drops_bad_rows(mem_db: DBConnection):
class Schema(LanceModel):
text: str
python/python/tests/test_util.py
+47 -7
View File
@@ -406,7 +406,7 @@ def test_handle_bad_vectors_jagged(on_bad_vectors):
).read_all()
if on_bad_vectors == "drop":
expected = pa.array([[1.0, 2.0], [4.0, 5.0], [None, 9.0]])
expected = pa.array([[1.0, 2.0], [4.0, 5.0]])
elif on_bad_vectors == "fill":
expected = pa.array(
[
@@ -414,12 +414,12 @@ def test_handle_bad_vectors_jagged(on_bad_vectors):
[3.0, 42.0],
[4.0, 5.0],
[6.0, 7.0],
[None, 9.0],
[42.0, 9.0],
[42.0, 42.0],
]
)
elif on_bad_vectors == "null":
expected = pa.array([[1.0, 2.0], None, [4.0, 5.0], None, [None, 9.0], None])
expected = pa.array([[1.0, 2.0], None, [4.0, 5.0], None, None, None])
assert output["vector"].combine_chunks() == expected
@@ -430,17 +430,17 @@ def test_handle_bad_vectors_jagged(on_bad_vectors):
(
pa.list_(pa.float64()),
[[1.0, float("nan")], [2.0], None, [None, 3.0], [4.0, 5.0, 6.0]],
[[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [None, 3.0], [4.0, 5.0]],
[[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [42.0, 3.0], [4.0, 5.0]],
),
(
pa.large_list(pa.float64()),
[[1.0, float("nan")], [2.0], None, [None, 3.0], [4.0, 5.0, 6.0]],
[[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [None, 3.0], [4.0, 5.0]],
[[1.0, 42.0], [2.0, 42.0], [42.0, 42.0], [42.0, 3.0], [4.0, 5.0]],
),
(
pa.list_(pa.float64(), 2),
[[1.0, float("nan")], None, [None, 3.0]],
[[1.0, 42.0], [42.0, 42.0], [None, 3.0]],
[[1.0, 42.0], [42.0, 42.0], [42.0, 3.0]],
),
],
)
@@ -467,7 +467,7 @@ def test_handle_bad_vectors_nan(on_bad_vectors):
).read_all()
output = exception_output(e)
assert output == (
"ValueError: Vector column 'vector' has NaNs. Set "
"ValueError: Vector column 'vector' has NaNs or null values. Set "
"on_bad_vectors='drop' to remove them, set on_bad_vectors='fill' "
"and fill_value=<value> to replace them, or set on_bad_vectors='null' "
"to replace them with null."
@@ -490,6 +490,46 @@ def test_handle_bad_vectors_nan(on_bad_vectors):
assert output["vector"].combine_chunks() == expected
@pytest.mark.parametrize("value_type", [pa.float16(), pa.float32(), pa.float64()])
@pytest.mark.parametrize(
"list_type", [pa.list_, pa.large_list, lambda t: pa.list_(t, 2)]
)
@pytest.mark.parametrize("chunked", [False, True])
def test_has_nan_values_with_null_elements(value_type, list_type, chunked):
arr = pa.array(
[[None, None], [1.0, 2.0], [None, 3.0], None, [4.0, float("nan")], [5.0, 6.0]],
type=list_type(value_type),
).slice(1, 5)
if chunked:
arr = pa.chunked_array([arr.slice(0, 2), arr.slice(2)])
# Whole null vectors have no components to inspect. Their handling is
# separate from missing or NaN components in a valid vector.
assert has_nan_values(arr).to_pylist() == [False, True, False, True, False]
@pytest.mark.parametrize("on_bad_vectors", ["error", "drop", "fill", "null"])
def test_sanitize_vectors_with_only_null_elements(on_bad_vectors):
data = pa.table({"vector": [[None, None]]})
schema = pa.schema({"vector": pa.list_(pa.float32(), 2)})
def sanitize():
return _sanitize_data(
data, schema, on_bad_vectors=on_bad_vectors, fill_value=42.0
).read_all()
if on_bad_vectors == "error":
with pytest.raises(ValueError, match="null"):
sanitize()
return
expected = {
"drop": [],
"fill": [{"vector": [42.0, 42.0]}],
"null": [{"vector": None}],
}
assert sanitize().to_pylist() == expected[on_bad_vectors]
@pytest.mark.parametrize("dtype", [pa.float16(), pa.float32(), pa.float64()])
@pytest.mark.parametrize("large", [False, True])
@pytest.mark.parametrize("chunked", [False, True])
rust/lancedb/src/table/add_data.rs
+21 -10
View File
@@ -45,10 +45,12 @@ pub struct AddResult {
#[derive(Debug, Default, Clone, Copy)]
pub enum NaNVectorBehavior {
/// Reject any vectors containing NaN values (the default)
/// Reject any vectors containing NaN values or null elements (the default).
/// Whole null vectors are allowed.
#[default]
Error,
/// Allow NaN values to be added, but they will not be indexed for search
/// Allow NaN values and null elements to be added, but they will not be indexed
/// for search.
Keep,
}
@@ -104,11 +106,12 @@ impl AddDataBuilder {
self
}
/// Configure how to handle NaN values in vector columns.
/// Configure how to handle NaN values and null elements in vector columns.
///
/// By default, any vectors containing NaN values will be rejected with an
/// error, since NaNs cannot be indexed for search. Setting this to `Keep`
/// will allow NaN values to be added to the table, but they will not be
/// By default, any vectors containing NaN values or null elements will be
/// rejected with an error, since these vectors cannot be indexed for search.
/// Whole null vectors are allowed. Setting this to `Keep` will allow NaN values
/// and null elements to be added to the table, but these vectors will not be
/// indexed and will not be searchable.
pub fn on_nan_vectors(mut self, behavior: NaNVectorBehavior) -> Self {
self.on_nan_vectors = behavior;
@@ -953,8 +956,11 @@ mod tests {
);
}
#[rstest::rstest]
#[case::nan(Some(f32::NAN))]
#[case::null(None)]
#[tokio::test]
async fn test_add_rejects_nan_vectors() {
async fn test_add_rejects_nan_or_null_vectors(#[case] bad_value: Option<f32>) {
let schema = Arc::new(Schema::new(vec![Field::new(
"embedding",
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4),
@@ -974,7 +980,12 @@ mod tests {
FixedSizeListArray::try_new(
Arc::new(Field::new("item", DataType::Float32, true)),
4,
Arc::new(Float32Array::from(vec![0.1, 0.2, f32::NAN, 0.4])),
Arc::new(Float32Array::from(vec![
Some(0.1),
Some(0.2),
bad_value,
Some(0.4),
])),
None,
)
.unwrap(),
@@ -984,8 +995,8 @@ mod tests {
let res = table.add(batch.clone()).execute().await;
let err = res.unwrap_err();
assert!(
err.to_string().contains("NaN"),
"Expected error mentioning NaN values, but got: {err:?}"
err.to_string().contains("NaN or null"),
"Expected error mentioning NaN or null values, but got: {err:?}"
);
table
rust/lancedb/src/table/datafusion/reject_nan.rs
+48 -23
View File
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! A DataFusion projection that rejects vectors containing NaN values.
//! A DataFusion projection that rejects vectors containing NaN or null elements.
use std::sync::{Arc, LazyLock};
@@ -31,10 +31,12 @@ fn is_vector_field(field: &Field) -> bool {
}
}
/// Wraps the input plan with a projection that checks vector columns for NaN values.
/// Wraps the input plan with a projection that checks vector columns for NaN or
/// null elements.
///
/// Non-vector columns pass through unchanged. Vector columns are wrapped with a
/// UDF that returns the column as-is if no NaNs are present, or errors otherwise.
/// UDF that returns the column as-is if no NaN or null elements are present, or
/// errors otherwise. Whole null vectors pass through unchanged.
pub fn reject_nan_vectors(input: Arc<dyn ExecutionPlan>) -> Result<Arc<dyn ExecutionPlan>> {
let schema = input.schema();
let config = Arc::new(ConfigOptions::default());
@@ -70,7 +72,7 @@ pub fn reject_nan_vectors(input: Arc<dyn ExecutionPlan>) -> Result<Arc<dyn Execu
}
/// A scalar UDF that passes through FixedSizeList arrays unchanged, but errors
/// if any float values in the list are NaN.
/// if any float values in a valid list are NaN or null.
#[derive(Debug, Hash, PartialEq, Eq)]
struct RejectNanUdf {
signature: Signature,
@@ -122,11 +124,14 @@ fn check_no_nans(array: &dyn Array) -> datafusion_common::Result<()> {
)
})?;
// Only inspect elements that are both in a valid parent row and non-null
// themselves. Values backing null parent rows or null child elements may
// contain garbage (including NaN) per the Arrow spec.
let has_nan = (0..fsl.len()).filter(|i| fsl.is_valid(*i)).any(|i| {
// Values backing null parent rows may contain garbage per the Arrow spec.
// A null child in a valid row is an invalid vector component, however:
// pandas can represent NaN components as Arrow nulls.
let has_bad_values = (0..fsl.len()).filter(|i| fsl.is_valid(*i)).any(|i| {
let row = fsl.value(i);
if row.null_count() > 0 {
return true;
}
match row.data_type() {
DataType::Float16 => row
.as_any()
@@ -150,10 +155,10 @@ fn check_no_nans(array: &dyn Array) -> datafusion_common::Result<()> {
}
});
if has_nan {
if has_bad_values {
return Err(datafusion_common::DataFusionError::ArrowError(
Box::new(arrow_schema::ArrowError::ComputeError(
"Vector column contains NaN values".to_string(),
"Vector column contains NaN or null values".to_string(),
)),
None,
));
@@ -165,7 +170,7 @@ fn check_no_nans(array: &dyn Array) -> datafusion_common::Result<()> {
#[cfg(test)]
mod tests {
use super::*;
use arrow_array::Float32Array;
use arrow_array::{Float16Array, Float32Array, Float64Array};
#[test]
fn test_passes_clean_vectors() {
@@ -209,23 +214,43 @@ mod tests {
}
#[test]
fn test_skips_null_elements_within_valid_row() {
// A valid row with null child elements: the underlying buffer may hold
// NaN but the null bitmap says they're absent — should not reject.
let values = Float32Array::from(vec![
Some(1.0),
None, // null element — buffer may contain NaN
Some(3.0),
None, // null element
]);
fn test_rejects_null_elements_within_valid_row() {
let arrays: Vec<Arc<dyn Array>> = vec![
Arc::new(Float16Array::from(vec![Some(half::f16::ONE), None])),
Arc::new(Float32Array::from(vec![Some(1.0), None])),
Arc::new(Float64Array::from(vec![Some(1.0), None])),
];
for values in arrays {
let fsl = FixedSizeListArray::try_new(
Arc::new(Field::new("item", values.data_type().clone(), true)),
2,
values,
None,
)
.unwrap();
let err = check_no_nans(&fsl).unwrap_err();
assert!(err.to_string().contains("null"));
}
}
#[test]
fn test_skips_null_elements_in_null_rows_and_outside_slice() {
let fsl = FixedSizeListArray::try_new(
Arc::new(Field::new("item", DataType::Float32, true)),
2,
Arc::new(values),
None, // both rows are valid
Arc::new(Float32Array::from(vec![
None,
Some(1.0),
Some(2.0),
Some(3.0),
None,
None,
])),
Some(vec![true, true, false].into()),
)
.unwrap();
assert!(check_no_nans(&fsl).is_ok());
assert!(check_no_nans(&fsl.slice(1, 2)).is_ok());
assert!(check_no_nans(&fsl.slice(0, 2)).is_err());
}
#[test]