feat: list a table's per-row Function errors from the client (#4208)

A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.

`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.

Python and Node expose the same call, with the same optional filters.
This commit is contained in:
Wyatt Alt
2026-09-17 05:12:50 -07:00
committed by GitHub
parent c72931b30f
commit 5231d37f8d
24 changed files with 1099 additions and 4 deletions
Generated
+2 -2
View File
@@ -5368,8 +5368,8 @@ dependencies = [
[[package]]
name = "lance-geo"
version = "13.0.0-beta.3"
source = "git+https://github.com/lance-format/lance.git?tag=v13.0.0-beta.3#565808c992c3d2adf1909b9753b50a28b6bd8a19"
version = "13.0.0-beta.4"
source = "git+https://github.com/lance-format/lance.git?tag=v13.0.0-beta.4#fea665e0d96d73acd1be330ce850973ff00b7126"
dependencies = [
"datafusion",
"geo-traits",
+35
View File
@@ -578,6 +578,41 @@ so this is safe to call repeatedly.
***
### functionErrors()
```ts
abstract functionErrors(options?): Promise<FunctionErrors>
```
The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped with
the input that failed and the error. This lists those records, newest
job first, plus a summary for any fragment whose per-row detail was
capped. LanceDB Cloud and Enterprise only; reading errors needs read
access to the table, since a message carries the value that failed.
#### Parameters
* **options?**: [`FunctionErrorsOptions`](../interfaces/FunctionErrorsOptions.md)
Optional filters: `jobId`,
`column`, and `limit` (server default 10000, cap 100000).
#### Returns
`Promise`&lt;[`FunctionErrors`](../interfaces/FunctionErrors.md)&gt;
The records, the capped fragments,
and whether the listing stopped at its limit.
#### Example
```ts
const { records, truncated } = await table.functionErrors({ column: "embedding" });
```
***
### getLsmStats()
```ts
+4
View File
@@ -92,6 +92,10 @@
- [FtsToken](interfaces/FtsToken.md)
- [FullTextQuery](interfaces/FullTextQuery.md)
- [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
- [FunctionErrorFragment](interfaces/FunctionErrorFragment.md)
- [FunctionErrorRecord](interfaces/FunctionErrorRecord.md)
- [FunctionErrors](interfaces/FunctionErrors.md)
- [FunctionErrorsOptions](interfaces/FunctionErrorsOptions.md)
- [GenerationStats](interfaces/GenerationStats.md)
- [HnswPqOptions](interfaces/HnswPqOptions.md)
- [HnswSqOptions](interfaces/HnswSqOptions.md)
@@ -0,0 +1,42 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionErrorFragment
# Interface: FunctionErrorFragment
A fragment whose per-row error detail was capped: `rowsSkipped` rows
failed, of which only `rowsRecorded` have a record of their own.
## Properties
### fragmentId
```ts
fragmentId: number;
```
***
### jobId
```ts
jobId: string;
```
***
### rowsRecorded
```ts
rowsRecorded: number;
```
***
### rowsSkipped
```ts
rowsSkipped: number;
```
@@ -0,0 +1,92 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionErrorRecord
# Interface: FunctionErrorRecord
One row a Function refresh skipped, as the server recorded it.
## Properties
### column
```ts
column: string;
```
***
### createdAtMillis
```ts
createdAtMillis: number;
```
***
### errorMessage
```ts
errorMessage: string;
```
***
### errorType
```ts
errorType: string;
```
***
### fragmentId
```ts
fragmentId: number;
```
***
### function
```ts
function: string;
```
***
### functionVersion
```ts
functionVersion: string;
```
***
### jobId
```ts
jobId: string;
```
***
### rowOffset?
```ts
optional rowOffset: number;
```
The row's offset within the fragment; absent when the fragment's
detail was capped.
***
### tableVersion
```ts
tableVersion: number;
```
+39
View File
@@ -0,0 +1,39 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionErrors
# Interface: FunctionErrors
A table's per-row Function errors.
## Properties
### fragments
```ts
fragments: FunctionErrorFragment[];
```
Fragments whose detail was capped.
***
### records
```ts
records: FunctionErrorRecord[];
```
The recorded rows, newest job first.
***
### truncated
```ts
truncated: boolean;
```
Whether the listing stopped at its limit.
@@ -0,0 +1,39 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionErrorsOptions
# Interface: FunctionErrorsOptions
Which per-row Function errors to list; every filter is optional.
## Properties
### column?
```ts
optional column: string;
```
Only errors on this column.
***
### jobId?
```ts
optional jobId: string;
```
Only errors recorded by this job.
***
### limit?
```ts
optional limit: number;
```
At most this many records (server default 10000, cap 100000).
+6
View File
@@ -170,6 +170,12 @@ listing a storage directory.
::: lancedb.functions.RefreshColumnResult
::: lancedb.FunctionErrors
::: lancedb.FunctionErrorRecord
::: lancedb.FunctionErrorFragment
::: lancedb.job.Job
::: lancedb.job.AsyncJob
+60
View File
@@ -195,6 +195,66 @@ describe("remote connection", () => {
);
});
it("lists the rows a Function refresh skipped", async () => {
const bodies: unknown[] = [];
await withMockDatabase(
(req, res) => {
const path = req.url ?? "";
if (path.endsWith("/describe/")) {
res.writeHead(200, { "Content-Type": "application/json" }).end(
JSON.stringify({
name: "docs",
version: 1,
schema: { fields: [] },
}),
);
return;
}
if (path === "/v1/table/docs/errors") {
let body = "";
req.on("data", (chunk) => {
body += chunk;
});
req.on("end", () => {
bodies.push(JSON.parse(body));
res.writeHead(200, { "Content-Type": "application/json" }).end(
`{"records": [{"job_id": "j-7", "fragment_id": 3, "row_offset": 9,
"column": "embedding", "function": "embed", "function_version": "2",
"table_version": 11, "error_type": "ValueError",
"error_message": "bad input 'x'", "created_at_millis": 1700000000000}],
"fragments": [{"job_id": "j-7", "fragment_id": 4, "rows_skipped": 500,
"rows_recorded": 100}], "truncated": true}`,
);
});
return;
}
res.writeHead(404).end();
},
async (db) => {
const table = await db.openTable("docs");
const errors = await table.functionErrors({
jobId: "j-7",
column: "embedding",
limit: 2,
});
expect(errors.truncated).toBe(true);
expect(errors.records.map((r) => r.errorMessage)).toEqual([
"bad input 'x'",
]);
expect(errors.records[0].rowOffset).toBe(9);
expect(errors.fragments[0].rowsSkipped).toBe(500);
await table.functionErrors();
await expect(table.functionErrors({ limit: -1 })).rejects.toThrow(
"limit must be a non-negative integer",
);
},
);
expect(bodies).toEqual([
JSON.parse('{"job_id": "j-7", "column": "embedding", "limit": 2}'),
{},
]);
});
it("surfaces JSON server errors from remote table operations", async () => {
await withMockDatabase(
(req, res) => {
+8
View File
@@ -4358,6 +4358,14 @@ describe("computed columns", () => {
expect(rows.map((r) => r.doubled).sort()).toEqual([2, 4]);
});
it("records Function errors only on remote tables", async () => {
const db = await connect(tmpDir.name);
const table = await db.createTable("errors_local", [{ x: 1 }]);
await expect(table.functionErrors()).rejects.toThrow(
"LanceDB Cloud and Enterprise",
);
});
it("returns a job handle from refreshColumnAsync", async () => {
const db = await connect(tmpDir.name);
const table = await db.createTable("computed_job", [{ x: 1 }, { x: 2 }]);
+4
View File
@@ -56,6 +56,10 @@ export {
AddResult,
AddColumnsResult,
RefreshColumnResult,
FunctionErrors,
FunctionErrorsOptions,
FunctionErrorRecord,
FunctionErrorFragment,
RefreshMaterializedViewResult,
AlterColumnsResult,
UpdateFieldMetadataResult,
+31
View File
@@ -21,6 +21,7 @@ import { BlobFile } from "./blob";
import { EmbeddingFunctionConfig, getRegistry } from "./embedding/registry";
import { IndexOptions } from "./indices";
import { Job } from "./job";
import { validateNonNegativeInteger } from "./materialized_view";
import { MergeInsertBuilder } from "./merge";
import {
AddColumnsResult,
@@ -30,6 +31,8 @@ import {
BranchContents,
DeleteResult,
DropColumnsResult,
FunctionErrors,
FunctionErrorsOptions,
IndexConfig,
IndexStatistics,
LsmStats,
@@ -636,6 +639,27 @@ export abstract class Table {
*/
abstract refreshColumnAsync(column: string): Promise<Job>;
/**
* The per-row errors Function refreshes recorded on this table.
*
* A refresh running under a skip policy records each row it skipped with
* the input that failed and the error. This lists those records, newest
* job first, plus a summary for any fragment whose per-row detail was
* capped. LanceDB Cloud and Enterprise only; reading errors needs read
* access to the table, since a message carries the value that failed.
* @param {FunctionErrorsOptions} options Optional filters: `jobId`,
* `column`, and `limit` (server default 10000, cap 100000).
* @returns {Promise<FunctionErrors>} The records, the capped fragments,
* and whether the listing stopped at its limit.
* @example
* ```ts
* const { records, truncated } = await table.functionErrors({ column: "embedding" });
* ```
*/
abstract functionErrors(
options?: FunctionErrorsOptions,
): Promise<FunctionErrors>;
/**
* Recompute this table's contents from its materialized-view definition.
*
@@ -1363,6 +1387,13 @@ export class LocalTable extends Table {
return new Job(await this.inner.refreshColumnAsync(column));
}
async functionErrors(
options?: FunctionErrorsOptions,
): Promise<FunctionErrors> {
validateNonNegativeInteger(options?.limit, "limit");
return await this.inner.functionErrors(options);
}
async refreshMaterializedView(
full?: boolean,
sourceVersion?: number,
+117
View File
@@ -427,6 +427,32 @@ impl Table {
Ok(crate::job::Job::new(job))
}
#[napi(catch_unwind)]
pub async fn function_errors(
&self,
options: Option<FunctionErrorsOptions>,
) -> napi::Result<FunctionErrors> {
let options = options.unwrap_or_default();
let limit = options
.limit
.map(|limit| {
usize::try_from(limit)
.map_err(|_| napi::Error::from_reason("limit must be a non-negative integer"))
})
.transpose()?;
let request = lancedb::function::FunctionErrorsRequest {
job_id: options.job_id,
column: options.column,
limit,
};
let errors = self
.inner_ref()?
.function_errors(request)
.await
.default_error()?;
Ok(errors.into())
}
#[napi(catch_unwind)]
pub async fn refresh_materialized_view(
&self,
@@ -1476,6 +1502,97 @@ pub struct RefreshColumnResult {
pub version: i64,
}
/// Which per-row Function errors to list; every filter is optional.
#[napi(object)]
#[derive(Clone, Debug, Default)]
pub struct FunctionErrorsOptions {
/// Only errors recorded by this job.
pub job_id: Option<String>,
/// Only errors on this column.
pub column: Option<String>,
/// At most this many records (server default 10000, cap 100000).
pub limit: Option<i64>,
}
/// One row a Function refresh skipped, as the server recorded it.
#[napi(object)]
#[derive(Clone, Debug)]
pub struct FunctionErrorRecord {
pub job_id: String,
pub fragment_id: i64,
/// The row's offset within the fragment; absent when the fragment's
/// detail was capped.
pub row_offset: Option<i64>,
pub column: String,
pub function: String,
pub function_version: String,
pub table_version: i64,
pub error_type: String,
pub error_message: String,
pub created_at_millis: i64,
}
impl From<lancedb::function::FunctionErrorRecord> for FunctionErrorRecord {
fn from(record: lancedb::function::FunctionErrorRecord) -> Self {
Self {
job_id: record.job_id,
fragment_id: record.fragment_id as i64,
row_offset: record.row_offset.map(i64::from),
column: record.column,
function: record.function,
function_version: record.function_version,
table_version: record.table_version as i64,
error_type: record.error_type,
error_message: record.error_message,
created_at_millis: record.created_at_millis,
}
}
}
/// A fragment whose per-row error detail was capped: `rowsSkipped` rows
/// failed, of which only `rowsRecorded` have a record of their own.
#[napi(object)]
#[derive(Clone, Debug)]
pub struct FunctionErrorFragment {
pub job_id: String,
pub fragment_id: i64,
pub rows_skipped: i64,
pub rows_recorded: i64,
}
impl From<lancedb::function::FunctionErrorFragment> for FunctionErrorFragment {
fn from(fragment: lancedb::function::FunctionErrorFragment) -> Self {
Self {
job_id: fragment.job_id,
fragment_id: fragment.fragment_id as i64,
rows_skipped: fragment.rows_skipped as i64,
rows_recorded: fragment.rows_recorded as i64,
}
}
}
/// A table's per-row Function errors.
#[napi(object)]
#[derive(Clone, Debug)]
pub struct FunctionErrors {
/// The recorded rows, newest job first.
pub records: Vec<FunctionErrorRecord>,
/// Fragments whose detail was capped.
pub fragments: Vec<FunctionErrorFragment>,
/// Whether the listing stopped at its limit.
pub truncated: bool,
}
impl From<lancedb::function::FunctionErrors> for FunctionErrors {
fn from(errors: lancedb::function::FunctionErrors) -> Self {
Self {
records: errors.records.into_iter().map(Into::into).collect(),
fragments: errors.fragments.into_iter().map(Into::into).collect(),
truncated: errors.truncated,
}
}
}
#[napi(object)]
pub struct RefreshMaterializedViewResult {
/// How the view was brought up to date: "rebuild", "incremental" or "no_op".
+3
View File
@@ -12,6 +12,9 @@ __version__ = importlib.metadata.version("lancedb")
from ._lancedb import connect as lancedb_connect
from ._lancedb import FtsToken
from ._lancedb import FunctionErrorFragment as FunctionErrorFragment
from ._lancedb import FunctionErrorRecord as FunctionErrorRecord
from ._lancedb import FunctionErrors as FunctionErrors
from ._lancedb import LsmWriteSpec
from ._lancedb import tokenize as _tokenize
from .common import URI, sanitize_uri
+29
View File
@@ -461,6 +461,12 @@ class Table:
) -> AddColumnsResult: ...
async def refresh_column(self, column: str) -> RefreshColumnResult: ...
async def refresh_column_async(self, column: str) -> Job: ...
async def function_errors(
self,
job_id: Optional[str] = None,
column: Optional[str] = None,
limit: Optional[int] = None,
) -> FunctionErrors: ...
async def refresh_materialized_view(
self, full: bool = False, source_version: Optional[int] = None
) -> RefreshMaterializedViewResult: ...
@@ -820,6 +826,29 @@ class RefreshColumnResult:
rows_filled: int
version: int
class FunctionErrorRecord:
job_id: str
fragment_id: int
row_offset: Optional[int]
column: str
function: str
function_version: str
table_version: int
error_type: str
error_message: str
created_at_millis: int
class FunctionErrorFragment:
job_id: str
fragment_id: int
rows_skipped: int
rows_recorded: int
class FunctionErrors:
records: list[FunctionErrorRecord]
fragments: list[FunctionErrorFragment]
truncated: bool
class RefreshMaterializedViewResult:
@staticmethod
def from_json(value: str) -> RefreshMaterializedViewResult: ...
+10
View File
@@ -994,6 +994,16 @@ class RemoteTable(Table):
def refresh_column_async(self, column: str) -> Job[RefreshColumnResult]:
return Job(LOOP.run(self._table.refresh_column_async(column)))
def function_errors(
self,
job_id: Optional[str] = None,
column: Optional[str] = None,
limit: Optional[int] = None,
):
return LOOP.run(
self._table.function_errors(job_id=job_id, column=column, limit=limit)
)
def alter_columns(
self, *alterations: Iterable[Dict[str, str]]
) -> AlterColumnsResult:
+97
View File
@@ -186,6 +186,7 @@ if TYPE_CHECKING:
CompactionStats,
Tag,
AddColumnsResult,
FunctionErrors,
RefreshColumnResult,
AddResult,
AlterColumnsResult,
@@ -2308,6 +2309,47 @@ class Table(ABC):
'finished'
"""
@abstractmethod
def function_errors(
self,
job_id: Optional[str] = None,
column: Optional[str] = None,
limit: Optional[int] = None,
) -> "FunctionErrors":
"""
The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped
with the input that failed and the error. This lists those records,
newest job first, plus a summary for any fragment whose per-row
detail was capped. LanceDB Cloud and Enterprise only; reading errors
needs read access to the table, since a message carries the value
that failed.
Parameters
----------
job_id: str, optional
Only errors recorded by this job.
column: str, optional
Only errors on this column.
limit: int, optional
At most this many records (server default 10000, cap 100000).
Returns
-------
FunctionErrors
``records``, ``fragments`` and ``truncated``, the last saying
whether the listing stopped at its limit.
Examples
--------
>>> errors = table.function_errors(column="embedding") # doctest: +SKIP
>>> for record in errors.records: # doctest: +SKIP
... print(record.job_id, record.row_offset, record.error_message)
>>> if errors.truncated: # doctest: +SKIP
... print("listing stopped at the limit")
"""
@abstractmethod
def alter_columns(self, *alterations: Iterable[Dict[str, str]]):
"""
@@ -4362,6 +4404,18 @@ class LanceTable(Table):
"""
return Job(LOOP.run(self._table.refresh_column_async(column)))
def function_errors(
self,
job_id: Optional[str] = None,
column: Optional[str] = None,
limit: Optional[int] = None,
) -> "FunctionErrors":
"""The per-row errors Function refreshes recorded on this table. See
[`Table.function_errors`][lancedb.table.Table.function_errors]."""
return LOOP.run(
self._table.function_errors(job_id=job_id, column=column, limit=limit)
)
def alter_columns(
self, *alterations: Iterable[Dict[str, str]]
) -> AlterColumnsResult:
@@ -6428,6 +6482,49 @@ class AsyncTable:
"""
return await self._inner.refresh_column(column)
async def function_errors(
self,
job_id: Optional[str] = None,
column: Optional[str] = None,
limit: Optional[int] = None,
) -> "FunctionErrors":
"""
The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped
with the input that failed and the error. This lists those records,
newest job first, plus a summary for any fragment whose per-row
detail was capped. LanceDB Cloud and Enterprise only; reading errors
needs read access to the table, since a message carries the value
that failed.
Parameters
----------
job_id: str, optional
Only errors recorded by this job.
column: str, optional
Only errors on this column.
limit: int, optional
At most this many records (server default 10000, cap 100000).
Returns
-------
FunctionErrors
``records``, ``fragments`` and ``truncated``, the last saying
whether the listing stopped at its limit.
Examples
--------
>>> errors = await table.function_errors(column="embedding") # doctest: +SKIP
>>> for record in errors.records: # doctest: +SKIP
... print(record.job_id, record.row_offset, record.error_message)
>>> if errors.truncated: # doctest: +SKIP
... print("listing stopped at the limit")
"""
return await self._inner.function_errors(
job_id=job_id, column=column, limit=limit
)
async def refresh_column_async(
self, column: str
) -> AsyncJob[RefreshColumnJobResult]:
+65
View File
@@ -982,6 +982,71 @@ def test_remote_refresh_async_returns_typed_terminal_result():
assert result.version == 8
def test_remote_function_errors_lists_the_rows_a_refresh_skipped():
listing = {
"records": [
{
"job_id": "j-7",
"fragment_id": 3,
"row_offset": 9,
"column": "embedding",
"function": "embed",
"function_version": "2",
"table_version": 11,
"error_type": "ValueError",
"error_message": "bad input 'x'",
"created_at_millis": 1700000000000,
}
],
"fragments": [
{
"job_id": "j-7",
"fragment_id": 4,
"rows_skipped": 500,
"rows_recorded": 100,
}
],
"truncated": True,
}
bodies = []
def handler(request):
content_len = int(request.headers.get("Content-Length", 0))
body = request.rfile.read(content_len) if content_len > 0 else b""
if request.path == "/v1/table/test/errors":
bodies.append(json.loads(body))
request.send_response(200)
request.send_header("Content-Type", "application/json")
request.end_headers()
request.wfile.write(json.dumps(listing).encode())
elif request.path == "/v1/table/test/describe/":
request.send_response(200)
request.send_header("Content-Type", "application/json")
request.end_headers()
request.wfile.write(
json.dumps({"version": 1, "schema": {"fields": []}}).encode()
)
else:
request.send_response(404)
request.end_headers()
with mock_lancedb_connection(handler) as db:
table = db.open_table("test")
errors = table.function_errors(job_id="j-7", column="embedding", limit=2)
everything = table.function_errors()
assert bodies == [{"job_id": "j-7", "column": "embedding", "limit": 2}, {}]
assert errors.truncated is True
assert [r.error_message for r in errors.records] == ["bad input 'x'"]
assert errors.records[0].row_offset == 9
assert errors.records[0].function_version == "2"
assert (errors.fragments[0].rows_skipped, errors.fragments[0].rows_recorded) == (
500,
100,
)
assert everything.truncated is True
def test_remote_job_wait_raises_on_failure():
from lancedb.exceptions import JobFailedError
from lancedb.index import BTree
+7
View File
@@ -4327,6 +4327,13 @@ async def test_computed_column_async(tmp_path):
assert (await table.to_arrow())["tripled"].to_pylist() == [9]
def test_function_errors_are_remote_only(tmp_path):
db = lancedb.connect(tmp_path)
table = db.create_table("t", [{"x": 1}])
with pytest.raises(NotImplementedError, match="LanceDB Cloud and Enterprise"):
table.function_errors()
def test_refresh_column_async_returns_job(tmp_path):
db = lancedb.connect(tmp_path)
table = db.create_table("computed_job", [{"x": 1}, {"x": 2}])
+6 -2
View File
@@ -16,8 +16,9 @@ use query::{FTSQuery, HybridQuery, Query, VectorQuery};
use session::Session;
use table::{
AddColumnsResult, AddResult, AlterColumnsResult, DeleteResult, DropColumnsResult, FtsToken,
LsmWriteSpec, MergeResult, PyBlobFile, RefreshColumnResult, RefreshMaterializedViewResult,
Table, UpdateFieldMetadataResult, UpdateResult,
FunctionErrorFragment, FunctionErrorRecord, FunctionErrors, LsmWriteSpec, MergeResult,
PyBlobFile, RefreshColumnResult, RefreshMaterializedViewResult, Table,
UpdateFieldMetadataResult, UpdateResult,
};
pub mod arrow;
@@ -84,6 +85,9 @@ pub fn _lancedb(py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<RecordBatchStream>()?;
m.add_class::<AddColumnsResult>()?;
m.add_class::<RefreshColumnResult>()?;
m.add_class::<FunctionErrors>()?;
m.add_class::<FunctionErrorRecord>()?;
m.add_class::<FunctionErrorFragment>()?;
m.add_class::<RefreshMaterializedViewResult>()?;
m.add_class::<AlterColumnsResult>()?;
m.add_class::<UpdateFieldMetadataResult>()?;
+130
View File
@@ -441,6 +441,117 @@ impl From<lancedb::table::RefreshColumnResult> for RefreshColumnResult {
}
}
/// One row a Function refresh skipped, as the server recorded it.
#[pyclass(module = "lancedb._lancedb", get_all, from_py_object)]
#[derive(Clone, Debug)]
pub struct FunctionErrorRecord {
pub job_id: String,
pub fragment_id: u64,
pub row_offset: Option<u32>,
pub column: String,
pub function: String,
pub function_version: String,
pub table_version: u64,
pub error_type: String,
pub error_message: String,
pub created_at_millis: i64,
}
#[pymethods]
impl FunctionErrorRecord {
pub fn __repr__(&self) -> String {
format!(
"FunctionErrorRecord(job_id={:?}, fragment_id={}, row_offset={:?}, column={:?}, \
error_type={:?}, error_message={:?})",
self.job_id,
self.fragment_id,
self.row_offset,
self.column,
self.error_type,
self.error_message
)
}
}
impl From<lancedb::function::FunctionErrorRecord> for FunctionErrorRecord {
fn from(record: lancedb::function::FunctionErrorRecord) -> Self {
Self {
job_id: record.job_id,
fragment_id: record.fragment_id,
row_offset: record.row_offset,
column: record.column,
function: record.function,
function_version: record.function_version,
table_version: record.table_version,
error_type: record.error_type,
error_message: record.error_message,
created_at_millis: record.created_at_millis,
}
}
}
/// A fragment whose per-row error detail was capped.
#[pyclass(module = "lancedb._lancedb", get_all, from_py_object)]
#[derive(Clone, Debug)]
pub struct FunctionErrorFragment {
pub job_id: String,
pub fragment_id: u64,
pub rows_skipped: u64,
pub rows_recorded: u64,
}
#[pymethods]
impl FunctionErrorFragment {
pub fn __repr__(&self) -> String {
format!(
"FunctionErrorFragment(job_id={:?}, fragment_id={}, rows_skipped={}, rows_recorded={})",
self.job_id, self.fragment_id, self.rows_skipped, self.rows_recorded
)
}
}
impl From<lancedb::function::FunctionErrorFragment> for FunctionErrorFragment {
fn from(fragment: lancedb::function::FunctionErrorFragment) -> Self {
Self {
job_id: fragment.job_id,
fragment_id: fragment.fragment_id,
rows_skipped: fragment.rows_skipped,
rows_recorded: fragment.rows_recorded,
}
}
}
/// A table's per-row Function errors.
#[pyclass(module = "lancedb._lancedb", get_all, from_py_object)]
#[derive(Clone, Debug)]
pub struct FunctionErrors {
pub records: Vec<FunctionErrorRecord>,
pub fragments: Vec<FunctionErrorFragment>,
pub truncated: bool,
}
#[pymethods]
impl FunctionErrors {
pub fn __repr__(&self) -> String {
format!(
"FunctionErrors(records={}, fragments={}, truncated={})",
self.records.len(),
self.fragments.len(),
self.truncated
)
}
}
impl From<lancedb::function::FunctionErrors> for FunctionErrors {
fn from(errors: lancedb::function::FunctionErrors) -> Self {
Self {
records: errors.records.into_iter().map(Into::into).collect(),
fragments: errors.fragments.into_iter().map(Into::into).collect(),
truncated: errors.truncated,
}
}
}
#[pyclass(get_all, from_py_object)]
#[derive(Clone, Debug)]
pub struct RefreshMaterializedViewResult {
@@ -1638,6 +1749,25 @@ impl Table {
})
}
#[pyo3(signature = (job_id=None, column=None, limit=None))]
pub fn function_errors(
self_: PyRef<'_, Self>,
job_id: Option<String>,
column: Option<String>,
limit: Option<usize>,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
let request = lancedb::function::FunctionErrorsRequest {
job_id,
column,
limit,
};
future_into_py(self_.py(), async move {
let errors = inner.function_errors(request).await.infer_error()?;
Ok(FunctionErrors::from(errors))
})
}
#[pyo3(signature = (full=false, source_version=None))]
pub fn refresh_materialized_view(
self_: PyRef<'_, Self>,
+95
View File
@@ -731,6 +731,101 @@ pub struct RefreshColumnResult {
pub published_version: Option<u64>,
}
/// Which per-row errors [`crate::Table::function_errors`] lists. Every
/// filter is optional; the listing is table-addressed, so with none set it
/// covers every refresh of every column.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct FunctionErrorsRequest {
/// Only errors recorded by this job.
pub job_id: Option<String>,
/// Only errors on this column.
pub column: Option<String>,
/// At most this many records; the server default is 10000 and its cap
/// 100000. [`FunctionErrors::truncated`] says whether the cap was hit.
pub limit: Option<usize>,
}
impl FunctionErrorsRequest {
/// A request with no filter.
pub fn new() -> Self {
Self::default()
}
/// Only errors recorded by `job_id`.
pub fn job_id(mut self, job_id: impl Into<String>) -> Self {
self.job_id = Some(job_id.into());
self
}
/// Only errors on `column`.
pub fn column(mut self, column: impl Into<String>) -> Self {
self.column = Some(column.into());
self
}
/// At most `limit` records.
pub fn limit(mut self, limit: usize) -> Self {
self.limit = Some(limit);
self
}
}
/// One row a Function refresh skipped, as the server recorded it. The
/// message carries the input that failed, which is why reading errors needs
/// read access to the table.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FunctionErrorRecord {
/// The refresh job that recorded the error.
pub job_id: String,
/// The fragment holding the row.
pub fragment_id: u64,
/// The row's offset within the fragment; `None` when the fragment's
/// detail was capped and only the fragment summary remains.
#[serde(default)]
pub row_offset: Option<u32>,
/// The column being computed.
pub column: String,
/// The Function that failed.
pub function: String,
/// The Function's version.
pub function_version: String,
/// The table version the refresh read.
pub table_version: u64,
/// The error's class, as the executor reported it.
pub error_type: String,
/// The error's text.
pub error_message: String,
/// When the error was recorded, in milliseconds since the epoch.
pub created_at_millis: i64,
}
/// A fragment whose per-row detail was capped: `rows_skipped` rows failed,
/// of which only `rows_recorded` have a [`FunctionErrorRecord`].
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FunctionErrorFragment {
/// The refresh job that recorded the errors.
pub job_id: String,
/// The fragment.
pub fragment_id: u64,
/// Rows the refresh skipped in this fragment.
pub rows_skipped: u64,
/// Rows with a record of their own.
pub rows_recorded: u64,
}
/// A table's per-row Function errors; see [`crate::Table::function_errors`].
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct FunctionErrors {
/// The recorded rows, newest job first.
pub records: Vec<FunctionErrorRecord>,
/// Fragments whose detail was capped.
#[serde(default)]
pub fragments: Vec<FunctionErrorFragment>,
/// Whether the listing stopped at its limit.
#[serde(default)]
pub truncated: bool,
}
impl RefreshColumnResult {
/// Deprecated compatibility alias for `rows_assigned`.
pub fn rows_filled(&self) -> u64 {
+111
View File
@@ -3519,6 +3519,35 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
})))
}
async fn function_errors(
&self,
request: &crate::function::FunctionErrorsRequest,
) -> Result<crate::function::FunctionErrors> {
let mut body = serde_json::json!({});
if let Some(job_id) = &request.job_id {
body["job_id"] = serde_json::json!(job_id);
}
if let Some(column) = &request.column {
body["column"] = serde_json::json!(column);
}
if let Some(limit) = request.limit {
body["limit"] = serde_json::json!(limit);
}
self.apply_branch_body(&mut body);
let request = self
.client
.post(&format!("/v1/table/{}/errors", self.identifier))
.json(&body);
let (request_id, response) = self.send(request, true).await?;
let response = self.check_table_response(&request_id, response).await?;
let body = response.text().await.err_to_http(request_id.clone())?;
serde_json::from_str(&body).map_err(|e| Error::Http {
source: format!("Failed to parse errors response: {}", e).into(),
request_id,
status_code: None,
})
}
async fn alter_columns(&self, alterations: &[ColumnAlteration]) -> Result<AlterColumnsResult> {
self.check_mutable().await?;
let body = alterations
@@ -8188,6 +8217,88 @@ mod tests {
);
}
/// The error listing is table-addressed with optional job and column
/// filters, mirroring the server's SQL surface, and the two non-record
/// signals come back as their own fields rather than as rows.
#[tokio::test]
async fn test_function_errors_lists_the_rows_a_refresh_skipped() {
use crate::function::{FunctionErrorFragment, FunctionErrorRecord, FunctionErrorsRequest};
let table = Table::new_with_handler("my_table", |request| {
assert_eq!(request.method(), "POST");
assert_eq!(request.url().path(), "/v1/table/my_table/errors");
let body = request.body().unwrap().as_bytes().unwrap();
let value: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(
value,
serde_json::json!({"job_id": "j-7", "column": "embedding", "limit": 2})
);
http::Response::builder()
.status(200)
.body(
r#"{"records": [{"job_id": "j-7", "fragment_id": 3, "row_offset": 9,
"column": "embedding", "function": "embed", "function_version": "2",
"table_version": 11, "error_type": "ValueError",
"error_message": "bad input 'x'", "created_at_millis": 1700000000000}],
"fragments": [{"job_id": "j-7", "fragment_id": 4, "rows_skipped": 500,
"rows_recorded": 100}],
"truncated": true}"#,
)
.unwrap()
});
let errors = table
.function_errors(
FunctionErrorsRequest::new()
.job_id("j-7")
.column("embedding")
.limit(2),
)
.await
.unwrap();
assert_eq!(
errors.records,
[FunctionErrorRecord {
job_id: "j-7".into(),
fragment_id: 3,
row_offset: Some(9),
column: "embedding".into(),
function: "embed".into(),
function_version: "2".into(),
table_version: 11,
error_type: "ValueError".into(),
error_message: "bad input 'x'".into(),
created_at_millis: 1_700_000_000_000,
}]
);
assert_eq!(
errors.fragments,
[FunctionErrorFragment {
job_id: "j-7".into(),
fragment_id: 4,
rows_skipped: 500,
rows_recorded: 100,
}]
);
assert!(errors.truncated);
// No filter sends no filter, and an empty listing reads as such.
let table = Table::new_with_handler("my_table", |request| {
let body = request.body().unwrap().as_bytes().unwrap();
let value: serde_json::Value = serde_json::from_slice(body).unwrap();
assert_eq!(value, serde_json::json!({}));
http::Response::builder()
.status(200)
.body(r#"{"records": []}"#)
.unwrap()
});
let errors = table
.function_errors(FunctionErrorsRequest::new())
.await
.unwrap();
assert_eq!(errors, crate::function::FunctionErrors::default());
}
/// The refresh handle is wrapped for read-freshness tracking, so it has to
/// forward the detail APIs too -- this is the job an operator is holding
/// when a backfill goes quiet.
+67
View File
@@ -53,6 +53,7 @@ use crate::database::Database;
use crate::database::read_freshness::TableFreshness;
use crate::embeddings::{EmbeddingDefinition, EmbeddingRegistry, MemoryRegistry};
use crate::error::{Error, Result};
use crate::function::FunctionErrorsRequest;
use crate::index::IndexStatistics;
use crate::index::{Index, IndexBuilder};
use crate::index::{IndexConfig, IndexStatisticsImpl, IndexType};
@@ -821,6 +822,17 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
message: "computed columns are supported only on local tables".into(),
})
}
/// The per-row errors Function refreshes recorded on this table; see
/// [`Table::function_errors`]. The default returns `NotSupported`.
async fn function_errors(
&self,
_request: &crate::function::FunctionErrorsRequest,
) -> Result<crate::function::FunctionErrors> {
Err(Error::NotSupported {
message: "per-row Function errors are recorded only on LanceDB Cloud and Enterprise"
.into(),
})
}
/// Alter columns in the table.
async fn alter_columns(&self, alterations: &[ColumnAlteration]) -> Result<AlterColumnsResult>;
/// Drop columns from the table.
@@ -1850,6 +1862,36 @@ impl Table {
self.inner.refresh_column_async(column.as_ref()).await
}
/// The per-row errors Function refreshes recorded on this table: the
/// rows a refresh skipped under its skip policy, with the failing input
/// and the error, plus a summary for any fragment whose detail was
/// capped. Filter by job or column through the request; a listing that
/// hit its limit reports [`FunctionErrors::truncated`].
///
/// LanceDB Cloud and Enterprise only, and the caller needs read access
/// to the table, since a message carries the value that failed.
///
/// ```
/// # use lancedb::Table;
/// use lancedb::function::FunctionErrorsRequest;
///
/// # async fn list_errors(table: &Table) -> Result<(), Box<dyn std::error::Error>> {
/// let errors = table
/// .function_errors(FunctionErrorsRequest::new().column("embedding"))
/// .await?;
/// for record in &errors.records {
/// println!("{}: {}", record.error_type, record.error_message);
/// }
/// # Ok(())
/// # }
/// ```
pub async fn function_errors(
&self,
request: FunctionErrorsRequest,
) -> Result<crate::function::FunctionErrors> {
self.inner.function_errors(&request).await
}
/// Change a column's name or nullability.
pub async fn alter_columns(
&self,
@@ -4049,6 +4091,31 @@ mod tests {
assert_eq!(table.name, "test")
}
/// The per-row error store is a server feature; a local table says so
/// rather than answering with an empty listing.
#[tokio::test]
async fn test_function_errors_are_remote_only() {
let tmp_dir = tempdir().unwrap();
let conn = connect(tmp_dir.path().to_str().unwrap())
.execute()
.await
.unwrap();
let batch = make_test_batches();
let table = conn
.create_table("t", batch.clone())
.execute()
.await
.unwrap();
let err = table
.function_errors(FunctionErrorsRequest::new())
.await
.unwrap_err();
assert!(
matches!(&err, Error::NotSupported { message } if message.contains("Cloud and Enterprise")),
"{err:?}"
);
}
#[tokio::test]
async fn test_open_not_found() {
let tmp_dir = tempdir().unwrap();