mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-17 16:14:00 +00:00
docs: clarify Function workflows and search contracts
Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
---
|
||||
name: lancedb-vector-dedup
|
||||
description: Plan and run Enterprise IVF vector deduplication with dedupe and duplicate_pairs, including sampling, index coverage, recall and resource limits. Use for near-duplicate embedding or perceptual-hash records; use MinHash guidance for text shingles.
|
||||
---
|
||||
|
||||
# Vector deduplication
|
||||
|
||||
Read the maintained [workflow](../../../docs/web/curation/dedupe.mdx),
|
||||
[DEDUPE reference](../../../docs/web/sql/dql/dedupe.mdx) and
|
||||
[DUPLICATE_PAIRS reference](../../../docs/web/sql/dql/duplicate-pairs.mdx)
|
||||
before constructing SQL. In a source checkout these paths are relative to this
|
||||
skill directory. When using an installed copy without the docs, read
|
||||
`https://docs.lancedb.com/curation/dedupe` and the linked SQL references.
|
||||
|
||||
## Establish the comparison contract
|
||||
|
||||
- Confirm an Enterprise deployment that supports these table functions. Use
|
||||
`dedupe` and `duplicate_pairs`; legacy `vector_dedup` names are rejected.
|
||||
- Inspect the vector type, metric, compatible index, indexed fragments and
|
||||
source version. Wait for index coverage after writes. Do not run a full
|
||||
scan over unindexed fragments or silently change the index to make it run.
|
||||
- Explain that candidates are compared only within physical IVF partitions
|
||||
and segments. Encoded/quantized distances are not an exact raw-vector
|
||||
comparison. L2 thresholds are squared distances; Hamming counts bits.
|
||||
- Use `id_column` on pair queries for business IDs. Retain `dataset_version`,
|
||||
segment and partition identities. Do not decode internal row IDs or assume
|
||||
that representatives stay the same after later data/index changes.
|
||||
|
||||
## Bound the first experiment
|
||||
|
||||
Use a reproducible partition sample before a full scan:
|
||||
|
||||
```sql
|
||||
SELECT id_a, id_b, distance, dataset_version, segment_id, partition_id
|
||||
FROM duplicate_pairs(
|
||||
'frames', 'embedding',
|
||||
distance_threshold => 0.02,
|
||||
id_column => 'frame_id',
|
||||
selection => 'random', count => 10, seed => 42
|
||||
)
|
||||
```
|
||||
|
||||
Adapt the threshold and sample size to the metric and actual partition count.
|
||||
Sampling partitions is not uniform sampling of rows. Inspect known duplicate
|
||||
and distinct examples, pair density, partition skew, memory, spill space and
|
||||
elapsed time. Do not infer global recall or ten-billion-row qualification
|
||||
from a small or unversioned benchmark. Pair work and output can be quadratic
|
||||
within a partition; an outer SQL `LIMIT` is not an upstream work budget.
|
||||
|
||||
## Choose the result
|
||||
|
||||
`dedupe` returns original representative rows and always scans all partitions.
|
||||
Use its default `connected_components` for transitive groups; choose
|
||||
`direct_representative` when chains should not merge transitively. It does not
|
||||
accept `id_column`, `selection`, `count` or `seed`; `duplicate_pairs` does not
|
||||
accept `resolution`.
|
||||
|
||||
Keep pair data on the server when possible; avoid collecting an unbounded
|
||||
edge set in client memory. Persist a new table/view only within the user's
|
||||
authorized scope, preserving source rows unless deletion was requested.
|
||||
Do not register a Python Function or fetch/rewrite Blob payloads to invoke
|
||||
these built-in table functions. For MinHash/text deduplication, follow the
|
||||
separate `https://docs.lancedb.com/enterprise/sql-minhash` workflow.
|
||||
@@ -33,6 +33,46 @@ print(job.status())
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Cached properties and live calls {#snapshots}
|
||||
|
||||
Python job properties describe the last observation; reading them does not
|
||||
contact the server. Submission need not fetch the full registry record, so
|
||||
some properties can initially be `None`.
|
||||
|
||||
| Surface | Contract |
|
||||
| --- | --- |
|
||||
| `state` | Last observed lifecycle state. It does not update on its own. |
|
||||
| `job_type`, `creation_ms` | Cached job kind and creation time in epoch milliseconds. |
|
||||
| `spec` | Complete, job-specific submitted specification. Read this separately from the bounded `repr(job)` display. |
|
||||
| `result` | Cached, job-specific reported data; it is not necessarily the typed object returned by `wait()`. |
|
||||
| `failure` | Cached `JobFailureInfo`, when the server reports a terminal failure. |
|
||||
| `status()` | Live lifecycle request. For registry jobs it also caches the full record. A failed state is returned without raising a job-failure exception; request errors can still raise. |
|
||||
| `refresh()` | Fetch and cache the current record without waiting for completion. |
|
||||
| `events(...)` | Separate live request returning a `pyarrow.Table` of recorded events; it does not refresh the properties. |
|
||||
| `wait(...)` | Poll until completion and return the operation's typed result, or raise on failure/cancellation. Refresh afterward when you need the complete final registry record. |
|
||||
|
||||
For example, poll a registry handle and then retrieve its final details:
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
state = job.status()
|
||||
while state == "running":
|
||||
print(job.state)
|
||||
time.sleep(2)
|
||||
state = job.status()
|
||||
|
||||
result = job.wait()
|
||||
job.refresh()
|
||||
print(result)
|
||||
print(job.spec)
|
||||
```
|
||||
|
||||
The four lifecycle strings below are the usual SDK contract; newer server
|
||||
states may pass through an older client. Do not infer execution progress from
|
||||
`state` alone. Structured progress methods and queue fields require matching
|
||||
SDK/server support; see [refresh admission](/feature-engineering/refresh-column#admission).
|
||||
|
||||
## Submit and wait for a job {#waiting}
|
||||
|
||||
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
|
||||
@@ -133,6 +173,39 @@ for info in db.list_jobs(page_limit=50):
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`list_jobs` yields `JobInfo` summaries, newest creation time first, rather than
|
||||
live `Job` handles. Open an ID to inspect its full record. `page_limit` bounds
|
||||
each server request, not the total yielded items, and `page_token` resumes a
|
||||
listing. It is a lazy iterator: iteration performs requests and can raise
|
||||
request errors. To materialize it, use `list(db.list_jobs(...))`; it does not
|
||||
support list indexing, `len()`, or a `.to_list()` method.
|
||||
|
||||
`Listing.num_page_results()` reports cached items remaining in the current
|
||||
page and `Listing.page_token()` reports the next request's token. Drain the
|
||||
cached items before saving that token, or resuming can skip them. A `None`
|
||||
token after the final page can coexist with unread cached items. A resumed
|
||||
listing is not a transactionally frozen view of jobs submitted in the meantime.
|
||||
|
||||
`events.schema` describes the Arrow response. Use `events.to_pylist()` for
|
||||
Python row dictionaries. Common event columns are `event_seq` (int64), `state`
|
||||
(string), `updated_at_millis` (int64), `updated_by` (string), and nullable
|
||||
`emitted_from`, `emitted_by` and `claim_entity` strings. Optional uint64 work
|
||||
counters such as `rows_processed` and diagnostic details depend on job type
|
||||
and server version. If a `payload` column is returned, its value is a nullable
|
||||
JSON string, not an already decoded Python dictionary.
|
||||
Empty results retain their schema. Events are newest first and belong to the
|
||||
requested job; child events are separate. An event count is not a completed-row
|
||||
count, and a running job can have no terminal `result` while producing events.
|
||||
|
||||
For a Function refresh, `wait()` returns a `RefreshColumnResult` such as
|
||||
`rows_assigned`, `rows_failed`, `rows_remaining`, `source_version` and optional
|
||||
`published_version`. Other job types have different results. Do not parse the
|
||||
printed representation to recover a spec, result or progress measurement.
|
||||
For object-shaped registry data, `job.spec` and `job.result` are decoded Python
|
||||
dictionaries: access fields with keys, such as `job.result.get("summary")`
|
||||
after refreshing a successful Function job. The separate typed object from
|
||||
`wait()` uses attributes, such as `result.rows_assigned`.
|
||||
|
||||
Event responses have [row limits](/basics/limits#jobs). An explicit limit can
|
||||
request more than the default, but it does not guarantee a complete event
|
||||
history.
|
||||
|
||||
@@ -44,6 +44,58 @@ table.add_columns({"doubled": version(value=col("value"))})
|
||||
immutable version, so changing one produces a new version rather than
|
||||
altering rows a previous version already filled.
|
||||
|
||||
## Supported Function signatures {#signature-types}
|
||||
|
||||
A table's [Arrow schema](/sql/data-types#supported) can contain types that a
|
||||
Function cannot accept or return. The matrix below describes the Python SDK
|
||||
`0.42.0-beta.12` source contract. Nested schemas and list-Blob outputs require
|
||||
a matching Enterprise server; an SDK upgrade alone does not enable them.
|
||||
|
||||
| Arrow type | Function input/output support |
|
||||
| --- | --- |
|
||||
| Boolean; signed/unsigned 8, 16, 32, 64-bit integers; float16/32/64 | Supported. Ordinary `int` and `float` annotations infer int64 and float64. |
|
||||
| utf8 / large_utf8; binary / large_binary | Supported. `str` and `bytes` infer utf8 and binary. Plain binary does not carry Blob semantics. |
|
||||
| date32 / date64 | Supported; Python `date` infers date32. |
|
||||
| timestamp, with or without timezone; time32/64; duration; interval | Unsupported by the Function signature grammar. A Python `datetime` annotation can resolve to an Arrow timestamp but registration still rejects it. |
|
||||
| decimal; map; dictionary; union; fixed-size binary | Unsupported. Decode dictionary values and use supported scalar/list/struct representations. |
|
||||
| struct; list / large_list; positive-length fixed-size list | Supported with supported children. Explicit schemas preserve child nullability; annotation-only `list[T]` uses non-nullable elements. Struct fields must be nonempty and uniquely named. |
|
||||
| Blob v2 at top level or in a struct | Supported in inputs and outputs; use `lancedb.blob(...)` in the schema. |
|
||||
| Blob v2 under list / large_list | Output only on supporting SDK/server versions. Inputs are rejected. |
|
||||
| Blob v2 under fixed-size list or map | Unsupported in both directions. |
|
||||
|
||||
For times, pass an int64 epoch value with an explicit unit such as microseconds,
|
||||
or a UTF-8 ISO 8601 string with its timezone. Convert inside the Function and
|
||||
cast the result in SQL if needed. Use scaled integers with a declared scale
|
||||
when exact decimal values matter; converting to float can lose precision.
|
||||
Represent a map as a list of key/value structs when that fits the calculation.
|
||||
|
||||
A scalar output is non-nullable because physical NULL represents an unfilled
|
||||
computed output. For an optional value, return a named struct with a nullable
|
||||
field, for example:
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
from lancedb import function
|
||||
|
||||
@function(
|
||||
input_schema=pa.schema([pa.field("text", pa.string(), nullable=False)]),
|
||||
output_schema=pa.schema([pa.field("score", pa.float64(), nullable=True)]),
|
||||
)
|
||||
def optional_score(text: str):
|
||||
return {"score": float(len(text)) if text else None}
|
||||
```
|
||||
|
||||
Return a dictionary by field name. A successfully computed nullable field can
|
||||
be NULL; `score IS NULL` alone cannot distinguish it from pending work or a
|
||||
skipped failure. Use the job result and error diagnostics. Do not assume an
|
||||
explicit assignment-state flag is available on every deployment.
|
||||
|
||||
List-nested Blob outputs are queryable as descriptors, but row-ID blob fetch
|
||||
APIs and `blob_mode="bytes"` do not fetch payloads inside lists. Prefer a
|
||||
separate row per media item when you need independent reads, retries or
|
||||
re-scoring. A plain struct shaped like a Blob without its extension metadata
|
||||
is not a substitute for `lancedb.blob`.
|
||||
|
||||
## Batch: once per call, many rows at a time {#batch}
|
||||
|
||||
A function is called once per batch instead of once per row when its
|
||||
@@ -170,9 +222,12 @@ in the function's source. A Secret is a database-scoped named credential:
|
||||
puts it in the environment variable the function reads, without ever
|
||||
returning it from any API or recording it in the function's own definition.
|
||||
Combine it with the stateful form so the HTTP client -- and any rate limiter
|
||||
around it -- is built once, not reopened on every row. There is no built-in
|
||||
retry or per-row error-skip policy -- a function that calls a flaky service
|
||||
should handle timeouts and retries itself, inside `__call__`.
|
||||
around it -- is built once, not reopened on every row. Configure HTTP timeouts
|
||||
inside `__call__`. Refresh-level `error_policy='skip'` can record row failures;
|
||||
see [handling failures](/management/function#failures). Native retry policies
|
||||
need matching SDK, server and Function-image support; on other deployments,
|
||||
handle bounded retries inside the Function only when repeating the request is
|
||||
safe.
|
||||
|
||||
See [Vend a Secret to a Function](/curation/vend-secret#specify) for the
|
||||
full worked example, how the bound value actually reaches a running
|
||||
|
||||
@@ -136,12 +136,60 @@ SELECT * FROM dedupe(
|
||||
See [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
|
||||
for manual and automatic refresh options to keep it current as new records land.
|
||||
|
||||
## At production scale {#production}
|
||||
## Index, snapshot and recall limits {#limits}
|
||||
|
||||
[LeRobotDataset](/integrations/lerobotdataset) runs the same `dedupe`
|
||||
call over 27.6 million real embedding rows to remove near-static frames from a
|
||||
robotics training set in under 30 seconds -- the mechanism does not change
|
||||
with scale, only the table it runs against.
|
||||
These are Enterprise SQL table functions, not standard SQL or a registered
|
||||
Python Function. They require one compatible current vector index covering
|
||||
the scanned snapshot. Wait for indexing to finish after ingestion and inspect
|
||||
index coverage before running a full scan. Unindexed fragments are rejected;
|
||||
deleted rows are excluded at the pinned dataset version.
|
||||
|
||||
The scan compares candidates **within each physical IVF partition and index
|
||||
segment**. It does not compare across partitions or segments. Scanning every
|
||||
partition therefore does not establish global all-pairs recall. Quantized
|
||||
indexes compare their encoded vector representations; `IVF_FLAT` compares the
|
||||
stored vectors. Thresholds must match the index's metric: L2 is squared L2,
|
||||
cosine is `1 - cosine`, dot distance is `1 - dot`, and Hamming counts differing
|
||||
bits. Inspect close examples and known duplicates before choosing a threshold.
|
||||
|
||||
`duplicate_pairs` reports `dataset_version`, `segment_id` and `partition_id`.
|
||||
Retain those with the index configuration and source version when comparing
|
||||
runs. Pass `id_column` for application IDs, rather than treating physical row
|
||||
addresses as durable identifiers. A new append, index change or bridge between
|
||||
components can change groups and representatives.
|
||||
|
||||
## Efficient use and scale qualification {#production}
|
||||
|
||||
Start with [`duplicate_pairs` partition sampling](/sql/dql/duplicate-pairs#sampling),
|
||||
using a recorded seed and a bounded partition count. Inspect pair density and
|
||||
threshold quality before running `dedupe`, which always scans every partition.
|
||||
Sampling partitions is not uniform row sampling and can miss rare groups.
|
||||
`resolution='connected_components'` merges transitive chains;
|
||||
`direct_representative` uses direct representative relationships instead.
|
||||
|
||||
Pair work grows quadratically within dense partitions, and a loose threshold
|
||||
can emit a quadratic number of edges. An outer `LIMIT` is not a resource
|
||||
budget for upstream pair finding or group resolution. Avoid repeatedly
|
||||
materializing all pairs in a Python list; select needed columns, aggregate on
|
||||
the server, and persist a reviewed result when repeated reads justify it.
|
||||
Measure peak memory, spill storage, elapsed time and pair count on
|
||||
representative partitions before increasing scope.
|
||||
|
||||
The [LeRobotDataset integration](/integrations/lerobotdataset) illustrates a
|
||||
27.6-million-row data workflow. Its timings are not a versioned qualification
|
||||
of this deduplication path on arbitrary indexes or hardware. A 10-billion-row
|
||||
run is not qualified by that example. Qualify source/index versions, partition
|
||||
sizes, metric, threshold, worker resources, spill limits and correctness on your
|
||||
deployment before extrapolating. There is no universal row-count or latency
|
||||
guarantee in this guide.
|
||||
|
||||
For text shingles and MinHash signatures, use the distinct
|
||||
[MinHash workflow](/enterprise/sql-minhash). Vector distance deduplication and
|
||||
MinHash/Jaccard text deduplication have different indexes and thresholds.
|
||||
|
||||
Agents can follow the repository's
|
||||
[vector deduplication skill](https://github.com/lancedb/lancedb/blob/main/.agents/skills/lancedb-vector-dedup/SKILL.md)
|
||||
to apply these checks without inventing Function options or scale guarantees.
|
||||
|
||||
## Continue with the curated data {#next}
|
||||
|
||||
|
||||
@@ -496,6 +496,7 @@
|
||||
"management/materialized-view",
|
||||
"management/secret",
|
||||
"management/function",
|
||||
"management/function-failures",
|
||||
"management/storage-integration"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -13,6 +13,13 @@ import {
|
||||
PySchemaAddSetup,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
<Note>
|
||||
Declare computed columns on ordinary tables. A materialized view derives its
|
||||
schema from its query; include the calculation in the view's `SELECT` instead
|
||||
of attaching a Function with `add_columns`. See
|
||||
[materialized-view schema changes](/feature-engineering/materialized-views#schema-changes).
|
||||
</Note>
|
||||
|
||||
Lance can write a new column without rewriting the columns already in the
|
||||
table. That matters most on a table that also holds large images, video or
|
||||
embeddings, where rewriting everything just to add one small derived field
|
||||
|
||||
@@ -176,6 +176,30 @@ definition is read with `await view.definition()`.
|
||||
Creation, refresh and drop also have async, job-returning SDK methods; see
|
||||
[Materialized view](/management/materialized-view#manage).
|
||||
|
||||
## Add derived values to a view {#schema-changes}
|
||||
|
||||
A materialized view's output schema belongs to its defining `SELECT`. Do not
|
||||
use `view.table.add_columns(...)` to attach another computed or Function column
|
||||
to that output. In particular, adding a Function binding is rejected: the
|
||||
column would not be produced by the next view refresh.
|
||||
|
||||
Choose where the calculation belongs:
|
||||
|
||||
- Include the expression or Function call in the view's `SELECT`, as in the
|
||||
[caption example](#functions). To change an existing definition, create a
|
||||
replacement view under a new name, check its results, indexes and grants,
|
||||
and move readers to it before dropping the old view.
|
||||
- Add a Function column to the ordinary source table, refresh it, then select
|
||||
that stored column in a view. This lets you manage column refresh separately
|
||||
from view refresh.
|
||||
- Create an ordinary table from a query when you need independently editable
|
||||
outputs. That table is a snapshot; it does not inherit the view's refresh
|
||||
relationship to the source.
|
||||
|
||||
Opening the underlying table gives access to reads and indexes; it does not
|
||||
make independent schema changes part of the materialized-view definition.
|
||||
Changing automatic-refresh settings also does not change that definition.
|
||||
|
||||
## Related guides {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
@@ -77,6 +77,11 @@ catch-up works when it's re-enabled after being off.
|
||||
|
||||
## Refreshing part of a table {#partial}
|
||||
|
||||
Python `where=` requires a supporting SDK, such as `0.42.0b9`, and an
|
||||
Enterprise server with filtered refresh support. Stable `0.41.0` does not
|
||||
accept this argument. The SQL form is available independently of that Python
|
||||
argument on supporting servers.
|
||||
|
||||
<Badge color="red">Enterprise</Badge> A refresh can be scoped to a predicate,
|
||||
with `where=` in Python or `WHERE` in SQL, which is useful for filling a slice
|
||||
first and checking it before committing to the whole table:
|
||||
@@ -123,6 +128,89 @@ to let LanceDB track which outputs need updating.
|
||||
`function_refresh` for one filled by a registered [function](/management/function).
|
||||
</Note>
|
||||
|
||||
### Re-score selected list elements {#list-elements}
|
||||
|
||||
`WHERE` selects **rows**, not elements within a list. Refreshing a row that
|
||||
contains ten answers passes the whole list to the Function. A mask can control
|
||||
which elements your code computes, but the Function still returns one complete
|
||||
list for that row. There is no element-level refresh or partial-list write API.
|
||||
|
||||
This example keeps `sample_id` as the link to an existing `samples` table with
|
||||
a Blob column named `media`. It creates only answer metadata and scores; it
|
||||
neither reads nor copies Blob payloads. Run it on a deployment supporting
|
||||
`@function`, list signatures and Python `where=`, with `samples` already
|
||||
containing sample IDs `s1` and `s2`.
|
||||
|
||||
```python
|
||||
from lancedb import col, function
|
||||
|
||||
media_before = db.execute_query(
|
||||
"SELECT sample_id, media FROM samples "
|
||||
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
|
||||
).read_all().to_pylist()
|
||||
|
||||
answers = db.create_table("answer_rescore_demo", [
|
||||
{"group_id": "g1", "sample_id": "s1",
|
||||
"answers": ["yes", "maybe", "no"],
|
||||
"selected": [False, True, False], "previous": [0.1, 0.2, 0.3]},
|
||||
{"group_id": "g1", "sample_id": "s2",
|
||||
"answers": ["other"], "selected": [False], "previous": [0.4]},
|
||||
])
|
||||
|
||||
@function
|
||||
def rescore_masked(
|
||||
answers: list[str], selected: list[bool], previous: list[float]
|
||||
) -> list[float]:
|
||||
if not (len(answers) == len(selected) == len(previous)):
|
||||
raise ValueError("answers, selected and previous must have equal lengths")
|
||||
return [
|
||||
float(len(answer)) if choose else old
|
||||
for answer, choose, old in zip(answers, selected, previous)
|
||||
]
|
||||
|
||||
version = db.create_function(rescore_masked)
|
||||
answers.add_columns(
|
||||
{"rescored": version(
|
||||
answers=col("answers"), selected=col("selected"), previous=col("previous")
|
||||
)},
|
||||
auto_refresh_enabled=False,
|
||||
)
|
||||
job = answers.refresh_column_async("rescored", where="sample_id = 's1'")
|
||||
job.wait()
|
||||
|
||||
rows = answers.search().select(
|
||||
["group_id", "sample_id", "previous", "rescored"]
|
||||
).to_list()
|
||||
by_sample = {row["sample_id"]: row for row in rows}
|
||||
assert by_sample["s1"]["group_id"] == "g1"
|
||||
assert by_sample["s1"]["rescored"] == [0.1, 5.0, 0.3]
|
||||
assert by_sample["s1"]["previous"] == [0.1, 0.2, 0.3]
|
||||
assert by_sample["s2"]["rescored"] is None
|
||||
assert by_sample["s2"]["previous"] == [0.4]
|
||||
|
||||
media_after = db.execute_query(
|
||||
"SELECT sample_id, media FROM samples "
|
||||
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
|
||||
).read_all().to_pylist()
|
||||
assert media_after == media_before
|
||||
```
|
||||
|
||||
Here length is a deterministic stand-in for a scorer. Only `"maybe"` is scored;
|
||||
the other scores are copied from `previous`. The final equality checks Blob
|
||||
descriptors on an otherwise unchanged source table, without fetching payloads.
|
||||
Keep the source table unchanged during this comparison. Use the same model,
|
||||
normalization and score type as the original computation in a real re-scoring
|
||||
pass. For another pass, explicitly supply the intended prior scores and mask;
|
||||
refresh does not merge the previous output into your Function's inputs.
|
||||
|
||||
If answers need independent retries or versions, prefer one ordinary table row
|
||||
per answer, with `group_id`, `sample_id`, a stable application `answer_id`,
|
||||
answer text and score. Keep media in `samples` and join by `sample_id` when
|
||||
needed. A row predicate can then select exact answer IDs, and Blob storage
|
||||
stays in the source table. Do not use `UNNEST` to assume that nested Blob values
|
||||
become supported Function inputs; see the
|
||||
[nested-Blob limits](/curation/create-function#signature-types).
|
||||
|
||||
## Run a refresh on Enterprise {#async}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
@@ -212,6 +300,31 @@ The wait helpers do not submit or cancel a refresh. They retry only the typed
|
||||
finalization-pending response and propagate other errors.
|
||||
*/}
|
||||
|
||||
### Admission and waiting {#admission}
|
||||
|
||||
Submission does not reserve a supervisor or GPU immediately. A refresh can wait
|
||||
for source preparation/compaction, an earlier refresh, executor admission or
|
||||
worker capacity. A lifecycle `running` value from `status()` alone does not
|
||||
prove that Python code has begun executing.
|
||||
|
||||
Use [`SHOW JOBS`](/sql/ddl/job#show) and the Console's job detail to inspect
|
||||
available phase and row progress. On SDK/server versions with `job.progress()`,
|
||||
the snapshot can also include `phase`, `queue_reason`, timestamps and row
|
||||
counters. These optional fields are version-dependent; missing values are
|
||||
unknown, not zero progress or an ETA. `creation_ms` is submission time, not
|
||||
worker start time. `DESCRIBE JOB` provides the finished run report, rather than
|
||||
a live queue-position API.
|
||||
|
||||
Limits are deployment settings: the Agent dispatcher concurrency, executor
|
||||
capacity, per-table automatic-refresh limit and Function-call
|
||||
`max_concurrency` control different things. There is no universal five-refresh
|
||||
limit per database. Ask your administrator for the effective settings and use
|
||||
the [Enterprise operations guide](/enterprise/function-operations#admission)
|
||||
for their configuration names. Repeated cancellation and resubmission does not
|
||||
bypass those limits. Cancellation requests a stop, including while queued;
|
||||
observe a terminal state before submitting replacement work. Committed work
|
||||
is not rolled back.
|
||||
|
||||
### Configure workers and resources {#resources}
|
||||
|
||||
SQL exposes the worker budget directly, which is the lever when a backfill is
|
||||
@@ -229,6 +342,12 @@ up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not
|
||||
per-worker allocations or a guaranteed worker count. `num_workers` is only
|
||||
accepted for SQL-expression columns; registered Function columns reject it.
|
||||
|
||||
In particular, `gpu_count = 2` does not request two devices inside one Python
|
||||
process or enable tensor parallelism. Inspect configured GPU classes and their
|
||||
per-worker limits with `db.list_gpu_classes()` or `SHOW GPU CLASSES`. Choose a
|
||||
model that fits the worker shape supported by your deployment; see
|
||||
[GPU placement and budgets](/enterprise/function-operations#gpu).
|
||||
|
||||
The SDKs accept the same nested budget. CPU and memory are required; the GPU
|
||||
count inside the budget defaults to zero, so a CPU-only Function can omit it:
|
||||
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
title: "Handle Function failures"
|
||||
sidebarTitle: "Function Failures"
|
||||
description: "Keep usable outputs, inspect row errors, retry by stable identity and isolate inputs that need attention."
|
||||
tag: "Early access"
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
A Function refresh can encounter a bad row, fail to initialize a model, lose
|
||||
a worker, or fail while publishing results. These need different remedies.
|
||||
This guide uses the existing [refresh error policy](/sql/dml/refresh-column#options)
|
||||
and Python error records; it does not require a new retry or quarantine API.
|
||||
|
||||
## Choose a row-error policy {#policy}
|
||||
|
||||
The default `fail` stops the job on a row error. `skip` writes null outputs for
|
||||
failed rows and records errors, subject to the refresh's failure limits:
|
||||
|
||||
```python
|
||||
job = table.refresh_column_async(
|
||||
"embedding",
|
||||
error_policy="skip",
|
||||
max_failed_rows=100,
|
||||
max_failed_fraction=0.1,
|
||||
)
|
||||
result = job.wait()
|
||||
print(result.rows_assigned, result.rows_failed, result.rows_remaining)
|
||||
```
|
||||
|
||||
Choose limits for your workload; these example limits stop work when more
|
||||
than 100 rows or more than 10% fail. `skip` does not make initialization,
|
||||
infrastructure or publication failures successful. If `wait()` raises,
|
||||
[refresh the job record](/basics/jobs#snapshots), inspect `job.failure`, and
|
||||
inspect any recorded row errors. A failed/cancelled job can have committed
|
||||
some output, but retention of every successful call is not guaranteed.
|
||||
External effects are not rolled back and can be repeated during recovery.
|
||||
|
||||
## Inspect and group recorded errors {#inspect}
|
||||
|
||||
Scope diagnostics to the job you are investigating:
|
||||
|
||||
```python
|
||||
from collections import Counter
|
||||
|
||||
errors = table.function_errors(job_id=job.id, column="embedding", limit=10000)
|
||||
groups = Counter((r.error_type, r.error_message) for r in errors.records)
|
||||
for cause, count in groups.most_common():
|
||||
print(count, cause)
|
||||
for fragment in errors.fragments:
|
||||
print(fragment.fragment_id, fragment.rows_skipped, fragment.rows_recorded)
|
||||
print("listing truncated:", errors.truncated)
|
||||
```
|
||||
|
||||
Each record includes the Function name/version, table version, error type and
|
||||
message, creation time, and available row identities. `attempts` and traceback
|
||||
details depend on the server/image version. This client-side grouping counts
|
||||
only returned detail records. `truncated=True` or fragment summaries with
|
||||
more skipped than recorded rows mean it is incomplete; do not label it a full
|
||||
error census. See [`SHOW ERRORS`](/sql/ddl/job#errors) for SQL and its limits.
|
||||
|
||||
Records are historical observations, not a deduplicated list of currently
|
||||
failed rows. Reading errors without `job_id` can include older attempts that
|
||||
have since succeeded. A null output can mean unfilled, skipped, or a computed
|
||||
nullable field; output nullness alone is not a current-error test.
|
||||
|
||||
## Retry exactly identified rows {#retry}
|
||||
|
||||
On tables with stable row IDs, supporting server/SDK versions return
|
||||
`record.row_id` from the refresh's pinned source version. Keep that table and
|
||||
branch identity with the records. Compaction can move physical addresses, so
|
||||
do not derive a row ID from fragment numbers or offsets.
|
||||
|
||||
The following deliberately refuses incomplete or unidentified error sets.
|
||||
Review historical records against the current inputs before using them as a
|
||||
retry selection. Keep this selection small enough for a SQL predicate.
|
||||
|
||||
```python
|
||||
if errors.truncated or any(f.rows_skipped > f.rows_recorded for f in errors.fragments):
|
||||
raise RuntimeError("Inspect a narrower job/error set before selecting retries")
|
||||
if any(r.row_id is None for r in errors.records):
|
||||
raise RuntimeError("This workflow requires stable row IDs in every error record")
|
||||
|
||||
row_ids = sorted({int(r.row_id) for r in errors.records})
|
||||
if row_ids:
|
||||
predicate = "_rowid IN (" + ", ".join(map(str, row_ids)) + ")"
|
||||
retry_job = table.refresh_column_async("embedding", where=predicate)
|
||||
retry_job.wait()
|
||||
```
|
||||
|
||||
This recomputes every selected live row with the existing binding. It can
|
||||
repeat a row already repaired since the error was recorded. Inspect the retry
|
||||
job and its errors afterward; an older record need not disappear. On tables
|
||||
without stable row IDs, resolve errors to application keys at the recorded
|
||||
snapshot before retrying. Do not substitute `_rowaddr` after a rewrite.
|
||||
Python `where=` requires a supporting SDK such as `0.42.0b9`, not stable
|
||||
`0.41.0`, plus a matching server.
|
||||
|
||||
## Repair with a newer Function {#new-version}
|
||||
|
||||
Registering a new version does not change an existing column's immutable
|
||||
binding. To evaluate a fix while retaining old outputs, register the corrected
|
||||
callable and declare a separate output column. For example, with a corrected
|
||||
`embed_fixed` callable already defined and a `text` input column:
|
||||
|
||||
```python
|
||||
from lancedb import col
|
||||
|
||||
fixed = db.create_function(embed_fixed)
|
||||
table.add_columns(
|
||||
{"embedding_repaired": fixed(text=col("text"))},
|
||||
auto_refresh_enabled=False,
|
||||
)
|
||||
if row_ids:
|
||||
repair_job = table.refresh_column_async("embedding_repaired", where=predicate)
|
||||
repair_job.wait()
|
||||
```
|
||||
|
||||
Only the reviewed failed-row selection is populated in the new column; other
|
||||
rows remain unfilled. Compare those outputs before deciding how readers
|
||||
should use them. This does not promise an in-place upgrade/retry operation or
|
||||
per-row producing-version lookup for a column containing mixed executions.
|
||||
Error records' Function versions describe failures, not the provenance of
|
||||
every successful row.
|
||||
|
||||
## Quarantine inputs for review {#quarantine}
|
||||
|
||||
Use an application-owned table of source table/branch, recorded table version,
|
||||
stable row ID or business key, Function version, cause and review disposition.
|
||||
Keep Blob payloads in the source table; a quarantine entry can refer to their
|
||||
source identity. Choose an ordinary table name such as `function_quarantine`,
|
||||
and explicitly exclude its reviewed IDs in subsequent input selections.
|
||||
|
||||
Quarantine is an application workflow here: there is no implied built-in
|
||||
quarantine lifecycle, automatic current-error clearing or exactly-once retry.
|
||||
Do not delete source rows merely to hide an error. For very large failure sets,
|
||||
bounded error listings cannot select every row; use a smaller reviewed scope
|
||||
or wait for a deployment with the required complete selection support.
|
||||
|
||||
Native retry policies, full current-error summaries, in-place version upgrades
|
||||
and materialized-view row-error handling have separate server/SDK prerequisites.
|
||||
Do not apply `REFRESH COLUMN` options to a materialized view and assume the
|
||||
same policy; consult the [MV refresh reference](/sql/dml/refresh-materialized-view)
|
||||
for your deployment's accepted options.
|
||||
|
||||
## Continue {#next}
|
||||
|
||||
- [Function lifecycle and failures](/management/function#failures)
|
||||
- [Conditional and list-element refresh](/feature-engineering/refresh-column#partial)
|
||||
- [Job results and events](/basics/jobs#snapshots)
|
||||
@@ -420,6 +420,9 @@ full worked example.
|
||||
|
||||
## Failure handling {#failures}
|
||||
|
||||
For the full skip, inspect, grouped-error, stable-row-ID retry and quarantine
|
||||
workflow, see [Handle Function failures](/management/function-failures).
|
||||
|
||||
By default, a Function that raises fails the refresh job. Choose how the
|
||||
refresh handles row errors with its error policy:
|
||||
|
||||
@@ -495,6 +498,47 @@ In practice this means a function should be self-contained: take its inputs as
|
||||
arguments, import what it needs at module level, and avoid capturing state from
|
||||
the surrounding scope.
|
||||
|
||||
### Supported global captures {#captures}
|
||||
|
||||
The Python SDK's source packager captures only these literal values:
|
||||
|
||||
| Module-level value | Packaging behavior |
|
||||
| --- | --- |
|
||||
| `None`, exact `bool`, `int`, `str`, `bytes`, finite `float` | Inlined as a snapshot at registration. |
|
||||
| Tuples containing only those values, including nested tuples | Recursively inlined. |
|
||||
| Imported modules, importable functions/classes | Reconstructed as imports; declare their packages in `pip` or `conda`, or ship local source with `code`. |
|
||||
| Arrow types/schemas, lists, dictionaries, sets, NumPy arrays, non-finite floats, other objects | Not captured as literals. Construct them inside the Function or class. |
|
||||
|
||||
For example, a module-level `OUT_TYPE = pa.list_(pa.float32(), 3)` referenced
|
||||
from the Function body is unsupported. Construct the type where it is used:
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
from lancedb import function
|
||||
|
||||
@function(
|
||||
input_schema=pa.schema([pa.field("value", pa.float32(), nullable=False)]),
|
||||
output_schema=pa.list_(pa.float32(), 3),
|
||||
)
|
||||
def make_vector(value: pa.Array) -> pa.Array:
|
||||
out_type = pa.list_(pa.float32(), 3)
|
||||
return pa.array([[v, v, v] for v in value.to_pylist()], type=out_type)
|
||||
```
|
||||
|
||||
Decorator schema arguments are serialized as the signature; this restriction
|
||||
concerns values referenced by the packaged body. For an expensive constant or
|
||||
model, build it in a class Function's `__init__` and keep it on `self`.
|
||||
|
||||
Later mutations of a client global do not update a registered version. Register
|
||||
a new version to change captured literals or source. The diagnostic fix in
|
||||
[SDK #4938](https://github.com/lancedb/lancedb/pull/4938) names the unsupported
|
||||
global and its type; it does not add Arrow/container capture support. There is
|
||||
no supported arbitrary-object serialization or documented container-size limit
|
||||
to rely on for those rejected types.
|
||||
|
||||
See the [Function type matrix](/curation/create-function#signature-types) for
|
||||
what can travel as row inputs and outputs rather than captured globals.
|
||||
|
||||
Pass `code=[module, ...]` to ship additional helper modules into the same
|
||||
image, for logic the function imports but that does not belong inlined into
|
||||
it -- a model-loading helper shared by several Functions, for example.
|
||||
|
||||
@@ -182,6 +182,56 @@ BM25 scores depend on the indexed corpus and its term statistics. Compare
|
||||
rankings on the dataset you will search; treat scores as relative relevance
|
||||
values rather than probabilities.
|
||||
|
||||
### Match any word or every word {#term-operators}
|
||||
|
||||
A multi-word string such as `"happy puppy"` uses **OR**: a row can match either
|
||||
term. Writing `"happy AND puppy"` does not select an AND operator; the string
|
||||
is analyzed as search text. To require both terms, construct a match query:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from lancedb.query import FullTextOperator, MatchQuery
|
||||
|
||||
query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND)
|
||||
results = table.search(query, query_type="fts").limit(10).to_list()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
import { MatchQuery, Operator } from "@lancedb/lancedb";
|
||||
|
||||
const query = new MatchQuery("happy puppy", "text", { operator: Operator.And });
|
||||
const results = await tbl.query().fullTextSearch(query).limit(10).toArray();
|
||||
```
|
||||
|
||||
```rust Rust icon="rust"
|
||||
use lance_index::scalar::{
|
||||
FullTextSearchQuery,
|
||||
inverted::query::{MatchQuery, Operator},
|
||||
};
|
||||
use lancedb::query::{ExecutableQuery, QueryBase};
|
||||
|
||||
let query = MatchQuery::new("happy puppy".to_owned())
|
||||
.with_column(Some("text".to_owned()))
|
||||
.with_operator(Operator::And);
|
||||
let results = tbl
|
||||
.query()
|
||||
.full_text_search(FullTextSearchQuery::new_query(query.into()))
|
||||
.limit(10)
|
||||
.execute()
|
||||
.await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
AND requires all analyzed terms, without requiring adjacency or order. Use
|
||||
[`PhraseQuery`](#phrase-vs-terms-queries) for an ordered phrase, or combine
|
||||
separate queries with [`&` and `|`](#boolean-queries). Python `and` and `or`
|
||||
do not build queries.
|
||||
|
||||
On Enterprise, the existing [`fts()` SQL reference](/sql/dql/fts#term-operators)
|
||||
shows the equivalent JSON (`"operator":"And"`), `_score`, and a
|
||||
[`COUNT(*)` workaround](/sql/dql/fts#count) for counting matches without a
|
||||
top-k limit. `len(results)` counts only the returned result page.
|
||||
|
||||
### Keeping the index up to date {#keeping-the-index-up-to-date}
|
||||
|
||||
In LanceDB Embedded, new rows can remain outside the FTS index until you optimize the table.
|
||||
@@ -395,6 +445,13 @@ table.create_index("text", config=FTS(with_position=True, remove_stop_words=Fals
|
||||
</CodeGroup>
|
||||
|
||||
Storing token positions increases the index size and adds indexing work.
|
||||
On Enterprise, wait for the replacement index to finish before issuing the
|
||||
phrase query, using `table.wait_for_index(["text_idx"])` with your actual
|
||||
index name. An existing index without positions must be rebuilt; changing the
|
||||
query does not add positions. A phrase query against such an index is rejected
|
||||
with an error asking you to recreate it with positions, rather than returning
|
||||
an empty match set. See [FTS index options](/indexing/fts-index#fts-parameters)
|
||||
and the [SQL phrase reference](/sql/dql/fts#phrase).
|
||||
|
||||
### Fuzzy Search {#fuzzy-search}
|
||||
|
||||
|
||||
@@ -55,6 +55,15 @@ current deployments.
|
||||
a subset of partitions and `id_column` are pair-query-only options, accepted
|
||||
by [`duplicate_pairs`](/sql/dql/duplicate-pairs#input) but refused here.
|
||||
|
||||
## Index and snapshot requirements {#requirements}
|
||||
|
||||
Use one compatible, current vector index covering the source snapshot.
|
||||
Full scans reject unindexed fragments. Comparisons stay within physical IVF
|
||||
partitions and index segments; this does not enumerate cross-partition or
|
||||
cross-segment duplicates. Quantized-index distances use encoded vector
|
||||
representations. See [recall and scale limits](/curation/dedupe#limits) before
|
||||
treating the result as globally exact deduplication.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
One row per duplicate group, **in the table's own schema** -- every column of
|
||||
|
||||
@@ -60,6 +60,12 @@ only [`dedupe`](/sql/dql/dedupe#input) accepts it.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
See [`dedupe` index prerequisites](/sql/dql/dedupe#requirements) and the
|
||||
[recall limits](/curation/dedupe#limits). Pairs are found within each physical
|
||||
partition and index segment, not across them. A full scan requires complete
|
||||
index coverage; sampling is a diagnostic view of the selected indexed
|
||||
partitions and does not prove coverage or recall for the rest of the table.
|
||||
|
||||
One row per close pair found, in a fixed schema independent of the source
|
||||
table:
|
||||
|
||||
|
||||
@@ -30,14 +30,16 @@ INSERT INTO my_docs VALUES
|
||||
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
|
||||
(5, 'Puppy training requires patience and dedication', 'training', 'Alice');
|
||||
|
||||
CREATE INDEX ON my_docs USING fts (text);
|
||||
CREATE INDEX ON my_docs USING fts (text) WITH (with_position = true);
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Warning>
|
||||
Phrase queries need position information, which costs extra index size.
|
||||
Add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
|
||||
[phrase queries](#phrase) find nothing. See
|
||||
The setup above enables it with `WITH (with_position = true)`. An existing
|
||||
index without positions must be rebuilt before running a phrase query; the
|
||||
server rejects the query and asks you to recreate the index with positions.
|
||||
It does not silently return no matches. See
|
||||
[FTS parameters](/indexing/fts-index#fts-parameters).
|
||||
</Warning>
|
||||
|
||||
@@ -86,6 +88,44 @@ LIMIT 5
|
||||
it behaves like any other column, but exists only in a relation that came
|
||||
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
## Match any term or every term {#term-operators}
|
||||
|
||||
The default match operator is OR. `"happy puppy"` matches either analyzed
|
||||
term. SQL JSON uses `"Or"` and `"And"`, with this casing; Python uses the
|
||||
`FullTextOperator.OR` and `FullTextOperator.AND` enum members:
|
||||
|
||||
```python
|
||||
from lancedb.query import FullTextOperator, MatchQuery
|
||||
|
||||
query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND)
|
||||
json_query = query.to_json()
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}')
|
||||
ORDER BY _score DESC
|
||||
```
|
||||
|
||||
Both terms must occur, but they need not be adjacent. Do not embed `AND` or
|
||||
`OR` in `terms` to select an operator.
|
||||
|
||||
## Count matches {#count}
|
||||
|
||||
Count the `fts()` relation before applying a top-k limit:
|
||||
|
||||
```sql
|
||||
SELECT COUNT(*) AS matching_rows
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}')
|
||||
```
|
||||
|
||||
This is an Enterprise SQL workaround when the SDK's FTS query builder does
|
||||
not expose a match-count operation. `table.count_rows()` without an FTS
|
||||
relation counts table rows, and taking the length of a limited search counts
|
||||
only that result page. For list-element indexes, multiple matching elements
|
||||
can produce multiple rows; use `COUNT(DISTINCT id)` when you need unique
|
||||
application records.
|
||||
|
||||
## Fuzzy {#fuzzy}
|
||||
|
||||
Tolerates typos, up to a character-edit budget:
|
||||
@@ -109,6 +149,12 @@ PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', '{"phrase":{"column":"text","terms":"happy puppy","slop":0}}')
|
||||
ORDER BY _score DESC
|
||||
```
|
||||
|
||||
## Boolean {#boolean}
|
||||
|
||||
Combine queries with `&` (AND) or `|` (OR):
|
||||
@@ -120,6 +166,17 @@ MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
For SQL, serialize the composed query rather than writing occurrence enums
|
||||
by hand:
|
||||
|
||||
```python
|
||||
query = MatchQuery("puppy", "text") & MatchQuery("happy", "text")
|
||||
json_query = query.to_json()
|
||||
```
|
||||
|
||||
Use that string as the second `fts()` argument. `.to_json()` supplies the
|
||||
serialized Boolean field names for the SDK version you use.
|
||||
|
||||
## Boost {#boost}
|
||||
|
||||
Weight a query up or down relative to another:
|
||||
|
||||
Reference in New Issue
Block a user