docs: clarify Function workflows and search contracts

Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
This commit is contained in:
Jack Ye
2026-11-14 01:09:40 -08:00
committed by GitHub
parent 97e62dc11b
commit dd449978d9
14 changed files with 725 additions and 11 deletions
@@ -0,0 +1,63 @@
---
name: lancedb-vector-dedup
description: Plan and run Enterprise IVF vector deduplication with dedupe and duplicate_pairs, including sampling, index coverage, recall and resource limits. Use for near-duplicate embedding or perceptual-hash records; use MinHash guidance for text shingles.
---
# Vector deduplication
Read the maintained [workflow](../../../docs/web/curation/dedupe.mdx),
[DEDUPE reference](../../../docs/web/sql/dql/dedupe.mdx) and
[DUPLICATE_PAIRS reference](../../../docs/web/sql/dql/duplicate-pairs.mdx)
before constructing SQL. In a source checkout these paths are relative to this
skill directory. When using an installed copy without the docs, read
`https://docs.lancedb.com/curation/dedupe` and the linked SQL references.
## Establish the comparison contract
- Confirm an Enterprise deployment that supports these table functions. Use
`dedupe` and `duplicate_pairs`; legacy `vector_dedup` names are rejected.
- Inspect the vector type, metric, compatible index, indexed fragments and
source version. Wait for index coverage after writes. Do not run a full
scan over unindexed fragments or silently change the index to make it run.
- Explain that candidates are compared only within physical IVF partitions
and segments. Encoded/quantized distances are not an exact raw-vector
comparison. L2 thresholds are squared distances; Hamming counts bits.
- Use `id_column` on pair queries for business IDs. Retain `dataset_version`,
segment and partition identities. Do not decode internal row IDs or assume
that representatives stay the same after later data/index changes.
## Bound the first experiment
Use a reproducible partition sample before a full scan:
```sql
SELECT id_a, id_b, distance, dataset_version, segment_id, partition_id
FROM duplicate_pairs(
'frames', 'embedding',
distance_threshold => 0.02,
id_column => 'frame_id',
selection => 'random', count => 10, seed => 42
)
```
Adapt the threshold and sample size to the metric and actual partition count.
Sampling partitions is not uniform sampling of rows. Inspect known duplicate
and distinct examples, pair density, partition skew, memory, spill space and
elapsed time. Do not infer global recall or ten-billion-row qualification
from a small or unversioned benchmark. Pair work and output can be quadratic
within a partition; an outer SQL `LIMIT` is not an upstream work budget.
## Choose the result
`dedupe` returns original representative rows and always scans all partitions.
Use its default `connected_components` for transitive groups; choose
`direct_representative` when chains should not merge transitively. It does not
accept `id_column`, `selection`, `count` or `seed`; `duplicate_pairs` does not
accept `resolution`.
Keep pair data on the server when possible; avoid collecting an unbounded
edge set in client memory. Persist a new table/view only within the user's
authorized scope, preserving source rows unless deletion was requested.
Do not register a Python Function or fetch/rewrite Blob payloads to invoke
these built-in table functions. For MinHash/text deduplication, follow the
separate `https://docs.lancedb.com/enterprise/sql-minhash` workflow.
+73
View File
@@ -33,6 +33,46 @@ print(job.status())
```
</CodeGroup>
## Cached properties and live calls {#snapshots}
Python job properties describe the last observation; reading them does not
contact the server. Submission need not fetch the full registry record, so
some properties can initially be `None`.
| Surface | Contract |
| --- | --- |
| `state` | Last observed lifecycle state. It does not update on its own. |
| `job_type`, `creation_ms` | Cached job kind and creation time in epoch milliseconds. |
| `spec` | Complete, job-specific submitted specification. Read this separately from the bounded `repr(job)` display. |
| `result` | Cached, job-specific reported data; it is not necessarily the typed object returned by `wait()`. |
| `failure` | Cached `JobFailureInfo`, when the server reports a terminal failure. |
| `status()` | Live lifecycle request. For registry jobs it also caches the full record. A failed state is returned without raising a job-failure exception; request errors can still raise. |
| `refresh()` | Fetch and cache the current record without waiting for completion. |
| `events(...)` | Separate live request returning a `pyarrow.Table` of recorded events; it does not refresh the properties. |
| `wait(...)` | Poll until completion and return the operation's typed result, or raise on failure/cancellation. Refresh afterward when you need the complete final registry record. |
For example, poll a registry handle and then retrieve its final details:
```python
import time
state = job.status()
while state == "running":
print(job.state)
time.sleep(2)
state = job.status()
result = job.wait()
job.refresh()
print(result)
print(job.spec)
```
The four lifecycle strings below are the usual SDK contract; newer server
states may pass through an older client. Do not infer execution progress from
`state` alone. Structured progress methods and queue fields require matching
SDK/server support; see [refresh admission](/feature-engineering/refresh-column#admission).
## Submit and wait for a job {#waiting}
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
@@ -133,6 +173,39 @@ for info in db.list_jobs(page_limit=50):
```
</CodeGroup>
`list_jobs` yields `JobInfo` summaries, newest creation time first, rather than
live `Job` handles. Open an ID to inspect its full record. `page_limit` bounds
each server request, not the total yielded items, and `page_token` resumes a
listing. It is a lazy iterator: iteration performs requests and can raise
request errors. To materialize it, use `list(db.list_jobs(...))`; it does not
support list indexing, `len()`, or a `.to_list()` method.
`Listing.num_page_results()` reports cached items remaining in the current
page and `Listing.page_token()` reports the next request's token. Drain the
cached items before saving that token, or resuming can skip them. A `None`
token after the final page can coexist with unread cached items. A resumed
listing is not a transactionally frozen view of jobs submitted in the meantime.
`events.schema` describes the Arrow response. Use `events.to_pylist()` for
Python row dictionaries. Common event columns are `event_seq` (int64), `state`
(string), `updated_at_millis` (int64), `updated_by` (string), and nullable
`emitted_from`, `emitted_by` and `claim_entity` strings. Optional uint64 work
counters such as `rows_processed` and diagnostic details depend on job type
and server version. If a `payload` column is returned, its value is a nullable
JSON string, not an already decoded Python dictionary.
Empty results retain their schema. Events are newest first and belong to the
requested job; child events are separate. An event count is not a completed-row
count, and a running job can have no terminal `result` while producing events.
For a Function refresh, `wait()` returns a `RefreshColumnResult` such as
`rows_assigned`, `rows_failed`, `rows_remaining`, `source_version` and optional
`published_version`. Other job types have different results. Do not parse the
printed representation to recover a spec, result or progress measurement.
For object-shaped registry data, `job.spec` and `job.result` are decoded Python
dictionaries: access fields with keys, such as `job.result.get("summary")`
after refreshing a successful Function job. The separate typed object from
`wait()` uses attributes, such as `result.rows_assigned`.
Event responses have [row limits](/basics/limits#jobs). An explicit limit can
request more than the default, but it does not guarantee a complete event
history.
+58 -3
View File
@@ -44,6 +44,58 @@ table.add_columns({"doubled": version(value=col("value"))})
immutable version, so changing one produces a new version rather than
altering rows a previous version already filled.
## Supported Function signatures {#signature-types}
A table's [Arrow schema](/sql/data-types#supported) can contain types that a
Function cannot accept or return. The matrix below describes the Python SDK
`0.42.0-beta.12` source contract. Nested schemas and list-Blob outputs require
a matching Enterprise server; an SDK upgrade alone does not enable them.
| Arrow type | Function input/output support |
| --- | --- |
| Boolean; signed/unsigned 8, 16, 32, 64-bit integers; float16/32/64 | Supported. Ordinary `int` and `float` annotations infer int64 and float64. |
| utf8 / large_utf8; binary / large_binary | Supported. `str` and `bytes` infer utf8 and binary. Plain binary does not carry Blob semantics. |
| date32 / date64 | Supported; Python `date` infers date32. |
| timestamp, with or without timezone; time32/64; duration; interval | Unsupported by the Function signature grammar. A Python `datetime` annotation can resolve to an Arrow timestamp but registration still rejects it. |
| decimal; map; dictionary; union; fixed-size binary | Unsupported. Decode dictionary values and use supported scalar/list/struct representations. |
| struct; list / large_list; positive-length fixed-size list | Supported with supported children. Explicit schemas preserve child nullability; annotation-only `list[T]` uses non-nullable elements. Struct fields must be nonempty and uniquely named. |
| Blob v2 at top level or in a struct | Supported in inputs and outputs; use `lancedb.blob(...)` in the schema. |
| Blob v2 under list / large_list | Output only on supporting SDK/server versions. Inputs are rejected. |
| Blob v2 under fixed-size list or map | Unsupported in both directions. |
For times, pass an int64 epoch value with an explicit unit such as microseconds,
or a UTF-8 ISO 8601 string with its timezone. Convert inside the Function and
cast the result in SQL if needed. Use scaled integers with a declared scale
when exact decimal values matter; converting to float can lose precision.
Represent a map as a list of key/value structs when that fits the calculation.
A scalar output is non-nullable because physical NULL represents an unfilled
computed output. For an optional value, return a named struct with a nullable
field, for example:
```python
import pyarrow as pa
from lancedb import function
@function(
input_schema=pa.schema([pa.field("text", pa.string(), nullable=False)]),
output_schema=pa.schema([pa.field("score", pa.float64(), nullable=True)]),
)
def optional_score(text: str):
return {"score": float(len(text)) if text else None}
```
Return a dictionary by field name. A successfully computed nullable field can
be NULL; `score IS NULL` alone cannot distinguish it from pending work or a
skipped failure. Use the job result and error diagnostics. Do not assume an
explicit assignment-state flag is available on every deployment.
List-nested Blob outputs are queryable as descriptors, but row-ID blob fetch
APIs and `blob_mode="bytes"` do not fetch payloads inside lists. Prefer a
separate row per media item when you need independent reads, retries or
re-scoring. A plain struct shaped like a Blob without its extension metadata
is not a substitute for `lancedb.blob`.
## Batch: once per call, many rows at a time {#batch}
A function is called once per batch instead of once per row when its
@@ -170,9 +222,12 @@ in the function's source. A Secret is a database-scoped named credential:
puts it in the environment variable the function reads, without ever
returning it from any API or recording it in the function's own definition.
Combine it with the stateful form so the HTTP client -- and any rate limiter
around it -- is built once, not reopened on every row. There is no built-in
retry or per-row error-skip policy -- a function that calls a flaky service
should handle timeouts and retries itself, inside `__call__`.
around it -- is built once, not reopened on every row. Configure HTTP timeouts
inside `__call__`. Refresh-level `error_policy='skip'` can record row failures;
see [handling failures](/management/function#failures). Native retry policies
need matching SDK, server and Function-image support; on other deployments,
handle bounded retries inside the Function only when repeating the request is
safe.
See [Vend a Secret to a Function](/curation/vend-secret#specify) for the
full worked example, how the bound value actually reaches a running
+53 -5
View File
@@ -136,12 +136,60 @@ SELECT * FROM dedupe(
See [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
for manual and automatic refresh options to keep it current as new records land.
## At production scale {#production}
## Index, snapshot and recall limits {#limits}
[LeRobotDataset](/integrations/lerobotdataset) runs the same `dedupe`
call over 27.6 million real embedding rows to remove near-static frames from a
robotics training set in under 30 seconds -- the mechanism does not change
with scale, only the table it runs against.
These are Enterprise SQL table functions, not standard SQL or a registered
Python Function. They require one compatible current vector index covering
the scanned snapshot. Wait for indexing to finish after ingestion and inspect
index coverage before running a full scan. Unindexed fragments are rejected;
deleted rows are excluded at the pinned dataset version.
The scan compares candidates **within each physical IVF partition and index
segment**. It does not compare across partitions or segments. Scanning every
partition therefore does not establish global all-pairs recall. Quantized
indexes compare their encoded vector representations; `IVF_FLAT` compares the
stored vectors. Thresholds must match the index's metric: L2 is squared L2,
cosine is `1 - cosine`, dot distance is `1 - dot`, and Hamming counts differing
bits. Inspect close examples and known duplicates before choosing a threshold.
`duplicate_pairs` reports `dataset_version`, `segment_id` and `partition_id`.
Retain those with the index configuration and source version when comparing
runs. Pass `id_column` for application IDs, rather than treating physical row
addresses as durable identifiers. A new append, index change or bridge between
components can change groups and representatives.
## Efficient use and scale qualification {#production}
Start with [`duplicate_pairs` partition sampling](/sql/dql/duplicate-pairs#sampling),
using a recorded seed and a bounded partition count. Inspect pair density and
threshold quality before running `dedupe`, which always scans every partition.
Sampling partitions is not uniform row sampling and can miss rare groups.
`resolution='connected_components'` merges transitive chains;
`direct_representative` uses direct representative relationships instead.
Pair work grows quadratically within dense partitions, and a loose threshold
can emit a quadratic number of edges. An outer `LIMIT` is not a resource
budget for upstream pair finding or group resolution. Avoid repeatedly
materializing all pairs in a Python list; select needed columns, aggregate on
the server, and persist a reviewed result when repeated reads justify it.
Measure peak memory, spill storage, elapsed time and pair count on
representative partitions before increasing scope.
The [LeRobotDataset integration](/integrations/lerobotdataset) illustrates a
27.6-million-row data workflow. Its timings are not a versioned qualification
of this deduplication path on arbitrary indexes or hardware. A 10-billion-row
run is not qualified by that example. Qualify source/index versions, partition
sizes, metric, threshold, worker resources, spill limits and correctness on your
deployment before extrapolating. There is no universal row-count or latency
guarantee in this guide.
For text shingles and MinHash signatures, use the distinct
[MinHash workflow](/enterprise/sql-minhash). Vector distance deduplication and
MinHash/Jaccard text deduplication have different indexes and thresholds.
Agents can follow the repository's
[vector deduplication skill](https://github.com/lancedb/lancedb/blob/main/.agents/skills/lancedb-vector-dedup/SKILL.md)
to apply these checks without inventing Function options or scale guarantees.
## Continue with the curated data {#next}
+1
View File
@@ -496,6 +496,7 @@
"management/materialized-view",
"management/secret",
"management/function",
"management/function-failures",
"management/storage-integration"
]
},
@@ -13,6 +13,13 @@ import {
PySchemaAddSetup,
} from '/snippets/tables.mdx';
<Note>
Declare computed columns on ordinary tables. A materialized view derives its
schema from its query; include the calculation in the view's `SELECT` instead
of attaching a Function with `add_columns`. See
[materialized-view schema changes](/feature-engineering/materialized-views#schema-changes).
</Note>
Lance can write a new column without rewriting the columns already in the
table. That matters most on a table that also holds large images, video or
embeddings, where rewriting everything just to add one small derived field
@@ -176,6 +176,30 @@ definition is read with `await view.definition()`.
Creation, refresh and drop also have async, job-returning SDK methods; see
[Materialized view](/management/materialized-view#manage).
## Add derived values to a view {#schema-changes}
A materialized view's output schema belongs to its defining `SELECT`. Do not
use `view.table.add_columns(...)` to attach another computed or Function column
to that output. In particular, adding a Function binding is rejected: the
column would not be produced by the next view refresh.
Choose where the calculation belongs:
- Include the expression or Function call in the view's `SELECT`, as in the
[caption example](#functions). To change an existing definition, create a
replacement view under a new name, check its results, indexes and grants,
and move readers to it before dropping the old view.
- Add a Function column to the ordinary source table, refresh it, then select
that stored column in a view. This lets you manage column refresh separately
from view refresh.
- Create an ordinary table from a query when you need independently editable
outputs. That table is a snapshot; it does not inherit the view's refresh
relationship to the source.
Opening the underlying table gives access to reads and indexes; it does not
make independent schema changes part of the materialized-view definition.
Changing automatic-refresh settings also does not change that definition.
## Related guides {#next}
<CardGroup cols={2}>
@@ -77,6 +77,11 @@ catch-up works when it's re-enabled after being off.
## Refreshing part of a table {#partial}
Python `where=` requires a supporting SDK, such as `0.42.0b9`, and an
Enterprise server with filtered refresh support. Stable `0.41.0` does not
accept this argument. The SQL form is available independently of that Python
argument on supporting servers.
<Badge color="red">Enterprise</Badge> A refresh can be scoped to a predicate,
with `where=` in Python or `WHERE` in SQL, which is useful for filling a slice
first and checking it before committing to the whole table:
@@ -123,6 +128,89 @@ to let LanceDB track which outputs need updating.
`function_refresh` for one filled by a registered [function](/management/function).
</Note>
### Re-score selected list elements {#list-elements}
`WHERE` selects **rows**, not elements within a list. Refreshing a row that
contains ten answers passes the whole list to the Function. A mask can control
which elements your code computes, but the Function still returns one complete
list for that row. There is no element-level refresh or partial-list write API.
This example keeps `sample_id` as the link to an existing `samples` table with
a Blob column named `media`. It creates only answer metadata and scores; it
neither reads nor copies Blob payloads. Run it on a deployment supporting
`@function`, list signatures and Python `where=`, with `samples` already
containing sample IDs `s1` and `s2`.
```python
from lancedb import col, function
media_before = db.execute_query(
"SELECT sample_id, media FROM samples "
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
).read_all().to_pylist()
answers = db.create_table("answer_rescore_demo", [
{"group_id": "g1", "sample_id": "s1",
"answers": ["yes", "maybe", "no"],
"selected": [False, True, False], "previous": [0.1, 0.2, 0.3]},
{"group_id": "g1", "sample_id": "s2",
"answers": ["other"], "selected": [False], "previous": [0.4]},
])
@function
def rescore_masked(
answers: list[str], selected: list[bool], previous: list[float]
) -> list[float]:
if not (len(answers) == len(selected) == len(previous)):
raise ValueError("answers, selected and previous must have equal lengths")
return [
float(len(answer)) if choose else old
for answer, choose, old in zip(answers, selected, previous)
]
version = db.create_function(rescore_masked)
answers.add_columns(
{"rescored": version(
answers=col("answers"), selected=col("selected"), previous=col("previous")
)},
auto_refresh_enabled=False,
)
job = answers.refresh_column_async("rescored", where="sample_id = 's1'")
job.wait()
rows = answers.search().select(
["group_id", "sample_id", "previous", "rescored"]
).to_list()
by_sample = {row["sample_id"]: row for row in rows}
assert by_sample["s1"]["group_id"] == "g1"
assert by_sample["s1"]["rescored"] == [0.1, 5.0, 0.3]
assert by_sample["s1"]["previous"] == [0.1, 0.2, 0.3]
assert by_sample["s2"]["rescored"] is None
assert by_sample["s2"]["previous"] == [0.4]
media_after = db.execute_query(
"SELECT sample_id, media FROM samples "
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
).read_all().to_pylist()
assert media_after == media_before
```
Here length is a deterministic stand-in for a scorer. Only `"maybe"` is scored;
the other scores are copied from `previous`. The final equality checks Blob
descriptors on an otherwise unchanged source table, without fetching payloads.
Keep the source table unchanged during this comparison. Use the same model,
normalization and score type as the original computation in a real re-scoring
pass. For another pass, explicitly supply the intended prior scores and mask;
refresh does not merge the previous output into your Function's inputs.
If answers need independent retries or versions, prefer one ordinary table row
per answer, with `group_id`, `sample_id`, a stable application `answer_id`,
answer text and score. Keep media in `samples` and join by `sample_id` when
needed. A row predicate can then select exact answer IDs, and Blob storage
stays in the source table. Do not use `UNNEST` to assume that nested Blob values
become supported Function inputs; see the
[nested-Blob limits](/curation/create-function#signature-types).
## Run a refresh on Enterprise {#async}
<Badge color="red">Enterprise</Badge>
@@ -212,6 +300,31 @@ The wait helpers do not submit or cancel a refresh. They retry only the typed
finalization-pending response and propagate other errors.
*/}
### Admission and waiting {#admission}
Submission does not reserve a supervisor or GPU immediately. A refresh can wait
for source preparation/compaction, an earlier refresh, executor admission or
worker capacity. A lifecycle `running` value from `status()` alone does not
prove that Python code has begun executing.
Use [`SHOW JOBS`](/sql/ddl/job#show) and the Console's job detail to inspect
available phase and row progress. On SDK/server versions with `job.progress()`,
the snapshot can also include `phase`, `queue_reason`, timestamps and row
counters. These optional fields are version-dependent; missing values are
unknown, not zero progress or an ETA. `creation_ms` is submission time, not
worker start time. `DESCRIBE JOB` provides the finished run report, rather than
a live queue-position API.
Limits are deployment settings: the Agent dispatcher concurrency, executor
capacity, per-table automatic-refresh limit and Function-call
`max_concurrency` control different things. There is no universal five-refresh
limit per database. Ask your administrator for the effective settings and use
the [Enterprise operations guide](/enterprise/function-operations#admission)
for their configuration names. Repeated cancellation and resubmission does not
bypass those limits. Cancellation requests a stop, including while queued;
observe a terminal state before submitting replacement work. Committed work
is not rolled back.
### Configure workers and resources {#resources}
SQL exposes the worker budget directly, which is the lever when a backfill is
@@ -229,6 +342,12 @@ up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not
per-worker allocations or a guaranteed worker count. `num_workers` is only
accepted for SQL-expression columns; registered Function columns reject it.
In particular, `gpu_count = 2` does not request two devices inside one Python
process or enable tensor parallelism. Inspect configured GPU classes and their
per-worker limits with `db.list_gpu_classes()` or `SHOW GPU CLASSES`. Choose a
model that fits the worker shape supported by your deployment; see
[GPU placement and budgets](/enterprise/function-operations#gpu).
The SDKs accept the same nested budget. CPU and memory are required; the GPU
count inside the budget defaults to zero, so a CPU-only Function can omit it:
+151
View File
@@ -0,0 +1,151 @@
---
title: "Handle Function failures"
sidebarTitle: "Function Failures"
description: "Keep usable outputs, inspect row errors, retry by stable identity and isolate inputs that need attention."
tag: "Early access"
availability:
embedded: unavailable
enterprise: available
---
A Function refresh can encounter a bad row, fail to initialize a model, lose
a worker, or fail while publishing results. These need different remedies.
This guide uses the existing [refresh error policy](/sql/dml/refresh-column#options)
and Python error records; it does not require a new retry or quarantine API.
## Choose a row-error policy {#policy}
The default `fail` stops the job on a row error. `skip` writes null outputs for
failed rows and records errors, subject to the refresh's failure limits:
```python
job = table.refresh_column_async(
"embedding",
error_policy="skip",
max_failed_rows=100,
max_failed_fraction=0.1,
)
result = job.wait()
print(result.rows_assigned, result.rows_failed, result.rows_remaining)
```
Choose limits for your workload; these example limits stop work when more
than 100 rows or more than 10% fail. `skip` does not make initialization,
infrastructure or publication failures successful. If `wait()` raises,
[refresh the job record](/basics/jobs#snapshots), inspect `job.failure`, and
inspect any recorded row errors. A failed/cancelled job can have committed
some output, but retention of every successful call is not guaranteed.
External effects are not rolled back and can be repeated during recovery.
## Inspect and group recorded errors {#inspect}
Scope diagnostics to the job you are investigating:
```python
from collections import Counter
errors = table.function_errors(job_id=job.id, column="embedding", limit=10000)
groups = Counter((r.error_type, r.error_message) for r in errors.records)
for cause, count in groups.most_common():
print(count, cause)
for fragment in errors.fragments:
print(fragment.fragment_id, fragment.rows_skipped, fragment.rows_recorded)
print("listing truncated:", errors.truncated)
```
Each record includes the Function name/version, table version, error type and
message, creation time, and available row identities. `attempts` and traceback
details depend on the server/image version. This client-side grouping counts
only returned detail records. `truncated=True` or fragment summaries with
more skipped than recorded rows mean it is incomplete; do not label it a full
error census. See [`SHOW ERRORS`](/sql/ddl/job#errors) for SQL and its limits.
Records are historical observations, not a deduplicated list of currently
failed rows. Reading errors without `job_id` can include older attempts that
have since succeeded. A null output can mean unfilled, skipped, or a computed
nullable field; output nullness alone is not a current-error test.
## Retry exactly identified rows {#retry}
On tables with stable row IDs, supporting server/SDK versions return
`record.row_id` from the refresh's pinned source version. Keep that table and
branch identity with the records. Compaction can move physical addresses, so
do not derive a row ID from fragment numbers or offsets.
The following deliberately refuses incomplete or unidentified error sets.
Review historical records against the current inputs before using them as a
retry selection. Keep this selection small enough for a SQL predicate.
```python
if errors.truncated or any(f.rows_skipped > f.rows_recorded for f in errors.fragments):
raise RuntimeError("Inspect a narrower job/error set before selecting retries")
if any(r.row_id is None for r in errors.records):
raise RuntimeError("This workflow requires stable row IDs in every error record")
row_ids = sorted({int(r.row_id) for r in errors.records})
if row_ids:
predicate = "_rowid IN (" + ", ".join(map(str, row_ids)) + ")"
retry_job = table.refresh_column_async("embedding", where=predicate)
retry_job.wait()
```
This recomputes every selected live row with the existing binding. It can
repeat a row already repaired since the error was recorded. Inspect the retry
job and its errors afterward; an older record need not disappear. On tables
without stable row IDs, resolve errors to application keys at the recorded
snapshot before retrying. Do not substitute `_rowaddr` after a rewrite.
Python `where=` requires a supporting SDK such as `0.42.0b9`, not stable
`0.41.0`, plus a matching server.
## Repair with a newer Function {#new-version}
Registering a new version does not change an existing column's immutable
binding. To evaluate a fix while retaining old outputs, register the corrected
callable and declare a separate output column. For example, with a corrected
`embed_fixed` callable already defined and a `text` input column:
```python
from lancedb import col
fixed = db.create_function(embed_fixed)
table.add_columns(
{"embedding_repaired": fixed(text=col("text"))},
auto_refresh_enabled=False,
)
if row_ids:
repair_job = table.refresh_column_async("embedding_repaired", where=predicate)
repair_job.wait()
```
Only the reviewed failed-row selection is populated in the new column; other
rows remain unfilled. Compare those outputs before deciding how readers
should use them. This does not promise an in-place upgrade/retry operation or
per-row producing-version lookup for a column containing mixed executions.
Error records' Function versions describe failures, not the provenance of
every successful row.
## Quarantine inputs for review {#quarantine}
Use an application-owned table of source table/branch, recorded table version,
stable row ID or business key, Function version, cause and review disposition.
Keep Blob payloads in the source table; a quarantine entry can refer to their
source identity. Choose an ordinary table name such as `function_quarantine`,
and explicitly exclude its reviewed IDs in subsequent input selections.
Quarantine is an application workflow here: there is no implied built-in
quarantine lifecycle, automatic current-error clearing or exactly-once retry.
Do not delete source rows merely to hide an error. For very large failure sets,
bounded error listings cannot select every row; use a smaller reviewed scope
or wait for a deployment with the required complete selection support.
Native retry policies, full current-error summaries, in-place version upgrades
and materialized-view row-error handling have separate server/SDK prerequisites.
Do not apply `REFRESH COLUMN` options to a materialized view and assume the
same policy; consult the [MV refresh reference](/sql/dml/refresh-materialized-view)
for your deployment's accepted options.
## Continue {#next}
- [Function lifecycle and failures](/management/function#failures)
- [Conditional and list-element refresh](/feature-engineering/refresh-column#partial)
- [Job results and events](/basics/jobs#snapshots)
+44
View File
@@ -420,6 +420,9 @@ full worked example.
## Failure handling {#failures}
For the full skip, inspect, grouped-error, stable-row-ID retry and quarantine
workflow, see [Handle Function failures](/management/function-failures).
By default, a Function that raises fails the refresh job. Choose how the
refresh handles row errors with its error policy:
@@ -495,6 +498,47 @@ In practice this means a function should be self-contained: take its inputs as
arguments, import what it needs at module level, and avoid capturing state from
the surrounding scope.
### Supported global captures {#captures}
The Python SDK's source packager captures only these literal values:
| Module-level value | Packaging behavior |
| --- | --- |
| `None`, exact `bool`, `int`, `str`, `bytes`, finite `float` | Inlined as a snapshot at registration. |
| Tuples containing only those values, including nested tuples | Recursively inlined. |
| Imported modules, importable functions/classes | Reconstructed as imports; declare their packages in `pip` or `conda`, or ship local source with `code`. |
| Arrow types/schemas, lists, dictionaries, sets, NumPy arrays, non-finite floats, other objects | Not captured as literals. Construct them inside the Function or class. |
For example, a module-level `OUT_TYPE = pa.list_(pa.float32(), 3)` referenced
from the Function body is unsupported. Construct the type where it is used:
```python
import pyarrow as pa
from lancedb import function
@function(
input_schema=pa.schema([pa.field("value", pa.float32(), nullable=False)]),
output_schema=pa.list_(pa.float32(), 3),
)
def make_vector(value: pa.Array) -> pa.Array:
out_type = pa.list_(pa.float32(), 3)
return pa.array([[v, v, v] for v in value.to_pylist()], type=out_type)
```
Decorator schema arguments are serialized as the signature; this restriction
concerns values referenced by the packaged body. For an expensive constant or
model, build it in a class Function's `__init__` and keep it on `self`.
Later mutations of a client global do not update a registered version. Register
a new version to change captured literals or source. The diagnostic fix in
[SDK #4938](https://github.com/lancedb/lancedb/pull/4938) names the unsupported
global and its type; it does not add Arrow/container capture support. There is
no supported arbitrary-object serialization or documented container-size limit
to rely on for those rejected types.
See the [Function type matrix](/curation/create-function#signature-types) for
what can travel as row inputs and outputs rather than captured globals.
Pass `code=[module, ...]` to ship additional helper modules into the same
image, for logic the function imports but that does not belong inlined into
it -- a model-loading helper shared by several Functions, for example.
+57
View File
@@ -182,6 +182,56 @@ BM25 scores depend on the indexed corpus and its term statistics. Compare
rankings on the dataset you will search; treat scores as relative relevance
values rather than probabilities.
### Match any word or every word {#term-operators}
A multi-word string such as `"happy puppy"` uses **OR**: a row can match either
term. Writing `"happy AND puppy"` does not select an AND operator; the string
is analyzed as search text. To require both terms, construct a match query:
<CodeGroup>
```python Python icon="python"
from lancedb.query import FullTextOperator, MatchQuery
query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND)
results = table.search(query, query_type="fts").limit(10).to_list()
```
```typescript TypeScript icon="square-js"
import { MatchQuery, Operator } from "@lancedb/lancedb";
const query = new MatchQuery("happy puppy", "text", { operator: Operator.And });
const results = await tbl.query().fullTextSearch(query).limit(10).toArray();
```
```rust Rust icon="rust"
use lance_index::scalar::{
FullTextSearchQuery,
inverted::query::{MatchQuery, Operator},
};
use lancedb::query::{ExecutableQuery, QueryBase};
let query = MatchQuery::new("happy puppy".to_owned())
.with_column(Some("text".to_owned()))
.with_operator(Operator::And);
let results = tbl
.query()
.full_text_search(FullTextSearchQuery::new_query(query.into()))
.limit(10)
.execute()
.await?;
```
</CodeGroup>
AND requires all analyzed terms, without requiring adjacency or order. Use
[`PhraseQuery`](#phrase-vs-terms-queries) for an ordered phrase, or combine
separate queries with [`&` and `|`](#boolean-queries). Python `and` and `or`
do not build queries.
On Enterprise, the existing [`fts()` SQL reference](/sql/dql/fts#term-operators)
shows the equivalent JSON (`"operator":"And"`), `_score`, and a
[`COUNT(*)` workaround](/sql/dql/fts#count) for counting matches without a
top-k limit. `len(results)` counts only the returned result page.
### Keeping the index up to date {#keeping-the-index-up-to-date}
In LanceDB Embedded, new rows can remain outside the FTS index until you optimize the table.
@@ -395,6 +445,13 @@ table.create_index("text", config=FTS(with_position=True, remove_stop_words=Fals
</CodeGroup>
Storing token positions increases the index size and adds indexing work.
On Enterprise, wait for the replacement index to finish before issuing the
phrase query, using `table.wait_for_index(["text_idx"])` with your actual
index name. An existing index without positions must be rebuilt; changing the
query does not add positions. A phrase query against such an index is rejected
with an error asking you to recreate it with positions, rather than returning
an empty match set. See [FTS index options](/indexing/fts-index#fts-parameters)
and the [SQL phrase reference](/sql/dql/fts#phrase).
### Fuzzy Search {#fuzzy-search}
+9
View File
@@ -55,6 +55,15 @@ current deployments.
a subset of partitions and `id_column` are pair-query-only options, accepted
by [`duplicate_pairs`](/sql/dql/duplicate-pairs#input) but refused here.
## Index and snapshot requirements {#requirements}
Use one compatible, current vector index covering the source snapshot.
Full scans reject unindexed fragments. Comparisons stay within physical IVF
partitions and index segments; this does not enumerate cross-partition or
cross-segment duplicates. Quantized-index distances use encoded vector
representations. See [recall and scale limits](/curation/dedupe#limits) before
treating the result as globally exact deduplication.
## Output {#output}
One row per duplicate group, **in the table's own schema** -- every column of
+6
View File
@@ -60,6 +60,12 @@ only [`dedupe`](/sql/dql/dedupe#input) accepts it.
## Output {#output}
See [`dedupe` index prerequisites](/sql/dql/dedupe#requirements) and the
[recall limits](/curation/dedupe#limits). Pairs are found within each physical
partition and index segment, not across them. A full scan requires complete
index coverage; sampling is a diagnostic view of the selected indexed
partitions and does not prove coverage or recall for the rest of the table.
One row per close pair found, in a fixed schema independent of the source
table:
+60 -3
View File
@@ -30,14 +30,16 @@ INSERT INTO my_docs VALUES
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
(5, 'Puppy training requires patience and dedication', 'training', 'Alice');
CREATE INDEX ON my_docs USING fts (text);
CREATE INDEX ON my_docs USING fts (text) WITH (with_position = true);
```
</CodeGroup>
<Warning>
Phrase queries need position information, which costs extra index size.
Add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
[phrase queries](#phrase) find nothing. See
The setup above enables it with `WITH (with_position = true)`. An existing
index without positions must be rebuilt before running a phrase query; the
server rejects the query and asks you to recreate the index with positions.
It does not silently return no matches. See
[FTS parameters](/indexing/fts-index#fts-parameters).
</Warning>
@@ -86,6 +88,44 @@ LIMIT 5
it behaves like any other column, but exists only in a relation that came
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
## Match any term or every term {#term-operators}
The default match operator is OR. `"happy puppy"` matches either analyzed
term. SQL JSON uses `"Or"` and `"And"`, with this casing; Python uses the
`FullTextOperator.OR` and `FullTextOperator.AND` enum members:
```python
from lancedb.query import FullTextOperator, MatchQuery
query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND)
json_query = query.to_json()
```
```sql
SELECT id, text, _score
FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}')
ORDER BY _score DESC
```
Both terms must occur, but they need not be adjacent. Do not embed `AND` or
`OR` in `terms` to select an operator.
## Count matches {#count}
Count the `fts()` relation before applying a top-k limit:
```sql
SELECT COUNT(*) AS matching_rows
FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}')
```
This is an Enterprise SQL workaround when the SDK's FTS query builder does
not expose a match-count operation. `table.count_rows()` without an FTS
relation counts table rows, and taking the length of a limited search counts
only that result page. For list-element indexes, multiple matching elements
can produce multiple rows; use `COUNT(DISTINCT id)` when you need unique
application records.
## Fuzzy {#fuzzy}
Tolerates typos, up to a character-edit budget:
@@ -109,6 +149,12 @@ PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
```
</CodeGroup>
```sql
SELECT id, text, _score
FROM fts('my_docs', '{"phrase":{"column":"text","terms":"happy puppy","slop":0}}')
ORDER BY _score DESC
```
## Boolean {#boolean}
Combine queries with `&` (AND) or `|` (OR):
@@ -120,6 +166,17 @@ MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
```
</CodeGroup>
For SQL, serialize the composed query rather than writing occurrence enums
by hand:
```python
query = MatchQuery("puppy", "text") & MatchQuery("happy", "text")
json_query = query.to_json()
```
Use that string as the second `fts()` argument. `.to_json()` supplies the
serialized Boolean field names for the SDK version you use.
## Boost {#boost}
Weight a query up or down relative to another: