From dd449978d9c43a286ae7bf739301be34827125c4 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Sat, 14 Nov 2026 01:09:40 -0800 Subject: [PATCH] docs: clarify Function workflows and search contracts Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits. --- .agents/skills/lancedb-vector-dedup/SKILL.md | 63 ++++++++ docs/web/basics/jobs.mdx | 73 +++++++++ docs/web/curation/create-function.mdx | 61 ++++++- docs/web/curation/dedupe.mdx | 58 ++++++- docs/web/docs.json | 1 + .../add-computed-column.mdx | 7 + .../materialized-views.mdx | 24 +++ .../feature-engineering/refresh-column.mdx | 119 ++++++++++++++ docs/web/management/function-failures.mdx | 151 ++++++++++++++++++ docs/web/management/function.mdx | 44 +++++ docs/web/search/full-text-search.mdx | 57 +++++++ docs/web/sql/dql/dedupe.mdx | 9 ++ docs/web/sql/dql/duplicate-pairs.mdx | 6 + docs/web/sql/dql/fts.mdx | 63 +++++++- 14 files changed, 725 insertions(+), 11 deletions(-) create mode 100644 .agents/skills/lancedb-vector-dedup/SKILL.md create mode 100644 docs/web/management/function-failures.mdx diff --git a/.agents/skills/lancedb-vector-dedup/SKILL.md b/.agents/skills/lancedb-vector-dedup/SKILL.md new file mode 100644 index 000000000..1221e4061 --- /dev/null +++ b/.agents/skills/lancedb-vector-dedup/SKILL.md @@ -0,0 +1,63 @@ +--- +name: lancedb-vector-dedup +description: Plan and run Enterprise IVF vector deduplication with dedupe and duplicate_pairs, including sampling, index coverage, recall and resource limits. Use for near-duplicate embedding or perceptual-hash records; use MinHash guidance for text shingles. +--- + +# Vector deduplication + +Read the maintained [workflow](../../../docs/web/curation/dedupe.mdx), +[DEDUPE reference](../../../docs/web/sql/dql/dedupe.mdx) and +[DUPLICATE_PAIRS reference](../../../docs/web/sql/dql/duplicate-pairs.mdx) +before constructing SQL. In a source checkout these paths are relative to this +skill directory. When using an installed copy without the docs, read +`https://docs.lancedb.com/curation/dedupe` and the linked SQL references. + +## Establish the comparison contract + +- Confirm an Enterprise deployment that supports these table functions. Use + `dedupe` and `duplicate_pairs`; legacy `vector_dedup` names are rejected. +- Inspect the vector type, metric, compatible index, indexed fragments and + source version. Wait for index coverage after writes. Do not run a full + scan over unindexed fragments or silently change the index to make it run. +- Explain that candidates are compared only within physical IVF partitions + and segments. Encoded/quantized distances are not an exact raw-vector + comparison. L2 thresholds are squared distances; Hamming counts bits. +- Use `id_column` on pair queries for business IDs. Retain `dataset_version`, + segment and partition identities. Do not decode internal row IDs or assume + that representatives stay the same after later data/index changes. + +## Bound the first experiment + +Use a reproducible partition sample before a full scan: + +```sql +SELECT id_a, id_b, distance, dataset_version, segment_id, partition_id +FROM duplicate_pairs( + 'frames', 'embedding', + distance_threshold => 0.02, + id_column => 'frame_id', + selection => 'random', count => 10, seed => 42 +) +``` + +Adapt the threshold and sample size to the metric and actual partition count. +Sampling partitions is not uniform sampling of rows. Inspect known duplicate +and distinct examples, pair density, partition skew, memory, spill space and +elapsed time. Do not infer global recall or ten-billion-row qualification +from a small or unversioned benchmark. Pair work and output can be quadratic +within a partition; an outer SQL `LIMIT` is not an upstream work budget. + +## Choose the result + +`dedupe` returns original representative rows and always scans all partitions. +Use its default `connected_components` for transitive groups; choose +`direct_representative` when chains should not merge transitively. It does not +accept `id_column`, `selection`, `count` or `seed`; `duplicate_pairs` does not +accept `resolution`. + +Keep pair data on the server when possible; avoid collecting an unbounded +edge set in client memory. Persist a new table/view only within the user's +authorized scope, preserving source rows unless deletion was requested. +Do not register a Python Function or fetch/rewrite Blob payloads to invoke +these built-in table functions. For MinHash/text deduplication, follow the +separate `https://docs.lancedb.com/enterprise/sql-minhash` workflow. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index 47285f5d0..264acd728 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -33,6 +33,46 @@ print(job.status()) ``` +## Cached properties and live calls {#snapshots} + +Python job properties describe the last observation; reading them does not +contact the server. Submission need not fetch the full registry record, so +some properties can initially be `None`. + +| Surface | Contract | +| --- | --- | +| `state` | Last observed lifecycle state. It does not update on its own. | +| `job_type`, `creation_ms` | Cached job kind and creation time in epoch milliseconds. | +| `spec` | Complete, job-specific submitted specification. Read this separately from the bounded `repr(job)` display. | +| `result` | Cached, job-specific reported data; it is not necessarily the typed object returned by `wait()`. | +| `failure` | Cached `JobFailureInfo`, when the server reports a terminal failure. | +| `status()` | Live lifecycle request. For registry jobs it also caches the full record. A failed state is returned without raising a job-failure exception; request errors can still raise. | +| `refresh()` | Fetch and cache the current record without waiting for completion. | +| `events(...)` | Separate live request returning a `pyarrow.Table` of recorded events; it does not refresh the properties. | +| `wait(...)` | Poll until completion and return the operation's typed result, or raise on failure/cancellation. Refresh afterward when you need the complete final registry record. | + +For example, poll a registry handle and then retrieve its final details: + +```python +import time + +state = job.status() +while state == "running": + print(job.state) + time.sleep(2) + state = job.status() + +result = job.wait() +job.refresh() +print(result) +print(job.spec) +``` + +The four lifecycle strings below are the usual SDK contract; newer server +states may pass through an older client. Do not infer execution progress from +`state` alone. Structured progress methods and queue fields require matching +SDK/server support; see [refresh admission](/feature-engineering/refresh-column#admission). + ## Submit and wait for a job {#waiting} 1. **Submit the operation.** Keep the returned job handle. For an Enterprise @@ -133,6 +173,39 @@ for info in db.list_jobs(page_limit=50): ``` +`list_jobs` yields `JobInfo` summaries, newest creation time first, rather than +live `Job` handles. Open an ID to inspect its full record. `page_limit` bounds +each server request, not the total yielded items, and `page_token` resumes a +listing. It is a lazy iterator: iteration performs requests and can raise +request errors. To materialize it, use `list(db.list_jobs(...))`; it does not +support list indexing, `len()`, or a `.to_list()` method. + +`Listing.num_page_results()` reports cached items remaining in the current +page and `Listing.page_token()` reports the next request's token. Drain the +cached items before saving that token, or resuming can skip them. A `None` +token after the final page can coexist with unread cached items. A resumed +listing is not a transactionally frozen view of jobs submitted in the meantime. + +`events.schema` describes the Arrow response. Use `events.to_pylist()` for +Python row dictionaries. Common event columns are `event_seq` (int64), `state` +(string), `updated_at_millis` (int64), `updated_by` (string), and nullable +`emitted_from`, `emitted_by` and `claim_entity` strings. Optional uint64 work +counters such as `rows_processed` and diagnostic details depend on job type +and server version. If a `payload` column is returned, its value is a nullable +JSON string, not an already decoded Python dictionary. +Empty results retain their schema. Events are newest first and belong to the +requested job; child events are separate. An event count is not a completed-row +count, and a running job can have no terminal `result` while producing events. + +For a Function refresh, `wait()` returns a `RefreshColumnResult` such as +`rows_assigned`, `rows_failed`, `rows_remaining`, `source_version` and optional +`published_version`. Other job types have different results. Do not parse the +printed representation to recover a spec, result or progress measurement. +For object-shaped registry data, `job.spec` and `job.result` are decoded Python +dictionaries: access fields with keys, such as `job.result.get("summary")` +after refreshing a successful Function job. The separate typed object from +`wait()` uses attributes, such as `result.rows_assigned`. + Event responses have [row limits](/basics/limits#jobs). An explicit limit can request more than the default, but it does not guarantee a complete event history. diff --git a/docs/web/curation/create-function.mdx b/docs/web/curation/create-function.mdx index 111eaf13f..9ee7d572a 100644 --- a/docs/web/curation/create-function.mdx +++ b/docs/web/curation/create-function.mdx @@ -44,6 +44,58 @@ table.add_columns({"doubled": version(value=col("value"))}) immutable version, so changing one produces a new version rather than altering rows a previous version already filled. +## Supported Function signatures {#signature-types} + +A table's [Arrow schema](/sql/data-types#supported) can contain types that a +Function cannot accept or return. The matrix below describes the Python SDK +`0.42.0-beta.12` source contract. Nested schemas and list-Blob outputs require +a matching Enterprise server; an SDK upgrade alone does not enable them. + +| Arrow type | Function input/output support | +| --- | --- | +| Boolean; signed/unsigned 8, 16, 32, 64-bit integers; float16/32/64 | Supported. Ordinary `int` and `float` annotations infer int64 and float64. | +| utf8 / large_utf8; binary / large_binary | Supported. `str` and `bytes` infer utf8 and binary. Plain binary does not carry Blob semantics. | +| date32 / date64 | Supported; Python `date` infers date32. | +| timestamp, with or without timezone; time32/64; duration; interval | Unsupported by the Function signature grammar. A Python `datetime` annotation can resolve to an Arrow timestamp but registration still rejects it. | +| decimal; map; dictionary; union; fixed-size binary | Unsupported. Decode dictionary values and use supported scalar/list/struct representations. | +| struct; list / large_list; positive-length fixed-size list | Supported with supported children. Explicit schemas preserve child nullability; annotation-only `list[T]` uses non-nullable elements. Struct fields must be nonempty and uniquely named. | +| Blob v2 at top level or in a struct | Supported in inputs and outputs; use `lancedb.blob(...)` in the schema. | +| Blob v2 under list / large_list | Output only on supporting SDK/server versions. Inputs are rejected. | +| Blob v2 under fixed-size list or map | Unsupported in both directions. | + +For times, pass an int64 epoch value with an explicit unit such as microseconds, +or a UTF-8 ISO 8601 string with its timezone. Convert inside the Function and +cast the result in SQL if needed. Use scaled integers with a declared scale +when exact decimal values matter; converting to float can lose precision. +Represent a map as a list of key/value structs when that fits the calculation. + +A scalar output is non-nullable because physical NULL represents an unfilled +computed output. For an optional value, return a named struct with a nullable +field, for example: + +```python +import pyarrow as pa +from lancedb import function + +@function( + input_schema=pa.schema([pa.field("text", pa.string(), nullable=False)]), + output_schema=pa.schema([pa.field("score", pa.float64(), nullable=True)]), +) +def optional_score(text: str): + return {"score": float(len(text)) if text else None} +``` + +Return a dictionary by field name. A successfully computed nullable field can +be NULL; `score IS NULL` alone cannot distinguish it from pending work or a +skipped failure. Use the job result and error diagnostics. Do not assume an +explicit assignment-state flag is available on every deployment. + +List-nested Blob outputs are queryable as descriptors, but row-ID blob fetch +APIs and `blob_mode="bytes"` do not fetch payloads inside lists. Prefer a +separate row per media item when you need independent reads, retries or +re-scoring. A plain struct shaped like a Blob without its extension metadata +is not a substitute for `lancedb.blob`. + ## Batch: once per call, many rows at a time {#batch} A function is called once per batch instead of once per row when its @@ -170,9 +222,12 @@ in the function's source. A Secret is a database-scoped named credential: puts it in the environment variable the function reads, without ever returning it from any API or recording it in the function's own definition. Combine it with the stateful form so the HTTP client -- and any rate limiter -around it -- is built once, not reopened on every row. There is no built-in -retry or per-row error-skip policy -- a function that calls a flaky service -should handle timeouts and retries itself, inside `__call__`. +around it -- is built once, not reopened on every row. Configure HTTP timeouts +inside `__call__`. Refresh-level `error_policy='skip'` can record row failures; +see [handling failures](/management/function#failures). Native retry policies +need matching SDK, server and Function-image support; on other deployments, +handle bounded retries inside the Function only when repeating the request is +safe. See [Vend a Secret to a Function](/curation/vend-secret#specify) for the full worked example, how the bound value actually reaches a running diff --git a/docs/web/curation/dedupe.mdx b/docs/web/curation/dedupe.mdx index 470e170c2..2e90bc231 100644 --- a/docs/web/curation/dedupe.mdx +++ b/docs/web/curation/dedupe.mdx @@ -136,12 +136,60 @@ SELECT * FROM dedupe( See [Refresh a materialized view](/feature-engineering/refresh-materialized-view) for manual and automatic refresh options to keep it current as new records land. -## At production scale {#production} +## Index, snapshot and recall limits {#limits} -[LeRobotDataset](/integrations/lerobotdataset) runs the same `dedupe` -call over 27.6 million real embedding rows to remove near-static frames from a -robotics training set in under 30 seconds -- the mechanism does not change -with scale, only the table it runs against. +These are Enterprise SQL table functions, not standard SQL or a registered +Python Function. They require one compatible current vector index covering +the scanned snapshot. Wait for indexing to finish after ingestion and inspect +index coverage before running a full scan. Unindexed fragments are rejected; +deleted rows are excluded at the pinned dataset version. + +The scan compares candidates **within each physical IVF partition and index +segment**. It does not compare across partitions or segments. Scanning every +partition therefore does not establish global all-pairs recall. Quantized +indexes compare their encoded vector representations; `IVF_FLAT` compares the +stored vectors. Thresholds must match the index's metric: L2 is squared L2, +cosine is `1 - cosine`, dot distance is `1 - dot`, and Hamming counts differing +bits. Inspect close examples and known duplicates before choosing a threshold. + +`duplicate_pairs` reports `dataset_version`, `segment_id` and `partition_id`. +Retain those with the index configuration and source version when comparing +runs. Pass `id_column` for application IDs, rather than treating physical row +addresses as durable identifiers. A new append, index change or bridge between +components can change groups and representatives. + +## Efficient use and scale qualification {#production} + +Start with [`duplicate_pairs` partition sampling](/sql/dql/duplicate-pairs#sampling), +using a recorded seed and a bounded partition count. Inspect pair density and +threshold quality before running `dedupe`, which always scans every partition. +Sampling partitions is not uniform row sampling and can miss rare groups. +`resolution='connected_components'` merges transitive chains; +`direct_representative` uses direct representative relationships instead. + +Pair work grows quadratically within dense partitions, and a loose threshold +can emit a quadratic number of edges. An outer `LIMIT` is not a resource +budget for upstream pair finding or group resolution. Avoid repeatedly +materializing all pairs in a Python list; select needed columns, aggregate on +the server, and persist a reviewed result when repeated reads justify it. +Measure peak memory, spill storage, elapsed time and pair count on +representative partitions before increasing scope. + +The [LeRobotDataset integration](/integrations/lerobotdataset) illustrates a +27.6-million-row data workflow. Its timings are not a versioned qualification +of this deduplication path on arbitrary indexes or hardware. A 10-billion-row +run is not qualified by that example. Qualify source/index versions, partition +sizes, metric, threshold, worker resources, spill limits and correctness on your +deployment before extrapolating. There is no universal row-count or latency +guarantee in this guide. + +For text shingles and MinHash signatures, use the distinct +[MinHash workflow](/enterprise/sql-minhash). Vector distance deduplication and +MinHash/Jaccard text deduplication have different indexes and thresholds. + +Agents can follow the repository's +[vector deduplication skill](https://github.com/lancedb/lancedb/blob/main/.agents/skills/lancedb-vector-dedup/SKILL.md) +to apply these checks without inventing Function options or scale guarantees. ## Continue with the curated data {#next} diff --git a/docs/web/docs.json b/docs/web/docs.json index 3414fd121..c586d4f62 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -496,6 +496,7 @@ "management/materialized-view", "management/secret", "management/function", + "management/function-failures", "management/storage-integration" ] }, diff --git a/docs/web/feature-engineering/add-computed-column.mdx b/docs/web/feature-engineering/add-computed-column.mdx index 851b37d9b..7cc9f6988 100644 --- a/docs/web/feature-engineering/add-computed-column.mdx +++ b/docs/web/feature-engineering/add-computed-column.mdx @@ -13,6 +13,13 @@ import { PySchemaAddSetup, } from '/snippets/tables.mdx'; + +Declare computed columns on ordinary tables. A materialized view derives its +schema from its query; include the calculation in the view's `SELECT` instead +of attaching a Function with `add_columns`. See +[materialized-view schema changes](/feature-engineering/materialized-views#schema-changes). + + Lance can write a new column without rewriting the columns already in the table. That matters most on a table that also holds large images, video or embeddings, where rewriting everything just to add one small derived field diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index 617eb7525..ccfd8429d 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -176,6 +176,30 @@ definition is read with `await view.definition()`. Creation, refresh and drop also have async, job-returning SDK methods; see [Materialized view](/management/materialized-view#manage). +## Add derived values to a view {#schema-changes} + +A materialized view's output schema belongs to its defining `SELECT`. Do not +use `view.table.add_columns(...)` to attach another computed or Function column +to that output. In particular, adding a Function binding is rejected: the +column would not be produced by the next view refresh. + +Choose where the calculation belongs: + +- Include the expression or Function call in the view's `SELECT`, as in the + [caption example](#functions). To change an existing definition, create a + replacement view under a new name, check its results, indexes and grants, + and move readers to it before dropping the old view. +- Add a Function column to the ordinary source table, refresh it, then select + that stored column in a view. This lets you manage column refresh separately + from view refresh. +- Create an ordinary table from a query when you need independently editable + outputs. That table is a snapshot; it does not inherit the view's refresh + relationship to the source. + +Opening the underlying table gives access to reads and indexes; it does not +make independent schema changes part of the materialized-view definition. +Changing automatic-refresh settings also does not change that definition. + ## Related guides {#next} diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index 1b809b91d..d82fb1eef 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -77,6 +77,11 @@ catch-up works when it's re-enabled after being off. ## Refreshing part of a table {#partial} +Python `where=` requires a supporting SDK, such as `0.42.0b9`, and an +Enterprise server with filtered refresh support. Stable `0.41.0` does not +accept this argument. The SQL form is available independently of that Python +argument on supporting servers. + Enterprise A refresh can be scoped to a predicate, with `where=` in Python or `WHERE` in SQL, which is useful for filling a slice first and checking it before committing to the whole table: @@ -123,6 +128,89 @@ to let LanceDB track which outputs need updating. `function_refresh` for one filled by a registered [function](/management/function). +### Re-score selected list elements {#list-elements} + +`WHERE` selects **rows**, not elements within a list. Refreshing a row that +contains ten answers passes the whole list to the Function. A mask can control +which elements your code computes, but the Function still returns one complete +list for that row. There is no element-level refresh or partial-list write API. + +This example keeps `sample_id` as the link to an existing `samples` table with +a Blob column named `media`. It creates only answer metadata and scores; it +neither reads nor copies Blob payloads. Run it on a deployment supporting +`@function`, list signatures and Python `where=`, with `samples` already +containing sample IDs `s1` and `s2`. + +```python +from lancedb import col, function + +media_before = db.execute_query( + "SELECT sample_id, media FROM samples " + "WHERE sample_id IN ('s1', 's2') ORDER BY sample_id" +).read_all().to_pylist() + +answers = db.create_table("answer_rescore_demo", [ + {"group_id": "g1", "sample_id": "s1", + "answers": ["yes", "maybe", "no"], + "selected": [False, True, False], "previous": [0.1, 0.2, 0.3]}, + {"group_id": "g1", "sample_id": "s2", + "answers": ["other"], "selected": [False], "previous": [0.4]}, +]) + +@function +def rescore_masked( + answers: list[str], selected: list[bool], previous: list[float] +) -> list[float]: + if not (len(answers) == len(selected) == len(previous)): + raise ValueError("answers, selected and previous must have equal lengths") + return [ + float(len(answer)) if choose else old + for answer, choose, old in zip(answers, selected, previous) + ] + +version = db.create_function(rescore_masked) +answers.add_columns( + {"rescored": version( + answers=col("answers"), selected=col("selected"), previous=col("previous") + )}, + auto_refresh_enabled=False, +) +job = answers.refresh_column_async("rescored", where="sample_id = 's1'") +job.wait() + +rows = answers.search().select( + ["group_id", "sample_id", "previous", "rescored"] +).to_list() +by_sample = {row["sample_id"]: row for row in rows} +assert by_sample["s1"]["group_id"] == "g1" +assert by_sample["s1"]["rescored"] == [0.1, 5.0, 0.3] +assert by_sample["s1"]["previous"] == [0.1, 0.2, 0.3] +assert by_sample["s2"]["rescored"] is None +assert by_sample["s2"]["previous"] == [0.4] + +media_after = db.execute_query( + "SELECT sample_id, media FROM samples " + "WHERE sample_id IN ('s1', 's2') ORDER BY sample_id" +).read_all().to_pylist() +assert media_after == media_before +``` + +Here length is a deterministic stand-in for a scorer. Only `"maybe"` is scored; +the other scores are copied from `previous`. The final equality checks Blob +descriptors on an otherwise unchanged source table, without fetching payloads. +Keep the source table unchanged during this comparison. Use the same model, +normalization and score type as the original computation in a real re-scoring +pass. For another pass, explicitly supply the intended prior scores and mask; +refresh does not merge the previous output into your Function's inputs. + +If answers need independent retries or versions, prefer one ordinary table row +per answer, with `group_id`, `sample_id`, a stable application `answer_id`, +answer text and score. Keep media in `samples` and join by `sample_id` when +needed. A row predicate can then select exact answer IDs, and Blob storage +stays in the source table. Do not use `UNNEST` to assume that nested Blob values +become supported Function inputs; see the +[nested-Blob limits](/curation/create-function#signature-types). + ## Run a refresh on Enterprise {#async} Enterprise @@ -212,6 +300,31 @@ The wait helpers do not submit or cancel a refresh. They retry only the typed finalization-pending response and propagate other errors. */} +### Admission and waiting {#admission} + +Submission does not reserve a supervisor or GPU immediately. A refresh can wait +for source preparation/compaction, an earlier refresh, executor admission or +worker capacity. A lifecycle `running` value from `status()` alone does not +prove that Python code has begun executing. + +Use [`SHOW JOBS`](/sql/ddl/job#show) and the Console's job detail to inspect +available phase and row progress. On SDK/server versions with `job.progress()`, +the snapshot can also include `phase`, `queue_reason`, timestamps and row +counters. These optional fields are version-dependent; missing values are +unknown, not zero progress or an ETA. `creation_ms` is submission time, not +worker start time. `DESCRIBE JOB` provides the finished run report, rather than +a live queue-position API. + +Limits are deployment settings: the Agent dispatcher concurrency, executor +capacity, per-table automatic-refresh limit and Function-call +`max_concurrency` control different things. There is no universal five-refresh +limit per database. Ask your administrator for the effective settings and use +the [Enterprise operations guide](/enterprise/function-operations#admission) +for their configuration names. Repeated cancellation and resubmission does not +bypass those limits. Cancellation requests a stop, including while queued; +observe a terminal state before submitting replacement work. Committed work +is not rolled back. + ### Configure workers and resources {#resources} SQL exposes the worker budget directly, which is the lever when a backfill is @@ -229,6 +342,12 @@ up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not per-worker allocations or a guaranteed worker count. `num_workers` is only accepted for SQL-expression columns; registered Function columns reject it. +In particular, `gpu_count = 2` does not request two devices inside one Python +process or enable tensor parallelism. Inspect configured GPU classes and their +per-worker limits with `db.list_gpu_classes()` or `SHOW GPU CLASSES`. Choose a +model that fits the worker shape supported by your deployment; see +[GPU placement and budgets](/enterprise/function-operations#gpu). + The SDKs accept the same nested budget. CPU and memory are required; the GPU count inside the budget defaults to zero, so a CPU-only Function can omit it: diff --git a/docs/web/management/function-failures.mdx b/docs/web/management/function-failures.mdx new file mode 100644 index 000000000..4546129fa --- /dev/null +++ b/docs/web/management/function-failures.mdx @@ -0,0 +1,151 @@ +--- +title: "Handle Function failures" +sidebarTitle: "Function Failures" +description: "Keep usable outputs, inspect row errors, retry by stable identity and isolate inputs that need attention." +tag: "Early access" +availability: + embedded: unavailable + enterprise: available +--- + +A Function refresh can encounter a bad row, fail to initialize a model, lose +a worker, or fail while publishing results. These need different remedies. +This guide uses the existing [refresh error policy](/sql/dml/refresh-column#options) +and Python error records; it does not require a new retry or quarantine API. + +## Choose a row-error policy {#policy} + +The default `fail` stops the job on a row error. `skip` writes null outputs for +failed rows and records errors, subject to the refresh's failure limits: + +```python +job = table.refresh_column_async( + "embedding", + error_policy="skip", + max_failed_rows=100, + max_failed_fraction=0.1, +) +result = job.wait() +print(result.rows_assigned, result.rows_failed, result.rows_remaining) +``` + +Choose limits for your workload; these example limits stop work when more +than 100 rows or more than 10% fail. `skip` does not make initialization, +infrastructure or publication failures successful. If `wait()` raises, +[refresh the job record](/basics/jobs#snapshots), inspect `job.failure`, and +inspect any recorded row errors. A failed/cancelled job can have committed +some output, but retention of every successful call is not guaranteed. +External effects are not rolled back and can be repeated during recovery. + +## Inspect and group recorded errors {#inspect} + +Scope diagnostics to the job you are investigating: + +```python +from collections import Counter + +errors = table.function_errors(job_id=job.id, column="embedding", limit=10000) +groups = Counter((r.error_type, r.error_message) for r in errors.records) +for cause, count in groups.most_common(): + print(count, cause) +for fragment in errors.fragments: + print(fragment.fragment_id, fragment.rows_skipped, fragment.rows_recorded) +print("listing truncated:", errors.truncated) +``` + +Each record includes the Function name/version, table version, error type and +message, creation time, and available row identities. `attempts` and traceback +details depend on the server/image version. This client-side grouping counts +only returned detail records. `truncated=True` or fragment summaries with +more skipped than recorded rows mean it is incomplete; do not label it a full +error census. See [`SHOW ERRORS`](/sql/ddl/job#errors) for SQL and its limits. + +Records are historical observations, not a deduplicated list of currently +failed rows. Reading errors without `job_id` can include older attempts that +have since succeeded. A null output can mean unfilled, skipped, or a computed +nullable field; output nullness alone is not a current-error test. + +## Retry exactly identified rows {#retry} + +On tables with stable row IDs, supporting server/SDK versions return +`record.row_id` from the refresh's pinned source version. Keep that table and +branch identity with the records. Compaction can move physical addresses, so +do not derive a row ID from fragment numbers or offsets. + +The following deliberately refuses incomplete or unidentified error sets. +Review historical records against the current inputs before using them as a +retry selection. Keep this selection small enough for a SQL predicate. + +```python +if errors.truncated or any(f.rows_skipped > f.rows_recorded for f in errors.fragments): + raise RuntimeError("Inspect a narrower job/error set before selecting retries") +if any(r.row_id is None for r in errors.records): + raise RuntimeError("This workflow requires stable row IDs in every error record") + +row_ids = sorted({int(r.row_id) for r in errors.records}) +if row_ids: + predicate = "_rowid IN (" + ", ".join(map(str, row_ids)) + ")" + retry_job = table.refresh_column_async("embedding", where=predicate) + retry_job.wait() +``` + +This recomputes every selected live row with the existing binding. It can +repeat a row already repaired since the error was recorded. Inspect the retry +job and its errors afterward; an older record need not disappear. On tables +without stable row IDs, resolve errors to application keys at the recorded +snapshot before retrying. Do not substitute `_rowaddr` after a rewrite. +Python `where=` requires a supporting SDK such as `0.42.0b9`, not stable +`0.41.0`, plus a matching server. + +## Repair with a newer Function {#new-version} + +Registering a new version does not change an existing column's immutable +binding. To evaluate a fix while retaining old outputs, register the corrected +callable and declare a separate output column. For example, with a corrected +`embed_fixed` callable already defined and a `text` input column: + +```python +from lancedb import col + +fixed = db.create_function(embed_fixed) +table.add_columns( + {"embedding_repaired": fixed(text=col("text"))}, + auto_refresh_enabled=False, +) +if row_ids: + repair_job = table.refresh_column_async("embedding_repaired", where=predicate) + repair_job.wait() +``` + +Only the reviewed failed-row selection is populated in the new column; other +rows remain unfilled. Compare those outputs before deciding how readers +should use them. This does not promise an in-place upgrade/retry operation or +per-row producing-version lookup for a column containing mixed executions. +Error records' Function versions describe failures, not the provenance of +every successful row. + +## Quarantine inputs for review {#quarantine} + +Use an application-owned table of source table/branch, recorded table version, +stable row ID or business key, Function version, cause and review disposition. +Keep Blob payloads in the source table; a quarantine entry can refer to their +source identity. Choose an ordinary table name such as `function_quarantine`, +and explicitly exclude its reviewed IDs in subsequent input selections. + +Quarantine is an application workflow here: there is no implied built-in +quarantine lifecycle, automatic current-error clearing or exactly-once retry. +Do not delete source rows merely to hide an error. For very large failure sets, +bounded error listings cannot select every row; use a smaller reviewed scope +or wait for a deployment with the required complete selection support. + +Native retry policies, full current-error summaries, in-place version upgrades +and materialized-view row-error handling have separate server/SDK prerequisites. +Do not apply `REFRESH COLUMN` options to a materialized view and assume the +same policy; consult the [MV refresh reference](/sql/dml/refresh-materialized-view) +for your deployment's accepted options. + +## Continue {#next} + +- [Function lifecycle and failures](/management/function#failures) +- [Conditional and list-element refresh](/feature-engineering/refresh-column#partial) +- [Job results and events](/basics/jobs#snapshots) diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index aae67ceb9..27f45cb6a 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -420,6 +420,9 @@ full worked example. ## Failure handling {#failures} +For the full skip, inspect, grouped-error, stable-row-ID retry and quarantine +workflow, see [Handle Function failures](/management/function-failures). + By default, a Function that raises fails the refresh job. Choose how the refresh handles row errors with its error policy: @@ -495,6 +498,47 @@ In practice this means a function should be self-contained: take its inputs as arguments, import what it needs at module level, and avoid capturing state from the surrounding scope. +### Supported global captures {#captures} + +The Python SDK's source packager captures only these literal values: + +| Module-level value | Packaging behavior | +| --- | --- | +| `None`, exact `bool`, `int`, `str`, `bytes`, finite `float` | Inlined as a snapshot at registration. | +| Tuples containing only those values, including nested tuples | Recursively inlined. | +| Imported modules, importable functions/classes | Reconstructed as imports; declare their packages in `pip` or `conda`, or ship local source with `code`. | +| Arrow types/schemas, lists, dictionaries, sets, NumPy arrays, non-finite floats, other objects | Not captured as literals. Construct them inside the Function or class. | + +For example, a module-level `OUT_TYPE = pa.list_(pa.float32(), 3)` referenced +from the Function body is unsupported. Construct the type where it is used: + +```python +import pyarrow as pa +from lancedb import function + +@function( + input_schema=pa.schema([pa.field("value", pa.float32(), nullable=False)]), + output_schema=pa.list_(pa.float32(), 3), +) +def make_vector(value: pa.Array) -> pa.Array: + out_type = pa.list_(pa.float32(), 3) + return pa.array([[v, v, v] for v in value.to_pylist()], type=out_type) +``` + +Decorator schema arguments are serialized as the signature; this restriction +concerns values referenced by the packaged body. For an expensive constant or +model, build it in a class Function's `__init__` and keep it on `self`. + +Later mutations of a client global do not update a registered version. Register +a new version to change captured literals or source. The diagnostic fix in +[SDK #4938](https://github.com/lancedb/lancedb/pull/4938) names the unsupported +global and its type; it does not add Arrow/container capture support. There is +no supported arbitrary-object serialization or documented container-size limit +to rely on for those rejected types. + +See the [Function type matrix](/curation/create-function#signature-types) for +what can travel as row inputs and outputs rather than captured globals. + Pass `code=[module, ...]` to ship additional helper modules into the same image, for logic the function imports but that does not belong inlined into it -- a model-loading helper shared by several Functions, for example. diff --git a/docs/web/search/full-text-search.mdx b/docs/web/search/full-text-search.mdx index 9c17fc50b..b7ceb15ad 100644 --- a/docs/web/search/full-text-search.mdx +++ b/docs/web/search/full-text-search.mdx @@ -182,6 +182,56 @@ BM25 scores depend on the indexed corpus and its term statistics. Compare rankings on the dataset you will search; treat scores as relative relevance values rather than probabilities. +### Match any word or every word {#term-operators} + +A multi-word string such as `"happy puppy"` uses **OR**: a row can match either +term. Writing `"happy AND puppy"` does not select an AND operator; the string +is analyzed as search text. To require both terms, construct a match query: + + +```python Python icon="python" +from lancedb.query import FullTextOperator, MatchQuery + +query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND) +results = table.search(query, query_type="fts").limit(10).to_list() +``` + +```typescript TypeScript icon="square-js" +import { MatchQuery, Operator } from "@lancedb/lancedb"; + +const query = new MatchQuery("happy puppy", "text", { operator: Operator.And }); +const results = await tbl.query().fullTextSearch(query).limit(10).toArray(); +``` + +```rust Rust icon="rust" +use lance_index::scalar::{ + FullTextSearchQuery, + inverted::query::{MatchQuery, Operator}, +}; +use lancedb::query::{ExecutableQuery, QueryBase}; + +let query = MatchQuery::new("happy puppy".to_owned()) + .with_column(Some("text".to_owned())) + .with_operator(Operator::And); +let results = tbl + .query() + .full_text_search(FullTextSearchQuery::new_query(query.into())) + .limit(10) + .execute() + .await?; +``` + + +AND requires all analyzed terms, without requiring adjacency or order. Use +[`PhraseQuery`](#phrase-vs-terms-queries) for an ordered phrase, or combine +separate queries with [`&` and `|`](#boolean-queries). Python `and` and `or` +do not build queries. + +On Enterprise, the existing [`fts()` SQL reference](/sql/dql/fts#term-operators) +shows the equivalent JSON (`"operator":"And"`), `_score`, and a +[`COUNT(*)` workaround](/sql/dql/fts#count) for counting matches without a +top-k limit. `len(results)` counts only the returned result page. + ### Keeping the index up to date {#keeping-the-index-up-to-date} In LanceDB Embedded, new rows can remain outside the FTS index until you optimize the table. @@ -395,6 +445,13 @@ table.create_index("text", config=FTS(with_position=True, remove_stop_words=Fals Storing token positions increases the index size and adds indexing work. +On Enterprise, wait for the replacement index to finish before issuing the +phrase query, using `table.wait_for_index(["text_idx"])` with your actual +index name. An existing index without positions must be rebuilt; changing the +query does not add positions. A phrase query against such an index is rejected +with an error asking you to recreate it with positions, rather than returning +an empty match set. See [FTS index options](/indexing/fts-index#fts-parameters) +and the [SQL phrase reference](/sql/dql/fts#phrase). ### Fuzzy Search {#fuzzy-search} diff --git a/docs/web/sql/dql/dedupe.mdx b/docs/web/sql/dql/dedupe.mdx index dbb6717b4..452638ea3 100644 --- a/docs/web/sql/dql/dedupe.mdx +++ b/docs/web/sql/dql/dedupe.mdx @@ -55,6 +55,15 @@ current deployments. a subset of partitions and `id_column` are pair-query-only options, accepted by [`duplicate_pairs`](/sql/dql/duplicate-pairs#input) but refused here. +## Index and snapshot requirements {#requirements} + +Use one compatible, current vector index covering the source snapshot. +Full scans reject unindexed fragments. Comparisons stay within physical IVF +partitions and index segments; this does not enumerate cross-partition or +cross-segment duplicates. Quantized-index distances use encoded vector +representations. See [recall and scale limits](/curation/dedupe#limits) before +treating the result as globally exact deduplication. + ## Output {#output} One row per duplicate group, **in the table's own schema** -- every column of diff --git a/docs/web/sql/dql/duplicate-pairs.mdx b/docs/web/sql/dql/duplicate-pairs.mdx index daa078955..dc93d5180 100644 --- a/docs/web/sql/dql/duplicate-pairs.mdx +++ b/docs/web/sql/dql/duplicate-pairs.mdx @@ -60,6 +60,12 @@ only [`dedupe`](/sql/dql/dedupe#input) accepts it. ## Output {#output} +See [`dedupe` index prerequisites](/sql/dql/dedupe#requirements) and the +[recall limits](/curation/dedupe#limits). Pairs are found within each physical +partition and index segment, not across them. A full scan requires complete +index coverage; sampling is a diagnostic view of the selected indexed +partitions and does not prove coverage or recall for the rest of the table. + One row per close pair found, in a fixed schema independent of the source table: diff --git a/docs/web/sql/dql/fts.mdx b/docs/web/sql/dql/fts.mdx index 7b52f5e17..667d92cd6 100644 --- a/docs/web/sql/dql/fts.mdx +++ b/docs/web/sql/dql/fts.mdx @@ -30,14 +30,16 @@ INSERT INTO my_docs VALUES (3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'), (5, 'Puppy training requires patience and dedication', 'training', 'Alice'); -CREATE INDEX ON my_docs USING fts (text); +CREATE INDEX ON my_docs USING fts (text) WITH (with_position = true); ``` Phrase queries need position information, which costs extra index size. -Add `WITH (with_position = true)` to `CREATE INDEX`. Without it, -[phrase queries](#phrase) find nothing. See +The setup above enables it with `WITH (with_position = true)`. An existing +index without positions must be rebuilt before running a phrase query; the +server rejects the query and asks you to recreate the index with positions. +It does not silently return no matches. See [FTS parameters](/indexing/fts-index#fts-parameters). @@ -86,6 +88,44 @@ LIMIT 5 it behaves like any other column, but exists only in a relation that came from `fts()`. See [Data types](/sql/data-types#pseudo-columns). +## Match any term or every term {#term-operators} + +The default match operator is OR. `"happy puppy"` matches either analyzed +term. SQL JSON uses `"Or"` and `"And"`, with this casing; Python uses the +`FullTextOperator.OR` and `FullTextOperator.AND` enum members: + +```python +from lancedb.query import FullTextOperator, MatchQuery + +query = MatchQuery("happy puppy", "text", operator=FullTextOperator.AND) +json_query = query.to_json() +``` + +```sql +SELECT id, text, _score +FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}') +ORDER BY _score DESC +``` + +Both terms must occur, but they need not be adjacent. Do not embed `AND` or +`OR` in `terms` to select an operator. + +## Count matches {#count} + +Count the `fts()` relation before applying a top-k limit: + +```sql +SELECT COUNT(*) AS matching_rows +FROM fts('my_docs', '{"match":{"column":"text","terms":"happy puppy","operator":"And"}}') +``` + +This is an Enterprise SQL workaround when the SDK's FTS query builder does +not expose a match-count operation. `table.count_rows()` without an FTS +relation counts table rows, and taking the length of a limited search counts +only that result page. For list-element indexes, multiple matching elements +can produce multiple rows; use `COUNT(DISTINCT id)` when you need unique +application records. + ## Fuzzy {#fuzzy} Tolerates typos, up to a character-edit budget: @@ -109,6 +149,12 @@ PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them ``` +```sql +SELECT id, text, _score +FROM fts('my_docs', '{"phrase":{"column":"text","terms":"happy puppy","slop":0}}') +ORDER BY _score DESC +``` + ## Boolean {#boolean} Combine queries with `&` (AND) or `|` (OR): @@ -120,6 +166,17 @@ MatchQuery("puppy", "text") | MatchQuery("kitten", "text") ``` +For SQL, serialize the composed query rather than writing occurrence enums +by hand: + +```python +query = MatchQuery("puppy", "text") & MatchQuery("happy", "text") +json_query = query.to_json() +``` + +Use that string as the second `fts()` argument. `.to_json()` supplies the +serialized Boolean field names for the SDK version you use. + ## Boost {#boost} Weight a query up or down relative to another: