diff --git a/docs/web/.mintignore b/docs/web/.mintignore index 388087951..8fea3d17d 100644 --- a/docs/web/.mintignore +++ b/docs/web/.mintignore @@ -1 +1,8 @@ tools/cli.mdx +# Branching is Beta; not documented until early access. tables/branching.mdx +# stays reachable (out of the nav only): SDK error messages link to it. +# To restore: docs.json Tutorials > Feature Engineering, after tutorials/feature-engineering/materialized-views. +agent-branch-experiments.mdx +# JSON columns are Beta; not documented until early access. +# To restore: docs.json Documentation > SQL Reference > DQL, after sql/dql/vector-distance. +sql/dql/json-functions.mdx diff --git a/docs/web/basics/embedded-vs-enterprise.mdx b/docs/web/basics/embedded-vs-enterprise.mdx index c11324300..da58acb7d 100644 --- a/docs/web/basics/embedded-vs-enterprise.mdx +++ b/docs/web/basics/embedded-vs-enterprise.mdx @@ -78,7 +78,6 @@ file format, so moving data from one edition to the other does not require a dat | | [Database](/management/database) | ✅ | ✅ | | | [Namespace](/management/namespace) | ✅ | ✅ | | | [Table](/management/table) | ✅ | ✅ | -| | [Branches](/tables/branching) | ✅ | ✅ | | | [View](/management/view) | ❌ | ✅ | | | [Materialized view](/management/materialized-view) | ❌ | ✅ | | | [Secret](/management/secret) | ❌ | ✅ | @@ -104,6 +103,11 @@ file format, so moving data from one edition to the other does not require a dat | | [OpenTelemetry](/deployment/opentelemetry) | ✅ | ✅ | | | [Kubernetes](/enterprise/deployment/kubernetes) | ❌ | ✅ | +{/* +Draft until branching reaches early access; restore to the Management rows. +| | [Branches](/tables/branching) | ✅ | ✅ | +*/} + ## Usage differences between Enterprise and Embedded {#usage-differences-between-enterprise-and-oss} The [quickstart](/quickstart) guide shows both LanceDB Embedded connections and Enterprise `db://...` diff --git a/docs/web/basics/naming.mdx b/docs/web/basics/naming.mdx index 165840218..b00f50bf4 100644 --- a/docs/web/basics/naming.mdx +++ b/docs/web/basics/naming.mdx @@ -10,7 +10,7 @@ keywords: ["names", "naming", "table name", "namespace", "identifier", "limits"] -Databases, namespaces, tables, views and branches all have names. Choose names +Databases, namespaces, tables and views all have names.{/* Draft until branching reaches early access: restore "Databases, namespaces, tables, views and branches all have names." */} Choose names that describe their purpose, such as `documents`, `prod` or `training_2026`. Consistent names make data easier to find and application code easier to read. @@ -21,10 +21,14 @@ Use the following rules for names that work across LanceDB Embedded and LanceDB | Table, materialized view, view | ASCII letters and digits, `-`, `_`, `.` | 1 to 128 characters | | Namespace component | ASCII letters and digits, `-`, `_`, `.` | 1 to 64 characters; at most 10 components in a path | | Database | ASCII letters and digits, `-`, `_`, `.`, with `/` between components | 1 to 64 characters for the whole name | -| Branch | Avoid `$` and `:`; use ASCII letters, digits, `-` and `_` for portability | Keep to 1 to 64 characters | | Secret | ASCII letters and digits, `-`, `_`, `.` | 1 to 128 characters | | Function | ASCII letters and digits, `_` | 1 to 128 characters | +{/* +Draft until branching reaches early access; restore after the Database row. +| Branch | Avoid `$` and `:`; use ASCII letters, digits, `-` and `_` for portability | Keep to 1 to 64 characters | +*/} + - Names and individual namespace or database components must not be empty. - Table, view, namespace and database components cannot be `.` or `..`. - Secret names cannot be `.` or `..`. @@ -60,5 +64,9 @@ The empty path `[]` identifies the root namespace. ## Learn more {#next} - [Namespaces](/management/namespace): create namespaces and address tables. -- [Branches](/tables/branching): name and manage separate lines of table history. - [Limits](/basics/limits): check other constraints before designing your schema. + +{/* +Draft until branching reaches early access; restore after Namespaces. +- [Branches](/tables/branching): name and manage separate lines of table history. +*/} diff --git a/docs/web/curation/create-function.mdx b/docs/web/curation/create-function.mdx index 232cf79dc..a2751cf8f 100644 --- a/docs/web/curation/create-function.mdx +++ b/docs/web/curation/create-function.mdx @@ -1,5 +1,6 @@ --- title: "Create a Function" +tag: "Early access" sidebarTitle: "Create Function" description: "Register Python as a column: CPU, GPU, and remote-service Functions, called once per row or once per batch." icon: code @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + Some curation steps cannot be written as a SQL expression: an embedding model, an image decode, a call to an external service. A Function is a Python callable registered with the database, versioned, and then applied as diff --git a/docs/web/curation/gpu-function-execution.mdx b/docs/web/curation/gpu-function-execution.mdx index f769faef4..e7f6f0f6a 100644 --- a/docs/web/curation/gpu-function-execution.mdx +++ b/docs/web/curation/gpu-function-execution.mdx @@ -1,6 +1,7 @@ --- title: "GPU Function Execution" -sidebarTitle: "GPU Function Execution" +tag: "Early access" +sidebarTitle: "GPU Functions" description: "How a GPU Function's worker gets scheduled onto a GPU-capable node, and what happens when none is free." icon: microchip keywords: ["gpu", "gpu class", "scheduling", "function", "udf", "kubernetes", "node selector"] @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + [Registering a GPU Function](/curation/create-function#gpu) sets `gpu=True` and an optional `gpu_class`; this page covers what happens after that, when a job actually needs a worker. An operator configures one or more named diff --git a/docs/web/curation/vend-secret.mdx b/docs/web/curation/vend-secret.mdx index 6a16fcdf0..801f16436 100644 --- a/docs/web/curation/vend-secret.mdx +++ b/docs/web/curation/vend-secret.mdx @@ -1,5 +1,6 @@ --- title: "Vend a Secret to a Function" +tag: "Early access" sidebarTitle: "Vend Secret" description: "Bind a Secret to a Function by reference, and how its value reaches running code during SQL evaluation and refresh." icon: key @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + A Function that calls an external service needs a credential without putting it in the function's source -- see [Create a Function](/curation/create-function#remote) for how to register diff --git a/docs/web/docs.json b/docs/web/docs.json index bdc849f63..58ad78a52 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -590,7 +590,6 @@ "sql/dql/dedupe", "sql/dql/duplicate-pairs", "sql/dql/vector-distance", - "sql/dql/json-functions", "sql/dql/blob-descriptor", "sql/dql/generate-series", "sql/dql/explain" @@ -708,8 +707,7 @@ "group": "Feature Engineering", "pages": [ "tutorials/feature-engineering/feature-engineering-101", - "tutorials/feature-engineering/materialized-views", - "agent-branch-experiments" + "tutorials/feature-engineering/materialized-views" ] }, { diff --git a/docs/web/enterprise/adaptive-execution.mdx b/docs/web/enterprise/adaptive-execution.mdx index 3584cc191..0c8c20b86 100644 --- a/docs/web/enterprise/adaptive-execution.mdx +++ b/docs/web/enterprise/adaptive-execution.mdx @@ -1,5 +1,6 @@ --- title: "Adaptive Execution" +tag: "Early access" sidebarTitle: "Adaptive Execution" description: "How a Function refresh's part size, resource class, worker count and checkpoint frequency adjust themselves to observed behavior." icon: "gauge-high" @@ -8,6 +9,8 @@ availability: enterprise: available --- +Early access + When a [computed column](/feature-engineering/add-computed-column) or [materialized view](/feature-engineering/materialized-views) is filled by a registered [Function](/curation/create-function), LanceDB Enterprise diff --git a/docs/web/enterprise/checkpointing.mdx b/docs/web/enterprise/checkpointing.mdx index 1569071d0..ca6eab814 100644 --- a/docs/web/enterprise/checkpointing.mdx +++ b/docs/web/enterprise/checkpointing.mdx @@ -1,5 +1,6 @@ --- title: "Function Refresh Checkpointing" +tag: "Early access" sidebarTitle: "Checkpointing" description: "How a Function refresh's progress survives a worker failure or restart, part by part." icon: "floppy-disk" @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + This page is about checkpointing a [Function refresh](/enterprise/adaptive-execution) specifically -- not [PyTorch training checkpointing](/training/checkpointing), a different, unrelated mechanism covered elsewhere. diff --git a/docs/web/enterprise/gpu-refresh-execution.mdx b/docs/web/enterprise/gpu-refresh-execution.mdx index b9c7e6acc..7fcacf4b5 100644 --- a/docs/web/enterprise/gpu-refresh-execution.mdx +++ b/docs/web/enterprise/gpu-refresh-execution.mdx @@ -1,5 +1,6 @@ --- title: "GPU Refresh Execution" +tag: "Early access" sidebarTitle: "GPU Refresh Execution" description: "A computed column or materialized view filled by a GPU Function schedules its refresh workers the same way GPU Function execution does in Data Curation." icon: microchip @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + A [computed column](/feature-engineering/add-computed-column) or [materialized view](/feature-engineering/materialized-views) filled by a registered [Function](/curation/create-function) with `gpu=True` refreshes diff --git a/docs/web/enterprise/tiered-index-cache.mdx b/docs/web/enterprise/tiered-index-cache.mdx index 439c52df2..f807f04ae 100644 --- a/docs/web/enterprise/tiered-index-cache.mdx +++ b/docs/web/enterprise/tiered-index-cache.mdx @@ -1,6 +1,7 @@ --- title: "Tiered Index Cache" -sidebarTitle: "Tiered Index Cache" +tag: "Early access" +sidebarTitle: "Index Cache" description: "How LanceDB Enterprise caches deserialized index structures across memory and local disk." icon: "layer-group" availability: @@ -8,6 +9,8 @@ availability: enterprise: available --- +Early access + The tiered index cache holds deserialized vector, full-text and scalar index structures in memory, backed by a local-disk tier beneath it. When memory fills, the cache serializes evicted entries and spills them to disk diff --git a/docs/web/index.mdx b/docs/web/index.mdx index 8b290b97a..7b6bc6eed 100644 --- a/docs/web/index.mdx +++ b/docs/web/index.mdx @@ -23,7 +23,7 @@ time stitching infrastructure together and more time improving datasets, testing ## Accelerate the data flywheel for model research {#accelerate-the-data-flywheel-for-model-research} -Keep your AI data in one table, and give each agent-driven experiment its own isolated branch. +Keep your AI data in one table.{/* Draft until branching reaches early access: restore ", and give each agent-driven experiment its own isolated branch" before the period. */} Find the failure, fix the data, and retrain. @@ -31,11 +31,15 @@ Find the failure, fix the data, and retrain. Start from one failed grasp or evaluation example. [Search](/search/) multimodal data, metadata, and embeddings together, retrieve similar examples, and [curate](/curation) them into a targeted training subset. + {/* + Draft until branching reaches early access. + Don't let an agent silently corrupt your data and ruin a training run. Give each agent its own [branch](/tables/branching), a writable fork of the table, to experiment on, and review the changes before they reach `main`. See [Run experiments on branches](/agent-branch-experiments). + */} Keep episodes, frames, metadata, features, and embeddings together in one schema-enforced table, without pipelines to keep them in sync. Table [versions](/management/table#versioning) let you trace each training diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx index d0ecab8b9..baf241b30 100644 --- a/docs/web/ingestion/merge-insert.mdx +++ b/docs/web/ingestion/merge-insert.mdx @@ -30,6 +30,8 @@ import { import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx'; +Early access in Enterprise + Use merge insert when incoming records should be compared with existing rows. Combine update and insert policies for an **upsert**. Use [append](/ingestion/insert) when every incoming row should simply be added. diff --git a/docs/web/maintenance/cleanup.mdx b/docs/web/maintenance/cleanup.mdx index c7fc2cabe..21ba079f9 100644 --- a/docs/web/maintenance/cleanup.mdx +++ b/docs/web/maintenance/cleanup.mdx @@ -37,7 +37,7 @@ retention period comes from its configuration instead. The latest version is never removed, whatever you pass. `timedelta(0)` makes every older version eligible for removal, but cleanup still keeps tagged -versions and the files branches still read, as the warning below describes. +versions{/* Draft until branching reaches early access: restore " and the files branches still read" */}, as the warning below describes. Pruning is what makes [time travel](/management/table#versioning) finite: an @@ -47,12 +47,18 @@ whatever its age, when it is: - the **latest** version; - a **tagged** version. +{/* +Draft until branching reaches early access. + Cleanup also keeps the files a **branch** still reads, and the version the branch was forked from while the branch's remaining versions read the parent's files. That does not keep the fork for good: once the branch has replaced the data it inherited and its own older versions are pruned, cleanup prunes the -fork version like any other. To keep a version recoverable, tag it, subject to -the policy below, or keep the window wide enough to cover it. +fork version like any other. +*/} + +To keep a version recoverable, tag it, subject to the policy below, or keep +the window wide enough to cover it. Under the default policy, a tag can also stop cleanup. When a tagged version is older than the window and is not the latest, cleanup refuses to run: it raises @@ -75,7 +81,7 @@ need before running this, not after. [Managed blobs](/management/table#managed-blobs) store out-of-line payloads under the table's `_blobs/` directory. Cleanup follows the references from -retained snapshots, including protected tags and branches, before selecting +retained snapshots, including protected tags{/* Draft until branching reaches early access: restore " and branches" */}, before selecting objects for deletion. Replacing a blob or deleting a row makes its old payload eligible only after no retained snapshot references it. diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 38a346c18..3a4f0736c 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -1,5 +1,6 @@ --- title: "Functions" +tag: "Early access" sidebarTitle: "Function" description: "Register a Python function once and apply it as a column, for work SQL cannot express." icon: code @@ -23,6 +24,8 @@ import { } from '/snippets/function.mdx'; import { PyConnectEnterprisePlaceholders as FunctionConnect } from '/snippets/connection.mdx'; +Early access + Some columns cannot be written as a SQL expression: an embedding, a model call, an image decode. A function is a Python callable registered with the database, versioned, and then applied as a column. The code runs where the data is diff --git a/docs/web/management/secret.mdx b/docs/web/management/secret.mdx index 0359ce6a3..b8a37fb51 100644 --- a/docs/web/management/secret.mdx +++ b/docs/web/management/secret.mdx @@ -1,5 +1,6 @@ --- title: "Secret" +tag: "Early access" sidebarTitle: "Secret" description: "Create, rotate, list, describe and drop a named credential that Functions bind by reference." icon: key @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + A **Secret** is a catalog object, like a table or a namespace: a named credential, such as a model provider's API key, stored once and addressed by name. A [Function](/curation/create-function) binds a Secret by reference @@ -109,12 +112,10 @@ Every Secret's value is encrypted directly with your deployment's key in AWS KMS, Google Cloud KMS, or Azure Key Vault -- there is no separate application data key to manage. Storage, backups and replicas hold only ciphertext. Decryption goes through your KMS, which records it in your cloud's audit -log. A deployment without a KMS key does not offer Secrets; see -[KMS integration](/enterprise/deployment/kms). +log. BYOC You are responsible for the network egress -policy for the jobs that run Functions, and for scoping the Function -identity's cloud role to the KMS key so it can decrypt but not encrypt. +policy for the jobs that run Functions. ### Redaction {#redaction} @@ -131,7 +132,7 @@ that holds that credential. | Item | Limit | | --- | --- | | Secret name | 1--128 characters from `A-Z a-z 0-9 _ . -`. `$` is reserved as the namespace delimiter. | -| Value | One nonempty UTF-8 string, at most 4,096 bytes. Azure Key Vault RSA keys allow less -- see [KMS integration](/enterprise/deployment/kms). | +| Value | One nonempty UTF-8 string, at most 4,096 bytes. On Azure, at most 158, 286 or 414 bytes for an RSA-2048, RSA-3072 or RSA-4096 key. | | Bindings per Function | 16 | | Environment variable name | `[A-Za-z_][A-Za-z0-9_]*`, and not [reserved](#reserved) | | Variables per Function | Each bound once. Two variables may bind the same Secret. | @@ -157,8 +158,8 @@ refreshing one: the Secret was never created in that namespace, was dropped, or its name is misspelled in the binding. Create it, or register a new Function version with the right name. -**501 from a Secret API call**: the deployment has no KMS key configured, so -Secrets are disabled. See [KMS integration](/enterprise/deployment/kms). +**501 from a Secret API call**: Secrets aren't enabled for this deployment. +Contact [LanceDB support](/deployment/support). ## Privileges {#privileges} diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index 14ff57cf4..c229e13f0 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -143,8 +143,8 @@ import { A table stores data in rows and columns and keeps a history of committed versions. Queries read table data, writes update it, and indexes accelerate -searches over its columns. Branches let you make changes separately from the -main table history. +searches over its columns.{/* Draft until branching reaches early access: restore "Branches let you make changes separately from the +main table history." */} This page covers table management, data types, schema changes and version history. For loading records, see [Ingestion](/ingestion). For changes to @@ -161,7 +161,11 @@ In LanceDB Enterprise, tables are also created and dropped in SQL with [`CREATE | **Columns** | including vectors, blobs, and [computed](/feature-engineering/add-computed-column) columns | | **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) | | **Indexes** | vector, full-text and scalar, built over its columns | + +{/* +Draft until branching reaches early access. | **Branches** | isolated lines of change you can [merge or discard](/tables/branching) | +*/} ## Create a table {#create} @@ -326,7 +330,7 @@ as it is to the format. | String and binary | `utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)` | | Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp`, `duration` | | Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` | -| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](#multimodal), bfloat16 | +| Lance extensions | vectors, [blobs](#multimodal), bfloat16{/* Draft until JSON columns reach early access: restore "[JSON](/sql/data-types#json), " after "vectors, " */} | A vector column is a `fixed_size_list`, an ordinary Arrow type, which is why it is selectable and joinable but why arithmetic over it is list @@ -345,9 +349,7 @@ file, up to 64 KiB by default, and larger values are stored outside it. ### Blob columns {#multimodal-blob-v2} - -Blob V2 is in early access. - +Early access A `blob` column keeps metadata separate from the payload bytes, so queries can filter rows without loading the blobs. @@ -1490,6 +1492,9 @@ Deleting a tag only removes the label, not the version it points to. After deletion, the underlying table version becomes eligible for cleanup again. +{/* +Draft until branching reaches early access. + ### Branches {#versioning-branches} Beyond linear history, LanceDB also supports **branches**: isolated, writable @@ -1499,6 +1504,7 @@ writable history, making it ideal for experiments, backfills, and migrations that you want to keep separate from production reads on `main`. Branches are covered in their own guide: see [Branches](/tables/branching). +*/} ### Delete Data From the Table {#versioning-delete-data-from-the-table} diff --git a/docs/web/search/realtime-read.mdx b/docs/web/search/realtime-read.mdx index cebfd6d77..e586a1eac 100644 --- a/docs/web/search/realtime-read.mdx +++ b/docs/web/search/realtime-read.mdx @@ -70,8 +70,8 @@ returning a result that omits un-compacted rows. ## Limitations {#limitations} -Support for all table operations on RTE tables is coming soon. Time travel, -branching, `count_rows`, row IDs (`with_row_id`), and `TABLESAMPLE` are not yet +Support for all table operations on RTE tables is coming soon. Time travel,{/* Draft until branching reaches early access: restore " branching," */} +`count_rows`, row IDs (`with_row_id`), and `TABLESAMPLE` are not yet available. diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx index 651d04d29..ef5dfcab7 100644 --- a/docs/web/sql/data-types.mdx +++ b/docs/web/sql/data-types.mdx @@ -3,7 +3,7 @@ title: "SQL data types" sidebarTitle: "Data Types" description: "How Lance columns appear to SQL, including the types SQL has no name for." icon: shapes -keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "json"] +keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector"] availability: embedded: unavailable enterprise: available @@ -84,6 +84,9 @@ calculations. SQL vector search depends on the Enterprise server version and que plan; see [SQL](/basics/sql#no-vector-sql). +{/* +Draft until JSON columns reach early access. + ### JSON {#json} Stored as JSONB in `large_binary` with the `lance.json` extension name. It reads @@ -98,6 +101,7 @@ WHERE json_get_int(data, 'status') = 200 `json_get`, its typed variants, and `json_extract` are **index-accelerated** by a JSON scalar index. See [JSON functions](/sql/dql/json-functions). +*/} ### bfloat16 {#bfloat16} @@ -105,6 +109,9 @@ Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. Useful as a vector element type when halving the memory matters more than the precision does. +{/* +Draft: geospatial queries and indexes are unsupported in LanceDB Enterprise. + ## Geospatial {#geospatial} @@ -118,6 +125,7 @@ spatial index is gated the same way. Geometry can still be stored (it is binary, and binary columns work), but nothing interprets it. Filter on a bounding box held in ordinary numeric columns if you need spatial predicates today. +*/} ## Pseudo-columns {#pseudo-columns} diff --git a/docs/web/sql/ddl/function.mdx b/docs/web/sql/ddl/function.mdx index 5dbdb58ae..8cd45f8bc 100644 --- a/docs/web/sql/ddl/function.mdx +++ b/docs/web/sql/ddl/function.mdx @@ -1,5 +1,6 @@ --- title: "Function" +tag: "Early access" sidebarTitle: "Function" description: "Create, describe, alter, list, and drop registered functions with SQL." icon: function @@ -9,6 +10,8 @@ availability: enterprise: available --- +Early access + [SQL runs in LanceDB Enterprise.](/sql) ## CREATE FUNCTION {#create} diff --git a/docs/web/sql/ddl/secret.mdx b/docs/web/sql/ddl/secret.mdx index 33ba009da..6c6e3c4c6 100644 --- a/docs/web/sql/ddl/secret.mdx +++ b/docs/web/sql/ddl/secret.mdx @@ -1,12 +1,13 @@ --- title: "Secret" +tag: "Early access" sidebarTitle: "Secret" description: "CREATE, ALTER, DROP, SHOW and DESCRIBE SECRET, and granting privileges on Secrets." icon: key keywords: ["secret", "create secret", "alter secret", "drop secret", "show secrets", "describe secret", "grant", "sql"] --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) +Early access Enterprise [SQL runs in LanceDB Enterprise.](/sql) A [Secret](/management/secret) is a named credential that Functions bind by reference. These statements store, rotate, list, describe and drop Secrets. diff --git a/docs/web/sql/dql/functions.mdx b/docs/web/sql/dql/functions.mdx index 945abbe03..0ea53a508 100644 --- a/docs/web/sql/dql/functions.mdx +++ b/docs/web/sql/dql/functions.mdx @@ -15,7 +15,7 @@ plus what Lance adds. | | | | --- | --- | -| [Scalar functions](/sql/dql/scalar-functions) | per row: math, string, regex, date/time, array, struct, JSON | +| [Scalar functions](/sql/dql/scalar-functions) | per row: math, string, regex, date/time, array, struct{/* Draft until JSON columns reach early access: restore ", JSON" */} | | [Aggregate functions](/sql/dql/aggregation#functions) | per group: `count`, `sum`, statistical, approximate | | [Window functions](/sql/dql/window#functions) | per window: ranking, offset, running aggregates | | [Table functions](#table-functions) | produce a relation rather than a value | @@ -45,13 +45,17 @@ see [Data types](/sql/data-types#pseudo-columns). ## What Lance accelerates {#accelerated} -Two function families use an index rather than scanning, which is the difference +`contains_tokens` uses an index rather than scanning, which is the difference between a fast predicate and a slow one: | | Index used | | --- | --- | | [`contains_tokens`](/sql/dql/contains-tokens) | full-text index | + +{/* +Draft until JSON columns reach early access; restore "Two function families" above. | [`json_get`, typed variants, `json_extract`](/sql/dql/json-functions) | JSON scalar index | +*/} Confirm with [`EXPLAIN`](/sql/dql/explain): an accelerated predicate shows an index node rather than a `LanceScan` followed by a filter. diff --git a/docs/web/sql/dql/scalar-functions.mdx b/docs/web/sql/dql/scalar-functions.mdx index b0910c01a..9bd2057b0 100644 --- a/docs/web/sql/dql/scalar-functions.mdx +++ b/docs/web/sql/dql/scalar-functions.mdx @@ -3,7 +3,7 @@ title: "Scalar functions" sidebarTitle: "Scalar Functions" description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds." icon: calculator -keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"] +keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions"] availability: embedded: unavailable enterprise: available @@ -133,7 +133,14 @@ reference page: | [`contains_tokens`](/sql/dql/contains-tokens) | index-accelerated keyword predicate, usable in `WHERE` | | [`ivf_partition`](/sql/dql/ivf-partition) | assign each row to its trained IVF index partition | | [`blob_descriptor`](/sql/dql/blob-descriptor) | a BlobV2 column's payload location, without reading it | + +{/* +Draft until JSON columns reach early access. | [JSON accessors](/sql/dql/json-functions) | `json_get` and its typed variants, `json_extract`, and more | +*/} + +{/* +Draft: geospatial queries and indexes are unsupported in LanceDB Enterprise. ### Geospatial {#geo} @@ -143,6 +150,7 @@ Lance defines `st_*` functions on GeoArrow, but they sit behind a build feature feature rather than an unknown-function error, and the spatial index is gated the same way. +*/} ## Additional scalar functions {#escape-hatch} diff --git a/docs/web/tutorials/index.mdx b/docs/web/tutorials/index.mdx index 6ada57c9a..b9aea29a2 100644 --- a/docs/web/tutorials/index.mdx +++ b/docs/web/tutorials/index.mdx @@ -21,7 +21,7 @@ are runnable in Colab; the rest are written up here. Retrieval-augmented generation and agent applications. - Deriving columns, materialized views, experimenting on branches. + Deriving columns and materialized views.{/* Draft until branching reaches early access: restore "Deriving columns, materialized views, experimenting on branches." */} Curating data and training models directly on a table.