mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
179 lines
4.7 KiB
Plaintext
179 lines
4.7 KiB
Plaintext
---
|
|
title: "Full-Text Search"
|
|
sidebarTitle: "Full-Text Search"
|
|
description: "The fts() table function: match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
|
|
icon: magnifying-glass
|
|
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
|
predicate: it produces a relevance score alongside the matching rows, and a
|
|
`WHERE` clause has nowhere to put one. The two string-literal arguments are
|
|
the table name and a JSON-encoded query, built with the same query-builder
|
|
classes used from the SDK and serialized with `.to_json()`.
|
|
|
|
## Setup {#setup}
|
|
|
|
Run these statements in order, sending one statement per SQL request.
|
|
Semicolons separate the statements for clients that execute a script.
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING);
|
|
|
|
INSERT INTO my_docs VALUES
|
|
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
|
|
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
|
|
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
|
|
(5, 'Puppy training requires patience and dedication', 'training', 'Alice');
|
|
|
|
CREATE INDEX ON my_docs USING fts (text);
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Warning>
|
|
Phrase queries need position information, which costs extra index size.
|
|
Add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
|
|
[phrase queries](#phrase) find nothing. See
|
|
[FTS parameters](/indexing/fts-index#fts-parameters).
|
|
</Warning>
|
|
|
|
## Basic search {#basic}
|
|
|
|
The query builder produces the JSON string used as the second SQL argument.
|
|
The SQL examples below spell out the equivalent match object so they can be
|
|
run directly after the setup above.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.query import MatchQuery
|
|
|
|
query = MatchQuery("puppy", "text")
|
|
json_query = query.to_json()
|
|
```
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SELECT id, text, category
|
|
FROM fts(
|
|
'my_docs',
|
|
'{"match":{"column":"text","terms":"puppy"}}'
|
|
)
|
|
LIMIT 5
|
|
```
|
|
</CodeGroup>
|
|
|
|
By default the top 5 matches come back in **arbitrary order**. Select
|
|
`_score` and order by it for relevance ranking:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SELECT id, text, _score
|
|
FROM fts(
|
|
'my_docs',
|
|
'{"match":{"column":"text","terms":"puppy"}}'
|
|
)
|
|
ORDER BY _score DESC
|
|
LIMIT 5
|
|
```
|
|
</CodeGroup>
|
|
|
|
`_score` is a pseudo-column carrying the query's BM25 relevance for the row;
|
|
it behaves like any other column, but exists only in a relation that came
|
|
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
|
|
|
|
## Fuzzy {#fuzzy}
|
|
|
|
Tolerates typos, up to a character-edit budget:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
MatchQuery("pupy", "text", fuzziness=2)
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Phrase {#phrase}
|
|
|
|
Exact phrase, or with `slop` to allow words between terms:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.query import PhraseQuery
|
|
|
|
PhraseQuery("happy puppy", "text")
|
|
PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Boolean {#boolean}
|
|
|
|
Combine queries with `&` (AND) or `|` (OR):
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
MatchQuery("puppy", "text") & MatchQuery("happy", "text")
|
|
MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Boost {#boost}
|
|
|
|
Weight a query up or down relative to another:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.query import BoostQuery
|
|
|
|
BoostQuery(
|
|
positive=MatchQuery("puppy", "text"),
|
|
negative=MatchQuery("kitten", "text"),
|
|
negative_boost=0.2,
|
|
)
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Multi-match {#multi-match}
|
|
|
|
Search several columns at once, optionally weighting them differently:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.query import MultiMatchQuery
|
|
|
|
MultiMatchQuery("puppy", ["text", "category"])
|
|
MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Combining with SQL {#combining}
|
|
|
|
`fts()` is a relation, so it composes with the rest of the SQL surface.
|
|
Filter, join, or aggregate over it same as any table:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SELECT id, text, category
|
|
FROM fts(
|
|
'my_docs',
|
|
'{"match":{"column":"text","terms":"puppy"}}'
|
|
)
|
|
WHERE category = 'animals'
|
|
LIMIT 5
|
|
```
|
|
</CodeGroup>
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Full-Text Search" icon="magnifying-glass" href="/search/full-text-search">
|
|
The query-builder equivalent, and every query form in full.
|
|
</Card>
|
|
<Card title="FTS Index" icon="bolt" href="/indexing/fts-index">
|
|
Tokenizers, and the position information phrase queries need.
|
|
</Card>
|
|
</CardGroup>
|