Files
lancedb/docs/web/integrations/data/duckdb.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

136 lines
4.0 KiB
Plaintext

---
title: "DuckDB"
sidebarTitle: "DuckDB"
description: "Learn how to use the DuckDB-Lance extension to query Lance tables with SQL."
---
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for LanceDB Embedded tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
## Install {#install}
Install the DuckDB CLI as per [their docs](https://duckdb.org/install) and alternatively, their Python package with `pip install duckdb`.
Then, open the DuckDB CLI and install and load the Lance extension as follows:
<CodeGroup>
```sql SQL icon="database"
INSTALL lance;
LOAD lance;
```
</CodeGroup>
## Attach the directory namespace in DuckDB {#attach-the-directory-namespace-in-duckdb}
Attach the LanceDB root directory as a Lance namespace:
<CodeGroup>
```sql SQL icon="database"
ATTACH './local_lancedb' AS lance_ns (TYPE LANCE);
```
</CodeGroup>
In this page, tables are referenced using `lance_ns.main.<table_name>`, so the table path is `lance_ns.main.lance_duck`.
## Write Lance table {#write-lance-table}
Create the `lance_duck` table using SQL and populate it with sample data:
<CodeGroup>
```sql SQL icon="database"
CREATE OR REPLACE TABLE lance_ns.main.lance_duck AS
SELECT *
FROM (
VALUES
('duck', 'quack', [0.9, 0.7, 0.1]::FLOAT[]),
('horse', 'neigh', [0.3, 0.1, 0.5]::FLOAT[]),
('dragon', 'roar', [0.5, 0.2, 0.7]::FLOAT[])
) AS t(animal, noise, vector);
```
</CodeGroup>
This table is the source of truth for all DuckDB queries below.
<Info>
The examples below show SQL entered in the DuckDB CLI. You can run the same SQL from
Python as well, using LanceDB and DuckDB's Python clients in your application code.
</Info>
## Query the table with SQL {#query-the-table-with-sql}
<CodeGroup>
```sql SQL icon="database"
SELECT *
FROM lance_ns.main.lance_duck
LIMIT 5;
```
</CodeGroup>
## Vector search {#vector-search}
<CodeGroup>
```sql SQL icon="database"
SELECT animal, noise, vector, _distance
FROM lance_vector_search(
'lance_ns.main.lance_duck',
'vector',
[0.8, 0.7, 0.2]::FLOAT[],
k = 1,
prefilter = true
)
ORDER BY _distance ASC;
```
</CodeGroup>
## Full-text search {#full-text-search}
<CodeGroup>
```sql SQL icon="database"
SELECT animal, noise, vector, _score
FROM lance_fts(
'lance_ns.main.lance_duck',
'animal',
'the brave knight faced the dragon',
k = 1,
prefilter = true
)
ORDER BY _score DESC;
```
</CodeGroup>
## Hybrid search {#hybrid-search}
<CodeGroup>
```sql SQL icon="database"
SELECT animal, noise, vector, _hybrid_score, _distance, _score
FROM lance_hybrid_search(
'lance_ns.main.lance_duck',
'vector',
[0.8, 0.7, 0.2]::FLOAT[],
'animal',
'the duck surprised the dragon',
k = 2,
prefilter = false,
alpha = 0.5,
oversample_factor = 4
)
ORDER BY _hybrid_score DESC;
```
</CodeGroup>
## Directory namespace model {#directory-namespace-model}
A directory namespace maps a LanceDB namespace root -- there's no catalog in
Embedded -- to namespace-qualified table identifiers in DuckDB. This keeps
table discovery and table naming stable as your project grows.
To learn more about the namespace and catalog model, see [Namespaces and the Catalog Model](/management/namespace).
## Advanced usage {#advanced-usage}
See the [docs](https://github.com/lance-format/lance-duckdb) directory in the Lance-DuckDB extension repo
for more advanced usage on SQL and REST API clients.