mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 16:22:27 +00:00
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
103 lines
4.3 KiB
Plaintext
103 lines
4.3 KiB
Plaintext
---
|
|
title: "Lance Lakehouse"
|
|
sidebarTitle: "Lance Lakehouse"
|
|
description: "What operates on the catalog's objects: storage, search, computation, and maintenance."
|
|
icon: layer-group
|
|
keywords: ["lakehouse", "multimodal", "search", "compute", "maintenance", "feature engineering"]
|
|
---
|
|
|
|
The [catalog](/basics/catalog) says what exists. The **lakehouse** is everything
|
|
that acts on it: storing the rows, searching them, computing new ones, and
|
|
keeping the whole thing fast as it grows.
|
|
|
|
The distinction matters because the two layers have different lifetimes. A
|
|
catalog entry is a name and a pointer. A lakehouse operation reads and writes
|
|
the [Lance](/basics/lance) files that name points at -- and those operations are
|
|
where almost all of the cost, and all of the tuning, lives.
|
|
|
|
## Multimodal is the point {#multimodal}
|
|
|
|
A lakehouse table holds images, video, audio, text, embeddings, annotations and
|
|
the features you derive from them **in one table**, side by side.
|
|
|
|
That is the difference from assembling a warehouse plus a vector store plus a
|
|
feature store. There is one copy of the data, one set of versions, and no
|
|
pipeline keeping three systems agreeing about the same rows. A search can filter
|
|
on a label, rank by a vector, and return the blob, without leaving the table.
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Columns** | vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
|
|
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
|
|
| **Indexes** | vector, full-text and scalar, built over those columns |
|
|
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
|
|
|
|
## The four capabilities {#capabilities}
|
|
|
|
Everything the lakehouse does falls into one of four groups, and the rest of
|
|
these docs is organized the same way.
|
|
|
|
### Ingest {#ingest}
|
|
|
|
Getting rows in, and changing the ones that are there: append, merge, update,
|
|
delete. Every write commits a version rather than mutating in place, which is
|
|
what makes the other three safe to run against a moving table. See
|
|
[Data Ingestion](/ingestion).
|
|
|
|
### Search {#search}
|
|
|
|
Vector, full-text and hybrid retrieval over the same rows, with filters that
|
|
push into the scan rather than running after it. The search is a lakehouse
|
|
operation, not a separate service holding a second copy of your embeddings. See
|
|
[Semantic Search](/search).
|
|
|
|
### Compute {#compute}
|
|
|
|
Deriving new columns and tables from what is already stored:
|
|
[computed columns](/feature-engineering/add-computed-column),
|
|
[functions](/management/function) that run Python where the data is, and
|
|
[materialized views](/management/materialized-view) that persist a query's
|
|
result.
|
|
|
|
The reason this belongs in the lakehouse rather than in a pipeline outside it is
|
|
the format. Adding a derived column writes only that column -- the images and
|
|
videos are never rewritten -- so deriving a feature is an ordinary operation
|
|
rather than a migration. See [Feature Engineering](/feature-engineering) and
|
|
[Data Curation](/curation).
|
|
|
|
### Maintenance {#maintenance}
|
|
|
|
The work that keeps the first three fast: building and repairing
|
|
[indexes](/indexing), and [compaction](/maintenance/compaction) to merge small
|
|
files, prune old versions and fold new rows into existing indexes.
|
|
|
|
<Warning>
|
|
None of it runs on its own. There is no background compactor -- `optimize()`
|
|
runs when you call it, and an index does not cover rows added after it was
|
|
built until you do. See [Guarantees](/basics/guarantees#not-guaranteed).
|
|
</Warning>
|
|
|
|
## Where the work happens {#where}
|
|
|
|
The same lakehouse runs in two shapes, and which one you use decides where these
|
|
capabilities execute rather than whether you have them:
|
|
|
|
- **Embedded** -- your process does all four itself, directly against object
|
|
storage. Nothing runs unless you call it.
|
|
- **Distributed** -- a cluster does them, with indexing and compaction on
|
|
workers separate from the ones serving queries, plus
|
|
[SQL](/sql) and server-side [jobs](/basics/jobs).
|
|
|
|
See [Architecture](/architecture) for the comparison in full.
|
|
|
|
## Next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Lance Catalog" icon="sitemap" href="/basics/catalog">
|
|
The objects these operations act on.
|
|
</Card>
|
|
<Card title="Lance format" icon="layer-group" href="/basics/lance">
|
|
The open format underneath, and why column evolution is cheap.
|
|
</Card>
|
|
</CardGroup>
|