The Documentation tab now reads as a manual over Python, TypeScript and SQL
rather than a Python guide with SQL bolted on.
Basics replaces the Catalog section: what the catalog holds, what each API
surface can do, jobs, agentic skills, and what Lance is. Management is new and
is the CRUD and listing reference for every catalog object, with a column per
language so the gaps are visible -- TypeScript has no function or SQL methods,
SQL needs a deployment.
Search becomes Query and gains a SELECT reference, since an analytical query is
a query whether it came from a builder or a statement.
Jobs is new. Every long-running operation already returned the same handle from
an _async method; nothing documented it.
The SQL function reference gains what Lance registers on top of DataFusion:
contains_tokens, backed by the FTS index, and the JSON accessors, backed by a
JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB
does not ship, which is now stated rather than left to a confusing error.
Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text
Search, and splits Query Optimization into Explain and Explain Analyze.
The tab opened on Quickstart, buried "What is LanceDB?" inside it, and put
everything else under a single Guides group that had to be expanded before any
section was reachable. Model training sat outside it for no reason.
Reorganize around what a user does, in order: Introduction, Catalog, Ingestion,
Search, Feature engineering, Training, Maintenance, SQL. Guides is gone as a
level, so each section is one click.
New pages: catalog overview and database, and a compaction page under
Maintenance. tables/update covered four separate operations, so it becomes the
Ingestion section — insert, merge insert, update, delete — with insert drawing
on the large-dataset material from tables/create. Its old URL redirects.
Agentic skills move to their own tab after API Reference.
DataFrame support is not going ahead, so the pages documenting it come out:
the relational operations and the expression system, both of which described
only that API.
The manual was framed around having two query surfaces and choosing between
them. With one, that framing is gone — the overview now leads with SQL over
Arrow Flight SQL and what the dialect is, rather than with a comparison to
something that will not exist.
The function reference keeps everything that was about SQL: the DataFusion
version, how Lance columns map onto its type system, the pseudo-columns, blob
handling, and `fts()`. Only the section on reaching functions through the
Python wrapper is replaced, since the wrapper is going too.
Mentions of pandas and polars DataFrames elsewhere in the documentation are
unrelated and untouched.
The manual said the SQL dialect's function and type reference was out of scope.
It is not: the engine is DataFusion 54 with its default function set registered,
so the dialect is documented already and the useful thing is to say which
version and point at it.
The types are worth being concrete about rather than deferring. Lance's
DataFusion table provider returns the dataset's Arrow schema directly — there is
no SQL type layer over storage — so DataFusion's type reference describes Lance
columns exactly, and a vector column is a FixedSizeList that behaves like one.
Two consequences of that mapping are documented because they surprise people.
The pseudo-columns the provider can append are real and typed — `_rowid`,
`_rowaddr` and `_rowoffset` are nullable UInt64, and the nullability is a reused
selection bitmap rather than a missing value, so a null there means nothing.
And a blob column's Arrow type depends on how it is read: descriptions by
default, binary on request, which changes the schema a query sees.
`fts()` is recorded as what it is, a DataFusion TableProvider over the dataset.
That explains the parts that otherwise look arbitrary: why it is a table
function rather than a predicate, why every column of the table is available in
its result, and why projections push into the scan.
Functions and materialized views are objects in the Lance Multimodal Lakehouse,
alongside tables and indexes — created, versioned and refreshed like anything
else in it. The pages introduced them as client methods, which undersold what
they are and left the capability sounding like a helper rather than part of the
storage layer.
The materialized view page also framed Enterprise availability around which
client you are holding. A reader does not care about that; they care whether
views exist where their data lives. They do, refreshed across the cluster, and
the page now says so and points at the Enterprise documentation rather than
implying the feature is missing.
Both the lancedb client and the geneva client expose
`create_materialized_view`, and they are not the same feature. The page said
materialized views are local only, which is true of this client and misleading
for anyone on Enterprise, where views are created through geneva and refreshed
on the cluster.
Found by assembling the combined site and reading the Geneva pages against the
new ones: they described the same call as distributed and Enterprise while this
one described it as raising. Both were accurate about their own client and
neither said which client it meant.
The feature engineering documentation describes the geneva package, a separate
SDK. The capability it covers now lives in this client — materialized views,
computed columns, and registered functions are all shipped API with no pages
describing them.
Four pages cover the three mechanisms and, more usefully, when each applies.
The overview leads with that choice rather than with concepts, because the
mechanisms overlap and picking the wrong one is the expensive mistake.
Two things they say plainly that the API reference cannot:
Declaring and filling are separate everywhere, and filling never revisits a row
it has already done. That makes refreshes idempotent and appends cheap, and it
means a filled value does not track its inputs — surprising if you expect a
spreadsheet, so it is stated rather than left to be discovered.
The availability split runs the opposite way to expectation. Materialized views
are local only and raise on a remote connection; functions are Cloud and
Enterprise only and raise on a local one. Someone reading about both in one
sitting will otherwise assume the usual direction and be wrong in one of them.
Examples are the tested ones from the docstrings, so they cannot drift from the
behaviour they describe.
The training and tutorial pages that pointed into the geneva documentation now
point here.
The README that carried this sat inside docs/web, which is the published site
root, so it would have shipped as a page. Contributor guidance belongs in the
contributing guide, which is also where someone changing behaviour will look for
whether the page describing it is theirs to update.
The pages describing this code lived in lancedb/docs, a repository with ten
stars that neither contributing guide mentioned. A user who hit a wrong page
while reading the source had to find a separate, near-invisible repository to
fix it. These 102 pages now sit beside the code, and a documentation fix is a
pull request in the repository the reader already had open.
`docs/web` is a complete Mintlify site: `cd docs/web && mint dev` serves it with
no other checkout. Enterprise pages stay private and are merged in when the
published site is assembled, so links into them do not resolve here — every
open-source page does.
`docs/web-tests` holds the tests the code examples are extracted from. They ran
against a published wheel before; here they can run against this repository's
own SDK, so an example that stops compiling fails the pull request that broke
it rather than being noticed later.
Both directories sit beside `docs/src` and are invisible to it: mkdocs reads
`docs_dir: src`, so the SDK reference builds and deploys exactly as before.
Jack Ye