SELECT and Enterprise SQL sat apart in Query while documenting the same engine.
They are now one SQL group: overview, Select, Aggregation, Join, Window,
Sampling, Functions -- one page per construct, with the Enterprise client-setup
and FTS pages merged in rather than parallel to them.
Sampling is new. TABLESAMPLE is implemented and was undocumented: BERNOULLI
picks rows, SYSTEM picks Lance fragments, REPEATABLE fixes the draw so a sample
works as a stable evaluation set. Its refusals are documented too -- a
percentage over 100, a fractional row count, BUCKET, the SAMPLE spelling and
the OFFSET modifier are all errors, because a sampling clause that were read and
not applied would be a wrong answer rather than a missing feature.
Schema & Data Types is one page again, with the multimodal content merged into
its blob section rather than sitting beside it.
Agentic Skills stops being a tab. Both its pages are tutorials, so they join the
Tutorials tab; the Basics page that introduces them stays.
The Documentation tab now reads as a manual over Python, TypeScript and SQL
rather than a Python guide with SQL bolted on.
Basics replaces the Catalog section: what the catalog holds, what each API
surface can do, jobs, agentic skills, and what Lance is. Management is new and
is the CRUD and listing reference for every catalog object, with a column per
language so the gaps are visible -- TypeScript has no function or SQL methods,
SQL needs a deployment.
Search becomes Query and gains a SELECT reference, since an analytical query is
a query whether it came from a builder or a statement.
Jobs is new. Every long-running operation already returned the same handle from
an _async method; nothing documented it.
The SQL function reference gains what Lance registers on top of DataFusion:
contains_tokens, backed by the FTS index, and the JSON accessors, backed by a
JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB
does not ship, which is now stated rather than left to a confusing error.
Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text
Search, and splits Query Optimization into Explain and Explain Analyze.
DataFrame support is not going ahead, so the pages documenting it come out:
the relational operations and the expression system, both of which described
only that API.
The manual was framed around having two query surfaces and choosing between
them. With one, that framing is gone — the overview now leads with SQL over
Arrow Flight SQL and what the dialect is, rather than with a comparison to
something that will not exist.
The function reference keeps everything that was about SQL: the DataFusion
version, how Lance columns map onto its type system, the pseudo-columns, blob
handling, and `fts()`. Only the section on reaching functions through the
Python wrapper is replaced, since the wrapper is going too.
Mentions of pandas and polars DataFrames elsewhere in the documentation are
unrelated and untouched.
The manual said the SQL dialect's function and type reference was out of scope.
It is not: the engine is DataFusion 54 with its default function set registered,
so the dialect is documented already and the useful thing is to say which
version and point at it.
The types are worth being concrete about rather than deferring. Lance's
DataFusion table provider returns the dataset's Arrow schema directly — there is
no SQL type layer over storage — so DataFusion's type reference describes Lance
columns exactly, and a vector column is a FixedSizeList that behaves like one.
Two consequences of that mapping are documented because they surprise people.
The pseudo-columns the provider can append are real and typed — `_rowid`,
`_rowaddr` and `_rowoffset` are nullable UInt64, and the nullability is a reused
selection bitmap rather than a missing value, so a null there means nothing.
And a blob column's Arrow type depends on how it is read: descriptions by
default, binary on request, which changes the schema a query sees.
`fts()` is recorded as what it is, a DataFusion TableProvider over the dataset.
That explains the parts that otherwise look arbitrary: why it is a table
function rather than a predicate, why every column of the table is available in
its result, and why projections push into the scan.
Jack Ye