The function catalogue is now real rather than a pointer. Scalar functions are
listed by taxonomy -- math, string, unicode, regex, date/time, array, map,
struct, crypto, type -- and the names come from what datafusion-functions 54
actually registers, not from what DataFusion documents on main. Aggregate and
window pages gain the same treatment.
Two things the audit corrected rather than copied.
SHOW TABLES does not work. DataFusion implements it over information_schema,
which this session does not enable, so it fails at plan time. It was documented
as working here, on the SHOW page and in three Manage tables; all four are
fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS.
Prepared statements are a Flight SQL protocol feature, not a PREPARE statement.
A script carrying PREPARE ... AS SELECT will not run, while the same intent
expressed through a client will. The new page says so rather than copying
DataFusion's syntax.
Every section of the Documentation tab that has a SQL counterpart now links to
it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather
than at the reference index.
The Documentation tab now reads as a manual over Python, TypeScript and SQL
rather than a Python guide with SQL bolted on.
Basics replaces the Catalog section: what the catalog holds, what each API
surface can do, jobs, agentic skills, and what Lance is. Management is new and
is the CRUD and listing reference for every catalog object, with a column per
language so the gaps are visible -- TypeScript has no function or SQL methods,
SQL needs a deployment.
Search becomes Query and gains a SELECT reference, since an analytical query is
a query whether it came from a builder or a statement.
Jobs is new. Every long-running operation already returned the same handle from
an _async method; nothing documented it.
The SQL function reference gains what Lance registers on top of DataFusion:
contains_tokens, backed by the FTS index, and the JSON accessors, backed by a
JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB
does not ship, which is now stated rather than left to a confusing error.
Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text
Search, and splits Query Optimization into Explain and Explain Analyze.
DataFrame support is not going ahead, so the pages documenting it come out:
the relational operations and the expression system, both of which described
only that API.
The manual was framed around having two query surfaces and choosing between
them. With one, that framing is gone — the overview now leads with SQL over
Arrow Flight SQL and what the dialect is, rather than with a comparison to
something that will not exist.
The function reference keeps everything that was about SQL: the DataFusion
version, how Lance columns map onto its type system, the pseudo-columns, blob
handling, and `fts()`. Only the section on reaching functions through the
Python wrapper is replaced, since the wrapper is going too.
Mentions of pandas and polars DataFrames elsewhere in the documentation are
unrelated and untouched.
The manual said the SQL dialect's function and type reference was out of scope.
It is not: the engine is DataFusion 54 with its default function set registered,
so the dialect is documented already and the useful thing is to say which
version and point at it.
The types are worth being concrete about rather than deferring. Lance's
DataFusion table provider returns the dataset's Arrow schema directly — there is
no SQL type layer over storage — so DataFusion's type reference describes Lance
columns exactly, and a vector column is a FixedSizeList that behaves like one.
Two consequences of that mapping are documented because they surprise people.
The pseudo-columns the provider can append are real and typed — `_rowid`,
`_rowaddr` and `_rowoffset` are nullable UInt64, and the nullability is a reused
selection bitmap rather than a missing value, so a null there means nothing.
And a blob column's Arrow type depends on how it is read: descriptions by
default, binary on request, which changes the schema a query sees.
`fts()` is recorded as what it is, a DataFusion TableProvider over the dataset.
That explains the parts that otherwise look arbitrary: why it is a table
function rather than a predicate, why every column of the table is available in
its result, and why projections push into the scan.
Jack Ye