lancedb now holds a page at every path the site publishes, including the twelve
Enterprise ones. Those are real open-source pages -- what the capability is, and
what the embedded form does instead -- rather than blank placeholders, so the
open-source site is complete on its own. sophon becomes an overlay whose page
replaces the one at the same path, which means navigation lives in one place and
the two repositories only have to agree on a path.
That removes the whole class of problem where a page could only be reached from
one of the two trees. The open-source site went from twenty-four dangling
Enterprise links to one, and that one points at the dataset pages the build
repository owns.
Ports four changes that landed on the docs repository after this branch left it:
#344 indexing and quantization rewritten
#354 LeRobot native reader
#361 training/index renamed to training/data-loading and rewritten
#362 distance metric mismatch falls back to a brute-force scan
None applied as patches, because these files were merged or split here --
quantization into the vector index page, multi-vector into vector search,
performance into four pages. Each was ported by taking the upstream page and
re-applying the restructure on top, which also picked up a correction that would
otherwise have been lost: the stricter brute-force threshold for multi-vector
columns is a Cloud limit, not an Enterprise one.
The upstream LeRobot page linked to a Geneva page that no longer exists; it now
points at the feature-engineering pages that replaced it.
Six groups were verbs -- Manage, Ingest, Train, Operate, Secure, Deploy -- and
now name the thing rather than the action. Four more were sentence case.
The page labels that were verb phrases follow: Add Computed Column becomes
Computed Columns, Refresh Column becomes Column Refresh, Configuring Storage
becomes Configuration, and Explain becomes Query Plans, which is also more
accurate now that the page covers analyze as well.
SQL statement pages take the keyword -- SELECT, DDL, DML, REFRESH, SHOW, EXPLAIN
-- because a statement name is a noun and lowercase Select and Show read as
instructions. Concept pages there keep noun phrases: Joins, Window Functions,
Subqueries, Operators.
Two duplicate labels are resolved on the way. Reranking sat in Semantic Search
labelled Overview despite being the only page on its subject, and the Enterprise
client page sat in SQL Reference labelled SQL next to the reference Overview; it
is Client Connection now.
Title case leaves prepositions and articles lowercase, so Needle in a Haystack
and Hybrid Search on BEIR are already correct and unchanged.
Ten merges and renames, all in one direction: fewer pages, each covering a whole
subject.
Table absorbs schema evolution and versioning. Vector search absorbs
multi-vector. Explain absorbs explain analyze. Reranking was seven pages -- an
overview and six rerankers of a few dozen lines each -- and is now one. The
vector index page absorbs quantization, which was only ever a parameter of it.
Indexing moves to Manage, where the other catalog objects are. The operational
half stays in Operate as one Indexing page, absorbing GPU indexing -- which was
filed in sophon despite importing lancedb's snippets and being mostly
open-source content with one Enterprise-badged section.
Performance was a single page aggregating tips for four unrelated operations,
which is hard to keep current. Each section now sits with the operation it is
about: ingestion tips on Insert, index tips on Indexing, compaction tips on
Compaction, query tips on Search.
Data Types leaves Concepts. The type reference belongs in the SQL reference,
which already described the same types from the query side; Table now carries a
short summary that points there. The multimodal material it had absorbed is its
own page again, since storing blobs is a task rather than a type definition.
Query becomes Semantic Search, Branches becomes Branch, and the Enterprise
Security page becomes Compliance.
Redirects cover every moved URL, and 30 files were repointed at the merged
anchors.
The function catalogue is now real rather than a pointer. Scalar functions are
listed by taxonomy -- math, string, unicode, regex, date/time, array, map,
struct, crypto, type -- and the names come from what datafusion-functions 54
actually registers, not from what DataFusion documents on main. Aggregate and
window pages gain the same treatment.
Two things the audit corrected rather than copied.
SHOW TABLES does not work. DataFusion implements it over information_schema,
which this session does not enable, so it fails at plan time. It was documented
as working here, on the SHOW page and in three Manage tables; all four are
fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS.
Prepared statements are a Flight SQL protocol feature, not a PREPARE statement.
A script carrying PREPARE ... AS SELECT will not run, while the same intent
expressed through a client will. The new page says so rather than copying
DataFusion's syntax.
Every section of the Documentation tab that has a SQL counterpart now links to
it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather
than at the reference index.
Namespace and Table were the only objects behind a nested group. Both are now a
single page like Database, Materialized view, View and Function, so Manage is a
flat list of the things a catalog holds.
Namespace merges its two pages, which were a concept page and a usage page for
the same object.
Table is a new page. The old tables/index was a 672-line basic-usage tutorial
whose every section had acquired a better home during the restructure --
connecting, ingesting, searching, filtering, adding and dropping columns -- to
the point that its own Next steps cards pointed at the pages that superseded it.
It is replaced by a table object page: what a table carries, create, open, list,
rename, clone, drop, and links out.
Schema evolution, versioning and branching stay as siblings rather than folding
in. Merging them would have made one 1,800-line page, which is not what the
other object pages look like.
Every group now answers exactly one question. Get started and Concepts cover
what the system is; the verb groups cover doing something with it; Operate,
Secure and Deploy cover running it; SQL reference is lookup. Maintenance becomes
Operate, Governance becomes Secure, and the rest take verbs to match.
Three new pages in Concepts and Get started:
Guarantees states what a write promises when it returns, which version a reader
sees, what survives a failure, and -- the half usually missing -- what is not
guaranteed: no multi-table transactions, no read-your-writes across processes
without an interval, unenforced primary keys that do not enforce, no background
compaction.
Limits collects bounds that were stated in passing across a dozen pages, with
the numbers verified against the code rather than inferred: name lengths, job
event caps, SHOW ERRORS bounds, the sampling range that is refused rather than
clamped.
Architecture illustrates embedded against distributed as the one deployment
decision, and is written to cover both rather than only Enterprise.
Indexing splits by question. Choosing an index is a Query decision and moves
there; building, repairing and GPU-building it stay in Operate.
SQL becomes a top-level reference of fifteen pages instead of a sub-group of
Query. Auditing sophon-sql turned up a large undocumented surface: ALTER TABLE
ADD COLUMN in its plain, computed and function forms with FILTER, IGNORE NULLS
and OVER modifiers; SET UNENFORCED PRIMARY KEY; SET and UNSET WAL with sharding;
CREATE FUNCTION with its dependency options; REFRESH COLUMN and REFRESH
MATERIALIZED VIEW; and the SHOW family. None of it was written down.
SELECT and Enterprise SQL sat apart in Query while documenting the same engine.
They are now one SQL group: overview, Select, Aggregation, Join, Window,
Sampling, Functions -- one page per construct, with the Enterprise client-setup
and FTS pages merged in rather than parallel to them.
Sampling is new. TABLESAMPLE is implemented and was undocumented: BERNOULLI
picks rows, SYSTEM picks Lance fragments, REPEATABLE fixes the draw so a sample
works as a stable evaluation set. Its refusals are documented too -- a
percentage over 100, a fractional row count, BUCKET, the SAMPLE spelling and
the OFFSET modifier are all errors, because a sampling clause that were read and
not applied would be a wrong answer rather than a missing feature.
Schema & Data Types is one page again, with the multimodal content merged into
its blob section rather than sitting beside it.
Agentic Skills stops being a tab. Both its pages are tutorials, so they join the
Tutorials tab; the Basics page that introduces them stays.
The Documentation tab now reads as a manual over Python, TypeScript and SQL
rather than a Python guide with SQL bolted on.
Basics replaces the Catalog section: what the catalog holds, what each API
surface can do, jobs, agentic skills, and what Lance is. Management is new and
is the CRUD and listing reference for every catalog object, with a column per
language so the gaps are visible -- TypeScript has no function or SQL methods,
SQL needs a deployment.
Search becomes Query and gains a SELECT reference, since an analytical query is
a query whether it came from a builder or a statement.
Jobs is new. Every long-running operation already returned the same handle from
an _async method; nothing documented it.
The SQL function reference gains what Lance registers on top of DataFusion:
contains_tokens, backed by the FTS index, and the JSON accessors, backed by a
JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB
does not ship, which is now stated rather than left to a confusing error.
Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text
Search, and splits Query Optimization into Explain and Explain Analyze.
DataFrame support is not going ahead, so the pages documenting it come out:
the relational operations and the expression system, both of which described
only that API.
The manual was framed around having two query surfaces and choosing between
them. With one, that framing is gone — the overview now leads with SQL over
Arrow Flight SQL and what the dialect is, rather than with a comparison to
something that will not exist.
The function reference keeps everything that was about SQL: the DataFusion
version, how Lance columns map onto its type system, the pseudo-columns, blob
handling, and `fts()`. Only the section on reaching functions through the
Python wrapper is replaced, since the wrapper is going too.
Mentions of pandas and polars DataFrames elsewhere in the documentation are
unrelated and untouched.
The manual said the SQL dialect's function and type reference was out of scope.
It is not: the engine is DataFusion 54 with its default function set registered,
so the dialect is documented already and the useful thing is to say which
version and point at it.
The types are worth being concrete about rather than deferring. Lance's
DataFusion table provider returns the dataset's Arrow schema directly — there is
no SQL type layer over storage — so DataFusion's type reference describes Lance
columns exactly, and a vector column is a FixedSizeList that behaves like one.
Two consequences of that mapping are documented because they surprise people.
The pseudo-columns the provider can append are real and typed — `_rowid`,
`_rowaddr` and `_rowoffset` are nullable UInt64, and the nullability is a reused
selection bitmap rather than a missing value, so a null there means nothing.
And a blob column's Arrow type depends on how it is read: descriptions by
default, binary on request, which changes the schema a query sees.
`fts()` is recorded as what it is, a DataFusion TableProvider over the dataset.
That explains the parts that otherwise look arbitrary: why it is a table
function rather than a predicate, why every column of the table is available in
its result, and why projections push into the scan.
Jack Ye