Mint sends a folder URL to the first sidebar page under it unless that page
is the folder's index. With Security listed first, /enterprise opened
Authentication, while production, whose sidebar starts with the Enterprise
overview, serves the overview at /enterprise. Listing Deployment, which
starts with the overview, before Security restores that. The groups, their
pages and the redirects are otherwise unchanged.
The previous commit restored production's sentence so the example is
attributed to Geneva. This brings back the newer page's customer-facing
framing, that feature engineering costs nothing until it is used and never
copies the dataset or touches the videos, while keeping the Geneva link and
naming the API the code calls: @udf, add_columns and backfill. The code and
its timings, measured with Geneva, are unchanged.
The example calls Geneva's @udf, add_columns and backfill, but the sentence
before it described lancedb computed columns and Functions, which take none
of those parameters and have no backfill. This restores the production
sentence, which names Geneva and links its UDF guide; the code and its
timings are unchanged.
Production served the same LeRobot page at /training/lerobot and
/integrations/lerobotdataset. Only the integrations page carries over, so
the old Training URL now redirects to it instead of returning 404.
The TypeScript and Rust Quickstarts read the image from ../../docs/static,
the layout before the documentation moved into this repository. The file is
docs/web/static/assets/images/quickstart/sir-lancelot.jpg, so the TypeScript
test failed with ENOENT and the Rust example panicked at the read. Both now
resolve the image from the example's own directory (__dirname and
CARGO_MANIFEST_DIR), and the two Quickstart exports are regenerated with
docs/web-tests/mdx_snippets_gen.py.
Production serves its "Query with SQL" page at /search/sql, and the same
page at /search/sql/index; both URLs return that page on
docs.lancedb.com. Only the /index form was redirected to /sql, where that
guide now lives, so the canonical URL would have 404ed. It now redirects
to the same destination.
Carries over production's #368 (deploy-freeze 113db11c), which landed after
this branch's gallery was split into a page per demo. Each demo gets a page
in that layout with production's card, badges and link, and a row in the
overview table and the sidebar. The two hero images are byte-identical to
production's.
Carries over production's #360 (deploy-freeze 78d55c2) where it was only
partly ported. The navbar's Support link still opened
mailto:contact@lancedb.com, and the home page linked the contact page but
showed the email address as link text. Both now match production. The
other passages #360 changed are already ported at their new paths.
The note dated the new behavior to "SDK 0.40", but it was verified on the
development SDK 0.40.0b12 built from main, and no stable 0.40 has been
released. The note now names that build and keeps the comparison with
the published 0.39.0, which creates the view empty, has no with_no_data,
and writes both matching rows on the first refresh.
The opt-in Enterprise walkthrough overwrote the table `function_demo`,
registered the Function `double`, then dropped both. On this SDK
drop_function removes the named Function, not only the version this run
created, so a run against a shared database could replace or remove
existing objects.
The walkthrough snippets keep their names. The test registers a copy of
`double` under a name unique to the run, creates the table under such a
name, and drops only those two objects. The table is no longer created
with mode="overwrite", so even a name collision cannot replace an
existing table. The remote test has not been run.
The page said every Function output must be non-nullable. On this SDK
only a scalar output must be. A named-struct output may declare nullable
fields: whether a row has a result at all is the struct's own validity,
which Enterprise keeps in a separate internal column when it expands the
result into sibling columns.
test_function_output_nullability checks both rules against the SDK,
including that a struct declared as a single nullable field is refused.
maintenance/background-jobs.mdx links to /management/function#drop, which
did not exist. The section follows drop_function and drop_function_async
on main: dropping removes the name and its object, and deleting the
content can finish later as a background job.
Found by `mint broken-links --check-anchors` on the assembled site. Three
point at sections that were renamed or moved: the tag-versioning heading,
the branches comparison table, and the table-creation patterns, which now
live on the Create a table page. Three named sections that no longer exist
and now link to the page they were on.
Five pages still said a refresh never revisits a filled row and that a
changed input needs the column dropped and declared again. Since
lancedb#4161 a refresh also recomputes rows whose inputs changed, so the
quickstart, the feature-engineering overview, Guarantees, and the SQL
REFRESH and UPDATE pages now say that.
Retry claims are qualified the same way. A local refresh commits the column
once, and a deployment's job commits finished batches. A retry fills what is
still empty or stale, but can recompute values that were already written (a
refresh whose input record was not saved recomputes next time), so no page
promises that a retry costs only the unfinished rows.
Since lancedb#4180, create_materialized_view runs the initial refresh
before it returns (with_no_data=True skips it), so the page's first manual
refresh now writes 0 rows, not 1. The example now reads the populated view
and refreshes after a source change, and its snippets come from
docs/web-tests/py/test_materialized_view.py. A note scopes the behavior:
0.39 and earlier create the view empty.
Pages that said a refresh computes the rows now say it brings them up to
date, and the SQL note no longer says an incremental refresh covers only
added rows.
Porting the upstream changes replaced snippets/indexing.mdx with a hand
copy from lancedb/docs, which dropped six exports the vector-index page
still imports (`mint validate` warns on each) and added
PyQuantizationCustomParams, which had no source here.
The quantization test region is ported from lancedb/docs and the module
is regenerated, so every export comes from a test again. The regenerated
PyQuantizationCustomParams matches the copy byte for byte. Also fixes
"convetion" and "entires" on the page.
Two pages imported PySchemaAddSetup under the alias SchemaAddSetup, and
Mint resolved only the first, failing `mint validate` on
management/table.mdx. Both pages now import it unaliased.
TsAddProgress was added to the generated tables snippets by hand in
lancedb/docs#243 and dropped by the next regeneration, leaving
ingestion/insert.mdx with a missing import. Its example is restored
unchanged as a test in docs/web-tests/ts/tables.test.ts, which checks
that the callback reports the finished write.
"explictly" and "Additionaly", in snippet sources and their generated
snippets. integrations.mdx is updated in place: regenerating it whole
would also drop PyRerankingWatsonxUsage, which has no source.
`view.table()` and `view.definition()` raise TypeError on a synchronous
view; both are properties. An async view reads its definition with
`await view.definition()`.
Since lancedb#4161 a refresh recomputes rows whose inputs changed, and it
records what it computed from in a second, metadata-only commit, so the
example's version is 4. A local refresh commits once, so a failure keeps
none of its values; the page no longer says each row commits as it
completes.
The guide applied the local decorated callable to `table["value"]`, which
raises TypeError: a table is not subscriptable, and add_columns takes an
application of a registered version, not the callable's result.
The guide now walks one example from definition to filled column: define
`double`, call it locally, connect to Enterprise, create a table whose
`value` column matches the parameter's type and nullability, register the
function, apply `version(value=col("value"))`, refresh, wait, and read the
values back. Its snippets come from docs/web-tests/py/test_function.py.
The local tests in that file check what `@udf` sends for registration, how
a registered version binds a column, and that a local connection refuses
both registration and Function columns. The Enterprise walkthrough runs
only when LANCEDB_URI, LANCEDB_API_KEY and LANCEDB_HOST_OVERRIDE are set.
The page also stops promising that a retry after a failed refresh costs
only the unfinished rows.
SQL now leads with why you'd reach for it beyond analytics: it's served over
Flight SQL, so it's the way in for any language without a native LanceDB SDK,
and broadly every public SDK method has a corresponding SQL statement. Closes
by pointing at the SQL Reference as the spec that defines the precise
behavior behind both the SQL surface and the SDK operations it mirrors.
Moving both into Getting Started (before Agentic Skills) means a reader knows
what the API surface even is before being told how to hand it to an agent.
FAQ content is redistributed into the pages it actually belongs to: license
and Parquet-comparison notes into the Lance format page, scale guidance into
Architecture, index-necessity and refine_factor guidance into the vector
index page, the multiprocessing/fork gotcha into SDKs and SQL, and the
weak-consistency recommendation into Consistency. General query-performance
issues are dropped since Explain and Explain Analyze already cover that
ground.
The new Support page under Deployment carries the OSS Discord and Enterprise
support-email channels that used to live in Troubleshooting.
OSS docs now own the full usage content for features that only run against
an Enterprise deployment, since the client interface itself lives here;
Enterprise docs keep only the parts specific to its own behavior. Sophon's
overlay for these five pages is dropped since the OSS versions are now the
complete reference.
SQL Reference is restructured to match an object-type SQL manual (DDL by
object, then DML, then DQL, then Expressions) instead of the old flat,
loosely-ordered page list, absorbing the SQL client-connection guide and
FTS-via-SQL examples into it.
Indexing loses its wrapper group -- Overview, Vector Index, FTS Index, Scalar
Index and Build & Repair sit directly in Maintenance now, followed by
Compaction, Cleanup and Background Jobs at the same level.
Cleanup is new, split out of Compaction: pruning old versions is a distinct
Enterprise job (cleanup) from merging files (compact), even though optimize()
in OSS bundles both into one call. The retention, disk-usage and
failed-transaction material that used to live under Compaction's umbrella
moves with it.
Background Jobs is new and covers what nothing else does: the job types a
deployment runs on its own -- WAL/SPFresh maintenance, cache prewarming,
registry archiving and metering export, and the object-teardown job that
follows a namespace or view drop. Every entry is grounded against the
job-coordinator's public job_type constants, not the internal two-letter wire
codes used for registry storage. Five existing pages (Build & Repair, Table,
Function, Column Refresh, Materialized View Refresh) gain a one-line job-type
note rather than duplicating a page for job types they already document.
Storage folds into Deployment: storage/index (choosing a backend) and
storage/configuration (configuring one) merge into a single Storage page --
they never overlapped, one is the decision and one is the how-to -- and move
under Deployment along with Monitoring. Cloud Providers loses its nesting;
its two pages plus Azure sit flat, and the former Overview is retitled
Deployment Models so it doesn't collide with the group's own Overview entry.
Redirects cover every moved URL.
One page was carrying two layers. The catalog answers what exists and what it is
called -- names, hierarchy, resolution, and the grants that follow the
hierarchy. The lakehouse is everything that acts on those names: storing rows,
searching them, deriving new ones, and keeping it all fast.
Separating them makes the dependency visible. The catalog is a specification
another engine can implement, which is why a Lance table is reachable without
LanceDB; the lakehouse is where the cost and the tuning live. Governance belongs
to the catalog because the hierarchy is what a grant attaches to, and
maintenance belongs to the lakehouse because it is the files being rewritten.
Also fixes a corruption from an earlier bulk link rewrite: "central
metadata/catalog services" in prose had become a path.
deploy-freeze is the branch currently serving the site, and it moved eight
commits while this work was in flight. Six carried content:
#359 LeRobot loader table replaced with same-batches measurements
#360contact@lancedb.com replaced with the contact page
#363 redirect for the legacy /core/update URL
#364 Stanford Cars source link
#365 cherry-pick of the data-loading rewrite, already ported from main
#366 opt-in cookie consent banner, gating RB2B
Each landed in its new location rather than its old one: the contact link
appears in pages that now live in two repositories, /core/update points at
/ingestion/update, and the consent script is site chrome so it ships with the
base navigation.
Query and Job were one page because they share the _async convention, but they
are different things: a query returns rows and streams them, a job returns an
outcome you wait for. Two pages now, query first, each carrying a short pointer
to the other rather than the shared framing.
Agentic Skills absorbs the agent plugin tutorial, which was a separate page in
Use Cases describing the same plugin in more detail -- the two had diverged, with
the tutorial carrying the fuller install instructions. It also moves to Getting
Started: installing the plugin is something you do before writing anything, not
a concept to understand.
lancedb now holds a page at every path the site publishes, including the twelve
Enterprise ones. Those are real open-source pages -- what the capability is, and
what the embedded form does instead -- rather than blank placeholders, so the
open-source site is complete on its own. sophon becomes an overlay whose page
replaces the one at the same path, which means navigation lives in one place and
the two repositories only have to agree on a path.
That removes the whole class of problem where a page could only be reached from
one of the two trees. The open-source site went from twenty-four dangling
Enterprise links to one, and that one points at the dataset pages the build
repository owns.
Ports four changes that landed on the docs repository after this branch left it:
#344 indexing and quantization rewritten
#354 LeRobot native reader
#361 training/index renamed to training/data-loading and rewritten
#362 distance metric mismatch falls back to a brute-force scan
None applied as patches, because these files were merged or split here --
quantization into the vector index page, multi-vector into vector search,
performance into four pages. Each was ported by taking the upstream page and
re-applying the restructure on top, which also picked up a correction that would
otherwise have been lost: the stricter brute-force threshold for multi-vector
columns is a Cloud limit, not an Enterprise one.
The upstream LeRobot page linked to a Geneva page that no longer exists; it now
points at the feature-engineering pages that replaced it.
Multimodal Data becomes a section of Table, between Data types -- which
introduces blobs -- and Schema evolution.
The rest is link repair. Earlier merges left links pointing at pages that had
moved, which the redirect table covered but the link checker does not follow,
so they were broken in the rendered site: /catalog, /lance, /namespaces,
/tables-and-namespaces, /search/multivector-search, /search/explain-analyze and
/basics/data-types#blob.
The redirect table itself had two conflicting entries for /tables/multimodal and
four pointing at reranking pages that no longer exist; both are fixed.
The tab now runs write, read, curate, derive, train before it reaches the
management side: Ingestion, Semantic Search, Data Curation, Feature Engineering,
Training, then Management and Operations. Indexing joins Operations, where the
index maintenance page already was, and gains it as Build & Repair.
Ingestion gains a Create Table page. It was a section inside the overview, so
the one thing a new reader needs first was the one thing the sidebar did not
show.
Data Curation is new: Aggregate, Sampling, Join, Windowing. The SQL reference
answers what the syntax is; these answer why you would reach for it, framed as
the questions curation actually asks -- what do I have, what can I afford, what
else do I know, which one do I keep. Each links to the reference for the
complete grammar rather than repeating it.
They carry the things a reference page would not: that count(*) minus count(col)
is your coverage, that sampling both sides of a join gives a percent rather than
ten, that a window ordered by a value you recompute moves rows between train and
test, and that a one percent sample tells you about the head and nothing about
the tail.
Six groups were verbs -- Manage, Ingest, Train, Operate, Secure, Deploy -- and
now name the thing rather than the action. Four more were sentence case.
The page labels that were verb phrases follow: Add Computed Column becomes
Computed Columns, Refresh Column becomes Column Refresh, Configuring Storage
becomes Configuration, and Explain becomes Query Plans, which is also more
accurate now that the page covers analyze as well.
SQL statement pages take the keyword -- SELECT, DDL, DML, REFRESH, SHOW, EXPLAIN
-- because a statement name is a noun and lowercase Select and Show read as
instructions. Concept pages there keep noun phrases: Joins, Window Functions,
Subqueries, Operators.
Two duplicate labels are resolved on the way. Reranking sat in Semantic Search
labelled Overview despite being the only page on its subject, and the Enterprise
client page sat in SQL Reference labelled SQL next to the reference Overview; it
is Client Connection now.
Title case leaves prepositions and articles lowercase, so Needle in a Haystack
and Hybrid Search on BEIR are already correct and unchanged.
Ten merges and renames, all in one direction: fewer pages, each covering a whole
subject.
Table absorbs schema evolution and versioning. Vector search absorbs
multi-vector. Explain absorbs explain analyze. Reranking was seven pages -- an
overview and six rerankers of a few dozen lines each -- and is now one. The
vector index page absorbs quantization, which was only ever a parameter of it.
Indexing moves to Manage, where the other catalog objects are. The operational
half stays in Operate as one Indexing page, absorbing GPU indexing -- which was
filed in sophon despite importing lancedb's snippets and being mostly
open-source content with one Enterprise-badged section.
Performance was a single page aggregating tips for four unrelated operations,
which is hard to keep current. Each section now sits with the operation it is
about: ingestion tips on Insert, index tips on Indexing, compaction tips on
Compaction, query tips on Search.
Data Types leaves Concepts. The type reference belongs in the SQL reference,
which already described the same types from the query side; Table now carries a
short summary that points there. The multimodal material it had absorbed is its
own page again, since storing blobs is a task rather than a type definition.
Query becomes Semantic Search, Branches becomes Branch, and the Enterprise
Security page becomes Compliance.
Redirects cover every moved URL, and 30 files were repointed at the merged
anchors.
The function catalogue is now real rather than a pointer. Scalar functions are
listed by taxonomy -- math, string, unicode, regex, date/time, array, map,
struct, crypto, type -- and the names come from what datafusion-functions 54
actually registers, not from what DataFusion documents on main. Aggregate and
window pages gain the same treatment.
Two things the audit corrected rather than copied.
SHOW TABLES does not work. DataFusion implements it over information_schema,
which this session does not enable, so it fails at plan time. It was documented
as working here, on the SHOW page and in three Manage tables; all four are
fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS.
Prepared statements are a Flight SQL protocol feature, not a PREPARE statement.
A script carrying PREPARE ... AS SELECT will not run, while the same intent
expressed through a client will. The new page says so rather than copying
DataFusion's syntax.
Every section of the Documentation tab that has a SQL counterpart now links to
it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather
than at the reference index.
Namespace and Table were the only objects behind a nested group. Both are now a
single page like Database, Materialized view, View and Function, so Manage is a
flat list of the things a catalog holds.
Namespace merges its two pages, which were a concept page and a usage page for
the same object.
Table is a new page. The old tables/index was a 672-line basic-usage tutorial
whose every section had acquired a better home during the restructure --
connecting, ingesting, searching, filtering, adding and dropping columns -- to
the point that its own Next steps cards pointed at the pages that superseded it.
It is replaced by a table object page: what a table carries, create, open, list,
rename, clone, drop, and links out.
Schema evolution, versioning and branching stay as siblings rather than folding
in. Merging them would have made one 1,800-line page, which is not what the
other object pages look like.
Every group now answers exactly one question. Get started and Concepts cover
what the system is; the verb groups cover doing something with it; Operate,
Secure and Deploy cover running it; SQL reference is lookup. Maintenance becomes
Operate, Governance becomes Secure, and the rest take verbs to match.
Three new pages in Concepts and Get started:
Guarantees states what a write promises when it returns, which version a reader
sees, what survives a failure, and -- the half usually missing -- what is not
guaranteed: no multi-table transactions, no read-your-writes across processes
without an interval, unenforced primary keys that do not enforce, no background
compaction.
Limits collects bounds that were stated in passing across a dozen pages, with
the numbers verified against the code rather than inferred: name lengths, job
event caps, SHOW ERRORS bounds, the sampling range that is refused rather than
clamped.
Architecture illustrates embedded against distributed as the one deployment
decision, and is written to cover both rather than only Enterprise.
Indexing splits by question. Choosing an index is a Query decision and moves
there; building, repairing and GPU-building it stay in Operate.
SQL becomes a top-level reference of fifteen pages instead of a sub-group of
Query. Auditing sophon-sql turned up a large undocumented surface: ALTER TABLE
ADD COLUMN in its plain, computed and function forms with FILTER, IGNORE NULLS
and OVER modifiers; SET UNENFORCED PRIMARY KEY; SET and UNSET WAL with sharding;
CREATE FUNCTION with its dependency options; REFRESH COLUMN and REFRESH
MATERIALIZED VIEW; and the SHOW family. None of it was written down.
Both tabs held one group of the same name as the tab, so every entry sat a level
deeper than it needed to and the group heading said nothing the tab had not
already said. The overview now sits above the real groups, which the tab can do
once it uses pages rather than groups.
Demos has no categories at all, so its four pages are simply flat.
The embedding overview is renamed from Overview, which would otherwise have been
the second entry with that label in the same sidebar.
The three gallery pages held thirteen examples as rows in a table, reachable
only by opening the gallery first. Each is now its own page, so the sidebar
lists them and search finds them by name.
A page for an example that lives in vectordb-recipes says so and links to the
notebook and its source, rather than pretending the tutorial is here.
Overview moves above the groups -- the tab uses pages rather than groups, which
is what allows an entry to sit outside every group. Stable World Model joins
Training, since that is what it is doing. The agent plugin page is now Using
LanceDB Agent Skill.
Mintlify cannot link outward from a sidebar: an {anchor, href} entry is ignored
and a bare URL renders with its scheme mangled to a single slash, so a real page
per example is the only way to list them.
The tab is Use Cases now, grouped General, Search, RAG, Feature Engineering,
Training, World Models -- one page per example, named for the example rather
than for its position in a tree.
The gallery pages that listed external notebooks become the More Examples entry
of the category they belong to, instead of standing in front of the real
tutorials. The top-level overview stops linking to itself through three levels
of index page and names the categories directly.
Branch experiments move to Feature Engineering, which is what they are an
experiment in. The agent plugin leads General, since it applies to everything
else in the tab.
LeRobotDataset moves to the Datasets tab, where the LeRobot datasets already
are, and Robotics moves to the top of that tab.
SELECT and Enterprise SQL sat apart in Query while documenting the same engine.
They are now one SQL group: overview, Select, Aggregation, Join, Window,
Sampling, Functions -- one page per construct, with the Enterprise client-setup
and FTS pages merged in rather than parallel to them.
Sampling is new. TABLESAMPLE is implemented and was undocumented: BERNOULLI
picks rows, SYSTEM picks Lance fragments, REPEATABLE fixes the draw so a sample
works as a stable evaluation set. Its refusals are documented too -- a
percentage over 100, a fractional row count, BUCKET, the SAMPLE spelling and
the OFFSET modifier are all errors, because a sampling clause that were read and
not applied would be a wrong answer rather than a missing feature.
Schema & Data Types is one page again, with the multimodal content merged into
its blob section rather than sitting beside it.
Agentic Skills stops being a tab. Both its pages are tutorials, so they join the
Tutorials tab; the Basics page that introduces them stays.
Basics gains a Schema & Data Types section -- the Arrow types a table can hold,
the extension types Lance adds on top (lance.json, lance.blob.v2,
lance.bfloat16), and where geospatial actually stands. The multimodal page moves
in beside it. Consistency joins Basics too, after Query & Job.
Query & Job now leads with the thing the _async convention actually gives you: a
handle. execute_query_async returns a query, everything else returns a job, and
both answer the same four questions -- which one, how is it going, give me the
result, stop. Only the third differs, because a query produces rows to stream
and a job produces an outcome to wait for.
Management gains a Materialized View page. Creating one was documented only on
the refresh page, and the stable-row-ids requirement is a creation-time
decision that cannot be undone.
Embeddings move to Integrations as the Embedding Registry, which is what they
are -- a client-side feature that wraps provider models, sitting above the
providers it registers.
The demo gallery was three apps on one page; it is now three pages. Use Cases
folds into Tutorials as Tutorials and Use Cases.
Versioning and branching are properties of a table, not steps in getting data
in, so they join the Table group. Consistency stays with Ingestion, where the
question it answers -- when do other writers' rows become visible -- is asked.
The Ingesting Data page was in the Table group while an Ingestion overview
existed separately, so the two are merged. The large-dataset section was
duplicated between it and the insert page; the insert page keeps it.
Schema and Data Evolution covered two different things under one title.
Changing a table's shape stays in Table management; deriving values from data
already there becomes a Feature engineering page, next to computed columns and
functions, which are the same job at larger scale.
Feature engineering is now the overview plus one page per operation: add a
computed column, refresh a column, refresh a materialized view. Computed
columns covered declaring and filling on one page, which are separate decisions
with separate failure modes.
Functions move to Management, where the other catalog objects are, and absorb
the environment-mismatch guidance from the Enterprise page -- a symptom table
worth having beside the parameters that cause it.
The Enterprise feature-engineering page is merged into the overview. Two of its
claims were stale: create_materialized_view is implemented for db:// connections
in remote/db.py, so views are not local-only, and the tier is Enterprise rather
than 'Cloud and Enterprise'.
The REST reference leaves the sidebar. Mintlify only generates per-endpoint
pages from a navigation group -- hidden, dropdowns, global anchors and a
top-level openapi key were all tried -- so removing the group removes those
pages. The overview survives as an unlisted page, linked from SDKs & SQL, and
the spec still ships for anything that wants to read it directly.
Brendan Clement
Jack Ye