mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-20 02:32:27 +00:00
Jack YeandBrendan Clement
47b7babc08
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
66 lines
3.1 KiB
Plaintext
66 lines
3.1 KiB
Plaintext
---
|
|
title: "Background jobs"
|
|
sidebarTitle: "Background Jobs"
|
|
description: "Jobs a deployment runs on its own — visible in SHOW JOBS, not started by a call you make."
|
|
icon: gears
|
|
keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Every job documented elsewhere in these docs -- [Build & Repair](/indexing/reindexing),
|
|
[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup),
|
|
[Column Refresh](/feature-engineering/refresh-column) -- is one you cause,
|
|
directly or by a schedule you set. This page is the rest: jobs the deployment
|
|
runs to keep itself correct and fast, which you may see named in
|
|
[`SHOW JOBS`](/sql/ddl/job#show) without having asked for any of them.
|
|
|
|
There is no action to take on these beyond recognizing the name. None of them
|
|
has a parameter you set or a call you make.
|
|
|
|
## Write-ahead log maintenance {#wal}
|
|
|
|
For a table with a [write-ahead log](/sql/ddl/table#properties) enabled, two jobs keep the
|
|
log from growing without bound:
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `sstable_compaction` | Folds a WAL bucket's SSTables into the base table by primary key. Distinct from `compact`, which merges base-table fragments -- the two are unrelated. |
|
|
| `wal_index_catchup` | Extends a base-table index over the rows `sstable_compaction` just folded in, and records how far the index has caught up. Nothing else advances that record. |
|
|
|
|
## Index maintenance {#index}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `index_remap` | Reassigns index entries after row ids shift underneath an index -- the kind of remap a delta compaction can require. |
|
|
| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. |
|
|
|
|
## Cache warming {#cache}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
|
|
| `prewarm_index_cache` | The same, for index structures. |
|
|
| `startup_prewarm_idx` | Runs cache prewarming as part of a query node's own startup, rather than on a schedule. |
|
|
|
|
## Registry housekeeping {#registry}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `job_registry_archive` | Sweeps terminal jobs out of the active registry and into archival storage, on a per-database schedule. |
|
|
| `job_events_export` | Copies terminal jobs' event histories into a curated system table for metering. Registry-scoped, like the archive sweep. |
|
|
|
|
These two are why an old job you completed weeks ago no longer shows in
|
|
`SHOW JOBS` -- it was archived, not deleted; its events survive in the exported
|
|
table.
|
|
|
|
## Object teardown {#teardown}
|
|
|
|
| Job type | What it does |
|
|
| --- | --- |
|
|
| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. |
|
|
|
|
You trigger this indirectly, by dropping a [namespace](/management/namespace),
|
|
[view](/sql/ddl/view#drop), or [function](/management/function#drop) -- the teardown
|
|
itself runs as this job rather than inline with the call.
|