Files
lancedb/docs/web/sql/ddl/table.mdx
Dan RammerandClaude Opus 5 fe34eb3549 refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What

Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.

### 1. MemWAL APIs drop "LSM"

The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.

| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |

TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.

Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.

### 2. Stats use SSTable and table shard

Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.

**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.

`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.

## Compatibility

Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:

- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies

## Validation

- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-16 13:33:57 -05:00

187 lines
5.9 KiB
Plaintext

---
title: "Table"
sidebarTitle: "Table"
description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE."
icon: table
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE TABLE {#create}
```sql
CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
```
Column types are Arrow types under SQL names -- see [Data types](/sql/data-types).
A table inside a [namespace](/sql/ddl/namespace) is written as a dotted path:
`prod.events.clicks`.
## DESCRIBE {#describe}
```sql
DESCRIBE documents
```
The table's columns and types, without a plan -- the SQL counterpart of
`table.schema`.
## ALTER TABLE {#alter}
What `ALTER TABLE ... ADD COLUMN` does depends on what follows the column
name.
### A plain column {#plain-column}
SQL `ADD COLUMN` requires an `AS` binding, so a plain column such as
`ALTER TABLE t ADD COLUMN c INT` is rejected. Add an empty, nullable column
with the SDK instead -- see
[Add nullable columns](/management/table#schema-add-nullable-columns).
### A computed column {#computed-column}
```sql
ALTER TABLE t ADD COLUMN doubled AS value * 2
```
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
The column is committed empty and the expression is stored, not evaluated --
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
{/*
Draft for the Functions release.
### A function column {#function-column}
```sql
ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
```
When the expression calls a [registered function](/sql/ddl/function), the
column is filled by running that function on workers. `version` pins which
immutable function version fills it, so a later registration does not silently
change what the column means.
Several modifiers narrow what gets computed:
| | |
| --- | --- |
| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
| `IGNORE NULLS` | rows with null inputs are skipped |
| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
A function returning several values fills several columns at once:
```sql
ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
```
```sql
ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
```
<Note>
`WITH` on a plain computed column accepts only `version`. Resource options such
as `num_workers` apply to function columns, where there are workers to size.
</Note>
*/}
### Table properties {#properties}
```sql
ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
```
Records which column identifies a row. **Unenforced** is not a hedge in the
naming: uniqueness is not checked on write, and declaring it does not make
duplicates impossible. It tells the planner what to assume. The column must be
declared `NOT NULL`.
```sql
ALTER TABLE t SET MEMWAL
ALTER TABLE t SET MEMWAL WITH (sharding = 'bucket', column = 'id', buckets = 4)
ALTER TABLE t SET MEMWAL WITH (maintained_indexes = '')
ALTER TABLE t UNSET MEMWAL
```
A write-ahead log makes small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `bucket` sharding names its `column`, which must be
the table's unenforced primary key, and inserts into the table then need a
scalar index on that key. `maintained_indexes` chooses which indexes are kept
current against WAL rows. A table with a computed column cannot take a
write-ahead log. See
[Background jobs](/maintenance/background-jobs#wal) for how the log gets
folded back into the table.
<Warning>
[`TABLESAMPLE`](/sql/dql/sampling) is refused on a WAL-backed table, because
rows a sample would draw from are not all in the fragments yet.
</Warning>
**Errors and limits**
| | |
| --- | --- |
| `ALTER TABLE ... DROP COLUMN` | not supported |
Dropping, renaming, or retyping a column is an SDK operation --
[`drop_columns`](/management/table#schema) and
[`alter_columns`](/management/table#schema) -- not a SQL statement.
## CLONE {#clone}
```sql
CREATE TABLE docs_snapshot CLONE documents
CREATE TABLE docs_at_v42 CLONE documents AT 42
CREATE TABLE docs_at_release CLONE documents AT TAG 'release-1'
CREATE TABLE docs_yesterday CLONE documents AT TIMESTAMP '2026-09-01T00:00:00Z'
```
A **shallow clone**: the new table shares the source's data files rather than
copying them, so it costs one metadata commit regardless of the source's size.
`AT` pins the point in the source's history to clone. A bare integer is a
version and a bare quoted string is a timestamp; `VERSION`, `TAG` and
`TIMESTAMP` say so explicitly, and a tag always needs its keyword -- a bare
quoted string cannot be both a tag name and a timestamp.
<Warning>
The clone depends on the source's files continuing to exist. If the source is
[compacted](/maintenance/cleanup) past the retention window, the version the
clone points at can be pruned. The SDK's
[`clone_table`](/management/table#rename-clone) also offers a deep clone for
when the copy has to outlive the source's history.
</Warning>
## DROP TABLE {#drop}
```sql
DROP TABLE documents
```
<Warning>
Removes the table and every version of it. There is no undrop.
</Warning>
<Note>
**Enterprise job type:** `drop_table` or `drop_object`, depending on how the
deployment stores its catalog. See
[Background jobs](/maintenance/background-jobs#teardown).
</Note>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Table" icon="table" href="/management/table">
The SDK equivalent, and what a table carries.
</Card>
<Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
Renaming, retyping and dropping columns from the SDK.
</Card>
</CardGroup>