mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-18 09:42:27 +00:00
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dan Rammer
co-authored by
Claude Opus 5
Claude Opus 5
parent
0968c21f4c
commit
fe34eb3549
@@ -425,19 +425,19 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
the SSTables), deduplicated by primary key; a table without a spec
|
||||
reads the base table.
|
||||
|
||||
#### Parameters
|
||||
@@ -448,7 +448,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -457,7 +457,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -76,15 +76,15 @@ the query optimizer chooses a suboptimal path.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): MergeInsertBuilder
|
||||
useMemWal(enable): MergeInsertBuilder
|
||||
```
|
||||
|
||||
Control MemWAL routing for this merge.
|
||||
|
||||
By default (unset), a `mergeInsert` on a table with an LSM write spec is
|
||||
By default (unset), a `mergeInsert` on a table with a MemWAL write spec is
|
||||
routed through Lance's MemWAL shard writer, and a table without one uses the
|
||||
standard path.
|
||||
|
||||
@@ -92,7 +92,7 @@ standard path.
|
||||
|
||||
* **enable**: `boolean`
|
||||
`true` forces MemWAL routing and errors if the table has no
|
||||
LSM write spec. `false` forces the standard write path even when a spec is set.
|
||||
MemWAL write spec. `false` forces the standard write path even when a spec is set.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -108,12 +108,12 @@ validateSingleShard(validateSingleShard): MergeInsertBuilder
|
||||
|
||||
Controls how an LSM merge checks that its input targets a single shard.
|
||||
|
||||
When a table has an LSM write spec, every row in a `mergeInsert` call must
|
||||
When a table has a MemWAL write spec, every row in a `mergeInsert` call must
|
||||
route to the same shard. When `true` (the default), every row is inspected
|
||||
to verify this. When `false`, only the first row is inspected and the
|
||||
shard it routes to is used for the whole input — a faster path for callers
|
||||
that have already pre-sharded their input. Has no effect on tables without
|
||||
an LSM write spec.
|
||||
a MemWAL write spec.
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
||||
@@ -499,19 +499,19 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
the SSTables), deduplicated by primary key; a table without a spec
|
||||
reads the base table.
|
||||
|
||||
#### Parameters
|
||||
@@ -522,7 +522,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -531,7 +531,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
+116
-116
@@ -228,17 +228,17 @@ version of the table.
|
||||
|
||||
***
|
||||
|
||||
### checkpointLsm()
|
||||
### checkpointMemWal()
|
||||
|
||||
```ts
|
||||
abstract checkpointLsm(): Promise<void>
|
||||
abstract checkpointMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Converge this table's LSM write path into its base table.
|
||||
Converge this table's MemWAL write path into its base table.
|
||||
|
||||
Seals once, then triggers compaction and polls until the L0 that existed
|
||||
Freezes once, then triggers compaction and polls until the SSTables that existed
|
||||
at the start is gone. The target set is fixed at the start, so
|
||||
generations created *during* the checkpoint are ignored — that is what
|
||||
SSTables created *during* the checkpoint are ignored — that is what
|
||||
lets it terminate under write load, and what makes it best-effort: it
|
||||
converges the fresh tier as of some instant. Idempotent, abandonable at
|
||||
any point, and safe to run on a cadence.
|
||||
@@ -254,9 +254,9 @@ is merging. The caller owns the deadline.
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
const before = await table.getLsmStats();
|
||||
await table.checkpointLsm();
|
||||
const after = await table.getLsmStats();
|
||||
const before = await table.getMemWalStats();
|
||||
await table.checkpointMemWal();
|
||||
const after = await table.getMemWalStats();
|
||||
```
|
||||
|
||||
***
|
||||
@@ -279,15 +279,15 @@ Any attempt to use the table after it is closed will result in an error.
|
||||
|
||||
***
|
||||
|
||||
### closeLsmWriters()
|
||||
### closeMemWalWriters()
|
||||
|
||||
```ts
|
||||
abstract closeLsmWriters(): Promise<void>
|
||||
abstract closeMemWalWriters(): Promise<void>
|
||||
```
|
||||
|
||||
Drain and close any cached MemWAL shard writers held for this table.
|
||||
|
||||
When an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) is installed, `mergeInsert` opens MemWAL
|
||||
When an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) is installed, `mergeInsert` opens MemWAL
|
||||
shard writers and caches them for reuse across calls. This closes them,
|
||||
flushing pending data; writers reopen lazily on the next `mergeInsert`.
|
||||
It is a no-op when no writers are cached.
|
||||
@@ -298,17 +298,17 @@ It is a no-op when no writers are cached.
|
||||
|
||||
***
|
||||
|
||||
### compactLsm()
|
||||
### compactMemWal()
|
||||
|
||||
```ts
|
||||
abstract compactLsm(): Promise<void>
|
||||
abstract compactMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Trigger a background L0 → base compaction pass per bucket.
|
||||
Trigger a background SSTable compaction pass per table shard.
|
||||
|
||||
Returns once the passes are *dispatched*, not once they finish — watch
|
||||
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
|
||||
[Table#checkpointLsm](Table.md#checkpointlsm) to wait for convergence.
|
||||
[Table#getMemWalStats](Table.md#getmemwalstats) for progress, or use
|
||||
[Table#checkpointMemWal](Table.md#checkpointmemwal) to wait for convergence.
|
||||
|
||||
#### Returns
|
||||
|
||||
@@ -449,6 +449,23 @@ containing the new version number of the table
|
||||
|
||||
***
|
||||
|
||||
### disableMemWal()
|
||||
|
||||
```ts
|
||||
abstract disableMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Remove the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) from this table, reverting to the standard
|
||||
`mergeInsert` write path.
|
||||
|
||||
Errors if no spec is currently set.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
***
|
||||
|
||||
### display()
|
||||
|
||||
```ts
|
||||
@@ -514,6 +531,54 @@ Drop an index from the table.
|
||||
|
||||
***
|
||||
|
||||
### enableMemWal()
|
||||
|
||||
```ts
|
||||
abstract enableMemWal(spec): Promise<void>
|
||||
```
|
||||
|
||||
Install an [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) on this table, selecting Lance's MemWAL
|
||||
LSM-style write path for future `mergeInsert` calls.
|
||||
|
||||
`MemWalWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
|
||||
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
|
||||
key (`column` and `numBuckets` required).
|
||||
- `"identity"` — shard by the raw value of a scalar `column`.
|
||||
- `"unsharded"` — route every write to a single shard.
|
||||
|
||||
All variants require the table to have an unenforced primary key
|
||||
([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
|
||||
requires it to be the single column being bucketed.
|
||||
|
||||
Omitting `maintainedIndexes` maintains every index the table has,
|
||||
including ones created later, and skips a kind the MemWAL cannot maintain.
|
||||
Naming them pins an exact set, and a still-building index is rejected
|
||||
rather than quietly omitted.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **spec**: [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)
|
||||
The sharding spec to install.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
await table.setUnenforcedPrimaryKey("id");
|
||||
await table.enableMemWal({
|
||||
specType: "bucket",
|
||||
column: "id",
|
||||
numBuckets: 16,
|
||||
maintainedIndexes: ["id_idx"],
|
||||
});
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### fetchBlobFiles()
|
||||
|
||||
```ts
|
||||
@@ -582,15 +647,15 @@ memory; use [Table.fetchBlobFiles](Table.md#fetchblobfiles) for large values.
|
||||
|
||||
***
|
||||
|
||||
### flushLsm()
|
||||
### flushMemWal()
|
||||
|
||||
```ts
|
||||
abstract flushLsm(): Promise<void>
|
||||
abstract flushMemWal(): Promise<void>
|
||||
```
|
||||
|
||||
Seal every bucket's active memtable into a new L0 generation.
|
||||
Freeze every table shard's active memtable into a new SSTable.
|
||||
|
||||
Returns once the seal is committed. Sealing an empty memtable is a no-op,
|
||||
Returns once the freeze is committed. Freezing an empty memtable is a no-op,
|
||||
so this is safe to call repeatedly.
|
||||
|
||||
#### Returns
|
||||
@@ -634,48 +699,48 @@ const { records, truncated } = await table.functionErrors({ column: "embedding"
|
||||
|
||||
***
|
||||
|
||||
### getLsmStats()
|
||||
### getMemWalSpec()
|
||||
|
||||
```ts
|
||||
abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
|
||||
abstract getMemWalSpec(): Promise<undefined | MemWalWriteSpec>
|
||||
```
|
||||
|
||||
Read live per-bucket LSM state.
|
||||
Read the [MemWalWriteSpec](../interfaces/MemWalWriteSpec.md) currently installed on this table.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which bucket is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
|
||||
Resolves to `undefined` only when the LSM write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **includeGenerationRows?**: `boolean`
|
||||
Also count rows per L0 generation.
|
||||
Off by default because each count opens an uncached Lance dataset.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`LsmStats`](../interfaces/LsmStats.md)>
|
||||
|
||||
***
|
||||
|
||||
### getLsmWriteSpec()
|
||||
|
||||
```ts
|
||||
abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
|
||||
```
|
||||
|
||||
Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
|
||||
|
||||
Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
|
||||
spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
|
||||
Resolves to `undefined` when the MemWAL write path is not enabled (no
|
||||
spec has been set, or it was removed with [Table#disableMemWal](Table.md#disablememwal)).
|
||||
The spec is the one installed, including its maintained-index selection:
|
||||
an absent `maintainedIndexes` for every index the table has, an empty
|
||||
array for none.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)>
|
||||
`Promise`<`undefined` \| [`MemWalWriteSpec`](../interfaces/MemWalWriteSpec.md)>
|
||||
|
||||
***
|
||||
|
||||
### getMemWalStats()
|
||||
|
||||
```ts
|
||||
abstract getMemWalStats(includeSstableRows?): Promise<undefined | MemWalStats>
|
||||
```
|
||||
|
||||
Read live per-table-shard MemWAL state.
|
||||
|
||||
Answers "how far behind is my fresh tier", "which table shard is hot", and
|
||||
"why is my fresh-tier vector search brute-force". Mutates no table state.
|
||||
|
||||
Resolves to `undefined` only when the MemWAL write path is not enabled.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **includeSstableRows?**: `boolean`
|
||||
Also count rows per SSTable.
|
||||
Off by default because each count opens an uncached Lance dataset.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`undefined` \| [`MemWalStats`](../interfaces/MemWalStats.md)>
|
||||
|
||||
***
|
||||
|
||||
@@ -1093,54 +1158,6 @@ of the given query
|
||||
|
||||
***
|
||||
|
||||
### setLsmWriteSpec()
|
||||
|
||||
```ts
|
||||
abstract setLsmWriteSpec(spec): Promise<void>
|
||||
```
|
||||
|
||||
Install an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) on this table, selecting Lance's MemWAL
|
||||
LSM-style write path for future `mergeInsert` calls.
|
||||
|
||||
`LsmWriteSpec` chooses one of three sharding strategies via `specType`:
|
||||
|
||||
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
|
||||
key (`column` and `numBuckets` required).
|
||||
- `"identity"` — shard by the raw value of a scalar `column`.
|
||||
- `"unsharded"` — route every write to a single shard.
|
||||
|
||||
All variants require the table to have an unenforced primary key
|
||||
([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
|
||||
requires it to be the single column being bucketed.
|
||||
|
||||
Omitting `maintainedIndexes` maintains every index the table has,
|
||||
including ones created later, and skips a kind the MemWAL cannot maintain.
|
||||
Naming them pins an exact set, and a still-building index is rejected
|
||||
rather than quietly omitted.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **spec**: [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)
|
||||
The sharding spec to install.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
await table.setUnenforcedPrimaryKey("id");
|
||||
await table.setLsmWriteSpec({
|
||||
specType: "bucket",
|
||||
column: "id",
|
||||
numBuckets: 16,
|
||||
maintainedIndexes: ["id_idx"],
|
||||
});
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### setUnenforcedPrimaryKey()
|
||||
|
||||
```ts
|
||||
@@ -1295,23 +1312,6 @@ same tokenizer model files must also exist locally.
|
||||
|
||||
***
|
||||
|
||||
### unsetLsmWriteSpec()
|
||||
|
||||
```ts
|
||||
abstract unsetLsmWriteSpec(): Promise<void>
|
||||
```
|
||||
|
||||
Remove the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) from this table, reverting to the standard
|
||||
`mergeInsert` write path.
|
||||
|
||||
Errors if no spec is currently set.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
***
|
||||
|
||||
### update()
|
||||
|
||||
#### update(opts)
|
||||
|
||||
@@ -275,10 +275,10 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this take query.
|
||||
|
||||
@@ -781,19 +781,19 @@ Empty results retain the query's output schema.
|
||||
|
||||
***
|
||||
|
||||
### useLsm()
|
||||
### useMemWal()
|
||||
|
||||
```ts
|
||||
useLsm(enable): this
|
||||
useMemWal(enable): this
|
||||
```
|
||||
|
||||
Control MemWAL read routing for this query.
|
||||
|
||||
By default (unset), when the table carries a MemWAL write spec (see
|
||||
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
|
||||
[Table#enableMemWal](Table.md#enablememwal)), reads are routed through the LSM scanner so
|
||||
they also return data written via the `mergeInsert` LSM path that has not yet
|
||||
been compacted into the base table (the active/frozen in-memory memtables and
|
||||
the flushed generations), deduplicated by primary key; a table without a spec
|
||||
the SSTables), deduplicated by primary key; a table without a spec
|
||||
reads the base table.
|
||||
|
||||
#### Parameters
|
||||
@@ -804,7 +804,7 @@ reads the base table.
|
||||
even when a spec is present.
|
||||
Note: the LSM scanner does not support every query shape (e.g. reranking,
|
||||
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
|
||||
`useLsm(false)` is set, because a base-only read would silently exclude
|
||||
`useMemWal(false)` is set, because a base-only read would silently exclude
|
||||
un-compacted MemWAL data.
|
||||
|
||||
#### Returns
|
||||
@@ -813,7 +813,7 @@ reads the base table.
|
||||
|
||||
#### Inherited from
|
||||
|
||||
`StandardQueryBase.useLsm`
|
||||
`StandardQueryBase.useMemWal`
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -66,7 +66,6 @@
|
||||
- [BranchDiff](interfaces/BranchDiff.md)
|
||||
- [BranchIndexSummary](interfaces/BranchIndexSummary.md)
|
||||
- [BranchRowCountSummary](interfaces/BranchRowCountSummary.md)
|
||||
- [BucketStats](interfaces/BucketStats.md)
|
||||
- [CatalogOptions](interfaces/CatalogOptions.md)
|
||||
- [CherryPickError](interfaces/CherryPickError.md)
|
||||
- [CherryPickPreview](interfaces/CherryPickPreview.md)
|
||||
@@ -98,7 +97,6 @@
|
||||
- [FunctionErrorRecord](interfaces/FunctionErrorRecord.md)
|
||||
- [FunctionErrors](interfaces/FunctionErrors.md)
|
||||
- [FunctionErrorsOptions](interfaces/FunctionErrorsOptions.md)
|
||||
- [GenerationStats](interfaces/GenerationStats.md)
|
||||
- [HnswPqOptions](interfaces/HnswPqOptions.md)
|
||||
- [HnswSqOptions](interfaces/HnswSqOptions.md)
|
||||
- [IndexConfig](interfaces/IndexConfig.md)
|
||||
@@ -115,9 +113,9 @@
|
||||
- [ListTablesOptions](interfaces/ListTablesOptions.md)
|
||||
- [ListTablesResponse](interfaces/ListTablesResponse.md)
|
||||
- [ListingOptions](interfaces/ListingOptions.md)
|
||||
- [LsmStats](interfaces/LsmStats.md)
|
||||
- [LsmWriteSpec](interfaces/LsmWriteSpec.md)
|
||||
- [MaterializedViewDefinition](interfaces/MaterializedViewDefinition.md)
|
||||
- [MemWalStats](interfaces/MemWalStats.md)
|
||||
- [MemWalWriteSpec](interfaces/MemWalWriteSpec.md)
|
||||
- [MemtableStats](interfaces/MemtableStats.md)
|
||||
- [MergeResult](interfaces/MergeResult.md)
|
||||
- [NativeOAuthConfig](interfaces/NativeOAuthConfig.md)
|
||||
@@ -140,7 +138,9 @@
|
||||
- [SplitHashOptions](interfaces/SplitHashOptions.md)
|
||||
- [SplitRandomOptions](interfaces/SplitRandomOptions.md)
|
||||
- [SplitSequentialOptions](interfaces/SplitSequentialOptions.md)
|
||||
- [SsTableStats](interfaces/SsTableStats.md)
|
||||
- [TableNamesOptions](interfaces/TableNamesOptions.md)
|
||||
- [TableShardStats](interfaces/TableShardStats.md)
|
||||
- [TableStatistics](interfaces/TableStatistics.md)
|
||||
- [TimeoutConfig](interfaces/TimeoutConfig.md)
|
||||
- [TlsConfig](interfaces/TlsConfig.md)
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / GenerationStats
|
||||
|
||||
# Interface: GenerationStats
|
||||
|
||||
One flushed L0 generation.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the generation.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are sealed into L0.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeGenerationRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -1,22 +0,0 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / LsmStats
|
||||
|
||||
# Interface: LsmStats
|
||||
|
||||
Live per-bucket LSM state, as returned by `Table#getLsmStats`.
|
||||
|
||||
Nothing here is derived: sums and differences (total L0 bytes, WAL lag) are
|
||||
the caller's to compute.
|
||||
|
||||
## Properties
|
||||
|
||||
### buckets
|
||||
|
||||
```ts
|
||||
buckets: BucketStats[];
|
||||
```
|
||||
|
||||
One entry per bucket backing this table.
|
||||
@@ -0,0 +1,22 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / MemWalStats
|
||||
|
||||
# Interface: MemWalStats
|
||||
|
||||
Live per-table-shard MemWAL state, as returned by `Table#getMemWalStats`.
|
||||
|
||||
Nothing here is derived: sums and differences (total SSTable bytes, WAL lag) are
|
||||
the caller's to compute.
|
||||
|
||||
## Properties
|
||||
|
||||
### tableShards
|
||||
|
||||
```ts
|
||||
tableShards: TableShardStats[];
|
||||
```
|
||||
|
||||
One entry per table shard backing this table.
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / LsmWriteSpec
|
||||
[@lancedb/lancedb](../globals.md) / MemWalWriteSpec
|
||||
|
||||
# Interface: LsmWriteSpec
|
||||
# Interface: MemWalWriteSpec
|
||||
|
||||
Specification selecting Lance's MemWAL LSM-style write path for
|
||||
`mergeInsert`.
|
||||
@@ -36,7 +36,7 @@ Estimated in-memory size.
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation this memtable will become once sealed.
|
||||
The generation this memtable will become once frozen.
|
||||
|
||||
***
|
||||
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / SsTableStats
|
||||
|
||||
# Interface: SsTableStats
|
||||
|
||||
One SSTable.
|
||||
|
||||
## Properties
|
||||
|
||||
### bytes
|
||||
|
||||
```ts
|
||||
bytes: number;
|
||||
```
|
||||
|
||||
On-disk size of the SSTable.
|
||||
|
||||
***
|
||||
|
||||
### generation
|
||||
|
||||
```ts
|
||||
generation: number;
|
||||
```
|
||||
|
||||
The generation number. Increases as memtables are frozen into SSTables.
|
||||
|
||||
***
|
||||
|
||||
### rows?
|
||||
|
||||
```ts
|
||||
optional rows: number;
|
||||
```
|
||||
|
||||
Present only when `includeSstableRows` was requested. Off by default
|
||||
because each count opens an uncached Lance dataset.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / BucketStats
|
||||
[@lancedb/lancedb](../globals.md) / TableShardStats
|
||||
|
||||
# Interface: BucketStats
|
||||
# Interface: TableShardStats
|
||||
|
||||
Live state of one bucket. A table is N buckets on one node; flattening to a
|
||||
single number hides the one hot bucket that is usually why someone opened
|
||||
Live state of one table shard. A table is N table shards on one node; flattening to a
|
||||
single number hides the one hot table shard that is usually why someone opened
|
||||
this endpoint.
|
||||
|
||||
## Properties
|
||||
@@ -18,7 +18,7 @@ this endpoint.
|
||||
compacting: boolean;
|
||||
```
|
||||
|
||||
Whether a pass owns this bucket's compaction latch right now. Says *a*
|
||||
Whether a pass owns this table shard's compaction latch right now. Says *a*
|
||||
driver is running, not *whose*, and the latch is held from dispatch —
|
||||
including while the pass queues for a pod-wide compactor permit. Read it
|
||||
as "do not pile on", never as "mine is progressing".
|
||||
@@ -35,16 +35,6 @@ The generation the active memtable will become.
|
||||
|
||||
***
|
||||
|
||||
### generations
|
||||
|
||||
```ts
|
||||
generations: GenerationStats[];
|
||||
```
|
||||
|
||||
Flushed L0 generations not yet merged into the base table.
|
||||
|
||||
***
|
||||
|
||||
### manifestVersion
|
||||
|
||||
```ts
|
||||
@@ -61,7 +51,7 @@ Version of the shard manifest these numbers were read from.
|
||||
optional memtables: MemtableStats[];
|
||||
```
|
||||
|
||||
Oldest first, active last. Absent for a `"Sealed"` bucket, whose
|
||||
Oldest first, active last. Absent for a `"Sealed"` table shard, whose
|
||||
in-memory state is torn down.
|
||||
|
||||
***
|
||||
@@ -82,7 +72,17 @@ WAL position replay resumes from.
|
||||
shardId: string;
|
||||
```
|
||||
|
||||
The shard this bucket writes.
|
||||
The id of this table shard.
|
||||
|
||||
***
|
||||
|
||||
### sstables
|
||||
|
||||
```ts
|
||||
sstables: SsTableStats[];
|
||||
```
|
||||
|
||||
SSTables not yet merged into the base table.
|
||||
|
||||
***
|
||||
|
||||
@@ -297,7 +297,7 @@ listing a storage directory.
|
||||
|
||||
::: lancedb.table.Branches
|
||||
|
||||
::: lancedb.LsmWriteSpec
|
||||
::: lancedb.MemWalWriteSpec
|
||||
|
||||
## Functions and Jobs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user