mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Xuanwo
725eee8a47
Ambiguous gateway failures can occur after a write commits, so replaying the operation can append duplicate rows. Non-idempotent writes now retry only known precommit failures. Multipart retries reuse the same upload ID, part ID, and encoded bytes; ambiguous completion returns an unknown-outcome error without abort or replay. A definite 409/429 completion rejection triggers best-effort cleanup using the call's retry budget. Replayable parts and the shared byte-based partition target default to 64 MiB. Dictionary-bearing batches are compacted before size estimation and again when actual compressed IPC framing requires further splitting: keys are remapped without changing their types or schema, view backing buffers are collected, and ListView/LargeListView child ranges are compacted as an interval union so overlapping views retain sharing. Parts are prepared on demand; a single row and its schema can still exceed the target. Multipart progress starts at zero and counts each acknowledged IPC body once. Rust write builders and Python/TypeScript `add` expose per-call retry overrides. Atomic `create_table` pre-encodes to a temporary file and rejects compressed IPC over 256 MiB before sending any request; the schema-first import guidance now includes Python, TypeScript, and Rust examples. The shared partition target also affects local automatic partitioning. At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases pass on source verified against the committed four-file patch. With a 16 KiB target and 128 unique 1 KiB dictionary values, the original ordinary dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 / 143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 / 149,632. Encoded HTTP bodies are decoded to check values, nulls, slices, metadata, changing dictionaries, and overlapping views; additional cases cover framing-driven splitting, oversized rows, stable retry bytes/IDs, acknowledged progress, and atomic create. Real HTTP fault tests commit Blob payloads to a local Lance table and verify byte equality, no duplicate append, and zero abort/replay after ambiguous completion. The earlier 811 remote and 7 progress cases at `f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote database/options results (437 cases plus 8 fixture consumers), Node retry-binding evidence, and Rust/TypeScript documentation example checks remain historical validation. Python/Node extensions were not rebuilt for the dictionary follow-up; these results do not establish current binding integration or fresh cross-platform/minimum-dependency CI acceptance. A separate real Query Node lifecycle experiment used SDK `b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon `58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener forced a connection closed, a surviving QN accepted a retry with identical upload ID, part ID, and IPC bytes. Ordinary insert and complete response-loss scenarios sent no replay or abort; fresh reads observed exactly one commit and exact Blob bytes in all three scenarios. This local experiment used a 1-second drain grace, two 32 KiB payloads, and one distinct staged part. It validates acknowledgement-loss recovery at those revisions, not concurrent overlapping attempts, large uploads, physical orphan cleanup, the later dictionary encoding change, or the original sixth-insert EOF. The part target is not a process memory limit: source/ancestor buffers and temporary compaction allocations may coexist. Struct/FixedSizeList parent nulls can retain hidden child keys; this is not complete logical-reachability GC for every Arrow combination, and retained hidden values may affect wire size. The current local Lance 15.0.0-beta.17 rejects ListView/LargeListView persistence; the tests establish remote client IPC encoding behavior, not native or Cloud persistence support. Live Cloud, full workspace/Clippy, the original ordinary-insert disconnect root cause, and Sophon's asynchronous transaction cleanup behavior are not established. Cancelling during staging retains the existing server-session cleanup limitation. Function reader delivery is separate.
LanceDB JavaScript SDK
A JavaScript library for LanceDB.
Installation
npm install @lancedb/lancedb
This will download the appropriate native library for your platform. We currently support:
- Linux (x86_64 and aarch64 on glibc and musl)
- MacOS (Intel and ARM/M1/M2)
- Windows (x86_64 and aarch64)
Usage
Basic Example
import * as lancedb from "@lancedb/lancedb";
const db = await lancedb.connect("data/sample-lancedb");
const table = await db.createTable("my_table", [
{ id: 1, vector: [0.1, 1.0], item: "foo", price: 10.0 },
{ id: 2, vector: [3.9, 0.5], item: "bar", price: 20.0 },
]);
const results = await table.vectorSearch([0.1, 0.3]).limit(20).toArray();
console.log(results);
The quickstart contains more complete examples.
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.