rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-15 16:22:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: revise getting started, concepts and shared setup

Browse Source
This commit is contained in:
Brendan Clement
2026-10-14 21:48:07 -07:00
parent 9f37f39267
commit cf355f14dd
33 changed files with 2310 additions and 2175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+47 -1
View File
@@ -94,7 +94,7 @@ what the capability is, and what the embedded form does instead. A separate
private repository supplies the fuller version of each, which replaces the page
at the same path when the published site is assembled.
Two things to know before editing:
Four things to know before editing:
**Code examples are not written into the pages.** They live in real tests under
`docs/web-tests/{py,ts,rs}`, are extracted into `docs/web/snippets/`, and are
@@ -116,6 +116,52 @@ Enterprise pages attach their additions to the right section, so they are
assigned once and never regenerated. Leave an existing anchor alone even when
you reword the heading above it; only new headings need a new one.
**A page can declare where it applies.** Its frontmatter says whether LanceDB
OSS and LanceDB Enterprise support the feature: use only `available` or
`unavailable`, and one sentence that explains its scope. Availability describes
product support, not whether a particular deployment has enabled or configured
the feature. A supported feature stays `available` when it requires setup, a
supported SDK or server version, or has differences between offerings. State
those requirements in the summary or the relevant section, including unsupported
subfeatures. Do not use `varies`, infer support from an unknown status, or mark an
absent capability available. Narrow a page to the capability it actually covers
when necessary. Publication of unlaunched features is controlled separately by
`launch`, not by an availability status:
```yaml
availability:
oss: unavailable
enterprise: available
summary: >-
Requests to a deployment carry an API key or an OAuth token. Embedded
LanceDB has no server to sign in to.
```
The assembled site renders this as a label above the page's content, as a
sidebar tag when only one offering has the feature, and as the page's row in
the comparison on [OSS and Enterprise](docs/web/basics/offerings.mdx). Do not
also open the page with a badge of its own, and keep a qualification that only
one section needs in that section's text. A `mint dev` preview of `docs/web`
alone renders neither the labels nor the comparison.
**A feature can be documented before it launches.** Functions are documented
ahead of their launch. A page that documents only them says so in its
frontmatter:
```yaml
launch: functions
```
Another page marks what it says about them. A section between
`{/* if functions launched */}` and `{/* end if */}`, each on a line of its own,
is published only once Functions launch; `{/* if functions not launched */}`
marks one published only until then. Put the marker at the end of a table row or
list item to gate that line alone. Until the launch, the assembled site leaves
out those pages, their examples, and every marked section, navigation entry and
redirect, and stops the build if anything still published links to them. A
`mint dev` preview of `docs/web` alone shows both sides of every marker. The
documentation repository's README shows how to assemble either site.
## Release process
For information on the release process, see: [release_process.md](release_process.md)
docs/web-tests/py/test_connection.py
+84 -22
View File
@@ -4,6 +4,8 @@
from pathlib import Path
import shutil
import pytest
def test_connection():
# --8<-- [start:connect]
@@ -29,19 +31,16 @@ async def connect_async_example():
def connect_enterprise_quickstart_config():
# --8<-- [start:connect_enterprise_quickstart]
import os
import lancedb
# --8<-- [start:connect_enterprise_quickstart]
uri = "db://your-database-uri"
api_key = "your-api-key"
region = "us-east-1"
host_override = "https://your-enterprise-endpoint.com"
db = lancedb.connect(
uri=uri,
api_key=api_key,
region=region,
host_override=host_override,
uri=os.environ["LANCEDB_URI"], # db://your-database
api_key=os.environ["LANCEDB_API_KEY"],
region=os.environ.get("LANCEDB_REGION", "us-east-1"),
host_override=os.environ["LANCEDB_HOST_OVERRIDE"], # https://your-endpoint
)
# --8<-- [end:connect_enterprise_quickstart]
return db
@@ -57,17 +56,29 @@ def test_connect_enterprise_quickstart(monkeypatch):
return object()
monkeypatch.setattr(lancedb, "connect", fake_connect)
monkeypatch.setenv("LANCEDB_URI", "db://your-database")
monkeypatch.setenv("LANCEDB_API_KEY", "your-api-key")
monkeypatch.delenv("LANCEDB_REGION", raising=False)
monkeypatch.setenv("LANCEDB_HOST_OVERRIDE", "https://your-endpoint")
db = connect_enterprise_quickstart_config()
assert db is not None
assert connect_enterprise_quickstart_config() is not None
assert captured == {
"uri": "db://your-database-uri",
"uri": "db://your-database",
"api_key": "your-api-key",
"region": "us-east-1",
"host_override": "https://your-enterprise-endpoint.com",
"host_override": "https://your-endpoint",
}
def test_connect_enterprise_quickstart_requires_an_endpoint(monkeypatch):
monkeypatch.setenv("LANCEDB_URI", "db://your-database")
monkeypatch.setenv("LANCEDB_API_KEY", "your-api-key")
monkeypatch.delenv("LANCEDB_HOST_OVERRIDE", raising=False)
with pytest.raises(KeyError, match="LANCEDB_HOST_OVERRIDE"):
connect_enterprise_quickstart_config()
def connect_object_storage_config():
# --8<-- [start:connect_object_storage]
import lancedb
@@ -105,14 +116,26 @@ def namespace_table_ops_example():
mode="create", # use "overwrite" only if you want to replace existing table
)
# Verify
print(db.list_namespaces()) # ['prod']
print(db.list_namespaces(namespace_path=["prod"])) # ['recommendations', 'search']
print(db.list_tables(namespace_path=["prod", "search"])) # ['user']
print(db.list_tables(namespace_path=["prod", "recommendations"])) # ['user']
# Verify: listings return a response with the names and a page token
print(db.list_namespaces().namespaces) # ['prod']
print(db.list_namespaces(namespace_path=["prod"]).namespaces)
# ['recommendations', 'search']
print(db.list_tables(namespace_path=["prod", "search"]).tables) # ['user']
print(db.list_tables(namespace_path=["prod", "recommendations"]).tables) # ['user']
# --8<-- [end:namespace_table_ops]
def test_namespace_table_ops(tmp_path, monkeypatch, capsys):
monkeypatch.chdir(tmp_path)
namespace_table_ops_example()
assert capsys.readouterr().out.splitlines() == [
"['prod']",
"['recommendations', 'search']",
"['user']",
"['user']",
]
def namespace_admin_ops_example():
# --8<-- [start:namespace_admin_ops]
import lancedb
@@ -124,8 +147,8 @@ def namespace_admin_ops_example():
db.create_namespace(["prod", "search"])
child_namespaces = db.list_namespaces(namespace_path=["prod"]).namespaces
print(f"Child namespaces under {namespace}: {child_namespaces}")
# Child namespaces under ['prod', 'search']: ['search']
print(f"Child namespaces under ['prod']: {child_namespaces}")
# Child namespaces under ['prod']: ['search']
metadata = db.describe_namespace(["prod", "search"])
print(f"Metadata for namespace {namespace}: {metadata}")
@@ -134,7 +157,17 @@ def namespace_admin_ops_example():
db.drop_namespace(["prod", "search"], mode="skip")
db.drop_namespace(["prod"], mode="skip")
# --8<-- [end:namespace_admin_ops]
return child_namespaces, metadata
return db, child_namespaces, metadata
def test_namespace_admin_ops(tmp_path, monkeypatch, capsys):
monkeypatch.chdir(tmp_path)
db, child_namespaces, metadata = namespace_admin_ops_example()
assert child_namespaces == ["search"]
assert capsys.readouterr().out.splitlines()[0] == (
"Child namespaces under ['prod']: ['search']"
)
assert db.list_namespaces().namespaces == []
async def connect_object_storage_config_async():
# --8<-- [start:connect_object_storage_async]
@@ -146,3 +179,32 @@ async def connect_object_storage_config_async():
# --8<-- [end:connect_object_storage_async]
return async_db
def test_connect_enterprise_placeholders(monkeypatch):
import lancedb
captured = {}
def fake_connect(**kwargs):
captured.update(kwargs)
return object()
monkeypatch.setattr(lancedb, "connect", fake_connect)
# --8<-- [start:connect_enterprise_placeholders]
import lancedb
db = lancedb.connect(
uri="db://your-database",
api_key="your-api-key",
region="us-east-1",
host_override="https://your-endpoint",
)
# --8<-- [end:connect_enterprise_placeholders]
assert db is not None
assert captured == {
"uri": "db://your-database",
"api_key": "your-api-key",
"region": "us-east-1",
"host_override": "https://your-endpoint",
}
docs/web-tests/py/test_quickstart.py
+255 -142
View File
@@ -1,14 +1,125 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import os
import shutil
import uuid
from pathlib import Path
import lancedb
import pytest
import pytest_asyncio
# What the quickstart's download step leaves in an application's directory.
IMAGE = (
Path(__file__).resolve().parents[2]
/ "web/static/assets/images/quickstart/sir-lancelot.jpg"
)
DEPLOYMENT = ("LANCEDB_URI", "LANCEDB_HOST_OVERRIDE", "LANCEDB_API_KEY")
# The connection settings alone never send these tests to a deployment.
OPT_IN = "LANCEDB_DOCS_TEST_DEPLOYMENT"
def test_quickstart(db_path_factory):
uri = db_path_factory("quickstart_db")
db = lancedb.connect(uri)
def deployment_settings():
if os.environ.get(OPT_IN) != "1":
pytest.skip(f"set {OPT_IN}=1 and {', '.join(DEPLOYMENT)} to use a deployment")
missing = [name for name in DEPLOYMENT if not os.environ.get(name)]
if missing:
pytest.fail(f"{OPT_IN}=1 needs {', '.join(missing)}")
return dict(
uri=os.environ["LANCEDB_URI"],
api_key=os.environ["LANCEDB_API_KEY"],
region=os.environ.get("LANCEDB_REGION", "us-east-1"),
host_override=os.environ["LANCEDB_HOST_OVERRIDE"],
)
class OwnedTables:
"""A deployment connection on which a test creates only tables of its own.
Each name gets a suffix unique to the test and is claimed only while no
table has it; the test drops what it claimed, whether or not it passed.
"""
def __init__(self, db):
self.db = db
self.suffix = uuid.uuid4().hex[:12]
self.claimed = []
def claim(self, name, existing, mode):
owned = f"{name}_{self.suffix}"
if owned in self.claimed:
return owned, mode
if owned in existing:
raise RuntimeError(f"{owned} already exists and is not this test's")
self.claimed.append(owned)
return owned, "create"
def table_names(self):
names, token = [], None
while True:
page = self.db.list_tables(page_token=token)
names += page.tables
if not (token := page.page_token):
return names
def create_table(self, name, *args, mode="create", **kwargs):
owned, mode = self.claim(name, self.table_names(), mode)
return self.db.create_table(owned, *args, mode=mode, **kwargs)
def drop_claimed(self):
existing = self.table_names()
for name in self.claimed:
if name in existing:
self.db.drop_table(name)
class AsyncOwnedTables(OwnedTables):
async def table_names(self):
names, token = [], None
while True:
page = await self.db.list_tables(page_token=token)
names += page.tables
if not (token := page.page_token):
return names
async def create_table(self, name, *args, mode="create", **kwargs):
owned, mode = self.claim(name, await self.table_names(), mode)
return await self.db.create_table(owned, *args, mode=mode, **kwargs)
async def drop_claimed(self):
existing = await self.table_names()
for name in self.claimed:
if name in existing:
await self.db.drop_table(name)
@pytest.fixture
def deployment():
owned = OwnedTables(lancedb.connect(**deployment_settings()))
yield owned
owned.drop_claimed()
@pytest.fixture(params=["local", "enterprise"])
def db(request, db_path_factory):
if request.param == "local":
yield lancedb.connect(db_path_factory("quickstart_db"))
else:
yield request.getfixturevalue("deployment")
@pytest_asyncio.fixture(params=["local", "enterprise"])
async def async_db(request, db_path_factory):
if request.param == "local":
yield await lancedb.connect_async(db_path_factory("quickstart_async_db"))
return
owned = AsyncOwnedTables(await lancedb.connect_async(**deployment_settings()))
yield owned
await owned.drop_claimed()
def create_characters(db):
# --8<-- [start:quickstart_data]
data = [
{
@@ -41,14 +152,29 @@ def test_quickstart(db_path_factory):
# --8<-- [start:quickstart_create_table]
table = db.create_table("characters", data=data, mode="overwrite")
# --8<-- [end:quickstart_create_table]
assert len(table) == 3
# Drop the table to test create without overwrite
db.drop_table("characters")
assert table.count_rows() == 3
# --8<-- [start:quickstart_add_data]
table.add(
[
{
"id": "4",
"name": "Morgana",
"role": "Sorceress",
"description": "Powerful sorceress of Avalon.",
"stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 4},
"vector": [0.3, 0.9, 0.6, 0.8],
}
]
)
# --8<-- [end:quickstart_add_data]
assert table.count_rows() == 4
return table
def test_quickstart(db):
table = create_characters(db)
# --8<-- [start:quickstart_create_table_no_overwrite]
table = db.create_table("characters", data=data)
# --8<-- [end:quickstart_create_table_no_overwrite]
assert len(table) == 3
# --8<-- [start:quickstart_vector_search_1]
# Search for examples similar to a "wise magical advisor"
query_vector = [0.2, 0.8, 0.4, 0.9]
@@ -62,7 +188,14 @@ def test_quickstart(db_path_factory):
)
print(result)
# --8<-- [end:quickstart_vector_search_1]
assert result.head(1)["name"][0] == "Merlin"
assert result["name"].to_list() == ["Merlin", "Morgana"]
# --8<-- [start:quickstart_output_pandas]
# Ensure you run `pip install pandas` beforehand
result = table.search(query_vector).limit(2).to_pandas()
print(result)
# --8<-- [end:quickstart_output_pandas]
assert result["name"].to_list() == ["Merlin", "Morgana"]
# --8<-- [start:quickstart_curate_with_metadata]
curated = (
@@ -74,131 +207,15 @@ def test_quickstart(db_path_factory):
)
print(curated)
# --8<-- [end:quickstart_curate_with_metadata]
assert curated.head(1)["name"][0] == "Merlin"
assert curated["name"].to_list() == ["Merlin", "Morgana"]
# --8<-- [start:quickstart_output_pandas]
# Ensure you run `pip install pandas` beforehand
result = table.search(query_vector).limit(2).to_pandas()
print(result)
# --8<-- [end:quickstart_output_pandas]
assert result.iloc[0]["name"] == "Merlin"
# --8<-- [start:quickstart_add_feature]
table.add_columns(
{
"power_score": "cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)"
}
)
# --8<-- [end:quickstart_add_feature]
assert "power_score" in table.schema.names
# --8<-- [start:quickstart_query_feature]
features = table.search().select(["name", "role", "power_score"]).to_polars()
print(features)
# --8<-- [end:quickstart_query_feature]
assert "power_score" in features.columns
# --8<-- [start:quickstart_multimodal_bytes]
from pathlib import Path
image_path = Path("docs/web/static/assets/images/quickstart/sir-lancelot.jpg")
image_bytes = image_path.read_bytes()
multimodal_table = db.create_table(
"character_images",
data=[
{
"id": "lancelot",
"description": "Portrait of Sir Lancelot",
"image": image_bytes,
"vector": [0.9, 0.1, 0.5, 0.4],
}
],
mode="overwrite",
)
# --8<-- [end:quickstart_multimodal_bytes]
assert len(multimodal_table) == 1
# --8<-- [start:quickstart_open_table]
table = db.open_table("characters")
# --8<-- [end:quickstart_open_table]
# --8<-- [start:quickstart_add_data]
more_data = [
{
"id": "4",
"name": "Morgana",
"role": "Sorceress",
"description": "Powerful sorceress of Avalon.",
"stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 4},
"vector": [0.3, 0.9, 0.6, 0.8],
"power_score": 3.75,
},
]
# Add data to table
table.add(more_data)
# --8<-- [end:quickstart_add_data]
assert len(table) == 4
# --8<-- [start:quickstart_vector_search_2]
# Search for examples similar to a "powerful sorceress"
query_vector = [0.3, 0.9, 0.6, 0.8]
results = table.search(query_vector).limit(2).to_polars()
print(results)
# --8<-- [end:quickstart_vector_search_2]
assert results.head(1)["name"][0] == "Morgana"
# --8<-- [start:quickstart_computed_feature]
table.add_columns(computed={"combat_score": "(stats.magic + stats.strength) / 2.0"})
table.refresh_column("combat_score")
# --8<-- [end:quickstart_computed_feature]
scores = (
table.search().select(["combat_score"]).to_arrow()["combat_score"].to_pylist()
)
assert scores == [2.5, 3.5, 3.0, 3.5]
# --8<-- [start:quickstart_training]
import torch
from lancedb.permutation import Permutation
dataset = Permutation.identity(table).select_columns(
["power_score", "combat_score"]
)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)
# --8<-- [end:quickstart_training]
batches = list(dataloader)
assert len(batches) == 1
assert set(batches[0]) == {"power_score", "combat_score"}
assert all(isinstance(value, torch.Tensor) for value in batches[0].values())
assert sorted(batches[0]["combat_score"].tolist()) == sorted(scores)
# --8<-- [start:quickstart_training_projection]
dataset = Permutation.identity(table).select_columns(["power_score"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=1024)
# --8<-- [end:quickstart_training_projection]
batch = next(iter(dataloader))
assert set(batch) == {"power_score"}
assert batch["power_score"].tolist() == [3.5, 4.0, 3.0, 3.75]
curated = table.search(query_vector).where("stats.magic < 4").limit(2).to_list()
assert [row["name"] for row in curated] == ["King Arthur", "Sir Lancelot"]
@pytest.mark.asyncio
async def test_quickstart_async_api(db_path_factory):
db_uri = db_path_factory("quickstart_async_db")
import lancedb
async_db = await lancedb.connect_async(db_uri)
# --8<-- [start:quickstart_data_async]
async def test_quickstart_async_api(async_db):
data = [
{
"id": "1",
"name": "King Arthur",
"role": "King",
"description": "Leader of Camelot and wielder of Excalibur.",
"stats": {"strength": 4, "magic": 1, "leadership": 5, "wisdom": 4},
"vector": [0.7, 0.1, 0.9, 0.7],
},
{
"id": "2",
"name": "Merlin",
@@ -206,17 +223,8 @@ async def test_quickstart_async_api(db_path_factory):
"description": "Advisor and prophet with deep magical knowledge.",
"stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 5},
"vector": [0.2, 0.9, 0.4, 0.9],
},
{
"id": "3",
"name": "Sir Lancelot",
"role": "Knight",
"description": "Legendary knight known for courage and combat skill.",
"stats": {"strength": 5, "magic": 1, "leadership": 3, "wisdom": 3},
"vector": [0.9, 0.1, 0.5, 0.4],
},
}
]
# --8<-- [end:quickstart_data_async]
# --8<-- [start:quickstart_create_table_async]
async_table = await async_db.create_table(
@@ -236,5 +244,110 @@ async def test_quickstart_async_api(db_path_factory):
).select(["name", "role", "description", "_distance"]).limit(2).to_polars()
print(async_result)
# --8<-- [end:quickstart_vector_search_1_async]
assert async_result["name"].to_list() == ["Merlin"]
assert async_result.head(1)["name"][0] == "Merlin"
def test_quickstart_derived_feature(db):
table = create_characters(db)
# --8<-- [start:quickstart_add_feature]
table.add_columns(
{
"power_score": "cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)"
}
)
# --8<-- [end:quickstart_add_feature]
# --8<-- [start:quickstart_query_feature]
features = table.search().select(["name", "role", "power_score"]).to_polars()
print(features)
# --8<-- [end:quickstart_query_feature]
assert sorted(zip(features["name"], features["power_score"])) == [
("King Arthur", 3.5),
("Merlin", 4.0),
("Morgana", 3.75),
("Sir Lancelot", 3.0),
]
def description_lengths(table):
rows = table.search().select(["description", "description_length"]).to_list()
return {row["description"]: row["description_length"] for row in rows}
def test_quickstart_computed_column(db_path_factory):
table = create_characters(lancedb.connect(db_path_factory("computed")))
# --8<-- [start:quickstart_computed_feature]
table.add_columns(computed={"description_length": "length(description)"})
table.refresh_column("description_length")
# --8<-- [end:quickstart_computed_feature]
lengths = description_lengths(table)
assert lengths == {text: len(text) for text in lengths}
def test_quickstart_computed_column_on_a_deployment(deployment):
table = create_characters(deployment)
# --8<-- [start:quickstart_computed_feature_enterprise]
table.add_columns(computed={"description_length": "length(description)"})
job = table.refresh_column_async("description_length")
job.wait()
# --8<-- [end:quickstart_computed_feature_enterprise]
lengths = description_lengths(table)
assert len(lengths) == 4
assert lengths == {text: len(text) for text in lengths}
with pytest.raises(Exception):
table.refresh_column("description_length")
def test_quickstart_multimodal(db, tmp_path, monkeypatch):
app = tmp_path / "app"
(app / "data").mkdir(parents=True)
shutil.copy(IMAGE, app / "data/sir-lancelot.jpg")
monkeypatch.chdir(app)
# --8<-- [start:quickstart_multimodal_bytes]
from pathlib import Path
image_bytes = Path("data/sir-lancelot.jpg").read_bytes()
multimodal_table = db.create_table(
"character_images",
data=[
{
"id": "lancelot",
"description": "Portrait of Sir Lancelot",
"image": image_bytes,
"vector": [0.9, 0.1, 0.5, 0.4],
}
],
mode="overwrite",
)
# --8<-- [end:quickstart_multimodal_bytes]
assert multimodal_table.count_rows() == 1
stored = multimodal_table.search().select(["image"]).to_list()[0]["image"]
assert stored == IMAGE.read_bytes()
def test_quickstart_training(db):
table = create_characters(db)
# --8<-- [start:quickstart_training]
import torch
from lancedb.permutation import Permutation
table.add_columns(
{
"magic": "cast(stats.magic as float)",
"strength": "cast(stats.strength as float)",
}
)
dataset = Permutation.identity(table).select_columns(["magic", "strength"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)
# --8<-- [end:quickstart_training]
batches = list(dataloader)
assert len(batches) == 1
assert set(batches[0]) == {"magic", "strength"}
assert all(isinstance(value, torch.Tensor) for value in batches[0].values())
assert sorted(batches[0]["magic"].tolist()) == [1.0, 1.0, 5.0, 5.0]
docs/web-tests/py/test_read_consistency.py
+35
View File
@@ -0,0 +1,35 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
def test_read_consistency():
# --8<-- [start:read_consistency]
from datetime import timedelta
from tempfile import TemporaryDirectory
import lancedb
with TemporaryDirectory(prefix="lancedb-consistency-") as uri:
writer = lancedb.connect(uri)
table = writer.create_table("documents", [{"id": 1, "text": "first"}])
default_reader = lancedb.connect(uri).open_table("documents")
latest_reader = lancedb.connect(
uri, read_consistency_interval=timedelta(0)
).open_table("documents")
table.add([{"id": 2, "text": "second"}])
default_count = default_reader.count_rows()
latest_count = latest_reader.count_rows()
print(f"Default reader: {default_count} rows") # 1
print(f"Zero-interval reader: {latest_count} rows") # 2
default_reader.checkout_latest()
refreshed_count = default_reader.count_rows()
print(f"After explicit refresh: {refreshed_count} rows") # 2
# --8<-- [end:read_consistency]
assert default_count == 1
assert latest_count == 2
assert refreshed_count == 2
docs/web-tests/rs/Cargo.lock
Generated
+207 -138
View File
@@ -452,6 +452,61 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "axum"
version = "0.7.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f"
dependencies = [
"async-trait",
"axum-core",
"bytes",
"futures-util",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"itoa",
"matchit",
"memchr",
"mime",
"percent-encoding",
"pin-project-lite",
"rustversion",
"serde",
"serde_json",
"serde_path_to_error",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tower",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "axum-core"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199"
dependencies = [
"async-trait",
"bytes",
"futures-util",
"http",
"http-body",
"http-body-util",
"mime",
"pin-project-lite",
"rustversion",
"sync_wrapper",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "backtrace"
version = "0.3.76"
@@ -845,6 +900,16 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6"
[[package]]
name = "core-foundation"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation"
version = "0.10.1"
@@ -1769,26 +1834,6 @@ dependencies = [
"uuid",
]
[[package]]
name = "deepsize"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1cdb987ec36f6bf7bfbea3f928b75590b736fc42af8e54d97592481351b2b96c"
dependencies = [
"deepsize_derive",
]
[[package]]
name = "deepsize_derive"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "990101d41f3bc8c1a45641024377ee284ecc338e5ecf3ea0f0e236d897c72796"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]]
name = "deranged"
version = "0.5.5"
@@ -2416,6 +2461,12 @@ version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "httpdate"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
[[package]]
name = "humantime"
version = "2.3.0"
@@ -2436,6 +2487,7 @@ dependencies = [
"http",
"http-body",
"httparse",
"httpdate",
"itoa",
"pin-project-lite",
"pin-utils",
@@ -2480,9 +2532,11 @@ dependencies = [
"percent-encoding",
"pin-project-lite",
"socket2",
"system-configuration",
"tokio",
"tower-service",
"tracing",
"windows-registry",
]
[[package]]
@@ -2938,15 +2992,15 @@ dependencies = [
"half",
"humantime",
"itertools 0.13.0",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"lance-datafusion",
"lance-encoding",
"lance-file",
"lance-index",
"lance-io",
"lance-linalg",
"lance-namespace 8.0.0",
"lance-namespace",
"lance-select",
"lance-table",
"lance-tokenizer",
@@ -2974,28 +3028,6 @@ dependencies = [
"uuid",
]
[[package]]
name = "lance-arrow"
version = "7.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "253f4a0a70580c985b91e65e9ca6cad644825a4078de28d8efbacf3ffbd7ecdc"
dependencies = [
"arrow-array",
"arrow-buffer",
"arrow-data",
"arrow-ipc",
"arrow-ord",
"arrow-schema",
"arrow-select",
"bytes",
"futures",
"getrandom 0.2.16",
"half",
"jsonb",
"num-traits",
"rand 0.9.5",
]
[[package]]
name = "lance-arrow"
version = "8.0.0"
@@ -3056,41 +3088,6 @@ dependencies = [
"seq-macro",
]
[[package]]
name = "lance-core"
version = "7.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13f84020da5a484e2f07dd1796e09785ed7cd889857ebc4cb77e32ef214ee594"
dependencies = [
"arrow-array",
"arrow-buffer",
"arrow-schema",
"async-trait",
"byteorder",
"bytes",
"deepsize",
"futures",
"itertools 0.13.0",
"lance-arrow 7.0.0",
"libc",
"log",
"moka",
"num_cpus",
"object_store",
"pin-project",
"prost",
"rand 0.9.5",
"roaring",
"serde_json",
"snafu 0.9.1",
"tempfile",
"tokio",
"tokio-stream",
"tokio-util",
"tracing",
"url",
]
[[package]]
name = "lance-core"
version = "8.0.0"
@@ -3108,7 +3105,7 @@ dependencies = [
"datafusion-sql",
"futures",
"itertools 0.13.0",
"lance-arrow 8.0.0",
"lance-arrow",
"lance-derive",
"libc",
"libm",
@@ -3152,8 +3149,8 @@ dependencies = [
"datafusion-physical-expr",
"futures",
"jsonb",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"lance-datagen",
"log",
"pin-project",
@@ -3214,9 +3211,9 @@ dependencies = [
"hex",
"hyperloglogplus",
"itertools 0.13.0",
"lance-arrow 8.0.0",
"lance-arrow",
"lance-bitpacking",
"lance-core 8.0.0",
"lance-core",
"log",
"lz4",
"num-traits",
@@ -3248,8 +3245,8 @@ dependencies = [
"bytes",
"datafusion-common",
"futures",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"lance-encoding",
"lance-io",
"log",
@@ -3294,9 +3291,9 @@ dependencies = [
"itertools 0.13.0",
"jieba-rs",
"jsonb",
"lance-arrow 8.0.0",
"lance-arrow",
"lance-arrow-stats",
"lance-core 8.0.0",
"lance-core",
"lance-datafusion",
"lance-datagen",
"lance-encoding",
@@ -3351,9 +3348,9 @@ dependencies = [
"futures",
"http",
"io-uring",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-namespace 8.0.0",
"lance-arrow",
"lance-core",
"lance-namespace",
"log",
"moka",
"object_store",
@@ -3379,26 +3376,12 @@ dependencies = [
"arrow-schema",
"cc",
"half",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"num-traits",
"rand 0.9.5",
]
[[package]]
name = "lance-namespace"
version = "7.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "014e8332ca0615506342e0d3af608639864b68396973be14239f09c9f21f1fc2"
dependencies = [
"arrow",
"async-trait",
"bytes",
"lance-core 7.0.0",
"lance-namespace-reqwest-client 0.7.7",
"snafu 0.9.1",
]
[[package]]
name = "lance-namespace"
version = "8.0.0"
@@ -3408,8 +3391,8 @@ dependencies = [
"arrow",
"async-trait",
"bytes",
"lance-core 8.0.0",
"lance-namespace-reqwest-client 0.8.6",
"lance-core",
"lance-namespace-reqwest-client",
"snafu 0.9.1",
]
@@ -3423,42 +3406,33 @@ dependencies = [
"arrow-ipc",
"arrow-schema",
"async-trait",
"axum",
"bytes",
"datafusion-common",
"datafusion-physical-plan",
"futures",
"lance",
"lance-core 8.0.0",
"lance-core",
"lance-index",
"lance-io",
"lance-linalg",
"lance-namespace 8.0.0",
"lance-namespace",
"lance-table",
"log",
"object_store",
"rand 0.9.5",
"reqwest",
"roaring",
"serde",
"serde_json",
"time",
"tokio",
"tower",
"tower-http 0.5.2",
"url",
"uuid",
]
[[package]]
name = "lance-namespace-reqwest-client"
version = "0.7.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6369eee4682fb11edf538388b43c61ce288b8302fe89bb40944d7daa7faaae99"
dependencies = [
"reqwest",
"serde",
"serde_json",
"serde_repr",
"serde_with",
"url",
]
[[package]]
name = "lance-namespace-reqwest-client"
version = "0.8.6"
@@ -3485,7 +3459,7 @@ dependencies = [
"byteorder",
"bytes",
"itertools 0.13.0",
"lance-core 8.0.0",
"lance-core",
"roaring",
"tracing",
]
@@ -3506,8 +3480,8 @@ dependencies = [
"bytes",
"chrono",
"futures",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"lance-file",
"lance-io",
"lance-select",
@@ -3538,7 +3512,7 @@ dependencies = [
"arrow-array",
"arrow-schema",
"criterion",
"lance-arrow 8.0.0",
"lance-arrow",
"num-traits",
"pprof",
"rand 0.9.5",
@@ -3589,9 +3563,10 @@ dependencies = [
"datafusion-sql",
"futures",
"half",
"http",
"lance",
"lance-arrow 8.0.0",
"lance-core 8.0.0",
"lance-arrow",
"lance-core",
"lance-datafusion",
"lance-datagen",
"lance-encoding",
@@ -3599,7 +3574,7 @@ dependencies = [
"lance-index",
"lance-io",
"lance-linalg",
"lance-namespace 8.0.0",
"lance-namespace",
"lance-namespace-impls",
"lance-table",
"lance-testing",
@@ -3613,6 +3588,7 @@ dependencies = [
"polars-arrow",
"rand 0.9.5",
"regex",
"reqwest",
"semver",
"serde",
"serde_json",
@@ -3621,6 +3597,7 @@ dependencies = [
"tempfile",
"tokio",
"url",
"urlencoding",
"uuid",
]
@@ -3867,6 +3844,12 @@ dependencies = [
"regex-automata",
]
[[package]]
name = "matchit"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
[[package]]
name = "matrixmultiply"
version = "0.3.10"
@@ -5391,7 +5374,7 @@ dependencies = [
"tokio-rustls",
"tokio-util",
"tower",
"tower-http",
"tower-http 0.6.8",
"tower-service",
"url",
"wasm-bindgen",
@@ -5472,7 +5455,7 @@ dependencies = [
"arrow-schema",
"futures",
"futures-util",
"lance-namespace 7.0.0",
"lance-namespace",
"lancedb",
"polars",
"polars-arrow",
@@ -5659,7 +5642,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3297343eaf830f66ede390ea39da1d462b6b0c1b000f420d0a83f898bbbe6ef"
dependencies = [
"bitflags 2.10.0",
"core-foundation",
"core-foundation 0.10.1",
"core-foundation-sys",
"libc",
"security-framework-sys",
@@ -5730,6 +5713,17 @@ dependencies = [
"zmij",
]
[[package]]
name = "serde_path_to_error"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457"
dependencies = [
"itoa",
"serde",
"serde_core",
]
[[package]]
name = "serde_repr"
version = "0.1.20"
@@ -6203,6 +6197,27 @@ dependencies = [
"windows 0.52.0",
]
[[package]]
name = "system-configuration"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3c879d448e9d986b661742763247d3693ed13609438cf3d006f51f5368a5ba6b"
dependencies = [
"bitflags 2.10.0",
"core-foundation 0.9.4",
"system-configuration-sys",
]
[[package]]
name = "system-configuration-sys"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "tagptr"
version = "0.2.0"
@@ -6444,6 +6459,24 @@ dependencies = [
"tokio",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower-http"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5"
dependencies = [
"bitflags 2.10.0",
"bytes",
"http",
"http-body",
"http-body-util",
"pin-project-lite",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@@ -6487,6 +6520,7 @@ version = "0.1.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d15d90a0b5c19378952d479dc858407149d7bb45a14de0142f6c534b16fc647"
dependencies = [
"log",
"pin-project-lite",
"tracing-attributes",
"tracing-core",
@@ -6641,6 +6675,12 @@ dependencies = [
"serde",
]
[[package]]
name = "urlencoding"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
[[package]]
name = "utf8-ranges"
version = "1.0.5"
@@ -6927,8 +6967,8 @@ dependencies = [
"windows-implement",
"windows-interface",
"windows-link 0.1.3",
"windows-result",
"windows-strings",
"windows-result 0.3.4",
"windows-strings 0.4.2",
]
[[package]]
@@ -6986,6 +7026,17 @@ dependencies = [
"windows-link 0.1.3",
]
[[package]]
name = "windows-registry"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
dependencies = [
"windows-link 0.2.1",
"windows-result 0.4.1",
"windows-strings 0.5.1",
]
[[package]]
name = "windows-result"
version = "0.3.4"
@@ -6995,6 +7046,15 @@ dependencies = [
"windows-link 0.1.3",
]
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link 0.2.1",
]
[[package]]
name = "windows-strings"
version = "0.4.2"
@@ -7004,6 +7064,15 @@ dependencies = [
"windows-link 0.1.3",
]
[[package]]
name = "windows-strings"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link 0.2.1",
]
[[package]]
name = "windows-sys"
version = "0.52.0"
docs/web-tests/rs/Cargo.toml
+2 -2
View File
@@ -9,8 +9,8 @@ arrow-json = "58.3.0"
arrow-schema = "58.3.0"
futures = "0.3.31"
futures-util = "0.3.31"
lance-namespace = "7.0.0"
lancedb = { version = "0.31.0", features = ["polars"] }
lance-namespace = "8.0.0"
lancedb = { version = "0.31.0", features = ["polars", "remote"] }
polars = ">=0.37,<0.40.0"
polars-arrow = ">=0.37,<0.40.0"
serde = { version = "1", features = ["derive"] }
docs/web-tests/rs/connection.rs
+66 -31
View File
@@ -3,48 +3,69 @@
use std::path::PathBuf;
use lancedb::connect;
use lancedb::{Connection, connect};
// --8<-- [start:connect]
async fn connect_example(uri: &str) {
let db = connect(uri).execute().await.unwrap();
let _ = db;
async fn connect_local() -> lancedb::Result<Connection> {
// --8<-- [start:connect]
let db = connect("ex_lancedb").execute().await?;
// --8<-- [end:connect]
Ok(db)
}
// --8<-- [end:connect]
#[tokio::main]
async fn main() {
let temp_dir = tempfile::tempdir().unwrap();
let uri = temp_dir.path().join("ex_lancedb");
connect_example(uri.to_str().unwrap()).await;
// Keep the enterprise snippet in this file, but don't run it in CI.
let _ = connect_enterprise_quickstart_config();
let _ = connect_object_storage_config();
}
fn connect_enterprise_quickstart_config() -> (String, String, String, String) {
// --8<-- [start:connect_enterprise_quickstart]
let uri = "db://your-database-uri";
let api_key = "your-api-key";
let region = "us-east-1";
let host_override = "https://your-enterprise-endpoint.com";
// --8<-- [end:connect_enterprise_quickstart]
(
uri.to_string(),
api_key.to_string(),
region.to_string(),
host_override.to_string(),
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let temp_dir = tempfile::tempdir()?;
std::env::set_current_dir(temp_dir.path())?;
let db = connect_local().await?;
db.create_empty_table(
"probe",
std::sync::Arc::new(arrow_schema::Schema::new(vec![arrow_schema::Field::new(
"id",
arrow_schema::DataType::Int64,
false,
)])),
)
.execute()
.await?;
let reopened = connect(temp_dir.path().join("ex_lancedb").to_str().unwrap())
.execute()
.await?;
assert_eq!(reopened.table_names().execute().await?, vec!["probe"]);
// Connects to a deployment only when asked, and only lists its tables.
if std::env::var("LANCEDB_DOCS_TEST_DEPLOYMENT").as_deref() == Ok("1") {
let db = connect_enterprise_quickstart().await?;
println!("{:?}", db.table_names().execute().await?);
}
Ok(())
}
fn connect_object_storage_config() -> &'static str {
async fn connect_enterprise_quickstart() -> Result<Connection, Box<dyn std::error::Error>> {
// --8<-- [start:connect_enterprise_quickstart]
let uri = std::env::var("LANCEDB_URI")?; // db://your-database
let api_key = std::env::var("LANCEDB_API_KEY")?;
let region = std::env::var("LANCEDB_REGION").unwrap_or_else(|_| "us-east-1".to_string());
let host_override = std::env::var("LANCEDB_HOST_OVERRIDE")?; // https://your-endpoint
let db = connect(&uri)
.api_key(&api_key)
.region(&region)
.host_override(&host_override)
.execute()
.await?;
// --8<-- [end:connect_enterprise_quickstart]
Ok(db)
}
// Compiled as documentation, never called by the example runner.
#[allow(dead_code)]
async fn connect_object_storage() -> lancedb::Result<Connection> {
// --8<-- [start:connect_object_storage]
let uri = "s3://your-bucket/path";
// You can also use "gs://your-bucket/path" or "az://your-container/path".
let db = connect(uri).execute().await?;
// --8<-- [end:connect_object_storage]
uri
Ok(db)
}
async fn namespace_table_ops_example(uri: &str) -> lancedb::Result<()> {
@@ -132,3 +153,17 @@ async fn namespace_admin_ops_example() -> lancedb::Result<()> {
fn repo_root() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..")
}
// Compiled as documentation, never called by the example runner.
#[allow(dead_code)]
async fn connect_enterprise_placeholders() -> Result<Connection, Box<dyn std::error::Error>> {
// --8<-- [start:connect_enterprise_placeholders]
let db = connect("db://your-database")
.api_key("your-api-key")
.region("us-east-1")
.host_override("https://your-endpoint")
.execute()
.await?;
// --8<-- [end:connect_enterprise_placeholders]
Ok(db)
}
docs/web-tests/rs/quickstart.rs
+64 -42
View File
@@ -1,23 +1,20 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
use std::sync::Arc;
use arrow_array::types::Float32Type;
use arrow_array::{
FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator, StructArray,
};
use arrow_schema::{DataType, Field, FieldRef, Schema};
use lancedb::arrow::IntoPolars;
use lancedb::database::CreateTableMode;
use lancedb::query::{ExecutableQuery, QueryBase, Select};
use lancedb::{connect, table::NewColumnTransform};
use polars::prelude::DataFrame;
use serde::{Deserialize, Serialize};
// --8<-- [start:quickstart_define_struct]
use std::sync::Arc;
use lancedb::arrow::arrow_array::types::Float32Type;
use lancedb::arrow::arrow_array::{
FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,
RecordBatchReader, StructArray,
};
use lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};
// Define structs representing the data schema
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Debug, Clone)]
struct Stats {
strength: i8,
magic: i8,
@@ -25,7 +22,7 @@ struct Stats {
wisdom: i8,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Debug, Clone)]
struct Character {
id: String,
name: String,
@@ -43,7 +40,7 @@ fn characters_schema() -> Arc<Schema> {
Field::new("description", DataType::LargeUtf8, false),
Field::new(
"stats",
DataType::Struct(arrow_schema::Fields::from(vec![
DataType::Struct(Fields::from(vec![
Arc::new(Field::new("strength", DataType::Int8, false)),
Arc::new(Field::new("magic", DataType::Int8, false)),
Arc::new(Field::new("leadership", DataType::Int8, false)),
@@ -58,11 +55,11 @@ fn characters_schema() -> Arc<Schema> {
),
]))
}
// --8<-- [end:quickstart_define_struct]
type BatchIter = Box<dyn arrow_array::RecordBatchReader + Send>;
fn characters_to_reader(schema: Arc<Schema>, rows: &[Character]) -> BatchIter {
fn characters_to_reader(
schema: Arc<Schema>,
rows: &[Character],
) -> Box<dyn RecordBatchReader + Send> {
let ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.id.as_str()));
let names = LargeStringArray::from_iter_values(rows.iter().map(|row| row.name.as_str()));
let roles = LargeStringArray::from_iter_values(rows.iter().map(|row| row.role.as_str()));
@@ -114,6 +111,7 @@ fn characters_to_reader(schema: Arc<Schema>, rows: &[Character]) -> BatchIter {
schema,
))
}
// --8<-- [end:quickstart_define_struct]
#[tokio::main]
async fn main() {
@@ -166,6 +164,8 @@ async fn main() {
// --8<-- [end:quickstart_data]
// --8<-- [start:quickstart_create_table]
use lancedb::database::CreateTableMode;
let schema = characters_schema();
let table = db
.create_table("characters", characters_to_reader(schema.clone(), &data))
@@ -175,22 +175,37 @@ async fn main() {
.unwrap();
// --8<-- [end:quickstart_create_table]
assert_eq!(table.count_rows(None).await.unwrap(), 3);
db.drop_table("characters", &[]).await.unwrap();
// --8<-- [start:quickstart_create_table_no_overwrite]
let table = db
.create_table("characters", characters_to_reader(schema.clone(), &data))
// --8<-- [start:quickstart_add_data]
let more_data = vec![Character {
id: "4".to_string(),
name: "Morgana".to_string(),
role: "Sorceress".to_string(),
description: "Powerful sorceress of Avalon.".to_string(),
stats: Stats {
strength: 2,
magic: 5,
leadership: 4,
wisdom: 4,
},
vector: [0.3, 0.9, 0.6, 0.8],
}];
table
.add(characters_to_reader(schema.clone(), &more_data))
.execute()
.await
.unwrap();
// --8<-- [end:quickstart_create_table_no_overwrite]
assert_eq!(table.count_rows(None).await.unwrap(), 3);
// --8<-- [end:quickstart_add_data]
assert_eq!(table.count_rows(None).await.unwrap(), 4);
// --8<-- [start:quickstart_vector_search_1]
use lancedb::arrow::IntoPolars;
use lancedb::query::{ExecutableQuery, QueryBase, Select};
// Search for examples similar to a "wise magical advisor"
let query_vector = [0.2, 0.8, 0.4, 0.9];
let result: DataFrame = table
let result = table
.query()
.nearest_to(&query_vector)
.unwrap()
@@ -211,9 +226,10 @@ async fn main() {
// --8<-- [end:quickstart_vector_search_1]
let name_col = result.column("name").unwrap().str().unwrap();
assert_eq!(name_col.get(0).unwrap(), "Merlin");
assert_eq!(name_col.get(1).unwrap(), "Morgana");
// --8<-- [start:quickstart_curate_with_metadata]
let curated: DataFrame = table
let curated = table
.query()
.nearest_to(&query_vector)
.unwrap()
@@ -235,9 +251,10 @@ async fn main() {
// --8<-- [end:quickstart_curate_with_metadata]
let curated_name_col = curated.column("name").unwrap().str().unwrap();
assert_eq!(curated_name_col.get(0).unwrap(), "Merlin");
assert_eq!(curated_name_col.get(1).unwrap(), "Morgana");
// --8<-- [start:quickstart_output_array]
let result: DataFrame = table
let result = table
.query()
.nearest_to(&query_vector)
.unwrap()
@@ -274,7 +291,7 @@ async fn main() {
// --8<-- [end:quickstart_add_feature]
// --8<-- [start:quickstart_query_feature]
let features: DataFrame = table
let features = table
.query()
.select(Select::Columns(vec![
"name".to_string(),
@@ -289,19 +306,23 @@ async fn main() {
.unwrap();
println!("{features:?}");
// --8<-- [end:quickstart_query_feature]
assert!(features.column("power_score").is_ok());
assert_eq!(features.column("power_score").unwrap().len(), 4);
// What the quickstart's download step leaves in an application's directory.
let app = tempfile::tempdir().unwrap();
std::fs::create_dir(app.path().join("data")).unwrap();
std::fs::copy(
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../web/static/assets/images/quickstart/sir-lancelot.jpg"),
app.path().join("data/sir-lancelot.jpg"),
)
.unwrap();
std::env::set_current_dir(app.path()).unwrap();
// --8<-- [start:quickstart_multimodal_bytes]
use std::sync::Arc;
use lancedb::arrow::arrow_array::BinaryArray;
use arrow_array::{
BinaryArray, FixedSizeListArray, LargeStringArray, RecordBatch, RecordBatchIterator,
};
use arrow_schema::{DataType, Field, Schema};
let image_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../web/static/assets/images/quickstart/sir-lancelot.jpg");
let image_bytes = std::fs::read(image_path).unwrap();
let image_bytes = std::fs::read("data/sir-lancelot.jpg").unwrap();
let image_schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::LargeUtf8, false),
@@ -333,9 +354,10 @@ async fn main() {
],
)
.unwrap();
let image_reader: Box<dyn arrow_array::RecordBatchReader + Send> = Box::new(
RecordBatchIterator::new(vec![Ok(image_batch)].into_iter(), image_schema),
);
let image_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(
vec![Ok(image_batch)].into_iter(),
image_schema,
));
let multimodal_table = db
.create_table("character_images", image_reader)
.mode(CreateTableMode::Overwrite)
docs/web-tests/ts/connection.test.ts
+82 -36
View File
@@ -1,65 +1,89 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import { expect, jest, test } from "@jest/globals";
import * as path from "node:path";
import { withTempDirectory } from "./util.ts";
// --8<-- [start:connect]
import { afterEach, describe, expect, jest, test } from "@jest/globals";
import * as lancedb from "@lancedb/lancedb";
async function connectExample(uri: string) {
const db = await lancedb.connect(uri);
return db;
}
// --8<-- [end:connect]
import { withTempDirectory } from "./util.ts";
test("connect to a local database", async () => {
await withTempDirectory(async (tempDir) => {
const uri = path.join(tempDir, "ex_lancedb");
const db = await connectExample(uri);
expect(db).toBeDefined();
const cwd = process.cwd();
process.chdir(tempDir);
try {
// --8<-- [start:connect]
const db = await lancedb.connect("ex_lancedb");
// --8<-- [end:connect]
await db.createTable("probe", [{ id: 1 }]);
expect((await db.listTables()).tables).toEqual(["probe"]);
} finally {
process.chdir(cwd);
}
const reopened = await lancedb.connect(`${tempDir}/ex_lancedb`);
expect((await reopened.listTables()).tables).toEqual(["probe"]);
});
});
async function connectEnterpriseQuickstart() {
// --8<-- [start:connect_enterprise_quickstart]
const uri = "db://your-database-uri";
const apiKey = "your-api-key";
const region = "us-east-1";
const hostOverride = "https://your-enterprise-endpoint.com";
const setting = (name: string) => {
const value = process.env[name];
if (!value) throw new Error(`Set ${name}`);
return value;
};
const db = await lancedb.connect(uri, {
apiKey,
region,
hostOverride,
const db = await lancedb.connect(setting("LANCEDB_URI"), {
apiKey: setting("LANCEDB_API_KEY"),
region: process.env.LANCEDB_REGION ?? "us-east-1",
hostOverride: setting("LANCEDB_HOST_OVERRIDE"),
});
// --8<-- [end:connect_enterprise_quickstart]
return db;
}
test("enterprise quickstart connect uses placeholder config", async () => {
const mockDb = { __mock: true } as unknown as Awaited<
ReturnType<typeof lancedb.connect>
>;
const spy = jest.spyOn(lancedb, "connect").mockResolvedValue(mockDb);
const db = await connectEnterpriseQuickstart();
expect(db).toBe(mockDb);
expect(spy).toHaveBeenCalledWith("db://your-database-uri", {
apiKey: "your-api-key",
region: "us-east-1",
hostOverride: "https://your-enterprise-endpoint.com",
describe("enterprise quickstart connect", () => {
const saved = { ...process.env };
afterEach(() => {
process.env = { ...saved };
});
test("reads the connection from the environment", async () => {
const mockDb = { __mock: true } as unknown as Awaited<
ReturnType<typeof lancedb.connect>
>;
const spy = jest.spyOn(lancedb, "connect").mockResolvedValue(mockDb);
process.env.LANCEDB_URI = "db://your-database";
process.env.LANCEDB_API_KEY = "your-api-key";
delete process.env.LANCEDB_REGION;
process.env.LANCEDB_HOST_OVERRIDE = "https://your-endpoint";
const db = await connectEnterpriseQuickstart();
expect(db).toBe(mockDb);
expect(spy).toHaveBeenCalledWith("db://your-database", {
apiKey: "your-api-key",
region: "us-east-1",
hostOverride: "https://your-endpoint",
});
spy.mockRestore();
});
test("requires an endpoint", async () => {
process.env.LANCEDB_URI = "db://your-database";
process.env.LANCEDB_API_KEY = "your-api-key";
delete process.env.LANCEDB_HOST_OVERRIDE;
await expect(connectEnterpriseQuickstart()).rejects.toThrow(
"Set LANCEDB_HOST_OVERRIDE",
);
});
spy.mockRestore();
});
// --8<-- [start:connect_object_storage]
async function connectObjectStorageExample() {
// --8<-- [start:connect_object_storage]
const uri = "s3://your-bucket/path";
// You can also use "gs://your-bucket/path" or "az://your-container/path".
const db = await lancedb.connect(uri);
// --8<-- [end:connect_object_storage]
return db;
}
// --8<-- [end:connect_object_storage]
async function namespaceTableOpsExample() {
// --8<-- [start:namespace_table_ops]
@@ -119,3 +143,25 @@ void [
namespaceTableOpsExample,
namespaceAdminOpsExample,
];
test("enterprise placeholders use an explicit endpoint", async () => {
const mockDb = {} as Awaited<ReturnType<typeof lancedb.connect>>;
const spy = jest.spyOn(lancedb, "connect").mockResolvedValue(mockDb);
try {
// --8<-- [start:connect_enterprise_placeholders]
const db = await lancedb.connect("db://your-database", {
apiKey: "your-api-key",
region: "us-east-1",
hostOverride: "https://your-endpoint",
});
// --8<-- [end:connect_enterprise_placeholders]
expect(db).toBe(mockDb);
expect(spy).toHaveBeenCalledWith("db://your-database", {
apiKey: "your-api-key",
region: "us-east-1",
hostOverride: "https://your-endpoint",
});
} finally {
spy.mockRestore();
}
});
docs/web-tests/ts/quickstart.test.ts
+241 -154
View File
@@ -1,176 +1,263 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import { expect, test } from "@jest/globals";
import { describe, expect, test } from "@jest/globals";
import { randomUUID } from "node:crypto";
import * as fs from "node:fs";
import * as path from "node:path";
import * as lancedb from "@lancedb/lancedb";
import { withTempDirectory } from "./util.ts";
test("quickstart example (async)", async () => {
// What the quickstart's download step leaves in an application's directory.
const IMAGE = path.join(
__dirname,
"../../web/static/assets/images/quickstart/sir-lancelot.jpg",
);
const deployment = ["LANCEDB_URI", "LANCEDB_HOST_OVERRIDE", "LANCEDB_API_KEY"];
// The connection settings alone never send these tests to a deployment.
const optIn = "LANCEDB_DOCS_TEST_DEPLOYMENT";
const connections =
process.env[optIn] === "1" ? ["local", "enterprise"] : ["local"];
async function tableNames(db: lancedb.Connection) {
const names: string[] = [];
let pageToken: string | undefined;
do {
const page = await db.listTables({ pageToken });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
return names;
}
// A deployment connection on which a test creates only tables of its own:
// each name gets a suffix unique to the test and is claimed only while no
// table has it; the test drops what it claimed, whether or not it passed.
async function withOwnedTables(
db: lancedb.Connection,
fn: (db: lancedb.Connection) => Promise<void>,
) {
const suffix = randomUUID().replaceAll("-", "").slice(0, 12);
const claimed: string[] = [];
const createTable = async (
name: string,
data: Parameters<lancedb.Connection["createTable"]>[1],
options: Partial<lancedb.CreateTableOptions> = {},
) => {
const owned = `${name}_${suffix}`;
let mode = options.mode;
if (!claimed.includes(owned)) {
if ((await tableNames(db)).includes(owned)) {
throw new Error(`${owned} already exists and is not this test's`);
}
claimed.push(owned);
mode = "create";
}
return db.createTable(owned, data, { ...options, mode });
};
try {
await fn(
new Proxy(db, {
get: (target, property, receiver) =>
property === "createTable"
? createTable
: Reflect.get(target, property, receiver),
}),
);
} finally {
const existing = await tableNames(db);
for (const name of claimed) {
if (existing.includes(name)) {
await db.dropTable(name);
}
}
}
}
async function withDatabase(
connection: string,
fn: (db: lancedb.Connection) => Promise<void>,
) {
if (connection === "enterprise") {
const missing = deployment.filter((name) => !process.env[name]);
if (missing.length) {
throw new Error(`${optIn}=1 needs ${missing.join(", ")}`);
}
const db = await lancedb.connect(process.env.LANCEDB_URI!, {
apiKey: process.env.LANCEDB_API_KEY!,
region: process.env.LANCEDB_REGION ?? "us-east-1",
hostOverride: process.env.LANCEDB_HOST_OVERRIDE!,
});
await withOwnedTables(db, fn);
return;
}
await withTempDirectory(async (databaseDir) => {
const db = await lancedb.connect(databaseDir);
await fn(await lancedb.connect(databaseDir));
});
}
// --8<-- [start:quickstart_data]
const data = [
{
id: "1",
name: "King Arthur",
role: "King",
description: "Leader of Camelot and wielder of Excalibur.",
stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 },
vector: [0.7, 0.1, 0.9, 0.7],
},
{
id: "2",
name: "Merlin",
role: "Wizard",
description: "Advisor and prophet with deep magical knowledge.",
stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 },
vector: [0.2, 0.9, 0.4, 0.9],
},
{
id: "3",
name: "Sir Lancelot",
role: "Knight",
description: "Legendary knight known for courage and combat skill.",
stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 },
vector: [0.9, 0.1, 0.5, 0.4],
},
];
// --8<-- [end:quickstart_data]
async function createCharacters(db: lancedb.Connection) {
// --8<-- [start:quickstart_data]
const data = [
{
id: "1",
name: "King Arthur",
role: "King",
description: "Leader of Camelot and wielder of Excalibur.",
stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 },
vector: [0.7, 0.1, 0.9, 0.7],
},
{
id: "2",
name: "Merlin",
role: "Wizard",
description: "Advisor and prophet with deep magical knowledge.",
stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 },
vector: [0.2, 0.9, 0.4, 0.9],
},
{
id: "3",
name: "Sir Lancelot",
role: "Knight",
description: "Legendary knight known for courage and combat skill.",
stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 },
vector: [0.9, 0.1, 0.5, 0.4],
},
];
// --8<-- [end:quickstart_data]
// --8<-- [start:quickstart_create_table]
let table = await db.createTable("characters", data, { mode: "overwrite" });
// --8<-- [end:quickstart_create_table]
expect(await table.countRows()).toBe(3);
await db.dropTable("characters");
// --8<-- [start:quickstart_create_table]
const table = await db.createTable("characters", data, { mode: "overwrite" });
// --8<-- [end:quickstart_create_table]
expect(await table.countRows()).toBe(3);
// --8<-- [start:quickstart_create_table_no_overwrite]
table = await db.createTable("characters", data);
// --8<-- [end:quickstart_create_table_no_overwrite]
expect(await table.countRows()).toBe(3);
// --8<-- [start:quickstart_add_data]
await table.add([
{
id: "4",
name: "Morgana",
role: "Sorceress",
description: "Powerful sorceress of Avalon.",
stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },
vector: [0.3, 0.9, 0.6, 0.8],
},
]);
// --8<-- [end:quickstart_add_data]
expect(await table.countRows()).toBe(4);
return table;
}
// --8<-- [start:quickstart_vector_search_1]
// Search for examples similar to a "wise magical advisor"
let queryVector = [0.2, 0.8, 0.4, 0.9];
describe.each(connections)("quickstart (%s)", (connection) => {
test("create, add and search", async () => {
await withDatabase(connection, async (db) => {
const table = await createCharacters(db);
let result = await table
.search(queryVector)
.select(["name", "role", "description", "_distance"])
.limit(2)
.toArray();
console.table(result);
// --8<-- [end:quickstart_vector_search_1]
expect(result[0].name).toBe("Merlin");
// --8<-- [start:quickstart_vector_search_1]
// Search for examples similar to a "wise magical advisor"
const queryVector = [0.2, 0.8, 0.4, 0.9];
// --8<-- [start:quickstart_curate_with_metadata]
const curated = await table
.search(queryVector)
.where("stats.magic >= 4")
.select(["name", "role", "description", "_distance"])
.limit(2)
.toArray();
console.table(curated);
// --8<-- [end:quickstart_curate_with_metadata]
expect(curated[0].name).toBe("Merlin");
const result = await table
.search(queryVector)
.select(["name", "role", "description", "_distance"])
.limit(2)
.toArray();
console.table(result);
// --8<-- [end:quickstart_vector_search_1]
expect(result.map((row) => row.name)).toEqual(["Merlin", "Morgana"]);
// --8<-- [start:quickstart_output_array]
result = await table.search(queryVector).limit(2).toArray();
console.table(result);
// --8<-- [end:quickstart_output_array]
expect(result[0].name).toBe("Merlin");
// --8<-- [start:quickstart_curate_with_metadata]
const curated = await table
.search(queryVector)
.where("stats.magic >= 4")
.select(["name", "role", "description", "_distance"])
.limit(2)
.toArray();
console.table(curated);
// --8<-- [end:quickstart_curate_with_metadata]
expect(curated.map((row) => row.name)).toEqual(["Merlin", "Morgana"]);
});
});
// --8<-- [start:quickstart_add_feature]
await table.addColumns([
{
name: "power_score",
valueSql:
"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)",
},
]);
// --8<-- [end:quickstart_add_feature]
const schemaWithFeature = await table.schema();
expect(schemaWithFeature.fields.some((f) => f.name === "power_score")).toBe(
true,
);
test("a derived feature", async () => {
await withDatabase(connection, async (db) => {
const table = await createCharacters(db);
// --8<-- [start:quickstart_query_feature]
const features = await table
.query()
.select(["name", "role", "power_score"])
.toArray();
console.table(features);
// --8<-- [end:quickstart_query_feature]
expect(features[0]).toHaveProperty("power_score");
// --8<-- [start:quickstart_multimodal_bytes]
const arrow = await import("apache-arrow");
const path = await import("node:path");
const { readFile } = await import("node:fs/promises");
const imagePath = path.join(
__dirname,
"../../web/static/assets/images/quickstart/sir-lancelot.jpg",
);
const imageBytes = await readFile(imagePath);
const imageSchema = new arrow.Schema([
new arrow.Field("id", new arrow.Utf8()),
new arrow.Field("description", new arrow.Utf8()),
new arrow.Field("image", new arrow.Binary()),
new arrow.Field(
"vector",
new arrow.FixedSizeList(
4,
new arrow.Field("item", new arrow.Float32(), true),
),
),
]);
const imageData = lancedb.makeArrowTable(
[
// --8<-- [start:quickstart_add_feature]
await table.addColumns([
{
id: "lancelot",
description: "Portrait of Sir Lancelot",
image: imageBytes,
vector: [0.9, 0.1, 0.5, 0.4],
name: "power_score",
valueSql:
"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)",
},
],
{ schema: imageSchema },
);
const multimodalTable = await db.createTable(
"character_images",
imageData,
{ mode: "overwrite" },
);
// --8<-- [end:quickstart_multimodal_bytes]
expect(await multimodalTable.countRows()).toBe(1);
]);
// --8<-- [end:quickstart_add_feature]
// --8<-- [start:quickstart_open_table]
table = await db.openTable("characters");
// --8<-- [end:quickstart_open_table]
// --8<-- [start:quickstart_query_feature]
const features = await table
.query()
.select(["name", "role", "power_score"])
.toArray();
console.table(features);
// --8<-- [end:quickstart_query_feature]
expect(
features.map((row) => [row.name, row.power_score]).sort(),
).toEqual([
["King Arthur", 3.5],
["Merlin", 4],
["Morgana", 3.75],
["Sir Lancelot", 3],
]);
});
});
// --8<-- [start:quickstart_add_data]
const moreData = [
{
id: "4",
name: "Morgana",
role: "Sorceress",
description: "Powerful sorceress of Avalon.",
stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },
vector: [0.3, 0.9, 0.6, 0.8],
power_score: 3.75,
},
];
test("multimodal data", async () => {
await withDatabase(connection, async (db) => {
await withTempDirectory(async (app) => {
fs.mkdirSync(path.join(app, "data"));
fs.copyFileSync(IMAGE, path.join(app, "data/sir-lancelot.jpg"));
const cwd = process.cwd();
process.chdir(app);
try {
// --8<-- [start:quickstart_multimodal_bytes]
const arrow = await import("apache-arrow");
const { readFile } = await import("node:fs/promises");
// Add data to table
await table.add(moreData);
// --8<-- [end:quickstart_add_data]
expect(await table.countRows()).toBe(4);
// --8<-- [start:quickstart_vector_search_2]
// Search for examples similar to a "powerful sorceress"
queryVector = [0.3, 0.9, 0.6, 0.8];
const results = await table.search(queryVector).limit(2).toArray();
console.table(results);
// --8<-- [end:quickstart_vector_search_2]
expect(results[0].name).toBe("Morgana");
const imageBytes = await readFile("data/sir-lancelot.jpg");
const imageSchema = new arrow.Schema([
new arrow.Field("id", new arrow.Utf8()),
new arrow.Field("description", new arrow.Utf8()),
new arrow.Field("image", new arrow.Binary()),
new arrow.Field(
"vector",
new arrow.FixedSizeList(
4,
new arrow.Field("item", new arrow.Float32(), true),
),
),
]);
const imageData = lancedb.makeArrowTable(
[
{
id: "lancelot",
description: "Portrait of Sir Lancelot",
image: imageBytes,
vector: [0.9, 0.1, 0.5, 0.4],
},
],
{ schema: imageSchema },
);
const multimodalTable = await db.createTable(
"character_images",
imageData,
{ mode: "overwrite" },
);
// --8<-- [end:quickstart_multimodal_bytes]
expect(await multimodalTable.countRows()).toBe(1);
const [row] = await multimodalTable.query().select(["image"]).toArray();
expect(Buffer.from(row.image)).toEqual(fs.readFileSync(IMAGE));
} finally {
process.chdir(cwd);
}
});
});
});
});
docs/web/architecture.mdx
+85 -114
View File
@@ -1,153 +1,124 @@
---
title: "Architecture"
sidebarTitle: "Architecture"
description: "How LanceDB is put together, and what changes between running it embedded and running it distributed."
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with embedded LanceDB OSS."
icon: cubes
keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
---
LanceDB has one storage layer and two ways to put compute on top of it. Which
one you run decides where queries execute, what runs on your behalf, and what
you have to operate — and almost nothing else. The data is the same
[Lance](/basics/lance) tables either way.
LanceDB OSS is an embedded library for working with Lance tables. LanceDB
Enterprise is a multimodal lakehouse for data curation, feature engineering,
search, and training data access. Its architecture separates query serving,
read execution, caching, and background work. Both use the open
[Lance](/basics/lance) format.
## One storage layer {#storage}
Teams can curate datasets, derive features, search multimodal records, and
read the resulting tables into training frameworks. Enterprise separates the
resources that serve shared data from those that build indexes and run
background work, so teams can size them for these different workloads.
Everything durable lives in object storage: table data, manifests, indexes. No
machine owns a dataset, and no server has to be running for the data to exist.
![LanceDB OSS runs the application and embedded library in one process, using local disk or object storage. Enterprise applications connect to query serving, with separate read execution and caching, indexing and background workers, and durable object storage. Both use the Lance format, but they do not share a storage location automatically.](/static/assets/images/overview/lancedb-architecture.svg)
```
┌──────────────────────────────────────────┐
│ object storage │
│ table data · manifests · indexes │
└──────────────────────────────────────────┘
```
The diagram shows the main compute and storage boundaries, not every request
path. The control plane governs the Enterprise service.
That is the whole persistence story, and it is why the two deployment shapes can
differ so much above it while staying compatible below it. A table written
embedded is opened by a deployment, and the reverse, without conversion.
## A shared format, different storage choices {#storage}
## Embedded {#embedded}
Lance stores table data, versions, and indexes. OSS can read and write tables on
local disk or in object storage. Enterprise keeps durable table data and index
artifacts in object storage, independently of the machines serving requests.
Its caches accelerate reads; they are not the durable source of truth.
```
your process
┌────────────────────────────────┐
│ application │
│ LanceDB library │ ← plans, executes, indexes, compacts
└────────────────────────────────┘
│
┌─────────▼─────────┐
│ object storage │
└───────────────────┘
```
Sharing a format does not mean that an Enterprise deployment automatically
opens an existing OSS database. Plan how to move or connect your data and
configure access for your deployment.
`lancedb.connect("./data")` or `connect("s3://bucket/path")` gives you a
**library**, not a client. Your process opens the tables, plans the queries,
runs the scans, builds the indexes, and writes the new versions. There is no
server in the picture at all.
## OSS: an embedded library inside your application {#embedded}
**What follows from that.** Nothing runs unless you call it — no background
compaction, no server-side [jobs](/basics/jobs), no
[SQL endpoint](/basics/sql). Scale is your process's scale: a query gets the
cores and memory your process has. Access control is whatever your object store
enforces, since there is no other gatekeeper.
With an embedded connection, the LanceDB library plans and executes queries,
builds indexes, and updates tables inside your process. There is no separate
database server to deploy.
**What it is good at.** Development, notebooks, single-node pipelines, embedding
LanceDB inside an application, and anything where "no service to operate" is
worth more than elasticity. Comfortably handles millions of vectors on one
node -- disk-based indexes keep it performant without extra infrastructure.
This fits local retrieval applications, notebooks, development, and
single-node data pipelines. Your application controls the compute resources and
when to run [table maintenance](/maintenance/compaction). You also manage access
to the local files or object storage that hold your tables.
## Distributed {#distributed}
## Enterprise: a multimodal lakehouse for shared workloads {#distributed}
<Badge color="red">Enterprise</Badge>
Applications connect to remote tables through a LanceDB SDK. The service handles
requests across a fleet, with components sized for different kinds of work:
```
your process
┌──────────────┐
│ application │
│ LanceDB SDK │ ← a thin client
└──────┬───────┘
│ db:// (REST / Flight SQL)
┌──────▼──────────────────────────────────────┐
│ control plane config · identity · policy │
├─────────────────────────────────────────────┤
│ data plane query serving · writes │
│ indexers indexing · compaction │
└──────┬──────────────────────────────────────┘
│
┌──────▼────────────┐
│ object storage │
└───────────────────┘
```
- **Query serving** receives requests, resolves tables, and plans operations.
Add serving capacity as more applications and users depend on your data.
- **Read execution and caching** use separately provisioned resources. Executors
cache table data to reduce repeated reads from object storage.
- **Indexing and background workers** build indexes and perform maintenance
outside the query request path. Size this work separately from query serving
as your data grows.
`connect("db://...")` gives you a **client**. Planning and execution move to a
cluster, and the work splits across tiers that scale independently: a control
plane governing configuration, identity and policy; a data plane serving reads
and writes; and indexers building indexes and compacting outside the request
path.
The control plane manages configuration, identity, policy, and cluster
lifecycle. It governs the service rather than executing table queries.
[Authorization](/governance/authorization) lets teams control access to catalog
objects when enabled and configured by the administrator. Deployment settings govern background
[compaction](/maintenance/compaction) and
[cleanup](/maintenance/cleanup), including retention.
**What follows from that.** Compute and storage are separated, so a query fleet
scales for interactive traffic without also scaling indexing capacity, and heavy
background work does not steal resources from user queries. Because a server
exists, so do the things that need one: [SQL](/sql),
[functions](/management/function), server-side jobs, and
[authorization](/governance/authorization) per object rather than per bucket.
These boundaries support different parts of an AI data workflow.
[SQL curation](/curation) requires a deployment with SQL enabled.
[Feature engineering](/feature-engineering) can produce derived columns and
tables; computed columns and materialized views require a deployment that
enables them. [Geneva](/geneva) runs distributed Python feature backfills while
its capabilities continue to transition to LanceDB APIs.
[Training data loaders](/training/data-loading) read tables into model training
frameworks in both offerings. LanceDB provides data access for training; the
training framework runs the model.
**What it is good at.** Many concurrent queries, continuous ingestion and
indexing, multi-team access with real permissions, and workloads too large for
one machine -- customers regularly operate on billions of rows and tens of
terabytes.
Size and monitor each tier for your workload. Read the
[Enterprise architecture](/enterprise/architecture) for the component details
and request flows.
## What changes, precisely {#differences}
## Familiar APIs, a different operating model {#differences}
| | Embedded | Distributed |
| --- | --- | --- |
| What you connect to | a library in your process | a cluster |
| Where queries plan and execute | your process | the data plane |
| Table handle | `LanceTable` | `RemoteTable` |
| Indexing and compaction | when you call it | dedicated indexers |
| [Jobs](/basics/jobs) | run in-process, no server record | real jobs: id, status, events, cancel |
| [SQL](/sql) | unavailable | Flight SQL endpoint |
| [Functions](/management/function) | unavailable | run on workers |
| Access control | object-store permissions | [per-object grants](/governance/authorization) |
| Scale bound | your process | the cluster |
| What you operate | nothing | a deployment |
Core operations remain familiar: create a table, add data, and run a search.
With OSS, the application executes those operations. With Enterprise, the SDK
sends requests to the service, and the service manages execution and configured
background maintenance.
Both expose the same table API — create, open, mutate, query — so application
code moves between them by changing the URI. What does not move is code that
depends on a row in the right-hand column.
Moving an application involves more than changing a connection string. Plan
for data migration, connection settings, credentials, and SDK methods that
behave differently on remote tables. See
[Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise) before adapting
an existing application. If you plan to use materialized views, check the
[stable row ID requirements](/basics/limits#schema) before creating source tables.
## Choosing, and changing your mind {#choosing}
## Choose for the workload you need to run {#choosing}
Start embedded. It has no operational cost and the API is the same one you will
keep using.
Choose OSS when you want an embedded library and can manage its resources and
maintenance in your application.
Move when you hit something in the right-hand column — usually concurrent
serving, ingestion that outgrows one machine, or a permissions requirement.
Because the storage layer is identical, moving is a connection-string change
plus a data migration, not a rewrite.
Choose Enterprise when multiple applications or teams need a shared multimodal
lakehouse for curation, feature engineering, search, and training data access.
It also fits workloads whose query and indexing demand grow at different rates,
or that need configured service-level access controls and background operations.
You can begin with Enterprise.
<Note>
The one thing to decide early is **stable row ids**: the SDK can give them to a
table only when it creates one on a local path or an object store, and a
deployment may require them on a
[materialized view](/management/materialized-view) source. See
[Limits](/basics/limits#schema).
</Note>
[Talk to us about your workload](https://www.lancedb.com/contact) to plan capacity,
deployment options, and the path from your existing data to Enterprise.
## Where to go next {#next}
<CardGroup cols={2}>
<Card title="Enterprise architecture" icon="cubes" href="/enterprise/architecture">
Query serving, execution, and background work in detail.
</Card>
<Card title="Connect to Enterprise" icon="plug" href="/basics/connect#enterprise">
Connect an application to your deployment.
</Card>
<Card title="Guarantees" icon="shield-check" href="/basics/guarantees">
What a write promises, and what a reader sees.
</Card>
<Card title="Lance format" icon="layer-group" href="/basics/lance">
The open format underneath both shapes.
</Card>
<Card title="Enterprise architecture" icon="cubes" href="/enterprise/architecture">
The distributed tiers in detail.
</Card>
<Card title="Limits" icon="ruler" href="/basics/limits">
The bounds you can actually hit.
The open format underneath both offerings.
</Card>
</CardGroup>
docs/web/basics/agentic-skills.mdx
+5 -4
View File
@@ -269,10 +269,11 @@ where appropriate.
Choose [LanceDB Enterprise](/enterprise) when the resulting table becomes
shared production infrastructure, and the workload needs distributed capacity,
private deployment, or platform-managed operations. The underlying data format and table
API stay the same, so the pipeline does not need to be redesigned. The agent
instead connects to a remote `db://` table and lets the cluster handle
maintenance and background work.
private deployment, or platform-managed operations. The data format and the core
table API stay the same, so the pipeline's table and search code carries over:
the agent connects to a remote `db://` table, leaves maintenance to the
deployment instead of calling `optimize()`, and submits long-running work as
jobs. See [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
## What agents are good at here {#workflows}
docs/web/basics/catalog.mdx
+46 -91
View File
@@ -1,107 +1,62 @@
---
title: "Lance Catalog"
sidebarTitle: "Lance Catalog"
description: "The naming layer: what objects exist, how they nest, and who may reach them."
description: "Organize, find and control access to your data in LanceDB."
icon: sitemap
keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view", "function"]
keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"]
---
A catalog answers one question: **what exists, and what is it called.** It holds
names, the hierarchy they sit in, and the rules about who may reach them. It
does not store rows, run queries, or build indexes -- that is the
[lakehouse](/basics/lakehouse) on top of it.
## What is a catalog? {#catalog}
Keeping the two apart is what lets a Lance table be reached by something that is
not LanceDB. The catalog is a specification -- the
[Lance Namespace](/basics/lance#standard) -- so any engine that implements it
resolves the same names to the same tables.
A catalog keeps track of named data objects, such as tables and views, so
applications can discover and access them. In LanceDB, it organizes those
objects into databases and namespaces.
## The objects {#objects}
## How is data organized? {#objects}
| Object | What it is |
A database contains namespaces that group related tables and views. Namespaces
can also contain other namespaces, allowing teams to organize data by project,
environment or purpose.
For example, a search application can keep production and staging data in
separate namespaces within the same database:
```mermaid
flowchart TD
database["ai_data · database"] --> prod["prod · namespace"]
database --> staging["staging · namespace"]
prod --> prod_docs["documents · table"]
staging --> staging_docs["documents · table"]
```
<span id="views" />
| Object | What it represents |
| --- | --- |
| [Database](/management/database) | The root. A connection opens exactly one. |
| [Namespace](/management/namespace) | A named grouping inside a database. Can nest. |
| [Table](/management/table) | Rows and columns, versioned. What most operations act on. |
| [Materialized view](/management/materialized-view) | A table whose contents come from a query, computed and stored. |
| [View](/management/view) | A saved query. Computed when read, not stored. |
| [Function](/management/function) | Registered code, applied to produce column values. |
| [Database](/management/database) | A container for namespaces and their data objects. |
| [Namespace](/management/namespace) | A named group of tables, views or nested namespaces. |
| [Table](/management/table) | Data stored as rows and columns. |
| [View](/management/view) | A saved query, evaluated when read. Available in Enterprise. |
| [Materialized view](/management/materialized-view) | The stored results of a query, updated by a refresh. |
| [Function](/management/function) {/* if functions launched */} | Registered code used to compute column values. Available in Enterprise. |
```
database
└── namespace
├── table
├── materialized view (a table, defined by a query)
├── view (a query, not stored)
└── function
```
<span id="resolution" />
A namespace can contain further namespaces, so the hierarchy goes as deep as
your organisation needs. A table is reached by its path through it:
`prod.events.clicks`.
## Why does the organization matter? {#governance}
This page is about what the objects *are*. For the operations on them, see
[Management](/management).
- **Separate projects and environments.** Keep production and staging tables
in distinct namespaces, even when they use the same table names.
- **Find shared datasets.** Give teams a consistent place to discover and
access the data they work with.
- **Control access in Enterprise.** Use permissions on databases, namespaces
and tables to manage who can access shared data and what they can do with it.
## Resolution {#resolution}
<span id="elsewhere" />
A name is resolved by walking the path, and the walk is the point: reaching a
table means traversing the namespaces above it. That is what makes the hierarchy
more than decoration -- it is the unit of both organisation and access.
## Learn more {#next}
A flat set of tables needs none of this. Namespaces start earning their keep when
several teams, environments or domains share one catalog and you want
`prod.search.documents` and `staging.search.documents` to be unambiguous.
## Governance {#governance}
<Badge color="red">Enterprise</Badge>
Because the catalog knows the hierarchy, it is the layer that can enforce
access. A grant names **one privilege** on **one object** for **one subject**,
and it inherits down the path: reaching a table requires `USAGE` on the
namespace containing it and on the database containing that.
That is the practical reason to put a boundary in the hierarchy -- a namespace
per team gives you something to grant on. See
[Authorization](/governance/authorization).
Embedded LanceDB has no such layer. Access is whatever the object store enforces
on the bucket, which has no concept of a table.
## Views and materialized views are different objects {#views}
The names are close enough to mislead. Both are catalog objects defined by a
query over other tables; the difference is when that query runs.
- A **materialized view** stores its rows, and a refresh brings them up to
date. Reads are reads of stored data -- fast, and as current as the last
refresh.
- A **view** stores only the query. Every read runs it, so a view is never stale
and never faster than the query it wraps.
Reach for a materialized view when the query is expensive and the data changes
less often than it is read. Reach for a view when correctness matters more than
latency, or when the query is cheap.
## What the catalog does not hold {#elsewhere}
**Indexes** belong to a table rather than standing alone. Creating one is a
performance decision, not a structural one -- see [Indexing](/indexing).
**Jobs** are not stored data at all; they are handles to work in progress. See
[Jobs](/basics/jobs).
**Rows** are in the [Lance](/basics/lance) files the catalog points at. The
catalog knows a table exists and where it lives; it does not know what is in it.
## Next {#next}
<CardGroup cols={2}>
<Card title="Lance Lakehouse" icon="layer-group" href="/basics/lakehouse">
What operates on these objects: search, compute, maintenance.
</Card>
<Card title="Management" icon="folder-tree" href="/management">
Creating, listing and dropping each one.
</Card>
</CardGroup>
- [Databases](/management/database): connect to and manage a database.
- [Namespaces](/management/namespace): create groups and address tables within them.
- [Tables](/management/table): create, inspect and manage your data.
- [Views](/management/view) and [materialized views](/management/materialized-view): work with query-defined data.
- [Authorization](/governance/authorization): configure Enterprise access permissions.
docs/web/basics/connect.mdx
+132
View File
@@ -0,0 +1,132 @@
---
title: "Connect to LanceDB"
sidebarTitle: "Connect"
description: "Use the embedded LanceDB OSS library with tables in a local directory or object storage, or connect to LanceDB Enterprise."
icon: plug
keywords: ["connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"]
---
import {
PyConnect,
PyConnectAsync,
PyConnectEnterprisePlaceholders,
PyConnectEnterpriseQuickstart,
PyConnectObjectStorage,
PyConnectObjectStorageAsync,
RsConnect,
TsConnect,
TsConnectEnterprisePlaceholders,
TsConnectEnterpriseQuickstart,
TsConnectObjectStorage,
} from '/snippets/connection.mdx';
The URI you pass to `connect` decides where the work runs. The same Python,
TypeScript and Rust SDKs open all three kinds;
[OSS and Enterprise](/basics/offerings) describes how the two offerings differ.
| URI | Where queries run |
| --- | --- |
| A local path, such as `./data` | OSS: in your process |
| `s3://`, `gs://` or `az://` | OSS: in your process, reading object storage |
| `db://` | LanceDB Enterprise: queries run on your cluster |
## A local directory {#local}
Embedded in your application, LanceDB reads and writes the tables in a
directory on disk.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnect}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::connect;\n" }
{ "\n" }
{RsConnect}
</CodeBlock>
</CodeGroup>
## Object storage {#object-storage}
With an object storage URI the library is still embedded: your process runs
every query and reads and writes the bucket directly, with the credentials from
its environment or its storage options. [Configuring storage](/deployment/storage)
covers credentials, endpoints and provider options.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectObjectStorageAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectObjectStorage}
</CodeBlock>
</CodeGroup>
In Rust, pass the same URI to `connect`.
## LanceDB Enterprise {#enterprise}
A `db://` URI connects to LanceDB Enterprise, where your cluster runs queries
and background work. Get the database URI, API key, region and endpoint from
your administrator, then replace the placeholders below. Use your cluster's region
in place of `us-east-1`. In application code, load credentials from environment
variables or a secrets manager.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterprisePlaceholders}
</CodeBlock>
</CodeGroup>
In Rust, enable the crate's `remote` feature; the connection builder then takes
the same settings through `api_key`, `region` and `host_override`. To use OAuth
instead of an API key, see
[Authentication](/enterprise/authentication); for every connection option, see
[Database](/management/database#deployment).
<Accordion title="Load production credentials from your environment">
Set these values in the shell that starts your application. Replace the URI, API
key, region and endpoint with your Enterprise connection settings. In production, a secrets
manager can supply these environment variables for you.
```shell
export LANCEDB_URI="db://your-database"
export LANCEDB_API_KEY="your-api-key"
export LANCEDB_REGION="us-east-1"
export LANCEDB_HOST_OVERRIDE="https://your-endpoint"
```
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterpriseQuickstart}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterpriseQuickstart}
</CodeBlock>
</CodeGroup>
</Accordion>
docs/web/basics/guarantees.mdx
+53 -106
View File
@@ -1,134 +1,81 @@
---
title: "Guarantees"
sidebarTitle: "Guarantees"
description: "What a write promises when it returns, what a reader sees, and what survives a failure."
description: "Understand table commits, concurrent writes and the boundaries of recovery."
icon: shield-check
keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency", "acid"]
keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency"]
---
What the system promises, and — just as usefully — what it does not. Most
surprises in practice come from assuming one of the guarantees below is
stronger or weaker than it is.
## What commits together? {#writes}
## Writes {#writes}
A table transaction commits its changes atomically: readers see the committed
table version, not a partially applied transaction. For example, a committed
append adds its rows together.
**A write is atomic.** Every write commits a new table version. Either the
version exists and holds all of the change, or it does not exist and holds none
of it. There is no state in which a reader sees half an `add()`.
This boundary is one table transaction. A workflow that updates several tables
or runs a background job can contain multiple commits. Do not assume the whole
workflow is atomic.
**A write returns only after the commit.** When the call returns, the version
is durable in object storage. Nothing is buffered in your process waiting to be
flushed.
<span id="schema" />
**Every write advances the version**, including one that affects zero rows. A
delete whose predicate matches nothing still commits. Version numbers count
commits, not changes.
Schema changes are also part of table history. Changing a column's type can
rewrite data; adding a derived column can write the new column independently.
**Concurrent writes are resolved optimistically.** A transaction records the
change and the version it was based on. At commit time, a conflicting
concurrent operation is detected rather than silently overwritten, and the
commit is retried against the newer version. Operations that do not collide —
appending while another process appends — both succeed.
## What does a read observe? {#reads}
<Note>
`merge_insert` reports its retry attempts in the result. A high count means
contention on the same keys, not an error.
</Note>
A read of a table uses a consistent snapshot. A consistent snapshot can still
be older than another writer's latest commit. [Consistency](/tables/consistency)
explains how OSS and Enterprise choose the version a reader sees.
## Reads {#reads}
For an experiment that must keep using the same data, select a specific retained
table version rather than relying on a connection's default freshness.
**A read sees a consistent snapshot.** A query reads one table version
throughout; it never sees a mixture of two.
## What happens with concurrent writes? {#concurrency}
**Which version, by default, is the one open at the time the table was opened.**
This is the guarantee most often assumed to be stronger than it is:
Lance uses optimistic concurrency control. Writers prepare changes, then check
compatibility with intervening commits. Compatible changes can be committed;
conflicts may be retried or returned as errors.
<Warning>
A long-lived connection **does not observe other processes' writes** until it
reopens the table or its `read_consistency_interval` elapses. The default is
right for a batch job reading a stable snapshot and wrong for a service
expected to see fresh data.
</Warning>
Applications must handle failed writes. Concurrent appends are compatible, but
that does not promise success through storage failures or exhausted retries.
Do not rely on a particular ordering between independent writers.
Set `read_consistency_interval` when several processes write the same table and
readers must observe those writes. The cost is a metadata check on that
interval. See [Consistency](/tables/consistency).
The [Lance transaction specification](https://lance.org/format/table/transaction/)
describes commit behavior and compatibility between operations.
**Your own writes are always visible to you.** The interval governs other
writers, not yourself.
<span id="jobs" />
## Durability and history {#durability}
## What survives completion or failure? {#durability}
**Old versions are retained, not overwritten.** A new version does not destroy
the one before it, which is what makes [time travel](/management/table#versioning) work
and what lets a reader finish a scan that started on an older version.
A successful commit records the change in the table's storage. With OSS, that
storage can be local disk or an object store; Enterprise uses its configured
storage infrastructure. Durability depends on that backend and its configuration.
**Deletion is a mark, not an erase.** A deleted row is recorded in a deletion
file. The data is recoverable for as long as the version that contains it
exists.
A job being accepted is not the same as its work completing. Wait for its
successful outcome before depending on the result. Some background operations
commit batches as they progress, so failure or cancellation can leave completed
work in place. See the operation's guide for its commit behavior.
<Warning>
Retention is finite. [Compaction](/maintenance/compaction) prunes versions
older than the retention window — **7 days by default** — and a tag or branch
pointing at a pruned version no longer resolves. If you rely on reading old
versions, widen the window before running it, not after.
</Warning>
A client timeout can leave the outcome unknown. Check the job or resulting data
before repeating work that could create duplicates.
**Dropping a table ends its history.** Versioning does not survive
`drop_table`; there is no undrop.
## What can history recover? {#history}
## Jobs {#jobs}
Retained table versions let you inspect earlier data and restore a prior state.
Cleanup can remove old versions and the files they need. Set retention for your
recovery and reproducibility requirements before cleanup runs.
**A refresh recomputes only what changed.** It gives values to rows that lack
them and recomputes rows whose inputs changed since they were computed; it
leaves the rest alone. Two consequences, both relied on elsewhere:
Dropping a table is different from deleting rows: table versioning does not
provide an undo for `drop_table`. Keep backups appropriate to your storage and
recovery requirements.
- **Retrying is safe.** Re-running a refresh fills what is still empty or stale.
It is not guaranteed to cost only the rows that did not complete: a retry can
recompute values that were already written.
- **A filled value catches up at the next refresh.** If a column derives from
`price` and a row's `price` later changes, the derived value keeps its old
value until the next refresh recomputes it.
<span id="not-guaranteed" />
**What a failure keeps depends on where the refresh ran.** `refresh_column` in
your process commits the column once, so a failure keeps none of its values. A
refresh job on a deployment commits finished batches as it goes, so some values
can remain.
## Learn more {#next}
**Cancellation is a request.** `cancel()` asks a job to stop; it does not
guarantee it stops before doing more work, and cancelling a finished job does
nothing.
**A job id is opaque.** It correlates with server logs and the jobs API.
Parsing it, or storing it to resume the job later, is not supported.
See [Jobs](/basics/jobs).
## Schema {#schema}
**Schema changes are versioned like data changes.** Adding, renaming, retyping
or dropping a column commits a version, so a schema change can be travelled back
through.
**A computed column's inputs are protected.** While a declaration references a
column, that column cannot be renamed, retyped, or dropped — the declaration is
part of the schema and will not be invalidated out from under itself.
**Most schema changes do not rewrite data.** Adding a derived column writes only
the new column. Type changes are the exception and can be substantial.
## What is not guaranteed {#not-guaranteed}
Worth stating plainly, because each is occasionally assumed:
| | |
| --- | --- |
| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. |
| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). |
| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. |
The last two are the most common source of "why is this query slow" — see
[Limits](/basics/limits) for the bounds that pair with them.
- [Consistency](/tables/consistency): control when reads observe updates.
- [Table versioning](/management/table#versioning): inspect and restore history.
- [Cleanup](/maintenance/cleanup): configure retention and understand protected versions.
- [Jobs](/basics/jobs): distinguish submission, completion and cancellation.
- [Column refresh](/feature-engineering/refresh-column): understand partial progress.
- [Architecture](/architecture): understand storage and background maintenance.
docs/web/basics/jobs.mdx
+59 -181
View File
@@ -1,196 +1,74 @@
---
title: "Jobs"
sidebarTitle: "Job"
description: "Work that outlives the call that started it: submitting, waiting, inspecting, cancelling."
description: "Track background work from submission to completion."
icon: hourglass-half
keywords: ["job", "async", "background", "events", "cancel", "refresh"]
keywords: ["job", "background", "progress", "events", "cancel"]
availability:
oss: available
enterprise: available
summary: >-
Job handles track operations in OSS and Enterprise. Enterprise also
supports listing server jobs, reopening them by ID and reading their events.
---
Some operations take longer than a request should. Building an index over a
large table, refreshing a computed column, registering a function -- these run
on the server after the call that started them has returned.
## What is a job? {#convention}
Each is submitted the same way and returns the same handle.
A job is a handle for tracking work, such as building an index or refreshing a
computed column. It lets you inspect progress and wait for the operation's
outcome after submission.
<Note>
A [query](/basics/query) also has an `_async` form, but its handle streams rows
rather than waiting for an outcome. Both answer the same four questions --
which one is this, how is it going, give me the result, stop it -- and differ
only in the third.
</Note>
Job-returning methods include Python's `create_index_async` and TypeScript's
`createIndexAsync`. Check the operation's reference for submission and waiting
behavior: a method's name alone does not tell you when the work is complete.
## The `_async` convention {#convention}
## What is its lifecycle? {#waiting}
An operation that can run in the background has two forms. The plain name blocks
until the work is done. The `_async` / `Async` name submits the work and returns
a **job** immediately.
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
job, also keep its ID if another process needs to look it up later.
2. **Check progress.** Read the current status while the operation runs.
3. **Wait for an outcome.** A job can finish, fail or be cancelled. Inspect its
result or failure information before continuing dependent work.
| Blocking | Returns a job | SQL |
In Python, `job.status()` reports the current state and `job.wait()` waits for
completion, raising if the job fails or is cancelled. A wait timeout ends the
wait; it does not request cancellation of the server job.
<span id="inspecting" />
<span id="events" />
## How do you manage a job? {#managing}
| Task | What to do |
| --- | --- |
| Inspect progress | Check status and refresh the handle to retrieve details. Enterprise job events can help explain ongoing work or failures. |
| Reattach from another process | Open an existing server job with its ID, such as Python's `db.open_job(job_id)`. The server must still retain the job record. |
| Diagnose a failure | Inspect failure details and relevant events. Enterprise SQL also provides `SHOW ERRORS`. |
| Request cancellation | Call the job's cancellation method. Cancellation may take time, and work already committed is not rolled back. |
Reopening a job retrieves the existing operation; it does not restart failed
work. Treat the ID as an opaque identifier and pass it unchanged to the jobs API.
Event responses have [row limits](/basics/limits#jobs). An explicit limit can
request more than the default, but it does not guarantee a complete event
history.
## Where does the work run? {#in-process}
| | OSS | Enterprise |
| --- | --- | --- |
| `create_index` | `create_index_async` | `CREATE INDEX` |
| `refresh_column` | `refresh_column_async` | `REFRESH COLUMN` |
| `create_function` | `create_function_async` | `CREATE FUNCTION` |
| `drop_table` | `drop_table_async` | `DROP TABLE` |
| Execution | In your application process | On server workers |
| Job identity | No server ID | A server ID for looking up the job |
| Event history | No server event history | Available through the jobs API |
| Reattaching after a client restart | No server job to reopen | Reopen by ID while the record is retained |
<Note>
The blocking form is not a different operation -- it submits the same job and
waits. Reach for the `_async` form when you want to do something else meanwhile,
run several at once, or survive the client going away.
</Note>
A handle can represent work that has already completed. Check its state rather
than assuming every returned job is still running.
## Submitting and waiting {#waiting}
## Learn more {#next}
<CodeGroup>
```python Python icon="python"
from lancedb.index import IvfPq
job = table.create_index_async("vector", config=IvfPq())
print(job.id) # the server's id for this operation
print(job.status()) # "running" | "finished" | "failed" | "cancelled"
job.wait() # blocks until terminal; raises if it failed
```
```typescript TypeScript icon="square-js"
const job = await table.createIndexAsync("vector");
console.log(job.id); // the server's id for this operation
console.log(await job.status()); // "running" | "finished" | "failed" | "cancelled"
await job.wait(); // resolves when terminal; rejects if it failed
```
</CodeGroup>
`wait` is the one call that raises on failure. `status` is a point snapshot and
returns the failed state rather than raising, which is what you want in a
polling loop.
In Python `wait` takes an optional timeout and returns the operation's typed
result -- a `RefreshColumnJobResult`, a `FunctionVersion`, or `None` for an
operation that has no result:
```python
from datetime import timedelta
result = table.refresh_column_async("embedding").wait(timeout=timedelta(minutes=30))
```
`TimeoutError` means the timeout elapsed, not that the job stopped. The job is
still running; the handle just gave up watching.
<Note>
The TypeScript `wait()` resolves to `void`. Read the outcome from `result` after
it resolves.
</Note>
## Inspecting a job {#inspecting}
Submitting an operation returns only an id, so the detail fields are empty until
the handle fetches the record. `refresh()` does that; `status()` does it as a
side effect.
<CodeGroup>
```python Python icon="python"
job.refresh()
job.state # last observed state, without another round trip
job.job_type # what kind of job, as the server names it
job.creation_ms # when it was created, epoch milliseconds
job.spec # what it was submitted with
job.result # its terminal result, once it succeeds
job.failure # why it failed, when it did
```
```typescript TypeScript icon="square-js"
await job.refresh();
job.state; // last observed state, without another round trip
job.jobType; // what kind of job, as the server names it
job.creationMs; // when it was created, epoch milliseconds
job.spec; // what it was submitted with
job.result; // its terminal result, once it succeeds
job.failure; // why it failed, when it did
```
</CodeGroup>
Printing the handle renders every field it knows, which is usually faster than
reading them one at a time.
## Progress, from events {#events}
`result` only tells you anything once the job has finished. Events are written
as it runs, and they outlive the workers that produced them -- so a job that is
still going, or one that never finished, can still account for what it did.
<CodeGroup>
```python Python icon="python"
events = job.events(limit=5000, filter="state = 'claim_complete'")
```
```typescript TypeScript icon="square-js"
const events = await job.events({ limit: 5000, filter: "state = 'claim_complete'" });
```
```sql SQL
SHOW ERRORS ON my_table FOR JOB 'job-id' LIMIT 100
```
</CodeGroup>
A distributed job records a `claim` / `claim_complete` pair per unit of work,
each carrying `rows_processed`. Summing `rows_processed` over the completed
claims is how you get progress on a job that has no percentage.
`filter` is a SQL-like expression over `state`, `updated_by`, `emitted_from`,
`emitted_by`, and `claim_entity`.
<Warning>
The server returns 1000 event rows by default and 10,000 at most, and
**truncates without telling you**. A job that emits an event per fragment will
silently lose its tail unless you pass `limit`.
</Warning>
Events are recorded server-side, so this raises for an operation that ran
in-process.
## Finding and cancelling jobs {#managing}
The connection lists and reopens jobs, so a handle lost to a restarted process
is recoverable from its id.
<CodeGroup>
```python Python icon="python"
db.list_jobs() # every job the server knows about
job = db.open_job(job_id) # reattach to one
db.cancel_job(job_id) # ask it to stop
```
```typescript TypeScript icon="square-js"
await db.listJobs();
const job = await db.openJob(jobId);
await db.cancelJob(jobId);
```
```sql SQL
SHOW JOBS
```
</CodeGroup>
Cancellation is a request, not a guarantee, and cancelling a job that already
finished does nothing. `JobNotFoundError` means the id does not name a job the
server still has.
<Warning>
A job id is opaque. It correlates with server logs and the calls above, but
parsing it or storing it to resume the job later is not supported.
</Warning>
## In-process jobs {#in-process}
A local database has no server to run anything, so its operations run in your
process. They still return a handle, so the same code works against both -- but
one with nothing behind it: `id` is `None`, `status()` reports `"finished"`,
`wait()` returns immediately, `cancel()` does nothing, and `events()` raises.
This is the difference to expect when moving between embedded and deployed --
see [SDKs](/basics/sdk).
- [Build and repair indexes](/indexing/reindexing): submit index work.
- [Column refresh](/feature-engineering/refresh-column): populate derived values.
- [Background jobs](/maintenance/background-jobs): understand Enterprise maintenance.
- [SQL job commands](/sql/ddl/job): list jobs and inspect errors.
- [SDKs](/basics/sdk): find the job API for your client.
docs/web/basics/lakehouse.mdx
+51 -83
View File
@@ -1,102 +1,70 @@
---
title: "Lance Lakehouse"
sidebarTitle: "Lance Lakehouse"
description: "What operates on the catalog's objects: storage, search, computation, and maintenance."
title: "LanceDB Lakehouse"
sidebarTitle: "LanceDB Lakehouse"
description: "Bring multimodal data, search and feature engineering together for AI workloads."
icon: layer-group
keywords: ["lakehouse", "multimodal", "search", "compute", "maintenance", "feature engineering"]
keywords: ["lakehouse", "multimodal", "search", "compute", "training", "feature engineering"]
---
The [catalog](/basics/catalog) says what exists. The **lakehouse** is everything
that acts on it: storing the rows, searching them, computing new ones, and
keeping the whole thing fast as it grows.
## What is the LanceDB lakehouse? {#multimodal}
The distinction matters because the two layers have different lifetimes. A
catalog entry is a name and a pointer. A lakehouse operation reads and writes
the [Lance](/basics/lance) files that name points at -- and those operations are
where almost all of the cost, and all of the tuning, lives.
LanceDB brings storage, search and data processing together for AI workloads.
You can store multimodal data, derive features, search your records and prepare
training datasets using tables built on the open [Lance format](/basics/lance).
## Multimodal is the point {#multimodal}
The format provides the storage foundation. The [catalog](/basics/catalog)
organizes your tables. LanceDB provides the tools to work with that data.
A lakehouse table holds images, video, audio, text, embeddings, annotations and
the features you derive from them **in one table**, side by side.
## How does the data flow? {#capabilities}
That is the difference from assembling a warehouse plus a vector store plus a
feature store. There is one copy of the data, one set of versions, and no
pipeline keeping three systems agreeing about the same rows. A search can filter
on a label, rank by a vector, and return the blob, without leaving the table.
Consider a team building a video search application:
| | |
| --- | --- |
| **Columns** | vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
| **Versions** | every write commits one, so reads can [travel back](/management/table#versioning) |
| **Indexes** | vector, full-text and scalar, built over those columns |
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
<span id="ingest" />
<span id="compute" />
<span id="search" />
## The four capabilities {#capabilities}
1. **Ingest the clips.** Store video clips with metadata such as timestamps,
source and labels.
2. **Add features.** Generate captions and embeddings with your models, and
store them alongside each clip.
3. **Use the dataset.** Search for relevant clips using embeddings and metadata
filters, or select a dataset to read into a training framework.
Everything the lakehouse does falls into one of four groups, and the rest of
these docs is organized the same way.
```mermaid
flowchart TD
clips["Video clips and metadata"] --> table["Store in a Lance table"]
table --> features["Add captions and embeddings"]
features --> search["Search for relevant clips"]
features --> training["Select data for training"]
```
### Ingest {#ingest}
The clips, metadata and derived features stay associated in the same table,
so retrieval and training workflows can use the same dataset. Your training
framework runs the model; LanceDB provides access to the training data.
Getting rows in, and changing the ones that are there: append, merge, update,
delete. Every write commits a version rather than mutating in place, which is
what makes the other three safe to run against a moving table. See
[Data Ingestion](/ingestion).
## What does Enterprise add? {#where}
### Search {#search}
LanceDB OSS runs as an embedded library in your application. LanceDB Enterprise
provides shared infrastructure for applications and teams working with larger
multimodal workloads.
Vector, full-text and hybrid retrieval over the same rows, with filters that
push into the scan rather than running after it. The search is a lakehouse
operation, not a separate service holding a second copy of your embeddings. See
[Semantic Search](/search).
<span id="maintenance" />
### Compute {#compute}
- **Scale resources for different workloads.** Provision query serving and
background workers independently as search traffic and indexing demand grow.
- **Share data across teams.** Give applications access to shared tables, with
Enterprise permissions controlling who can read or change them.
- **Manage ongoing maintenance.** Configure background indexing, compaction
and cleanup as your datasets grow.
Deriving new columns and tables from what is already stored:
[computed columns](/feature-engineering/add-computed-column),
[functions](/management/function) that run Python where the data is, and
[materialized views](/management/materialized-view) that persist a query's
result.
See [Architecture](/architecture) for how Enterprise separates storage, query
serving and background work.
The reason this belongs in the lakehouse rather than in a pipeline outside it is
the format. Adding a derived column writes only that column -- the images and
videos are never rewritten -- so deriving a feature is an ordinary operation
rather than a migration. See [Feature Engineering](/feature-engineering) and
[Data Curation](/curation).
## Learn more {#next}
### Maintenance {#maintenance}
The work that keeps the first three fast: building and repairing
[indexes](/indexing), and [compaction](/maintenance/compaction) to merge small
files, prune old versions and fold new rows into existing indexes.
<Warning>
None of it runs on its own. There is no background compactor -- `optimize()`
runs when you call it, and an index does not cover rows added after it was
built until you do. See [Guarantees](/basics/guarantees#not-guaranteed).
</Warning>
## Where the work happens {#where}
The same lakehouse runs in two shapes, and which one you use decides where these
capabilities execute rather than whether you have them:
- **Embedded** -- your process does all four itself, directly against object
storage. Nothing runs unless you call it.
- **Distributed** -- a cluster does them, with indexing and compaction on
workers separate from the ones serving queries, plus
[SQL](/sql) and server-side [jobs](/basics/jobs).
See [Architecture](/architecture) for the comparison in full.
## Next {#next}
<CardGroup cols={2}>
<Card title="Lance Catalog" icon="sitemap" href="/basics/catalog">
The objects these operations act on.
</Card>
<Card title="Lance format" icon="layer-group" href="/basics/lance">
The open format underneath, and why column evolution is cheap.
</Card>
</CardGroup>
- [Data ingestion](/ingestion): load and update your data.
- [Search](/search): retrieve records with vector, full-text and hybrid search.
- [Feature engineering](/feature-engineering): derive columns and datasets.
- [Functions](/management/function): run Python to compute column values. {/* if functions launched */}
- [Training data access](/training/data-loading): read datasets into training workflows.
- [Enterprise](/enterprise): explore deployment options for shared workloads.
docs/web/basics/lance.mdx
+29 -106
View File
@@ -1,122 +1,45 @@
---
title: "Lance format"
sidebarTitle: "Lance Format"
description: "Lance is the open format underneath. LanceDB is what implements a lakehouse on top of it."
description: "The open format for multimodal data behind LanceDB."
icon: "/static/assets/logo/lance-logo-gray.svg"
keywords: ["lance", "format", "open standard", "lakehouse", "namespace", "catalog"]
keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
---
Two names, two things. [Lance](https://lance.org/) is an open format and
specification. LanceDB is an implementation built on it.
## What is Lance? {#standard}
The distinction matters when you are deciding what you are committing to: the
format is a standard other engines read, and your data stays readable by them
whichever LanceDB you run.
[Lance](https://lance.org/) is an open-source, columnar format designed for
multimodal AI. It defines how data is stored and organized into versioned tables.
## Lance is the open standard {#standard}
<span id="embedded" />
<span id="enterprise" />
Lance is an open-source, columnar lakehouse format for multimodal AI. It
specifies three things:
LanceDB OSS and LanceDB Enterprise both use Lance as their storage foundation.
The format is developed as an independent open-source project, with its own
specification and tools.
| | |
| --- | --- |
| **File format** | how bytes are laid out on disk, optimized for random access without giving up scan throughput |
| **Table format** | how files compose into a versioned table -- schema evolution, deletions, indexes |
| **Namespace specification** | how tables are named, grouped, and resolved -- the catalog layer |
## What can it store? {#storage}
Being a specification rather than a product is the point. Object storage holds
the bytes, and open engines -- PyTorch, Ray, Spark, Trino, DuckDB, Polars --
read them without going through LanceDB at all.
A Lance table can hold images, video, audio, text, embeddings and structured
metadata together. For example, a video dataset can keep each clip alongside
its caption, labels and embeddings, so applications can search and work with
those fields together.
Lance exists because columnar formats derived from Parquet were not designed
for random access on vectors: [benchmarks](https://lancedb.com/blog/benchmarking-random-access-in-lance/)
show Lance up to 1000x faster than Parquet for that access pattern.
## What does the format enable? {#capabilities}
<Card
title="Lance format documentation"
icon="/static/assets/logo/lance-logo-gray.svg"
href="https://lance.org/format"
>
The format specification: design, layout, and how it enables the multimodal lakehouse.
</Card>
- **Efficient data access.** Read individual rows for retrieval and training,
or scan columns for analytics.
- **Versioned data.** Read an earlier table version to reproduce an experiment
or inspect previous results, while that version is retained.
- **Independent column updates.** Add embeddings or other derived features
without rewriting the existing image, video or text columns.
## LanceDB embedded implements the storage half {#embedded}
## Learn more {#next}
Connecting to a directory or a bucket gives you a **storage-only lakehouse**.
Your process opens Lance tables directly and does the work itself: it plans and
executes queries, builds indexes, and writes new versions.
<span id="versioning" />
<span id="evolution" />
```python
db = lancedb.connect("./.lancedb")
db = lancedb.connect("s3://bucket/path")
```
There is no server, which is what makes it embedded -- and also what bounds it.
There is nothing to run work on your behalf, so there are no server-side
[jobs](/basics/jobs), no [SQL endpoint](/basics/sql), and no shared catalog
beyond what the storage layer itself provides.
LanceDB OSS is Apache 2.0 licensed. LanceDB Enterprise is the commercial,
deployment-only product that implements the catalog specification below.
## LanceDB Enterprise implements the catalog specification {#enterprise}
<Badge color="red">Enterprise</Badge> A `db://` connection talks to a deployment
that implements the **Lance REST Namespace** -- the REST server defined by the
Lance namespace specification, covering the catalog operations in the
[REST API reference](/api-reference/rest).
```python
db = lancedb.connect("db://my-database", api_key="...")
```
Implementing the specification rather than inventing an interface is what makes
the catalog reachable by anything that speaks it, and what makes
[authorization](/governance/authorization), server-side jobs, and
[SQL](/basics/sql) possible: there is now a server to enforce and to run them.
The tables are the same Lance tables either way. What changes is who resolves
their names and who does the work.
## What the format gives you {#capabilities}
These are properties of Lance, so they hold in both forms:
| Capability | What it enables |
| --- | --- |
| Multimodal storage | Images, video, audio, text, embeddings, annotations and metadata in one table. |
| First-class blob API | Large binary objects in blob columns, with lazy reads and streaming byte access. |
| Fast random access and scans | Sample, shuffle, and fetch individual rows without giving up sequential throughput. |
| Zero-copy schema evolution | Add, drop, rename, or alter columns without rewriting existing data files. |
| Versioned tables | Reproduce experiments and tie artifacts to the exact table version they used. |
| Hybrid search and indexing | Vector search, full-text search, and scalar filters over the same dataset. |
| Open interoperability | Object storage underneath, open engines on top. |
Two of them explain behaviour you will meet elsewhere in these docs.
### Versioning is why old data lingers {#versioning}
Every write creates a version, and old versions are not immediately destroyed --
another reader may be part-way through one. A hundred versions are not a hundred
copies of the data, but they do carry a hundred times the metadata, which
eventually slows queries down.
Deletion works the same way: a deleted row is marked in a deletion file rather
than erased, so the data is recoverable for as long as the version is.
[Compaction](/maintenance/compaction) is what resolves both.
### Column evolution is why feature engineering is cheap {#evolution}
Adding a derived column writes only the new column. Existing data -- the
expensive part, the images and videos -- is untouched. That is what makes
[computed columns](/feature-engineering/add-computed-column) a reasonable thing to
do repeatedly on a large table rather than a full rewrite each time.
<Card
title="Learn more about Lance"
icon="/static/assets/logo/lance-logo-gray.svg"
href="https://lance.org/quickstart"
>
Lance is a separate open source project with its own documentation.
</Card>
- [Lance format specification](https://lance.org/format): how the format works.
- [Table versioning](/management/table#versioning): read and manage table versions.
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
- [Architecture](/architecture): how LanceDB OSS and Enterprise build on Lance.
docs/web/basics/limits.mdx
+51 -86
View File
@@ -1,106 +1,71 @@
---
title: "Limits"
sidebarTitle: "Limits"
description: "The bounds you can actually hit, in one place."
description: "Find fixed constraints, configurable bounds and what happens when you reach them."
icon: ruler
keywords: ["limits", "maximum", "quota", "constraints", "bounds"]
---
Limits worth designing around, collected so you do not have to find them by
hitting them. Each one is enforced -- where a value is refused rather than
clamped, that is stated, because the two fail very differently.
## Which limits apply? {#scope}
This page covers bounds that are the same for everyone. Deployment-specific
quotas -- storage, throughput, concurrency -- come from your deployment.
Some limits come from an API or data type; others come from Enterprise
configuration. Use the scope and linked reference below to identify which
applies to your operation. SQL and server job limits apply to Enterprise.
## Names {#names}
<span id="names" />
| Object | Maximum | |
[Naming Rules](/basics/naming) is the reference for name lengths, characters and
namespace depth. Client and server versions can differ in which rules they
enforce.
<span id="jobs" />
<span id="sql" />
## What are the fixed limits? {#fixed-limits}
| Operation | Scope | Limit and behavior |
| --- | --- | --- |
| Database | 64 characters | |
| Namespace | 64 characters | |
| Table | 128 characters | |
| Branch | 64 characters | |
| Job event retrieval | Enterprise | At most 10,000 rows per response. Results can be truncated without a truncation indicator. |
| `SHOW ERRORS ... LIMIT` | Enterprise SQL | Accepts 1 through 100,000. Values outside the range are rejected. |
| `SHOW ERRORS` without `FOR JOB` | Enterprise SQL | Inspects up to the 32 most recent failed jobs. Use `FOR JOB` to inspect a specific job. |
| Percentage sampling | Enterprise SQL | `TABLESAMPLE` accepts percentages from 0 through 100. Values outside the range are rejected. |
| Row-count sampling | Enterprise SQL | `TABLESAMPLE (... ROWS)` requires a whole number. |
A name must not be empty, and must not contain `$` or `:` -- both are reserved
as path delimiters, so a name holding one could not be addressed unambiguously.
Length is counted in characters, not bytes.
See [SQL job commands](/sql/ddl/job#errors) and
[Sampling](/sql/dql/sampling) for syntax and restrictions on supported table types.
## Jobs {#jobs}
<span id="schema" />
| | |
Data-type and index constraints belong to the operation you choose. Check
[Data types](/sql/data-types) and [Vector indexes](/indexing/vector-index)
before choosing a schema or index. For materialized views, check
[stable row ID requirements](/feature-engineering/materialized-views#stable-row-ids)
before creating source tables.
## What can you configure? {#not-quotas}
| Setting | What it controls |
| --- | --- |
| `job.events()` default | 1,000 rows |
| `job.events()` maximum | 10,000 rows |
| `SHOW ERRORS ... LIMIT` | between 1 and 100,000 |
| Failed jobs scanned by one `SHOW ERRORS` | 32 |
| Job event `limit` | Requests more or fewer event rows. The default is 1,000; the 10,000-row maximum still applies. |
| FTS `max_token_length` | The maximum indexed token length. The default is 40; longer tokens are excluded from the index. |
| Enterprise capacity and quotas | Storage, request rate and concurrency are governed by your deployment's resources and configuration. Confirm these with your administrator or LanceDB support. |
<Warning>
`job.events()` **truncates silently**. A job that emits an event per fragment
will lose its tail unless you pass `limit` explicitly. This is the limit on this
page most likely to mislead you, because nothing in the result says it was cut.
</Warning>
## What happens at a limit? {#behavior}
`SHOW ERRORS` without `FOR JOB` scans the most recent failed jobs and stops at
32 of them. Narrow with `FOR JOB` when you need a specific one rather than a
survey.
- **Rejected requests:** correct the input before retrying, such as a sampling
percentage outside the accepted range.
- **Truncated responses:** do not assume that a returned event list is the full
history. Increasing `limit` cannot exceed the maximum; use filters to narrow
the events you inspect.
- **Excluded data:** an FTS token-length setting can affect which words are
searchable. Choose it for your text and indexing requirements.
See [Jobs](/basics/jobs).
<span id="availability" />
## Schema and types {#schema}
## Learn more {#next}
| | |
| --- | --- |
| `map` with `keys_sorted = true` | rejected |
| `IVF_RQ` vector dimensions | must be divisible by 8 |
| FTS `max_token_length` | 40 by default; longer tokens are dropped |
Through the SDK, a table gets **stable row ids** only at creation, from the
`new_table_enable_stable_row_ids` storage option on a local path or an object
store; no SDK call adds them later, so decide early. A local
[materialized view](/feature-engineering/materialized-views#stable-row-ids) does
not need them. On a deployment, whether a view's source needs them and how a
table gets them depend on the deployment; see
[Against a deployment](/feature-engineering/materialized-views#availability).
See [Data types](/sql/data-types).
## SQL {#sql}
<Badge color="red">Enterprise</Badge>
| | |
| --- | --- |
| `TABLESAMPLE` percentage | 0 to 100, **refused** outside -- not clamped |
| `TABLESAMPLE (n ROWS)` | whole numbers only |
| `TABLESAMPLE BUCKET` | not supported |
| `TABLESAMPLE` on a WAL-backed table | not supported |
| `CREATE INDEX` | one column, server-chosen name |
| `REFRESH MATERIALIZED VIEW ... CASCADE` | depends on the deployment's version; see [REFRESH](/sql/dml/refresh#refresh-materialized-view) |
A sampling percentage over 100 is an error rather than a full scan, because a
query asking for 150 percent has a mistake in it and reading it as 100 would
hide the mistake. See [Sampling](/sql/dql/sampling).
## What is unavailable where {#availability}
Not quotas, but the same class of surprise -- things that exist in one place and
not another.
| | |
| --- | --- |
| SQL on a local database | unavailable; `execute_query` raises |
| SQL methods in the TypeScript SDK | none; use a Flight SQL client |
| Function methods in the TypeScript SDK | none; use Python or SQL |
| Vector search in SQL | no SQL form; use a query builder |
| Geospatial `st_*` functions | behind a build feature LanceDB does not ship |
| `tableExists` / `namespaceExists` in TypeScript | not present |
| Server-side jobs on a local database | run in-process; `events()` raises |
See [SQL](/basics/sql) for why the first two hold.
## Nothing here is a quota {#not-quotas}
None of the above is a billing or capacity limit. Rows per table, table count,
storage, and request rate are bounded by your deployment rather than by the
format -- ask your deployment, not this page.
- [Naming Rules](/basics/naming): choose portable names and paths.
- [Jobs](/basics/jobs): track work and inspect events.
- [Full-text indexes](/indexing/fts-index): configure tokenization.
- [SQL](/basics/sql) and [SDKs](/basics/sdk): check interface support.
- [Enterprise support](/deployment/support): discuss deployment capacity.
docs/web/basics/naming.mdx
+64
View File
@@ -0,0 +1,64 @@
---
title: "Naming rules"
sidebarTitle: "Naming Rules"
description: "Choose portable names for your data objects and namespace paths."
icon: signature
keywords: ["names", "naming", "table name", "namespace", "identifier", "limits"]
---
## Naming rules {#rules}
<span id="objects" />
Databases, namespaces, tables, views and branches all have names. Choose names
that describe their purpose, such as `documents`, `prod` or `training_2026`.
Consistent names make data easier to find and application code easier to read.
Use the following rules for names that work across LanceDB OSS and Enterprise.
| Object | Characters | Length |
| --- | --- | --- |
| Table, materialized view, view | ASCII letters and digits, `-`, `_`, `.` | 1 to 128 characters |
| Namespace component | ASCII letters and digits, `-`, `_`, `.` | 1 to 64 characters; at most 10 components in a path |
| Database | ASCII letters and digits, `-`, `_`, `.`, with `/` between components | 1 to 64 characters for the whole name |
| Branch | Avoid `$` and `:`; use ASCII letters, digits, `-` and `_` for portability | Keep to 1 to 64 characters |
| Secret {/* if functions launched */} | ASCII letters and digits, `-`, `_`, `.` | 1 to 128 characters |
| Function {/* if functions launched */} | ASCII letters and digits, `_` | 1 to 128 characters |
- Names and individual namespace or database components must not be empty.
- Table, view, namespace and database components cannot be `.` or `..`.
- Secret names cannot be `.` or `..`. {/* if functions launched */}
- Avoid `$` and `:` because they separate parts of qualified identifiers.
- A namespace path beginning with `public` cannot contain another component.
## Namespace paths {#paths}
A namespace path is a list of names. `["prod", "search"]` identifies the
`search` namespace within `prod`. A table named `documents` belongs to that
namespace; its table name remains `documents`.
Use the SDK's namespace-path argument to supply the list. A period inside an
SDK name is a character in that name, not a separator that creates another
namespace. SQL uses its own qualified-name syntax; see
[SQL namespace paths](/sql/dql/select#namespaces).
The empty path `[]` identifies the root namespace.
## Examples {#examples}
| Name or path | Guidance |
| --- | --- |
| `documents`, `user-events.v2`, `train_2026` | Portable table names |
| `["prod", "search"]` | A two-component namespace path |
| `my table` | Use `my_table` instead of a space |
| `orders$2026` | Use `orders_2026`; `$` is reserved |
| `résumés` | Use an ASCII name such as `resumes` |
| `["public", "search"]` | Invalid nesting under `public`; use a different parent namespace |
<span id="which-versions-check-them" />
## Learn more {#next}
- [Namespaces](/management/namespace): create namespaces and address tables.
- [Branches](/tables/branching): name and manage separate lines of table history.
- [Limits](/basics/limits): check other constraints before designing your schema.
docs/web/basics/offerings.mdx
+70
View File
@@ -0,0 +1,70 @@
---
title: "LanceDB OSS or Enterprise?"
sidebarTitle: "OSS and Enterprise"
description: "Start with an embedded library, or use the LanceDB Enterprise multimodal lakehouse for shared AI workloads."
icon: compass
keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "comparison"]
---
LanceDB gives you two ways to work with multimodal data, both built on the open
[Lance](/basics/lance) format.
**LanceDB OSS** is an embedded library. Run it inside your application
with tables on local disk or in object storage. You control the application,
its resources, and table maintenance, without a separate database service.
[Try the quickstart](/quickstart) to create a table and run your first search.
**LanceDB Enterprise** is a multimodal lakehouse for data curation, feature
engineering, search, and training data access. It adds query-serving capacity,
data caching, and background operations for teams working with shared
multimodal datasets.
[Connect to Enterprise](/basics/connect#enterprise) when your cluster is ready.
<span id="labels" />
<span id="availability" />
## Why choose Enterprise? {#why-enterprise}
### Serve growing workloads {#serving}
Add query-serving capacity as more applications and users depend on your data.
Enterprise spreads query traffic across a fleet and caches frequently read
table data near compute, reducing repeated reads from object storage. Query
serving and background work use separate resources, so you can size each for
your workload. See [Enterprise architecture](/enterprise/architecture).
### Reduce operational work {#operations}
Let the service handle routine table maintenance instead of scheduling it in
application code. Enterprise handles background [compaction](/maintenance/compaction) and can
[clean up old table versions](/maintenance/cleanup) according to your retention
policy. Configure these maintenance tasks for your Enterprise cluster so each
application does not have to schedule them.
### Fit your deployment requirements {#deployment}
Choose a managed deployment or bring LanceDB into your own cloud account.
Enterprise integrates with OpenTelemetry for metrics and traces, helping your
team monitor the service alongside your applications. Talk to us about your
cloud, networking, and operational requirements to plan the right deployment.
The [Enterprise overview](/enterprise) explains the available options.
### Connect your data workflows {#workflows}
Use the same tables for retrieval, dataset preparation, and training data access.
With its [SQL endpoint](/basics/sql) enabled, Enterprise supports analytical
[curation](/curation), including joins, grouping, and sampling.
[Feature engineering](/feature-engineering) adds derived columns and tables;
computed columns and materialized views require the corresponding features to
be enabled for your Enterprise cluster.
[Geneva](/geneva) runs Python feature backfills as distributed jobs.
[Training data loaders](/training/data-loading), available in both offerings,
read the resulting datasets for model training.
## Plan your Enterprise deployment {#get-started}
[Talk to us about your workload](https://www.lancedb.com/contact) to discuss your
data, query traffic, pipelines, and deployment requirements.
For a closer look at how the service runs, read the
[Enterprise architecture](/enterprise/architecture).
docs/web/basics/query.mdx
+36 -76
View File
@@ -1,94 +1,54 @@
---
title: "Queries"
sidebarTitle: "Query"
description: "Running a query, and reading its results before it finishes."
description: "Choose how to query your data and read the results."
icon: magnifying-glass
keywords: ["query", "async", "streaming", "arrow", "reader", "execute_query"]
keywords: ["query", "search", "SQL", "async", "streaming", "results"]
---
A query returns rows. How you get them depends on two choices that are easy to
conflate: whether **your thread** blocks, and whether you take the results all
at once or stream them as they arrive.
## What is a query? {#queries}
## Blocking or not is a language concern {#async-client}
A query selects data from your tables. You can filter rows, choose columns,
search for similar content or combine those operations. For example, a video
application might find clips similar to a sample and return only those recorded
in the last month.
A concern of your language, not of LanceDB. The work is identical either way.
## How can you query? {#searches}
<CodeGroup>
```python Python icon="python"
db = lancedb.connect("db://my-db", api_key="...") # blocking
tbl = db.open_table("documents")
| Interface | When to use it |
| --- | --- |
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in OSS and Enterprise. |
| **Enterprise SQL** | Write SQL queries for filtering, aggregation, joins and other data curation tasks. Connect through Python or a Flight SQL client. |
db = await lancedb.connect_async("db://my-db", api_key="...") # coroutines
tbl = await db.open_table("documents")
```
Both interfaces work with LanceDB tables. Choose the one that supports your
operation and fits your application. See [SQL](/basics/sql) for connection
instructions and SQL search support, and [Search](/search) for the query builders.
```typescript TypeScript icon="square-js"
// TypeScript is async throughout; there is no blocking form.
const db = await lancedb.connect("db://my-db", { apiKey: "..." });
const tbl = await db.openTable("documents");
```
</CodeGroup>
## How do results arrive? {#results}
Python offers both. `AsyncConnection` exposes the same surface as coroutines, so
choosing between them is about the program you are writing -- a service wants
the async form, a script does not care.
- **Collect the results** when you want a list, Arrow table or DataFrame that
fits in memory. Use a result limit and select only the columns you need.
- **Read batches** when you want to process a large result incrementally.
Streaming avoids collecting the entire result in application memory, although
some query operations still need to do substantial work before producing rows.
## Reading before the query finishes {#queries}
<span id="async-client" />
A query is not a job, but it has the same shape of choice: wait for everything,
or take a handle and start reading.
An async client lets your application do other work while a request is waiting.
That is separate from whether you collect results or read them in batches.
Python offers synchronous and asynchronous clients; TypeScript uses async APIs.
Check the SDK reference for the methods available in your client.
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 1000")
table = reader.read_all()
<span id="jobs" />
query = db.execute_query_async("SELECT id, text FROM my_docs")
```
Enterprise SQL can also return a query handle for inspecting execution and
requesting cancellation. A [job](/basics/jobs) tracks an operation such as an
index build. Use the result API for the operation you submitted.
```sql SQL
SELECT id, text FROM my_docs LIMIT 1000
```
</CodeGroup>
## Learn more {#next}
`execute_query` returns a blocking Arrow reader **once results start arriving**
-- it does not wait for the whole query to finish. Results stream as record
batches, so a large result set does not have to fit in memory before you see its
first row.
`execute_query_async` returns the query handle instead:
```python
query = db.execute_query_async("SELECT id, text FROM my_docs")
query.id # identifies it on the server
query.describe() # what it is doing, without consuming results
reader = query.reader() # stream the rows when you want them
query.cancel() # stop it
```
`describe_query(id)` reports on one from elsewhere, the way `open_job(id)` does
for a job.
<Note>
SQL requires a deployment. See [SQL](/basics/sql#sql-is-remote).
</Note>
Query-builder searches -- vector, full-text, hybrid -- are ordinary calls that
return results. They are fast enough not to need a handle. See
[Query](/search).
## Searches are ordinary calls {#searches}
Query-builder searches -- [vector](/search/vector-search),
[full-text](/search/full-text-search), [hybrid](/search/hybrid-search) -- return
results directly. They are fast enough not to need a handle, and they have no
SQL form. See [Semantic Search](/search).
## When the work outlives the call {#jobs}
Some operations take longer than a query should: building an index, refreshing a
computed column, registering a function. Those return a **job** rather than
rows, and a job is a different handle with a different shape. See
[Jobs](/basics/jobs).
- [Quickstart](/quickstart): run a complete create, insert and search example.
- [Search](/search): choose vector, full-text or hybrid retrieval.
- [Filtering](/search/filtering): narrow results with predicates.
- [SQL](/basics/sql): connect a client and run SQL queries.
- [SDKs](/basics/sdk): choose a language and find its API reference.
docs/web/basics/sdk.mdx
+82 -64
View File
@@ -1,99 +1,117 @@
---
title: "SDKs"
sidebarTitle: "SDKs"
description: "The language clients — Python, TypeScript, Rust — and what stays true across all of them."
description: "Choose a LanceDB SDK and find the API reference for your language."
icon: code-compare
keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"]
availability:
oss: available
enterprise: available
summary: >-
Python, TypeScript and Rust support both LanceDB OSS and LanceDB Enterprise.
Java provides a client for the Enterprise REST API.
---
A language SDK is the primary way to work with LanceDB: create and manage
tables, write and query data, build indexes, run feature engineering. The same
table API works whether you're connected to an embedded database or an
Enterprise deployment -- application code moves between the two by changing
the connection URI.
Use a LanceDB SDK to create tables, add data, build indexes and run searches
from your application. Python, TypeScript and Rust work with both the embedded
OSS library and LanceDB Enterprise.
## Officially supported SDKs {#supported-sdks}
The [quickstart](/quickstart) includes installation instructions and complete
examples in all three languages. For connection settings, see
[Connect to LanceDB](/basics/connect).
Python, TypeScript and Rust are supported against both embedded and deployed
LanceDB.
## Official SDKs {#supported-sdks}
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="python" /> [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration |
| <Icon icon="square-js" /> [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` |
| <Icon icon="rust" /> [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance |
| SDK | Use it with | API reference |
| --- | --- | --- |
| Python | Python applications, notebooks, pandas and PyArrow | [Python reference](https://lancedb.github.io/lancedb/python/python/) |
| TypeScript | Node.js and TypeScript applications | [TypeScript reference](https://lancedb.github.io/lancedb/js/) |
| Rust | Rust applications using the native library | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) |
| Java | Applications using the Enterprise REST API | [Java reference](https://lancedb.github.io/lancedb/java/java/) |
Not every SDK covers the same surface, though:
Guides show the available language examples together. Use the API reference for
method signatures and options; SDKs do not expose every feature through the same
methods. In Rust, enable the `remote` feature to connect to Enterprise.
| | Python | TypeScript |
| --- | :---: | :---: |
| Create and manage catalog objects | yes | yes |
| Insert, merge, update, delete | yes | yes |
| Filter, project, aggregate, join | yes | yes |
| Vector search | yes | yes |
| Full-text search | yes | yes |
| Feature engineering, jobs | yes | partly |
| Runs against a local database | yes | yes |
TypeScript has no methods for [functions](/management/function). Reach the
gap with SQL instead -- see [SQL](/basics/sql).
For SQL joins, aggregations and window functions, use
[SQL](/basics/sql). Python provides a SQL client, and applications in other
languages can use a Flight SQL client. TypeScript's table query API does not
provide a SQL execution method.
## REST API {#rest}
<Badge color="red">Enterprise</Badge> A deployment implements the
[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable
over HTTP by anything that speaks it -- see the
[REST API reference](/api-reference/rest).
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="java" /> [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java |
LanceDB Enterprise also exposes an HTTP API for catalog and table operations.
Use it when integrating a language or tool without a LanceDB SDK. The
[REST API reference](/api-reference/rest) describes its endpoints and request
formats. For SQL queries, connect through [Flight SQL](/basics/sql).
## Community SDKs {#community-sdks}
Contributed by the community rather than supported by LanceDB, so feature parity
and release cadence vary.
These clients are maintained by the community. Check each project's supported
features, installation instructions and releases before choosing one.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="golang" /> [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK |
| <Icon icon="gem" /> [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings |
| <Icon icon="swift" /> [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK |
| <Icon icon="R" /> [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package |
| <Icon icon="flutter" /> [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings |
For any other language, [SQL](/basics/sql) reaches the same tables without a
LanceDB-specific SDK at all.
| Language | Project |
| --- | --- |
| Go | [lancedb-go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) |
| Ruby | [Lancelot](https://github.com/scientist-labs/lancelot) |
| Swift | [LanceDbSwiftKit](https://github.com/RyanLisse/LanceDbSwiftKit) |
| R | [lancedb](https://github.com/CathalByrneGit/lancedb) |
| Flutter | [flutter_lancedb](https://github.com/Alexcn/flutter_lancedb) |
## Python multiprocessing {#multiprocessing}
<Warning>
Do not use `fork` as the start method with Python's `multiprocessing` module.
Lance is multi-threaded internally, and `fork` with alive threads is unsafe --
use `spawn` instead. See
[this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555)
for why.
Use `spawn` when starting Python worker processes with `multiprocessing`.
LanceDB uses threads internally, and forking a process with active threads can
cause deadlocks. Create a connection inside each worker instead of sharing an
open connection from the parent process.
</Warning>
## What stays the same {#invariants}
For PyTorch worker setup and connection factories, see
[PyTorch integration](/training/torch).
Whichever you pick:
## Moving to LanceDB Enterprise {#moving-to-enterprise}
- **Types are Arrow types.** A vector column is a `FixedSizeList<Float32>`
everywhere, not a SQL type mapped onto storage. See
[Functions and types](/sql/functions).
- **Reads are Arrow.** Nothing is serialized into language objects on the way out
unless you convert it.
- **The catalog is one catalog.** A table created from TypeScript is visible to
Python and to SQL immediately -- there is no per-language namespace.
The core table workflow stays familiar: connect, create or open a table, write
records, and query them. Enterprise runs those operations on remote compute and
manages background maintenance.
## What to read next {#next}
- **Configure the connection.** Replace the local path with your database URI,
and supply the endpoint, region and credentials provided by your administrator.
[Connect to Enterprise](/basics/connect#enterprise) shows these settings.
- **Retrieve query results.** Python's remote tables do not support whole-table
`to_arrow()` or `to_pandas()` calls. Use a query and select the columns and
number of rows you need, for example `table.search().limit(10).to_arrow()`.
- **Handle table listings and namespaces.** Remote table listings are paginated.
Use the namespace and pagination options described in
[Namespaces](/management/namespace).
- **Use server-side jobs.** Index builds and feature refreshes can run as
[jobs](/basics/jobs). Each operation's guide explains how to submit work and
track completion.
- **Let Enterprise manage maintenance.** Use
[background maintenance](/maintenance/background-jobs) rather than calling
the embedded table's `optimize()` method.
For SQL connection details, see [SQL](/basics/sql). For the operating model and
deployment options, see [LanceDB Enterprise](/enterprise).
## Shared data across languages {#invariants}
LanceDB uses Arrow schemas across its SDKs. A fixed-size vector column stores
vectors of a consistent dimension; the [data types reference](/sql/data-types)
describes the supported element types and other column types.
Applications in different languages can work with the same tables. Connect to
the same database and namespace, use the appropriate credentials, and account
for [read consistency](/tables/consistency) when another process writes data.
## Related guides {#next}
<CardGroup cols={2}>
<Card title="SQL" icon="table-list" href="/basics/sql">
The same tables, reached by any Flight SQL client -- and the one way in for a language with no SDK.
Query Enterprise tables with SQL from Python or a Flight SQL client.
</Card>
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
Give a coding agent a current reference to this same API surface.
Use the LanceDB plugin to help a coding agent work with your data.
</Card>
</CardGroup>
docs/web/basics/sql.mdx
+115 -81
View File
@@ -1,125 +1,159 @@
---
title: "SQL"
sidebarTitle: "SQL"
description: "The same tables, reached with SQL — the universal access path, and where every operation is formally documented."
description: "Query LanceDB Enterprise with SQL from Python or an Arrow Flight SQL client."
icon: table-list
keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"]
availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for embedded OSS tables. See [Connection settings](#sql-is-remote) to connect.
---
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local
database has no engine to plan a statement, so `execute_query` raises there.
Use SQL in LanceDB Enterprise to filter records, join tables, calculate
aggregates, and prepare datasets. SQL queries work with the same Lance tables
you create and update through the SDKs, without exporting the data to another
system.
SQL is a second way to reach the same tables the [SDKs](/basics/sdk) do --
not a different product, a different interface onto one engine. A table
written from Python is queried from SQL without an export step, and a table
created from SQL is visible to every SDK immediately.
You can run SQL through the Python SDK or an Arrow Flight SQL client. This
guide covers connection settings, a first query, and how SQL fits alongside
the SDK table APIs.
## Any language, via Flight SQL {#flight-sql}
## Connect a SQL client {#flight-sql}
SQL is served over an
Enterprise exposes an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it -- which makes SQL the way in
for a language LanceDB has no SDK for. You don't need a LanceDB-specific
client at all, just a Flight SQL driver for your language.
endpoint. A compatible client sends SQL statements to this endpoint and
receives results as Arrow record batches.
The TypeScript SDK itself has no SQL method, for the same reason a Go or C#
application would reach for SQL first: connect a Flight SQL client instead of
waiting on a native binding. See [SDKs](/basics/sdk) for the officially
supported and community clients, and reach for Flight SQL whenever yours
isn't among them.
- **Python:** use `execute_query()` on an Enterprise connection, as shown below.
- **TypeScript:** use a separate Flight SQL client. The LanceDB TypeScript SDK
does not expose a SQL execution method.
- **Other languages and tools:** use a Flight SQL client or a tool with Flight
SQL integration. A LanceDB-specific SDK is not required to submit SQL.
## Every SDK operation, in SQL {#sdk-parity}
The [SQL reference](/sql#connecting) shows a query from Python. See
[SDKs](/basics/sdk) for the LanceDB language clients.
Broadly, a public SDK method has a corresponding SQL statement -- creating a
table is `CREATE TABLE`, inserting rows is `INSERT INTO`, building an index is
`CREATE INDEX`. The two describe the same operations, so whichever one your
code speaks, the other is available too.
### Connection settings {#sql-is-remote}
| | SDK | SQL |
| --- | :---: | :---: |
| Create and manage catalog objects | yes | partly |
| Insert, merge, update, delete | yes | yes |
| Filter, project, aggregate, join | yes | yes |
| Vector search | yes | version-dependent |
| Full-text search | yes | yes |
| Feature engineering, jobs | yes | yes |
| Runs against a local database | yes | **no** |
Get the following values from your Enterprise administrator:
## Vector search depends on the deployment {#no-vector-sql}
- Database URI, such as `db://your-database`.
- API key and region.
- API endpoint for `host_override`.
- Flight SQL endpoint for `sql_host_override`, including its scheme and port.
The SDK query builder provides vector search across embedded and deployed
databases. SQL support depends on the deployment version and supported query
shape. Some versions recognize `ORDER BY distance(vector, query_vector) LIMIT k`
over a table scan and plan vector search, including indexed and flat search.
Confirm support on your deployment and use SQL
[`EXPLAIN`](/sql/dql/explain) to check the chosen plan; this is not a guarantee
that every deployed release supports that form.
The Flight SQL endpoint must be enabled and reachable from your client. Pass
it explicitly as `sql_host_override` in Python, as in the example below. The
API and SQL endpoints may use different hosts or ports.
Full-text search is available through the [`fts()`](/sql/functions#fts) table
function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens)
predicate.
See [Connect to Enterprise](/basics/connect#enterprise) for other connection options.
The reverse gap also exists: a [view](/management/view) is a saved SQL query,
so SQL is the only way to create one -- there's nothing for an SDK method to
call.
SQL runs in Enterprise. Connections to a local directory or an object-storage
bucket use the embedded OSS library and cannot execute SQL statements. Use
SDK table operations to query those tables.
## SQL needs a deployment {#sql-is-remote}
## Run a query in Python {#sql-from-python}
A connection to a directory or a bucket has no server to plan and execute a
statement, and asking it to raises `NotImplementedError`. This is the largest
behavioral difference between the embedded and deployed forms, and the reason
the SDKs are the portable choice: SDK code moves between the two by changing
the URI, SQL does not.
Install the SDK with `pip install lancedb`. Replace the connection placeholders
with your Enterprise settings, including the region and both endpoint URLs.
This example reads an existing table named `my_docs` with `id` and `text`
columns. Change the query to match your table.
```python
db = lancedb.connect("./.lancedb") # no SQL
db = lancedb.connect("db://my-database") # SQL
```
import lancedb
## Running SQL from Python {#sql-from-python}
db = lancedb.connect(
"db://your-database",
api_key="your-api-key",
region="us-east-1",
host_override="https://your-api-endpoint",
sql_host_override="https://your-sql-endpoint:443",
)
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
table = reader.read_all()
print(table.to_pylist())
```
`execute_query` returns a blocking Arrow reader once results start arriving; it
does not wait for the whole query. `execute_query_async` returns a handle
instead -- see [Queries](/basics/query).
In application code, load credentials from environment variables or a secrets
manager. The [connection guide](/basics/connect#enterprise) shows that setup.
## Choosing between them {#choosing}
`execute_query()` returns a blocking Arrow reader when results start arriving.
The example uses `read_all()` to collect the small result in memory. For larger
results, consume the reader in batches. To submit a query and check its status
separately, use `execute_query_async()`. See [Queries](/basics/query) for query
handles, streaming, and cancellation.
Reach for **the SDKs** when you are writing data, managing the catalog, running
vector search, or building an application that should work the same embedded and
deployed.
## Choose SQL or SDK table operations {#choosing}
Reach for **SQL** when the question is analytical -- joins across tables,
aggregates, window functions -- when the thing asking is a BI tool, a
notebook, or a person who already thinks in SQL, or when it's the only client
available for your language.
Use **SQL** for joins, aggregations, window functions, and analytical queries
written in SQL. Use **SDK table operations** to build applications around table
management, ingestion, filtering, and search, including applications that run
with the embedded OSS library.
The two compose. A common shape is SDK code to build and maintain tables, and
SQL for the analysis on top.
You can combine them. For example, create and populate a table with an SDK,
then query it with SQL to prepare a training dataset. In Python, both workflows
can use the same Enterprise connection.
## The complete reference {#reference}
### Common SQL operations {#sdk-parity}
This page is the concept; [SQL Reference](/sql) is the specification. It has
one page per statement, syntax first -- and since every SDK operation has a
SQL form, those pages end up defining the precise behavior of the SDK
operations too, not just the SQL dialect.
Several tasks are available through both interfaces, but the methods and SQL
statements do not have identical coverage. SDK support also differs by
language. These SQL reference pages describe the supported statements:
| Task | SQL reference |
| --- | --- |
| Create a table | [CREATE TABLE](/sql/ddl/table) |
| Insert or change rows | [INSERT](/sql/dml/insert), [UPDATE](/sql/dml/update), [DELETE](/sql/dml/delete) |
| Build an index | [CREATE INDEX](/sql/ddl/index) |
| Join and aggregate data | [SELECT](/sql/dql/select) |
| Save a query as a view | [CREATE VIEW](/sql/ddl/view) |
[Computed columns](/feature-engineering/add-computed-column) store SQL
expressions whose values are populated by a refresh.
They are separate from registered Python Functions. {/* if functions launched */}
See the computed-column and
[materialized-view](/management/materialized-view) guides for their SDK and SQL
operations and Enterprise setup requirements.
## Vector and full-text search {#no-vector-sql}
SQL vector search support depends on the Enterprise server version and query
form. Versions that support `ORDER BY distance(vector, query_vector) LIMIT k`
over a table scan can plan indexed or flat vector search. Check that your
server version supports this form, then use
[`EXPLAIN`](/sql/dql/explain) to inspect the query plan.
The [SDK vector search API](/search/vector-search) is also available in
Enterprise and OSS. Use it when you prefer the SDK query builder or need to
search embedded tables.
For full-text search in SQL, use the [`fts()`](/sql/functions#fts) table
function or the [`contains_tokens`](/sql/scalar-functions#contains-tokens)
predicate.
## SQL syntax and API references {#reference}
The [SQL reference](/sql) documents statement syntax, supported expressions,
and data types. For SDK methods and their arguments, use the
[Python](https://lancedb.github.io/lancedb/python/python/),
[TypeScript](https://lancedb.github.io/lancedb/js/), or
[Rust](https://docs.rs/lancedb/latest/lancedb/index.html) API reference.
<Card title="SQL Reference" icon="table-list" href="/sql">
DDL by object, then DML, then DQL -- the detailed technical documentation
that defines the behavior of every operation and SDK API.
Statement syntax, expressions, and data types for LanceDB SQL.
</Card>
## What to read next {#next}
## Next steps {#next}
<CardGroup cols={2}>
<Card title="SDKs" icon="code-compare" href="/basics/sdk">
The language clients, and what stays true across all of them.
<Card title="Queries" icon="magnifying-glass" href="/basics/query">
Read results, track long-running queries, and cancel work.
</Card>
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
Give a coding agent a current reference to this same API surface.
<Card title="SDKs" icon="code-compare" href="/basics/sdk">
Compare language clients and find their API references.
</Card>
</CardGroup>
docs/web/docs.json
+4 -5
View File
@@ -163,10 +163,6 @@
"source": "/storage/monitoring",
"destination": "/deployment/monitoring"
},
{
"source": "/enterprise/deployment",
"destination": "/enterprise/deployment/index"
},
{
"source": "/search/sql",
"destination": "/sql"
@@ -253,6 +249,8 @@
"group": "Getting Started",
"pages": [
"index",
"basics/offerings",
"basics/connect",
"quickstart",
"architecture",
"basics/sdk",
@@ -270,7 +268,8 @@
"basics/jobs",
"tables/consistency",
"basics/guarantees",
"basics/limits"
"basics/limits",
"basics/naming"
]
},
{
docs/web/quickstart.mdx
+175 -415
View File
@@ -1,96 +1,55 @@
---
title: Quickstart
sidebarTitle: "Quickstart"
description: "Get started with LanceDB in minutes."
description: "Install LanceDB, create a table, and run your first vector search."
icon: rocket
---
import {
PyConnect,
PyConnectAsync,
PyConnectEnterpriseQuickstart,
PyConnectObjectStorage,
PyConnectObjectStorageAsync,
RsConnect,
RsConnectEnterpriseQuickstart,
RsConnectObjectStorage,
TsConnect,
TsConnectEnterpriseQuickstart,
TsConnectObjectStorage,
} from '/snippets/connection.mdx';
import {
PyQuickstartData,
PyQuickstartComputedFeature,
PyQuickstartTraining,
PyQuickstartTrainingProjection,
PyQuickstartCreateTable,
PyQuickstartCreateTableAsync,
PyQuickstartAddFeature,
PyQuickstartCurateWithMetadata,
PyQuickstartMultimodalBytes,
PyQuickstartQueryFeature,
PyQuickstartVectorSearch1,
PyQuickstartVectorSearch1Async,
PyQuickstartOutputPandas,
RsQuickstartAddFeature,
RsQuickstartCurateWithMetadata,
RsQuickstartCreateTable,
RsQuickstartData,
RsQuickstartDefineStruct,
RsQuickstartMultimodalBytes,
RsQuickstartQueryFeature,
RsQuickstartVectorSearch1,
TsQuickstartAddFeature,
TsQuickstartCurateWithMetadata,
TsQuickstartCreateTable,
TsQuickstartData,
TsQuickstartMultimodalBytes,
TsQuickstartQueryFeature,
TsQuickstartVectorSearch1,
} from '/snippets/quickstart.mdx';
import { PyConnect, TsConnect, RsConnect, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartVectorSearch1, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1 } from '/snippets/quickstart.mdx';
As described in [the landing page](/), LanceDB provides one data layer for
curation, feature engineering, search and retrieval, and model training. Whether you are preparing
training data, building a RAG or agentic retrieval system, reviewing examples, or adding model-generated
features, you'll work with the same underlying table and search primitives.
This quickstart uses a local database and works without an account or API key.
Let's get started in just a few steps!
Follow the steps below, or jump to the [complete examples](#code) to copy a runnable program.
## 1\. Install LanceDB {#1-install-lancedb}
Install LanceDB in your client SDK.
Choose your language. Python uses Polars to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
<CodeGroup>
```bash pip icon="terminal"
pip install lancedb
```bash Python (pip) icon="python"
pip install lancedb polars
```
```bash uv icon="terminal"
uv add lancedb
# Or, in an existing virtual environment:
uv pip install lancedb
```bash Python (uv) icon="python"
uv venv
uv pip install lancedb polars
```
```bash TypeScript icon=js
```bash TypeScript icon="square-js"
npm install @lancedb/lancedb
```
```bash Rust icon=Rust
cargo add lancedb
```bash Rust icon="rust"
cargo init --name lancedb_quickstart
cargo add lancedb@0.31.0 --features polars,remote
cargo add tokio --features macros,rt-multi-thread
```
</CodeGroup>
Run the Rust commands in a new, empty directory. The `polars` feature provides DataFrame output; `remote` also lets this example connect to Enterprise.
<Accordion title="Python pre-release builds and older CPUs">
### Python pre-release builds {#python-pre-release-builds}
To pick up the latest features and bug fixes
before the next stable release, install a pre-release from LanceDB's Fury index.
<CodeGroup>
```bash pip icon="terminal"
```bash Python (pip) icon="terminal"
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
```
```bash uv icon="terminal"
```bash Python (uv) icon="terminal"
uv venv
uv pip install --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
@@ -105,436 +64,237 @@ for more than 6 months after release. For real-world workloads, we recommend you
as far as possible.
</Note>
<Accordion title="Running on older x86_64 CPUs without AVX2?">
### Older x86_64 CPUs {#older-x86-cpus}
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C). On older x86_64 CPUs without AVX2
(Intel Sandy Bridge, Ivy Bridge, Westmere; AMD Bulldozer, Piledriver, Steamroller), `import lancedb` crashes
with `Illegal instruction`.
Install `lancedb-compat` instead. It exposes the same API (`import lancedb` still works), is built at the
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still leverage AVX2, FMA, or AVX-512 when available:
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still use AVX2, FMA, or AVX-512 when available:
```bash pip icon="terminal"
```bash Python (pip) icon="terminal"
pip install lancedb-compat
```
The two packages share the same `lancedb/` namespace and conflict at install time, so pick one. To switch,
uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`).
</Accordion>
## 2\. Connect to a LanceDB database {#2-connect-to-a-lancedb-database}
## 2\. Connect to a database {#2-connect-to-a-lancedb-database}
LanceDB supports several URI patterns to connect to a database.
### Use a local directory {#connect-via-local-directory-path}
- A local filesystem path (when using it as an embedded library)
- A `db://...` URI (when using LanceDB Enterprise)
- An object storage URI: `s3://...`, `gs://...`, or `az://...` (when connecting directly from the client SDK)
LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed.
### Connect via local directory path {#connect-via-local-directory-path}
The simplest way to begin is to use LanceDB as an embedded library. Import LanceDB in your
client SDK of choice and point to a local directory path.
<CodeGroup >
<CodeBlock filename="Python (sync)" language="Python" icon="python">
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnect}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::connect;\n" }
{ "\n" }
{"use lancedb::connect;\n\n"}
{RsConnect}
</CodeBlock>
</CodeGroup >
</CodeGroup>
The [complete examples](#code) include the imports and entry points for each language. For Python's async API, see [Connecting to a database](/basics/connect).
### Connect via object storage URIs {#connect-via-object-storage-uris}
You can also connect directly to object storage from the client SDK:
<CodeGroup >
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectObjectStorageAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConnectObjectStorage}
</CodeBlock>
</CodeGroup >
For credentials, endpoints, and provider-specific options, see
[Configuring storage](/deployment/storage).
<Accordion title="Connect to LanceDB Enterprise instead">
### Connect to LanceDB Enterprise {#connect-to-lancedb-enterprise}
If you're using LanceDB Enterprise, you can connect to the remote database using the
`db://` URI along with the API key, region, and cluster endpoint you received from the
LanceDB team. Pass the cluster endpoint via `host_override` so the client routes
requests to your deployment.
Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`.
<CodeGroup >
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{ "import lancedb\n\n" }
{PyConnectEnterpriseQuickstart}
{PyConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterpriseQuickstart}
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::connect;\n\n" }
{RsConnectEnterpriseQuickstart}
{"use lancedb::connect;\n\n"}
{RsConnectEnterprisePlaceholders}
</CodeBlock>
</CodeGroup >
</CodeGroup>
<Note>
`host_override` is the full URL of your cluster endpoint, including the scheme
(`https://`) and a port if your deployment listens on a non-default one
(e.g. `https://your-enterprise-endpoint.com:443`). If you don't have the
endpoint, [contact the LanceDB team](https://www.lancedb.com/contact).
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/connect#enterprise) for an example.
</Note>
To learn more about `RemoteTable` semantics and how Enterprise differs operationally from
embedded LanceDB, see the [Enterprise overview](/enterprise).
## 3\. Create a new table {#3-create-a-new-table}
Let's create a small table of characters from the kingdom of Camelot. Each row stores source text,
metadata, structured fields, and a vector embedding in the same LanceDB table.
<Info>
The embeddings we use in this example are synthetic and for demonstration purposes only. In a real AI
data workflow, you would generate them from text, images, audio, or video using an embedding model of choice.
</Info>
Each row has source text, metadata, structured fields, and a vector:
```json
{
"id": "2",
"name": "Merlin",
"role": "Wizard",
"description": "Advisor and prophet with deep magical knowledge.",
"stats": { "strength": 2, "magic": 5, "leadership": 4, "wisdom": 5 },
"vector": [0.2, 0.9, 0.4, 0.9]
}
```
The full raw records are included below:
<Accordion title="View raw character records">
<CodeGroup >
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyQuickstartData}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartData}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartDefineStruct}
{RsQuickstartData}
</CodeBlock>
</CodeGroup >
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
</Accordion>
You can now create a LanceDB table from those records. The code below creates a LanceDB table
with the appropriate schema and ingests the data.
<Accordion title="Connect directly to object storage instead">
### Connect via object storage URIs {#connect-via-object-storage-uris}
<CodeGroup >
<CodeBlock filename="Python (sync)" language="Python" icon="python">
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Configuring storage](/deployment/storage) for credentials and provider-specific options.
</Accordion>
## 3\. Create a table and add data {#3-create-a-new-table}
Each row describes a character and stores a vector alongside their name, description, and stats.
<Info>
These vectors are made up for this example. In your application, use an embedding model to generate vectors from your data.
</Info>
<Accordion title="Character records">
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartData}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartData}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartDefineStruct}
{RsQuickstartData}
</CodeBlock>
</CodeGroup>
</Accordion>
Create a `characters` table from those records:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCreateTable}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyQuickstartCreateTableAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartCreateTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartCreateTable}
</CodeBlock>
</CodeGroup >
</CodeGroup>
## 4\. Semantic search {#4-semantic-search}
This example uses overwrite mode so you can run it again. It replaces any existing `characters` table in the database you connect to.
Search is a useful capability for all kinds of AI data pipelines. Below, we do a vector similarity
search for samples similar to a "_wise magical advisor_" (transforming the natural language query to
an embedding), and project only the columns needed by the next step.
Add one more character:
Search (which requires random access) is a ubiquitous access pattern that appears in many workloads:
whether you're building a RAG or recommendation system, serving agent memory, or curating a training
dataset.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddData}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartAddData}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartAddData}
</CodeBlock>
</CodeGroup>
<CodeGroup >
<CodeBlock filename="Python (sync)" language="Python" icon="python">
## 4\. Search the table {#4-semantic-search}
Find the two vectors closest to the query vector. This example uses a fixed vector; it does not call an embedding model.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartVectorSearch1}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyQuickstartVectorSearch1Async}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartVectorSearch1}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartVectorSearch1}
</CodeBlock>
</CodeGroup >
</CodeGroup>
The example for Python above shows how to convert results to a Polars DataFrame.
Depending on your language, you can collect query results as a list/array of objects or DataFrames
to be used downstream in your application.
Merlin and Morgana come back first. Python and Rust display a Polars DataFrame; TypeScript displays an array of results. The `_distance` column measures how far each result is from your query vector, with smaller values indicating a closer match.
<Accordion title="Pandas users in Python can get results as a Pandas DataFrame">
Use the `to_pandas()` method to convert query results into a Pandas DataFrame.
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartOutputPandas}
</CodeBlock>
## Complete examples {#code}
Each example below creates the local database, adds four characters, and prints the two closest matches. Use the dependencies from step 1.
<Accordion title="Python: quickstart.py">
Save this as `quickstart.py` and run `python quickstart.py`. If you installed with uv, run `uv run quickstart.py`.
<CodeGroup>
<CodeBlock filename="quickstart.py" language="Python" icon="python">
{PyConnect}
{PyQuickstartData}
{PyQuickstartCreateTable}
{PyQuickstartAddData}
{PyQuickstartVectorSearch1}
</CodeBlock>
</CodeGroup>
</Accordion>
## 5\. Curation {#5-curation}
<Accordion title="TypeScript: quickstart.mts">
Save this as `quickstart.mts` and run `node quickstart.mts` from the directory where you installed LanceDB.
Searching for relevant results can be more useful when combined with metadata filters.
In this tiny example, we filter to examples with high `magic` stats.
<CodeGroup >
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCurateWithMetadata}
<CodeGroup>
<CodeBlock filename="quickstart.mts" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnect}
{TsQuickstartData}
{TsQuickstartCreateTable}
{TsQuickstartAddData}
{TsQuickstartVectorSearch1}
</CodeBlock>
</CodeGroup>
</Accordion>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartCurateWithMetadata}
<Accordion title="Rust: src/main.rs">
Replace `src/main.rs` with this program, then run `cargo run` from your project directory. The schema and Arrow conversion helpers are included.
<CodeGroup>
<CodeBlock filename="src/main.rs" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsQuickstartDefineStruct}
{"\n#[tokio::main]\nasync fn main() -> Result<(), Box<dyn std::error::Error>> {\n"}
{RsConnect}
{RsQuickstartData}
{RsQuickstartCreateTable}
{RsQuickstartAddData}
{RsQuickstartVectorSearch1}
{"\nOk(())\n}\n"}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartCurateWithMetadata}
</CodeBlock>
</CodeGroup >
When working with large datasets, it's common to use the same pattern to filter on quality labels,
train/eval splits, numeric fields, categorical values, timestamp windows, or generated tags and labels.
## 6\. Add a derived feature {#6-add-a-derived-feature}
Feature engineering is the process of cleaning up your data and creating new signals that
help your model learn, make better predictions, or your agent retrieve more useful information.
In the example below, we add a `power_score` column from the structured `stats` fields.
Lance supports data evolution, so you can add new columns without rewriting the entire table.
<CodeGroup >
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddFeature}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartAddFeature}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::table::NewColumnTransform;\n\n" }
{RsQuickstartAddFeature}
</CodeBlock>
</CodeGroup >
Next, you can query a compact view of the new feature:
<CodeGroup >
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartQueryFeature}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartQueryFeature}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartQueryFeature}
</CodeBlock>
</CodeGroup >
| name | role | power_score |
| --- | --- | --- |
| King Arthur | King | 3.5 |
| Merlin | Wizard | 4.0 |
| Sir Lancelot | Knight | 3.0 |
The same workflow is used for data preparation tasks when adding derived features, cached model signals, review scores, or dataset
quality indicators.
### Declare now, compute later {#computed-columns}
The call above computes every row as part of the call, which is fine for three
rows. On a table large enough that you would rather choose *when* the work
happens, declare a column instead and fill it separately. The stored `power_score`
column stays available; declare a separate
`combat_score` column from magic and strength, then refresh it:
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartComputedFeature}
</CodeBlock>
A refresh gives values to rows that have none and recomputes only rows whose
inputs changed, so appending more characters and refreshing again fills just
the new ones -- it does not recompute the rows that are already current. That
is what makes a backfill safe to retry and cheap to repeat as data arrives.
On a deployment the same refresh runs server-side, on a cluster, as a
[job](/basics/jobs) you can walk away from:
```python
job = table.refresh_column_async("combat_score")
job.wait()
```
See [Computed columns](/feature-engineering/add-computed-column) for expressions,
and [Functions](/management/function) for filling a column with Python
-- an embedding model, a captioner, anything you can register.
## 7\. Store multimodal data {#7-store-multimodal-data}
Multimodal data is a first-class citizen in LanceDB. Binary data (image, audio, video, etc.) is
stored as blobs or inline Arrow binary types in a LanceDB column, and they benefit from the same
table operations and data versioning semantics as other data types. All the data is governed
in the same table, so you can search, filter, and retrieve multimodal records together with structured
fields, metadata, and embeddings.
In this example, the
[`lancedb/magical_kingdom`](https://huggingface.co/datasets/lancedb/magical_kingdom) dataset stores
character images, descriptions, structured stats, image embeddings, and text embeddings together.
Say we downloaded the image for Sir Lancelot from that dataset locally. You can read the image bytes
in your client SDK and store them in a LanceDB column. The image bytes can be used for downstream tasks
like retrieval, evaluation, or training.
<div style={{ display: "flex", justifyContent: "center" }}>
<img
src="/static/assets/images/quickstart/sir-lancelot.jpg"
alt="Sir Lancelot from the lancedb/magical_kingdom dataset"
width="320"
/>
</div>
These snippets load the local image file and store the bytes in an `image` column:
<CodeGroup >
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartMultimodalBytes}
</CodeBlock>
</CodeGroup >
For more examples, see [Data types](/management/table#multimodal).
## 8\. Train on the result {#8-train}
The point of the previous steps is this one. The table you searched, filtered,
and added features to is directly usable as a PyTorch
[Dataset](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset)
-- there is no export step, and no second copy of the data to keep in sync.
Install PyTorch with `pip install torch`. Select the numeric feature columns
before constructing the loader; the full table also contains strings,
structured stats, and vectors.
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartTraining}
</CodeBlock>
```python
for batch in dataloader:
... # train with batch["power_score"] and batch["combat_score"]
```
Because the loader reads from the table, the feature columns you just
materialized are read straight off storage rather than recomputed every epoch.
That is the whole argument for doing feature engineering in the lakehouse: an
expensive transform runs once, is versioned with the data, and every subsequent
epoch is a read.
Projection matters here. If a training step only needs `power_score`, select
that column so the descriptions, stats, and vectors are excluded:
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartTrainingProjection}
</CodeBlock>
See [Training](/training/data-loading) for shuffling, multi-worker loading, and reading from
a remote table.
## Code {#code}
See the full code for these examples (including helper functions) in the
`quickstart` file for the appropriate client language in the
[files provided in the repo](https://github.com/lancedb/docs/tree/main/tests).
</CodeGroup>
</Accordion>
## Next steps {#next-steps}
You've gone from a table, through search and curation, to derived features and a
training loop -- in one table, with no export step between any two stages. That
arc is the argument for a multimodal lakehouse, and every stage of it scales
past this toy example without changing shape.
Continue to the guides to build on it with schema options, versioning, indexing,
full-text and hybrid search, reranking, and server-side feature engineering.
You've created a table, added rows, and searched it. Continue with the guides below.
### Filter results {#5-curation}
Combine vector search with metadata filters, such as `stats.magic >= 4`. See [SQL filtering](/search/filtering).
### Extend your table {#extensions}
#### Add a derived feature {#6-add-a-derived-feature}
Add columns calculated from existing fields. See [Adding columns](/management/table#schema-add-new-columns).
#### Computed columns {#computed-columns}
Declare a column and choose when to fill or refresh it. See [Computed columns](/feature-engineering/add-computed-column).
#### Store multimodal data {#7-store-multimodal-data}
Store images, audio, and other binary data alongside vectors and metadata. See [Multimodal data types](/management/table#multimodal).
#### Train on your data {#8-train}
Read table data into PyTorch for training. See [Data loading](/training/data-loading).
<Columns cols={2}>
<Card
title="Basic table operations"
icon="table"
href="/management/table"
>
Build on this quickstart with table creation, updates, and schema tips.
<Card title="Basic table operations" icon="table" href="/management/table">
Create, update, and manage your tables.
</Card>
<Card
title="Build a RAG App"
icon="search"
href="/tutorials/agents/contextual-rag"
>
Learn how to build Retrieval-Augmented Generation (RAG) applications using LanceDB.
<Card title="Build a RAG app" icon="search" href="/tutorials/agents/contextual-rag">
Use retrieved context in a language model application.
</Card>
<Card
title="Indexing"
icon="layer-group"
href="/indexing/"
>
Create vector, full-text, and scalar indexes to speed up queries on larger datasets.
<Card title="Indexing" icon="layer-group" href="/indexing/">
Speed up queries on larger datasets.
</Card>
<Card
title="Data loading and shuffles"
icon="boxes-stacked"
href="/training/data-loading"
>
Use LanceDB for projected, shuffled, random-access reads in training workflows.
<Card title="Embedding models" icon="vector-square" href="/embedding/">
Generate vectors from your own data.
</Card>
</Columns>
docs/web/snippets/connection.mdx
+15 -9
View File
@@ -4,31 +4,37 @@ export const PyConnect = "import lancedb\n\nuri = \"ex_lancedb\"\ndb = lancedb.c
export const PyConnectAsync = "import lancedb\n\nuri = \"ex_lancedb\"\nasync_db = await lancedb.connect_async(uri)\n";
export const PyConnectEnterpriseQuickstart = "uri = \"db://your-database-uri\"\napi_key = \"your-api-key\"\nregion = \"us-east-1\"\nhost_override = \"https://your-enterprise-endpoint.com\"\n\ndb = lancedb.connect(\n uri=uri,\n api_key=api_key,\n region=region,\n host_override=host_override,\n)\n";
export const PyConnectEnterprisePlaceholders = "import lancedb\n\ndb = lancedb.connect(\n uri=\"db://your-database\",\n api_key=\"your-api-key\",\n region=\"us-east-1\",\n host_override=\"https://your-endpoint\",\n)\n";
export const PyConnectEnterpriseQuickstart = "import os\n\nimport lancedb\n\ndb = lancedb.connect(\n uri=os.environ[\"LANCEDB_URI\"], # db://your-database\n api_key=os.environ[\"LANCEDB_API_KEY\"],\n region=os.environ.get(\"LANCEDB_REGION\", \"us-east-1\"),\n host_override=os.environ[\"LANCEDB_HOST_OVERRIDE\"], # https://your-endpoint\n)\n";
export const PyConnectObjectStorage = "import lancedb\n\nuri = \"s3://your-bucket/path\"\n# You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\ndb = lancedb.connect(uri)\n";
export const PyConnectObjectStorageAsync = "import lancedb\n\nuri = \"s3://your-bucket/path\"\n# You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\nasync_db = await lancedb.connect_async(uri)\n";
export const PyNamespaceAdminOps = "import lancedb\n\ndb = lancedb.connect_namespace(\"dir\", {\"root\": \"./local_lancedb\"})\nnamespace = [\"prod\", \"search\"]\n\ndb.create_namespace([\"prod\"])\ndb.create_namespace([\"prod\", \"search\"])\n\nchild_namespaces = db.list_namespaces(namespace_path=[\"prod\"]).namespaces\nprint(f\"Child namespaces under {namespace}: {child_namespaces}\")\n# Child namespaces under ['prod', 'search']: ['search']\n\nmetadata = db.describe_namespace([\"prod\", \"search\"])\nprint(f\"Metadata for namespace {namespace}: {metadata}\")\n# Metadata for namespace ['prod', 'search']: properties=None\n\ndb.drop_namespace([\"prod\", \"search\"], mode=\"skip\")\ndb.drop_namespace([\"prod\"], mode=\"skip\")\n";
export const PyNamespaceAdminOps = "import lancedb\n\ndb = lancedb.connect_namespace(\"dir\", {\"root\": \"./local_lancedb\"})\nnamespace = [\"prod\", \"search\"]\n\ndb.create_namespace([\"prod\"])\ndb.create_namespace([\"prod\", \"search\"])\n\nchild_namespaces = db.list_namespaces(namespace_path=[\"prod\"]).namespaces\nprint(f\"Child namespaces under ['prod']: {child_namespaces}\")\n# Child namespaces under ['prod']: ['search']\n\nmetadata = db.describe_namespace([\"prod\", \"search\"])\nprint(f\"Metadata for namespace {namespace}: {metadata}\")\n# Metadata for namespace ['prod', 'search']: properties=None\n\ndb.drop_namespace([\"prod\", \"search\"], mode=\"skip\")\ndb.drop_namespace([\"prod\"], mode=\"skip\")\n";
export const PyNamespaceTableOps = "import lancedb\n\ndb = lancedb.connect_namespace(\"dir\", {\"root\": \"./local_lancedb\"})\n\n# Create namespace tree: prod/search\ndb.create_namespace([\"prod\"], mode=\"exist_ok\")\ndb.create_namespace([\"prod\", \"search\"], mode=\"exist_ok\")\ndb.create_namespace([\"prod\", \"recommendations\"], mode=\"exist_ok\")\n\ndb.create_table(\n \"user\",\n data=[{\"id\": 1, \"vector\": [0.1, 0.2], \"name\": \"alice\"}],\n namespace_path=[\"prod\", \"search\"],\n mode=\"create\", # use \"overwrite\" only if you want to replace existing table\n)\n\ndb.create_table(\n \"user\",\n data=[{\"id\": 2, \"vector\": [0.3, 0.4], \"name\": \"bob\"}],\n namespace_path=[\"prod\", \"recommendations\"],\n mode=\"create\", # use \"overwrite\" only if you want to replace existing table\n)\n\n# Verify\nprint(db.list_namespaces()) # ['prod']\nprint(db.list_namespaces(namespace_path=[\"prod\"])) # ['recommendations', 'search']\nprint(db.list_tables(namespace_path=[\"prod\", \"search\"])) # ['user']\nprint(db.list_tables(namespace_path=[\"prod\", \"recommendations\"])) # ['user']\n";
export const PyNamespaceTableOps = "import lancedb\n\ndb = lancedb.connect_namespace(\"dir\", {\"root\": \"./local_lancedb\"})\n\n# Create namespace tree: prod/search\ndb.create_namespace([\"prod\"], mode=\"exist_ok\")\ndb.create_namespace([\"prod\", \"search\"], mode=\"exist_ok\")\ndb.create_namespace([\"prod\", \"recommendations\"], mode=\"exist_ok\")\n\ndb.create_table(\n \"user\",\n data=[{\"id\": 1, \"vector\": [0.1, 0.2], \"name\": \"alice\"}],\n namespace_path=[\"prod\", \"search\"],\n mode=\"create\", # use \"overwrite\" only if you want to replace existing table\n)\n\ndb.create_table(\n \"user\",\n data=[{\"id\": 2, \"vector\": [0.3, 0.4], \"name\": \"bob\"}],\n namespace_path=[\"prod\", \"recommendations\"],\n mode=\"create\", # use \"overwrite\" only if you want to replace existing table\n)\n\n# Verify: listings return a response with the names and a page token\nprint(db.list_namespaces().namespaces) # ['prod']\nprint(db.list_namespaces(namespace_path=[\"prod\"]).namespaces)\n# ['recommendations', 'search']\nprint(db.list_tables(namespace_path=[\"prod\", \"search\"]).tables) # ['user']\nprint(db.list_tables(namespace_path=[\"prod\", \"recommendations\"]).tables) # ['user']\n";
export const TsConnect = "import * as lancedb from \"@lancedb/lancedb\";\n\nasync function connectExample(uri: string) {\n const db = await lancedb.connect(uri);\n return db;\n}\n";
export const TsConnect = "const db = await lancedb.connect(\"ex_lancedb\");\n";
export const TsConnectEnterpriseQuickstart = "const uri = \"db://your-database-uri\";\nconst apiKey = \"your-api-key\";\nconst region = \"us-east-1\";\nconst hostOverride = \"https://your-enterprise-endpoint.com\";\n\nconst db = await lancedb.connect(uri, {\n apiKey,\n region,\n hostOverride,\n});\n";
export const TsConnectEnterprisePlaceholders = "const db = await lancedb.connect(\"db://your-database\", {\n apiKey: \"your-api-key\",\n region: \"us-east-1\",\n hostOverride: \"https://your-endpoint\",\n});\n";
export const TsConnectObjectStorage = "async function connectObjectStorageExample() {\n const uri = \"s3://your-bucket/path\";\n // You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\n const db = await lancedb.connect(uri);\n return db;\n}\n";
export const TsConnectEnterpriseQuickstart = "const setting = (name: string) => {\n const value = process.env[name];\n if (!value) throw new Error(`Set ${name}`);\n return value;\n};\n\nconst db = await lancedb.connect(setting(\"LANCEDB_URI\"), {\n apiKey: setting(\"LANCEDB_API_KEY\"),\n region: process.env.LANCEDB_REGION ?? \"us-east-1\",\n hostOverride: setting(\"LANCEDB_HOST_OVERRIDE\"),\n});\n";
export const TsConnectObjectStorage = "const uri = \"s3://your-bucket/path\";\n// You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\nconst db = await lancedb.connect(uri);\n";
export const TsNamespaceAdminOps = "const db = await lancedb.connectNamespace(\"dir\", { root: \"./local_lancedb\" });\nconst namespace = [\"prod\", \"search\"];\n\nawait db.createNamespace([\"prod\"]);\nawait db.createNamespace([\"prod\", \"search\"]);\n\nconst childNamespaces = (await db.listNamespaces([\"prod\"])).namespaces;\nconsole.log(`Child namespaces under ${JSON.stringify(namespace)}:`, childNamespaces);\n// Child namespaces under [\"prod\",\"search\"]: [ 'search' ]\n\nconst metadata = await db.describeNamespace([\"prod\", \"search\"]);\nconsole.log(`Metadata for namespace ${JSON.stringify(namespace)}:`, metadata);\n\nawait db.dropNamespace([\"prod\", \"search\"], { mode: \"skip\" });\nawait db.dropNamespace([\"prod\"], { mode: \"skip\" });\n";
export const TsNamespaceTableOps = "const db = await lancedb.connectNamespace(\"dir\", { root: \"./local_lancedb\" });\n\n// Create namespace tree: prod/search and prod/recommendations\nawait db.createNamespace([\"prod\"], { mode: \"exist_ok\" });\nawait db.createNamespace([\"prod\", \"search\"], { mode: \"exist_ok\" });\nawait db.createNamespace([\"prod\", \"recommendations\"], { mode: \"exist_ok\" });\n\nawait db.createTable(\n \"user\",\n [{ id: 1, vector: [0.1, 0.2], name: \"alice\" }],\n [\"prod\", \"search\"],\n { mode: \"create\" }, // use \"overwrite\" only if you want to replace existing table\n);\n\nawait db.createTable(\n \"user\",\n [{ id: 2, vector: [0.3, 0.4], name: \"bob\" }],\n [\"prod\", \"recommendations\"],\n { mode: \"create\" },\n);\n\n// Verify\nconsole.log((await db.listNamespaces()).namespaces); // [\"prod\"]\nconsole.log((await db.listNamespaces([\"prod\"])).namespaces); // [\"recommendations\", \"search\"]\nconsole.log(await db.tableNames([\"prod\", \"search\"])); // [\"user\"]\nconsole.log(await db.tableNames([\"prod\", \"recommendations\"])); // [\"user\"]\n";
export const RsConnect = "async fn connect_example(uri: &str) {\n let db = connect(uri).execute().await.unwrap();\n let _ = db;\n}\n";
export const RsConnect = "let db = connect(\"ex_lancedb\").execute().await?;\n";
export const RsConnectEnterpriseQuickstart = "let uri = \"db://your-database-uri\";\nlet api_key = \"your-api-key\";\nlet region = \"us-east-1\";\nlet host_override = \"https://your-enterprise-endpoint.com\";\n";
export const RsConnectEnterprisePlaceholders = "let db = connect(\"db://your-database\")\n .api_key(\"your-api-key\")\n .region(\"us-east-1\")\n .host_override(\"https://your-endpoint\")\n .execute()\n .await?;\n";
export const RsConnectObjectStorage = "let uri = \"s3://your-bucket/path\";\n// You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\n";
export const RsConnectEnterpriseQuickstart = "let uri = std::env::var(\"LANCEDB_URI\")?; // db://your-database\nlet api_key = std::env::var(\"LANCEDB_API_KEY\")?;\nlet region = std::env::var(\"LANCEDB_REGION\").unwrap_or_else(|_| \"us-east-1\".to_string());\nlet host_override = std::env::var(\"LANCEDB_HOST_OVERRIDE\")?; // https://your-endpoint\n\nlet db = connect(&uri)\n .api_key(&api_key)\n .region(&region)\n .host_override(&host_override)\n .execute()\n .await?;\n";
export const RsConnectObjectStorage = "let uri = \"s3://your-bucket/path\";\n// You can also use \"gs://your-bucket/path\" or \"az://your-container/path\".\nlet db = connect(uri).execute().await?;\n";
export const RsNamespaceAdminOps = "let mut properties = std::collections::HashMap::new();\nproperties.insert(\"root\".to_string(), \"./local_lancedb\".to_string());\nlet db = lancedb::connect_namespace(\"dir\", properties).execute().await?;\nlet namespace = vec![\"prod\".to_string(), \"search\".to_string()];\n\ndb.create_namespace(lance_namespace::models::CreateNamespaceRequest {\n id: Some(vec![\"prod\".to_string()]),\n ..Default::default()\n})\n.await?;\ndb.create_namespace(lance_namespace::models::CreateNamespaceRequest {\n id: Some(namespace.clone()),\n ..Default::default()\n})\n.await?;\n\nlet child_namespaces = db\n .list_namespaces(lance_namespace::models::ListNamespacesRequest {\n id: Some(vec![\"prod\".to_string()]),\n ..Default::default()\n })\n .await?;\nprintln!(\n \"Child namespaces under {:?}: {:?}\",\n namespace, child_namespaces\n);\n// Child namespaces under [\"prod\", \"search\"]: [\"search\"]\n\ndb.drop_namespace(lance_namespace::models::DropNamespaceRequest {\n id: Some(namespace.clone()),\n ..Default::default()\n})\n.await?;\ndb.drop_namespace(lance_namespace::models::DropNamespaceRequest {\n id: Some(vec![\"prod\".to_string()]),\n ..Default::default()\n})\n.await?;\n";
docs/web/snippets/quickstart.mdx
+18 -34
View File
@@ -1,82 +1,66 @@
{/* Auto-generated by scripts/mdx_snippets_gen.py. Do not edit manually. */}
export const PyQuickstartAddData = "more_data = [\n {\n \"id\": \"4\",\n \"name\": \"Morgana\",\n \"role\": \"Sorceress\",\n \"description\": \"Powerful sorceress of Avalon.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 4},\n \"vector\": [0.3, 0.9, 0.6, 0.8],\n \"power_score\": 3.75,\n },\n]\n\n# Add data to table\ntable.add(more_data)\n";
export const PyQuickstartAddData = "table.add(\n [\n {\n \"id\": \"4\",\n \"name\": \"Morgana\",\n \"role\": \"Sorceress\",\n \"description\": \"Powerful sorceress of Avalon.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 4},\n \"vector\": [0.3, 0.9, 0.6, 0.8],\n }\n ]\n)\n";
export const PyQuickstartAddFeature = "table.add_columns(\n {\n \"power_score\": \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n }\n)\n";
export const PyQuickstartComputedFeature = "table.add_columns(computed={\"combat_score\": \"(stats.magic + stats.strength) / 2.0\"})\ntable.refresh_column(\"combat_score\")\n";
export const PyQuickstartComputedFeature = "table.add_columns(computed={\"description_length\": \"length(description)\"})\ntable.refresh_column(\"description_length\")\n";
export const PyQuickstartComputedFeatureEnterprise = "table.add_columns(computed={\"description_length\": \"length(description)\"})\njob = table.refresh_column_async(\"description_length\")\njob.wait()\n";
export const PyQuickstartCreateTable = "table = db.create_table(\"characters\", data=data, mode=\"overwrite\")\n";
export const PyQuickstartCreateTableAsync = "async_table = await async_db.create_table(\n \"characters\",\n data=data,\n mode=\"overwrite\",\n)\n";
export const PyQuickstartCreateTableNoOverwrite = "table = db.create_table(\"characters\", data=data)\n";
export const PyQuickstartCurateWithMetadata = "curated = (\n table.search(query_vector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(curated)\n";
export const PyQuickstartData = "data = [\n {\n \"id\": \"1\",\n \"name\": \"King Arthur\",\n \"role\": \"King\",\n \"description\": \"Leader of Camelot and wielder of Excalibur.\",\n \"stats\": {\"strength\": 4, \"magic\": 1, \"leadership\": 5, \"wisdom\": 4},\n \"vector\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"id\": \"2\",\n \"name\": \"Merlin\",\n \"role\": \"Wizard\",\n \"description\": \"Advisor and prophet with deep magical knowledge.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 5},\n \"vector\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"id\": \"3\",\n \"name\": \"Sir Lancelot\",\n \"role\": \"Knight\",\n \"description\": \"Legendary knight known for courage and combat skill.\",\n \"stats\": {\"strength\": 5, \"magic\": 1, \"leadership\": 3, \"wisdom\": 3},\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n";
export const PyQuickstartDataAsync = "data = [\n {\n \"id\": \"1\",\n \"name\": \"King Arthur\",\n \"role\": \"King\",\n \"description\": \"Leader of Camelot and wielder of Excalibur.\",\n \"stats\": {\"strength\": 4, \"magic\": 1, \"leadership\": 5, \"wisdom\": 4},\n \"vector\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"id\": \"2\",\n \"name\": \"Merlin\",\n \"role\": \"Wizard\",\n \"description\": \"Advisor and prophet with deep magical knowledge.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 5},\n \"vector\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"id\": \"3\",\n \"name\": \"Sir Lancelot\",\n \"role\": \"Knight\",\n \"description\": \"Legendary knight known for courage and combat skill.\",\n \"stats\": {\"strength\": 5, \"magic\": 1, \"leadership\": 3, \"wisdom\": 3},\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n";
export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_path = Path(\"docs/web/static/assets/images/quickstart/sir-lancelot.jpg\")\nimage_bytes = image_path.read_bytes()\n\nmultimodal_table = db.create_table(\n \"character_images\",\n data=[\n {\n \"id\": \"lancelot\",\n \"description\": \"Portrait of Sir Lancelot\",\n \"image\": image_bytes,\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n";
export const PyQuickstartOpenTable = "table = db.open_table(\"characters\")\n";
export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_bytes = Path(\"data/sir-lancelot.jpg\").read_bytes()\n\nmultimodal_table = db.create_table(\n \"character_images\",\n data=[\n {\n \"id\": \"lancelot\",\n \"description\": \"Portrait of Sir Lancelot\",\n \"image\": image_bytes,\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n";
export const PyQuickstartOutputPandas = "# Ensure you run `pip install pandas` beforehand\nresult = table.search(query_vector).limit(2).to_pandas()\nprint(result)\n";
export const PyQuickstartQueryFeature = "features = table.search().select([\"name\", \"role\", \"power_score\"]).to_polars()\nprint(features)\n";
export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ndataset = Permutation.identity(table).select_columns(\n [\"power_score\", \"combat_score\"]\n)\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n";
export const PyQuickstartTrainingProjection = "dataset = Permutation.identity(table).select_columns([\"power_score\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024)\n";
export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ntable.add_columns(\n {\n \"magic\": \"cast(stats.magic as float)\",\n \"strength\": \"cast(stats.strength as float)\",\n }\n)\ndataset = Permutation.identity(table).select_columns([\"magic\", \"strength\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n";
export const PyQuickstartVectorSearch1 = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nresult = (\n table.search(query_vector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(result)\n";
export const PyQuickstartVectorSearch1Async = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nasync_result = await (\n await async_table.search(query_vector)\n).select([\"name\", \"role\", \"description\", \"_distance\"]).limit(2).to_polars()\nprint(async_result)\n";
export const PyQuickstartVectorSearch2 = "# Search for examples similar to a \"powerful sorceress\"\nquery_vector = [0.3, 0.9, 0.6, 0.8]\n\nresults = table.search(query_vector).limit(2).to_polars()\nprint(results)\n";
export const TsQuickstartAddData = "const moreData = [\n {\n id: \"4\",\n name: \"Morgana\",\n role: \"Sorceress\",\n description: \"Powerful sorceress of Avalon.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },\n vector: [0.3, 0.9, 0.6, 0.8],\n power_score: 3.75,\n },\n];\n\n// Add data to table\nawait table.add(moreData);\n";
export const TsQuickstartAddData = "await table.add([\n {\n id: \"4\",\n name: \"Morgana\",\n role: \"Sorceress\",\n description: \"Powerful sorceress of Avalon.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },\n vector: [0.3, 0.9, 0.6, 0.8],\n },\n]);\n";
export const TsQuickstartAddFeature = "await table.addColumns([\n {\n name: \"power_score\",\n valueSql:\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\",\n },\n]);\n";
export const TsQuickstartCreateTable = "let table = await db.createTable(\"characters\", data, { mode: \"overwrite\" });\n";
export const TsQuickstartCreateTableNoOverwrite = "table = await db.createTable(\"characters\", data);\n";
export const TsQuickstartCreateTable = "const table = await db.createTable(\"characters\", data, { mode: \"overwrite\" });\n";
export const TsQuickstartCurateWithMetadata = "const curated = await table\n .search(queryVector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(curated);\n";
export const TsQuickstartData = "const data = [\n {\n id: \"1\",\n name: \"King Arthur\",\n role: \"King\",\n description: \"Leader of Camelot and wielder of Excalibur.\",\n stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n {\n id: \"2\",\n name: \"Merlin\",\n role: \"Wizard\",\n description: \"Advisor and prophet with deep magical knowledge.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n {\n id: \"3\",\n name: \"Sir Lancelot\",\n role: \"Knight\",\n description: \"Legendary knight known for courage and combat skill.\",\n stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst path = await import(\"node:path\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imagePath = path.join(\n __dirname,\n \"../../web/static/assets/images/quickstart/sir-lancelot.jpg\",\n);\nconst imageBytes = await readFile(imagePath);\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"id\", new arrow.Utf8()),\n new arrow.Field(\"description\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"vector\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n id: \"lancelot\",\n description: \"Portrait of Sir Lancelot\",\n image: imageBytes,\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"character_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n";
export const TsQuickstartOpenTable = "table = await db.openTable(\"characters\");\n";
export const TsQuickstartOutputArray = "result = await table.search(queryVector).limit(2).toArray();\nconsole.table(result);\n";
export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imageBytes = await readFile(\"data/sir-lancelot.jpg\");\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"id\", new arrow.Utf8()),\n new arrow.Field(\"description\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"vector\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n id: \"lancelot\",\n description: \"Portrait of Sir Lancelot\",\n image: imageBytes,\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"character_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n";
export const TsQuickstartQueryFeature = "const features = await table\n .query()\n .select([\"name\", \"role\", \"power_score\"])\n .toArray();\nconsole.table(features);\n";
export const TsQuickstartVectorSearch1 = "// Search for examples similar to a \"wise magical advisor\"\nlet queryVector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = await table\n .search(queryVector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n";
export const TsQuickstartVectorSearch1 = "// Search for examples similar to a \"wise magical advisor\"\nconst queryVector = [0.2, 0.8, 0.4, 0.9];\n\nconst result = await table\n .search(queryVector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n";
export const TsQuickstartVectorSearch2 = "// Search for examples similar to a \"powerful sorceress\"\nqueryVector = [0.3, 0.9, 0.6, 0.8];\n\nconst results = await table.search(queryVector).limit(2).toArray();\nconsole.table(results);\n";
export const RsQuickstartAddData = "let more_data = vec![Character {\n id: \"4\".to_string(),\n name: \"Morgana\".to_string(),\n role: \"Sorceress\".to_string(),\n description: \"Powerful sorceress of Avalon.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 4,\n },\n vector: [0.3, 0.9, 0.6, 0.8],\n}];\ntable\n .add(characters_to_reader(schema.clone(), &more_data))\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartAddFeature = "table\n .add_columns(\n NewColumnTransform::SqlExpressions(vec![(\n \"power_score\".to_string(),\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n .to_string(),\n )]),\n None,\n )\n .await\n .unwrap();\n";
export const RsQuickstartCreateTable = "let schema = characters_schema();\nlet table = db\n .create_table(\"characters\", characters_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartCreateTable = "use lancedb::database::CreateTableMode;\n\nlet schema = characters_schema();\nlet table = db\n .create_table(\"characters\", characters_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartCreateTableNoOverwrite = "let table = db\n .create_table(\"characters\", characters_to_reader(schema.clone(), &data))\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartCurateWithMetadata = "let curated: DataFrame = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.magic >= 4\")\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n";
export const RsQuickstartCurateWithMetadata = "let curated = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.magic >= 4\")\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n";
export const RsQuickstartData = "let data = vec![\n Character {\n id: \"1\".to_string(),\n name: \"King Arthur\".to_string(),\n role: \"King\".to_string(),\n description: \"Leader of Camelot and wielder of Excalibur.\".to_string(),\n stats: Stats {\n strength: 4,\n magic: 1,\n leadership: 5,\n wisdom: 4,\n },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n Character {\n id: \"2\".to_string(),\n name: \"Merlin\".to_string(),\n role: \"Wizard\".to_string(),\n description: \"Advisor and prophet with deep magical knowledge.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 5,\n },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n Character {\n id: \"3\".to_string(),\n name: \"Sir Lancelot\".to_string(),\n role: \"Knight\".to_string(),\n description: \"Legendary knight known for courage and combat skill.\".to_string(),\n stats: Stats {\n strength: 5,\n magic: 1,\n leadership: 3,\n wisdom: 3,\n },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const RsQuickstartDefineStruct = "// Define structs representing the data schema\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct Stats {\n strength: i8,\n magic: i8,\n leadership: i8,\n wisdom: i8,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct Character {\n id: String,\n name: String,\n role: String,\n description: String,\n stats: Stats,\n vector: [f32; 4],\n}\n\nfn characters_schema() -> Arc<Schema> {\n Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"name\", DataType::LargeUtf8, false),\n Field::new(\"role\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(arrow_schema::Fields::from(vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n";
export const RsQuickstartDefineStruct = "use std::sync::Arc;\n\nuse lancedb::arrow::arrow_array::types::Float32Type;\nuse lancedb::arrow::arrow_array::{\n FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,\n RecordBatchReader, StructArray,\n};\nuse lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};\n\n// Define structs representing the data schema\n#[derive(Debug, Clone)]\nstruct Stats {\n strength: i8,\n magic: i8,\n leadership: i8,\n wisdom: i8,\n}\n\n#[derive(Debug, Clone)]\nstruct Character {\n id: String,\n name: String,\n role: String,\n description: String,\n stats: Stats,\n vector: [f32; 4],\n}\n\nfn characters_schema() -> Arc<Schema> {\n Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"name\", DataType::LargeUtf8, false),\n Field::new(\"role\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(Fields::from(vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n\nfn characters_to_reader(\n schema: Arc<Schema>,\n rows: &[Character],\n) -> Box<dyn RecordBatchReader + Send> {\n let ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.id.as_str()));\n let names = LargeStringArray::from_iter_values(rows.iter().map(|row| row.name.as_str()));\n let roles = LargeStringArray::from_iter_values(rows.iter().map(|row| row.role.as_str()));\n let descriptions =\n LargeStringArray::from_iter_values(rows.iter().map(|row| row.description.as_str()));\n\n let strength = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.strength));\n let magic = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.magic));\n let leadership = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.leadership));\n let wisdom = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.wisdom));\n let stats_fields: Vec<FieldRef> = vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ];\n let stats = StructArray::new(\n stats_fields.into(),\n vec![\n Arc::new(strength),\n Arc::new(magic),\n Arc::new(leadership),\n Arc::new(wisdom),\n ],\n None,\n );\n\n let vectors = FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n rows.iter()\n .map(|row| Some(row.vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n );\n\n let batch = RecordBatch::try_new(\n schema.clone(),\n vec![\n Arc::new(ids),\n Arc::new(names),\n Arc::new(roles),\n Arc::new(descriptions),\n Arc::new(stats),\n Arc::new(vectors),\n ],\n )\n .unwrap();\n\n Box::new(RecordBatchIterator::new(\n vec![Ok(batch)].into_iter(),\n schema,\n ))\n}\n";
export const RsQuickstartMultimodalBytes = "use std::sync::Arc;\n\nuse arrow_array::{\n BinaryArray, FixedSizeListArray, LargeStringArray, RecordBatch, RecordBatchIterator,\n};\nuse arrow_schema::{DataType, Field, Schema};\n\nlet image_path = std::path::Path::new(env!(\"CARGO_MANIFEST_DIR\"))\n .join(\"../../web/static/assets/images/quickstart/sir-lancelot.jpg\");\nlet image_bytes = std::fs::read(image_path).unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_vectors = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"lancelot\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Portrait of Sir Lancelot\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n image_vectors\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box<dyn arrow_array::RecordBatchReader + Send> = Box::new(\n RecordBatchIterator::new(vec![Ok(image_batch)].into_iter(), image_schema),\n);\nlet multimodal_table = db\n .create_table(\"character_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartMultimodalBytes = "use lancedb::arrow::arrow_array::BinaryArray;\n\nlet image_bytes = std::fs::read(\"data/sir-lancelot.jpg\").unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_vectors = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"lancelot\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Portrait of Sir Lancelot\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n image_vectors\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(\n vec![Ok(image_batch)].into_iter(),\n image_schema,\n));\nlet multimodal_table = db\n .create_table(\"character_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartOutputArray = "let result: DataFrame = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartOutputArray = "let result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartQueryFeature = "let features: DataFrame = table\n .query()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"power_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n";
export const RsQuickstartQueryFeature = "let features = table\n .query()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"power_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n";
export const RsQuickstartVectorSearch1 = "// Search for examples similar to a \"wise magical advisor\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result: DataFrame = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartVectorSearch1 = "use lancedb::arrow::IntoPolars;\nuse lancedb::query::{ExecutableQuery, QueryBase, Select};\n\n// Search for examples similar to a \"wise magical advisor\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
docs/web/snippets/read_consistency.mdx
+4
View File
@@ -0,0 +1,4 @@
{/* Auto-generated by scripts/mdx_snippets_gen.py. Do not edit manually. */}
export const PyReadConsistency = "from datetime import timedelta\nfrom tempfile import TemporaryDirectory\n\nimport lancedb\n\nwith TemporaryDirectory(prefix=\"lancedb-consistency-\") as uri:\n writer = lancedb.connect(uri)\n table = writer.create_table(\"documents\", [{\"id\": 1, \"text\": \"first\"}])\n\n default_reader = lancedb.connect(uri).open_table(\"documents\")\n latest_reader = lancedb.connect(\n uri, read_consistency_interval=timedelta(0)\n ).open_table(\"documents\")\n\n table.add([{\"id\": 2, \"text\": \"second\"}])\n\n default_count = default_reader.count_rows()\n latest_count = latest_reader.count_rows()\n print(f\"Default reader: {default_count} rows\") # 1\n print(f\"Zero-interval reader: {latest_count} rows\") # 2\n\n default_reader.checkout_latest()\n refreshed_count = default_reader.count_rows()\n print(f\"After explicit refresh: {refreshed_count} rows\") # 2\n";
docs/web/static/assets/images/overview/lancedb-architecture.svg
+60
View File
@@ -0,0 +1,60 @@
<svg xmlns="http://www.w3.org/2000/svg" width="800" height="676" viewBox="0 0 800 676" role="img" aria-labelledby="title desc">
<title id="title">LanceDB OSS and Enterprise architecture</title>
<desc id="desc">OSS runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage.</desc>
<defs>
<marker id="arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto" markerUnits="userSpaceOnUse"><path d="M1 1 7 4 1 7" fill="none" stroke="#596776" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"/></marker>
<style>
text { font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1e293b; }
.title { font-size: 27px; font-weight: 650; }
.label { font-size: 23px; font-weight: 550; }
.small { font-size: 21px; font-weight: 550; }
.note { font-size: 20px; fill: #475569; }
.flow { fill: none; stroke: #596776; stroke-width: 2; marker-end: url(#arrow); }
</style>
</defs>
<rect width="800" height="676" rx="20" fill="#ffffff"/>
<rect x="12" y="12" width="380" height="580" rx="16" fill="#f3f6f9" stroke="#d4dce6"/>
<text class="title" x="36" y="53">LanceDB OSS</text>
<text class="note" x="36" y="83">Embedded library</text>
<rect x="36" y="110" width="332" height="296" rx="10" fill="#ffffff" stroke="#c4cfdd"/>
<text class="label" x="202" y="164" text-anchor="middle">Application +</text>
<text class="label" x="202" y="196" text-anchor="middle">LanceDB library</text>
<text class="note" x="202" y="262" text-anchor="middle">Queries</text>
<text class="note" x="202" y="293" text-anchor="middle">Indexing</text>
<text class="note" x="202" y="324" text-anchor="middle">Maintenance</text>
<path class="flow" d="M202 406 V441"/>
<rect x="36" y="444" width="332" height="50" rx="10" fill="#e5ecf3"/>
<text class="label" x="202" y="476" text-anchor="middle">Local disk or object storage</text>
<rect x="408" y="12" width="380" height="580" rx="16" fill="#fff5ed" stroke="#efc5a4"/>
<text class="title" x="432" y="53">LanceDB Enterprise</text>
<text class="note" x="432" y="79">Distributed multimodal</text>
<text class="note" x="432" y="102">lakehouse</text>
<rect x="432" y="110" width="332" height="74" rx="10" fill="#ffffff" stroke="#e9c8b0"/>
<text class="label" x="598" y="140" text-anchor="middle">Applications +</text>
<text class="label" x="598" y="168" text-anchor="middle">LanceDB SDK</text>
<path class="flow" d="M598 184 V219"/>
<rect x="432" y="222" width="332" height="50" rx="10" fill="#ffdfc5"/>
<text class="label" x="598" y="254" text-anchor="middle">Query serving</text>
<path class="flow" d="M598 272 V288 H510 V304"/>
<path class="flow" d="M598 288 H686 V304"/>
<rect x="432" y="308" width="156" height="98" rx="10" fill="#ffffff" stroke="#e9c8b0"/>
<text class="small" x="510" y="337" text-anchor="middle">Read</text>
<text class="small" x="510" y="363" text-anchor="middle">execution</text>
<text class="small" x="510" y="389" text-anchor="middle">+ caching</text>
<rect x="608" y="308" width="156" height="98" rx="10" fill="#ffffff" stroke="#e9c8b0"/>
<text class="small" x="686" y="337" text-anchor="middle">Indexing +</text>
<text class="small" x="686" y="363" text-anchor="middle">background</text>
<text class="small" x="686" y="389" text-anchor="middle">work</text>
<path class="flow" d="M510 406 V441"/>
<path class="flow" d="M686 406 V441"/>
<rect x="432" y="444" width="332" height="50" rx="10" fill="#ffdfc5"/>
<text class="label" x="598" y="476" text-anchor="middle">Durable object storage</text>
<rect x="432" y="510" width="332" height="62" rx="10" fill="#ffffff" stroke="#e9c8b0" stroke-dasharray="5 4"/>
<text class="small" x="598" y="537" text-anchor="middle">Control plane</text>
<text class="note" x="598" y="560" text-anchor="middle">Manages the service</text>
<rect x="12" y="610" width="776" height="54" rx="16" fill="#edf5f1" stroke="#c3dacf"/>
<text class="label" x="400" y="644" text-anchor="middle">Shared foundation: Lance format</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.3 KiB

docs/web/static/assets/images/overview/lancedb-suite.svg
+8 -11
View File
@@ -2,9 +2,6 @@
<title id="title">LanceDB suite</title>
<desc id="desc">LanceDB OSS supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI.</desc>
<defs>
<filter id="cardShadow" x="-8%" y="-18%" width="116%" height="136%" color-interpolation-filters="sRGB">
<feDropShadow dx="0" dy="16" stdDeviation="18" flood-color="#4F2A1A" flood-opacity="0.16"/>
</filter>
<linearGradient id="enterpriseGradient" x1="512" y1="90" x2="2020" y2="90" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#E9C9F6"/>
<stop offset="0.52" stop-color="#F39A9A"/>
@@ -28,21 +25,21 @@
</linearGradient>
</defs>
<rect x="28" y="28" width="464" height="146" rx="8" style="fill:#FFFFFF !important;stroke:#3E3A35 !important;" stroke-width="2" filter="url(#cardShadow)"/>
<rect x="28" y="202" width="464" height="146" rx="8" style="fill:#FFFFFF !important;stroke:#3E3A35 !important;" stroke-width="2" filter="url(#cardShadow)"/>
<rect x="28" y="28" width="464" height="146" rx="8" style="fill:#FFFFFF !important;stroke:#3E3A35 !important;" stroke-width="2"/>
<rect x="28" y="202" width="464" height="146" rx="8" style="fill:#FFFFFF !important;stroke:#3E3A35 !important;" stroke-width="2"/>
<rect x="512" y="28" width="1506" height="146" rx="8" style="fill:url(#enterpriseGradient) !important;" filter="url(#cardShadow)"/>
<rect x="512" y="28" width="1506" height="146" rx="8" style="fill:url(#enterpriseGradient) !important;"/>
<rect x="516" y="32" width="1498" height="138" rx="6" style="fill:#FFFFFF !important;"/>
<rect x="512" y="202" width="360" height="146" rx="8" style="fill:url(#curationGradient) !important;" filter="url(#cardShadow)"/>
<rect x="512" y="202" width="360" height="146" rx="8" style="fill:url(#curationGradient) !important;"/>
<rect x="516" y="206" width="352" height="138" rx="6" style="fill:#FFFFFF !important;"/>
<rect x="894" y="202" width="360" height="146" rx="8" style="fill:url(#featureGradient) !important;" filter="url(#cardShadow)"/>
<rect x="894" y="202" width="360" height="146" rx="8" style="fill:url(#featureGradient) !important;"/>
<rect x="898" y="206" width="352" height="138" rx="6" style="fill:#FFFFFF !important;"/>
<rect x="1276" y="202" width="360" height="146" rx="8" style="fill:url(#searchGradient) !important;" filter="url(#cardShadow)"/>
<rect x="1276" y="202" width="360" height="146" rx="8" style="fill:url(#searchGradient) !important;"/>
<rect x="1280" y="206" width="352" height="138" rx="6" style="fill:#FFFFFF !important;"/>
<rect x="1658" y="202" width="360" height="146" rx="8" style="fill:url(#trainingGradient) !important;" filter="url(#cardShadow)"/>
<rect x="1658" y="202" width="360" height="146" rx="8" style="fill:url(#trainingGradient) !important;"/>
<rect x="1662" y="206" width="352" height="138" rx="6" style="fill:#FFFFFF !important;"/>
<rect x="28" y="376" width="1990" height="116" rx="8" style="fill:#FFFFFF !important;stroke:#665FFF !important;" stroke-width="3" filter="url(#cardShadow)"/>
<rect x="28" y="376" width="1990" height="116" rx="8" style="fill:#FFFFFF !important;stroke:#665FFF !important;" stroke-width="3"/>
<g style="fill:#241712 !important;">
<circle cx="78" cy="78" r="9"/>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 6.3 KiB

After

Width:  |  Height:  |  Size: 5.8 KiB

docs/web/static/assets/images/overview/training-data-lifecycle.svg
+4 -7
View File
@@ -2,22 +2,19 @@
<title id="title">Training data lifecycle</title>
<desc id="desc">Four pillars of the LanceDB training data lifecycle: Curation, Feature Engineering, Search and Retrieval, and Training.</desc>
<defs>
<filter id="shadow" x="-20%" y="-20%" width="140%" height="140%" color-interpolation-filters="sRGB">
<feDropShadow dx="0" dy="14" stdDeviation="18" flood-color="#4f2a1a" flood-opacity="0.10"/>
</filter>
<linearGradient id="accent" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#FFB08E"/>
<stop offset="1" stop-color="#FF7E4F"/>
</linearGradient>
</defs>
<rect x="20" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5" filter="url(#shadow)"/>
<rect x="20" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5"/>
<rect x="20" y="48" width="5" height="170" rx="2.5" fill="url(#accent)"/>
<rect x="350" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5" filter="url(#shadow)"/>
<rect x="350" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5"/>
<rect x="350" y="48" width="5" height="170" rx="2.5" fill="url(#accent)"/>
<rect x="680" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5" filter="url(#shadow)"/>
<rect x="680" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5"/>
<rect x="680" y="48" width="5" height="170" rx="2.5" fill="url(#accent)"/>
<rect x="1010" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5" filter="url(#shadow)"/>
<rect x="1010" y="24" width="260" height="220" rx="16" style="fill:#FFFFFF !important;stroke:#F29A75 !important;" stroke-width="1.5"/>
<rect x="1010" y="48" width="5" height="170" rx="2.5" fill="url(#accent)"/>
<path d="M292 134H338" stroke="#FF8A5C" stroke-width="2.5" stroke-linecap="round"/>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.7 KiB

After

Width:  |  Height:  |  Size: 3.4 KiB

docs/web/tables/consistency.mdx
+61 -134
View File
@@ -1,161 +1,88 @@
---
title: Consistency
title: "Consistency"
sidebarTitle: "Consistency"
description: Learn about consistency settings and versioning in LanceDB tables.
description: "Understand when readers see changes and choose the freshness your application needs."
icon: "water"
availability:
oss: available
enterprise: available
summary: >-
Configure read freshness in OSS and Enterprise. OSS checks for table updates;
Enterprise uses the connection setting to tighten its configured freshness bound.
---
import {
PyConsistencyStrong as ConsistencyStrong,
TsConsistencyStrong as TsConsistencyStrong,
RsConsistencyStrong as RsConsistencyStrong,
PyConsistencyEventual as ConsistencyEventual,
TsConsistencyEventual as TsConsistencyEventual,
RsConsistencyEventual as RsConsistencyEventual,
PyConsistencyCheckoutLatest as ConsistencyCheckoutLatest,
TsConsistencyCheckoutLatest as TsConsistencyCheckoutLatest,
RsConsistencyCheckoutLatest as RsConsistencyCheckoutLatest,
RsUpdateMakeUsersReader as RsConsistencyMakeUsersReader,
} from '/snippets/tables.mdx';
You can set `read_consistency_interval` on the connection to control how often reads check for updates from other writers.
import { PyReadConsistency } from '/snippets/read_consistency.mdx';
There are three possible settings for `read_consistency_interval`:
## What does consistency mean here? {#consistency}
1. **Unset (default)**: no automatic cross-process refresh checks.
2. **Zero seconds**: check for updates on every read (strongest freshness).
3. **Non-zero interval**: check for updates after the interval elapses (eventual refresh).
A table read uses a consistent snapshot of the data. **Freshness** describes how
recent that snapshot is. A reader can see a coherent earlier version while
another process has already committed newer data.
The value you set depends on your application's consistency needs and performance requirements.
For example, a real-time dashboard might require strong consistency, while a batch analytics job might be
fine with eventual consistency. Stronger consistency is not free — the smaller the interval, the more
often each read pays the cost of refreshing against object storage, which raises per-read latency and cost.
For example, a dashboard may need the latest updates, while an experiment may
need to keep reading a specific table version. These are different requirements.
This setting works for both local ([LanceTable](/management/database#uris)) and remote
tables. It only affects read operations —
write operations are always consistent.
## When do changes become visible? {#visibility}
<Callout color="blue">
**Consistency in Remote Tables**
In **OSS**, a table handle sees its own committed writes. By default, it does
not automatically check for changes made through other handles or processes.
Use a read consistency interval or `checkout_latest()` to pick up those changes.
For remote tables (`db://` connections), `read_consistency_interval` is also
respected by the client. The interval is sent to the server as a freshness bound on each read:
In **Enterprise**, readers use the service's configured freshness policy.
A connection can request fresher data. When a read must include a completed
write, use a zero interval or call `checkout_latest()` after the write, with a
client and server that support these freshness controls.
- **Unset (default)**: no freshness header is sent; reads use the server's cached view of the table.
- **Zero seconds**: every read asks the server for the latest committed version.
- **Non-zero interval**: reads accept data at least as fresh as `now - interval`.
## How do you control freshness? {#configure-consistency-parameters}
In addition, after any write or after a `checkout_latest` / `restore` on a table handle, subsequent
reads on that same handle carry a freshness floor so you read your own writes without extra
configuration. The floor is the later of the configured interval and the moment of the last
write or refresh, and it is shared across handles to the same table on the same connection.
Set `read_consistency_interval` when connecting. The setting applies to reads;
it does not change the transaction boundary of a write.
Each remote table handle also tracks the highest dataset version it has observed in a read
response and sends it back with every subsequent read, so successive reads on the same handle
never observe an older version even when a load balancer routes them to query nodes with
differently-cached views. `checkout_latest` resets this watermark.
| Setting | OSS | Enterprise |
| --- | --- | --- |
| **Unset** | Keep the open table version until explicitly refreshed or changed by that handle. | Use the service's configured freshness policy. |
| **Zero** | Check for the latest committed version on every read. | Request the latest committed version on every read. |
| **Non-zero interval** | Check for updates on a read once the interval since the last check has elapsed. | Request data no older than the interval allows; the service may return fresher data. |
Stronger consistency is not free — the smaller the interval, the more often each read pays the cost
of refreshing against storage, which raises per-read latency and cost.
`checkout_latest()` explicitly refreshes an OSS table handle. On Enterprise it
asks subsequent reads on that handle to reflect data at least as recent as the
refresh request. To read a fixed historical version instead, use
[table versioning](/management/table#versioning).
In <Badge color="red">Enterprise</Badge> deployments, the server-side default freshness is still
controlled by the cluster-level `weak_read_consistency_interval_seconds` parameter; the client setting
tightens that bound on a per-connection basis.
<Accordion title="Complete Python example: compare two readers">
<Tip>
30-60 seconds is a reasonable starting point for
`weak_read_consistency_interval_seconds`: it cuts unnecessary object-store
checks while keeping data fresh enough for most applications. This setting
only affects reads -- writes always remain strongly consistent.
</Tip>
</Callout>
Install the Python SDK with `pip install lancedb`, then save this as
`consistency.py` and run `python consistency.py`. It creates a temporary local
database and removes it when finished. No Enterprise connection is needed.
## Configure Consistency Parameters {#configure-consistency-parameters}
<CodeBlock filename="consistency.py" language="python" icon="python">
{PyReadConsistency}
</CodeBlock>
To set strong consistency, set the interval to 0:
The default reader sees one row until refreshed. The reader with a zero
interval sees the new row on its next read.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{ConsistencyStrong}
</CodeBlock>
</Accordion>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConsistencyStrong}
</CodeBlock>
## How should you choose? {#choosing}
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConsistencyStrong}
</CodeBlock>
</CodeGroup>
- **Reads that must include recent writes:** use a zero interval. Checking more
frequently can increase read latency and storage requests.
- **Dashboards with an acceptable delay:** choose an interval within that delay
budget, then measure the effect on latency and cost.
- **Reproducible experiments:** read an explicit table version and retain it for
as long as the experiment needs it.
<Expandable title="In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.">
<CodeGroup>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConsistencyMakeUsersReader}
</CodeBlock>
</CodeGroup>
</Expandable>
There is no single interval that suits every application. Choose based on how
stale a result your users can accept.
For eventual consistency, use a non-zero interval:
## Learn more {#next}
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{ConsistencyEventual}
</CodeBlock>
- [Connect](/basics/connect): configure your client.
- [Table versioning](/management/table#versioning): read and restore retained versions.
- [Guarantees](/basics/guarantees): understand commit and failure boundaries.
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConsistencyEventual}
</CodeBlock>
<span id="handle-bad-vectors" />
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConsistencyEventual}
</CodeBlock>
</CodeGroup>
With the default unset interval, tables do not auto-refresh from other writers.
To manually check for updates, use `checkout_latest` / `checkoutLatest`:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{ConsistencyCheckoutLatest}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConsistencyCheckoutLatest}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConsistencyCheckoutLatest}
</CodeBlock>
</CodeGroup>
For reproducible reads, you can also pin a table to a specific snapshot with `checkout(...)` or
a tag, restore a table to a prior version, then return to the live table with
`checkout_latest` / `checkoutLatest`. See
[Versioning](/management/table#versioning) for the full version and tag workflow.
## Handle bad vectors {#handle-bad-vectors}
<Note>
This section is currently specific to the Python SDK.
</Note>
In LanceDB Python, you can use the `on_bad_vectors` parameter to choose how
invalid vector values are handled. Invalid vectors are vectors that are not valid
because:
1. They are the wrong dimension
2. They contain NaN values
3. They are null but are on a non-nullable field
By default, LanceDB will raise an error if it encounters a bad vector. You can
also choose one of the following options:
* `drop`: Ignore rows with bad vectors
* `fill`: Replace bad values (NaNs) or missing values (too few dimensions) with
the fill value specified in the `fill_value` parameter. An input like
`[1.0, NaN, 3.0]` will be replaced with `[1.0, 0.0, 3.0]` if `fill_value=0.0`.
* `null`: Replace bad vectors with null (only works if the column is nullable).
A bad vector `[1.0, NaN, 3.0]` will be replaced with `null` if the column is
nullable. If the vector column is non-nullable, then bad vectors will cause an
error
For invalid vector handling during ingestion, see
[Create a table](/ingestion/create-table#from-a-custom-schema).