rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-14 06:36:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge main into table overrides SDK

Browse Source
This commit is contained in:
Xuanwo
2026-10-31 09:50:14 +08:00
parent 495aa08731 1022e91434
commit 43bccbadc5
877 changed files with 158149 additions and 17839 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/lancedb-update-lance-dependency/SKILL.md
+27
View File
@@ -48,6 +48,23 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
This updates the Rust workspace Lance dependencies through `ci/set_lance_version.py`, updates `java/pom.xml`, refreshes Cargo metadata, and prints JSON metadata containing `branch_name`, `commit_message`, and `pr_title`.
Also synchronize the Python requirements in `python/pyproject.toml`: set the
`tests` extra's exact `pylance` pin to the matching published Python release,
and update the `pylance` extra's minimum when needed to exclude older bundled
Lance majors. The minimum must still accept the test pin. Python prerelease
spelling differs from Rust (for example, `15.0.0-beta.1` becomes `15.0.0b1`).
Confirm the matching Python release is published before changing the pin.
Refresh and check `python/uv.lock` from the repository root:
```bash
uv lock --project python --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
uv lock --project python --check --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
```
This strategy considers versions across indexes; use it only with trusted
indexes. Commit the Python requirements and lockfile together with the bump.
5. Run validation:
```bash
@@ -57,6 +74,16 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
Fix real diagnostics and rerun clippy until it succeeds. Do not skip warnings.
Run the Python dependency guards with Python 3.11+ and `packaging` installed:
```bash
python3 -m unittest discover -s ci/tests -p test_python_dependencies.py -v
```
These checks reject older Lance majors and ensure the `pylance` and `tests`
extras remain compatible. The full CI-script suite uses
`python3 -m unittest discover -s ci/tests -v`.
6. Inspect `git status --short` and `git diff` to ensure only the Lance dependency update and required compatibility fixes are present.
7. If the task only asks to prepare local changes, stop here and report the changed files and validation result.
.bumpversion.toml
+6 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion]
current_version = "0.40.0-beta.3"
current_version = "0.41.0-beta.17"
parse = """(?x)
(?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\.
@@ -70,6 +70,11 @@ filename = "rust/lancedb/Cargo.toml"
replace = "\nversion = \"{new_version}\""
search = "\nversion = \"{current_version}\""
[[tool.bumpversion.files]]
filename = "rust/lancedb/cli/Cargo.toml"
replace = "\nversion = \"{new_version}\""
search = "\nversion = \"{current_version}\""
[[tool.bumpversion.files]]
filename = "nodejs/Cargo.toml"
replace = "\nversion = \"{new_version}\""
.claude-plugin/marketplace.json
+1 -1
View File
@@ -8,7 +8,7 @@
{
"name": "lancedb",
"source": "./plugins/lancedb",
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
"description": "Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work with both LanceDB Embedded and LanceDB Enterprise.",
"version": "0.1.0",
"author": {
"name": "LanceDB"
.github/actions/setup-protoc/action.yml
+81
View File
@@ -0,0 +1,81 @@
name: Setup protoc
description: Install the protoc version pinned for wheel builds, and expose it for builds that install protoc themselves
inputs:
install:
description: "Install protoc on the runner. Set to false when the build runs in a container (e.g. manylinux) and installs protoc there."
required: false
default: "true"
outputs:
version:
description: "The pinned protoc version, e.g. 36.2"
value: ${{ steps.pin.outputs.version }}
linux-x86_64-sha256:
description: "SHA-256 of the protoc linux-x86_64 zip"
value: ${{ steps.pin.outputs.linux-x86_64-sha256 }}
linux-aarch_64-sha256:
description: "SHA-256 of the protoc linux-aarch_64 zip"
value: ${{ steps.pin.outputs.linux-aarch_64-sha256 }}
runs:
using: composite
steps:
# protobuf does not publish checksum files for protoc. These are the digests
# GitHub records for the release assets; when bumping, copy them from
# `gh api repos/protocolbuffers/protobuf/releases/tags/v<version>`.
- name: Set protoc pin
id: pin
shell: bash
run: |
{
echo "version=36.2"
echo "linux-x86_64-sha256=121f6c7afe1d4d0e3ea6aab9432038599250134cbf4474cb1167d2c7decd4278"
echo "linux-aarch_64-sha256=8b8f18bd2b30346efbc698dd5a73dd7c805f3ef8380f6dfc95c768f3f1852f6a"
echo "osx-universal_binary-sha256=83d0dc80672f53486f54299bca6177159f9cad1fd54d4d3143eeea53c4423e1b"
echo "win64-sha256=f0c128dc0d8492eceece83bb459a4c0e316764b929ffbf1aa416357fd644edd3"
} >> "$GITHUB_OUTPUT"
- name: Install protoc
if: ${{ inputs.install == 'true' }}
shell: bash
env:
PROTOC_VERSION: ${{ steps.pin.outputs.version }}
RUNNER_OS_NAME: ${{ runner.os }}
RUNNER_ARCH_NAME: ${{ runner.arch }}
LINUX_X86_64_SHA256: ${{ steps.pin.outputs.linux-x86_64-sha256 }}
LINUX_AARCH_64_SHA256: ${{ steps.pin.outputs.linux-aarch_64-sha256 }}
OSX_SHA256: ${{ steps.pin.outputs.osx-universal_binary-sha256 }}
WIN64_SHA256: ${{ steps.pin.outputs.win64-sha256 }}
run: |
set -euo pipefail
case "$RUNNER_OS_NAME/$RUNNER_ARCH_NAME" in
Linux/X64) platform="linux-x86_64"; sha256="$LINUX_X86_64_SHA256" ;;
Linux/ARM64) platform="linux-aarch_64"; sha256="$LINUX_AARCH_64_SHA256" ;;
macOS/*) platform="osx-universal_binary"; sha256="$OSX_SHA256" ;;
# Windows 11 Arm runners can execute the upstream x64 protoc binary.
Windows/X64|Windows/ARM64) platform="win64"; sha256="$WIN64_SHA256" ;;
*) echo "Unsupported runner: $RUNNER_OS_NAME/$RUNNER_ARCH_NAME" >&2; exit 1 ;;
esac
zip="$RUNNER_TEMP/protoc.zip"
dest="$RUNNER_TEMP/protoc"
curl -fsSL "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${platform}.zip" -o "$zip"
# Hash stdin rather than a path: Windows paths contain backslashes,
# which make sha256sum escape its output.
if [ "$RUNNER_OS_NAME" = "macOS" ]; then
actual=$(shasum -a 256 < "$zip" | cut -d' ' -f1)
else
actual=$(sha256sum < "$zip" | cut -d' ' -f1)
fi
if [ "$actual" != "$sha256" ]; then
echo "protoc checksum mismatch: expected $sha256, got $actual" >&2
exit 1
fi
if [ "$RUNNER_OS_NAME" = "Windows" ]; then
# Git Bash on Windows runners does not reliably ship unzip.
7z x -y -o"$dest" "$zip" > /dev/null
else
unzip -q -o "$zip" -d "$dest"
fi
rm "$zip"
echo "$dest/bin" >> "$GITHUB_PATH"
"$dest/bin/protoc" --version
.github/release_notes.json
+5 -1
View File
@@ -6,6 +6,10 @@
"title": "## 🏆 Highlights",
"labels": ["highlight"]
},
{
"title": "## 🚨 Critical Fixes",
"labels": ["critical-fix"]
},
{
"title": "## 🛠 Breaking Changes",
"labels": ["breaking-change"]
@@ -38,4 +42,4 @@
"labels": ["ci"]
}
]
}
}
.github/vale/README.md
+25
View File
@@ -0,0 +1,25 @@
# LanceDB prose rules
`Dashes`, `PlainLanguage`, `ProductNames`, `VagueHeadings` and `VaguePhrases`
come from the `LanceDB.zip` asset of `lancedb/docs` release
`vale-lancedb-v0.1.0`, whose archive SHA-256 is
`0730a43e919774121068811d5687c61dfc7a0421cff78e1393f5e565b622cb0f`.
`ProductNames` no longer swaps in "LanceDB OSS".
The product-terminology rules were added here: `EmbeddedName`,
`RemoteExecution`, `OSSProduct`, `DistributedEnterprise`, `InternalNames`
(warnings) and `LegacyCloud` (suggestion). They enforce the names in the
docs repository's `STYLE.md#names`. Run every rule's tests from
`.github/vale` with `vale test --coverage styles`.
Keeping the rules here lets contributors and CI run Vale without access to the
source repository. The root `.vale.ini` preserves the release's MDX settings
and built-in repeated-word check. No `vale sync` is needed.
The rules check product names, dashes, plain language, vague headings and vague
phrases. Findings are suggestions and do not fail the CI check.
To update, a maintainer downloads the next published package, verifies its
checksum and replaces `styles/LanceDB` with the release's files. Check the
package's `.vale.ini` for configuration changes, and update the version and
checksum here in the same PR.
.github/vale/styles/LanceDB/Dashes.yml
+40
View File
@@ -0,0 +1,40 @@
extends: existence
message: "Avoid '%s' in prose. Use a comma, a colon, parentheses or two sentences."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#punctuation
level: warning
nonword: true
# An em dash in prose, and a double hyphen or en dash used as one, with a
# space on each side. A table cell holding only an em dash, meaning "none",
# ranges such as 1–3 and flags such as --force are left alone, and code is
# never prose.
raw:
- '(?<!^\s*)—|—(?!\s*$)|(?<=\s)(?:--|–)(?=\s|$)'
tests:
- name: flags an em dash
input: Indexes are optional—queries work without them.
want: |
1:21:LanceDB.Dashes:Avoid '—' in prose. Use a comma, a colon, parentheses or two sentences.
- name: flags a double hyphen used as a dash
input: It has one page per statement, syntax first -- and every SDK operation has a SQL form.
want: |
1:45:LanceDB.Dashes:Avoid '--' in prose. Use a comma, a colon, parentheses or two sentences.
- name: flags a spaced en dash
input: Compaction is cheap – it rewrites only small fragments.
want: |
1:21:LanceDB.Dashes:Avoid '–' in prose. Use a comma, a colon, parentheses or two sentences.
- name: leaves an empty table cell alone
input: |
| Provider | Text | Images |
| --- | --- | --- |
| OpenAI | ✓ | — |
| Gemini | ✓ | per request — batched |
want: |
4:28:LanceDB.Dashes:Avoid '—' in prose. Use a comma, a colon, parentheses or two sentences.
- name: leaves ranges, flags and code alone
input: |
Use 1–3 replicas and pass --force to overwrite. Run `lancedb --version`.
```bash
uv run pytest -- -k search
```
want: ''
.github/vale/styles/LanceDB/DistributedEnterprise.yml
+19
View File
@@ -0,0 +1,19 @@
extends: existence
message: "'%s' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
tokens:
- 'Distributed(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
- 'Enterprise(?:\s+(?:and|or)\s+|\s*/\s*)(?:LanceDB\s+)?Distributed'
tests:
- name: flags Distributed and Enterprise side by side
input: Choose Distributed or Enterprise. LanceDB Enterprise and LanceDB Distributed differ, and so does Distributed/Enterprise.
want: |
1:8:LanceDB.DistributedEnterprise:'Distributed or Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
1:43:LanceDB.DistributedEnterprise:'Enterprise and LanceDB Distributed' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
1:98:LanceDB.DistributedEnterprise:'Distributed/Enterprise' reads as two products. LanceDB Enterprise provides LanceDB Distributed: name the product or the execution model, not both.
- name: leaves the defining sentence and the Embedded pair alone
input: |
LanceDB Enterprise runs LanceDB Distributed. Compare Embedded and Distributed execution. Distributed search and enterprise teams are fine.
want: ''
.github/vale/styles/LanceDB/EmbeddedName.yml
+22
View File
@@ -0,0 +1,22 @@
extends: substitution
message: "Use '%s' instead of '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# The in-process execution model is LanceDB Embedded, whether the data is on
# local disk or in object storage. Identifiers such as the `oss` product key
# are code, and code is never prose.
swap:
'LanceDB\s+(?:OSS|Oss|oss|Local|local)': LanceDB Embedded
'[Ll]ocal\s+mode': LanceDB Embedded
tests:
- name: replaces the old names of the execution model
input: Start with LanceDB OSS, also called LanceDB Local, and move off local mode later.
want: |
1:12:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB OSS'.
1:37:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'LanceDB Local'.
1:65:LanceDB.EmbeddedName:Use 'LanceDB Embedded' instead of 'local mode'.
- name: leaves the current name, other uses of local, and code alone
input: |
LanceDB Embedded reads local disk or object storage. Run LanceDB locally, set `availability.oss`, or call `LanceDB OSS` in a literal.
want: ''
.github/vale/styles/LanceDB/InternalNames.yml
+25
View File
@@ -0,0 +1,25 @@
extends: substitution
message: "Use '%s' instead of the internal name '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Internal component names stay out of public text. Environment variables such
# as `PHALANX_*` and Helm keys such as `planExecutorNodes` are code. Geneva is
# a product name and is not listed.
swap:
Phalanx: query node
'[Pp]lan\s+[Ee]xecutor': executor
'[Pp]lan\s+[Ee]xecutors': executors
Sophon: LanceDB Enterprise
tests:
- name: replaces internal component names
input: Phalanx sends work to a Plan Executor. Plan executors cache data in Sophon.
want: |
1:1:LanceDB.InternalNames:Use 'query node' instead of the internal name 'Phalanx'.
1:25:LanceDB.InternalNames:Use 'executor' instead of the internal name 'Plan Executor'.
1:40:LanceDB.InternalNames:Use 'executors' instead of the internal name 'Plan executors'.
1:69:LanceDB.InternalNames:Use 'LanceDB Enterprise' instead of the internal name 'Sophon'.
- name: leaves Geneva, configuration keys and code alone
input: |
Geneva runs on the executor + cache fleet. Set `PHALANX_REMOTE_FTS_SCORER` and `planExecutorNodes`.
want: ''
.github/vale/styles/LanceDB/LegacyCloud.yml
+21
View File
@@ -0,0 +1,21 @@
extends: existence
message: "'%s' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: suggestion
ignorecase: false
# A suggestion, because `db://` URIs, `api_key` and `region` for the hosted
# service are still documented as LanceDB Cloud.
tokens:
- 'LanceDB\s+Cloud'
- 'Cloud\s*/\s*Enterprise'
- 'Enterprise\s*/\s*Cloud'
tests:
- name: suggests checking each LanceDB Cloud mention
input: Connect to LanceDB Cloud with an API key. Cloud/Enterprise users get the cache.
want: |
1:12:LanceDB.LegacyCloud:'LanceDB Cloud' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
1:43:LanceDB.LegacyCloud:'Cloud/Enterprise' is the legacy hosted service. Unless the text is specific to it, write 'LanceDB Enterprise' or 'LanceDB Distributed'.
- name: leaves other clouds and code alone
input: |
Deploy into your own cloud account on Google Cloud. Pass `LanceDB Cloud` as a literal.
want: ''
.github/vale/styles/LanceDB/OSSProduct.yml
+28
View File
@@ -0,0 +1,28 @@
extends: existence
message: "Avoid '%s'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Bare OSS used as a product, as in "OSS and Enterprise" or "in OSS". "LanceDB
# OSS" is reported by LanceDB.EmbeddedName.
tokens:
- 'OSS(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)(?:LanceDB\s+)?Enterprise'
- 'Enterprise(?:\s+(?:and|or|vs\.?|versus)\s+|\s*/\s*)OSS'
- '(?:in|on|from|using|than)\s+OSS'
tests:
- name: flags OSS presented as a product
input: Compare OSS vs Enterprise. It works in OSS and on OSS, unlike OSS/Enterprise.
want: |
1:9:LanceDB.OSSProduct:Avoid 'OSS vs Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:37:LanceDB.OSSProduct:Avoid 'in OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:48:LanceDB.OSSProduct:Avoid 'on OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:63:LanceDB.OSSProduct:Avoid 'OSS/Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
- name: flags the pair in either order
input: Both OSS and LanceDB Enterprise ship it, as do Enterprise or OSS.
want: |
1:6:LanceDB.OSSProduct:Avoid 'OSS and LanceDB Enterprise'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
1:48:LanceDB.OSSProduct:Avoid 'Enterprise or OSS'. OSS is not a product name: write 'LanceDB Embedded', or 'open source' as a property.
- name: leaves open source, identifiers and code alone
input: |
LanceDB Embedded (open source) and LanceDB Enterprise share the format. Set `oss: available` or read `in OSS` literally.
want: ''
.github/vale/styles/LanceDB/PlainLanguage.yml
+32
View File
@@ -0,0 +1,32 @@
extends: substitution
message: "Consider '%s' instead of '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#say-it-directly
level: suggestion
ignorecase: true
# Keep this list small. Technical nouns such as GPU utilization stay intact.
swap:
utilize: use
utilizes: uses
utilized: used
utilizing: using
in order to: to
tests:
- name: suggests familiar verbs
input: Utilize the index. The query utilizes it. We utilized it. Try utilizing it.
want: |
1:1:LanceDB.PlainLanguage:Consider 'use' instead of 'Utilize'.
1:30:LanceDB.PlainLanguage:Consider 'uses' instead of 'utilizes'.
1:46:LanceDB.PlainLanguage:Consider 'used' instead of 'utilized'.
1:63:LanceDB.PlainLanguage:Consider 'using' instead of 'utilizing'.
- name: shortens an unnecessary phrase
input: Create an index in order to search faster.
want: |
1:17:LanceDB.PlainLanguage:Consider 'to' instead of 'in order to'.
- name: preserves technical terms and identifiers
input: |
Measure GPU utilization and resource underutilization. Keep the rows in order. Call `utilize()` and `in order to` only as literal examples.
```python
utilized = "in order to"
```
want: ''
.github/vale/styles/LanceDB/ProductNames.yml
+33
View File
@@ -0,0 +1,33 @@
extends: substitution
message: "Use '%s' instead of '%s'."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Case-sensitive. A lowercase package, repository or domain name, such as
# lancedb or github.com, is left alone; write identifiers in code font.
swap:
'(?:Lancedb|LanceDb|lanceDB|LANCEDB|Lance DB)': LanceDB
'LanceDB (?:cloud|CLOUD)': LanceDB Cloud
'LanceDB (?:enterprise|ENTERPRISE)': LanceDB Enterprise
'(?:HuggingFace|Huggingface|Hugging face)': Hugging Face
Langchain: LangChain
Typescript: TypeScript
VoyageAI: Voyage AI
tests:
- name: corrects the product name
input: Install Lancedb and connect to LanceDB cloud or LanceDB enterprise.
want: |
1:9:LanceDB.ProductNames:Use 'LanceDB' instead of 'Lancedb'.
1:32:LanceDB.ProductNames:Use 'LanceDB Cloud' instead of 'LanceDB cloud'.
1:49:LanceDB.ProductNames:Use 'LanceDB Enterprise' instead of 'LanceDB enterprise'.
- name: corrects names seen misspelled in the current pages
input: Load a HuggingFace model, build a Langchain retriever in Typescript and call VoyageAI.
want: |
1:8:LanceDB.ProductNames:Use 'Hugging Face' instead of 'HuggingFace'.
1:35:LanceDB.ProductNames:Use 'LangChain' instead of 'Langchain'.
1:58:LanceDB.ProductNames:Use 'TypeScript' instead of 'Typescript'.
1:78:LanceDB.ProductNames:Use 'Voyage AI' instead of 'VoyageAI'.
- name: leaves package names, identifiers and code alone
input: |
The lancedb package exposes `HuggingFaceEmbeddings` and LanceDBConnection; see `Typescript` in code.
want: ''
.github/vale/styles/LanceDB/RemoteExecution.yml
+33
View File
@@ -0,0 +1,33 @@
extends: existence
message: "Avoid '%s'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#names
level: warning
ignorecase: false
# Remote storage, remote URLs and a git remote are other things and stay. Class
# names such as `RemoteTable` are code, and code is never prose.
tokens:
- '[Rr]emote\s+(?:mode|tables?|connections?|databases?)'
tests:
- name: flags remote as the execution model
input: In remote mode, a remote table is opened through a remote connection.
want: |
1:4:LanceDB.RemoteExecution:Avoid 'remote mode'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
1:19:LanceDB.RemoteExecution:Avoid 'remote table'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
1:52:LanceDB.RemoteExecution:Avoid 'remote connection'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
- name: flags plurals and a heading
input: |
## Remote tables
Remote databases hold remote tables.
want: |
1:4:LanceDB.RemoteExecution:Avoid 'Remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
3:1:LanceDB.RemoteExecution:Avoid 'Remote databases'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
3:23:LanceDB.RemoteExecution:Avoid 'remote tables'. Write 'LanceDB Distributed' for the execution model or 'LanceDB Enterprise' for the product.
- name: leaves other remotes and code alone
input: |
Push to the git remote, read remote storage and fetch a remote URL. `open_table` returns a `RemoteTable`; see `remote table` in the API.
```python
remote_table = db.open_table("remote table")
```
want: ''
.github/vale/styles/LanceDB/VagueHeadings.yml
+19
View File
@@ -0,0 +1,19 @@
extends: existence
message: "'%s' doesn't say what the section covers. Name the task or topic instead."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#headings
level: suggestion
scope: heading
ignorecase: true
tokens:
- then what
tests:
- name: flags a heading that names no topic
input: |
## Then what {#next}
Choose the output according to how you will use it.
want: |
1:4:LanceDB.VagueHeadings:'Then what' doesn't say what the section covers. Name the task or topic instead.
- name: leaves the phrase alone in body text
input: Then what happens to the old fragments? Cleanup removes them.
want: ''
.github/vale/styles/LanceDB/VaguePhrases.yml
+31
View File
@@ -0,0 +1,31 @@
extends: existence
message: "'%s' talks about the writing, not the subject. State the point directly."
link: https://github.com/lancedb/docs/blob/main/STYLE.md#say-it-directly
level: suggestion
ignorecase: false
scope: sentence
tokens:
# The page's role instead of its content, as in "This page is the concept".
- '^[Tt]his page is the concept'
# A filler sentence opener, listed in skills/docs-writer/SKILL.md.
- '^That matters'
tests:
- name: flags a page describing its own role
input: This page is the concept; the SQL reference is the specification.
want: |
1:1:LanceDB.VaguePhrases:'This page is the concept' talks about the writing, not the subject. State the point directly.
- name: flags the filler opener
input: That matters for partner workloads because names repeat.
want: |
1:1:LanceDB.VaguePhrases:'That matters' talks about the writing, not the subject. State the point directly.
- name: leaves ordinary uses alone
input: This page describes compaction. It is the operation that matters for video.
want: ''
- name: leaves specific descriptions and embedded phrases alone
input: |
This page is the entry point for the Python API. This page is the conceptual overview of the table model. The heading "That matters" is quoted here. Only the setting That matters is highlighted.
want: ''
- name: catches a filler opener after another sentence
input: Names repeat. That matters for partner workloads.
want: |
1:15:LanceDB.VaguePhrases:'That matters' talks about the writing, not the subject. State the point directly.
.github/workflows/build_linux_wheel/action.yml
+11 -4
View File
@@ -29,6 +29,11 @@ inputs:
runs:
using: "composite"
steps:
- name: Get protoc version
id: protoc
uses: ./.github/actions/setup-protoc
with:
install: "false"
- name: CONFIRM ARM BUILD
shell: bash
env:
@@ -54,13 +59,14 @@ runs:
maturin-version: "1.12.4"
command: build
working-directory: python
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc -e PROTOC_VERSION=${{ steps.protoc.outputs.version }} -e PROTOC_SHA256=${{ steps.protoc.outputs.linux-x86_64-sha256 }} ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
target: x86_64-unknown-linux-gnu
manylinux: ${{ inputs.manylinux }}
args: ${{ inputs.args }}
before-script-linux: |
set -e
curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-x86_64.zip -o /tmp/protoc.zip
curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-x86_64.zip -o /tmp/protoc.zip
echo "${PROTOC_SHA256} /tmp/protoc.zip" | sha256sum -c -
unzip /tmp/protoc.zip -d /usr/local
rm /tmp/protoc.zip
/usr/local/bin/protoc --version
@@ -71,14 +77,15 @@ runs:
maturin-version: "1.12.4"
command: build
working-directory: python
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/' -e PROTOC=/usr/local/bin/protoc -e PROTOC_VERSION=${{ steps.protoc.outputs.version }} -e PROTOC_SHA256=${{ steps.protoc.outputs.linux-aarch_64-sha256 }} ${{ inputs.rustflags != '' && format('-e RUSTFLAGS={0}', inputs.rustflags) || '' }}"
target: aarch64-unknown-linux-gnu
manylinux: ${{ inputs.manylinux }}
args: ${{ inputs.args }}
before-script-linux: |
set -e
yum install -y clang
curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v24.4/protoc-24.4-linux-aarch_64.zip -o /tmp/protoc.zip
curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-aarch_64.zip -o /tmp/protoc.zip
echo "${PROTOC_SHA256} /tmp/protoc.zip" | sha256sum -c -
unzip /tmp/protoc.zip -d /usr/local
rm /tmp/protoc.zip
/usr/local/bin/protoc --version
.github/workflows/build_mac_wheel/action.yml
+1 -4
View File
@@ -12,10 +12,7 @@ inputs:
runs:
using: "composite"
steps:
- name: Install macos dependency
shell: bash
run: |
brew install protobuf
- uses: ./.github/actions/setup-protoc
- name: Build wheel
uses: PyO3/maturin-action@v1
with:
.github/workflows/build_windows_wheel/action.yml
+1 -9
View File
@@ -12,15 +12,7 @@ inputs:
runs:
using: "composite"
steps:
- name: Install Protoc v21.12
working-directory: C:\
run: |
New-Item -Path 'C:\protoc' -ItemType Directory
Set-Location C:\protoc
Invoke-WebRequest https://github.com/protocolbuffers/protobuf/releases/download/v21.12/protoc-21.12-win64.zip -OutFile C:\protoc\protoc.zip
7z x protoc.zip
Add-Content $env:GITHUB_PATH "C:\protoc\bin"
shell: powershell
- uses: ./.github/actions/setup-protoc
- name: Build wheel
uses: PyO3/maturin-action@v1
with:
.github/workflows/ci-scripts.yml
+18
View File
@@ -5,11 +5,27 @@ on:
branches:
- main
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/build_cli_linux.sh
- ci/package_cli.py
- .bumpversion.toml
- rust/lancedb/cli/Cargo.toml
- ci/set_lance_version.py
- ci/tests/**
- .github/workflows/ci-scripts.yml
pull_request:
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/build_cli_linux.sh
- ci/package_cli.py
- .bumpversion.toml
- rust/lancedb/cli/Cargo.toml
- ci/set_lance_version.py
- ci/tests/**
- .github/workflows/ci-scripts.yml
@@ -26,5 +42,7 @@ jobs:
- uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install test dependencies
run: python -m pip install packaging
- name: Run tests
run: python -m unittest discover -s ci/tests -v
.github/workflows/cli-release.yml
+167
View File
@@ -0,0 +1,167 @@
name: CLI release packages
on:
workflow_call:
workflow_dispatch:
pull_request:
paths:
- 'rust/lancedb/**'
- 'Cargo.toml'
- 'Cargo.lock'
- 'rust-toolchain.toml'
- '.cargo/config.toml'
- '.bumpversion.toml'
- 'LICENSE'
- 'RUST_THIRD_PARTY_LICENSES.html'
- 'ci/build_cli_linux.sh'
- 'ci/package_cli.py'
- 'ci/tests/test_package_cli.py'
- '.github/actions/setup-protoc/**'
- '.github/workflows/cli-release.yml'
- '.github/workflows/gh-release.yml'
permissions:
contents: read
concurrency:
group: cli-packages-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
version:
runs-on: ubuntu-24.04
outputs:
version: ${{ steps.version.outputs.version }}
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- name: Validate release version before building
id: version
run: |
version=$(python ci/package_cli.py version)
echo "version=$version" >> "$GITHUB_OUTPUT"
build:
needs: version
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
include:
- target: x86_64-unknown-linux-gnu
runner: ubuntu-2404-8x-x64
image: quay.io/pypa/manylinux_2_28_x86_64
protoc: linux-x86_64
- target: aarch64-unknown-linux-gnu
runner: ubuntu-2404-8x-arm64
image: quay.io/pypa/manylinux_2_28_aarch64
protoc: linux-aarch_64
- target: aarch64-apple-darwin
runner: warp-macos-14-arm64-6x
runs-on: ${{ matrix.runner }}
env:
CLI_TARGET: ${{ matrix.target }}
CLI_VERSION: ${{ needs.version.outputs.version }}
MACOSX_DEPLOYMENT_TARGET: '14.0'
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1
with:
# Leave .cargo/config.toml's CPU/feature flags in effect.
rustflags: ''
cache: false
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2
with:
key: cli-release-${{ matrix.target }}
save-if: ${{ github.event_name == 'pull_request' || github.ref == 'refs/heads/main' }}
- uses: ./.github/actions/setup-protoc
id: protoc
with:
install: ${{ runner.os == 'macOS' }}
- name: Add swap for Arm fat LTO
if: matrix.target == 'aarch64-unknown-linux-gnu'
run: |
swap_file="$RUNNER_TEMP/lancedb-cli-swap"
sudo fallocate --length 16G "$swap_file"
sudo chmod 600 "$swap_file"
sudo mkswap "$swap_file"
sudo swapon "$swap_file"
- name: Build Linux CLI against glibc 2.28
if: runner.os == 'Linux'
env:
MANYLINUX_IMAGE: ${{ matrix.image }}
PROTOC_PLATFORM: ${{ matrix.protoc }}
PROTOC_VERSION: ${{ steps.protoc.outputs.version }}
PROTOC_SHA256: ${{ matrix.protoc == 'linux-x86_64' && steps.protoc.outputs.linux-x86_64-sha256 || steps.protoc.outputs.linux-aarch_64-sha256 }}
run: |
docker run --rm \
--volume "$PWD:/workspace" --workdir /workspace \
--volume "$HOME/.cargo:/cargo" --env CARGO_HOME=/cargo \
--volume "$HOME/.rustup:/rustup" --env RUSTUP_HOME=/rustup \
--env CLI_TARGET --env PROTOC_PLATFORM --env PROTOC_VERSION --env PROTOC_SHA256 \
"$MANYLINUX_IMAGE" bash ci/build_cli_linux.sh
- name: Build macOS CLI
if: runner.os == 'macOS'
run: |
cargo build --locked --release -p lancedb-cli --bin lancedb --target "$CLI_TARGET"
strip -S "target/$CLI_TARGET/release/lancedb"
# Stripping can invalidate the linker's ad-hoc signature on Apple Silicon.
codesign --force --sign - "target/$CLI_TARGET/release/lancedb"
- name: Package, inspect runtime dependencies, and smoke-test archive
run: |
python ci/package_cli.py package \
--binary "target/$CLI_TARGET/release/lancedb" --target "$CLI_TARGET" --output dist
- name: Smoke-test Linux archive on a clean glibc 2.28 system
if: runner.os == 'Linux'
run: |
docker run --rm --volume "$PWD/dist:/artifacts:ro" \
--env CLI_TARGET --env CLI_VERSION almalinux:8 bash -euo pipefail -c '
dnf install -y libstdc++
mkdir -p /tmp/check /tmp/cli-home
export HOME=/tmp/cli-home XDG_CONFIG_HOME=/tmp/cli-home/.config
stem="lancedb-v${CLI_VERSION}-${CLI_TARGET}"
tar -xzf "/artifacts/${stem}.tar.gz" -C /tmp/check
cli="/tmp/check/${stem}/lancedb"
"$cli" --help
test "$("$cli" --version)" = "lancedb ${CLI_VERSION}"
"$cli" profile list
'
- uses: actions/upload-artifact@v7
with:
name: cli-archive-${{ matrix.target }}
path: dist/*.tar.gz
if-no-files-found: error
assemble:
needs: [version, build]
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- uses: actions/download-artifact@v8
with:
pattern: cli-archive-*
merge-multiple: true
path: dist
- name: Require all three archives and generate checksums
run: |
python ci/package_cli.py checksums --output dist
python ci/package_cli.py verify --output dist
- uses: actions/upload-artifact@v7
with:
name: cli-release
path: dist/*
if-no-files-found: error
.github/workflows/cli.yml
+45
View File
@@ -0,0 +1,45 @@
name: CLI
on:
push:
branches: [main]
pull_request:
paths:
- 'rust/lancedb/**'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/cli.yml'
permissions:
contents: read
jobs:
test:
strategy:
matrix:
os: [ubuntu-24.04, macos-15]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1
with:
components: clippy
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2
- name: Install Linux prerequisites
if: runner.os == 'Linux'
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler libssl-dev clang
- name: Install macOS prerequisites
if: runner.os == 'macOS'
run: brew install protobuf
- name: Build standalone CLI binary
run: cargo build --locked --profile ci -p lancedb-cli --bin lancedb
- name: Test standalone CLI
run: cargo test --locked --profile ci -p lancedb-cli
- name: Test SDK SQL
run: cargo test --locked --profile ci -p lancedb --features remote --lib remote::sql
- name: Lint CLI
run: cargo clippy --locked --profile ci -p lancedb-cli --all-targets --all-features -- -D warnings
- name: Check SDK without remote support
# Feature-disabled SDK code has existing unused-code warnings; this is a compile check.
env:
RUSTFLAGS: ""
run: cargo check --locked --profile ci -p lancedb --no-default-features --tests
.github/workflows/codex-fix-ci.yml
-179
View File
@@ -1,179 +0,0 @@
name: Codex Fix CI
on:
workflow_dispatch:
inputs:
workflow_run_url:
description: "Failing CI workflow run URL (e.g., https://github.com/lancedb/lancedb/actions/runs/12345678)"
required: true
type: string
branch:
description: "Branch to fix (e.g., main, release/v2.0, or feature-branch)"
required: true
type: string
guidelines:
description: "Additional guidelines for the fix (optional)"
required: false
type: string
permissions:
contents: write
pull-requests: write
actions: read
jobs:
fix-ci:
runs-on: warp-ubuntu-latest-x64-4x
timeout-minutes: 60
env:
CC: clang
CXX: clang++
steps:
- name: Show inputs
run: |
echo "workflow_run_url = ${{ inputs.workflow_run_url }}"
echo "branch = ${{ inputs.branch }}"
echo "guidelines = ${{ inputs.guidelines }}"
- name: Checkout Repo
uses: actions/checkout@v6
with:
ref: ${{ inputs.branch }}
fetch-depth: 0
persist-credentials: true
- name: Set up Node.js
uses: actions/setup-node@v6
with:
# pnpm 11 (used by the nodejs install step below) requires
# Node >= 22.13; use 24 since 22 hits EOL in October.
node-version: 24
- name: Install Codex CLI
run: npm install -g @openai/codex
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: stable
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y protobuf-compiler libssl-dev
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install Python dependencies
run: |
pip install maturin ruff pytest pyarrow pandas polars
- name: Set up Java
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '11'
cache: maven
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
version: 11.1.1
- name: Install Node.js dependencies for TypeScript bindings
run: |
cd nodejs
pnpm install --frozen-lockfile
- name: Configure git user
run: |
git config user.name "lancedb automation"
git config user.email "robot@lancedb.com"
- name: Run Codex to fix CI failure
env:
WORKFLOW_RUN_URL: ${{ inputs.workflow_run_url }}
BRANCH: ${{ inputs.branch }}
GUIDELINES: ${{ inputs.guidelines }}
GITHUB_TOKEN: ${{ secrets.ROBOT_TOKEN }}
GH_TOKEN: ${{ secrets.ROBOT_TOKEN }}
OPENAI_API_KEY: ${{ secrets.CODEX_TOKEN }}
run: |
set -euo pipefail
cat <<EOF >/tmp/codex-prompt.txt
You are running inside the lancedb repository on a GitHub Actions runner. Your task is to fix a CI failure.
Input parameters:
- Failing workflow run URL: ${WORKFLOW_RUN_URL}
- Branch to fix: ${BRANCH}
- Additional guidelines: ${GUIDELINES:-"None provided"}
Follow these steps exactly:
1. Extract the run ID from the workflow URL. The URL format is https://github.com/lancedb/lancedb/actions/runs/<run_id>.
2. Use "gh run view <run_id> --json jobs,conclusion,name" to get information about the failed run.
3. Identify which jobs failed. For each failed job, use "gh run view <run_id> --job <job_id> --log-failed" to get the failure logs.
4. Analyze the failure logs to understand what went wrong. Common failures include:
- Compilation errors
- Test failures
- Clippy warnings treated as errors
- Formatting issues
- Dependency issues
5. Based on the analysis, fix the issues in the codebase:
- For compilation errors: Fix the code that doesn't compile
- For test failures: Fix the failing tests or the code they test
- For clippy warnings: Apply the suggested fixes
- For formatting issues: Run "cargo fmt --all"
- For other issues: Apply appropriate fixes
6. After making fixes, verify them locally:
- Run "cargo fmt --all" to ensure formatting is correct
- Run "cargo clippy --workspace --tests --all-features -- -D warnings" to check for issues
- Run ONLY the specific failing tests to confirm they pass now:
- For Rust test failures: Run the specific test with "cargo test -p <crate> <test_name>"
- For Python test failures: Build with "cd python && maturin develop" then run "pytest <specific_test_file>::<test_name>"
- For Java test failures: Run "cd java && mvn test -Dtest=<TestClass>#<testMethod>"
- For TypeScript test failures: Run "cd nodejs && pnpm build && pnpm test -- --testNamePattern='<test_name>'"
- Do NOT run the full test suite - only run the tests that were failing
7. If the additional guidelines are provided, follow them as well.
8. Inspect "git status --short" and "git diff" to review your changes.
9. Create a fix branch: "git checkout -b codex/fix-ci-<run_id>".
10. Stage all changes with "git add -A" and commit with message "fix: resolve CI failures from run <run_id>".
11. Push the branch: "git push origin codex/fix-ci-<run_id>". If the remote branch exists, delete it first with "gh api -X DELETE repos/lancedb/lancedb/git/refs/heads/codex/fix-ci-<run_id>" then push. Do NOT use "git push --force" or "git push -f".
12. Create a pull request targeting "${BRANCH}":
- Title: "ci: <short summary describing the fix>" (e.g., "ci: fix clippy warnings in lancedb" or "ci: resolve test flakiness in vector search")
- First, write the PR body to /tmp/pr-body.md using a heredoc (cat <<'PREOF' > /tmp/pr-body.md). The body should include:
- Link to the failing workflow run
- Summary of what failed
- Description of the fixes applied
- Then run "gh pr create --base ${BRANCH} --body-file /tmp/pr-body.md".
13. Display the new PR URL, "git status --short", and a summary of what was fixed.
Constraints:
- Use bash commands for all operations.
- Do not merge the PR.
- Do not modify GitHub workflow files unless they are the cause of the failure.
- If any command fails, diagnose and attempt to fix the issue instead of aborting immediately.
- If you cannot fix the issue automatically, create the PR anyway with a clear explanation of what you tried and what remains to be fixed.
- env "GH_TOKEN" is available, use "gh" tools for GitHub-related operations.
EOF
printenv OPENAI_API_KEY | codex login --with-api-key
codex --config shell_environment_policy.ignore_default_excludes=true exec --dangerously-bypass-approvals-and-sandbox "$(cat /tmp/codex-prompt.txt)"
.github/workflows/docs-style.yml
+33
View File
@@ -0,0 +1,33 @@
name: Docs prose style
# Style findings are advisory; installation and configuration errors fail.
on:
pull_request:
paths:
- "docs/web/**/*.mdx"
- ".vale.ini"
- ".github/vale/**"
- ".github/workflows/docs-style.yml"
permissions:
contents: read
pull-requests: read
jobs:
prose:
name: Lint changed prose
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Annotate changed lines
uses: vale-cli/vale-action@518a9136acc6e6668ce7c00d367051e0941e87ff # v3.0.0
with:
version: '3.24.0'
sync: false
files: docs/web
reporter: github-pr-annotations
filter_mode: added
fail_on_error: false
.github/workflows/docs.yml
+1
View File
@@ -38,6 +38,7 @@ jobs:
uses: actions/checkout@v6
- name: Install dependencies needed for ubuntu
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
rustup update && rustup default
- name: Set up Python
.github/workflows/gh-release.yml
+34 -2
View File
@@ -1,8 +1,8 @@
name: GitHub Release
# All SDKs share one version, so a single `vX.Y.Z` tag produces a single GitHub
# release covering all of them. The per-package publish workflows (PyPI, NPM,
# Cargo, Maven) trigger off the same tag independently.
# release covering all of them, including CLI binaries. The per-package publish
# workflows (PyPI, NPM, Cargo, Maven) trigger off the same tag independently.
on:
push:
@@ -13,7 +13,11 @@ permissions:
contents: read
jobs:
cli:
uses: ./.github/workflows/cli-release.yml
gh-release:
needs: cli
runs-on: ubuntu-latest
permissions:
contents: write
@@ -22,6 +26,15 @@ jobs:
with:
fetch-depth: 0
lfs: true
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- uses: actions/download-artifact@v8
with:
name: cli-release
path: dist
- name: Verify CLI release assets
run: python ci/package_cli.py verify --output dist
- name: Extract version
id: extract_version
env:
@@ -83,3 +96,22 @@ jobs:
generate_release_notes: false
name: LanceDB v${{ steps.extract_version.outputs.version }}
body: ${{ steps.release_notes.outputs.changelog }}
files: |
dist/lancedb-*.tar.gz
dist/SHA256SUMS
fail_on_unmatched_files: true
overwrite_files: true
report-failure:
runs-on: ubuntu-24.04
needs: [cli, gh-release]
if: always() && failure()
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/create-failure-issue
with:
job-results: ${{ toJSON(needs) }}
workflow-name: ${{ github.workflow }}
.github/workflows/java.yml
+22 -3
View File
@@ -28,9 +28,9 @@ permissions:
contents: read
jobs:
build-java:
style-check:
runs-on: ubuntu-24.04
name: Build
name: Style Check
defaults:
run:
working-directory: ./java
@@ -45,5 +45,24 @@ jobs:
cache: "maven"
- name: Java Style Check
run: ./mvnw checkstyle:check
build-java:
runs-on: ubuntu-24.04
name: Build (Java ${{ matrix.java-version }})
strategy:
matrix:
java-version: [11, 17]
defaults:
run:
working-directory: ./java
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Set up Java ${{ matrix.java-version }}
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: ${{ matrix.java-version }}
cache: "maven"
- name: Build and install
run: ./mvnw clean install
run: ./mvnw clean install -Dcheckstyle.skip=true
.github/workflows/license-header-check.yml
+1
View File
@@ -9,6 +9,7 @@ on:
- python/**
- nodejs/**
- java/**
- docs/web-tests/**
- .github/workflows/license-header-check.yml
permissions:
.github/workflows/nodejs.yml
+71 -2
View File
@@ -79,11 +79,15 @@ jobs:
# that dir is missing, so create an empty one for lint-only runs.
run: mkdir -p ../dist && pnpm install --frozen-lockfile && pnpm lint-ci
linux:
name: Linux (NodeJS ${{ matrix.node-version }})
name: Linux (NodeJS ${{ matrix.node-version }}, ${{ matrix.features }})
timeout-minutes: 30
strategy:
matrix:
node-version: [ "22", "24", "26" ]
features: [ "default" ]
include:
- node-version: "24"
features: "experimental-functions"
runs-on: "ubuntu-22.04"
defaults:
run:
@@ -121,7 +125,7 @@ jobs:
pnpm install --frozen-lockfile
# No `--` separator: pnpm forwards it literally, which would
# make napi-rs treat `--profile ci` as a cargo passthrough arg.
pnpm build:debug --profile ci
pnpm build:debug --profile ci --features ${{ matrix.features }}
pnpm tsc
- name: Setup examples
working-directory: nodejs/examples
@@ -164,6 +168,71 @@ jobs:
python ci/mock_openai.py &
cd nodejs/examples
node_modules/.bin/jest --testEnvironment jest-environment-node-single-context --verbose
arrow-compat:
# The main jobs resolve `apache-arrow` to the version pinned in
# devDependencies (the newest supported release). This job re-runs the
# suite with the library itself running on each older Arrow release the
# peer range still accepts, which is what a consumer on that version gets.
name: Linux (Arrow ${{ matrix.arrow-version }})
timeout-minutes: 30
strategy:
matrix:
arrow-version: [ "19.0.1", "20.0.0" ]
runs-on: "ubuntu-22.04"
defaults:
run:
shell: bash
working-directory: nodejs
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- uses: pnpm/action-setup@v6
with:
version: 11.1.1
- uses: actions/setup-node@v6
with:
node-version: 24
cache: 'pnpm'
cache-dependency-path: nodejs/pnpm-lock.yaml
- uses: Swatinem/rust-cache@v2
with:
# Same crate, profile and target as the `linux` job above, so reuse
# its cache: rust-cache keys on the job id by default, and this job
# would otherwise start cold on every run until it had saved its own.
shared-key: linux
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Build
run: |
pnpm install --frozen-lockfile
# No `--` separator: pnpm forwards it literally, which would
# make napi-rs treat `--profile ci` as a cargo passthrough arg.
pnpm build:debug --profile ci
- name: Resolve apache-arrow to ${{ matrix.arrow-version }}
run: |
# The older releases are already installed as the `apache-arrow-NN`
# test aliases. With pnpm's hoisted linker each alias is a complete,
# self-contained directory, so swap it in for the library's own
# `apache-arrow` instead of running a second install, which has been
# seen to hang after printing "Done" in CI. A copy, not a symlink: the
# cross-version tests rely on the alias being a separate module
# instance from the one the library loads.
major="$(echo '${{ matrix.arrow-version }}' | cut -d. -f1)"
mv node_modules/apache-arrow node_modules/apache-arrow.pinned
cp -R "node_modules/apache-arrow-${major}" node_modules/apache-arrow
# `require("apache-arrow/package.json")` is blocked by the package's exports map.
node -p "'apache-arrow resolved to ' + JSON.parse(require('fs').readFileSync('node_modules/apache-arrow/package.json')).version"
pnpm tsc
- name: Test
env:
# Newer @smithy/core uses dynamic ESM imports.
NODE_OPTIONS: "--experimental-vm-modules"
run: node_modules/.bin/jest --verbose
macos:
timeout-minutes: 30
# macos-15 ships a newer linker; the older macos-14 linker fails to insert
.github/workflows/npm-publish.yml
+16 -16
View File
@@ -71,9 +71,9 @@ jobs:
# https://github.com/napi-rs/napi-rs/blob/main/debian.Dockerfile
docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-debian
pre_build: |-
set -e &&
apt-get update &&
apt-get install -y protobuf-compiler pkg-config &&
set -e
apt-get update
apt-get install -y protobuf-compiler pkg-config
# The base image (manylinux2014-cross) sets TARGET_CC to the old
# GCC 4.8 cross-compiler. aws-lc-sys checks TARGET_CC before CC,
# so it picks up GCC even though the napi-rs image sets CC=clang.
@@ -84,10 +84,10 @@ jobs:
host: ubuntu-2404-8x-x64
features: fp16kernels
pre_build: |-
set -e &&
sudo apt-get update &&
sudo apt-get install -y protobuf-compiler pkg-config &&
rustup target add x86_64-unknown-linux-musl &&
set -e
sudo apt-get update
sudo apt-get install -y protobuf-compiler pkg-config
rustup target add x86_64-unknown-linux-musl
export EXTRA_ARGS="-x"
- target: aarch64-unknown-linux-gnu
host: ubuntu-2404-8x-x64
@@ -103,13 +103,13 @@ jobs:
# linker process 7.0 -> 4.0 GiB (lancedb/sophon#7313).
linker: /tmp/aarch64-lld-clang
pre_build: |-
set -e &&
apt-get update &&
apt-get install -y protobuf-compiler pkg-config &&
export TARGET_CC=clang TARGET_CXX=clang++ &&
set -e
apt-get update
apt-get install -y protobuf-compiler pkg-config
export TARGET_CC=clang TARGET_CXX=clang++
# The manylinux2014 sysroot has glibc 2.17 headers which lack
# AT_HWCAP2 (added in Linux 3.17). Define it for aws-lc-sys.
export CFLAGS="$CFLAGS -DAT_HWCAP2=26" &&
export CFLAGS="$CFLAGS -DAT_HWCAP2=26"
rustup target add aarch64-unknown-linux-gnu
# Not `&&`-chained: in dash, errexit does not fire for a
# non-final command in an `&&` list, so failures were ignored.
@@ -136,10 +136,10 @@ jobs:
lto: thin
codegen_units: 16
pre_build: |-
set -e &&
sudo apt-get update &&
sudo apt-get install -y protobuf-compiler &&
rustup target add aarch64-unknown-linux-musl &&
set -e
sudo apt-get update
sudo apt-get install -y protobuf-compiler
rustup target add aarch64-unknown-linux-musl
export EXTRA_ARGS="-x"
name: build - ${{ matrix.settings.target }}
runs-on: ${{ matrix.settings.host }}
.github/workflows/pypi-publish.yml
+116 -17
View File
@@ -11,6 +11,13 @@ on:
- .github/workflows/build_linux_wheel/action.yml
- .github/workflows/build_mac_wheel/action.yml
- .github/workflows/build_windows_wheel/action.yml
- .github/actions/setup-protoc/action.yml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- python/pyproject.toml
- python/Cargo.toml
- python/vendor/**
- rust/lancedb/Cargo.toml
- Cargo.toml # Change in dependency frequently breaks builds
- Cargo.lock
@@ -21,7 +28,7 @@ permissions:
contents: read
# Without this, a force-push to a PR leaves the previous run going -- including
# a ~74 minute Windows job and a billed arm64 wheel build.
# a long fat-LTO Windows job and a billed arm64 wheel build.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
@@ -123,18 +130,17 @@ jobs:
path: target/wheels/lancedb-*.whl
if-no-files-found: error
windows:
timeout-minutes: 90
runs-on: windows-latest
# The fat-LTO build of the `lancedb` crate does not finish within 90 minutes
# on the 4-core standard runner, so use the 8-core larger runner with extra
# headroom.
timeout-minutes: 150
runs-on: windows-2025-8x-x64
env:
# link.exe is single-threaded and the long pole on Windows builds. Use
# rustc's bundled lld-link instead.
CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER: rust-lld
# Fat LTO of the cdylib is single-threaded and the peak-memory step of the
# build. ThinLTO parallelizes it across the runner's cores, at some cost
# to runtime performance on our least performance-sensitive platform.
# Matches what the nodejs Windows builds already do in npm-publish.yml.
CARGO_PROFILE_RELEASE_LTO: thin
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: 16
# Keep the repository's fat LTO / 1 codegen unit release profile to leave
# headroom below PyPI's file-size limit for Windows wheels.
steps:
- uses: actions/checkout@v6
with:
@@ -149,7 +155,7 @@ jobs:
# or the default branch -- so with no run on main there is nothing that
# can populate an entry the release build would be allowed to read. Fixing
# this needs a main/nightly trigger (which would also catch wheel-build
# breakage before a release); the ~74 minutes here is otherwise dominated
# breakage before a release); the build time here is otherwise dominated
# by the fat-LTO link, which no cache avoids.
- uses: ./.github/workflows/build_windows_wheel
with:
@@ -161,10 +167,93 @@ jobs:
name: wheels-windows
path: target/wheels/lancedb-*.whl
if-no-files-found: error
sdist:
name: Python sdist
timeout-minutes: 60
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
lfs: true
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.12"
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
maturin-version: "1.12.4"
command: sdist
working-directory: python
args: --out dist
# maturin ships pins used only by other workspace members in Cargo.lock.
# Remove this step when the pinned maturin fixes
# https://github.com/PyO3/maturin/issues/2609.
- name: Sync Cargo.lock to the packaged workspace
shell: bash
run: |
# A synced repo lock lets the prune drop packages and optional edges
# without changing the remaining pins or adding dependency edges.
cargo metadata --locked --format-version 1 > /dev/null
mkdir "$RUNNER_TEMP/sdist-src"
tar -xzf python/dist/lancedb-*.tar.gz -C "$RUNNER_TEMP/sdist-src"
lock=$(echo "$RUNNER_TEMP"/sdist-src/lancedb-*/Cargo.lock)
cp "$lock" "$RUNNER_TEMP/Cargo.lock.before"
cargo metadata --format-version 1 \
--manifest-path "$RUNNER_TEMP"/sdist-src/lancedb-*/Cargo.toml > /dev/null
python ci/check_cargo_lock.py "$RUNNER_TEMP/Cargo.lock.before" "$lock"
tar -czf python/dist/lancedb-*.tar.gz -C "$RUNNER_TEMP/sdist-src" \
"$(basename python/dist/lancedb-*.tar.gz .tar.gz)"
- name: Install sdist check tools
shell: bash
run: python -m pip install twine==7.0.0 pydistcheck==0.11.3
- name: Check sdist size
shell: bash
run: |
pydistcheck --select distro-too-large-compressed \
--max-allowed-size-compressed 100M python/dist/lancedb-*.tar.gz
- name: Check sdist contents
shell: bash
run: python ci/check_sdist.py python/dist/lancedb-*.tar.gz
- name: Check sdist with twine
shell: bash
run: python -m twine check --strict python/dist/lancedb-*.tar.gz
- uses: ./.github/actions/setup-protoc
# Build the way `pip install lancedb` does when no wheel matches. The dev
# profile keeps the build within the runner's disk.
- name: Build wheel from sdist
shell: bash
env:
MATURIN_PEP517_ARGS: "--profile dev --locked"
CARGO_PROFILE_DEV_DEBUG: "0"
run: |
python -m pip wheel --no-deps --wheel-dir "$RUNNER_TEMP/sdist-wheel" \
python/dist/lancedb-*.tar.gz
- name: Smoke test wheel built from sdist
shell: bash
working-directory: ${{ runner.temp }}
run: |
python -m venv sdist-venv
sdist-venv/bin/pip install sdist-wheel/lancedb-*.whl
sdist-venv/bin/python -c '
import tempfile
import lancedb
db = lancedb.connect(tempfile.mkdtemp())
table = db.create_table("t", [{"vector": [1.0, 2.0], "id": 1}, {"vector": [3.0, 4.0], "id": 2}])
assert table.search([1.0, 2.0]).limit(1).to_list()[0]["id"] == 1
print("lancedb", lancedb.__version__, "built from sdist OK")
'
- uses: actions/upload-artifact@v7
if: startsWith(github.ref, 'refs/tags/v')
with:
name: sdist
path: python/dist/lancedb-*.tar.gz
if-no-files-found: error
publish:
name: Publish wheels
name: Publish wheels and sdist
if: startsWith(github.ref, 'refs/tags/v')
needs: [linux, mac, windows]
needs: [linux, mac, windows, sdist]
runs-on: ubuntu-latest
permissions:
id-token: write
@@ -177,7 +266,12 @@ jobs:
pattern: wheels-*
path: target/wheels
merge-multiple: true
- name: List wheels
- name: Download sdist artifact
uses: actions/download-artifact@v8
with:
name: sdist
path: target/wheels
- name: List packages
run: ls -la target/wheels
- name: Choose repo
id: choose_repo
@@ -194,13 +288,18 @@ jobs:
run: |
shopt -s nullglob
WHEELS=(target/wheels/*.whl)
SDISTS=(target/wheels/*.tar.gz)
if [[ ${#WHEELS[@]} -eq 0 ]]; then
echo "No wheels found in target/wheels/" >&2
exit 1
fi
for WHEEL in "${WHEELS[@]}"; do
echo "Uploading $WHEEL to Fury"
curl -f -F package=@"$WHEEL" "https://$FURY_TOKEN@push.fury.io/lancedb/"
if [[ ${#SDISTS[@]} -ne 1 ]]; then
echo "Expected one sdist in target/wheels/, found ${#SDISTS[@]}" >&2
exit 1
fi
for PACKAGE in "${WHEELS[@]}" "${SDISTS[@]}"; do
echo "Uploading $PACKAGE to Fury"
curl -f -F package=@"$PACKAGE" "https://$FURY_TOKEN@push.fury.io/lancedb/"
done
# NOTE: pypa/gh-action-pypi-publish must be invoked directly from a
# workflow file, not from inside a composite action. When called from a
@@ -215,7 +314,7 @@ jobs:
report-failure:
name: Report Workflow Failure
runs-on: ubuntu-latest
needs: [linux, mac, windows, publish]
needs: [linux, mac, windows, sdist, publish]
permissions:
contents: read
issues: write
.github/workflows/python.yml
+11 -3
View File
@@ -15,6 +15,7 @@ on:
- .github/workflows/build_linux_wheel/**
- .github/workflows/build_mac_wheel/**
- .github/workflows/build_windows_wheel/**
- .github/actions/setup-protoc/**
- .github/workflows/run_tests/**
permissions:
@@ -121,6 +122,10 @@ jobs:
run: |
pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -e .[tests,dev,embeddings]
pip install mlx
- name: Test remote torch data loader
run: |
python -c "import torch"
pytest python/tests/test_torch.py::test_remote_permutation_dataloader_multiprocessing -q -rs
- name: Doctest
run: pytest --doctest-modules python/lancedb
linux:
@@ -161,9 +166,12 @@ jobs:
- name: Delete wheels
run: rm -rf target/wheels
platform:
name: "Mac"
name: "Mac: ${{ matrix.features }}"
strategy:
matrix:
features: ["default", "experimental-functions"]
timeout-minutes: 30
runs-on: macos-14
runs-on: macos-15
defaults:
run:
shell: bash
@@ -187,7 +195,7 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- uses: ./.github/workflows/build_mac_wheel
with:
args: --profile ci
args: --profile ci --features ${{ matrix.features }}
- uses: ./.github/workflows/run_tests
# Make sure wheels are not included in the Rust cache
- name: Delete wheels
.github/workflows/rust.yml
+120 -45
View File
@@ -11,8 +11,10 @@ on:
- rust-toolchain.toml
- deny.toml
- rust/**
- docs/web-tests/blob-v2/rs/**
- nodejs/Cargo.toml
- python/Cargo.toml
- .github/actions/setup-protoc/**
- .github/workflows/rust.yml
permissions:
@@ -29,6 +31,63 @@ env:
RUST_BACKTRACE: "1"
jobs:
dependency-overrides:
name: Downstream Lance override
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Read the pinned Lance tag
id: lance
run: |
python3 - <<'PYTHON'
import os
import tomllib
from pathlib import Path
manifest = tomllib.loads(Path("Cargo.toml").read_text())
tag = manifest["workspace"]["dependencies"]["lance"]["tag"]
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write(f"tag={tag}\n")
PYTHON
- uses: actions/checkout@v6
with:
repository: lance-format/lance
ref: ${{ steps.lance.outputs.tag }}
path: local-lance
- name: Resolve a downstream workspace with a local Lance override
run: |
mkdir -p downstream/src
touch downstream/src/lib.rs
cat > downstream/Cargo.toml <<'TOML'
[package]
name = "lancedb-local-lance-consumer"
version = "0.1.0"
edition = "2024"
[workspace]
[dependencies]
lancedb = { path = "../rust/lancedb", default-features = false }
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
[patch."https://github.com/lance-format/lance.git"]
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
TOML
cargo metadata --manifest-path downstream/Cargo.toml --format-version 1 > downstream/graph.json
python3 - <<'PYTHON'
import json
from pathlib import Path
graph = json.loads(Path("downstream/graph.json").read_text())
packages = [p for p in graph["packages"] if p["name"] == "lance-arrow"]
assert len(packages) == 1, [p["id"] for p in packages]
assert Path(packages[0]["manifest_path"]).resolve() == Path(
"local-lance/rust/lance-arrow/Cargo.toml"
).resolve()
PYTHON
lint:
timeout-minutes: 30
runs-on: ubuntu-24.04
@@ -165,6 +224,11 @@ jobs:
run: CARGO_ARGS="--profile ci" make -C ./lancedb feature-tests
- name: Run examples
run: cargo run --profile ci --all-features --example simple --locked
- name: Run blob v2 docs example
# Python and TypeScript blob docs tests install preview packages.
# Those builds are not guaranteed to remain available for more than
# 6 months, so they stay a manual run. This example uses the workspace crate.
run: cargo run --profile ci --all-features -p lancedb-blob-v2-docs --example multimodal --locked
remote:
timeout-minutes: 30
@@ -180,7 +244,8 @@ jobs:
env:
CC: clang-18
CXX: clang++-18
GH_TOKEN: ${{ secrets.SOPHON_READ_TOKEN }}
GH_TOKEN: ${{ github.token }}
SOPHON_READ_TOKEN: ${{ secrets.SOPHON_READ_TOKEN }}
steps:
- uses: actions/checkout@v6
with:
@@ -197,6 +262,8 @@ jobs:
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- uses: rui314/setup-mold@v1
- name: Test the default local Functions gate
run: cargo test --profile ci -p lancedb --features remote --test experimental_functions --locked
- name: Run remote tests
run: CARGO_ARGS="--profile ci" make -C ./lancedb remote-tests
@@ -204,7 +271,7 @@ jobs:
timeout-minutes: 60
strategy:
matrix:
mac-runner: ["macos-14", "macos-15"]
mac-runner: ["macos-15", "macos-26"]
runs-on: "${{ matrix.mac-runner }}"
defaults:
run:
@@ -244,12 +311,6 @@ jobs:
include:
- target: x86_64-pc-windows-msvc
runner: windows-2022
# windows-11-arm is a standard runner, so it is free on public repos.
# Running natively lets the aarch64 tests actually execute -- this
# job used to cross-compile them and then skip the test step, paying
# full codegen and link cost for a compile check.
- target: aarch64-pc-windows-msvc
runner: windows-11-arm
runs-on: ${{ matrix.runner }}
defaults:
run:
@@ -264,8 +325,8 @@ jobs:
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install Protoc v21.12
run: choco install --no-progress protoc
- name: Install protoc
uses: ./.github/actions/setup-protoc
- name: Build
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
@@ -278,13 +339,50 @@ jobs:
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target ${{ matrix.target }}
# Run the aarch64 Windows tests only on main so PRs wait for a single
# Windows platform. windows-11-arm is a standard runner, so it is free on
# public repos. Running natively lets the aarch64 tests actually execute --
# this job used to cross-compile them and then skip the test step, paying
# full codegen and link cost for a compile check.
windows-arm:
# Keep the historical name so the check shows up as before on main.
name: windows (aarch64-pc-windows-msvc, windows-11-arm)
if: github.ref == 'refs/heads/main'
runs-on: windows-11-arm
defaults:
run:
working-directory: rust/lancedb
steps:
- uses: actions/checkout@v6
- name: Set target
run: rustup target add aarch64-pc-windows-msvc
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install protoc
uses: ./.github/actions/setup-protoc
- name: Build
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
cargo build --profile ci --features aws,remote --tests --locked --target aarch64-pc-windows-msvc
- name: Run tests
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
# `--target` has to match the build step above. Without it cargo uses
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target aarch64-pc-windows-msvc
msrv:
# Check the minimum supported Rust version
name: MSRV Check - Rust v${{ matrix.msrv }}
runs-on: ubuntu-24.04
strategy:
matrix:
msrv: ["1.91.0"] # This should match up with rust-version in Cargo.toml
msrv: ["1.95.0"] # This should match up with rust-version in Cargo.toml
env:
# Need up-to-date compilers for kernels
CC: clang-18
@@ -307,41 +405,18 @@ jobs:
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Downgrade dependencies
# These packages have newer requirements for MSRV
- name: Resolve MSRV-compatible dependencies
# Resolve the AWS SDK and Smithy crates together: newer SDK releases
# require newer Smithy versions, so downgrading only Smithy can fail.
run: cargo +${{ matrix.msrv }} update --config 'resolver.incompatible-rust-versions="fallback"'
- name: Keep Xet dependencies compatible with the MSRV
# Xet 1.7 does not declare its MSRV, but requires sysinfo 0.39 (Rust 1.95).
# Downgrade parents before their dependencies to satisfy their constraints.
run: |
cargo update -p aws-sdk-bedrockruntime --precise 1.77.0
cargo update -p aws-sdk-dynamodb --precise 1.68.0
cargo update -p aws-config --precise 1.6.0
cargo update -p aws-sdk-kms --precise 1.63.0
cargo update -p aws-sdk-s3 --precise 1.79.0
cargo update -p aws-sdk-sso --precise 1.62.0
cargo update -p aws-sdk-ssooidc --precise 1.63.0
cargo update -p aws-sdk-sts --precise 1.63.0
# aws-runtime/sigv4/credential-types/types and the aws-smithy-*
# crates bumped their MSRV to 1.91.1 in late 2026; pin to the last
# 1.91.0-compatible versions. The order matters — each downgrade
# only succeeds once everything that still pins it at a higher
# version has itself been downgraded.
cargo update -p aws-runtime --precise 1.5.12
cargo update -p aws-types --precise 1.3.9
cargo update -p aws-sigv4 --precise 1.3.5
cargo update -p aws-credential-types --precise 1.2.8
# aws-smithy-checksums must stay at or above 0.63.13: OpenDAL's S3
# service needs crc-fast ~1.9, and older releases pin it to ~1.3.
cargo update -p aws-smithy-checksums --precise 0.63.13
cargo update -p aws-smithy-runtime --precise 1.9.3
cargo update -p aws-smithy-http --precise 0.62.6
cargo update -p aws-smithy-eventstream --precise 0.60.14
cargo update -p aws-smithy-http-client --precise 1.1.3
cargo update -p aws-smithy-observability --precise 0.1.4
cargo update -p aws-smithy-query --precise 0.60.8
cargo update -p aws-smithy-runtime-api --precise 1.9.3
cargo update -p aws-smithy-async --precise 1.2.7
cargo update -p aws-smithy-types --precise 1.3.6
cargo update -p aws-smithy-xml --precise 0.60.11
cargo update -p home --precise 0.5.9
for crate in hf-xet xet-data xet-client xet-core-structures xet-runtime; do
cargo +${{ matrix.msrv }} update -p "$crate" --precise 1.6.0
done
- name: cargo +${{ matrix.msrv }} check
env:
RUSTUP_TOOLCHAIN: ${{ matrix.msrv }}
run: cargo check --profile ci --workspace --tests --benches --all-features
run: cargo check --locked --profile ci --workspace --tests --benches --all-features
.gitignore
+3
View File
@@ -40,3 +40,6 @@ dist
target
**/sccache.log
# Vale styles installed by `vale sync`
/.vale/
.typos.toml
+15 -1
View File
@@ -1,5 +1,15 @@
[default]
extend-ignore-re = ["(?Rm)^.*(#|//)\\s*spellchecker:disable-line$"]
extend-ignore-re = [
"(?Rm)^.*(#|//)\\s*spellchecker:disable-line$",
# Conventional import aliases in the docs examples: `import fiftyone as fo`
# and `import torchvision.io as tio`.
"\\bas (fo|tio)\\b",
"\\btio\\.",
# The substring query in the n-gram full-text search example.
'search\("nce"',
# Garbled OCR output quoted verbatim in the VLM fine-tuning tutorial.
"permission nstallation",
]
[default.extend-words]
# Azure Kubernetes Service, mentioned in rust/lancedb/src/remote/oauth.rs.
@@ -12,6 +22,10 @@ mmaped = "mmaped"
# `WriteableBuffer` is the real name of a type from Python's `_typeshed` stubs,
# used in python/python/lancedb/_blob.py.
Writeable = "Writeable"
# Superlinked's self-hosted inference engine (SIE), its `sie-lancedb` package,
# and the `sie` embedding function it registers, in
# docs/web/integrations/embedding/superlinked.mdx.
SIE = "SIE"
[files]
extend-exclude = [
.vale.ini
+6
View File
@@ -0,0 +1,6 @@
StylesPath = .github/vale/styles
MinAlertLevel = suggestion
[*.mdx]
BasedOnStyles = LanceDB
Vale.Repetition = warning
AGENTS.md
+2 -2
View File
@@ -102,8 +102,8 @@ TypeScript bindings changes:
1. Add napi-rs method binding on `Table` in `nodejs/src/table.rs`.
2. Run `pnpm build` to generate TypeScript definitions.
3. Add typescript method on abstract class `Table` in `nodejs/src/table.ts`.
4. Add concrete method on `LocalTable` class in `nodejs/src/native_table.ts`.
3. Add typescript method on abstract class `Table` in `nodejs/lancedb/table.ts`.
4. Add concrete method on `LocalTable` class in `nodejs/lancedb/table.ts`.
* Note: despite the name, this class is also used for remote tables.
5. Add test in `nodejs/__test__/table.test.ts`.
6. Run `pnpm run docs` to generate TypeScript documentation.
CLAUDE.md
-1
View File
@@ -1 +0,0 @@
AGENTS.md
CONTRIBUTING.md
+123 -1
View File
@@ -71,7 +71,129 @@ are wrappers around the Rust library.
* `nodejs`: Typescript package source code
* `node`: **Deprecated** Typescript package source code
* `java`: Java package source code
* `docs`: Documentation source code
* `docs/src`: SDK reference, built with mkdocs
* `docs/web`: website guides for LanceDB Embedded and the APIs used by both Embedded and Distributed
* `docs/web-tests`: the tests the documentation's code examples are extracted from
## Documentation
Update the docs in the same pull request as the behavior they describe. You can
merge docs before a feature launches; keep the unfinished material hidden until
it is ready for readers.
### Choose the right place
- `docs/web/` holds website guides for LanceDB Embedded and the APIs used by both
Embedded and Distributed.
- `docs/src/` holds the SDK reference. See [docs/README.md](docs/README.md) for
its build and API-generation instructions.
- `docs/web-tests/` and `rust/lancedb/tests/` hold tests used to generate the
website's code examples.
Document SDK APIs here, and explain which operations work in Embedded and Distributed.
Put Enterprise-specific deployment and operating instructions in the Enterprise
documentation.
Some pages are placeholders that the website build replaces with an Enterprise
page. Changes to a placeholder won't appear on the combined site. Put
instructions that both products need on a separate page and link to it.
### Preview and check your change
From the repository root:
```bash
npm i -g mint@4.2.888
(cd docs/web && mint dev)
```
This previews this checkout, including your uncommitted changes. It doesn't
include the other documentation sources or render the availability labels.
Use the [combined preview](https://github.com/lancedb/docs#development) when
checking Enterprise replacements or links to pages from another source.
Staging shows saved release editions, so it won't show a new source change
just because that change merged.
Follow the [style guide](https://github.com/lancedb/docs/blob/main/STYLE.md).
[Vale](https://vale.sh) checks wording, names and punctuation using the
[bundled LanceDB rules](.github/vale/README.md). Pull requests show suggestions
on the lines they change. With Vale 3.24.0 installed, check a page locally with:
```bash
vale docs/web/indexing/vector-index.mdx
```
Keep existing `{#anchor}` values when changing headings, so links to those
sections keep working. Give new headings stable anchors. For links in rendered
components, write the path directly, such as `href="/quickstart"`, and spell out
attributes rather than using property spreads. That lets versioned docs keep
links inside the selected edition. Code examples aren't subject to this rule.
### Code examples
Edit the test that produces an example, then regenerate its snippet. The
sources are `docs/web-tests/{py,ts,rs}`, `docs/web-tests/blob-v2/{py,ts,rs}` and
`rust/lancedb/tests`. Run this from the repository root:
```bash
uv run docs/web-tests/mdx_snippets_gen.py \
-s docs/web-tests/py -s docs/web-tests/ts -s docs/web-tests/rs \
-s docs/web-tests/blob-v2/py -s docs/web-tests/blob-v2/ts -s docs/web-tests/blob-v2/rs \
-s rust/lancedb/tests -o docs/web/snippets
```
Run the tests for the examples you changed; generating a snippet doesn't run
its example. Review the generated diff and commit only the relevant snippets
with their tests. Don't assume the PR checks execute every documentation
example.
### Say where a feature works
Use `availability` in a page's frontmatter to describe the support covered by
that page. Each product (`oss` for LanceDB Embedded, `enterprise` for LanceDB
Enterprise) is either `available` or `unavailable`; use the summary
and body for setup requirements, SDK differences and unsupported operations.
For example, a page about signing in to Enterprise could declare:
```yaml
availability:
oss: unavailable
enterprise: available
summary: >-
These sign-in instructions apply to LanceDB Enterprise. LanceDB
Embedded has no server to sign in to.
```
Check these claims with the feature owner. Don't use `varies`, treat an API
stub as working support, or mark a supported feature unavailable just because
it needs configuration. If one label can't describe the whole page accurately,
make the page's scope narrower and explain the differences.
Assembly turns this declaration into a page label and, for a feature supported
by only one product, a sidebar tag. Don't add those badges by hand. Availability
labels do not hide unreleased content.
### Write docs before launch
You don't need to hold a documentation PR until launch. Use these controls:
- For a whole page, add its path to `docs/web/.mintignore`. Leave it out of
`docs/web/docs.json`, redirects and links from published pages. Removing a
sidebar entry alone doesn't hide a page.
- For a section, wrap it in an MDX comment, `{/* ... */}`, with a note such as
`Draft for the Feature X release.` Put draft table rows and list items in a
comment below the table or list, not between entries.
Drafts remain in the source. Use them to avoid advertising an unfinished
feature, not to keep information confidential. See
[Writing docs before release](https://github.com/lancedb/docs#writing-docs-before-release)
for examples and how to preview a draft.
Before the intended release, remove the draft controls, restore the navigation
and links, and check the result. Those changes must be in the source commit the
edition uses. A later change to `main` doesn't update an edition already saved.
For a mistake in an existing edition, fix the source and request a
[recorded correction](https://github.com/lancedb/docs#correcting-a-saved-edition).
## Release process
Cargo.lock
Generated
+1161 -509
View File
File diff suppressed because it is too large Load Diff
Cargo.toml
+56 -37
View File
@@ -1,5 +1,11 @@
[workspace]
members = ["rust/lancedb", "nodejs", "python"]
members = [
"rust/lancedb/cli",
"rust/lancedb",
"nodejs",
"python",
"docs/web-tests/blob-v2/rs",
]
resolver = "2"
[workspace.package]
@@ -10,60 +16,66 @@ repository = "https://github.com/lancedb/lancedb"
description = "Serverless, low-latency vector database for AI applications"
keywords = ["lancedb", "lance", "database", "vector", "search"]
categories = ["database-implementations"]
rust-version = "1.91.0"
rust-version = "1.95.0"
[workspace.dependencies]
lance = { "version" = "=13.0.0-beta.4", default-features = false, "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=13.0.0-beta.4", default-features = false, "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=13.0.0-beta.4", default-features = false, "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=13.0.0-beta.4", "tag" = "v13.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance = { "version" = "=15.0.0-beta.7", default-features = false, "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=15.0.0-beta.7", default-features = false, "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-jsonb = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=15.0.0-beta.7", default-features = false, "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=15.0.0-beta.7", "tag" = "v15.0.0-beta.7", "git" = "https://github.com/lance-format/lance.git" }
lancedb = { path = "rust/lancedb", default-features = false }
ahash = "0.8"
anyhow = "1"
# Note that this one does not include pyarrow
arrow = { version = "58.0.0", optional = false }
arrow-array = "58.0.0"
arrow-buffer = "58.0.0"
arrow-data = "58.0.0"
arrow-ipc = "58.0.0"
arrow-ord = "58.0.0"
arrow-schema = "58.0.0"
arrow-select = "58.0.0"
arrow-cast = "58.0.0"
arrow-flight = { version = "58.0.0", features = ["flight-sql-experimental"] }
arrow = { version = "59.2.0", optional = false }
arrow-array = "59.2.0"
arrow-buffer = "59.2.0"
arrow-data = "59.2.0"
arrow-ipc = "59.2.0"
arrow-ord = "59.2.0"
arrow-schema = "59.2.0"
arrow-select = "59.2.0"
arrow-cast = "59.2.0"
arrow-flight = { version = "59.2.0", features = ["flight-sql-experimental"] }
async-trait = "0"
# Smithy JSON 0.63 requires the pre-1.7 Document representation; allow the MSRV pin.
aws-smithy-types = ">=1.3.6, <1.7"
base64 = "0.22"
bytes = "1"
datafusion = { version = "54.0.0", default-features = false }
datafusion-catalog = "54.0.0"
datafusion-common = { version = "54.0.0", default-features = false }
datafusion-execution = "54.0.0"
datafusion-expr = "54.0.0"
datafusion-functions = "54.0.0"
datafusion-physical-plan = "54.0.0"
datafusion-physical-expr = "54.0.0"
datafusion-sql = "54.0.0"
datafusion = { version = "55.0.0", default-features = false }
datafusion-catalog = "55.0.0"
datafusion-common = { version = "55.0.0", default-features = false }
datafusion-execution = "55.0.0"
datafusion-expr = "55.0.0"
datafusion-functions = "55.0.0"
datafusion-physical-plan = "55.0.0"
datafusion-physical-expr = "55.0.0"
datafusion-sql = "55.0.0"
env_logger = "0.11"
half = { "version" = "2.7.1", default-features = false, features = [
"num-traits",
] }
futures = "0.3"
hf-hub = { version = "0.4.1", default-features = false }
http = "1"
log = "0.4"
metrics = "0.24"
metrics-util = "0.19"
moka = { version = "0.12", features = ["future"] }
object_store = "0.14.1"
pin-project = "1.0.7"
prost = "0.14"
rand = "0.9"
snafu = "0.8"
url = "2"
@@ -72,13 +84,20 @@ oauth2 = { version = "5.0", default-features = false }
regex = "1.10"
semver = "1.0.25"
serde = "1"
serde_json = "1"
serde_json = { version = "1", features = ["preserve_order"] }
serial_test = "3"
sha2 = "0.10"
tempfile = "3.5.0"
tokio = { version = "1.23", features = ["rt-multi-thread", "sync"] }
tonic = { version = "0.14", features = ["tls-native-roots", "tls-ring"] }
tonic = { version = "0.14", features = ["tls-native-roots", "tls-aws-lc"] }
tracing = "0.1"
uuid = { version = "1.7.0", features = ["v4", "v7"] }
chrono = { version = "0.4", default-features = false, features = ["clock"] }
[patch.crates-io]
# Lance v14.0.0-beta.18 requires multipart write modes from this upstream revision.
object_store = { git = "https://github.com/apache/arrow-rs-object-store.git", rev = "c10758c207c0103355f940f0c736fb8ecb157b04" }
[profile.ci]
debug = "line-tables-only"
inherits = "dev"
README.md
+4 -6
View File
@@ -1,8 +1,6 @@
<a href="https://cloud.lancedb.com" target="_blank">
<img src="https://github.com/user-attachments/assets/92dad0a2-2a37-4ce1-b783-0d1b4f30a00c" alt="LanceDB Cloud Public Beta" width="100%" style="max-width: 100%;">
</a>
<div align="center">
[![Reverie](docs/src/assets/reverie-banner.png)](https://www.reveriesummit.com/?utm_medium=social&utm_source=github&utm_campaign=reverie-2026)
[![LanceDB](docs/src/assets/hero-header.png)](https://lancedb.com)
[![Website](https://img.shields.io/badge/-Website-100000?style=for-the-badge&labelColor=645cfb&color=645cfb)](https://lancedb.com/)
[![Blog](https://img.shields.io/badge/Blog-100000?style=for-the-badge&labelColor=645cfb&color=645cfb)](https://blog.lancedb.com/)
@@ -13,7 +11,7 @@
<img src="docs/src/assets/lancedb.png" alt="LanceDB" width="50%">
# **The Multimodal AI Lakehouse**
# **The Multimodal Lakehouse for AI**
[**How to Install** ](#how-to-install) ✦ [**Detailed Documentation**](https://docs.lancedb.com) ✦ [**Tutorials and Recipes**](https://github.com/lancedb/vectordb-recipes/tree/main) ✦ [**Contributors**](#contributors)
@@ -47,8 +45,8 @@ LanceDB is a central location where developers can build, train and analyze thei
- **Advanced Features**: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
### **Products**:
- **Open Source & Local**: 100% open source, runs locally or in your cloud. No vendor lock-in.
- **Cloud and Enterprise**: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
- **LanceDB Embedded (open source)**: 100% open source, runs in your application's process; your data can live on local disk or in object storage. No vendor lock-in.
- **LanceDB Enterprise**: Production-scale distributed search, indexing, and maintenance. Plus, complete data sovereignty and security.
### **Ecosystem**:
- **Columnar Storage**: Built on the Lance columnar format for efficient storage and analytics.
ci/build_cli_linux.sh
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# Run inside a native-architecture manylinux_2_28 container. Cargo and rustup
# are mounted from the runner; keep the image's compiler/toolchain PATH intact.
set -euo pipefail
export PATH="/cargo/bin:$PATH"
# The manylinux images provide GCC, but do not guarantee Clang or LLD. Native
# dependencies use Clang, and the Arm linker wrapper below also requires LLD.
dnf install -y clang lld
export CC=clang CXX=clang++
"$CC" --version
"$CXX" --version
ld.lld --version
# Fail before Cargo downloads or compiles dependencies if either compiler or
# the linker cannot produce a runnable binary in this container.
printf 'int main(void) { return 0; }\n' > /tmp/cli-toolchain-probe.c
"$CC" -fuse-ld=lld /tmp/cli-toolchain-probe.c -o /tmp/cli-toolchain-probe
/tmp/cli-toolchain-probe
"$CXX" -fuse-ld=lld -x c++ /tmp/cli-toolchain-probe.c -o /tmp/cli-toolchain-probe
/tmp/cli-toolchain-probe
git config --global --add safe.directory /workspace
curl -fsSL "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${PROTOC_PLATFORM}.zip" -o /tmp/protoc.zip
echo "${PROTOC_SHA256} /tmp/protoc.zip" | sha256sum -c -
unzip -q /tmp/protoc.zip -d /tmp/protoc
export PATH="/tmp/protoc/bin:$PATH"
# GNU ld on Arm has a much higher memory peak for Lance's final link. Use a
# linker wrapper without replacing the repository's CPU and feature rustflags.
if [[ "$CLI_TARGET" == aarch64-unknown-linux-gnu ]]; then
cat > /tmp/cli-linker <<'LINKER'
#!/bin/sh
exec clang -fuse-ld=lld "$@"
LINKER
chmod 755 /tmp/cli-linker
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=/tmp/cli-linker
fi
cargo build --locked --release -p lancedb-cli --bin lancedb --target "$CLI_TARGET"
strip "target/$CLI_TARGET/release/lancedb"
ci/check_cargo_lock.py
+59
View File
@@ -0,0 +1,59 @@
"""Fail if pruning Cargo.lock changes pins or adds dependency edges.
Removing workspace members can disable optional dependency edges even when
their target packages remain in use elsewhere in the packaged workspace.
Usage: python ci/check_cargo_lock.py <lock-before> <lock-after>
"""
import sys
import tomllib
from pathlib import Path
def key(package: dict) -> tuple:
return tuple(package.get(k) for k in ("name", "version", "source", "checksum"))
def graph(lock: dict) -> dict[tuple, set[tuple]]:
"""Map each package to the packages its dependency strings resolve to."""
by_name = {}
for p in lock["package"]:
by_name.setdefault(p["name"], []).append(p)
def resolve(dep: str) -> tuple:
# Cargo writes "name", "name version" or "name version (source)".
name, *rest = dep.split(" ", 2)
(match,) = [
p
for p in by_name[name]
if (not rest or p["version"] == rest[0])
and (len(rest) < 2 or rest[1] == f"({p.get('source')})")
]
return key(match)
return {
key(p): {resolve(d) for d in p.get("dependencies", [])} for p in lock["package"]
}
def main() -> None:
before, after = (tomllib.loads(Path(p).read_text()) for p in sys.argv[1:])
errors = []
for field in (before.keys() | after.keys()) - {"package"}:
if before.get(field) != after.get(field):
errors.append(f"top-level {field!r} changed")
edges_before, edges_after = graph(before), graph(after)
removed = edges_before.keys() - edges_after.keys()
for package, deps in edges_after.items():
if package not in edges_before:
errors.append(f"added or changed {package}")
elif not deps <= edges_before[package]:
errors.append(f"dependencies of {package[:2]} changed")
if errors:
sys.exit("error: the prune changed Cargo.lock:\n" + "\n".join(errors))
print(f"Cargo.lock prune only removed {len(removed)} packages")
if __name__ == "__main__":
main()
ci/check_sdist.py
+42
View File
@@ -0,0 +1,42 @@
"""Fail if the lancedb sdist lacks a git-tracked file of the Python package.
Usage: python ci/check_sdist.py <sdist> (run from the repository root)
"""
import subprocess
import sys
import tarfile
from pathlib import Path
# Where git tracks the package, and where maturin puts it under the sdist root.
TRACKED_PACKAGE = "python/python/lancedb"
SDIST_PACKAGE = "python/lancedb"
def main() -> None:
sdist = Path(sys.argv[1])
root = sdist.name.removesuffix(".tar.gz")
listing = subprocess.run(
["git", "ls-files", "-z", "--", TRACKED_PACKAGE],
check=True,
capture_output=True,
text=True,
).stdout
tracked = [p for p in listing.split("\0") if p]
if not tracked:
sys.exit(f"error: git tracks no files under {TRACKED_PACKAGE}")
with tarfile.open(sdist) as tar:
members = {m.name for m in tar if m.isfile()}
missing = [
p
for p in tracked
if f"{root}/{SDIST_PACKAGE}{p.removeprefix(TRACKED_PACKAGE)}" not in members
]
if missing:
sys.exit(f"error: {sdist.name} lacks tracked sources: {', '.join(missing)}")
print(f"{sdist.name} OK: all {len(tracked)} tracked sources present")
if __name__ == "__main__":
main()
ci/package_cli.py
+230
View File
@@ -0,0 +1,230 @@
"""Package and validate standalone CLI release assets (Python 3.12+).
Run `version` before building, `package` on each native runner, then `checksums`
after gathering the archives. Run `verify` again immediately before publication.
"""
import argparse
import hashlib
import os
import re
import subprocess
import tarfile
import tempfile
import tomllib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
TARGETS = (
"x86_64-unknown-linux-gnu",
"aarch64-unknown-linux-gnu",
"aarch64-apple-darwin",
)
NOTICES = ("LICENSE", "RUST_THIRD_PARTY_LICENSES.html")
def release_version(root: Path, ref: str) -> str:
config = tomllib.loads((root / ".bumpversion.toml").read_text())["tool"][
"bumpversion"
]
version = config["current_version"]
if not re.fullmatch(r"\d+\.\d+\.\d+(?:-beta\.\d+)?", version):
raise ValueError(f"Unsupported release version: {version}")
for manifest in ("rust/lancedb/Cargo.toml", "rust/lancedb/cli/Cargo.toml"):
package = tomllib.loads((root / manifest).read_text())["package"]
if package["version"] != version:
raise ValueError(f"{manifest} version does not match {version}")
if ref.startswith("refs/tags/") and ref != f"refs/tags/v{version}":
raise ValueError(f"Tag {ref} does not match v{version}")
return version
def archive_stem(version: str, target: str) -> str:
return f"lancedb-v{version}-{target}"
def command(*args: str) -> str:
return subprocess.check_output(args, text=True)
def check_linux_metadata(header: str, dynamic: str, symbols: str, target: str):
machine = (
"AArch64" if target.startswith("aarch64") else "Advanced Micro Devices X86-64"
)
if not re.search(rf"Machine:\s+{re.escape(machine)}\s*$", header, re.MULTILINE):
raise ValueError(f"ELF architecture does not match {target}")
allowed = {
"libc.so.6",
"libdl.so.2",
"libm.so.6",
"libpthread.so.0",
"librt.so.1",
"libgcc_s.so.1",
"libstdc++.so.6",
"ld-linux-x86-64.so.2",
"ld-linux-aarch64.so.1",
}
needed = set(re.findall(r"\(NEEDED\).*\[([^\]]+)\]", dynamic))
if not needed or needed - allowed:
raise ValueError(f"Unexpected runtime libraries: {sorted(needed - allowed)}")
if re.search(r"\((?:RPATH|RUNPATH)\)", dynamic):
raise ValueError("CLI must not depend on a build-container runtime path")
# AlmaLinux 8 provides glibc 2.28, GCC 8's libstdc++, and libgcc.
limits = {
"GLIBC": (2, 28),
"GLIBCXX": (3, 4, 25),
"CXXABI": (1, 3, 11),
"GCC": (7, 0, 0),
}
for namespace, limit in limits.items():
for value in re.findall(rf"\b{namespace}_([\d.]+)\b", symbols):
if tuple(map(int, value.split("."))) > limit:
raise ValueError(
f"Runtime symbol {namespace}_{value} exceeds Linux baseline"
)
if "GLIBC_PRIVATE" in symbols:
raise ValueError("CLI must not require private glibc symbols")
def check_macos_metadata(arches: str, libraries: str, build: str):
if arches.strip() != "arm64":
raise ValueError("CLI must contain exactly the arm64 architecture")
for library in re.findall(r"^\s+(\S+) \(", libraries, re.MULTILINE):
if not library.startswith(("/usr/lib/", "/System/Library/")):
raise ValueError(f"Non-system macOS runtime library: {library}")
minimum = re.findall(r"^\s*minos ([\d.]+)\s*$", build, re.MULTILINE)
if len(minimum) != 1 or tuple(map(int, minimum[0].split("."))) != (14, 0):
raise ValueError("CLI must target macOS 14.0")
def check_runtime(binary: Path, target: str):
path = str(binary.resolve())
if target.endswith("linux-gnu"):
check_linux_metadata(
command("readelf", "-h", path),
command("readelf", "-d", path),
command("readelf", "--version-info", path),
target,
)
else:
command("codesign", "--verify", "--strict", path)
check_macos_metadata(
command("lipo", "-archs", path),
command("otool", "-L", path),
command("vtool", "-show-build", path),
)
def verify_archive(archive: Path, version: str, target: str):
stem = archive_stem(version, target)
expected = {f"{stem}/{name}" for name in ("lancedb", *NOTICES)}
with tarfile.open(archive, "r:gz") as tar:
members = tar.getmembers()
if len(members) != len(expected) or {m.name for m in members} != expected:
raise ValueError(f"Unexpected archive contents: {archive.name}")
for member in members:
mode = 0o755 if member.name.endswith("/lancedb") else 0o644
if not member.isfile() or member.mode != mode or member.size == 0:
raise ValueError(f"Invalid archive member: {member.name}")
def smoke_test(archive: Path, version: str, target: str):
verify_archive(archive, version, target)
with tempfile.TemporaryDirectory() as tmp:
directory = Path(tmp)
with tarfile.open(archive, "r:gz") as tar:
tar.extractall(directory, filter="data")
binary = directory / archive_stem(version, target) / "lancedb"
home = directory / "home"
home.mkdir()
env = {k: v for k, v in os.environ.items() if not k.startswith("LANCEDB_")}
env.update(HOME=str(home), XDG_CONFIG_HOME=str(home / ".config"))
for args in (("--help",), ("--version",), ("profile", "list")):
output = subprocess.run(
[str(binary), *args],
cwd=home,
env=env,
check=True,
capture_output=True,
text=True,
timeout=30,
)
if args == ("--version",) and output.stdout.strip() != f"lancedb {version}":
raise ValueError(
f"Packaged CLI reports the wrong version: {output.stdout!r}"
)
if args == ("--help",) and "Usage:" not in output.stdout:
raise ValueError("Packaged CLI did not print help")
if (
args == ("profile", "list")
and "Configured profiles:" not in output.stdout
):
raise ValueError("Packaged CLI did not list profiles")
def package(root: Path, binary: Path, target: str, output: Path, version: str) -> Path:
check_runtime(binary, target)
output.mkdir(parents=True, exist_ok=True)
stem = archive_stem(version, target)
archive = output / f"{stem}.tar.gz"
with tarfile.open(archive, "w:gz") as tar:
for name, source in [
("lancedb", binary),
*((name, root / name) for name in NOTICES),
]:
member = tar.gettarinfo(str(source), arcname=f"{stem}/{name}")
member.mode = 0o755 if name == "lancedb" else 0o644
member.uid = member.gid = 0
member.uname = member.gname = ""
with source.open("rb") as file:
tar.addfile(member, file)
smoke_test(archive, version, target)
return archive
def checksum_lines(output: Path, version: str) -> str:
expected = {f"{archive_stem(version, target)}.tar.gz": target for target in TARGETS}
found = {path.name for path in output.glob("*.tar.gz")}
if found != expected.keys():
raise ValueError(
f"Expected exactly these CLI archives: {sorted(expected)}; found {sorted(found)}"
)
lines = []
for name, target in sorted(expected.items()):
archive = output / name
verify_archive(archive, version, target)
with archive.open("rb") as file:
digest = hashlib.file_digest(file, "sha256").hexdigest()
lines.append(f"{digest} {name}\n")
return "".join(lines)
def verify_checksums(output: Path, version: str):
if (output / "SHA256SUMS").read_text() != checksum_lines(output, version):
raise ValueError("SHA256SUMS does not match the CLI release archives")
def main():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest="command", required=True)
commands.add_parser("version")
build = commands.add_parser("package")
build.add_argument("--binary", type=Path, required=True)
build.add_argument("--target", choices=TARGETS, required=True)
build.add_argument("--output", type=Path, required=True)
for name in ("checksums", "verify"):
commands.add_parser(name).add_argument("--output", type=Path, required=True)
args = parser.parse_args()
version = release_version(ROOT, os.environ.get("GITHUB_REF", ""))
if args.command == "version":
print(version)
elif args.command == "package":
print(package(ROOT, args.binary, args.target, args.output, version))
elif args.command == "checksums":
(args.output / "SHA256SUMS").write_text(checksum_lines(args.output, version))
else:
verify_checksums(args.output, version)
if __name__ == "__main__":
main()
ci/set_lance_version.py
+14
View File
@@ -2,6 +2,8 @@ import argparse
import re
import sys
import json
import tomllib
from pathlib import Path
LANCE_GIT_URL = "https://github.com/lance-format/lance.git"
@@ -148,6 +150,18 @@ def update_cargo_toml(line_updater):
with open("Cargo.toml", "w") as f:
f.writelines(new_lines)
core_manifest = Path("rust/lancedb/Cargo.toml")
if core_manifest.exists():
lines = core_manifest.read_text().splitlines(keepends=True)
for index, line in enumerate(lines):
if line.startswith("lance-arrow ="):
updated = tomllib.loads(line_updater(line))["lance-arrow"]
config = {"version": updated} if isinstance(updated, str) else updated
if "path" in config:
config["path"] = "../../" + config["path"]
lines[index] = dict_to_toml_line("lance-arrow", config)
core_manifest.write_text("".join(lines))
def set_stable_version(version: str):
"""
ci/tests/test_build_cli_linux.py
+65
View File
@@ -0,0 +1,65 @@
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
SCRIPT = Path(__file__).resolve().parents[1] / "build_cli_linux.sh"
class BuildCliLinuxTest(unittest.TestCase):
def test_toolchain_failures_stop_before_downloads_and_cargo(self):
# Start with controlled tools so the host's compilers cannot mask a
# missing container dependency. No command can download or build code.
for failed_tool, exit_code in (
("dnf", 42),
("clang", 127),
("clang++", 127),
("ld.lld", 127),
):
with (
self.subTest(failed_tool=failed_tool),
tempfile.TemporaryDirectory() as tmp,
):
workdir = Path(tmp)
bin_dir = workdir / "bin"
bin_dir.mkdir()
log = workdir / "commands.log"
for name in (
"dnf",
"clang",
"clang++",
"ld.lld",
"git",
"curl",
"cargo",
):
status = exit_code if name == failed_tool else 0
if name in ("git", "curl", "cargo"):
status = 99
command = bin_dir / name
command.write_text(
f'#!/bin/sh\necho "{name}" >> "$CLI_TEST_LOG"\nexit {status}\n'
)
command.chmod(0o755)
env = dict(os.environ, PATH=str(bin_dir), CLI_TEST_LOG=str(log))
result = subprocess.run(
["/bin/bash", str(SCRIPT)],
check=False,
cwd=workdir,
env=env,
capture_output=True,
text=True,
timeout=10,
)
self.assertEqual(result.returncode, exit_code, result.stderr)
commands = log.read_text().splitlines()
self.assertEqual(commands[0], "dnf")
self.assertEqual(commands[-1], failed_tool)
self.assertNotIn("git", commands)
self.assertNotIn("curl", commands)
self.assertNotIn("cargo", commands)
if __name__ == "__main__":
unittest.main()
ci/tests/test_check_cargo_lock.py
+125
View File
@@ -0,0 +1,125 @@
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT = REPO_ROOT / "ci" / "check_cargo_lock.py"
CRATES_IO = "registry+https://github.com/rust-lang/crates.io-index"
def package(name, version, deps=(), source=CRATES_IO, checksum="aa"):
text = f'[[package]]\nname = "{name}"\nversion = "{version}"\n'
if source:
text += f'source = "{source}"\nchecksum = "{checksum}"\n'
if deps:
text += f"dependencies = {list(deps)!r}\n".replace("'", '"')
return text
LIBC = package("libc", "0.2.1")
UNTRUSTED = package("untrusted", "0.9.0")
# The prune drops napi and untrusted 0.7.1, the edge from app to it, and the
# version qualifier on app's remaining untrusted edge.
BEFORE = "version = 4\n\n" + "".join(
[
package(
"app", "1.0.0", ["libc", "untrusted 0.7.1", "untrusted 0.9.0"], source=None
),
package("napi", "3.0.0", ["untrusted 0.7.1"]),
LIBC,
package("untrusted", "0.7.1"),
UNTRUSTED,
]
)
APP = package("app", "1.0.0", ["libc", "untrusted"], source=None)
class CheckCargoLockTest(unittest.TestCase):
def run_check(self, after, header="version = 4\n\n", before=BEFORE):
with tempfile.TemporaryDirectory() as tmp:
before_path = Path(tmp, "before.lock")
after_path = Path(tmp, "after.lock")
before_path.write_text(before)
after_path.write_text(header + after)
return subprocess.run(
[sys.executable, SCRIPT, before_path, after_path],
check=False,
capture_output=True,
text=True,
)
def test_prune_passes(self):
result = self.run_check(APP + LIBC + UNTRUSTED)
self.assertEqual(result.returncode, 0, result.stderr)
def test_added_entry_fails(self):
result = self.run_check(APP + LIBC + UNTRUSTED + package("ring", "0.17.0"))
self.assertNotEqual(result.returncode, 0)
self.assertIn("ring", result.stderr)
def test_changed_version_fails(self):
result = self.run_check(APP + package("libc", "0.2.2") + UNTRUSTED)
self.assertNotEqual(result.returncode, 0)
self.assertIn("0.2.2", result.stderr)
def test_changed_source_fails(self):
libc = package("libc", "0.2.1", source="git+https://example.com/libc")
result = self.run_check(APP + libc + UNTRUSTED)
self.assertNotEqual(result.returncode, 0)
self.assertIn("example.com", result.stderr)
def test_changed_checksum_fails(self):
libc = package("libc", "0.2.1", checksum="bb")
result = self.run_check(APP + libc + UNTRUSTED)
self.assertNotEqual(result.returncode, 0)
self.assertIn("'bb'", result.stderr)
def test_added_dependency_fails(self):
libc = package("libc", "0.2.1", ["untrusted"])
result = self.run_check(APP + libc + UNTRUSTED)
self.assertNotEqual(result.returncode, 0)
self.assertIn("dependencies of ('libc'", result.stderr)
def test_removed_dependency_passes(self):
app = package("app", "1.0.0", ["untrusted"], source=None)
result = self.run_check(app + LIBC + UNTRUSTED)
self.assertEqual(result.returncode, 0, result.stderr)
def test_disabled_optional_dependency_still_used_elsewhere_passes(self):
# Removing the CLI disables axum's tracing feature, but other SDK
# dependencies still need tracing. Its package therefore stays pinned.
app = package("app", "1.0.0", ["axum", "tracing"], source=None)
tracing = package("tracing", "0.1.44")
before = (
"version = 4\n\n" + app + package("axum", "0.8.9", ["tracing"]) + tracing
)
after = app + package("axum", "0.8.9") + tracing
result = self.run_check(after, before=before)
self.assertEqual(result.returncode, 0, result.stderr)
def test_dependency_redirected_to_an_existing_version_fails(self):
before = (
"version = 4\n\n"
+ package("app", "1.0.0", ["untrusted 0.7.1"], source=None)
+ package("untrusted", "0.7.1")
+ UNTRUSTED
)
after = (
package("app", "1.0.0", ["untrusted 0.9.0"], source=None)
+ package("untrusted", "0.7.1")
+ UNTRUSTED
)
result = self.run_check(after, before=before)
self.assertNotEqual(result.returncode, 0)
self.assertIn("dependencies of ('app'", result.stderr)
def test_changed_lock_version_fails(self):
result = self.run_check(APP + LIBC + UNTRUSTED, header="version = 3\n\n")
self.assertNotEqual(result.returncode, 0)
self.assertIn("top-level 'version'", result.stderr)
if __name__ == "__main__":
unittest.main()
ci/tests/test_check_sdist.py
+71
View File
@@ -0,0 +1,71 @@
import subprocess
import sys
import tarfile
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT = REPO_ROOT / "ci" / "check_sdist.py"
ROOT = "lancedb-1.0.0"
class CheckSdistTest(unittest.TestCase):
def run_check(self, tracked, shipped, links=(), root=ROOT):
"""Run the check in a git repo tracking `tracked`, on an sdist holding `shipped`
as files and `links` as symlinks, under the folder `root`."""
with tempfile.TemporaryDirectory() as tmp:
subprocess.run(["git", "init", "-q", tmp], check=True)
for name in tracked:
path = Path(tmp, "python/python/lancedb", name)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("x = 1\n")
if tracked:
subprocess.run(["git", "-C", tmp, "add", "."], check=True)
sdist = Path(tmp, f"{ROOT}.tar.gz")
with tarfile.open(sdist, "w:gz") as tar:
for name in shipped:
member = Path(tmp, "member")
member.write_text("x = 1\n")
tar.add(member, arcname=f"{root}/python/lancedb/{name}")
for name in links:
link = tarfile.TarInfo(f"{root}/python/lancedb/{name}")
link.type = tarfile.SYMTYPE
link.linkname = "__init__.py"
tar.addfile(link)
return subprocess.run(
[sys.executable, SCRIPT, sdist],
cwd=tmp,
capture_output=True,
text=True,
)
def test_complete_sdist_passes(self):
result = self.run_check(["__init__.py", "db.py"], ["__init__.py", "db.py"])
self.assertEqual(result.returncode, 0, result.stderr)
def test_missing_tracked_file_fails(self):
result = self.run_check(["__init__.py", "db.py"], ["__init__.py"])
self.assertNotEqual(result.returncode, 0)
self.assertIn("python/python/lancedb/db.py", result.stderr)
def test_tracked_file_shipped_as_symlink_fails(self):
result = self.run_check(
["__init__.py", "db.py"], ["__init__.py"], links=["db.py"]
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("python/python/lancedb/db.py", result.stderr)
def test_files_outside_the_sdist_root_fail(self):
result = self.run_check(["__init__.py"], ["__init__.py"], root="other-1.0.0")
self.assertNotEqual(result.returncode, 0)
self.assertIn("python/python/lancedb/__init__.py", result.stderr)
def test_no_tracked_files_fails(self):
result = self.run_check([], ["__init__.py"])
self.assertNotEqual(result.returncode, 0)
self.assertIn("git tracks no files", result.stderr)
if __name__ == "__main__":
unittest.main()
ci/tests/test_package_cli.py
+228
View File
@@ -0,0 +1,228 @@
import io
import os
import tarfile
import tempfile
import tomllib
import unittest
from pathlib import Path
from unittest.mock import patch
from ci import package_cli as cli
VERSION = "1.2.3-beta.4"
class PackageCliTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
self.root = Path(self.tmp.name)
self.output = self.root / "dist"
self.write_versions(VERSION)
for name in cli.NOTICES:
(self.root / name).write_text(f"Test {name}\n")
self.binary = self.root / "lancedb"
self.binary.write_text(
"#!/usr/bin/env python3\n"
"import os, sys\n"
"assert not any(k.startswith('LANCEDB_') for k in os.environ)\n"
"assert os.environ['HOME'] != os.environ.get('ORIGINAL_HOME')\n"
"assert not os.listdir(os.environ['HOME'])\n"
f"print({{('--version',): 'lancedb {VERSION}', "
"('--help',): 'Usage: lancedb', "
"('profile', 'list'): 'Configured profiles:'}[tuple(sys.argv[1:])])\n"
)
# Packaging must restore executable permissions even if the input lacks them.
self.binary.chmod(0o644)
def write_versions(self, version):
(self.root / ".bumpversion.toml").write_text(
f'[tool.bumpversion]\ncurrent_version = "{version}"\n'
)
for manifest in ("rust/lancedb/Cargo.toml", "rust/lancedb/cli/Cargo.toml"):
path = self.root / manifest
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(f'[package]\nversion = "{version}"\n')
def make_package(self, target=cli.TARGETS[0]):
with patch.object(cli, "check_runtime"):
return cli.package(self.root, self.binary, target, self.output, VERSION)
def test_stable_and_beta_versions_and_tags(self):
for version in ("1.2.3", VERSION):
self.write_versions(version)
for ref in (
f"refs/tags/v{version}",
"refs/heads/main",
"refs/pull/1/merge",
):
with self.subTest(version=version, ref=ref):
self.assertEqual(cli.release_version(self.root, ref), version)
def test_mismatched_tag_and_manifests_fail(self):
with self.assertRaisesRegex(ValueError, "Tag"):
cli.release_version(self.root, "refs/tags/v1.2.3")
for manifest in ("rust/lancedb/Cargo.toml", "rust/lancedb/cli/Cargo.toml"):
self.write_versions(VERSION)
(self.root / manifest).write_text('[package]\nversion = "0.1.0"\n')
with (
self.subTest(manifest=manifest),
self.assertRaisesRegex(ValueError, "version does not match"),
):
cli.release_version(self.root, "refs/heads/main")
def test_cli_participates_in_repository_version_bumps(self):
config = tomllib.loads((cli.ROOT / ".bumpversion.toml").read_text())["tool"][
"bumpversion"
]
entry = next(
f
for f in config["files"]
if f.get("filename") == "rust/lancedb/cli/Cargo.toml"
)
manifest = (cli.ROOT / entry["filename"]).read_text()
version = cli.release_version(cli.ROOT, "")
self.assertIn(entry["search"].format(current_version=version), manifest)
lock = tomllib.loads((cli.ROOT / "Cargo.lock").read_text())
package = next(p for p in lock["package"] if p["name"] == "lancedb-cli")
self.assertEqual(package["version"], version)
def test_archive_layout_permissions_and_isolated_smoke_test(self):
with patch.dict(
os.environ,
{"LANCEDB_API_KEY": "unused", "ORIGINAL_HOME": os.environ["HOME"]},
):
archive = self.make_package()
self.assertEqual(archive.name, f"lancedb-v{VERSION}-{cli.TARGETS[0]}.tar.gz")
cli.verify_archive(archive, VERSION, cli.TARGETS[0])
with tarfile.open(archive) as tar:
for member in tar.getmembers():
self.assertEqual(member.uid, 0)
self.assertEqual(member.gid, 0)
def test_wrong_binary_version_fails_smoke_test(self):
self.binary.write_text(
self.binary.read_text().replace(f"lancedb {VERSION}", "lancedb 0.1.0")
)
with self.assertRaisesRegex(ValueError, "wrong version"):
self.make_package()
def test_bad_archive_members_are_rejected(self):
archive = self.make_package()
stem = cli.archive_stem(VERSION, cli.TARGETS[0])
for bad in (
"missing",
"extra",
"duplicate",
"symlink",
"permissions",
"empty",
"traversal",
):
with self.subTest(bad=bad):
with tarfile.open(archive, "w:gz") as tar:
for name in ("lancedb", *cli.NOTICES):
if bad == "missing" and name == "LICENSE":
continue
member = tarfile.TarInfo(f"{stem}/{name}")
member.mode = 0o755 if name == "lancedb" else 0o644
member.size = 1
if name == "lancedb":
if bad == "symlink":
member.type = tarfile.SYMTYPE
member.linkname = "/usr/bin/true"
member.size = 0
elif bad == "permissions":
member.mode = 0o644
elif bad == "empty":
member.size = 0
elif bad == "traversal":
member.name = "../lancedb"
tar.addfile(member, io.BytesIO(b"x"))
if bad in ("extra", "duplicate"):
member = tarfile.TarInfo(
f"{stem}/{'LICENSE' if bad == 'duplicate' else 'extra'}"
)
tar.addfile(member)
with self.assertRaises(ValueError):
cli.verify_archive(archive, VERSION, cli.TARGETS[0])
def test_checksums_require_exact_target_set_and_detect_tampering(self):
self.make_package()
with self.assertRaisesRegex(ValueError, "exactly"):
cli.checksum_lines(self.output, VERSION)
for target in cli.TARGETS[1:]:
self.make_package(target)
manifest = self.output / "SHA256SUMS"
lines = cli.checksum_lines(self.output, VERSION)
manifest.write_text(lines)
self.assertEqual(len(lines.splitlines()), 3)
cli.verify_checksums(self.output, VERSION)
# Still a valid archive, but a different payload from the published digest.
self.binary.write_text(self.binary.read_text() + "# changed\n")
self.make_package()
with self.assertRaisesRegex(ValueError, "SHA256SUMS"):
cli.verify_checksums(self.output, VERSION)
(self.output / "unexpected.tar.gz").write_bytes(b"extra")
with self.assertRaisesRegex(ValueError, "exactly"):
cli.checksum_lines(self.output, VERSION)
class RuntimeMetadataTest(unittest.TestCase):
def test_linux_baseline_and_architectures(self):
for target, machine in (
(cli.TARGETS[0], "Advanced Micro Devices X86-64"),
(cli.TARGETS[1], "AArch64"),
):
cli.check_linux_metadata(
f" Machine: {machine}\n",
" (NEEDED) Shared library: [libc.so.6]\n (NEEDED) Shared library: [libgcc_s.so.1]",
"GLIBC_2.28 GLIBCXX_3.4.25 CXXABI_1.3.11 GCC_7.0.0",
target,
)
def test_linux_rejects_wrong_arch_new_symbols_and_non_system_libraries(self):
header = " Machine: Advanced Micro Devices X86-64\n"
dynamic = " (NEEDED) Shared library: [libc.so.6]"
cases = [(" Machine: AArch64\n", dynamic, "GLIBC_2.28")]
for symbol in (
"GLIBC_2.29",
"GLIBCXX_3.4.26",
"CXXABI_1.3.12",
"GCC_12.0.0",
"GLIBC_PRIVATE",
):
cases.append((header, dynamic, symbol))
for extra in (
" (NEEDED) Shared library: [libssl.so.3]",
" (RUNPATH) Library runpath: [/opt/build/lib]",
):
cases.append((header, dynamic + extra, "GLIBC_2.28"))
for args in cases:
with self.subTest(args=args), self.assertRaises(ValueError):
cli.check_linux_metadata(*args, cli.TARGETS[0])
def test_macos_baseline_and_system_libraries(self):
cli.check_macos_metadata(
"arm64\n",
"cli:\n\t/usr/lib/libSystem.B.dylib (compatibility version 1.0.0)",
" minos 14.0\n",
)
def test_macos_rejects_wrong_arch_version_and_homebrew_dependencies(self):
for args in (
("x86_64", "", " minos 14.0\n"),
("arm64", "", " minos 15.0\n"),
("arm64", "", ""),
(
"arm64",
"\t/opt/homebrew/lib/libssl.dylib (compatibility version 3.0.0)",
" minos 14.0\n",
),
):
with self.subTest(args=args), self.assertRaises(ValueError):
cli.check_macos_metadata(*args)
if __name__ == "__main__":
unittest.main()
ci/tests/test_python_dependencies.py
+47
View File
@@ -0,0 +1,47 @@
import tomllib
import unittest
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
REPO_ROOT = Path(__file__).resolve().parents[2]
class PythonDependenciesTest(unittest.TestCase):
def setUp(self):
with (REPO_ROOT / "python/pyproject.toml").open("rb") as f:
self.project = tomllib.load(f)["project"]
self.pylance = Requirement(self.project["optional-dependencies"]["pylance"][0])
def test_pylance_extra_excludes_older_lance_majors(self):
with (REPO_ROOT / "Cargo.toml").open("rb") as f:
lance = tomllib.load(f)["workspace"]["dependencies"]["lance"]
if "version" not in lance:
self.skipTest("Local Lance checkout has no version requirement")
major = Version(lance["version"].lstrip("=")).major
for older_major in range(major):
for suffix in (".0.0", ".0.0b6", ".999.999"):
version = f"{older_major}{suffix}"
with self.subTest(version=version):
self.assertFalse(
self.pylance.specifier.contains(version, prereleases=True),
f"The pylance extra must exclude Lance {version}; "
f"the bundled Lance is major {major}",
)
def test_pylance_extra_accepts_tested_version(self):
requirements = [
Requirement(dep) for dep in self.project["optional-dependencies"]["tests"]
]
tested = next(dep for dep in requirements if dep.name == "pylance")
pin = next(spec for spec in tested.specifier if spec.operator == "==")
self.assertTrue(
self.pylance.specifier.contains(pin.version, prereleases=True),
f"The tests and pylance extras must be installable together: {tested}",
)
if __name__ == "__main__":
unittest.main()
ci/tests/test_set_lance_version.py
+28
View File
@@ -4,6 +4,7 @@ import subprocess
import sys
import tempfile
import textwrap
import tomllib
import unittest
from pathlib import Path
@@ -83,6 +84,13 @@ lance_datafusion = {{ "version" = "=8.2.0-beta.4", "features" = ["substrait"], "
with self.subTest(version=version), tempfile.TemporaryDirectory() as tmp:
workdir = Path(tmp)
(workdir / "Cargo.toml").write_text(CARGO_TOML)
core_manifest = workdir / "rust/lancedb/Cargo.toml"
core_manifest.parent.mkdir(parents=True)
core_manifest.write_text(
"[dependencies]\nlance-arrow = { "
'version = "=1.0.0", git = "https://github.com/lance-format/lance.git", '
'tag = "v1.0.0" }\narrow.workspace = true\n'
)
command_log = workdir / "commands.log"
fake_bin = workdir / "bin"
fake_bin.mkdir()
@@ -111,6 +119,26 @@ lance_datafusion = {{ "version" = "=8.2.0-beta.4", "features" = ["substrait"], "
+ updated_dependencies
+ UNTOUCHED_DEPENDENCIES,
)
core = tomllib.loads(core_manifest.read_text())["dependencies"]
dependency = core["lance-arrow"]
if isinstance(dependency, str):
dependency = {"version": dependency}
if version == "local":
self.assertEqual(
dependency["path"], "../../../lance/rust/lance-arrow"
)
else:
expected_version = {
"stable": "9.9.9",
"preview": "10.0.0-beta.3",
}.get(version, version.removeprefix("v"))
self.assertEqual(dependency["version"], f"={expected_version}")
if "beta" in expected_version:
self.assertEqual(dependency["git"], LANCE_GIT_URL)
self.assertEqual(dependency["tag"], f"v{expected_version}")
else:
self.assertNotIn("git", dependency)
self.assertEqual(core["arrow"], {"workspace": True})
commands = command_log.read_text().splitlines()
for command in expected_commands:
self.assertTrue(
deny.toml
+43 -21
View File
@@ -68,21 +68,11 @@ ignore = [
# https://rustsec.org/advisories/RUSTSEC-2025-0119
{ id = "RUSTSEC-2025-0119", reason = "transitive via hf-hub/indicatif; cosmetic formatting crate" },
# rustls-webpki 0.101.7 (old major line): name-constraint checks for
# URI / wildcard names. Pulled in only via the legacy rustls 0.21 chain
# from aws-smithy-http-client. The 0.103 line we actively use is patched.
# Clearing the 0.101 copy requires the aws-sdk chain to migrate off
# rustls 0.21.
# https://rustsec.org/advisories/RUSTSEC-2026-0098
# https://rustsec.org/advisories/RUSTSEC-2026-0099
{ id = "RUSTSEC-2026-0098", reason = "only affects rustls-webpki 0.101 from legacy aws-smithy/rustls 0.21 chain" },
{ id = "RUSTSEC-2026-0099", reason = "only affects rustls-webpki 0.101 from legacy aws-smithy/rustls 0.21 chain" },
# rustls-webpki 0.101.7: reachable panic in CRL parsing. Same legacy
# rustls 0.21 chain from aws-smithy-http-client as above. The 0.103 line
# we actively use is upgraded to 0.103.13 which contains the fix.
# https://rustsec.org/advisories/RUSTSEC-2026-0104
{ id = "RUSTSEC-2026-0104", reason = "only affects rustls-webpki 0.101 from legacy aws-smithy/rustls 0.21 chain" },
# unsafe-libyaml: unmaintained. Transitive via lance-tokenizer → lindera
# → serde_yaml_ng. No maintained upgrade exists; clearing it requires
# lindera to move off serde_yaml_ng.
# https://rustsec.org/advisories/RUSTSEC-2024-0449
{ id = "RUSTSEC-2024-0449", reason = "transitive via lindera (lance-tokenizer); no maintained upgrade available" },
# pyo3 advisories in the Python bindings; tracked pending a patched pyo3 release.
# https://rustsec.org/advisories/RUSTSEC-2026-0176
@@ -90,6 +80,18 @@ ignore = [
{ id = "RUSTSEC-2026-0176", reason = "pyo3 in Python bindings; awaiting patched pyo3 release" },
{ id = "RUSTSEC-2026-0177", reason = "pyo3 in Python bindings; awaiting patched pyo3 release" },
# zip < 9.0.0: SUID/SGID bits, symlink targets and permissions applied through
# symlinks during archive extraction. Transitive via candle-core (optional
# local-embedding feature); every published candle release pins zip < 9
# (0.9.x/0.10.x ^7.2, 0.11.0 ^8.6), so no dependency bump clears it.
# LanceDB never extracts archives; the zip crate is only linked.
# https://rustsec.org/advisories/RUSTSEC-2026-0367
# https://rustsec.org/advisories/RUSTSEC-2026-0368
# https://rustsec.org/advisories/RUSTSEC-2026-0369
{ id = "RUSTSEC-2026-0367", reason = "zip via candle-core; candle pins zip < 9 and lancedb never extracts archives" },
{ id = "RUSTSEC-2026-0368", reason = "zip via candle-core; candle pins zip < 9 and lancedb never extracts archives" },
{ id = "RUSTSEC-2026-0369", reason = "zip via candle-core; candle pins zip < 9 and lancedb never extracts archives" },
# quick-xml < 0.41.0: quadratic runtime on duplicate attribute names (DoS).
# quick-xml < 0.41.0: unbounded namespace-declaration allocation in NsReader (DoS).
# Pulled in transitively by inferno (dev-only flame-graph dep), lance-namespace-impls
@@ -108,14 +110,30 @@ ignore = [
# compact_str/smol_str, so clearing this requires polars to migrate.
# https://rustsec.org/advisories/RUSTSEC-2026-0249
{ id = "RUSTSEC-2026-0249", reason = "smartstring unmaintained via polars; no fixed upstream release" },
# crypto-bigint < 0.7.6: timing side-channels in constant-time ops,
# incl. subtraction, zero counting, division and Montgomery multiplication.
# Reached only through aws-sdk-s3 sigv4a -> aws-sigv4 (p256/ecdsa); signed
# requests carry the SigV4a credential-derived key. The patched 0.7.6 was
# published 2026-10-23, but no published aws-sigv4 release uses it yet
# (latest 1.6.3 pins crypto-bigint ^0.5), so clearing requires smithy-rs to
# ship an updated aws-sigv4.
# https://rustsec.org/advisories/RUSTSEC-2026-0387
{ id = "RUSTSEC-2026-0387", reason = "crypto-bigint via aws-sigv4 (sigv4a); no fixed upstream aws-sigv4 release yet" },
# h2 0.3: empty DATA frames can be queued without limit. The patched
# h2 0.4 line is locked to 0.4.16, but no patched 0.3 release exists.
# The old copy is pulled in by aws-smithy's legacy hyper 0.14 client.
# https://rustsec.org/advisories/RUSTSEC-2026-0258
{ id = "RUSTSEC-2026-0258", reason = "h2 0.3 via legacy aws-smithy/hyper 0.14; no patched 0.3 release" },
# zstd: arrow-ipc 58.x pins zstd ^0.13, while the patches require zstd
# >=0.14.3 and zstd-safe >=8.2.0. Arrow 59+ updates the dependency, but it
# cannot be used until DataFusion and Lance move to the matching Arrow line.
# LanceDB uses zstd::bulk through arrow-ipc and does not use the affected
# with_context, seekable, or decompress_stream APIs.
# https://rustsec.org/advisories/RUSTSEC-2026-0391
# https://rustsec.org/advisories/RUSTSEC-2026-0398
# https://rustsec.org/advisories/RUSTSEC-2026-0403
{ id = "RUSTSEC-2026-0391", reason = "zstd 0.13 via arrow-ipc 58; affected with_context API is unused" },
{ id = "RUSTSEC-2026-0398", reason = "zstd-safe 7 via arrow-ipc 58; affected seekable API is not enabled" },
{ id = "RUSTSEC-2026-0403", reason = "zstd-safe 7 via arrow-ipc 58; affected streaming decoder API is unused" },
]
# ---------------------------------------------------------------------------
# Licenses: only allow licenses we've reviewed as compatible with Apache-2.0.
# ---------------------------------------------------------------------------
@@ -171,7 +189,9 @@ wildcards = "deny"
# `[workspace.dependencies]` table: any crate used by more than one member must
# go through `workspace = true`, and entries nothing uses are an error. This
# keeps versions from drifting between the core crate and the bindings.
workspace-dependencies = { duplicates = "deny", unused = "deny" }
# Python bundles a patched path crate while the Rust SDK uses the overridable
# upstream source. Check shared external declarations, not these path bindings.
workspace-dependencies = { duplicates = "deny", unused = "deny", include-path-dependencies = false }
# Internal workspace crates reference each other via `path = "..."`, which
# cargo-deny sees as a wildcard version. That's fine for private workspace
# members (not published to crates.io), so allow it specifically for paths.
@@ -197,4 +217,6 @@ allow-registry = ["https://github.com/rust-lang/crates.io-index"]
# releases are cut to crates.io. Allow that specific host.
allow-git = [
"https://github.com/lance-format/lance",
# Pinned by Lance for the multipart write mode API (see [patch.crates-io] in Cargo.toml).
"https://github.com/apache/arrow-rs-object-store",
]
docs/README.md
+14 -8
View File
@@ -1,10 +1,13 @@
# LanceDB Documentation
LanceDB docs are available at [docs.lancedb.com](https://docs.lancedb.com).
For website guides in `docs/web`, start with
[Documentation in CONTRIBUTING.md](../CONTRIBUTING.md#documentation). That
covers editing pages, generated examples, previews, availability and drafts.
The SDK docs are built and deployed automatically by [Github Actions](../.github/workflows/docs.yml)
whenever a commit is pushed to the `main` branch. So it is possible for the docs to show
unreleased features.
This page covers the SDK reference in `docs/src`, built with MkDocs. Its
[GitHub Actions workflow](../.github/workflows/docs.yml) builds and deploys on
pushes to `main`, so it can include unreleased APIs. That is separate from the
release editions of the Mintlify website.
## Building the docs
@@ -15,18 +18,21 @@ unreleased features.
### Preview the docs
Run these commands from the LanceDB repository root. The subshell changes into
`docs` to load `mkdocs.yml`, while leaving your shell in the repository root.
```shell
cd docs
mkdocs serve
(cd docs && PYTHONPATH=. mkdocs serve)
```
If you want to just generate the HTML files:
```shell
PYTHONPATH=. mkdocs build -f docs/mkdocs.yml
(cd docs && PYTHONPATH=. mkdocs build)
```
If successful, you should see a `docs/site` directory that you can verify locally.
If successful, you should see a `docs/site` directory in the repository root that
you can verify locally.
## Adding examples
docs/openapi.yml
+3 -3
View File
@@ -1,9 +1,9 @@
openapi: 3.1.0
info:
version: 1.0.0
title: LanceDB Cloud API
title: LanceDB REST API
description: |
LanceDB Cloud API is a RESTful API that allows users to access and modify data stored in LanceDB Cloud.
The LanceDB REST API is a RESTful API that allows users to access and modify data stored in LanceDB Enterprise or LanceDB Cloud (legacy).
Table actions are considered temporary resource creations and all use POST method.
contact:
name: LanceDB support
@@ -12,7 +12,7 @@ info:
servers:
- url: https://{db}.{region}.api.lancedb.com
description: LanceDB Cloud REST endpoint.
description: LanceDB Cloud (legacy) REST endpoint. For LanceDB Enterprise, use your deployment's endpoint.
variables:
db:
default: ""
docs/src/assets/hero-header.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.7 MiB

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/src/assets/reverie-banner.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 722 KiB

docs/src/java/java.md
+7 -7
View File
@@ -1,10 +1,10 @@
# Java SDK
The LanceDB Java SDK provides a convenient way to interact with LanceDB Cloud and Enterprise deployments using the Lance REST Namespace API.
The LanceDB Java SDK provides a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API.
!!! note
The Java SDK currently only works for LanceDB remote database that connects to LanceDB Cloud and Enterprise.
Local database support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
The Java SDK currently only works with LanceDB Distributed, connecting to LanceDB Enterprise or LanceDB Cloud (legacy).
LanceDB Embedded support is a work in progress. Check [LANCEDB-2848](https://github.com/lancedb/lancedb/issues/2848) for the latest progress.
## Installation
@@ -14,13 +14,13 @@ Add the following dependency to your `pom.xml`:
<dependency>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId>
<version>0.40.0-beta.3</version>
<version>0.41.0-beta.17</version>
</dependency>
```
## Quick Start
### Connecting to LanceDB Cloud
### Connecting to LanceDB Cloud (legacy)
```java
import com.lancedb.LanceDbNamespaceClientBuilder;
@@ -51,8 +51,8 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
|--------|-------------|----------|
| `apiKey(String)` | LanceDB API key | Yes |
| `database(String)` | Database name | Yes |
| `endpoint(String)` | Custom endpoint URL for Enterprise deployments | No |
| `region(String)` | AWS region (default: "us-east-1") | No |
| `endpoint(String)` | Custom endpoint URL for LanceDB Enterprise deployments | No |
| `region(String)` | LanceDB Cloud (legacy) region (default: "us-east-1") | No |
| `config(String, String)` | Additional configuration parameters | No |
### Opening a Table with Vended Credentials
docs/src/js/_media/CONTRIBUTING.md
+13
View File
@@ -50,6 +50,19 @@ pnpm install
pnpm build
```
Local SQL computed columns and materialized-view creation/refresh are experimental
and disabled in published packages. To opt in when building from this directory:
```shell
pnpm build:debug --features experimental-functions
pnpm tsc
pnpm test --runInBand table.test.ts materialized_view.test.ts
```
Rebuild without `--features experimental-functions` to test the default package.
The tests detect the compiled feature; remote operations, metadata protections,
and ordinary eager `addColumns` transforms remain available in both builds.
Lint:
```shell
docs/src/js/classes/AutoQuery.md
+23 -16
View File
@@ -350,7 +350,7 @@ input to this method would be:
#### Parameters
* **columns**: `string` \| `string`[] \| `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **columns**: `string` \| `string`[] \| `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
#### Returns
@@ -417,37 +417,39 @@ Collect the results as an Arrow
ArrowTable.
Empty results retain the query's output schema.
#### Inherited from
`StandardQueryBase.toArrow`
***
### useLsm()
### useRTE()
```ts
useLsm(enable): this
useRTE(enable): this
```
Control MemWAL read routing for this query.
Control RTE read routing for this query.
By default (unset), when the table carries a MemWAL write spec (see
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
By default (unset), when the table carries an RTE write spec (see
[Table#enableRTE](Table.md#enablerte)), reads are routed through the LSM scanner so
they also return data written via the `mergeInsert` LSM path that has not yet
been compacted into the base table (the active/frozen in-memory memtables and
the flushed generations), deduplicated by primary key; a table without a spec
the SSTables), deduplicated by primary key; a table without a spec
reads the base table.
#### Parameters
* **enable**: `boolean`
`true` forces the LSM scanner and errors if the table has no
MemWAL write spec. `false` bypasses the MemWAL and reads the base table only,
RTE write spec. `false` bypasses the RTE and reads the base table only,
even when a spec is present.
Note: the LSM scanner does not support every query shape (e.g. reranking,
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
`useLsm(false)` is set, because a base-only read would silently exclude
un-compacted MemWAL data.
hybrid search, `orderBy`). On an RTE table those shapes error unless
`useRTE(false)` is set, because a base-only read would silently exclude
un-compacted RTE data.
#### Returns
@@ -455,7 +457,7 @@ reads the base table.
#### Inherited from
`StandardQueryBase.useLsm`
`StandardQueryBase.useRTE`
***
@@ -467,29 +469,34 @@ where(predicate): this
A filter statement to be applied to this query.
The filter should be supplied as an SQL query string. For example:
The filter may be an SQL query string or an expression built with `col`,
which preserves case-sensitive column names. For example:
#### Parameters
* **predicate**: `string`
* **predicate**: `string` \| [`Expr`](Expr.md)
#### Returns
`this`
#### Example
#### Examples
```ts
x > 10
y > 0 AND y < 100
x > 5 OR y = 'test'
```
```ts
table.query().where(col("firstName").eq("Alice"));
```
Filtering performance can often be improved by creating a scalar index
on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather
than replacing the previous filter.
```
#### Inherited from
docs/src/js/classes/BlobFile.md
+101 -5
View File
@@ -10,16 +10,52 @@ A lazy handle to blob bytes. Create one with [Table.fetchBlobFiles](Table.md#fet
## Methods
### close()
```ts
close(): Promise<void>
```
Releases the handle. Reads after `close()` fail. Calling it again does
nothing.
#### Returns
`Promise`&lt;`void`&gt;
***
### isClosed()
```ts
isClosed(): boolean
```
Returns true after [BlobFile.close](BlobFile.md#close).
#### Returns
`boolean`
***
### read()
```ts
read(): Promise<Buffer>
read(maxBytes?, options?): Promise<Buffer>
```
Reads from the cursor to the end and advances the cursor.
Reads from the cursor and advances the cursor.
A second call returns an empty buffer. [BlobFile.readRange](BlobFile.md#readrange) does
not move the cursor.
Reads to the end when `maxBytes` is omitted, or at most `maxBytes` bytes
otherwise. Returns an empty buffer at the end of the blob.
[BlobFile.readRange](BlobFile.md#readrange) does not move the cursor.
#### Parameters
* **maxBytes?**: `bigint`
* **options?**: [`BlobReadOptions`](../type-aliases/BlobReadOptions.md)
#### Returns
@@ -30,7 +66,10 @@ not move the cursor.
### readRange()
```ts
readRange(start, end): Promise<Buffer>
readRange(
start,
end,
options?): Promise<Buffer>
```
Reads the half-open byte range `[start, end)`.
@@ -43,12 +82,55 @@ Fails when `end` is past the blob size. Does not move the cursor.
* **end**: `bigint`
* **options?**: [`BlobReadOptions`](../type-aliases/BlobReadOptions.md)
#### Returns
`Promise`&lt;`Buffer`&gt;
***
### readRanges()
```ts
readRanges(ranges, options?): Promise<Buffer[]>
```
Reads several half-open byte ranges. Returns one buffer per range, in
the order given.
Fails when any `end` is past the blob size. Does not move the cursor.
#### Parameters
* **ranges**: [`BlobRange`](../type-aliases/BlobRange.md)[]
* **options?**: [`BlobReadOptions`](../type-aliases/BlobReadOptions.md)
#### Returns
`Promise`&lt;`Buffer`[]&gt;
***
### seek()
```ts
seek(position): Promise<void>
```
Moves the cursor to `position`, in bytes from the start of the blob.
#### Parameters
* **position**: `bigint`
#### Returns
`Promise`&lt;`void`&gt;
***
### size()
```ts
@@ -60,3 +142,17 @@ Returns the blob size in bytes.
#### Returns
`bigint`
***
### tell()
```ts
tell(): Promise<bigint>
```
Returns the cursor position in bytes.
#### Returns
`Promise`&lt;`bigint`&gt;
docs/src/js/classes/Catalog.md
+20 -5
View File
@@ -70,7 +70,9 @@ Create a database, or open an existing database when existOk is true.
dropDatabase(name, options): Promise<void>
```
Drop an empty database. The server rejects nonempty databases.
Drop a database. A nonempty database is an error unless `cascade` is set,
which drops everything in it first; the name stays taken until the server
finishes cleaning up their storage.
#### Parameters
@@ -78,6 +80,8 @@ Drop an empty database. The server rejects nonempty databases.
* **options** = `{}`
* **options.cascade?**: `boolean`
* **options.ignoreMissing?**: `boolean`
#### Returns
@@ -89,19 +93,30 @@ Drop an empty database. The server rejects nonempty databases.
### listDatabases()
```ts
listDatabases(options): Promise<ListDatabasesResponse>
listDatabases(options): DatabaseNames
```
List one page of databases; pass pageToken from a response for the next page.
Iterate lazily over all database names using `for await...of`.
pageToken resumes from a saved token; omitted starts at the beginning.
pageLimit limits each REST response, not the total; omitted uses the server default.
Request errors are raised during iteration and terminate the iterator.
#### Parameters
* **options** = `{}`
* **options.limit?**: `number`
* **options.pageLimit?**: `number`
* **options.pageToken?**: `string`
#### Returns
`Promise`&lt;[`ListDatabasesResponse`](../interfaces/ListDatabasesResponse.md)&gt;
[`DatabaseNames`](DatabaseNames.md)
#### Example
```ts
for await (const name of catalog.listDatabases({ pageLimit: 20 })) {
console.log(name);
}
```
docs/src/js/classes/Connection.md
+212 -78
View File
@@ -180,13 +180,13 @@ abstract createMaterializedView(
Define a materialized view named `name` over the table `source`.
LanceDB Embedded creation and refresh require a source build with the
`experimental-functions` Cargo feature, disabled in published packages.
LanceDB Distributed operations do not require this feature.
The view is populated before creation returns. Set `withNoData` to create
only its definition and empty backing table. The view is a normal table:
it can be queried, indexed and searched, and it appears in `tableNames`.
The source table must have stable row ids (create it with
the `newTableEnableStableRowIds` storage option); they keep the view's
provenance valid across source compactions and cannot be enabled after
a table exists.
#### Parameters
@@ -319,6 +319,38 @@ Creates a new Table and initialize it with new data.
***
### createView()
```ts
abstract createView(
name,
query,
namespacePath?): Promise<ViewDescription>
```
Create a view: a named query the database plans on every read.
The query is planned once, at creation, so one that cannot be planned is
rejected now rather than at the first read. A view holds no rows, and its
readers see its sources as they are at read time.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create.
#### Parameters
* **name**: `string`
* **query**: `string`
* **namespacePath?**: `string`[]
#### Returns
`Promise`&lt;[`ViewDescription`](../interfaces/ViewDescription.md)&gt;
***
### describeNamespace()
```ts
@@ -342,6 +374,27 @@ The namespace's properties
***
### describeView()
```ts
abstract describeView(name, namespacePath?): Promise<ViewDescription>
```
What this database records about the view named `name`: its defining
query and the schema that query resolved to.
#### Parameters
* **name**: `string`
* **namespacePath?**: `string`[]
#### Returns
`Promise`&lt;[`ViewDescription`](../interfaces/ViewDescription.md)&gt;
***
### display()
```ts
@@ -383,8 +436,8 @@ abstract dropMaterializedView(name, namespacePath?): Promise<void>
Drop the materialized view named `name`.
The view may become unavailable before physical cleanup finishes. Use
[dropMaterializedViewAsync](Connection.md#dropmaterializedviewasync) to retain and wait for the cleanup job.
Remote drops wait for the SQL command to remove the catalog binding.
Physical cleanup continues in the background.
Rejects a table that exists but is not a materialized view.
@@ -406,8 +459,9 @@ Rejects a table that exists but is not a materialized view.
abstract dropMaterializedViewAsync(name, namespacePath?): Promise<Job>
```
Start dropping the materialized view named `name` and return its cleanup
job without waiting for completion.
Start dropping the materialized view named `name` and return its job.
Remote jobs have type `query` and use the SQL query ID as `id`. Waiting
confirms catalog removal; physical cleanup continues in the background.
Rejects a table that exists but is not a materialized view.
@@ -498,6 +552,52 @@ on the returned job to know when cleanup has finished.
***
### dropView()
```ts
abstract dropView(name, namespacePath?): Promise<void>
```
Drop the view named `name` and wait for catalog removal.
The tables it reads are untouched: a view holds no rows of its own. Use
[dropViewAsync](Connection.md#dropviewasync) to retain its SQL query job. Completion confirms
catalog removal; physical cleanup continues in the background.
#### Parameters
* **name**: `string`
* **namespacePath?**: `string`[]
#### Returns
`Promise`&lt;`void`&gt;
***
### dropViewAsync()
```ts
abstract dropViewAsync(name, namespacePath?): Promise<Job>
```
Drop a view and return its SQL query job.
The job has type `query` and its ID is the SQL query UUID. Waiting confirms
catalog removal; physical cleanup continues in the background.
#### Parameters
* **name**: `string`
* **namespacePath?**: `string`[]
#### Returns
`Promise`&lt;[`Job`](Job.md)&gt;
***
### isOpen()
```ts
@@ -515,124 +615,114 @@ Return true if the connection has not been closed
### listJobs()
```ts
abstract listJobs(): Promise<JobInfo[]>
abstract listJobs(options?): Listing<JobInfo>
```
List server-side jobs across the database's tables.
Iterate lazily over server-side jobs across the database's tables.
Use `for await...of`. Options control page size and the starting token.
See [Listing](Listing.md) for pagination state and error behavior.
#### Parameters
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
`Promise`&lt;[`JobInfo`](../interfaces/JobInfo.md)[]&gt;
[`Listing`](Listing.md)&lt;[`JobInfo`](../interfaces/JobInfo.md)&gt;
***
### listMaterializedViews()
```ts
abstract listMaterializedViews(): Promise<string[]>
abstract listMaterializedViews(options?): Listing<string>
```
The names of the materialized views in this database.
Found by reading every table's schema, so this costs an open per table.
Iterate lazily with `for await...of`. Local listings inspect table schemas
one page at a time. Options control page size and the starting token.
See [Listing](Listing.md) for pagination state and error behavior.
#### Parameters
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
`Promise`&lt;`string`[]&gt;
[`Listing`](Listing.md)&lt;`string`&gt;
***
### listNamespaces()
```ts
abstract listNamespaces(namespacePath?, options?): Promise<ListNamespacesResponse>
abstract listNamespaces(namespacePath?, options?): Listing<string>
```
List the immediate child namespaces under the given parent.
Results may be paginated. To retrieve subsequent pages, pass the
`pageToken` returned by a previous call.
Lazily list immediate child namespaces, 100 per page by default.
#### Parameters
* **namespacePath?**: `string`[]
The parent namespace path. Defaults
to the root namespace if omitted.
* **options?**: `Partial`&lt;[`ListNamespacesOptions`](../interfaces/ListNamespacesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
`Promise`&lt;[`ListNamespacesResponse`](../interfaces/ListNamespacesResponse.md)&gt;
Child namespace names and
an optional token for fetching the next page.
[`Listing`](Listing.md)&lt;`string`&gt;
***
### listTables()
#### listTables(options)
```ts
abstract listTables(options?): Promise<ListTablesResponse>
abstract listTables(namespacePath?, options?): Listing<string>
```
List a page of the tables in this database.
Lazily list tables in a namespace, requesting 100 per page by default.
To retrieve the tables after the page, pass the `pageToken` the response
carries back in. A page can be shorter than `limit` without being the last
one, so walk until a response carries no page token:
```ts
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
```
##### Parameters
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
##### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
A page of table names and an
optional token for the tables after it.
#### listTables(namespacePath, options)
```ts
abstract listTables(namespacePath?, options?): Promise<ListTablesResponse>
```
List a page of the tables in this database.
##### Parameters
#### Parameters
* **namespacePath?**: `string`[]
The namespace path to list tables from
(defaults to root namespace)
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
##### Returns
#### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
[`Listing`](Listing.md)&lt;`string`&gt;
A page of table names and an
optional token for the tables after it.
#### Example
```ts
for await (const name of conn.listTables([], { pageLimit: 20 })) {
console.log(name);
}
```
***
### listViews()
```ts
abstract listViews(namespacePath?, options?): Listing<string>
```
The names of the views in one namespace.
Names only; a definition comes from [describeView](Connection.md#describeview).
Iterate lazily with `for await...of`. See [Listing](Listing.md) for pagination
state and error behavior; options control page size and the starting token.
#### Parameters
* **namespacePath?**: `string`[]
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
[`Listing`](Listing.md)&lt;`string`&gt;
***
@@ -703,6 +793,28 @@ abstract openTable(
***
### pauseJob()
```ts
abstract pauseJob(jobId): Promise<string>
```
Pause a server-side job by id.
The job's workers drain and it stays parked until resumed. Resolves to
"pausing", "already_paused", or "committing" -- a job finalizing its
results cannot be parked; retry shortly.
#### Parameters
* **jobId**: `string`
#### Returns
`Promise`&lt;`string`&gt;
***
### renameTable()
```ts
@@ -714,7 +826,7 @@ abstract renameTable(
Rename a table.
Currently only supported by LanceDB Cloud. Local OSS connections and
Currently only supported in LanceDB Distributed. LanceDB Embedded connections and
namespace-backed connections (via [connectNamespace](../functions/connectNamespace.md)) reject with
a "not supported" error.
@@ -736,6 +848,28 @@ a "not supported" error.
***
### resumeJob()
```ts
abstract resumeJob(jobId): Promise<string>
```
Resume a paused server-side job by id.
Its workers pick their work back up from checkpoints. Resolves to
"resumed", "still_pausing" -- the pause's worker drain is not confirmed
yet; retry shortly -- or "not_paused".
#### Parameters
* **jobId**: `string`
#### Returns
`Promise`&lt;`string`&gt;
***
### ~~tableNames()~~
#### tableNames(options)
docs/src/js/classes/DatabaseNames.md
+79
View File
@@ -0,0 +1,79 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / DatabaseNames
# Class: DatabaseNames
A lazy async iterator of database names with pagination state.
Returned by [Catalog.listDatabases](Catalog.md#listdatabases).
## Implements
- `AsyncIterableIterator`&lt;`string`&gt;
## Methods
### \[asyncIterator\]()
```ts
asyncIterator: AsyncIterableIterator<string>
```
#### Returns
`AsyncIterableIterator`&lt;`string`&gt;
#### Implementation of
`AsyncIterableIterator.[asyncIterator]`
***
### next()
```ts
next(): Promise<IteratorResult<string, any>>
```
Fetch the next name, requesting another page only when needed.
#### Returns
`Promise`&lt;`IteratorResult`&lt;`string`, `any`&gt;&gt;
#### Implementation of
`AsyncIterableIterator.next`
***
### numPageResults()
```ts
numPageResults(): number
```
Number of names available without another REST request.
#### Returns
`number`
***
### pageToken()
```ts
pageToken(): undefined | string
```
Token for the next REST request. Initially this is the supplied starting token.
Returns undefined after the final page is fetched, even if names remain cached.
Drain the cache before saving a token to avoid skipping names when resuming.
A failed request terminates iteration and retains its token for a new iterator.
#### Returns
`undefined` \| `string`
docs/src/js/classes/Expr.md
+365
View File
@@ -0,0 +1,365 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / Expr
# Class: Expr
A SQL expression backed by the shared LanceDB expression builder.
## Example
```ts
table.query().where(col("firstName").eq("Alice")).toArray();
```
## Constructors
### new Expr()
```ts
new Expr(inner): Expr
```
#### Parameters
* **inner**: `Expr`
#### Returns
[`Expr`](Expr.md)
## Properties
### inner
```ts
inner: Expr;
```
## Methods
### add()
```ts
add(other): Expr
```
Add a value.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### and()
```ts
and(other): Expr
```
Combine two predicates with AND.
#### Parameters
* **other**: [`Expr`](Expr.md)
#### Returns
[`Expr`](Expr.md)
***
### cast()
```ts
cast(dataType): Expr
```
Cast to an Arrow data type, such as "Int64".
#### Parameters
* **dataType**: `string`
#### Returns
[`Expr`](Expr.md)
***
### contains()
```ts
contains(substr): Expr
```
Test whether a string contains a substring.
#### Parameters
* **substr**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### div()
```ts
div(other): Expr
```
Divide by a value.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### eq()
```ts
eq(other): Expr
```
Compare for equality.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### gt()
```ts
gt(other): Expr
```
Compare for greater than.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### gte()
```ts
gte(other): Expr
```
Compare for greater than or equal to.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### isIn()
```ts
isIn(values): Expr
```
Test membership in a list of values.
#### Parameters
* **values**: [`ExprLike`](../type-aliases/ExprLike.md)[]
#### Returns
[`Expr`](Expr.md)
***
### lower()
```ts
lower(): Expr
```
Convert a string to lowercase.
#### Returns
[`Expr`](Expr.md)
***
### lt()
```ts
lt(other): Expr
```
Compare for less than.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### lte()
```ts
lte(other): Expr
```
Compare for less than or equal to.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### mul()
```ts
mul(other): Expr
```
Multiply by a value.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### ne()
```ts
ne(other): Expr
```
Compare for inequality.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### not()
```ts
not(): Expr
```
Negate this predicate.
#### Returns
[`Expr`](Expr.md)
***
### or()
```ts
or(other): Expr
```
Combine two predicates with OR.
#### Parameters
* **other**: [`Expr`](Expr.md)
#### Returns
[`Expr`](Expr.md)
***
### sub()
```ts
sub(other): Expr
```
Subtract a value.
#### Parameters
* **other**: [`ExprLike`](../type-aliases/ExprLike.md)
#### Returns
[`Expr`](Expr.md)
***
### toSql()
```ts
toSql(): string
```
Serialize with the LanceDB SQL dialect, preserving column case.
#### Returns
`string`
***
### upper()
```ts
upper(): Expr
```
Convert a string to uppercase.
#### Returns
[`Expr`](Expr.md)
docs/src/js/classes/Index.md
+1 -1
View File
@@ -252,7 +252,7 @@ static labelList(): Index
Create a label list index.
LabelList index is a scalar index that can be used on `List<T>` columns to
support queries with `array_contains_all` and `array_contains_any`
support queries with `array_has_all` and `array_has_any`
using an underlying bitmap index.
#### Returns
docs/src/js/classes/Job.md
+8 -5
View File
@@ -17,7 +17,8 @@ extra round trip on every call:
- [Job.refresh](Job.md#refresh) and [Job.status](Job.md#status) fetch the whole record.
- [Job.wait](Job.md#wait) records the terminal state it establishes, but not the
rest of the record.
- Everything is null until one of those runs.
- Registry details are null until one of those runs. Query handles report
jobType "query" immediately.
## Accessors
@@ -57,8 +58,10 @@ get id(): null | string
Identifies the operation on the server that is running it.
Operations that run in this process have no server id. The value is
opaque: parsing it or storing it to resume the job later is not supported.
Registry handles expose a registry job ID; query handles expose a
connection-scoped SQL query UUID string. Local handles have no ID.
Query handles report jobType "query". Their IDs are not registry IDs or
durable resume tokens.
#### Returns
@@ -72,8 +75,8 @@ opaque: parsing it or storing it to resume the job later is not supported.
get jobType(): null | string
```
The job's type, as the server names it. Null for an in-process job, which
has no server-side record.
"query" immediately for SQL-backed handles; otherwise the cached
server-reported job type. Null for in-process handles.
#### Returns
docs/src/js/classes/Listing.md
+91
View File
@@ -0,0 +1,91 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / Listing
# Class: Listing&lt;T&gt;
A lazy async iterator over database resources with pagination state.
Requests occur during iteration; request errors terminate the iterator.
## Example
```ts
for await (const name of connection.listViews([], { pageLimit: 20 })) {
console.log(name);
}
```
## Type Parameters
• **T**
## Implements
- `AsyncIterableIterator`&lt;`T`&gt;
## Methods
### \[asyncIterator\]()
```ts
asyncIterator: AsyncIterableIterator<T>
```
#### Returns
`AsyncIterableIterator`&lt;`T`&gt;
#### Implementation of
`AsyncIterableIterator.[asyncIterator]`
***
### next()
```ts
next(): Promise<IteratorResult<T, any>>
```
Fetch the next item, requesting another page only when needed.
#### Returns
`Promise`&lt;`IteratorResult`&lt;`T`, `any`&gt;&gt;
#### Implementation of
`AsyncIterableIterator.next`
***
### numPageResults()
```ts
numPageResults(): number
```
Number of cached items available without another request.
#### Returns
`number`
***
### pageToken()
```ts
pageToken(): undefined | string
```
Token for the next request. Initially this is the supplied starting token.
Returns undefined after fetching the final page, even with cached items.
Drain the cache before saving a token to avoid skipping items on resumption.
Failed requests terminate iteration and retain their token for a new iterator.
#### Returns
`undefined` \| `string`
docs/src/js/classes/MaterializedView.md
+6 -8
View File
@@ -64,15 +64,13 @@ refresh(options?): Promise<RefreshMaterializedViewResult>
```
Recompute the view from its source.
Local refresh requires a source build with `experimental-functions`.
The refresh is incremental when the source's changes can be reconciled
into the view -- rows added, changed or removed since the last one --
and otherwise rebuilds. `full` forces a rebuild; `sourceVersion`
refreshes to that source version instead of the latest.
Concurrent refreshes of one view do not duplicate its rows. Two that
plan the same source rows conflict on commit, and the loser throws
rather than writing them a second time.
`full: true` forces a remote rebuild; otherwise the server chooses the
refresh mode. Local views always rebuild, and `sourceVersion` selects a
source version instead of the latest. Remote SQL refreshes do not support
source-version pinning. Their results leave `mode`, `rowsWritten`, and
`sourceVersion` absent because the SQL job adapter does not expose these metrics.
#### Parameters
docs/src/js/classes/MergeInsertBuilder.md
+8 -8
View File
@@ -76,23 +76,23 @@ the query optimizer chooses a suboptimal path.
***
### useLsm()
### useRTE()
```ts
useLsm(enable): MergeInsertBuilder
useRTE(enable): MergeInsertBuilder
```
Control MemWAL routing for this merge.
Control RTE routing for this merge.
By default (unset), a `mergeInsert` on a table with an LSM write spec is
By default (unset), a `mergeInsert` on a table with an RTE write spec is
routed through Lance's MemWAL shard writer, and a table without one uses the
standard path.
#### Parameters
* **enable**: `boolean`
`true` forces MemWAL routing and errors if the table has no
LSM write spec. `false` forces the standard write path even when a spec is set.
`true` forces RTE routing and errors if the table has no
RTE write spec. `false` forces the standard write path even when a spec is set.
#### Returns
@@ -108,12 +108,12 @@ validateSingleShard(validateSingleShard): MergeInsertBuilder
Controls how an LSM merge checks that its input targets a single shard.
When a table has an LSM write spec, every row in a `mergeInsert` call must
When a table has an RTE write spec, every row in a `mergeInsert` call must
route to the same shard. When `true` (the default), every row is inspected
to verify this. When `false`, only the first row is inspected and the
shard it routes to is used for the whole input — a faster path for callers
that have already pre-sharded their input. Has no effect on tables without
an LSM write spec.
an RTE write spec.
#### Parameters
docs/src/js/classes/Query.md
+23 -16
View File
@@ -424,7 +424,7 @@ input to this method would be:
#### Parameters
* **columns**: `string` \| `string`[] \| `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **columns**: `string` \| `string`[] \| `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
#### Returns
@@ -491,37 +491,39 @@ Collect the results as an Arrow
ArrowTable.
Empty results retain the query's output schema.
#### Inherited from
`StandardQueryBase.toArrow`
***
### useLsm()
### useRTE()
```ts
useLsm(enable): this
useRTE(enable): this
```
Control MemWAL read routing for this query.
Control RTE read routing for this query.
By default (unset), when the table carries a MemWAL write spec (see
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
By default (unset), when the table carries an RTE write spec (see
[Table#enableRTE](Table.md#enablerte)), reads are routed through the LSM scanner so
they also return data written via the `mergeInsert` LSM path that has not yet
been compacted into the base table (the active/frozen in-memory memtables and
the flushed generations), deduplicated by primary key; a table without a spec
the SSTables), deduplicated by primary key; a table without a spec
reads the base table.
#### Parameters
* **enable**: `boolean`
`true` forces the LSM scanner and errors if the table has no
MemWAL write spec. `false` bypasses the MemWAL and reads the base table only,
RTE write spec. `false` bypasses the RTE and reads the base table only,
even when a spec is present.
Note: the LSM scanner does not support every query shape (e.g. reranking,
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
`useLsm(false)` is set, because a base-only read would silently exclude
un-compacted MemWAL data.
hybrid search, `orderBy`). On an RTE table those shapes error unless
`useRTE(false)` is set, because a base-only read would silently exclude
un-compacted RTE data.
#### Returns
@@ -529,7 +531,7 @@ reads the base table.
#### Inherited from
`StandardQueryBase.useLsm`
`StandardQueryBase.useRTE`
***
@@ -541,29 +543,34 @@ where(predicate): this
A filter statement to be applied to this query.
The filter should be supplied as an SQL query string. For example:
The filter may be an SQL query string or an expression built with `col`,
which preserves case-sensitive column names. For example:
#### Parameters
* **predicate**: `string`
* **predicate**: `string` \| [`Expr`](Expr.md)
#### Returns
`this`
#### Example
#### Examples
```ts
x > 10
y > 0 AND y < 100
x > 5 OR y = 'test'
```
```ts
table.query().where(col("firstName").eq("Alice"));
```
Filtering performance can often be improved by creating a scalar index
on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather
than replacing the previous filter.
```
#### Inherited from
docs/src/js/classes/QueryBase.md
+3 -1
View File
@@ -193,7 +193,7 @@ input to this method would be:
#### Parameters
* **columns**: `string` \| `string`[] \| `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **columns**: `string` \| `string`[] \| `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
#### Returns
@@ -252,6 +252,8 @@ Collect the results as an Arrow
ArrowTable.
Empty results retain the query's output schema.
***
### withRowId()
docs/src/js/classes/Table.md
+167 -130
View File
@@ -69,6 +69,10 @@ abstract addColumns(newColumnTransforms): Promise<AddColumnsResult>
Add new columns with defined values.
LanceDB Embedded computed-column declaration and refresh require a source
build with the `experimental-functions` Cargo feature. Published packages
disable it; LanceDB Distributed operations remain available.
The `{ computed }` form stores the expression rather than evaluating it
now: the column is committed with no values, and rows get them from
[Table#refreshColumn](Table.md#refreshcolumn). Declaring one therefore costs the same on a
@@ -79,7 +83,7 @@ computed, so a mutated input is reflected by the next refresh. While a
declaration reads a column, that column cannot be renamed, retyped or
dropped.
On LanceDB Cloud and Enterprise the expression is planned by the
In LanceDB Distributed the expression is planned by the
server, and the refresh runs as a server job -- see
[Table#refreshColumnAsync](Table.md#refreshcolumnasync).
@@ -143,7 +147,8 @@ containing the new version number of the table after altering the columns.
abstract blobColumns(): Promise<string[]>
```
Blob v2 columns, including nested dotted paths.
Blob v2 columns addressable by row ID, including struct-nested dotted paths.
Blobs inside lists are omitted because a row ID does not identify an element.
#### Returns
@@ -227,17 +232,17 @@ version of the table.
***
### checkpointLsm()
### checkpointRTE()
```ts
abstract checkpointLsm(): Promise<void>
abstract checkpointRTE(): Promise<void>
```
Converge this table's LSM write path into its base table.
Converge this table's RTE write path into its base table.
Seals once, then triggers compaction and polls until the L0 that existed
Freezes once, then triggers compaction and polls until the SSTables that existed
at the start is gone. The target set is fixed at the start, so
generations created *during* the checkpoint are ignored — that is what
SSTables created *during* the checkpoint are ignored — that is what
lets it terminate under write load, and what makes it best-effort: it
converges the fresh tier as of some instant. Idempotent, abandonable at
any point, and safe to run on a cadence.
@@ -253,9 +258,9 @@ is merging. The caller owns the deadline.
#### Example
```ts
const before = await table.getLsmStats();
await table.checkpointLsm();
const after = await table.getLsmStats();
const before = await table.getRTEStats();
await table.checkpointRTE();
const after = await table.getRTEStats();
```
***
@@ -278,15 +283,15 @@ Any attempt to use the table after it is closed will result in an error.
***
### closeLsmWriters()
### closeRTEWriters()
```ts
abstract closeLsmWriters(): Promise<void>
abstract closeRTEWriters(): Promise<void>
```
Drain and close any cached MemWAL shard writers held for this table.
Drain and close any cached RTE shard writers held for this table.
When an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) is installed, `mergeInsert` opens MemWAL
When an [RTEWriteSpec](../interfaces/RTEWriteSpec.md) is installed, `mergeInsert` opens RTE
shard writers and caches them for reuse across calls. This closes them,
flushing pending data; writers reopen lazily on the next `mergeInsert`.
It is a no-op when no writers are cached.
@@ -297,17 +302,17 @@ It is a no-op when no writers are cached.
***
### compactLsm()
### compactRTE()
```ts
abstract compactLsm(): Promise<void>
abstract compactRTE(): Promise<void>
```
Trigger a background L0 → base compaction pass per bucket.
Trigger a background SSTable compaction pass per table shard.
Returns once the passes are *dispatched*, not once they finish — watch
[Table#getLsmStats](Table.md#getlsmstats) for progress, or use
[Table#checkpointLsm](Table.md#checkpointlsm) to wait for convergence.
[Table#getRTEStats](Table.md#getrtestats) for progress, or use
[Table#checkpointRTE](Table.md#checkpointrte) to wait for convergence.
#### Returns
@@ -380,6 +385,13 @@ await table.createIndex("vector", {
});
```
```ts
// Or size the partitions by row count instead of choosing how many.
await table.createIndex("vector", {
config: lancedb.Index.ivfPq({ targetPartitionSize: 4096 }),
});
```
```ts
// Or create a Scalar index
await table.createIndex("my_float_col");
@@ -448,6 +460,23 @@ containing the new version number of the table
***
### disableRTE()
```ts
abstract disableRTE(): Promise<void>
```
Remove the [RTEWriteSpec](../interfaces/RTEWriteSpec.md) from this table, reverting to the standard
`mergeInsert` write path.
Errors if no spec is currently set.
#### Returns
`Promise`&lt;`void`&gt;
***
### display()
```ts
@@ -513,10 +542,61 @@ Drop an index from the table.
***
### enableRTE()
```ts
abstract enableRTE(spec): Promise<void>
```
Install an [RTEWriteSpec](../interfaces/RTEWriteSpec.md) on this table, selecting Lance's MemWAL
LSM-style write path for future `mergeInsert` calls.
`RTEWriteSpec` chooses one of three sharding strategies via `specType`:
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
key (`column` and `numBuckets` required).
- `"identity"` — shard by the raw value of a scalar `column`.
- `"unsharded"` — route every write to a single shard.
All variants require the table to have an unenforced primary key
([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
requires it to be the single column being bucketed.
Omitting `maintainedIndexes` maintains every index the table has,
including ones created later, and skips a kind the RTE cannot maintain.
Naming them pins an exact set, and a still-building index is rejected
rather than quietly omitted.
#### Parameters
* **spec**: [`RTEWriteSpec`](../interfaces/RTEWriteSpec.md)
The sharding spec to install.
#### Returns
`Promise`&lt;`void`&gt;
#### Example
```ts
await table.setUnenforcedPrimaryKey("id");
await table.enableRTE({
specType: "bucket",
column: "id",
numBuckets: 16,
maintainedIndexes: ["id_idx"],
});
```
***
### fetchBlobFiles()
```ts
abstract fetchBlobFiles(column, rowIds): Promise<(null | BlobFile)[]>
abstract fetchBlobFiles(
column,
rowIds,
options?): Promise<(null | BlobFile)[]>
```
Opens lazy blob handles for `column` at the given row IDs using the
@@ -524,6 +604,8 @@ table's current checkout.
Preserves input order, duplicates, and nulls. Use this for large payloads.
See [Table.fetchBlobs](Table.md#fetchblobs) for row-ID validity across versions.
One missing or deleted row ID rejects the entire batch, including on remote
tables; no partial handles are returned.
#### Parameters
@@ -531,6 +613,8 @@ See [Table.fetchBlobs](Table.md#fetchblobs) for row-ID validity across versions.
* **rowIds**: readonly (`number` \| `bigint`)[]
* **options?**: [`BlobReadOptions`](../type-aliases/BlobReadOptions.md)
#### Returns
`Promise`&lt;(`null` \| [`BlobFile`](BlobFile.md))[]&gt;
@@ -540,7 +624,10 @@ See [Table.fetchBlobs](Table.md#fetchblobs) for row-ID validity across versions.
### fetchBlobs()
```ts
abstract fetchBlobs(column, rowIds): Promise<(null | Buffer)[]>
abstract fetchBlobs(
column,
rowIds,
options?): Promise<(null | Buffer)[]>
```
Bytes for `column` at row IDs from [Query.withRowId](Query.md#withrowid).
@@ -548,6 +635,14 @@ Bytes for `column` at row IDs from [Query.withRowId](Query.md#withrowid).
Reads the table's current checkout. IDs from another version can fail after
compaction unless stable row ids are enabled. Results keep input order and
duplicates. Null blobs are `null`. Empty blobs are empty buffers.
One missing or deleted row ID rejects the entire batch, including on remote
tables; no partial results are returned.
Blobs inside lists cannot be fetched by row ID.
Remote servers limit each request to 1024 row IDs and 64 MiB of blob bytes.
The client splits requests automatically and reads an individual larger
blob through the Range route. This method still materializes all bytes in
memory; use [Table.fetchBlobFiles](Table.md#fetchblobfiles) for large values.
#### Parameters
@@ -555,21 +650,23 @@ duplicates. Null blobs are `null`. Empty blobs are empty buffers.
* **rowIds**: readonly (`number` \| `bigint`)[]
* **options?**: [`BlobReadOptions`](../type-aliases/BlobReadOptions.md)
#### Returns
`Promise`&lt;(`null` \| `Buffer`)[]&gt;
***
### flushLsm()
### flushRTE()
```ts
abstract flushLsm(): Promise<void>
abstract flushRTE(): Promise<void>
```
Seal every bucket's active memtable into a new L0 generation.
Freeze every table shard's active memtable into a new SSTable.
Returns once the seal is committed. Sealing an empty memtable is a no-op,
Returns once the freeze is committed. Freezing an empty memtable is a no-op,
so this is safe to call repeatedly.
#### Returns
@@ -589,7 +686,7 @@ The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped with
the input that failed and the error. This lists those records, newest
job first, plus a summary for any fragment whose per-row detail was
capped. LanceDB Cloud and Enterprise only; reading errors needs read
capped. LanceDB Distributed only; reading errors needs read
access to the table, since a message carries the value that failed.
#### Parameters
@@ -613,49 +710,48 @@ const { records, truncated } = await table.functionErrors({ column: "embedding"
***
### getLsmStats()
### getRTESpec()
```ts
abstract getLsmStats(includeGenerationRows?): Promise<undefined | LsmStats>
abstract getRTESpec(): Promise<undefined | RTEWriteSpec>
```
Read live per-bucket LSM state.
Read the [RTEWriteSpec](../interfaces/RTEWriteSpec.md) currently installed on this table.
Answers "how far behind is my fresh tier", "which bucket is hot", and
Resolves to `undefined` when the RTE write path is not enabled (no
spec has been set, or it was removed with [Table#disableRTE](Table.md#disablerte)).
The spec is the one installed, including its maintained-index selection:
an absent `maintainedIndexes` for every index the table has, an empty
array for none.
#### Returns
`Promise`&lt;`undefined` \| [`RTEWriteSpec`](../interfaces/RTEWriteSpec.md)&gt;
***
### getRTEStats()
```ts
abstract getRTEStats(includeSstableRows?): Promise<undefined | RTEStats>
```
Read live per-table-shard RTE state.
Answers "how far behind is my fresh tier", "which table shard is hot", and
"why is my fresh-tier vector search brute-force". Mutates no table state.
Resolves to `undefined` only when the LSM write path is not enabled.
Resolves to `undefined` only when the RTE write path is not enabled.
#### Parameters
* **includeGenerationRows?**: `boolean`
Also count rows per L0 generation.
* **includeSstableRows?**: `boolean`
Also count rows per SSTable.
Off by default because each count opens an uncached Lance dataset.
#### Returns
`Promise`&lt;`undefined` \| [`LsmStats`](../interfaces/LsmStats.md)&gt;
***
### getLsmWriteSpec()
```ts
abstract getLsmWriteSpec(): Promise<undefined | LsmWriteSpec>
```
Read the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) currently installed on this table.
Resolves to `undefined` when the MemWAL LSM write path is not enabled (no
spec has been set, or it was removed with [Table#unsetLsmWriteSpec](Table.md#unsetlsmwritespec)).
The returned spec mirrors what was passed to
[Table#setLsmWriteSpec](Table.md#setlsmwritespec), except that `maintainedIndexes` always
reports the concrete list resolved when the spec was set — `undefined`
never round-trips.
#### Returns
`Promise`&lt;`undefined` \| [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)&gt;
`Promise`&lt;`undefined` \| [`RTEStats`](../interfaces/RTEStats.md)&gt;
***
@@ -843,7 +939,7 @@ Prewarm one or more columns of data in the table.
Prewarming is generally useful for columns used in filters or projections.
Large columns (e.g. high-dimensional vectors or binary data) may not be
practical to prewarm.
This feature is currently only supported on remote tables.
This feature is currently only supported in LanceDB Distributed.
#### Returns
@@ -950,11 +1046,12 @@ abstract refreshColumn(column): Promise<RefreshColumnResult>
```
Fill the rows of a computed column that hold no value yet.
Local refresh requires a source build with `experimental-functions`.
Rows appended since the last refresh are filled by the next one, and
rows whose inputs changed since they were computed are recomputed;
everything else is left as it is. Local tables only: a remote refresh
runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
everything else is left as it is. LanceDB Embedded only: in LanceDB
Distributed a refresh runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
#### Parameters
@@ -978,12 +1075,15 @@ abstract refreshColumnAsync(column): Promise<Job>
Like [Table#refreshColumn](Table.md#refreshcolumn), but returns a handle to the refresh
job instead of blocking until it completes.
LanceDB Embedded refresh requires a source build with
`experimental-functions`. LanceDB Distributed jobs do not require this
feature.
The job may already be complete when returned; callers must not assume
the column is filled until [Job.wait](Job.md#wait) resolves. Invalid input --
an unknown column, or one that is not computed -- rejects here rather
than failing the job. On local tables the job runs in-process; on
LanceDB Cloud and Enterprise it is the server's backfill job.
than failing the job. In LanceDB Embedded the job runs in-process; in
LanceDB Distributed it is the server's backfill job.
#### Parameters
@@ -1073,54 +1173,6 @@ of the given query
***
### setLsmWriteSpec()
```ts
abstract setLsmWriteSpec(spec): Promise<void>
```
Install an [LsmWriteSpec](../interfaces/LsmWriteSpec.md) on this table, selecting Lance's MemWAL
LSM-style write path for future `mergeInsert` calls.
`LsmWriteSpec` chooses one of three sharding strategies via `specType`:
- `"bucket"` — hash-bucket writes by the single-column unenforced primary
key (`column` and `numBuckets` required).
- `"identity"` — shard by the raw value of a scalar `column`.
- `"unsharded"` — route every write to a single shard.
All variants require the table to have an unenforced primary key
([Table#setUnenforcedPrimaryKey](Table.md#setunenforcedprimarykey)); bucket sharding additionally
requires it to be the single column being bucketed.
Omitting `maintainedIndexes` maintains every index on the table, resolved
here, failing if one cannot be maintained — name them to install anyway.
Naming them pins an exact set, and a still-building index is rejected
rather than quietly omitted.
#### Parameters
* **spec**: [`LsmWriteSpec`](../interfaces/LsmWriteSpec.md)
The sharding spec to install.
#### Returns
`Promise`&lt;`void`&gt;
#### Example
```ts
await table.setUnenforcedPrimaryKey("id");
await table.setLsmWriteSpec({
specType: "bucket",
column: "id",
numBuckets: 16,
maintainedIndexes: ["id_idx"],
});
```
***
### setUnenforcedPrimaryKey()
```ts
@@ -1260,8 +1312,12 @@ Tokenize a full-text search query using the tokenizer configured on an FTS index
Specify exactly one of `column` or `indexName`.
Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
the client process from index metadata. For remote tables, this means the
same tokenizer model files must also exist locally.
the client process from index metadata. In LanceDB Distributed, this means the
same tokenizer model files must also exist locally (under
`LANCE_LANGUAGE_MODEL_HOME`) and must match the files the server uses. A
mismatch cannot be detected: the returned tokens may differ from what the
server's index produces. If the model files are missing, an error naming
the tokenizer is raised.
#### Parameters
@@ -1275,23 +1331,6 @@ same tokenizer model files must also exist locally.
***
### unsetLsmWriteSpec()
```ts
abstract unsetLsmWriteSpec(): Promise<void>
```
Remove the [LsmWriteSpec](../interfaces/LsmWriteSpec.md) from this table, reverting to the standard
`mergeInsert` write path.
Errors if no spec is currently set.
#### Returns
`Promise`&lt;`void`&gt;
***
### update()
#### update(opts)
@@ -1367,7 +1406,7 @@ repeatedly calling this method.
##### Parameters
* **updates**: `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **updates**: `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
the
columns to update
@@ -1403,8 +1442,6 @@ used when appropriate:
- `lancedb:description`: for a human-readable description of a field.
- `lancedb:tag:<name>`: for a user-defined key-value tag, where the suffix
names the tag category; e.g. `lancedb:tag:model: "clip"`.
- `lancedb:logical-column`: for a column grouping; e.g. `feature_v1` and
`feature_v2` might be in the same logical column.
- `lancedb:status`: for status options (`production`, `candidate`,
`deprecated`, `archived`) to designate the current life cycle state of
this column.
docs/src/js/classes/TakeQuery.md
+8 -6
View File
@@ -200,7 +200,7 @@ input to this method would be:
#### Parameters
* **columns**: `string` \| `string`[] \| `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **columns**: `string` \| `string`[] \| `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
#### Returns
@@ -267,23 +267,25 @@ Collect the results as an Arrow
ArrowTable.
Empty results retain the query's output schema.
#### Inherited from
[`QueryBase`](QueryBase.md).[`toArrow`](QueryBase.md#toarrow)
***
### useLsm()
### useRTE()
```ts
useLsm(enable): this
useRTE(enable): this
```
Control MemWAL read routing for this take query.
Control RTE read routing for this take query.
`false` bypasses the MemWAL and reads the base table only — the escape hatch,
`false` bypasses the RTE and reads the base table only — the escape hatch,
since take-by-row-id/offset is not supported on the LSM scanner and, on a
MemWAL table, auto-routes to it and errors otherwise.
RTE table, auto-routes to it and errors otherwise.
#### Parameters
docs/src/js/classes/VectorQuery.md
+62 -22
View File
@@ -447,6 +447,8 @@ nprobes(nprobes): VectorQuery
Set the number of partitions to search (probe)
The number of probes must be greater than 0.
This argument is only used when the vector column has an IVF PQ index.
If there is no index then this value is ignored.
@@ -458,17 +460,16 @@ exhaustiely searched to find matches. This parameter controls how many
partitions should be searched.
Increasing this value will increase the recall of your query but will
also increase the latency of your query. The default value is 20. This
default is good for many cases but the best value to use will depend on
your data and the recall that you need to achieve.
also increase the latency of your query. If this method is not called,
Lance's adaptive probe defaults are used.
For best results we recommend tuning this parameter with a benchmark against
your actual data to find the smallest possible value that will still give
you the desired recall.
For more fine grained control over behavior when you have a very narrow filter
you can use `minimumNprobes` and `maximumNprobes`. This method sets both
the minimum and maximum to the same value.
The value is retained as `nprobes` through client and server request
construction. Lance sets both probe bounds to this value. Explicit minimum
or maximum settings can override their respective bounds.
#### Parameters
@@ -645,6 +646,38 @@ rerank(reranker): VectorQuery
***
### searchEffort()
```ts
searchEffort(searchEffort): VectorQuery
```
Set the initial IVF search effort to a finite value in [0, 1].
The default, null, uses Auto (internally 0.5). Zero starts at the caller
minimum; one searches all partitions within the caller maximum. Later
probing may continue when too few candidates are found.
Any explicit effort, including 0.5, overrides nprobes in either call order.
Explicit minimum and maximum bounds still apply. This is not a recall
guarantee and does not change refinement or HNSW ef. The RTE LSM
scanner does not yet support this option.
#### Parameters
* **searchEffort**: `null` \| `number`
#### Returns
[`VectorQuery`](VectorQuery.md)
#### Example
```ts
table.vectorSearch([0.1, 0.2]).searchEffort(0.75).limit(10);
```
***
### select()
```ts
@@ -673,7 +706,7 @@ input to this method would be:
#### Parameters
* **columns**: `string` \| `string`[] \| `Record`&lt;`string`, `string`&gt; \| `Map`&lt;`string`, `string`&gt;
* **columns**: `string` \| `string`[] \| `Map`&lt;`string`, `string`&gt; \| `Record`&lt;`string`, `string`&gt;
#### Returns
@@ -740,37 +773,39 @@ Collect the results as an Arrow
ArrowTable.
Empty results retain the query's output schema.
#### Inherited from
`StandardQueryBase.toArrow`
***
### useLsm()
### useRTE()
```ts
useLsm(enable): this
useRTE(enable): this
```
Control MemWAL read routing for this query.
Control RTE read routing for this query.
By default (unset), when the table carries a MemWAL write spec (see
[Table#setLsmWriteSpec](Table.md#setlsmwritespec)), reads are routed through the LSM scanner so
By default (unset), when the table carries an RTE write spec (see
[Table#enableRTE](Table.md#enablerte)), reads are routed through the LSM scanner so
they also return data written via the `mergeInsert` LSM path that has not yet
been compacted into the base table (the active/frozen in-memory memtables and
the flushed generations), deduplicated by primary key; a table without a spec
the SSTables), deduplicated by primary key; a table without a spec
reads the base table.
#### Parameters
* **enable**: `boolean`
`true` forces the LSM scanner and errors if the table has no
MemWAL write spec. `false` bypasses the MemWAL and reads the base table only,
RTE write spec. `false` bypasses the RTE and reads the base table only,
even when a spec is present.
Note: the LSM scanner does not support every query shape (e.g. reranking,
hybrid search, `orderBy`). On a MemWAL table those shapes error unless
`useLsm(false)` is set, because a base-only read would silently exclude
un-compacted MemWAL data.
hybrid search, `orderBy`). On an RTE table those shapes error unless
`useRTE(false)` is set, because a base-only read would silently exclude
un-compacted RTE data.
#### Returns
@@ -778,7 +813,7 @@ reads the base table.
#### Inherited from
`StandardQueryBase.useLsm`
`StandardQueryBase.useRTE`
***
@@ -790,29 +825,34 @@ where(predicate): this
A filter statement to be applied to this query.
The filter should be supplied as an SQL query string. For example:
The filter may be an SQL query string or an expression built with `col`,
which preserves case-sensitive column names. For example:
#### Parameters
* **predicate**: `string`
* **predicate**: `string` \| [`Expr`](Expr.md)
#### Returns
`this`
#### Example
#### Examples
```ts
x > 10
y > 0 AND y < 100
x > 5 OR y = 'test'
```
```ts
table.query().where(col("firstName").eq("Alice"));
```
Filtering performance can often be improved by creating a scalar index
on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather
than replacing the previous filter.
```
#### Inherited from
docs/src/js/functions/blob.md
+11
View File
@@ -11,9 +11,20 @@ function blob(name, options): Field
```
Declares a `lance.blob.v2` column.
Uses nullable `data`, `uri`, `position`, and `size` fields for inline bytes,
whole external files, and external byte ranges. Set both `position` and
positive `size` for a range, or omit both for a whole file.
Existing two-field columns remain writable. Appends and merge inserts fill
missing range fields with nulls for four-field targets and drop all-null
range fields for two-field targets. Non-null ranges require a four-field
target. Overwrite with Arrow input replaces the table's schema; row objects
are converted using the current table schema.
Query results are descriptors, not payload bytes. Use [Table.fetchBlobs](../classes/Table.md#fetchblobs)
or [Table.fetchBlobFiles](../classes/Table.md#fetchblobfiles) to read bytes.
Blobs inside lists can be written and queried as descriptors, but cannot
currently be fetched because a row ID does not identify a list element.
## Parameters
docs/src/js/functions/col.md
+21
View File
@@ -0,0 +1,21 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / col
# Function: col()
```ts
function col(name): Expr
```
Reference a column by its exact, case-sensitive name.
## Parameters
* **name**: `string`
## Returns
[`Expr`](../classes/Expr.md)
docs/src/js/functions/connect.md
+5 -5
View File
@@ -20,15 +20,15 @@ Connect to a LanceDB instance at the given URI.
Accepted formats:
- `/path/to/database` - local database
- `/path/to/database` - database on the local file system
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (LanceDB cloud)
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
### Parameters
* **uri**: `string`
The uri of the database. If the database uri starts
with `db://` then it connects to a remote database.
with `db://` then it connects to LanceDB Enterprise or LanceDB Cloud (legacy).
* **options?**: `Partial`&lt;[`ConnectionOptions`](../interfaces/ConnectionOptions.md)&gt;
The options to use when connecting to the database
@@ -80,9 +80,9 @@ Connect to a LanceDB instance at the given URI.
Accepted formats:
- `/path/to/database` - local database
- `/path/to/database` - database on the local file system
- `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (LanceDB cloud)
- `db://host:port` - LanceDB Enterprise or LanceDB Cloud (legacy) database
### Parameters
docs/src/js/functions/connectCatalog.md
+3 -1
View File
@@ -28,5 +28,7 @@ headers; opened database connections inherit authentication and client options.
```ts
const catalog = await connectCatalog("https://my-server.example", { apiKey: "secret" });
const db = await catalog.createDatabase("analytics", { existOk: true });
const page = await catalog.listDatabases({ limit: 20 });
for await (const name of catalog.listDatabases({ pageLimit: 20 })) {
console.log(name);
}
```
docs/src/js/functions/connectNamespace.md
+1 -1
View File
@@ -18,7 +18,7 @@ function connectNamespace(
Connect to a LanceDB database through a namespace.
Unlike [connect](connect.md), which routes by URI scheme (local path vs.
`db://` cloud), `connectNamespace` always returns a namespace-backed
`db://`), `connectNamespace` always returns a namespace-backed
connection. The `implName` selects the namespace implementation:
- `"dir"` — directory namespace, configured with [DirNamespaceConfig](../interfaces/DirNamespaceConfig.md).
docs/src/js/functions/func.md
+23
View File
@@ -0,0 +1,23 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / func
# Function: func()
```ts
function func(name, ...args): Expr
```
Call a registered SQL function, such as `func("lower", col("firstName"))`.
## Parameters
* **name**: `string`
* ...**args**: [`ExprLike`](../type-aliases/ExprLike.md)[]
## Returns
[`Expr`](../classes/Expr.md)
docs/src/js/functions/lit.md
+21
View File
@@ -0,0 +1,21 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / lit
# Function: lit()
```ts
function lit(value): Expr
```
Create a string or numeric literal. Unsafe integers are represented as Float64.
## Parameters
* **value**: `string` \| `number`
## Returns
[`Expr`](../classes/Expr.md)
docs/src/js/functions/litBool.md
+21
View File
@@ -0,0 +1,21 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / litBool
# Function: litBool()
```ts
function litBool(value): Expr
```
Create a boolean literal.
## Parameters
* **value**: `boolean`
## Returns
[`Expr`](../classes/Expr.md)
docs/src/js/functions/litInt.md
+21
View File
@@ -0,0 +1,21 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / litInt
# Function: litInt()
```ts
function litInt(value): Expr
```
Create an Int64 literal from a JavaScript safe integer. Other numbers throw RangeError.
## Parameters
* **value**: `number`
## Returns
[`Expr`](../classes/Expr.md)
docs/src/js/globals.md
+22 -9
View File
@@ -27,9 +27,12 @@
- [Branches](classes/Branches.md)
- [Catalog](classes/Catalog.md)
- [Connection](classes/Connection.md)
- [DatabaseNames](classes/DatabaseNames.md)
- [Expr](classes/Expr.md)
- [HeaderProvider](classes/HeaderProvider.md)
- [Index](classes/Index.md)
- [Job](classes/Job.md)
- [Listing](classes/Listing.md)
- [MakeArrowTableOptions](classes/MakeArrowTableOptions.md)
- [MatchQuery](classes/MatchQuery.md)
- [MaterializedView](classes/MaterializedView.md)
@@ -64,7 +67,6 @@
- [BranchDiff](interfaces/BranchDiff.md)
- [BranchIndexSummary](interfaces/BranchIndexSummary.md)
- [BranchRowCountSummary](interfaces/BranchRowCountSummary.md)
- [BucketStats](interfaces/BucketStats.md)
- [CatalogOptions](interfaces/CatalogOptions.md)
- [CherryPickError](interfaces/CherryPickError.md)
- [CherryPickPreview](interfaces/CherryPickPreview.md)
@@ -96,7 +98,6 @@
- [FunctionErrorRecord](interfaces/FunctionErrorRecord.md)
- [FunctionErrors](interfaces/FunctionErrors.md)
- [FunctionErrorsOptions](interfaces/FunctionErrorsOptions.md)
- [GenerationStats](interfaces/GenerationStats.md)
- [HnswPqOptions](interfaces/HnswPqOptions.md)
- [HnswSqOptions](interfaces/HnswSqOptions.md)
- [IndexConfig](interfaces/IndexConfig.md)
@@ -108,13 +109,7 @@
- [JobEventsOptions](interfaces/JobEventsOptions.md)
- [JobFailureInfo](interfaces/JobFailureInfo.md)
- [JobInfo](interfaces/JobInfo.md)
- [ListDatabasesResponse](interfaces/ListDatabasesResponse.md)
- [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
- [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
- [ListTablesOptions](interfaces/ListTablesOptions.md)
- [ListTablesResponse](interfaces/ListTablesResponse.md)
- [LsmStats](interfaces/LsmStats.md)
- [LsmWriteSpec](interfaces/LsmWriteSpec.md)
- [ListingOptions](interfaces/ListingOptions.md)
- [MaterializedViewDefinition](interfaces/MaterializedViewDefinition.md)
- [MemtableStats](interfaces/MemtableStats.md)
- [MergeResult](interfaces/MergeResult.md)
@@ -124,6 +119,8 @@
- [OptimizeOptions](interfaces/OptimizeOptions.md)
- [OptimizeStats](interfaces/OptimizeStats.md)
- [QueryExecutionOptions](interfaces/QueryExecutionOptions.md)
- [RTEStats](interfaces/RTEStats.md)
- [RTEWriteSpec](interfaces/RTEWriteSpec.md)
- [RefreshColumnResult](interfaces/RefreshColumnResult.md)
- [RefreshMaterializedViewResult](interfaces/RefreshMaterializedViewResult.md)
- [RemovalStats](interfaces/RemovalStats.md)
@@ -138,7 +135,9 @@
- [SplitHashOptions](interfaces/SplitHashOptions.md)
- [SplitRandomOptions](interfaces/SplitRandomOptions.md)
- [SplitSequentialOptions](interfaces/SplitSequentialOptions.md)
- [SsTableStats](interfaces/SsTableStats.md)
- [TableNamesOptions](interfaces/TableNamesOptions.md)
- [TableShardStats](interfaces/TableShardStats.md)
- [TableStatistics](interfaces/TableStatistics.md)
- [TimeoutConfig](interfaces/TimeoutConfig.md)
- [TlsConfig](interfaces/TlsConfig.md)
@@ -149,6 +148,7 @@
- [UpdateOptions](interfaces/UpdateOptions.md)
- [UpdateResult](interfaces/UpdateResult.md)
- [Version](interfaces/Version.md)
- [ViewDescription](interfaces/ViewDescription.md)
- [WriteExecutionOptions](interfaces/WriteExecutionOptions.md)
- [WriteProgress](interfaces/WriteProgress.md)
@@ -156,9 +156,17 @@
- [AnalyzePlanDistributedMetrics](type-aliases/AnalyzePlanDistributedMetrics.md)
- [BaseTokenizer](type-aliases/BaseTokenizer.md)
- [BlobData](type-aliases/BlobData.md)
- [BlobInput](type-aliases/BlobInput.md)
- [BlobMode](type-aliases/BlobMode.md)
- [BlobOptions](type-aliases/BlobOptions.md)
- [BlobRange](type-aliases/BlobRange.md)
- [BlobReadOptions](type-aliases/BlobReadOptions.md)
- [BlobUri](type-aliases/BlobUri.md)
- [Data](type-aliases/Data.md)
- [DataLike](type-aliases/DataLike.md)
- [DataTypeLike](type-aliases/DataTypeLike.md)
- [ExprLike](type-aliases/ExprLike.md)
- [FieldLike](type-aliases/FieldLike.md)
- [IntoSql](type-aliases/IntoSql.md)
- [IntoVector](type-aliases/IntoVector.md)
@@ -173,11 +181,16 @@
- [RecordBatchIterator](functions/RecordBatchIterator.md)
- [blob](functions/blob.md)
- [col](functions/col.md)
- [connect](functions/connect.md)
- [connectCatalog](functions/connectCatalog.md)
- [connectNamespace](functions/connectNamespace.md)
- [func](functions/func.md)
- [instrumentLanceDbMetrics](functions/instrumentLanceDbMetrics.md)
- [isBlobField](functions/isBlobField.md)
- [lit](functions/lit.md)
- [litBool](functions/litBool.md)
- [litInt](functions/litInt.md)
- [makeArrowTable](functions/makeArrowTable.md)
- [makeJsonField](functions/makeJsonField.md)
- [packBits](functions/packBits.md)
docs/src/js/interfaces/AddDataOptions.md
+16
View File
@@ -10,6 +10,22 @@ Options for adding data to a table.
## Properties
### allowExternalBlobOutsideBases?
```ts
optional allowExternalBlobOutsideBases: boolean;
```
Whether blob URIs outside registered external bases may be written.
Defaults to `false`. This option is supported only in LanceDB Embedded;
LanceDB Distributed returns an error when it is enabled. An enabled write
stores the absolute URI reference without registering a base or copying
the external object into the database. The object must remain accessible
when the blob is read later.
***
### mode
```ts
docs/src/js/interfaces/CherryPickResult.md
+3 -3
View File
@@ -38,9 +38,9 @@ preview: CherryPickPreview;
```ts
status:
| "failed"
| "unknown"
| "ready"
| "failed"
| "notImplemented"
| "cherryPicked";
| "cherryPicked"
| "unknown";
```
docs/src/js/interfaces/ClientConfig.md
+13 -1
View File
@@ -8,6 +8,18 @@
## Properties
### blobRequestConcurrency?
```ts
optional blobRequestConcurrency: number;
```
Maximum number of concurrent blob HTTP requests across all tables and
blob handles on this connection. Defaults to 8. Can also be set with
`LANCE_CLIENT_BLOB_REQUEST_CONCURRENCY`. Must be greater than zero.
***
### extraHeaders?
```ts
@@ -68,7 +80,7 @@ optional userId: string;
User identifier for tracking purposes.
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Cloud/Enterprise.
This is sent as the `x-lancedb-user-id` header in requests to LanceDB Enterprise.
It can be set directly, or via the `LANCEDB_USER_ID` environment variable.
Alternatively, set `LANCEDB_USER_ID_ENV_KEY` to specify another environment
variable that contains the user ID value.
docs/src/js/interfaces/ConnectionOptions.md
+10 -10
View File
@@ -14,7 +14,7 @@
optional apiKey: string;
```
(For LanceDB cloud only): the API key to use with LanceDB Cloud.
(For LanceDB Enterprise or LanceDB Cloud only): the API key to use.
Can also be set via the environment variable `LANCEDB_API_KEY`.
@@ -26,7 +26,7 @@ Can also be set via the environment variable `LANCEDB_API_KEY`.
optional clientConfig: ClientConfig;
```
(For LanceDB cloud only): configuration for the remote HTTP client.
(For LanceDB Enterprise or LanceDB Cloud only): configuration for the HTTP client.
***
@@ -36,8 +36,8 @@ optional clientConfig: ClientConfig;
optional hostOverride: string;
```
(For LanceDB cloud only): the host to use for LanceDB cloud. Used
for testing purposes.
(For LanceDB Enterprise only): the host of the LanceDB Enterprise
deployment to connect to.
***
@@ -47,7 +47,7 @@ for testing purposes.
optional manifestEnabled: boolean;
```
(For LanceDB OSS only): use directory namespace manifests as the source
(For LanceDB Embedded only): use directory namespace manifests as the source
of truth for table metadata. Existing directory-listed root tables are
migrated into the manifest on access.
@@ -59,7 +59,7 @@ migrated into the manifest on access.
optional namespaceClientProperties: Record<string, string>;
```
(For LanceDB OSS only): extra properties for the backing namespace
(For LanceDB Embedded only): extra properties for the backing namespace
client used by manifest-enabled native connections.
***
@@ -70,7 +70,7 @@ client used by manifest-enabled native connections.
optional oauthConfig: NativeOAuthConfig;
```
(For LanceDB cloud only): OAuth configuration for IdP-based
(For LanceDB Enterprise only): OAuth configuration for IdP-based
authentication (e.g., Azure Entra ID). When set, token acquisition
and refresh are handled entirely in Rust. TypeScript users should pass
the public `OAuthConfig` type exported from `@lancedb/lancedb`.
@@ -105,7 +105,7 @@ storage, raising per-read latency and cost.
optional region: string;
```
(For LanceDB cloud only): the region to use for LanceDB cloud.
(For LanceDB Cloud only): the region to use for LanceDB Cloud (legacy).
Defaults to 'us-east-1'.
***
@@ -116,7 +116,7 @@ Defaults to 'us-east-1'.
optional session: Session;
```
(For LanceDB OSS only): the session to use for this connection. Holds
(For LanceDB Embedded only): the session to use for this connection. Holds
shared caches and other session-specific state.
***
@@ -127,6 +127,6 @@ shared caches and other session-specific state.
optional storageOptions: Record<string, string>;
```
(For LanceDB OSS only): configuration for object storage.
(For LanceDB Embedded only): configuration for object storage.
The available options are described at https://docs.lancedb.com/storage/
docs/src/js/interfaces/CreateTableOptions.md
+4 -1
View File
@@ -65,7 +65,7 @@ then no error will be raised.
### mode
```ts
mode: "overwrite" | "create";
mode: "overwrite" | "create" | "exist_ok";
```
The mode to use when creating the table.
@@ -76,6 +76,9 @@ happen. Any provided data will be ignored.
If this is set to "overwrite" then any existing table will be replaced.
If this is set to "exist_ok" then an existing table is opened and any
provided data is ignored.
***
### schema?
docs/src/js/interfaces/DropNamespaceOptions.md
+1 -1
View File
@@ -21,7 +21,7 @@ Refuse to drop if non-empty (restrict) or drop recursively (cascade).
### mode?
```ts
optional mode: "fail" | "skip";
optional mode: "skip" | "fail";
```
Whether to skip if the namespace doesn't exist, or fail.
docs/src/js/interfaces/FtsOptions.md
+2
View File
@@ -37,6 +37,8 @@ The following tokenizers are available:
"raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
"code" - Code tokenizer. This tokenizer keeps snake_case and camelCase identifiers as single tokens.
"icu" - ICU dictionary-based word segmentation.
"icu/split" - ICU segmentation with simple-style delimiter splitting.
docs/src/js/interfaces/GenerationStats.md
-40
View File
@@ -1,40 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / GenerationStats
# Interface: GenerationStats
One flushed L0 generation.
## Properties
### bytes
```ts
bytes: number;
```
On-disk size of the generation.
***
### generation
```ts
generation: number;
```
The generation number. Increases as memtables are sealed into L0.
***
### rows?
```ts
optional rows: number;
```
Present only when `includeGenerationRows` was requested. Off by default
because each count opens an uncached Lance dataset.
docs/src/js/interfaces/HnswPqOptions.md
+22
View File
@@ -100,6 +100,8 @@ optional numPartitions: number;
The number of IVF partitions to create.
When set, this takes precedence over `targetPartitionSize`.
For HNSW, we recommend a small number of partitions. Setting this to 1 works
well for most tables. For very large tables, training just one HNSW graph
will require too much memory. Each partition becomes its own HNSW graph, so
@@ -147,3 +149,23 @@ number of vectors used to train the index is `sample_rate * num_partitions`.
Increasing this value might improve the quality of the index but in most cases the
default should be sufficient.
***
### targetPartitionSize?
```ts
optional targetPartitionSize: number;
```
The target number of rows in each IVF partition.
This is an alternative to `numPartitions`. When it is set and
`numPartitions` is not, the number of partitions is derived by dividing
the number of rows by this value. When both are set, `numPartitions` is
used and this value is ignored. When neither is set, the target is
1,048,576 rows.
Larger partitions mean fewer of them, which makes the first part of the
search (picking the right partitions) faster and the second part
(searching within a partition) slower.
docs/src/js/interfaces/HnswSqOptions.md
+22
View File
@@ -100,6 +100,8 @@ optional numPartitions: number;
The number of IVF partitions to create.
When set, this takes precedence over `targetPartitionSize`.
For HNSW, we recommend a small number of partitions. Setting this to 1 works
well for most tables. For very large tables, training just one HNSW graph
will require too much memory. Each partition becomes its own HNSW graph, so
@@ -126,3 +128,23 @@ number of vectors used to train the index is `sample_rate * num_partitions`.
Increasing this value might improve the quality of the index but in most cases the
default should be sufficient.
***
### targetPartitionSize?
```ts
optional targetPartitionSize: number;
```
The target number of rows in each IVF partition.
This is an alternative to `numPartitions`. When it is set and
`numPartitions` is not, the number of partitions is derived by dividing
the number of rows by this value. When both are set, `numPartitions` is
used and this value is ignored. When neither is set, the target is
1,048,576 rows.
Larger partitions mean fewer of them, which makes the first part of the
search (picking the right partitions) faster and the second part
(searching within a partition) slower.
docs/src/js/interfaces/IndexConfig.md
+10 -10
View File
@@ -31,7 +31,7 @@ optional createdAt: Date;
When the index was created.
`undefined` for remote tables or indices created before timestamps were tracked.
`undefined` in LanceDB Distributed or indices created before timestamps were tracked.
***
@@ -43,8 +43,8 @@ optional indexDetails: any;
Index-type-specific details parsed as a JavaScript object.
Falls back to a raw string if JSON parsing fails. `undefined` for
remote tables or when details are unavailable.
Falls back to a raw string if JSON parsing fails. `undefined` in
LanceDB Distributed or when details are unavailable.
***
@@ -66,7 +66,7 @@ optional indexUuid: string;
The UUID of the first segment of the index.
`undefined` for remote tables, which do not yet surface this.
`undefined` in LanceDB Distributed, which does not yet surface this.
***
@@ -78,7 +78,7 @@ optional indexVersion: number;
The on-disk index format version.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
***
@@ -100,7 +100,7 @@ optional numIndexedRows: number;
The number of rows indexed, across all segments.
`undefined` for remote tables.
May be `undefined` in LanceDB Distributed if the server does not report it.
***
@@ -112,7 +112,7 @@ optional numSegments: number;
The number of segments that make up the index.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
***
@@ -124,7 +124,7 @@ optional numUnindexedRows: number;
The number of rows not yet covered by this index.
`undefined` for remote tables.
May be `undefined` in LanceDB Distributed if the server does not report it.
***
@@ -136,7 +136,7 @@ optional sizeBytes: number;
The total size in bytes of all index files across all segments.
`undefined` for remote tables or indices without size tracking.
`undefined` in LanceDB Distributed or indices without size tracking.
***
@@ -148,4 +148,4 @@ optional typeUrl: string;
The protobuf type URL, a precise type identifier for the index.
`undefined` for remote tables.
`undefined` in LanceDB Distributed.
docs/src/js/interfaces/IvfFlatOptions.md
+23 -2
View File
@@ -82,8 +82,9 @@ optional numPartitions: number;
The number of IVF partitions to create.
This value should generally scale with the number of rows in the dataset.
By default the number of partitions is the square root of the number of
rows.
When set, this takes precedence over `targetPartitionSize`. When omitted,
the partition count is derived from `targetPartitionSize`, which defaults
to 4096 rows for this index type.
If this value is too large then the first part of the search (picking the
right partition) will be slow. If this value is too small then the second
@@ -110,3 +111,23 @@ Increasing this value might improve the quality of the index but in most cases t
default should be sufficient.
The default value is 256.
***
### targetPartitionSize?
```ts
optional targetPartitionSize: number;
```
The target number of rows in each IVF partition.
This is an alternative to `numPartitions`. When it is set and
`numPartitions` is not, the number of partitions is derived by dividing
the number of rows by this value. When both are set, `numPartitions` is
used and this value is ignored. When neither is set, the target is
4096 rows.
Larger partitions mean fewer of them, which makes the first part of the
search (picking the right partitions) faster and the second part
(searching within a partition) slower.
docs/src/js/interfaces/IvfPqOptions.md
+23 -2
View File
@@ -92,8 +92,9 @@ optional numPartitions: number;
The number of IVF partitions to create.
This value should generally scale with the number of rows in the dataset.
By default the number of partitions is the square root of the number of
rows.
When set, this takes precedence over `targetPartitionSize`. When omitted,
the partition count is derived from `targetPartitionSize`, which defaults
to 8192 rows for this index type.
If this value is too large then the first part of the search (picking the
right partition) will be slow. If this value is too small then the second
@@ -141,3 +142,23 @@ Increasing this value might improve the quality of the index but in most cases t
default should be sufficient.
The default value is 256.
***
### targetPartitionSize?
```ts
optional targetPartitionSize: number;
```
The target number of rows in each IVF partition.
This is an alternative to `numPartitions`. When it is set and
`numPartitions` is not, the number of partitions is derived by dividing
the number of rows by this value. When both are set, `numPartitions` is
used and this value is ignored. When neither is set, the target is
8192 rows.
Larger partitions mean fewer of them, which makes the first part of the
search (picking the right partitions) faster and the second part
(searching within a partition) slower.

Some files were not shown because too many files have changed in this diff Show More