mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
e6b04c5da7822c48bddb7f1a7982ffdaf00a53df
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Jack YeandClaude Sonnet 5
|
de95ed3f6d |
docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 alexcastgonz
|
e9b2f7e6e1 |
fix(python): use dim as the embedding size for OpenAI-compatible models
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model names outside the three built-in OpenAI names, so schemas can be built for compatible models. Preserve existing model defaults and request behavior. Add regression coverage for the resulting Arrow schema dimensions and for all known OpenAI model dimensions. |
||
 Brendan ClementandClaude Opus 5.5
|
ca343a5c06 |
docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)
## Summary Fixes the problems a page-by-page review of the docs found before launch, after checking each one against the current source and, where noted, the code. The matching fixes for the dataset and Geneva pages, which live in `lancedb/docs`, are in lancedb/docs#402. **Tables** - `integrations/embedding/cohere`: each input-type description was split across two table rows and wrapped in literal quotes (`"`search_document`"`). Now one row per type. - `integrations/embedding/gemini`: removes the same stray quotes from the task-type table. - `integrations/ai/langchain`: the MMR search table's `kwargs` row was missing a cell, so its description sat under Type. - Six reranker pages (answerdotai, cohere, colbert, jina, openai, voyageai): closes the backtick before `_relevance_score`. - `indexing/vector-index`: finishes a cell cut off at "with scalar". - `sql/dql/dedupe`: labels the empty header cell in the resolution table. **Corrections** - `curation/sampling`: sampling works in LanceDB Embedded through the Python SDK, so the page is available in both, with the SQL sections marked Enterprise. - `feature-engineering/index`, `add-computed-column`: automatic refresh is on by default for each computed column and takes effect once a deployment enables its scheduler. `refresh-column` already said so; these two implied it was off. - `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates `array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also merges scalar-index's two "nested fields" sections into one. - `integrations/embedding/ibm`: lists the four current models from `CURRENT_MODELS` (it listed a deprecated one) and explains that the default stays a legacy ID so existing tables reload with their model. - `integrations/reranking/openai`: the class is `OpenaiReranker`, not `OpenAI()`. - `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit` throughout, each step runs one command, each step passes the file it writes to the next, and `save-as` uses `-f`/`-o`. - `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the pipeline reads, and the run command runs the file the page creates. - `search/vector-search`: the multivector walkthrough connected to Enterprise in step 1 and a local path in step 2. It now runs locally throughout, with a note on connecting to Enterprise, and is renamed so it doesn't read as a duplicate of the short section above. **Structure and wording** - `integrations/embedding/nvidia`: adds headings, and drops test-suite details aimed at contributors. - `integrations/ai/genkit`: makes the retrieval test a real heading and describes the query step. - `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a `<Note>`. - `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading, not bold text. - `curation/windowing`: sidebar title matches the curation overview ("Rank and Deduplicate"). - `deployment/storage`, `basics/agentic-skills`: fixes a garbled sentence and a stale reference to the quickstart. **Checked:** no new Vale alerts on the changed pages. Assembled together with lancedb/docs#402, the site passes `mint validate` and `mint broken-links --check-anchors --check-redirects`. Every changed page was reviewed in a local preview. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 Taylor
|
a065b4b9c6 |
feat(python): add NVIDIA Nemotron embedding provider
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions. - Documents are embedded as `passage`, searches as `query`. - Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`. - Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request. - Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately. - `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`. - Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`. Python and text only. Docs are in `integrations/embedding/nvidia`. Originally authored by Taylor (@tsmithv11). ### Testing - `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables. - Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected. - Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment. |
||
|
659ea11ac4 |
fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0` on April 4, 2026 ([deprecations](https://docs.cohere.com/docs/deprecations), which lists `embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the embedding alternatives). `CohereEmbeddingFunction` still defaulted to `embed-multilingual-v2.0`, so every call without an explicit `name` fails. The default is now `embed-multilingual-v3.0`, the multilingual v3 model, and the retired models are removed from the docstring and the Cohere integration page (the page also listed `embed-english-v2.0` as the default, which did not match the code). The docs example, its generated snippet, the docs test and the slow Cohere test use the new name. This changes the default vector size: `ndims()` returns 1024 for `embed-multilingual-v3.0`, where the old default gave 768. Tables created from a schema that uses `cohere.ndims()` get 1024-dimension vectors from now on. A table that already holds 768-dimension vectors from the retired model needs a re-embed with a current model, since the old model can no longer be called. The `ndims()` entries for the v2 names are left as they are. Test: `test_cohere_default_model_is_not_a_retired_v2_model` in `python/python/tests/test_embeddings.py` checks the default name and that `ndims()` is 1024. The repository's pytest session could not be run for this change (its `conftest.py` needs a newer native module than the released wheel); the body of that test was run against the released `lancedb` 0.40.0 with this version of `cohere.py` in place, where it passes, and it returns `embed-multilingual-v2.0` / 768 without the change. ruff check and format are clean on the touched Python files. No Cohere request was made. This change was made with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Captain
|
a6011dcd95 |
fix(python): accept NumPy text arrays in Ollama embeddings (#4605)
`OllamaEmbeddings.compute_source_embeddings(np.array(["hello", "world"]))` raises a validation error before sending a request. The embedding interface accepts NumPy arrays, but the adapter passes the array directly to Ollama's string-or-sequence request field. Convert NumPy text arrays to lists before the batched request. This preserves input order, the single request, model options and `keep_alive`. The tests cover Unicode/object/sliced arrays, empty input, existing list/Arrow/query inputs and API error propagation. The added input-contract tests also cover two-dimensional arrays and byte strings, with equivalent list inputs as controls. The real Ollama SDK rejects 2-D inputs and invalid UTF-8 locally, before transport. Valid UTF-8 bytes are decoded normally. No extra adapter validation is needed for these cases. Validation on Python 3.12.14 / macOS arm64: - Built the Rust/PyO3 bindings using the repository's uv/maturin environment. - The initial regression run had 5 failures and 4 passing controls on the base; all 9 passed with the fix. With the 6 additional input-contract cases, the related embedding and trust suites now total 55 passed / 11 skipped. - All Python Ruff lint and formatting checks pass. Root checks still report the same 49 lint errors and 11 documentation formatting files reproduced on the untouched base. - Separate review exercised 13 public API cases. Another 7 groups passed through a real loopback HTTP transport, including local validation, UTF-8 serialization and empty-batch model-error propagation. The regression tests use the real Ollama 0.6.2 SDK with HTTPX MockTransport. The loopback check uses an HTTP surrogate, not an Ollama daemon. No live model inference was run. A live Ollama server was unavailable locally. [The current server handler](https://github.com/ollama/ollama/blob/2cd63f7a15039ccd790e97b31edc091fa1707c56/server/routes.go#L1212) explicitly returns HTTP 200 with empty embeddings for empty input after model lookup and runner setup. The adapter preserves the existing empty-list request and its model/setup errors. This is a source check, not a live-server test. Developed with OpenAI Codex, with separate agent review and executed local checks. |
||
|
Captain
|
f2c6aa35c2 |
fix(python): normalize VoyageAI image modes before JPEG encoding (#4576)
PIL images, PNG bytes and local image paths in VoyageAI source embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA) and palette (P) images therefore raise `OSError: cannot write mode ... as JPEG` before the provider is called. Convert to RGB at all three JPEG encoding sites. The caller’s original PIL image remains unchanged. Document that alpha is discarded and callers should composite onto a chosen background when it matters. Validation: - New image regressions + existing VoyageAI tests: **18 failed / 26 passed before → 44 passed after**. Tests cover both multimodal models, PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images. - Independent local LanceDB table ingestion smoke: three PNG modes stored with 1024-dimensional vectors, original bytes preserved. Uses the freshly built current native extension. - Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key), 9 deselected. The separate Flickr download test failed with `RemoteDisconnected`; it is excluded from the local result. - Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting passed. External VoyageAI calls are stubbed; real Pillow conversion and LanceDB ingestion execute locally. No live provider or full-repository test result is claimed. AI assistance was used for implementation and validation. |
||
|
Captain
|
0cb5bd0a5d |
fix(python): accept VoyageAI text batch containers (#4558)
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]` and NumPy text batches before reaching the provider, although both belong to the declared `TEXT` input type. For example, `["first", "second"]` raises `ValueError: Input type <class 'list'> not allowed with text model.` Normalize these containers to a list before the existing string-entry validation. The regression cases exercise the public embedding API with a stub provider: input order, list/NumPy batches, Arrow controls, empty batches, and rejection of non-text entries and ordinary two-dimensional arrays. The VoyageAI integration page now documents the accepted text containers. Verified on current main `7b3ba4c` with a locally compiled native extension (0.41.0b7, Python 3.12.12): - VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed afterward. - Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases deselected. - Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured Pyright reports zero errors/warnings; all applicable pre-commit hooks pass. The provider is stubbed; no live Voyage API request or model-quality evaluation was performed. AI disclosure: I used Codex for investigation, implementation, tests, review, and drafting this description. |
||
|
Brendan Clement
|
198812fd53 |
docs: clean up prose flagged by the shared Vale rules (#4596)
The existing docs have 254 findings from our shared Vale rules. Contributors would otherwise inherit these when editing the same lines. Clean up the prose across 80 pages: correct product names and a repeated word, simplify a few phrases, and replace dashes with punctuation that fits each sentence. The pinned Vale scan now reports zero findings, with no suppressions. |
||
 Will JonesandClaude Opus 5.5
|
9020640538 |
fix(python)!: don't trust embedding settings from table metadata (#4550)
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Brendan Clement
|
758f738b8c |
docs: complete ingestion, search and integration examples (#4533)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. Ingestion, search and integration examples are incomplete or inconsistent across languages, and some guidance misstates SDK behavior. This brings the guides and their example sources together. It includes complete Python, TypeScript and Rust workflows, isolated provider and reranker examples, the Superlinked corrections, and guidance on retries that can duplicate writes. Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after #4491 is squash-merged. Local assembly, Mint validation and link checks pass. Example code is unchanged from the previously validated package. Publication stays held through integration. |
||
|
Jack YeandBrendan Clement
|
47b7babc08 |
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (
|