mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
171 lines
8.9 KiB
Plaintext
171 lines
8.9 KiB
Plaintext
---
|
|
title: Sentence Transformers
|
|
sidebarTitle: Sentence Transformers
|
|
---
|
|
|
|
import { PyEmbeddingSentenceTransformersBaai } from '/snippets/integrations.mdx';
|
|
|
|
Allows you to set parameters when registering a `sentence-transformers` object.
|
|
|
|
<Info>
|
|
Sentence transformer embeddings are normalized by default. It is recommended to use normalized embeddings for similarity search.
|
|
</Info>
|
|
|
|
<Warning>
|
|
Setting `trust_remote_code=True` allows the model to execute arbitrary code from its HuggingFace repository during loading. Only enable it for repositories you trust, and pin `revision` to a commit hash. When a table using such a model is opened in a new process, the model must be approved with `get_registry().allow_remote_code(...)`. See [Trusting remote code and custom endpoints](/embedding#trusting-remote-code-and-custom-endpoints).
|
|
</Warning>
|
|
|
|
| Parameter | Type | Default Value | Description |
|
|
|---|---|---|---|
|
|
| `name` | `str` | `all-MiniLM-L6-v2` | The name of the model |
|
|
| `device` | `str` | `cpu` | The device to run the model on (can be `cpu` or `gpu`) |
|
|
| `normalize` | `bool` | `True` | Whether to normalize the input text before feeding it to the model |
|
|
| `trust_remote_code` | `bool` | `False` | Whether to trust and execute remote code from the model's HuggingFace repository |
|
|
| `revision` | `str` | `None` | The model revision to load, such as a commit hash |
|
|
|
|
|
|
<Details title="Check out available sentence-transformer models here!">
|
|
```markdown
|
|
- sentence-transformers/all-MiniLM-L12-v2
|
|
- sentence-transformers/paraphrase-mpnet-base-v2
|
|
- sentence-transformers/gtr-t5-base
|
|
- sentence-transformers/LaBSE
|
|
- sentence-transformers/all-MiniLM-L6-v2
|
|
- sentence-transformers/bert-base-nli-max-tokens
|
|
- sentence-transformers/bert-base-nli-mean-tokens
|
|
- sentence-transformers/bert-base-nli-stsb-mean-tokens
|
|
- sentence-transformers/bert-base-wikipedia-sections-mean-tokens
|
|
- sentence-transformers/bert-large-nli-cls-token
|
|
- sentence-transformers/bert-large-nli-max-tokens
|
|
- sentence-transformers/bert-large-nli-mean-tokens
|
|
- sentence-transformers/bert-large-nli-stsb-mean-tokens
|
|
- sentence-transformers/distilbert-base-nli-max-tokens
|
|
- sentence-transformers/distilbert-base-nli-mean-tokens
|
|
- sentence-transformers/distilbert-base-nli-stsb-mean-tokens
|
|
- sentence-transformers/distilroberta-base-msmarco-v1
|
|
- sentence-transformers/distilroberta-base-msmarco-v2
|
|
- sentence-transformers/nli-bert-base-cls-pooling
|
|
- sentence-transformers/nli-bert-base-max-pooling
|
|
- sentence-transformers/nli-bert-base
|
|
- sentence-transformers/nli-bert-large-cls-pooling
|
|
- sentence-transformers/nli-bert-large-max-pooling
|
|
- sentence-transformers/nli-bert-large
|
|
- sentence-transformers/nli-distilbert-base-max-pooling
|
|
- sentence-transformers/nli-distilbert-base
|
|
- sentence-transformers/nli-roberta-base
|
|
- sentence-transformers/nli-roberta-large
|
|
- sentence-transformers/roberta-base-nli-mean-tokens
|
|
- sentence-transformers/roberta-base-nli-stsb-mean-tokens
|
|
- sentence-transformers/roberta-large-nli-mean-tokens
|
|
- sentence-transformers/roberta-large-nli-stsb-mean-tokens
|
|
- sentence-transformers/stsb-bert-base
|
|
- sentence-transformers/stsb-bert-large
|
|
- sentence-transformers/stsb-distilbert-base
|
|
- sentence-transformers/stsb-roberta-base
|
|
- sentence-transformers/stsb-roberta-large
|
|
- sentence-transformers/xlm-r-100langs-bert-base-nli-mean-tokens
|
|
- sentence-transformers/xlm-r-100langs-bert-base-nli-stsb-mean-tokens
|
|
- sentence-transformers/xlm-r-base-en-ko-nli-ststb
|
|
- sentence-transformers/xlm-r-bert-base-nli-mean-tokens
|
|
- sentence-transformers/xlm-r-bert-base-nli-stsb-mean-tokens
|
|
- sentence-transformers/xlm-r-large-en-ko-nli-ststb
|
|
- sentence-transformers/bert-base-nli-cls-token
|
|
- sentence-transformers/all-distilroberta-v1
|
|
- sentence-transformers/multi-qa-MiniLM-L6-dot-v1
|
|
- sentence-transformers/multi-qa-distilbert-cos-v1
|
|
- sentence-transformers/multi-qa-distilbert-dot-v1
|
|
- sentence-transformers/multi-qa-mpnet-base-cos-v1
|
|
- sentence-transformers/multi-qa-mpnet-base-dot-v1
|
|
- sentence-transformers/nli-distilroberta-base-v2
|
|
- sentence-transformers/all-MiniLM-L6-v1
|
|
- sentence-transformers/all-mpnet-base-v1
|
|
- sentence-transformers/all-mpnet-base-v2
|
|
- sentence-transformers/all-roberta-large-v1
|
|
- sentence-transformers/allenai-specter
|
|
- sentence-transformers/average_word_embeddings_glove.6B.300d
|
|
- sentence-transformers/average_word_embeddings_glove.840B.300d
|
|
- sentence-transformers/average_word_embeddings_komninos
|
|
- sentence-transformers/average_word_embeddings_levy_dependency
|
|
- sentence-transformers/clip-ViT-B-32-multilingual-v1
|
|
- sentence-transformers/clip-ViT-B-32
|
|
- sentence-transformers/distilbert-base-nli-stsb-quora-ranking
|
|
- sentence-transformers/distilbert-multilingual-nli-stsb-quora-ranking
|
|
- sentence-transformers/distilroberta-base-paraphrase-v1
|
|
- sentence-transformers/distiluse-base-multilingual-cased-v1
|
|
- sentence-transformers/distiluse-base-multilingual-cased-v2
|
|
- sentence-transformers/distiluse-base-multilingual-cased
|
|
- sentence-transformers/facebook-dpr-ctx_encoder-multiset-base
|
|
- sentence-transformers/facebook-dpr-ctx_encoder-single-nq-base
|
|
- sentence-transformers/facebook-dpr-question_encoder-multiset-base
|
|
- sentence-transformers/facebook-dpr-question_encoder-single-nq-base
|
|
- sentence-transformers/gtr-t5-large
|
|
- sentence-transformers/gtr-t5-xl
|
|
- sentence-transformers/gtr-t5-xxl
|
|
- sentence-transformers/msmarco-MiniLM-L-12-v3
|
|
- sentence-transformers/msmarco-MiniLM-L-6-v3
|
|
- sentence-transformers/msmarco-MiniLM-L12-cos-v5
|
|
- sentence-transformers/msmarco-MiniLM-L6-cos-v5
|
|
- sentence-transformers/msmarco-bert-base-dot-v5
|
|
- sentence-transformers/msmarco-bert-co-condensor
|
|
- sentence-transformers/msmarco-distilbert-base-dot-prod-v3
|
|
- sentence-transformers/msmarco-distilbert-base-tas-b
|
|
- sentence-transformers/msmarco-distilbert-base-v2
|
|
- sentence-transformers/msmarco-distilbert-base-v3
|
|
- sentence-transformers/msmarco-distilbert-base-v4
|
|
- sentence-transformers/msmarco-distilbert-cos-v5
|
|
- sentence-transformers/msmarco-distilbert-dot-v5
|
|
- sentence-transformers/msmarco-distilbert-multilingual-en-de-v2-tmp-lng-aligned
|
|
- sentence-transformers/msmarco-distilbert-multilingual-en-de-v2-tmp-trained-scratch
|
|
- sentence-transformers/msmarco-distilroberta-base-v2
|
|
- sentence-transformers/msmarco-roberta-base-ance-firstp
|
|
- sentence-transformers/msmarco-roberta-base-v2
|
|
- sentence-transformers/msmarco-roberta-base-v3
|
|
- sentence-transformers/multi-qa-MiniLM-L6-cos-v1
|
|
- sentence-transformers/nli-mpnet-base-v2
|
|
- sentence-transformers/nli-roberta-base-v2
|
|
- sentence-transformers/nq-distilbert-base-v1
|
|
- sentence-transformers/paraphrase-MiniLM-L12-v2
|
|
- sentence-transformers/paraphrase-MiniLM-L3-v2
|
|
- sentence-transformers/paraphrase-MiniLM-L6-v2
|
|
- sentence-transformers/paraphrase-TinyBERT-L6-v2
|
|
- sentence-transformers/paraphrase-albert-base-v2
|
|
- sentence-transformers/paraphrase-albert-small-v2
|
|
- sentence-transformers/paraphrase-distilroberta-base-v1
|
|
- sentence-transformers/paraphrase-distilroberta-base-v2
|
|
- sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
|
- sentence-transformers/paraphrase-multilingual-mpnet-base-v2
|
|
- sentence-transformers/paraphrase-xlm-r-multilingual-v1
|
|
- sentence-transformers/quora-distilbert-base
|
|
- sentence-transformers/quora-distilbert-multilingual
|
|
- sentence-transformers/sentence-t5-base
|
|
- sentence-transformers/sentence-t5-large
|
|
- sentence-transformers/sentence-t5-xxl
|
|
- sentence-transformers/sentence-t5-xl
|
|
- sentence-transformers/stsb-distilroberta-base-v2
|
|
- sentence-transformers/stsb-mpnet-base-v2
|
|
- sentence-transformers/stsb-roberta-base-v2
|
|
- sentence-transformers/stsb-xlm-r-multilingual
|
|
- sentence-transformers/xlm-r-distilroberta-base-paraphrase-v1
|
|
- sentence-transformers/clip-ViT-L-14
|
|
- sentence-transformers/clip-ViT-B-16
|
|
- sentence-transformers/use-cmlm-multilingual
|
|
- sentence-transformers/all-MiniLM-L12-v1
|
|
```
|
|
</Details>
|
|
|
|
<Info>
|
|
You can also load many other model architectures from the library. For example models from sources such as BAAI, Nomic, Salesforce Research, etc. See this HF hub page for all [supported models](https://huggingface.co/models?library=sentence-transformers).
|
|
</Info>
|
|
|
|
<Note title="BAAI embeddings example">
|
|
Here is an example that uses the BAAI embedding model from the HuggingFace Hub [supported models](https://huggingface.co/models?library=sentence-transformers).
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyEmbeddingSentenceTransformersBaai}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
</Note>
|
|
Visit the sentence-transformers [HuggingFace Hub](https://huggingface.co/sentence-transformers) page for more information on the available models.
|
|
|