docs: complete ingestion, search and integration examples (#4533)

This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.

Ingestion, search and integration examples are incomplete or
inconsistent across languages, and some guidance misstates SDK behavior.

This brings the guides and their example sources together. It includes
complete Python, TypeScript and Rust workflows, isolated provider and
reranker examples, the Superlinked corrections, and guidance on retries
that can duplicate writes.

Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after
#4491 is squash-merged.

Local assembly, Mint validation and link checks pass. Example code is
unchanged from the previously validated package. Publication stays held
through integration.

docs/web/integrations/embedding/cohere.mdx

@@ -8,6 +8,10 @@ import { PyEmbeddingCohereUsage } from '/snippets/integrations.mdx';
 Using cohere API requires cohere package, which can be installed using `pip install cohere`. Cohere embeddings are used to generate embeddings for text data. The embeddings can be used for various tasks like semantic search, clustering, and classification.
 You also need to set the `COHERE_API_KEY` environment variable to use the Cohere API.
 
+```shell
+export COHERE_API_KEY="your-api-key"
+```
+
 Supported models are:
 
 - embed-english-v3.0

docs/web/integrations/embedding/ibm.mdx

@@ -3,7 +3,7 @@ title: IBM watsonx
 sidebarTitle: IBM WatsonX
 ---
 
-import { PyEmbeddingIbmUsage } from '/snippets/integrations.mdx';
+import { PyEmbeddingIbmUsage } from '/snippets/provider_examples.mdx';
 
 Generate text embeddings using IBM's watsonx.ai platform.
 
@@ -54,9 +54,13 @@ Optionally set environment variables (if not passing credentials to `create` dir
 export WATSONX_API_KEY="YOUR_WATSONX_API_KEY"
 # Provide exactly one of the following:
 export WATSONX_PROJECT_ID="YOUR_WATSONX_PROJECT_ID"
-export WATSONX_SPACE_ID="YOUR_WATSONX_SPACE_ID"
+# export WATSONX_SPACE_ID="YOUR_WATSONX_SPACE_ID"
 ```
 
+Replace the credential placeholders with your provider credentials. The registry
+variable keeps the API key out of stored embedding configuration. In application
+code, load its value from environment variables or a secrets manager.
+
 <CodeGroup>
   <CodeBlock filename="Python" language="Python" icon="python">
     {PyEmbeddingIbmUsage}

docs/web/integrations/embedding/jina.mdx

@@ -6,7 +6,7 @@ sidebarTitle: Jina
 import {
   PyEmbeddingJinaText,
   PyEmbeddingJinaMultimodal,
-} from '/snippets/integrations.mdx';
+} from '/snippets/provider_examples.mdx';
 
 ## Text Embedding Models {#text-embedding-models}
 
@@ -23,6 +23,8 @@ Supported parameters (to be passed in `create` method) are:
 
 Usage Example:
 
+Replace `your-api-key` with your Jina API key. This provider reads it from `JINA_API_KEY`; the example sets it explicitly. In application code, supply it through your environment or a secrets manager.
+
 <CodeGroup>
   <CodeBlock filename="Python" language="Python" icon="python">
     {PyEmbeddingJinaText}

docs/web/integrations/embedding/openai.mdx

@@ -10,9 +10,15 @@ LanceDB registers the OpenAI embeddings function in the registry by default, as
 | Parameter | Type | Default Value | Description |
 |---|---|---|---|
 | `name` | `str` | `"text-embedding-ada-002"` | The name of the model. |
-| `dim` | `int` |  Model default   | For OpenAI's newer text-embedding-3 model, we can specify a dimensionality that is smaller than the 1536 size. This feature supports it |
+| `dim` | `int` |  Model default   | Requested output dimensions for models that support shortening embeddings. Use `func.ndims()` when defining the vector field. |
 | `use_azure` | bool | `False` | Set true to use Azure OpenAI SDK |
 
+Set your OpenAI API key before running the example:
+
+```shell
+export OPENAI_API_KEY="your-api-key"
+```
+
 <CodeGroup>
   <CodeBlock filename="Python" language="Python" icon="python">
     {PyEmbeddingOpenaiBasic}

docs/web/integrations/embedding/superlinked.mdx

@@ -36,6 +36,9 @@ Supported parameters on `.create()`:
 ## Usage {#usage}
 
 ```py Python icon=Python
+import tempfile
+from pathlib import Path
+
 import lancedb
 from lancedb.embeddings import get_registry
 from lancedb.pydantic import LanceModel, Vector
@@ -47,22 +50,23 @@ sie = get_registry().get("sie").create(
 )
 
 class Documents(LanceModel):
+    id: int
     text: str = sie.SourceField()
     vector: Vector(sie.ndims()) = sie.VectorField()
 
-db = lancedb.connect("~/.lancedb")
-table = db.create_table("docs", schema=Documents, mode="overwrite")
+db = lancedb.connect(str(Path(tempfile.mkdtemp()) / "superlinked-demo"))
+table = db.create_table("docs", schema=Documents)
 
 table.add([
-    {"text": "Machine learning is a subset of AI."},
-    {"text": "Neural networks use multiple layers."},
-    {"text": "Python is popular for ML development."},
+    {"id": 1, "text": "Machine learning is a subset of AI."},
+    {"id": 2, "text": "Neural networks use multiple layers."},
+    {"id": 3, "text": "Python is popular for ML development."},
 ])
 
 results = table.search("What is deep learning?").limit(3).to_list()
 ```
 
-LanceDB handles embedding generation for both inserts and queries automatically, based on the `SourceField` / `VectorField` declarations on the schema.
+LanceDB handles embedding generation for both inserts and queries automatically, based on the `SourceField` / `VectorField` declarations on the schema. The `id` column gives each document a stable key, which [entity extraction](#entity-extraction) uses to merge its results back.
 
 ## Hybrid search with reranker {#hybrid-search-with-reranker}
 
@@ -88,7 +92,7 @@ for r in results:
 The reranker also works with pure vector or pure FTS search via `.rerank()`.
 
 ## ColBERT and multivector {#colbert-and-multivector}
-`SIEMultiVectorEmbeddingFunction` (registered as `"sie-multivector"`) works with LanceDB's native `MultiVector` type and MaxSim scoring for ColBERT and ColPali models:
+`SIEMultiVectorEmbeddingFunction` (registered as `"sie-multivector"`) works with LanceDB's native `MultiVector` type and MaxSim scoring for ColBERT and ColPali models. In LanceDB Python 0.40.0, a multivector that LanceDB generates on insert is stored as null unless the document's token count happens to equal the per-token dimension, so this example computes the document multivectors with `compute_source_embeddings()` and passes them in the `vector` column. LanceDB still generates the query multivector automatically:
 
 ```py Python icon=Python
 from lancedb.pydantic import MultiVector
@@ -102,8 +106,12 @@ class ColBERTDocs(LanceModel):
     text: str = sie_colbert.SourceField()
     vector: MultiVector(sie_colbert.ndims()) = sie_colbert.VectorField()
 
-table = db.create_table("colbert_docs", schema=ColBERTDocs, mode="overwrite")
-table.add([{"text": "Machine learning is a subset of AI."}])
+table = db.create_table("colbert_docs", schema=ColBERTDocs)
+
+# Compute the document multivectors and store them with their text
+texts = ["Machine learning is a subset of AI."]
+vectors = sie_colbert.compute_source_embeddings(texts)
+table.add([{"text": text, "vector": vector} for text, vector in zip(texts, vectors)])
 
 # MaxSim search - query and document multivectors compared token-by-token
 results = table.search("What is ML?").limit(5).to_list()
@@ -111,7 +119,7 @@ results = table.search("What is ML?").limit(5).to_list()
 
 ## Entity extraction {#entity-extraction}
 
-`SIEExtractor` adds entity extraction to LanceDB's data-enrichment workflows. Extract entities from a text column and merge the results back as a structured Arrow column - enabling filtered search on extracted entities:
+`SIEExtractor` adds entity extraction to LanceDB's data-enrichment workflows. Extract entities from a text column and merge the results back as a structured Arrow column - enabling filtered search on extracted entities. This example enriches the `docs` table from [Usage](#usage), matching rows on its `id` column:
 
 ```py Python icon=Python
 from sie_lancedb import SIEExtractor
@@ -121,8 +129,9 @@ extractor = SIEExtractor(
     model="urchade/gliner_multi-v2.1",
 )
 
+docs = db.open_table("docs")
 extractor.enrich_table(
-    table,
+    docs,
     source_column="text",
     target_column="entities",
     labels=["person", "technology", "organization"],

docs/web/integrations/embedding/voyageai.mdx

@@ -11,6 +11,10 @@ Voyage AI provides cutting-edge embedding and rerankers.
 Using voyageai API requires voyageai package, which can be installed using `pip install voyageai`. Voyage AI embeddings are used to generate embeddings for text data. The embeddings can be used for various tasks like semantic search, clustering, and classification.
 You also need to set the `VOYAGE_API_KEY` environment variable to use the VoyageAI API.
 
+```shell
+export VOYAGE_API_KEY="your-api-key"
+```
+
 Supported models are:
 
 - voyage-4-large (best retrieval quality, 1024 default dimensions, supports 256/512/1024/2048)