mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-15 07:06:52 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
33 lines
9.8 KiB
Plaintext
33 lines
9.8 KiB
Plaintext
{/* Auto-generated by scripts/mdx_snippets_gen.py. Do not edit manually. */}
|
|
|
|
export const PyFilterSearch = "table = db.create_table(\n \"documents\",\n [\n {\n \"id\": 1,\n \"text\": \"A puppy playing in the park\",\n \"category\": \"animals\",\n \"vector\": [1.0, 0.0],\n },\n {\n \"id\": 2,\n \"text\": \"A dog resting at home\",\n \"category\": \"animals\",\n \"vector\": [0.9, 0.1],\n },\n {\n \"id\": 3,\n \"text\": \"A guide to growing tomatoes\",\n \"category\": \"gardening\",\n \"vector\": [0.0, 1.0],\n },\n ],\n)\nrows = (\n table.search([0.0, 1.0])\n .where(\"category = 'animals'\", prefilter=True)\n .select([\"id\", \"text\", \"category\", \"_distance\"])\n .limit(2)\n .to_list()\n)\nprint(rows)\n";
|
|
|
|
export const PyHybridSearch = "table = db.create_table(\n \"documents\",\n [\n {\n \"id\": 1,\n \"text\": \"A puppy playing in the park\",\n \"category\": \"animals\",\n \"vector\": [1.0, 0.0],\n },\n {\n \"id\": 2,\n \"text\": \"A dog resting at home\",\n \"category\": \"animals\",\n \"vector\": [0.9, 0.1],\n },\n {\n \"id\": 3,\n \"text\": \"A guide to growing tomatoes\",\n \"category\": \"gardening\",\n \"vector\": [0.0, 1.0],\n },\n ],\n)\nfrom lancedb.rerankers import RRFReranker\n\ntable.create_index(\"text\", config=FTS())\nrows = (\n table.search(\n query_type=\"hybrid\", vector_column_name=\"vector\", fts_columns=\"text\"\n )\n .vector([1.0, 0.0])\n .text(\"puppy\")\n .rerank(RRFReranker())\n .limit(2)\n .to_list()\n)\nprint(rows)\n";
|
|
|
|
export const PyQueryPlan = "table = db.create_table(\n \"documents\",\n [\n {\n \"id\": 1,\n \"text\": \"A puppy playing in the park\",\n \"category\": \"animals\",\n \"vector\": [1.0, 0.0],\n },\n {\n \"id\": 2,\n \"text\": \"A dog resting at home\",\n \"category\": \"animals\",\n \"vector\": [0.9, 0.1],\n },\n {\n \"id\": 3,\n \"text\": \"A guide to growing tomatoes\",\n \"category\": \"gardening\",\n \"vector\": [0.0, 1.0],\n },\n ],\n)\nquery = table.search([1.0, 0.0]).select([\"id\", \"text\", \"_distance\"]).limit(2)\nprint(\"Plan:\", query.explain_plan())\nprint(\"Measured execution:\", query.analyze_plan())\nrows = query.to_list()\nprint(rows)\n";
|
|
|
|
export const PyTextSearch = "table = db.create_table(\n \"documents\",\n [\n {\n \"id\": 1,\n \"text\": \"A puppy playing in the park\",\n \"category\": \"animals\",\n },\n {\n \"id\": 2,\n \"text\": \"A dog resting at home\",\n \"category\": \"animals\",\n },\n {\n \"id\": 3,\n \"text\": \"A guide to growing tomatoes\",\n \"category\": \"gardening\",\n },\n ],\n)\ntable.create_index(\"text\", config=FTS())\nrows = (\n table.search(\"puppy\", query_type=\"fts\", fts_columns=\"text\")\n .select([\"id\", \"text\", \"_score\"])\n .limit(2)\n .to_list()\n)\nprint(rows)\n";
|
|
|
|
export const PyVectorSearch = "table = db.create_table(\n \"documents\",\n [\n {\n \"id\": 1,\n \"text\": \"A puppy playing in the park\",\n \"category\": \"animals\",\n \"vector\": [1.0, 0.0],\n },\n {\n \"id\": 2,\n \"text\": \"A dog resting at home\",\n \"category\": \"animals\",\n \"vector\": [0.9, 0.1],\n },\n {\n \"id\": 3,\n \"text\": \"A guide to growing tomatoes\",\n \"category\": \"gardening\",\n \"vector\": [0.0, 1.0],\n },\n ],\n)\nrows = (\n table.search([1.0, 0.0])\n .distance_type(\"l2\")\n .select([\"id\", \"text\", \"_distance\"])\n .limit(2)\n .to_list()\n)\nprint(rows)\n";
|
|
|
|
export const TsFilterSearchBody = "const lancedb = await import(\"@lancedb/lancedb\");\nconst { mkdtemp, rm } = await import(\"node:fs/promises\");\nconst { tmpdir } = await import(\"node:os\");\nconst { join } = await import(\"node:path\");\nconst uri = await mkdtemp(join(tmpdir(), \"lancedb-search-\"));\ntry {\n const db = await lancedb.connect(uri);\n const table = await db.createTable(\"documents\", [\n {\n id: 1,\n text: \"A puppy playing in the park\",\n category: \"animals\",\n vector: [1.0, 0.0],\n },\n {\n id: 2,\n text: \"A dog resting at home\",\n category: \"animals\",\n vector: [0.9, 0.1],\n },\n {\n id: 3,\n text: \"A guide to growing tomatoes\",\n category: \"gardening\",\n vector: [0.0, 1.0],\n },\n ]);\n const rows = await table\n .vectorSearch([0.0, 1.0])\n .where(\"category = 'animals'\")\n .select([\"id\", \"text\", \"category\", \"_distance\"])\n .limit(2)\n .toArray();\n console.log(rows);\n";
|
|
|
|
export const TsFilterSearchTail = "} finally {\n await rm(uri, { recursive: true, force: true });\n}\n";
|
|
|
|
export const TsHybridSearchBody = "const lancedb = await import(\"@lancedb/lancedb\");\nconst { mkdtemp, rm } = await import(\"node:fs/promises\");\nconst { tmpdir } = await import(\"node:os\");\nconst { join } = await import(\"node:path\");\nconst uri = await mkdtemp(join(tmpdir(), \"lancedb-search-\"));\ntry {\n const db = await lancedb.connect(uri);\n const table = await db.createTable(\"documents\", [\n {\n id: 1,\n text: \"A puppy playing in the park\",\n category: \"animals\",\n vector: [1.0, 0.0],\n },\n {\n id: 2,\n text: \"A dog resting at home\",\n category: \"animals\",\n vector: [0.9, 0.1],\n },\n {\n id: 3,\n text: \"A guide to growing tomatoes\",\n category: \"gardening\",\n vector: [0.0, 1.0],\n },\n ]);\n await table.createIndex(\"text\", { config: lancedb.Index.fts() });\n const reranker = await lancedb.rerankers.RRFReranker.create();\n const rows = await table\n .vectorSearch([1.0, 0.0])\n .fullTextSearch(\"puppy\", { columns: [\"text\"] })\n .rerank(reranker)\n .limit(2)\n .toArray();\n console.log(rows);\n";
|
|
|
|
export const TsHybridSearchTail = "} finally {\n await rm(uri, { recursive: true, force: true });\n}\n";
|
|
|
|
export const TsQueryPlanBody = "const lancedb = await import(\"@lancedb/lancedb\");\nconst { mkdtemp, rm } = await import(\"node:fs/promises\");\nconst { tmpdir } = await import(\"node:os\");\nconst { join } = await import(\"node:path\");\nconst uri = await mkdtemp(join(tmpdir(), \"lancedb-search-\"));\ntry {\n const db = await lancedb.connect(uri);\n const table = await db.createTable(\"documents\", [\n {\n id: 1,\n text: \"A puppy playing in the park\",\n category: \"animals\",\n vector: [1.0, 0.0],\n },\n {\n id: 2,\n text: \"A dog resting at home\",\n category: \"animals\",\n vector: [0.9, 0.1],\n },\n {\n id: 3,\n text: \"A guide to growing tomatoes\",\n category: \"gardening\",\n vector: [0.0, 1.0],\n },\n ]);\n const query = table\n .vectorSearch([1.0, 0.0])\n .select([\"id\", \"text\", \"_distance\"])\n .limit(2);\n console.log(\"Plan:\", await query.explainPlan());\n console.log(\"Measured execution:\", await query.analyzePlan());\n const rows = await query.toArray();\n console.log(rows);\n";
|
|
|
|
export const TsQueryPlanTail = "} finally {\n await rm(uri, { recursive: true, force: true });\n}\n";
|
|
|
|
export const TsTextSearchBody = "const lancedb = await import(\"@lancedb/lancedb\");\nconst { mkdtemp, rm } = await import(\"node:fs/promises\");\nconst { tmpdir } = await import(\"node:os\");\nconst { join } = await import(\"node:path\");\nconst uri = await mkdtemp(join(tmpdir(), \"lancedb-search-\"));\ntry {\n const db = await lancedb.connect(uri);\n const table = await db.createTable(\"documents\", [\n { id: 1, text: \"A puppy playing in the park\", category: \"animals\" },\n { id: 2, text: \"A dog resting at home\", category: \"animals\" },\n { id: 3, text: \"A guide to growing tomatoes\", category: \"gardening\" },\n ]);\n await table.createIndex(\"text\", { config: lancedb.Index.fts() });\n const rows = await table\n .query()\n .fullTextSearch(\"puppy\", { columns: [\"text\"] })\n .select([\"id\", \"text\", \"_score\"])\n .limit(2)\n .toArray();\n console.log(rows);\n";
|
|
|
|
export const TsTextSearchTail = "} finally {\n await rm(uri, { recursive: true, force: true });\n}\n";
|
|
|
|
export const TsVectorSearchBody = "const lancedb = await import(\"@lancedb/lancedb\");\nconst { mkdtemp, rm } = await import(\"node:fs/promises\");\nconst { tmpdir } = await import(\"node:os\");\nconst { join } = await import(\"node:path\");\nconst uri = await mkdtemp(join(tmpdir(), \"lancedb-search-\"));\ntry {\n const db = await lancedb.connect(uri);\n const table = await db.createTable(\"documents\", [\n {\n id: 1,\n text: \"A puppy playing in the park\",\n category: \"animals\",\n vector: [1.0, 0.0],\n },\n {\n id: 2,\n text: \"A dog resting at home\",\n category: \"animals\",\n vector: [0.9, 0.1],\n },\n {\n id: 3,\n text: \"A guide to growing tomatoes\",\n category: \"gardening\",\n vector: [0.0, 1.0],\n },\n ]);\n const rows = await table\n .vectorSearch([1.0, 0.0])\n .distanceType(\"l2\")\n .select([\"id\", \"text\", \"_distance\"])\n .limit(2)\n .toArray();\n console.log(rows);\n";
|
|
|
|
export const TsVectorSearchTail = "} finally {\n await rm(uri, { recursive: true, force: true });\n}\n";
|
|
|