From ddb318567a11211a9b75aca6dcf2902ff615f7d8 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Thu, 29 Oct 2026 10:16:03 -0700 Subject: [PATCH] docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table. --- docs/web-tests/py/test_quickstart.py | 151 +++--- docs/web-tests/rs/quickstart.rs | 240 +++++----- docs/web-tests/ts/quickstart.test.ts | 98 ++-- docs/web/.mintignore | 22 +- docs/web/architecture.mdx | 128 ------ docs/web/basics/agentic-skills.mdx | 2 +- docs/web/basics/catalog.mdx | 124 ++++- docs/web/basics/connect.mdx | 135 ------ docs/web/basics/guarantees.mdx | 4 +- docs/web/basics/jobs.mdx | 49 +- docs/web/basics/lakehouse.mdx | 75 --- docs/web/basics/lance.mdx | 4 +- docs/web/basics/limits.mdx | 6 +- docs/web/basics/offerings.mdx | 73 --- docs/web/basics/query.mdx | 55 --- docs/web/basics/sdk.mdx | 109 ++--- docs/web/basics/sql.mdx | 148 ++++-- docs/web/curation/aggregate.mdx | 4 +- docs/web/curation/changes.mdx | 93 ++++ docs/web/curation/chunking.mdx | 210 +++++++++ docs/web/curation/create-function.mdx | 213 +++++++++ docs/web/curation/dedupe.mdx | 142 ++++++ docs/web/curation/index.mdx | 6 +- docs/web/curation/join.mdx | 36 +- docs/web/curation/sampling.mdx | 10 +- docs/web/curation/windowing.mdx | 4 +- docs/web/deployment/storage.mdx | 17 +- docs/web/docs.json | 105 +++-- docs/web/embedding/index.mdx | 6 +- docs/web/enterprise/architecture.mdx | 23 - docs/web/enterprise/authentication.mdx | 6 +- docs/web/enterprise/benchmarks.mdx | 2 +- docs/web/enterprise/deployment/azure.mdx | 17 - docs/web/enterprise/deployment/index.mdx | 20 - docs/web/enterprise/deployment/kubernetes.mdx | 17 + docs/web/enterprise/distributed-search.mdx | 9 +- docs/web/enterprise/index.mdx | 13 +- docs/web/enterprise/security.mdx | 2 +- .../add-computed-column.mdx | 117 ++++- .../feature-engineering/data-evolution.mdx | 134 ------ docs/web/feature-engineering/index.mdx | 39 +- .../materialized-views.mdx | 151 +++--- .../feature-engineering/refresh-column.mdx | 57 ++- .../refresh-materialized-view.mdx | 88 ++++ docs/web/feature-engineering/view.mdx | 73 +++ docs/web/governance/authorization.mdx | 4 +- docs/web/index.mdx | 218 +++++---- docs/web/indexing/fts-index.mdx | 6 +- docs/web/indexing/index.mdx | 5 +- docs/web/indexing/reindexing.mdx | 7 +- docs/web/indexing/scalar-index.mdx | 6 +- docs/web/indexing/vector-index.mdx | 7 +- docs/web/ingestion/create-table.mdx | 4 +- docs/web/ingestion/delete.mdx | 4 +- docs/web/ingestion/index.mdx | 6 +- docs/web/ingestion/insert.mdx | 4 +- docs/web/ingestion/merge-insert.mdx | 4 +- docs/web/ingestion/realtime-write.mdx | 196 ++++++++ docs/web/ingestion/update.mdx | 6 +- docs/web/integrations/ai/huggingface.mdx | 7 +- .../integrations/data/pandas_and_pyarrow.mdx | 6 +- docs/web/integrations/data/polars_arrow.mdx | 7 +- docs/web/integrations/data/pydantic.mdx | 7 +- docs/web/integrations/lerobotdataset.mdx | 19 + docs/web/maintenance/background-jobs.mdx | 9 +- docs/web/maintenance/cleanup.mdx | 4 +- docs/web/maintenance/compaction.mdx | 8 +- docs/web/management/database.mdx | 15 +- docs/web/management/function.mdx | 327 ++++++++++--- docs/web/management/index.mdx | 2 +- docs/web/management/materialized-view.mdx | 64 ++- docs/web/management/namespace.mdx | 5 +- docs/web/management/table.mdx | 10 +- docs/web/management/view.mdx | 6 +- docs/web/quickstart.mdx | 187 ++++---- docs/web/reranking/index.mdx | 4 +- docs/web/search/explain.mdx | 4 +- docs/web/search/filtering.mdx | 4 +- docs/web/search/full-text-search.mdx | 4 +- docs/web/search/hybrid-search.mdx | 4 +- docs/web/search/index.mdx | 6 +- docs/web/search/realtime-read.mdx | 74 +++ docs/web/search/vector-search.mdx | 4 +- docs/web/snippets/quickstart.mdx | 62 +-- docs/web/sql/data-types.mdx | 5 +- docs/web/sql/ddl/database.mdx | 50 ++ docs/web/sql/ddl/function.mdx | 23 + docs/web/sql/ddl/grant.mdx | 77 ++++ docs/web/sql/ddl/index.mdx | 5 +- docs/web/sql/ddl/job.mdx | 7 +- docs/web/sql/ddl/materialized-view.mdx | 67 ++- docs/web/sql/ddl/namespace.mdx | 8 +- docs/web/sql/ddl/table.mdx | 37 +- docs/web/sql/ddl/view.mdx | 5 +- docs/web/sql/dml/delete.mdx | 5 +- docs/web/sql/dml/insert.mdx | 5 +- .../dml/{refresh.mdx => refresh-column.mdx} | 46 +- .../web/sql/dml/refresh-materialized-view.mdx | 50 ++ docs/web/sql/dml/update.mdx | 5 +- docs/web/sql/dql/aggregation.mdx | 5 +- docs/web/sql/dql/changes.mdx | 76 +++ docs/web/sql/dql/dedupe.mdx | 99 ++++ docs/web/sql/dql/duplicate-pairs.mdx | 97 ++++ docs/web/sql/dql/explain.mdx | 5 +- docs/web/sql/dql/fts.mdx | 5 +- docs/web/sql/dql/join.mdx | 31 +- docs/web/sql/dql/sampling.mdx | 5 +- docs/web/sql/dql/select.mdx | 5 +- docs/web/sql/dql/subqueries.mdx | 5 +- docs/web/sql/dql/window.mdx | 5 +- docs/web/sql/functions.mdx | 28 +- docs/web/sql/index.mdx | 7 +- docs/web/sql/operators.mdx | 5 +- docs/web/sql/prepared-statements.mdx | 12 +- docs/web/sql/scalar-functions.mdx | 58 ++- .../assets/images/enterprise/architecture.png | Bin 99871 -> 0 bytes .../images/overview/lancedb-architecture.svg | 60 --- .../assets/images/overview/lancedb-suite.svg | 92 ---- .../overview/training-data-lifecycle.svg | 49 -- .../static/assets/logo/lance-logo-gray.png | Bin 0 -> 57024 bytes .../static/assets/logo/lance-logo-gray.svg | 5 - .../static/assets/logo/light-lancedb-logo.svg | 17 +- docs/web/tables/branching.mdx | 6 +- docs/web/tables/consistency.mdx | 8 +- docs/web/tables/realtime-engine.mdx | 219 +-------- docs/web/training/data-loading.mdx | 435 +++++++----------- docs/web/training/distributed-training.mdx | 216 +++++++++ docs/web/training/object-detection.mdx | 4 +- docs/web/training/permutation.mdx | 123 +++++ docs/web/training/torch.mdx | 334 -------------- docs/web/training/why-lancedb.mdx | 119 ----- .../feature-engineering-101.mdx | 1 - .../materialized-views.mdx | 2 +- 133 files changed, 3978 insertions(+), 2918 deletions(-) delete mode 100644 docs/web/architecture.mdx delete mode 100644 docs/web/basics/connect.mdx delete mode 100644 docs/web/basics/lakehouse.mdx delete mode 100644 docs/web/basics/offerings.mdx delete mode 100644 docs/web/basics/query.mdx create mode 100644 docs/web/curation/changes.mdx create mode 100644 docs/web/curation/chunking.mdx create mode 100644 docs/web/curation/create-function.mdx create mode 100644 docs/web/curation/dedupe.mdx delete mode 100644 docs/web/enterprise/architecture.mdx delete mode 100644 docs/web/enterprise/deployment/azure.mdx delete mode 100644 docs/web/enterprise/deployment/index.mdx create mode 100644 docs/web/enterprise/deployment/kubernetes.mdx delete mode 100644 docs/web/feature-engineering/data-evolution.mdx create mode 100644 docs/web/feature-engineering/refresh-materialized-view.mdx create mode 100644 docs/web/feature-engineering/view.mdx create mode 100644 docs/web/ingestion/realtime-write.mdx create mode 100644 docs/web/search/realtime-read.mdx create mode 100644 docs/web/sql/ddl/database.mdx create mode 100644 docs/web/sql/ddl/grant.mdx rename docs/web/sql/dml/{refresh.mdx => refresh-column.mdx} (63%) create mode 100644 docs/web/sql/dml/refresh-materialized-view.mdx create mode 100644 docs/web/sql/dql/changes.mdx create mode 100644 docs/web/sql/dql/dedupe.mdx create mode 100644 docs/web/sql/dql/duplicate-pairs.mdx delete mode 100644 docs/web/static/assets/images/enterprise/architecture.png delete mode 100644 docs/web/static/assets/images/overview/lancedb-architecture.svg delete mode 100644 docs/web/static/assets/images/overview/lancedb-suite.svg delete mode 100644 docs/web/static/assets/images/overview/training-data-lifecycle.svg create mode 100644 docs/web/static/assets/logo/lance-logo-gray.png delete mode 100644 docs/web/static/assets/logo/lance-logo-gray.svg create mode 100644 docs/web/training/distributed-training.mdx create mode 100644 docs/web/training/permutation.mdx delete mode 100644 docs/web/training/torch.mdx delete mode 100644 docs/web/training/why-lancedb.mdx diff --git a/docs/web-tests/py/test_quickstart.py b/docs/web-tests/py/test_quickstart.py index 779a268d4..bdcb8abff 100644 --- a/docs/web-tests/py/test_quickstart.py +++ b/docs/web-tests/py/test_quickstart.py @@ -119,38 +119,35 @@ async def async_db(request, db_path_factory): await owned.drop_claimed() -def create_characters(db): +def create_frames(db): # --8<-- [start:quickstart_data] data = [ { - "id": "1", - "name": "King Arthur", - "role": "King", - "description": "Leader of Camelot and wielder of Excalibur.", - "stats": {"strength": 4, "magic": 1, "leadership": 5, "wisdom": 4}, - "vector": [0.7, 0.1, 0.9, 0.7], + "frame_id": "1", + "video_id": "clip-1", + "caption": "a robot arm reaching toward a cup", + "stats": {"motion": 3, "sharpness": 4, "exposure": 4, "framing": 3}, + "embedding": [0.7, 0.1, 0.9, 0.7], }, { - "id": "2", - "name": "Merlin", - "role": "Wizard", - "description": "Advisor and prophet with deep magical knowledge.", - "stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 5}, - "vector": [0.2, 0.9, 0.4, 0.9], + "frame_id": "2", + "video_id": "clip-1", + "caption": "a two-finger gripper closing on a cup", + "stats": {"motion": 5, "sharpness": 4, "exposure": 5, "framing": 2}, + "embedding": [0.2, 0.9, 0.4, 0.9], }, { - "id": "3", - "name": "Sir Lancelot", - "role": "Knight", - "description": "Legendary knight known for courage and combat skill.", - "stats": {"strength": 5, "magic": 1, "leadership": 3, "wisdom": 3}, - "vector": [0.9, 0.1, 0.5, 0.4], + "frame_id": "3", + "video_id": "clip-2", + "caption": "a robot arm holding a cup steady", + "stats": {"motion": 2, "sharpness": 3, "exposure": 3, "framing": 4}, + "embedding": [0.9, 0.1, 0.5, 0.4], }, ] # --8<-- [end:quickstart_data] # --8<-- [start:quickstart_create_table] - table = db.create_table("characters", data=data, mode="overwrite") + table = db.create_table("frames", data=data, mode="overwrite") # --8<-- [end:quickstart_create_table] assert table.count_rows() == 3 @@ -158,12 +155,11 @@ def create_characters(db): table.add( [ { - "id": "4", - "name": "Morgana", - "role": "Sorceress", - "description": "Powerful sorceress of Avalon.", - "stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 4}, - "vector": [0.3, 0.9, 0.6, 0.8], + "frame_id": "4", + "video_id": "clip-2", + "caption": "a gripper lowering a cup onto the table", + "stats": {"motion": 5, "sharpness": 4, "exposure": 4, "framing": 2}, + "embedding": [0.3, 0.9, 0.6, 0.8], } ] ) @@ -173,153 +169,152 @@ def create_characters(db): def test_quickstart(db): - table = create_characters(db) + table = create_frames(db) # --8<-- [start:quickstart_vector_search_1] - # Search for examples similar to a "wise magical advisor" + # Search for frames similar to "a gripper closing on an object" query_vector = [0.2, 0.8, 0.4, 0.9] # Ensure you run `pip install polars` beforehand result = ( table.search(query_vector) - .select(["name", "role", "description", "_distance"]) + .select(["frame_id", "video_id", "caption", "_distance"]) .limit(2) .to_polars() ) print(result) # --8<-- [end:quickstart_vector_search_1] - assert result["name"].to_list() == ["Merlin", "Morgana"] + assert result["frame_id"].to_list() == ["2", "4"] # --8<-- [start:quickstart_output_pandas] # Ensure you run `pip install pandas` beforehand result = table.search(query_vector).limit(2).to_pandas() print(result) # --8<-- [end:quickstart_output_pandas] - assert result["name"].to_list() == ["Merlin", "Morgana"] + assert result["frame_id"].to_list() == ["2", "4"] # --8<-- [start:quickstart_curate_with_metadata] curated = ( table.search(query_vector) - .where("stats.magic >= 4") - .select(["name", "role", "description", "_distance"]) + .where("stats.motion >= 4") + .select(["frame_id", "video_id", "caption", "_distance"]) .limit(2) .to_polars() ) print(curated) # --8<-- [end:quickstart_curate_with_metadata] - assert curated["name"].to_list() == ["Merlin", "Morgana"] + assert curated["frame_id"].to_list() == ["2", "4"] - curated = table.search(query_vector).where("stats.magic < 4").limit(2).to_list() - assert [row["name"] for row in curated] == ["King Arthur", "Sir Lancelot"] + curated = table.search(query_vector).where("stats.motion < 4").limit(2).to_list() + assert [row["frame_id"] for row in curated] == ["1", "3"] @pytest.mark.asyncio async def test_quickstart_async_api(async_db): data = [ { - "id": "2", - "name": "Merlin", - "role": "Wizard", - "description": "Advisor and prophet with deep magical knowledge.", - "stats": {"strength": 2, "magic": 5, "leadership": 4, "wisdom": 5}, - "vector": [0.2, 0.9, 0.4, 0.9], + "frame_id": "2", + "video_id": "clip-1", + "caption": "a two-finger gripper closing on a cup", + "stats": {"motion": 5, "sharpness": 4, "exposure": 5, "framing": 2}, + "embedding": [0.2, 0.9, 0.4, 0.9], } ] # --8<-- [start:quickstart_create_table_async] async_table = await async_db.create_table( - "characters", + "frames", data=data, mode="overwrite", ) # --8<-- [end:quickstart_create_table_async] # --8<-- [start:quickstart_vector_search_1_async] - # Search for examples similar to a "wise magical advisor" + # Search for frames similar to "a gripper closing on an object" query_vector = [0.2, 0.8, 0.4, 0.9] # Ensure you run `pip install polars` beforehand async_result = await ( await async_table.search(query_vector) - ).select(["name", "role", "description", "_distance"]).limit(2).to_polars() + ).select(["frame_id", "video_id", "caption", "_distance"]).limit(2).to_polars() print(async_result) # --8<-- [end:quickstart_vector_search_1_async] - assert async_result["name"].to_list() == ["Merlin"] + assert async_result["frame_id"].to_list() == ["2"] def test_quickstart_derived_feature(db): - table = create_characters(db) + table = create_frames(db) # --8<-- [start:quickstart_add_feature] table.add_columns( { - "power_score": "cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)" + "quality_score": "cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)" } ) # --8<-- [end:quickstart_add_feature] # --8<-- [start:quickstart_query_feature] - features = table.search().select(["name", "role", "power_score"]).to_polars() + features = table.search().select(["frame_id", "video_id", "quality_score"]).to_polars() print(features) # --8<-- [end:quickstart_query_feature] - assert sorted(zip(features["name"], features["power_score"])) == [ - ("King Arthur", 3.5), - ("Merlin", 4.0), - ("Morgana", 3.75), - ("Sir Lancelot", 3.0), + assert sorted(zip(features["frame_id"], features["quality_score"])) == [ + ("1", 3.5), + ("2", 4.0), + ("3", 3.0), + ("4", 3.75), ] -def description_lengths(table): - rows = table.search().select(["description", "description_length"]).to_list() - return {row["description"]: row["description_length"] for row in rows} +def caption_lengths(table): + rows = table.search().select(["caption", "caption_length"]).to_list() + return {row["caption"]: row["caption_length"] for row in rows} def test_quickstart_computed_column(db_path_factory): - table = create_characters(lancedb.connect(db_path_factory("computed"))) + table = create_frames(lancedb.connect(db_path_factory("computed"))) # --8<-- [start:quickstart_computed_feature] - table.add_columns(computed={"description_length": "length(description)"}) - table.refresh_column("description_length") + table.add_columns(computed={"caption_length": "length(caption)"}) + table.refresh_column("caption_length") # --8<-- [end:quickstart_computed_feature] - lengths = description_lengths(table) + lengths = caption_lengths(table) assert lengths == {text: len(text) for text in lengths} def test_quickstart_computed_column_on_a_deployment(deployment): - table = create_characters(deployment) + table = create_frames(deployment) # --8<-- [start:quickstart_computed_feature_enterprise] - table.add_columns(computed={"description_length": "length(description)"}) - job = table.refresh_column_async("description_length") + table.add_columns(computed={"caption_length": "length(caption)"}) + job = table.refresh_column_async("caption_length") job.wait() # --8<-- [end:quickstart_computed_feature_enterprise] - lengths = description_lengths(table) + lengths = caption_lengths(table) assert len(lengths) == 4 assert lengths == {text: len(text) for text in lengths} with pytest.raises(Exception): - table.refresh_column("description_length") + table.refresh_column("caption_length") def test_quickstart_multimodal(db, tmp_path, monkeypatch): app = tmp_path / "app" (app / "data").mkdir(parents=True) - shutil.copy(IMAGE, app / "data/sir-lancelot.jpg") + shutil.copy(IMAGE, app / "data/sample-frame.jpg") monkeypatch.chdir(app) # --8<-- [start:quickstart_multimodal_bytes] from pathlib import Path - image_bytes = Path("data/sir-lancelot.jpg").read_bytes() + image_bytes = Path("data/sample-frame.jpg").read_bytes() multimodal_table = db.create_table( - "character_images", + "frame_images", data=[ { - "id": "lancelot", - "description": "Portrait of Sir Lancelot", + "frame_id": "thumb-1", + "caption": "Sample frame thumbnail", "image": image_bytes, - "vector": [0.9, 0.1, 0.5, 0.4], + "embedding": [0.9, 0.1, 0.5, 0.4], } ], mode="overwrite", @@ -331,7 +326,7 @@ def test_quickstart_multimodal(db, tmp_path, monkeypatch): def test_quickstart_training(db): - table = create_characters(db) + table = create_frames(db) # --8<-- [start:quickstart_training] import torch @@ -339,15 +334,15 @@ def test_quickstart_training(db): table.add_columns( { - "magic": "cast(stats.magic as float)", - "strength": "cast(stats.strength as float)", + "motion": "cast(stats.motion as float)", + "sharpness": "cast(stats.sharpness as float)", } ) - dataset = Permutation.identity(table).select_columns(["magic", "strength"]) + dataset = Permutation.identity(table).select_columns(["motion", "sharpness"]) dataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True) # --8<-- [end:quickstart_training] batches = list(dataloader) assert len(batches) == 1 - assert set(batches[0]) == {"magic", "strength"} + assert set(batches[0]) == {"motion", "sharpness"} assert all(isinstance(value, torch.Tensor) for value in batches[0].values()) - assert sorted(batches[0]["magic"].tolist()) == [1.0, 1.0, 5.0, 5.0] + assert sorted(batches[0]["motion"].tolist()) == [2.0, 3.0, 5.0, 5.0] diff --git a/docs/web-tests/rs/quickstart.rs b/docs/web-tests/rs/quickstart.rs index f6ea7e0f9..b0df4daba 100644 --- a/docs/web-tests/rs/quickstart.rs +++ b/docs/web-tests/rs/quickstart.rs @@ -16,92 +16,88 @@ use lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema}; // Define structs representing the data schema #[derive(Debug, Clone)] struct Stats { - strength: i8, - magic: i8, - leadership: i8, - wisdom: i8, + motion: i8, + sharpness: i8, + exposure: i8, + framing: i8, } #[derive(Debug, Clone)] -struct Character { - id: String, - name: String, - role: String, - description: String, +struct Frame { + frame_id: String, + video_id: String, + caption: String, stats: Stats, - vector: [f32; 4], + embedding: [f32; 4], } -fn characters_schema() -> Arc { +fn frames_schema() -> Arc { Arc::new(Schema::new(vec![ - Field::new("id", DataType::LargeUtf8, false), - Field::new("name", DataType::LargeUtf8, false), - Field::new("role", DataType::LargeUtf8, false), - Field::new("description", DataType::LargeUtf8, false), + Field::new("frame_id", DataType::LargeUtf8, false), + Field::new("video_id", DataType::LargeUtf8, false), + Field::new("caption", DataType::LargeUtf8, false), Field::new( "stats", DataType::Struct(Fields::from(vec![ - Arc::new(Field::new("strength", DataType::Int8, false)), - Arc::new(Field::new("magic", DataType::Int8, false)), - Arc::new(Field::new("leadership", DataType::Int8, false)), - Arc::new(Field::new("wisdom", DataType::Int8, false)), + Arc::new(Field::new("motion", DataType::Int8, false)), + Arc::new(Field::new("sharpness", DataType::Int8, false)), + Arc::new(Field::new("exposure", DataType::Int8, false)), + Arc::new(Field::new("framing", DataType::Int8, false)), ])), false, ), Field::new( - "vector", + "embedding", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4), false, ), ])) } -fn characters_to_reader( +fn frames_to_reader( schema: Arc, - rows: &[Character], + rows: &[Frame], ) -> Box { - let ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.id.as_str())); - let names = LargeStringArray::from_iter_values(rows.iter().map(|row| row.name.as_str())); - let roles = LargeStringArray::from_iter_values(rows.iter().map(|row| row.role.as_str())); - let descriptions = - LargeStringArray::from_iter_values(rows.iter().map(|row| row.description.as_str())); + let frame_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.frame_id.as_str())); + let video_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.video_id.as_str())); + let captions = + LargeStringArray::from_iter_values(rows.iter().map(|row| row.caption.as_str())); - let strength = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.strength)); - let magic = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.magic)); - let leadership = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.leadership)); - let wisdom = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.wisdom)); + let motion = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.motion)); + let sharpness = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.sharpness)); + let exposure = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.exposure)); + let framing = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.framing)); let stats_fields: Vec = vec![ - Arc::new(Field::new("strength", DataType::Int8, false)), - Arc::new(Field::new("magic", DataType::Int8, false)), - Arc::new(Field::new("leadership", DataType::Int8, false)), - Arc::new(Field::new("wisdom", DataType::Int8, false)), + Arc::new(Field::new("motion", DataType::Int8, false)), + Arc::new(Field::new("sharpness", DataType::Int8, false)), + Arc::new(Field::new("exposure", DataType::Int8, false)), + Arc::new(Field::new("framing", DataType::Int8, false)), ]; let stats = StructArray::new( stats_fields.into(), vec![ - Arc::new(strength), - Arc::new(magic), - Arc::new(leadership), - Arc::new(wisdom), + Arc::new(motion), + Arc::new(sharpness), + Arc::new(exposure), + Arc::new(framing), ], None, ); - let vectors = FixedSizeListArray::from_iter_primitive::( + let embeddings = FixedSizeListArray::from_iter_primitive::( rows.iter() - .map(|row| Some(row.vector.iter().copied().map(Some).collect::>())), + .map(|row| Some(row.embedding.iter().copied().map(Some).collect::>())), 4, ); let batch = RecordBatch::try_new( schema.clone(), vec![ - Arc::new(ids), - Arc::new(names), - Arc::new(roles), - Arc::new(descriptions), + Arc::new(frame_ids), + Arc::new(video_ids), + Arc::new(captions), Arc::new(stats), - Arc::new(vectors), + Arc::new(embeddings), ], ) .unwrap(); @@ -121,44 +117,41 @@ async fn main() { // --8<-- [start:quickstart_data] let data = vec![ - Character { - id: "1".to_string(), - name: "King Arthur".to_string(), - role: "King".to_string(), - description: "Leader of Camelot and wielder of Excalibur.".to_string(), + Frame { + frame_id: "1".to_string(), + video_id: "clip-1".to_string(), + caption: "a robot arm reaching toward a cup".to_string(), stats: Stats { - strength: 4, - magic: 1, - leadership: 5, - wisdom: 4, + motion: 3, + sharpness: 4, + exposure: 4, + framing: 3, }, - vector: [0.7, 0.1, 0.9, 0.7], + embedding: [0.7, 0.1, 0.9, 0.7], }, - Character { - id: "2".to_string(), - name: "Merlin".to_string(), - role: "Wizard".to_string(), - description: "Advisor and prophet with deep magical knowledge.".to_string(), + Frame { + frame_id: "2".to_string(), + video_id: "clip-1".to_string(), + caption: "a two-finger gripper closing on a cup".to_string(), stats: Stats { - strength: 2, - magic: 5, - leadership: 4, - wisdom: 5, + motion: 5, + sharpness: 4, + exposure: 5, + framing: 2, }, - vector: [0.2, 0.9, 0.4, 0.9], + embedding: [0.2, 0.9, 0.4, 0.9], }, - Character { - id: "3".to_string(), - name: "Sir Lancelot".to_string(), - role: "Knight".to_string(), - description: "Legendary knight known for courage and combat skill.".to_string(), + Frame { + frame_id: "3".to_string(), + video_id: "clip-2".to_string(), + caption: "a robot arm holding a cup steady".to_string(), stats: Stats { - strength: 5, - magic: 1, - leadership: 3, - wisdom: 3, + motion: 2, + sharpness: 3, + exposure: 3, + framing: 4, }, - vector: [0.9, 0.1, 0.5, 0.4], + embedding: [0.9, 0.1, 0.5, 0.4], }, ]; // --8<-- [end:quickstart_data] @@ -166,9 +159,9 @@ async fn main() { // --8<-- [start:quickstart_create_table] use lancedb::database::CreateTableMode; - let schema = characters_schema(); + let schema = frames_schema(); let table = db - .create_table("characters", characters_to_reader(schema.clone(), &data)) + .create_table("frames", frames_to_reader(schema.clone(), &data)) .mode(CreateTableMode::Overwrite) .execute() .await @@ -177,21 +170,20 @@ async fn main() { assert_eq!(table.count_rows(None).await.unwrap(), 3); // --8<-- [start:quickstart_add_data] - let more_data = vec![Character { - id: "4".to_string(), - name: "Morgana".to_string(), - role: "Sorceress".to_string(), - description: "Powerful sorceress of Avalon.".to_string(), + let more_data = vec![Frame { + frame_id: "4".to_string(), + video_id: "clip-2".to_string(), + caption: "a gripper lowering a cup onto the table".to_string(), stats: Stats { - strength: 2, - magic: 5, - leadership: 4, - wisdom: 4, + motion: 5, + sharpness: 4, + exposure: 4, + framing: 2, }, - vector: [0.3, 0.9, 0.6, 0.8], + embedding: [0.3, 0.9, 0.6, 0.8], }]; table - .add(characters_to_reader(schema.clone(), &more_data)) + .add(frames_to_reader(schema.clone(), &more_data)) .execute() .await .unwrap(); @@ -202,7 +194,7 @@ async fn main() { use lancedb::arrow::IntoPolars; use lancedb::query::{ExecutableQuery, QueryBase, Select}; - // Search for examples similar to a "wise magical advisor" + // Search for frames similar to "a gripper closing on an object" let query_vector = [0.2, 0.8, 0.4, 0.9]; let result = table @@ -210,9 +202,9 @@ async fn main() { .nearest_to(&query_vector) .unwrap() .select(Select::Columns(vec![ - "name".to_string(), - "role".to_string(), - "description".to_string(), + "frame_id".to_string(), + "video_id".to_string(), + "caption".to_string(), "_distance".to_string(), ])) .limit(2) @@ -224,20 +216,20 @@ async fn main() { .unwrap(); println!("{result:?}"); // --8<-- [end:quickstart_vector_search_1] - let name_col = result.column("name").unwrap().str().unwrap(); - assert_eq!(name_col.get(0).unwrap(), "Merlin"); - assert_eq!(name_col.get(1).unwrap(), "Morgana"); + let frame_id_col = result.column("frame_id").unwrap().str().unwrap(); + assert_eq!(frame_id_col.get(0).unwrap(), "2"); + assert_eq!(frame_id_col.get(1).unwrap(), "4"); // --8<-- [start:quickstart_curate_with_metadata] let curated = table .query() .nearest_to(&query_vector) .unwrap() - .only_if("stats.magic >= 4") + .only_if("stats.motion >= 4") .select(Select::Columns(vec![ - "name".to_string(), - "role".to_string(), - "description".to_string(), + "frame_id".to_string(), + "video_id".to_string(), + "caption".to_string(), "_distance".to_string(), ])) .limit(2) @@ -249,9 +241,9 @@ async fn main() { .unwrap(); println!("{curated:?}"); // --8<-- [end:quickstart_curate_with_metadata] - let curated_name_col = curated.column("name").unwrap().str().unwrap(); - assert_eq!(curated_name_col.get(0).unwrap(), "Merlin"); - assert_eq!(curated_name_col.get(1).unwrap(), "Morgana"); + let curated_frame_id_col = curated.column("frame_id").unwrap().str().unwrap(); + assert_eq!(curated_frame_id_col.get(0).unwrap(), "2"); + assert_eq!(curated_frame_id_col.get(1).unwrap(), "4"); // --8<-- [start:quickstart_output_array] let result = table @@ -259,9 +251,9 @@ async fn main() { .nearest_to(&query_vector) .unwrap() .select(Select::Columns(vec![ - "name".to_string(), - "role".to_string(), - "description".to_string(), + "frame_id".to_string(), + "video_id".to_string(), + "caption".to_string(), "_distance".to_string(), ])) .limit(2) @@ -273,15 +265,15 @@ async fn main() { .unwrap(); println!("{result:?}"); // --8<-- [end:quickstart_output_array] - let name_col = result.column("name").unwrap().str().unwrap(); - assert_eq!(name_col.get(0).unwrap(), "Merlin"); + let frame_id_col = result.column("frame_id").unwrap().str().unwrap(); + assert_eq!(frame_id_col.get(0).unwrap(), "2"); // --8<-- [start:quickstart_add_feature] table .add_columns( NewColumnTransform::SqlExpressions(vec![( - "power_score".to_string(), - "cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)" + "quality_score".to_string(), + "cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)" .to_string(), )]), None, @@ -294,9 +286,9 @@ async fn main() { let features = table .query() .select(Select::Columns(vec![ - "name".to_string(), - "role".to_string(), - "power_score".to_string(), + "frame_id".to_string(), + "video_id".to_string(), + "quality_score".to_string(), ])) .execute() .await @@ -306,7 +298,7 @@ async fn main() { .unwrap(); println!("{features:?}"); // --8<-- [end:quickstart_query_feature] - assert_eq!(features.column("power_score").unwrap().len(), 4); + assert_eq!(features.column("quality_score").unwrap().len(), 4); // What the quickstart's download step leaves in an application's directory. let app = tempfile::tempdir().unwrap(); @@ -314,7 +306,7 @@ async fn main() { std::fs::copy( std::path::Path::new(env!("CARGO_MANIFEST_DIR")) .join("../../web/static/assets/images/quickstart/sir-lancelot.jpg"), - app.path().join("data/sir-lancelot.jpg"), + app.path().join("data/sample-frame.jpg"), ) .unwrap(); std::env::set_current_dir(app.path()).unwrap(); @@ -322,30 +314,30 @@ async fn main() { // --8<-- [start:quickstart_multimodal_bytes] use lancedb::arrow::arrow_array::BinaryArray; - let image_bytes = std::fs::read("data/sir-lancelot.jpg").unwrap(); + let image_bytes = std::fs::read("data/sample-frame.jpg").unwrap(); let image_schema = Arc::new(Schema::new(vec![ - Field::new("id", DataType::LargeUtf8, false), - Field::new("description", DataType::LargeUtf8, false), + Field::new("frame_id", DataType::LargeUtf8, false), + Field::new("caption", DataType::LargeUtf8, false), Field::new("image", DataType::Binary, false), Field::new( - "vector", + "embedding", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4), false, ), ])); - let image_vectors = [[0.9_f32, 0.1, 0.5, 0.4]]; + let image_embeddings = [[0.9_f32, 0.1, 0.5, 0.4]]; let image_batch = RecordBatch::try_new( image_schema.clone(), vec![ - Arc::new(LargeStringArray::from_iter_values(["lancelot"])), + Arc::new(LargeStringArray::from_iter_values(["thumb-1"])), Arc::new(LargeStringArray::from_iter_values([ - "Portrait of Sir Lancelot", + "Sample frame thumbnail", ])), Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])), Arc::new( FixedSizeListArray::from_iter_primitive::( - image_vectors + image_embeddings .iter() .map(|vector| Some(vector.iter().copied().map(Some).collect::>())), 4, @@ -359,7 +351,7 @@ async fn main() { image_schema, )); let multimodal_table = db - .create_table("character_images", image_reader) + .create_table("frame_images", image_reader) .mode(CreateTableMode::Overwrite) .execute() .await diff --git a/docs/web-tests/ts/quickstart.test.ts b/docs/web-tests/ts/quickstart.test.ts index 2c3ee8252..6796e0613 100644 --- a/docs/web-tests/ts/quickstart.test.ts +++ b/docs/web-tests/ts/quickstart.test.ts @@ -90,50 +90,46 @@ async function withDatabase( }); } -async function createCharacters(db: lancedb.Connection) { +async function createFrames(db: lancedb.Connection) { // --8<-- [start:quickstart_data] const data = [ { - id: "1", - name: "King Arthur", - role: "King", - description: "Leader of Camelot and wielder of Excalibur.", - stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 }, - vector: [0.7, 0.1, 0.9, 0.7], + frame_id: "1", + video_id: "clip-1", + caption: "a robot arm reaching toward a cup", + stats: { motion: 3, sharpness: 4, exposure: 4, framing: 3 }, + embedding: [0.7, 0.1, 0.9, 0.7], }, { - id: "2", - name: "Merlin", - role: "Wizard", - description: "Advisor and prophet with deep magical knowledge.", - stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 }, - vector: [0.2, 0.9, 0.4, 0.9], + frame_id: "2", + video_id: "clip-1", + caption: "a two-finger gripper closing on a cup", + stats: { motion: 5, sharpness: 4, exposure: 5, framing: 2 }, + embedding: [0.2, 0.9, 0.4, 0.9], }, { - id: "3", - name: "Sir Lancelot", - role: "Knight", - description: "Legendary knight known for courage and combat skill.", - stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 }, - vector: [0.9, 0.1, 0.5, 0.4], + frame_id: "3", + video_id: "clip-2", + caption: "a robot arm holding a cup steady", + stats: { motion: 2, sharpness: 3, exposure: 3, framing: 4 }, + embedding: [0.9, 0.1, 0.5, 0.4], }, ]; // --8<-- [end:quickstart_data] // --8<-- [start:quickstart_create_table] - const table = await db.createTable("characters", data, { mode: "overwrite" }); + const table = await db.createTable("frames", data, { mode: "overwrite" }); // --8<-- [end:quickstart_create_table] expect(await table.countRows()).toBe(3); // --8<-- [start:quickstart_add_data] await table.add([ { - id: "4", - name: "Morgana", - role: "Sorceress", - description: "Powerful sorceress of Avalon.", - stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 }, - vector: [0.3, 0.9, 0.6, 0.8], + frame_id: "4", + video_id: "clip-2", + caption: "a gripper lowering a cup onto the table", + stats: { motion: 5, sharpness: 4, exposure: 4, framing: 2 }, + embedding: [0.3, 0.9, 0.6, 0.8], }, ]); // --8<-- [end:quickstart_add_data] @@ -144,44 +140,44 @@ async function createCharacters(db: lancedb.Connection) { describe.each(connections)("quickstart (%s)", (connection) => { test("create, add and search", async () => { await withDatabase(connection, async (db) => { - const table = await createCharacters(db); + const table = await createFrames(db); // --8<-- [start:quickstart_vector_search_1] - // Search for examples similar to a "wise magical advisor" + // Search for frames similar to "a gripper closing on an object" const queryVector = [0.2, 0.8, 0.4, 0.9]; const result = await table .search(queryVector) - .select(["name", "role", "description", "_distance"]) + .select(["frame_id", "video_id", "caption", "_distance"]) .limit(2) .toArray(); console.table(result); // --8<-- [end:quickstart_vector_search_1] - expect(result.map((row) => row.name)).toEqual(["Merlin", "Morgana"]); + expect(result.map((row) => row.frame_id)).toEqual(["2", "4"]); // --8<-- [start:quickstart_curate_with_metadata] const curated = await table .search(queryVector) - .where("stats.magic >= 4") - .select(["name", "role", "description", "_distance"]) + .where("stats.motion >= 4") + .select(["frame_id", "video_id", "caption", "_distance"]) .limit(2) .toArray(); console.table(curated); // --8<-- [end:quickstart_curate_with_metadata] - expect(curated.map((row) => row.name)).toEqual(["Merlin", "Morgana"]); + expect(curated.map((row) => row.frame_id)).toEqual(["2", "4"]); }); }); test("a derived feature", async () => { await withDatabase(connection, async (db) => { - const table = await createCharacters(db); + const table = await createFrames(db); // --8<-- [start:quickstart_add_feature] await table.addColumns([ { - name: "power_score", + name: "quality_score", valueSql: - "cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)", + "cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)", }, ]); // --8<-- [end:quickstart_add_feature] @@ -189,17 +185,17 @@ describe.each(connections)("quickstart (%s)", (connection) => { // --8<-- [start:quickstart_query_feature] const features = await table .query() - .select(["name", "role", "power_score"]) + .select(["frame_id", "video_id", "quality_score"]) .toArray(); console.table(features); // --8<-- [end:quickstart_query_feature] expect( - features.map((row) => [row.name, row.power_score]).sort(), + features.map((row) => [row.frame_id, row.quality_score]).sort(), ).toEqual([ - ["King Arthur", 3.5], - ["Merlin", 4], - ["Morgana", 3.75], - ["Sir Lancelot", 3], + ["1", 3.5], + ["2", 4], + ["3", 3], + ["4", 3.75], ]); }); }); @@ -208,7 +204,7 @@ describe.each(connections)("quickstart (%s)", (connection) => { await withDatabase(connection, async (db) => { await withTempDirectory(async (app) => { fs.mkdirSync(path.join(app, "data")); - fs.copyFileSync(IMAGE, path.join(app, "data/sir-lancelot.jpg")); + fs.copyFileSync(IMAGE, path.join(app, "data/sample-frame.jpg")); const cwd = process.cwd(); process.chdir(app); try { @@ -216,13 +212,13 @@ describe.each(connections)("quickstart (%s)", (connection) => { const arrow = await import("apache-arrow"); const { readFile } = await import("node:fs/promises"); - const imageBytes = await readFile("data/sir-lancelot.jpg"); + const imageBytes = await readFile("data/sample-frame.jpg"); const imageSchema = new arrow.Schema([ - new arrow.Field("id", new arrow.Utf8()), - new arrow.Field("description", new arrow.Utf8()), + new arrow.Field("frame_id", new arrow.Utf8()), + new arrow.Field("caption", new arrow.Utf8()), new arrow.Field("image", new arrow.Binary()), new arrow.Field( - "vector", + "embedding", new arrow.FixedSizeList( 4, new arrow.Field("item", new arrow.Float32(), true), @@ -232,16 +228,16 @@ describe.each(connections)("quickstart (%s)", (connection) => { const imageData = lancedb.makeArrowTable( [ { - id: "lancelot", - description: "Portrait of Sir Lancelot", + frame_id: "thumb-1", + caption: "Sample frame thumbnail", image: imageBytes, - vector: [0.9, 0.1, 0.5, 0.4], + embedding: [0.9, 0.1, 0.5, 0.4], }, ], { schema: imageSchema }, ); const multimodalTable = await db.createTable( - "character_images", + "frame_images", imageData, { mode: "overwrite" }, ); diff --git a/docs/web/.mintignore b/docs/web/.mintignore index b8da6c17a..79cce2724 100644 --- a/docs/web/.mintignore +++ b/docs/web/.mintignore @@ -1,6 +1,22 @@ # Functions are documented before their release. At release, remove these -# entries, add the pages back to the navigation in docs.json, restore the -# /feature-engineering/functions redirect to /management/function, and publish -# the drafts marked "Draft for the Functions release". +# entries, restore the /feature-engineering/functions redirect to +# /management/function, publish the drafts marked "Draft for the Functions +# release", restore `availability` frontmatter (removed while gated, since +# the comparison table on index.mdx links every page that declares one, +# draft or not) to management/function.mdx and sql/ddl/function.mdx -- each +# currently carries a plain `Enterprise` instead +# -- and re-link curation/chunking.mdx's plain-text `CREATE FUNCTION`/ +# `add_columns` mentions to /management/function and /sql/ddl/function#create +# now that those pages have a route. Chunking stays published ahead of +# Functions shipping, so its examples intentionally do not link to either +# gated page -- only an unrouted page would break. management/function.mdx sql/ddl/function.mdx + +# GRANT/REVOKE currently only covers privileges on Secrets and Functions +# (CREATE SECRET, CREATE FUNCTION, USAGE on a secret or function), which are +# themselves part of the Functions release: see the "Draft for the Functions +# release" block in governance/authorization.mdx#privileges. At release, +# remove this entry, add the page to the navigation in docs.json, and give it +# `availability` frontmatter like the rest of the SQL reference. +sql/ddl/grant.mdx diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx deleted file mode 100644 index be6cc84af..000000000 --- a/docs/web/architecture.mdx +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: "Architecture" -sidebarTitle: "Architecture" -description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with LanceDB Embedded." -icon: cubes -keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"] ---- - -LanceDB Embedded is an open-source library for working with Lance tables. It runs -in your application's process; your data can live on local disk or in object -storage. LanceDB Enterprise is a multimodal lakehouse for data curation, feature -engineering, search, and training data access. LanceDB Enterprise runs LanceDB -Distributed: query execution, indexing and maintenance run on the Enterprise -service, outside your application. Its architecture separates query serving, -read execution, caching, and background work. Both use the open -[Lance](/basics/lance) format. - -Teams can curate datasets, derive features, search multimodal records, and -read the resulting tables into training frameworks. Enterprise separates the -resources that serve shared data from those that build indexes and run -background work, so teams can size them for these different workloads. - -![LanceDB Embedded runs the application and embedded library in one process, using local disk or object storage. Enterprise applications connect to query serving, with separate read execution and caching, indexing and background workers, and durable object storage. Both use the Lance format, but they do not share a storage location automatically.](/static/assets/images/overview/lancedb-architecture.svg) - -The diagram shows the main compute and storage boundaries, not every request -path. The control plane governs the Enterprise service. - -## A shared format, different storage choices {#storage} - -Lance stores table data, versions, and indexes. LanceDB Embedded can read and write tables on -local disk or in object storage. Enterprise keeps durable table data and index -artifacts in object storage, independently of the machines serving requests. -Its caches accelerate reads; they are not the durable source of truth. - -Sharing a format does not mean that an Enterprise deployment automatically -opens an existing LanceDB Embedded database. Plan how to move or connect your data and -configure access for your deployment. - -## LanceDB Embedded: a library inside your application {#embedded} - -With an embedded connection, the LanceDB library plans and executes queries, -builds indexes, and updates tables inside your process. There is no separate -database server to deploy. - -This fits local retrieval applications, notebooks, development, and -single-node data pipelines. Your application controls the compute resources and -when to run [table maintenance](/maintenance/compaction). You also manage access -to the local files or object storage that hold your tables. - -## LanceDB Enterprise: distributed execution for shared workloads {#distributed} - -A LanceDB SDK connects to LanceDB Enterprise, and the service runs operations -with LanceDB Distributed. The service handles -requests across a fleet, with components sized for different kinds of work: - -- **Query serving** receives requests, resolves tables, and plans operations. - Add serving capacity as more applications and users depend on your data. -- **Read execution and caching** use separately provisioned resources. Executors - cache table data to reduce repeated reads from object storage. -- **Indexing and background workers** build indexes and perform maintenance - outside the query request path. Size this work separately from query serving - as your data grows. - -The control plane manages configuration, identity, policy, and cluster -lifecycle. It governs the service rather than executing table queries. -[Authorization](/governance/authorization) lets teams control access to catalog -objects when enabled and configured by the administrator. Deployment settings govern background -[compaction](/maintenance/compaction) and -[cleanup](/maintenance/cleanup), including retention. - -These boundaries support different parts of an AI data workflow. -[SQL curation](/curation) requires a deployment with SQL enabled. -[Feature engineering](/feature-engineering) can produce derived columns and -tables; computed columns and materialized views require a deployment that -enables them. [Geneva](/geneva) runs distributed Python feature backfills while -its capabilities continue to transition to LanceDB APIs. -[Training data loaders](/training/data-loading) read tables into model training -frameworks in both offerings. LanceDB provides data access for training; the -training framework runs the model. - -Size and monitor each tier for your workload. Read the -[Enterprise architecture](/enterprise/architecture) for the component details -and request flows. - -## Application changes for Enterprise {#differences} - -Core operations remain familiar: create a table, add data, and run a search. -With LanceDB Embedded, the application executes those operations. With Enterprise, the SDK -sends requests to the service, and the service manages execution and configured -background maintenance. - -To use Enterprise, configure the cluster endpoint, database URI and credentials, -and make your data available in the Enterprise database. Review SDK methods -that behave differently with LanceDB Distributed, including result retrieval and -background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise) -describes these application changes. If you plan to use materialized views, check the -[stable row ID requirements](/basics/limits#schema) before creating source tables. - -## Choose for the workload you need to run {#choosing} - -Choose LanceDB Embedded when you want an embedded library and can manage its resources and -maintenance in your application. - -Choose Enterprise when multiple applications or teams need a shared multimodal -lakehouse for curation, feature engineering, search, and training data access. -It also fits workloads whose query and indexing demand grow at different rates, -or that need configured service-level access controls and background operations. -You can begin with Enterprise. - -[Talk to us about your workload](https://www.lancedb.com/contact) to plan capacity, -deployment options, and the path from your existing data to Enterprise. - -## Where to go next {#next} - - - - Query serving, execution, and background work in detail. - - - Connect an application to your deployment. - - - What a write promises, and what a reader sees. - - - The open format underneath both offerings. - - diff --git a/docs/web/basics/agentic-skills.mdx b/docs/web/basics/agentic-skills.mdx index 96222e27a..b87574dff 100644 --- a/docs/web/basics/agentic-skills.mdx +++ b/docs/web/basics/agentic-skills.mdx @@ -273,7 +273,7 @@ private deployment, or platform-managed operations. The data format and the core table API stay the same, so the pipeline's table and search code carries over: the agent connects to an Enterprise `db://` table, leaves maintenance to the deployment instead of calling `optimize()`, and submits long-running work as -jobs. See [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise). +[jobs](/basics/jobs). ## What agents are good at here {#workflows} diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index c6950b9de..976aac300 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -1,16 +1,128 @@ --- -title: "Lance Catalog" -sidebarTitle: "Lance Catalog" -description: "Organize, find and control access to your data in LanceDB." +title: "LanceDB Catalog" +sidebarTitle: "LanceDB Catalog" +description: "LanceDB Lakehouse objects are organized into a catalog of databases, namespaces and tables -- connect to a database to work with them." icon: sitemap -keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"] +keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view", "connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"] --- +import { + PyConnect, + PyConnectAsync, + PyConnectEnterprisePlaceholders, + PyConnectObjectStorage, + PyConnectObjectStorageAsync, + RsConnect, + TsConnect, + TsConnectEnterprisePlaceholders, + TsConnectObjectStorage, +} from '/snippets/connection.mdx'; + ## Catalog overview {#catalog} A catalog keeps track of named data objects, such as tables and views, so applications can discover and access them. In LanceDB, it organizes those -objects into databases and namespaces. +objects into databases and namespaces -- similar to a catalog in Postgres, +Snowflake, or other database systems. + +## Connecting {#connecting} + +You always run table and query operations within a **database**; cross-database +operations go through SQL instead. The same Python, TypeScript and Rust SDKs +reach a database two ways: + +- **Directly, the common case.** `lancedb.connect(uri)` opens one database in a + single call -- a local directory, an object-storage path, or (with a `db://` + URI) a specific Enterprise database. +- **Through the catalog, for admin and cross-database work.** `connect_catalog(endpoint)` + connects to Enterprise's root catalog, which can list, create and drop + databases across the whole deployment. Its `connect_database(name)` and + `create_database(name)` then return the same kind of database connection + `lancedb.connect()` does. Local filesystem and object-store deployments have + no catalog to connect to -- each is already scoped to one database. + +The table below shows which URI reaches which kind of database directly. + +| URI | Where queries run | +| --- | --- | +| A local path, such as `./data` | Embedded: in your process | +| `s3://`, `gs://` or `az://` | Embedded: in your process, reading object storage | +| `db://` | LanceDB Enterprise: queries run on your cluster | + +### A local directory {#local} + +Embedded in your application, LanceDB reads and writes the tables in a +directory on disk. + + + + {PyConnect} + + + + {PyConnectAsync} + + + + { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } + {TsConnect} + + + + { "use lancedb::connect;\n" } + { "\n" } + {RsConnect} + + + +### Object storage {#object-storage} + +With an object storage URI the library is still embedded: your process runs +every query and reads and writes the bucket directly, with the credentials from +its environment or its storage options. [Configuring storage](/deployment/storage) +covers credentials, endpoints and provider options. + + + + {PyConnectObjectStorage} + + + + {PyConnectObjectStorageAsync} + + + + { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } + {TsConnectObjectStorage} + + + +In Rust, pass the same URI to `connect`. + +### LanceDB Enterprise {#enterprise} + +A `db://` URI connects directly to one database in LanceDB Enterprise, where +your cluster runs queries and background work. Get the database URI, API key, +region and endpoint from your administrator, then replace the placeholders +below. Use your cluster's region in place of `us-east-1`. In application code, +load credentials from environment variables or a secrets manager. + + + + {PyConnectEnterprisePlaceholders} + + + + { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } + {TsConnectEnterprisePlaceholders} + + + +In Rust, enable the crate's `remote` feature; the connection builder then takes +the same settings through `api_key`, `region` and `host_override`. To use OAuth +instead of an API key, see +[Authentication](/enterprise/authentication); for every connection option, see +[Database](/management/database#deployment). ## Databases, namespaces and tables {#objects} @@ -37,7 +149,7 @@ flowchart TD | [Namespace](/management/namespace) | A named group of tables, views or nested namespaces. | | [Table](/management/table) | Data stored as rows and columns. | | [View](/management/view) | A saved query, evaluated when read. Available in Enterprise. | -| [Materialized view](/management/materialized-view) | The stored results of a query, updated by a refresh. | +| [Materialized view](/management/materialized-view) | The stored results of a query, updated by a refresh. Available in Enterprise. | {/* Draft for the Functions release. Add this row to the table above: diff --git a/docs/web/basics/connect.mdx b/docs/web/basics/connect.mdx deleted file mode 100644 index d576b62d8..000000000 --- a/docs/web/basics/connect.mdx +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: "Connect to LanceDB" -sidebarTitle: "Connect" -description: "Use LanceDB Embedded with tables in a local directory or object storage, or connect to LanceDB Enterprise." -icon: plug -keywords: ["connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"] ---- - -import { - PyConnect, - PyConnectAsync, - PyConnectEnterprisePlaceholders, - PyConnectEnterpriseQuickstart, - PyConnectObjectStorage, - PyConnectObjectStorageAsync, - RsConnect, - TsConnect, - TsConnectEnterprisePlaceholders, - TsConnectEnterpriseQuickstart, - TsConnectObjectStorage, -} from '/snippets/connection.mdx'; - -The URI you pass to `connect` decides where the work runs. LanceDB Embedded -runs in your application's process; your data can live on local disk or in -object storage. LanceDB Enterprise runs LanceDB Distributed: query execution, -indexing and maintenance run on the Enterprise service, outside your -application. The same Python, TypeScript and Rust SDKs open all three kinds; -[Embedded and Enterprise](/basics/offerings) describes how the two offerings differ. - -| URI | Where queries run | -| --- | --- | -| A local path, such as `./data` | LanceDB Embedded: in your process | -| `s3://`, `gs://` or `az://` | LanceDB Embedded: in your process, reading object storage | -| `db://` | LanceDB Enterprise: LanceDB Distributed runs queries on your cluster | - -## A local directory {#local} - -Embedded in your application, LanceDB reads and writes the tables in a -directory on disk. - - - - {PyConnect} - - - - {PyConnectAsync} - - - - { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } - {TsConnect} - - - - { "use lancedb::connect;\n" } - { "\n" } - {RsConnect} - - - -## Object storage {#object-storage} - -With an object storage URI the library is still embedded: your process runs -every query and reads and writes the bucket directly, with the credentials from -its environment or its storage options. [Configuring storage](/deployment/storage) -covers credentials, endpoints and provider options. - - - - {PyConnectObjectStorage} - - - - {PyConnectObjectStorageAsync} - - - - { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } - {TsConnectObjectStorage} - - - -In Rust, pass the same URI to `connect`. - -## LanceDB Enterprise {#enterprise} - -A `db://` URI connects to LanceDB Enterprise, where your cluster runs queries -and background work. Get the database URI, API key, region and endpoint from -your administrator, then replace the placeholders below. Use your cluster's region -in place of `us-east-1`. In application code, load credentials from environment -variables or a secrets manager. - - - - {PyConnectEnterprisePlaceholders} - - - - { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } - {TsConnectEnterprisePlaceholders} - - - -In Rust, enable the crate's `remote` feature; the connection builder then takes -the same settings through `api_key`, `region` and `host_override`. To use OAuth -instead of an API key, see -[Authentication](/enterprise/authentication); for every connection option, see -[Database](/management/database#deployment). - - - - -Set these values in the shell that starts your application. Replace the URI, API -key, region and endpoint with your Enterprise connection settings. In production, a secrets -manager can supply these environment variables for you. - -```shell -export LANCEDB_URI="db://your-database" -export LANCEDB_API_KEY="your-api-key" -export LANCEDB_REGION="us-east-1" -export LANCEDB_HOST_OVERRIDE="https://your-endpoint" -``` - - - - {PyConnectEnterpriseQuickstart} - - - { "import * as lancedb from \"@lancedb/lancedb\";\n\n" } - {TsConnectEnterpriseQuickstart} - - - - diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx index 2774d749b..42392c5ba 100644 --- a/docs/web/basics/guarantees.mdx +++ b/docs/web/basics/guarantees.mdx @@ -87,5 +87,5 @@ cannot undo `drop_table`. - [Table versioning](/management/table#versioning): inspect and restore history. - [Cleanup](/maintenance/cleanup): configure retention and understand protected versions. - [Jobs](/basics/jobs): distinguish submission, completion and cancellation. -- [Column refresh](/feature-engineering/refresh-column): understand partial progress. -- [Architecture](/architecture): understand storage and background maintenance. +- [Refresh a computed column](/feature-engineering/refresh-column): understand partial progress. +- [Architecture](/#architecture): understand storage and background maintenance. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index 20445843e..a15f10fee 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -5,11 +5,8 @@ description: "Track background work from submission to completion." icon: hourglass-half keywords: ["job", "background", "progress", "events", "cancel"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Job handles track operations in LanceDB Embedded and LanceDB Enterprise. Enterprise also - supports listing server jobs, reopening them by ID and reading their events. --- ## Job handles {#convention} @@ -26,6 +23,14 @@ so check its status before waiting or requesting cancellation. See the operation's API reference for when the method returns. +```python +job = tbl.create_index_async("vector", replace=True) + +print(job.id) +print(job.job_type) +print(job.status()) +``` + ## Submit and wait for a job {#waiting} 1. **Submit the operation.** Keep the returned job handle. For an Enterprise @@ -39,6 +44,22 @@ In Python, `job.status()` reports the current state and `job.wait()` waits for completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting. The timeout does not cancel the server job. +```python +from datetime import timedelta +from lancedb.exceptions import JobFailedError, JobCancelledError + +job = tbl.create_index_async("vector", replace=True) + +try: + result = job.wait(timeout=timedelta(minutes=10)) +except JobFailedError: + print(f"job failed: {job.failure}") +except JobCancelledError: + print("job was cancelled") +except TimeoutError: + print(f"still running: {job.status()}") +``` + @@ -54,6 +75,24 @@ The timeout does not cancel the server job. Reopening a job retrieves the existing operation; it does not restart failed work. Treat the ID as an opaque identifier and pass it unchanged to the jobs API. +```python +job = db.open_job(job_id) # raises JobNotFoundError if unknown +job.refresh() # pull the latest status from the server +print(job.status()) + +events = job.events(limit=100) +print(events.to_pylist()) + +job.cancel() +``` + +List jobs on a connection, paging through the results: + +```python +for info in db.list_jobs(page_limit=50): + print(info.job_id, info.job_type, info.state) +``` + Event responses have [row limits](/basics/limits#jobs). An explicit limit can request more than the default, but it does not guarantee a complete event history. @@ -97,7 +136,7 @@ workers. That placement does not change the handle's query-backed identity. ## Learn more {#next} - [Build and repair indexes](/indexing/reindexing): submit index work. -- [Column refresh](/feature-engineering/refresh-column): populate derived values. +- [Refresh a computed column](/feature-engineering/refresh-column): populate derived values. - [Background jobs](/maintenance/background-jobs): understand Enterprise maintenance. - [SQL job commands](/sql/ddl/job): list jobs and inspect errors. - [SDKs](/basics/sdk): find the job API for your client. diff --git a/docs/web/basics/lakehouse.mdx b/docs/web/basics/lakehouse.mdx deleted file mode 100644 index 9ecf6800a..000000000 --- a/docs/web/basics/lakehouse.mdx +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: "LanceDB Lakehouse" -sidebarTitle: "LanceDB Lakehouse" -description: "Bring multimodal data, search and feature engineering together for AI workloads." -icon: layer-group -keywords: ["lakehouse", "multimodal", "search", "compute", "training", "feature engineering"] ---- - -## Multimodal storage and processing {#multimodal} - -LanceDB brings storage, search and data processing together for AI workloads. -You can store multimodal data, derive features, search your records and prepare -training datasets using tables built on the open [Lance format](/basics/lance). - -The format provides the storage foundation. The [catalog](/basics/catalog) -organizes your tables. LanceDB provides the tools to work with that data. - -## Example: a video search workflow {#capabilities} - -Consider a team building a video search application: - - - - - -1. **Ingest the clips.** Store video clips with metadata such as timestamps, - source and labels. -2. **Add features.** Generate captions and embeddings with your models, and - store them alongside each clip. -3. **Use the dataset.** Search for relevant clips using embeddings and metadata - filters, or select a dataset to read into a training framework. - -```mermaid -flowchart TD - clips["Video clips and metadata"] --> table["Store in a Lance table"] - table --> features["Add captions and embeddings"] - features --> search["Search for relevant clips"] - features --> training["Select data for training"] -``` - -The clips, metadata and derived features stay associated in the same table, -so retrieval and training workflows can use the same dataset. Your training -framework runs the model; LanceDB provides access to the training data. - -## Shared compute and maintenance in Enterprise {#where} - -LanceDB Embedded runs as an embedded library in your application. LanceDB Enterprise -provides shared infrastructure for applications and teams working with larger -multimodal workloads. - - - -- **Scale resources for different workloads.** Provision query serving and - background workers independently as search traffic and indexing demand grow. -- **Share data across teams.** Give applications access to shared tables, with - Enterprise permissions controlling who can read or change them. -- **Manage ongoing maintenance.** Configure background indexing, compaction - and cleanup as your datasets grow. - -See [Architecture](/architecture) for how Enterprise separates storage, query -serving and background work. - -## Learn more {#next} - -- [Data ingestion](/ingestion): load and update your data. -- [Search](/search): retrieve records with vector, full-text and hybrid search. -- [Feature engineering](/feature-engineering): derive columns and datasets. -- [Training data access](/training/data-loading): read datasets into training workflows. -- [Enterprise](/enterprise): explore deployment options for shared workloads. - -{/* -Draft for the Functions release. Add this item to the list above, after -Feature engineering: -- [Functions](/management/function): run Python to compute column values. -*/} diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 13b7755be..883992afd 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -2,7 +2,7 @@ title: "Lance format" sidebarTitle: "Lance Format" description: "The open format for multimodal data behind LanceDB." -icon: "/static/assets/logo/lance-logo-gray.svg" +icon: "/static/assets/logo/lance-logo-gray.png" keywords: ["lance", "format", "multimodal", "versioning", "column evolution"] --- @@ -42,4 +42,4 @@ those fields together. - [Lance format specification](https://lance.org/format): how the format works. - [Table versioning](/management/table#versioning): read and manage table versions. - [Computed columns](/feature-engineering/add-computed-column): add derived features to your data. -- [Architecture](/architecture): how LanceDB Embedded and LanceDB Enterprise build on Lance. +- [Architecture](/#architecture): how LanceDB Embedded and Enterprise build on Lance. diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index 0d4f1b028..c4af28c45 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -38,9 +38,9 @@ See [SQL job commands](/sql/ddl/job#errors) and Data-type and index constraints belong to the operation you choose. Check [Data types](/sql/data-types) and [Vector indexes](/indexing/vector-index) -before choosing a schema or index. For materialized views, check -[stable row ID requirements](/feature-engineering/materialized-views#stable-row-ids) -before creating source tables. +before choosing a schema or index. For materialized views, confirm your +source table meets your deployment's requirements with your administrator +before creating one over it. ## Configurable limits {#not-quotas} diff --git a/docs/web/basics/offerings.mdx b/docs/web/basics/offerings.mdx deleted file mode 100644 index a2df61b21..000000000 --- a/docs/web/basics/offerings.mdx +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: "LanceDB Embedded or Enterprise?" -sidebarTitle: "Embedded and Enterprise" -description: "Start with an embedded library, or use the LanceDB Enterprise multimodal lakehouse for shared AI workloads." -icon: compass -keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "comparison"] ---- - -LanceDB gives you two ways to work with multimodal data, both built on the open -[Lance](/basics/lance) format. - -**LanceDB Embedded (open source)** is a library that runs in your -application's process; your data can live on local disk or in object storage. -You control the application, -its resources, and table maintenance, without a separate database service. -[Try the quickstart](/quickstart) to create a table and run your first search. - -**LanceDB Enterprise** is a multimodal lakehouse for data curation, feature -engineering, search, and training data access. It adds query-serving capacity, -data caching, and background operations for teams working with shared -multimodal datasets. LanceDB Enterprise runs LanceDB Distributed: query -execution, indexing and maintenance run on the Enterprise service, outside your -application. -[Connect to Enterprise](/basics/connect#enterprise) when your cluster is ready. - - - - -## Why choose Enterprise? {#why-enterprise} - -### Serve growing workloads {#serving} - -Add query-serving capacity as more applications and users depend on your data. -Enterprise spreads query traffic across a fleet and caches frequently read -table data near compute, reducing repeated reads from object storage. Query -serving and background work use separate resources, so you can size each for -your workload. See [Enterprise architecture](/enterprise/architecture). - -### Reduce operational work {#operations} - -Let the service handle routine table maintenance instead of scheduling it in -application code. Enterprise handles background [compaction](/maintenance/compaction) and can -[clean up old table versions](/maintenance/cleanup) according to your retention -policy. Configure these maintenance tasks for your Enterprise cluster so each -application does not have to schedule them. - -### Fit your deployment requirements {#deployment} - -Choose a managed deployment or bring LanceDB into your own cloud account. -Enterprise integrates with OpenTelemetry for metrics and traces, helping your -team monitor the service alongside your applications. Talk to us about your -cloud, networking, and operational requirements to plan the right deployment. -The [Enterprise overview](/enterprise) explains the available options. - -### Connect your data workflows {#workflows} - -Use the same tables for retrieval, dataset preparation, and training data access. -With its [SQL endpoint](/basics/sql) enabled, Enterprise supports analytical -[curation](/curation), including joins, grouping, and sampling. -[Feature engineering](/feature-engineering) adds derived columns and tables; -computed columns and materialized views require the corresponding features to -be enabled for your Enterprise cluster. -[Geneva](/geneva) runs Python feature backfills as distributed jobs. -[Training data loaders](/training/data-loading), available in both offerings, -read the resulting datasets for model training. - -## Plan your Enterprise deployment {#get-started} - -[Talk to us about your workload](https://www.lancedb.com/contact) to discuss your -data, query traffic, pipelines, and deployment requirements. - -For a closer look at how the service runs, read the -[Enterprise architecture](/enterprise/architecture). diff --git a/docs/web/basics/query.mdx b/docs/web/basics/query.mdx deleted file mode 100644 index aa8e9b9df..000000000 --- a/docs/web/basics/query.mdx +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: "Queries" -sidebarTitle: "Query" -description: "Choose how to query your data and read the results." -icon: magnifying-glass -keywords: ["query", "search", "SQL", "async", "streaming", "results"] ---- - -## Querying tables {#queries} - -A query selects data from your tables. You can filter rows, choose columns, -search for similar content or combine those operations. For example, a video -application might find clips similar to a sample and return only those recorded -in the last month. - -## SDK searches and SQL queries {#searches} - -| Interface | When to use it | -| --- | --- | -| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in LanceDB Embedded and LanceDB Enterprise. | -| **Enterprise SQL** | Write SQL queries for filtering, aggregation, joins and other data curation tasks. Connect through Python or a Flight SQL client. | - -Both interfaces work with LanceDB tables. Choose the one that supports your -operation and fits your application. See [SQL](/basics/sql) for connection -instructions and SQL search support, and [Search](/search) for the query builders. - -## Collect or stream results {#results} - -- **Collect the results** when you want a list, Arrow table or DataFrame that - fits in memory. Use a result limit and select only the columns you need. -- **Read batches** when you want to process a large result incrementally. - Reading batches avoids loading the entire result into application memory. - It does not guarantee that the query can return its first batch immediately. - -### Asynchronous clients {#async-client} - -An async client lets your application do other work while a request is waiting. -That is separate from whether you collect results or read them in batches. -Python offers synchronous and asynchronous clients; TypeScript uses async APIs. -Check the SDK reference for the methods available in your client. - -### Query handles and background jobs {#jobs} - -An Enterprise SQL query handle lets you inspect query execution and request -cancellation. Background operations such as index builds return a -[job](/basics/jobs) handle instead. Use the API for the returned handle to -check status and retrieve results. - -## Learn more {#next} - -- [Quickstart](/quickstart): run a complete create, insert and search example. -- [Search](/search): choose vector, full-text or hybrid retrieval. -- [Filtering](/search/filtering): narrow results with predicates. -- [SQL](/basics/sql): connect a client and run SQL queries. -- [SDKs](/basics/sdk): choose a language and find its API reference. diff --git a/docs/web/basics/sdk.mdx b/docs/web/basics/sdk.mdx index 3144c11cf..1d9312f77 100644 --- a/docs/web/basics/sdk.mdx +++ b/docs/web/basics/sdk.mdx @@ -4,12 +4,6 @@ sidebarTitle: "SDKs" description: "Choose a LanceDB SDK and find the API reference for your language." icon: code-compare keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"] -availability: - oss: available - enterprise: available - summary: >- - Python, TypeScript and Rust support both LanceDB Embedded and LanceDB Enterprise. - Java provides a client for the Enterprise REST API. --- Use a LanceDB SDK to create tables, add data, build indexes and run searches @@ -21,16 +15,15 @@ run on the Enterprise service, outside your application. The [quickstart](/quickstart) includes installation instructions and complete examples in all three languages. For connection settings, see -[Connect to LanceDB](/basics/connect). +[Connect to LanceDB](/basics/catalog). ## Official SDKs {#supported-sdks} -| SDK | Use it with | API reference | -| --- | --- | --- | -| Python | Python applications, notebooks, pandas and PyArrow | [Python reference](https://lancedb.github.io/lancedb/python/python/) | -| TypeScript | Node.js and TypeScript applications | [TypeScript reference](https://lancedb.github.io/lancedb/js/) | -| Rust | Rust applications using the native library | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) | -| Java | Applications using the Enterprise REST API | [Java reference](https://lancedb.github.io/lancedb/java/java/) | +| SDK | API reference | +| --- | --- | +| Python | [Python reference](https://lancedb.github.io/lancedb/python/python/) | +| TypeScript | [TypeScript reference](https://lancedb.github.io/lancedb/js/) | +| Rust | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) | Guides show the available language examples together. Use the API reference for method signatures and options; SDKs do not expose every feature through the same @@ -41,12 +34,35 @@ For SQL joins, aggregations and window functions, use languages can use a Flight SQL client. TypeScript's table query API does not provide a SQL execution method. -## REST API {#rest} +## Lance Namespace SDKs {#lance-namespace-sdks} -LanceDB Enterprise also exposes an HTTP API for catalog and table operations. -Use it when integrating a language or tool without a LanceDB SDK. The -[REST API reference](/api-reference/rest) describes its endpoints and request -formats. For SQL queries, connect through [Flight SQL](/basics/sql). +LanceDB Enterprise's catalog and table operations follow +[Lance Namespace](https://lance.org/format/namespace), an open specification +for describing access to tables in a multimodal lakehouse, including its REST +Namespace Spec. LanceDB Enterprise exposes this as an HTTP API for catalog and +table operations -- the [REST API reference](/api-reference/rest) describes +its endpoints and request formats. Use it when integrating a language or tool +without a LanceDB SDK; for SQL queries, connect through +[Flight SQL](/basics/sql) instead. Any client that implements a Lance +Namespace REST client can integrate with LanceDB this way, not only the +official SDKs above. + +| Language | Package | Used by | +| --- | --- | --- | +| Java | `org.lance:lance-namespace-core` | Spark, Flink, Kafka, Trino, Presto; also LanceDB's Java client | +| Python | `lance-namespace` | Core interface and connect functionality | +| Rust | `lance-namespace` | Core interface, in [lance-format/lance](https://github.com/lance-format/lance) | + +LanceDB's Java client builds on this Lance Namespace Java core: table creation, +listing and most other catalog operations go through the generated +`LanceNamespace` interface talking to Enterprise's REST Namespace endpoint. +The [real time engine](/tables/realtime-engine) is the one area with its own, +LanceDB-specific Java client on top, since RTE's LSM write path is not part of +the Lance Namespace specification. + +See [lance.org/format/namespace](https://lance.org/format/namespace) for the +full spec and its other catalog implementations (Directory, Apache Polaris, +Unity Catalog, Apache Hive Metastore, Apache Iceberg REST Catalog, and more). ## Community SDKs {#community-sdks} @@ -61,66 +77,15 @@ features, installation instructions and releases before choosing one. | R | [lancedb](https://github.com/CathalByrneGit/lancedb) | | Flutter | [flutter_lancedb](https://github.com/Alexcn/flutter_lancedb) | -## Python multiprocessing {#multiprocessing} - - -Use `spawn` when starting Python worker processes with `multiprocessing`. -LanceDB uses threads internally, and forking a process with active threads can -cause deadlocks. Create a connection inside each worker instead of sharing an -open connection from the parent process. - - -For PyTorch worker setup and connection factories, see -[PyTorch integration](/training/torch). - -## Moving to LanceDB Enterprise {#moving-to-enterprise} - -The core table workflow stays familiar: connect, create or open a table, write -records, and query them. Enterprise runs those operations with LanceDB -Distributed and manages background maintenance. - -- **Configure the connection.** Replace the local path with your database URI, - and supply the endpoint, region and credentials provided by your administrator. - [Connect to Enterprise](/basics/connect#enterprise) shows these settings. -- **Retrieve query results.** Python tables on Enterprise do not support whole-table - `to_arrow()` or `to_pandas()` calls. Use a query and select the columns and - number of rows you need, for example `table.search().limit(10).to_arrow()`. -- **Handle table listings and namespaces.** Enterprise table listings are paginated. - Use the namespace and pagination options described in - [Namespaces](/management/namespace). -- **Use server-side jobs.** Index builds and feature refreshes can run as - [jobs](/basics/jobs). Each operation's guide explains how to submit work and - track completion. -- **Let Enterprise manage maintenance.** Use - [background maintenance](/maintenance/background-jobs) rather than calling - the embedded table's `optimize()` method. - -{/* Draft for the remote option validation release. -- **Remove local storage and cache settings.** `db://` connections reject local - sessions, table cache settings, and per-table storage and write options. Omit - these options so Enterprise manages storage and caching. Connection-level - storage options supported by your Enterprise deployment remain available. -*/} - -For SQL connection details, see [SQL](/basics/sql). For the operating model and -deployment options, see [LanceDB Enterprise](/enterprise). - -## Shared data across languages {#invariants} - -LanceDB uses Arrow schemas across its SDKs. A fixed-size vector column stores -vectors of a consistent dimension; the [data types reference](/sql/data-types) -describes the supported element types and other column types. - -Applications in different languages can work with the same tables. Connect to -the same database and namespace, use the appropriate credentials, and account -for [read consistency](/tables/consistency) when another process writes data. - ## Related guides {#next} Query Enterprise tables with SQL from Python or a Flight SQL client. + + Use LanceDB tables in a DataLoader, including multiple worker processes. + Use the LanceDB plugin to help a coding agent work with your data. diff --git a/docs/web/basics/sql.mdx b/docs/web/basics/sql.mdx index ce34237b5..590e3989e 100644 --- a/docs/web/basics/sql.mdx +++ b/docs/web/basics/sql.mdx @@ -4,11 +4,6 @@ sidebarTitle: "SQL" description: "Query LanceDB Enterprise with SQL from Python or an Arrow Flight SQL client." icon: table-list keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"] -availability: - oss: unavailable - enterprise: available - summary: >- - Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for LanceDB Embedded tables. See [Connection settings](#sql-is-remote) to connect. --- Use SQL in LanceDB Enterprise to filter records, join tables, calculate @@ -16,26 +11,42 @@ aggregates, and prepare datasets. SQL queries work with the same Lance tables you create and update through the SDKs, without exporting the data to another system. -You can run SQL through the Python SDK or an Arrow Flight SQL client. This -guide covers connection settings, a first query, and how SQL fits alongside -the SDK table APIs. +This guide covers the two ways to run SQL, connection settings, a first query, +parameterized queries, and how to read and manage the results. -## Connect a SQL client {#flight-sql} +## Access LanceDB with SQL {#access} -Enterprise exposes an -[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) -endpoint. A compatible client sends SQL statements to this endpoint and -receives results as Arrow record batches. +There are two ways to run SQL against LanceDB Enterprise, both against the +same [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) +endpoint, returning results as Arrow record batches: -- **Python:** use `execute_query()` on an Enterprise connection, as shown below. -- **TypeScript:** use a separate Flight SQL client. The LanceDB TypeScript SDK - does not expose a SQL execution method. -- **Other languages and tools:** use a Flight SQL client or a tool with Flight - SQL integration. A LanceDB-specific SDK is not required to submit SQL. +1. **`db.execute_query()`** on the Python SDK's Enterprise connection, shown + below. This is LanceDB's native SQL execution method, and we recommend it + for the best performance -- see [Adaptive query execution](#adaptive). +2. **An Arrow Flight SQL client.** Use this from TypeScript and other + languages, whose LanceDB SDKs do not expose a SQL execution method, or from + any tool with its own Flight SQL integration. A LanceDB-specific SDK is not + required to submit SQL. The [SQL reference](/sql#connecting) shows a query from Python. See [SDKs](/basics/sdk) for the LanceDB language clients. +### Adaptive query execution {#adaptive} + +A standard Flight SQL client calls `GetFlightInfo` to plan a query and then +`DoGet` to fetch rows -- two round trips every time, even when the server +could stream results back immediately. + +`execute_query()` skips that extra round trip when it can. It calls `DoGet` +directly for a query with no parameters, or `DoExchange` when +[binding parameters](#parameters), and sends a +`lancedb-sql-protocol: adaptive-v1` header. The server decides per query +whether to stream results on that same call or defer execution and return a +poll ticket; when it defers, `execute_query()` follows up with +`PollFlightInfo` automatically, with no extra code on your side. A generic +Flight SQL client does not send that header, so it always takes the slower +`GetFlightInfo` path. + ### Connection settings {#sql-is-remote} Get the following values from your Enterprise administrator: @@ -49,7 +60,7 @@ The Flight SQL endpoint must be enabled and reachable from your client. Pass it explicitly as `sql_host_override` in Python, as in the example below. The API and SQL endpoints may use different hosts or ports. -See [Connect to Enterprise](/basics/connect#enterprise) for other connection options. +See [Connect to Enterprise](/basics/catalog#enterprise) for other connection options. SQL runs in Enterprise. Connections to a local directory or an object-storage bucket use LanceDB Embedded and cannot execute SQL statements. Use @@ -59,10 +70,11 @@ SDK table operations to query those tables. Install the SDK with `pip install lancedb`. Replace the connection placeholders with your Enterprise settings, including the region and both endpoint URLs. -This example reads an existing table named `my_docs` with `id` and `text` -columns. Change the query to match your table. +This example reads an existing `frames` table, with a `video_id` and a +`caption` column describing each frame. Change the query to match your table. -```python + +```python Python icon="python" import lancedb db = lancedb.connect( @@ -73,21 +85,77 @@ db = lancedb.connect( sql_host_override="https://your-sql-endpoint:443", ) -reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") +reader = db.execute_query("SELECT frame_id, video_id, caption FROM frames LIMIT 10") table = reader.read_all() print(table.to_pylist()) ``` + In application code, load credentials from environment variables or a secrets -manager. The [connection guide](/basics/connect#enterprise) shows that setup. +manager. The [connection guide](/basics/catalog#enterprise) shows that setup. -`execute_query()` returns a blocking Arrow reader when results start arriving. -The server can stream rows on the first call or return result tickets that the -SDK follows automatically, with the same behavior for parameterized queries. -The example uses `read_all()` to collect the small result in memory. For larger -results, consume the reader in batches. To submit a query and check its status -separately, use `execute_query_async()`. See [Queries](/basics/query) for query -handles, streaming, and cancellation. +`execute_query()` returns a blocking Arrow reader; see +[Adaptive query execution](#adaptive) for how it starts streaming. The example +above uses `read_all()` to collect the small result in memory. For larger +results, or to track and cancel a running query, see +[Read and manage results](#results). To submit a query and check its status +separately, use `execute_query_async()`. + +## Parameters in SQL queries {#parameters} + +Pass a `parameters` argument to bind values into a query, instead of +formatting them into the SQL string: + +```python +import numpy as np + +reader = db.execute_query( + "SELECT id FROM docs ORDER BY distance(vector, $1) LIMIT $2", + parameters=[np.array([0.1, 0.2, 0.3], dtype=np.float32), 10], +) +``` + +Values travel as Arrow, not interpolated SQL text, so a vector binds as a +fixed-size list with no manual formatting and no injection risk. Reference a +parameter with `$1`, `$2`, ... for positional binding, or `$name` for named +binding: + +- A list or tuple binds positionally, in order, to `$1`, `$2`, ... +- A dict binds by key to `$name`. +- A one-row `pyarrow.RecordBatch` or `Table` binds its columns, by name and by + position. +- A 1-D NumPy or Arrow array becomes a single fixed-size-list value -- use + this form for a vector argument, as shown above. + + +`parameters` is a LanceDB-specific mechanism, separate from the Flight SQL +protocol's `PREPARE`/bind step, which +[a deployment does not implement](/sql/prepared-statements#parameters). Bind +values with `parameters` from the Python SDK; a generic Flight SQL client has +no equivalent. + + +## Read and manage results {#results} + +- **Collect the results** when you want a list, Arrow table or DataFrame that + fits in memory. Use a result limit and select only the columns you need. +- **Read batches** when you want to process a large result incrementally. + Reading batches avoids loading the entire result into application memory. + It does not guarantee that the query can return its first batch immediately. + +### Asynchronous clients {#async-client} + +An async client lets your application do other work while a request is +waiting. That is separate from whether you collect results or read them in +batches. Python offers synchronous and asynchronous clients; TypeScript uses +async APIs. Check the SDK reference for the methods available in your client. + +### Query handles and background jobs {#jobs} + +An Enterprise SQL query handle lets you inspect query execution and request +cancellation. Background operations such as index builds return a +[job](/basics/jobs) handle instead. Use the API for the returned handle to +check status and retrieve results. ## Choose SQL or SDK table operations {#choosing} @@ -100,6 +168,14 @@ You can combine them. For example, create and populate a table with an SDK, then query it with SQL to prepare a training dataset. In Python, both workflows can use the same Enterprise connection. +Both interfaces work with the same Lance tables, through the same Arrow +schema. Applications in different languages can work with the same tables: +connect to the same database and namespace, use the appropriate credentials, +and account for [read consistency](/tables/consistency) when another process +writes data. A fixed-size vector column stores vectors of a consistent +dimension; see the [data types reference](/sql/data-types) for the supported +element types and other column types. + ### Common SQL operations {#sdk-parity} Several tasks are available through both interfaces, but the methods and SQL @@ -152,8 +228,14 @@ and data types. For SDK methods and their arguments, use the ## Next steps {#next} - - Read results, track long-running queries, and cancel work. + + Run a complete create, insert and search example. + + + Choose vector, full-text or hybrid retrieval. + + + Narrow results with predicates. Compare language clients and find their API references. diff --git a/docs/web/curation/aggregate.mdx b/docs/web/curation/aggregate.mdx index b6d02d7fb..4fbca58f4 100644 --- a/docs/web/curation/aggregate.mdx +++ b/docs/web/curation/aggregate.mdx @@ -5,10 +5,8 @@ description: "Measure coverage, missing values and class balance before selectin icon: chart-column keywords: ["aggregate", "group by", "class balance", "count", "profiling", "distinct"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports SQL aggregation. --- Use SQL aggregates to summarize a dataset: count records, find missing values, diff --git a/docs/web/curation/changes.mdx b/docs/web/curation/changes.mdx new file mode 100644 index 000000000..28f98f784 --- /dev/null +++ b/docs/web/curation/changes.mdx @@ -0,0 +1,93 @@ +--- +title: "Scope curation to what changed" +sidebarTitle: "Changes" +description: "Process only the rows added since your last curation run, instead of rescanning the whole table." +icon: clock-rotate-left +keywords: ["changes", "incremental", "cdc", "version", "append-only"] +availability: + embedded: unavailable + enterprise: available +--- + +A curation task you run repeatedly -- chunking, deduplication, profiling -- +does not need to rescan rows it already processed. `CHANGES` reads only the +rows appended to a table since an earlier +[version](/management/table#versioning), so a recurring job can scope itself +to what is new. + +## Example data {#example-data} + +`curation_batches` gets a batch of new documents appended periodically, with +no updates or deletes between appends: + +```sql +CREATE TABLE curation_batches (doc_id BIGINT, source VARCHAR, text VARCHAR); +INSERT INTO curation_batches VALUES + (1, 'web', 'A cat on a chair'), + (2, 'web', 'A dog in a field'); +-- note the version after this insert, for example 5 +INSERT INTO curation_batches VALUES + (3, 'books', 'A bird in a tree'), + (4, 'books', 'A cat by a window'); +-- the table is now at a later version, for example 7 +``` + +## Read only what is new {#read-new} + +`CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)` reads the rows +appended after version 5 -- documents 3 and 4 -- without rescanning 1 and 2: + +```sql +SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5) +``` + +Record the table's version after each curation run, then start the next +run's `CHANGES` from it. See [Versioning](/management/table#versioning) for +the APIs that read a table's current version. + +## Chain it into a curation task {#chain} + +`CHANGES` is a relation, like any other table reference, so it composes with +the rest of this page: chunk, dedupe, or join only the new rows, instead of +the whole table. Materializing the result keeps a running, append-only table +of work already done: + +```sql +INSERT INTO curated_batches +SELECT doc_id, source, text +FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5) +WHERE source = 'books' +``` + +A [chunker Function](/curation/chunking) or a [`dedupe`](/sql/dql/dedupe) +call works the same way: put the `CHANGES` relation where the source table +would otherwise go, and only the new rows flow through it. + +## Strictly append-only {#strict} + +The clause is named for what it requires, not just what it does. If anything +other than a pure append happened to the table between the starting version +and the current one -- an update, a delete, a merge -- the query is refused +rather than silently returning an incomplete or wrong result: + +```sql +-- refused if any row in curation_batches was updated or deleted after version 5 +SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5) +``` + +When that happens, either re-run from a later starting version that is +itself after the non-append write, or read the table in full instead of +incrementally for that run. See +[Changes since a version](/sql/dql/changes#strict) for the complete set of +refused forms. + +## What to read next {#next} + + + + The full SQL syntax, placement rules, and unsupported forms. + + + How table versions work, and how to find one to start from. + + diff --git a/docs/web/curation/chunking.mdx b/docs/web/curation/chunking.mdx new file mode 100644 index 000000000..f4c0197b7 --- /dev/null +++ b/docs/web/curation/chunking.mdx @@ -0,0 +1,210 @@ +--- +title: "Chunk records into multiple rows" +sidebarTitle: "Chunking" +description: "Expand one source row into many derived rows -- documents into passages, videos into clips, images into tiles." +icon: diagram-subtask +keywords: ["chunking", "function", "udf", "lateral", "unnest", "1:N", "passages", "clips", "video", "frames"] +availability: + embedded: unavailable + enterprise: available +--- + +Every other curation task on this page keeps the row count the same or +smaller: profiling, sampling, joining, deduplicating. Chunking goes the other +way -- one source row becomes many derived rows, each carrying its parent's +columns along with it. Use it to split a document into passages for +embedding, a video into clips, or an image into tiles. + +## Define a chunking Function {#function} + +A Function that returns a `list` produces more than one value per input row. +A chunker is a Function whose output is a list -- of structs, of strings, of +anything -- registered like any other Function. See +[Create a Function](/curation/create-function) for CPU, GPU and remote-service +Functions, and scalar versus batch calling, if you have not registered one +before. + +A function that splits text into fixed-size passages turns a `documents` +table into one row per passage: + +**Source: `documents`** + +| doc_id | title | text | +| --- | --- | --- | +| 1 | "Intro to AI" | "Machine learning is a method of..." (long text) | + + +```python Python icon="python" +import pyarrow as pa +from lancedb.functions import udf + +@udf( + output_schema=pa.list_( + pa.struct([ + pa.field("chunk_index", pa.int64(), nullable=False), + pa.field("chunk_text", pa.utf8(), nullable=False), + ]) + ), +) +def split_into_passages(text: str) -> list[dict]: + size = 200 + return [ + {"chunk_index": i, "chunk_text": text[i : i + size]} + for i in range(0, len(text), size) + ] + +version = db.create_function_async(split_into_passages).wait() +``` + + +Registration details, environment and dependency declarations, and how a +version stays immutable once rows are filled from it, are the same for a +chunker as for any other Function. The same function can be registered from +SQL with `CREATE FUNCTION`: + + +```sql SQL +CREATE FUNCTION split_into_passages(text utf8) +RETURNS list> +LANGUAGE python +AS 'def split_into_passages(text): size = 200; return [{"chunk_index": i, "chunk_text": text[i:i+size]} for i in range(0, len(text), size)]' +``` + + +## Explode it in a query {#explode} + +Call the function in the `FROM` clause, after the source table, with an +alias: a Function there yields one row per element of the list it returns, +with every other column repeated. Reference the function's output fields +through the alias: + + +```python Python icon="python" +reader = db.execute_query(""" + SELECT doc_id, title, p.chunk_index, p.chunk_text + FROM documents, split_into_passages(text) AS p +""") +print(reader.read_all().to_pylist()) +``` + + +A document with five 200-character passages becomes five output rows, each +carrying `doc_id` and `title` from its parent. This is the fastest way to +check a chunker before committing to a materialized view: run it over a few +rows and inspect the output shape. + +## Video into frames {#video} + +Chunking a blob column is the same shape, with the chunker producing +timestamps rather than text spans, and a second Function pulling the actual +frame for each one. `videos` holds raw clips in a +[Blob v2](/management/table#managed-blobs) column, with each row's duration +already known: + +**Source: `videos`** + +| video_id | video | duration_seconds | +| --- | --- | --- | +| "clip-1" | (Blob v2 bytes) | 12.0 | + +A chunker that samples one timestamp per second needs no blob access at all +-- it only looks at `duration_seconds`: + + +```python Python icon="python" +@udf( + output_schema=pa.list_( + pa.struct([ + pa.field("frame_index", pa.int64(), nullable=False), + pa.field("ts", pa.float64(), nullable=False), + ]) + ), +) +def sample_frame_timestamps(duration_seconds: float) -> list[dict]: + every = 1.0 # one frame per second + count = int(duration_seconds // every) + return [{"frame_index": i, "ts": i * every} for i in range(count)] + +version = db.create_function_async(sample_frame_timestamps).wait() +``` + + +A second Function reads the blob -- incrementally, through a `BlobReader`, +rather than loading the whole clip -- and returns one JPEG per call: + + +```python Python icon="python" +import lancedb +from datetime import timedelta +from lancedb import BlobReader + +@udf( + name="extract_frame", + gpu=True, + pip=["av==13.1.0", "pillow==11.0.0"], + input_schema=pa.schema([ + lancedb.blob("video"), + pa.field("ts", pa.float64(), nullable=False), + ]), + output_schema=lancedb.blob("jpeg", nullable=False), +) +def extract_frame(video: BlobReader, ts: float) -> bytes: + ... # decode the frame at ts, encode as JPEG, return its bytes + +extract_version = db.create_function_async(extract_frame).wait(timeout=timedelta(minutes=15)) +``` + + +Compose both in one query: explode `videos` into one row per sampled +timestamp, then call `extract_frame` on each row to produce its image: + + +```python Python icon="python" +reader = db.execute_query(""" + SELECT video_id, f.frame_index, f.ts, extract_frame(video, f.ts) AS jpeg + FROM videos, sample_frame_timestamps(duration_seconds) AS f +""") +``` + + +A 12-second clip becomes 12 output rows, each with its own extracted JPEG. A +Function in `FROM` position yields one row per list element the same way +regardless of what the list holds; whether the second step produces a blob, +a plain value, or another struct does not change the explosion. + +## Persist it as a materialized view {#persist} + +Wrap either query in [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) +to keep a real table that stays current as the source changes, instead of +recomputing the explosion on every query: + + +```sql SQL +CREATE MATERIALIZED VIEW chunks AS +SELECT doc_id, title, p.chunk_index, p.chunk_text +FROM documents, split_into_passages(text) AS p +``` + + +**Derived: `chunks`** (1:N expansion) + +| doc_id | title | chunk_index | chunk_text | +| --- | --- | --- | --- | +| 1 | "Intro to AI" | 0 | "Machine learning is a method..." | +| 1 | "Intro to AI" | 1 | "...of teaching computers..." | + +The view refreshes like any other materialized view: new source rows are +chunked and inserted, deleted source rows drop their children, and an +updated row's old children are replaced by freshly chunked ones on the next +[refresh](/feature-engineering/refresh-column). A single Function call or +`UNNEST(column)` is the only lateral source a materialized view can carry -- +not both, and not combined with `GROUP BY` -- so keep aggregation or further +curation in a second step over the chunked table. + +## Chunk, then curate {#next} + +A freshly chunked table is a normal table: [dedupe](/curation/dedupe) it to +collapse near-identical chunks, [sample](/curation/sampling) it for a quick +look, or [profile it](/curation/aggregate) to check how many chunks each +source row produced. Continue with +[choosing a curation task](/curation#questions). diff --git a/docs/web/curation/create-function.mdx b/docs/web/curation/create-function.mdx new file mode 100644 index 000000000..2d161e84b --- /dev/null +++ b/docs/web/curation/create-function.mdx @@ -0,0 +1,213 @@ +--- +title: "Create a Function" +sidebarTitle: "Create Function" +description: "Register Python as a column: CPU, GPU, and remote-service Functions, called once per row or once per batch." +icon: code +keywords: ["function", "udf", "cpu", "gpu", "batch", "scalar", "blob", "remote api", "embedding", "rate limiter"] +availability: + embedded: unavailable + enterprise: available +--- + +Some curation steps cannot be written as a SQL expression: an embedding +model, an image decode, a call to an external service. A Function is a +Python callable registered with the database, versioned, and then applied as +a column or called in a query -- the code runs where the data is, not pulled +into your process first. This page covers how to shape one; later curation +tasks such as [Chunking](/curation/chunking) assume you have read it. + +## Scalar: once per row {#scalar} + +The plain form takes one row's values and returns one result. Types are +inferred from the annotations: + + +```python Python icon="python" +from lancedb import udf + +@udf(pip=["numpy==2.2.0"]) +def score(value: float) -> float: + return value * 2 + +version = db.create_function_async(score).wait() +table.add_columns({"doubled": version(value=col("value"))}) +``` + + +`pip` declares the worker's dependencies; they are part of the function's +immutable version, so changing one produces a new version rather than +altering rows a previous version already filled. + +## Batch: once per call, many rows at a time {#batch} + +A function is called once per batch instead of once per row when its +parameters are annotated `pyarrow.Array`, or when it takes a single +`pyarrow.RecordBatch` parameter to read more than one column together. The +same doubling logic, vectorized over a whole batch at once: + + +```python Python icon="python" +import pyarrow as pa +import pyarrow.compute as pc +from lancedb import udf + +@udf(pip=["numpy==2.2.0"]) +def score_batch(value: pa.Array) -> pa.Array: + return pc.multiply(value, 2) +``` + + +Batching amortizes per-call overhead across many rows, which matters most +once a function also does something with real fixed cost per call -- loading +a model, opening a connection, making a network round trip. + +## Stateful Functions: load once, call many times {#stateful} + +A `@udf`-decorated **class** is the same callable with state. Each remote +instance calls `__init__` once, calls `__call__` for every row or batch it +processes, and calls `close()`, if defined, once when it retires: + + +```python Python icon="python" +from lancedb import udf + +@udf +class scale: + def __init__(self, factor: float, offset: float = 0.0): + self.factor = factor + self.offset = offset + + def __call__(self, value: float) -> float: + return value * self.factor + self.offset +``` + + +Constructor parameters are **initialization fields**, bound once per binding +rather than read per row -- mixed with per-row columns in the same call: +`version(value=col("value"), factor=2.0)`. Load models, open clients, and +build rate limiters in `__init__`; every batch the instance serves then +reuses them, since an instance's state is shared across its own batches but +never across other instances or workers. This is the pattern GPU inference +and remote-API Functions both build on below. + +## GPU: batch inference, loaded once {#gpu} + +`gpu=True` requires a GPU-capable worker for every execution. Combined with +the batch and stateful forms above, a realistic embedding model loads once +in `__init__` and runs one forward pass per batch, not one per row: + + +```python Python icon="python" +import pyarrow as pa +from lancedb import udf + +@udf(gpu=True, pip=["torch==2.12.0", "sentence-transformers==5.7.0"]) +class embed_text: + def __init__(self, model_name: str = "all-MiniLM-L6-v2"): + import torch + from sentence_transformers import SentenceTransformer + + device = "cuda" if torch.cuda.is_available() else "cpu" + self.model = SentenceTransformer(model_name, device=device) + + def __call__(self, text: pa.Array) -> pa.Array: + texts = [t or "" for t in text.to_pylist()] + embeddings = self.model.encode(texts) + return pa.array(embeddings.tolist(), type=pa.list_(pa.float32(), list_size=384)) + +version = db.create_function_async(embed_text).wait() +table.add_columns({"embedding": version(text=col("caption"))}) +``` + + +The model loads once per worker instance, in `__init__`; `__call__` only +encodes the batch it was handed. Registering the same logic as a plain +function instead would reload the model on every single row. + +## Calling a remote service {#remote} + +A function that calls an external API needs a credential without putting it +in the function's source. A Secret is a database-scoped named credential: +`db.create_secret` stores the value once, and binding it at registration time +puts it in the environment variable the function reads, without ever +returning it from any API or recording it in the function's own definition. +Combine it with the stateful form so the HTTP client -- and any rate limiter +around it -- is built once, not reopened on every row: + + +```python Python icon="python" +import os +from lancedb import EnvVarSecret, udf + +db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"]) + +@udf +class embed_remote: + def __init__(self, model: str = "text-embedding-3-small"): + import os + from openai import OpenAI + + self.client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) + self.model = model + + def __call__(self, text: str) -> list[float]: + response = self.client.embeddings.create(input=text, model=self.model) + return response.data[0].embedding + +version = db.create_function_async( + embed_remote, + secrets=[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")], +).wait() +``` + + +A mistyped Secret name is caught at registration; a mistyped environment +variable name only surfaces on the first rows, since nothing records which +variables a function reads. There is no built-in retry or per-row +error-skip policy -- a function that calls a flaky service should handle +timeouts and retries itself, inside `__call__`. + +## Blob v2 input and output {#blob} + +A Function can bind a [Blob v2 column](/management/table#managed-blobs) and +return another one, including a blob nested inside a struct field, not only +at the top level. In row mode, a blob parameter arrives as Python `bytes` +(or a `BlobReader` for incremental, seekable access, so a function never +has to materialize a payload it only partially needs): + + +```python Python icon="python" +import pyarrow as pa +import lancedb +from lancedb import BlobReader, udf + +@udf( + gpu=True, + pip=["av==13.1.0", "pillow==11.0.0"], + input_schema=pa.schema([ + lancedb.blob("video"), + pa.field("ts", pa.float64(), nullable=False), + ]), + output_schema=lancedb.blob("jpeg", nullable=False), +) +def extract_frame(video: BlobReader, ts: float) -> bytes: + ... # decode the frame at ts, encode as JPEG, return its bytes +``` + + +`output_schema=lancedb.blob(...)` is what makes the result Blob v2 rather +than plain Arrow `binary` -- returning `bytes` with no explicit blob output +schema still holds the right bytes, but skips the Managed storage path and +`fetch_blobs`. Declare it explicitly on every function whose output should +be a blob. + +## What to read next {#next} + + + + A Function whose output is a list, exploded into one row per element. + + + Filling the column a Function is bound to, and re-filling it as inputs change. + + diff --git a/docs/web/curation/dedupe.mdx b/docs/web/curation/dedupe.mdx new file mode 100644 index 000000000..bdcf0b4b2 --- /dev/null +++ b/docs/web/curation/dedupe.mdx @@ -0,0 +1,142 @@ +--- +title: "Deduplicate near-duplicate records" +sidebarTitle: "Dedupe" +description: "Collapse records that are only approximately alike, judged by embedding distance, without a pairwise scan." +icon: clone +keywords: ["dedupe", "duplicate_pairs", "deduplication", "near-duplicate", "embedding", "distance_threshold"] +availability: + embedded: unavailable + enterprise: available +--- + +[Ranking and deduplication](/curation/windowing#dedup) removes records by an +explicit rule, such as keeping the latest version of a known key. This page +covers the other kind of duplicate: records with no shared key that are still +nearly identical, judged by distance between their embeddings -- a stopped +camera holding one pose for a few seconds, or two scraped pages with the same +paragraph reworded. `dedupe` and `duplicate_pairs` find these directly over a +vector index, without a pairwise scan. + +## Example data {#example-data} + +Connect with an [Enterprise connection](/basics/catalog#enterprise). A small +fixture keeps the distances easy to check by hand. + + + +`curation_embeddings` has eight records. Rows 10 and 20 are near-identical; +rows 30, 40 and 50 form one looser cluster; the rest are distinct. + + +```python Python icon="python" +import pyarrow as pa +from lancedb.index import IvfFlat + +schema = pa.schema([ + pa.field("id", pa.int64(), nullable=False), + pa.field("caption", pa.utf8()), + pa.field("embedding", pa.list_(pa.float32(), 2)), +]) +table = db.create_table("curation_embeddings", [ + {"id": 10, "caption": "a cat on a chair", "embedding": [0.0, 0.0]}, + {"id": 20, "caption": "a cat on the chair", "embedding": [0.0, 0.03]}, + {"id": 30, "caption": "a dog in a field", "embedding": [5.0, 5.0]}, + {"id": 40, "caption": "a dog in the field", "embedding": [5.0, 5.05]}, + {"id": 50, "caption": "a dog out in a field", "embedding": [5.0, 5.10]}, + {"id": 60, "caption": "a bird in a tree", "embedding": [10.0, 10.0]}, + {"id": 70, "caption": "a boat on the water", "embedding": [20.0, 20.0]}, + {"id": 80, "caption": "a car on the road", "embedding": [30.0, 30.0]}, +], schema=schema) + +table.create_index("embedding", config=IvfFlat(num_partitions=1, distance_type="l2")) +table.wait_for_index(["embedding_idx"]) +``` + + +`dedupe` and `duplicate_pairs` read the vector index, so the column needs one +before either function runs; a query against unindexed fragments is rejected +rather than silently returning stale or wrong results. + + + +## Collapse near-duplicates {#dedupe} + +`dedupe` returns one representative row per cluster of records within +`distance_threshold` of each other -- rows 10/20 collapse to one, 30/40/50 +collapse to one, and the four distinct rows pass through unchanged: + + +```python Python icon="python" +reader = db.execute_query(""" + SELECT id, caption FROM dedupe('curation_embeddings', 'embedding', distance_threshold => 0.06) + ORDER BY id +""") +print(reader.read_all().to_pylist()) +# [{'id': 10, 'caption': 'a cat on a chair'}, {'id': 30, 'caption': 'a dog in a field'}, +# {'id': 60, ...}, {'id': 70, ...}, {'id': 80, ...}] +``` + + +Every other column survives unchanged, since `dedupe` returns full rows, not +just which ones matched. Clustering is transitive: 30 and 40 are within +`distance_threshold` of each other, and so are 40 and 50, so all three +collapse into one representative even though 30 and 50 alone sit outside the +threshold. Pass `resolution => 'direct_representative'` +for a simpler, non-transitive selection instead, if a chained cluster like that +should not merge into one representative. + +Which row survives as the representative is not guaranteed to stay the same +across appends: a later row that bridges two previously separate clusters can +change which id represents the merged cluster. Do not treat a representative's +id as a stable identifier for its cluster. + +## Inspect the pairs instead {#pairs} + +`duplicate_pairs` returns the matching edges themselves, without collapsing +anything -- useful for review, or for a threshold that is still being tuned: + + +```python Python icon="python" +db.execute_query(""" + SELECT id_a, id_b, distance FROM duplicate_pairs( + 'curation_embeddings', 'embedding', distance_threshold => 0.06, id_column => 'id' + ) + ORDER BY id_a +""") +``` + + +This returns the pairs `(10, 20)`, `(30, 40)` and `(40, 50)` -- note 30 and 50 +are not a direct pair here (they sit `0.10` apart, outside the threshold), +even though `dedupe` merges all three transitively above. + +## Keep it current with a materialized view {#materialized} + +Wrap either call in a [materialized view](/management/materialized-view) so +the deduplicated set stays current as new records arrive, instead of +recomputing it from scratch on every curation pass: + + +```python Python icon="python" +db.execute_query(""" + CREATE MATERIALIZED VIEW curation_embeddings_deduped AS + SELECT * FROM dedupe('curation_embeddings', 'embedding', distance_threshold => 0.06) +""") +``` + + +See [Refresh a materialized view](/feature-engineering/refresh-materialized-view) +for manual and automatic refresh options to keep it current as new records land. + +## At production scale {#production} + +[LeRobotDataset](/integrations/lerobotdataset) runs the same `dedupe` +call over 27.6 million real embedding rows to remove near-static frames from a +robotics training set in under 30 seconds -- the mechanism does not change +with scale, only the table it runs against. + +## Continue with the curated data {#next} + +[Profile the result](/curation/aggregate) to check how much a threshold +removes before committing to it, or go back to +[choosing a curation task](/curation#questions). diff --git a/docs/web/curation/index.mdx b/docs/web/curation/index.mdx index 191ff2fd1..c4d6a48e5 100644 --- a/docs/web/curation/index.mdx +++ b/docs/web/curation/index.mdx @@ -17,10 +17,14 @@ records belong, and save the selected data or the rules used to produce it. | Task | What it helps you decide | | --- | --- | +| [Create a Function](/curation/create-function) | How to run your own Python -- on CPU, on GPU, or against a remote service -- as part of SQL. | +| [Chunk records](/curation/chunking) | How to split one record into several -- passages, clips, or tiles. | +| [Scope to what changed](/curation/changes) | How to process only the rows added since your last run. | | [Profile a dataset](/curation/aggregate) | Whether coverage, missing values and class balance suit the task. | | [Sample records](/curation/sampling) | Which subset to inspect or use for an experiment. | | [Join datasets](/curation/join) | How to attach labels, metadata or scores stored elsewhere. | -| [Rank and deduplicate](/curation/windowing) | Which records to keep when several describe the same item. | +| [Rank and deduplicate](/curation/windowing) | Which records to keep when several share an explicit key. | +| [Deduplicate near-duplicates](/curation/dedupe) | Which records to collapse when they only look alike, with no shared key. | The analytical examples in this section use **Enterprise SQL**. Connect with the [SQL guide](/basics/sql). For sampling through the Python SDK in LanceDB Embedded or diff --git a/docs/web/curation/join.mdx b/docs/web/curation/join.mdx index 03a45326e..3306ca511 100644 --- a/docs/web/curation/join.mdx +++ b/docs/web/curation/join.mdx @@ -5,10 +5,8 @@ description: "Combine records with labels or metadata, and check for missing or icon: link keywords: ["join", "labels", "metadata", "enrichment", "namespace"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports SQL joins. --- A SQL join combines records from two tables using a shared key. Use joins to @@ -127,6 +125,38 @@ Use [Explain a query plan](/search/explain) to see how the query reads and combines data. Check the plan to see whether the query uses indexes and how it performs the join. An index on a key does not guarantee that the join will use it. +### Defer wide columns until after the join {#late-materialization} + +When one side of a join carries a BlobV2 column, an embedding, or both, join +on a narrow key first -- `row_id`, `_rowid`, or another selective column -- +and project the wide columns only in the final select list, rather than +selecting them alongside the join condition. The planner can then defer +reading those columns until after the join has already narrowed down which +rows survive, instead of reading them for every candidate row. + +`frames` is wide: a BlobV2 `image` and a vector `embedding` alongside its +key. `selected` is narrow: the frame IDs an earlier curation step already +kept, such as a sample or a dedupe result. Join on the key, project the wide +columns last: + +```sql +SELECT s.frame_id, f.image, f.embedding +FROM selected AS s +JOIN frames AS f ON f.frame_id = s.frame_id +``` + +`selected` narrows the row set before `image` or `embedding` is ever touched +on the `frames` side, rather than reading every row's blob and vector just +to discard most of them after the join. + +This is query planning the optimizer may or may not apply to a given query, +not a guarantee from writing the join this way. Confirm it per query with +[`EXPLAIN`](/search/explain): a plan containing `RemoteTake` deferred the +wide read; a plan without it read the wide columns eagerly, even for a query +shaped exactly like the one above. See +[Late materialization for wide columns](/sql/dql/join#late-materialization) +in the SQL reference for the underlying mechanism. + ## Save the result or continue curating {#reference} Keep the query for reuse or [save the selected data](/curation#next). diff --git a/docs/web/curation/sampling.mdx b/docs/web/curation/sampling.mdx index 1289275a1..82f1c1ecb 100644 --- a/docs/web/curation/sampling.mdx +++ b/docs/web/curation/sampling.mdx @@ -1,14 +1,12 @@ --- -title: "SQL sampling" -sidebarTitle: "SQL Sampling" +title: "Sampling" +sidebarTitle: "Sampling" description: "Choose a sample for exploration or evaluation and preserve the selected records when reproducibility matters." icon: dice keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports SQL `TABLESAMPLE`. For sampling in LanceDB Embedded and LanceDB Enterprise through the Python SDK, see [Sample without SQL](#training). --- import { PySampleDataset } from '/snippets/curation_sampling.mdx'; @@ -163,7 +161,7 @@ promise the same IDs across SDK versions or changes to the source dataset. For shuffled training reads, worker configuration and data loaders, see -[Loading data for model training](/training/data-loading). +[PyTorch Data Loader](/training/data-loading). ## Related guides {#reference} diff --git a/docs/web/curation/windowing.mdx b/docs/web/curation/windowing.mdx index fa84ed5fe..6cb5322d6 100644 --- a/docs/web/curation/windowing.mdx +++ b/docs/web/curation/windowing.mdx @@ -5,10 +5,8 @@ description: "Choose records with explicit ranking rules and deterministic tie h icon: chart-simple keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports SQL window functions. --- Window functions calculate values across related rows while keeping each row diff --git a/docs/web/deployment/storage.mdx b/docs/web/deployment/storage.mdx index dd370846b..50746d778 100644 --- a/docs/web/deployment/storage.mdx +++ b/docs/web/deployment/storage.mdx @@ -1,23 +1,16 @@ --- -title: "Embedded storage configuration" -sidebarTitle: "Embedded Storage" -description: "Choose and configure local disk or object storage for tables accessed by LanceDB Embedded." +title: "Storage configuration" +sidebarTitle: "Storage Configuration" +description: "Choose and configure local disk or object storage for your tables." icon: database keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"] -availability: - oss: available - enterprise: unavailable - summary: >- - The embedded library chooses and configures its storage backend from the - connection URI and `storage_options`. In LanceDB Enterprise, storage is part of - the deployment's own configuration instead. --- This page is about LanceDB Embedded, where your connection chooses and configures the storage backend. A LanceDB Enterprise deployment sets up its own storage when it is deployed, so a `db://` connection passes none of these options except an Azure storage account name. See -[Deployment](/enterprise/deployment) for how a deployment is set up. +[LanceDB Enterprise](/enterprise) for how a deployment is set up. LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs). @@ -124,7 +117,7 @@ import { **LanceDB Enterprise storage configuration** -In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow. +In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Kubernetes deployment guide](/enterprise/deployment/kubernetes) for the Enterprise flow. ### Object stores {#object-stores} diff --git a/docs/web/docs.json b/docs/web/docs.json index 917eb6406..5729dd8eb 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -27,6 +27,42 @@ }, "favicon": "/static/favicon.ico", "redirects": [ + { + "source": "/architecture", + "destination": "/#architecture" + }, + { + "source": "/enterprise/architecture", + "destination": "/#architecture" + }, + { + "source": "/enterprise/deployment/azure", + "destination": "/enterprise/deployment/kubernetes" + }, + { + "source": "/basics/query", + "destination": "/basics/sql#results" + }, + { + "source": "/basics/offerings", + "destination": "/#embedded-and-enterprise" + }, + { + "source": "/enterprise/deployment", + "destination": "/enterprise#deployment" + }, + { + "source": "/feature-engineering/data-evolution", + "destination": "/feature-engineering/add-computed-column" + }, + { + "source": "/training/torch", + "destination": "/training/data-loading" + }, + { + "source": "/basics/lakehouse", + "destination": "/" + }, { "source": "/tables/update", "destination": "/ingestion/update" @@ -209,7 +245,7 @@ }, { "source": "/sql/refresh", - "destination": "/sql/dml/refresh" + "destination": "/sql/dml/refresh-column" }, { "source": "/sql/show", @@ -245,13 +281,11 @@ "group": "Getting Started", "pages": [ "index", - "basics/offerings", - "basics/connect", "quickstart", - "architecture", "basics/sdk", "basics/sql", - "basics/agentic-skills" + "basics/agentic-skills", + "deployment/support" ] }, { @@ -259,9 +293,8 @@ "pages": [ "basics/lance", "basics/catalog", - "basics/lakehouse", - "basics/query", "basics/jobs", + "tables/realtime-engine", "tables/consistency", "basics/guarantees", "basics/limits", @@ -276,7 +309,8 @@ "ingestion/insert", "ingestion/merge-insert", "ingestion/update", - "ingestion/delete" + "ingestion/delete", + "ingestion/realtime-write" ] }, { @@ -288,35 +322,42 @@ "search/hybrid-search", "search/filtering", "reranking/index", - "search/explain" + "search/explain", + "search/realtime-read", + "enterprise/distributed-search" ] }, { "group": "Data Curation", "pages": [ "curation/index", + "curation/create-function", + "curation/chunking", + "curation/changes", "curation/aggregate", "curation/sampling", "curation/join", - "curation/windowing" + "curation/windowing", + "curation/dedupe" ] }, { "group": "Feature Engineering", "pages": [ "feature-engineering/index", - "feature-engineering/data-evolution", "feature-engineering/add-computed-column", "feature-engineering/refresh-column", - "feature-engineering/materialized-views" + "feature-engineering/view", + "feature-engineering/materialized-views", + "feature-engineering/refresh-materialized-view" ] }, { "group": "Training", "pages": [ - "training/why-lancedb", + "training/permutation", "training/data-loading", - "training/torch" + "training/distributed-training" ] }, { @@ -327,19 +368,24 @@ "management/namespace", "management/table", "tables/branching", - "tables/realtime-engine", - "management/materialized-view", - "management/view" + "management/view", + "management/materialized-view" ] }, { "group": "Maintenance", "pages": [ - "indexing/index", - "indexing/vector-index", - "indexing/fts-index", - "indexing/scalar-index", - "indexing/reindexing", + { + "group": "Indexing", + "icon": "layer-group", + "pages": [ + "indexing/index", + "indexing/vector-index", + "indexing/fts-index", + "indexing/scalar-index", + "indexing/reindexing" + ] + }, "maintenance/compaction", "maintenance/cleanup", "maintenance/background-jobs" @@ -349,14 +395,10 @@ "group": "Deployment", "pages": [ "enterprise/index", - "enterprise/architecture", - "enterprise/distributed-search", - "enterprise/deployment/index", "deployment/storage", "deployment/monitoring", - "enterprise/deployment/azure", - "enterprise/benchmarks", - "deployment/support" + "enterprise/deployment/kubernetes", + "enterprise/benchmarks" ] }, { @@ -375,6 +417,7 @@ { "group": "DDL", "pages": [ + "sql/ddl/database", "sql/ddl/namespace", "sql/ddl/table", "sql/ddl/view", @@ -391,7 +434,8 @@ "sql/dml/update", "sql/dml/delete", "sql/dml/copy", - "sql/dml/refresh" + "sql/dml/refresh-column", + "sql/dml/refresh-materialized-view" ], "icon": "database" }, @@ -404,7 +448,10 @@ "sql/dql/aggregation", "sql/dql/window", "sql/dql/sampling", + "sql/dql/changes", "sql/dql/fts", + "sql/dql/dedupe", + "sql/dql/duplicate-pairs", "sql/dql/explain" ], "icon": "filter" diff --git a/docs/web/embedding/index.mdx b/docs/web/embedding/index.mdx index c35074b20..91efcc566 100644 --- a/docs/web/embedding/index.mdx +++ b/docs/web/embedding/index.mdx @@ -4,12 +4,8 @@ sidebarTitle: "Managing Embeddings" description: "Use the embedding API in LanceDB: registry, functions, schemas, and multi-language SDK support." icon: "bars" availability: - oss: available + embedded: available enterprise: available - summary: >- - Embedding functions run in your process, whether the table is local or on a - deployment, which stores their configuration but never runs them. Python - and TypeScript also embed query text; in Rust you embed queries yourself. --- import { diff --git a/docs/web/enterprise/architecture.mdx b/docs/web/enterprise/architecture.mdx deleted file mode 100644 index 8f887c142..000000000 --- a/docs/web/enterprise/architecture.mdx +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Enterprise architecture" -sidebarTitle: "Architecture" -description: "The tiers a deployment runs, and how they scale independently." -icon: cubes -keywords: ["architecture", "control plane", "data plane", "indexer"] ---- - -Enterprise - -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. - -A deployment separates compute from storage so each scales on its own: table -data and indexes live in object storage, while query serving and background work -run on machines that can be replaced without any of them owning the data. - -[Architecture](/architecture) covers this alongside the embedded form, which is -the comparison most readers want first. - -See [Distributed search](/enterprise/distributed-search) for how Enterprise -distributes index execution and table-data reads. diff --git a/docs/web/enterprise/authentication.mdx b/docs/web/enterprise/authentication.mdx index 74d68ecae..920275d16 100644 --- a/docs/web/enterprise/authentication.mdx +++ b/docs/web/enterprise/authentication.mdx @@ -5,12 +5,8 @@ description: "Authentication modes for LanceDB Enterprise: API keys and OAuth cr icon: key keywords: ["authentication", "oauth", "api key", "authorization code", "device code", "azure managed identity", "client credentials"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Requests to a deployment carry an API key or an OAuth token, whichever it is - configured to accept. LanceDB Embedded has no server to sign in to; see - [Architecture](/architecture). --- A deployment authenticates each request over a `db://` connection before diff --git a/docs/web/enterprise/benchmarks.mdx b/docs/web/enterprise/benchmarks.mdx index feec4dfc4..b733aef78 100644 --- a/docs/web/enterprise/benchmarks.mdx +++ b/docs/web/enterprise/benchmarks.mdx @@ -10,7 +10,7 @@ keywords: ["benchmarks", "performance", "throughput", "latency"] This page describes a LanceDB Enterprise capability. The embedded form of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. +[Architecture](/#architecture) for what differs between the two. Benchmark results are specific to a deployment's shape (cluster size, storage class, dataset), so they are published with LanceDB Enterprise. diff --git a/docs/web/enterprise/deployment/azure.mdx b/docs/web/enterprise/deployment/azure.mdx deleted file mode 100644 index 702c6a2eb..000000000 --- a/docs/web/enterprise/deployment/azure.mdx +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Azure deployment guide" -sidebarTitle: "Azure" -description: "Deploying LanceDB Enterprise on Azure." -icon: cloud -keywords: ["azure", "deployment", "aks"] ---- - -Enterprise - -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. - -The Azure deployment procedure is published with LanceDB Enterprise. See -[Deployment](/enterprise/deployment) for the general shape, and -[contact us](https://www.lancedb.com/contact) to get started. diff --git a/docs/web/enterprise/deployment/index.mdx b/docs/web/enterprise/deployment/index.mdx deleted file mode 100644 index 38e854282..000000000 --- a/docs/web/enterprise/deployment/index.mdx +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Deployment guide" -sidebarTitle: "Deployment Models" -description: "Standing up a LanceDB Enterprise deployment." -icon: server -keywords: ["deployment", "install", "cluster", "kubernetes"] ---- - -Enterprise - -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. - -Deployment is specific to your cloud and account, so the procedure is published -with LanceDB Enterprise. [Contact us](https://www.lancedb.com/contact) to get -started. - -LanceDB Embedded has nothing to deploy: it is a library your process links -against. See [Architecture](/architecture#embedded). diff --git a/docs/web/enterprise/deployment/kubernetes.mdx b/docs/web/enterprise/deployment/kubernetes.mdx new file mode 100644 index 000000000..3b1f4dea9 --- /dev/null +++ b/docs/web/enterprise/deployment/kubernetes.mdx @@ -0,0 +1,17 @@ +--- +title: "Kubernetes deployment guide" +sidebarTitle: "Kubernetes" +description: "Deploying LanceDB Enterprise on Kubernetes." +icon: cloud +keywords: ["kubernetes", "k8s", "aks", "deployment", "helm"] +--- + +Enterprise + +This page describes a LanceDB Enterprise capability. The embedded form +of LanceDB has no server to provide it. See +[Architecture](/#architecture) for what differs between the two. + +The Kubernetes deployment procedure is published with LanceDB Enterprise. See +[LanceDB Enterprise](/enterprise#deployment) for the general shape, and +[contact us](https://www.lancedb.com/contact) to get started. diff --git a/docs/web/enterprise/distributed-search.mdx b/docs/web/enterprise/distributed-search.mdx index 86837e157..f7c6001a5 100644 --- a/docs/web/enterprise/distributed-search.mdx +++ b/docs/web/enterprise/distributed-search.mdx @@ -4,17 +4,14 @@ sidebarTitle: "Distributed search" description: "How LanceDB Enterprise distributes index execution and table-data reads." icon: "network-wired" availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - LanceDB Enterprise distributes index execution and table-data reads across - executors. LanceDB Embedded executes searches in the application process. --- Distributed search spreads index execution, table-data reads, and caching across multiple serving machines in LanceDB Enterprise. Query nodes plan queries and -combine results. Executors cache index structures and table data and execute -the distributed work. See [Enterprise architecture](/enterprise/architecture). +combine results. Plan executors cache index structures and table data and execute +the remote work. See [Architecture](/#architecture). ## Embedded search {#embedded-search} diff --git a/docs/web/enterprise/index.mdx b/docs/web/enterprise/index.mdx index 888804756..59ab7e3a0 100644 --- a/docs/web/enterprise/index.mdx +++ b/docs/web/enterprise/index.mdx @@ -10,7 +10,7 @@ keywords: ["enterprise", "deployment", "cluster"] This page describes a LanceDB Enterprise capability. The embedded form of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. +[Architecture](/#architecture) for what differs between the two. LanceDB Enterprise runs the lakehouse as a service: a control plane for configuration, identity and policy; a data plane serving reads and writes; and @@ -30,4 +30,13 @@ Draft for the Functions release. Add this row to the table above, after SQL: | [Functions](/management/function) | Python running on workers | */} -See [Architecture](/architecture) for the two shapes side by side. +See [Architecture](/#architecture) for the two shapes side by side. + +## Getting a deployment {#deployment} + +Deployment is specific to your cloud and account, so the procedure is +published with LanceDB Enterprise rather than here. +[Contact us](https://www.lancedb.com/contact) to get started; see the +[Kubernetes deployment guide](/enterprise/deployment/kubernetes) for a worked example. +Embedded LanceDB has nothing to deploy: it is a library your process links +against -- see [LanceDB Embedded](/#1-lancedb-oss). diff --git a/docs/web/enterprise/security.mdx b/docs/web/enterprise/security.mdx index c848be23f..b8d9d72c5 100644 --- a/docs/web/enterprise/security.mdx +++ b/docs/web/enterprise/security.mdx @@ -10,7 +10,7 @@ keywords: ["compliance", "security", "soc2", "encryption", "audit"] This page describes a LanceDB Enterprise capability. The embedded form of LanceDB has no server to provide it. See -[Architecture](/architecture) for what differs between the two. +[Architecture](/#architecture) for what differs between the two. Compliance posture (certifications, data residency, encryption at rest and in transit, audit logging) is a property of a deployment, so it is documented diff --git a/docs/web/feature-engineering/add-computed-column.mdx b/docs/web/feature-engineering/add-computed-column.mdx index 49820a454..aecb0acf0 100644 --- a/docs/web/feature-engineering/add-computed-column.mdx +++ b/docs/web/feature-engineering/add-computed-column.mdx @@ -1,18 +1,33 @@ --- -title: "Add a computed column" -sidebarTitle: "Computed Columns" +title: "Create a computed column" +sidebarTitle: "Create Computed Column" description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it." icon: table-columns keywords: ["computed columns", "add_columns", "transforms", "schema"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Local computed columns require a source build with the experimental-functions Cargo feature. Enterprise requires the deployment to enable them. --- -A computed column is declared with an expression and starts with no values. -Declaring it is cheap on any table size, because nothing is evaluated until +import { + PySchemaAddSetup, + TsSchemaAddSetup, + RsSchemaAddSetup, + PyAddColumnsCalculated, + TsAddColumnsCalculated, + RsAddColumnsCalculated, +} from '/snippets/tables.mdx'; + +Lance can write a new column without rewriting the columns already in the +table. That matters most on a table that also holds large images, video or +embeddings, where rewriting everything just to add one small derived field +would be wasteful. See [Lance format](/basics/lance) for the storage model +this relies on. + +A computed column is how LanceDB formalizes that capability for feature +engineering specifically: instead of writing the derived values immediately, +it is declared with an expression and starts with no values. Declaring it is +cheap on any table size, because nothing is evaluated until [a refresh](/feature-engineering/refresh-column) asks for it. @@ -56,6 +71,69 @@ including that you can travel back to before the column existed. `computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is large enough that you would rather control when the work happens. +Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` alongside +`computed` to have the server keep the column refreshed instead of calling +[`refresh_column`](/feature-engineering/refresh-column) yourself: + +```python +table.add_columns( + computed={"doubled": "x * 2"}, + auto_refresh_enabled=True, + auto_refresh_interval_seconds=300, +) +``` + +A local table stores these settings but does not act on them -- automatic +scheduling runs on LanceDB Cloud or Enterprise. See +[Automatic refresh](/feature-engineering/refresh-column#automatic) for the +full behavior, including how to change the setting later. + +An eager `transforms` call looks like this, starting from a small product +table: + + + + {PySchemaAddSetup} + + + + {TsSchemaAddSetup} + + + + {RsSchemaAddSetup} + + + + + + {PyAddColumnsCalculated} + + + + {TsAddColumnsCalculated} + + + + {RsAddColumnsCalculated} + + + +For feature engineering on large existing tables, group related derived +features into one `add_columns` call instead of running many separate +writes: this creates one new table version for the schema change and +computes every new column from the existing rows in it, rather than growing +the table's version history with many small updates. + + +`add_columns` does not accept Python callables, batch UDFs, or PyArrow +`RecordBatch` iterators for populating new columns, whether through +`transforms` or `computed`. New column values must be defined with SQL +expressions, or added as NULL columns from an Arrow field or schema. If your +transformation cannot be expressed in SQL, compute the values outside +`add_columns` before writing them back through another workflow. + + ## The expression {#expressions} The expression is SQL, evaluated over the row. It can reference any column in @@ -111,12 +189,35 @@ image decode -- the column is filled by a registered [function](/management/function) instead of an expression. Declaring it looks the same; what differs is what fills it. It needs a deployment with Functions enabled. + +The call passed to `add_columns` is the tell: a SQL expression in `computed` +is a string, keyed by the column name -- + + +```python Python icon="python" +table.add_columns(computed={"caption_length": "length(caption)"}) +``` + + +-- while a Function application is the registered version itself, called with +its parameters bound to source columns through `col(...)`, keyed the same way +but passed positionally rather than through `computed`: + + +```python Python icon="python" +table.add_columns({"caption": caption_version(jpeg=col("jpeg"))}) +``` + + +Both accept the same `auto_refresh_enabled` and `auto_refresh_interval_seconds` +options, and both refresh and toggle the same way; see +[Refresh a computed column](/feature-engineering/refresh-column#automatic). */} ## Related guides {#next} - + Fill the rows that lack values. diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx deleted file mode 100644 index 37e396b8b..000000000 --- a/docs/web/feature-engineering/data-evolution.mdx +++ /dev/null @@ -1,134 +0,0 @@ ---- -title: "Data evolution" -sidebarTitle: "Data Evolution" -description: "Derive new columns from data already in the table, without rewriting what is already there." -icon: wand-magic-sparkles -keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"] -availability: - oss: available - enterprise: available - summary: >- - Adding columns computed from SQL expressions works in the embedded library - and in LanceDB Enterprise. ---- - -import { - PySchemaAddSetup, - TsSchemaAddSetup as TsSchemaAddSetup, - RsSchemaAddSetup as RsSchemaAddSetup, - PyAddColumnsCalculated as AddColumnsCalculated, - TsAddColumnsCalculated as TsAddColumnsCalculated, - RsAddColumnsCalculated as RsAddColumnsCalculated, - PyAddFeatureColumnsSql as AddFeatureColumnsSql, - TsAddFeatureColumnsSql as TsAddFeatureColumnsSql, - RsAddFeatureColumnsSql as RsAddFeatureColumnsSql, -} from '/snippets/tables.mdx'; - -Add derived columns to calculate features from data already in a table. -For example, compute a product score from existing fields without reading -and rewriting the table in your application. - -Lance can write the new columns without rewriting the existing columns. -This is useful when the table also contains large images, video or embeddings. -See [Lance format](/basics/lance) for the storage model. - - -In SQL, on an Enterprise cluster that enables computed columns, this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column). - - -## Set up the example table {#set-up-the-example-table} - -The examples use `db`, a connection opened as in -[Connect to LanceDB](/basics/connect), and create a small product table in it: - - - - {PySchemaAddSetup} - - - - {TsSchemaAddSetup} - - - - {RsSchemaAddSetup} - - - -## Add derived columns {#add-derived-columns} - -You can add new columns that are derived from existing data using SQL expressions. -For feature engineering on large existing tables, group related derived features into -one `add_columns` operation instead of running many separate writes. This creates one -new table version for the schema change and computes the new columns from the existing -rows, which avoids growing the table's version history with many small updates. - - - - {AddColumnsCalculated} - - - - {TsAddColumnsCalculated} - - - - {RsAddColumnsCalculated} - - - -The same call can add multiple derived columns at once. For example, if you are -building several lightweight features from existing product fields, pass all of the -new column expressions together: - - - - {AddFeatureColumnsSql} - - - - {TsAddFeatureColumnsSql} - - - - {RsAddFeatureColumnsSql} - - - - -LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or -PyArrow `RecordBatch` iterators for populating new columns. New column values must be -defined with SQL expressions, or added as NULL columns from an Arrow field or schema. -If your transformation cannot be expressed in SQL, compute the values outside -`add_columns` before writing them back through another workflow. - - -## When SQL is not enough {#beyond-sql} - -The expressions above are evaluated once, as part of the call. For other workflows: - -- A [computed column](/feature-engineering/add-computed-column) declares the - expression and leaves the column empty, so you choose when the work happens. - Use it when you want to schedule the computation separately. -- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for - values SQL cannot express: an embedding, a model call, an image decode. - -{/* -Draft for the Functions release. Replaces the Geneva item in the list above: -- A [function](/management/function) fills a column with Python, for values SQL - cannot express -- an embedding, a model call, an image decode. -*/} - -A deployment has to enable computed columns before you use them. -{/* Draft for the Functions release: Functions need enabling there too. */} - -## Related guides {#next} - - - - Declare now, fill later. - - - Change the table's shape instead of its contents. - - diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 30a165af4..6807998ec 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -24,7 +24,7 @@ then refresh the derived data when its inputs change. */} -In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it. +In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH COLUMN](/sql/dml/refresh-column) to fill it. ## Computed columns and materialized views {#the-shape} @@ -36,8 +36,8 @@ refresh costs: | | Computed column | Materialized view | | --- | --- | --- | | Declaring or creating it | records the expression; no values are written | runs the query and stores its rows, unless created empty | -| A refresh | fills rows with no value and recomputes rows whose inputs changed | applies source changes or rebuilds the view, depending on the query and SDK version | -| A refresh when nothing changed | fills nothing | can be a no-op when incremental refresh is supported | +| A refresh | fills rows with no value and recomputes rows whose inputs changed | rebuilds the view completely from its source query | +| A refresh when nothing changed | fills nothing | still rebuilds the view and commits a new version | | In LanceDB Enterprise | runs as a [job](/basics/jobs) | runs as a server-side materialization job | ### Computed columns fill incrementally {#incremental} @@ -65,21 +65,19 @@ up-to-date features. A [materialized view](/feature-engineering/materialized-views) has no separate declaration step: creating one runs its query and stores the result, so the view -is ready to read. A refresh updates those stored results. Python SDK 0.40.0 can apply -changes incrementally for supported queries; other queries require a full rebuild. -See [refresh behavior](/feature-engineering/materialized-views#incremental) for -version-specific details. +is ready to read. A refresh rebuilds the stored result completely from the +source query; see [Refresh a materialized view](/feature-engineering/refresh-materialized-view) +for the full behavior, including automatic refresh. -Enterprise runs materialization on the Enterprise service. Use the SDK and refresh -options supported by your server version; see -[Enterprise refreshes](/feature-engineering/materialized-views#availability). +Enterprise runs materialization on remote compute, as a server-side job. ## Choose a feature engineering workflow {#which-one} | | What it produces | Filled by | Available | | --- | --- | --- | --- | +| [View](/feature-engineering/view) | a saved query, no stored rows of its own | nothing -- it runs on every read | Enterprise | | [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise | -| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Embedded and Enterprise | +| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Enterprise | | [Geneva](/geneva) | columns filled by Python | the `geneva` package | Enterprise | {/* @@ -88,6 +86,11 @@ Draft for the Functions release. Replace the Geneva row of the table above with: | [Geneva](/geneva) | columns filled by Python, for existing pipelines | the `geneva` package | Enterprise | */} +**A view** has no refresh and no staleness window, because it stores the +query rather than its result. Reach for one first, before a computed column +or a materialized view, when the query is cheap enough to simply re-run on +every read. + **Computed columns** are the common case: a column declared with a SQL expression, filled by [a refresh](/feature-engineering/refresh-column). @@ -147,20 +150,26 @@ should handle that itself, inside the function body. When a refresh fails, fix the cause and run the refresh again. The new refresh fills what is still empty or stale. How much of the failed run was kept depends on where it ran (see -[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry +[Refresh a computed column](/feature-engineering/refresh-column#failures)), and a retry can recompute values that were already written, so do not count on it costing only the rows that did not complete. ## Related guides {#next} - + + A saved query, with no storage or refresh of its own. + + Declare a column from a SQL expression. - + Fill the rows that lack values, locally or as a job. - + + Store a query's result as a table. + + Bring a derived table up to date with its source. {/* diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index e90f3233c..86a3667f3 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -1,122 +1,95 @@ --- -title: "Refresh a materialized view" -sidebarTitle: "Materialized View Refresh" -description: "Create a materialized view and refresh it when the source data changes." +title: "Create a materialized view" +sidebarTitle: "Create Materialized View" +description: "Store the result of a query as a table, ready to query, index and search." icon: table -keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"] +keywords: ["materialized view", "create_materialized_view"] availability: - oss: available + embedded: unavailable enterprise: available - summary: >- - Creation and refresh in LanceDB Embedded require a source build with the experimental-functions Cargo feature. Enterprise runs the work on the Enterprise service. See [Enterprise refreshes](#availability) for server compatibility requirements. --- -import { - PyMaterializedViewCreate as MaterializedViewCreate, - PyMaterializedViewRefresh as MaterializedViewRefresh, -} from '/snippets/materialized_view.mdx'; - A materialized view stores the result of a query as a table. Use it to keep a -filtered or transformed dataset that you can query, index and search repeatedly. -Refresh the view after its source data changes. +filtered or transformed dataset that you can query, index and search +repeatedly, instead of recomputing the query every time you read it. -In the 0.41 SDK release, materialized-view creation and refresh in LanceDB Embedded require a -source build with the `experimental-functions` Cargo feature. Published Python -and TypeScript packages leave it disabled. Opening, reading, listing and dropping -existing views remain available, as do LanceDB Enterprise operations. +In SQL this is [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view#create). - -In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view). - +## Create a view {#create} -## Create and refresh {#create-and-refresh} + +```python Python icon="python" +view = db.create_materialized_view( + "frames_captioned", + source="frames", + select=["frame_id", "video_id", "ts", "caption"], + where="selected = true", +) +``` - -{MaterializedViewCreate} - +```typescript TypeScript icon="square-js" +const view = await db.createMaterializedView("frames_captioned", "frames", { + select: ["frame_id", "video_id", "ts", "caption"], + where: "selected = true", +}); +``` + + +`source` names the table the view reads. `select` takes column names, +`(alias, expression)` pairs, or a dict of the same; omitting it selects every +source column, expanded against the schema as it was at creation time. +`where` filters source rows, and `limit` caps the view. Creating the view computes its rows, so it is ready to read. Pass -`with_no_data=True` to create it empty instead. Later changes to the source -reach the view through a refresh: +`with_no_data=True` to create it empty instead; a later +[refresh](/feature-engineering/refresh-materialized-view) populates it. - -{MaterializedViewRefresh} - +Your administrator manages stable row IDs and other source-table +requirements for materialized views on the server; confirm support for your +source table's shape before creating a view over it. -The example works with Python SDK 0.40.0. It creates the source with stable -row IDs, which that release requires for ordinary materialized-view sources. +## Automatic refresh at creation {#auto-refresh} -`select` takes column names, `(alias, expression)` pairs, or a dict of the same. -Omitting it selects every source column, expanded against the schema as it was -at creation time. `where` filters source rows; `limit` caps the view. +Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` to schedule +refreshes instead of calling +[`refresh`](/feature-engineering/refresh-materialized-view) yourself: -## Prepare the source table {#stable-row-ids} +```python +view = db.create_materialized_view( + "frames_captioned", + source="frames", + select=["frame_id", "video_id", "ts", "caption"], + where="selected = true", + auto_refresh_enabled=True, + auto_refresh_interval_seconds=300, +) +``` -In Python SDK 0.40.0, create the source with -`storage_options={"new_table_enable_stable_row_ids": "true"}`, as the example -above does. Stable row IDs let the view track source rows across updates and -compaction. Set this option when creating the table; it cannot be added later. - -A source with the [realtime engine](/tables/realtime-engine) enabled or retained, uncompacted write-ahead-log -rows is not accepted by the local materialized-view implementation. - -## Refresh behavior {#incremental} - -With Python SDK 0.40.0, a refresh applies source changes incrementally when the -query supports it, and rebuilds the view otherwise. In the example, adding Bea -writes one new view row. Refreshing again without source changes writes none. -Use `refresh(full=True)` to force a rebuild. - -`rows_written` counts rows written by that refresh, not all rows in the view. -Use `view.table.count_rows()` to get the total. Pass `source_version=N` to -refresh against a particular source version instead of the latest. - - -Refresh behavior has changed between SDK versions. The earlier 0.40.0b13 -development build rebuilt every local view on refresh and did not accept -`full`; 0.39.0 created views empty. Use the reference for your installed SDK -when maintaining an existing pipeline. - - -## Enterprise refreshes {#availability} - -Enterprise - -Enterprise runs materialized-view creation and refresh on the Enterprise service. -For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and -[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view). - -Use an SDK version supported by your Enterprise deployment. Materialized-view -APIs and stored definitions have changed across releases; a local SDK example -alone does not establish compatibility with an older server. Before upgrading -an existing pipeline, confirm the supported refresh options and whether its -views need to be recreated with your administrator. - -Stable row IDs and feature configuration for Enterprise sources are managed on -the server. The source-table storage option in the local example does not -configure the Enterprise deployment, and Python SDK 0.40.0's `create_table` on a -`db://` connection does not accept `storage_options`: leave it out, and confirm -with your administrator that the deployment gives new tables stable row IDs. - -`list_materialized_views` requests the server's view listing. On a local -database it reads table schemas to find views. +Omit `auto_refresh_enabled` to let the server decide whether to enable it, +and omit `auto_refresh_interval_seconds` to let the server decide the +pacing. See +[Automatic refresh](/feature-engineering/refresh-materialized-view#auto-refresh) +for how scheduling behaves once the view exists. ## Working with the view {#working-with-the-view} `view.table` is the underlying table, for querying, indexing, or search. `view.definition` is the recorded query: the select list, predicate, and -limit. Use it to inspect the query without inferring it from the rows. Both are properties; on an async view, the definition is read -with `await view.definition()`. +limit. Use it to inspect the query without inferring it from the rows. Both +are properties; on an async view, the definition is read with +`await view.definition()`. -An existing view is reopened with `db.open_materialized_view(name)`. +An existing view is reopened with `db.open_materialized_view(name)`. Creation, +refresh and drop also have async, job-returning SDK methods; see +[Materialized view](/management/materialized-view#manage). ## Related guides {#next} - - Derive a column instead of a whole table. + + Bring the view up to date with its source. Inspect the table history and read a particular version. diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index 10ab9fdc8..ba3934f1e 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -1,14 +1,12 @@ --- -title: "Refresh a column" -sidebarTitle: "Column Refresh" +title: "Refresh a computed column" +sidebarTitle: "Refresh Computed Column" description: "Populate a computed column and update it when its inputs change." icon: arrows-rotate keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Local column refresh requires a source build with the experimental-functions Cargo feature. Enterprise runs it as a server job that must be enabled for your Enterprise cluster. --- Refresh a [computed column](/feature-engineering/add-computed-column) to fill @@ -64,10 +62,6 @@ That is what makes the call idempotent and safe to retry. However, unlike a spreadsheet formula, nothing is recomputed until you refresh, so a derived column is stale between an update and the next refresh. -{/* -Draft for automatic computed-column refresh. Keep the released manual-refresh -instructions above until the server feature is available. - ## Automatic refresh {#automatic} Set scheduling options when declaring a computed column: @@ -80,17 +74,17 @@ table.add_columns( ) ``` -Both options apply to every computed output declared in that call, including -registered Function outputs. Automatic refresh is enabled by default. When -the interval is omitted, the server determines when to refresh. LanceDB Embedded -tables persist only an explicitly configured interval; automatic execution requires a -LanceDB Enterprise deployment with its separate `column_refresh.enabled` switch -turned on. That server switch is disabled in the base configuration. +These options apply to every computed output declared in that call. Automatic +refresh is enabled by default. When the interval is omitted, the server +determines when to refresh. Local tables store the setting, but automatic +execution requires a Cloud or Enterprise server with its separate +`column_refresh.enabled` switch turned on. That server switch is disabled in +the base configuration. The interval is a positive minimum delay after the last successful refresh of that output. A first refresh is immediately eligible. The scheduler still checks input freshness and waits for computed ancestors to settle; the timer -does not make unchanged inputs require recomputation. Each Function output's +does not make unchanged inputs require recomputation. Each declared output's interval is independent, so refreshing a sibling does not postpone it. Pacing uses retained live Job Registry history. If completed jobs are archived @@ -107,10 +101,31 @@ ALTER TABLE my_table ALTER COLUMN doubled SET auto_refresh_interval_seconds = 60 ALTER TABLE my_table ALTER COLUMN doubled SET auto_refresh_enabled = true; ``` -Disabling automatic refresh prevents new automatic jobs. Inflight jobs may -finish, and manual refresh remains available. Re-enabling wakes maintenance -to consider accumulated changes under the interval and dependency rules. -*/} + +The current setting is in the column's own field metadata, +`computed_column.auto_refresh_enabled`, readable from `table.schema` without +submitting a job to ask. + + +Setting the flag is a metadata-only change and always succeeds, independent of +whether the scheduler can actually refresh the column -- a deployment issue +that keeps the background job from running (for example, a worker that cannot +reach object storage) does not stop the `ALTER` from committing. Check the +column's job history, not just its metadata, if values stop appearing after +re-enabling. + +**Disabling does not skip or drop writes.** Rows written while disabled +commit with no computed value, the same as a row written before the column +existed; they do not queue anywhere separate. Inflight jobs may finish, and +manual refresh remains available throughout. + +**Re-enabling catches up the entire backlog in one pass**, not just rows +written after that point: the next scheduler tick processes every row left +unfilled since the column was disabled, in a single refresh, with no +further write required to trigger it. This is "catch up," not "resume +cadence" -- a column disabled for an hour and re-enabled fills everything +that accumulated during that hour, immediately, rather than only picking up +new rows from the moment it is re-enabled. ## Refreshing part of a table {#partial} @@ -196,7 +211,7 @@ symptoms, which rarely name their cause. Waiting on, inspecting, and cancelling server-side work. - + The same idea, for a whole derived table. diff --git a/docs/web/feature-engineering/refresh-materialized-view.mdx b/docs/web/feature-engineering/refresh-materialized-view.mdx new file mode 100644 index 000000000..05ebff109 --- /dev/null +++ b/docs/web/feature-engineering/refresh-materialized-view.mdx @@ -0,0 +1,88 @@ +--- +title: "Refresh a materialized view" +sidebarTitle: "Refresh Materialized View" +description: "Bring a materialized view up to date after its source data changes." +icon: arrows-rotate +keywords: ["materialized view", "refresh", "auto_refresh_enabled", "auto_refresh_interval_seconds"] +availability: + embedded: unavailable + enterprise: available +--- + +Refresh a [materialized view](/feature-engineering/materialized-views) to +bring its stored rows up to date after the source table changes. + + +In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh-materialized-view). + + +## Refresh a view {#refresh} + +```python +view.refresh() +``` + +Every refresh rebuilds the view completely from its source query -- there is +no incremental refresh. Calling it again with no source changes still +rebuilds and commits a new table version. + +The result reports what the refresh did: + +```python +result = view.refresh() +result.rows_written # rows in the rebuilt view +result.source_version # the source table version the view now reflects +result.version # the view's own new table version +``` + +Pass `source_version=N` to refresh against a particular source version +instead of the latest: + +```python +view.refresh(source_version=12) +``` + +Use `view.table.count_rows()` for the view's row count at any time; it +always matches `rows_written` from the most recent refresh. + +## Automatic refresh {#auto-refresh} + +Set scheduling options when [creating the view](/feature-engineering/materialized-views#auto-refresh): + +```python +view = db.create_materialized_view( + "frames_captioned", + source="frames", + select=["frame_id", "video_id", "ts", "caption"], + where="selected = true", + auto_refresh_enabled=True, + auto_refresh_interval_seconds=300, +) +``` + +`auto_refresh_interval_seconds` is a positive minimum delay after the last +successful refresh before the next automatic one is eligible; omit it to let +the server decide. The whole view toggles together -- there is no +per-column granularity the way a computed column has one setting per output. + +The view's *initial* population, at creation or when the server first runs +it, happens regardless of these settings; only refreshes after that are +gated. Manual `view.refresh()` keeps working whether or not automatic +refresh is enabled. + +## When a refresh fails {#failures} + +A refresh replaces the view's stored rows in one commit, so a failure leaves +the view as it was after its last successful refresh. Fix the cause and +refresh again. + +## Related guides {#next} + + + + Define the query a view stores. + + + The same idea, for a single derived column. + + diff --git a/docs/web/feature-engineering/view.mdx b/docs/web/feature-engineering/view.mdx new file mode 100644 index 000000000..e61b2118a --- /dev/null +++ b/docs/web/feature-engineering/view.mdx @@ -0,0 +1,73 @@ +--- +title: "Create a view" +sidebarTitle: "Create View" +description: "Define a derived dataset as a saved query, with no storage or refresh of its own." +icon: eye +keywords: ["view", "create view", "feature engineering", "saved query"] +availability: + embedded: unavailable + enterprise: available +--- + +A view is the lightest way to derive a feature-engineered dataset: it stores +a **query**, not rows, so there is nothing to refresh and nothing that can +fall out of date. Reach for one before +[a computed column](/feature-engineering/add-computed-column) or +[a materialized view](/feature-engineering/materialized-views) when the +underlying query is cheap enough to simply re-run on every read. + +## Define a view {#create} + +A view over a products table, selecting and deriving a few columns: + +```sql +CREATE VIEW "discounted_products" AS +SELECT id, name, price, price * 0.9 AS sale_price +FROM "products" +WHERE in_stock = true +``` + +The statement resolves the view's schema from the query at creation time. +Anything the [SQL dialect](/sql/functions) supports can define one: joins, +predicates, aggregates, window functions -- the same expressiveness as +[a computed column's](/feature-engineering/add-computed-column#expressions) +SQL expression, just over a whole derived table instead of one column. + +## Read it like a table {#reading} + +```sql +SELECT * FROM "discounted_products" WHERE sale_price < 20 +``` + +Every read runs the saved query against current data, so `sale_price` +reflects whatever `price` is right now -- there is no staleness window to +reason about, unlike a materialized view between refreshes. A predicate +applied when reading the view composes with the query inside it, rather than +filtering the view's output afterward, so a selective filter does not pay +for rows the view would have produced and then discarded. + +## When to use a view instead {#choosing} + +| | View | Computed column / materialized view | +| --- | --- | --- | +| Stores | the query | derived values | +| Freshness | always current | as of the last refresh | +| Read cost | the full query, every read | a column or table scan | +| Write cost | none | a refresh | + +Reach for a view when the query is cheap, or when reading a stale value would +be wrong for the workload. Reach for a computed column or a materialized view +when the query is expensive and the data changes less often than it is read. +See [View](/sql/ddl/view#choosing) for the full SQL reference comparison, +including the separate privileges each one needs. + +## What to read next {#next} + + + + CREATE VIEW, DROP VIEW, and listing views alongside tables. + + + The stored-rows counterpart, and when a refresh is worth paying for. + + diff --git a/docs/web/governance/authorization.mdx b/docs/web/governance/authorization.mdx index 72e4e9534..c7831590b 100644 --- a/docs/web/governance/authorization.mdx +++ b/docs/web/governance/authorization.mdx @@ -5,10 +5,8 @@ description: "Who may do what: privileges granted on catalog objects to principa icon: shield-halved keywords: ["authorization", "authz", "privileges", "rbac", "access control"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports catalog privileges when its administrator configures enforcement. LanceDB Embedded has no catalog authorization layer: access is whatever the object store allows. --- [Authentication](/enterprise/authentication) establishes who is calling. diff --git a/docs/web/index.mdx b/docs/web/index.mdx index 11b72505d..3cd157897 100644 --- a/docs/web/index.mdx +++ b/docs/web/index.mdx @@ -19,29 +19,6 @@ Training data and experimentation slow down when raw data, metadata, embeddings, artifacts live in separate systems. LanceDB keeps them together in one versioned multimodal table, so AI teams spend less time stitching infrastructure together and more time improving datasets, testing features, and keeping GPUs fed. -![Training data lifecycle: Curation, Feature Engineering, Search and Retrieval, Training](/static/assets/images/overview/training-data-lifecycle.svg) - -Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs. -Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble -GPU-ready batches from a tagged dataset version. - -For a deeper look at how this works in training pipelines, start with [Why LanceDB for training](/training/why-lancedb). - -## LanceDB suite {#lancedb-suite} - -The LanceDB suite includes LanceDB Embedded (open source), a retrieval library, and LanceDB Enterprise, -a multimodal lakehouse platform for the full AI data lifecycle. -LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage. -It is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built -for teams that need scale without building bespoke infrastructure for curation, -feature engineering, search and retrieval, and efficient training data access. -LanceDB Enterprise runs LanceDB Distributed: query execution, indexing and maintenance -run on the Enterprise service, outside your application. - -![LanceDB suite: LanceDB Embedded search and Enterprise multimodal lakehouse on Lance format](/static/assets/images/overview/lancedb-suite.svg) - -## Why teams use LanceDB {#why-teams-use-lancedb} - Store images, video, audio, text, annotations, embeddings, and model-generated features together in one schema-enforced table. @@ -58,70 +35,149 @@ run on the Enterprise service, outside your application. +Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs. +Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble +GPU-ready batches from a tagged dataset version. Training pipelines need curation, derived features, reproducible +splits, fast random access, and a clean path into frameworks such as PyTorch -- LanceDB keeps all of it connected +through the same table model, whether a workflow uses one table or several related ones. + +## Architecture {#architecture} + +LanceDB Embedded and Enterprise are both built on [Lance](/basics/lance), which stores table data, versions, and +indexes in open, columnar files. The format and table model are the same across both offerings -- what changes is +how storage and compute are organized around them. + +```mermaid +%%{init: {"flowchart": {"curve": "basis", "nodeSpacing": 45, "rankSpacing": 55, "padding": 12}}}%% +flowchart TB + subgraph Embedded["LanceDB Embedded"] + EmbApp(["Your application
+ LanceDB SDK"]) + end + + subgraph Enterprise["LanceDB Enterprise"] + EntSdk(["LanceDB SDK"]) --> EntEngine{{"Enterprise Distributed Engine"}} + end + + Lance[("Lance format
table data, versions, indexes")] + + Embedded --> Lance + Enterprise --> Lance +``` + +Embedded reads and writes tables on local disk or in object storage from inside your application process. +Enterprise keeps durable table data and index artifacts in object storage, independently of the machines serving +requests, and separates query serving, read execution and caching, and indexing and background work into their +own independently sized components. Caches accelerate reads; they are never the durable source of truth. With +multiple serving replicas, Enterprise routes traffic to surviving nodes when one fails; losing a cache node +temporarily raises latency while its replacement warms, rather than causing an outage. + +Sharing a format does not mean an Enterprise deployment automatically opens an existing Embedded database -- +plan how to move or connect your data and configure access for your deployment. + +## Embedded and Enterprise {#embedded-and-enterprise} + +LanceDB ships as two offerings on the same Lance format and table model. Start locally with Embedded, then move to +Enterprise when your team needs distributed scale, managed infrastructure, private deployment, or higher-throughput +curation, feature engineering, search and retrieval, and training workflows. + +### 1\. LanceDB Embedded {#1-lancedb-oss} + +**LanceDB Embedded** is an open-source, embedded retrieval library, with client SDKs in Python, TypeScript and +Rust -- the application itself plans queries, builds indexes, and runs background work, with no separate database +server to deploy. This fits local retrieval applications, notebooks, development, and single-node data pipelines, +where your application controls its own compute resources and [table maintenance](/maintenance/compaction). + +### 2\. LanceDB Enterprise {#2-lancedb-enterprise} + +[LanceDB Enterprise](/enterprise) is a petabyte-scale (and beyond), distributed **multimodal lakehouse** platform +built for search, curation, feature engineering, and high-throughput training data access workflows on top of the +same core table abstraction, without building bespoke infrastructure. Applications connect through a LanceDB SDK +while the service handles requests across a fleet, with query serving, read execution and caching, and indexing +and background work sized and scaled as separate components. A control plane governs configuration, identity +and cluster lifecycle, separately from the data-plane components in the diagram below. See +[Distributed search](/enterprise/distributed-search) for how query nodes and plan executors divide a request. + +```mermaid +%%{init: {"flowchart": {"curve": "basis", "nodeSpacing": 45, "rankSpacing": 55, "padding": 12}}}%% +flowchart TB + Clients(["Clients"]) + QueryFleet{{"Query Fleet
query serving"}} + ExecutorRing{{"Executor Ring
read execution + cache"}} + MonitoringFleet{{"Monitoring Fleet"}} + JobWorkerFleet{{"Job Worker Fleet
indexing, column + MV refresh, compaction"}} + Storage[("Cloud Object Storage")] + + Clients --> QueryFleet + QueryFleet -- reads --> ExecutorRing + QueryFleet -- writes --> Storage + QueryFleet -- notifies --> MonitoringFleet + MonitoringFleet -- creates jobs --> JobWorkerFleet + ExecutorRing -- reads --> Storage + JobWorkerFleet -- writes --> Storage +``` + +Teams choose Enterprise to: + +- **Serve growing workloads** -- add query-serving capacity and let Enterprise cache frequently read table data near + compute as traffic grows. +- **Reduce operational work** -- let the service handle background [compaction](/maintenance/compaction) and + [cleanup](/maintenance/cleanup) instead of scheduling them in application code. +- **Fit deployment requirements** -- run a managed deployment or bring LanceDB into your own cloud account, with + OpenTelemetry metrics and traces. +- **Connect data workflows** -- use the same tables for retrieval, curation, and training data access, with + [SQL](/basics/sql), [feature engineering](/feature-engineering), and [Geneva](/geneva) feature backfills. + +To set up LanceDB Enterprise in your organization, [reach out to us](https://www.lancedb.com/contact). + +### Compare features {#compare} + +{/* availability-comparison */} + +## Projects using LanceDB {#projects-using-lancedb} + +### Training {#training} + +- [stable-worldmodel](https://github.com/galilai-group/stable-worldmodel) -- a platform for reproducible world-model + research built on a LanceDB data layer, reporting 3-4x faster data loading on Push-T versus HDF5 / MP4 at a + fraction of the disk footprint. +- [le-wm](https://github.com/lucas-maes/le-wm) -- a joint-embedding predictive world model from pixels, trained on + the stable-worldmodel platform and its LanceDB data layer. +- [lerobot-lancedb](https://github.com/lancedb/lerobot-lancedb) -- a drop-in LanceDB backend for Hugging Face + LeRobot datasets with faster loading across robotics datasets. + +### Agentic applications {#agentic-applications} + +- [OpenClaw LanceDB Plugin](https://docs.openclaw.ai/plugins/memory-lancedb) -- stores an agent's long-term memory + in LanceDB, automatically recalling relevant memories before a model turn and capturing new facts afterward. +- [Cognee](https://github.com/topoteretes/cognee) -- an open-source AI memory platform that gives agents persistent + memory across sessions, using LanceDB as one of its vector store backends. +- [CrewAI](https://github.com/crewAIInc/crewAI) -- a multi-agent framework that + [rebuilt its agent memory on LanceDB](https://www.lancedb.com/blog/crewai-rebuilt-agent-memory-on-lancedb). + +Across these projects, LanceDB and Lance provide the multimodal data layer that keeps raw observations, annotations, +features, and training or memory access patterns in one format instead of scattering them across task-specific +stores. + ## Start with your workload {#start-with-your-workload} - - Learn why LanceDB works well as the data layer for training workloads. - - - Use LanceDB tables and permutations for projected, shuffled, random-access training reads. - - - Explore Lance-formatted multimodal datasets with raw bytes, metadata, embeddings, and indices. + + Create tables, append records, and update or delete rows as your data changes. Use vector search, full-text search, hybrid search, reranking, filtering, and SQL. + + Profile, sample, chunk, join, and deduplicate records before training on them. + + + Derive computed columns and materialized views without moving data elsewhere. + + + Use LanceDB tables and permutations for projected, shuffled, random-access training reads. + + + Create, list, and drop tables, namespaces, and views from Python, TypeScript, or SQL. + -## From local development to production scale {#from-local-development-to-production-scale} - -LanceDB Embedded and LanceDB Enterprise share the same Lance format and table model. Start locally with -LanceDB Embedded, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment, -or higher-throughput curation, feature engineering, search and retrieval, and training workflows. - -### 1\. LanceDB Embedded {#1-lancedb-oss} -The fastest way to get started is the open-source embedded library, with client SDKs in Python, TypeScript -and Rust. Run it locally in just a few steps, which lets you explore datasets, curate data, and run search and retrieval workloads -for agents. Start here: - - - - Get started with LanceDB in minutes. - - - Create tables, evolve schemas, version data, and modify rows in LanceDB. - - - -### 2\. LanceDB Enterprise {#2-lancedb-enterprise} - -[LanceDB Enterprise](/enterprise) is a petabyte-scale (and beyond), distributed **multimodal lakehouse** platform built for -search, curation, feature engineering, and high-throughput training data access workflows on top of the same core table -abstraction. This eliminates the need for teams to build bespoke infrastructure to manage large multimodal datasets. -To set up LanceDB Enterprise in your organization, -[reach out to us](https://www.lancedb.com/contact). - - -**Built with scale, performance, and security in mind.** - -LanceDB Enterprise is designed for very large-scale, high-performance, distributed workloads in -private deployments, and can operate under strict [security requirements](/enterprise/security). - - - - Get started with LanceDB in minutes, including Enterprise `db://` connections. - diff --git a/docs/web/indexing/fts-index.mdx b/docs/web/indexing/fts-index.mdx index f112c996f..ac3ff36e5 100644 --- a/docs/web/indexing/fts-index.mdx +++ b/docs/web/indexing/fts-index.mdx @@ -4,12 +4,8 @@ sidebarTitle: "FTS Index" description: "Create and tune BM25-based full-text search indexes in LanceDB." icon: "book" availability: - oss: available + embedded: available enterprise: available - summary: >- - Full-text indexes are built in the embedded library and in LanceDB Enterprise. - The embedded library builds the index before the call returns; a - deployment builds it in the background. --- import { PyFtsIndexAsync as FtsIndexAsync, PyFtsIndexCreate as FtsIndexCreate, PyFtsIndexNested as FtsIndexNested, PyFtsIndexWait as FtsIndexWait } from '/snippets/indexing.mdx'; diff --git a/docs/web/indexing/index.mdx b/docs/web/indexing/index.mdx index 78844527f..e408acb78 100644 --- a/docs/web/indexing/index.mdx +++ b/docs/web/indexing/index.mdx @@ -4,11 +4,8 @@ sidebarTitle: "Overview" description: "Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more." icon: "list" availability: - oss: available + embedded: available enterprise: available - summary: >- - Indexes are built in the embedded library and in LanceDB Enterprise. Enterprise builds them in the background, and builds fewer vector index types than the - embedded library. --- Indexes speed up searches and filters by organizing data for specific query diff --git a/docs/web/indexing/reindexing.mdx b/docs/web/indexing/reindexing.mdx index 27a51a3d3..68d0f5d73 100644 --- a/docs/web/indexing/reindexing.mdx +++ b/docs/web/indexing/reindexing.mdx @@ -5,13 +5,8 @@ description: "Learn how to keep your indexes up-to-date in LanceDB using increme icon: "refresh" keywords: ["reindexing", "re-index", "incremental indexing", "reindex"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Indexes are kept up to date in the embedded library and in LanceDB Enterprise. In - the embedded library you add new rows to them with `optimize()`; a - deployment does it in the background, and `optimize()` is not available - there. --- import { diff --git a/docs/web/indexing/scalar-index.mdx b/docs/web/indexing/scalar-index.mdx index 9f1f9a176..d99bb6003 100644 --- a/docs/web/indexing/scalar-index.mdx +++ b/docs/web/indexing/scalar-index.mdx @@ -4,12 +4,8 @@ sidebarTitle: "Scalar Index" description: "Learn how to use scalar indexes in LanceDB for efficient metadata filtering and query optimization." icon: "tree" availability: - oss: available + embedded: available enterprise: available - summary: >- - Scalar indexes are built in the embedded library and in LanceDB Enterprise. A - deployment builds them in the background and keeps most of them up to date - itself; in the embedded library you update them with `optimize()`. --- import { diff --git a/docs/web/indexing/vector-index.mdx b/docs/web/indexing/vector-index.mdx index 33d6a8575..3e2b7d6f0 100644 --- a/docs/web/indexing/vector-index.mdx +++ b/docs/web/indexing/vector-index.mdx @@ -4,13 +4,8 @@ sidebarTitle: "Vector Index" description: "Build and manage LanceDB vector indexes." icon: "arrow-up-right-dots" availability: - oss: available + embedded: available enterprise: available - summary: >- - Vector indexes are built in the embedded library and in LanceDB Enterprise. The - embedded library builds an index before `create_index()` returns; a - deployment builds it in the background, keeps it up to date, and builds - only `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and `IVF_HNSW_SQ`. --- import { diff --git a/docs/web/ingestion/create-table.mdx b/docs/web/ingestion/create-table.mdx index 00425a1f9..ec13c02b7 100644 --- a/docs/web/ingestion/create-table.mdx +++ b/docs/web/ingestion/create-table.mdx @@ -5,10 +5,8 @@ description: "Create a table from records or a schema and choose input formats." icon: table keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Create and open tables in LanceDB Embedded and LanceDB Enterprise. --- import { diff --git a/docs/web/ingestion/delete.mdx b/docs/web/ingestion/delete.mdx index f02320733..2b961fc4b 100644 --- a/docs/web/ingestion/delete.mdx +++ b/docs/web/ingestion/delete.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Delete" description: "Select and delete records, verify the remaining rows, and understand storage retention." icon: trash availability: - oss: available + embedded: available enterprise: available - summary: >- - Delete rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Storage reclamation follows compaction and retention policy. --- import { RsExample as RsDeleteRows } from '/snippets/ingestion_delete.mdx'; diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx index 9eb453cd6..28cfce650 100644 --- a/docs/web/ingestion/index.mdx +++ b/docs/web/ingestion/index.mdx @@ -5,10 +5,8 @@ description: "Choose how to create a table, append records or change existing da icon: inbox keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Create tables and write records through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also provides SQL write statements. --- import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx'; @@ -32,7 +30,7 @@ with a stable key when the incoming data should update existing records. ## Connect and prepare the data {#connect-to-lancedb} -Use the [connection guide](/basics/connect) to connect to LanceDB Embedded or LanceDB Enterprise. +Use the [connection guide](/basics/catalog) to connect to Embedded or Enterprise. The SDK write operations in this section work with both. Enterprise also supports [SQL write statements](/sql#dml); an SDK filter expression does not require a separate SQL connection. diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx index 2c5823f4e..cd15bc65b 100644 --- a/docs/web/ingestion/insert.mdx +++ b/docs/web/ingestion/insert.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Insert" description: "Append records, load file-backed data or batches, and verify ingestion." icon: plus availability: - oss: available + embedded: available enterprise: available - summary: >- - Append records through the SDKs in LanceDB Embedded and LanceDB Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table). --- import { diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx index 74b161c29..a9ad3071c 100644 --- a/docs/web/ingestion/merge-insert.mdx +++ b/docs/web/ingestion/merge-insert.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Merge Insert" description: "Match incoming records by key and choose which rows to update, insert or remove." icon: code-merge availability: - oss: available + embedded: available enterprise: available - summary: >- - Merge insert is available through the SDKs in LanceDB Embedded and LanceDB Enterprise. Use it for key-based updates and inserts. --- import { diff --git a/docs/web/ingestion/realtime-write.mdx b/docs/web/ingestion/realtime-write.mdx new file mode 100644 index 000000000..bed77eed5 --- /dev/null +++ b/docs/web/ingestion/realtime-write.mdx @@ -0,0 +1,196 @@ +--- +title: "Real Time writes" +sidebarTitle: "Real Time Write" +description: "Enable the real time engine on a table and absorb small, frequent writes cheaply, each one committed in milliseconds." +icon: "bolt" +keywords: ["rte", "real time engine", "wal", "memwal", "lsm", "write-ahead log", "streaming writes", "upsert"] +availability: + embedded: unavailable + enterprise: available +--- + +The [real time engine](/tables/realtime-engine) puts an LSM-style write tier in +front of a table, turning many small writes into a few large commits. This +page covers enabling it and writing to a table; for querying an RTE table, see +[Real Time reads](/search/realtime-read). + +## Getting started {#getting-started} + +### 1. Create a table with a primary key {#primary-key} + +RTE deduplicates writes by primary key. Create the table with a non-nullable +key column, then declare it with `set_unenforced_primary_key`. The key is a +single `int32`, `int64`, string, or binary column, and cannot be changed once +set. + +The table below holds frames captioned and embedded as they stream in from a +live camera feed, such as a robot recording episodes into a dataset shaped +like [LeRobotDataset](/integrations/lerobotdataset)'s `frames` table — +`frame_id` is the primary key so a re-captioned frame upserts in place: + + +```python Python icon="python" +from datetime import timedelta + +import lancedb +import numpy as np +import pyarrow as pa +from lancedb import RTEWriteSpec +from lancedb.index import FTS, BTree, IvfPq + +db = lancedb.connect("db://my_db", api_key="...", host_override="https://...") + +dim = 768 +schema = pa.schema([ + pa.field("frame_id", pa.int64(), nullable=False), + pa.field("video_id", pa.utf8(), nullable=False), + pa.field("embedding", pa.list_(pa.float32(), dim)), + pa.field("caption", pa.utf8()), +]) +frames = db.create_table("frames", schema=schema) +frames.set_unenforced_primary_key("frame_id") +``` + + +Or in SQL: + + +```sql SQL +ALTER TABLE frames SET UNENFORCED PRIMARY KEY (frame_id) +``` + + +### 2. Create indexes {#indexes} + +By default, RTE maintains every supported index on the table, including +indexes created after it is enabled. A scalar index on the primary key is +required for partial updates and deletes. + + +```python Python icon="python" +frames.create_index("embedding", config=IvfPq()) +frames.create_index("caption", config=FTS()) +frames.create_index("frame_id", config=BTree()) +frames.wait_for_index(["embedding_idx", "caption_idx", "frame_id_idx"], timeout=timedelta(minutes=5)) +``` + + +### 3. Enable RTE {#enable} + + +```python Python icon="python" +frames.enable_rte(RTEWriteSpec.unsharded()) +``` + + +Or in SQL: + + +```sql SQL +ALTER TABLE frames SET RTE +``` + + +### 4. Write {#write} + + +On an RTE table, an upsert through `merge_insert` is the only way to write +rows. Use the form below exactly: `when_matched_update_all()` with no filter, +plus `when_not_matched_insert_all()`. Other `merge_insert` forms, such as +conditional updates or deleting rows missing from the source, are rejected. + + +Writes go through `merge_insert` on the primary key, and are visible to queries +immediately — see [Real Time reads](/search/realtime-read) for querying them +back: + + +```python Python icon="python" +new_frames = pa.table({ + "frame_id": pa.array([2000, 2001], pa.int64()), + "video_id": pa.array(["clip-9", "clip-9"]), + "embedding": pa.FixedSizeListArray.from_arrays( + pa.array(np.random.rand(2 * dim).astype("float32")), dim), + "caption": pa.array(["a gripper closing on a cup", "a cup set down on the table"]), +}, schema=schema) + +frames.merge_insert("frame_id") \ + .when_matched_update_all() \ + .when_not_matched_insert_all() \ + .execute(new_frames) +``` + + +**Partial updates.** Send the primary key plus only the columns you want to +change; omitted columns keep their current values. This requires a scalar +index on the primary key. A re-captioning job, for example, only touches +`caption`: + + +```python Python icon="python" +patch = pa.table( + {"frame_id": [2000, 2001], "caption": ["a two-finger gripper closing on a cup", "the cup resting upright"]}, + schema=pa.schema([schema.field("frame_id"), schema.field("caption")]), +) +frames.merge_insert("frame_id").when_matched_update_all().when_not_matched_insert_all().execute(patch) +``` + + +**Deletes.** The predicate must bind the primary key to literal values. Ranges, +`OR`, and filters on other columns are rejected. Deletes require a scalar index +on the primary key. + + +```python Python icon="python" +frames.delete("frame_id IN (2000, 2001)") +``` + + +### Disabling RTE {#disable} + +Stop writes and wait for in-flight writes to finish, then flush pending writes +into the base table and disable. `checkpoint_rte()` only drains writes that +existed when it started, so anything written during the checkpoint can be left +behind. `disable_rte()` is rejected while the write-ahead log still holds rows, +so run `checkpoint_rte()` first. + + +```python Python icon="python" +frames.checkpoint_rte() +frames.disable_rte() +``` + + +## Write performance {#write-performance} + +RTE acknowledges a write as soon as it is durable in the write-ahead log, so +ingest is no longer limited by how fast the table can commit. + +**Compared with writing directly to the table.** On tables ranging from +hundreds of thousands to tens of millions of rows, RTE delivered 4–12× the +write throughput and 24–52× lower p99 write latency, depending on the mix of +inserts, updates, and deletes. + +**Scaling with shards.** An RTE table can be split into +[shards](/sql/ddl/table#properties) that write independently, and a single +table's ingest grows with its shard count. In our testing on AWS, with every +index on the table (vector, full-text, and BTree) kept current, a single table +reached these peak rates: + +| Shards | Peak ingest | +| --: | --: | +| 1 | 90 MB/s | +| 2 | 170 MB/s | +| 4 | 340 MB/s | +| 8 | 625 MB/s | +| 16 | 690 MB/s | +| 32 | 885 MB/s | + +Adding shards scales a table's ingest nearly linearly. + +## Next steps {#next-steps} + +- [Real Time reads](/search/realtime-read) — query an RTE table +- [How the real time engine works](/tables/realtime-engine#how-it-works) +- [Merge insert](/ingestion/merge-insert) +- [`ALTER TABLE ... SET RTE`](/sql/ddl/table#properties) diff --git a/docs/web/ingestion/update.mdx b/docs/web/ingestion/update.mdx index 4a5fdaa24..ad4ae23b2 100644 --- a/docs/web/ingestion/update.mdx +++ b/docs/web/ingestion/update.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Update" description: "Select existing rows, change values or expressions, and verify the changes." icon: pen availability: - oss: available + embedded: available enterprise: available - summary: >- - Update rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also supports the SQL UPDATE statement. --- import { @@ -83,7 +81,7 @@ connection or table handle, account for its Nested-field updates are not supported by these operations. For schema changes, see [schema evolution](/management/table#schema). Treat derived values separately: changing an input field is not a guarantee that every stored feature has been -recomputed. See [Refresh a column](/feature-engineering/refresh-column). +recomputed. See [Refresh a computed column](/feature-engineering/refresh-column). Updates can change index coverage and query cost. After a large update, inspect the workload and follow [index maintenance](/indexing/reindexing) diff --git a/docs/web/integrations/ai/huggingface.mdx b/docs/web/integrations/ai/huggingface.mdx index 0a38c208b..b284d38a2 100644 --- a/docs/web/integrations/ai/huggingface.mdx +++ b/docs/web/integrations/ai/huggingface.mdx @@ -3,11 +3,8 @@ title: "Hugging Face Hub" sidebarTitle: "Hugging Face" description: "Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval." availability: - oss: available - enterprise: unavailable - summary: >- - The embedded library opens Lance datasets on the Hugging Face Hub in place - through `hf://` URIs. A deployment does not read `hf://` datasets. + embedded: available + enterprise: available --- [Hugging Face Hub](https://huggingface.co/datasets?format=format:lance&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources. diff --git a/docs/web/integrations/data/pandas_and_pyarrow.mdx b/docs/web/integrations/data/pandas_and_pyarrow.mdx index c646e7f3b..810536587 100644 --- a/docs/web/integrations/data/pandas_and_pyarrow.mdx +++ b/docs/web/integrations/data/pandas_and_pyarrow.mdx @@ -2,12 +2,8 @@ title: "Pandas and PyArrow" sidebarTitle: "Pandas & PyArrow" availability: - oss: available + embedded: available enterprise: available - summary: >- - DataFrames go into `create_table` and come back from query results the same - way for local and deployment tables. - --- import { diff --git a/docs/web/integrations/data/polars_arrow.mdx b/docs/web/integrations/data/polars_arrow.mdx index 8eaaa6440..4ff759211 100644 --- a/docs/web/integrations/data/polars_arrow.mdx +++ b/docs/web/integrations/data/polars_arrow.mdx @@ -2,13 +2,8 @@ title: "Polars" sidebarTitle: "Polars & Arrow" availability: - oss: available + embedded: available enterprise: available - summary: >- - Polars data goes into `create_table`, and query results come back as Polars - DataFrames, for local and deployment tables alike; a LazyFrame over a whole - table is local only. - --- import { diff --git a/docs/web/integrations/data/pydantic.mdx b/docs/web/integrations/data/pydantic.mdx index 3ac660774..132ce7801 100644 --- a/docs/web/integrations/data/pydantic.mdx +++ b/docs/web/integrations/data/pydantic.mdx @@ -2,13 +2,8 @@ title: "Pydantic" sidebarTitle: "Pydantic" availability: - oss: available + embedded: available enterprise: available - summary: >- - The SDK converts `LanceModel` schemas to Arrow, and query rows back to - models, in your process, so they work the same with local and deployment - tables. - --- import { diff --git a/docs/web/integrations/lerobotdataset.mdx b/docs/web/integrations/lerobotdataset.mdx index cfb66b69b..260921034 100644 --- a/docs/web/integrations/lerobotdataset.mdx +++ b/docs/web/integrations/lerobotdataset.mdx @@ -138,6 +138,25 @@ class EmbedFromBlob: # SigLIP2 loads once per worker ... ``` +**Deduplication** removes near-static runs before they skew curation or +training. A stopped gripper holding one pose for a few seconds produces many +frames whose `emb_siglip2` vectors are nearly identical; LanceDB Enterprise's +native [`dedupe`](/sql/dql/dedupe) collapses them into one +representative per cluster, directly over the index, without a pairwise scan: + +```python +db.execute_query(""" + CREATE MATERIALIZED VIEW frames_deduped AS + SELECT * FROM dedupe('frames', 'emb_siglip2', distance_threshold => 0.01) +""") +``` + +Every column survives the dedup unchanged -- `jerk_score`, the episode and +camera metadata, the video blob reference -- since `dedupe` returns full +rows, not just which ones matched. Wrap it in a materialized view, as above, +to keep the deduplicated set current as new episodes land; see +[Materialized views](/feature-engineering/materialized-views). + **Search** is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms. **Curation** is one query across all of it. Semantic search with a predicate on the derived score, no join, no second system: diff --git a/docs/web/maintenance/background-jobs.mdx b/docs/web/maintenance/background-jobs.mdx index cfeab74df..3c2718cfe 100644 --- a/docs/web/maintenance/background-jobs.mdx +++ b/docs/web/maintenance/background-jobs.mdx @@ -5,16 +5,13 @@ description: "Understand the Enterprise background jobs that maintain tables, in icon: gears keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - LanceDB Enterprise schedules background maintenance jobs according to your - cluster configuration. --- The jobs documented elsewhere in these docs ([Build & Repair](/indexing/reindexing), [Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup), -[Column Refresh](/feature-engineering/refresh-column)) start from a call you +[Refresh Computed Column](/feature-engineering/refresh-column)) start from a call you make or, for compaction, cleanup and index updates, from the deployment's own maintenance. This page is the rest: jobs the deployment runs to keep itself correct and fast, which you may see named in [`SHOW JOBS`](/sql/ddl/job#show) @@ -27,7 +24,7 @@ also start cache-warming jobs through the admin API. ## Write-ahead log maintenance {#wal} -For a table with the [realtime engine](/tables/realtime-engine) enabled, two jobs keep the +For a table with the [real time engine](/tables/realtime-engine) enabled, two jobs keep the log from growing without bound: | Job type | What it does | diff --git a/docs/web/maintenance/cleanup.mdx b/docs/web/maintenance/cleanup.mdx index 4b9e24796..365eb66e8 100644 --- a/docs/web/maintenance/cleanup.mdx +++ b/docs/web/maintenance/cleanup.mdx @@ -5,10 +5,8 @@ description: "Pruning old versions and the files only they referenced." icon: broom keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Both offerings support cleanup. Embedded `optimize()` uses a retention window of 7 days by default; Enterprise uses configured background cleanup and retention, and does not support `optimize()`. --- Every write keeps the version before it, which is what makes diff --git a/docs/web/maintenance/compaction.mdx b/docs/web/maintenance/compaction.mdx index be6685d9c..64f18b095 100644 --- a/docs/web/maintenance/compaction.mdx +++ b/docs/web/maintenance/compaction.mdx @@ -4,12 +4,8 @@ sidebarTitle: "Compaction" description: "Merge small files, prune old versions, and fold new rows into existing indexes." icon: broom availability: - oss: available + embedded: available enterprise: available - summary: >- - Tables are compacted in the embedded library and in LanceDB Enterprise. In the - embedded library you call `optimize()`; a deployment compacts tables in the - background, and `optimize()` is not available there. --- import { PyStorageCompactionTuning, PyStorageRebuildVectorIndex, RsStorageCompactionTuning } from '/snippets/doc_storage_examples.mdx'; @@ -98,7 +94,7 @@ continuous small writes needs it on a schedule. Compaction rewrites files, which conflicts with a concurrent [computed-column refresh](/feature-engineering/add-computed-column) or -[view refresh](/feature-engineering/materialized-views) planned against the +[view refresh](/feature-engineering/refresh-materialized-view) planned against the version it replaces. Sequence them rather than running both at once. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 35b63c308..6651d6409 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -5,12 +5,8 @@ description: "Connect to LanceDB and configure storage, credentials and read con icon: database keywords: ["connect", "database", "uri", "api key", "storage options", "consistency"] availability: - oss: available + embedded: available enterprise: available - summary: >- - LanceDB Embedded opens tables in a local directory or object storage. - LanceDB Enterprise connects with a database URI and credentials and runs - operations with LanceDB Distributed. --- import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx'; @@ -27,11 +23,10 @@ objects, and configure how your application reaches the data. | `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | LanceDB Embedded, with object storage | In your application | | `db://my-database` | LanceDB Enterprise | On the Enterprise service (LanceDB Distributed) | -[Connect to LanceDB](/basics/connect) has examples for each connection type. -A connection to object storage still uses LanceDB Embedded: your process -reads and writes the bucket directly. LanceDB Enterprise runs LanceDB -Distributed: query execution, indexing and maintenance run on the Enterprise -service, outside your application. +[Connect to LanceDB](/basics/catalog) has examples for each connection type. +A connection to object storage still uses the embedded library: your process +reads and writes the bucket directly. Enterprise runs operations remotely and +manages the supporting compute and maintenance. ## Connect to Enterprise {#deployment} diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 7825b0bf6..415de7613 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -4,11 +4,6 @@ sidebarTitle: "Function" description: "Register a Python function once and apply it as a column, for work SQL cannot express." icon: code keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"] -availability: - oss: unavailable - enterprise: available - summary: >- - Enterprise supports Functions when enabled for your Enterprise cluster; LanceDB Embedded cannot register or run them. --- import { @@ -110,6 +105,26 @@ with `db.get_function(name, version=...)`. the function is baked into an image before the version becomes usable. +## Try it in SQL {#try-sql} + +A registered function is callable in SQL by its name, the same as any other +scalar function. Before applying it as a column, call it directly over the +table to check the registration succeeded, without committing a schema +change: + + +```python Python icon="python" +reader = db.execute_query("SELECT value, double(value) AS doubled FROM function_demo") +print(reader.read_all().to_pylist()) +# [{'value': 1.0, 'doubled': 2.0}, {'value': 2.0, 'doubled': 4.0}, {'value': 3.0, 'doubled': 6.0}] +``` + + + +Equivalent SQL: `SELECT value, double(value) AS doubled FROM function_demo`. +See [Querying with SQL](/basics/sql) for `execute_query` and other languages. + + ## Apply it as a column {#apply} Calling the registered version binds each parameter, by name, to a column with @@ -124,6 +139,51 @@ function application keeps its scalar or named-struct result in the named column. A bare named-struct application expands its result fields as one atomic binding, with aliases from `rename(columns=...)`. +### Bind more than one input, or fill more than one column {#multi} + +Parameters bind by name, so a function that takes several arguments binds +several columns in one call: + + +```python Python icon="python" +@udf +def ratio(numerator: float, denominator: float) -> float: + return numerator / denominator + +version = db.create_function_async(ratio).wait() +table.add_columns({"ratio": version(numerator=col("value"), denominator=col("count"))}) +``` + + +A named-struct output fills more than one column from a single application. +Declare the fields with `output_schema`, then alias each field to the column +it fills with `rename(columns=...)` and pass the application itself, not a +mapping, so both columns come from one atomic binding: + + +```python Python icon="python" +import pyarrow as pa + +@udf( + output_schema=pa.schema([ + pa.field("half", pa.float64(), nullable=False), + pa.field("double", pa.float64(), nullable=False), + ]), +) +def scale(value: float): + return {"half": value / 2, "double": value * 2} + +version = db.create_function_async(scale).wait() +application = version(value=col("value")).rename( + columns={"half": "value_half", "double": "value_double"} +) +table.add_columns(application) +``` + + +A single refresh fills `value_half` and `value_double` together: there is no +state where one is filled and the other is still empty. + ## Fill the column {#refresh} A refresh runs the function on the cluster as a [job](/basics/jobs). `wait()` @@ -188,46 +248,61 @@ undecorated class, so local tests can create an instance and call it directly. A Function can bind a [Blob v2 column](/management/table#managed-blobs) and return another blob. In row mode, a top-level blob parameter arrives as Python -`bytes`, or `None` for a nullable input. The worker reads the payload before -calling your function. Return bytes and let the destination table store the -Managed payload and descriptor. +`bytes`, or `None` for a nullable input, after the worker has read the +payload. Return bytes and let the destination table store the Managed +payload and descriptor. -Declare blob inputs and outputs explicitly. This example reverses the bytes -to demonstrate the round trip without requiring an image library. Here `db` -is an Enterprise database and `table` contains an `image` Blob v2 column -with table schema metadata `lance.blob.managed=true`, as in the -[table creation example](/management/table#managed-blobs-read-write): - -```python -from datetime import timedelta +The rest of this section extracts a frame from a stored video and captions +it, continuing as one example across chaining, persisting and incremental +reads below. `videos` holds raw clips in a `video` Blob v2 column; `frames` +names one timestamp per clip to extract, with a `selected` flag to skip some: + +```python Python icon="python" import lancedb -from lancedb import col -from lancedb.functions import udf import pyarrow as pa -@udf( - name="reverse_image", - python_version="3.12", - pip=["pyarrow==26.0.0"], - input_schema=pa.schema([lancedb.blob("image")]), - output_schema=lancedb.blob("reversed", nullable=False), -) -def reverse_image(image): - return b"missing" if image is None else image[::-1] - -function = db.create_function_async(reverse_image).wait( - timeout=timedelta(minutes=15) -) -table.add_columns({"reversed": function(image=col("image"))}) -result = table.refresh_column_async("reversed").wait( - timeout=timedelta(minutes=15) -) -assert result.rows_failed == 0 - -hits = table.search().select(["id", "reversed"]).with_row_id(True).to_arrow() -values = table.fetch_blobs("reversed", hits).to_pylist() +videos = db.create_table("videos", schema=pa.schema([ + pa.field("id", pa.utf8(), nullable=False), + lancedb.blob("video", nullable=False), +])) +frames = db.create_table("frames", [ + {"frame_id": 1, "video_id": "clip-1", "ts": 0.5, "selected": True}, + {"frame_id": 2, "video_id": "clip-1", "ts": 1.5, "selected": True}, +]) ``` + + +### Produce a blob {#produce-blobs} + +Declare blob inputs and outputs explicitly. `extract_frame` reads one frame +out of a video at a timestamp and returns it as a JPEG, without loading the +whole clip first -- see [Read a blob incrementally](#incremental) below for +how the `video: BlobReader` parameter does that: + + +```python Python icon="python" +from datetime import timedelta + +from lancedb import BlobReader, col +from lancedb.functions import udf + +@udf( + name="extract_frame", + gpu=True, + pip=["av==13.1.0", "pillow==11.0.0"], + input_schema=pa.schema([ + lancedb.blob("video"), + pa.field("ts", pa.float64(), nullable=False), + ]), + output_schema=lancedb.blob("jpeg", nullable=False), +) +def extract_frame(video: BlobReader, ts: float) -> bytes: + ... # decode the frame at ts, encode as JPEG, return its bytes + +version = db.create_function_async(extract_frame).wait(timeout=timedelta(minutes=15)) +``` + An empty byte string is a filled result. A scalar output cannot be nullable, because a physical NULL marks an unfilled computed row. To return a nullable @@ -235,29 +310,93 @@ blob as a value, use a named-struct output schema with a nullable blob field; the struct itself records whether the result has been filled. Read a nested blob with `table.fetch_blobs("payload.image", hits)`. -On deployments that support computed-column inputs, another Function can bind -to `col("reversed")`. Refresh the producer first, then the consumer. Once -outputs are current, a later refresh fills appended rows. A filtered refresh -preserves payload references for unselected Managed outputs. An unfiltered -refresh can recompute a whole fragment when it cannot establish that the -fragment's outputs are current. + +`output_schema=lancedb.blob(...)` is what makes a persisted result Blob v2 +rather than plain Arrow `binary`. Returning `bytes` from a function whose +output type is inferred from a bare annotation, with no explicit +`lancedb.blob(...)` output schema, persists as ordinary `binary`: the column +still holds the right bytes, but it skips the Managed storage path, descriptor +reads, and `fetch_blobs`. Declare it explicitly on every function whose output +should be a blob, not just the first one in a chain. + + +### Chain a Function on its output {#chain} + +A second Function can bind a blob another Function produced. Register +`caption` the same way, consuming a blob instead of producing one: + + +```python Python icon="python" +@udf(gpu=True, pip=["transformers==4.47.0"]) +def caption(jpeg: bytes) -> str: + ... # run a vision-language model over the JPEG, return one sentence + +caption_version = db.create_function_async(caption).wait(timeout=timedelta(minutes=15)) +``` + + +Compose both in one SQL statement over a join, instead of applying each as +its own column. The join supplies `video` and `ts` from two different +tables, and `caption` reads `extract_frame`'s result as it is computed, +before anything is written: + + +```sql SQL +SELECT f.frame_id, extract_frame(v.video, f.ts) AS jpeg, + caption(extract_frame(v.video, f.ts)) AS caption +FROM videos v JOIN frames f ON v.id = f.video_id +WHERE f.selected = true +``` + + +[`EXPLAIN`](/sql/dql/explain) on a query like this shows `RemoteTake`: the +blob is read once, where the worker runs, and never pulled into the query +node first. + +### Persist it as a column, or a materialized view {#persist} + +Apply the same chained expression with `add_columns` to persist it as an +ordinary table column: + + +```python Python icon="python" +table.add_columns({"jpeg": extract_frame(video=col("video"), ts=col("ts"))}) +result = table.refresh_column_async("jpeg").wait(timeout=timedelta(minutes=15)) +assert result.rows_failed == 0 + +hits = table.search().select(["frame_id", "jpeg"]).with_row_id(True).to_arrow() +values = table.fetch_blobs("jpeg", hits).to_pylist() +``` + + +Or wrap the join and both Functions in a +[materialized view](/sql/ddl/materialized-view) instead, so the extracted +frame and its caption stay current as new videos and frames arrive. See +[Automatic refresh](/feature-engineering/refresh-column#automatic) for +keeping either one fresh without calling refresh yourself. + +On deployments that support computed-column inputs, another Function can +bind to `col("jpeg")` directly, the same as the SQL join above does through +its own expression. Refresh the producer first, then the consumer. Once +outputs are current, a later refresh fills appended rows. A filtered +refresh preserves payload references for unselected Managed outputs. An +unfiltered refresh can recompute a whole fragment when it cannot establish +that the fragment's outputs are current. Function output blobs follow the same compaction and cleanup lifecycle as ingested Managed blobs. Retained versions continue to own their payloads; cleanup reclaims an object only after its retained references are gone. -### Draft: incremental blob inputs (upcoming) - -This feature requires an SDK that exports `BlobReader` and a compatible -Enterprise runtime with reader input support enabled. Coordinate the runtime -upgrade before registering or executing reader Functions. +### Read a blob incrementally {#incremental} Annotate a row Function parameter with `BlobReader` to read a Blob v2 input -incrementally. The signature selects the access mode without another -registration option. For example, read the first MP4 box type without loading -the whole video: +incrementally, the way `extract_frame` above does, instead of materializing +the whole payload first. The signature selects the access mode without +another registration option. A minimal example, reading only the first MP4 +box type without loading the rest of the video: -```python + +```python Python icon="python" from lancedb import BlobReader, udf @@ -266,12 +405,15 @@ def first_box_type(video: BlobReader) -> bytes: video.seek(4) return video.read(4) ``` + -The worker passes a seekable reader backed by the stored blob. `read(size)` -reads from the current cursor; `read_range(offset, length)` reads a byte range -without moving it. `size()` returns the payload size without reading it. -Each row call receives its own reader, which closes when the call returns or -raises. Do not retain it for another invocation. +The worker passes a seekable reader backed by the stored blob -- +`video.seekable()` is `True`. `read(size)` reads from the current cursor; +`read_range(offset, length)` reads a byte range without moving it, the same +bytes a `seek(offset)` followed by `read(length)` would read. `size()` +returns the payload size without reading it. Each row call receives its own +reader, which closes when the call returns or raises. Do not retain it for +another invocation. Use `video: bytes` when the Function needs the complete payload. Bytes inputs materialize before invocation and remain subject to eager input limits. @@ -293,6 +435,14 @@ dependencies are part of its definition: | `python_version` | Python major/minor for the execution environment, defaulting to yours | | `gpu` | whether every execution requires a GPU | + +```python Python icon="python" +@udf(gpu=True) +def extract_frame(video: BlobReader, ts: float) -> bytes: + ... # runs only on a GPU-capable worker +``` + + These are part of the **immutable function version**. Changing a dependency produces a new version rather than altering the one already registered, so a column filled by version 1 is unaffected by version 2. @@ -322,6 +472,46 @@ The client version defaults `python_version`, so a function registered from a right and occasionally surprising. Pin it when the function is sensitive to it. +## Remote calls and credentials {#secrets} + +A function that calls an external API needs a credential without putting it +in the function's source or its `env` map in plain text. A Secret is a +database-scoped named credential; `db.create_secret` stores the value once, +and an `EnvVarSecret` binds it to the environment variable the function +reads. Binding is a registration-time decision, passed to +`create_function_async`, not to `@udf`: + + +```python Python icon="python" +import os + +from lancedb import EnvVarSecret + +db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"]) + +@udf +def embed(text: str) -> list[float]: + import os + + from openai import OpenAI + + client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) + response = client.embeddings.create(input=text, model="text-embedding-3-small") + return response.data[0].embedding + +version = db.create_function_async( + embed, + secrets=[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")], +).wait() +``` + + +No API returns a Secret's value, and it never appears in the function's +registration: `embed` reads `OPENAI_API_KEY` the way it always did, and the +binding is what puts a value there. A mistyped Secret name is caught at +registration; a mistyped environment variable name surfaces only on the +first rows, since nothing records which variables a function reads. + ## Failure handling {#failures} A function that raises fails the job. There is no per-row retry or skip policy @@ -350,6 +540,25 @@ In practice this means a function should be self-contained: take its inputs as arguments, import what it needs at module level, and avoid capturing state from the surrounding scope. +Pass `code=[module, ...]` to ship additional helper modules into the same +image, for logic the function imports but that does not belong inlined into +it -- a model-loading helper shared by several Functions, for example. +Continuing `caption` from [above](#chain), with its model-loading code +factored into a sibling module instead of inlined: + + +```python Python icon="python" +import caption_worker # a sibling module with the model-loading logic + +@udf(gpu=True, code=[caption_worker]) +def caption(jpeg: bytes) -> str: + return caption_worker.run(jpeg) +``` + + +Each module in `code` is captured and shipped the same way the function +itself is: as source, snapshotted at registration. + ## Drop it {#drop} `db.drop_function(name, version=...)` drops the function's name and the object diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index 21222f418..9fc02ad6b 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -64,7 +64,7 @@ Enterprise, it requests the server's view listing. See [Materialized views](/management/materialized-view) for creating one, and -[Refresh a materialized view](/feature-engineering/materialized-views) for keeping +[Refresh a materialized view](/feature-engineering/refresh-materialized-view) for keeping it current. ### Views {#views} diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx index 42245a51a..90283e198 100644 --- a/docs/web/management/materialized-view.mdx +++ b/docs/web/management/materialized-view.mdx @@ -5,10 +5,8 @@ description: "Create, open, list and drop a table whose contents are defined by icon: table keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"] availability: - oss: available + embedded: unavailable enterprise: available - summary: >- - Both offerings support materialized views. See [Enterprise compatibility](#deployment) for connection and server requirements. --- A materialized view is a table whose contents are defined by a query over @@ -19,10 +17,10 @@ The result is an ordinary table: queryable, indexable, searchable, and present in `table_names()`. Nothing about reading it is special. This page covers creating and managing one. Keeping it current is -[Refresh a materialized view](/feature-engineering/materialized-views). +[Refresh a materialized view](/feature-engineering/refresh-materialized-view). -In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show). +In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show). ## Create a materialized view {#create} @@ -30,25 +28,25 @@ In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized- ```python Python icon="python" view = db.create_materialized_view( - "recent_docs", - source="documents", - select=["id", "title", "embedding"], - where="created_at > TIMESTAMP '2026-01-01 00:00:00'", + "frames_captioned", + source="frames", + select=["frame_id", "video_id", "ts", "caption"], + where="selected = true", ) ``` ```typescript TypeScript icon="square-js" -const view = await db.createMaterializedView("recent_docs", "documents", { - select: ["id", "title", "embedding"], - where: "created_at > TIMESTAMP '2026-01-01 00:00:00'", +const view = await db.createMaterializedView("frames_captioned", "frames", { + select: ["frame_id", "video_id", "ts", "caption"], + where: "selected = true", }); ``` -`source` names the table the view reads. With Python SDK 0.40.0, create that -table with stable row IDs before creating the view. The -[complete example](/feature-engineering/materialized-views#create-and-refresh) -shows the setup. `select`, `where` and `limit` define the query. +`source` names the table the view reads — here, a `frames` table with a +`caption` column, filtered to the frames selected for captioning. See +[Create a materialized view](/feature-engineering/materialized-views#create) +for the full walkthrough. `select`, `where` and `limit` define the query. ## Open, list and drop views {#manage} @@ -60,6 +58,30 @@ shows the setup. `select`, `where` and `limit` define the query. | refresh | `refresh` | `refresh` | `REFRESH MATERIALIZED VIEW` | | drop | `drop_table` | `dropTable` | `DROP TABLE` | +Enterprise Creating, refreshing and dropping a +materialized view each have an async, job-returning counterpart -- +`create_materialized_view_async`, `view.refresh_async()`, +`drop_materialized_view_async` -- the same pattern as +[`refresh_column_async`](/feature-engineering/refresh-column#async): each +returns a [job](/basics/jobs) you `wait()` on instead of blocking. The async +counterpart of the example [above](#create): + + +```python Python icon="python" +from datetime import timedelta + +job = db.create_materialized_view_async( + "frames_captioned", source="frames", + select=["frame_id", "video_id", "ts", "caption"], where="selected = true", +) +view = job.wait(timeout=timedelta(minutes=15)) +``` + + +`create_materialized_view_async` also accepts `auto_refresh_enabled` and +`auto_refresh_interval_seconds`; see +[Automatic refresh](/feature-engineering/refresh-materialized-view#auto-refresh). + On a local database, `list_materialized_views` reads table schemas to find views. On Enterprise, it asks the server for the view listing, as @@ -78,16 +100,14 @@ That also means dropping one removes stored rows, unlike dropping a ## Enterprise compatibility {#deployment} -Enterprise supports creating and refreshing materialized views on the -Enterprise service. Use the SDK version supported by your deployment, or the -[SQL interface](/sql/ddl/materialized-view). See -[Enterprise refreshes](/feature-engineering/materialized-views#availability) -for compatibility considerations when upgrading an existing pipeline. +Enterprise supports creating and refreshing materialized views on remote +compute. Use the SDK version supported by your deployment, or the +[SQL interface](/sql/ddl/materialized-view). ## Related guides {#next} - + How a refresh works, and what it writes. diff --git a/docs/web/management/namespace.mdx b/docs/web/management/namespace.mdx index e5222ab84..99e53d793 100644 --- a/docs/web/management/namespace.mdx +++ b/docs/web/management/namespace.mdx @@ -5,11 +5,8 @@ description: "Grouping tables into a hierarchy, and addressing them through it." icon: folder-tree keywords: ["namespace", "catalog", "hierarchy", "directory namespace", "rest namespace"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Namespaces work in the embedded library, through directory or REST - catalogs, and in LanceDB Enterprise. --- import { diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index a28f63d40..0b71fb315 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -5,12 +5,8 @@ description: "The object most operations act on: creating, opening, renaming, cl icon: table keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Tables are created, evolved, versioned and dropped in the embedded library - and in LanceDB Enterprise. Renaming, cloning, dropping every table and pruning old - versions differ between the two. --- import { @@ -953,8 +949,8 @@ First, let's create a sample table with product data to demonstrate schema evolu Columns derived from existing data (SQL expressions over other columns) are -covered in [Data evolution](/feature-engineering/data-evolution), because -deriving values is a feature-engineering job rather than a schema one. +covered in [Create a computed column](/feature-engineering/add-computed-column), +because deriving values is a feature-engineering job rather than a schema one. #### Add columns with default values {#schema-add-columns-with-default-values} diff --git a/docs/web/management/view.mdx b/docs/web/management/view.mdx index de6cd38a5..e01120034 100644 --- a/docs/web/management/view.mdx +++ b/docs/web/management/view.mdx @@ -1,14 +1,12 @@ --- title: "Views" sidebarTitle: "View" -description: "A saved query, evaluated when read. The non-materialized counterpart to a materialized view." +description: "A saved query, evaluated when read." icon: eye keywords: ["view", "create view", "sql", "catalog"] availability: - oss: unavailable + embedded: unavailable enterprise: available - summary: >- - Enterprise supports views when enabled for your Enterprise cluster; the embedded library cannot create them. --- A view saves a query that you can refer to by name. Reading the view executes diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 645e0565e..c84ae191a 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -4,26 +4,24 @@ sidebarTitle: "Quickstart" description: "Install LanceDB, create a table, and run your first vector search." icon: rocket --- -import { PyConnect, TsConnect, RsConnect, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx'; -import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartVectorSearch1, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1 } from '/snippets/quickstart.mdx'; +import { PyConnect, TsConnect, RsConnect, PyConnectObjectStorage, TsConnectObjectStorage, RsConnectObjectStorage, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx'; +import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartAddFeature, PyQuickstartComputedFeature, PyQuickstartComputedFeatureEnterprise, PyQuickstartMultimodalBytes, PyQuickstartTraining, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, TsQuickstartCurateWithMetadata, TsQuickstartAddFeature, TsQuickstartQueryFeature, TsQuickstartMultimodalBytes, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1, RsQuickstartCurateWithMetadata, RsQuickstartAddFeature, RsQuickstartQueryFeature, RsQuickstartMultimodalBytes } from '/snippets/quickstart.mdx'; This quickstart uses LanceDB Embedded with a local database and works without an account or API key. LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage. -Follow the steps below, or jump to the [complete examples](#code) to copy a runnable program. - ## 1\. Install LanceDB {#1-install-lancedb} -Choose your language. Python uses Polars to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly. +Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly. ```bash Python (pip) icon="python" -pip install lancedb polars +pip install lancedb pandas ``` ```bash Python (uv) icon="python" uv venv -uv pip install lancedb polars +uv pip install lancedb pandas ``` ```bash TypeScript icon="square-js" @@ -112,7 +110,7 @@ uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat` ## 2\. Connect to a database {#2-connect-to-a-lancedb-database} -### Use a local directory {#connect-via-local-directory-path} +### Local directory {#connect-via-local-directory-path} LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed. @@ -130,10 +128,27 @@ LanceDB creates an `ex_lancedb` directory to store your tables. No server is nee -The [complete examples](#code) include the imports and entry points for each language. For Python's async API, see [Connecting to a database](/basics/connect). +For Python's async API, see [Connecting to a database](/basics/catalog). - -### Connect to LanceDB Enterprise {#connect-to-lancedb-enterprise} +### Object storage {#connect-via-object-storage-uris} + +Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Configuring storage](/deployment/storage) for credentials and provider-specific options. + + + + {PyConnectObjectStorage} + + + {"import * as lancedb from \"@lancedb/lancedb\";\n\n"} + {TsConnectObjectStorage} + + + {"use lancedb::connect;\n\n"} + {RsConnectObjectStorage} + + + +### Enterprise {#connect-to-lancedb-enterprise} Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`. @@ -152,27 +167,20 @@ Use the database URI, API key, and endpoint provided by your administrator or th -In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/connect#enterprise) for an example. +In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/catalog#enterprise) for an example. -Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between Embedded and Distributed execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise). - - - -### Connect via object storage URIs {#connect-via-object-storage-uris} - -Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Configuring storage](/deployment/storage) for credentials and provider-specific options. - +Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [LanceDB Enterprise](/enterprise). ## 3\. Create a table and add data {#3-create-a-new-table} -Each row describes a character and stores a vector alongside their name, description, and stats. +Each row describes a frame from a robot video clip and stores an embedding alongside its video, caption, and quality stats. -These vectors are made up for this example. In your application, use an embedding model to generate vectors from your data. +These embeddings are made up for this example. In your application, use an embedding model to generate them from your data. - + {PyQuickstartData} @@ -187,7 +195,7 @@ These vectors are made up for this example. In your application, use an embeddin -Create a `characters` table from those records: +Create a `frames` table from those records: @@ -201,9 +209,9 @@ Create a `characters` table from those records: -This example uses overwrite mode so you can run it again. It replaces any existing `characters` table in the database you connect to. +This example uses overwrite mode so you can run it again. It replaces any existing `frames` table in the database you connect to. -Add one more character: +Add one more frame: @@ -219,11 +227,11 @@ Add one more character: ## 4\. Search the table {#4-semantic-search} -Find the two vectors closest to the query vector. This example uses a fixed vector; it does not call an embedding model. +Find the two embeddings closest to the query embedding. This example uses a fixed embedding; it does not call an embedding model. - {PyQuickstartVectorSearch1} + {"# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install pandas` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(result)\n"} {TsQuickstartVectorSearch1} @@ -233,84 +241,109 @@ Find the two vectors closest to the query vector. This example uses a fixed vect -Merlin and Morgana come back first. Python and Rust display a Polars DataFrame; TypeScript displays an array of results. The `_distance` column measures how far each result is from your query vector, with smaller values indicating a closer match. +Frames 2 and 4 come back first -- the gripper closing on the cup, then lowering it. Python displays a Pandas DataFrame, Rust displays a Polars DataFrame, and TypeScript displays an array of results. The `_distance` column measures how far each result is from your query embedding, with smaller values indicating a closer match. -## Complete examples {#code} +## 5\. Filter results {#5-curation} -Each example below creates the local database, adds four characters, and prints the two closest matches. Use the dependencies from step 1. - - -Save this as `quickstart.py` and run `python quickstart.py`. If you installed with uv, run `uv run quickstart.py`. +Combine vector search with a metadata filter, such as `stats.motion >= 4`, to curate results instead of just ranking them: - - {PyConnect} - {PyQuickstartData} - {PyQuickstartCreateTable} - {PyQuickstartAddData} - {PyQuickstartVectorSearch1} + + {"curated = (\n table.search(query_vector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(curated)\n"} + + + {TsQuickstartCurateWithMetadata} + + + {RsQuickstartCurateWithMetadata} - - -Save this as `quickstart.mts` and run `node quickstart.mts` from the directory where you installed LanceDB. +Frames 2 and 4 come back again here, since both already have `motion >= 4`; frames 1 and 3 are excluded from ranking entirely, rather than ranked and then dropped. See [SQL filtering](/search/filtering) for the full predicate syntax. + +## 6\. Add a derived feature {#6-add-a-derived-feature} + +Add a column calculated from existing fields with a SQL expression. This fills every existing row once, immediately: - - {"import * as lancedb from \"@lancedb/lancedb\";\n\n"} - {TsConnect} - {TsQuickstartData} - {TsQuickstartCreateTable} - {TsQuickstartAddData} - {TsQuickstartVectorSearch1} + + {PyQuickstartAddFeature} + + + {TsQuickstartAddFeature} + + + {RsQuickstartAddFeature} - - -Replace `src/main.rs` with this program, then run `cargo run` from your project directory. The schema and Arrow conversion helpers are included. +Read it back like any other column: - - {"use lancedb::connect;\n\n"} - {RsQuickstartDefineStruct} - {"\n#[tokio::main]\nasync fn main() -> Result<(), Box> {\n"} - {RsConnect} - {RsQuickstartData} - {RsQuickstartCreateTable} - {RsQuickstartAddData} - {RsQuickstartVectorSearch1} - {"\nOk(())\n}\n"} + + {"features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).limit(10).to_pandas()\nprint(features)\n"} + + + {TsQuickstartQueryFeature} + + + {RsQuickstartQueryFeature} - -## Next steps {#next-steps} +See [Adding columns](/management/table#schema-add-new-columns) for more on this form. -You've created a table, added rows, and searched it. Continue with the guides below. +## 7\. Set up computed columns {#7-computed-columns} -### Filter results {#5-curation} +A computed column is declared up front and filled -- or refilled -- on demand, instead of being calculated once like step 6's. Declare `caption_length`, then fill it explicitly: -Combine vector search with metadata filters, such as `stats.magic >= 4`. See [SQL filtering](/search/filtering). + + + {PyQuickstartComputedFeature} + + -### Extend your table {#extensions} +A later `refresh_column` call fills only rows added since the last refresh and recomputes rows whose inputs changed, leaving unchanged rows alone. On LanceDB Enterprise, the equivalent refresh runs as a background job instead of blocking: -#### Add a derived feature {#6-add-a-derived-feature} + + + {PyQuickstartComputedFeatureEnterprise} + + -Add columns calculated from existing fields. See [Adding columns](/management/table#schema-add-new-columns). +TypeScript and Rust declare and refresh computed columns the same way, through the same `add_columns`/`refresh_column` calls. See [Computed columns](/feature-engineering/add-computed-column) for the full refresh semantics and the automatic-refresh option. -#### Computed columns {#computed-columns} +## 8\. Store multimodal data {#8-store-multimodal-data} -Declare a column and choose when to fill or refresh it. See [Computed columns](/feature-engineering/add-computed-column). +Store images, audio, or other binary data alongside vectors and metadata, in a column of raw bytes. This creates a new table from one image file -- replace the path with one of your own: -#### Store multimodal data {#7-store-multimodal-data} + + + {PyQuickstartMultimodalBytes} + + + {TsQuickstartMultimodalBytes} + + + {RsQuickstartMultimodalBytes} + + -Store images, audio, and other binary data alongside vectors and metadata. See [Multimodal data types](/management/table#multimodal). +See [Multimodal data types](/management/table#multimodal) for larger payloads and the Blob v2 column type, which avoids loading binary data on every scan. -#### Train on your data {#8-train} +## 9\. Train on your data {#9-train} -Read table data into PyTorch for training. See [Data loading](/training/data-loading). +Read table data into PyTorch, projecting and shuffling only the columns a training step needs: + + + + {PyQuickstartTraining} + + + +`Permutation` is LanceDB's row-ordering primitive for training reads; `select_columns` projects before reading, so a training step touches only the columns it selects, not unused ones such as `caption` or `embedding`. See [Data loading](/training/data-loading) for the full `StreamingDataset` API, including resuming and multi-worker loading. + +## Where to go next {#next-steps} diff --git a/docs/web/reranking/index.mdx b/docs/web/reranking/index.mdx index ccaa70e9b..1de69a2a8 100644 --- a/docs/web/reranking/index.mdx +++ b/docs/web/reranking/index.mdx @@ -5,10 +5,8 @@ description: "Choose a reranker, apply it to search candidates and evaluate rele icon: "sort-amount-down" keywords: ["re-ranking", "reranker", "rerank"] availability: - oss: available + embedded: available enterprise: available - summary: >- - Reranking is available with LanceDB Embedded and LanceDB Enterprise. --- import { diff --git a/docs/web/search/explain.mdx b/docs/web/search/explain.mdx index 1a1fa3330..653eecb52 100644 --- a/docs/web/search/explain.mdx +++ b/docs/web/search/explain.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Query Plans" description: "Inspect query execution, identify expensive work and compare changes." icon: "map" availability: - oss: available + embedded: available enterprise: available - summary: >- - Query plans are available in LanceDB Embedded and LanceDB Enterprise. --- import { PyQueryPlan, TsQueryPlanBody, TsQueryPlanTail } from '/snippets/search_examples.mdx'; diff --git a/docs/web/search/filtering.mdx b/docs/web/search/filtering.mdx index a3ad6f735..099aea73c 100644 --- a/docs/web/search/filtering.mdx +++ b/docs/web/search/filtering.mdx @@ -4,10 +4,8 @@ sidebarTitle: Filtering description: "Use predicates to select rows and control when filters apply to search." icon: "filter" availability: - oss: available + embedded: available enterprise: available - summary: >- - Metadata filtering is available in LanceDB Embedded and LanceDB Enterprise. --- import { PyFilterSearch, TsFilterSearchBody, TsFilterSearchTail } from '/snippets/search_examples.mdx'; diff --git a/docs/web/search/full-text-search.mdx b/docs/web/search/full-text-search.mdx index 858492576..0f9930dc3 100644 --- a/docs/web/search/full-text-search.mdx +++ b/docs/web/search/full-text-search.mdx @@ -4,10 +4,8 @@ sidebarTitle: Full-Text Search (FTS) description: "Find records by words, phrases or partial matches using a full-text index." icon: "book" availability: - oss: available + embedded: available enterprise: available - summary: >- - Full-text search is available in LanceDB Embedded and LanceDB Enterprise. --- import { diff --git a/docs/web/search/hybrid-search.mdx b/docs/web/search/hybrid-search.mdx index 1100c073a..6b769b6a8 100644 --- a/docs/web/search/hybrid-search.mdx +++ b/docs/web/search/hybrid-search.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Hybrid Search" description: "Combine vector similarity and keyword matches in a single ranked result set." icon: "search" availability: - oss: available + embedded: available enterprise: available - summary: >- - Hybrid search is available in LanceDB Embedded and LanceDB Enterprise. --- import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx'; diff --git a/docs/web/search/index.mdx b/docs/web/search/index.mdx index c4bdd5cae..86a0b1b17 100644 --- a/docs/web/search/index.mdx +++ b/docs/web/search/index.mdx @@ -4,10 +4,8 @@ sidebarTitle: Overview description: "Choose vector, full-text or hybrid search and return the records your application needs." icon: "list" availability: - oss: available + embedded: available enterprise: available - summary: >- - Vector, full-text and hybrid search are available in LanceDB Embedded and LanceDB Enterprise. --- import { PyVectorSearch, TsVectorSearchBody, TsVectorSearchTail } from '/snippets/search_examples.mdx'; @@ -68,7 +66,7 @@ and removes it when finished. This query returns the puppy and dog records, ordered by vector distance. It searches the three-row table directly, without an ANN index or an embedding -service. For connection options, see [Connect](/basics/connect). +service. For connection options, see [Connect](/basics/catalog). diff --git a/docs/web/search/realtime-read.mdx b/docs/web/search/realtime-read.mdx new file mode 100644 index 000000000..8f51995d3 --- /dev/null +++ b/docs/web/search/realtime-read.mdx @@ -0,0 +1,74 @@ +--- +title: "Real Time reads" +sidebarTitle: "Real Time Read" +description: "Query a table with the real time engine enabled and get writes back within milliseconds of being written, merged transparently with the base table." +icon: "magnifying-glass" +keywords: ["rte", "real time engine", "real time search", "real time query", "fresh reads", "memwal", "upsert"] +availability: + embedded: unavailable + enterprise: available +--- + +A table with the [real time engine](/tables/realtime-engine) enabled queries +the same way as any other table — nothing about `search`, filtering, or SQL +changes. This page covers what you get back and how it performs; for enabling +RTE and writing to a table, see [Real Time writes](/ingestion/realtime-write). + +## Query a real time table {#query} + +Every query runs against the base table and the fresh tier (unflushed +memtables and SSTables) in parallel, and the two are merged into one result. +Fresh rows supersede base rows with the same primary key, so a row written a +moment ago is already reflected: + + +```python Python icon="python" +# continuing the `frames` table connected and written to in Real Time writes +import numpy as np + +query_vector = np.random.rand(frames.schema.field("embedding").type.list_size).astype("float32") + +frames.search(query_vector, vector_column_name="embedding").limit(10).to_list() +frames.search("gripper closing on a cup", fts_columns="caption").limit(10).to_list() +``` + + +Vector, full-text, BTree, bitmap, and most other scalar indexes are maintained +on the memtable, so new rows are searchable through the index as soon as +they're written — not just after the next compaction. + +## Read performance {#read-performance} + +Reads on an RTE table stay close to those on a fully compacted table, even +with the fresh tier at its fullest before compaction. + +The table below compares a 5M-row table with a 128-dimension IVF-PQ index plus +BTree and FTS indexes, with and without RTE. The RTE table holds one +unflushed memtable and three SSTables, each from a 100k-row write wave, half of +which are updates to existing rows. Latencies are p50 in milliseconds. + +| Query | Without RTE | With RTE | +| --- | --: | --: | +| Primary-key lookup | 62 | 63 | +| Vector search | 68 | 71 | +| Vector search with prefilter | 90 | 91 | +| Full-text term | 63 | 64 | +| Full-text phrase | 68 | 82 | +| Hybrid search (RRF) | 68 | 83 | +| SQL point lookup | 189 | 187 | + +Most queries are within a few milliseconds of the base table. Hybrid and phrase +queries add roughly 15 ms to merge the two tiers. + + +Support for all table operations on RTE tables is coming soon. Time travel, +branching, `count_rows`, row IDs (`with_row_id`), and `TABLESAMPLE` are not yet +available. + + +## Next steps {#next-steps} + +- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table +- [How the real time engine works](/tables/realtime-engine#how-it-works) +- [Hybrid search](/search/hybrid-search) +- [Vector search](/search/vector-search) diff --git a/docs/web/search/vector-search.mdx b/docs/web/search/vector-search.mdx index 59fbc1cc9..8c43fd0e6 100644 --- a/docs/web/search/vector-search.mdx +++ b/docs/web/search/vector-search.mdx @@ -4,10 +4,8 @@ sidebarTitle: "Vector Search" description: "Search by vector similarity, choose a distance metric and tune your queries." icon: "arrow-up-right-dots" availability: - oss: available + embedded: available enterprise: available - summary: >- - Vector search is available in LanceDB Embedded and LanceDB Enterprise. --- import { diff --git a/docs/web/snippets/quickstart.mdx b/docs/web/snippets/quickstart.mdx index c7cf98e32..cd8976768 100644 --- a/docs/web/snippets/quickstart.mdx +++ b/docs/web/snippets/quickstart.mdx @@ -1,66 +1,66 @@ {/* Auto-generated by scripts/mdx_snippets_gen.py. Do not edit manually. */} -export const PyQuickstartAddData = "table.add(\n [\n {\n \"id\": \"4\",\n \"name\": \"Morgana\",\n \"role\": \"Sorceress\",\n \"description\": \"Powerful sorceress of Avalon.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 4},\n \"vector\": [0.3, 0.9, 0.6, 0.8],\n }\n ]\n)\n"; +export const PyQuickstartAddData = "table.add(\n [\n {\n \"frame_id\": \"4\",\n \"video_id\": \"clip-2\",\n \"caption\": \"a gripper lowering a cup onto the table\",\n \"stats\": {\"motion\": 5, \"sharpness\": 4, \"exposure\": 4, \"framing\": 2},\n \"embedding\": [0.3, 0.9, 0.6, 0.8],\n }\n ]\n)\n"; -export const PyQuickstartAddFeature = "table.add_columns(\n {\n \"power_score\": \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n }\n)\n"; +export const PyQuickstartAddFeature = "table.add_columns(\n {\n \"quality_score\": \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\"\n }\n)\n"; -export const PyQuickstartComputedFeature = "table.add_columns(computed={\"description_length\": \"length(description)\"})\ntable.refresh_column(\"description_length\")\n"; +export const PyQuickstartComputedFeature = "table.add_columns(computed={\"caption_length\": \"length(caption)\"})\ntable.refresh_column(\"caption_length\")\n"; -export const PyQuickstartComputedFeatureEnterprise = "table.add_columns(computed={\"description_length\": \"length(description)\"})\njob = table.refresh_column_async(\"description_length\")\njob.wait()\n"; +export const PyQuickstartComputedFeatureEnterprise = "table.add_columns(computed={\"caption_length\": \"length(caption)\"})\njob = table.refresh_column_async(\"caption_length\")\njob.wait()\n"; -export const PyQuickstartCreateTable = "table = db.create_table(\"characters\", data=data, mode=\"overwrite\")\n"; +export const PyQuickstartCreateTable = "table = db.create_table(\"frames\", data=data, mode=\"overwrite\")\n"; -export const PyQuickstartCreateTableAsync = "async_table = await async_db.create_table(\n \"characters\",\n data=data,\n mode=\"overwrite\",\n)\n"; +export const PyQuickstartCreateTableAsync = "async_table = await async_db.create_table(\n \"frames\",\n data=data,\n mode=\"overwrite\",\n)\n"; -export const PyQuickstartCurateWithMetadata = "curated = (\n table.search(query_vector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(curated)\n"; +export const PyQuickstartCurateWithMetadata = "curated = (\n table.search(query_vector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(curated)\n"; -export const PyQuickstartData = "data = [\n {\n \"id\": \"1\",\n \"name\": \"King Arthur\",\n \"role\": \"King\",\n \"description\": \"Leader of Camelot and wielder of Excalibur.\",\n \"stats\": {\"strength\": 4, \"magic\": 1, \"leadership\": 5, \"wisdom\": 4},\n \"vector\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"id\": \"2\",\n \"name\": \"Merlin\",\n \"role\": \"Wizard\",\n \"description\": \"Advisor and prophet with deep magical knowledge.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 5},\n \"vector\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"id\": \"3\",\n \"name\": \"Sir Lancelot\",\n \"role\": \"Knight\",\n \"description\": \"Legendary knight known for courage and combat skill.\",\n \"stats\": {\"strength\": 5, \"magic\": 1, \"leadership\": 3, \"wisdom\": 3},\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n"; +export const PyQuickstartData = "data = [\n {\n \"frame_id\": \"1\",\n \"video_id\": \"clip-1\",\n \"caption\": \"a robot arm reaching toward a cup\",\n \"stats\": {\"motion\": 3, \"sharpness\": 4, \"exposure\": 4, \"framing\": 3},\n \"embedding\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"frame_id\": \"2\",\n \"video_id\": \"clip-1\",\n \"caption\": \"a two-finger gripper closing on a cup\",\n \"stats\": {\"motion\": 5, \"sharpness\": 4, \"exposure\": 5, \"framing\": 2},\n \"embedding\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"frame_id\": \"3\",\n \"video_id\": \"clip-2\",\n \"caption\": \"a robot arm holding a cup steady\",\n \"stats\": {\"motion\": 2, \"sharpness\": 3, \"exposure\": 3, \"framing\": 4},\n \"embedding\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n"; -export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_bytes = Path(\"data/sir-lancelot.jpg\").read_bytes()\n\nmultimodal_table = db.create_table(\n \"character_images\",\n data=[\n {\n \"id\": \"lancelot\",\n \"description\": \"Portrait of Sir Lancelot\",\n \"image\": image_bytes,\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n"; +export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_bytes = Path(\"data/sample-frame.jpg\").read_bytes()\n\nmultimodal_table = db.create_table(\n \"frame_images\",\n data=[\n {\n \"frame_id\": \"thumb-1\",\n \"caption\": \"Sample frame thumbnail\",\n \"image\": image_bytes,\n \"embedding\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n"; export const PyQuickstartOutputPandas = "# Ensure you run `pip install pandas` beforehand\nresult = table.search(query_vector).limit(2).to_pandas()\nprint(result)\n"; -export const PyQuickstartQueryFeature = "features = table.search().select([\"name\", \"role\", \"power_score\"]).to_polars()\nprint(features)\n"; +export const PyQuickstartQueryFeature = "features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).to_polars()\nprint(features)\n"; -export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ntable.add_columns(\n {\n \"magic\": \"cast(stats.magic as float)\",\n \"strength\": \"cast(stats.strength as float)\",\n }\n)\ndataset = Permutation.identity(table).select_columns([\"magic\", \"strength\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n"; +export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ntable.add_columns(\n {\n \"motion\": \"cast(stats.motion as float)\",\n \"sharpness\": \"cast(stats.sharpness as float)\",\n }\n)\ndataset = Permutation.identity(table).select_columns([\"motion\", \"sharpness\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n"; -export const PyQuickstartVectorSearch1 = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nresult = (\n table.search(query_vector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(result)\n"; +export const PyQuickstartVectorSearch1 = "# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(result)\n"; -export const PyQuickstartVectorSearch1Async = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nasync_result = await (\n await async_table.search(query_vector)\n).select([\"name\", \"role\", \"description\", \"_distance\"]).limit(2).to_polars()\nprint(async_result)\n"; +export const PyQuickstartVectorSearch1Async = "# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nasync_result = await (\n await async_table.search(query_vector)\n).select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"]).limit(2).to_polars()\nprint(async_result)\n"; -export const TsQuickstartAddData = "await table.add([\n {\n id: \"4\",\n name: \"Morgana\",\n role: \"Sorceress\",\n description: \"Powerful sorceress of Avalon.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },\n vector: [0.3, 0.9, 0.6, 0.8],\n },\n]);\n"; +export const TsQuickstartAddData = "await table.add([\n {\n frame_id: \"4\",\n video_id: \"clip-2\",\n caption: \"a gripper lowering a cup onto the table\",\n stats: { motion: 5, sharpness: 4, exposure: 4, framing: 2 },\n embedding: [0.3, 0.9, 0.6, 0.8],\n },\n]);\n"; -export const TsQuickstartAddFeature = "await table.addColumns([\n {\n name: \"power_score\",\n valueSql:\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\",\n },\n]);\n"; +export const TsQuickstartAddFeature = "await table.addColumns([\n {\n name: \"quality_score\",\n valueSql:\n \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\",\n },\n]);\n"; -export const TsQuickstartCreateTable = "const table = await db.createTable(\"characters\", data, { mode: \"overwrite\" });\n"; +export const TsQuickstartCreateTable = "const table = await db.createTable(\"frames\", data, { mode: \"overwrite\" });\n"; -export const TsQuickstartCurateWithMetadata = "const curated = await table\n .search(queryVector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(curated);\n"; +export const TsQuickstartCurateWithMetadata = "const curated = await table\n .search(queryVector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(curated);\n"; -export const TsQuickstartData = "const data = [\n {\n id: \"1\",\n name: \"King Arthur\",\n role: \"King\",\n description: \"Leader of Camelot and wielder of Excalibur.\",\n stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n {\n id: \"2\",\n name: \"Merlin\",\n role: \"Wizard\",\n description: \"Advisor and prophet with deep magical knowledge.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n {\n id: \"3\",\n name: \"Sir Lancelot\",\n role: \"Knight\",\n description: \"Legendary knight known for courage and combat skill.\",\n stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n"; +export const TsQuickstartData = "const data = [\n {\n frame_id: \"1\",\n video_id: \"clip-1\",\n caption: \"a robot arm reaching toward a cup\",\n stats: { motion: 3, sharpness: 4, exposure: 4, framing: 3 },\n embedding: [0.7, 0.1, 0.9, 0.7],\n },\n {\n frame_id: \"2\",\n video_id: \"clip-1\",\n caption: \"a two-finger gripper closing on a cup\",\n stats: { motion: 5, sharpness: 4, exposure: 5, framing: 2 },\n embedding: [0.2, 0.9, 0.4, 0.9],\n },\n {\n frame_id: \"3\",\n video_id: \"clip-2\",\n caption: \"a robot arm holding a cup steady\",\n stats: { motion: 2, sharpness: 3, exposure: 3, framing: 4 },\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n];\n"; -export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imageBytes = await readFile(\"data/sir-lancelot.jpg\");\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"id\", new arrow.Utf8()),\n new arrow.Field(\"description\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"vector\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n id: \"lancelot\",\n description: \"Portrait of Sir Lancelot\",\n image: imageBytes,\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"character_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n"; +export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imageBytes = await readFile(\"data/sample-frame.jpg\");\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"frame_id\", new arrow.Utf8()),\n new arrow.Field(\"caption\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"embedding\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n frame_id: \"thumb-1\",\n caption: \"Sample frame thumbnail\",\n image: imageBytes,\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"frame_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n"; -export const TsQuickstartQueryFeature = "const features = await table\n .query()\n .select([\"name\", \"role\", \"power_score\"])\n .toArray();\nconsole.table(features);\n"; +export const TsQuickstartQueryFeature = "const features = await table\n .query()\n .select([\"frame_id\", \"video_id\", \"quality_score\"])\n .toArray();\nconsole.table(features);\n"; -export const TsQuickstartVectorSearch1 = "// Search for examples similar to a \"wise magical advisor\"\nconst queryVector = [0.2, 0.8, 0.4, 0.9];\n\nconst result = await table\n .search(queryVector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n"; +export const TsQuickstartVectorSearch1 = "// Search for frames similar to \"a gripper closing on an object\"\nconst queryVector = [0.2, 0.8, 0.4, 0.9];\n\nconst result = await table\n .search(queryVector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n"; -export const RsQuickstartAddData = "let more_data = vec![Character {\n id: \"4\".to_string(),\n name: \"Morgana\".to_string(),\n role: \"Sorceress\".to_string(),\n description: \"Powerful sorceress of Avalon.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 4,\n },\n vector: [0.3, 0.9, 0.6, 0.8],\n}];\ntable\n .add(characters_to_reader(schema.clone(), &more_data))\n .execute()\n .await\n .unwrap();\n"; +export const RsQuickstartAddData = "let more_data = vec![Frame {\n frame_id: \"4\".to_string(),\n video_id: \"clip-2\".to_string(),\n caption: \"a gripper lowering a cup onto the table\".to_string(),\n stats: Stats {\n motion: 5,\n sharpness: 4,\n exposure: 4,\n framing: 2,\n },\n embedding: [0.3, 0.9, 0.6, 0.8],\n}];\ntable\n .add(frames_to_reader(schema.clone(), &more_data))\n .execute()\n .await\n .unwrap();\n"; -export const RsQuickstartAddFeature = "table\n .add_columns(\n NewColumnTransform::SqlExpressions(vec![(\n \"power_score\".to_string(),\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n .to_string(),\n )]),\n None,\n )\n .await\n .unwrap();\n"; +export const RsQuickstartAddFeature = "table\n .add_columns(\n NewColumnTransform::SqlExpressions(vec![(\n \"quality_score\".to_string(),\n \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\"\n .to_string(),\n )]),\n None,\n )\n .await\n .unwrap();\n"; -export const RsQuickstartCreateTable = "use lancedb::database::CreateTableMode;\n\nlet schema = characters_schema();\nlet table = db\n .create_table(\"characters\", characters_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n"; +export const RsQuickstartCreateTable = "use lancedb::database::CreateTableMode;\n\nlet schema = frames_schema();\nlet table = db\n .create_table(\"frames\", frames_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n"; -export const RsQuickstartCurateWithMetadata = "let curated = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.magic >= 4\")\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n"; +export const RsQuickstartCurateWithMetadata = "let curated = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.motion >= 4\")\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n"; -export const RsQuickstartData = "let data = vec![\n Character {\n id: \"1\".to_string(),\n name: \"King Arthur\".to_string(),\n role: \"King\".to_string(),\n description: \"Leader of Camelot and wielder of Excalibur.\".to_string(),\n stats: Stats {\n strength: 4,\n magic: 1,\n leadership: 5,\n wisdom: 4,\n },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n Character {\n id: \"2\".to_string(),\n name: \"Merlin\".to_string(),\n role: \"Wizard\".to_string(),\n description: \"Advisor and prophet with deep magical knowledge.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 5,\n },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n Character {\n id: \"3\".to_string(),\n name: \"Sir Lancelot\".to_string(),\n role: \"Knight\".to_string(),\n description: \"Legendary knight known for courage and combat skill.\".to_string(),\n stats: Stats {\n strength: 5,\n magic: 1,\n leadership: 3,\n wisdom: 3,\n },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n"; +export const RsQuickstartData = "let data = vec![\n Frame {\n frame_id: \"1\".to_string(),\n video_id: \"clip-1\".to_string(),\n caption: \"a robot arm reaching toward a cup\".to_string(),\n stats: Stats {\n motion: 3,\n sharpness: 4,\n exposure: 4,\n framing: 3,\n },\n embedding: [0.7, 0.1, 0.9, 0.7],\n },\n Frame {\n frame_id: \"2\".to_string(),\n video_id: \"clip-1\".to_string(),\n caption: \"a two-finger gripper closing on a cup\".to_string(),\n stats: Stats {\n motion: 5,\n sharpness: 4,\n exposure: 5,\n framing: 2,\n },\n embedding: [0.2, 0.9, 0.4, 0.9],\n },\n Frame {\n frame_id: \"3\".to_string(),\n video_id: \"clip-2\".to_string(),\n caption: \"a robot arm holding a cup steady\".to_string(),\n stats: Stats {\n motion: 2,\n sharpness: 3,\n exposure: 3,\n framing: 4,\n },\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n];\n"; -export const RsQuickstartDefineStruct = "use std::sync::Arc;\n\nuse lancedb::arrow::arrow_array::types::Float32Type;\nuse lancedb::arrow::arrow_array::{\n FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,\n RecordBatchReader, StructArray,\n};\nuse lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};\n\n// Define structs representing the data schema\n#[derive(Debug, Clone)]\nstruct Stats {\n strength: i8,\n magic: i8,\n leadership: i8,\n wisdom: i8,\n}\n\n#[derive(Debug, Clone)]\nstruct Character {\n id: String,\n name: String,\n role: String,\n description: String,\n stats: Stats,\n vector: [f32; 4],\n}\n\nfn characters_schema() -> Arc {\n Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"name\", DataType::LargeUtf8, false),\n Field::new(\"role\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(Fields::from(vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n\nfn characters_to_reader(\n schema: Arc,\n rows: &[Character],\n) -> Box {\n let ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.id.as_str()));\n let names = LargeStringArray::from_iter_values(rows.iter().map(|row| row.name.as_str()));\n let roles = LargeStringArray::from_iter_values(rows.iter().map(|row| row.role.as_str()));\n let descriptions =\n LargeStringArray::from_iter_values(rows.iter().map(|row| row.description.as_str()));\n\n let strength = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.strength));\n let magic = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.magic));\n let leadership = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.leadership));\n let wisdom = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.wisdom));\n let stats_fields: Vec = vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ];\n let stats = StructArray::new(\n stats_fields.into(),\n vec![\n Arc::new(strength),\n Arc::new(magic),\n Arc::new(leadership),\n Arc::new(wisdom),\n ],\n None,\n );\n\n let vectors = FixedSizeListArray::from_iter_primitive::(\n rows.iter()\n .map(|row| Some(row.vector.iter().copied().map(Some).collect::>())),\n 4,\n );\n\n let batch = RecordBatch::try_new(\n schema.clone(),\n vec![\n Arc::new(ids),\n Arc::new(names),\n Arc::new(roles),\n Arc::new(descriptions),\n Arc::new(stats),\n Arc::new(vectors),\n ],\n )\n .unwrap();\n\n Box::new(RecordBatchIterator::new(\n vec![Ok(batch)].into_iter(),\n schema,\n ))\n}\n"; +export const RsQuickstartDefineStruct = "use std::sync::Arc;\n\nuse lancedb::arrow::arrow_array::types::Float32Type;\nuse lancedb::arrow::arrow_array::{\n FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,\n RecordBatchReader, StructArray,\n};\nuse lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};\n\n// Define structs representing the data schema\n#[derive(Debug, Clone)]\nstruct Stats {\n motion: i8,\n sharpness: i8,\n exposure: i8,\n framing: i8,\n}\n\n#[derive(Debug, Clone)]\nstruct Frame {\n frame_id: String,\n video_id: String,\n caption: String,\n stats: Stats,\n embedding: [f32; 4],\n}\n\nfn frames_schema() -> Arc {\n Arc::new(Schema::new(vec![\n Field::new(\"frame_id\", DataType::LargeUtf8, false),\n Field::new(\"video_id\", DataType::LargeUtf8, false),\n Field::new(\"caption\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(Fields::from(vec![\n Arc::new(Field::new(\"motion\", DataType::Int8, false)),\n Arc::new(Field::new(\"sharpness\", DataType::Int8, false)),\n Arc::new(Field::new(\"exposure\", DataType::Int8, false)),\n Arc::new(Field::new(\"framing\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"embedding\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n\nfn frames_to_reader(\n schema: Arc,\n rows: &[Frame],\n) -> Box {\n let frame_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.frame_id.as_str()));\n let video_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.video_id.as_str()));\n let captions =\n LargeStringArray::from_iter_values(rows.iter().map(|row| row.caption.as_str()));\n\n let motion = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.motion));\n let sharpness = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.sharpness));\n let exposure = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.exposure));\n let framing = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.framing));\n let stats_fields: Vec = vec![\n Arc::new(Field::new(\"motion\", DataType::Int8, false)),\n Arc::new(Field::new(\"sharpness\", DataType::Int8, false)),\n Arc::new(Field::new(\"exposure\", DataType::Int8, false)),\n Arc::new(Field::new(\"framing\", DataType::Int8, false)),\n ];\n let stats = StructArray::new(\n stats_fields.into(),\n vec![\n Arc::new(motion),\n Arc::new(sharpness),\n Arc::new(exposure),\n Arc::new(framing),\n ],\n None,\n );\n\n let embeddings = FixedSizeListArray::from_iter_primitive::(\n rows.iter()\n .map(|row| Some(row.embedding.iter().copied().map(Some).collect::>())),\n 4,\n );\n\n let batch = RecordBatch::try_new(\n schema.clone(),\n vec![\n Arc::new(frame_ids),\n Arc::new(video_ids),\n Arc::new(captions),\n Arc::new(stats),\n Arc::new(embeddings),\n ],\n )\n .unwrap();\n\n Box::new(RecordBatchIterator::new(\n vec![Ok(batch)].into_iter(),\n schema,\n ))\n}\n"; -export const RsQuickstartMultimodalBytes = "use lancedb::arrow::arrow_array::BinaryArray;\n\nlet image_bytes = std::fs::read(\"data/sir-lancelot.jpg\").unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_vectors = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"lancelot\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Portrait of Sir Lancelot\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::(\n image_vectors\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box = Box::new(RecordBatchIterator::new(\n vec![Ok(image_batch)].into_iter(),\n image_schema,\n));\nlet multimodal_table = db\n .create_table(\"character_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n"; +export const RsQuickstartMultimodalBytes = "use lancedb::arrow::arrow_array::BinaryArray;\n\nlet image_bytes = std::fs::read(\"data/sample-frame.jpg\").unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"frame_id\", DataType::LargeUtf8, false),\n Field::new(\"caption\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"embedding\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_embeddings = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"thumb-1\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Sample frame thumbnail\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::(\n image_embeddings\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box = Box::new(RecordBatchIterator::new(\n vec![Ok(image_batch)].into_iter(),\n image_schema,\n));\nlet multimodal_table = db\n .create_table(\"frame_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n"; -export const RsQuickstartOutputArray = "let result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n"; +export const RsQuickstartOutputArray = "let result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n"; -export const RsQuickstartQueryFeature = "let features = table\n .query()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"power_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n"; +export const RsQuickstartQueryFeature = "let features = table\n .query()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"quality_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n"; -export const RsQuickstartVectorSearch1 = "use lancedb::arrow::IntoPolars;\nuse lancedb::query::{ExecutableQuery, QueryBase, Select};\n\n// Search for examples similar to a \"wise magical advisor\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n"; +export const RsQuickstartVectorSearch1 = "use lancedb::arrow::IntoPolars;\nuse lancedb::query::{ExecutableQuery, QueryBase, Select};\n\n// Search for frames similar to \"a gripper closing on an object\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n"; diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx index 051368525..c5f362a6b 100644 --- a/docs/web/sql/data-types.mdx +++ b/docs/web/sql/data-types.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Data Types" description: "How Lance columns appear to SQL, including the types SQL has no name for." icon: shapes keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "json"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - There is no SQL type layer over Lance storage. The table provider hands DataFusion the dataset's Arrow schema directly, so a column is the same object to the query engine as it is to the format. diff --git a/docs/web/sql/ddl/database.mdx b/docs/web/sql/ddl/database.mdx new file mode 100644 index 000000000..6f4ab557f --- /dev/null +++ b/docs/web/sql/ddl/database.mdx @@ -0,0 +1,50 @@ +--- +title: "Database" +sidebarTitle: "Database" +description: "ATTACH DATABASE: bind a database to a storage location." +icon: database +keywords: ["attach database", "database", "sql", "ddl"] +availability: + embedded: unavailable + enterprise: available +--- + +A database is the catalog's top-level container: it holds +[namespaces](/sql/ddl/namespace), which hold tables, views and other objects. +`ATTACH DATABASE` binds a name to a storage location so its objects become +reachable. + +## ATTACH DATABASE {#attach} + +```sql +ATTACH DATABASE analytics WITH (location = 'analytics') +ATTACH DATABASE IF NOT EXISTS analytics WITH (location = 'analytics') +``` + +`location` is the only supported option, and the only way to say where a +database lives: a relative subdirectory, never an absolute path or a URI. + + +An absolute path (`/tmp/db`), an `s3://` or `file://` URI, a Windows path +(`C:/db`), a `..` segment, an empty string and a doubled slash (`sub//db`) +are all refused. + + +`IF NOT EXISTS` makes the statement a no-op rather than an error when the +database is already attached. + + +A quoted name reaches characters a bare identifier cannot, such as +`` ATTACH DATABASE "team/search" WITH (location = 'team-search') ``. + + +## What to read next {#next} + + + + CREATE SCHEMA inside an attached database. + + + Creating tables once a database and namespace exist. + + diff --git a/docs/web/sql/ddl/function.mdx b/docs/web/sql/ddl/function.mdx index caf4c4cd5..1593c2d7b 100644 --- a/docs/web/sql/ddl/function.mdx +++ b/docs/web/sql/ddl/function.mdx @@ -17,6 +17,15 @@ AS 'def f(x): return x * 2' WITH (pip = 'numpy==1.26.4', python_version = '3.11') ``` +A function takes more than one parameter; each one binds a separate column +or literal where the function is called: + +```sql +CREATE FUNCTION ratio(numerator float64, denominator float64) RETURNS float64 +LANGUAGE python +AS 'def ratio(numerator, denominator): return numerator / denominator' +``` + Parameter and result types are Arrow type names, not SQL ones: `int64`, `float64`, `utf8`, `bool`, `date32`, and lists such as `list` or `fixed_size_list`. `BIGINT` is refused. A parameter accepts nulls @@ -51,6 +60,20 @@ version is unaffected. the function is baked into an image before the version becomes usable. +## Call it {#call} + +Once registered, a function is callable by name in any query, the same as a +built-in scalar function: + +```sql +SELECT id, f(x) AS doubled FROM my_table +``` + +This is also the fastest way to check a new registration before binding it to +a column: call it over a few rows, or over literal arguments, before +committing to [a computed column](/sql/ddl/table#computed-column) that fills +from it. + ## SHOW FUNCTIONS {#show} ```sql diff --git a/docs/web/sql/ddl/grant.mdx b/docs/web/sql/ddl/grant.mdx new file mode 100644 index 000000000..85da7832a --- /dev/null +++ b/docs/web/sql/ddl/grant.mdx @@ -0,0 +1,77 @@ +--- +title: "Grant" +sidebarTitle: "Grant" +description: "GRANT and REVOKE: privileges on Secrets and Functions." +icon: user-lock +keywords: ["grant", "revoke", "privileges", "authorization", "sql", "ownership"] +--- + +Enterprise [SQL runs in LanceDB Enterprise.](/sql) + +```sql +GRANT [, ...] ON { SECRET | FUNCTION } TO [, ...] +GRANT [, ...] ON SCHEMA TO [, ...] +REVOKE [, ...] ON ... FROM [, ...] +``` + +```sql +GRANT USAGE, UPDATE ON SECRET api_key TO ROLE readers, writers +GRANT CREATE FUNCTION ON SCHEMA analytics TO ROLE data_science +REVOKE USAGE ON FUNCTION embed FROM ROLE readers +``` + +## Privileges {#privileges} + +| Privilege | Allows | +| --- | --- | +| `OWNERSHIP` | everything on the object | +| `USAGE` | using it: binding a Secret's value, calling a Function | +| `UPDATE` | changing it | +| `CREATE SECRET` | creating a Secret inside a `SCHEMA` | +| `CREATE FUNCTION` | creating a Function inside a `SCHEMA` | + + +`OWNERSHIP` cannot be revoked, only granted to someone else, which transfers +it: an object always has exactly one owner. `GRANT OWNERSHIP` is refused with +more than one grantee for the same reason. + + +These are a subset of the full privilege model; see +[Authorization](/governance/authorization#privileges) for privileges on +tables, views and namespaces, which this statement does not grant. + +## Grantees {#grantees} + +```sql + ::= [ ROLE ] | [ { USER | GROUP } ] { | ID '' } +``` + +```sql +GRANT USAGE ON SECRET api_key TO ROLE readers, USER alice, GROUP ID 'g1' +``` + +A grantee is a role, or a user or group named by its identity-directory name +or its directory ID. A grantee with no leading `ROLE`/`USER`/`GROUP` keyword +takes the kind of the one before it, so `TO ROLE a, b` names two roles -- +but the first grantee in the list always needs its keyword. An unquoted name +is folded to lowercase before it is looked up; a quoted name is looked up +exactly as written. + +## Naming the target {#targets} + +A `SECRET` or `FUNCTION` target can be qualified with its namespace and +database, the same as anywhere else: `SECRET analytics.api_key`, +`SECRET db1.analytics.api_key`. A `SCHEMA` target is the namespace itself. +There is no `TABLE` target: table privileges are granted through +[Authorization](/governance/authorization), not this statement. + +## What to read next {#next} + + + + The full privilege model, and how creation privileges resolve upward. + + + Creating the schemas these privileges apply to. + + diff --git a/docs/web/sql/ddl/index.mdx b/docs/web/sql/ddl/index.mdx index fb84e881c..dd900ee52 100644 --- a/docs/web/sql/ddl/index.mdx +++ b/docs/web/sql/ddl/index.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Index" description: "CREATE INDEX and DROP INDEX: one column, with the algorithm chosen by USING." icon: bolt keywords: ["create index", "drop index", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ## CREATE INDEX {#create} ```sql diff --git a/docs/web/sql/ddl/job.mdx b/docs/web/sql/ddl/job.mdx index 323dc1b1e..1b66e1e29 100644 --- a/docs/web/sql/ddl/job.mdx +++ b/docs/web/sql/ddl/job.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Job" description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL." icon: hourglass-half keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A [job](/basics/jobs) is not a catalog object (it has no schema and is not namespaced), but it has its own SQL statements, grouped here the same way the catalog objects are. @@ -32,7 +33,7 @@ SHOW ERRORS ON documents SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100 ``` -Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports +Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports fewer filled rows than you expected. **Errors and limits** diff --git a/docs/web/sql/ddl/materialized-view.mdx b/docs/web/sql/ddl/materialized-view.mdx index a8dfb8fbe..25fcab76e 100644 --- a/docs/web/sql/ddl/materialized-view.mdx +++ b/docs/web/sql/ddl/materialized-view.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Materialized View" description: "CREATE MATERIALIZED VIEW and SHOW MATERIALIZED VIEWS; refreshing is a REFRESH statement." icon: table keywords: ["materialized view", "create materialized view", "show materialized views", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A materialized view stores a query's result as rows, kept current by a refresh. It is the counterpart to a [view](/sql/ddl/view), which stores only the query. @@ -28,6 +29,29 @@ Once created, a materialized view can be queried like a table. Use the statements below to list, refresh and drop it. For SDK methods, see [Materialized views](/management/materialized-view#create). +{/* +Draft for automatic materialized-view refresh. Enable this section, and +"ALTER MATERIALIZED VIEW" below, when the server feature is released. + +Set scheduling options at creation, continuing the join and chained +Functions from [Functions](/management/function#chain): + +```sql +CREATE MATERIALIZED VIEW frames_captioned AS +SELECT f.frame_id, extract_frame(v.video, f.ts) AS jpeg, + caption(extract_frame(v.video, f.ts)) AS caption +FROM videos v JOIN frames f ON v.id = f.video_id +WHERE f.selected = true +WITH (auto_refresh_enabled = false, auto_refresh_interval_seconds = 5) +``` + +`auto_refresh_enabled` and `auto_refresh_interval_seconds` work the same way +here as for a [computed column](/sql/ddl/table#automatic-refresh-options): +the interval is a minimum pacing after the last successful refresh, and +automatic execution needs the same server-side switch enabled. Omitting +`WITH` leaves automatic refresh at the server's default. +*/} + ## SHOW MATERIALIZED VIEWS {#show} ```sql @@ -45,9 +69,37 @@ on a `db://` connection. REFRESH MATERIALIZED VIEW chunks ``` -See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax, +See [REFRESH MATERIALIZED VIEW](/sql/dml/refresh-materialized-view) for the full syntax, and compatibility notes for older server versions. +{/* +Draft for automatic materialized-view refresh. Enable this section when the +server feature is released; see the note on "CREATE MATERIALIZED VIEW" above. + +## ALTER MATERIALIZED VIEW {#alter} + +```sql +ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = true; +ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_interval_seconds = 300; +ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = false; +``` + +The whole view toggles together; there is no per-output granularity the way +a multi-output Function column has one setting per output. SET takes effect +immediately and does not itself run a refresh: + +- **Disabling** prevents new automatic refreshes. An inflight one may still + finish, and `REFRESH MATERIALIZED VIEW` keeps working manually. +- **Re-enabling catches up the entire backlog in one pass**, the same as a + computed column's [automatic refresh](/sql/ddl/table#automatic-refresh-options): + the next scheduler tick picks up every source change accumulated since the + view was disabled, immediately, with no further write needed to trigger it. + +A materialized view's *initial* population, at `CREATE` or when the server +runs it, happens regardless of `auto_refresh_enabled`; only refreshes after +that are gated by the setting. +*/} + ## Dropping one {#drop} A materialized view **is** a table, so it is dropped with @@ -55,9 +107,8 @@ A materialized view **is** a table, so it is dropped with stored rows, unlike dropping a [view](/sql/ddl/view#drop). -**Enterprise job type:** `mat_view_refresh` for a refresh, or `sql` on a -deployment that rebuilds views through SQL. Dropping one runs the same job as -dropping any table, `drop_table` or `drop_object`. +**Enterprise job type for a refresh:** `sql`. Dropping, by contrast, runs the +same job as dropping any table, `drop_table` or `drop_object`. ## What to read next {#next} @@ -66,7 +117,7 @@ dropping any table, `drop_table` or `drop_object`. Creating one from the SDK. - - How a refresh works, locally and in LanceDB Enterprise. + + How a refresh works, in LanceDB Enterprise. diff --git a/docs/web/sql/ddl/namespace.mdx b/docs/web/sql/ddl/namespace.mdx index 472f3c1ad..dd50031ac 100644 --- a/docs/web/sql/ddl/namespace.mdx +++ b/docs/web/sql/ddl/namespace.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Namespace" description: "CREATE SCHEMA and DROP SCHEMA: the SQL spelling of namespace DDL." icon: folder-tree keywords: ["namespace", "schema", "create schema", "drop schema", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA`, the standard's name for the same concept. @@ -46,6 +47,9 @@ default) states that requirement rather than changing it. ## What to read next {#next} + + Attaching the database a namespace lives in. + The SDK equivalent, and how a namespace fits the catalog hierarchy. diff --git a/docs/web/sql/ddl/table.mdx b/docs/web/sql/ddl/table.mdx index 2d1ac6cbc..b1a5e8f1e 100644 --- a/docs/web/sql/ddl/table.mdx +++ b/docs/web/sql/ddl/table.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Table" description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE." icon: table keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ## CREATE TABLE {#create} ```sql @@ -50,7 +51,7 @@ ALTER TABLE t ADD COLUMN doubled AS value * 2 `AS ` declares a [computed column](/feature-engineering/add-computed-column). The column is committed empty and the expression is stored, not evaluated: declaring one on a billion-row table costs what declaring one on an empty table -costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it. +costs. [`REFRESH COLUMN`](/sql/dml/refresh-column) fills it. {/* Draft for automatic computed-column refresh. Enable this section when the @@ -74,19 +75,35 @@ and is minimum pacing after a successful refresh of that output; unchanged inputs do not force work. A new column is immediately eligible for its first refresh. The server's separate automatic-refresh switch must also be enabled. -These options also work beside a Function binding's `version` and `branch`: +These options also work beside a Function binding's `version` and `branch`, +continuing the [`caption` Function](/management/function#chain) from Functions: ```sql -ALTER TABLE t ADD COLUMN (label, score) AS classify(text) +ALTER TABLE frames ADD COLUMN caption AS caption(extract_frame(video, ts)) WITH (version = 'fv_1', auto_refresh_enabled = true, auto_refresh_interval_seconds = 300); -ALTER TABLE t ALTER COLUMN score SET auto_refresh_enabled = false; +ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_enabled = false; ``` Declaration settings initialize every output in the group; later SET commands -change one output. Disabling automatic refresh prevents new submissions while -allowing inflight jobs to finish and explicit REFRESH COLUMN to run. Column -SUSPEND/RESUME syntax is not supported. +change one output. SET takes effect immediately and does not itself read or +write any row: + +- **Disabling** prevents new automatic submissions. Inflight jobs may still + finish, explicit `REFRESH COLUMN` keeps working, and rows written while + disabled commit with no computed value -- they do not queue anywhere + separate. +- **Re-enabling catches up the entire backlog in one pass.** The next + scheduler tick processes every row left unfilled since the column was + disabled, immediately, with no further write required to trigger it: a + column disabled for an hour and re-enabled fills everything that + accumulated during that hour, not just rows written after the moment it + is re-enabled. + +Column SUSPEND/RESUME syntax is not supported; `SET auto_refresh_enabled` is +the only toggle. The current setting is in the column's own field metadata, +`computed_column.auto_refresh_enabled`, readable from `DESCRIBE` or the +SDK's `table.schema` without submitting a job to ask. */} {/* @@ -148,7 +165,7 @@ ALTER TABLE t SET RTE WITH (maintained_indexes = '') ALTER TABLE t UNSET RTE ``` -The [realtime engine](/tables/realtime-engine) puts a write-ahead log in front +The [real time engine](/tables/realtime-engine) puts a write-ahead log in front of the table, making small frequent writes cheap by batching them out of the commit path. `sharding` and `buckets` spread the log so concurrent writers do not contend on one place; `bucket` sharding names its `column`, which must be diff --git a/docs/web/sql/ddl/view.mdx b/docs/web/sql/ddl/view.mdx index ec923063b..c6c1f7ecb 100644 --- a/docs/web/sql/ddl/view.mdx +++ b/docs/web/sql/ddl/view.mdx @@ -4,10 +4,11 @@ sidebarTitle: "View" description: "CREATE VIEW and DROP VIEW: a saved query, evaluated when read." icon: eye keywords: ["create view", "drop view", "sql", "catalog"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A view stores a **query**, not rows. Reading it runs the query against current data, so a view is never stale, and never faster than the query it wraps. It is the counterpart to a [materialized view](/sql/ddl/materialized-view), which diff --git a/docs/web/sql/dml/delete.mdx b/docs/web/sql/dml/delete.mdx index 429271de3..29b8f65d9 100644 --- a/docs/web/sql/dml/delete.mdx +++ b/docs/web/sql/dml/delete.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Delete" description: "DELETE FROM ... WHERE: rows are marked, not erased." icon: trash keywords: ["delete", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ```sql DELETE FROM documents WHERE created_at < TIMESTAMP '2025-01-01 00:00:00' ``` diff --git a/docs/web/sql/dml/insert.mdx b/docs/web/sql/dml/insert.mdx index 44007c6a3..71a7d1182 100644 --- a/docs/web/sql/dml/insert.mdx +++ b/docs/web/sql/dml/insert.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Insert" description: "INSERT INTO, including INSERT ... SELECT to move rows server-side." icon: plus keywords: ["insert", "insert into", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ```sql INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur') diff --git a/docs/web/sql/dml/refresh.mdx b/docs/web/sql/dml/refresh-column.mdx similarity index 63% rename from docs/web/sql/dml/refresh.mdx rename to docs/web/sql/dml/refresh-column.mdx index d454069a9..db5d26475 100644 --- a/docs/web/sql/dml/refresh.mdx +++ b/docs/web/sql/dml/refresh-column.mdx @@ -1,19 +1,18 @@ --- -title: "REFRESH" -sidebarTitle: "REFRESH" -description: "Filling computed columns and bringing materialized views up to date." +title: "REFRESH COLUMN" +sidebarTitle: "REFRESH COLUMN" +description: "Filling a computed column: idempotent, scoped, and resumable." icon: arrows-rotate -keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"] +keywords: ["refresh column", "backfill", "computed column"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - -LanceDB extensions with no DataFusion equivalent. They are what makes +A LanceDB extension with no DataFusion equivalent. It is what makes [feature engineering](/feature-engineering) reachable from SQL: a column declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it. -## REFRESH COLUMN {#refresh-column} - ```sql REFRESH COLUMN ON [TABLE] [WHERE ] @@ -33,7 +32,7 @@ can recompute values the failed run had already written. `WHERE` scopes the refresh, which is how you fill a slice and check it before committing to the whole table. -### Options {#refresh-column-options} +## Options {#options} | Option | Meaning | | --- | --- | @@ -62,39 +61,22 @@ memory and GPU options there. */} -## REFRESH MATERIALIZED VIEW {#refresh-materialized-view} - -```sql -REFRESH MATERIALIZED VIEW -``` - -```sql -REFRESH MATERIALIZED VIEW chunks -``` - -A refresh updates the view's stored result. Refresh views in dependency order, -starting with their source views. - - -Some older server versions accepted `WITH (full = true)` and `CASCADE`. -Those options are not accepted by the SQL implementation described here. -Use the basic statement above; do not assume that the options accepted by a -local SDK's `refresh()` method are also SQL options. - - ## Watching a refresh {#watching} -Both statements submit work that outlives the call. Track it with +A refresh submits work that outlives the call. Track it with [`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with [`SHOW ERRORS`](/sql/ddl/job#errors). ## See also {#see-also} - + The same operation from the SDKs, and what a refresh guarantees. Declaring the columns this fills. + + Bringing a view's stored result up to date, instead of a column. + diff --git a/docs/web/sql/dml/refresh-materialized-view.mdx b/docs/web/sql/dml/refresh-materialized-view.mdx new file mode 100644 index 000000000..8e1886f23 --- /dev/null +++ b/docs/web/sql/dml/refresh-materialized-view.mdx @@ -0,0 +1,50 @@ +--- +title: "REFRESH MATERIALIZED VIEW" +sidebarTitle: "REFRESH MATERIALIZED VIEW" +description: "Bringing a materialized view's stored result up to date." +icon: arrows-rotate +keywords: ["refresh materialized view", "cascade"] +availability: + embedded: unavailable + enterprise: available +--- + +A LanceDB extension with no DataFusion equivalent. + +```sql +REFRESH MATERIALIZED VIEW +``` + +```sql +REFRESH MATERIALIZED VIEW chunks +``` + +A refresh updates the view's stored result. Refresh views in dependency order, +starting with their source views. + + +Some older server versions accepted `WITH (full = true)` and `CASCADE`. +Those options are not accepted by the SQL implementation described here. +Use the basic statement above; do not assume that the options accepted by a +local SDK's `refresh()` method are also SQL options. + + +## Watching a refresh {#watching} + +A refresh submits work that outlives the call. Track it with +[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with +[`SHOW ERRORS`](/sql/ddl/job#errors). + +## See also {#see-also} + + + + Creating the view this refreshes. + + + The same operation from the SDKs, and automatic refresh. + + + Filling a computed column, instead of a view. + + diff --git a/docs/web/sql/dml/update.mdx b/docs/web/sql/dml/update.mdx index b527d1ec0..68512e155 100644 --- a/docs/web/sql/dml/update.mdx +++ b/docs/web/sql/dml/update.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Update" description: "UPDATE ... SET ... WHERE, and why it does not recompute derived columns." icon: pen keywords: ["update", "sql"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ```sql UPDATE documents SET title = upper(title) WHERE category = 'film' ``` diff --git a/docs/web/sql/dql/aggregation.mdx b/docs/web/sql/dql/aggregation.mdx index fe02dc17a..cc41397b4 100644 --- a/docs/web/sql/dql/aggregation.mdx +++ b/docs/web/sql/dql/aggregation.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Aggregation" description: "GROUP BY, HAVING, and the aggregate function set." icon: layer-group keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - Use SQL aggregates to summarize records, either across a whole table or within groups. Common examples include counts by category, totals and averages. diff --git a/docs/web/sql/dql/changes.mdx b/docs/web/sql/dql/changes.mdx new file mode 100644 index 000000000..012c06ff8 --- /dev/null +++ b/docs/web/sql/dql/changes.mdx @@ -0,0 +1,76 @@ +--- +title: "Changes since a version" +sidebarTitle: "Changes" +description: "CHANGES: read the rows appended to a table since an earlier version." +icon: clock-rotate-left +keywords: ["changes", "append", "incremental", "cdc", "version"] +availability: + embedded: unavailable + enterprise: available +--- + +`CHANGES` reads the rows appended to a table since an earlier +[version](/management/table#versioning), without rescanning rows that were +already there: + +```sql +SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42) +``` + +This returns the rows appended after version 42, up to the table's current +version at the time the query runs. + +## Strictly append-only {#strict} + +The clause is named for what it requires, not just what it does: every +change to the table between the starting version and the current one must +be an append. If any of them inserted with changes other than a pure append +-- an update, a delete, a merge -- the query is refused rather than silently +returning an incomplete or wrong result. Re-run it with a later starting +version that is itself after the non-append write, or read the table in full +instead of incrementally. + +`INFORMATION => APPEND_ONLY` is the only supported value: it names what kind +of history the clause is willing to read, not a choice among several. `AT +(VERSION => ...)` takes a positive integer; there is no `AT (TIMESTAMP => +...)` form for this clause, unlike other version qualifiers. + +## Where the clause can go {#placement} + +It attaches to a table in `FROM`, including inside a subquery, a CTE, a join, +a set operation, or `EXPLAIN`: + +```sql +WITH recent AS ( + SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42) +) +SELECT count(*) FROM recent + +SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 10) a +JOIN events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 20) b + ON a.id = b.id +``` + +Each occurrence of a table keeps its own starting version, even when the same +table is referenced more than once in one statement. + +## Unsupported forms {#refused} + +| | | +| --- | --- | +| `CHANGES (INFORMATION => DEFAULT)` | only `APPEND_ONLY` is supported | +| `AT (TIMESTAMP => ...)` | only `AT (VERSION => ...)` is supported | +| `AT (VERSION => 0)` | the starting version must be positive | +| `... END (VERSION => ...)` | not supported; the current version is always the end | +| `CHANGES ... TABLESAMPLE (...)` | cannot be combined on the same table reference | + +## What to read next {#next} + + + + CREATE, ALTER, CLONE and versioning basics. + + + The rest of the query grammar this clause attaches to. + + diff --git a/docs/web/sql/dql/dedupe.mdx b/docs/web/sql/dql/dedupe.mdx new file mode 100644 index 000000000..c82e72c13 --- /dev/null +++ b/docs/web/sql/dql/dedupe.mdx @@ -0,0 +1,99 @@ +--- +title: "DEDUPE" +sidebarTitle: "DEDUPE" +description: "The dedupe() table function: one representative row per near-duplicate group, scanned over the index." +icon: layer-group +keywords: ["dedupe", "vector_dedup", "sql", "deduplication", "near-duplicate", "table function"] +availability: + embedded: unavailable + enterprise: available +--- + +`dedupe` scans a vector column's [index](/indexing/vector-index) partitions +for rows closer than a distance threshold, resolves the overlapping pairs it +finds into duplicate groups, and returns one representative row per group -- +without a full pairwise scan or a shuffle. See +[`duplicate_pairs`](/sql/dql/duplicate-pairs) to inspect the pairs themselves +before committing to this resolution. + +```sql +SELECT * FROM dedupe(
, , + distance_threshold => + [, resolution => 'connected_components' | 'direct_representative'] + [, selection => 'all' | 'first' | 'random', count => , seed => ] +) +``` + +```sql +SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02) +``` + + +Earlier releases spelled this `vector_dedup`; that name is rejected on +current deployments. + + +## Input {#input} + +| Argument | Type | Required | Meaning | +| --- | --- | --- | --- | +| table name | string literal | yes | the table to scan | +| vector column | string literal | yes | the indexed vector column to compare | +| `distance_threshold` | float | yes | maximum distance for a pair to count as close | +| `resolution` | `'connected_components'` (default) \| `'direct_representative'` | no | how overlapping pairs group together -- see [Resolution](#resolution) | +| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan | +| `count` | int | no | number of partitions to sample when `selection => 'random'` | +| `seed` | int | no | random seed when `selection => 'random'` | + +`id_column` is refused on `dedupe`: it has no separate pair-identifier output +to project onto, only [`duplicate_pairs`](/sql/dql/duplicate-pairs#input) +accepts it. + +## Output {#output} + +One row per duplicate group, **in the table's own schema** -- every column of +`
`, including a BlobV2 column, comes through unchanged on the +representative row Lance selected for that group. Unlike +[`duplicate_pairs`](/sql/dql/duplicate-pairs#output), there is no fixed +`dedupe`-specific schema to document: the output schema is the source +table's schema. + +Because the result is already shaped like the table, it composes directly +into a new table or a materialized view: + +```sql +CREATE MATERIALIZED VIEW frames_deduped AS +SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02) +``` + +## Resolution {#resolution} + +`resolution` chooses how overlapping pairs group together: + +| `resolution` | | +| --- | --- | +| `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together | +| `direct_representative` | only pairs within the threshold of each other resolve together | + +## Sampling partitions {#sampling} + +`selection` scopes the scan to a sample of partitions instead of all of them, +the same `all` (default) / `first` / `random` choice as +[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a +duplicate rate on a large table before scanning it in full: + +```sql +SELECT * FROM dedupe('frames', 'vec', + distance_threshold => 0.02, selection => 'random', count => 100, seed => 42) +``` + +## What to read next {#next} + + + + Inspect the close pairs themselves instead of the resolved groups. + + + When to reach for this instead of exact-key deduplication. + + diff --git a/docs/web/sql/dql/duplicate-pairs.mdx b/docs/web/sql/dql/duplicate-pairs.mdx new file mode 100644 index 000000000..e8651a28c --- /dev/null +++ b/docs/web/sql/dql/duplicate-pairs.mdx @@ -0,0 +1,97 @@ +--- +title: "DUPLICATE_PAIRS" +sidebarTitle: "DUPLICATE_PAIRS" +description: "The duplicate_pairs() table function: every close pair in a vector column, scanned over the index." +icon: clone +keywords: ["duplicate_pairs", "vector_duplicate_pairs", "sql", "deduplication", "near-duplicate", "table function"] +availability: + embedded: unavailable + enterprise: available +--- + +`duplicate_pairs` scans a vector column's [index](/indexing/vector-index) +partitions for every pair of rows closer than a distance threshold, without a +full pairwise scan or a shuffle. It returns one row per close pair, not one +row per source row -- use it to inspect what would be collapsed before +committing to [`dedupe`](/sql/dql/dedupe), which resolves these same pairs +into groups. + +```sql +SELECT * FROM duplicate_pairs(
, , + distance_threshold => + [, id_column => ] + [, selection => 'all' | 'first' | 'random', count => , seed => ] +) +``` + +```sql +SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02) +``` + + +Earlier releases spelled this `vector_duplicate_pairs`; that name is rejected +on current deployments. + + +## Input {#input} + +| Argument | Type | Required | Meaning | +| --- | --- | --- | --- | +| table name | string literal | yes | the table to scan | +| vector column | string literal | yes | the indexed vector column to compare | +| `distance_threshold` | float | yes | maximum distance for a pair to count as close | +| `id_column` | column name | no | project this column's value instead of the internal row ID -- see [Application-level identifiers](#id-column) | +| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan | +| `count` | int | no | number of partitions to sample when `selection => 'random'` | +| `seed` | int | no | random seed when `selection => 'random'` | + +`resolution` is refused on `duplicate_pairs`: a pair has no notion of a group, +only [`dedupe`](/sql/dql/dedupe#input) accepts it. + +## Output {#output} + +One row per close pair found, in a fixed schema independent of the source +table: + +| Column | Type | Meaning | +| --- | --- | --- | +| `row_id_a` | uint64 | internal row ID of the first row in the pair, the same identifier as the [`_rowid`](/sql/data-types#pseudo-columns) pseudo-column | +| `row_id_b` | uint64 | internal row ID of the second row in the pair | +| `distance` | float32 | distance between the pair's vectors, at or below `distance_threshold` | +| `segment_id` | int64 | which index segment the pair was found in | +| `partition_id` | int64 | which index partition the pair was found in | +| `dataset_version` | int64 | the dataset version the scan ran against | + +### Application-level identifiers {#id-column} + +A row ID is internal and not stable across compaction. Pass `id_column`, a +top-level scalar column such as a primary key, to project its value instead: +`row_id_a`/`row_id_b` are replaced by `id_a`/`id_b`, typed like `id_column`. + +```sql +SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02, id_column => 'frame_id') +``` + +## Sampling partitions {#sampling} + +`selection` scopes the scan to a sample of partitions instead of all of them, +the same `all` (default) / `first` / `random` choice as +[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a +distance threshold or duplicate rate on a large table before scanning it in +full: + +```sql +SELECT * FROM duplicate_pairs('frames', 'vec', + distance_threshold => 0.02, selection => 'random', count => 100, seed => 42) +``` + +## What to read next {#next} + + + + Resolve these same pairs into one representative row per group. + + + When to reach for this instead of exact-key deduplication. + + diff --git a/docs/web/sql/dql/explain.mdx b/docs/web/sql/dql/explain.mdx index 1fc4dc191..151a861bc 100644 --- a/docs/web/sql/dql/explain.mdx +++ b/docs/web/sql/dql/explain.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Explain" description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE: reading and measuring a plan from SQL." icon: map keywords: ["explain", "explain analyze", "explain verbose", "query plan"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ```sql EXPLAIN SELECT * FROM documents WHERE identifier < 1000 EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000 diff --git a/docs/web/sql/dql/fts.mdx b/docs/web/sql/dql/fts.mdx index f6d5bc696..3dc6d3a3e 100644 --- a/docs/web/sql/dql/fts.mdx +++ b/docs/web/sql/dql/fts.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Full-Text Search" description: "The fts() table function: match, fuzzy, phrase, boolean, boost and multi-match queries from SQL." icon: magnifying-glass keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - Full-text search is a **table function**, `fts(table, query)`, rather than a predicate: it produces a relevance score alongside the matching rows, and a `WHERE` clause has nowhere to put one. The two string-literal arguments are diff --git a/docs/web/sql/dql/join.mdx b/docs/web/sql/dql/join.mdx index 5ccf49f50..ecc81ec31 100644 --- a/docs/web/sql/dql/join.mdx +++ b/docs/web/sql/dql/join.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Joins" description: "Combining rows from more than one table, including across namespaces." icon: link keywords: ["join", "inner", "left", "outer", "cross", "namespace"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - Use SQL joins to combine rows from two or more tables. ## The join types {#types} @@ -70,3 +71,29 @@ Avoid selecting blob or vector columns you do not need on either side of a join. The join multiplies rows before projection is applied in some plans, so a wide column can be materialized many more times than the result has rows. + +## Late materialization for wide columns {#late-materialization} + +The planner can avoid the multiplication above: when a join narrows down to +a key column (such as `_rowid` or another narrow, selective column) before a +wide one is needed, it can defer reading a BlobV2 or vector column until +after the join has already settled which rows survive, instead of reading it +for every candidate row first. In a plan where this applies, the wide column +is read once per surviving row, where the worker runs, and never pulled into +the query node to be joined directly. + +```sql +SELECT d.title, b.caption +FROM documents AS d +JOIN (SELECT _rowid, caption FROM captions WHERE source = 'model-v2') AS b + ON b._rowid = d._rowid +``` + +This is query planning, not a syntax you opt into: a join written this way +is a **candidate** for the planner to defer the wide read, not a guarantee. +Confirm it for a specific query with [`EXPLAIN`](/sql/dql/explain) -- +`RemoteTake` in the plan means the deferral happened; its absence means the +wide column was read eagerly, even if the SQL above matches exactly. Neither +a passing query nor an `EXPLAIN` that omits `RemoteTake` tells you how much +I/O a plan did or did not save -- read the plan's own row and byte counts for +that. diff --git a/docs/web/sql/dql/sampling.mdx b/docs/web/sql/dql/sampling.mdx index 03e518f1b..954247142 100644 --- a/docs/web/sql/dql/sampling.mdx +++ b/docs/web/sql/dql/sampling.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Sampling" description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments." icon: dice keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - Use `TABLESAMPLE` to explore a subset of a table or estimate aggregate statistics. The sampling method determines how much data is read and whether the table's physical layout influences the sample. diff --git a/docs/web/sql/dql/select.mdx b/docs/web/sql/dql/select.mdx index cfc02c9c4..621061204 100644 --- a/docs/web/sql/dql/select.mdx +++ b/docs/web/sql/dql/select.mdx @@ -4,10 +4,11 @@ sidebarTitle: "SELECT" description: "Projection, filtering, ordering, and composing queries out of other queries." icon: table-columns keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - ```sql [WITH cte AS (...)] SELECT [ALL | DISTINCT] select_expr [, ...] diff --git a/docs/web/sql/dql/subqueries.mdx b/docs/web/sql/dql/subqueries.mdx index 48887da72..3c5b5d11d 100644 --- a/docs/web/sql/dql/subqueries.mdx +++ b/docs/web/sql/dql/subqueries.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Subqueries" description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables." icon: code-branch keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A subquery is allowed anywhere an expression or a relation is. ## As a relation {#derived} diff --git a/docs/web/sql/dql/window.mdx b/docs/web/sql/dql/window.mdx index 14b052861..e90f601c6 100644 --- a/docs/web/sql/dql/window.mdx +++ b/docs/web/sql/dql/window.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Window Functions" description: "Compute over a set of rows related to the current one, without collapsing them." icon: chart-simple keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A window function computes across a set of rows related to the current one and returns a value **per row**. That is the difference from [aggregation](/sql/dql/aggregation), which collapses the rows it covers. diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx index 9f7b8985b..9f6a25229 100644 --- a/docs/web/sql/functions.mdx +++ b/docs/web/sql/functions.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Functions" description: "Where to find each kind of function, and the dialect version they come from." icon: function keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org), currently **version 54**. The function set is DataFusion's default registration plus what Lance adds. @@ -77,6 +78,29 @@ like any other column (order by it, filter on it, project it) and exists only in a relation that came from `fts()`. See [Data types](/sql/data-types#pseudo-columns). +### `dedupe` and `duplicate_pairs` {#dedupe} + +Scan a vector column's [index](/indexing/vector-index) partitions for rows +closer than a distance threshold, without a full pairwise scan or a shuffle: + +```sql +SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02) +SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02) +``` + + +Earlier releases spelled these `vector_duplicate_pairs` and `vector_dedup`; +those names are rejected on current deployments. + + +`duplicate_pairs` returns every close pair it finds, in a fixed schema. +`dedupe` instead resolves overlapping pairs into duplicate groups and returns +one representative row **in the table's own schema** per group, so its result +composes directly into `CREATE TABLE ... AS` or +`CREATE MATERIALIZED VIEW ... AS`. See [`DEDUPE`](/sql/dql/dedupe) and +[`DUPLICATE_PAIRS`](/sql/dql/duplicate-pairs) for the full input and output +schema, resolution grouping, and partition sampling. + ### `generate_series` {#generate-series} ```sql diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 5793b5235..74561fca3 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -32,7 +32,7 @@ for batch in reader: SQL results arrive as Arrow record batches. Iterating over the reader avoids collecting the entire result in application memory. See -[Queries](/basics/query) for result handling. +[Read and manage results](/basics/sql#results) for more. ## Usage {#usage} @@ -47,6 +47,7 @@ operations that have no SQL form, such as a merge insert. See | | | | --- | --- | +| [Database](/sql/ddl/database) | `ATTACH DATABASE` | | [Namespace](/sql/ddl/namespace) | `CREATE SCHEMA`, `DROP SCHEMA` | | [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` | | [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` | @@ -67,7 +68,8 @@ Draft for the Functions release. Add this row to the table above, after Index: | [Update](/sql/dml/update) | `UPDATE ... SET` | | [Delete](/sql/dml/delete) | `DELETE FROM` | | [Copy](/sql/dml/copy) | `COPY` (refused by recent deployment versions) | -| [Refresh](/sql/dml/refresh) | filling computed columns and materialized views | +| [Refresh column](/sql/dml/refresh-column) | filling a computed column | +| [Refresh materialized view](/sql/dml/refresh-materialized-view) | bringing a view's stored result up to date | ### DQL {#dql} @@ -80,6 +82,7 @@ Draft for the Functions release. Add this row to the table above, after Index: | [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set | | [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions | | [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments | +| [Changes](/sql/dql/changes) | `CHANGES`: rows appended since an earlier version | | [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms | | [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` | diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx index 3e27e6bb2..156fab3b0 100644 --- a/docs/web/sql/operators.mdx +++ b/docs/web/sql/operators.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Operators" description: "Comparison, logical and arithmetic operators, literals, and precedence." icon: plus-minus keywords: ["operators", "literals", "precedence", "null", "cast", "interval"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - LanceDB uses DataFusion's SQL operators. This page describes comparison, logical and arithmetic operators, literal values, type conversions and operator precedence. diff --git a/docs/web/sql/prepared-statements.mdx b/docs/web/sql/prepared-statements.mdx index 2b94b9c9a..cc0b1257e 100644 --- a/docs/web/sql/prepared-statements.mdx +++ b/docs/web/sql/prepared-statements.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Prepared Statements" description: "What a deployment does when a Flight SQL client prepares a statement or binds parameter values." icon: bolt keywords: ["prepared statement", "flight sql", "parameters", "bind"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - A Flight SQL client can prepare a statement before it runs it. A deployment accepts that, but it keeps no plan between runs and does not accept parameter values. @@ -50,6 +51,13 @@ Do not put untrusted input, such as a value from a user or a request, directly into the statement text. Quoting it does not protect the statement the way binding a parameter would. + +This limitation is specific to the Flight SQL protocol's `PREPARE`/bind step. +The Python SDK's `execute_query(sql, parameters=...)` binds parameter values +through a separate, LanceDB-specific mechanism -- see +[Parameters in SQL queries](/basics/sql#parameters). + + ## When to use prepared statements {#when} Prepare a statement when your client requires it, or to get a statement's diff --git a/docs/web/sql/scalar-functions.mdx b/docs/web/sql/scalar-functions.mdx index f44c935c4..1af20bb96 100644 --- a/docs/web/sql/scalar-functions.mdx +++ b/docs/web/sql/scalar-functions.mdx @@ -4,10 +4,11 @@ sidebarTitle: "Scalar Functions" description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds." icon: calculator keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise [SQL runs in LanceDB Enterprise.](/sql) - Functions that take a value and return a value, one row at a time. The set is based on DataFusion's registration plus Lance and deployment-specific functions. The names below are selected examples, not an exhaustive catalog @@ -98,6 +99,21 @@ functions supported by your deployment and inspect SQL index-backed plan. See [SQL vector search](/basics/sql#no-vector-sql). +A third, optional argument is a comma-separated options string. `metric` +selects `l2` (default, alias `euclidean`), `cosine`, or `dot`; the +index-search rewrite also reads `nprobes` and `use_index`: + +```sql +SELECT id FROM documents +ORDER BY distance(embedding, ARRAY[0.1, 0.2, 0.3], 'metric=cosine,nprobes=20') +LIMIT 10 + +-- Force a flat scan, bypassing the index: +SELECT id FROM documents +ORDER BY exact_distance(embedding, ARRAY[0.1, 0.2, 0.3], 'use_index=false') +LIMIT 10 +``` + ## Structs and fields {#structs} ```text @@ -146,6 +162,44 @@ semantics and the simple tokenizer. query, the planner uses it instead of scanning, which is the reason to prefer it over `LIKE` for keyword tests. +### `ivf_partition(vector, ...)` {#ivf-partition} + +```sql +SELECT ivf_partition(embedding, selection => 'random', count => 100, seed => 42) AS partition, + array_agg(id) AS ids +FROM documents +GROUP BY partition +``` + +Assigns each row to one of `count` IVF-style partitions over `vector`, using +the named arguments `selection` (partition-center strategy; `'random'` is +sampled with `seed`) and `seed`. Grouping by its result buckets rows by +vector proximity without building or querying a vector index — useful for +blocking a large table into comparable chunks, such as before a pairwise +similarity pass. + +### `blob_descriptor(column)` {#blob-descriptor} + +```sql +SELECT id, blob_descriptor(video) AS descriptor FROM videos +``` + +A [BlobV2](/management/table#managed-blobs) column's bytes are not fetched by +a plain `SELECT`: that already returns `large_binary`, the same as any other +binary column. `blob_descriptor` instead returns a small struct describing +where the payload lives, without reading it. Pass one row's descriptor to +the SDK's blob file reader to open that payload directly: + +```python +rows = db.execute_query("SELECT id, blob_descriptor(video) AS descriptor FROM videos").read_all().to_pylist() +with table.open_blob_file(rows[0]["descriptor"]) as reader: + payload = reader.read() +``` + +A descriptor is pinned to the table version it was read at: it keeps +resolving correctly even after the table is appended to and checked out to a +newer version. + ### JSON accessors {#json} | Function | Returns | diff --git a/docs/web/static/assets/images/enterprise/architecture.png b/docs/web/static/assets/images/enterprise/architecture.png deleted file mode 100644 index aa00854da4ffddafd3861f6fcb2866cec653f7f9..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 99871 zcmeFZ30RE%`!_u9(2eMhC?(kVP-eNV?{a?6^K*XA#qY#% zt>sHamf~=@k{zx27=-QLFCsp8$#TSoiY2Y~MPkc%8BY0)WHSKfGIGk_{`ny2KFo%gi+z9Kv6&nR!AJ z@U$b9&^H#jjdX~?D7n{?s1Vl$^ZzMr*%Y>L{y$rEC6~0o=9t?N?EW#>x|Zbr zVsnRgkrE9~x`oEExU;KHy;vt?wQAc7Q=|iMi9R64e1~d99m#tr-pa8zZyu#S$KejV zI=*ivsbD(c?!$c-E~8g)xD$V^_)_9YAI|Z|`4}{_hnlBvcE4(=Yk1_k9p2pKa1LK$ zfgK4RkZvEAyOXi?j<-Ey#q76*$;qZ7I9x!aQipr?*H_QnI*5ud^iy?;kOsl&hQJVp zw7PRUgPNC%I@2;`N_Nz6hKeO)_R7=h(Xg%DXb%Y9V?PS6&+I8A>V$`n@~ac2(R(py zopbazoKNUAG9mV<^052xOim>_*zpVDfS2N?*p=b6vExzbMW0Zop^@3+RM4=nLS3K? zEdKCj`fje5nbQOojeq>NB0!4+5{KzZAMf+$&v%6!Uh#u&e)MRiPp0FQD<3`a*4Eb4 zVD6eFr6A$P`MevX(E^8uho>n77RCL?^N8!L+}vEJG65|OO-;?yXU;q~1^We;Tn$pZ z2mA)VfA{qqDNzy;73G~|3=9ky6}vl`Tzj|Zq_J^~>kRhU(!H0Mib$ru*eRlcl&#Va z(A%o;{eXdnhM}S1Zbii-TOav-dbF*)B3wGLs7RTn#nt0v0cq?ZKZb?^^u4-P)ipJ_ ze5O!Wop{^(*tqMV*81%g6*}3Mt1AXmQ&NNfsu~QWFU$aw0&Lr&9rLI@084vXF1E)@&ITP+ieEwWw(s?%27MVA+v2A-atDifa%+o-b_OioR`xXH0o z^!!Y5pQzJ1g-7THhtq!lnO(RC?v--y-o5Vw0~LdgHa7ddp0&S|>oqf009!RS?A#rw zL@xFiU%_C&kVor}%1#a{!>AGx5-xtGd-wtND4;S_)l$4ZomZZLV)z#))8@OoMnjyE| zSNL+vPAA!%>-AyVPpuF>FgQFMG&x8q=iyiRqrJVo4t8q93y;TFMemV*aV%y{a-26K zge;CtCv+=94%>gwXw}OeYiR6NQrfmg`++;*Lz1XN!NxT*Cc;sATV+Zn>Zkt_B8If6 zHfNZM4mPI`Qu6W4se|_~A}L_c&MbO=&ePW3zB|F8&sbSWV*(NcrF?a`-oav`Tb!Zvh& zPYe#q%gak><_g{2-_dE*btt0c6%#>jSq{q;O{}b}Bm^|5=-9G<@^j>nBJer}b8dtg zxD!)TrC%I>dh^7y80n0H1dHWOO-)R^Ih;5X_iuMK#S)DjOI%3Zt47&NkAk=}@zqcE zIWI(j7jUia^D*&umFB{y9%O6&B1zVtR8^(}z3!YQnAgjX=y)7Q<*Kdi_*wr~DJi}Z zM<16}%92d3S;fOWv=n2+48%Z$D#etPl**4OwF_oj*9JX!@W6y{?%cV?=4LW{cpXUH zh)CybpOXKsBVh5neJqt6JBm|YzWhNUCD))^>Ju0Z#dKK4b33sO^AW&6@OD^fp)tm; zD7}+;4$^xJO`J5%%t83mjIz3XYR%7{UBjxq(1-yZ+}!{cX%rduiArO{XTqXxoGctl&9g6HfQzWIO{h7l>vMbbQe|obzIcz$=ung4@EP%&& z-|~T6G_2~<99m-+#9n46^UupfVB!;DTzT3q7n?jaDJiLhmoHzMo7>vjPE1X$__6$pJK^}-IMn)@=J8fUM9yzyDrhBw^#=EPU62UGZ${o zl_~goyO_AsIFIj#i1kyK=7Z{|w6wG~S!U-Sl`Yww=?DtoVfE!zBq8ihf_IPsk=q0( z5mapdB){9;l+8hVH)W1Ik1!LNV5X*~CZ^e4Q9fTAy(jf!;smilUtKjb6ZRuWdAK0q z&v@)E8*C0USGj}M7Z7Lvyg$xx9cp>#dQNk)LON4$Xk=(;UGrp~LwhuZ4)#?8b}zoe z{EBoQPt&xqfGS@ort zt0Es#@?k%a3ZXs^EO>b;d8=zW*ouQB*QB$mB0*pvke$`r`bbFcf$tT2`@rVrW?$cr zmv%Xk*IJh)tFwMgx0I_X4uvOk2d|Uo$O8vtolE0Mn+UnCrRjEIFCLzsyi)KAG;iqLFVP#Mgz34IxsL$-esT>kiZ|uu^VhyJvR=|p{zwr5V7*6 zDPvGdPJL0GIzsIWDk>@xo;}+Lng&$jW<`fKVR|Ql;K0mdoy&JHRbhNjtn27d z%TP1KW|pLn#KhnPoFh1T9`8c8#d+-lO&{Uvp?OkLt`SB7r~_2jRXE8O`#vI<2u{n=DK|H_;45O5X8`Y^ky2Y*6W_kwKQh}LtRCy?#~s#Y zOf!eVgJHuxSOE8Jy688*Kf8Ys5S73v`_(?rzjQXQ{T=pJs-?5FRr%tc0A2(nYgOqz zllggh=Ovb*JNXD~#*8hagHDW~P^eT9SSr7EWK3j;(_O`Eg<2Ko()bfuU`+Za+dcZ9 zZa)9sUCwiA==QCALo*KvuGfJR-iE;;P0$@O1x?L=-Cfq*7vre*eGn$21)2pGqu4pW z-L=(MedMl?StNzh(y}*gwHMgxgy+w#T2fU@e*U%0Po)9i!b-73kn^_eJ-c^nXla$d z&NvmYIv<<1#cM9;0@>DIHYIENcDLH(Oa0}Zp4qHLp`aNP0nC)eO{QhnNVO2??QZ-( z_JakHx2}1Dt?}Q)IV58gsmX`djL-Mgb|+EPf0pv_#8zI;1~YuU7EbZo+Xpk=Neb>) zo@%jN<#8E%{N6thhSfd?$~Wy?_KUBDaN;86Ey*EgD?eS%&br=Pu@^*h+2T+*Nl1Rf zt#^WOfSoR8-(4m|KFqJWI|V60@;0?mKW{vk=+O}ldxVE*-JEvrrHUhnrdcQZtg94! z>w~`#eb5#Tc(}CT3Y$GI0R_NuAEAKZ76Oa^G60h*Tz4=R8H4rF0E_a@8Ey(wzz}|n zBXQsU{ree>ShKJH z3C}&p{mRPSv*We3%Z$$$8kWC0lWKs^I25GLo-QWBoLLzIIgXOsyI4DiYt)!SS8Q#S zzp9GE%_9}8kDGC9lAk`Fx`wToT+p=j8}Jb~A+O^-XicA62_XAMZ*Tl~o!QygC@6h4 zO`9p7n3lFN4qm=`A4Nf za6Yww++A|e+XUL0GL|5h}O!C-(* zSgnoeT#K@qM>fd+1Ifv{_~Ai8*8}N`m5+Bi!5JwBET)>$^avTFqvi~Q9Y(J$j4r@2 ze1QbYtn&a2BAlWWmfi^J_#z0ZQ~y)96$Px84jAXfx&+OWr%ow+_dI?2bdTz(XK(K- zD1vX$v!F8{p0l?%>{M5mzwG*Jp1FrC6+vsbPc|7Qf=(YUp?56EpOW5v1g))(gAFfQ ziqh$89|0b^C7Uk9g(Z+}Nn+`uibO*3Y{lNx;Q-(#d_gfFU&YrKykkNjsJ-nk^rBXZ z!ztdh>7HQY_A{Qbqv`n_4*;^Jq^AcW8xOKT*8BJZ62TyEpi-xjez9X;MJoq!NAN3WIEp(I+^Z1X))H@WTV}zk3gD+p}lS zOGneQL)K~O>BowmBnUgWr-201JF@;bfE`PVq3HkSS~1y%fBrwt7W*8Wo-}rRZptN0 zaHT055wVuS`Z!wH9n>GIU$hPm-dcNgW`Eo0Nv-u!uKUo!glzy*B3wG6mr)~@@6aOr z#k8m%5p%kS2JZzHcI(G_oTv7p51zOUs>lK-u9}r+&LqaEgt>wshPxuvXL(ySiHTxJ&&%_L@8||F@&j06O%N(qTOJ$KJ>+@t~Hb)ZKcGjrEjgMMDRc@#-LznuHW9F9FQZ{NL>JnFNE5u?JRysw{QH|C*AIZ@*CDERf2^k(?K+MeNF+YZ zviG#sd$M<CyPmCIX$tQppVn;)E4j z?~nB-O^=!PQQg{Zp7NTeTPK;WL2Cef=@J{{C|uh>_{mEWGJpG8jpD#A5>H0tDO|6` z5!^cPNqOk=;chnqy~V`yuRjlC8wT&T@FQ5jz5W-5Ni0rB!cw`ly}e+`6%8DYiDV5I zuvC23!h_4k;&68mu{(GDFAVGiEY%Y8xI}UVhJ8*);^c5>9V|{u%BrWCd-m5nwTa-4 zgzv!rmiUtJo~>uj_5~Gjz#TVTfi1z@kQ(>SBu2G48BzwhB7wvG$04P>$ytW;^+iU8 zIYJ_tcY-CnI5qUeCW6bg!B-!is26T%Z53U;dbLf&g2T=2?PAM>WQHh)=H{}$);=-h zM7?Aqe9Ab_&X2%ICOj}_k40nwuv}nn3>Z$qx+Vb81*2KP_#I%P>Sf|qHh=ysw~1c( z<3)_unt3LjZO~uhrw!ZsHm77Yv^SE z=Ad&v_i8x8h1eBv0r*ZI9>f!`(pQlO7D^%MfWbxpiin8aJ6zBW2pqAk)L>pd1B9b; za(dWYcnJ{^iLldMfZjGe`lJC&WCEpq)$#qzq3lJEEW!}^v%{o7j?U%n;EYKz-NI9V zrA0>Fp*8o`4&%HXY5>;D;i2O-C6mp@P9r6Baxh`Gd%@vj0L+})oXRylJ&R~Lk}1GA zj@BLloI?%XBc%#}+$I7nZ!g8MBz%r^EyevAs@n)~bV^kFs)Iui(49PH$$5E-XVZ)X z0O>n*c=WI4+Xlc<6Cf7CQWX_mTfUFvwK^alEvItBruE1e8ws-U8$jELTRPp8={-MJ&`MXP2BhSX=libd1OTs9z>MqktRL^I_K#yvBqO)F*twI@ zoPV!)=_x8;d`p0Ih}r!RB&K{3J_gMA>g3p)S2c=BciVv!GtWu?0m$AG-YT4$muvpY zf@Tm?Q6+}}aN{jYYc;?P>7utYv%w*gEt4h8R7UQxJ*yYJdwbCSffO#MaN?V+_o)BG zV!=5XyQeMr^3O}zUl6byV<<%FnYezj^6N+iD2@$0LWV}^?C4kzx8}Kb6~-|Bm)_*$ z9%BM78ZJF-%6>esVM+po|v2Kaw5}wj&Kz=knDlr$=aXifG_9<$L)HW zalXvR9@6v&*)eh+2u^dRxq6ZlwH16UK}207P}bRk@6!cS2mptJhnNR0^qL(;f52gR ze{fMVh)7>fPOHv4XEZ_e~q+q@~yv2#6OJ)ITObJOID zqP6ON+#IZ8AlG<}V(PZqDm<=IrYP;Jq61-6i12Xg#JUqlj~>-~y5HsVtt4OHrc^`Z z`~nyS&<`gIf*i%ApVpf*>LbP&aE8In_H3aSkCH{l8Hmffsd=SC{I^jXr% zv0%=0um?U$?}cQmo~A}vc=!i!d?t(8up*($kAipt$9T9y>gXMCHV`3&9K;Khkm}2c z&z@Z%Z43hjG$k{$c98RppRjkRzeCCAjH&6<)*w5T)>V90?d;YyGmGB7?V4k$Hdrk^{M^1viM%j3+dfe2t z9&n<;n+~|0x(d8bty8B?A&&}e0{A+?yy9N3E#xFObv<#e@k35=;l2J?PnlTvF293| zPRfvG$BK!F-!!TvI(3taiQqsBSuIMhOUlo8E4BE1AkGgC=xs_gDBk$|jdnIRe*2vJ z1o>XT)HR)*orhy-G#YZaZl{Lkf+Gn|=hBQu!27F#%qrw8{lc7$<=lItx^ZrS1uLhI z?+yzAf8_Rp`_8qh`6=}DloZK=-X?OOsDIv}hHL5Pb6_kaaZM#J&w2bb9LQ>et~=fnITfBGy&T0P6m_B9H|{-_9>C}1)N-G z>_B2o6%#?e4rZ=D4K(B+C9{^PhjbKZX={Cp+gRG(4^S z)q5xb6My_xnz9?v_HcpAd-6B)-?24@fN9c7#STl4IV%{s5;=d^kcF3r&$*;22#fvd3Ikye`?QRp9A~B+LxU; zTeH>?qt)GjAzs0}$#lrhb^P9cU}?Hnj{01z!W++Q-DKyeKKqS-oFP-#6M8|Nc@5O? zo)!CgLdw{pLsM6or><0%fYQ0qLTRN_ZU9ZX$lUPCqO3#Di0}4x`EDL#EVb|EERy_zBy+wP^CL7*M@5#v8;_MNnkAWg5&`ij`D zcmH~;n5gq;tHjMBKghMxSI?#P?XJjJou*lSL9xdR zgsn@{eI_6dF|ll&i<_H}m*Yxl^|@s%HQF4e@h<%}ocP5o>TZaC@O!l8vSv_%!NE-T zQ&(Ekd0s-W8pId2pGN0{U0Y=6@IbQIr4{(zh!%z5%Xe>Ec?Q$8MXsdgf0Gmpn)7vi z>cH%4pn!r09-?RdK#C`HdX@)?dEj{#S&?5MdIktF(?(Ad=juHk@1juHBIR*w-=EyjRLk1#200_Bk%FWev#mz-= z->AUbn04NV4bQoMscIkW5P+$-=0AJjKuBX@+mFjqOuT1$UpAJ2j!Ymdo?8u{EI?)& ze1eRYw=TAIJR9%k#W8xesYgwP3>kvO*>OHcGHYdiCIQ#OkC$L?9rXvuc5^Ed;lNCFaI0pp}d=k)=+KA5Of@h+R z#qzL&hB=;jxw)%{I1qeeC8)vD8S@XY6fRe3+fT;AdAZ=|$dxl(J@HwRu|wf=bLOBq z=cZKc`BzI>X$Xpi*Oq0KuNSqlVjC{SZSGCk96UrB_UNuoQ5pDn!E8BKgFFVWwv>R- zn;8nf-2lO$#X=z5&CGo3qqTB3vK`$o7v$NuDt@P|dDRQ436j)E!Be1EA!IT+XbYyv zPBF$-i29bc%zXL~^?9%-PkIQ7b|pdDQ-E}D@BK;Jk+wnw8jz)M`TPdk^f7`s@%%Q6 zV&-Xoa+m_H5rCJPLDImh)Wu%40$S@Hek_5gsmgas?9G9fnjY*NOf>MIvNVR>JJVZ1 zO7E5IYA`w@S!y^GA8kGlsD)KP9U6d!1B_caXb$KCc4Lb-30#2NEMO$f>f$Erw8X_G zY?m!Mfk3`@>axJC2>pVhq7SyNI$d!&+rv12^-}pjH*wZ?t#JlAVhM2Gh-#URIT_SN z*d%bc4_m@cUnGKMTQj5%d~^a3hgnP%zX!lJ&Y+~7Q~0WQMnIr^y{Da7CxG{RH%7o$ zD5iayO8%^h;1Hz6tZmK~a#k*Whk`>~CLiK+u(JSGwlM;aK|D6nc)Qe+ZBK6)n3;pg z%E`@LK6vvEOHXj=M^m?Um-kCQT4jF$Cv@lVfq8AH3#+RKCj~+eTQf=Aus({`@ZwR7 z;0UM)Gej%{9lz!wC|&`wbe?UUgNJ1-A{DMg#=sdBCEk3-uPk_0p8hm zcLyNFA>b>QCU?lu(Gk#wAm0{EjW$TixU;nM31m-r;F%8l6hlC(K;%RGIt@~PDt?7T z7$OJ?H=ncCEMp!F1)yCHc?)Rd;@4GqAs0mo1c@<;A2jZK0B_ORC^CF~G2hzdg7>1;3G9&D9-f(}n<01r`5A#I^NGnSlo4F))f;a3 zVE5a(`CPjOTXF(21R(OThHs-rv3T*EJ8gC1pf8|)2%K82^Pc|3TH-B^V4i0Q|!BAh9SSo^Njg`wDqA zd@mm-q2t1$GVkR=s1jv^8MI!(qHznJ0i5NNN_$*=dD%SRer@ky%<0Myg~25Eq1 zcmk(3%d$KO@#Y$3J{yxt?5R06q>TZ1kXJPMvES4L1Q4S8r0=~;ZQ>a}`16{xVB;T{ z&vpqpaaeS^XAod{ITUH~9kE7L0R`P7_c>o4?)N6BYvJV9 zRK+sIox4`D*4c3|C7c78Hl#l^8Z?&Px!~Ar^m9SeV2x^o%IwTwBv;iCrL}YyU zc3(V}LWq~snDue-&7&ysepG+Sv8d-5wxnxUV-U)*;Cg$Q?uwPb? z`E23D;-yQLgSUodJ{4|gXUC4iaaa2~xme*cPL4X4w0+=5Hs07z8nhIkYRuIF?!?bT z0mvTki6GwO-O_qzr_t;9udY{l-)wWg5sxDu0Q7nhGH{TybU~c}UjR4Frp$1o)Fp@p zEn>z*M;}oM*?k@X^jZRY93a2Q)E8BPnU;@^;Q(XM+qD6v9a!khBSdG#ft@Uq@+q5MTELaZaLxO)h_9Rf&RL z$b(onF?HPYF9UwH>96r-Bx{M7q`dI+kUwG9NmPaB-`~9$eN)KH{et$lM;Bi~fNXPVvEGy_bt7y_@;st}aE^cg z63^wH&tI3s^?uqG7hvC-_W@vyv$`#uUSK3m8{^z?l8XW%@%p@yofpS_Cfw<4``URr zc<8XC4c`nh5#XA!7z2ZYiHQHI7d`;_umIX~%{yav4nx#LZ;m$Bjia@aLu@k&!CwU2 zUf16K)YXRXD_PXB3d>=vm%(xwKr4TZ|6L+*BeMGM*nkXBibh1a8seMs+JK1ta?p`B zZ{|Z3pC!pB0UHNi@fZWLq`HT)YuD&WZG_k$gy;2pCS`$Bol9?kTd#-)+Be&MvK#x8nRc>Y|V|*xdPyotrig z*$0KKp@D*|86?qEal8xCjb|BLa) z{ogEHIT8+;@Yl;B=KNwVesBVn5#Z!rdHuZ8i`Zy%t;1g9{{CMdR9Y~qJ!pykLTmv0 zkfj;yKK#~gXx>>fJt%rsOGBeUiTx&pJ^+CI%6S*OhDO_@oB9Js)1pmV4aFZ#{egY2 zG!l#CBy}}qP`XGa-kK1V^Yisp-^XeD_(U%I3Y|Gz&DrVJ)-Zl~olDO{8-DKN0?wiQ zDxAT|Gm`{Ot6svwJH(CxDF~*m?lsGFEm;Gz>!ThOGG7Vs{kbYPKqGentCyq{RK9 zjuKBds;1oQSf!2t+YFYg%H+gjjSNcOw|5svZc}Z}M`mZVj7t);`s&V(TuJlDX;SXE zy54>=h#U>w8;lGhkLbt{VN(kld8CYccM_<>yH&QK^X0 zjnQ7eMd1F#3x-FCuYqL^MX^s1nFP>$0h$pe>uI$P$BG1f68_MYIw|Wg81SJoql!uC za;}}Oec9MR$ycClTst1%G+1eU^s3K{Q0ax(>lCX>3@8f!@P-0@|Tm5vW6KpE~G)6EB>)`Rq-9oiD%EBu%KTSTpKBnW;;FfW;Xt$5Xx$Q3( zLSf5*koH^eradUGvIde+Ec9X^zFS55nq4@hN@s6@f)#rs%D`aj7uk^AoZGcaw;-uF zu@VB){^!IwMlcAgLSk}k$Hg-$7FFOuACxVbxK5hV(oZ{vV3e@HQBGHAGH0yNBR}Hl(y^4kjELs$thB`nEtD@DyEP%+y{3;YKmR% zV%_xgA!GnRp`4Tx5+;;35Kus3P~WtT2F1n}(^?kSgIswNLb7j+N}Pm8Q-it6zdH!2 zq7&CaL#z@q&yk}zz-F4YHG7oO@%=@%&gF2sX^Itk6fR^Xip0im*?E5YF1AKW>2K9Q z1!!t;R%_^hgH7wXekCePVDq%*}+E zNzB_ptH;!og~=*ZA*4$+d_3zxn&NIAfN;MTXDIzt7iaz8Pfg`Xrd;z7V9Om?Dv%q~ zAidXVP*4T+3>{Uu1V-5@mKn#;&`1KZC*Z2biU-1q$JLc(p%&{?ZfhJ~#%;Ex+rF^d zU-rB((t4&`3NFBt8`YPd*LtM!nj(+PsMF`xPR4OesL#2}!i}-7gyq#}Z)@OHA;HF+ z)xqBLS;>7e+*6z12da)hFQYLUX z&V8q}>s%hqj*?a2#zP_I4*;|pR zKr#$hsaXXo7Q_^FR0 zOG!(-`y*GqXb-YUt`ieUrb|mBT`C6#H$#D1PNlF=l`ZarwVAXkJhn@L6R6s!4?hXN z$u<8DUn~ZBTw{9O!;G`7R%IhVvm@!Wr|ntxOj{NMLjT~zr3F?b2I&3MrLJO7Gcu~H0kbpwBCCrRvcNeRxeJ~^8cwc6YJZ-7(SZjz+ zggPP^Ss5xDbYRrDK;H0+C<^Yu`{J|IDgy4q1-57_ECl>qXpt5JXi4t;C8*=WO8gv6 zTmK+jXTS9BULN8zHg>psRWjQf3L&zc@%pMI5^x+|m{KFH%y=Z%!`OCl76=B0~*ptKH09TY`Q0ETMyNd)E?uiWBC|1Wg0f695%%ZHl-Xk z#S7b%XIq4mAXTYWj5AvN2zgyAU;}+dKhE;v4#T41aQBW%F1hzDpHPHC0vqJ}_Tb2y z$Hj4cPiIzvDqf0|4fpzI`U?o9k5_ho>?37|)}I#h;1G9WA=^P%xDupt{%ijW^HjKx z_}}}0(+Kc=?sSYM35+;+Y}vF)?OOf0(RX(#eIj_nUjm>&l@Gd}bkndWQn$IeFabU4S`E-1YfIfxVHuX4l5?e05{^T{elxrp5B ze_n1KmiHoD+kq$UbfuRxWja!9TNz@c@d?gH&rVMc8>CQKogle&HraCAECTO+ibGjl z&`hMa(N%dmvP*L-yg7}L*W9h7ESDUx*+p3(w?1+NXK|(KDFEL)U@)&D36Q{Tw&-WH zb`#XcI_b=4x*2{Wqd9%=*W491o~kZ(PIt>!SqUq4$%wMQ$Co2`pF>$gTEM$uQCxEAKzq44Wy9H69 za{-bf=WV#s4p_qZ8UDxffy81%=TKpPPf)9(N+dUV5xymOizK?3ybD!_&Ru?Q3|9XC zXI=hdX9Q%3Eq2^=EGK<1?o?5y?J}s#chH>))jwouKgrN0jNp4E-8=F>%4lV)qw_m&!6f0at}!q;}CZUqq=hL6&49* zr$Oe{${V#MGx+2uUItGTncbtG6gbYO*SlPzN*uyKBlh7^$;rkBy`#4t)rkC@819Ju$gX^3V|Y$*)z(^Hzj8g%X|LX% zoTCdjEa`g-GB=1LuNZGH?XPyj30@pt2V%KO_#U~}@P+Ab5`BgJ9BTwZ0BKIhQZ2Rg z^*d|)CT=}W1(ZhG?^i9Mft@0hJ!FWFb%SLwX)9R$*pM~r~dL+%_vIO=ox6=Ba! zaivBR9j&Ta<80BzDx{+PZM2pCeFSw^>q~K+yT8sHw#KVfyoil-jl!+l@?Tcar~Wh~ z$2tM(vK)OQy?N6`*qqtP<*A83LYM z85Fnbn^Gq(F|N>|bi8lBk9?Vz(pQghG2EgkWnLA>;e{LQqgGue8%CI|QOYRXvuP9Z zg7qm+*Ji7+vvQ7F@@3;i=k}pg*o|%7|6#&rPvMkOCTTmkC(ljSSVn@{eemt`*mS9m z^>X){ZjSnwIK7{8Qg=L9^2oZZ%<{!@FXHv6{|3(oTXjKkM)(6USA-Bx&Y zv@JQXWRAyJ>n~HNvZQvJK~Ym5Yu z;mxVU#eb}FfeDV5dRY^BkR1cltJ?{q1&_8CZ8@0r;S$a>qnw}1=FZRJFGOzO6@d7Y zg6|x51Lo7UFlIS=MO(;{(%^ySE$4^hF}DPoeIxgD4)>naB5&UoCx{Rt+;jS%bKAtz z!Fm~g`H%lH^xYpn2<@yAQwJ&kKG+)NsuOS!IWOF+`(sr%X9G%prJ`hMTBz_HRKLQ# z$PZDsh~c%#y;OabnHy9I`NiS?PX(y1cXK)CV4L#@vs`q&rwZ}yL~VnNpZpOpK*AkA z{K1Tex~_idKGG!ARbprkhVjd&^QPH8QHu!$%5wJ(P<;0yM;5C%5_#e`IQRSXS3iYv z59+F*S}{aC0k3q@%-e`+C4aBwZz~!F1OjPF)RBm0FMn3|H_3a4E4z8NJ>B#5M|+9t zLsrDLhKx6oTlHcERs}~X{+y}!WE1iGft6J*JaI~t1p#7xMWD1saFS(h`+jX z$)*c%f)Rx)kk|qCwgk)|YzSLvAouNOMar~)YrV{7Co2*?Yr51A9FqkHmdO1RY57Bb z;^)6v>jMwNj;YG8;AHO18f+3`OmarbhXd$r{mwr>DIjSNqj%8Rtnlg1;90r}Grxu* zEB{ShEQbDeXgsE?eP?hFwRci_S@RpZsyaZ}ldiK*0PKCvRl9c4bltAH#@|Yr2%7Q! z6PkG){!P2C?32R9uAa3l#+zEl3f2+#F8PM8&yUm}(TM=sj7gopi!<(zU;HbnbGv(R z31Qsy`*~(Z_R#dcJ4O9C%5Od;QUAcaqwoIHqRs`fkgWd?bsw6;=4H`(@4CHL>k|jZFEBWhT(nBu8(~kUV?jn>7O8S{j)7TE60tqg*2+nUUyRFMTEyJ)o#G= zq~c_mK*=`TI^Tb8o#+1u!upR$;$KJjpZN3t6+M>FGKOT@zrQojisKGEoGH3su_N1H z9-RCa7G91Is_x9e2{>>O-24EoK6RY%esEuLa)O6+%h>n2o*z;k_jfO*IaUq3E> zU^I$DUJnGwD6j~^seguAgdm#o)N!@#6rv67{b<8B@6~wx2xP^SrbM*oN_%^Cu5xNp zI)5FHh%Ag|wXfQh3EsX(`i&?K?m_vik}KQ`el96Ba+{C22dq(bEC!oJQ~?AtF&L*f z89CkUv8~*Z;|kRszXeF4!;c7JY{kT26`!qa)cKK`lgD|eP?FX zhI4)T@nO5UgzqS;yVsBmSSiVLrKK+r1t)=$M~1&ahR32R8ptlno}Ix}OB5PptW4x! zkP#l~yU`#kXO$GuAS-8=Bxy?eT?D^}?<6pOPVn4B;CrBaN3QP)qJI0uY*}Q_g@`37 zFFyMgu_Tsk(IUi>07YSu2A;vXKZi0*QEqxU@Ugz!x+tHMFQ+z_mqQIyP;jh~D8|gy z6kIk-3@e%G%k2KKlO})RuRmK#m?e>lE>~xVoy2vEc6_un6UD<5c!AYnN2)Ja5j(R= za`jo=gh+H=d_w7yw(R|ioFPAVND8|6*PlzA+U|RA8=j3B(fm^8H&?|WcXf%xNj)p9 z=I6CK0)6G|&T?lOL^@^7Zq9_e)1SH}rkTu#A0ZiVNwg~cNt;vaJs`#)Ds9k|Gr+jb z8uXr^k6QbS)XVTA1$3{vb2*RYmp`F`1ZWZMU!jje>$q@bB7oCPMYe{hk_ugmFt{i& zsN889ql{{@h;MpoooQmKs#)La%I`a$!#)-NkvpxN5h)$S)gr@)kjG$2Hl zQ^^hcv$3+O_)BTbx=y;W#1HACY_C#E4Unb9bne)mAH<%u1O+34#CP{o5ic$ky#IPB zCF0lgwcJo$0gms~nW9x}KgndPTnQ`MZXQo!{rz(?p83^iRmN54I0q(BEL*!)q ztd5kd`jF3vw5<9LjMC05?48OH=LV2}f0h(V87IeJ`cISI9>Xynb~mu_Nh3DII8{wqj;K z!1yVqp_L~u5I88jrP?u-?M(y17zz%#UmY8J&#+fVQK9ZwPtHb++!TQ*;J_=#v??)v zx9}uYnc0Tg{vcx|y9pMGiur^#LcELJSMwM}Ju@;viM(f&-^4vngWgw?L(z`2Ry*EK zk3sNmm4E>99~?}aZ5#FW$-D~Qw;$z|T`;AC6sVvr{x~uSB=j_dK<&%QpCLUmyNvB_ zG*Z&vK|uONd^9@HX>_?;z~)UNnXD;ail>}q8*1)0isuh-uTHdwH=>E75$|uBpSPUd z)*mMdGhQ)RLUz9fF|@#B`Qx^+YLV#Q^%cVT>P5MoV~q_jj{9-?W0d%q9U~?A$}3bS zct)Wy#1V}$!nrYXOKM{@i)gKa7aGGm|3o**8;Kz{1oL&?eE?CqPIr;+x}SJwM`8ni z?8u?H*j?upCh>!YxK=iPhCWJt$44(a~*QE=7AYIa^VlIya>ov?##e>Ik}xto+( zvawHT7Cg$D2(uvgyMD?=wy4KnKz>42PT6gEC_kO7EixQYh7|1c%Md4gFV^iV>|)v0 zH2GG3ebP7f|Wti zzGn7)qTb z;?Q{}RECw!QO2T27L6$5K(}b&gG(doN*9Kr3Gb}CYm}QIhG^M5U(L!_1M;6pS0X{{ z3CewZV8dmC3RrE?F(wEqx!Yb{bksT{L+xhElHMun{Pm4S%Om7vh~I*oG|C{J9q3-d zt3XEKa?(Txwn^TN;fVwGFV<-|O8%`XoobV;SH-VnN=UzVA{2T0d2ha(6jr zZ8(?(7Vm_djaTbKV?vI7^>V+!(f7f#r)I}b=|l7%5+S^3l?3^(w&e|O8#|SFk$C(~ z5MW~h?#su5#ktJ_g|Y78O|5SFv45?tQY^WJUj1%upry)s;xd|i2YDKW&N^A`mX>{^ z1iA04%b!Jp)Vn%NW&$6xf@!^WDaI-sX@t3HHoO{irlAN@9bx$tsBTu?L88^<icOv@`qyKU6rShwLr}jO_<1nO<7`5%F+U? zok&3B@?f>t*1VmYfMX&Wg)#r7ksucUx~4#M7O>8~uZ6eHo&a6$Q>TH{varDBEU*TM zLBMDM2-769rg1pd5K1yexH3KkvvdMZps|?gp%dW}1hYn_>2z{jEh?if=xpL<@cUfC z`~po1=o{owXW&OQmnFgx^1oRNI&uwWkvs`w7S#7hy#YrJo*zV4Zy<}#qN4C$-N-qJ zzIu)RCEG)Fn~mScVAwt4nw`yDv-9du>J|w4D{i8z^(({;P`rz2);9cH+ZC`3v#=*w zK&vowwJw4#M34~zkdd~?ia;1Oe2SaMQX07p1wsU?A-ad^5er=*s;p%P->6X=d{o$t z0Kw!8Cu-_7Hqc)mPtWJT5jC)~Lg7P!%nM*TP}73B?fB=1KM+&5>@@Y#2|M;AQgBBg8-ar$%5fMo(` zvcf6YtqpT&@+h4F1ZlfionH;CL+`pA$ivwyg}?#8 z0B_?;)-;iKzVe(bh?e(jJgi87viIjFw&q~@L(L>?f-~{GV{J|#PF!rpjkH-}LVoQ; zGwb^ZGj~v(QwSbXSQJn>dXT4(EAHJB_v_Al=*|``OVdM7V_eK~U4YM75c)2h5>vkt z1Yp7@f{TPKZ)0I>yZjDzK8u=Y1(JI_hBfL<hy5I}*;M;<#f!jBD$%=|rvSvnc* z1+x_=vq8!L_J2U!RxW2MLmb02mE5nK#M(LuKQ1xUL0*b;t;$YX=)Km>7$V;nF`6oUm@N&94y?jzMNf$q)x8h`Yr%(H5761csjpoih4ATmBR*k$_JJ zP<(k6$Tx(xG1?bX2J>5a8p~q1*YgX+gN*?GZRO~{VPo$4CqBkTc?D2mARNCwswc0< zk)mJg&l%RDKc64pd2D>~v?5&tlw}bS7Kv!QA6-YU)6n|;{S1U-XW*a*B(KO9!HU94 zr9)@UV7A(?VPc%Y3Q2H3(m?x|6oE`yUPVd?vR^Qz1R=h|7)*YHEssSbS4OY=%H2Sa z)>TLI6oBU5AjEv%(-Qx6yj-vmu%vQy>p|k{Wm2yoZ3q2v`>AfM_TYJ;IwFP%6=|6H z>2UYu@T5bv)Bh&(DVWUjr2y02R?ydM;>lhVC)SXyH?$H&0ZZ|L`N&Wo*m-%{!UfCax{` z3`(fpK~b4Qumo)a=xfBCl*;`Kt9_!w8{flx(bpHpK8K4G%ADSRp9*=+&{Yr1bzsh0{duh280 zi{&6W@gms5jFMq3Lfl+d%!yF6^cGKA|L*33@d-TxhTY^^N3jVLyhbhLM$M_5Pz=3v z8^KHeaCU7X&eRWpy7Z)7qrwSuQpDKm4CDiK}`Vs#NjwmO%-!i5`f8bKPu z$LSSF^lr$AqU7n;e`WpXujNwR06pisXM|z8TsWDloH+vztU&Ot>iq*{%|F?I`(#L; zTZ^Cxg$NC;xs-XJ6!Lm8TjHKOnEPwpuso&izlrn!5`AoMk!1c7ltgY9;)j!f3(xB4 ze!NRy%y3gw!BG`9KR4FTZL9*SA5zUlQF8(k5qd?HLDyO=nNWBe^jIzw~08S0ksxW%v>nSJlqBS;f@!jl|fZnziMYPG;+hTwa*$rvxzY> zY70PnV^KC#l2DseQGuc?XgR6@>tG3OoyV47E{1%?KdLnHNt(P3$_L6~K4sj{&6t8&SV6 zff0^$K0p5ZVS#V`E_=5giw<*L;)iM^QQk8Y3SGye))gOC8}-JlB>`MdawvOeym{B z+JVA=oOhcr##jVt9VB>*WK`?nc-R%j)PosCTMKCu$^1_a32Ngc9$~X0?qkPpv93gNh>c znMdk=`rc!Yoa%M9Es=ir2N;|BHU(%Ylyou^MmU%+)&Y)^fA&C)HcCjFI{Cf!lwtaL z45gje0;v?*+K@L7#Zyy*sKW&mtn<&DhqXYR@TN<7)&77^3^nmQIsyWa5|28%ok~6) zhv+iCFJDh=c8IzRF{SP{k@}QtIErq-)XSdjb;<>5a+8bKCke-l3f)eqG7Wxqqk8#4hFk#?f?eDXpDsQVtY-9&v@|!$nzUNiiD+mBUfv;t->SNa> ziB!~U6jp-87$$MplG38HoXgo??4acIyN|H~bhz!^DeWCkhXP&HGy)1Otc|3Vb8Y(w47vLw%Tl)vU@T zO@&vchla&NG9^l0be2Aeaz(pqS}8B)S|24Hfnp2~XhvcQjbUse-l3i~%1N|$-!q|0 zo2+1CW8?0m(!;PssCR`$!+I#c^oLS3)RXnR-1kOku?xovHDM^|i?&UG)-)q~V2Y^a zBI9l1?AQ<_cO(t~%)8IU&(Xd2`}_U=9*rJ`Fg%q zs`AStlwBWKG=oD!6QR>=<=q}L5K-P|qpx+9Kc4D^e@XDUQ~V>H0DWj%&g;7B*&7Sy zXryOMi($q5+WL!Z@}PAcob>vUoE+yBQ|SnfH=}$%`Nfn2PdOetu#qe~?}*RS3CRT-pG)~*~%<1SvO>>=2XjgNB~ z7v`<4Ibt@Z65+||>P@}oMP~k@r_cQU{b50Vf4>&|ZWyh`OZ9XWMn=Y|dH3Ct;YFhu zOxNtG=dr_9yen$GltfGl184fh_JqQOoC1QJbuB7KbkCIrWqmfdR@g_8{kyEKZSJpM zzZxiRFqm~`#i5AM7dheEw{PdxBep;vE*m2~Auxk(p#L;V9kazEz(LXBCK+_WoLUc�LWX%B_Z(57ZL^nLQ= zNk^tG&I8lTdi!<<)V`%dvjqAKeTlZO&kt}~{rqakBQ)i`^F3mOINkXeshyS}{xvo> zhWvhDJTyd1&B~H{d6--Qnrcym zaaX=7LBo*WNf~F(oN14Zdj?<2jod>Vj=J#ItP zx-czG3TnQJpx;6Yl+k@%IkFI8$^aE+zh`_#)^(lHC#gAp%rhLtZg;%HuJS@(76;8; z==ckj@*ir-g94{_Uura-pM)B7{N0`euK^1@Xs)=Dx#IdvJk(lAS(dLS;AfOs!*YQn3T zhUb}qFkcze4=qCr9A)G=#Gm=V^S!?!Y74CtIGr8u>I$(!5pKkN>rZz|X1^SE5UZ-G zskbO7iITOq&tF_z6mgquM$Twf+4Z9O*EpgUc1Sr-NC;}AH43fb`sO{Uj~zQUJ%qzS zrRm7hM3Y3L%Q#fowL;bLRPm8c*Xz)j;xy^3ViY!`2caOiGf-5e1S>_<+IFr%wDw71hIA2?oUUDk-B*CqBE%N?YC?;V@!!9H zH$_k%;=QPG7yFa;`2X9iXQmU!Xg%42+eeZ`_&U{USF z#Jrko9?QyR3ffI8m2EaQG%R;>v9q_&ulcSjZkFlgzGD?yscmfJbk*%3jx;;b3A%RA ztR~7tEjn){thsWm+7v=3AVYt^e|gSuaymmoLIlknX1lYeJ35qbx{8sFU9rUZ^{sVZ z{H5m5rIZ-~C2eg2eu9sUt#tyet|=uZYQ+uj&Z#G~Jbgw71mjdU(bBZLPdNoj`D4;IAbv6B85Dq4F}&DAQV*x9XX>IlU|6 z0PKr^^n1yk?YzOmQ&f>zyFLe{<%el;SKuBeV}chGe>GUTZ)*DIgFIYM*2ppIxd&~s zBmoPw{j7M2Nnuh_Q%myn^yFn;a}0`!$pk&@cY*EOyg4m}H}pF1D95%ot zRKzAc>smKe6)`>j>eZ`aXh#Qo7aSFp?!#K%ojDXb2uA%n3yU<;jUp&U=!NE^Dw7n3 za0VtO1;xaz1qR#dz_DldqYS;l{qZ16S&tr{ci7C$<$%7x)roc|PMqk(ic{jeZ(7be ze2XYA_xh;o2O}%u1}`ACB_Lq9)eRZ|t@W$20T}SVK(Kcy>-+>I#qk;1CC=%kx-oF6 z0dPg!Fa_LiE0Vq;7NsTdeX!?Y8cx$-bkY~<_4V}#K}@l1pJi`u?Z}BdX!!-LwrBb? z%7w;nFfiPNBmL&h8)$CkHN(LW5L^Dv%hU59BLjouz>XJ2tW9c1#+Mie zIzj{AZ6=~t(ECxswnagoc1%vO4t{plOkj6w$*qdleaF>5uq)4>K*-S!%58uDox$>H z3%qPp3<2fb9@C$|=?gez3Dw~?7)NuBdxfBfH znY&%r$3p4M!S3;|baXR@%6`Mg!ww)`+6E`!&q{XwiIf>o9HP_gVrRr)Xe){zU zzg0=O^KJIMMu?{P;B3SlraN8Y^zMfC60n*_-@9+m=u!*0m1=4}dGhJry?eotk*QI9 z4osj|W*jyF`E+hZ4Gvp55+YF&638L4pX*NsGsVo!jkha}nD_EqmvNj)FK{NzJeHP5 zntj==E$Gn-x~f6^O$k0fTnq<XZKEpW&rrP8Way;oo>>#-&+CX#?8L zWO=gHtA5tLS!vekKv7gwWJj^jkmFL*aHV${g%H>~SEig9^B*5UiB7|9{YRh}*qLQ} zCf(K=BPzLG7dqTG&6gqc^kW&hI_NC&OleFe3baO4<{SA^Q2umyK2Vmt?Cd$DKk)NL zcecKp?qm1nzL@6Ng0_Z=Z<^{>mN7Zg(>9sX`rge;_S*0Usi4CJ25G{>!@X*9d(3on zZLmZ+!V;)#4&QaSf=;F%>x3Gc+nLm;tKP~A*!1p9y&c#g>X*ZPEx|7)Sbr`&&SGFxR;9UdyfbuG|E#;1m{?TJ?+;|4 zjeaVALutQxWcPdZQC>L(nWe7pnbW6F;wh`*87%R+Dn+mDP(YAF}k9=@#=2q!e%T8};2RN-B4GQ4n7T z9-A0OXA6U)(Pp(Ieik;FhU?Eq6(78MYif%qjhPrpS1c?i`n9^GXT+CmWT0#qKUKIg z>onPJzEp8?dIdI0Qg?Q}jbOL-q&d{k8#vJyM^UIoRyh>v`I?MnLElV&^(3DGrmrK3 znJqa(klJ!Pvz)E~JZIi(*DB}Be@&Z?7$g)0jEYBN1==*PY!vB(cHI+t?3>@lp81d!|(-V=XhqdW=Ty=O+|29>O|Fmd*uAP>byWN$Rv$Y8~26n zIS1U5aZVq>8G5Nd&C{3}a07SLYb>g)%%L>dNtXc>Co}Wo-@kuLI}eJ9Hgs}&h6e!3 zbjjshky9+cM;&no^eQ-N84i7ulg5?ASFP*O&7dQK1w!LNlDk~`3k&a85(8T0%vRPn zhjufbA2`xg^sdvi6Iw<7YSqD?@(M)Il2b0IVq<7~Bj~PUI)rAN>M@F0nYlFb>leS| zyNU7fl(e*l`5CH{h{YV@%kxe~OTrOf8FT=@Xc>AsIOXob7@M@Ee&4|w!2KCS;vksL zJirT?^{ghq)KtkW+n0o7j_*9}HFomU@cu(TETn6ZqAp&Ew1fzz))ycCb zX!x%}yS`0Vzor8CpBR56Ysm9zV4%i(paX8Pva-esmG1;N1B8~8kg!5)f~yCOY(~1q zt?qL?8mbyv+jAq(@AEu<6wbo{s0=N;` z%yuO8+c%NQrZc)a^-CAE>$kRC{fX~bk?!w?JVdysQ|6H8j^hIh7bW zmxPwWa0NGv4xQ`5gGv{3ZG6{{EWJwpUJ*9+d#tRo4uejK!xObPtEUV zD^onOb@P*#{EOHDi%f3ojB41@ZJDr#^$Yz=?)K6S@BEa1d0KUa3Eu9)|}6l z1NU_ni5{OK^A}hRYaW^!85t4SkYt!2s={aL&bNLD3!7A>|JJFhOCa)G6fn-W8WzC? z8TxLRxH=`7LNh)({4E}y9xRUW)~#CoKIkcjcfphjyyiCTxQT0OPHHZ=vdvyDcn}Gu zd)`Afrxhzc^U3GV@yTYGDlyfbovxCq_o6Ny71Uqu+ZZD>G~!^6H)vlQP z&vgIR&|feP>O#gQcwC)W7EaF2ObQR+Fyy%+r2(Z#N=YI168Z%`^Ix$i^o|jW)~0oL zH&nk8i=Nh>^Unw960>_>x6(9^T7`pc+BMln$UVt24G@e|?xCx3NkzXhn_(tY5{SM? z=Y-c<2e>L6*1sO&R`r6H7h$Y{llD+jdwooNAe`FIiHXctu2ih3*Z=q-=S@NDI_NEb z(vHnJ(`81Aqd0UU=%Ae-_PE*LY34Gvl)BDfVyy79`4sTBiH2C6% zaZhQ=&dyjb;I`!CLtLYS9=M@q-wjD$x-S8~snaqM9JQGvRaI3N1#he4MNwiy+1r>? zvKw(R39IEZk9(nqELR#diQKToFT=bv{$J1uY$aK@aq_vCdnVdXc)6Klxf?DkeDk+J z&i*8EEv=7WmIow3tBjjYwS7jsx{%P&3AWOJ0Sj)|FQ2)v45vy424mau;dH`Q*rv*4 zJN|ycqLKSUe*gyPv|$^EMtv(@9+?4$d|%-=av~_-gImcoSk|5ihf`}KMnaq6HK!ig zk((IAUpZz<{xrDsuh((-EmW+D%+B=)ryR>6!ODk!St=K7VLri}ET)T)bsZO+c2%VC5EyEedQ&v~oT9eiXNgA>`qy^EX}mA5E$iZ#~7^PIRl0({p_X z5GasZ^$9j1Sa8<_)%G?dT=L`HcLr<#-%kPey8qs27f@g)gG_u=rn%x5TbHNWL%_-D z8Yj5e>F7j@Yl?2_GLGm~;$U8uGyPR+FjQ^r?E@PDs+l_J(Bsi$vL!A8{HVSSaDEUL zK$gvXsC)ox4u4P1i=*FNtD!UBS*_q2qw%^ME; zJ;-4c632?$YQy?v56!45C_L$&OP7?24UBWuTOR!z13FZ3qD6CpeWUaOOcB#t)>-1N z_^$am9%-fffQ|DZfuU143(O*v(nk=My5Hlo>=n|u2cxyjB{+F?{ok}0DKXZXW{1(9 zV<5li8r8Q|?T~N0)!o-Xfh4$+8z`;Zl-u7U`%CueJSfsbqhWUIaMZ^XpN@BZx}?oL27hobT(qK?Z4g;%d2YCG|W6`I3n2x-3m_qo8pQ&zw)x5E&Z zfpVxFsBniSGlK3@smX!X<)1!Xt-Rp~Z&owdD?5Eq^wYz;>?uE6<(^+KOtu^s2@!_&8$Z+2x}RPFu(D9Lj+&9xExuC&-|>K#T(8 zLv`09=SneWQTp)gSC~#w<&-6(9XZ|CmrljBRvG{`Mr5uXcNN2kmaClp0+<9Q4%)Ro zI$zWa&Z)pg8~7fsbE?a$0wr-1M$5eGOTC;9|E`0^dxRNdD8go&R_|n_E=mAY7W1R- zOd5&*{FxK=*^m19ESa%Pne)m$Licl-Zr~}G)&&UlXXGmky0i4?LLQfrZ$G^bjydS| z8E}nFzDP2Qg|tLyil(}9R+Dr0pUT3#dh$Y3A4nsZ_@@Xh-IRyQG15gx4oe1QYJgbSfP2 zF8z81g@lH3>q-Dm)ipFIJ$v?L_*uyS-Fxq#q}wOqqa5ii8_a?t)qn>=A|g_tJ)+BP z2!javFOlzJV3DNVPFl*IWAjw64-5(U3?5ltooA@9sqv$N1X~DmIOlRY8GimSWYd5Fd&IGUvF?lrq}xGDJ2h@p-?D|tiPY@9h!z_ZtHoO!YXO2s){OI2v_}r z7SQTFq0OG=le@3^|NcsbW7goPIm2A8k|MmI901(7q4&=Mpd%eiGl8XS`5D;yd=xN? z)1pE2S#*W(Tt#D)D;NA<4D(CrkN4%F6PdLSn~;!hd!nqWii%23^?;Ud-8gr!(AdUb z5vHmr=jGue4HU<8q+}u{B_(+sZ^{W zy_zFxtIuH&#eFi50zTFCBA$Wc4V*UVDBeyhb>J|8M%j`qmynWLmUu4#-tF$r_JMOt z5W}74lI^6Q-!CAg`PirCyclYKqjqw2Wbx@hQ>G~LOZVkyd2Qvuq6`}$cwr(w0m!W! z*`BlHv~N>6LA1RpUu5W%X#`!DHeI%gHAGLx$Nhcb>R-||!?p^Hw>+1pXfp-NMzm1| znYirdc+ee}4{|!_dKG;CD16V^qsgLRQ#&B(&d%~MkWGGj+Vw zjznQ%$7<{96kvBWBU=|V>`SyMys;HEx0skr7i~=5{q8+rPMjM^pxuPa1%*MNz=gZlvq4Zx7X2=%j@BdlN*P> zENUsL=bMGsV0b|A;E7BZ^(m=2lGK|!Z_3xk+*_DHzZ(j@>s>je1VmSc)8;}PLxHEq z{||Tx5MI6J3TM@?w>)l0l!h7DVrZ0Wj|L~Jx6m4^sp;lc4pIO9B>lX2!JF`^moEea zdwPJHQUraV840`<49ZIf1YfWUZXt_yaYy1dwO@7_v{$??6!YfH4eDt9yP!jLb1v1~ zj7x|sWn|A=zOoEsqjdc9Vb~c5lX`RHmoNAH=!JA@Yio;PlwU|tkbYrw>n1Y$*X_o& zbza*@+c~?8C}Rb_Y_^VzjQ6=#O{%(X^73jx$NXhuUFhB%L@D1@WMh;gi>nyh@N?7M zSi1OVp%=E8it2pHxRF#z&FuT12lfzz))KRdValgpxqJJ2P)zw**@`zOgVQ)DGQmvr z@?9C5lQgQD()23k>N#g5;t+}u_3!~0B5GjoPMR~SjBdcS5|57ouWasaN2D!IWJ5w&Q8#-?32{JNPB{IaMW)#<9(|Mww6s`861$wCi zmxDq>@v*FhG+bs2E|KGs<9Df_rN$kT7puMaT*~=)Zf_L5LS{-TUP zJb07E!(KET0Gj+)d6>aKRuJ!+giB8kkmKsUjGXQ?q@WGP7lwxx`FNlEXu> z@^IWv&Qk)Fda@2*9(Sj7L8Nlk%k_?guxWTw;MrcUljTk3ZY$^+P+0If*YBnTGG8S6 zV77kOxXlm6yzL(>_vRuVhyYzjypYcVT$}O(Q%F^y>w|WGor&}HV)#5BI#_78DgVF!}~1R(nCL zQnRya*lH4d6b0xFN_Fns_{Q%swH?zYnHMqJvqpa9T%2GgMq0THew?bb^NT)*p1W5s z_B59!aP%))+H|ld{;QyMjx6VKFgdZHAh+VRY@f?a@M3jRpfm=F;U`E(UkN)Rnc zmA^&sgkkWbW-SO)@A7yeX;njuT-<>`Iqjep&d&Twi z;>4fHYNx5BLcoQ;7m_bJV{MFoh@xTC^rB-{HAJUY=D$}}-GU`Gj{@H|X2ZjGgNCM8 zXvFHiMn|PQtfa8wm0QR5A-3Xke0*UZ7<~UxPhcpRz#&U-3>9Li0T0&g*6M^w3^fE4 zf&2CZPFiU`bOhqQm3p7I*1wr79~th&3*VEm{q4}d3gB)7BS38m7?;kxr>Dow;r{(g z6qTOg0PuROFxZ)neo@Rt>RKR?07kkr9nSs0a>oT@DVacq!hdy!ore$e?d&&L(ztu>fcc3O@J z*-7404istW?h`Lb@OkT#XTEuaL~(gsW&_XBKOY>{wNA>P^OZV(7@NdV0u^P1k8}76>c@|lz>x-l?9#MU ze)!N+%=bbZ%!o5|H%9VLk5Vx(#-9;2*<*!6E?^pDVA z(3+VUf)QA8e-#jEWW+<)dR6%1~O*4ou#{S!!hf#|suHs@tygpwzh;qS>c z3DOe@WXO!-5UX{E83x2DN5mMtu@;-Z;m-RCo|hEgv&S0TxJG`Fsg>T^+S)7TJhrL4 z7-ox1K}1BuP>~8}yhM9|udke7^=lT|EfxPIV_}g2iBmC-1LM#Oe#IElR8onxG?q0%zRUy(1qdyO zN3x230yQ)Zt3sTj?l4@gSwPNb`NLh9{IF<@5L=Kob1ddv6xgk`gG!=ayQVT?^P!Z2 z#*TIeu!$w6&?r_7veQ26(q40NasrEfuQWd}7{W+-qKWRiJ9Osd47z$+mrjAED8SPSgXTTeJzy zW{^RVS-SXE+y(6RhdbJ)_Kt2bJ^=`;`8^e4a?bO|222po`uW?do|gJzN@2i-@&rON)2AE7c!$YOlL7Fuf_Lc0RS!bZtQkkqk)Ch}mF8Lepy=QrgFq zA-GA`*Fgu%jn(Tn%}-^P!PfPIMiIo|KQsv(G*)7CCG_U$;D#XXO_qKUg<|;u$(U1;va-a8 zSa};Qtlo>jprEo3-Si6m<4YVy|t`q>2H;s!~#+g5l1cjMmDH&ehdE zOBW0Q5)nG{(5l+bQ=gr8)+d04Nd!%X{|%-#7*ry3phXgAz;G>UaH4_1M5vmw>S6Q3 z98dr>*N=3T9)UoCRuoJrmj(k)HW1%Ah@;Z2j#9|52c+4{MFV7w;gdzW7q z>q*7##D@c8HUpB8o-+c;BQ5B}LB#!5y(HF&pS5?9u>W{*`%XZ7aPZlf)P_TR*dZTI z91+w$@Se5x_3bc_XTBKMYsO{XU)ZWoZ?`=tYPC5cLfmzbXS|hbS74_YGx$4Wed$aS zY|N=rU#3$>B3kA2+lqEV0FO^X<8VCiDo8{Z6d@88gn*c*rhBCP6Gh+|`9mS(N+;lq ze2zGocnef1a9XyY;HU+&K7@x)tvpYGC4v*`lDoMmoFt!yM7jji&eGFccH<)pxepBA3Ty_3 z*Bc2n5x6b-)h6sAY+zSyQqY&me(zoiJ4CYJ&rGR`eMwpImoIIJY6SZd{C2llKBNqE zOhf3J7ynpFDvhWHe2-mqWLTJ5R78Ynp$b-J|`q_t(DD7W9Ds-;P`|)GJ*-b3wFI~aNyr+hST`n=p zM9%ykvrb@GGd5L`_|pue5*%TeE6z~8ytc^qIj(g34#YN-mUb*`X{U=-H%iL6#c)B> zd_vewhzfNl32<|B|2^*%Ha{f3lzN1mW^BXBq24}6RD{emd*X2ceg*!Xl)S(Z)VqsI z-G2EM{e##pjPd8%7*rX`c9`q$`RN8@tj6=)69miM&aaFmW|Wh{dpor5<(4#<3G>OB z-uLvn8=>nuuwO3*N{3iuhb~HcNurFi;23Xmfs8@gBJTy!uoJk`XCUta3D4trpPHVIX_941MN1TK zWIWJ*V=>rqQgQ6D(|iKtn@_i|3qWMc&AkcTQJS%1+tLZ~z$J-iU_Fnnf2YC@F)B=+ z98tk%d2zzyPS|{VA|mnvrqZh!UH!!vs`X^mq*9i|_?vs91)q*{sI#2QsTA>uZ}>PK zp;NNMqOA0hYmfAQ1BCi!r}ul{y}+3&+%VoC&JX?6_owDWB7JTv^AMJUNW}~wV8m&} zuR!bz3t1ult6s}94Pk7RSbF9?*FD9i1k5v%mgy#R8GvvS;zn!azV<|d3$b)t{yY!K z!L{%4R}Z=ijPf!}ZE@V%w+1eZ`O9)or_{PHM$OT@yK;13t-%CA;wTF8pEHIX$0i<2 zNVLG9?9Tz$#JQ&HpMZfpOimrIoT_313FLI0Pnv`V!B4Y6GTG)0A|v8c!6E2!(t#8r z=g|OhP}1}E+NL-zASY9ZVMu-}#AN_sQeIT)2kt4&K?q4*1!b9M+R0S5voWK?-1E^L zgFt5oyLhkb*h7Jk1OkyE%?Xs7n_JnurtTwgaWuG#NCukV?v@Rt+lT>w_ItxEH2}^a zj4r4h!3<~qNmJFxG4z8;e?M39q~JvWwHt?R$pvZULZU1}yRVc(dJHv}nOf9*V#LD& zpc+i!P3+t#AlQu13qa1bCN}4{8pU5BcIcEjJIzkOAW)*yCRlug3_>oC9Lp~C04ZLg zQw*2626j-Flu+0Q;RBjdA>E4@!vj&Cg}}&FuEMfi6bvz0-soJ3UUtA(`VOMYb$no6 zg$aO7gcHuwt2~UFU}AuDTL}M{w8TCr-de23q`b9o-U05O-BCr@><$9(V|3=oSc9lO7^g`~*T)&(<{&NPFH|GA>j;T`onVV| z-L{>#A$lBao3UsCstE}x=frGF0b{$e_Rhps%N^WE6JX!LZ*7P9k;^bzqL^tX+I+?U~(Q!g|V4jd`Bjif!dfA+Jox<>Y*)}tl} z++YWe3vY-CBLRifb&~ez68}>FiHs?rS5XqCai?$i+AuN#d4-02 z<+VjpQxbTbqb9?KP>&lo3+Y=6NG{wP;IQc2*6VeFOiys-6R81I-9H}r=@Wf~47@pz z&CQ&dnq&OE|>tP{UyLy%YW5Cc|$j9L%u2yovP=HS7TQ&-X%GYzIN zO|X!ji_z_-aR#JTCgvJ$kKlrG@+TP!S%>E=*zU>HN~I%>r$kix)t za?%K#3K8~%fsqKo|8*;n+mVrrL1Q6~HkkIo--|?Kfhu8$dTe$PQn|!SS~t&C7{Pgx zM4?*2j)nk2L?D1xVM`*CJ~&(q;SJ8s+@DzUJ|U~rz+f(id~h}2{D)(M+DJEnZ$~*H zEFgRhQh%8tnGb_2oJd>q11qoNG%k*21hVZtXYwvmE-B@0*`Wj{zbU*3X^DoTg+ zWf(p`4Hy{`Mh+aI;MnZ=dBD3p*_W|iduer8bRbZ<m_6z{LWZmQ(8kpm-p3 z9%z@?E32u60UriY^7F{nVtGF>m=BhQA6LKf4RX63G`YbZvP*{ZG;&6jnR_!a$wZPI zM$)a_u7M!!2d7B}X3*NA>zkU8TtFyvFn$G|;#fe~7f2KV)d=st#o@T{Hvy1x)|0?Y z@Y#S%^ep||bsa?R89*NE)DO8Ed3Gd1hjeL=AqmO-9%P6qMZ&hduv6f2>BFTTA-*?Y z#O`!pBNf3&aq)3*v_R^G-OsQ4zyfcEd<@=$cL*{tjM?w11RPSc;cdG{*{DFNka(3J zrrE}`v{(`?TsLeRLCXK%d!g(ITBr))fH^nq5NiF&^|22EZU|Il3W=>H^yG-cVX-g= z&ja!4;l2vM)c~+a1UqU9lo*n90C*`DX6MAv&AhwFE{3H3ySkS4=vB(*G5=zaxnN{3 zaTf(?AsX5D+BtDRVv)4WzvUeZ;3|`~9E802C-z5;KOd&Jlr%kUno}u!26#vaVqZqR zW08a0)&hp$WW_+FP{Cg9E()TX)&6DLXgrP)1sUldNh#dO8wpG3ts$M*FCV=PLes~u z9B4)K_sPy&Gv(rd#v(tSZU)V>=(U&!=JqV3u(_N2graL(G5Rkg+=+{cT~ z|EEI8Vgg7lpu=Xjm{3vr`gRipEd1+(Cy=>>U;clN$eJhVfxbTcf1%cUi3=^V80VUj zaWv)9UgrUL>a43ESWxRCPhdj*0mR`b2U#m=Bg?+Hw{P43lM~@GkQ4Z*RA7k%y>g&j z=5VD~l8sp!`aez^c|{NBKMy<_KA~eJO@z2sz-lD&$p0%03M|N7k*`zBe=hs}-xw;S z@ECe~Uj6`NSKrnV9{Py?&Vk*Z$9({g3#B_isk~6z+mZp!m~4ZhOEQV1&;#kxKdE4c z;T_rgK5o+~2mUN5xzRsG%5^Q@*4<-cNT1W$dt#1j|D%PH0}sa77T5)0*5vS8>z(1(wgp&Tnx?nl-M`MMnTp0s-$XRu5tQr*Id zD-57qR;Fm@BXmRZs{a#>c zzXkF6l%)IS_G+Xn!?!}jRzV*86Dq32|1&x6oJjYirdVO4`}?K1|Fq%HYEygUcZDCT zLj64{{mS#yc`B-8I4#<>78f6XK(A$8`J!>#(ECpd<;xPIa|4N6Tl#hxXY}$+g>jD= zP(yCSC@v%1gxvs`=T3-1_aweLC9<~^o^V3s7>`N+>I&d48H7CZs@{TK<~48^%zGoUANBB8Sq(Q;VV6?Qm!~dDid17M*Av=#mBtqrKG3 zPkiz3n;g9gl2o*Eh06;z8;&9SX7H&edqNrUqjhWlQ_H^L2-$ERzu~-7azZ#8U-ZV$ zIR3G@0Z=fchk(cU0qN9NFK>n;vLJit%4^aojD6FQ5y;`%eG0jE5abv3U6_ODZpFWw zelP%H?UIpvCGwkLyo}*YDC1Lm8+P4lK@KFg<+h`_xl=4VN z1w!AV^$=X`|CkPs%crTRrc~_8?jRdWT3oT$*KXbNyaW&vew&LA$)9jUs_Hu0eYBQT zwXHA0N}5#<3ifhpD`}B^r$!F_kkX~d50&Nr0@0>UNpg>;Mdg4 zX148DMd8$7)f*|u>}>8wX@snEr@Pw(0aTzV!+kWTp(!Q&9LCa=nxo$ou(5e-l*V1K zNvcAtG{l{=yjgw8^f%Y_aXBCEv-%Wyj*V}7(j0}>XO|9JHgi%8hP~t*+)K4cfpkR% z)Z@_1j@X$Dis`VjW5|bjI)0H?g^98Z%K5^N=2FNd_4Vgqy&-|97eWit`R|%M@#FxTRktQDUj8zq=59S1=Zx)A77Z0mdu{Lx*~fWq z{{&OlHjGXvNXz`R=0WP0lBfr1g`@Fvn~zdd(i%?wIB$aN6-(b}{F{s^DCZ0Y{fwL$ zjpT^(w7i+2wuX+L-jCbSu)~M2^i+_omm0Af>AKi6fs|xjm&FJ}mTe=&2{}ME$WGd= zV=0CHwtsDWMMCnp0{k1N)ARuZMW+jJU9S#yyvh8!_h!Z5a$&W!zYC4&hz)-4N6nco zPz(R2<_bDAdZ7E43kx{Q{~e=2GS%6r^SSpx6k4TP6 zZ$CY60VQ1>R^#;kcWz2xTN~2O`M}EIuie76tYsIJ^g?NAtj6gVBEWf({FMSqdP$Mf z``^aLUq%vU&D~}R0NN2dprfSp5lQ08SpOIO7+1mN6bxl-8Z*_xQ9RCM9wuVFmd_%G zcAH^YjS|V>cpR|%tQ**NrRIhk2+~P_IY8!9r>bO-42lp#-aHC0dK(sCKU`{1M!kq`% z1EU!W5gL-Mns5G-rWOpZ*AoX49!Gb6=LhGuThow0%8h;GkA#>IpC`t3wFq6jD*Dv; zQcXb>uUn@`PUTVcZRPZxz(smY|E!$I2SPVJlnURJzjf3R{^vTI9Yc=8wxXbE`2+p-G1)0LF>awamas+^YW(oOX~LVphtR!-p|kd zu_XJz6y5|pfOoBHFj0zoAos>^YLrW>i8-@;uV@E3s~#$Q=r4BP|HW~Bt^B_aG4gRp z6FNpRH)KNcjK|_WfyD0N62Qy?$BD2O=$isdHRLM|4X(a57Kg+r2josSms{b z$(Z`|>z6Yg!NFSX09cs;Fy+YZOHzeb&zJ(uqX4MB(i^s+tvR&jY< z(V3af{;#zWJ|Rv*6eK^hrfs}r9QI%CNuC;qo#Yp^+je9mxYuy7kw<~AmKS>#aqVxw zc>P`unxuN>gKLZ+lz3Zt>+F2<-M!p$(#Ki;7(M|2pZ7l8c4+MLrc?%{7gHeg$3w?Z z8;?P1=2LpX4$j@W)Xe{|XY3)wu_Gf(5(hwwc=he}6p4z>hFj`Ei2||ksG#g~dlFx$ z%xw|x)IKBK4>mes4=ifr{rDkKu-^Md6fhf&5~B&;n;t#;-6a?FSYQ)*=t|8ipGHCL z+-CqiJ3xt|-JnkPe^yRTB?k!_k~N$KvtV96I*NMqX#0H<;vZTl2hzDMoVT0Hga_Cu zw91C$Ei3dqSf7|7p6c25L(4#Y7#mKTq1tZ5U@}0(xsi(>CG)_UqYB&hykRCu1Y-N9fHG;q#gUO{?@lttz%6@QfV=g9;Xj4 zD?)y<>X(7GF;+wo0=GmKd+0H^FeKQ&hbWB=_qJXq4bhLTfI+=x05NyW?<@#yjOH{C z^lpOkdm+?-Jx=~Y1`(fw#b9ffBh`~FsJ_9&i0q?ufX*{=mAkrRxm+Xy2<#a4S6~pJ zFVh?5_dm*{YwzNy+3(a|yD}9Je9?KT*#{*dB1eq?FjvAse$p$#H)@|f$Lp(2^DZg8 z(1n$3gX1wk^idLs%&-kTQ6;Wlf267LoJ^dISiE%_&_g>)8atV3$HmU19SmySlr=ieye6u{2{0;1(U8AZqQo)!@yJS_R6Q1+fj_{0-Vq$&;^ zkl;{NW92StyeljxUQQMKld0hz|J0niC$a~ug#hk*a z0NwJKmMk?O-N)aX-lJ<2SvfESQ{#Hvi(h3utlalC4e#tH8{b2{_nS3ohU`c+AW9`S zr9W~sX7RRxsa(vZ648KTsBKsE6cGkS6oUFySllZ}Egm#p!7e*bwieMZ4I#EnuMk3}g@~qqAzi;$^fU1*D8`<3l zprRsanNFPolU))R2x-9Wu4~4@#2P2WMYerrhVcW4sLYVZPV1uM-$wc&Y!0`KSPvd8FfJ& zu>P}-6<7La5n1v08xK07%djjSS~Uc`_dI9PHgHC(y{FGU2~k2wmPxKL5FcX^(ayx? zke$@Crw_Leg9hX6%&h+F!ic_&{CY{pVeLIz=*^?=r z`!d$S4>j~Y`wW)58vb}V{gGa7g9UJvO>B(Nt?aT)$jOM?*LWQ?f5G}cN6h2gt%*v1 zt#)p0cfX!tvImR_ibrxpkD9=<%_EoMNdqQX_NBMKz4U}xpD{@fvgp)MOz<+!R(emj zdeh5!ueO4l&QSth4g%%7GTa{r1_OWLMM)#_`7W6-7Tn83{80C06m>T)6Bwt)+4ml? z)_M++Jy*z#)!RPzl1|g+TsQ=TK_Ba(9<1PgoT{w=~8>|v*cNwuuv2Ne_HW!qMP{f^ppEM-vK%R=&gWi3#^h={9PlCoT51=L`{K~3oM5HlcVN$%|-JX z^@)-2TT&v!oqnqAqza5R!jmIuV$8GCZPWQ zGkTtvnGp5rn7z1L2AQf*3H`O6wRKciN~zh$&CK4IZ2G4y?@FfSQXVT2y}bf`Q3(E@ z8}SjU9~7_l9br$9Zk;W{1u+0A5a&(5^$Iev_PUxNCrId3o{yYksIj_RoAPkf)lU0S zSw_^b_NPw!$ttftrL;|vKqb3RS|KrX0ZS#Pm1(zin%v!4c~Tn$89nzr|I+T&!6O+{ z$99LzsUsazZv!CJ+n3~B^+XT(hhG4N$=QbK?evQ&=y*j#4m$_ zwy?i{MSq#)TK~AvkM)*+A?nAPwrgUv(U(~IT5ohbt=!h})<;XWwrAoG@hO}wFnagN zh+I*vkHTL%Rkse8!3(mU$dP%t#5zy3S^T*0|N#$*khCeq;?}F zK26Dw1dxFBq_UQl&5IG}&tzaQdXU^SekpKd_g%fynkPf@;)ta{*9r7cm-70}Uv+fH z=i8bv_qkbb5AY^cevItYYidkUBfgRmD0=PMxsqMw107RU#{|ji?G0=8ML8Man&#?)MrK56)yI~sFKwP+ zKqcbGBm-rd9HsRLR}E-ew9UPiD&o*?B z;&imXxF=`bAv92&Z}cF+Lo(s0jG~R>q}UR!%n5yM^$#`rHF8qD5)s}nwlhDo)69B? zluyOb+15(rbu!$b(pmRyn!W!soBtC|dSJC#8E)K2K_yKRBT$f0;J40&+lG?hJt>a@ z>9_@7v13`Dsg*%kLj=d zNAJ_IJ>Q2c2~XmD|M9&^*JF9fsFj^5qXL6B1kNdgI;+Xt4gwKuc=x3O%Bz(h46UbZ z-5b(Zhm8sf>-ghFktL}DMGt`Az7e-oH}t=*)=~3a2+6|w`9ET0@0uL-#fzly26A`a zsI&XF^QrCwH0!?j2F0=mz{tpAe9JJo=;EY8HG0fe_`f?5m-@~#{*3<4r}mj=jqmPe zeN<+`m|-Q)o2{rle4T_?bo=)lbuy1;U}scGX@X$v{|$8t zOT4)h8cShgv1-gpy<(&sbCa8|)KGD~wZMxisMxGf7nDK+1?4^x0Dw9gR+2}K#0zK} zE!Sl&_D%%Z^HXq0pL~#6HWp$VXMIL@iw&;gU_XP0v)YLjux~tDDj5n*Pl4J_`5wLt+xZSRQ2BOM!D4X^Uk8s(j)OL*53&2u;T zH}?ClFyXf4j0APYBP@59s8;B52%YUztxLtu{!`|Sj) zzEufBk+MHXeJq7+e@a^o7pxBT^W1yL19|9zVE@B6?UA!cPZ^c3@ElEZ7;PhB`P_!6 zn|Wu*(+oT0FW?8S8RaPcU-{9jHEFy~-xKxs9~A&t#^G(Rn5o-+0PFL>t+;PIDD-fx?oU zC^NS=uBcG|9C#7UAHgTUt)i#h8%|40y~o**c?^Ni&xtS$J7cOl*S*^zfQY;n0%=Tl zyN~+k|9ZpBXcU4_gwtonm?X8vx(uhf9?-^If(sk}BW~pPR#)kH4*=!7(pnSqFLiLr z_G-pdG8pmVJj`z2Ziv47bGT4cjwVi}s+hIE_fxet6FhE?sOSMTgaiWzJeYx`9L^$- zuz)~YK%-0`G^n<=F=o!m;#7nJIc0E|Vlp+iD2EZnrKw6nTo6FQTUQV)mYQBDgk)}Y z&t|>oBI2Ay%?}eleAYaos@af&;$@q4$`O4TlN5aRJG^!y1xW^{Ie1Jy{cpI$<8nJ# zplbwf6YwobksK09(2~~MzjX$1Nl{)%LRn*$xn0toOnwKyrZcTWdq;+P>(4Qct@!uI zP%*Rbah32sN2#NtX=b=|*PoVyCNf`ktL%OB18~6yI=168ZB4W7N@}6@`}`)BBVERB z-?T&fs0Z9sm#{g;cye#Rr*ZIJnOZY>{Zg$_%nWGy;=bZNXeXkj++(e*`k~X5Z`tPrMbG}V9jyn9htz<(*Et_lTj?8 zWAzSY7MeLj)7>YOsoeAWzGQv6c6@jI%*ghBW6q70?PT7X>J-qRfy>iq%gKOX-97U! zqBd4QpR$};&{6bmWWj~EXtK?jXvc(pQZ^&8XrfCsFYx&-G1|1HJnj5IJVj7f;C-?% z-^~*q;~j3RRt4*ItV;(z$my4T-;Vo=sJZ-tW#w9d@RU25X3=MIJyY9H2&s15C%PyS zhNr1h7O|Gfuu*+Gz0GtPzm73SPHeRpcjN(jG``YX9+8rtnNS?)*<>`Z@&)Z=u9qm& zQDokDZzdyD;KM8Tr8jtZ6@h^t4t%#YqZmQyQvoxsQQ zxbSKXqjz`VZ=UPVN0h_;#=9J3E z%HA}bdgveb{}J{TU{P(~+hZqSfdT>sinK@xNEoPqfV8xVGz=mPT`EdQDBU2<458G} zsFXC4(gM;9-8J7jgWli$*YnMD?{mEfoPEyOd&Rrnwcc^YYSo8W1?MjMPdds@>ycHq zG@9yrnT%FRCyJ&X@>_rd(jJ8}OCS&l-~v+0?<%I6;1C=e>|#p$aA`V8&+W%!T{B_E zs)3a{@;xrQ0GedvH|;r?u%-|5WjA6_jbOP%J|h$Jxqv27E5Ve$2ghx z#MFTU!$0j?M+bVp!66ciDeO8r9>>?QG@rcrEZ$~N68sUxm~D{oN|c}oCcHa`UHwd6 zITH`vAOE=P0~GFAnNwhsb>}+AkgESkCV0N##EOFIQ#;7aUcb3#%>{6v7GOC|0WwGJ zwcwn3M%d(cN$1~9>e#&at14MUEZ7gDScVdbVOG)RQ0a}Z-)9VHcHY+c%Tr@c;==sR zQ&(w*r`)xxpE#FKT|M@AhrG2@g@c9~nm^Gi${;7=o}^;#b9;Z~B?0i_N=~899OK_9 zJ()=j0u8ALy5M{l?fT{Wza>+vHlHcB#VX3u9m7TLnf( z9`j9e_5om{mOC?AlmH2pwXp?NfTUl4o6p1$+ZCm@xof_R$?$i&%erF6c7U;i=i%gB znNs}N`DE{i28&D>-R>|k0K|~|9kOml3Fy@{0+3^emGkPp(wcc-S~%M3Oyc#z6-;FB zMnT8$kD;IsJSd*sE2pP9T~yhGpPuR~j=Qkm%9`w@^4c68+hd{V^SVmOF}6s;R~zCJ z`ZhnTPw;s8r%_!(x3ua03f94iD6({S%YWrA;Hd=JBY#=3ffHpB-AY$Y&l+aZq%`mJ z)!oW~bjF7$lwQ+FyfoTT_4|f=^KY&%qNMldnhd6_lE5^7Bj`U{RFgnmwx@b@=Tk@6ffQvmz%Vv6%U z+(!0OJd$?7-W?4hGTCD9+L#1jJy|ste75#sMKF~Ye=wzLy54)CVA$2|msJxm=}>}f zn)%?2NFnXL`$*^I)29ocWIsoaqpv`1*NCwDoM+nO_I zVg9q8HkOLBHT|}v(lzW>~8NX$rbWO(_@t1z@&)WOa(BqfqjQqT4)Rd4@f3G`~ z?m*b@h|S`L<6EW$kMDUxK(3jv5XL8WxP(L&Ik(qV>i*%&rrOdBuLs%!9Z9??5xW}gtviT`%^}8kr4U5_Hf95x&=tAi!gJD51F8b zIv0-=Nu_X^JR;*FC;VRBfmDL`K|sppHErG6?Q}3nitv1qrxvI^VxQUpFpv@duu-XR zH+mumK}H}Ou+vN$4D{_C={`D5H@o0_wx!>6dr{qfQbTlo;NwK*fD;0dJew01Pe`BC zD0qzXJfVUF(2(8m7BvK+>00Mxt@m$k4ls~lE!V4~75rmUjl`;{t3BDaTOFBae;Qhy zK|RphDK|i6KLEx}g{A;!gEX}S%FFN(yZ;j^`B1!4fzZG6<1bObWSAi+Mw=obUt%#q z?nAPk$P0~E;JPf;+vifcEv;0dAoFt&L@%a3nE(eOAs zEdNdnid;uoNN%hym$*=X4i9%8^b|s(ZSJuPl-|xM(>0P>=~;|r12YhltLYxWv%*CL zr?>zFSM4BI5qJ> z6gBG=;q)6q&a~Ma9w#gF{rngW!4~i|JNf@5lI_g_Yl#%8=xzk)TV1&>3t&=3dMFIQ zVY`xM?fwa}PNh#nE*v(T73@zgLskVCgJtod+s8W=EiWy%&IX12UM*w-fXs*iU{dsVVPnPnN{ijKg&KPLvXYOsJ9u06In&%t35d-dQU~T|ot>E_uxF!U z`OND58-k=vyCh`Xa4dd!)0c0@(36Qs9Rr~l{5M?gf#Uxe7(DYB*R=`}`#!imNVgLF z>ml$@Ae2Fy7K&^>0-@~t;2HGX-0U7XpBUwH)>}|z!iu<|4OJ7Jy!r6`0 zE2dxFMT(OQa%yU8FtM|MgGy zEX%gho5&k#Gr3~75mM03&=w%Kwf45BnF~RBpVkM2D$N#wZa@OaHyCOdq*4skG~IPN}!#T|-at&*!S$Dhl=FjZA%m8R&k9ycwY@XenKOM!=f=bZOBe zos+ikc1>1eiG;Is6O3|q8Kv>=PtuinyDs?a8=X?5iKwO*e5%HX4772Aht z=!@dmS3OC#z<6{l2^j+uY4TsHd$5@kZJ|AlCi1=O$jNpNpcF+A5lD*TWfe_(OA^cRo(E9(ohWqS#i!e_v3A zvU_yAl64p)x>{j}a!b2cfoNQq6H9ldEku6=(Uz1S5+v?wBR`Ppr zD}mH^_2!W1JVn{>oY|R@>nk%aBa!^|iSDRhfYDn=oN+e|V#KB0DCy){U){u{?`b^) zK>Q!q#ApcfGc5nsHe3#|aTT`;q?J#;(68o)lvO^f%;W@wtSN*qhrIibqi*+T=assK zP3NN37VG@GmIkWqCdS3nolZJm^)g)`dt%Ku`aZUD>p6wTnV*`03~T(Z>g9pLJuCT> zpBSsxW@eOZhPg3lT7kJ2n2;gW@e)9Z0h1<#QMA5)zMagv{b|{>4X}BwlObTZ$EVse zdef`dCUfc56e6{PCsl4Bw#9VYq~W7p>E-Kgk`l;UIU?7aQpjkV=JafBU06aMVm^7; zPPzi06q@s&C4`xl9ocT2Y7skv@B@~8AT$yV;SmET^%p}N0dXkJ!6zX{nfWLtP#4MqEqtA&l|^fS_amIA6Zsz#RJdE>a+%@0i1UL_q!ak+fTWOHAhUrbx(}M=)#yPWTZf{i4ePI4?YFhGMlBFWM9d{4Q+-XD+(W(WuvB z(uEy*S4y|ERoX7|h?3HZSAi`bAr5#e`th6bri@$X$DBuPy&Krg742~(^L_C+iFM%0 zW+5k|L_;4m<;{DfZ@N7b4<-!lrL^&Du!Z+- zo``TV&WKIYo(V(J0b{=aC}`v}DG(TFQ<`nf?9+Un#X}`ab5_d}Dpq+p)&Sk`jOz6- zbFER9`gnZVVU)4Azb5v-wgd9_dNz7OI7PC!%3dj2Xu9BbBeC?DRGwi=Lxkx|y9fU` zVJNxF`9M|^LzyC?o|vU=I$uzP9edw-o-ec}qd&zWKc|W}vwZQI(1UZ&%$?_aHlz#? z2?nzlen72OYZNBxLromY!f9a(_dO10INasJUA8@1tsD)N#0c74!X4W^}?%06>Z)L-*KDYcNzap#Wf^f8&>{|Q1VG$Z>5%yZ}YfTb|=5j5z=bqv!4`Sx=;U3KSj^JBk ztLw_~9P5B~BDDmhO$QV8zYr0`nkHL>v^ckHKTArw7n>rVB9b>YNJ`Q9GJ7-a?;t4am%PLgU6}a zCSx-9o!<4Vpu~xvT&nhSt}&`Wk%@&~K_x{EHUL1|@AmQwRR05eylQ(ouq)%-zzz8q z>ARcZo&|<@-(Unff_q}lchNvWpz`8*?1W9I;0=K;)8w4D1^~xf9J+6T4zlEsNP$GA zTM%|5*b^&`Ls$_PDunuyF^MA(y z)7H;3+Vxz^p}YQd>Z%JBaOzf0pF6{&e@)(d+1Gp4;2q#jcbwI~_ztHv%P-#MpMX+n z7$HL9?qRY&tVL9P^?LL`SFH1zRMu_7rMvxueX8>}8{Gb;e5kzvIcEHTIgencP8xvr z(WV@nDwcLfb98b(=K^g}G}Mx6y721$5piq-fJmo~FZ*7;O$ymiX_CP!{VHv9Ly8iI zcSDX5CU{8|&_hW-h|4-=pWJ?y^9X8ZaA)u_2tZ2N=o(%q>YbB#K`$~uW5@QGZD68k z{Ip1#-M@f95{W@+rX)?&Lm~X;oDo>S3`)O-)8M&%v^SDmRJe>RS}G850YRsse9fZ0 z_5l(#im9AC{xKv*uZuF%<}2uo*+n#;{2or~wnKV^{<AVwujLd?=l`af% zP=~Mm+~!6gdXIpmgI&!hB(4iwBqU55U2S z&`+T>$HmDc74#)B*3!lyzMCtRC{9EtlNKWONl=}mHh3nCPCV`r1Pq^gJBj-8;Z}NX z2hJV*(?LG$_xiC=uvfJ!Pww^l>KWx!TxEpNX%J8kj`uEs)P#Fp7uLIbx@xo!*OmO( zX{4+N4(k-a8(BzYlG+eXpY(`hx)<i4?Zq|*jZdkB7ja~;2tKoN zmrk6Yw4TsF6U@CK(yt>wc~G@SyJ%>8cSzh+{z@yrP#Ozh$;ce`#SyP(4Y^28S1V#{ zb+aO)5f{X1uNJAbn%{wy_I@&cd`Qk4$xZZxM0x1{B(CWN+brx(wx5k@Tx_AyDz?+Y znMtAsnyn~ASC3RwT3$oiI#A`BO%+JvkO}bkUWxtxJc-z+6sSL_%lk3rIGfxN!(NKq z95x1fa#;c^XsZTTf+=+B<1AlP8YAWybH8*+!-OJ(;(L0xDBrg@0tCGE0w^Wnh5S@V>*kAI0JTc7Rz zI{}O)wV+5I&>Jk!ku*(TVuN6hpR3y5ohAEY7{QpFEHZwl9{#=!73Fa`#}v!2?Rsf@ z-&7vqCvk{T2X*zWXO!p0ez4GpXf&8n*A{*JKN%--HNjJh29v8vLiOZ0!@XC|OZW+L zp;HGN)mm#a$ZVNgunyhiM4dVWw~C-)HB zBYu=7el}fGxC`e)r|`p@e*zK(|ak0sfz4AB>GjBK*2(nDiOq^w3BKxlc+)}$M3N*(Q{8nn5COxT7l`#K|~ zLc^9;tKav_uoC+H&f0H+HDW{BaBWuQ2eZn^j1absUSdNKi3CdBGA##1HInTVLn%hH zw{&Lk*=lMjS$>~77XDzkW2QB~r)!9YRQSAnEHkgvrfO^7L3$vIVW0k%rG$J*B^=-< zB%BAY$D|aj5=Qql3rT69%(p_y{(uivPfp=YVnDcjU|Hav!F5~z2rWEK%VE4>gzX$y zD_7raBNW0jYKk|)UJexANUfn_n|-9^7m}Pj-qJR(QmPM(#Q#$MUmi(Af_D`Syn4IeCDCY2`$9u)^iaDNF3T4_`71al{RDj%yoF!a6QpVPjQJ(9? zsYzpE;CY?`uYJ#c{8bu~*-gzJLoGT1PTeYx+01Jw+R8oE*4HD}mF*8@tjgM3jLg-> zn_{fT^m4FUQ=LXacG;4ZzACVB(6y{E+dLE`Opt(MbssRO&`qD!*kCMzG&Hvck!ly^ zhlWUXB?Ba6{*5rK55jwM8X$5$H-}8?KdDhzH+k_RgoGD6$C@RoH zKK8@Xkq2(CIOCpnUdy_@t7-Qe)wGGl;L_y!?;qBkKox2D=%^mszgVfwwHcez)_o$9 zf9w5C2ZKViCL)vvG5J})X(Kf(Q#zmUd5oFYwp#syW;c`?+cam7w#!2W{}9E(r+$BS znI}_v0Kyh}vR{$JK8YOrG0k?xROxm{4iyQOPt$@(4^GqvS4NaHhtZ~u=4_{a=JMp| z=Tul~;Dd-{>`+xbId}5^PRH9dl(uuNPZz~W)H(N;PY$UhC7e8Z(hm@in4yks57+pe z&bB73(sXyyN&dd%teH|Xt1HlwCe;`!<;m?>dq=07zwlSoUPpm@7kOFRAqdu!qgnmvz*!gbAR_e)5O1qdt zX-^D?(W|8jWV}qn@h)%OX#C3)i#?WtrvN0#=i-qEEURpJLZ{yq+I)u;%l(-*&iq8I^zJk?z@k3JPNlh>qVKH$fMX6ibt zj2Gck-nYJ*3ywJl?U58^CrLXU=u+ADlcJoURz_u$?Pg8DHp~=Vti`HG%Ss*=vQ-3N zE0pH8j_teBof6j*D;t;bSU6na#_4iD*zw7bN8xoZd3bn6ao;28H}VNRXh8@806Jr! zRLnTO#LtHmFV4pUrJmHhxJK$RxsST#1-u}*qnI~?+q=(=T_-^|Wq0Tinv$xZTp%+) z7C;UX%dkr@Cx4A;VzK9uY60e|hSoJ5X@g6TKHfZ20y#o93phN_DsS!w_l+!uyeHHe46Q;?YR8dx5C4yRoxT<<*u%LzbF9_%I`S6Fe^DGck0}i3YraqE8HMKE=dp39nvtFxx%IWJqJvXDUKa;^(SU7?hi`Tno?HGqo zlcIl@%g3nDN^TOyO54*qSPs$YTx8E+IfPVndDseY7dU7JKRYUQ9>PyBwZB7OI~!GP zI~DzcexsSUU++sJ*^mG+Bq)7Lcic{*xQeGsg~Y-vSlW)JKe!(#kFI_-z9nsQ#>99Z z=K>_(UwTBpLkCxc#NuIR?8@*@v%%XALC?iCY(GMbrU{9!mI~_*jr(z4uCi-s!7gB1 zewpSp)F{O-wm~7RO7)lrG2B$=cOFwuZ^C5M6J-DC9f^a&TW0nvQ07a5n z6~Hctp@vA0WU|~Do1fEO&iHSgk$*D;z>=d3c0U9bRpnRKQrxO!~#Uzk8d! z)u>{VIz(a!Y|XkAj&+}|KG9OEQ_O9zrO;V0o^BiG*e0U)n~VYnsQ(guOLqw>#21)p z59KwrQvA8K6d=E%%)0R7)BHo}7f87WYcs5)RP^~CSnTjiBF6n_BOz=w5RWP+ubTQH zL={IOj{lX@cY(d6L?NaE*Mpb6Gz1Dpj1}mfWWEky1G`tUDMJDI2@rfvOw8@kGvGH3 zOWkt-YqMy8U&iWGSq>s5!z%5<`VzO$1BIg`We{j`8ZW7q_)qGW0?YK6` zW{!4vK%QT&fJf|E{)vDcrvEn=rORO0&QH2b8*#_`UZ`5k$YOy^6Zb56`xFvGjZ;$_Q~}z?SwIz4g!~u4fzKCa&cuub&`wHIA*6DMRLuRWNNp2#cHtz-ajomQn@CP- z@S(w^GuE&Cj_)!bIG+$+<5PO3)I_qwLP}RAPn(6&?RU9+0uFG&IewK(Cz&STTo7q(`b(}irX)oxcfQ(3Phe+0KR!i}O|Dm8s3VV) z34A@nE40_ zfO^16Vo}f2gea(0?_m)Dd-!VXMTEw$&*z)~sla=8SHZc{VD*|}ZKUpdhKXp>IT1Ig zKD-A`*4Gu5sq!aK?s<{ZruYD1Zt~HvrO=IRtfwTpgT>#A^Bm^yIEEE|hLE4isGSKi$8p(9VV8G}^0CaQ&u?=Ky zCdXY!pJuKV*L(&{vQ|CNV8k(lN$0Fm*@pm~0`s23o9iLK>Z=nGcRjXk2fs`$pr(}A z8<&=#d_1LS>kxh|?Dxl-a3|)`VV&NfoNBwKoK2JF1$6D7Bna-!WOR!AyN-ITXgdyI zmA4z>#{@AK9s^DE85Hq?dWcIljWLF<5X-H$Djc5a>Abp@nPW>Lpe^+jZ~Zp{T9X&T zISOk)^5N&OoC#fqEsG8wqVvtH=gQRpKH6XS;p`XG!*_oC;4V@EU?68iVcsf#($D4X zK_MyX-k_L~C}rrjP@AzpP!?zb1FHBXiHwrvxAYQgZ%FZdj7LtP?#o!Th)O6w+l8*@ zk>D?tliW(4hx0d&B*i@wHWSwNq3)aB9qn`I+#*8CWZ7aeJY>LvB(5%033~2vejX<& z&9|1)<|&q3GFV2YjGC-WwTL;!-$g@)Htp4oOBxgEA`$^Rm-0ip8KIZ`_907zUB>~f z0TQQM^JO08DM7fpG|ieSoy}$%JE_U}zH0BWBBYn`n9a_Gm>~6S#v7kjuUBN!wi#-TKrRYw0`-PI^(n_-A+5SwFd7^ zLnf;HL*o1|P0h#!Vw83YKnI@yFOPR=wVkssVCfxyeaTT5ddJmW^$#N1C6LH!pnE;j)_J_tsMRonZjM!nYms5M$^YmqMpiC{tC%%K z{+n#rFHT9^3QakGeaiPc!qR1hsXYG+4WM*c4oEM@O)nq-w8%~y3($%TNofp(ng^BN ziR;Xq84lC_d`*EosU^d4(|q758J&3{oVv97&Y4h0N#x4#IhMHtrJ%CUAUv>ovw!ao zF&t@w9F)}0g&{_JQ3gLCcZ;-JhnB~Kd4GC%Nn55gmpH|cpFX|}V^sdL9#zurj!`Ia z$d^2E5(y`j2+u>Hmp1!4fEJSg)gZC%+6$te)vd4pF1beI*;zQOq_t1=w@fLJ_}_4% z-iss8`XCFWMJbE?r5_k0weT`kCr>Csstu=I?47yQ3Yt>ug$LxU{)~Zuj$nI6?VDw; z{9zf@@e-j!cQre*Y@A03j%d^2#R=Pw#)6~%jdjoudF0y`&~WzBBJNDADVc*+p?Y-= zZ3=l%wfSrebVG~1Q@-X@gsrWCXgdaJ>yl=;5s(BEAePVykH(_*cK5c|Pqn}!Wp8>T zy8gjM~Le3h6sPcJVL$Bm+?OVhD!7!J4} z&^^XQNj>EK4lA7aHR41|_tg7aew7?}=BINK~70;;2+!a8j&9Ua{STOtCP z<>kEDkQz^d?+ZVe0B8v%b zBX!_oeHc1cxb^aK-auX_l=T4)*;eKN~yM1yE14;M~v*TpGa?V>l~0IX$v9v0wc6Al_)} z%EP#cRQFD{^Q{L#j6(@%)m~tJk3hzU8X3xPfL=32$lhf7oJ;T7SBbwd?Jf(I=J{gAd{1m!th;-Eckriy1Jo%x2YolM!?I^fj4?qEt zIitvc9wVXE`s2d2@3UO!B|<-vng%XHop~1m_oM242fAVBr?4oew|1`dTN}SS%P(Vk zZ?^m}iYF)c;@vARdamPh0YbV7eV)Vcp|oPvgw%2#VtLje2~4e&PXb zYwyk^7#*aR2iI7Wk+{J@f4k{%(PY_~ zRXe-~Nx`N%-H0?)-W|Qs}+=f%Pjoo24AQ!t^YQpfUY0vf`w(dB`dZ*!FfFpMFiA&nLw0;_`xVvZhe9%dsgtC%Wp~UQWbOv4t{gGbUrtdPch~`N%IjDFX^!zP`duh8 zO2Euw6J@Q9g*LV~u#XC%$*;$9qV=okHWWg`c=e-zciNt>2@b5&jRI3I3fze$dYoqA zl`L5j_zG6z7ODRtrc)_6aD0P$dZ-HlIgq5g^J6~AP_4K0<^x|S=D4R!KnUHf`(v{k z{j5dCD)`(P{kn%3)lm4x9)1A{DJs!47`&98i%cc(PZEjgDc+dqSsu0aF-I)cD3{4i zLU|%g(D|R&b|ikkNN*YmBdlX+atYrUwqP1mFFz4seS3deVIW#cYm-=UC$o`scYRVD zByOfV^$^2ktfe}=tVRW6JzzLaaW;z-xSdI>QHttZ7Gv-3c~EyMtgnLF(uU$6dwwpS z)H4#+2GtylU9TWM_q+K}>qwU|?Bn_`yBsxJ`sufWeTEtM0reouwjb3`A3y&(gIg#f zrX9YEQUD#H-aU?)v;?c4b=%@3@RY<_7f@blD!^?b+Rq+VVz_X>B4i5jsv26U``y zwQYQ>>&A8DtYg%@{!;pl$~k2)hn61M%DAA?UtAI0A?6YY*N%Qt+hU0lJ(=!o9A~_~ zInV<|h%iT^Ki57s|4QErS`nKDBb|oGp!Q>7Jeg;i@+}tTkJUH%k6Z9^FKL`;s~(); z-Acd5JC=y!{XJrgdD1uyHPtWgS}66iVL*ea3G}3%mCa2~HbFJIV2-vG zZl~m1zklLollpCL%MHPVmMAyCxN=4Uhu7+dJ4jMcMHLxBhNwe30!cn*7qpg{Y|6mu!;TXnK<;u?Qc;1C*N`q5tbsGMOjY5; z^BI>qA>ie=)UV-8Tg_NDcmW9go(3tevfJsEkj6zw-cPo=uE^t%ckhB>hgr#?t~>G) z$Rs_)47Mw8%$)6~ znp{;nQcu1oE#%K6GA;*NSRa)xXUW))cJuM(7@7Ni$aojzuHpN{E^5F&-Wh5%e>mwCxmrdJS`?~u6hO}Dvl3({RQ>YH-lZ{gSnU+?|9mEz~Hg4J7h&cDoB(Mjh&v{>% zTxT#l`y<^j%6&H~doM&bxLH4`o-UwRDS-C^qO5~5G9u4<_I)mzDAwJ#Ax2^-?~Ngc zp`5&*oV7Kb#>F78HG?@1lVG?e7Yi_GQ*+{G(l^e$FH=nu&^o@fkwjI%!qE1Kf&ok1 zCa_Ub(Ocsl%pnuVL9G&rYnv(B{86Zva)lR;ZE1>LFJ<_|mwpnK_^Z&!O#$X@KrS{wqASW zNRH)NZF+nH!qJ6{58=m>T8SJ`23&vQ$C#bz8BOs}ew+O8{*qtLw8$Yl!ocyZ~JX)k? z;1X00lcHhE$UU;mMkducJk(T9)G4;h&v_%C!#Lja98Jy1(E-R>eh%K(r#&U}n(vae zi7lx?@YAlCiWqtMAq`BM(py5b!@LLb8{qSh35U#3a0#ka4`e*yhdO8JDt?a@mKwSY3+f z=GQp(_25z-D80KxdrkM=U0Hq153R{LPc(zNpQCf2J)KwR0sI}Pair&u(3I(d?9&pO zIN-)$rINqiC3d?6B_t?H`8l0w zFs&X;qNJrA%1zB)4Cox#g`)WdvJ*Te#9yx|CVJ9iV`Cf26JRX*_kriF`M%cujvb%2 z?uo9;i8T=5iE(YFOh$;TkHxv1g@cVm5TrRhdo@VhoZT)|mFhY1r0PuCSD2yJLR6oN zwf+M`OPRE zeo_YTC6Kt3L2ruOLzBKGCx5}Hn;%5FYWZxpVOmN!-CurYPTeo=qMY0ee9 zsHM~;P2)4|c`N!+Oj-_e6<&mUbQr~_D^Fq6y+=L{PZ=>gi+s8>nHK%gb)vv$xR{{^+`OG z`B03%4Q9>r`dW5YOV@#!!g~&}D@=YI6yFCKW{4MqZmhg`ovZG|%Wc(XUUfZo>Q{X2 zp0H|*GlER8G$)@M^WSYiQ;jAYZjKEu*Y!dtG)NId4ZYp(3sr)eP~YR_ICpnSR^?}@ zPeA6k9=lY{6m4RKN-plTn@lszV&<)Sb6V@nOexXnb!cR6{0BE5oU<@-VBz5= zno7ry+d0wNF4JOHV5NDc?GgLCwKwNS2gl~zs+83oZ;Us))UVE#IahkBs8q^2tuEL& z6Z$V+c)C8UQnBmyvUe=Oj!Bcg>1b40ldUcJ?Ufg>+{AsJ7x?zO(=^5}!W)ez;6apC z@5VH1Mv-5xqH3<#k0|@;&Q!J1r8KkQ-(fIWa1eHhx4U~4gY%7{rD*d?YOcp$T=wfs zD5|LFlQC{BCW(6YT6MD@K74qv;Goi-=;@(S%WUMO|1`q+R?*v&dl8G~V)qlAoqm6gm)d#CobZ&jCiDc2*c4nqmXhA+ zld$Djll%9x55oA(wA8x6YDcxVprj_fbcjT(e+_Ozci?3Dk!Z{I#H zy}FPy+DVpdKn%&hdsi>%ZHh0J;Q6HIF&c#Bd4f%3euJ}1qFhpmC--q?sb%+&lcHEq$D-_gc{zBEu%2&Y&?YMh=Z0_u@hAyc#YJK zCi?bt&G0!vWJh^PWm29xRSrt?OWDiXTN5hksd}&qL5h!zE;7IJE2Jj0XXaBP>*`RE zKbt;mw3lkZsgVfYLs|kIvV^m?zdsn7dcWoe%|pp95l~jP-h&)VYo=s*AUXQR@B%vuxHmAek+`2luVGHYgLft9TVD|Eu%F# zed<&^oeti8M(1s>dUSJ*~ygfs{q&{RDaHGH#GuG&Bu;B?*d4dhHf^YV&E#>So25W;`>ZrjbkhbK26x ztBk6fG05YixP^2lp~QkSN2BcZOte2g6~D2)I%G%?n$FWvQ%{VMPo(y{z}H~cN_|_P zGw6M{DZckwA7X+H4t)d+0x~2iK(#ze95*V%c#UbYIl-Dyv8%Pl5WZjZt+VlZ@G;`S z%S3ngGIAldaVc~lXe)UUpoBxZZnkP;{7z-(dk%zI(r!^6s z{jz87%N~k&g7x?FZj6rk(rV!BDwnsnHuOzOVaLCijz!G6&DoeKO8s<2yexTq;h|VB9 zXRjk1dFPPxe;vMXt$}X|S;U9O?PT2Po+9Gn$B%OnniJTI?AFe$GyZO-Ut8fs82eek=-jn3+kf2}7O^|kowIn39@ zM4ul$)Sm3WQ`-!W#~E?y%V{uRS7uJpfj)X}T_;_2uyB@Fh*s8nPvvw6vxk z7CCN3vyZ#%Avef$1}_*cTsiVeiq5n%6I9qRS=@JcmBH+muGO0j#{0_{{m27Z`bsb= zQ9pC&uRiNv@0dx~q&*VlAoy7gBEr$74i&zi^oyarV2NXT4*MAiV|R}w&=XAFy*uR~ z8kX1j8S(_L8HZ7!j_h*X^G0qP4JT9^Pege8&o)G!@~ZS>NvG(-$TLO!Hnz zFW4LmfxB5UMIIx5(@3vp9v7M>GTpP`1g2Cy$8Pic?W#*JL`Os&w3Bf8S?#?+r5C3$ zE6Lsc$1!FNMQ+oEOAiF5F{A1BU26(M?wPMo7M@0_`$6=Ee8#fx9u+;;5bB_}$jnmH z$<=-0eyshhwQKzx{QKEbWyP}03%A!#@Du34!a1&@+St?kG876F$j`P`G7$66IB$*) zrq$qes*iA1(BhPH3=@GPq;W;a($b`((E6;^_m^B_+=WNtcJ=x^gajKtn%~GPx~t${ zc$~~k!A#2c)PvmREIKj@x!Y5x6NAlamWs!xSJZknv-B`~=){$jlpGeq`&0W%Ab5<^ zE_IR!ZqxmP=E4I>iKT=vUT>x5d)QjIGqbpQEFo@6u6?r zcT5%hXv_yl?F7Iy`*FNJYq$90o1T1cNJ>U!tZl6WbAo(#9vm*k)U!;-q! z^UrO-w`g|F57)xabbki8hj3vgzwp9^!8lE~JEqaQ$ky%BZ3OL79WP$EV6-e;GVq0r z;ZEN>if44{)-WY3awyJk56$9|jkWHggQMKY&~7pQ0crr}QK-mTPfFIOUtjGb>#_nt z>N5f_p5c7+ownBVz(cll)z;RA^DZOiPQYU14m=z%I~V0tRjH332TkA~8QP^kb$o>f zSBHFkLC_JOm#dnkXS%GWkfL}4)S?huj{Hzvl(5i+3zu(k@qPpIjgq)?Q}IE_F}TZj zPs-j_Kvqu=e>PsupDteM%iiur7qWv;)6WIpktBH`kqX=T5n( zY}j#LZ;{qI?b*ri+C_72F%&e8&pUN`!!B2~FC{FC@bU4ii)D{azMI$(oqc%))f=FH z6-D>swFN_(v*WMt19A^PDK#VJ!14^@JRd|{=SnNi&CPAADq3n+*O~uaP1Xe(UwpKZ z1)$25-IKqRkt7{Xb^JKU!rytfThTiM^Ud|p(h%ILhSS>cQWZLtBSLYsWMldr zB-?x&8s55sji4BKIUwNTEeX0NI67^mxl~-Fhu*tbn}Kz998J|ej4Z?CDTo)#X=rGO z(#l0%k_yT;x*Y574r-3fX@tRdoDBs;&{`b=KW2PP&?sFl?rsH)0v-aPWTO^G18K6M zO7C~f3W?>QOW2)g(IR4ZSrfSm@)5JPYFE`y$U(Hm(*%{_T_^&}J%@spk8NmMDu~Sk z0*+jk_lo);?<}XKMW3RGt`ygY9$PM@j5?@W&PPb`TZ+0$-*xqw;TDQLnmj;o;u(|6e zFF$afaVBnjhWFgYsUIYnF!=QVgBS88!>>u5S^D?jTYH>|Thxt#(hy4Z*bI4-=>bv2 z(e5bqHAeF5zg)&TGPF~*OGU6)>>$Jt9@N}=@pMD9{i}CDDg1ic7CZ=?GolEZ^d<__ z8%xgc7(9HP8)@B_U$GKVS2TG5l`YXoDO;t??;tXMt&1^d<@4u{o9pqlz+_lByML?* zoTHr5l1ya@ohM$l;09D*j<7N_dw^=%{+rRTGnCl}K!`m!A%_ke0`&?zNKPBGa&Y*l zqw_CS`p|{92S-OUfYhEhYy~h}t5s(pm3X}kvSyVL0v6Zk@iHLzt#;i2cG}XY6Ubb* zr00M7boKC&BSVm~dSCRtag9&}aTD_Jkt5mVnAQSkEn5xrM_V5=yr_i?No+Dry=7eV zptt*XZ3pW`hqVD@{XuiE{#Gou&tq;l>Deate3QAUC~UE&am2kCnzCztGX?QdGQG2& z?pHvd?u?0A<}-=P;pWVJel)Z~Rwg&Vy%`c9);)>>qTH}`)7nFWJtYc*CRY48KvSdK;tcGB7)N%^0-n(-Ha508nwqt)juK^2 zorjS@90NhjY%H@9r=8OWDDDck zry9JNN~|)tVb!ulyV4fm#BrffC!<&?8h>v5gj#%uSt+e;ZIr>rOVo+q)DP$ypTL4$ zdW_E6RlhlO@^X2qYSy)7(BDjCH*Ige)Pv3&c$=M@;YctxGzGi{(!&q!PK2C}+|{rK zco6$P#=E^!TkVC%QL46>p)MbE1Dt_ONKuDzS_*{rbE zr`rH4_KaX`WYSm@9DcSt&#Wwr@&k(y)3|!uaN9EVd&@)YIQ%EG=1Zt-5B;m*kmJA- z$2J=TMkBH>wM`dHrt7`&yT;xNde~HN-n=o~D4DCIH8h2EpgtiSYYNwuuU43$^QZTC%qzqfSXtaf@ClR+}>C`E5lyKip7_c5GluN|IQRG4{>BH_9 zeeN6A@@&Xm4h=#Vt`5HAo-TZRb3L^iH2j{+WLyfA&Gms(kudGjsFP?r0mzQVdt@9iONQVR$OFqvD3dj9PY0Bc_R zO~ci{v9~mvDkMIr7L0ocmP4((+~cIqV7Z6-rrpI0R$YckcN(NNt|8W|#7O?bN{^=8 zY}wY+$%XypRc2vLF&jE78mH8w`Okp|E$Gx)h$>{w6)_Wgt!ZCWcn$f}B6!U^CYx6s z;78)OcDbM{b~bqnH5|-Sr%&@6boZP}V#_AUNo4;pMd8r?_w?K7m{Td3%QN zMN^c^9UF2W527r!^dn)!KsaBTQz1N1lPo~u*eT)S0$K9+5YrGZ6+?IM(yk^p;XY03 zeJVcp3%vNMm%G>6M%mrFM4T*2g&n43Rqpy?l;#FhOL4@Q1?}+ZPiX1IM=F##-HY!w zFFAWDb_y(M7Ec$)kAJRj{Sm~y+km<8v0ClU?JfvHrbz4Rci{pEGzN1S1Q?FzB8kYj z6beaF2~5s&y>f-prTe)#O|YZ1?*LjV`yuKc707unRXyu-++|AS*u7vwHSz46xdvlU zOa+(F6+=i2N7v+;B_?#sfc2q@wg5R)IbdmIi4G4BOZ4A8aT#)Gy7C%I)}i!H>k7@R z86W^CU)Gcfhlpl~x79&#V~XC^*^$R+zmKP{72S#B#O>$VRU2leR&1!>exsj8ufih~ zyJrC&f&XVZV5ZQc$DwGZ`tLti z;Jh+wYGpmcm0E?DNCZO+@NPTvN}_}AP;tcEBsEt|yfK@;!a`_S=JGIS3I8@`?Du+t z{rvX2fkSrH$9fvt4~_ycCL50~9~|?#@fv@$@yX5;@EHM-Qm<-%CVqoLK-Qi%$4e}A zFN`+IsuvIz*;bT|x`&%>Y9i3L^bydUsW9@&n3Tjz={Ifk}kRno~OGipFDkLJ(I}xG;q=rZfB_uiP z3C!&MUFUq~IzP{k@4YVf9zuA_Q`WlIz3#PM={G+P_rtEXYjZdbFa11gAEV;wgnT~u z$Z&!~X+Rt>K^QmP!V1rTfPiv??UtkUtdTc@Gl#w%wLd{ui2v5XA%?bw(&w05F)Tl( z;ehwzMq2+yEA+fQ8t*-O%zu1*V24eO^)Sj0fi6Lt5TimjFvGfM3@^|`xH}GA%{=q# zd?O!%`1HL}>M)Lis%=+J0d`1rAAqSw9|(TQ30>dcPFB~OV+4OPop;qV#kH*4iI#hS zuye)psTUZH0h$s*+YO>ywrwLlKcFvA(CyZ&%lC}nX6)kPvQGF|Nhev2Q9jw!g&YoG zFNySVc-d^I3yoBjsy3b0ar2Q-Zwz(8`mkZ%z9D0+)w^Iv6vW_#?s&8~HA@wh{+P^d zm4Bp_V;GJsnARrmI&4jLY1=Pl(f{i($0~j>2lUu zjr=JuZr1*iJDN>a1=>aP8TH106k?4yF<_E!qtu?Cdl9=iEzea! zrO28}YbLcb`Tr2PKF8QtF%t1++2H80`*xcbciUBX!S8i=D;xmTK-fvOV+b$)ZQ%)7 z!1R7lgJ;S5z3q8qBpKeMECAcmvB4AqFE(o?G%qjjC{SfIU>yTvJ-QB{RFD^Yx}63| z#&a3F&1swNw43@mmh}|32}ti@V*#;#0k9W*a)Z)7CYThjx&BD7&yb|jj--f!+xTmn z%ur2u@0iM%QyW2_AR`LK?Q9#zLhqPDHkWapdECs{`s0znRT#%UrP0Nu9EG6o`V-p0 zv95JW5=V_rZd9GR@HJZgkRvH6AUI#FHddv$+O-`rrDF~G8?4s)U%7f!;pK5V2%5Ab zyR4?pmBKVG9(jVtylzA4%9^g;8?d*_Z2<@XF92E5$F%g}&dodXtb`PvULHWB1&m@+ z&7i_U>0V6jC5rE(w`$JENrbw*q;o~{u+W~m*$Z|+Jx7qw|_ z&M_1Z_@Pp^{@7-)soo3BVq$H{F=GqR>x4pZ{@$>r!TmRQxikddeBgTQblO^n20x_N zX+rS%v$`g~np6Xjs^$<4GR-p0_Ofn`!{K_bLka!xMpb45s7b ze?*R$hQg(Dqw_0S%_(4_ZMvfe-RldIGHk;Eok2qL_SdNZ$9XXmVT|R^MLt18eBblm zV9#{?I^HKmP(jg^F`FykYTEK}qvo=~m&fk&?sqWg@M24Dbn?FE!1CBebeS4!-C|hC zGTkFO1@l_XDE1>sI)MQJyW@p6!3e+M$YV6URyiM73^y_Ffmzp>bZAyocy1P-o00Fg zD>{o4Vy;o`zvF_;t|1xBpe;DtfAKaPApATel36^S>Mt>1@T120fAcpOr6sZ+eXZ~L zKWG;7RG*i37-#@{cJX$9KvIoucvijO)7*;r@|71UHX3U~S7D4FE-pJv?6V7(R?9)) zHJvj6Oc|s@&~;+Yl}(YSbBOXoLHoP1J$dv4hJJ3593@b? zqZnw2t>EWF0|HjA-PTgffQTj+nD0}Xm9X@8&dr%`Yq(A@Ub+{5Co5RS(#}2QD1mBT z_eu%qr_M|lFpn;yxn#unTL6&MHe2Op3)hg-abyT3!X5CllxxCdI)`)U6(^gLmuXyG ziXv3dl{0b($|<@rZRz7+SA!ueKdq?=N%#ejtnfAr*05nW8vkS7b7*>-t`un7-xBgPB#2?B1l>w?ACiJzeZsl|a?HUDu91u!X69o%b@Z*=WxXE%u#6%2UWaDB&^(VO-o^+%*H$|>^ zcglNJ^Htu?)iCe?S7l?8M~8V21Y4Y6z3Y-fxR~w}$bpjiw01s|{^eH5r&S3_nnWN) zhKwYtg+Tp?O?<=xOH=fZ4=p`Sol=*=p;yY#2m_N}U&@p>+IK?N^kgq~` zUb`*Voy-Q$2l~6fAz+Y7l#Z;7qx_5Z1YRc%nU;7 zNwRdmBBMgXroa0H=1%V$)eVjvzwjd0+x&jMd)qlPyUN5{iv6CmqLG9}A=)@zB*0n# z*xAq`s%nK)BK+bU*6qkhTi@vY1^~2HdYYzHB7XF{PGs4aSAo=l`6?|z)TjkZqWD|W zPmN2lkd!Kcw3h|wL+oV5bYm1G5GRqDFPa57c?_7yRgNgShUv^ZC2y%*7r?hSF)1nO z6ge$D{Yh#(P8kPGcV(ayVFxUe_4DFOob2ZFM|VG5FWA=0pJH7b{i5xvscBEIZ_n(} ztO}cYb*>*kD)9(Oljv;+4;?xb2&=s+k}#FLL)E8%Kc@?&=FTi~gv&8*#!E9^1mAk8 z;AmvT`g6@@^U;o6B~=4RJ}mT0;UDW^hxP5Sy0sCP!7OYRKYuPGCwW=SIdYVet99TB z?RGOfmSmW$AP<~-K)_3Bi_>f+qfG~IUANF2Iic-XVS@*#PJ`rZZON!i5 z^ROc+R?w;Bb$8@#U*M>WRg1twf2pZy9(D?Oyjh$CYqED?vYu*wGZq#{fsX@hY*6(S zt6gs0By0UOAXz7p1WcxHzF6Ufzq6e%bDv-MGAb0avYfQDnRmvkwx<>X8afSZMVaXc z5`Uz-K-d=w^rF}bZ3seig)M-Ob!oY%RRwOGGA%%Xfb}4gis!=bW6IhPkNDT!6_hPO z!y%wKSUmhC-#B$<=^{8r_o;Kx2~cHEHS<%#U%8~hAM^A{Yw{7r1*G&gBnvYi@2!F` zTd=UO5E&m$pd`CE?l)s0vDq;#2Q1Gi_yp0#(=R|)1WatEadGyCLX;Shw)*nCfDO}6 z(~qBje&f>G|FCr@ilQDPnsRI)4rnJM%8HH0p1QzWQW7P6Efj!bj%&Mm{rXOj2$~^i z1jF7GVJQey$}vc+p!-M)&Vn5Uq?(*}U2Uyqi>XujI4S6EQk1L>2_ZMwedpH(yZ{^r zcu+9IASVQDVL|RdJRY~Fy@~(d?%d==N+%xlE4p12Souv|c@L6$fJ=V$%C+6Z)3FiJ z-_<3S6xdChfU4~todu|5y>1S&0lC#y-(-UA(jw|WF_df$EUg_KV$sKdOAEeVu|Tj2 z!ia(!O{1Op7Mz7SM?&1#E+YLA#@{WiA`}ELkaB6BSL<@;epa|^;h2*Zv(vbJ27HVh1QAT>q52zW)=Kk%?N2JSGV;h&9WA0b**S(Dn>bSz=L#|DtwxR_)z8(40$V0} z>zQ;}uV3F^0tt>tpyMo1ni;vhiWr7aCXk6o406{KFjB)AIZlmA5J}b{EyR5$45Ot< zUMPMu7*G?3{X{i;vs-=bdrspdTn7}ez&>84YKg9{d>FPOj4$IuHMS4Oa#s z>eb9^xu+slj(+_!OgUk0BXv6;Uu&y~m{MfxnrTWd006ytnZ4!~ zH{Rw#D3{bq7L+#EG@^h7iJP+W@Uli^jCBm*BKyA0W{KTbJkBS5snec6kNGQNfEToB z$Vyy$w=2sL1;i1eHC0N=A4iWzEk6l^T33=}Y%g)W05J3O1T{GE*gw{K7`F{_n0W2rN|KVQ7x2rOHZsGAqU zeDute6fxn_ncQnrkpmXhu2PDFf@Zk001ge;bvnl%z1{cgTCtN!Q!+X)ruMmAR2Ia1 z*N8YQa3w_}?<0YmYb!!}m2aMqxTyY?@Y}x_Zf3kwzlBq0m)tmQv}4WSEYcVtg#lzi z&|#DSgcd&~Qp!viEQV9Nt2$D;I7Xeg)L_C3KK-+Y84UK`y69QeyNk%T;{=dPvXW5> z05&~Tox!w&p-m}dKn96_h6)+9@b|}qzrIqEy=8Cj%1kXUKStk4S3P;f6%rY{Q`?n6 z50NC5_6Y=q7!8#z#hTJ^&hPOrYo0NCu*`2SrH)J`3XMK}bv#^otLeyt{Ec zh9m`(X7O}?WL~Xl>m}GRw#~Cz!rnhbOZjihz4rv0tb)G8{MPMKKs2`7TD#?Zkshm zMGl7fGqnZJuOkQ67O8`BsOW>sW>(2Fp2czyNcemFnwjA=-iSz&dLFv<^u6=8p{lD% zFD*7^QE{B1?O~EU0Z&H|GP&13(oyOTSie{C&Tu*YNM6!`!ZtF4#HuwT#_qs-kl5yS z+kbrlXrQ8#H3p`VpF*LH=y~wgK|*gQXwPOOyA+DL$2Nn!q@c-0i($s~=^x1mlSIlq zjO{}pi=?uETG@mA$*VYVfbg^9wB|0rb{3!@GbCF@AGc$Uc1oyu>W~)5_ysMJZ&RC9 zWAvMnlA~E!$@y|fiCNoI43>zdH1IX5d93{l5xJX!`!?G+zFEWd!gBkD?)WPVHhY$# zL_QowgMV&5|9QvRsAJxDuHUH^Rf>{E7e>g@#qb?U7fwu0=7-P`BW0mrptC{&oJ-Bo z`z@fnYD*vCA{G+-l-0VeXi$_8Y zKd?a8J-glcSs;rR1Q;Z-;jN}70tu_u?RYBv^fJH(#cntK{E&Q5fhHB2K$<#6sJ&&K zE}yPcpa+BumamQvqb#|{8GBeq(%ut&s~o|$g}`7cjJ(9obSq}j9|}6zwjI`qynaVJ zFtRAH-uof)9zSJrsShmxpl5c&Nk+OxpC(1p*QIJu5keba6Pfyb>-3k1P zU=&0~=i=2=EzBTt3KEb%$n5cDlz?0SorT|Koe@|tG&Yj&`@#@?YYs-2f5ma+Fd)o= zBwcMfy)RtvJdfB0md|GW2>7kU59@TJBH-1gfR4Z2sgdBXH#RaV?Yz+xCaPn;FgtEX zN9l72Lu`Z;9Xj3>9JvfvA0=`*hV@9av0q9m)UG+js-g{?`3d5VF#ZbsV4>L~DmH8xkNrJrLjW=To%%p; z@Fd{v#$vgvpiNPj)YEzK;spvh6T#5d3hM)Z4)=B@W~D7`qH&NdkU$;G4+|qe4Le^6 z$__z5@UL*eMk53QR2lDMbU3qWi zS`fMbvl)ai7Y&kLWB`7YKBdXYNrZP0E>G!Z!2&zl_w=MWWEKM;N+x}Lu$I=SC8Dd+ z-t9>hil1*uZMx-jmWX5bB8dhN>&er>Fl&Gh(?g)vVh1d~%+0M9 zRh-5VN_1UI7B@5ay-1J7Q`@UR=&%!(pabcKt}*fl;4_WonG-cq^*eBBkY0it&HV7P z)G*3u)4i@11XE!;EJ^Rq!(JAztT5?lWj05FZ)@s zXmd7QhM5T5E)@a-o}8-uwekpV1L8G7@V)hJQfc%-C`^t}pnQ7|3aoTl$m*k#g6A(@ z6y91uhM@n;i`s0&Qo{F){0P^7y9*#4eTt^)YOoaW7a&htyRYym=zhiD(Z$s+a_6h6 zazJ7T8JV;AsF`!@k-U>y;wBs14q>4rz|5c`Uku?Wej6zP=wWxfW9@3bOK7w{G3)l?s(erwS|i**1wDFV3S9 zlBH*Uc=PdHG<>goc1Ga28_WR)si6>GaxPl#z@9yAIc6fT<@h7#N`ZPAp0VEu4~6O` zfaZeKN-qLTD4Vlq&*`8Y$Q~G;l!i#mQ5i`AVWVn+ZDp%oD|NFghvmq}mkY*~1eFOb zmg^-{si(G^Y(Z2F#LOr}v;YAYU#@h*^kY50Juq7`A+d|pbB6AK2;gqS=pex(gg=VC zwTjdr@T01G_JA8L18oeAn{S%gTFl~8WD=`$=YXXLfErsPxiuxX_uajKq+p) zZ2GRWF1DUZWodn#ZLA&VbP?N&9R(JUcMa&+vF_mE!$J9V%7+lGmcRhmV&8tDDP{QM zk^8Hi=Vo?|jD3jsy>AM*Y`ERXGnoCbzEH@WlvJ7LKnH?7A^fEWrv;T)j1=dZC1+Sn~!OkL{y%2Esua!m`ri5*C}0> z5WA=~&}ff$^s`1}{>>jBy=D^bo7Wy+y+&-ANs|_%e6#%q<^63Tg5ocEeqQ4du<4-I zm&db=Gldi-_v}$Ks!(p*>2_hkeyAy_X8GqRJgA$b-QN4SYyNp-6YumwVGzkSBxjaQ zHLaqEWSI&vgx#KQ9K9>~+DWVg5LWE8foT+ErqI#g&-<YdL++u{)mZdFH_T&C2d1aHswUT!@u@c>B<-vpwr@B|LC(!x) zS0Sz*vn$f4Z-DIK`m@p(e5atWZ{~4omkZ6jtTUfRdn6L)2JgK4@B-way@;{}*?@CH z`$(MKGve%6K)~$xFAtX`LejRiucMT(q`?uK>aUp9@LOOW;169ZH&qXHGBe=c?cwcx z)1POE9^ehh@|K^U2GK#(e6~~$gXwzanZ3+kGBxc+lka9R%^%(aRSNp4VkTl5`I%+I z=|MI5YxXU}XtYDcgEWonyo$^oQ$ zKv|abgexaQR}u4bLhgyK;na#;sFkU{ZfW7@_U z#beBC<@!bfSP|dDKDdc+?Z`F8b*1vT(;H{BOP?ro-kvSq3lTjep#wvEyv5Oq^egaj zH;V5UhXSw>0X09bZ#Wj+j*g050FS0u=jy3WUv48Hg}!9jXr35Z3d;2?e6_W;&u7!2 zP|I=rr3#~5R(xO;W|tD$AI0x#ThbawV1Lj{T-JA~FD-lr&w<$S+~fDdBsX_aA8t(i z9{q`ed-dnE&!G5tW1TNq%XeqA+X1ENoMatZ#W+oNRoIiUGAW<+f)Q?yVUehG0zoQU zWV3c%t6kl?6Y%t3N`JfV?(TjVibt}rgAK2r#lPVk&U8#mjfzL#Izl$in>qd z-29X{NMI$}a5;;2)T26>@Rlj)<+eh#E$gxHNP3v$Ud8rHY+RFQ-{t98Pf{7b^_k&L zB2feOrrQc|^ozwGRy7f);G%6VwT5w7X`?L8ipF9%#r6?%Itu)Y?$QhQ)11BPRq z&W?=0-HM9AED9=(zWXFoXEgOQ#@Tw^;ZvGWbhS1{bxcR9;<@{p5}P=0F$-w9!JsB~ zWn^Sn^ptxeH^ukN-8$vx=l&)~7+YTDz>>#-N>EpY70`Y`cT(SAwHy=+qG+(RATEBv z(r;oYl~y5zS5km=cb@?F@g&;7^l=Yy_EhBhWm8@w$@^!A`1n~!9T@Nk7v}B&)+3K8 zFR$R-xmh>F@!L#&xV2_2fh>;AD1A`p5kCpam&o5mv0#D6Iqk$art?KjO)zjz=o}XD zjn+*t#mhBfpsRJ6H2Wh%%w(0uBTuM#s;tLzCJUJ8(d_pR{ZqN;1!Zso1?;(cd`PpN zx+-nfK~@4N-xeaLIi9m+yQ$aBE;TSaPc9$aTHNDfZ`2(lsiKIC(ZE}Z4JyKcQcgK< zXMq`1bIX5U544*&7KKVO@$7yf{e3GRS`(-;)NjO?y6FP2BwEO$l_m>6c#zu-DaNNi z*1uoRRSa(;Mo8+Ge)Y(0t=$|hc{5)P$9kTmhaDz}+0+g{>p<*v#8kEAGu@{88ll)K0QOmWpQK_5XkdE1t zjO@~b_Pf%?aA+$_JAEs;2ES>tT7hvht0uL=Cr1RFD*EizSY2uZ3U*+Uk)nb_-!=5YytHz5{xu=c~G%4 zkh*a*Vhyk3a&vFBrU#L0<mUwlNsSPC-ktiW`e zewZD9rfz$oToLDIr14#Z$}(M;48@Oz>FjR>36vn=7Y%ad2}`2eGo|Zn6$?k&ZLTZJ z>_Iy^w;T2=8Pr{+=fHyhTFeFfv*>d9?N?^EnzYhr3Da1}^m<7Xs5adm}oNA-x zc?HI(j_K9)wHOQ+#rA*ji3rN=SDS-e$*1MuHxKtHfxsL)lGSM6@h(ZHHmd6*ftLfe zN@~?4H@H?cH;!aaf+LmpasYZYjZxz}mi^gxsD3<0dcbNy<++^$$iuWa_sqy3WaU=S zO64KXXqWrra|90mlSA~OCJr_qLKD+O95M*g!LF!uAwgW!9^$7gFIBmyCGqK1mMwzaOu zXfz{T#xK0jmTe>e6Uc`rPh-K)AS~2o;(i7EpLXQ$bem7Bk>HG{p!dl%-dsrWlkrh) z`K4&44pwdtXhEONG<|z$(VkVUTtx0tM4znpVC4R|&l9VIIz&@*%8wmYfGxcTo)w)L zUBO~k+OP6S^D-q@Vps>A0kOf{%+fO>S7q2$OEI#-=Lsg8 z6*D%__i}UNwLrmb@#zOOq-A&b-U$J=)F*I{VBwJt_A~z0)uhw7EW(JzMl;G(hxPD5 z*}bPUTNdcf<2{jtEosR*&GSA?2DRV%1^~IP?1Y3}U?!rs9lf=4mr?AGNMR@hH0Aoh zQ?Iw)Xq0cQ3q{oJQodJBOk!O;UjLNrz6zqrIy>gqv-F$O!>zfHt1V#apI1u$aK}03 zpoE03J+B543b!5J9R|wE(E>8|{_DUmZ{uMMA-4-gX19i#yljZ<#Q8cEupk+w{I)Xe zp~E1)L67*x$@&dwg85ILQZ=a9!N8F%9HUb*I&{|?)b#AvV^0_K3dZhHz z7Jv>49KpR$0ZCgP@tGPdC|dHmY!C3mP+v?hGG44NHU-*p9(tV2S(99Cejl|&<&XuU zKZ*(fLfJUfE6^@$GW&<}x}o;eh%64d5W}g3KJhn$p2Wb7P6Q>+cm*SQn#=wwOZx-_ z)Ql94PgQ-cI;loIk(N4?6F;>V)A@B^ui!NBn#Jca371X_ZO+r3zBN!k(A%D zXQ=8HtQcp4A@T5HNjcH3!@Al3QmlcgwR@w{6=>A zbrgWdgyYm_TEd+-RP;-10dctWJ{CW%`7l=FuRJsg2`_^3W_7(++8oR}rQ6%a&!Hh9 zNe33OaulR175~KtH8q<|pnwRmq$%`xyXs> zIT8Zl2Rc?l#Sf3yZvc<^MDOb5N_E_?`)>(DDFqz7AH#P%=cJKBx02g$e@mjdo48*Z z;%C~mpv0j4q`F;0&t!1bc%#!N#j(jQ{by(APZT9gw_o3kteu6~2armN<7Ph%K4aj^ z!1hMeY>|1=vE!f*Sj?s_PpG4?7P6R|=;sfmfX&_Z$V>SiIOHR=Og-dWNTQ>lAH0K{ zN5k*#`C`-Fqq&2fM7$A~Y&0A5pka4kKN)R1Fqnd0ahIW3pyHzJ?aO|*`@aO^akF1Y zH$lFi*xrVcvq+>K07Yh4QR8zz6ba9m&KmU`_M3W((5fPX>%7@7FoGzC!~|nOj)w~T zRgS1e_TIgF1z6dYFXH?rs=X$v#hbe}sxSOXl2S^|dys?f4&{SU-wZC`DF?Vogxs){ zBG7T|4$u%U{4rYt!paemvCUxRpCCsKa_T*cK}Mo=10T0%j(hJ6l<{;QWx|Hq$-3Vy z7~=T4d|>R>uhhLG7q9K0KvIG2!5hg#2M@Qe!UTWwZWl4{N>&fCi{u%=8NSTgL9bz&8%(cN!bv`G|+Y> ze_$@^uk9`Y+T+DSAM@*;l$&^U$I`Mnwmj7)#T33Z2G>^7K&iPt&Gw$ z(=;04!36$%DX==Zs zdjF0KIIP1_>v_^c$NAHmm8%U>R#$P{zlx6MSV(q`b^!UIw{|;JS~Yxsxg_2f$IfNt z0ZiV9s?i%(TE(nTxWKSt~Hs*#sI3*&@{QAr#ewJ)_!s48yW3 znQGFZAxGOl054nu#S=2C!X+!Fnvi6%X}M`AwAN~h+z1pA(#dFS3nv^qFh-z_S<_Ye z59ynKF-Wnbt1~MtX!G1L%eyh$wObs=bAj7|T309m@5p3F|Gq`uW>pgEnlPl3RSe9& z4LwH2>MXl34$JJEo=FAe5;V$+z6LPiJ{uyccruug?T874?1w7ywlAR%$}zBwcH}r^ z_jE&mS9g}grsUX$4OxtZ+Su`M9o!~9J}P05BF)jE<&TxQ|7g=TgPa5= z_5z4I!`WaYzKu#p|E{op=h1hcB#af1E|Sfp81FnH09D%yoEc&$gxUqwe_Yknq!O?+ zz*mn!P;wXLvwKqygN`fFnMi0XHlt>;MdKyye)(uQ~+05%fhS6AE1obRMc7D(! zD#rTct&!Qbws_!h;ej3694zw_Fug(3jX&Grm_8UnJyJEJTgoyP^F7ckW=U; z$h*A8BXNW`wod;e&R!#Gl?CxL9R3^Xq0VgS`0r)tmvK#i9N;jad{v#~HheHKXjN2MHE2vF?@KQk}1@42-n-d8j5suP5@$kQHvS}y9v_dzw zK~)r^t+=)JKg_~QxHww%YHymxY=SM+}^y9EJ1_GAwD}^7Djf%x=?n2%L$$;x4 zMM6TEN9BeNEv8)VWZLjnMC?h)2nQV%>uHv@*`gU}*nrS2rr_+f_gW)RyE+q%iKB~) ztwDZ`sJq7AjY0-oAI_&4QIx8+ybJ{82x%)*x6}=^UNIg+p%!DGCKU*3_**+KdfZ+r zQTY1*zDl2#L+$_|vSVYN9ae;1iciW5%AgmzLnec#hMdYhrP4mL_!x`jC4YbeuJ4Kz zm@5o;W>)jpuS^sR|0BSF9NnpjyAhbCcDJZMkvI#HOC&#p(UL z>@`@hK@u7`&H8xQw}@Y8g7t}x&$&EGALF2eA}Oew=>L8+OZ{|8|ANt#lgAmax)<9K z?Utt9FnkdA4BQ|;I_1_t(I|Add zYJ2ve>~l&S-X$i)8+v4rpnXg6tx@Qp;W+Ml!DDdnC6p`~%X=VP{eELTrB}X~DNMXF z=Mf~OZWY+B0@_WK-pmq=X;6*h{=#MnpfPRk`B(R*t~;UOvxLAwznKoABfXyJh*|HA zlhdHMRpp2&#>#Z}<0@lqn1~VG=(1u$`R|4t4yM7G?s3z5h;r=q2O=Jyy&Im$6Ra zOih=}|FzIAIFbSr=1>;d4^&lI0u*TFWXxXGVEn?_dECDHRz_ALBz)oX2v*CstO{~k zboASFADh7TG80BmR-A%rn(w!+eqq!qXr>2`4{WDI_`a%~G0_|-BXb<>@v(GjQOKQ8 zyia#M?J@B%An)pdSb;Yigi5OzE1wEx9HKt1HhZ6dV?q0t!lRX~&gLB12M@R3M>@oerPFA5f)670{?QaQKOXIEAk`vdg%XqATJo940zP1^!dijWnQT zlhKYQamIw@8cH#<$w0e%(2G~oreRPpBB|0a}wh5FO|x;d@Bw8=2aLSRGuJ>E@* zsI(DrwCO9*BCp_s{$xBfScJ6ul7JASyps2|BVIZ6kcObxJClaLc!H48Hu-_xF=Ouh zU(cZ9A5Uxe_gCMY9B0DRzQwUVI-Bt0@eC2|$dv^ey1KgG=JQp0opUM-sR|SMCy~?y z9m)@QDeu8_g8Fm}>TbDrPcOR=AKI+9a6U+h+0fzpIibTl643NU+mHEcBP-(>`10es z(xCMdYMljDPUtKesi-)(;XG=#be=t2S<-JHJ3`^O&3qb9EfYCkDwps8R&AAI@q)qM z+40nyP~GS6;r6Av{~~NJP|$YWK#f67Tya9DDAVUbCJ=^nMzZY5z zn*V67C-M4DZO}-+{XSwsP%%{@_2L@BfSv?AiZ6*OLD>A5Xfk3n|H!##zmM<#u5G zH}`rh@8|A926zmONAJE@r_%bG>q5YNHYLkFO;3C5j2pQ3*6XGnlFJvw{H>4~JgoNZ z-EOD3erNNxS5%-pR2T%oM?3v)IDk;EDY@#Aph9J^A~cR~%L=T=-ZXz6|Leig>E}}J z0bQ7)0@PP`yV*SL8gvKc;~QuBa(2){x!0SlT774qku8uf_-1R7WIs)lgBa9N+z?_D;wjbayITerFq@8J2;nF0eys^IU({#Ly=}zh^ip7lO`-YpY3Ts|=NN#Atp5gAf#XD!UZs#pGVeF?IQACvG#!Vo|xm>QUqQ znj>XXTI$tZp7602xZuR;Wex`rW&u+QvJ%#>ax^Q6(-nY1kVofdAsbu0S?^qd8g!^z z3P)`p(n9&S1CEnrsvezb^qR}5*_1ZYNd8Hl+$v=`N=bFgIm&tI^0;N?(?6jv^(?2w z*_uwg(bfOBPS~^OL8K);#;?upL=vb*_8_DWT-7;!qWl4Ss=IobU10m0LOIp?QG zV{y<9qhF z0~p@Xe%;fejpMAVy!RPv1*V~A$$c0NQuGOYAdhq;5d@tH60pBso796yufB4vj#*s? zbV9CjVMe9y!vgyE_7E?y3k8F;U%XPle~$fgETb^vNwyTw1o< z<|`E*j6bj-Ta6lnM)li74Yh#sUSD3`WjZL{imu+C=ZKj`KJv>3AsFuNLVH?KlxM-n z95vlf<)_*)DGRgs#KfcCy^po2d%$h)U0Mpin?-w_*s>j(bt~G}In%(36xMj--6^Bj z3e#mQsHXRs0Y6U8=*_ob-1h5AgnHEESd#!J7E|8Wm;DOMgivC>_H%uW&QuTFtoYK+ zUhPOX$T>>SF#FPl?_RIizwnBYjg=Z(U{@Hja%O*Q3?oQjgwBHkllH3G#+uG;#Mjw= zXpCl{cmdjuT8^?{!lcb-muXW4;9i#s>>JK%3AtSJOSfzf!lH)i+oZguQwLb^#{sCE zlb`iPZ}!C0fiB4*Hgw`dQ(zcD6aw$_ups=M=&k*S z?r=yJ3R|TPJGLKy-uq`rEv|KST{AUi^M_*E33`&_0scK-M^ZZ9u!ZV8o4UM5*cYj z&F5nZxg)7@ucrD}xCsPPH`bG~OX4RmZ08yl&0~Qx09iI61*c(Heo1DM*hM6*46zvUY zj!@d3fEt>znb0Hh354{n9t*nolXz>)5y(?8F)wp7B<7!y&zmFSjpT-yv-b>qJ-YVF*%=!hO9OOatwmJFn!uuw!L3IXa-o?<*i`az%)S%IQ|#$2u^^oW z!#Pif3Ev7k{Hh3JW@aJxgYy|2II%E-CbVfz$cgv4(?-b1p3A8XA&wDoe9*j6U4}`Q zw+-E)=EhBu;!Du6MWf0qH}skqsuiY@V(_tr&d~Dq4J!PU+Kk!7>*4W-U}R^B!~EPd zB}dYXZ=?EzS4PMc6ZTDnbfzg*k3tmCB2vtUb%}cA$}%g8dZ&?n&<)|TMaJs8JgYY1 zpHB5JbK~c27Ep&dPDDmP7Eu6=TW-!jOJWa3!yGxT?_AW!7u<#kQI$QBfdJDsk<@YY zfY~Htnr;yRC~#1}z@#a1RYS5IPW^^H>igTy9{rhV!JSO)v>}$k z-GgC_gnPj3JB>zBVLVIQMy`X?*eYuz$3b|ju52ugss{UCw|N7mL3bLAHZtBMLMtQF zZhRc&D-hHa#TsHvF41OIC=?EH2NYrTY~`ms8ZFjCcRWu0@znKZ8S<53@kab8eT94 z%evw7e;SX2;4)%_48}h;y#%;6Pp`I#wT|{cH7OaU6}^G+-0Lx+`Fca9E-ieIM6pWH zZx{Ww9^-1$PZ|*DFFRg)v67@#V|MYjOv4#Kf!7t-3-ffagaMC4yvwT*rk$;i=NPb= zf8(n8OAt0=&a=N%?w1^+rdXWgkcOaDiAE&C49#`%T`3mO4Ec$y9_wrqbmMN=%}Wrj zO7>m<1n_pdvmS9F{cSeq$Os3mpZiulgvqS@Z~|aI<)>-X=$7GJt>;2JeHX}oB}pwy zFh=NTea}YQG%_8!xH6{a)L zT4w?WUQ-!ipN=3P?oM^Sek04Qh3qza1o+nAd-t-SGy#=}=rwvsb4uPQmQfpav#Nf* zW{I0fY7`^VFOMMdE)D3=Mzjq1r`~2|W$BgzWjr{Dy!lS{+(afDs-k+cm596%0AWbN zeb((y4`2RDZZ@A*vCkWud_=7aQaeQvH8}{@5=_SPVAhE?4sv+|ii#{zQP6SmsQ%gu zOQsY60&qslVbROgT&?h_T0;x9zaw&YpMS_}ut{Q(E+}w7zveZJ(vbZdQnhj5j?F87 ztpH95cpA%(YvT1vf5Su%HJscK1ZzKx_k~{g=^V7d_1~a_j6n1O;D04!a5y7heWEmc z(77<$C6ZcbGF#zHUU z)eL^$^<8gpzJEiK(!&@Tw2l!IHSfp(M$N%=iRr{({Oc^~z6>%1jnF;F7q55->Xl*< zb?tTuoS+b`M(=SzfU#4#0tyPCphg4Uupn}}vqVbsPn36v-pk7cCgdMj!ZNcgb!AVul1v)7!DhTOal%>^-7?DdZSLj6CwZOKs-8g*lPU3|v+= z3O%8guJ5)RUM4aAF-&y4@1wZI_+-7}VA)O572fXF`K?sxb<$2HsIH;`mVqWgW%MV# z>d>{Oh>BxDRm4~+;|I7?Z#lk z6152D^Vj@$+0p-{gBP*~i^qSTYyB@Rzy9YB{+##XaN6fW3>Nml%8IseiWX@ zpXc5JqGjLDIu&dAlI?h3?5QSE_Y(U#%E%#soip&MC^ik9P2(f4db5ye zJKUkXl)LOx@7p)LF7rt`D0l5|&Fhag2;B%SADf)|STd0pXwkFRRd#H$j2qScoyabh z3dxD}xcW+N-12V321}^8A2(|^6)NuWJ885(xendN(AuqhjlV|bJ{`&qp%HGM=o?^} zE!6eJrl%uHrwuQWpT;evzH?!X*yuja7mKH{P^#i@K|pMS%9SfFA6>MnI`S?;XRfGB zT$xOExr5QD%AkHk$MCLg94p*t6Fvs+%?^IwiGE*q)XIDvCv;ivFR97RWe*AQqubTF z{a@$RPd*AglNoVciJj3uUDMd#^^y2Bv|DH1S>fsPYAL6ee5?)gm-7id@#^!xoQMm1 z(416{+dqf@i~Wken{$3=y!2l(^H1}3sP+EU;Yr+`>&WQIop8<}PZ&g6y}COPq%TI2 z)sup^ISt;*AKlkSjx^h3V5%TtK~1UgSl?mQer>L}opzVS$TF*Q)Q9m8%C&maJ3xHJ zPt*H0vFtf?_}8qX#2IbgaVvyaLD)lz2tI4C&bT+^G9^tWXUBJC;T{*EU-TGFTQ6{m zU3<4Hezkqah0$?gIWq!3k%MY|vLk-mhSDFq+RI1QfxUiw7<*IODB(4iSv8&!u)xY<{j}gCFDal|o8x=&`>1^+j4Dm$ng0+^_oOL3ehnXJ_F) z5BT>G>tT8QD0^ad;xSqNUjn*wq_o_=oI|n?f9EpipY%EHf*ZyB$rH&a2YW&j`?chq zjR}rZyRF7&KR&U3Z~>j*7~$QXEcR-ux+0Vk$4~XUAA$>4X`Ah6Psl<~1>Zc%{;C}l z`~D!*(q_-r+tWf_;{!O9?TUk};mUDBoa5AR9yO_@iCCAfb8I&M3hiKM za87zfrHZ`Scp$^snC)S8OlIao5KDc!@1BGoq~m~A+l_ixW85~GagDt1YkV3Y9^LI! zGFP-sqJmisMf7kf+ar`Fc-PBCoAJnH*{nK6)njIu zMWi2q`kSj2bEnxJuH*{+D_fzL3FAl|>`n)duTa@Zn{JP2Zx`T`wypmlZJYB!>HYRj zKm4$wpu-H6aC9U{Q8#UEG~Kj`NXRNQtB0DO(687dJ$P=#d=c8^Au9Cp&Qc?meBc=_ zYg6tx<7>bQEk)Lj&IS7P*726`wdQ|GmXS87jBPP?HhC$JDHxJR%cvr-Zv&?4g~<6hv1jgA2Jj3KmRC@0`yf9L*;4bDHNdYfA@O}Fn!yEQIvIbXU? zNG)Rm@X4yg*Dc(4GWz{;)}Li)fsWff|V!G!p}DjX3D9xUB=v^A#-fRZ~$GW zlX|SU^JZ|2Q#qkJr+e_^7xOv>qu$6uhiKgy5pO>_Y4++5M0N(A*a@pLnQdb0_f_N^ z@tA~v)2x6(p&eWu*2{crrgeLM+6%mI#RsO-3a(jTsPt(5UfPST8L~a$EJlS2GFwKh za;={VEP=6di>o9HIgh11?RY0%w|;IqKc#Xj*V~a~wJ+Wn-g-lAh3O}9>90zd;s(g( zM<1*A-hM9B89%dNjo;YqMw@owR(PgjxoGKxuol+1*mgdfZZER?o}*;Os>Mp3(p=Y- zAvad)nSC1pl7@am0@RKtY%ZCJD-&?Qa{@O<%WuA*Kvm@?dRbKsd-?t7Mf=Qg{`LYG z@S{IU-YBq+!z&C8{lHK?5i!GQgRhYA;0m;>SpNDe^KO#v$o1LCIVE*24EdW69$b1L ztwicoLyAQlyVkWi!ljJcQwA?p0Uzkdv{z(2d++l3Gn)qIRxslt%MU*(3PL7jwwwDk zWW}Qw@RSj9w9kC%goUthoB59f1d0KH~+L_Q6SqMT%y1q5up+d<8*`#-k}h;bfD_N&xx zl|;Z$XQ>@$GPiba&1rbrKG$`%`e!R_S{QXCAG_{ExUV$Aj3nw@7nU3 ziZRF+F5t*ST`|SNgeqKD$mvketJE>M=Q;R!puAGqF-y>G5VS103toucDS+9q z!i0o*27GRUcddPDcB)uTOhcOi>0Mk?lSXXV;^p3wMNUiZyO9fKSK5FJoGyC!aJBWF zXvSBSwjY=GCIQ*e%V=pRYb8YiN}1CK(cV=VhA91OK=J5H#rQmXwjkQC*LZ5fpM!J- zb_Xj8{|y$uy0aj+)=wSPZ2?b-6xcT$jqU0IN6=voS-)k_q4Bl!+fxAq{ONL z#)hlU$eZoH{Yu~vTe8UuOe4O#E_Y*SsaKgY9Db(+Xd`02Y`3~{P=@^bPZqu9cpP5a zDYpYb&N@@|Zm$;-ZiEZZK<{Zs^ zL*boehA3gExDc53>X!KIjHBM`nqAWzGN*;rTOBtDp-wCpuB91;C(PXL;~;YlDPm}q zvX9~pJ`74?hbF{JPIlZ&z+|fGoF1MiF1fi|$3r)^(5^~Jk(5gH;7#E=n!&9_O7|(U*`g+CXs{3wi1`K-e>LFlQ>4T@g5&v z+Gu?~evU_6Ab3hk#x^;5R}OD5A~2pP?CcsWN{PG(!-km?IABlZUHSvoOuH+ZzCn*1 zUPgAzN|?~*1b+emvW6c0n|u`*|l;g&uMy(<(u z$G`-V%YzP5iKGgZ!`riTVmu6=M;n!3DOK)46H#}zr5+2l=FYvzd8Dy_f7(g7d9|4V zWq!UM^*OaIiZj8dDr2M0-hIvPcdT9iT+eI1n#fuM72R32zTVyj+fpBue{dp1Ugf{t zujbpa(8g=AUAazD4i_`ZX49LjqPnFcj~Gh^20Ng}Ybx1ua-1KhpD3nvwuMz-Q>DU0 zuRr=4gOrB9@F@-*2gSfsvF|)qO1?JxdY`Lb{sxiyc`j6{=w*K4Yya zMHlpxPgG(r$gC8j<|J5;aL?p^hIqFhn$(aH}@NUKeryrOQS&h?z6dBnLVda?mfl#0Q`)A?WFy=_Y z-zxaKDQ2{vr6l7U$&1;jC^z-&NBu>mqnGvYt*`0i+73My68-M5$>Xh>JBR`LG3QP? zoq=Wi>Gx>nu>||8WRQJ{Q9UBCb2c9t5|Urw$m&>uM$aI2_|5%}dzQ&MCA^-U0i2j5 z#Ax!!$S$+oLy=}Nh;9F7QrebR(B zGXA`{>F|6d=0tb>#9%IgGJQ>9v50fg-``ZRd0l@cCNdi+FwWhcy0vzpL}q?cyrnebM`1Ap%3mA1^jLodX`WVJKMRb>|11>-#M4GME6z!%_7>`k|sq%ZY2lq^m6x zGKp%hOwh7PFU*AAo~sIsK%C<)u)0Q7IBO9EcZ1HD4!BO<_cJRWuC=X+rJ#!kJUj03 zUfi@g&`k)#57h&Ho|_kMIz05rrpnO{@Ul4g7TAnG`sUz$jK%N@j~hbrgd-`nnoesU z4BY9^_N})n{|w0TT#${4$x-!e!}jw_b-L7^&xyvlO6b>#+#i&pPwCM-_DG}2F6=Skz9;gHr&fUG!G4_IZwW< zA}wG()ivmdj#2(lB2$2&g@08RH~M_2bgB#J-=jpyLk6N+1?uRreJ1qDqwe&~D7X`k zS}MyKF~8e&)7}nqmYY*m+bv`&qC*^`SsQ;;IW{#35=|rrT}}fpv4s~e%ogkd6MUgP zk^|FFi*=h`(iZ&9)YB=_qO6)D)+?K*`TOA4&$`^#SonI`_ge#1ScQ-^&m1muvi!`R z-aghPl9mj z_6#ys2@^9@d@mhUizFYmc8a~R%!s)?B2v_4dc*SV2lVO{ghQFHoqj=y^mLb-I&($> zb<+X;aJ=7$Xq8+L>~MPjG>$j6IAJzxqg$_B^A<`n{(fV#+^4fjro``aH)&-ATR^kk zQ7sG8p3b5@D&l)?T$XKpWVWkjxkO`|Sw*fcrQ$Yk?ACQ4C;`*cnZa}KUckqml+T!# z=#4kue*gaB2ya;0!fT@bXog@Oj$q{2Cic_ z9@WoaH}vxlSXqbE>@qD@b?VOO?d+fHXn>*xUuEti&4?mQ;MR^xJ|j+u~NC z89wh3;)_qF^_zSPdwZSM=VM-1rnLj#zgLd2Dxj0A3+FLFgb?$~*^nGKX}-b-HIt{Q zg@@*E(xQsX%atGuPzFTh-+evW8lpCHFX^3ty#rn5Ffm{bP4D&Gn@~I}3J>_nbO48% z+QBtEX&(B<1DR&;_`2v>l+b{WiZw{643xqFkvmMkon zzjcJunVl?&p-|qOFH*SomOT55DablInf9)P#7&s*Up|w;EXo>VR-#?&reFqTCf7KxH@L=1#-Q( zjCH+{Fb09j?~dy~W+YqztdGMOXuoDPH5ib{;=aaLd2&P6)%?~6lBwdTGGs^&>f5Sc z|0`>pQ^Z{Y-w_CXlQUTn6)vCdOg@M%SB@f&h2E7>+;5>+l=@|~2-~y!M7j6GAbs&g z#r9ah#rT1i*}l?-hLh6VFXv$_icF_y~?ms0eyJizGk70bF-|8pfn zDUk_4KlSeGzzQl+`1~=o2|SBlHS^A!q$|>kf+Nd!0)n5M+4Li0E&xTm$r=P1N~l35 zy3Q@YoV{b&U082lX0mS4lxy&w(`B}Vl7%OO#}zZ>ItWoR&&6eeZ&@N7j0^Im$_p-= z#&9Jw#Lx43%w;*9ZtBH59CwX(OL!P94yy%mWPmpOT-OksR8xpm`jfWmZ6&ZrfcJ@DMD$ujmXZJhCrzx99*q#T&M2)8NBZt>|xC|Xlihq zJBz;RkNLZ*|D_SuR~EiZQdEHTfmuz4R0wlo{QMe4L`JSoy~`YpllN}HDmy$8$Eq&B zZqwE>G&dG0w`YdP>oW)r>`%W|PFCnP2ABO7Hl8h(!v&fm zvj&NGsS>cd3lf>{O{6WE=XCFXJ=P_h226xccii*FcG(E3f&+?(-EQh??1g?}I&dcV zoQppOX5U`%u3GGEXf|jqGd!7Q{Zy{*z^#ftOkRLk%vm+%S=&bsvrx0>15}N8Q+mp3 z+f&)PwTZGX-Z22K5|22arN!!ddpjxOFBu<#sjDe#{n_xUMspa(ck}uN9oLB?JcX9{ z-ovKWcN1qVr|WMoh*7z;3MJkHx;~0@Z?P2nqNDqKr(?b4Maex zBN;|mxTy#Vl&E8$&c% z3QcYbh)LGv^gOyG4@l|Ajq;dqFsRYCe-~-_Iq3H7^=O3C?!{bqmE9N6$rV)yfd+*o zKGVIQfwf^z4uw3&-py!u=j{LH)4H64+F3P*&$y?0c9w=qv~5ryU)mvjaXq!)|72R? z)0L0_D{Ap)97i9~^J5?Z(xq{HD7_aL-S3zLqlmkSQ8N^2q2oXf+l@X$hBKNBh=L{~ zxlCZ_putW`Ytbsj8O;QglkY4pb?Hn0%`3 zsQzp((&Dmfw5ykU`6QS$5mxL^!aWojJke8Qk3o~QpGcv@uiN+Dv@r03tF-?D&0?R# zc-HUxAPe@ly4*9=#t{;^&i66vnwF@8O0R_0_nPktgU1#I`_G){8281ARW>F#`X!WU zmRh$zaZUMij21E^ru@ zyxCb^8{EH6PJ&?%sz_jkzE5voALg_1O)=N7F;0kcKw80D6;D=@aX}20?R8)mj>S;& z+g5Y~K7P)vpaslgvTaC^{5vIG-kgQj2O#9f=R`#w;5FD%uwtdL?;4F-pV68RK^44A zOaS+52QNgLuDl_kRNa5R2$%ElkH zA-oFMIAH3@tAkhwFLQJ~9HJ|j$*VX*6p<@7-2SfrTsjHi^X0quRL2XXVr0s%UVHm= zi*SZn;s78;b;Fva6f*_tQ9^~iSyBRKm&^eybGCL}Fa}L7ac@ay6sqSpG2$W5loVi` z828lh9X3@UIiwd@s)h0-z_i-PVAmqwd@t@w|9!VWoQo{y^f5%|RN1CV4r;81DvcYBKH)dAbOpIpRmZe|m$UB3}nEKGu_(Bn#< zwO6t@b6(}BNTg_&=Hd>DqTPr$cn5t-^N`x|Ywp8`3z2`aB~4za=@uGeZGYMT={k*j zWK|jT5g_09tm7=$D?WEf`uvIK@UeZsYQbn_A}B)p_XAFVFJ&*BO6$Dx;bC4x|A(yI z$|;H!>t0P0RkG5y_;Ue0BBhvzB>UPE3~TyigLcUGJ5YyhI!ypB4)zBUsaW8qv^A8@ zkPs+XXp* zahO}Dv`s6(pc1tTjTbfEn;D;y&u@BSQGv)yPuZ2AuqSfRkxGrMz29ucNl8d!=gw^_LY;oKa* zi{I0C$wdmcNcARxrQ4cxLPcXB|53se7?Fm{r*?jQaeLEFMe6_;0CK#jrn`!!6b z+#cERGt7mZt_i&e`DQG5Su*5}e6IzjBf^nYSoMp>L0p55v;)__hz!LGYISZ!g=Cv; z-y``48s|v4<6ZbXmhakLB8LFlipl(dzP|2Dst7Z~`_Us;=0(hl`jY{tBY^yLc^w*x zRVo5v0(>4NTBK;uMgrAfqLDpmop-^kfNNzmk;22vqn4D>){ z@I*y~PSo@VKtwHGEdbv7L*K>?I0PGSq#R;-t-w^?94VHBlhUDNMbBKZhFB^4ry-$2 zGQg5O=@b$&?@RRBA)E;4(IF@|Du$M$fj@iZ1I#YT1-xQ!FlBjs#sOJOcczG)Ndq&fEd#R2FT$f z2AsDN3ARClt^R-%DAD9oJ6>}NcA&ZA_{Kyn{k!>nPwY1^d5zKiy3#!89&>#e^Cg=E zOyf-)1pr_8BB6GnUXgN9rDaf(7R!ka<&ywzfJvGQ#)37~ZNccTR5R$<6Nr+P>E@mIYCl~;%b+hZm>0l`=7lh@ zLm=wlW|%C_FM`&q1>$74% z9WL9b!(e_1_5y%hzqDjfMTOXrSwl;z4f7tUIx9M%dtP_r%URbb?&d)$_SsFw*fRb_ ztqJ4^j#__$!5lb|qZ{flxJ<)JLE#1{xN$P=hEHav8=}=QtYc>lHW>szJSZ*EQ{OQ# z=)`wiOvK^{53p~y|6A)45Ez6sR?uuuS2GD% zSgp>EjW^P8i32VERwgD=poKH`bqsWn{mCOWF$YOXmJXq)XzNuY02X$&EfOF?I4~FL zs$8#aRTc@d>aW#eFzVp$_b`J3uqLqOB?5xPZWT=;h?-nkoOsaj^XV&4@vfw(=r>uw z&!B=9MU;Z!$FH(dh8J}F!e)L1mAQk4aeqL&;I?cLkpR{q@I&8#ULB#LVW814R6zZ` zbp9q?;42_bArIa$${N4Is{xN^~=O*+yU z2VHwFfMO>gK)b~a-24(Hc)marixu!zX>h^lfH|~o0j|j{RUmh4_-td38idqZtkD^-(q=# zrtwn?DzQiV^mu>XJ>iTy`mFi^p(e!M!~_x?aK-Tl_0Mxn%SYn~o8HTRetl9R_%d+t zKi|P$w#t9S<)6X$ufYGq?OV3|7Zd-tfc(Fyjd{VZmmcAl_YqshKttK!t2V)IF2U|P zu7U30V~e_~x`vXPnv$xPh1yviHBB9LEk#vT9aYtW4S)attAQ`l?T$yp|J-2k)yE#t k;MD)=A^483dr+{8umAttCcbuVQ{Hb}HNH}D$@$*@0!Lf4qW}N^ diff --git a/docs/web/static/assets/images/overview/lancedb-architecture.svg b/docs/web/static/assets/images/overview/lancedb-architecture.svg deleted file mode 100644 index 585dfe991..000000000 --- a/docs/web/static/assets/images/overview/lancedb-architecture.svg +++ /dev/null @@ -1,60 +0,0 @@ - - LanceDB Embedded and Enterprise architecture - LanceDB Embedded runs the application and embedded library in one process with local disk or object storage. Enterprise separates query serving, read execution and caching, and indexing and background workers. The control plane governs the service. Enterprise persists tables in object storage. Both offerings use the Lance format, without automatically sharing storage. - - - - - - - - LanceDB Embedded - Embedded library - - Application + - LanceDB library - Queries - Indexing - Maintenance - - - Local disk or object storage - - - LanceDB Enterprise - Distributed multimodal - lakehouse - - Applications + - LanceDB SDK - - - Query serving - - - - Read - execution - + caching - - Indexing + - background - work - - - - Durable object storage - - Control plane - Manages the service - - - Shared foundation: Lance format - diff --git a/docs/web/static/assets/images/overview/lancedb-suite.svg b/docs/web/static/assets/images/overview/lancedb-suite.svg deleted file mode 100644 index 2eb7378a2..000000000 --- a/docs/web/static/assets/images/overview/lancedb-suite.svg +++ /dev/null @@ -1,92 +0,0 @@ - - LanceDB suite - LanceDB Embedded supports search. LanceDB Enterprise is a multimodal lakehouse for curation, feature engineering, search and retrieval, and training. Both are built on the Lance open lakehouse format for multimodal AI. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - LanceDB Embedded - Search - - - - - - - - - - - - - - - LanceDB Enterprise - Multimodal Lakehouse - - Curation - Feature - Engineering - Search & - Retrieval - Training - - - - - - - - - - Lance Open Lakehouse Format for Multimodal AI - - diff --git a/docs/web/static/assets/images/overview/training-data-lifecycle.svg b/docs/web/static/assets/images/overview/training-data-lifecycle.svg deleted file mode 100644 index 4998ee2e3..000000000 --- a/docs/web/static/assets/images/overview/training-data-lifecycle.svg +++ /dev/null @@ -1,49 +0,0 @@ - - Training data lifecycle - Four pillars of the LanceDB training data lifecycle: Curation, Feature Engineering, Search and Retrieval, and Training. - - - - - - - - - - - - - - - - - - - - - - - - - Curation - - - - - - Feature - Engineering - - - - - - Search & - Retrieval - - - - - - Training - diff --git a/docs/web/static/assets/logo/lance-logo-gray.png b/docs/web/static/assets/logo/lance-logo-gray.png new file mode 100644 index 0000000000000000000000000000000000000000..79252d4950f8f4528c1a507359b130aa1afc37de GIT binary patch literal 57024 zcmeFZ`6E?()IYutiByJLl#J!3h>#rf&|GB5lq8Ns;mDY{#A&37d&}4{b<12F8KP1r ziZVn=PEtrn8B>NNpSAX>`#j&j;rsMMKRA28hc&#`YpwNu@3WTX+ZGBg6C?;?;g0Q_ ztq6i&iXi6x6qpNtarNACogm&t?%2G+COCQEZHHeA=auTf>$~57YKVp@@U)B4RuFvY zp4l&s70ENAiSdfubW=+9Swd<9Q()vQk+#6x%xUWFVXcS(&5~Cy|4z-SiQJZ^+GJn@qhpRPXhlZf&Y`h|4HEg&k`6P%9@#71rm5wAY0uX`=X`w`PXln3C*or zewKJK$9Pr6;5YY`zH}2@TfWL#USpB7a`uwWqf|bv*M843!&qU^Xe@UYd1Z{Lq zG(2q7B94wkQp@@iw)gYxe~*l-ftz6W*5MF z{L67^r84X9C!VO*oa&g0X)EA?hkbaq!Gr-kCFUu0w%=S+XR~LN;PIkAwBc9g3*qj| zu`iex=UjBY{N`q=^L)7yXFixH3Kzk#ntmy@KJ}GTU1f5IfSjtAssKSS>~ZbvPdT3V z0;XGv6vOHt@WWq+$?@`D33umt*vTI~8Y0hFKMP|pB1CW^uftb$Gh@v?3*5G>Jv`^{ zUL*LM94^>Sl6v1lvB&v!+L$7veoVHpAp-u&Q4qsPa=%suuRDElTS@D!4Xst%#$-Xb z)Qz36iW^&vEr2$2zRe?=~h6YkbkxpAz!dEx=>(c(&$xi(sOWkW9EqZtPBX?9LmdBdq&OB zOtB)TLP)&pOWTi{0CF=z7UX#Q56j<9oDDW$)JFj6p~aAIte#N=$)vn;;&(}Q=Hu>g zAd+12rZu+iz7^k}{{Yo8Bm|@=1Vg0Q%B^us*60xKj3r5~XQR;tA1f0JSWiQJL_Otz z(4Qc`NecN}#+^~?!6Pgk*-QzpHp0lE?e$wODzkfRC$(*jdOLW3DJkCki2Qt?2YOk&y^~Q*Lmp{Qm{+&mzjPy6UT5H)uC zegovyM%wTDLs^1HJs698g3~u8 z@@`A7e|%ApyXy+>uJ?z_4>U|IHtrRCLaLQfPO4-}f8^*9lm6Q<*upXyLvpNvo#f_C zj4ljYn`FV}C6jUUq&>Eu{XLISR&2Cq0WWn3y_1`&o~IK?dL`9_ zT*gzsK|Vrn-|eZ-JAQ65y6{_@v>sAKm$H3NO`2yGWK9W^?;f_@#MwXtp{qBm65-?m=v{l{KJ27L4~$560M^Av3J4?+oNV>oCRCc; z>nJQcHVlrS$K4I!YG01@3gd<4ZKGA|XGh!7RVUTa^;qE8KjW-)0SXvTVFT-CB_O15 zodWE_$J*op_&i&<*XVw8I0T)aT*n0SOUsINYJs0r_JMOwkyyoO5Jh-B|3cXHaMqPK zxjdWy7mU*uaZMc*rIQf|%Oo_O0SuAOKt?m;zL3~0Kxg+{ z9M40X7Kta#GC6!{jI~J`m}UCL=@t$2`Y7(A0rqr!>mSOq#Z#P_v%!ZEBuo`ZcLGOl zRm|vsmuSm>;R0(ZX(0cAsxDbA^MYTl0&F=lXA|EP@({+Fzkk&?Q9PCpA~FW+;dvsp zorhrmvL0~uV4nbRq z?->h#qq5ccV~9r+E>|d1=HMHQ$=|=+H&>jo10xuC{OwbX)Mn5u=}rPyf1@p-ZU0NN z`$>T=2m6k-`tAttNVQMdPTQi=<9CoD>$!0EbnFWc*s)~`42Q0yvGl`uyndm6_NHHNx#l0Pw~Z!7DyCv~Qx5#pAc#;>=o|YZ)f@fl$D}Bo+nyEhY#-cB zA;!0nFYS+(C;0MnWl<1Yc?dOLREG$-N~5(&0hkuUwgIgAI^v-oY1CI?-IuJs=iw5* zq4au<{~#E9$vt&}ZL=vKM$~Wb(*${U@{`hm=zQNyamE35O0S0KcL=P3ZA9wU7Yr4c zo;Cb*{U#U%LIcSH=DZZ{-euAs03u+2`0*>GGj8re)t-XivSMHO;cvSg3&N2j*=v&P zMv?$74t`9u?+07?RO7gC9>G#V-O61LUut1(G7Mfp1d)LOqv9#9fY#NBGD;dqgO4KN z8)%4YzHfb8G{F+z-4YW*h6G~)a-bXGOPzrpy@Xi?b{xL_SGWoTJ;nY{at-l1(FDDI z>LT&w*7ym6SV#WR0RpnM32-e65!Y6bYSx8()L4uht~h*YP3()yfSrumwLV<_#xIX9 zz7b46#H8OAcUtM#&Lu?SBru!;GU#X04>SXQO0l3ENaccq+A4CsR9J-XK`yOhtRSbk z7rwM3GdBDtn6`l7Z(7H7%1W5W!=&E~2dnMlXboVK;!>=s*x}@#!b^S>;2|1nts{5oH+Ts(u`U-wMLY%>-vxDuq?@^ZNIxn zS({jZ*%O>MzmZ}73RHz%hHQ18V#aP1gB1OztB_oP+&seOzuzFK$$~8rLQ0V_<%78p ztu2p}qDX-V*O_>78$kmXQB(DX2ZXla?9#4h@X8{TFda z|NHlFF;*sNKs!@rQmy_Aya%%XzB0?@hABLq8$u zw4Ide<;ldr-u$2+Y7G(kjT7>+I>Zkyt6v!)wQ2fjD>X!@uLJyBpYQLOEN9$cBrs<# zebge^h3%(Eb$%=Gp_3^e;)7xiPiZ`JhV;Pq<5Udt0mdwe5qaxNx_?|3TuaxxJH7AC z2WOuxfR#$+x#9JZZtxVXh+L*8EHgZpA)mT!Wt=<=V!7czU;0IP^B@=C(hAzD+5R?# z*&ps#fqRN!Zs4=8T$p%HE~;f+je7A|zIXWR8m(91Vhj`io}XYnLynJy2~)25KpxWA zt9|CkutPW7@tg+uHJ@_${@P_gn@5c$}*5zHmTNxZ!Q z-bBXp(I2-yl-qHaQbO`9@eSls;DKKdrZ)u|nd-Y8j8BF9?6xI|HK_1Oz@ths+ zw2v(wEdn~jXd9D-471acYhPw@2T42mpmcHCEBjRwN(KV>+EOzeqL74NvyZ={J?H7Q z65B(N-xxps#bX(JxR0oTY-a<#$MkO|qDcM)`4^F3&r~ZZf@t$7;9W_Bs>deKYr569 z?#TZ1m{)lOmu_=T<(%%xinaO~7d&*+CAxbpK6GMCF*xjVGk+(QT&8JGtc-K%c?H+{KhMrq z&;FQod}IIAIiZ(x1L!sg2h-7%51&U;8;7RR>Kf<75MgoPMU0X&CRvIZ{E8Wuy_R=N z716&a)a^0pKPj>BYf%D+}8yqtq;wAzAb6+N_#tm6ZRhLlG44S#f#u37xsqTk?EWu z7a0?AJ@)57n}QFUW&B^>8g%L8%r?WjA)o>FP8&TQ@|6RjY{!)Gu!r920i4M!s`dGw zdS7$Esi)fRs5W&7Llj^Ax85=R7o)tiyY)DSl_Yz~{9N~YrY)|AHRB@OmP6`t+*L^C z>?$AkJE=m(AmO0&86EakRIPn6dqcO`i5fo0q`tk3P6+UC9n6@=hY<#nKk;Y;1^#4rnX-F&e!Q{ZF7a%IxVA%PCVP?1Rvn}x&F>R&xl`PpH0b-{1bnJU`FTX;77)PpQEH3DHpE6gvmnN9iiGeI=-=voE*el}BO4|B+Si9U_-=1eJGJ$M>p#?p2E`jToGMQ04UDV(m0l z`8GHSqs8}-0PQsDMd|{}Z`H;c0}b%eZTsGYWmgD%0ymdi|5l%{Lz=+e072$}=e63m zum6614I5aZGB|* z^wo`P^CL@^u>C)Hw&r^~nHL!hXT)BKeIeU?PmPFQ+2}tJPOw&B$wGTGXVNeuHX>tS zaqwZ;kfBZOb&h3?hOI16`I8vbj=fncHR6@i9D+=M2lSpynR2w5=M@8OZfX>P-%W)s zdsUS0?Z4Nttb}B0S=Mp%pJEK%wVAJSr+lS};~qJJ1lzDe}9pnCZ$sXq3oZk4{fo+Fxet#ILCmbMERdxzUx$2MCiJbu?BuRu{x_(RIZFH6EJK`t;DD@hu>Yi z+&QK*c2w&|jpEAV@}XKK)6)Up(OwLvjJJiy_!{vGnBe*YZd*09TvRd?TGe>>X-o8C z2yg3cMU_!sTMPk;pBxkrX@20T3Z)C%kmYp!GOfs2i~Q#Gi!}ImKe};FJEub6$!zoo zi381NrYd5sE?a$f^*p(dJ&|J0uSE!K$85g$#^HMJzq48D8mHZ2r_~zVl#Z4N)F>mF@u zKA`MVn~%Nwu_|@d?iN>(v#`mCYd4jQjnS%B|GR>|D|v3(28f!h??Vrf(#qG@0SW3`o=8 ztL|Ofms?ZQT;Q;g*K1@LJfRir_%@a>7%AVX67$ZUx>)z+-Z+olI#V;6SVGWc`r&bU zE44huSEEF(-uCYYd9UW4sEb@p2z=;6FOyp@NvXTs7rJXm8Gz^Hg6ZK)=LD#4!@nuf zla)#X1ZEH1G1C5X&gL$i1#(MOY$*h1kI{wl!RxQQ>jI2+-+Sq_`2ZxN3%^fb$XT1^ zB{wVRcx^^Wr2bmJ2p>Vo1_`HMPswHZC4Fh!dU|(KyOk!t-OuQnfc3UpIO(S(Gpknn zWEJF9iDg-TyB73_{o+GJLVF|SgjyBsXrD*c9mjC$>3ToMEgyw`80SlUYU_0a0S-Zd zn?p)3=MsNBJDXq_V||CV3JCSO`JT^*JQVk2%2p>RW-QT|(bv9#lx*~B6~;pGN(~ib z5p=eox2Nil%h$Q}nk@$}E7~wm13V!6dVgE5MVcY|3+dx)zl5!iH#2=I;jqv=K`aTzGy3)QC2Ic`xf4-96TJ-u}3hu3-NAXUG298!F1$YF#(v==U|_)Angw zT8w@(&z!3nP^>%(*;43LR{;kTaUigfr3JlHe!jGXeQ(;wa;mxjKk5gc`1A?A}^h1aQE?~{(K%0Mtm+9*bYdFLgYJY#81RuI{n4t zI}m`g-@8M?|Fq$x>rdotG5|l)7}+^rgbYV_t<)FiRgUf%+;W3`H}-`OgrH{uX-?Av zG@Z=SD4zoN8&IQcKXK53L7+f8^KG=MJr#5KY<0x^{Ze+?zGiQPHMXCi_H%XHnNNRL zlec}wGE{U;V)tYwfX`INnpR>Cu;UTww(Gpg$$}v2p<3m;!73{iJBZyqOYbg}6e*tZ zY^6fv^|(`7o?!JClpo%1SjPFYv6z^T6;cbc2)v_)rcIkEj#WMo?HZg<;Vd28?Gh*R(up8C zFj-`L(-?D?8Fv3}_%bJopoSf~?!`0$EN&}lm;qM#7qz7B)nSSE+6~?>yi|D@1*Goo z)~eJ+Z8Lou(w6&2^Dbb5``%1S@U#_dq^EjN#7TgqpNzkxQQT)eZ_yQr!|dH4+pEbq zsWa)(cdfdXPOOJ0wk>3Di}SIlh*_w{AXcbP4m zzIJCR?Ko**&meWDpV_tD!Q>gRxv+nK-qyLlg#RcqdBdZfBE`<9E1tT{8lcHReEvAyhx5Ev3&{`V8al)7Faz*D?4o#bV7Di}o|P zQNhrpFPo@J)GAB95u@nL%d?F(Pvdxp(@f~~8Kw3laFf%|NiWqllS+ucG-f;X$L&f{ z+AdhOo?eTBG;I%(qk+GajBd}BS2%83uBtLBpf~YU*_!q|?(4ocKOkN-=X(!j#eQim z>VMZgHUDSbj1+AH31+*8txO(#pWOJu%0IQVGNie+{$`@Tp5x7{$5AXk>zMGM4X$;K zw{5pom-eP_XW2@{HK2ahn!(*&5#?#T%1^k~u(P>77V0JspAA`vEgHz?77z^Ce*<6} zdu^mawLw2Mw@lXJLGGi2gG2b@Np#zwe=FDM+eY`;OjXBebv7L1v0Ejp01)imN(Blv zVPygZfCqOE7VbxT#)h5oSq)iddk0G~c*rwaUj23Y96O6?TPm!T!H2n;<$DWwV3qY= z$J%>ilrwH8|KKzi#H&B{MNxmqGs?1m{TTwZ-psC+NxdGOyA3f{Jq`x~ciPpE@bDX6qqt1hwD9G&PQNITa; zT)wSOYc6Sclka^6^wm>30QGJI{#zoeK-?sIx* z`u-!RRk*t@_Qm`(m+cA7n(3PB*{Nnl*DSK0E!B~$ZTwp;A#|vZ_(n!%&SJZHH{QPx zx}7XoQ}(6z_Nz*Q|L>~w|Ix?T@fgrYj&$Bh*;uGdrmrB$SV-Ts1*WD?H$p{){d}v3 z2%RpvBBRIQtsN1M{9vdFSjoV}7}(4|oHl%hns z+H3bO+5QX6+?;fdd(+!;d6c0T(0*xJ$9N?wwl}c1`}fk~b9=R`dzKLbe;#%JNb-gl z|G2+Qa&{=r{9|<|yMmhgSLYZ%S+Zs)|NQEA=7?^|sf3NmkYG#WJX*kIyJ zqe7|YU_fTFQU}`Ln(H&4dbN$`kOetLxrd@eYj@jx`yC$-2iZ90Qy6j0;YSt$`%#+A zfrJr)&J~BH6$PYXZ!aI48qDYRS;EdN-)G*+2yrGPp*EaK(S}CS7JSVT&3-^7FEb%* ze=U<9`y5s#@691fZ+bDqWG9p*nII=VGSS$ux23F}wdSZ{)kM0?RXrlrtvkSoUt10J zJ1ysX_Or_%-)60n?)tdZ#$`z6)=-Jk zx*~cQ7#2TyElNZviPLVm$g&rGhpTY1&JNNMtZ!%%#8d$+V0y*>Nw%!#Pu@9oa@^!9; z(%qAI=!vl7t|W2pX$GltI?;YGK(-fCO55q#Sa&*4D}lD9ks$H=3Bdx6N_MoD>V7^) zUcZz>RdZP(;HSWMem}rkNmW@{hY0--Nc|1Wt_B}M=Z>RgOxCd{R zK{?*|_K8U_$nni+kPo2}!Mm-<=RI_68DiQ66~wvhuY4Rec^IAt@ zev}tzq^w{VU}W<>1YVKB*P<<2 z7O=dhI{y}>6sC+j$7LW4~+1qZPXn~A8kyP!nV%v*# z3+_UU;nc6=V}s8D=q6zmEt_;{KjrJ|R_KG!(bfo2c0Er1l6F}E7L1-)r)^KYVzofCE##%y*saBRisPm z-lEG*(u6$EdyN`V+E&C9PlOsS2=3=QHRPAnEx4MDlT?Y9ec}odVX08B<1o?z zJ2mb@i`Zw(bD<fIb%9-|Q^{Hf{8LRvSZG-@m4^!cL*cZ>n)O0YE@Q77|{ zGE9(~wBCEbxYgJUpv{AK-2&WGbu!CKl+@?Fr8LPHAk1#WO~ z(?5yDF1%>7F!%qwALI~9S(9F?MPt!FzJ_)g zz=A{kHd-EXj`uglc{oXsIo>H%uTiH;#u6VnqG_7~xkn$APlL#!{Xg16b_Z_e2>y;@ zPd!KxB+H*K6c+mBj3{N@^yET;C?!wF_(Ql5i?kWjw(Fv0kG`ctF@oJO%lWl<5GdKc zW&H_Wf20i#G~BF>29ZG_M0NkVy2WT#q4t1CyU5b;A7W009(fMtZkus-dV65->Z5~l zYa>4rL8J)Pe`_{bKy_7MjM+yaM%AWKk32yZ**`Xw_dRRMv3dscpBf^{5^4bEIHcY( zV83Xw5H42S0<%6{aIqFa0hKG$>O>fCHiG{NcAR3ic2X~`r(-svdH8{XPwi|+f9u!} zFKCGDepR?*EO&gx>RkV@!Lc<$Z&Pl>99wwrCWNC;(lU_V%gXqx#(dU(z%OMr=cB=n6 zmTtb+_P9K|^w?tUGYK8L-ZoN(urn>S88UzCmYVMVp259w=QH34k806}?=JUp89(HG z-s@#7+bVQ{FIv6la}3W^`lGKtt@nSX9C+|$rY}iYLe9IwwFa{P4WeMCLm#v5{*;A0 zMP%L2JL!w=m-}y@Ec5-J1af@)5x*qp*{Xi*FdG_L;(ed;;#J22cg)Z8A9kOVdD*>;;7m%@ zn^{Z zs(B8J5+aAnr6tups^+3y8)R1yjMd;m?0xQB zYwB1T>IS1zhbA9A;%1;-_gam9|B`F31NmQ+n7vDXDyjLXIK%Vfv0Z?=kfnru6+MMf zLj~NM;dA9giGIPZy^m(U|G^%wnrwGYIk^kNYX;KUUlC3ynWr6(vQFlLF)QS-M&CFk zeGXQGIO(tFZX3;4ynpXwP1@x@Xm7q0-H4Vh1dJyb0=NSsXb53px$|QIWkZ{vq*-emUm| zb`)c>cn3du^wGEIVlHcR=v4Id!OIHy&EZRJO_zLZQ{azwD13b6J=mW<0C966_hblr ziS_wJEyL)4o78tl-jao2;(ii1drg^t=V3T(bjD0tPfF&TS*s937KORnr8`tZ@_fZ$ zVmx{7%E~e^%;>KT3FhPDMXJisBo*+7KM|KK6Vh zp@Im<7RK5>+xJ)d3;8D{m|fvO;VMgPFCB7-h0TSoV&;G&MM;!NK`?guOB-)Ir(-uz zE+Jc_2d8jfCqG@IkBz44rzDa9m=>f+sK*6d2MmJ4?@l<6*SjO zs4q`^L9%ood&6(MGjiJw&D#9)%)<0(S+vbeyy%^R_scBBC@gQ?phtZWII&4jb1k+q z(7tKg2cerTq#LWp>3>Tj>g9}a3~HSIqn63 znP7Yh9Eh^mYs)AAxwFS}B?I(qg+AM?dAjHI^|QxRq*|Od`vMaJ7Wj(}gq;RkEZvf^ z&%j6N=k3;|Wg-a@L12XQRr!ZdIHc7C?g^D%3B=#2|Z{B$a5d>keQDbYlOI z&iGXQSgp>_5Jt)$KeFPUF%+vWTdgT2=_Vr3zr1Cn1k>aC{Il1mL|S(h0ylW3DZ)g# zWG2-Q^%N#OUEq5X_pxD3qnoXcObZYIPWsuw4TF3)7-(mCg6wUw8zyz4v!Iy8`V)3x zU*5lR`*fvSh*K!#xsI4j>Zg2vu=HLaD;4^N5IyjqjJ;9I#k%f;>x9Le{9sZb>16F= zNHbkbl3kj@d@7Y1%}2DN=4NbmDK{35>lHyJL3J@LKZ-873-s$^T@x*(`$vMG1oy$m znU;0;*lxF;J3a%|LEvNya;{%!fQ$y>bprg*L~kPb*S^ zqz?6rg4H3%v)59YpI|zj5(4hDTS#(7($@H+Oe)Bv8)O3P$gkg?Py?RIHu+W+OxK7Q zt#4I680|{WKmH$FKng^!X}+t)?nO(kun|&8OWOsM5jY`IPY_3D1(gtLdJG81fby2T zX3?MV7Myj$S(UD70$MeN7n%&HTxYw_k7?c(MH4GNIWYw4Rx9K*#nz+7A+~!ED*BP+4rShQxf$`jXWzGu7s* zW{xEbK1l4V*}%7P0!zaQUeIg(_&%U>2}d7n=teUf*Y)*M+S_}za?SWzU&-}+z;V1e z{v+m?Bm-w1#`C-{AgsXi#b*FeN~Nfu+9>oZn>N=1wP>wDNj(CdiZc zq}LLDQ2gLnn52^LihGujGAm|4js>!~2(oZrSJNw}$npS9GTfT%N}!(f+mB(*?RBmI zwVd_1yUX#pqXJU%k6^ZyRoRW%7>J!~3lP@HCDSC(kW7++fawLm17@{YJ%Q+%#=DvA zj@nJBA3fJsCc8B*r>FSpIZ!VT)S?8PuYy5yuM}RQDfu5+FVJs=aRa!wpuyMpKgN25&K! z%;kDZ_Lqz3EukTFp-VhW;Ixb*e?Zx4J$-prWtASBh{zsaal*Ps67YevlD`*oq)?7p zzM{I-flBnG$FKD}&ItN*4n?~j2V)OE2g9-aU`I!FT-WUn`2Bo>gp=RAy|v(iZ!Sa4 zi2~@))i6q;7rnINr^O&2OGC1uXU-6U60#Z)JCNo+P^}_5sp@f4N5Ey;O7GL&kh+}N zrxP)YhLR3-6hErZfYo^|M()Rk7uO(z(Iy{bHuRQ3C&oAs%Z=*%!;k(9M5>Qc%h$xJ z2@?b8k)qd92}_zo%I3b;ec@1+_+S=_{g+{o^xOdckCkYD3vf~xjU(9rz)3DnAy}0* z-mFw6jATQ#@v6LENGcx{!|WROA|3rW6*R9w@V6Tbu9a3Fc$Dc%VM(AgjF=66uMe$# zS<XmBakC7A0E??{o5rD!KnEz{w!laowjMCiyo`GcTIahQZo5bEg zH{71Wj3xb=v6jle0!B5#8@;_Tqa#{J(>`&-LlCZ2J%Jihu z5L9x$sx*~ceAPCJ2twC2?n_xqm7dRNfr=k{E{87ExHUirfKm^uBbNO#OfwVA(KkYl zZf^i`rALC6=<5@#yTq8UDtm0UJnHS7hhUp^Q;FpX(&7klrF9^>TghR*Y%4@=s-mmj zr^sCys_ar75EfgMJUjbqq{dBgKJ*h4Uz%q2-VFcy>g%YOhz;RFs6c6gNT&3iu0wUP~3VDvs65Vu4<$<4t$Ba{w(F9@h} z2;1_6nk+AVmIJO$9DM)QbA92fp#!ffbEWFQDOCVBw0Y?E&^tG{!y8ygQtx9Jfl39q z4<9!Lvijv!+YoOdl20J2?RCdn0CU)9P+oN&I^Nm)o9#;4UoYr5`4?IQHc-wmZ(yxK^$eC!d>LyERbCvo#bL%lfpuHQ^uYwp_O>9xg)+bva)S6dYWuwG zx-bk3AW@)Jy9We=A3@Cwsp!d7_ze2ddawm@DC#`N_rFjF=~YP(T_~044560Fd)pq@ z#L1`-9}+J1ZfvtWeY71;^bT&BbMW%IQpk`S!g)BQVGNt|Upe<$DS2=nnBjd3oDB^0 z&?YhF!Yi;cN6z&H7G5q87loSo9TqX4X|g||;h&^aBM)9HCn4TgCvK?FLuCPC9f+13 zXb`{x2J#2WjQ}l-G^23GJO*#zXnz~!#yQwrX&q_l^F20|oFkP3{9z8ngtVpC=?hl~hw0ZGjJOs4##GSplmtKu74J*&Q=_zqFG>oc< zl~{*swd$oX8o?cH?>o8zm?4JUc~bKa-K-GIeM=S&2d_hIuNMJxVZaKiK0E^J1{vtw zEO8AOnL{)djrM8lZsxy%n7|^%4PlB~R0p~m>=-Kg(4$Ix@4M?Z5* zO~rF%x+2n}6BDG_7^lu|cEt7VJpa|UseGobRdZ<=QacqfS&R0HTh%|xuluGO_zypg)W39WE{%=?t z(U*L~4Ji~$>@zxI9tvjD2i6+odFHKhE8*u+un^+`z$kf=K3`i@5-Y(v(L4zp=ybSV zUT284@_NTF7=g&+wvL6brI;`bRBn5htwxK1A{yqvxLgF0Fc$~{E&EV4<7GhpjlBjr zbSs`t5f3_4=Kd<+twag6g5g1tDcegD`Vo56n!-dNEcjRm)i6NnI1Av4eT2$%jZ^`< zDC7Yiky)8{!K`)?o}nk^Yvy&zDU#Q3%AUCg&@1Ep;R*LM`vB7OAajPOzhCcp2`_mY z;D;)+p$lS878LwI8vy#Rug?Jsrf-`HcM26H?qYtbg^qB(U;pa#V>~<5z-qR-b?-$= z8BXU2k!KIpsQHhCO@YSS0sA@?0GH?O?OhJVvk|9sFJ`+PN5PE_#)3kJ9x1`WoN7`M zCe9gB%35ekcbK10v_pak#rZ^2nK#fQHQtN^d66Dbj>Q3xci*xojL_)F_QWXw4dQ-p zea#>7&8)F{zr}ZIkUTJBM`8IKwZ@`6+D<|w8s}NYY#+lZfBcE?t~ZMRZ^n|Wi*Swm%b*J&gf zG*k zd4Gpu;03c1Xr3K(9n+tG3JclyYXOSQtKuObDFmuc_(5;Tz+pZwm_qpZz~LjtcrB_E z!5mbVi9Px!&zddlr_pp@I(M zVVqaPZBD>8p$PSC!CjQQ&&`x!-bZd=H?VGxe}qoqA=vxCO))^^0NI4LnXM>t1<)^3 zyq~+GZO8C&W|$Mq=|iA`^U`rl@X>UmxBqDNN35N6@=dGu#|}0-hF#} zD}W;AC%$_Dfc_|5{0VDC31dIBv)Ldp=ygGZ?*`dtT=5O$Hla&8)Lm3vyj7l)nYs>3)CL~_as`#y`nC+F#NB?5bHXGhx~mpAk_{SWxZwev^S9U};DTi0fggfdE^Ra_rr@fX=U zIH>^k*YmCcN*;RCJ4*C`v29)k_Q75OWQK7U=s;^B9Je|(l$Q?BrqscOHc*kTD!({h zC2NlZF~EE%hIS?drR;N27T`KU|5Sr*DPsavwg~hCDT+tlGK|l3tLmVC+oWk@AckSb z0El;l7EZPg%-!^~s}3{dex9wTek7O%HCmK!~FGHg99zAs*Nj?I=rN5l@4e)t?YlHPO1-CFIc_h=D zb8FS9KvTBhP<1+yR&ma9BR9S25rk^j{BgIV2MzCy+&o>fpBJgxvqVN2#8dGtAoy714$Um! z!DNf&{R_L23nmkjKu4ilw4$X%9$Efl^xBnRV!b`ZB%1C8C%QaL$z8EuV|pGU2~K?m zrv^HJBr0HqJ^~@X#eRTEhB^fy@;pL~2{28oF8gP5&FH@z*Vnd#mPmIM}1xNGuXz zs#CZFtp)2`S5pYJ6zZY(&1Z5iU`Op;a^(8THu5~`E?4lRRA*~ zIw}VN7y_i`|2CFpCrM&BkenFDad!o;fr1FwDWEel0;z2TAb^pA-0dCw+IUb?O^cfr z_6p?BGC3A5}lMK}i6zG|*v&Dj|Dp&Q-8J zTQrt(aKi=<4z|bXO^e|G4|!Yxkqwsh`=LFr=|aO~F;(2AFy%_9^}t&xBZ1sfaXX0k z-hGN05M&dY}8MnYI@1IQXDqXRA83q?<+_>_zH)8Ek8^86z z@;PYuz6pvHK23nVpoOh*K4A`plyuh(C?SHp0>ttGu>rM20!LSb+c9KqCjm~z1!tXa zpy+nW;Xa(ege;)(ciQzwxfbwY5I>YX2uZDqW=O(A6^bBDe1{_51n{dz*U7JldO~)p z7hd5c3#!To#^9kJ(0%<%{&8%ANL7}{=&a`^ig@l6Y=#U}Mloi6%V@(m=kmlUQ6`v2 zvH-NXk*Pa&c|p+tD%J>>p*pOih)% z;}<|d?~jsojN!n%bf~9k;ZT3WQ0~{t84Iu2vVJKCm^0L&ATCy*T|@#dQIETp7hx#i zq=R-Yg0F#%G4iDgb&UB3E`mj9F9nm$0Wt{Cgj8J=YYtct#oU?h{{GnkLpbz8OCdK1 z=J=UY=bon!RhYJzHO|>tAhDCtQ@b%gKeC~A}E~} zApmHCyzA;bkVN$Ee0P2MS_p`A%j&+7c_heztGT1{Ji(@LPB=^h#MsyuNGBc_2ksUS zERdLj%YF!nB!mPk2x0%rWiQ{&(-iI268U0ig~mYX#2#xMzP{X0_3Z2j=S%MelE9H_ zVfCl1jf}>4P>&Jj<$jR6iz(U)9=|DrZsWBV28J+XaGyLT%K!T*^lii&I8^b)KyLR| z5%ispZKBY(XWsyk8LPO5KSG`X1=X9U>!60g^_(MvzKqrRu8=V6pyMdj|FC~t*N=6m zzRkf9gX<1ei6g{Kj3HpYMzp_jFS!~-KpyZ55$TX{c5x$Fsr=2;=X1a{E3g%pFD-PbIy@Cm7t*A0-QEBU z50c?KNt=C@?!AX1Md1dCG&?%CV2~n|NZLyS2;RI(T&>1WjMNP*Y0({CToTMDj`LAf zvoLaUGx(5?;p!bRCk2c5^BT8CbLtPwbw?7F3xql~p*e(l2&}0KqS+)o07XB&N)rtT|wqC~Ux;-y%Z7#=Ds;Xv#rr z5T4^Ga7pwh9Z33bLYP~uMaeR}V|=+9q_Cfb6r)1G5he|BWj=6k#jv%k;S z0DLBJFbe8yt1$(RujCq^#i>43@-$ti4V>?T%F<9DMvHT(xr`D2WmtIuG{`LR1;hGc zU@Te6!9#f<24GTK(79GX9n}J;K2&FKi-XrjbxP@LHD3$T+fNy5CNi$*$Pc*?udbn)Uf1w4P1!p~z7Io!yE?bQNtg!Oeih`h(M#N)g z>mDA9#I|K0wA{+!;J1Fu5b+^jvv*xh4`3M{z_VX~0TFJd+d%zo-rMd-74y>xyZU9o zGD(}+>h|crN5Qh-D9LlFnsurzNbU<{tx$CoVN%=as} zLR!QXRS+r8YFHTfvW_U!T3%J&U3915gy&JlMGYl+eMnEpLAGG}2#5gI20XI<^-d%H zPrHx=2Kwpf&lw=a^bce?P}hVWJDhNu6YXYm2c<+re#8X1o!r|BAazh&0)L}Jmxp8w zzs%{VZs#-9h?N%eERBf^cd^X>34BRqeP*oTMAhVdxcN1sCfH3Wt#X9SPMhcGm*%%W z`v@Yk8NS!!QFSTU-_ke2Gp0_+-?piEmxW_YX6FTIxJEGh^UX(xth{okDca-6oLotU zSS`PKV7P$FgXVGz-rHZs27ewT2-^=0+nEjOJ?Ywwev3;7o*i7`od4lVfU?)kK^fM& z&_|c^(tM0AEP-!kMcNXtfiGXW6Hj30K{nhfV@b}pa;_X zF7~+tQx-0%)24uIp)u83+@4OO5HHnBOs5CNggV@}jThjb6qrG-`Q!7=enxX_P&kB?BrlIl}9;1IDtl1a$XTZ(Dfh5`T8R@%Fyia>o z38v(}8w93YQ`0s-Jqqd&!f0~>Kp;rcW+v2b+w*7~jvk(9K69}gF8x}@fM|h*Muz_! zorR;9qN-W&ah5q-GzOfef1LXe#CY(1!KTQeMo-lhth^?J+nh7yQf95a37gbs)Zj`S z%Xz48w4?t^Hb27$FPCgiuUu0~_EF*P&28}@rLb)dUIF+)hk<{lLxO8SiR709#Gws@ zT5zy@yvkR(;LGVvc*`~%+*=}!5|$93OBi2Iu$CP6>q$#dsyNYf%yS}Pjc-&xoW8%2 z{zBRlrFwr;U_n@-OOmNjSOOedkP`$i#ZE5Z0dySQMTX!YhcMJB?cru1C8!CG85#G5 z%fO?ZcQI@UAo&t2T@2S0pyhw%pu^UW&#d60nJ-)eDct53oXU592VBIJyX4sQ-p0*w z>fl7__zE8Dp1aX-H>m5DS7)n4gbYgbL>fV=)4hII2^&){>)RAu>#`V zGrz@lrzfC94y|g_&G77x6p%MyoX_|EH?z(JQ3s!{dRIIRU#A)v_}7a$ZTR3Vd~^f8 zAXIs9Xkf zd1PUOcUy~JA<42&4c$l@&tuyig4`J}E-(D&_gM}2=$VA2?B2sEU*R&fo$ta!8^l6S z_QIXXISRf9OwF?jmXJ1aV=!dgKQ~@J6)Q=~mnA^ftF6x$lJ~6E-YqTf8+)^lYw=-c zuFkwDxmdMG5AL=%EH(81TixOk1f{%9$SgmZ{sVMla-PPtPlJPGzS`%$lr>^*SenkqZh7? zT3*JiGK=>oW|fBX z*k&2Gt6yk3YetFXBYW(Ti*yqI0!JjyDl!PLMUzYbtLP~8y`723J__MC_g2!8N^Cs48Z;Fsgx zLHT}BYuECBWMCd+90=qVQ9_T|&A&lYhyL=6=eFj1aubv&GqAzta)o@?;Xq@IOB;*0 zdPO4z!Xoh2WglEe^!8`0h6)+pUxD%*IGR-5UdtZU_iz;B!a@^!Ex1lA$qgV_kh8=5 z1jttN38nwaN4d=$H^VkmNRc&M&Wb6FgNl1II)h>Ag%i`!aSG(KHB|ipbkEL;}|3876om`+Q zNZJ<$6N!1IiD)c|7V-+YE8sPG1hw?)3x{C%ngu6)PWa|&TYGKr3XKFQD(34qPBiq| zo>FlbiA6a)43&r*V()|H*L?_=xqGW!YTK_bGtzn!nw^jRdsXYFP43W8ZsnQ>9~a$T z*z&pUynyoTbe9YfTkJgJKLX#EIp5zA`-48^l6!|o=xEeXoUoqqa3ISCVOCSr?(sNl zoM%Pz2~RQZ<`U13=UMJ(rRt0<`BKx9B1FZ*{x_g54DirXoPN$sGWSc9fsYkWJ?Kwc zxv|&}DMYs*Jr*?t{THaT_j=@+Kqo%F5Q;&$0H6O01ULsGUw?F5H2e2l)&`9C-v33` zcge6}ES`%bn;h+I0vS)0r`0IwE=R=y9-9_7;bZ+hB)?Ks+x<= zt%-Qk%)%xF$cOBVGZvc2ErZ*#X(?V81f%Nor0Mu|D|=}W5SUk@#g8IArUrB=6}vm< ziKL9@b&Bje@Ox&5d5-&`GJPRQ3oaZuZ1wV{dBZKp^I&qlpb&(w8*%ZMFjnuigd#|J3QC-ebv~xRjKH$4PnT2;_4DU68iaXNlor%5aGfG~)avqK_ zqbu%JNoS#?laQosaGdC!nuGBkNk;D9b%@RI!3D2>y|$5$kzXZX7^IzkQ63O?Rr+vgl2{cobR6; zJ*KexdmP161!hKyZ>PB-S<#&gD)J3&+Qx^+#N7oYXNMa zKnt!Q?DDg4gFGAsijDspZ}$xHS#F-63d(bA@cR?*@3BstTonOdC^NQ%+qR2SJ&=M* zVix`CnFoiHd`2Y;a=)6kHnC8H8etv+?#^!NyE1aG`5}J8hoO>uh77wF!2Gm<%oXjh zhSJ)6+h_tlWEHn5AZqQnUP?KOML{>1pXI{0wh&qk)g7}?s_$`nHex_bQU`ax)=Kf( z4-u1!H#6wRp&<3ZxtNKwjVtC6|qCa=Diz%-Ld z;0^pWflct^ApFYbo5w?7_8n(x6&m0MHdTot1oxko`%U$pv>xLsG1C)4D?`YLbJy%T?gq z0xX{V1`!<(gMrDexqB+QFvU|vpLuZX*c!q;b4ID-LlRXVFbO|UNn4YA=3;|X4b~=- z1=BXbF{#SYj({*#VjMn4dHDNB`FV8|U3(og?|Hw@t@xproOTW4sR`MecSnJE5A2qI z2OcRSXc4%jc&N3Tk-GihyAm4qq;@M`OZDjXk6LVpXh%Qm8PB&+}kflPBBFm_$lonJN!f2znY$ei$XeUOpWtmAzh0&rYLaRcd zRas_~j6^C#vW&(OvTrjC^PT(fe!c(l%zf@X_uO-S=XcIM4;%F5`^nN>{05#j!G7ad zRadFkAVi}eR~IzfZ7&)xc5SKIG0F7L&MhxCGN;?Yb=y2RttO_iY1hsuAo~^HwDymt zq(3#i`=5))iA-W-Z zPl34wc!@bv(5k+ZA^5sD*XRS<&^_eW7hDgcCo4j9)!;dxEtjJt%`1p;1ilU$C|rJ( z3T4qFn|@2eZRrF452J$VpVz6TBg0fP$6IAr!Nr=@c%*+kI*ftUC69Nk4tN*^+_VE~ zVVb$Qwg(S=>}S1gd!t|IllroE{RTFzhffhTwW!Tl)ppY!uGFIP`Z#o-wC4C;yI-?` z{sBHfx+>A=>-8J%$C!X2WH!Kcjl!nnv%KG)M;tVs$~h3xy9~f0 zT+8HdT7mB^xZZuD5-0*m?k`@ifO-5gR`tH$LU~=fA81!A-f-s8qHu=ZA8!B2J!aQ? z3~OzSGx9F1gn;w4`UJqLy2C@8yN0k?PsclA=cbQqJHEuqZZ9#h7@1d+bWAIy_mY4A zSoW)Iywel?i6J-> z->>kMZw@H{fx=GNI>tJwyA|B$DZzd>4{Z7|rlIWfk6c3^y3BW!HFf`1`EHByJ#eOn zUT9tF0r%pbTiX{4{ia@4i?EId9m|7ep??9D;lYuOQ`&*dq5I8>OjIBON})n-)!SKk z=@Y5OZLz%f7HpTa| zy&YopZJR#LRBMEuQ)k)cwmEid&Jt?ob?q{)Fgs#_r5EBIB5Z&aK+XR6%fg24xW(t* zDTfXQ0;3Z<6lRu2#KTyddlSY)oaViF1ISv3|yI<{{{DUGMb%boI-q zFLL%Ae2*PW&&9HI(5;*3u$3Q9*wJoxO1vv*$s|kcPOCTF7Xfr%X{BJuP?)1TUf17R zyAj%p)ZD8`!*|@rKR|!lWw%t8a4B`>a>@+Ca)cOPt@4~`o#zQR7h0s}JX!A-s}BTz zR?C;I9L47kMq-nGuFd&<8t;)mcAPo-;1=|Vh)^aRY-UKozM!)0_FK!T2hg=}*29+A z4!E=a`%&KheNezzc2j@N_=-pGANO(|PQMnB_Fmq0tm>Lu*Dlpd1ZQGDAvUFASH5tQ z2eU){m^(i}qqfn2BMz+#=w$%9D8pa&Da8(`9Q_5?(dMqdWr-bjFML$`O)lYza%EFh9uxwtIBD7LdWd8qExT|Fje_BB?PMMFY> zZbf;h;mKKI&*M|s!d%131JJisg1T1q6#t|;w2Jr%N>DY*7h1dZ#wG6a*^9a8KvONw zE$E2bTJgdpCi6c^`%lXb|B)ASANC8(CHydQnCGiWZ5;>#o|{z<;2C>*v(+wd#_r!$j)3;>@7GZxX8f!DGM$~Vwb9oD`e;h&E{YElW zCp_Vs8-G&AWxQ^#cLGG`BE2~_+vi#WSNTrA@X3Cb4MC}E7 zt@VstD?(H7qV2Z{kfa~0ss?+?>C>bOq8XxC=5F9yQ+J$mM%$`?3eUeo&H+j`5&e}6 z&UlC){bn5X-OV&^Xjg};{m>rIxd5ynTnzSoIL-QI+L!vrzp}0RANM}qyeiCtd27ep zY(q(%morRmZWzjyYj0eJIiv(7`OTUw!hZ}^!q^Sy*fZJjX8o1GcQ+7o*6{dOuUgC2 ztl9`VT(Z^iG)uISu-v%;W1VikS^lPiFe?nWMa-#f(V?I>BIloJI=KV_nULuTwk;Og zy|X%0CToaUU8v7lufm(vpfPpt%4S$WTG(gT?!9&{fi{5z>K0qOvG6q7Sc+vyr^4m# zc5K$R{BS0$gyV~v6!Ff3HbH;o0m8Wl)2#1+#fXodoA~{m_DyKs?8!V18oA=<)WM>z zL*#c*d`EW83(hjQG!AtO@1TzsG-q@4J5d4ETUzA?(4M7x^**#YT1qng%ES>y) z1;8EIMQQCiNq8NPH-78&Bw{OyzE9aLGn1bP;3vO(n{c@1 z@y>dfbI-Z%rWU=+ddHF{-~`8EobFPuFua9Z_WpD8ya1_obKw@k+c1^PNPaYBf4uO~ zAziHQGZgP;LI(+qP6P;6?z%&D9&?&3LP#S(H_&ONG{FaWeYP&3eku}I-RGQ`31Z+# zpvs4rtAYZ%Y{$k#X(evG zGALU22H~GI<42x2R@*(8eY!k^$`p+>8|d_DY`-Xxw#VhuvCpKpZ54NS2JOoJ`p%|m zf-U$lunH4=bUe8}b%J#@yPL>{wdEypa!`F88><%F$o;peHt8 zhH<0uQCZ4H^0&h|WA9C}I_f`>lwRLav)eCBZ??)-BmN6m+5Nc3@_z->boCJ3l*s3D?J> zF7%~k{Go|CAJV^a+IYsf(UEWSFwCZ+`+k{k?AZt+wc|RwwB>8Iia9ksDd*jt`u&x} z;c{+#S9iR>vG?5a8GJWr5Hiuw(l&ePWmQ=7>&zeBjt^oSPk%}Ekz4nt{|E#A5HeD9 zYDlf+n)Nm(F&l=WL4I^Jf8h3m3?LWPlQ`wB@6Wtb1ZzQE*Kj{$Z-{7>8oQ!~>p)3) zai+YB7UU`RaJ4Jdq`g8Ox6g+Wd7)?Xs0q)2ihs`>6rXzk>!0GwE}RF7uJp0Zsngn ztpqxup@MY6xnw-Lobp`w)lCB{Y6e{>oz#W-Ctl^eUiLs^o0Eij&IMaVm1EEAQFkdW zA-y&)`;3V5Tu=e+yG9LTH_hjiG=;(5dL{Ix{=8l0>ql#kcvQ##HjnkOPC+K-1$=P8 zttD>~oSBBi2`nuzcz^O+TjtoxwO;V1r-B zT(vzpRI=`2zjD0F0;>zHG||e=g&v8;ePm|4XRfAAB)m(*Bhp_bCziQ;^cJ@Cb3_og zxk%&ue zcV72h5pCWbL~NAUg4MeInl-0T3O2pMEO@+IW-N7_JXk;QKR^KsoS)a{`Rxj>isTrKOE*x8#<0wl{yZ$u zw0Qv%5|;4n(2w4O5=d%*!dT53JKMj^ykp4ycaHT~a&}&s??}-q+oxoTBJ=dB-=gR8 zL)Jx9M8p18mcwr_q&S=4+(*ZpN(ssXlQiNE8+`ruXK9u0kW!u5$%^o3`Ud^M`kV(^ zB6~)9pmY&F=;8UWO?7?cGT{#?23!7e_UDO3+e2K1Trfa2A{$8)h3KFBu@ zy-)**ZfSRzuwHcc%Xu3cTlR;k8~{mfU6wb<@wyu!dATS<@tjM`@Vjgw|#k_%FW^sL5GQHtGbj;?F9R|V)?|0t;>?mi#aR(_gDN=bt7t} z$M??+_VWWN{6O|G&R~ge_SjdRT{0OYv7)2vi(k1wiFHy8v_TGRDj=LBqd2?3MJ_7k zYUYQ1y;=rcO{Ar4IwsgXzXtD(M`IOU^2=5r!>;Ig-`=EYbDpDYaIBx)$uQU&qX&a$ zC8EzMGN)Ir4h^D&=nCi4z201$Gj|cJJF-wJqvHqHe^l{J?x#6hss5vVLG2NQY%shV zb65nk1JR@vThO={jTvcFzu!4VsaHrf+${wuMDJ5;p)2eGAvZ*Dd}2QRCz=BOsGr66 zK-r^JiPFI%zOck9(68wFYY#F+pL*p}2**5^+kBib=k6kSENL`44H}XeNz)o^gs`YH zzJ@Q6V6-s8utk4_?8qR8l5_>@%8BLrKle}6BJt?R7}9dzJQM0fI}M$l#u?}?^gb98 zEKypD4vyRpu0vk*#>V=+qYE_~&4<|EstFnlr+SIvfjd3@)!61Np&TSKRYtqtuq9j3 zu16kyUf)7=59v*|Um4AjQ)0XTQ}5j$s-OcaJAqDjt@xK`K@~M;k)Gi4VtQ&w%EBLW!pdW%uYpa_z?G`j4cg z-u`m+yK{`*^&Sgq)*ti^66pSNozOLv`(@DnzJOO1G}`Bt+xpQ-G~F^JL|)~dPndx`gMXj&0#0-CLmw=hyQ^MXEWLnzr>A>mu-upVa7vvIdbRO-)M`@m z)JD6jK<;8*xpX1x_M+q6KgCF4$m0xf@p16Id&3p%MZ#ZTh; z{W>DeAt3nq$(760!Zp!5pvBw1S~k7XbHY zd&DXX>c)Q`G`4vU$9JL0>0qq1X?;SG_+jV!ni%U;e>^kPwO*}<9p)7iH2u>m+qlnQ z_9Oe!!c&L?$F|VFjZnHp?~g~b=JXFla3t3t7yj@XE%T!%7AX3_ksqa9(B~hL9nP=2 z7{AD*%c|bp56sugE?Ql*)61)vVnou117>CrZ&(zou{-{?LqW6(B5ZT-$B9Ai8xX_h zF(Pq3b9w>?JVij2l(r~Rj}737fY*_bJz1p|;634we5~Rju^6sv!iv=)3QH4VR#I@j zRtYU+N?Ay7?bb0iuYA8VUe&<3)E0dQ?0OdsEC}zGu&wC~pA5Al z+F77)=38r3w`a(1Y&708y70Kdx|nGQ$bQSiURc|1 zr356W9ggNo>HXYq(sZ<0Dg7)g{RupN#MV4{Whki>M`7)G@+xVu>{DPSbAwiIrH1)r z6Om1-|8Tt`8f5#R2X?i5caz+WRb@-OW9!IiXR zIF*eNqwb1tB{FYu!E|RjW`ySlz8u_ax*VB%r6~w)Tw|AW#P3I+-p_hsQQUg(ljykL zv;Ac%&D|v{dOv>Joe-__`JX5eqHDAP?>MXRqwEW^JosfAlyo(%B>&T^e)W3G+v>)ymo=w6AJbzLh<2Jd#faypHEccDDz^FF6si8`SxJdP zx?~MDVvrEC6kuGScF52KSmA*5wc1+p^3J#gECmxk-XtBK+Hy@V8f^mLQWe>dJ)?&7 z4khOFogXI59SXr|7?0GvM^pT6@_k9;?pqX^a?;4X6Z!mitNR3fIwnGm?h9i_V`q^j z@?Nk(&TpC$jny@})|Gr!b^g;Z$GWTh&K6x89T ztcNu>{MGX-BZv2G3M%zHd|SEggA~nrK=lU2Ukum6O@7WU#Nmt|xsIZmKZU&d0_o?h zcD3vPmDa^%k<9bOykG(S2BkGWe&UGdVXs2v>CC{&)Wd7bL4E6$169u4+_%qQJkE(p zn9~KY$ix(^-s@NmyZQlEK`yN9c*B{#ldZ+P#r(3g>W1Q95C3?jClVd4X%nCCog=g& zxYy~H0wSFqGI4HxuIgy_4}ii?z`i^xLAq?4G-vee_X*4N>h9z=H+bhyUZ6!@cdfkd zw~vECCDrw#tZ&09#Hdda$LoZ=FFZTm!ZGCo1c=&n>#O*-izzlOefvFK{^Dth}{paH+?WhbQ37eFS^Y1YqMoRYe#eJ(n~N1jG|qSL!eo- zR^;xMFMzaGK8R_Ym23qVuYNyZ1lpPvLS^jH2|-8rot~4CVvisOR%3P?k==7(zVwPs zFp)~zsGRHLrp+%%wC6BhO?{8?3cEm#Wg8dN2-d7-o@_RjSI;z~mpX~gDVs2#-{j?(?4U z0Lw00zcfO&5>}j8#BikRz99n;?_OkN^MhAqbf|d3{J3v<;L1#u0y+AQcb{a(atf6f z7+w{2Goy1ZJ|3}b5s03cC|!B6+tii2Yh;WVT#g+NxP1dzr^;r8MIll+dXpn*Dn%*I zb^4KC6#w?ZHHO)0z2gW} zczg#7ZkCtzDR+m+l_f&)N7d-aIl-hCzZH#Ir^37?dJrMn-2cZLyKni-L!kqG^MxoH z$xsg<)OeG`U~0?wK3O=%B|SHU!R?*c(!zisglzpuExQv!UaSsXTvIPO4Gvt~?Hb;0 zUqdvE6O3t&)_W2G3VH67M#9 z_p50jYg*pZ1&$<3aQN-ICc?xqn#)q=`3e6+I8eCqtL=v%XsF5g?gubfV??E&ZnDc_ z3IK>6Ch0X?$PO^#ZhPS9$ay3Cs%|1ZeqCb{9nfb*{3xc-MKe-icIc|7_!|$_ zFPB{qvz{vEJlBQ$BzOMkXVufvKL9Me%C30rIHjz|k%NmD(*kaVehC>az|dB`NZHiZ zFAWDiskyM|p3%wIr`LfqQ)BZFws&$C1*kk5-(!BVM9z695lE6!t~B9zfTjQ3X>w$(KmL@Tltz6 zN$qvqU1w?m@vt*mohph(p5%rWNw2X<#(`jZbPYL`*dNjEMcX!gD2)Tx2<0LN9?LOe z+22BXJbdUsnk!nYxg;i|F)768wYAF9{YKWoywVAFcu~gF9r#fhOxdq3-w7=ocL<7!ho9FQN5`+k9B#nEU;d%(d*Jaq#}7_-?;$X2_vDl{T=n;@ z6NZHNJxe>zUt0`ef{QfBpV3p~LWpSzVhX>OQB7OMh*EH%dHOJAuBCg}CD_%3P3{R*qJGgC$V~M5{?0kF z&8ct?d1!{sN{}los5Hs4-r;<_!JXBx9&>r+BW<7a#bF~T0@J-OVZz<3g5{QvnOrkZ zYt;|N9#w1>D^7B4%mQgd>7Q@SN*ktp9tpnpTuZS7fHieamY3WHGf{92t9&ooWCWUh zj5-1)Lf@}pMjw`@%zWD@jB9idf7G6LSuk2T!EVntH|iglQweMf=X+t%=9YO4eChf2 zB@J)w`~gnHgETE+Fes)3-c16#Mh4vMv&x3ct9p+eTg~n>Dg0=rf7kQ@D3Qlz_kDF! zHYH9V@x9Tw$nKM8oC(;)@34y~mftsOh{aVR%*MYz!}LUjfeZ=^{MUHYhADu-$kxfq zk|c)b=Ri!M{REMR_tv>~hCE*zZRc~OyLgj1y)}Uy4l4)zR{m?yb)vK}IMOz1cy<7Ss}33;Oj*qr)#O z`qiAXun~l8QKEdbljnC_wo0r&em0OL!f*a|^x;3&(_50Zn>ZSbA4P0T9KHX5(0fCA z1=)!3qa-F?AQYAxdA=V7x4_zUF>3QOc2*S4kG2B6U5$Owpk0!s&^9K2d1K?7c8Nzm zz?JZ-AQC7oFUv4Ce}5XzZqSt)hja*!)mFe9+(1yTx%W={b$KOxKb#YIYsct;bDl*Q z)ey8L^~q7;y)TkcAS&L9tgNF`b}8-#@<41cC7Mx|;^v|JJ90a*x(4=H?51bK8i=C% zjBlx^5>sL(4bI|ETvr-=s>A!PT2BlOt|>!}p+hv-egwCL4pU(nxyt+=4HDe9PJ2$36S}?k!9kp7gw% zw*B!D;}qMHA;P^KB6hq%m@RwR?{JXJ_$krwVoBI5!DjdLOWUKQfz4A$FiX5Cmxe`> zLv$sI1&0kyqvXB2*%NF}GnIn3G={)B^3i9eI1jJ@M?Ndd5HNS(^GW>}F)G@$LrZ8s z?-_0|9p|ofoK6SgqJkF1`yJ(zJZyWpvC!Ovz-RW+-g;9P_V>n&(72uB%Zg^`AHba8 zyr}DaZ=WLK4)f3S>PLH5^~YaBAhpPPU29tjaHlh{G-m-sh4WzAQQpt8_Jq2U;6xCI z$@VK7rv(9nbo6$wBx}=W*`kcrub}`Ouri3p=P%K-x1_ok!X9>>vL8}YDzF!ClA~W# z#rZZYSxM+PDuOwjhZc{;p7u%-yI()|7rlB;8aSgbMi4M&{3`#zFvp5c%_Z{No<#2& zrl@sqhQzOGfn4%>7t+oa;QzxPXoX%g8suGH+%Q;{65d$!v=VqK09yfVBrLKio;T6I zHbi{PO_2Ig0%{Qf$$wGI^3sjmd7>DQ*nY{!4f_Lcm(1(yTLCo25aQD5b|pD#-gJS1 zFN!#@o#j&I^z#>@(p(nV?(JWd3~2>PtCMq6?3Ov?HpNpKnlm5-3t8Lne*A(FrdV)W zE7GaHhgQ0NHZ+zFC7F^6i>msTY}(teYyAM;v89tM)QFrE;r`eYcG-6*h7Fx zz`Lm(g)Z9=5wA6M-}k`0A?Rp#V=oGZH4sB+9Nlyp45H`L$>hjmkF`cd<43YyO#3Zp8g&GOx0JYHh$jD$~bUjiB+J+3Bu&%cMI8g}IRj??X48MRmv>F$}$E-8o%Q-(NEV3Ig+mPeAL<(Ty*MDF)VH->!V5ZC6 z?(%d8d_d$LnFcf(9SBnR1gqZD1{!2x=6b2E3p$ydY-OxL-mHoDbLJ^+l$Je~^$tS)E%oOS*UCQfe5^=r@fe;p8GPr*Nn$;Zz z{wY1IrY2Hwm<4_@bhGvP@z%bKT7cSWKsZ&BKx1iRC<(9_E#ewz_;PhUy_TsHaRc!7 zL}t&E(|)cg#JXF+ah7s^0e6)`B+%Vp!Pn)L;|J8udcHmaic=3hhmP>haOUb;ip*Qq zrXfd~S7Y*g|HY?7gui+nP5N9_6XOS_|#0iuHM zAj}GUZK9G}M4ovWN%Q>p$PGbW>m*MOG&=0xBiM$uDaw2Z5_dHLbD0C{4Z4U#Y7nC9 z5lX-`m%#K*-Q5%Sdj@p}H08-%s0wT@(}gJ1e=e{g87R?YF;7lC3O*mZ6@-5dgx_|G z}7*3BG@#5EYp|h>GilK(4YMbD>tSzUo+#! zkcLV;6A`_`*$IgoB0B`=Qi9^orf$RvC!2zF?i89q)CR_udxf*ST%M-r=8%`uU328a z7;yxcplVF}M#S4b6n&m=jLaJ*tzwXzY_iF4>AbXMk?@-#{ z6TPkmfc#I5%3(zi$|PUEb48}L(Q!NPRHb|t2=DNG<}KRXPdc+P z@>Pyxrgqkqv)d^Fo~{jzlCJ|~0>C4G+-U#uH3UyUawk6(83#mwpFE<}lC zUC~Lj4)Ok$a+?3}Y>>GzbrG@vKx*R*7fT_O#1bhq3#}=3C|~BeV{2m|Ic*p;Nzf)4 zxh_OEi7r?LAC{EHY$Fl9^`r<(4uD^j;=hP1U(Syj98vq0+M=I|j(%o;wG%jJT&WC9 zCgx|fTRg#%|A8Mw;y;*EzH*rSePyEpt-QWZdyH50T=FtxX%z!;BCCypIe@^Om$;DD zyB|z#iwk=9wV3r5#61IS;z$O}K>sAhQvA&2w$%s$HeqRCg-b!`7osfqMgy7;2ET3G z_Y#m^VE*7{j24yJkQ8!ijRfXuXfz(TaS>c#;7DrgWL;6$TRr3*Wy~1*0IAE|M|2-&mog8b-Tyf6bK9`FVzJ( zU&Q771H1{i&#ds={>IWULU4O8c=o=0Fgfa6G>55tP#!=|HQg(GLf^NxGMlwZN) zY&7;o;GRb)j{f%vOE1EFSEKnptBhkd8V}m#fOd5QcN3WprrAXJ9rVgk|Yu^y5hx} zEwhw&R{|L{CQRhHK0-dM0e~ax528#lMQ_-kD*;{a?g)hANBZRGSzszivy`P=ND)(f z>qf}@2FL*JvSpkF$|+!>fYjHUQ8aVy(%o6{{*c!KDNwD6sxPwF_ca>qL6(;4x`@XEb4Fa)mwNknz2Sc(|MF zS#d`N#{o;;j=Qztz6V)dKJ9t0=rNR zyAbl|t|9m%*+7*vzjD=P)&}mb@`a|Y83fLk%?YQ1+71w$t*K(X>G%dbDD!@w0y)kMq@r71BGe!|N!VjgBmcm8Dor9B{h0=o0HkfO z6=>L*xPqR~@@JyhXhn4D=;B|lA&w$T+jR#H04E@qY% zU;HTeEH+ZbB=NEB{>GHgjcF|yKMkU~i?9z4c@Z(+Nc!&xFF@O8DZg2y_w0N8GDK%M zgux?Ozs#Zu;{RPeapy+54-}wO z$)hEefywZ+-D-V{rXF_Tzuj8KK);kktvTf)r+Rk_I09OH${js1WQSac+;Bhu-owHZqA} zS8*ymiAmV1$~--n@NLAF{?i+1G@u%seUQ>iCeau{@J+>rvm2YhU|)Luqr2Nl383dD zkzSu3V7H96V4V=-rQx#V^$#3WoAG)8+kSJrosF{58HBB1Z7JhNL6>4R_SQWwS9T7D z!FBP$t0}uCn%3WSL5aH{rZMhx_MHY~n3yZV0%C`yiIiW?V>p4|@NtHHa}Z7BTf2Gk zPOxMG%Z4B0#jYZmt3pXO#^}(!D@)RMLIq7M2Ry^)O3a zsB%i!F$$i>`N6*rT}2w)#B19gsB)|x5k~_@Ks$h)LS$YB;d}3%KZ#kXSSKiPl^ojf zgyDUoc(H=qgieSLQ>ep~61N#eP%ha%8eCKc6LTQI+b^Mk4rwy3Cn+GxNpqV0i z4o(|k_9G}%z?OpzufiQq_>E=g*jL^0JzBwUeas3&2yK*UwI#Gerww+A;-lOB3ATTa#BoZK>2?u30 zO`0kXMtm!1lN>(PPeu}La%Nh_Jex9DkGl6|v87N~l|zH9q=}qVuL0kuj3g^<3iia= z3l~I7V)%ga83+<=qE(2rDE)SHR%lPM@UudN0uu+R0y&2md=dwMDC_Y$HHvir)VKLr zP7f!1n=n5LIn2!2p`=Vjr2ZQSf4su%3{Y(npnIFukfuf06lTa;h|lmGxR7$l1p{lw zS~OL%0NV5%ygluNh-msxEpTm)V7|$>p`VWxGT$~0?V*jp#5>>UVFD0@cm$OP+4oE( zOE4=iQO|M1Jf~RA{Ai${1LDUurqb}}u}7X=v@kow`lmrr3B5v!!E&FR2c(LS!Mg$D z>P!Y($A2P3c8e+A(GOr-AHp^)1vYCpzFE`3mQFGC^j}~G@k#`o(>>Ggv#~J`0t9@@ z7L!wE4-6qlWs?r^B(f`iCgIl|i|E{^7*g3dT&8z`eC*aAyG&QkMyx$8L*mgS^17qw zXSsdyK?tX2Dc@O$gm29cfh#KhBQmw6v|4z{d>3AN~B*6%~ahpN1NS z^4NSudM6aTMDPy%&I#&JqyzhU<|_g|a+(LDN!nlCLRC}&TSFER;8ugv=xOH%jX zmD|R@!k*!5EN0F*i0i2WL>sjGG2NgxrUT#tjtlra{Iv82z)=k377!d{3*^W+W|{;tzJbZ3{F&)|d4a*Mj9Q{_l1%51f@$Swy%5TaDby@w;Lx zOI$06fVr{AZB#J^?p+*4>UjuNVXr5di-8Qn@gK;!#8EcRn5x3UocD#2sBZYouY+fW z;WAMtcFDi)uX5 zD}O>k4nhl&fEGL~(24kR0ce=D4+bls=1ThvsF;D7NaZtO-wH5zPCtcPqSTavWiqW` z4kUaI!--=;Ko;0x|6p%CY_%~OBR;lCT;1!8wi)6d>Pmc@&mv%zeN8PHs#2_+F$I7% zla6LODnHD9MW8FpRymV_AGWoS>@&7rHl8`?gRN*@C~_ZQ!;>k-AoD(C2kHmRu}sHM zQtxJb66clDruY6pq}9!{i%d_95iK*ErL6Qt(i1q6LAZ}2xWdIzp_L^*(zGoP zham~{`TfjH4;V-m4J0<)IIll``#M<()~@|{4t^V<@|CdfPG6b@C$T32>jZlMj5Uar z(=l?s5lU#+K*_d1GxI$7;OEu!gkLyy5H|TB?hJcn;dn9s>HW4*l==cd0}w}NZq*2j zr4WZzPLlj0ULeQYghbCeVgQMP`wYyk;$ue{L z#{`H+4aX&SPGPP$B}?t-MUo7_)5!l)D|(cC4Ma-yLx$de@q-OSj%9-yae{S^@MUg+ z9I(o01$vF^{+twx6TYl%f)fvN0YXNq7>Hkfb$Zf>A~t_9dBZ)>Wuh3}HJ>K9v7?0h z94=JCUX*2!@M5fN6X;zn;lUsry~dH0#bF z8j10Ad6B9gij`c)={}{$fX6|4x z`D{qi$9r}I7woVO9{}>TJ?r?4{t-0~x}71OS%x_Ik(Ussx8VhgXtMsT zW^*tTb846ZE@bVIkt9?*Ve|(}J2I(#5Kq{P)J~~$r{m(kTZ58TV4~e>@hJYy zt1rA--cF3MfV`mvm(CohByY(npzUdj*TE7zXrp~~@jp$Y>|Y4C9~I*q3?=QvEt09U zUntY_sF(uw8m;vqqRNglt~{XhOeE|$-S$uxD^e$Kcz_RCIT+jt4WK@Ls}?a;_&lPa z4vD9oK^67mB8Y~KWvY&@GB?K*&v7It1!>di$c{Bj7oj#QHxmJOK@GF!4B#`8)P;5j z#hkao9L&(sH0|fyPNTkU#CumN*B%xD+qd+mKyq->AT@^D2S{Es-ZlZ5pILyxN z6i2JCH3vw043+C?;sKX)S5u$e2)J z@eD_D8Xkas>1VKme)YMxyAhm+@USu?hk(&Zk_?-rxL%h<*hWW_Od32V2n1)%w!Q4DWQp4WiPMLGKQNR3GB~&Vjuy$sY z%R&JdQ0bV2m^q?fXDJgHLdVl{xx0w)XI2>67C6>P1T6U&%zgusFUDm!<F?2q8jg$9ZWewSwVhsT?rFhjVBg83DA(djgES#Z)W@ zUt%Yb-&x?=nRya1R0F&!;8o==}|wSE6_sw6GBQg780EQBb*=K?fmhImJc{iF~hyq4mCdts2 z!#9{3UoG3X529K~b1cIHZ76Ps@_Kl^2>yEK(s6_(wa&qjmQbxnP)G+SHx;` zBGpqapqmofuTZeVivyK2Jp&IACLb8eh)05eG!UEv0yZ6lILQ!T>^z3yBpLm10IF6| zaEGk`OP?{h+(?v6K(`8j5Cc_MNPagDmoWAa`g8oNF6l`vP<08j!{R_tNehry!QyED zdJqxqDqa7K${NXQ6uw7+qe9LB*+9%%&Sf;#C4& zEYbjv;*r7eSWgX`l*uDc$VJYCj7S|SX~0<0k_>gdlmEDj(MRFl5uV<}Ar#F6zPdfV z;7Y5(53~o)x5tRyboB2mh2C= zpn#{9EpeZxB2;-xBTZNZ33`r9qyU%^MwIiTVrGoF(@P-MmW7`yjs^T?FpMd%QaUGMwY){8qo6P?> z9gDG|-Unm`{6J^Y@}#2p-($9s-gQ-7(sPoByinWZJuw4@!)=$%NX`SPG$Eb z=M$TxF(7q&ydiF}Fsi$!;nj2TM*+xaHBaU&rJ^y@D`MvMI2qZ?Z)5kkI+a#xD0z0C zsLNYvB;r;h%#MzBjeICh&of%qe6zX~GhGJHZ$swM#`2IfBzrB<)^D;(F z)%SYda4vi{=5Uo$tclBbVxlh2FLKkPV?aa%3M`K)HsU_w4{1>h!Ly}mgWWkk5 z>Sv#cRx68V+mdo@)vwQ8K@$3`4zzQUIL}U!!d9%O9<;;vC$>ByAgCj4dQnP446V@= zI9?{)HF673lW^Vx*yi&d5zHBYc^Xn?$?PDjI=hh6KDQLr0S~s(afRdu00Y(zzg`~f zA>mw?6ybfUv=0iP=fV$g^~^R>c%DYx#XX^yCfB=s-B8ggb=qJ*b4n%owA8g@-myKGaZ%YWaAdM5z64RP+$Lyb5Fe7Bf!; z5eaO>T(~C?$cbZDX&_57OWCt?Ary7B$zl(|hfsjO zeRS=yI>tH*PvD`%Eam7!NhDS-DrsMYwFIsmtcR)+_^^8tXG<2o+b-P4k@nLZT3-sF za!KHZl?ZOGv!3>`Vm*d2mXLQWLRK=ghAe*1j!;tbe`CA!?AKI`^$}i!i)>jOd$9Hj#TJ@5}afa9)N$|)SnIrL{E6u2LUwYG>GW1K~yqz||$y%1&Fw`97h zF=ZYe2M}7D(7%s{k~aVQ?*aJtA)K979>nPPq!=r#CMEf~Abu&<`2#n&KuS0pd3+H? zS*vF_eu8Cn>Hg&cBZvtix2N`Es*1ISCMBF?-LL#$cJbDZ} z283}J1bkTCEl%+yf#t%Tz3O~^PNx>p)b`)}c+hQN%2`jO7?XsRy56df`+s86Zvg<( z#Npl)1d&YzaQCyLQGdeeo#d3Tzx6-LlSM4kz$MjuKDP|++Q|YF!;5cQw%AP)qbtj+ z+>Am#alQMBf@}Wr)9B|R5q5wkS^`fRAkBQSf+7o-q2sQkmf{FaUGGNo`h-&idgqj9 zf6?3kD%a+Z(cKK1ikl~2?0T#}o()x9NeZ6Nm{DLp&&)M$h)x#AvKgG>`%}+c%-;`1 zOOL`45#2B42^SZ#cPnfBrEGwgDJuJ z6vWnSwgAAIO^*4-#$d?qV)TBc$0fg~-X-0CaZVr<5g$h>8ZNTQHWg~-=#@7(#gC>Q zT8Xaekg#i)D%SJ4mV(k#5LQE$rWv#V!JXCfC$-bzn&6=pR=ULZPM0X$ZNiOkds%Su zB7T4r#@XXbUX8O)p*0Ti|0bnbAkHyf93{s<+~HPU8^(H>KyAipC$X$7GodFGg3z`8 z(1N@(g?-wlBBf;w-Y`-qNdxyzfQwl#KoKZ3RTR0NF{lB{vu_)<`6%ubv%qwC3b{Ev z@PTm=_0M2*G4n|#X_m4Z+)9MOU7-}rDViA-{eC>U@xw(Z1~{YRzY1DiEGD1CNhGl> zFKI}OfwBGdf7MN3O-*BxK?1K$H~jaqlI6|mBKRDNCWXhnC1^w!wP*C;_mb{O^JYU% z4^{R+B_*z4`{D(y%dxd>Q2GOG3C4<&V(i1+kGkH|p*Palts?-nF)BcWN%H4Hlp7RR zu(~FTq%bUdE44Y|-B_9y(ffAZ)Q*U|>d2CeairM5qe)gVg0|8E1 zdd0sim+23XK5=TOT7!B?6m|rA&r*lN5eR8IhH0qM-^Qfi3Uf;SD|zK;angTif+csM zjNT^0$X*HKq=}hNOc|WLeaB=KFzwp_ugd2UWrryZz+FJ~(bA-kU?E{cO#U6JuSg0j z1*%Ihu20J?11?R{jtb3Qv%fQ+3X?K@C$}_fF@&LkFMuM!guEYvI1g!aK4)X&MX&<9 zq!>{+R=N=NBmjz9Qm!drrZ=JPYzhvv`arSmepy3|s!Fm`p0dn)2M>TEU>*NHL6s3TwaO=>QL7&SZ>f57v`pXX9@sO}+V30>H;_1(O+AV$%R*R!54?iehzPstoBVyZt0E zZ*Urh9fD)vQP>R!s3?KD4`*ybYibNrKzo1UB?@Iu8{JR^Kqivz5cQqcBuV2!0;Ukt zUkkY!CC#3okTJNaQIzrR5)obAOkQP0ev0$ykXm}+x3SSj4P4{>q}N@v`_o1i=EF=b zaI!Z{Zr+s`_;`S9?ryk}#~Tdq=t!+mOdU#&J1bNk9l@ZQp^gElZ=SzhJEGAPYPOfs&u3KPTYR3gE);fhxzmYd1sg&vlZFaUsP*?O;QI zDMr5qRSRPf1=R|R>k#Yw{k;sPvXKh)kNA0RzN-HZMn~5!ZU)0Du9&N0Tk%?vrF*k zrqFCvz*#Y10mh>fIH+37C=_Eo7Gq!f_rDjT;D7`PIlV|>o%)@B=XlG?VpPq7tSSF+ z1tL2YVeRVZhEHRx;iQTJa(={eBER|UZ7a-&nM73tv~#e$^BCs5Hq>?6q_&Y_a}7lF zX)6Qfdaj}=d+IeBT!?(_rIukn0SV&EOtsNf5CUns2uTZQL>lKSh7~O&&z*t@5c!04 zI#En4mj%ze(-)WRZ);#QaTWNfIg3oIpwRsAX#RSoTsDS9T2WVkBzwt)7Te>Ro;o zSqTom*DxB^UhG$Gykrr(2UpC`GNRLnsapn*X|vOg)BMB+ddAju7s*k zG4>>fgCy=MRzmseui7N%JdTQ@R>q?z?u)TQ|CKsCvC3(=3Bxw3acAzDjxs2di_M>f zJv<6Q_qA;L1s{oF0W0~Re?@Y&5Y^Cml{L9#rCjG0t zYx0o!{|RlJdGow!^cG=y#)3b1cNDW z7kY1PzbzkynYN3u(FNF~9b2v2c%IuAX#mi~S6*1OH?Y~#9$NyttEk>H@6;{Sa&RyA z1A)Frl7VjX8mw*QY>7uv2bAr6Gs^1wU2QvTAd(4*Z~h`~PRzFquUlVbT(M-sP;OZ6 zol2uD$WJ5jl)JsZ)~xT8(<+=z$b|4o(03xp#7ClebFLFLamT3jpA~Qc0DE+cLtb!i zXZ(`Ci@vA5M}fX0KZPdW-=1@3BmH(F-k8T2jYI3IH$s7#>Tk zAvd_C1KQ9P&I76wn?6X{=ES;`w{KL$A*=?Syp7L_w7Z)D8ifC#Do2b>1}1gX`;|a;2pV%ygEfWe$n5l zUxQi}^mc9-m-l%I&v;qv-*{ZDr^FL1lefBwh#`#&-Eklf!zEUdps0SJdVuMg!)>ixW3lWp>!m#u}o zApGw@D3{zFIXGq`{+}TeQ410*Q@g-R1wg=mWM|(*L)ZYrj^5^wWi(W(A#<3898Q_M zavJJ@0}HdDs)TbKD1V1bzXV-x04K|kb+@zmyo}$ z{yPZB93wM0ThjiD8$HGG4<)VqyK!dlCwc{JO}1kD$br*;2SpR08p3))T~+^%;H8^8 zR|0+k4N_6(B8lWlz|3N|;NRp#zS^f8SN1(03q-4v(;eQN` z|BUGeH30wLE~8Lx|KH_hNim_e9)3yZkZ;}TS+$A1`QpC^3`!)dwjPV!G5U}%;aZQQ zT>QCLPJ#z}{(p|=W%UJbZ4VicEgHSK8dMvbroF>wpQri6cy{97BcayZn>XRE)v9Pj!TNq^1;s!Ec+N$Y~YbGJPG9Klx%qs>-HfqqOn1wEg@9saMYtB-n>sQ@FZg+FI(e&KqgtktYHhM2BWaw-#F~yHW7MI9^oz1Y@=`Hd=B?T| zKgP;y)>tQJmdVh{gw*eQ-}iIQbDpO^d_JD%dhYwWulu_1`?{~|dtFDr=O zdS(5=EvbVx?klhNmL>n*CJZxcUGuv|RJsb4F5}*nY`f)MUc-$hCmkmLnZM%+qc2P} zyTIs$K6$tu>SgHP=VO;CZS?&>*abiCZYbN?vDD6A80g8r9jn&kHJ_nwi z+=yhy8&@aAgj7C6_??H~k2K~t7a~mcz?`|~v9-Z}jm8eA zYoq?zcr(@c>3gNm-`Zyf8m&2di+{aSj^w(<{tgR+^Suw-64~_{1geKM{FNIrwktPm zwJzMHlR`8NBPbuMiW^GerS``^N=UJGXc|XsS1Ov@WNHCrxi!G4E%)Z}aI?KuXK8Rk z_&5wXVH2GSke4`Hdm~Pr(cJ3#a>K6fUqemFeFXLc3G&11u*pI_9JOs8AISOpsAfa| zJ*eE!?|x;+^sYo7m(*vC^PSQel598!H&ScWHAlyCcD55dYdqeuFf9VY!fm=Fz5(A1@mME6nFr=5E!P!Ge}{= zLR><7a{-O}_g5r600ROd3qx@Lc;ya_gc-r>f%-Ixw?qO( z;-+8E*%vwH@MtLZ;enAp+LLEiI z8#Ih70!YnOEI0d10GH_$3mxD-S5+#W7<=Fm8NOK{dyjyF#bk&Rm>;HyP$M$Znz++k zfSwLhY^Ol>p&n{1%xNxo3#Yq?An?OLu_lPrjavYvxhWPKqCHk660w~JPL#wjjMgpg zJ|?Bqk71a;*d$>tORZoG!W)0rrfX+NG_+l zgtd8?(Ihcz*1)l|h=GB%LYPpYk{OKvYtCZfmVh-D6);H6Hv*8?nPRDxg0;=mB_^;2 zgAP{%<3+Db!0bH|vlb547KnHM!qvb^I*PUV5XuA|r~WTP2j#%pjizodK_?FZKfO&X zB3x%&Mh}p*2DBgy17k@N5XI;T5;82IB1vn6H|if3I-LQr%@m6*9%9R3-K2d6h@S0$08X zKwyNJHB_83IZJ&Ch7oWDK)e@7J{^U@r-?;b^#N92qgWtXOx9C7v3TzQ!JaB!FiJ-x zfmdW96l~bVNv2hgJFvRn+5LEi+juK+Hi zZd74?=OX`+0ojbm(3-QPH2|GpvcXak$abN%9uhL}hFoJ~-6HjcN+e>l)nmy71Ejn! zKv)v7nOdkrC5s4GzI5uJ6y^#SXP0n^*w*)z?Uk_R4$uZNagAB?oeS`kgtcBn_i8KT%!L+WWaZjUeBUCOj$*1-G zWP>GNI$9w;lU+rj)Wp;$u|dz;iy{@TI%2M7xpo4-2$=^S7t5^*211^AB@kr6HJMo5 z6#c5R_DlHKhE62{$?)UhI)cOzCvZ6=-rG3;q4HxAdnq;)_O_E#;aWTWIMW6AxiFeKiNSFS`pMfcegh1ghXu4fP+ZH zX6qsk8z>fAxqEe%#LQHl2ku28HtYW=_DNVPv~RW6$)tVIB?LBoB5)Jgzdm?vRv zyGekpWLioA6qqQMTIf5~Ss#k`HswPgIVhx9&}QVR|8Io2gQ^5`HB0v~BHP46D1?)# z>LJLatpT?+MB18#IwP1k42)U>>V)DSj9UYlq%}|AwX7v#17y!lBDNV;GH=O_9S^Zx z74zY{O^;6yls-=nL=aXQfdw<IC4|JzH>; zBUR4sD|2j~FVafo?0zy9yB9_;S7~oksFAS8R%MxJ%q08jwqBY6E z9BYQwBxJz69+mRu&xfszCI;1rhf~OdAhswaAC@)}n;!}YN!UgG>_~~&;J~ern3>8R z=IZSdvB9UxvK<7mA=DDrBvyM;oxit)HTSLBD9Mca?nOLQ!dfp^xvhk?PLT8n(M#9>5E18{Sk1c`6tFL$?LuEFNDVY|luJ$R#$&Kr-dQy)P zsWTt>~lrYE(T zA(>&H?x2ei{0@mik%qZVA`d#D@p=_jKm%Yj5D<0(XZirRwaF>CfrI)19cgUJQX zy-Cz**VM8nMVW`}B@p}mV4|5#cd+Ou6+I8)=75KsXRzWlb$ixaxp45eR^KCtPLSv3 zYA#C_3twN08lfd+F$|K@IXs#0+VP}rn#C1SLM5tO5x@v!r;7a zr7Dt-5p|R2NB>OF+ZcvvjQ1w@*Dy+%!MAW2<)#sWUbFd?Bc%GeZmf= z&Yb^=hgx@H;iHU16t9N%cN2KxHO)@#&IH~9jU5s>Kz5$!;7=W8J1Jof`L>TD7z-n&D@O2weSgV2Oa$fShXN z{3U!4>+O?OsDADFKAVUI&&tAS5gos^n2A9i#S^E;1jI3gi2dy59@cA4OEbz&k6|2^ zZyd%3vo{ezumIMM?W{SB)EZfrmLA2O)!*=q%#2{;@teHn!FZVVbgMcIIk^{0Oz16Nl{K%*X4RPy}mY^h%#eaGef9)0OVWJI|`YD&g8UqU*%(Y0*Tp>rgUn=MOskB0~{Q1UjVM z=lPjULMR9OYvP|R?n<^G!WU&_m6GmEj*To%{;lI(wv})<^CmNe?wHTBc&#>b$`J2w z$&be%TMmE9hnFXWMUXD5cAgD6jfw@0bw9KXE_j;pfPf#k(BfZMw`*I@#+*ip&P46p zmwY`o^4^!9v8oy8mgX1dtO@e?;&{!5N6vdJ+ZF{A0_DRQPE2}t{&^Lk&h*)r<5ZRy z6X&!yA%+gD++mv;Rr($WQb(eqT)~ac=`@Rtgh~!n? z*aSA;qo0@D+|c_;%IQFlGx%|zM^hQ8yWiF#H8^1Mbci!ESb?DWW?o9KO z-SYM^_vUqVEKS|5o2RDp&t>p?0%@45E9v(Br+Y8m#OMQu&%znY^nbp(C6mKeAU0SK zwtC!^`=M5Bod6MJssS)%NOAiu-7O$!C^37u(x;f5@y+|YmqsBB&A<6eaVK#lPUX)G zXFMRKgJdq_|9B91imhY%0F@kn$_aBi5F|W2_rn2Xu;27a#RD|lkl0}8ugZTNdPb+lJDez{5CMux4dK-N8b zCclet$|RL;!KsPUJ-%4pury$rz8U|?q?w6^|5~bFq`S1}$25A0!ZY{;(t=ng+wv93 z-zUA+6T9|STWY$Gcx+^A5nDTOnLvk7V&%ooTI8>eJM5<(qB`TYi7% zMo(8H*Cc(Rf4`u&y7A|>f_bj`4v(8@Cbqq9hDVGZs$A9KM~Y}meGk8QH&yo0lWy)S zGq7DAuh_>Qs&QkFJVMzqU`i-e{HUQMuV8(+ZM&E2`vSJd-g?Bg`z8-M!BmNNNl=r-wqFnWFhIQ7~t+cyP zYCDbX<6L2BfAueAy<>XvCprwxvGTLCJ0|FRFOkbp>FDqHh#gJy`FVI|7Vx>u&d~d7 z3(ccu)3PFX_nDjzS*TZSDXJ)K(+@2wM3GTZ8tnZuZ-{WQE8Jwy56E`uhg3CT9e^I# zUS6zQ#EcT`AurS8{#m`YC4Q}5`Y+cNQLqKm$oN0fMuSL73?9XvRLIc~kIqrZud@@pvR`pqi-&MN)^7lF0E%p87K1kQ?WtA+J znIdUy(S?QnC2aGl`q52h2_VO3cpjrU=>N(Azo41+`O4&LzJ1rvZ67sw=el)82O{eae`~to!a_&3gLCCo+LW;w*T-L+m92?x{2+Vh zkfr1KFJeurTQw1fj?~1RQp}$e8Gq-A)8khIXk%Q|EJeQKg+;TjwvN2`Y4%yuRl$=- zJ7u4K{^5LjT5-xWYgxDX)?Z&P8RoYn;c<6k>g9X)7VdsEVl*2$YlZW*ec+g^+3Vt$ Sz0sXKdT(0r)T4f}s{aGkd+>Mw literal 0 HcmV?d00001 diff --git a/docs/web/static/assets/logo/lance-logo-gray.svg b/docs/web/static/assets/logo/lance-logo-gray.svg deleted file mode 100644 index 764b68db7..000000000 --- a/docs/web/static/assets/logo/lance-logo-gray.svg +++ /dev/null @@ -1,5 +0,0 @@ - - - diff --git a/docs/web/static/assets/logo/light-lancedb-logo.svg b/docs/web/static/assets/logo/light-lancedb-logo.svg index b0746745e..99e50467d 100644 --- a/docs/web/static/assets/logo/light-lancedb-logo.svg +++ b/docs/web/static/assets/logo/light-lancedb-logo.svg @@ -1,14 +1,3 @@ - - - - - - - - - - - - - - \ No newline at end of file + + + diff --git a/docs/web/tables/branching.mdx b/docs/web/tables/branching.mdx index af06ec7a9..baf46b265 100644 --- a/docs/web/tables/branching.mdx +++ b/docs/web/tables/branching.mdx @@ -4,12 +4,8 @@ sidebarTitle: "Branch" description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main." icon: "code-branch" availability: - oss: available + embedded: available enterprise: available - summary: >- - Branches work in the embedded library and in LanceDB Enterprise. Comparing a - branch with `main` and promoting its new columns onto `main` is available - only in LanceDB Enterprise. --- import { PyConnect, diff --git a/docs/web/tables/consistency.mdx b/docs/web/tables/consistency.mdx index 87df024ab..78aedb23f 100644 --- a/docs/web/tables/consistency.mdx +++ b/docs/web/tables/consistency.mdx @@ -4,12 +4,8 @@ sidebarTitle: "Consistency" description: "Understand when readers see changes and choose the freshness your application needs." icon: "water" availability: - oss: available + embedded: available enterprise: available - summary: >- - Configure read freshness in LanceDB Embedded and LanceDB Enterprise. Embedded - checks for table updates; Enterprise uses the connection setting to tighten its - configured freshness bound. --- import { PyReadConsistency } from '/snippets/read_consistency.mdx'; @@ -82,7 +78,7 @@ a write completing and a reader seeing it. ## Learn more {#next} -- [Connect](/basics/connect): configure your client. +- [Connect](/basics/catalog): configure your client. - [Table versioning](/management/table#versioning): read and restore retained versions. - [Guarantees](/basics/guarantees): understand commit and failure boundaries. diff --git a/docs/web/tables/realtime-engine.mdx b/docs/web/tables/realtime-engine.mdx index 88738791c..b3ca31346 100644 --- a/docs/web/tables/realtime-engine.mdx +++ b/docs/web/tables/realtime-engine.mdx @@ -1,229 +1,42 @@ --- -title: "Realtime engine (RTE)" -sidebarTitle: "Realtime Engine" +title: "Real Time engine (RTE)" +sidebarTitle: "Real Time Engine" description: "Absorb small, frequent writes cheaply and make them queryable within milliseconds." icon: "bolt" -keywords: ["rte", "realtime engine", "wal", "memwal", "lsm", "write-ahead log", "streaming writes", "upsert"] +keywords: ["rte", "real time engine", "wal", "memwal", "lsm", "write-ahead log", "streaming writes", "upsert"] +availability: + embedded: unavailable + enterprise: available --- -Enterprise - -## What is the realtime engine? {#what-is-rte} +## What is the real time engine? {#what-is-rte} Every write to a LanceDB table is normally a commit to the underlying Lance dataset. That is ideal for bulk loads, but a stream of small upserts becomes a stream of small commits, and each write is only visible once its commit lands. -The realtime engine (RTE) puts an LSM-style write tier in front of the table. +The real time engine (RTE) puts an LSM-style write tier in front of the table. It turns many small writes into a few large commits, and makes each write **searchable within milliseconds** of being acknowledged. It is enabled per table, and nothing about how you query that table changes. ## How it works {#how-it-works} -- **Writes.** A `merge_insert` or `delete` lands in an in-memory **memtable** - and is acknowledged once it is durable in the write-ahead log on object - storage. +- **[Writes](/ingestion/realtime-write).** A `merge_insert` or `delete` lands in + an in-memory **memtable** and is acknowledged once it is durable in the + write-ahead log on object storage. - **Flush and compaction.** Memtables freeze into **SSTables**, which background jobs compact into the base table, catching its indexes up as they go. See [Background jobs](/maintenance/background-jobs#wal). -- **Reads.** Every query runs against the base table and the fresh tier - (memtables and SSTables) in parallel. Fresh rows supersede base rows with the - same primary key, and the two are merged into one result. +- **[Reads](/search/realtime-read).** Every query runs against the base table + and the fresh tier (memtables and SSTables) in parallel. Fresh rows supersede + base rows with the same primary key, and the two are merged into one result. - **Indexes stay fresh.** Vector, full-text, BTree, bitmap, and most other scalar indexes are maintained on the memtable, so new rows are searchable through the index as soon as they are written. -## Getting started {#getting-started} - -### 1. Create a table with a primary key {#primary-key} - -RTE deduplicates writes by primary key. Create the table with a non-nullable -key column, then declare it with `set_unenforced_primary_key`. The key is a -single `int32`, `int64`, string, or binary column, and cannot be changed once -set. - -```python -from datetime import timedelta - -import lancedb -import numpy as np -import pyarrow as pa -from lancedb import RTEWriteSpec -from lancedb.index import FTS, BTree, IvfPq - -db = lancedb.connect("db://my_db", api_key="...", host_override="https://...") - -n, dim = 1024, 128 -schema = pa.schema([ - pa.field("id", pa.int64(), nullable=False), - pa.field("vector", pa.list_(pa.float32(), dim)), - pa.field("text", pa.utf8()), -]) -initial_data = pa.table({ - "id": pa.array(range(n), pa.int64()), - "vector": pa.FixedSizeListArray.from_arrays( - pa.array(np.random.rand(n * dim).astype("float32")), dim), - "text": pa.array([f"doc {i} alpha bravo" for i in range(n)]), -}, schema=schema) -table = db.create_table("docs", initial_data, schema=schema) -table.set_unenforced_primary_key("id") -``` - -Or in SQL: - -```sql -ALTER TABLE docs SET UNENFORCED PRIMARY KEY (id) -``` - -### 2. Create indexes {#indexes} - -By default, RTE maintains every supported index on the table, including -indexes created after it is enabled. A scalar index on the primary key is -required for partial updates and deletes. - -```python -table.create_index("vector", config=IvfPq()) -table.create_index("text", config=FTS()) -table.create_index("id", config=BTree()) -table.wait_for_index(["vector_idx", "text_idx", "id_idx"], timeout=timedelta(minutes=5)) -``` - -### 3. Enable RTE {#enable} - -```python -table.enable_rte(RTEWriteSpec.unsharded()) -``` - -Or in SQL: - -```sql -ALTER TABLE docs SET RTE -``` - -### 4. Write and query {#write-and-query} - - -On an RTE table, an upsert through `merge_insert` is the only way to write -rows. Use the form below exactly: `when_matched_update_all()` with no filter, -plus `when_not_matched_insert_all()`. Other `merge_insert` forms, such as -conditional updates or deleting rows missing from the source, are rejected. - - -Writes go through `merge_insert` on the primary key, and are visible to queries -immediately: - -```python -new_rows = pa.table({ - "id": pa.array([2000, 2001], pa.int64()), - "vector": pa.FixedSizeListArray.from_arrays( - pa.array(np.random.rand(2 * dim).astype("float32")), dim), - "text": pa.array(["fresh charlie delta", "fresh echo foxtrot"]), -}, schema=schema) -query_vector = np.random.rand(dim).astype("float32") - -table.merge_insert("id") \ - .when_matched_update_all() \ - .when_not_matched_insert_all() \ - .execute(new_rows) - -table.search(query_vector).limit(10).to_list() -table.search("charlie").limit(10).to_list() -``` - -**Partial updates.** Send the primary key plus only the columns you want to -change; omitted columns keep their current values. This requires a scalar -index on the primary key. - -```python -patch = pa.table( - {"id": [0, 1], "text": ["doc 0 revised", "doc 1 revised"]}, - schema=pa.schema([schema.field("id"), schema.field("text")]), -) -table.merge_insert("id").when_matched_update_all().when_not_matched_insert_all().execute(patch) -``` - -**Deletes.** The predicate must bind the primary key to literal values. Ranges, -`OR`, and filters on other columns are rejected. Deletes require a scalar index -on the primary key. - -```python -table.delete("id IN (2000, 2001)") -``` - -### Disabling RTE {#disable} - -Stop writes and wait for in-flight writes to finish, then flush pending writes -into the base table and disable. `checkpoint_rte()` only drains writes that -existed when it started, so anything written during the checkpoint can be left -behind. `disable_rte()` is rejected while the write-ahead log still holds rows, -so run `checkpoint_rte()` first. - -```python -table.checkpoint_rte() -table.disable_rte() -``` - -## Performance {#performance} - -### Writes {#write-performance} - -RTE acknowledges a write as soon as it is durable in the write-ahead log, so -ingest is no longer limited by how fast the table can commit. - -**Compared with writing directly to the table.** On tables ranging from -hundreds of thousands to tens of millions of rows, RTE delivered 4–12× the -write throughput and 24–52× lower p99 write latency, depending on the mix of -inserts, updates, and deletes. - -**Scaling with shards.** An RTE table can be split into -[shards](/sql/ddl/table#properties) that write independently, and a single -table's ingest grows with its shard count. In our testing on AWS, with every -index on the table (vector, full-text, and BTree) kept current, a single table -reached these peak rates: - -| Shards | Peak ingest | -| --: | --: | -| 1 | 90 MB/s | -| 2 | 170 MB/s | -| 4 | 340 MB/s | -| 8 | 625 MB/s | -| 16 | 690 MB/s | -| 32 | 885 MB/s | - -Adding shards scales a table's ingest nearly linearly. - -### Reads {#read-performance} - -Reads on an RTE table stay close to those on a fully compacted table, even -with the fresh tier at its fullest before compaction. - -The table below compares a 5M-row table with a 128-dimension IVF-PQ index plus -BTree and FTS indexes, with and without RTE. The RTE table holds one -unflushed memtable and three SSTables, each from a 100k-row write wave, half of -which are updates to existing rows. Latencies are p50 in milliseconds. - -| Query | Without RTE | With RTE | -| --- | --: | --: | -| Primary-key lookup | 62 | 63 | -| Vector search | 68 | 71 | -| Vector search with prefilter | 90 | 91 | -| Full-text term | 63 | 64 | -| Full-text phrase | 68 | 82 | -| Hybrid search (RRF) | 68 | 83 | -| SQL point lookup | 189 | 187 | - -Most queries are within a few milliseconds of the base table. Hybrid and phrase -queries add roughly 15 ms to merge the two tiers. - - -Support for all table operations on RTE tables is coming soon. Time travel, -branching, `count_rows`, row IDs (`with_row_id`), and `TABLESAMPLE` are not yet -available. - - ## Next steps {#next-steps} -- [Merge insert](/ingestion/merge-insert) +- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table +- [Real Time reads](/search/realtime-read) — query an RTE table and its performance - [Background jobs](/maintenance/background-jobs#wal) -- [`ALTER TABLE ... SET RTE`](/sql/ddl/table#properties) diff --git a/docs/web/training/data-loading.mdx b/docs/web/training/data-loading.mdx index a839f5d8c..47a3a6703 100644 --- a/docs/web/training/data-loading.mdx +++ b/docs/web/training/data-loading.mdx @@ -1,38 +1,13 @@ --- -title: "Loading Data for Model Training" -sidebarTitle: "Data Loading & Shuffles" -description: "Stream, shuffle, transform, and resume model training data with LanceDB." +title: "PyTorch Data Loader" +sidebarTitle: "PyTorch Data Loader" +description: "Stream, shuffle, transform, and resume model training data with StreamingDataset." icon: boxes-stacked availability: - oss: available + embedded: available enterprise: available - summary: >- - Streaming, shuffling and resuming training data work with local, - object-storage and deployment tables in the Python SDK. --- -LanceDB provides an excellent data backend for training machine learning models. For basic use cases, a `Table` can be fed directly into a data loader. For a more complete solution, a `StreamingDataset` adds prefetching, elastic determinism, resumability, multithreaded transformations, and more. - -## Basic Data Loading - -Most model training frameworks feed data into a model by sequencing through successive batches, a process -often referred to as **data loading**. The simplest way to perform data loading is to read a LanceDB table in batches, -in the order its rows are stored, through an identity `Permutation`. A table itself is not iterable. - - -```py Python icon=Python -import lancedb -from lancedb.permutation import Permutation - -db = lancedb.connect("file://some/db/path") -table = db.open_table("some_table") - -for batch in Permutation.identity(table).with_format("arrow").iter(batch_size=1024): - print(batch.to_pydict()) -``` - - -## Advanced Data Loading Advanced data loading scenarios can be handled by the `StreamingDataset`. This object allows you to apply preprocessing before training, pipelined reads, and more. Use this when the training data does not fit in memory, or when you need deterministic global batches across cluster sizes. @@ -56,30 +31,15 @@ for batch in dataloader: ``` -The `StreamingDataset` wraps a LanceDB `Table` and yields plain Python dictionaries by default. Whether iterating directly or through a PyTorch `DataLoader`, only one iterator can be simultaneously active on a `StreamingDataset` instance. +The `StreamingDataset` wraps a LanceDB `Table` and yields plain Python dictionaries by default. Whether iterating directly or through a PyTorch `DataLoader`, only one iterator can be simultaneously active on a `StreamingDataset` instance. -`StreamingDataset` is built on the `permutation` API and works with both LanceDB Embedded and LanceDB Enterprise. The underlying table data can live on local disk or object storage. +`StreamingDataset` is built on the [`Permutation`](/training/permutation) API and works with both LanceDB Embedded and LanceDB Enterprise. The underlying table data can live on local disk or object storage. -To understand the wide range of scenarios handled by `StreamingDataset`, consider a model trained with stochastic gradient descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs -across multiple servers. After each batch is processed, the GPUs exchange weights and the next batch is loaded. We -fix the following terminology: +## Filtering {#filtering} -- **World size** - The number of GPUs being loaded simultaneously. For example, if we have $2$ servers with $4$ - GPUs each, the world size is $8$. -- **Rank** - The identifier of an individual GPU. It is an integer in the range `[0, world_size)`. - Each rank is allocated a disjoint portion of the source data. -- **Batch size** - The number of rows processed by one GPU in a single training step. For example, $128$. -- **Global batch size** - The total number of rows processed across all GPUs in a training step (sometimes called a "minibatch" in machine learning literature). For - example, if we have $8$ GPUs with batch size $128$, the global batch size is $1024$. - - -Other concepts, such as read batch size and `num_workers`, are introduced in the relevant sections below. - -### Filtering - -The `StreamingDataset` object allows you to train models on only the subset of rows that satisfy a user-defined filter, specified using the `filter` parameter. This can be useful for removing large rows, badly-sourced data, and a wide range of other applications. +The `StreamingDataset` object allows you to train models on only the subset of rows that satisfy a user-defined filter, specified using the `filter` parameter. This can be useful for removing large rows, badly-sourced data, and a wide range of other applications. You can also use the `columns` parameter to load only a subset of columns. @@ -93,7 +53,7 @@ ds = StreamingDataset( ``` -### Shuffling Rows +## Shuffling Rows {#shuffling-rows} In model training, model quality can be hurt by unintentionally learning patterns in the ordering of the source data. Shuffling the dataset before training alleviates this problem. However, fetching rows with scattered sparse reads across object storage can be dramatically slower than fetching contiguous batches. `StreamingDataset` shuffles at one of two granularities, selected by the `shuffle_clump_size` parameter. @@ -102,21 +62,20 @@ In model training, model quality can be hurt by unintentionally learning pattern with the same inputs remain reproducible. Set `shuffle_seed = None` to generate a random seed upon pipeline initialization. -#### Row-Level Shuffle {#one-phase-shuffle} +### Row-Level Shuffle {#one-phase-shuffle} By default, `shuffle_clump_size` is `None` and rows are shuffled individually: the rows of the source dataset are shuffled according to a pseudorandom permutation, then fed in batches to the accelerator. This gives the best possible randomness, but can exhibit low throughput. - - -```py Python icon=Python + +```py Python icon=Python # 1-phase shuffled read: rows shuffled individually. -for epoch in range(num_epochs): - ds = StreamingDataset( - table, - shuffle_seed=42, - epoch=epoch, # changes the permutation each epoch - ) - for sample in ds: +for epoch in range(num_epochs): + ds = StreamingDataset( + table, + shuffle_seed=42, + epoch=epoch, # changes the permutation each epoch + ) + for sample in ds: train_step(sample) #Evaluation: deterministic sequential order, no shuffle. @@ -124,11 +83,9 @@ eval_ds = StreamingDataset(table, shuffle=False) for sample in eval_ds: eval_step(sample) ``` - - -#### Clumped Shuffle {#two-phase-shuffle} +### Clumped Shuffle {#two-phase-shuffle} Setting `shuffle_clump_size` to an integer shuffles contiguous groups ("clumps") of that many rows instead of single rows. Each clump keeps its rows in storage order and moves as a unit, so storage I/O consists mostly of contiguous reads rather than single-row fetches. Larger clumps improve I/O locality, which matters most on object storage, at the cost of randomness. @@ -153,57 +110,7 @@ for sample in eval_ds: ``` -### Parallelism & Elasticity - -When loading data, `StreamingDataset` partitions the source dataset among disjoint parallel threads called **splits**. Each rank is allocated a contiguous group of splits, and each DataLoader worker associated to a rank gets a contiguous subgroup of its rank's splits. - -If using `shuffle=False`, data is allocated to splits naively in ordered contiguous blocks of equal size. With `shuffle=True` -(the default), the rows of the source dataset are shuffled and distributed among splits such that each split receives the same -number of rows; with `shuffle_clump_size` set, they are shuffled in clumps of that many contiguous rows. - -The number of splits defaults to `world_size` if `num_splits` is omitted during initialization. This is not recommended, as a later run using a different world size will then construct a different layout and structure of global batches. - -To get **elastic determinism**, choose a fixed `num_splits` that is compatible with every topology you plan to use. -For example, a run with `num_splits=8` and $4$ GPUs assigns $2$ splits to each rank. The ranks pull from their splits in -round-robin order, producing the same global batches as a run with $8$ GPUs and the same `num_splits`, `shuffle_seed`, -and `epoch`. - -The following constraints apply: - -- `num_splits` must be divisible by `world_size`. -- With DataLoader worker processes, `num_splits` must be divisible by `world_size * num_workers`. -- For the same samples to form each global training step across topologies, `global_batch_size` must be a multiple of - `num_splits`. -- The filtered row count must be at least `num_splits`. If it is not evenly divisible by `num_splits`, up to - `num_splits - 1` surplus rows are dropped so that every split has the same length. - -For example, to switch between $8$ and $6$ GPUs, use a common multiple such as $24$ for `num_splits`. Highly composite values can support many layouts; for example, `num_splits = 48` is compatible with $10$ distinct possible GPU counts. Remember to include `num_workers` when checking divisibility. - - -```py Python icon=Python -import torch.distributed as dist - -dist.init_process_group("nccl") -rank = dist.get_rank() -world_size = dist.get_world_size() - -# num_splits=48 is divisible by many integers, -# so this dataset works unchanged as you add or remove GPUs. -ds = StreamingDataset( - table, - num_splits=48, - shuffle_seed=42, - epoch=current_epoch, - rank=rank, - world_size=world_size, -) - -for sample in ds: - train_step(sample) -``` - - -### Fetching +## Fetching {#fetching} Traditional datasets have often been designed to yield a single sample at a time upon iteration. However, in modern use cases, accessing data on object storage one row at a time introduces too much per-call overhead. Thus, `StreamingDataset` fetches data in read batches, whose size is controlled by the `read_batch_size` parameter (defaults = `64`). @@ -220,13 +127,12 @@ ds = StreamingDataset( ``` -### Transforming +## Transforming {#transforming} -Many model training workloads require performing a transformation to the data after it is loaded from object storage and before it is passed to the GPU. For example, we may need to decode images, tokenize text, or normalize vectors. A transformation function can be provided using the `transform` parameter. +Many model training workloads require performing a transformation to the data after it is loaded from object storage and before it is passed to the GPU. For example, we may need to decode images, tokenize text, or normalize vectors. A transformation function can be provided using the `transform` parameter. A transformation function must receive a PyArrow `RecordBatch` and return an iterable. For example, the default transform function returns a list of Python dictionaries. Furthermore, the input expected by PyTorch must match the collation function used in the PyTorch data loader. PyTorch's default collation function accepts a Python dictionary and several other types, and is thus compatible with the default `StreamingDataset` transform function. If your custom transform function returns something different (for example, Arrow record batches), you may need to provide a compatible PyTorch collation function. - ```py Python icon=Python import pyarrow as pa @@ -245,9 +151,50 @@ ds = StreamingDataset(table, shuffle_seed=42, transform=normalize) `StreamingDataset` manages expensive transforms through a -`ThreadPoolExecutor` whose worker count equals the number of available CPUs. Furthermore, transformations are applied to batches (instead of individual samples) to amortize per-batch overhead. +`ThreadPoolExecutor` whose worker count equals the number of available CPUs (`transform_parallelism`). Furthermore, transformations are applied to batches (instead of individual samples) to amortize per-batch overhead. -##### DataLoader Workers +Heavier per-sample work belongs in the same function, such as decoding +video. Store each clip in a Blob v2 column (`lancedb.blob("video")`). A scan +returns small descriptors instead of the video bytes, so reading other +columns never pulls video in. `fetch_blob_files` opens the clips for a batch +of rows as file handles, and `torchcodec`'s `VideoDecoder` decodes just the +frames you ask for. `ExtractFrames` samples 8 evenly spaced frames from each +clip: + + +```py Python icon=Python +import functools +from torchcodec.decoders import VideoDecoder + +class ExtractFrames: + def __init__(self, uri: str, name: str, num_frames: int = 8) -> None: + self.uri, self.name, self.num_frames = uri, name, num_frames + + @functools.cached_property + def table(self) -> lancedb.table.Table: + return lancedb.connect(self.uri).open_table(self.name) + + def __call__(self, rb: pa.RecordBatch) -> list[dict[str, torch.Tensor]]: + samples = [] + for blob in self.table.fetch_blob_files("video", rb["_rowid"]): + decoder = VideoDecoder(blob) + idx = torch.linspace(0, len(decoder) - 1, self.num_frames).long() + samples.append({"frames": decoder.get_frames_at(idx.tolist()).data}) + return samples + +# `clips` is the table "clips" in the database at `uri`, with a Blob v2 column "video" +train = StreamingDataset(clips, columns=["_rowid"], transform=ExtractFrames(uri, "clips")) +``` + + +Real data has bad rows, such as a truncated frame. Raise inside the +transform and set `on_transform_error="skip"` (or `"warn"` to log each +failure). `StreamingDataset` re-runs the failing batch one row at a time, +drops only the rows that raise, and counts them in `dataset.rows_skipped`. +When the bad rows match a SQL predicate, prefer `filter=` instead: it runs +before the splits are built, so every split keeps the same length. + +### DataLoader Workers {#dataloader-workers} The thread-based transformation model that `StreamingDataset` uses by default is only effective when the transform function releases the GIL. This is true for many Python scientific libraries, including NumPy, PyArrow, and @@ -259,6 +206,7 @@ loading in the rank's main process, and add workers only after confirming an una workers, `num_splits` must be divisible by `world_size * num_workers`. Use the `forkserver` or `spawn` multiprocessing context because LanceDB uses internal threads. + ```py Python icon=Python dataloader = torch.utils.data.DataLoader( dataset, @@ -268,12 +216,87 @@ dataloader = torch.utils.data.DataLoader( persistent_workers=True, ) ``` + -By default, `StreamingDataset` serializes enough table state to reopen the table in each worker. For custom connection -setup, pass a picklable `connection_factory` callable that accepts a table name and returns an open table. This avoids -serializing connection credentials into worker state. +By default, `StreamingDataset` serializes enough table state to reopen the +table in each worker. Remote LanceDB Enterprise tables (`db://...`) work the +same way: workers reopen the table from the pickled connection state. +Replace the URI, API key, region and endpoint below with your deployment +details. In application code, load credentials from environment variables +or a secrets manager: -### Observability + +```py Python icon=Python +import lancedb +import torch +from lancedb.util import tbl_to_tensor + +db = lancedb.connect( + "db://my-database", + api_key="your-api-key", + region="us-east-1", + host_override="https://your-endpoint", +) +table = db.open_table("my_table") + +dataloader = torch.utils.data.DataLoader( + table, + batch_size=512, + num_workers=4, + multiprocessing_context="forkserver", + collate_fn=tbl_to_tensor, +) +``` + + + +This sends the connection state, including the API key, to each worker. Use a connection factory if credentials should be loaded inside the worker or your `client_config` contains a non-serializable `header_provider`. + + +#### Providing a custom connection factory {#providing-a-custom-connection-factory} + +For custom connection setup, pass a picklable `connection_factory` callable +that accepts a table name and returns an open table. This avoids +serializing connection credentials into worker state: it loads the API key +inside each worker instead. Set it in the shell before starting the Python +program, then replace the database URI, region and endpoint in the factory +with your deployment details: + +```shell +export LANCEDB_API_KEY="your-api-key" +``` + + +```py Python icon=Python +import os +import lancedb +import torch +from lancedb.permutation import Permutation + +def open_table(name: str): + db = lancedb.connect( + "db://my-database", + api_key=os.environ["LANCEDB_API_KEY"], + region="us-east-1", + host_override="https://your-endpoint", + ) + return db.open_table(name) + +table = open_table("my_table") +permutation = ( + Permutation.identity(table) + .with_connection_factory(open_table) +) +dataloader = torch.utils.data.DataLoader( + permutation, + batch_size=512, + num_workers=4, + multiprocessing_context="forkserver", +) +``` + + +## Observability {#observability} Optimizing data loader performance can be tricky, as runtime bottlenecks are often difficult to locate across various stages of the I/O and CPU pipelines. To help distinguish bottlenecks, `StreamingDataset` exposes several pipeline counters: `raw_queue_depth` shows the number of loaded rows waiting to be transformed, and `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and @@ -320,161 +343,13 @@ are cumulative across iterations of the same dataset instance; because work runs can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters reflect the latest iterator snapshot. -### Checkpointing -`StreamingDataset` provides checkpointing in case your model training fails partway through a run. Use `StreamingDataset.state_dict()` to capture the current number of samples consumed from each split in a Python dictionary, and `load_state_dict()` to restore the last saved position. +## Next steps {#next} -A stored `StreamingDataset` state reflects only round-robin cycles that have been completed across all splits; samples completed partway through a cycle at the time of saving can be replayed. To ensure efficient saving, make sure `global_batch_size` is a multiple of `num_splits`, as in the example below. - - -```py Python icon=Python -import torch - -global_batch_size = 48 # one sample per split in each global step -batch_size = global_batch_size // world_size - -dataset = StreamingDataset( - table, - num_splits=48, - shuffle_seed=42, - epoch=current_epoch, - rank=rank, - world_size=world_size, -) -dataloader = torch.utils.data.DataLoader( - dataset, - batch_size=batch_size, - num_workers=0, -) - -for step, batch in enumerate(dataloader): - train_step(batch) - - if (step + 1) % checkpoint_interval == 0: - torch.save( - {"model": model.state_dict(), "dataset": dataset.state_dict()}, - f"checkpoint_{step + 1}.pt", - ) - -# --- resuming after a crash --- -checkpoint = torch.load("checkpoint_100.pt") -model.load_state_dict(checkpoint["model"]) - -dataset_state = checkpoint["dataset"] -dataset = StreamingDataset( - table, - num_splits=dataset_state["num_splits"], - shuffle_seed=dataset_state["shuffle_seed"], - epoch=dataset_state["epoch"], - rank=rank, - world_size=world_size, # may differ from the run that saved the checkpoint -) -dataset.load_state_dict(dataset_state) -dataloader = torch.utils.data.DataLoader( - dataset, - batch_size=global_batch_size // world_size, - num_workers=0, -) - -for batch in dataloader: - train_step(batch) -``` - - -`load_state_dict()` rejects a checkpoint whose `num_splits` or `shuffle_seed` differs from the new dataset. For an -exact resume, also use the same table snapshot, `epoch`, `shuffle`, `filter`, and other data-selection settings. The -`world_size` and number of DataLoader workers may change as long as the split and batch-size divisibility constraints -still hold. - -##### Checkpointing with Multiple DataLoader Workers - -Python's built-in `torch.utils.data.DataLoader` produces a safe checkpoint when `num_workers = 0`. With `num_workers > 0`, calling `state_dict()` often raises `RuntimeError`. In this case, use `StreamingDataLoader` for multi-worker prefetch and exact resumability. It carries a state snapshot -alongside every internal batch and commits it to the parent `StreamingDataset` only when that batch is returned to the -trainer. The trainer receives the same collated batch it would receive from a standard `DataLoader`. - - -```py Python icon=Python -from lancedb.streaming import StreamingDataset, StreamingDataLoader - -dataset = StreamingDataset( - table, - num_splits=48, - shuffle_seed=42, - epoch=current_epoch, - rank=rank, - world_size=world_size, -) -dataloader = StreamingDataLoader( - dataset, - batch_size=batch_size, - num_workers=4, -) - -for step, batch in enumerate(dataloader): - train_step(batch) - - if (step + 1) % checkpoint_interval == 0: - torch.save( - {"model": model.state_dict(), "dataset": dataset.state_dict()}, - f"checkpoint_{step + 1}.pt", - ) -``` - - -`StreamingDataLoader` accepts the same arguments as `torch.utils.data.DataLoader`, with a few restrictions: - -- `dataset` must be a `StreamingDataset` (subclasses that override `__iter__` are not supported). -- `in_order = True` is required so that consumer-committed checkpoints stay deterministic. -- `persistent_workers = True` is not supported, because prefetched worker copies cannot be restored from parent-committed state. -- `drop_last = True` is not supported, because incomplete tails discarded by worker replicas cannot be checkpointed topology-independently. - -With more than one worker, call `state_dict()` at a complete logical step boundary where every split assigned to the -rank has the same consumed-sample count. Calling it mid-step raises `RuntimeError` asking you to consume more batches -first. - -##### Resuming Across Different Topologies - -When training across ranks, each rank owns its own subset of splits and only its own splits have exact progress. To -resume on a different `world_size`, collect the `state_dict()` from every rank of the previous run and merge them with -`StreamingDataset.merge_state_dicts` before calling `load_state_dict()` on the new run. The merge is topology-agnostic: -pass the full list of per-rank states in, and hand the identical merged dict to every rank of the resumed job, -regardless of whether the topology grew, shrank, or stayed the same. - - -```py Python icon=Python -# On each rank of the previous run: -state = dataset.state_dict() -# ...gather `state` from every rank into a single list `states` (for example -# via torch.distributed all-gather or by saving one file per rank). - -merged = StreamingDataset.merge_state_dicts(states) - -# On each rank of the resumed run (world_size may differ): -dataset = StreamingDataset( - table, - num_splits=merged["num_splits"], - shuffle_seed=merged["shuffle_seed"], - epoch=merged["epoch"], - rank=rank, - world_size=new_world_size, -) -dataset.load_state_dict(merged) -``` - - -## Permutations - -In specific use cases, you may need the flexibility to shuffle, split, and select data without using the full `StreamingDataset` object. In these cases, use `Permutation`, the lower-level functionality on which `StreamingDataset` -is built. A `Permutation` defines a customizable row ordering, and supports map-style access through `__getitem__()` -and batched access through `__getitems__()`. - -##### Version Pinning - -A `Permutation` is pinned to the version of its base table at the time it was constructed, and every read (including future reads -from a `StreamingDataset` that wraps it, and every DataLoader worker after it starts) resolves against that pinned -version. This makes iteration deterministic across compactions and worker restarts: the permutation only addresses rows -that existed when it was built, and rows appended to the base table afterwards are not visible through the existing -permutation. - -To include newly appended rows, build a new `Permutation` (and any `StreamingDataset` that depends on it) -against the updated table. Permutations written before this behavior shipped continue to -read as they did before. + + + The lower-level row ordering StreamingDataset is built on. + + + Multi-GPU sharding, elastic determinism, and checkpointing. + + diff --git a/docs/web/training/distributed-training.mdx b/docs/web/training/distributed-training.mdx new file mode 100644 index 000000000..549539591 --- /dev/null +++ b/docs/web/training/distributed-training.mdx @@ -0,0 +1,216 @@ +--- +title: "Distributed Training" +sidebarTitle: "Distributed Training" +description: "Shard StreamingDataset across ranks and workers, with elastic determinism and checkpointing." +icon: server +availability: + embedded: available + enterprise: available +--- + +To understand the wide range of scenarios handled by [`StreamingDataset`](/training/data-loading) across a +distributed job, consider a model trained with stochastic gradient descent (SGD) and distributed data parallelism +(DDP). In this example, we need to load batches onto multiple GPUs across multiple servers. After each batch is +processed, the GPUs exchange weights and the next batch is loaded. We fix the following terminology: + +- **World size** - The number of GPUs being loaded simultaneously. For example, if we have $2$ servers with $4$ + GPUs each, the world size is $8$. +- **Rank** - The identifier of an individual GPU. It is an integer in the range `[0, world_size)`. + Each rank is allocated a disjoint portion of the source data. +- **Batch size** - The number of rows processed by one GPU in a single training step. For example, $128$. +- **Global batch size** - The total number of rows processed across all GPUs in a training step (sometimes called a "minibatch" in machine learning literature). For + example, if we have $8$ GPUs with batch size $128$, the global batch size is $1024$. + +Other concepts, such as read batch size and `num_workers`, are introduced in [PyTorch Data Loader](/training/data-loading). + +## Parallelism & Elasticity {#parallelism-&-elasticity} + +When loading data, `StreamingDataset` partitions the source dataset among disjoint parallel threads called **splits**. Each rank is allocated a contiguous group of splits, and each DataLoader worker associated to a rank gets a contiguous subgroup of its rank's splits. + +If using `shuffle=False`, data is allocated to splits naively in ordered contiguous blocks of equal size. With `shuffle=True` +(the default), the rows of the source dataset are shuffled and distributed among splits such that each split receives the same +number of rows; with `shuffle_clump_size` set, they are shuffled in clumps of that many contiguous rows. + +The number of splits defaults to `world_size` if `num_splits` is omitted during initialization. This is not recommended, as a later run using a different world size will then construct a different layout and structure of global batches. + +To get **elastic determinism**, choose a fixed `num_splits` that is compatible with every topology you plan to use. +For example, a run with `num_splits=8` and $4$ GPUs assigns $2$ splits to each rank. The ranks pull from their splits in +round-robin order, producing the same global batches as a run with $8$ GPUs and the same `num_splits`, `shuffle_seed`, +and `epoch`. + +The following constraints apply: + +- `num_splits` must be divisible by `world_size`. +- With DataLoader worker processes, `num_splits` must be divisible by `world_size * num_workers`. +- For the same samples to form each global training step across topologies, `global_batch_size` must be a multiple of + `num_splits`. +- The filtered row count must be at least `num_splits`. If it is not evenly divisible by `num_splits`, up to + `num_splits - 1` surplus rows are dropped so that every split has the same length. + +For example, to switch between $8$ and $6$ GPUs, use a common multiple such as $24$ for `num_splits`. Highly composite values can support many layouts; for example, `num_splits = 48` is compatible with $10$ distinct possible GPU counts. Remember to include `num_workers` when checking divisibility. + + +```py Python icon=Python +import torch.distributed as dist + +dist.init_process_group("nccl") +rank = dist.get_rank() +world_size = dist.get_world_size() + +# num_splits=48 is divisible by many integers, +# so this dataset works unchanged as you add or remove GPUs. +ds = StreamingDataset( + table, + num_splits=48, + shuffle_seed=42, + epoch=current_epoch, + rank=rank, + world_size=world_size, +) + +for sample in ds: + train_step(sample) +``` + + +## Checkpointing {#checkpointing} + +`StreamingDataset` provides checkpointing in case your model training fails partway through a run. Use `StreamingDataset.state_dict()` to capture the current number of samples consumed from each split in a Python dictionary, and `load_state_dict()` to restore the last saved position. + +A stored `StreamingDataset` state reflects only round-robin cycles that have been completed across all splits; samples completed partway through a cycle at the time of saving can be replayed. To ensure efficient saving, make sure `global_batch_size` is a multiple of `num_splits`, as in the example below. + + +```py Python icon=Python +import torch + +global_batch_size = 48 # one sample per split in each global step +batch_size = global_batch_size // world_size + +dataset = StreamingDataset( + table, + num_splits=48, + shuffle_seed=42, + epoch=current_epoch, + rank=rank, + world_size=world_size, +) +dataloader = torch.utils.data.DataLoader( + dataset, + batch_size=batch_size, + num_workers=0, +) + +for step, batch in enumerate(dataloader): + train_step(batch) + + if (step + 1) % checkpoint_interval == 0: + torch.save( + {"model": model.state_dict(), "dataset": dataset.state_dict()}, + f"checkpoint_{step + 1}.pt", + ) + +# --- resuming after a crash --- +checkpoint = torch.load("checkpoint_100.pt") +model.load_state_dict(checkpoint["model"]) + +dataset_state = checkpoint["dataset"] +dataset = StreamingDataset( + table, + num_splits=dataset_state["num_splits"], + shuffle_seed=dataset_state["shuffle_seed"], + epoch=dataset_state["epoch"], + rank=rank, + world_size=world_size, # may differ from the run that saved the checkpoint +) +dataset.load_state_dict(dataset_state) +dataloader = torch.utils.data.DataLoader( + dataset, + batch_size=global_batch_size // world_size, + num_workers=0, +) + +for batch in dataloader: + train_step(batch) +``` + + +`load_state_dict()` rejects a checkpoint whose `num_splits` or `shuffle_seed` differs from the new dataset. For an +exact resume, also use the same table snapshot, `epoch`, `shuffle`, `filter`, and other data-selection settings. The +`world_size` and number of DataLoader workers may change as long as the split and batch-size divisibility constraints +still hold. + +### Checkpointing with Multiple DataLoader Workers {#checkpointing-with-multiple-dataloader-workers} + +Python's built-in `torch.utils.data.DataLoader` produces a safe checkpoint when `num_workers = 0`. With `num_workers > 0`, calling `state_dict()` often raises `RuntimeError`. In this case, use `StreamingDataLoader` for multi-worker prefetch and exact resumability. It carries a state snapshot +alongside every internal batch and commits it to the parent `StreamingDataset` only when that batch is returned to the +trainer. The trainer receives the same collated batch it would receive from a standard `DataLoader`. + + +```py Python icon=Python +from lancedb.streaming import StreamingDataset, StreamingDataLoader + +dataset = StreamingDataset( + table, + num_splits=48, + shuffle_seed=42, + epoch=current_epoch, + rank=rank, + world_size=world_size, +) +dataloader = StreamingDataLoader( + dataset, + batch_size=batch_size, + num_workers=4, +) + +for step, batch in enumerate(dataloader): + train_step(batch) + + if (step + 1) % checkpoint_interval == 0: + torch.save( + {"model": model.state_dict(), "dataset": dataset.state_dict()}, + f"checkpoint_{step + 1}.pt", + ) +``` + + +`StreamingDataLoader` accepts the same arguments as `torch.utils.data.DataLoader`, with a few restrictions: + +- `dataset` must be a `StreamingDataset` (subclasses that override `__iter__` are not supported). +- `in_order = True` is required so that consumer-committed checkpoints stay deterministic. +- `persistent_workers = True` is not supported, because prefetched worker copies cannot be restored from parent-committed state. +- `drop_last = True` is not supported, because incomplete tails discarded by worker replicas cannot be checkpointed topology-independently. + +With more than one worker, call `state_dict()` at a complete logical step boundary where every split assigned to the +rank has the same consumed-sample count. Calling it mid-step raises `RuntimeError` asking you to consume more batches +first. + +### Resuming Across Different Topologies {#resuming-across-different-topologies} + +When training across ranks, each rank owns its own subset of splits and only its own splits have exact progress. To +resume on a different `world_size`, collect the `state_dict()` from every rank of the previous run and merge them with +`StreamingDataset.merge_state_dicts` before calling `load_state_dict()` on the new run. The merge is topology-agnostic: +pass the full list of per-rank states in, and hand the identical merged dict to every rank of the resumed job, +regardless of whether the topology grew, shrank, or stayed the same. + + +```py Python icon=Python +# On each rank of the previous run: +state = dataset.state_dict() +# ...gather `state` from every rank into a single list `states` (for example +# via torch.distributed all-gather or by saving one file per rank). + +merged = StreamingDataset.merge_state_dicts(states) + +# On each rank of the resumed run (world_size may differ): +dataset = StreamingDataset( + table, + num_splits=merged["num_splits"], + shuffle_seed=merged["shuffle_seed"], + epoch=merged["epoch"], + rank=rank, + world_size=new_world_size, +) +dataset.load_state_dict(merged) +``` + diff --git a/docs/web/training/object-detection.mdx b/docs/web/training/object-detection.mdx index 620409728..6e9213a57 100644 --- a/docs/web/training/object-detection.mdx +++ b/docs/web/training/object-detection.mdx @@ -7,7 +7,7 @@ icon: car This example walks through fine-tuning an autonomous vehicle (AV) perception model on targeted failure-mode slices of [BDD100K](https://www.bdd100k.com/) (riders, nighttime pedestrians, and distant pedestrians) using LanceDB as a single multimodal table from raw JPEG bytes through to the PyTorch training loop. -The full pipeline lives in the [lancedb/training](https://github.com/lancedb/training/tree/main/object-detection) repository. This page focuses on the parts most relevant to training: defining curated splits as materialized views, loading them through the [`Permutation`](/training/data-loading) API, and pinning checkpoints to an exact data version. +The full pipeline lives in the [lancedb/training](https://github.com/lancedb/training/tree/main/object-detection) repository. This page focuses on the parts most relevant to training: defining curated splits as materialized views, loading them through the [`Permutation`](/training/permutation) API, and pinning checkpoints to an exact data version. ## What you get {#what-you-get} @@ -155,7 +155,7 @@ with gconn.local_ray_context(): ## 4\. PyTorch DataLoader via the Permutation API {#4-pytorch-dataloader-via-the-permutation-api} -The training script doesn't know about the filter: it opens a view by name and reads through the [`Permutation`](/training/data-loading) API. Each DataLoader worker reopens its own connection lazily, reads Arrow batches directly from Lance (zero-copy, no intermediate file format), and the collate function decodes the whole batch in one pass. `Permutation` provides random-access indexing over the table, so shuffling is a cheap pointer rewrite rather than a full-dataset shuffle on disk. +The training script doesn't know about the filter: it opens a view by name and reads through the [`Permutation`](/training/permutation) API. Each DataLoader worker reopens its own connection lazily, reads Arrow batches directly from Lance (zero-copy, no intermediate file format), and the collate function decodes the whole batch in one pass. `Permutation` provides random-access indexing over the table, so shuffling is a cheap pointer rewrite rather than a full-dataset shuffle on disk. ```py Python icon=Python import lancedb diff --git a/docs/web/training/permutation.mdx b/docs/web/training/permutation.mdx new file mode 100644 index 000000000..751056541 --- /dev/null +++ b/docs/web/training/permutation.mdx @@ -0,0 +1,123 @@ +--- +title: "Permutation" +sidebarTitle: "Permutation" +description: "The low-level, customizable row ordering that StreamingDataset is built on." +icon: shuffle +availability: + embedded: available + enterprise: available +--- + +Most model training frameworks feed data into a model by sequencing through successive batches, a process +often referred to as **data loading**. The simplest way to perform data loading is to read a LanceDB table in batches, +in the order its rows are stored, through an identity `Permutation`. A table itself is not iterable. + + +```py Python icon=Python +import lancedb +from lancedb.permutation import Permutation + +db = lancedb.connect("file://some/db/path") +table = db.open_table("some_table") + +for batch in Permutation.identity(table).with_format("arrow").iter(batch_size=1024): + print(batch.to_pydict()) +``` + + +In specific use cases, you may need the flexibility to shuffle, split, and select data without using the full +[`StreamingDataset`](/training/data-loading) object. In these cases, use `Permutation`, the lower-level +functionality `StreamingDataset` is built on. A `Permutation` defines a customizable row ordering, and supports +map-style access through `__getitem__()` and batched access through `__getitems__()`. + +## Version Pinning {#version-pinning} + +A `Permutation` is pinned to the version of its base table at the time it was constructed, and every read (including future reads +from a `StreamingDataset` that wraps it, and every DataLoader worker after it starts) resolves against that pinned +version. This makes iteration deterministic across compactions and worker restarts: the permutation only addresses rows +that existed when it was built, and rows appended to the base table afterwards are not visible through the existing +permutation. + +To include newly appended rows, build a new `Permutation` (and any `StreamingDataset` that depends on it) +against the updated table. Permutations written before this behavior shipped continue to +read as they did before. + +## Output formats {#output-formats} + +By default, a `Table` data loader emits Arrow data. `collate_fn` is PyTorch's batching hook: PyTorch calls it to +turn the fetched items into one batch. PyTorch's default collate function only knows how to combine tensors, NumPy +arrays, numbers, dicts, and lists, so it does not accept Arrow data directly. When using a `Table` directly, pass +LanceDB's `lancedb.util.tbl_to_tensor` helper as PyTorch's `collate_fn`; it converts numeric Arrow columns into a +column-major `torch.Tensor` with shape `(columns, rows)`. + +`Permutation` works differently: its default output is a list of Python dicts, which PyTorch's default collate function +can batch into a dict of tensors. This is usually more convenient when you are getting started. However, there is a +significant performance penalty converting from Arrow, Lance's internal representation, to this default format. If you +want the default PyTorch dict-of-tensors behavior, use a `Permutation` as-is; if you want direct Arrow-to-tensor +conversion, either pass `lancedb.util.tbl_to_tensor` as `collate_fn` with a direct `Table` or configure a `Permutation` +with one of the transform formats described below. + +To address this, the `Permutation` class provides a set of builtin transform functions that can be applied to map +the Arrow data in different ways. The `arrow` and `polars` formats will always avoid data copies. However, `numpy`, +`pandas`, and `torch_col` formats will also avoid data copies in most cases. The `python`, `python_col`, and +`torch` formats will all require at least one full copy of the data and are the slowest options. + +### Using the torch column format with a torch data loader {#using-the-torch-column-format-with-a-torch-data-loader} + +The `torch_col` format is the most efficient way to convert from Arrow to a `torch.Tensor`. It will convert the +entire Arrow batch to a _column-major_ `torch.Tensor`. In other words, given C columns and R rows, the resulting +Tensor will have shape `(C, R)`. However, this format generates an error if you are using a +`torch.utils.data.DataLoader` with the default collation function: + +```text +TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor +``` + +This error occurs because the default collation function does not currently expect a single two-dimensional tensor. +It expects a list of tensors which it will then stack. This is what is output by the `torch` format but that format +requires a data copy. To avoid this error, and avoid data copies, you will need to provide a custom collation function +in addition to specifying the `torch_col` format. + +```py Python icon=Python +from lancedb.permutation import Permutation + +permutation = Permutation.identity(table).with_format("torch_col") +dataloader = torch.utils.data.DataLoader(permutation, collate_fn=lambda x: x) +``` + +This will now output a single two-dimensional tensor for each batch. + +## Selecting columns {#selecting-columns} + +By default, the `Table` class will return all columns in the table when used as input to PyTorch. If you only need +a subset of columns, you can significantly reduce your I/O requirements by selecting only the columns you need. The +`Permutation` class provides a `select_columns` method that provides this functionality. + +```py Python icon=Python +import lancedb +import pyarrow as pa +import torch +from lancedb.permutation import Permutation + +db = lancedb.connect("memory://") +prompts = db.create_table( + "prompts", + pa.table( + { + "id": range(4), + "prompt": ["north", "south", "east", "west"], + "embedding": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]], + } + ), +) + +permutation = Permutation.identity(prompts).select_columns(["id", "prompt"]) +dataloader = torch.utils.data.DataLoader( + permutation, batch_size=1024, shuffle=True +) + +for batch in dataloader: + print(sorted(batch)) # ['id', 'prompt']: the embedding is never read +``` + +Each batch is a dict of the selected columns, the default `Permutation` format described above. diff --git a/docs/web/training/torch.mdx b/docs/web/training/torch.mdx deleted file mode 100644 index 6a6f0c12f..000000000 --- a/docs/web/training/torch.mdx +++ /dev/null @@ -1,334 +0,0 @@ ---- -title: "PyTorch Integration" -sidebarTitle: "PyTorch" -description: Use a LanceDB table as a PyTorch IterableDataset, mapped onto the `torch.utils.data` concepts you already know. -icon: fire -availability: - oss: available - enterprise: available - summary: >- - The PyTorch integration works with local, object-storage and deployment - tables in the Python SDK. ---- - -`StreamingDataset` is a LanceDB table exposed as a PyTorch -[`IterableDataset`](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset), -so it drops into a stock [`DataLoader`](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). -It handles per-epoch shuffling, sharding across ranks and workers, prefetching, and exact resume, so you -don't write those yourself. The shuffle is fixed when the dataset is constructed, so each epoch needs a new -`StreamingDataset` built with that epoch's number. - -## Quickstart {#quickstart} - -```py Python icon=Python -import lancedb, pyarrow as pa, torch -from torch import nn -from lancedb.streaming import StreamingDataset - -def to_samples(rb: pa.RecordBatch): - f = lambda c: torch.from_dlpack(rb[c].flatten()).view(len(rb), -1) - rows = zip(f("observation_state"), f("action")) - return [{"state": s, "action": a} for s, a in rows] - -uri = "hf://datasets/lance-format/lerobot-xvla-soft-fold/data" -db = lancedb.connect(uri, storage_options={"hf_enable_resolve_cache": "true"}) -table = db.open_table("frames") -train = StreamingDataset( - table, columns=["observation_state", "action"], - transform=to_samples) - -model = nn.Sequential(nn.Linear(96, 256), nn.ReLU(), nn.Linear(256, 14)) -opt = torch.optim.AdamW(model.parameters(), lr=1e-3) -for batch in torch.utils.data.DataLoader(train, batch_size=256): - loss = nn.MSELoss()(model(batch["state"]), batch["action"]) - opt.zero_grad() - loss.backward() - opt.step() -``` - -`StreamingDataset` is a standard PyTorch `IterableDataset`, so it works with everything built for one: a stock -`DataLoader`, its default `collate_fn`, and worker processes. The transform returns plain tensors, so the -model, loss and optimizer are unmodified PyTorch. Here it streams the -[`lerobot-xvla-soft-fold`](https://huggingface.co/datasets/lance-format/lerobot-xvla-soft-fold) dataset -straight from the Hugging Face Hub to train a policy that predicts each `action` from the robot's -`observation_state`. - -## PyTorch concepts in LanceDB {#concept-map} - -| In PyTorch you would write | In LanceDB you use | -| --- | --- | -| An `IterableDataset` that reads files in `__iter__` | `StreamingDataset(table)` | -| `DistributedSampler`, or sharding by `get_worker_info()` in `__iter__` | `rank=` and `world_size=`. Each rank and each DataLoader worker reads its own disjoint splits | -| Decoding in `__getitem__` or `collate_fn` | `transform=`, which receives a whole Arrow `RecordBatch` and runs in a thread pool | -| `random_split`, `Subset`, or reading only the fields a model needs | `filter=""` and `columns=[...]`. Unselected columns are never read from storage | -| `StatefulDataLoader` | `state_dict()` and `load_state_dict()`, plus `StreamingDataLoader` for workers | - -The dataset does not copy your data. At construction it computes a shuffled assignment of row IDs to -`num_splits` fixed splits, then reads rows from the table as it iterates. It reads the table version -that exists at construction, so rows appended later do not appear mid-epoch. - -## Optimizations {#optimizations} - -The quickstart works as written. These sections cover what to tune as training scales up: the transform that -builds each sample, running across many GPUs, and the output format of a `Table` or `Permutation`. - -### Transform {#transform} - -A `transform` turns a batch of Arrow rows into the per-sample items your `DataLoader` collates. It receives a -whole `pyarrow.RecordBatch` (`read_batch_size` rows, 64 by default) rather than one sample at a time, so -vectorized code is cheap, and it must return exactly one item per row. The quickstart's transform is the -whole pattern: columns become tensors and each row becomes a dict, which the default `collate_fn` stacks into -a batch. Without a transform, rows arrive as plain Python dicts. - -Heavier per-sample work belongs in the same function, such as decoding video. Store each clip in a blob v2 -column (`lancedb.blob("video")`). A scan returns small descriptors instead of the video bytes, so reading -other columns never pulls video in. `fetch_blob_files` opens the clips for a batch of rows as file handles, -and torchcodec's `VideoDecoder` decodes just the frames you ask for. `ExtractFrames` samples 8 evenly spaced -frames from each clip: - -```py Python icon=Python -import functools -from torchcodec.decoders import VideoDecoder - -class ExtractFrames: - def __init__(self, uri: str, name: str, num_frames: int = 8) -> None: - self.uri, self.name, self.num_frames = uri, name, num_frames - - @functools.cached_property - def table(self) -> lancedb.table.Table: - return lancedb.connect(self.uri).open_table(self.name) - - def __call__(self, rb: pa.RecordBatch) -> list[dict[str, torch.Tensor]]: - samples = [] - for blob in self.table.fetch_blob_files("video", rb["_rowid"]): - decoder = VideoDecoder(blob) - idx = torch.linspace(0, len(decoder) - 1, self.num_frames).long() - samples.append({"frames": decoder.get_frames_at(idx.tolist()).data}) - return samples - -# `clips` is the table "clips" in the database at `uri`, with a blob v2 column "video" -train = StreamingDataset(clips, columns=["_rowid"], transform=ExtractFrames(uri, "clips")) -``` - -Transforms run in a thread pool, one thread per CPU by default (`transform_parallelism`), overlapping with -storage reads and your training step. NumPy, PyArrow and PyTorch release the GIL, so vectorized -work runs in parallel. A pure-Python loop does not. If a profile shows the transform is the bottleneck, move -it into [DataLoader workers](#using-multiple-dataloader-workers). - -Real data has bad rows, such as a truncated frame. Raise inside the transform and set -`on_transform_error="skip"` (or `"warn"` to log each failure). LanceDB re-runs the failing batch one row at a -time, drops only the rows that raise, and counts them in `dataset.rows_skipped`. When the bad rows match a SQL -predicate, prefer `filter=`: it runs before the splits are built, so every split keeps the same length. - -### Distributed Training {#distributed-training} - -`StreamingDataset` takes the place of `DistributedSampler`. Pass the rank and world size, and each rank reads -its own disjoint part of the table, together covering it once per epoch: - -```py Python icon=Python -import torch.distributed as dist - -dist.init_process_group("nccl") -train = StreamingDataset( - table, columns=["observation_state", "action"], transform=to_samples, - num_splits=48, shuffle_seed=42, epoch=epoch, - rank=dist.get_rank(), world_size=dist.get_world_size()) -``` - -Everything after the dataset is standard distributed PyTorch. Wrap the model in `DistributedDataParallel` -and pass `batch_size` to the `DataLoader` per GPU, so the global batch is `batch_size * world_size`. Nothing -in the data path changes. - -Three arguments matter in distributed runs: - -- **`num_splits`** is how many fixed streams the table is divided into. It must be divisible by `world_size`, - and by `world_size * num_workers` when you use DataLoader workers. Choose a number with many divisors, such - as 48, and runs on 8, 4 or 2 GPUs see the same data at every step, provided the global batch size is a multiple of - `num_splits`. -- **`shuffle_seed` and `epoch`** must be identical on every rank. Pass a fixed `shuffle_seed` and increase - `epoch` each epoch. `shuffle_seed=None` draws a different random seed in each process, so ranks would read - overlapping rows and miss others. -- **`rank` and `world_size`** come from your launcher, for example `torchrun`. - -To resume after a failure, save `train.state_dict()` next to the model. It records progress per split, so the -resumed run continues from the same row even on a different number of GPUs. See -[Parallelism & Elasticity](/training/data-loading#parallelism-&-elasticity) and -[Checkpointing](/training/data-loading#checkpointing) for the batch-size rules and the multi-worker case. - -### Output Formats {#output-formats} - -By default, a `Table` data loader will emit Arrow data. `collate_fn` is PyTorch's batching hook: PyTorch calls it to -turn the fetched items into one batch. PyTorch's default collate function only knows how to combine tensors, NumPy -arrays, numbers, dicts, and lists, so it does not accept Arrow data directly. When using a `Table` directly, pass -LanceDB's `lancedb.util.tbl_to_tensor` helper as PyTorch's `collate_fn`; it converts numeric Arrow columns into a -column-major `torch.Tensor` with shape `(columns, rows)`. - -`Permutation` works differently: its default output is a list of Python dicts, which PyTorch's default collate function -can batch into a dict of tensors. This is usually more convenient when you are getting started. However, there is a -significant performance penalty converting from Arrow, Lance's internal representation, to this default format. If you -want the default PyTorch dict-of-tensors behavior, use a `Permutation` as-is; if you want direct Arrow-to-tensor -conversion, either pass `lancedb.util.tbl_to_tensor` as `collate_fn` with a direct `Table` or configure a `Permutation` -with one of the transform formats described below. - -To address this, the `Permutation` class provides a set of builtin transform functions that can be applied to map -the Arrow data in different ways. The `arrow` and `polars` formats will always avoid data copies. However, `numpy`, -`pandas`, and `torch_col` formats will also avoid data copies in most cases. The `python`, `python_col`, and -`torch` formats will all require at least one full copy of the data and are the slowest options. - -#### Using the torch column format with a torch data loader {#using-the-torch-column-format-with-a-torch-data-loader} - -The `torch_col` format is the most efficient way to convert from Arrow to a `torch.Tensor`. It will convert the -entire Arrow batch to a _column-major_ `torch.Tensor`. In other words, given C columns and R rows, the resulting -Tensor will have shape `(C, R)`. However, this format generates an error if you are using a -`torch.utils.data.DataLoader` with the default collation function: - -```text -TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor -``` - -This error occurs because the default collation function does not currently expect a single two-dimensional tensor. -It expects a list of tensors which it will then stack. This is what is output by the `torch` format but that format -requires a data copy. To avoid this error, and avoid data copies, you will need to provide a custom collation function -in addition to specifying the `torch_col` format. - -```py Python icon=Python -from lancedb.permutation import Permutation - -permutation = Permutation.identity(table).with_format("torch_col") -dataloader = torch.utils.data.DataLoader(permutation, collate_fn=lambda x: x) -``` - -This will now output a single two-dimensional tensor for each batch. - -## Selecting columns {#selecting-columns} - -By default, the `Table` class will return all columns in the table when used as input to PyTorch. If you only need -a subset of columns, you can significantly reduce your I/O requirements by selecting only the columns you need. The -`Permutation` class provides a `select_columns` method that provides this functionality. - -```py Python icon=Python -import lancedb -import pyarrow as pa -import torch -from lancedb.permutation import Permutation - -db = lancedb.connect("memory://") -prompts = db.create_table( - "prompts", - pa.table( - { - "id": range(4), - "prompt": ["north", "south", "east", "west"], - "embedding": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]], - } - ), -) - -permutation = Permutation.identity(prompts).select_columns(["id", "prompt"]) -dataloader = torch.utils.data.DataLoader( - permutation, batch_size=1024, shuffle=True -) - -for batch in dataloader: - print(sorted(batch)) # ['id', 'prompt']: the embedding is never read -``` - -Each batch is a dict of the selected columns, the default `Permutation` format -described above. - -## Using multiple DataLoader workers {#using-multiple-dataloader-workers} - -Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables in LanceDB Enterprise are picklable, so each worker reopens the table after it starts. A LanceDB Embedded table is not, so wrap it in a `Permutation`, as below, to read it from several workers. - -Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and became the default on Linux and other supported POSIX platforms in Python -3.14. macOS and Windows default to `spawn`. See -[Python's start-method documentation](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods). See [the performance guide](/search#performance) for more multiprocessing guidance. - - -`forkserver` is only available on POSIX systems (Linux and macOS). On Windows, use `spawn`, the only available start method. - - -```py Python icon=Python -import torch -from lancedb.permutation import Permutation - -permutation = Permutation.identity(table) -dataloader = torch.utils.data.DataLoader( - permutation, - batch_size=1024, - shuffle=True, - num_workers=4, - multiprocessing_context="forkserver", - persistent_workers=True, -) -``` - -### Enterprise tables in DataLoader workers {#remote-tables-in-dataloader-workers} - -LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state. Replace the URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager. - -```py Python icon=Python -import lancedb -import torch -from lancedb.util import tbl_to_tensor - -db = lancedb.connect( - "db://my-database", - api_key="your-api-key", - region="us-east-1", - host_override="https://your-endpoint", -) -table = db.open_table("my_table") - -dataloader = torch.utils.data.DataLoader( - table, - batch_size=512, - num_workers=4, - multiprocessing_context="forkserver", - collate_fn=tbl_to_tensor, -) -``` - - -This sends the connection state, including the API key, to each worker. Use a connection factory if credentials should be loaded inside the worker or your `client_config` contains a non-serializable `header_provider`. - - -### Providing a custom connection factory {#providing-a-custom-connection-factory} - -`Permutation.with_connection_factory` lets each worker reopen the base table with custom logic. The factory takes the table name, returns a LanceDB table, and must be picklable. - -This example loads the API key inside each worker. Set it in the shell before -starting the Python program, then replace the database URI, region and endpoint -in the factory with your deployment details: - -```shell -export LANCEDB_API_KEY="your-api-key" -``` - -```py Python icon=Python -import os -import lancedb -import torch -from lancedb.permutation import Permutation - -def open_table(name: str): - db = lancedb.connect( - "db://my-database", - api_key=os.environ["LANCEDB_API_KEY"], - region="us-east-1", - host_override="https://your-endpoint", - ) - return db.open_table(name) - -table = open_table("my_table") -permutation = ( - Permutation.identity(table) - .with_connection_factory(open_table) -) -dataloader = torch.utils.data.DataLoader( - permutation, - batch_size=512, - num_workers=4, - multiprocessing_context="forkserver", -) -``` diff --git a/docs/web/training/why-lancedb.mdx b/docs/web/training/why-lancedb.mdx deleted file mode 100644 index af10d4edc..000000000 --- a/docs/web/training/why-lancedb.mdx +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: "Why LanceDB for Training" -sidebarTitle: "Why LanceDB" -description: "Use LanceDB as the multimodal data layer for model training, fine-tuning, curation, and feature engineering workflows." -icon: fire ---- - -LanceDB is built for AI teams that need a practical data layer between raw multimodal datasets and model training. -Instead of moving data through separate systems for curation, feature engineering, search, manifests, and training, -you can keep the whole workflow attached to one versioned LanceDB table. - -That table can hold images, video, audio, text, annotations, metadata, embeddings, tokenized fields, model outputs, -quality signals, and training-ready tensors. As the dataset evolves, LanceDB lets you add new columns, filter rows, -pin versions, and read batches without rewriting the original data. - -![Training data lifecycle: Curation, Feature Engineering, Search and Retrieval, Training](/static/assets/images/overview/training-data-lifecycle.svg) - -The LanceDB multimodal lakehouse connects curation, feature engineering, retrieval, and training data access -through Lance tables. Data loaders read these tables into training frameworks, which run the model training. - -## A connected data lifecycle {#a-connected-data-lifecycle} - -Training pipelines usually need more than a pile of files. They need curation, derived features, reproducible splits, -fast random access, and a clean path into frameworks such as PyTorch. LanceDB keeps these pieces connected through -the same table model, whether you organize a workflow as one table or several related tables. - - - - Use filters, vector search, full-text search, and retrieval workflows to find the examples that matter: hard negatives, - long-tail failure modes, duplicate clusters, low-quality samples, or targeted fine-tuning slices. - - - Add embeddings, detections, OCR output, labels, token IDs, hidden states, deduplication flags, or quality scores as - new columns. Lance's columnar layout and schema evolution avoid rewriting large raw media columns when you add features. - - - Build filtered splits and materialized views from the table instead of exporting CSV manifests. Data versions and tags - make it possible to tie a checkpoint back to the exact rows and features used for training. - - - Use fast random access and column projection to read only the columns a training step needs. LanceDB tables can be read - from local storage or object storage, and integrate with data loading patterns such as PyTorch datasets. - - - -## Lance as the foundation {#lance-as-the-foundation} - -LanceDB is built on [Lance](https://lance.org/), an open-source lakehouse format designed for multimodal AI data. -The table below highlights the Lance features that enable the multimodal lakehouse on top. - -| Capability | Why it matters for training | -|---|---| -| **Multimodal columns** | Store raw bytes, annotations, metadata, embeddings, and features together. | -| **Fast random access** | Support shuffled and sampled reads without reshuffling the dataset on disk. | -| **Column projection** | Read only images, tokens, labels, embeddings, or hidden states needed by a given run. | -| **Schema evolution** | Add new feature columns without rewriting existing media columns. | -| **Versioning** | Reproduce experiments against the same table snapshot, even as the dataset evolves. | -| **Search and filtering** | Find and materialize useful training slices directly from the table. | - -## Search inside training workflows {#search-inside-training-workflows} - -Search is not limited to QA systems, agents, or production retrieval apps. It is also a practical way to inspect, -curate, and improve training data: - -- Find visually similar examples when debugging model failures. -- Retrieve hard negatives or near-duplicates for contrastive training. -- Combine vector search, full-text search, and metadata filters to build targeted fine-tuning slices. -- Reuse the same table for both offline curation and production retrieval. - -In LanceDB, retrieval and training workflows can operate over the same multimodal tables instead of forcing teams to -manage separate data systems for each stage. - -## Projects using LanceDB for training workflows {#projects-using-lancedb-for-training-workflows} - - - - A platform for reproducible world-model research built on a LanceDB data layer, reporting faster data loading on Push-T workloads. - - - A joint-embedding predictive world model from pixels, trained on the stable-worldmodel platform and its LanceDB data layer. - - - A drop-in LanceDB backend for Hugging Face LeRobot datasets with faster loading across robotics datasets. - - - -In the world-model ecosystem, [stable-worldmodel](https://github.com/galilai-group/stable-worldmodel) reports -3-4x faster data loading on Push-T versus HDF5 / MP4 at a fraction of the disk footprint. Across these projects, -LanceDB and Lance provide the multimodal data layer that keeps raw observations, annotations, features, and training -access patterns in one format instead of scattering them across task-specific stores. - -## Next steps {#next-steps} - - - - Learn how to use LanceDB permutations to select rows, project columns, split datasets, and shuffle training reads. - - - Use LanceDB tables and permutations with `torch.utils.data.DataLoader`. - - - Fine-tune an AV perception model on curated failure-mode slices backed by one LanceDB table. - - - Fine-tune a VLM on TextVQA using LanceDB and Geneva to cache expensive training features. - - diff --git a/docs/web/tutorials/feature-engineering/feature-engineering-101.mdx b/docs/web/tutorials/feature-engineering/feature-engineering-101.mdx index 7f6110162..31f847473 100644 --- a/docs/web/tutorials/feature-engineering/feature-engineering-101.mdx +++ b/docs/web/tutorials/feature-engineering/feature-engineering-101.mdx @@ -5,7 +5,6 @@ description: "Add derived feature columns to an existing table." --- Adds new derived features to a table, the pattern documented under -[data evolution](/feature-engineering/data-evolution) and [computed columns](/feature-engineering/add-computed-column). diff --git a/docs/web/tutorials/feature-engineering/materialized-views.mdx b/docs/web/tutorials/feature-engineering/materialized-views.mdx index 5e1a77bd4..32641a2d1 100644 --- a/docs/web/tutorials/feature-engineering/materialized-views.mdx +++ b/docs/web/tutorials/feature-engineering/materialized-views.mdx @@ -6,7 +6,7 @@ description: "Persist query results as physical tables and keep them current." Creates [materialized views](/management/materialized-view): query results persisted as physical tables, brought up to date by a -[refresh](/feature-engineering/materialized-views). +[refresh](/feature-engineering/refresh-materialized-view). This example is a runnable notebook in the