Bump version: 0.13.0-beta.0 → 0.13.0-beta.1

docs: add dlt-lancedb integration page (#1551 )
Co-authored-by: Akela Drissner-Schmid <32450038+akelad@users.noreply.github.com>
2025-12-23 13:29:57 +00:00 · 2024-08-23 13:56:30 +00:00 · 2024-08-22 15:18:49 +05:30 · 2024-08-22 11:46:36 +05:30 · 2024-08-22 00:54:05 +05:30 · 2024-08-21 20:37:04 +05:30
80 changed files with 2062 additions and 406 deletions
--- a/.bumpversion.toml
+++ b/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.8.0"
+current_version = "0.10.0-beta.0"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -20,12 +20,12 @@ keywords = ["lancedb", "lance", "database", "vector", "search"]
 categories = ["database-implementations"]

 [workspace.dependencies]
-lance = { "version" = "=0.16.0", "features" = ["dynamodb"] }
-lance-index = { "version" = "=0.16.0" }
-lance-linalg = { "version" = "=0.16.0" }
-lance-testing = { "version" = "=0.16.0" }
-lance-datafusion = { "version" = "=0.16.0" }
-lance-encoding = { "version" = "=0.16.0" }
+lance = { "version" = "=0.16.1", "features" = ["dynamodb"] }
+lance-index = { "version" = "=0.16.1" }
+lance-linalg = { "version" = "=0.16.1" }
+lance-testing = { "version" = "=0.16.1" }
+lance-datafusion = { "version" = "=0.16.1" }
+lance-encoding = { "version" = "=0.16.1" }
 # Note that this one does not include pyarrow
 arrow = { version = "52.2", optional = false }
 arrow-array = "52.2"
@@ -43,7 +43,7 @@ half = { "version" = "=2.4.1", default-features = false, features = [
 ] }
 futures = "0"
 log = "0.4"
-object_store = "0.10.1"
+object_store = "0.10.2"
 pin-project = "1.0.7"
 snafu = "0.7.4"
 url = "2"
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -58,7 +58,7 @@ plugins:
            - https://pandas.pydata.org/docs/objects.inv
  - mkdocs-jupyter
  - render_swagger:
-      allow_arbitrary_locations : true
+      allow_arbitrary_locations: true

 markdown_extensions:
  - admonition
@@ -89,9 +89,10 @@ nav:
          - Data management: concepts/data_management.md
      - 🔨 Guides:
          - Working with tables: guides/tables.md
-          - Building an ANN index: ann_indexes.md
+          - Building a vector index: ann_indexes.md
          - Vector Search: search.md
          - Full-text search: fts.md
+          - Building a scalar index: guides/scalar_index.md
          - Hybrid search:
              - Overview: hybrid_search/hybrid_search.md
              - Comparing Rerankers: hybrid_search/eval.md
@@ -128,15 +129,16 @@ nav:
          - Polars: python/polars_arrow.md
          - DuckDB: python/duckdb.md
          - LangChain:
-            - LangChain 🔗: integrations/langchain.md
-            - LangChain demo: notebooks/langchain_demo.ipynb
-            - LangChain JS/TS 🔗: https://js.langchain.com/docs/integrations/vectorstores/lancedb
+              - LangChain 🔗: integrations/langchain.md
+              - LangChain demo: notebooks/langchain_demo.ipynb
+              - LangChain JS/TS 🔗: https://js.langchain.com/docs/integrations/vectorstores/lancedb
          - LlamaIndex 🦙:
-            - LlamaIndex docs: integrations/llamaIndex.md
-            - LlamaIndex demo: notebooks/llamaIndex_demo.ipynb
+              - LlamaIndex docs: integrations/llamaIndex.md
+              - LlamaIndex demo: notebooks/llamaIndex_demo.ipynb
          - Pydantic: python/pydantic.md
          - Voxel51: integrations/voxel51.md
          - PromptTools: integrations/prompttools.md
+          - dlt: integrations/dlt.md
      - 🎯 Examples:
          - Overview: examples/index.md
          - 🐍 Python:
@@ -144,6 +146,10 @@ nav:
              - Build From Scratch: examples/python_examples/build_from_scratch.md
              - Multimodal: examples/python_examples/multimodal.md
              - Rag: examples/python_examples/rag.md
+              - Vector Search: examples/python_examples/vector_search.md
+              - Chatbot: examples/python_examples/chatbot.md
+              - Evaluation: examples/python_examples/evaluations.md
+              - AI Agent: examples/python_examples/aiagent.md
              - Miscellaneous: 
                  - YouTube Transcript Search: notebooks/youtube_transcript_search.ipynb
                  - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
@@ -181,6 +187,7 @@ nav:
      - Building an ANN index: ann_indexes.md
      - Vector Search: search.md
      - Full-text search: fts.md
+      - Building a scalar index: guides/scalar_index.md
      - Hybrid search:
          - Overview: hybrid_search/hybrid_search.md
          - Comparing Rerankers: hybrid_search/eval.md
@@ -222,6 +229,7 @@ nav:
      - Pydantic: python/pydantic.md
      - Voxel51: integrations/voxel51.md
      - PromptTools: integrations/prompttools.md
+      - dlt: integrations/dlt.md
  - Examples:
      - examples/index.md
      - 🐍 Python:
@@ -229,6 +237,10 @@ nav:
          - Build From Scratch: examples/python_examples/build_from_scratch.md
          - Multimodal: examples/python_examples/multimodal.md
          - Rag: examples/python_examples/rag.md
+          - Vector Search: examples/python_examples/vector_search.md
+          - Chatbot: examples/python_examples/chatbot.md
+          - Evaluation: examples/python_examples/evaluations.md
+          - AI Agent: examples/python_examples/aiagent.md
          - Miscellaneous: 
              - YouTube Transcript Search: notebooks/youtube_transcript_search.ipynb
              - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,6 +1,7 @@
 mkdocs==1.5.3
 mkdocs-jupyter==0.24.1
 mkdocs-material==9.5.3
-mkdocstrings[python]==0.20.0
+mkdocstrings[python]==0.25.2
+griffe
 mkdocs-render-swagger-plugin
 pydantic
--- a/docs/src/assets/open_hf_space.svg
+++ b/docs/src/assets/open_hf_space.svg
@@ -0,0 +1,22 @@
+<svg width="147" height="20" viewBox="0 0 147 20" fill="none" xmlns="http://www.w3.org/2000/svg">
+<rect x="0.5" y="0.5" width="145.482" height="19" rx="9.5" fill="white" stroke="#EFEFEF"/>
+<path d="M14.1863 10.9251V12.7593H16.0205V10.9251H14.1863Z" fill="#FF3270"/>
+<path d="M17.8707 10.9251V12.7593H19.7049V10.9251H17.8707Z" fill="#861FFF"/>
+<path d="M14.1863 7.24078V9.07496H16.0205V7.24078H14.1863Z" fill="#097EFF"/>
+<path fill-rule="evenodd" clip-rule="evenodd" d="M12.903 6.77179C12.903 6.32194 13.2676 5.95728 13.7175 5.95728C14.1703 5.95728 15.2556 5.95728 16.1094 5.95728C16.7538 5.95728 17.2758 6.47963 17.2758 7.12398V9.6698H19.8217C20.4661 9.6698 20.9884 10.1922 20.9884 10.8365C20.9884 11.6337 20.9884 12.4309 20.9884 13.2282C20.9884 13.678 20.6237 14.0427 20.1738 14.0427H17.3039H16.5874H13.7175C13.2676 14.0427 12.903 13.678 12.903 13.2282V9.71653V9.64174V6.77179ZM14.1863 7.24066V9.07485H16.0205V7.24066H14.1863ZM14.1863 12.7593V10.9251H16.0205V12.7593H14.1863ZM17.8708 12.7593V10.9251H19.705V12.7593H17.8708Z" fill="black"/>
+<path d="M18.614 8.35468L20.7796 6.18905M20.7796 6.18905V7.66073M20.7796 6.18905L19.2724 6.18905" stroke="black" stroke-width="0.686298" stroke-linecap="round" stroke-linejoin="round"/>
+<path d="M31.6082 13.9838C30.8546 13.9838 30.1895 13.802 29.6132 13.4385C29.0368 13.066 28.5846 12.5429 28.2565 11.869C27.9373 11.1862 27.7777 10.3749 27.7777 9.43501C27.7777 8.49511 27.9373 7.69265 28.2565 7.02762C28.5846 6.3626 29.0368 5.85275 29.6132 5.49807C30.1895 5.14339 30.8546 4.96605 31.6082 4.96605C32.3708 4.96605 33.0403 5.14339 33.6166 5.49807C34.193 5.85275 34.6408 6.3626 34.96 7.02762C35.2881 7.69265 35.4521 8.49511 35.4521 9.43501C35.4521 10.3749 35.2881 11.1862 34.96 11.869C34.6408 12.5429 34.193 13.066 33.6166 13.4385C33.0403 13.802 32.3708 13.9838 31.6082 13.9838ZM31.6082 12.6404C32.291 12.6404 32.8363 12.3523 33.2442 11.7759C33.6521 11.1907 33.856 10.4104 33.856 9.43501C33.856 8.45964 33.6521 7.69708 33.2442 7.14733C32.8363 6.58871 32.291 6.3094 31.6082 6.3094C30.9255 6.3094 30.3802 6.58871 29.9723 7.14733C29.5644 7.69708 29.3605 8.45964 29.3605 9.43501C29.3605 10.4104 29.5644 11.1907 29.9723 11.7759C30.3802 12.3523 30.9255 12.6404 31.6082 12.6404Z" fill="#2C3236"/>
+<path d="M37.0592 16.4045V7.29363H38.3227L38.4291 7.98526H38.4823C38.7572 7.75472 39.0631 7.55521 39.4 7.38674C39.7459 7.21826 40.0961 7.13403 40.4508 7.13403C41.2665 7.13403 41.8961 7.43551 42.3395 8.03846C42.7917 8.64142 43.0178 9.44831 43.0178 10.4591C43.0178 11.204 42.8848 11.8424 42.6188 12.3744C42.3528 12.8976 42.0069 13.2966 41.5813 13.5715C41.1646 13.8463 40.7124 13.9838 40.2247 13.9838C39.9409 13.9838 39.6572 13.9217 39.3734 13.7976C39.0897 13.6646 38.8148 13.4872 38.5488 13.2656L38.5887 14.3562V16.4045H37.0592ZM39.9055 12.7202C40.3399 12.7202 40.7035 12.5296 40.9961 12.1483C41.2887 11.767 41.435 11.2084 41.435 10.4724C41.435 9.81629 41.3242 9.30644 41.1025 8.94289C40.8808 8.57935 40.5217 8.39757 40.0252 8.39757C39.5641 8.39757 39.0853 8.64142 38.5887 9.1291V12.1749C38.8281 12.37 39.0587 12.5119 39.2803 12.6005C39.502 12.6803 39.7104 12.7202 39.9055 12.7202Z" fill="#2C3236"/>
+<path d="M47.3598 13.9838C46.7568 13.9838 46.2115 13.8508 45.7238 13.5848C45.2361 13.3099 44.8504 12.9197 44.5667 12.4143C44.2829 11.9 44.141 11.2838 44.141 10.5656C44.141 9.85619 44.2829 9.24437 44.5667 8.73009C44.8593 8.2158 45.2361 7.82122 45.6972 7.54634C46.1583 7.27147 46.6415 7.13403 47.147 7.13403C47.741 7.13403 48.2376 7.26703 48.6366 7.53304C49.0356 7.79018 49.3371 8.15373 49.541 8.62368C49.745 9.08476 49.847 9.62122 49.847 10.233C49.847 10.5523 49.8248 10.8005 49.7805 10.9779H45.6307C45.7016 11.5542 45.91 12.002 46.2558 12.3212C46.6016 12.6404 47.0361 12.8 47.5593 12.8C47.843 12.8 48.1046 12.7601 48.344 12.6803C48.5923 12.5917 48.8361 12.472 49.0755 12.3212L49.5942 13.2789C49.2839 13.4828 48.9381 13.6513 48.5568 13.7843C48.1755 13.9173 47.7765 13.9838 47.3598 13.9838ZM45.6174 9.94043H48.5169C48.5169 9.43501 48.4061 9.04043 48.1844 8.75669C47.9627 8.46408 47.6302 8.31777 47.1869 8.31777C46.8056 8.31777 46.4642 8.45964 46.1627 8.74339C45.8701 9.01826 45.6883 9.41728 45.6174 9.94043Z" fill="#2C3236"/>
+<path d="M51.3078 13.8242V7.29363H52.5714L52.6778 8.17147H52.731C53.0236 7.88772 53.3428 7.64388 53.6886 7.43994C54.0344 7.236 54.429 7.13403 54.8724 7.13403C55.5728 7.13403 56.0827 7.36014 56.4019 7.81235C56.7211 8.26457 56.8807 8.90299 56.8807 9.72762V13.8242H55.3512V9.92713C55.3512 9.38624 55.2714 9.00496 55.1118 8.78329C54.9522 8.56161 54.6906 8.45078 54.327 8.45078C54.0433 8.45078 53.7906 8.52171 53.5689 8.66358C53.3561 8.79659 53.1123 8.99609 52.8374 9.2621V13.8242H51.3078Z" fill="#2C3236"/>
+<path d="M61.4131 13.8242V7.29363H62.9426V13.8242H61.4131ZM62.1845 6.14979C61.9096 6.14979 61.6879 6.06999 61.5195 5.91038C61.351 5.75078 61.2668 5.53797 61.2668 5.27196C61.2668 5.01482 61.351 4.80644 61.5195 4.64684C61.6879 4.48723 61.9096 4.40743 62.1845 4.40743C62.4594 4.40743 62.6811 4.48723 62.8495 4.64684C63.018 4.80644 63.1022 5.01482 63.1022 5.27196C63.1022 5.53797 63.018 5.75078 62.8495 5.91038C62.6811 6.06999 62.4594 6.14979 62.1845 6.14979Z" fill="#2C3236"/>
+<path d="M64.8941 13.8242V7.29363H66.1576L66.264 8.17147H66.3172C66.6098 7.88772 66.929 7.64388 67.2748 7.43994C67.6207 7.236 68.0152 7.13403 68.4586 7.13403C69.1591 7.13403 69.6689 7.36014 69.9881 7.81235C70.3074 8.26457 70.467 8.90299 70.467 9.72762V13.8242H68.9374V9.92713C68.9374 9.38624 68.8576 9.00496 68.698 8.78329C68.5384 8.56161 68.2768 8.45078 67.9133 8.45078C67.6295 8.45078 67.3768 8.52171 67.1551 8.66358C66.9423 8.79659 66.6985 8.99609 66.4236 9.2621V13.8242H64.8941Z" fill="#2C3236"/>
+<path d="M75.1323 13.8242V5.12565H76.6752V8.62368H80.1998V5.12565H81.7427V13.8242H80.1998V9.96703H76.6752V13.8242H75.1323Z" fill="#2C3236"/>
+<path d="M83.9517 13.8242V5.12565H89.2054V6.4291H85.4945V8.88969H88.6601V10.1931H85.4945V13.8242H83.9517Z" fill="#2C3236"/>
+<path d="M95.9349 13.9838C95.3497 13.9838 94.7822 13.8729 94.2324 13.6513C93.6915 13.4296 93.2127 13.1148 92.796 12.7069L93.7004 11.6562C94.0108 11.9488 94.3654 12.1882 94.7645 12.3744C95.1635 12.5518 95.5625 12.6404 95.9615 12.6404C96.458 12.6404 96.8349 12.5385 97.092 12.3345C97.3492 12.1306 97.4778 11.8601 97.4778 11.5232C97.4778 11.1596 97.3492 10.8981 97.092 10.7385C96.8438 10.5789 96.5245 10.4148 96.1344 10.2463L94.9374 9.72762C94.6536 9.60348 94.3743 9.44388 94.0994 9.2488C93.8334 9.05373 93.6117 8.80546 93.4344 8.50398C93.2659 8.2025 93.1817 7.83895 93.1817 7.41334C93.1817 6.95225 93.3058 6.53994 93.5541 6.17639C93.8113 5.80398 94.1571 5.51137 94.5915 5.29856C95.0349 5.07689 95.5403 4.96605 96.1078 4.96605C96.6132 4.96605 97.1009 5.06802 97.5709 5.27196C98.0408 5.46703 98.4442 5.73304 98.7812 6.06999L97.9965 7.05423C97.7216 6.82368 97.429 6.64191 97.1186 6.5089C96.8172 6.3759 96.4802 6.3094 96.1078 6.3094C95.6999 6.3094 95.3674 6.4025 95.1103 6.58871C94.862 6.76605 94.7379 7.01432 94.7379 7.33353C94.7379 7.55521 94.7999 7.74142 94.9241 7.89215C95.0571 8.03403 95.23 8.15816 95.4428 8.26457C95.6556 8.36211 95.8817 8.45964 96.1211 8.55718L97.3048 9.0493C97.8191 9.27097 98.2403 9.56358 98.5684 9.92713C98.8965 10.2818 99.0605 10.7739 99.0605 11.4035C99.0605 11.8734 98.9364 12.3035 98.6881 12.6936C98.4398 13.0838 98.0807 13.3986 97.6108 13.638C97.1497 13.8685 96.591 13.9838 95.9349 13.9838Z" fill="#2C3236"/>
+<path d="M100.509 16.4045V7.29363H101.773L101.879 7.98526H101.932C102.207 7.75472 102.513 7.55521 102.85 7.38674C103.196 7.21826 103.546 7.13403 103.901 7.13403C104.717 7.13403 105.346 7.43551 105.79 8.03846C106.242 8.64142 106.468 9.44831 106.468 10.4591C106.468 11.204 106.335 11.8424 106.069 12.3744C105.803 12.8976 105.457 13.2966 105.031 13.5715C104.615 13.8463 104.162 13.9838 103.675 13.9838C103.391 13.9838 103.107 13.9217 102.824 13.7976C102.54 13.6646 102.265 13.4872 101.999 13.2656L102.039 14.3562V16.4045H100.509ZM103.356 12.7202C103.79 12.7202 104.154 12.5296 104.446 12.1483C104.739 11.767 104.885 11.2084 104.885 10.4724C104.885 9.81629 104.774 9.30644 104.553 8.94289C104.331 8.57935 103.972 8.39757 103.475 8.39757C103.014 8.39757 102.535 8.64142 102.039 9.1291V12.1749C102.278 12.37 102.509 12.5119 102.73 12.6005C102.952 12.6803 103.16 12.7202 103.356 12.7202Z" fill="#2C3236"/>
+<path d="M109.444 13.9838C108.876 13.9838 108.411 13.8064 108.047 13.4518C107.692 13.0971 107.515 12.636 107.515 12.0685C107.515 11.368 107.821 10.8271 108.433 10.4458C109.045 10.0557 110.02 9.78969 111.359 9.64782C111.35 9.30201 111.257 9.00496 111.08 8.75669C110.911 8.49954 110.605 8.37097 110.162 8.37097C109.843 8.37097 109.528 8.43304 109.218 8.55718C108.916 8.68132 108.619 8.83206 108.326 9.0094L107.768 7.98526C108.131 7.75472 108.539 7.55521 108.991 7.38674C109.452 7.21826 109.94 7.13403 110.454 7.13403C111.27 7.13403 111.878 7.37787 112.277 7.86555C112.685 8.34437 112.888 9.04043 112.888 9.95373V13.8242H111.625L111.518 13.1059H111.465C111.173 13.3542 110.858 13.5626 110.521 13.7311C110.193 13.8995 109.834 13.9838 109.444 13.9838ZM109.936 12.7867C110.202 12.7867 110.441 12.7247 110.654 12.6005C110.876 12.4675 111.111 12.2902 111.359 12.0685V10.6055C110.472 10.7207 109.856 10.8936 109.51 11.1242C109.164 11.3458 108.991 11.6207 108.991 11.9488C108.991 12.2414 109.08 12.4542 109.257 12.5872C109.435 12.7202 109.661 12.7867 109.936 12.7867Z" fill="#2C3236"/>
+<path d="M117.446 13.9838C116.851 13.9838 116.315 13.8508 115.836 13.5848C115.366 13.3099 114.989 12.9197 114.706 12.4143C114.431 11.9 114.293 11.2838 114.293 10.5656C114.293 9.83846 114.444 9.2222 114.746 8.71679C115.047 8.2025 115.446 7.81235 115.943 7.54634C116.448 7.27147 116.989 7.13403 117.565 7.13403C117.982 7.13403 118.346 7.20496 118.656 7.34684C118.966 7.48871 119.241 7.66161 119.48 7.86555L118.736 8.86309C118.567 8.71235 118.394 8.59708 118.217 8.51728C118.04 8.42861 117.849 8.38427 117.645 8.38427C117.122 8.38427 116.692 8.58378 116.355 8.98279C116.027 9.38181 115.863 9.9094 115.863 10.5656C115.863 11.2128 116.022 11.736 116.342 12.135C116.67 12.534 117.091 12.7335 117.605 12.7335C117.862 12.7335 118.102 12.6803 118.323 12.5739C118.554 12.4587 118.762 12.3256 118.948 12.1749L119.574 13.1857C119.272 13.4518 118.935 13.6513 118.563 13.7843C118.19 13.9173 117.818 13.9838 117.446 13.9838Z" fill="#2C3236"/>
+<path d="M123.331 13.9838C122.728 13.9838 122.183 13.8508 121.695 13.5848C121.207 13.3099 120.822 12.9197 120.538 12.4143C120.254 11.9 120.112 11.2838 120.112 10.5656C120.112 9.85619 120.254 9.24437 120.538 8.73009C120.83 8.2158 121.207 7.82122 121.668 7.54634C122.13 7.27147 122.613 7.13403 123.118 7.13403C123.712 7.13403 124.209 7.26703 124.608 7.53304C125.007 7.79018 125.308 8.15373 125.512 8.62368C125.716 9.08476 125.818 9.62122 125.818 10.233C125.818 10.5523 125.796 10.8005 125.752 10.9779H121.602C121.673 11.5542 121.881 12.002 122.227 12.3212C122.573 12.6404 123.007 12.8 123.53 12.8C123.814 12.8 124.076 12.7601 124.315 12.6803C124.563 12.5917 124.807 12.472 125.047 12.3212L125.565 13.2789C125.255 13.4828 124.909 13.6513 124.528 13.7843C124.147 13.9173 123.748 13.9838 123.331 13.9838ZM121.589 9.94043H124.488C124.488 9.43501 124.377 9.04043 124.156 8.75669C123.934 8.46408 123.601 8.31777 123.158 8.31777C122.777 8.31777 122.435 8.45964 122.134 8.74339C121.841 9.01826 121.66 9.41728 121.589 9.94043Z" fill="#2C3236"/>
+<path d="M129.101 13.9838C128.658 13.9838 128.215 13.8995 127.771 13.7311C127.328 13.5537 126.947 13.3365 126.627 13.0793L127.346 12.0951C127.638 12.3168 127.931 12.4941 128.223 12.6271C128.516 12.7601 128.826 12.8266 129.154 12.8266C129.509 12.8266 129.771 12.7513 129.939 12.6005C130.108 12.4498 130.192 12.2636 130.192 12.0419C130.192 11.8557 130.121 11.705 129.979 11.5897C129.846 11.4656 129.673 11.3591 129.46 11.2705C129.248 11.1729 129.026 11.0798 128.795 10.9912C128.512 10.8848 128.228 10.7562 127.944 10.6055C127.669 10.4458 127.443 10.2463 127.266 10.0069C127.088 9.75866 127 9.45274 127 9.0892C127 8.51284 127.213 8.04289 127.638 7.67935C128.064 7.3158 128.64 7.13403 129.367 7.13403C129.828 7.13403 130.241 7.21383 130.604 7.37344C130.968 7.53304 131.282 7.71482 131.548 7.91876L130.844 8.84979C130.613 8.68132 130.378 8.54831 130.139 8.45078C129.908 8.34437 129.664 8.29117 129.407 8.29117C129.079 8.29117 128.835 8.36211 128.676 8.50398C128.516 8.63698 128.436 8.80545 128.436 9.0094C128.436 9.26654 128.569 9.46161 128.835 9.59462C129.101 9.72762 129.412 9.85619 129.766 9.98033C130.068 10.0867 130.36 10.2197 130.644 10.3793C130.928 10.5301 131.163 10.7296 131.349 10.9779C131.544 11.2261 131.642 11.5542 131.642 11.9621C131.642 12.5207 131.424 12.9995 130.99 13.3986C130.555 13.7887 129.926 13.9838 129.101 13.9838Z" fill="#2C3236"/>
+</svg>
--- a/docs/src/embeddings/embedding_functions.md
+++ b/docs/src/embeddings/embedding_functions.md
@@ -2,8 +2,8 @@ Representing multi-modal data as vector embeddings is becoming a standard practi

 For this purpose, LanceDB introduces an **embedding functions API**, that allow you simply set up once, during the configuration stage of your project. After this, the table remembers it, effectively making the embedding functions *disappear in the background* so you don't have to worry about manually passing callables, and instead, simply focus on the rest of your data engineering pipeline.

-!!! Note "LanceDB cloud doesn't support embedding functions yet"
-    LanceDB Cloud does not support embedding functions yet. You need to generate embeddings before ingesting into the table or querying.
+!!! Note "Embedding functions on LanceDB cloud"
+    When using embedding functions with LanceDB cloud, the embeddings will be generated on the source device and sent to the cloud. This means that the source device must have the necessary resources to generate the embeddings.

 !!! warning
    Using the embedding function registry means that you don't have to explicitly generate the embeddings yourself.
--- a/docs/src/embeddings/index.md
+++ b/docs/src/embeddings/index.md
@@ -99,28 +99,28 @@ LanceDB registers the Sentence Transformers embeddings function in the registry

    Coming Soon!

-### Jina Embeddings
-
-LanceDB registers the JinaAI embeddings function in the registry as `jina`. You can pass any supported model name to the `create`. By default it uses `"jina-clip-v1"`.
-`jina-clip-v1` can handle both text and images and other models only support `text`.
-
-You need to pass `JINA_API_KEY` in the environment variable or pass it as `api_key` to `create` method.
+### Embedding function with LanceDB cloud
+Embedding functions are now supported on LanceDB cloud. The embeddings will be generated on the source device and sent to the cloud. This means that the source device must have the necessary resources to generate the embeddings. Here's an example using the OpenAI embedding function:

 ```python
 import os
 import lancedb
 from lancedb.pydantic import LanceModel, Vector
 from lancedb.embeddings import get_registry
-os.environ['JINA_API_KEY'] = "jina_*"
+os.environ['OPENAI_API_KEY'] = "..."

-db = lancedb.connect("/tmp/db")
-func = get_registry().get("jina").create(name="jina-clip-v1")
+db = lancedb.connect(
+  uri="db://....",
+  api_key="sk_...",
+  region="us-east-1"
+)
+func = get_registry().get("openai").create()

 class Words(LanceModel):
    text: str = func.SourceField()
    vector: Vector(func.ndims()) = func.VectorField()

-table = db.create_table("words", schema=Words, mode="overwrite")
+table = db.create_table("words", schema=Words)
 table.add(
    [
        {"text": "hello world"},
--- a/docs/src/examples/python_examples/aiagent.md
+++ b/docs/src/examples/python_examples/aiagent.md
@@ -0,0 +1,27 @@
+# AI Agents: Intelligent Collaboration🤖
+
+Think of a platform💻 where AI Agents🤖 can seamlessly exchange information, coordinate over tasks, and achieve shared targets with great efficiency📈🚀.
+
+## Vector-Based Coordination: The Technical Advantage
+Leveraging LanceDB's vector-based capabilities, our coordination application enables AI agents to communicate and collaborate through dense vector representations 🤖. AI agents can exchange information, coordinate on a task or work towards a common goal, just by giving queries📝.
+
+| **AI Agents** | **Description** | **Links** |
+|:--------------|:----------------|:----------|
+| **AI Agents: Reducing Hallucinationt📊** | 🤖💡 Reduce AI hallucinations using Critique-Based Contexting! Learn by Simplifying and Automating tedious workflows by going through fitness trainer agent example.💪 | [![Github](../../assets/github.svg)][hullucination_github] <br>[![Open In Collab](../../assets/colab.svg)][hullucination_colab] <br>[![Python](../../assets/python.svg)][hullucination_python] <br>[![Ghost](../../assets/ghost.svg)][hullucination_ghost] |                                                                                                                                                                                                                                                
+| **AI Trends Searcher: CrewAI🔍️** | 🔍️ Learn about CrewAI Agents ! Utilize the features of CrewAI - Role-based Agents, Task Management, and Inter-agent Delegation ! Make AI agents work together to do tricky stuff 😺| [![Github](../../assets/github.svg)][trend_github] <br>[![Open In Collab](../../assets/colab.svg)][trend_colab] <br>[![Ghost](../../assets/ghost.svg)][trend_ghost] |                                                                                                                                                                                                                                                
+| **SuperAgent Autogen🤖** | 💻 AI interactions with the Super Agent! Integrating Autogen, LanceDB, LangChain, LiteLLM, and Ollama to create AI agent that excels in understanding and processing complex queries.🤖 | [![Github](../../assets/github.svg)][superagent_github] <br>[![Open In Collab](../../assets/colab.svg)][superagent_colab] |
+
+
+[hullucination_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/reducing_hallucinations_ai_agents
+[hullucination_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/reducing_hallucinations_ai_agents/main.ipynb
+[hullucination_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/reducing_hallucinations_ai_agents/main.py
+[hullucination_ghost]: https://blog.lancedb.com/how-to-reduce-hallucinations-from-llm-powered-agents-using-long-term-memory-72f262c3cc1f/
+
+[trend_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/AI-Trends-with-CrewAI
+[trend_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/AI-Trends-with-CrewAI/CrewAI_AI_Trends.ipynb 
+[trend_ghost]: https://blog.lancedb.com/track-ai-trends-crewai-agents-rag/
+
+[superagent_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen
+[superagent_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen/main.ipynb
+
+
--- a/docs/src/examples/python_examples/build_from_scratch.md
+++ b/docs/src/examples/python_examples/build_from_scratch.md
@@ -1,8 +1,8 @@
-# Build from Scratch with LanceDB 🚀
+# **Build from Scratch with LanceDB 🛠️🚀**

-Start building your GenAI applications from the ground up using LanceDB's efficient vector-based document retrieval capabilities! 📄
+Start building your GenAI applications from the ground up using LanceDB's efficient vector-based document retrieval capabilities! 📑

-#### Get Started in Minutes ⏱️
+**Get Started in Minutes ⏱️**

 These examples provide a solid foundation for building your own GenAI applications using LanceDB. Jump from idea to proof of concept quickly with applied examples. Get started and see what you can create! 💻

--- a/docs/src/examples/python_examples/chatbot.md
+++ b/docs/src/examples/python_examples/chatbot.md
@@ -0,0 +1,41 @@
+**Chatbot Application with LanceDB 🤖**
+====================================================================
+
+ Create an innovative chatbot application that utilizes LanceDB for efficient vector-based response generation! 🌐✨ 
+ 
+**Introduction 👋✨**
+
+ Users can input their queries, allowing the chatbot to retrieve relevant context seamlessly. 🔍📚 This enables the generation of coherent and context-aware replies that enhance user experience. 🌟🤝 Dive into the world of advanced conversational AI and streamline interactions with powerful data management! 🚀💡
+
+
+| **Chatbot**  | **Description**  | **Links**  |
+|:----------------|:-----------------|:-----------|
+| **Databricks DBRX Website Bot ⚡️** | Unlock magical conversations with the Hogwarts chatbot, powered by Open-source RAG, DBRX, LanceDB, LLama-index, and Hugging Face Embeddings, delivering enchanting user experiences and spellbinding interactions ✨ | [![GitHub](../../assets/github.svg)][databricks_github] <br>[![Python](../../assets/python.svg)][databricks_python] |
+| **CLI SDK Manual Chatbot Locally 💻** | CLI chatbot for SDK/hardware documents, powered by Local RAG, LLama3, Ollama, LanceDB, and Openhermes Embeddings, built with Phidata Assistant and Knowledge Base for instant technical support 🤖 | [![GitHub](../../assets/github.svg)][clisdk_github] <br>[![Python](../../assets/python.svg)][clisdk_python] |
+| **Youtube Transcript Search QA Bot 📹** | Unlock the power of YouTube transcripts with a Q&A bot, leveraging natural language search and LanceDB for effortless data management and instant answers 💬 | [![GitHub](../../assets/github.svg)][youtube_github] <br>[![Open In Collab](../../assets/colab.svg)][youtube_colab] <br>[![Python](../../assets/python.svg)][youtube_python] |
+| **Code Documentation Q&A Bot with LangChain 🤖** | Revolutionize code documentation with a Q&A bot, powered by LangChain and LanceDB, allowing effortless querying of documentation using natural language, demonstrated with Numpy 1.26 docs 📚 | [![GitHub](../../assets/github.svg)][docs_github] <br>[![Open In Collab](../../assets/colab.svg)][docs_colab] <br>[![Python](../../assets/python.svg)][docs_python] |
+| **Context-aware Chatbot using Llama 2 & LanceDB 🤖** | Experience the future of conversational AI with a context-aware chatbot, powered by Llama 2, LanceDB, and LangChain, enabling intuitive and meaningful conversations with your data 📚💬 | [![GitHub](../../assets/github.svg)][aware_github] <br>[![Open In Collab](../../assets/colab.svg)][aware_colab] <br>[![Ghost](../../assets/ghost.svg)][aware_ghost] |
+| **Chat with csv using Hybrid Search 📊** | Revolutionize data interaction with a chat application that harnesses LanceDB's hybrid search capabilities to converse with CSV and Excel files, enabling efficient and scalable data exploration and analysis 🚀 | [![GitHub](../../assets/github.svg)][csv_github] <br>[![Open In Collab](../../assets/colab.svg)][csv_colab] <br>[![Ghost](../../assets/ghost.svg)][csv_ghost] |
+
+
+[databricks_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/databricks_DBRX_website_bot
+[databricks_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/databricks_DBRX_website_bot/main.py
+
+[clisdk_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/CLI-SDK-Manual-Chatbot-Locally
+[clisdk_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/CLI-SDK-Manual-Chatbot-Locally/assistant.py
+
+[youtube_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Youtube-Search-QA-Bot
+[youtube_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Youtube-Search-QA-Bot/main.ipynb
+[youtube_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Youtube-Search-QA-Bot/main.py
+
+[docs_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot
+[docs_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.ipynb
+[docs_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.py
+
+[aware_github]: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/chatbot_using_Llama2_&_lanceDB
+[aware_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/chatbot_using_Llama2_&_lanceDB/main.ipynb
+[aware_ghost]: https://blog.lancedb.com/context-aware-chatbot-using-llama-2-lancedb-as-vector-database-4d771d95c755
+
+[csv_github]: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/Chat_with_csv_file
+[csv_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Chat_with_csv_file/main.ipynb
+[csv_ghost]: https://blog.lancedb.com/p/d8c71df4-e55f-479a-819e-cde13354a6a3/
--- a/docs/src/examples/python_examples/evaluations.md
+++ b/docs/src/examples/python_examples/evaluations.md
@@ -0,0 +1,23 @@
+**Evaluation: Assessing Text Performance with Precision 📊💡**
+====================================================================
+
+**Evaluation Fundamentals 📊**
+
+Evaluation is a comprehensive tool designed to measure the performance of text-based inputs, enabling data-driven optimization and improvement 📈. 
+
+**Text Evaluation 101 📚**
+
+By leveraging cutting-edge technologies, this provides a robust framework for evaluating reference and candidate texts across various metrics 📊, ensuring high-quality text outputs that meet specific requirements and standards 📝.
+
+| **Evaluation** | **Description** | **Links** |
+| -------------- | --------------- | --------- |
+| **Evaluating Prompts with Prompttools 🤖** | Compare, visualize & evaluate embedding functions (incl. OpenAI) across metrics like latency & custom evaluation 📈📊 | [![Github](../../assets/github.svg)][prompttools_github] <br>[![Open In Collab](../../assets/colab.svg)][prompttools_colab] |
+| **Evaluating RAG with RAGAs and GPT-4o 📊** | Evaluate RAG pipelines with cutting-edge metrics and tools, integrate with CI/CD for continuous performance checks, and generate responses with GPT-4o 🤖📈 | [![Github](../../assets/github.svg)][RAGAs_github] <br>[![Open In Collab](../../assets/colab.svg)][RAGAs_colab] |
+
+
+
+[prompttools_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/prompttools-eval-prompts
+[prompttools_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/prompttools-eval-prompts/main.ipynb
+
+[RAGAs_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Evaluating_RAG_with_RAGAs
+[RAGAs_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Evaluating_RAG_with_RAGAs/Evaluating_RAG_with_RAGAs.ipynb
--- a/docs/src/examples/python_examples/multimodal.md
+++ b/docs/src/examples/python_examples/multimodal.md
@@ -1,10 +1,10 @@
-# Multimodal Search with LanceDB 🔍💡
+# **Multimodal Search with LanceDB 🤹‍♂️🔍**

-Experience the future of search with LanceDB's multimodal capabilities. Combine text and image queries to find the most relevant results in your corpus and unlock new possibilities! 🔓💡
+Experience the future of search with LanceDB's multimodal capabilities. Combine text and image queries to find the most relevant results in your corpus ! 🔓💡

-#### Explore the Future of Search 🚀
+**Explore the Future of Search 🚀**

-Unlock the power of multimodal search with LanceDB, enabling efficient vector-based retrieval of text and image data! 📊💻
+LanceDB supports multimodal search by indexing and querying vector representations of text and image data 🤖. This enables efficient retrieval of relevant documents and images using vector-based similarity search 📊. The platform facilitates cross-modal search, allowing for text-image and image-text retrieval, and supports scalable indexing of high-dimensional vector spaces 💻.



@@ -13,7 +13,7 @@ Unlock the power of multimodal search with LanceDB, enabling efficient vector-ba
 | **Multimodal CLIP: DiffusionDB 🌐💥**    | Revolutionize search with Multimodal CLIP and DiffusionDB, combining text and image understanding for a new dimension of discovery! 🔓 | [![GitHub](../../assets/github.svg)][Clip_diffusionDB_github] <br>[![Open In Collab](../../assets/colab.svg)][Clip_diffusionDB_colab] <br>[![Python](../../assets/python.svg)][Clip_diffusionDB_python] <br>[![Ghost](../../assets/ghost.svg)][Clip_diffusionDB_ghost] |
 | **Multimodal CLIP: Youtube Videos 📹👀** | Search Youtube videos using Multimodal CLIP, finding relevant content with ease and accuracy! 🎯                                                                                                                                                                | [![Github](../../assets/github.svg)][Clip_youtube_github]                                                                                                    <br>[![Open In Collab](../../assets/colab.svg)][Clip_youtube_colab]                                                                                           <br> [![Python](../../assets/python.svg)][Clip_youtube_python]         <br>[![Ghost](../../assets/ghost.svg)][Clip_youtube_python] |
 | **Multimodal Image + Text Search 📸🔍**               | Discover relevant documents and images with a single query, using LanceDB's multimodal search capabilities to bridge the gap between text and visuals! 🌉 | [![GitHub](../../assets/github.svg)](https://github.com/lancedb/vectordb-recipes/blob/main/examples/multimodal_search) <br>[![Open In Collab](../../assets/colab.svg)](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multimodal_search/main.ipynb) <br> [![Python](../../assets/python.svg)](https://github.com/lancedb/vectordb-recipes/blob/main/examples/multimodal_search/main.py)<br> [![Ghost](../../assets/ghost.svg)](https://blog.lancedb.com/multi-modal-ai-made-easy-with-lancedb-clip-5aaf8801c939/) |
-| **Cambrian-1: Vision-Centric Image Exploration 🔍👀** | Dive into vision-centric exploration of images with Cambrian-1, powered by LanceDB's multimodal search to uncover new insights! 🔎                        | [![GitHub](../../assets/github.svg)](https://www.kaggle.com/code/prasantdixit/cambrian-1-vision-centric-exploration-of-images/)<br>[![Open In Collab](../../assets/colab.svg)]() <br> [![Python](../../assets/python.svg)]() <br> [![Ghost](../../assets/ghost.svg)](https://blog.lancedb.com/cambrian-1-vision-centric-exploration/)                                                                                                                                                                                                                   |
+| **Cambrian-1: Vision-Centric Image Exploration 🔍👀** | Dive into vision-centric exploration of images with Cambrian-1, powered by LanceDB's multimodal search to uncover new insights! 🔎                        | [![Kaggle](https://img.shields.io/badge/Kaggle-035a7d?style=for-the-badge&logo=kaggle&logoColor=white)](https://www.kaggle.com/code/prasantdixit/cambrian-1-vision-centric-exploration-of-images/)<br> [![Ghost](../../assets/ghost.svg)](https://blog.lancedb.com/cambrian-1-vision-centric-exploration/)                                                                                                                                                                                                                   |


 [Clip_diffusionDB_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multimodal_clip_diffusiondb
--- a/docs/src/examples/python_examples/rag.md
+++ b/docs/src/examples/python_examples/rag.md
@@ -1,13 +1,12 @@

-**🔍💡 RAG: Revolutionize Information Retrieval with LanceDB 🔓**
+**RAG: Revolutionize Information Retrieval with LanceDB 🔓🧐**
 ====================================================================

-Unlock the full potential of Retrieval-Augmented Generation (RAG) with LanceDB, the ultimate solution for efficient vector-based information retrieval 📊. Input text queries and retrieve relevant documents with lightning-fast speed ⚡️ and accuracy ✅. Generate comprehensive answers by combining retrieved information, uncovering new insights 🔍 and connections.
+Unlock the full potential of Retrieval-Augmented Generation (RAG) with LanceDB, a solution for efficient vector-based information retrieval 📊. 

-### Experience the Future of Search 🔄
-
-Experience the future of search with RAG, transforming information retrieval and answer generation. Apply RAG to various industries, streamlining processes 📈, saving time ⏰, and resources 💰. Stay ahead of the curve with innovative technology 🔝, powered by LanceDB. Discover the power of RAG with LanceDB and transform your industry with innovative solutions 💡.
+**Experience the Future of Search 🔄**

+RAG integrates large language models (LLMs) with scalable knowledge bases, enabling efficient information retrieval and answer generation 🤖. By applying RAG to industry-specific use cases, developers can optimize query processing 📊, reduce response latency ⏱️, and improve resource utilization 💻. LanceDB provides a robust framework for integrating LLMs with external knowledge sources, facilitating accurate and informative responses 📝.

 | **RAG**                                      | **Description**                                                                                                                                                  | **Links**                  |
 |----------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------|                                                                                                                                                     
--- a/docs/src/examples/python_examples/vector_search.md
+++ b/docs/src/examples/python_examples/vector_search.md
@@ -0,0 +1,80 @@
+**Vector Search: Unlock Efficient Document Retrieval 🔓👀**
+====================================================================
+
+Unlock the power of vector search with LanceDB, a cutting-edge solution for efficient vector-based document retrieval 📊. 
+
+**Vector Search Capabilities in LanceDB🔝**
+
+LanceDB implements vector search algorithms for efficient document retrieval and analysis 📊. This enables fast and accurate discovery of relevant documents, leveraging dense vector representations 🤖. The platform supports scalable indexing and querying of high-dimensional vector spaces, facilitating precise document matching and retrieval 📈.
+
+| **Vector Search** | **Description** | **Links** |
+|:-----------------|:---------------|:---------|
+| **Inbuilt Hybrid Search 🔄** | Combine the power of traditional search algorithms with LanceDB's vector-based search for a robust and efficient search experience 📊 | [![Github](../../assets/github.svg)][inbuilt_hybrid_search_github] <br>[![Open In Collab](../../assets/colab.svg)][inbuilt_hybrid_search_colab] |                                                                                                                                                                                                                                                                                                    
+| **Hybrid Search with BM25 and LanceDB 💡** | Synergizes BM25's keyword-focused precision (term frequency, document length normalization, bias-free retrieval) with LanceDB's semantic understanding (contextual analysis, query intent alignment) for nuanced search results in complex datasets 📈 | [![Github](../../assets/github.svg)][BM25_github] <br>[![Open In Collab](../../assets/colab.svg)][BM25_colab] <br>[![Ghost](../../assets/ghost.svg)][BM25_ghost] |                                                                                                                                                                                                                                                                                                    
+| **NER-powered Semantic Search 🔎** | Unlock contextual understanding with Named Entity Recognition (NER) methods: Dictionary-Based, Rule-Based, and Deep Learning-Based, to accurately identify and extract entities, enabling precise semantic search results 🗂️ | [![Github](../../assets/github.svg)][NER_github] <br>[![Open In Collab](../../assets/colab.svg)][NER_colab] <br>[![Ghost](../../assets/ghost.svg)][NER_ghost]|                                                                                                                                                                                                                                                                                                    
+| **Audio Similarity Search using Vector Embeddings 🎵** | Create vector embeddings of audio files to find similar audio content, enabling efficient audio similarity search and retrieval in LanceDB's vector store 📻 |[![Github](../../assets/github.svg)][audio_search_github] <br>[![Open In Collab](../../assets/colab.svg)][audio_search_colab] <br>[![Python](../../assets/python.svg)][audio_search_python]|                                                                                                                                                                                                                                                                                                    
+| **LanceDB Embeddings API: Multi-lingual Semantic Search 🌎** | Build a universal semantic search table with LanceDB's Embeddings API, supporting multiple languages (e.g., English, French) using cohere's multi-lingual model, for accurate cross-lingual search results 📄 | [![Github](../../assets/github.svg)][mls_github] <br>[![Open In Collab](../../assets/colab.svg)][mls_colab] <br>[![Python](../../assets/python.svg)][mls_python] |                                                                                                                                                                                                                                                                                                    
+| **Facial Recognition: Face Embeddings 🤖** | Detect, crop, and embed faces using Facenet, then store and query face embeddings in LanceDB for efficient facial recognition and top-K matching results 👥 | [![Github](../../assets/github.svg)][fr_github] <br>[![Open In Collab](../../assets/colab.svg)][fr_colab] |                                                                                                                                                                                                                                                                                                    
+| **Sentiment Analysis: Hotel Reviews 🏨** | Analyze customer sentiments towards the hotel industry using BERT models, storing sentiment labels, scores, and embeddings in LanceDB, enabling queries on customer opinions and potential areas for improvement 💬 | [![Github](../../assets/github.svg)][sentiment_analysis_github] <br>[![Open In Collab](../../assets/colab.svg)][sentiment_analysis_colab] <br>[![Ghost](../../assets/ghost.svg)][sentiment_analysis_ghost] |                                                                                                                                                                                                                                                                                                    
+| **Vector Arithmetic with LanceDB ⚖️** | Unlock powerful semantic search capabilities by performing vector arithmetic on embeddings, enabling complex relationships and nuances in data to be captured, and simplifying the process of retrieving semantically similar results 📊 | [![Github](../../assets/github.svg)][arithmetic_github] <br>[![Open In Collab](../../assets/colab.svg)][arithmetic_colab] <br>[![Ghost](../../assets/ghost.svg)][arithmetic_ghost] |                                                                                                                                                                                                                                                                                                    
+| **Imagebind Demo 🖼️** | Explore the multi-modal capabilities of Imagebind through a Gradio app, leveraging LanceDB API for seamless image search and retrieval experiences 📸 | [![Github](../../assets/github.svg)][imagebind_github] <br> [![Open in Spaces](../../assets/open_hf_space.svg)][imagebind_huggingface] |                                                                                                                                                                                                                                                                                                    
+| **Search Engine using SAM & CLIP 🔍** | Build a search engine within an image using SAM and CLIP models, enabling object-level search and retrieval, with LanceDB indexing and search capabilities to find the closest match between image embeddings and user queries 📸 | [![Github](../../assets/github.svg)][swi_github] <br>[![Open In Collab](../../assets/colab.svg)][swi_colab] <br>[![Ghost](../../assets/ghost.svg)][swi_ghost] |                                                                                                                                                                                                                                                                                                    
+| **Zero Shot Object Localization and Detection with CLIP 🔎** | Perform object detection on images using OpenAI's CLIP, enabling zero-shot localization and detection of objects, with capabilities to split images into patches, parse with CLIP, and plot bounding boxes 📊 | [![Github](../../assets/github.svg)][zsod_github] <br>[![Open In Collab](../../assets/colab.svg)][zsod_colab] |                                                                                                                                                                                                                                                                                                    
+| **Accelerate Vector Search with OpenVINO 🚀** | Boost vector search applications using OpenVINO, achieving significant speedups with CLIP for text-to-image and image-to-image searching, through PyTorch model optimization, FP16 and INT8 format conversion, and quantization with OpenVINO NNCF 📈 | [![Github](../../assets/github.svg)][openvino_github] <br>[![Open In Collab](../../assets/colab.svg)][openvino_colab] <br>[![Ghost](../../assets/ghost.svg)][openvino_ghost] |                                                                                                                                                                                                                                                                                                    
+| **Zero-Shot Image Classification with CLIP and LanceDB 📸** | Achieve zero-shot image classification using CLIP and LanceDB, enabling models to classify images without prior training on specific use cases, unlocking flexible and adaptable image classification capabilities 🔓 | [![Github](../../assets/github.svg)][zsic_github] <br>[![Open In Collab](../../assets/colab.svg)][zsic_colab] <br>[![Ghost](../../assets/ghost.svg)][zsic_ghost] |                                                                                                                                                                                                                                                                                                    
+
+
+
+
+[inbuilt_hybrid_search_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Inbuilt-Hybrid-Search
+[inbuilt_hybrid_search_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Inbuilt-Hybrid-Search/Inbuilt_Hybrid_Search_with_LanceDB.ipynb
+
+[BM25_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Hybrid_search_bm25_lancedb
+[BM25_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Hybrid_search_bm25_lancedb/main.ipynb
+[BM25_ghost]: https://blog.lancedb.com/hybrid-search-combining-bm25-and-semantic-search-for-better-results-with-lan-1358038fe7e6
+
+[NER_github]: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search
+[NER_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search/NER_powered_Semantic_Search_with_LanceDB.ipynb
+[NER_ghost]: https://blog.lancedb.com/ner-powered-semantic-search-using-lancedb-51051dc3e493
+
+[audio_search_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/audio_search
+[audio_search_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/audio_search/main.ipynb
+[audio_search_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/audio_search/main.py
+
+[mls_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-lingual-wiki-qa
+[mls_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multi-lingual-wiki-qa/main.ipynb
+[mls_python]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-lingual-wiki-qa/main.py
+
+[fr_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/facial_recognition
+[fr_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/facial_recognition/main.ipynb
+
+[sentiment_analysis_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Sentiment-Analysis-Analyse-Hotel-Reviews
+[sentiment_analysis_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Sentiment-Analysis-Analyse-Hotel-Reviews/Sentiment_Analysis_using_LanceDB.ipynb
+[sentiment_analysis_ghost]: https://blog.lancedb.com/sentiment-analysis-using-lancedb-2da3cb1e3fa6
+
+[arithmetic_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Vector-Arithmetic-with-LanceDB
+[arithmetic_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Vector-Arithmetic-with-LanceDB/main.ipynb
+[arithmetic_ghost]: https://blog.lancedb.com/vector-arithmetic-with-lancedb-an-intro-to-vector-embeddings/
+
+[imagebind_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/imagebind_demo
+[imagebind_huggingface]: https://huggingface.co/spaces/raghavd99/imagebind2
+
+[swi_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/search-within-images-with-sam-and-clip
+[swi_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/search-within-images-with-sam-and-clip/main.ipynb
+[swi_ghost]: https://blog.lancedb.com/search-within-an-image-331b54e4285e
+
+[zsod_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/zero-shot-object-detection-CLIP
+[zsod_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/zero-shot-object-detection-CLIP/zero_shot_object_detection_clip.ipynb
+
+[openvino_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO
+[openvino_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/clip_text_image_search.ipynb
+[openvino_ghost]: https://blog.lancedb.com/accelerate-vector-search-applications-using-openvino-lancedb/
+
+[zsic_github]: https://github.com/lancedb/vectordb-recipes/blob/main/examples/zero-shot-image-classification
+[zsic_colab]: https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/zero-shot-image-classification/main.ipynb
+[zsic_ghost]: https://blog.lancedb.com/zero-shot-image-classification-with-vector-search/
+
+
+
+
+
--- a/docs/src/fts.md
+++ b/docs/src/fts.md
@@ -1,9 +1,14 @@
 # Full-text search

-LanceDB provides support for full-text search via [Tantivy](https://github.com/quickwit-oss/tantivy) (currently Python only), allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions. Our goal is to push the FTS integration down to the Rust level in the future, so that it's available for Rust and JavaScript users as well.  Follow along at [this Github issue](https://github.com/lancedb/lance/issues/1195)
+LanceDB provides support for full-text search via Lance (before via [Tantivy](https://github.com/quickwit-oss/tantivy) (Python only)), allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.
+
+Currently, the Lance full text search is missing some features that are in the Tantivy full text search. This includes phrase queries, re-ranking, and customizing the tokenizer. Thus, in Python, Tantivy is still the default way to do full text search and many of the instructions below apply just to Tantivy-based indices.


-## Installation
+## Installation (Only for Tantivy-based FTS)
+
+!!! note
+    No need to install the tantivy dependency if using native FTS

 To use full-text search, install the dependency [`tantivy-py`](https://github.com/quickwit-oss/tantivy-py):

@@ -14,42 +19,83 @@ pip install tantivy==0.20.1

 ## Example

-Consider that we have a LanceDB table named `my_table`, whose string column `text` we want to index and query via keyword search.
+Consider that we have a LanceDB table named `my_table`, whose string column `text` we want to index and query via keyword search, the FTS index must be created before you can search via keywords.

-```python
-import lancedb
+=== "Python"

-uri = "data/sample-lancedb"
-db = lancedb.connect(uri)
+    ```python
+    import lancedb

-table = db.create_table(
-    "my_table",
-    data=[
-        {"vector": [3.1, 4.1], "text": "Frodo was a happy puppy"},
-        {"vector": [5.9, 26.5], "text": "There are several kittens playing"},
-    ],
-)
-```
+    uri = "data/sample-lancedb"
+    db = lancedb.connect(uri)

-## Create FTS index on single column
+    table = db.create_table(
+        "my_table",
+        data=[
+            {"vector": [3.1, 4.1], "text": "Frodo was a happy puppy"},
+            {"vector": [5.9, 26.5], "text": "There are several kittens playing"},
+        ],
+    )

-The FTS index must be created before you can search via keywords.
+    # passing `use_tantivy=False` to use lance FTS index
+    # `use_tantivy=True` by default
+    table.create_fts_index("text")
+    table.search("puppy").limit(10).select(["text"]).to_list()
+    # [{'text': 'Frodo was a happy puppy', '_score': 0.6931471824645996}]
+    # ...
+    ```

-```python
-table.create_fts_index("text")
-```
+=== "TypeScript"

-To search an FTS index via keywords, LanceDB's `table.search` accepts a string as input:
+    ```typescript
+    import * as lancedb from "@lancedb/lancedb";
+    const uri = "data/sample-lancedb"
+    const db = await lancedb.connect(uri);

-```python
-table.search("puppy").limit(10).select(["text"]).to_list()
-```
+    const data = [
+    { vector: [3.1, 4.1], text: "Frodo was a happy puppy" },
+    { vector: [5.9, 26.5], text: "There are several kittens playing" },
+    ];
+    const tbl = await db.createTable("my_table", data, { mode: "overwrite" });
+    await tbl.createIndex("text", {
+        config: lancedb.Index.fts(),
+    });

-This returns the result as a list of dictionaries as follows.
+    await tbl
+        .search("puppy")
+        .select(["text"])
+        .limit(10)
+        .toArray();
+    ```

-```python
-[{'text': 'Frodo was a happy puppy', 'score': 0.6931471824645996}]
-```
+=== "Rust"
+
+    ```rust
+    let uri = "data/sample-lancedb";
+    let db = connect(uri).execute().await?;
+    let initial_data: Box<dyn RecordBatchReader + Send> = create_some_records()?;
+    let tbl = db
+        .create_table("my_table", initial_data)
+        .execute()
+        .await?;
+    tbl
+        .create_index(&["text"], Index::FTS(FtsIndexBuilder::default()))
+        .execute()
+        .await?;
+
+    tbl
+        .query()
+        .full_text_search(FullTextSearchQuery::new("puppy".to_owned()))
+        .select(lancedb::query::Select::Columns(vec!["text".to_owned()]))
+        .limit(10)
+        .execute()
+        .await?;
+    ```
+
+It would search on all indexed columns by default, so it's useful when there are multiple indexed columns.
+For now, this is supported in tantivy way only.
+
+Passing `fts_columns="text"` if you want to specify the columns to search, but it's not available for Tantivy-based full text search.

 !!! note
    LanceDB automatically searches on the existing FTS index if the input to the search is of type `str`. If you provide a vector as input, LanceDB will search the ANN index instead.
@@ -57,20 +103,33 @@ This returns the result as a list of dictionaries as follows.
 ## Tokenization
 By default the text is tokenized by splitting on punctuation and whitespaces and then removing tokens that are longer than 40 chars. For more language specific tokenization then provide the argument tokenizer_name with the 2 letter language code followed by "_stem". So for english it would be "en_stem".

-```python
-table.create_fts_index("text", tokenizer_name="en_stem")
-```
+For now, only the Tantivy-based FTS index supports to specify the tokenizer, so it's only available in Python with `use_tantivy=True`.

-The following [languages](https://docs.rs/tantivy/latest/tantivy/tokenizer/enum.Language.html) are currently supported.
+=== "use_tantivy=True"

+    ```python
+    table.create_fts_index("text", use_tantivy=True, tokenizer_name="en_stem")
+    ```
+
+=== "use_tantivy=False"
+
+    [**Not supported yet**](https://github.com/lancedb/lance/issues/1195)
+
+the following [languages](https://docs.rs/tantivy/latest/tantivy/tokenizer/enum.Language.html) are currently supported.

 ## Index multiple columns

 If you have multiple string columns to index, there's no need to combine them manually -- simply pass them all as a list to `create_fts_index`:

-```python
-table.create_fts_index(["text1", "text2"])
-```
+=== "use_tantivy=True"
+
+    ```python
+    table.create_fts_index(["text1", "text2"])
+    ```
+
+=== "use_tantivy=False"
+
+    [**Not supported yet**](https://github.com/lancedb/lance/issues/1195)

 Note that the search API call does not change - you can search over all indexed columns at once.

@@ -80,19 +139,48 @@ Currently the LanceDB full text search feature supports *post-filtering*, meanin
 applied on top of the full text search results. This can be invoked via the familiar
 `where` syntax:

-```python
-table.search("puppy").limit(10).where("meta='foo'").to_list()
-```
+=== "Python"
+
+    ```python
+    table.search("puppy").limit(10).where("meta='foo'").to_list()
+    ```
+
+=== "TypeScript"
+
+    ```typescript
+    await tbl
+    .search("apple")
+    .select(["id", "doc"])
+    .limit(10)
+    .where("meta='foo'")
+    .toArray();
+    ```
+
+=== "Rust"
+
+    ```rust
+    table
+        .query()
+        .full_text_search(FullTextSearchQuery::new(words[0].to_owned()))
+        .select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
+        .limit(10)
+        .only_if("meta='foo'")
+        .execute()
+        .await?;
+    ```

 ## Sorting

+!!! warning "Warn"
+    Sorting is available for only Tantivy-based FTS
+
 You can pre-sort the documents by specifying `ordering_field_names` when
 creating the full-text search index. Once pre-sorted, you can then specify
 `ordering_field_name` while searching to return results sorted by the given
-field. For example, 
+field. For example,

-```
-table.create_fts_index(["text_field"], ordering_field_names=["sort_by_field"])
+```python
+table.create_fts_index(["text_field"], use_tantivy=True, ordering_field_names=["sort_by_field"])

 (table.search("terms", ordering_field_name="sort_by_field")
 .limit(20)
@@ -105,8 +193,8 @@ table.create_fts_index(["text_field"], ordering_field_names=["sort_by_field"])
    error will be raised that looks like `ValueError: The field does not exist: xxx`

 !!! note
-    The fields to sort on must be of typed unsigned integer, or else you will see 
-    an error during indexing that looks like 
+    The fields to sort on must be of typed unsigned integer, or else you will see
+    an error during indexing that looks like
    `TypeError: argument 'value': 'float' object cannot be interpreted as an integer`.

 !!! note
@@ -116,6 +204,9 @@ table.create_fts_index(["text_field"], ordering_field_names=["sort_by_field"])

 ## Phrase queries vs. terms queries

+!!! warning "Warn"
+    Phrase queries are available for only Tantivy-based FTS
+
 For full-text search you can specify either a **phrase** query like `"the old man and the sea"`,
 or a **terms** search query like `"(Old AND Man) AND Sea"`. For more details on the terms
 query syntax, see Tantivy's [query parser rules](https://docs.rs/tantivy/latest/tantivy/query/struct.QueryParser.html).
@@ -142,7 +233,7 @@ enforce it in one of two ways:

 1. Place the double-quoted query inside single quotes. For example, `table.search('"they could have been dogs OR cats"')` is treated as
 a phrase query.
-2. Explicitly declare the `phrase_query()` method. This is useful when you have a phrase query that
+1. Explicitly declare the `phrase_query()` method. This is useful when you have a phrase query that
 itself contains double quotes. For example, `table.search('the cats OR dogs were not really "pets" at all').phrase_query()`
 is treated as a phrase query.

@@ -150,7 +241,7 @@ In general, a query that's declared as a phrase query will be wrapped in double
 double quotes replaced by single quotes.


-## Configurations
+## Configurations (Only for Tantivy-based FTS)

 By default, LanceDB configures a 1GB heap size limit for creating the index. You can
 reduce this if running on a smaller node, or increase this for faster performance while
@@ -164,6 +255,8 @@ table.create_fts_index(["text1", "text2"], writer_heap_size=heap, replace=True)

 ## Current limitations

+For that Tantivy-based FTS:
+
 1. Currently we do not yet support incremental writes.
   If you add data after FTS index creation, it won't be reflected
   in search results until you do a full reindex.
--- a/docs/src/guides/scalar_index.md
+++ b/docs/src/guides/scalar_index.md
@@ -0,0 +1,108 @@
+# Building Scalar Index
+
+Similar to many SQL databases, LanceDB supports several types of Scalar indices to accelerate search
+over scalar columns.
+
+- `BTREE`: The most common type is BTREE. This index is inspired by the btree data structure
+  although only the first few layers of the btree are cached in memory.
+  It will perform well on columns with a large number of unique values and few rows per value.
+- `BITMAP`: this index stores a bitmap for each unique value in the column.
+  This index is useful for columns with a finite number of unique values and many rows per value.
+  For example, columns that represent "categories", "labels", or "tags"
+- `LABEL_LIST`: a special index that is used to index list columns whose values have a finite set of possibilities.
+  For example, a column that contains lists of tags (e.g. `["tag1", "tag2", "tag3"]`) can be indexed with a `LABEL_LIST` index.
+
+| Data Type                                                       | Filter                                    | Index Type   |
+| --------------------------------------------------------------- | ----------------------------------------- | ------------ |
+| Numeric, String, Temporal                                       | `<`, `=`, `>`, `in`, `between`, `is null` | `BTREE`      |
+| Boolean, numbers or strings with fewer than 1,000 unique values | `<`, `=`, `>`, `in`, `between`, `is null` | `BITMAP`     |
+| List of low cardinality of numbers or strings                   | `array_has_any`, `array_has_all`          | `LABEL_LIST` |
+
+=== "Python"
+
+    ```python
+    import lancedb
+    books = [
+      {"book_id": 1, "publisher": "plenty of books", "tags": ["fantasy", "adventure"]},
+      {"book_id": 2, "publisher": "book town", "tags": ["non-fiction"]},
+      {"book_id": 3, "publisher": "oreilly", "tags": ["textbook"]}
+    ]
+
+    db = lancedb.connect("./db")
+    table = db.create_table("books", books)
+    table.create_scalar_index("book_id")  # BTree by default
+    table.create_scalar_index("publisher", index_type="BITMAP")
+    ```
+
+=== "Typescript"
+
+    === "@lancedb/lancedb"
+
+        ```js
+        const db = await lancedb.connect("data");
+        const tbl = await db.openTable("my_vectors");
+
+        await tbl.create_index("book_id");
+        await tlb.create_index("publisher", { config: lancedb.Index.bitmap() })
+        ```
+
+For example, the following scan will be faster if the column `my_col` has a scalar index:
+
+=== "Python"
+
+    ```python
+    import lancedb
+
+    table = db.open_table("books")
+    my_df = table.search().where("book_id = 2").to_pandas()
+    ```
+
+=== "Typescript"
+
+    === "@lancedb/lancedb"
+
+        ```js
+        const db = await lancedb.connect("data");
+        const tbl = await db.openTable("books");
+
+        await tbl
+          .query()
+          .where("book_id = 2")
+          .limit(10)
+          .toArray();
+        ```
+
+Scalar indices can also speed up scans containing a vector search or full text search, and a prefilter:
+
+=== "Python"
+
+    ```python
+    import lancedb
+
+    data = [
+      {"book_id": 1, "vector": [1, 2]},
+      {"book_id": 2, "vector": [3, 4]},
+      {"book_id": 3, "vector": [5, 6]}
+    ]
+    table = db.create_table("book_with_embeddings", data)
+
+    (
+        table.search([1, 2])
+        .where("book_id != 3", prefilter=True)
+        .to_pandas()
+    )
+    ```
+
+=== "Typescript"
+
+    === "@lancedb/lancedb"
+
+        ```js
+        const db = await lancedb.connect("data/lance");
+        const tbl = await db.openTable("book_with_embeddings");
+
+        await tbl.search(Array(1536).fill(1.2))
+          .where("book_id != 3")  // prefilter is default behavior.
+          .limit(10)
+          .toArray();
+        ```
--- a/docs/src/integrations/dlt.md
+++ b/docs/src/integrations/dlt.md
@@ -0,0 +1,142 @@
+# dlt  
+
+[dlt](https://dlthub.com/docs/intro) is an open-source library that you can add to your Python scripts to load data from various and often messy data sources into well-structured, live datasets. dlt's [integration with LanceDB](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb) lets you ingest data from any source (databases, APIs, CSVs, dataframes, JSONs, and more) into LanceDB with a few lines of simple python code. The integration enables automatic normalization of nested data, schema inference, incremental loading and embedding the data. dlt also has integrations with several other tools like dbt, airflow, dagster etc. that can be inserted into your LanceDB workflow.
+
+## How to ingest data into LanceDB  
+
+In this example, we will be fetching movie information from the [Open Movie Database (OMDb) API](https://www.omdbapi.com/) and loading it into a local LanceDB instance. To implement it, you will need an API key for the OMDb API (which can be created freely [here](https://www.omdbapi.com/apikey.aspx)).
+  
+1. **Install `dlt` with LanceDB extras:**  
+    ```sh 
+    pip install dlt[lancedb]
+    ```
+
+2. **Inside an empty directory, initialize a `dlt` project with:**  
+    ```sh
+    dlt init rest_api lancedb
+    ```
+    This will add all the files necessary to create a `dlt` pipeline that can ingest data from any REST API (ex: OMDb API) and load into LanceDB.
+    ```text
+    ├── .dlt
+    │   ├── config.toml
+    │   └── secrets.toml
+    ├── rest_api
+    ├── rest_api_pipeline.py
+    └── requirements.txt
+    ```
+  
+    dlt has a list of pre-built [sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [SQL databases](https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_database), [REST APIs](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api), [Google Sheets](https://dlthub.com/docs/dlt-ecosystem/verified-sources/google_sheets), [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion) etc., that can be used out-of-the-box by running `dlt init <source_name> lancedb`. Since dlt is a python library, it is also very easy to modify these pre-built sources or to write your own custom source from scratch.
+
+
+3. **Specify necessary credentials and/or embedding model details:**  
+    
+    In order to fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB OSS or LanceDB cloud, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/sercrets.toml`. 
+
+    dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/sercrets.toml`:
+    ```toml
+    [sources.rest_api]
+    api_key = "api_key" # Enter the API key for the OMDb API
+
+    [destination.lancedb]
+    embedding_model_provider = "sentence-transformers"
+    embedding_model = "all-MiniLM-L6-v2"
+    [destination.lancedb.credentials]
+    uri = ".lancedb"
+    api_key = "api_key" # API key to connect to LanceDB Cloud. Leave out if you are using LanceDB OSS.
+    embedding_model_provider_api_key = "embedding_model_provider_api_key" # Not needed for providers that don't need authentication (ollama, sentence-transformers).
+    ```
+    See [here](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb#configure-the-destination) for more information and for a list of available models and model providers.  
+
+
+4. **Write the pipeline code inside `rest_api_pipeline.py`:**  
+
+    The following code shows how you can configure dlt's REST API source to connect to the [OMDb API](https://www.omdbapi.com/), fetch all movies with the word "godzilla" in the title, and load it into a LanceDB table. The REST API source allows you to pull data from any API with minimal code, to learn more read the [dlt docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api).
+
+    ```python   
+
+    # Import necessary modules
+    import dlt
+    from rest_api import rest_api_source
+
+    # Configure the REST API source
+    movies_source = rest_api_source(
+        {
+            "client": {
+                "base_url": "https://www.omdbapi.com/",
+                "auth": { # authentication strategy for the OMDb API
+                    "type": "api_key",
+                    "name": "apikey",
+                    "api_key": dlt.secrets["sources.rest_api.api_token"], # read API credentials directly from secrets.toml
+                    "location": "query"
+                },
+                "paginator": { # pagination strategy for the OMDb API 
+                    "type": "page_number",
+                    "base_page": 1,
+                    "total_path": "totalResults",
+                    "maximum_page": 5
+                }
+            },
+            "resources": [ # list of API endpoints to request
+                {
+                    "name": "movie_search",
+                    "endpoint": {
+                        "path": "/",
+                        "params": {
+                            "s": "godzilla",
+                            "type": "movie"
+                        }
+                    }
+                }
+            ]
+        })
+
+
+    if __name__ == "__main__":
+        # Create a pipeline object
+        pipeline = dlt.pipeline(
+            pipeline_name='movies_pipeline',
+            destination='lancedb', # this tells dlt to load the data into LanceDB
+            dataset_name='movies_data_pipeline',
+        )
+
+        # Run the pipeline
+        load_info = pipeline.run(movies_source)
+
+        # pretty print the information on data that was loaded
+        print(load_info)
+    ```
+
+    The script above will ingest the data into LanceDB as it is, i.e. without creating any embeddings. If we want to embed one of the fields (for example, `"Title"` that contains the movie titles), then we will use dlt's `lancedb_adapter` and modify the script as follows:  
+    
+    - Add the following import statement:
+        ```python
+        from dlt.destinations.adapters import lancedb_adapter
+        ```
+    - Modify the pipeline run like this:
+        ```python
+        load_info = pipeline.run(
+            lancedb_adapter(
+                movies_source,
+                embed="Title",
+            )
+        )
+        ```
+    This will use the embedding model specified inside `.dlt/secrets.toml` to embed the field `"Title"`.
+
+5. **Install necessary dependencies:**  
+    ```sh
+    pip install -r requirements.txt
+    ```  
+
+    Note: You may need to install the dependencies for your embedding models separately.
+    ```sh
+    pip install sentence-transformers
+    ```
+
+6. **Run the pipeline:**
+    Finally, running the following command will ingest the data into your LanceDB instance.
+    ```sh
+    python custom_source.py
+    ```
+
+For more information and advanced usage of dlt's LanceDB integration, read [the dlt documentation](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb).
--- a/docs/src/python/python.md
+++ b/docs/src/python/python.md
@@ -113,6 +113,10 @@ lists the indices that LanceDb supports.

 ::: lancedb.index.BTree

+::: lancedb.index.Bitmap
+
+::: lancedb.index.LabelList
+
 ::: lancedb.index.IvfPq

 ## Querying (Asynchronous)
--- a/docs/test/md_testing.py
+++ b/docs/test/md_testing.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import glob
 from typing import Iterator, List
 from pathlib import Path
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.8.0",
+  "version": "0.10.0-beta.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.8.0",
+      "version": "0.10.0-beta.0",
      "cpu": [
        "x64",
        "arm64"
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.8.0",
+  "version": "0.10.0-beta.0",
  "description": " Serverless, low-latency vector database for AI applications",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
--- a/nodejs/.npmignore
+++ b/nodejs/.npmignore
@@ -20,6 +20,5 @@ Cargo.toml
 biome.json
 build.rs
 jest.config.js
-native.d.ts
 tsconfig.json
-typedoc.json
+typedoc.json
--- a/nodejs/test/table.test.ts
+++ b/nodejs/test/table.test.ts
@@ -31,7 +31,9 @@ import {
  Float64,
  Int32,
  Int64,
+  List,
  Schema,
+  Utf8,
  makeArrowTable,
 } from "../lancedb/arrow";
 import {
@@ -331,6 +333,7 @@ describe("When creating an index", () => {
  const schema = new Schema([
    new Field("id", new Int32(), true),
    new Field("vec", new FixedSizeList(32, new Field("item", new Float32()))),
+    new Field("tags", new List(new Field("item", new Utf8(), true))),
  ]);
  let tbl: Table;
  let queryVec: number[];
@@ -346,6 +349,7 @@ describe("When creating an index", () => {
          vec: Array(32)
            .fill(1)
            .map(() => Math.random()),
+          tags: ["tag1", "tag2", "tag3"],
        })),
      {
        schema,
@@ -428,6 +432,22 @@ describe("When creating an index", () => {
    }
  });

+  test("create a bitmap index", async () => {
+    await tbl.createIndex("id", {
+      config: Index.bitmap(),
+    });
+    const indexDir = path.join(tmpDir.name, "test.lance", "_indices");
+    expect(fs.readdirSync(indexDir)).toHaveLength(1);
+  });
+
+  test("create a label list index", async () => {
+    await tbl.createIndex("tags", {
+      config: Index.labelList(),
+    });
+    const indexDir = path.join(tmpDir.name, "test.lance", "_indices");
+    expect(fs.readdirSync(indexDir)).toHaveLength(1);
+  });
+
  test("should be able to get index stats", async () => {
    await tbl.createIndex("id");

@@ -706,6 +726,21 @@ describe("when optimizing a dataset", () => {
    expect(stats.prune.bytesRemoved).toBeGreaterThan(0);
    expect(stats.prune.oldVersionsRemoved).toBe(3);
  });
+
+  it("delete unverified", async () => {
+    const version = await table.version();
+    const versionFile = `${tmpDir.name}/${table.name}.lance/_versions/${version - 1}.manifest`;
+    fs.rmSync(versionFile);
+
+    let stats = await table.optimize({ deleteUnverified: false });
+    expect(stats.prune.oldVersionsRemoved).toBe(0);
+
+    stats = await table.optimize({
+      cleanupOlderThan: new Date(),
+      deleteUnverified: true,
+    });
+    expect(stats.prune.oldVersionsRemoved).toBeGreaterThan(1);
+  });
 });

 describe.each([arrow13, arrow14, arrow15, arrow16, arrow17])(
@@ -785,11 +820,26 @@ describe.each([arrow13, arrow14, arrow15, arrow16, arrow17])(
      ];
      const table = await db.createTable("test", data);

-      expect(table.search("hello").toArray()).rejects.toThrow(
+      expect(table.search("hello", "vector").toArray()).rejects.toThrow(
        "No embedding functions are defined in the table",
      );
    });

+    test("full text search if no embedding function provided", async () => {
+      const db = await connect(tmpDir.name);
+      const data = [
+        { text: "hello world", vector: [0.1, 0.2, 0.3] },
+        { text: "goodbye world", vector: [0.4, 0.5, 0.6] },
+      ];
+      const table = await db.createTable("test", data);
+      await table.createIndex("text", {
+        config: Index.fts(),
+      });
+
+      const results = await table.search("hello").toArray();
+      expect(results[0].text).toBe(data[0].text);
+    });
+
    test.each([
      [0.4, 0.5, 0.599], // number[]
      Float32Array.of(0.4, 0.5, 0.599), // Float32Array
--- a/nodejs/examples/full_text_search.ts
+++ b/nodejs/examples/full_text_search.ts
@@ -0,0 +1,52 @@
+// Copyright 2024 Lance Developers.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+import * as lancedb from "@lancedb/lancedb";
+
+const db = await lancedb.connect("data/sample-lancedb");
+
+const words = [
+  "apple",
+  "banana",
+  "cherry",
+  "date",
+  "elderberry",
+  "fig",
+  "grape",
+];
+
+const data = Array.from({ length: 10_000 }, (_, i) => ({
+  vector: Array(1536).fill(i),
+  id: i,
+  item: `item ${i}`,
+  strId: `${i}`,
+  doc: words[i % words.length],
+}));
+
+const tbl = await db.createTable("myVectors", data, { mode: "overwrite" });
+
+await tbl.createIndex("doc", {
+  config: lancedb.Index.fts(),
+});
+
+// --8<-- [start:full_text_search]
+let result = await tbl
+  .search("apple")
+  .select(["id", "doc"])
+  .limit(10)
+  .toArray();
+console.log(result);
+// --8<-- [end:full_text_search]
+
+console.log("SQL search: done");
--- a/nodejs/examples/package-lock.json
+++ b/nodejs/examples/package-lock.json
@@ -10,7 +10,11 @@
      "license": "Apache-2.0",
      "dependencies": {
        "@lancedb/lancedb": "file:../",
-        "@xenova/transformers": "^2.17.2"
+        "@xenova/transformers": "^2.17.2",
+        "tsc": "^2.0.4"
+      },
+      "devDependencies": {
+        "typescript": "^5.5.4"
      },
      "peerDependencies": {
        "typescript": "^5.0.0"
@@ -18,7 +22,7 @@
    },
    "..": {
      "name": "@lancedb/lancedb",
-      "version": "0.7.1",
+      "version": "0.8.0",
      "cpu": [
        "x64",
        "arm64"
@@ -43,26 +47,30 @@
        "@types/axios": "^0.14.0",
        "@types/jest": "^29.1.2",
        "@types/tmp": "^0.2.6",
-        "apache-arrow-old": "npm:apache-arrow@13.0.0",
+        "apache-arrow-13": "npm:apache-arrow@13.0.0",
+        "apache-arrow-14": "npm:apache-arrow@14.0.0",
+        "apache-arrow-15": "npm:apache-arrow@15.0.0",
+        "apache-arrow-16": "npm:apache-arrow@16.0.0",
+        "apache-arrow-17": "npm:apache-arrow@17.0.0",
        "eslint": "^8.57.0",
        "jest": "^29.7.0",
        "shx": "^0.3.4",
        "tmp": "^0.2.3",
        "ts-jest": "^29.1.2",
-        "typedoc": "^0.25.7",
-        "typedoc-plugin-markdown": "^3.17.1",
-        "typescript": "^5.3.3",
+        "typedoc": "^0.26.4",
+        "typedoc-plugin-markdown": "^4.2.1",
+        "typescript": "^5.5.4",
        "typescript-eslint": "^7.1.0"
      },
      "engines": {
        "node": ">= 18"
      },
      "optionalDependencies": {
-        "@xenova/transformers": "^2.17.2",
+        "@xenova/transformers": ">=2.17 < 3",
        "openai": "^4.29.2"
      },
      "peerDependencies": {
-        "apache-arrow": "^15.0.0"
+        "apache-arrow": ">=13.0.0 <=17.0.0"
      }
    },
    "node_modules/@huggingface/jinja": {
@@ -785,6 +793,15 @@
        "b4a": "^1.6.4"
      }
    },
+    "node_modules/tsc": {
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/tsc/-/tsc-2.0.4.tgz",
+      "integrity": "sha512-fzoSieZI5KKJVBYGvwbVZs/J5za84f2lSTLPYf6AGiIf43tZ3GNrI1QzTLcjtyDDP4aLxd46RTZq1nQxe7+k5Q==",
+      "license": "MIT",
+      "bin": {
+        "tsc": "bin/tsc"
+      }
+    },
    "node_modules/tunnel-agent": {
      "version": "0.6.0",
      "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
@@ -797,10 +814,11 @@
      }
    },
    "node_modules/typescript": {
-      "version": "5.5.2",
-      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.5.2.tgz",
-      "integrity": "sha512-NcRtPEOsPFFWjobJEtfihkLCZCXZt/os3zf8nTxjVH3RvTSxjrCamJpbExGvYOF+tFHc3pA65qpdwPbzjohhew==",
-      "peer": true,
+      "version": "5.5.4",
+      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.5.4.tgz",
+      "integrity": "sha512-Mtq29sKDAEYP7aljRgtPOpTvOfbwRWlS6dPRzwjdE+C0R4brX/GUyhHSecbHMFLNBLcJIPt9nl9yG5TZ1weH+Q==",
+      "dev": true,
+      "license": "Apache-2.0",
      "bin": {
        "tsc": "bin/tsc",
        "tsserver": "bin/tsserver"
--- a/nodejs/examples/package.json
+++ b/nodejs/examples/package.json
@@ -13,7 +13,16 @@
    "@lancedb/lancedb": "file:../",
    "@xenova/transformers": "^2.17.2"
  },
-  "peerDependencies": {
-    "typescript": "^5.0.0"
+  "devDependencies": {
+    "typescript": "^5.5.4"
+  },
+  "compilerOptions": {
+    "target": "ESNext",
+    "module": "ESNext",
+    "moduleResolution": "Node",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "forceConsistentCasingInFileNames": true
  }
 }
--- a/nodejs/examples/search.ts
+++ b/nodejs/examples/search.ts
@@ -32,6 +32,7 @@ const _results2 = await tbl
  .distanceType("cosine")
  .limit(10)
  .toArray();
+console.log(_results2);
 // --8<-- [end:search2]

 console.log("search: done");
--- a/nodejs/lancedb/embedding/registry.ts
+++ b/nodejs/lancedb/embedding/registry.ts
@@ -37,6 +37,13 @@ interface EmbeddingFunctionCreate<T extends EmbeddingFunction> {
 export class EmbeddingFunctionRegistry {
  #functions = new Map<string, EmbeddingFunctionConstructor>();

+  /**
+   * Get the number of registered functions
+   */
+  length() {
+    return this.#functions.size;
+  }
+
  /**
   * Register an embedding function
   * @param name The name of the function
--- a/nodejs/lancedb/indices.ts
+++ b/nodejs/lancedb/indices.ts
@@ -175,6 +175,45 @@ export class Index {
  static btree() {
    return new Index(LanceDbIndex.btree());
  }
+
+  /**
+   * Create a bitmap index.
+   *
+   * A `Bitmap` index stores a bitmap for each distinct value in the column for every row.
+   *
+   * This index works best for low-cardinality columns, where the number of unique values
+   * is small (i.e., less than a few hundreds).
+   */
+  static bitmap() {
+    return new Index(LanceDbIndex.bitmap());
+  }
+
+  /**
+   * Create a label list index.
+   *
+   * LabelList index is a scalar index that can be used on `List<T>` columns to
+   * support queries with `array_contains_all` and `array_contains_any`
+   * using an underlying bitmap index.
+   */
+  static labelList() {
+    return new Index(LanceDbIndex.labelList());
+  }
+
+  /**
+   * Create a full text search index
+   *
+   * A full text search index is an index on a string column, so that you can conduct full
+   * text searches on the column.
+   *
+   * The results of a full text search are ordered by relevance measured by BM25.
+   *
+   * You can combine filters with full text search.
+   *
+   * For now, the full text search index only supports English, and doesn't support phrase search.
+   */
+  static fts() {
+    return new Index(LanceDbIndex.fts());
+  }
 }

 export interface IndexOptions {
--- a/nodejs/lancedb/query.ts
+++ b/nodejs/lancedb/query.ts
@@ -88,6 +88,19 @@ export interface QueryExecutionOptions {
  maxBatchLength?: number;
 }

+/**
+ * Options that control the behavior of a full text search
+ */
+export interface FullTextSearchOptions {
+  /**
+   * The columns to search
+   *
+   * If not specified, all indexed columns will be searched.
+   * For now, only one column can be searched.
+   */
+  columns?: string | string[];
+}
+
 /** Common methods supported by all query types */
 export class QueryBase<NativeQueryType extends NativeQuery | NativeVectorQuery>
  implements AsyncIterable<RecordBatch>
@@ -134,6 +147,25 @@ export class QueryBase<NativeQueryType extends NativeQuery | NativeVectorQuery>
    return this.where(predicate);
  }

+  fullTextSearch(
+    query: string,
+    options?: Partial<FullTextSearchOptions>,
+  ): this {
+    let columns: string[] | null = null;
+    if (options) {
+      if (typeof options.columns === "string") {
+        columns = [options.columns];
+      } else if (Array.isArray(options.columns)) {
+        columns = options.columns;
+      }
+    }
+
+    this.doCall((inner: NativeQueryType) =>
+      inner.fullTextSearch(query, columns),
+    );
+    return this;
+  }
+
  /**
   * Return only the specified columns.
   *
--- a/nodejs/lancedb/table.ts
+++ b/nodejs/lancedb/table.ts
@@ -84,6 +84,7 @@ export interface OptimizeOptions {
   * tbl.cleanupOlderVersions(new Date());
   */
  cleanupOlderThan: Date;
+  deleteUnverified: boolean;
 }

 /**
@@ -270,22 +271,23 @@ export abstract class Table {
   * @returns {Query} A builder that can be used to parameterize the query
   */
  abstract query(): Query;
+
  /**
   * Create a search query to find the nearest neighbors
-   * of the given query vector
-   * @param {string} query - the query. This will be converted to a vector using the table's provided embedding function
-   * @note  If no embedding functions are defined in the table, this will error when collecting the results.
+   * of the given query
+   * @param {string | IntoVector} query - the query, a vector or string
+   * @param {string} queryType - the type of the query, "vector", "fts", or "auto"
+   * @param {string | string[]} ftsColumns - the columns to search in for full text search
+   *    for now, only one column can be searched at a time.
   *
-   * This is just a convenience method for calling `.query().nearestTo(await myEmbeddingFunction(query))`
+   * when "auto" is used, if the query is a string and an embedding function is defined, it will be treated as a vector query
+   * if the query is a string and no embedding function is defined, it will be treated as a full text search query
   */
-  abstract search(query: string): VectorQuery;
-  /**
-   * Create a search query to find the nearest neighbors
-   * of the given query vector
-   * @param {IntoVector} query - the query vector
-   * This is just a convenience method for calling `.query().nearestTo(query)`
-   */
-  abstract search(query: IntoVector): VectorQuery;
+  abstract search(
+    query: string | IntoVector,
+    queryType?: string,
+    ftsColumns?: string | string[],
+  ): VectorQuery | Query;
  /**
   * Search the table with a given query vector.
   *
@@ -581,27 +583,50 @@ export class LocalTable extends Table {
  query(): Query {
    return new Query(this.inner);
  }
-  search(query: string | IntoVector): VectorQuery {
-    if (typeof query !== "string") {
-      return this.vectorSearch(query);
-    } else {
-      const queryPromise = this.getEmbeddingFunctions().then(
-        async (functions) => {
-          // TODO: Support multiple embedding functions
-          const embeddingFunc: EmbeddingFunctionConfig | undefined = functions
-            .values()
-            .next().value;
-          if (!embeddingFunc) {
-            return Promise.reject(
-              new Error("No embedding functions are defined in the table"),
-            );
-          }
-          return await embeddingFunc.function.computeQueryEmbeddings(query);
-        },
-      );

-      return this.query().nearestTo(queryPromise);
+  search(
+    query: string | IntoVector,
+    queryType: string = "auto",
+    ftsColumns?: string | string[],
+  ): VectorQuery | Query {
+    if (typeof query !== "string") {
+      if (queryType === "fts") {
+        throw new Error("Cannot perform full text search on a vector query");
+      }
+      return this.vectorSearch(query);
    }
+
+    // If the query is a string, we need to determine if it is a vector query or a full text search query
+    if (queryType === "fts") {
+      return this.query().fullTextSearch(query, {
+        columns: ftsColumns,
+      });
+    }
+
+    // The query type is auto or vector
+    // fall back to full text search if no embedding functions are defined and the query is a string
+    if (queryType === "auto" && getRegistry().length() === 0) {
+      return this.query().fullTextSearch(query, {
+        columns: ftsColumns,
+      });
+    }
+
+    const queryPromise = this.getEmbeddingFunctions().then(
+      async (functions) => {
+        // TODO: Support multiple embedding functions
+        const embeddingFunc: EmbeddingFunctionConfig | undefined = functions
+          .values()
+          .next().value;
+        if (!embeddingFunc) {
+          return Promise.reject(
+            new Error("No embedding functions are defined in the table"),
+          );
+        }
+        return await embeddingFunc.function.computeQueryEmbeddings(query);
+      },
+    );
+
+    return this.query().nearestTo(queryPromise);
  }

  vectorSearch(vector: IntoVector): VectorQuery {
@@ -647,7 +672,10 @@ export class LocalTable extends Table {
      cleanupOlderThanMs =
        new Date().getTime() - options.cleanupOlderThan.getTime();
    }
-    return await this.inner.optimize(cleanupOlderThanMs);
+    return await this.inner.optimize(
+      cleanupOlderThanMs,
+      options?.deleteUnverified,
+    );
  }

  async listIndices(): Promise<IndexConfig[]> {
--- a/nodejs/npm/darwin-arm64/package.json
+++ b/nodejs/npm/darwin-arm64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.8.0",
+	"version": "0.10.0-beta.0",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
--- a/nodejs/npm/darwin-x64/package.json
+++ b/nodejs/npm/darwin-x64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-x64",
-	"version": "0.8.0",
+	"version": "0.10.0-beta.0",
 	"os": ["darwin"],
 	"cpu": ["x64"],
 	"main": "lancedb.darwin-x64.node",
--- a/nodejs/npm/linux-arm64-gnu/package.json
+++ b/nodejs/npm/linux-arm64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.8.0",
+	"version": "0.10.0-beta.0",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
--- a/nodejs/npm/linux-x64-gnu/package.json
+++ b/nodejs/npm/linux-x64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.8.0",
+	"version": "0.10.0-beta.0",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
--- a/nodejs/npm/win32-x64-msvc/package.json
+++ b/nodejs/npm/win32-x64-msvc/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.8.0",
+	"version": "0.10.0-beta.0",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
--- a/nodejs/package-lock.json
+++ b/nodejs/package-lock.json
@@ -43,7 +43,7 @@
        "ts-jest": "^29.1.2",
        "typedoc": "^0.26.4",
        "typedoc-plugin-markdown": "^4.2.1",
-        "typescript": "^5.3.3",
+        "typescript": "^5.5.4",
        "typescript-eslint": "^7.1.0"
      },
      "engines": {
@@ -9292,10 +9292,11 @@
      }
    },
    "node_modules/typescript": {
-      "version": "5.3.3",
-      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.3.3.tgz",
-      "integrity": "sha512-pXWcraxM0uxAS+tN0AG/BF2TyqmHO014Z070UsJ+pFvYuRSq8KH8DmWpnbXe0pEPDHXZV3FcAbJkijJ5oNEnWw==",
+      "version": "5.5.4",
+      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.5.4.tgz",
+      "integrity": "sha512-Mtq29sKDAEYP7aljRgtPOpTvOfbwRWlS6dPRzwjdE+C0R4brX/GUyhHSecbHMFLNBLcJIPt9nl9yG5TZ1weH+Q==",
      "dev": true,
+      "license": "Apache-2.0",
      "bin": {
        "tsc": "bin/tsc",
        "tsserver": "bin/tsserver"
--- a/nodejs/package.json
+++ b/nodejs/package.json
@@ -10,7 +10,7 @@
    "vector database",
    "ann"
  ],
-  "version": "0.8.0",
+  "version": "0.10.0-beta.0",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
@@ -53,7 +53,7 @@
    "ts-jest": "^29.1.2",
    "typedoc": "^0.26.4",
    "typedoc-plugin-markdown": "^4.2.1",
-    "typescript": "^5.3.3",
+    "typescript": "^5.5.4",
    "typescript-eslint": "^7.1.0"
  },
  "ava": {
--- a/nodejs/src/index.rs
+++ b/nodejs/src/index.rs
@@ -14,7 +14,7 @@

 use std::sync::Mutex;

-use lancedb::index::scalar::BTreeIndexBuilder;
+use lancedb::index::scalar::{BTreeIndexBuilder, FtsIndexBuilder};
 use lancedb::index::vector::IvfPqIndexBuilder;
 use lancedb::index::Index as LanceDbIndex;
 use napi_derive::napi;
@@ -76,4 +76,25 @@ impl Index {
            inner: Mutex::new(Some(LanceDbIndex::BTree(BTreeIndexBuilder::default()))),
        }
    }
+
+    #[napi(factory)]
+    pub fn bitmap() -> Self {
+        Self {
+            inner: Mutex::new(Some(LanceDbIndex::Bitmap(Default::default()))),
+        }
+    }
+
+    #[napi(factory)]
+    pub fn label_list() -> Self {
+        Self {
+            inner: Mutex::new(Some(LanceDbIndex::LabelList(Default::default()))),
+        }
+    }
+
+    #[napi(factory)]
+    pub fn fts() -> Self {
+        Self {
+            inner: Mutex::new(Some(LanceDbIndex::FTS(FtsIndexBuilder::default()))),
+        }
+    }
 }
--- a/nodejs/src/query.rs
+++ b/nodejs/src/query.rs
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+use lancedb::index::scalar::FullTextSearchQuery;
 use lancedb::query::ExecutableQuery;
 use lancedb::query::Query as LanceDbQuery;
 use lancedb::query::QueryBase;
@@ -42,6 +43,12 @@ impl Query {
        self.inner = self.inner.clone().only_if(predicate);
    }

+    #[napi]
+    pub fn full_text_search(&mut self, query: String, columns: Option<Vec<String>>) {
+        let query = FullTextSearchQuery::new(query).columns(columns);
+        self.inner = self.inner.clone().full_text_search(query);
+    }
+
    #[napi]
    pub fn select(&mut self, columns: Vec<(String, String)>) {
        self.inner = self.inner.clone().select(Select::dynamic(&columns));
@@ -138,6 +145,12 @@ impl VectorQuery {
        self.inner = self.inner.clone().only_if(predicate);
    }

+    #[napi]
+    pub fn full_text_search(&mut self, query: String, columns: Option<Vec<String>>) {
+        let query = FullTextSearchQuery::new(query).columns(columns);
+        self.inner = self.inner.clone().full_text_search(query);
+    }
+
    #[napi]
    pub fn select(&mut self, columns: Vec<(String, String)>) {
        self.inner = self.inner.clone().select(Select::dynamic(&columns));
--- a/nodejs/src/table.rs
+++ b/nodejs/src/table.rs
@@ -265,7 +265,11 @@ impl Table {
    }

    #[napi(catch_unwind)]
-    pub async fn optimize(&self, older_than_ms: Option<i64>) -> napi::Result<OptimizeStats> {
+    pub async fn optimize(
+        &self,
+        older_than_ms: Option<i64>,
+        delete_unverified: Option<bool>,
+    ) -> napi::Result<OptimizeStats> {
        let inner = self.inner_ref()?;

        let older_than = if let Some(ms) = older_than_ms {
@@ -292,7 +296,7 @@ impl Table {
        let prune_stats = inner
            .optimize(OptimizeAction::Prune {
                older_than,
-                delete_unverified: None,
+                delete_unverified,
                error_if_tagged_old_versions: None,
            })
            .await
--- a/nodejs/tsconfig.json
+++ b/nodejs/tsconfig.json
@@ -9,7 +9,8 @@
    "allowJs": true,
    "resolveJsonModule": true,
    "emitDecoratorMetadata": true,
-    "experimentalDecorators": true
+    "experimentalDecorators": true,
+    "moduleResolution": "Node"
  },
  "exclude": ["./dist/*"],
  "typedocOptions": {
--- a/python/.bumpversion.toml
+++ b/python/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.12.0"
+current_version = "0.13.0-beta.1"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/python/Cargo.toml
+++ b/python/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.12.0"
+version = "0.13.0-beta.1"
 edition.workspace = true
 description = "Python bindings for LanceDB"
 license.workspace = true
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -3,7 +3,7 @@ name = "lancedb"
 # version in Cargo.toml
 dependencies = [
    "deprecation",
-    "pylance==0.16.0",
+    "pylance==0.16.1",
    "ratelimiter~=1.0",
    "requests>=2.31.0",
    "retry>=0.9.2",
@@ -18,7 +18,7 @@ description = "lancedb"
 authors = [{ name = "LanceDB Devs", email = "dev@lancedb.com" }]
 license = { file = "LICENSE" }
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 keywords = [
    "data-format",
    "data-science",
--- a/python/python/lancedb/_lancedb.pyi
+++ b/python/python/lancedb/_lancedb.pyi
@@ -74,6 +74,7 @@ class Query:
    def select(self, columns: Tuple[str, str]): ...
    def limit(self, limit: int): ...
    def nearest_to(self, query_vec: pa.Array) -> VectorQuery: ...
+    def nearest_to_text(self, query: dict) -> Query: ...
    async def execute(self, max_batch_legnth: Optional[int]) -> RecordBatchStream: ...

 class VectorQuery:
--- a/python/python/lancedb/db.py
+++ b/python/python/lancedb/db.py
@@ -276,6 +276,10 @@ class DBConnection(EnforceOverrides):
        """
        raise NotImplementedError

+    @property
+    def uri(self) -> str:
+        return self._uri
+

 class LanceDBConnection(DBConnection):
    """
@@ -340,10 +344,6 @@ class LanceDBConnection(DBConnection):
        val += ")"
        return val

-    @property
-    def uri(self) -> str:
-        return self._uri
-
    async def _async_get_table_names(self, start_after: Optional[str], limit: int):
        conn = AsyncConnection(await lancedb_connect(self.uri))
        return await conn.table_names(start_after=start_after, limit=limit)
--- a/python/python/lancedb/embeddings/instructor.py
+++ b/python/python/lancedb/embeddings/instructor.py
@@ -127,6 +127,7 @@ class InstructorEmbeddingFunction(TextEmbeddingFunction):
            batch_size=self.batch_size,
            show_progress_bar=self.show_progress_bar,
            normalize_embeddings=self.normalize_embeddings,
+            device=self.device,
        ).tolist()
        return res

--- a/python/python/lancedb/embeddings/transformers.py
+++ b/python/python/lancedb/embeddings/transformers.py
@@ -44,6 +44,7 @@ class TransformersEmbeddingFunction(EmbeddingFunction):
    """

    name: str = "colbert-ir/colbertv2.0"
+    device: str = "cpu"
    _tokenizer: Any = PrivateAttr()
    _model: Any = PrivateAttr()

@@ -53,6 +54,7 @@ class TransformersEmbeddingFunction(EmbeddingFunction):
        transformers = attempt_import_or_raise("transformers")
        self._tokenizer = transformers.AutoTokenizer.from_pretrained(self.name)
        self._model = transformers.AutoModel.from_pretrained(self.name)
+        self._model.to(self.device)

    if PYDANTIC_VERSION.major < 2:  # Pydantic 1.x compat

@@ -75,9 +77,9 @@ class TransformersEmbeddingFunction(EmbeddingFunction):
        for text in texts:
            encoding = self._tokenizer(
                text, return_tensors="pt", padding=True, truncation=True
-            )
+            ).to(self.device)
            emb = self._model(**encoding).last_hidden_state.mean(dim=1).squeeze()
-            embedding.append(emb.detach().numpy())
+            embedding.append(emb.tolist())

        return embedding

--- a/python/python/lancedb/index.py
+++ b/python/python/lancedb/index.py
@@ -8,7 +8,7 @@ from ._lancedb import (
 )


-class BTree(object):
+class BTree:
    """Describes a btree index configuration

    A btree index is an index on scalar columns.  The index stores a copy of the
@@ -22,7 +22,8 @@ class BTree(object):
    sizeof(Scalar) * 4096 bytes to find the correct row ids.

    This index is good for scalar columns with mostly distinct values and does best
-    when the query is highly selective.
+    when the query is highly selective. It works with numeric, temporal, and string
+    columns.

    The btree index does not currently have any parameters though parameters such as
    the block size may be added in the future.
@@ -32,7 +33,56 @@ class BTree(object):
        self._inner = LanceDbIndex.btree()


-class IvfPq(object):
+class Bitmap:
+    """Describe a Bitmap index configuration.
+
+    A `Bitmap` index stores a bitmap for each distinct value in the column for
+    every row.
+
+    This index works best for low-cardinality numeric or string columns,
+    where the number of unique values is small (i.e., less than a few thousands).
+    `Bitmap` index can accelerate the following filters:
+
+    - `<`, `<=`, `=`, `>`, `>=`
+    - `IN (value1, value2, ...)`
+    - `between (value1, value2)`
+    - `is null`
+
+    For example, a bitmap index with a table with 1Bi rows, and 128 distinct values,
+    requires 128 / 8 * 1Bi bytes on disk.
+    """
+
+    def __init__(self):
+        self._inner = LanceDbIndex.bitmap()
+
+
+class LabelList:
+    """Describe a LabelList index configuration.
+
+    `LabelList` is a scalar index that can be used on `List<T>` columns to
+    support queries with `array_contains_all` and `array_contains_any`
+    using an underlying bitmap index.
+
+    For example, it works with `tags`, `categories`, `keywords`, etc.
+    """
+
+    def __init__(self):
+        self._inner = LanceDbIndex.label_list()
+
+
+class FTS:
+    """Describe a FTS index configuration.
+
+    `FTS` is a full-text search index that can be used on `String` columns
+
+    For example, it works with `title`, `description`, `content`, etc.
+    """
+
+    def __init__(self):
+        self._inner = LanceDbIndex.fts()
+
+
+class IvfPq:
    """Describes an IVF PQ Index

    This index stores a compressed (quantized) copy of every vector.  These vectors
--- a/python/python/lancedb/query.py
+++ b/python/python/lancedb/query.py
@@ -15,7 +15,6 @@ from __future__ import annotations

 from abc import ABC, abstractmethod
 from concurrent.futures import ThreadPoolExecutor
-from pathlib import Path
 from typing import (
    TYPE_CHECKING,
    Dict,
@@ -38,7 +37,7 @@ from .arrow import AsyncRecordBatchReader
 from .common import VEC
 from .rerankers.base import Reranker
 from .rerankers.linear_combination import LinearCombinationReranker
-from .util import fs_from_uri, safe_import_pandas
+from .util import safe_import_pandas

 if TYPE_CHECKING:
    import PIL
@@ -99,6 +98,9 @@ class Query(pydantic.BaseModel):
    # if True then apply the filter before vector search
    prefilter: bool = False

+    # full text search query
+    full_text_query: Optional[Union[str, dict]] = None
+
    # top k results to return
    k: int

@@ -131,6 +133,7 @@ class LanceQueryBuilder(ABC):
        query_type: str,
        vector_column_name: str,
        ordering_field_name: str = None,
+        fts_columns: Union[str, List[str]] = None,
    ) -> LanceQueryBuilder:
        """
        Create a query builder based on the given query and query type.
@@ -170,7 +173,9 @@ class LanceQueryBuilder(ABC):
        if isinstance(query, str):
            # fts
            return LanceFtsQueryBuilder(
-                table, query, ordering_field_name=ordering_field_name
+                table,
+                query,
+                ordering_field_name=ordering_field_name,
            )

        if isinstance(query, list):
@@ -226,6 +231,7 @@ class LanceQueryBuilder(ABC):
        self._limit = 10
        self._columns = None
        self._where = None
+        self._prefilter = False
        self._with_row_id = False

    @deprecation.deprecated(
@@ -451,6 +457,22 @@ class LanceQueryBuilder(ABC):
            },
        ).explain_plan(verbose)

+    @abstractmethod
+    def rerank(self, reranker: Reranker) -> LanceQueryBuilder:
+        """Rerank the results using the specified reranker.
+
+        Parameters
+        ----------
+        reranker: Reranker
+            The reranker to use.
+
+        Returns
+        -------
+
+        The LanceQueryBuilder object.
+        """
+        raise NotImplementedError
+

 class LanceVectorQueryBuilder(LanceQueryBuilder):
    """
@@ -664,12 +686,21 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
 class LanceFtsQueryBuilder(LanceQueryBuilder):
    """A builder for full text search for LanceDB."""

-    def __init__(self, table: "Table", query: str, ordering_field_name: str = None):
+    def __init__(
+        self,
+        table: "Table",
+        query: str,
+        ordering_field_name: str = None,
+        fts_columns: Union[str, List[str]] = None,
+    ):
        super().__init__(table)
        self._query = query
        self._phrase_query = False
        self.ordering_field_name = ordering_field_name
        self._reranker = None
+        if isinstance(fts_columns, str):
+            fts_columns = [fts_columns]
+        self._fts_columns = fts_columns

    def phrase_query(self, phrase_query: bool = True) -> LanceFtsQueryBuilder:
        """Set whether to use phrase query.
@@ -689,6 +720,35 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
        return self

    def to_arrow(self) -> pa.Table:
+        path, fs, exist = self._table._get_fts_index_path()
+        if exist:
+            return self.tantivy_to_arrow()
+
+        query = self._query
+        if self._phrase_query:
+            raise NotImplementedError(
+                "Phrase query is not yet supported in Lance FTS. "
+                "Use tantivy-based index instead for now."
+            )
+        query = Query(
+            columns=self._columns,
+            filter=self._where,
+            k=self._limit,
+            prefilter=self._prefilter,
+            with_row_id=self._with_row_id,
+            full_text_query={
+                "query": query,
+                "columns": self._fts_columns,
+            },
+            vector=[],
+        )
+        results = self._table._execute_query(query)
+        results = results.read_all()
+        if self._reranker is not None:
+            results = self._reranker.rerank_fts(self._query, results)
+        return results
+
+    def tantivy_to_arrow(self) -> pa.Table:
        try:
            import tantivy
        except ImportError:
@@ -699,24 +759,24 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
        from .fts import search_index

        # get the index path
-        index_path = self._table._get_fts_index_path()
-
-        # Check that we are on local filesystem
-        fs, _path = fs_from_uri(index_path)
-        if not isinstance(fs, pa_fs.LocalFileSystem):
-            raise NotImplementedError(
-                "Full-text search is only supported on the local filesystem"
-            )
+        path, fs, exist = self._table._get_fts_index_path()

        # check if the index exist
-        if not Path(index_path).exists():
+        if not exist:
            raise FileNotFoundError(
                "Fts index does not exist. "
                "Please first call table.create_fts_index(['<field_names>']) to "
                "create the fts index."
            )
+
+        # Check that we are on local filesystem
+        if not isinstance(fs, pa_fs.LocalFileSystem):
+            raise NotImplementedError(
+                "Tantivy-based full text search "
+                "is only supported on the local filesystem"
+            )
        # open the index
-        index = tantivy.Index.open(index_path)
+        index = tantivy.Index.open(path)
        # get the scores and doc ids
        query = self._query
        if self._phrase_query:
@@ -726,11 +786,11 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
            index, query, self._limit, ordering_field=self.ordering_field_name
        )
        if len(row_ids) == 0:
-            empty_schema = pa.schema([pa.field("score", pa.float32())])
+            empty_schema = pa.schema([pa.field("_score", pa.float32())])
            return pa.Table.from_pylist([], schema=empty_schema)
        scores = pa.array(scores)
        output_tbl = self._table.to_lance().take(row_ids, columns=self._columns)
-        output_tbl = output_tbl.append_column("score", scores)
+        output_tbl = output_tbl.append_column("_score", scores)
        # this needs to match vector search results which are uint64
        row_ids = pa.array(row_ids, type=pa.uint64())

@@ -797,6 +857,21 @@ class LanceEmptyQueryBuilder(LanceQueryBuilder):
            limit=self._limit,
        )

+    def rerank(self, reranker: Reranker) -> LanceEmptyQueryBuilder:
+        """Rerank the results using the specified reranker.
+
+        Parameters
+        ----------
+        reranker: Reranker
+            The reranker to use.
+
+        Returns
+        -------
+        LanceEmptyQueryBuilder
+            The LanceQueryBuilder object.
+        """
+        raise NotImplementedError("Reranking is not yet supported.")
+

 class LanceHybridQueryBuilder(LanceQueryBuilder):
    """
@@ -811,7 +886,6 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):

    def __init__(self, table: "Table", query: str, vector_column: str):
        super().__init__(table)
-        self._validate_fts_index()
        vector_query, fts_query = self._validate_query(query)
        self._fts_query = LanceFtsQueryBuilder(table, fts_query)
        vector_query = self._query_to_vector(table, vector_query, vector_column)
@@ -819,12 +893,6 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        self._norm = "score"
        self._reranker = LinearCombinationReranker(weight=0.7, fill=1.0)

-    def _validate_fts_index(self):
-        if self._table._get_fts_index_path() is None:
-            raise ValueError(
-                "Please create a full-text search index " "to perform hybrid search."
-            )
-
    def _validate_query(self, query):
        # Temp hack to support vectorized queries for hybrid search
        if isinstance(query, str):
@@ -856,13 +924,13 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        # convert to ranks first if needed
        if self._norm == "rank":
            vector_results = self._rank(vector_results, "_distance")
-            fts_results = self._rank(fts_results, "score")
+            fts_results = self._rank(fts_results, "_score")
        # normalize the scores to be between 0 and 1, 0 being most relevant
        vector_results = self._normalize_scores(vector_results, "_distance")

        # In fts higher scores represent relevance. Not inverting them here as
        # rerankers might need to preserve this score to support `return_score="all"`
-        fts_results = self._normalize_scores(fts_results, "score")
+        fts_results = self._normalize_scores(fts_results, "_score")

        results = self._reranker.rerank_hybrid(
            self._fts_query._query, vector_results, fts_results
@@ -1177,6 +1245,16 @@ class AsyncQueryBase(object):
            await batch_iter.read_all(), schema=batch_iter.schema
        )

+    async def to_list(self) -> List[dict]:
+        """
+        Execute the query and return the results as a list of dictionaries.
+
+        Each list entry is a dictionary with the selected column names as keys,
+        or all table columns if `select` is not called. The vector and the "_distance"
+        fields are returned whether or not they're explicitly selected.
+        """
+        return (await self.to_arrow()).to_pylist()
+
    async def to_pandas(self) -> "pd.DataFrame":
        """
        Execute the query and collect the results into a pandas DataFrame.
@@ -1304,6 +1382,35 @@ class AsyncQuery(AsyncQueryBase):
            self._inner.nearest_to(AsyncQuery._query_vec_to_array(query_vector))
        )

+    def nearest_to_text(
+        self, query: str, columns: Union[str, List[str]] = None
+    ) -> AsyncQuery:
+        """
+        Find the documents that are most relevant to the given text query.
+
+        This method will perform a full text search on the table and return
+        the most relevant documents.  The relevance is determined by BM25.
+
+        The columns to search must be with native FTS index
+        (Tantivy-based can't work with this method).
+
+        By default, all indexed columns are searched,
+        now only one column can be searched at a time.
+
+        Parameters
+        ----------
+        query: str
+            The text query to search for.
+        columns: str or list of str, default None
+            The columns to search in. If None, all indexed columns are searched.
+            For now only one column can be searched at a time.
+        """
+        if isinstance(columns, str):
+            columns = [columns]
+        return AsyncQuery(
+            self._inner.nearest_to_text({"query": query, "columns": columns})
+        )
+

 class AsyncVectorQuery(AsyncQueryBase):
    def __init__(self, inner: LanceVectorQuery):
--- a/python/python/lancedb/remote/db.py
+++ b/python/python/lancedb/remote/db.py
@@ -49,6 +49,7 @@ class RemoteDBConnection(DBConnection):
        parsed = urlparse(db_url)
        if parsed.scheme != "db":
            raise ValueError(f"Invalid scheme: {parsed.scheme}, only accepts db://")
+        self._uri = str(db_url)
        self.db_name = parsed.netloc
        self.api_key = api_key
        self._client = RestfulLanceDBClient(
--- a/python/python/lancedb/remote/table.py
+++ b/python/python/lancedb/remote/table.py
@@ -15,7 +15,7 @@ import logging
 import uuid
 from concurrent.futures import Future
 from functools import cached_property
-from typing import Dict, Iterable, Optional, Union
+from typing import Dict, Iterable, Optional, Union, Literal

 import pyarrow as pa
 from lance import json_to_schema
@@ -35,10 +35,10 @@ from .db import RemoteDBConnection
 class RemoteTable(Table):
    def __init__(self, conn: RemoteDBConnection, name: str):
        self._conn = conn
-        self._name = name
+        self.name = name

    def __repr__(self) -> str:
-        return f"RemoteTable({self._conn.db_name}.{self._name})"
+        return f"RemoteTable({self._conn.db_name}.{self.name})"

    def __len__(self) -> int:
        self.count_rows(None)
@@ -49,14 +49,14 @@ class RemoteTable(Table):
        of this Table

        """
-        resp = self._conn._client.post(f"/v1/table/{self._name}/describe/")
+        resp = self._conn._client.post(f"/v1/table/{self.name}/describe/")
        schema = json_to_schema(resp["schema"])
        return schema

    @property
    def version(self) -> int:
        """Get the current version of the table"""
-        resp = self._conn._client.post(f"/v1/table/{self._name}/describe/")
+        resp = self._conn._client.post(f"/v1/table/{self.name}/describe/")
        return resp["version"]

    @cached_property
@@ -84,19 +84,20 @@ class RemoteTable(Table):

    def list_indices(self):
        """List all the indices on the table"""
-        resp = self._conn._client.post(f"/v1/table/{self._name}/index/list/")
+        resp = self._conn._client.post(f"/v1/table/{self.name}/index/list/")
        return resp

    def index_stats(self, index_uuid: str):
        """List all the stats of a specified index"""
        resp = self._conn._client.post(
-            f"/v1/table/{self._name}/index/{index_uuid}/stats/"
+            f"/v1/table/{self.name}/index/{index_uuid}/stats/"
        )
        return resp

    def create_scalar_index(
        self,
        column: str,
+        index_type: Literal["BTREE", "BITMAP", "LABEL_LIST", "scalar"] = "scalar",
    ):
        """Creates a scalar index
        Parameters
@@ -104,8 +105,10 @@ class RemoteTable(Table):
        column : str
            The column to be indexed.  Must be a boolean, integer, float,
            or string column.
+        index_type : str
+            The index type of the scalar index. Must be "scalar" (BTREE),
+            "BTREE", "BITMAP", or "LABEL_LIST"
        """
-        index_type = "scalar"

        data = {
            "column": column,
@@ -113,11 +116,27 @@ class RemoteTable(Table):
            "replace": True,
        }
        resp = self._conn._client.post(
-            f"/v1/table/{self._name}/create_scalar_index/", data=data
+            f"/v1/table/{self.name}/create_scalar_index/", data=data
        )

        return resp

+    def create_fts_index(
+        self,
+        column: str,
+        *,
+        replace: bool = False,
+    ):
+        data = {
+            "column": column,
+            "index_type": "FTS",
+            "replace": replace,
+        }
+        resp = self._conn._client.post(
+            f"/v1/table/{self.name}/create_index/", data=data
+        )
+        return resp
+
    def create_index(
        self,
        metric="L2",
@@ -191,7 +210,7 @@ class RemoteTable(Table):
            "index_cache_size": index_cache_size,
        }
        resp = self._conn._client.post(
-            f"/v1/table/{self._name}/create_index/", data=data
+            f"/v1/table/{self.name}/create_index/", data=data
        )

        return resp
@@ -238,7 +257,7 @@ class RemoteTable(Table):
        request_id = uuid.uuid4().hex

        self._conn._client.post(
-            f"/v1/table/{self._name}/insert/",
+            f"/v1/table/{self.name}/insert/",
            data=payload,
            params={"request_id": request_id, "mode": mode},
            content_type=ARROW_STREAM_CONTENT_TYPE,
@@ -248,6 +267,7 @@ class RemoteTable(Table):
        self,
        query: Union[VEC, str],
        vector_column_name: Optional[str] = None,
+        query_type="auto",
    ) -> LanceVectorQueryBuilder:
        """Create a search query to find the nearest neighbors
        of the given query vector. We currently support [vector search][search]
@@ -307,10 +327,18 @@ class RemoteTable(Table):
            - and also the "_distance" column which is the distance between the query
            vector and the returned vector.
        """
-        if vector_column_name is None:
-            vector_column_name = inf_vector_column_query(self.schema)
-        query = LanceQueryBuilder._query_to_vector(self, query, vector_column_name)
-        return LanceVectorQueryBuilder(self, query, vector_column_name)
+        if vector_column_name is None and query is not None and query_type != "fts":
+            try:
+                vector_column_name = inf_vector_column_query(self.schema)
+            except Exception as e:
+                raise e
+
+        return LanceQueryBuilder.create(
+            self,
+            query,
+            query_type,
+            vector_column_name=vector_column_name,
+        )

    def _execute_query(
        self, query: Query, batch_size: Optional[int] = None
@@ -339,12 +367,12 @@ class RemoteTable(Table):
                v = list(v)
                q = query.copy()
                q.vector = v
-                results.append(submit(self._name, q))
+                results.append(submit(self.name, q))
            return pa.concat_tables(
                [add_index(r.result().to_arrow(), i) for i, r in enumerate(results)]
            ).to_reader()
        else:
-            result = self._conn._client.query(self._name, query)
+            result = self._conn._client.query(self.name, query)
            return result.to_arrow().to_reader()

    def merge_insert(self, on: Union[str, Iterable[str]]) -> LanceMergeInsertBuilder:
@@ -394,7 +422,7 @@ class RemoteTable(Table):
            )

        self._conn._client.post(
-            f"/v1/table/{self._name}/merge_insert/",
+            f"/v1/table/{self.name}/merge_insert/",
            data=payload,
            params=params,
            content_type=ARROW_STREAM_CONTENT_TYPE,
@@ -448,7 +476,7 @@ class RemoteTable(Table):
        0  2  [3.0, 4.0]       85.0 # doctest: +SKIP
        """
        payload = {"predicate": predicate}
-        self._conn._client.post(f"/v1/table/{self._name}/delete/", data=payload)
+        self._conn._client.post(f"/v1/table/{self.name}/delete/", data=payload)

    def update(
        self,
@@ -509,7 +537,7 @@ class RemoteTable(Table):
            updates = [[k, v] for k, v in values_sql.items()]

        payload = {"predicate": where, "updates": updates}
-        self._conn._client.post(f"/v1/table/{self._name}/update/", data=payload)
+        self._conn._client.post(f"/v1/table/{self.name}/update/", data=payload)

    def cleanup_old_versions(self, *_):
        """cleanup_old_versions() is not supported on the LanceDB cloud"""
@@ -526,7 +554,7 @@ class RemoteTable(Table):
    def count_rows(self, filter: Optional[str] = None) -> int:
        payload = {"predicate": filter}
        resp = self._conn._client.post(
-            f"/v1/table/{self._name}/count_rows/", data=payload
+            f"/v1/table/{self.name}/count_rows/", data=payload
        )
        return resp

--- a/python/python/lancedb/rerankers/base.py
+++ b/python/python/lancedb/rerankers/base.py
@@ -220,8 +220,8 @@ class Reranker(ABC):

    def _keep_relevance_score(self, combined_results: pa.Table):
        if self.score == "relevance":
-            if "score" in combined_results.column_names:
-                combined_results = combined_results.drop_columns(["score"])
+            if "_score" in combined_results.column_names:
+                combined_results = combined_results.drop_columns(["_score"])
            if "_distance" in combined_results.column_names:
                combined_results = combined_results.drop_columns(["_distance"])
        return combined_results
--- a/python/python/lancedb/rerankers/cohere.py
+++ b/python/python/lancedb/rerankers/cohere.py
@@ -113,6 +113,6 @@ class CohereReranker(Reranker):
    ):
        result_set = self._rerank(fts_results, query)
        if self.score == "relevance":
-            result_set = result_set.drop_columns(["score"])
+            result_set = result_set.drop_columns(["_score"])

        return result_set
--- a/python/python/lancedb/rerankers/colbert.py
+++ b/python/python/lancedb/rerankers/colbert.py
@@ -1,5 +1,3 @@
-from functools import cached_property
-
 import pyarrow as pa

 from ..util import attempt_import_or_raise
@@ -12,7 +10,7 @@ class ColbertReranker(Reranker):

    Parameters
    ----------
-    model_name : str, default "colbert-ir/colbertv2.0"
+    model_name : str, default "colbert" (colbert-ir/colbert-v2.0)
        The name of the cross encoder model to use.
    column : str, default "text"
        The name of the column to use as input to the cross encoder model.
@@ -22,41 +20,26 @@ class ColbertReranker(Reranker):

    def __init__(
        self,
-        model_name: str = "colbert-ir/colbertv2.0",
+        model_name: str = "colbert",
        column: str = "text",
        return_score="relevance",
    ):
        super().__init__(return_score)
        self.model_name = model_name
        self.column = column
-        self.torch = attempt_import_or_raise(
-            "torch"
+        rerankers = attempt_import_or_raise(
+            "rerankers"
        )  # import here for faster ops later
+        self.colbert = rerankers.Reranker(self.model_name, model_type="colbert")

    def _rerank(self, result_set: pa.Table, query: str):
        docs = result_set[self.column].to_pylist()
+        doc_ids = list(range(len(docs)))
+        result = self.colbert.rank(query, docs, doc_ids=doc_ids)

-        tokenizer, model = self._model
+        # get the scores of each document in the same order as the input
+        scores = [result.get_result_by_docid(i).score for i in doc_ids]

-        # Encode the query
-        query_encoding = tokenizer(query, return_tensors="pt")
-        query_embedding = model(**query_encoding).last_hidden_state.mean(dim=1)
-        scores = []
-        # Get score for each document
-        for document in docs:
-            document_encoding = tokenizer(
-                document, return_tensors="pt", truncation=True, max_length=512
-            )
-            document_embedding = model(**document_encoding).last_hidden_state
-            # Calculate MaxSim score
-            score = self.maxsim(query_embedding.unsqueeze(0), document_embedding)
-            scores.append(score.item())
-
-        # replace the self.column column with the docs
-        result_set = result_set.drop(self.column)
-        result_set = result_set.append_column(
-            self.column, pa.array(docs, type=pa.string())
-        )
        # add the scores
        result_set = result_set.append_column(
            "_relevance_score", pa.array(scores, type=pa.float32())
@@ -105,36 +88,8 @@ class ColbertReranker(Reranker):
    ):
        result_set = self._rerank(fts_results, query)
        if self.score == "relevance":
-            result_set = result_set.drop_columns(["score"])
+            result_set = result_set.drop_columns(["_score"])

        result_set = result_set.sort_by([("_relevance_score", "descending")])

        return result_set
-
-    @cached_property
-    def _model(self):
-        transformers = attempt_import_or_raise("transformers")
-        tokenizer = transformers.AutoTokenizer.from_pretrained(self.model_name)
-        model = transformers.AutoModel.from_pretrained(self.model_name)
-
-        return tokenizer, model
-
-    def maxsim(self, query_embedding, document_embedding):
-        # Expand dimensions for broadcasting
-        # Query: [batch, length, size] -> [batch, query, 1, size]
-        # Document: [batch, length, size] -> [batch, 1, length, size]
-        expanded_query = query_embedding.unsqueeze(2)
-        expanded_doc = document_embedding.unsqueeze(1)
-
-        # Compute cosine similarity across the embedding dimension
-        sim_matrix = self.torch.nn.functional.cosine_similarity(
-            expanded_query, expanded_doc, dim=-1
-        )
-
-        # Take the maximum similarity for each query token (across all document tokens)
-        # sim_matrix shape: [batch_size, query_length, doc_length]
-        max_sim_scores, _ = self.torch.max(sim_matrix, dim=2)
-
-        # Average these maximum scores across all query tokens
-        avg_max_sim = self.torch.mean(max_sim_scores, dim=1)
-        return avg_max_sim
--- a/python/python/lancedb/rerankers/cross_encoder.py
+++ b/python/python/lancedb/rerankers/cross_encoder.py
@@ -42,7 +42,8 @@ class CrossEncoderReranker(Reranker):
    @cached_property
    def model(self):
        sbert = attempt_import_or_raise("sentence_transformers")
-        cross_encoder = sbert.CrossEncoder(self.model_name)
+        # Allows overriding the automatically selected device
+        cross_encoder = sbert.CrossEncoder(self.model_name, device=self.device)

        return cross_encoder

@@ -96,7 +97,7 @@ class CrossEncoderReranker(Reranker):
    ):
        fts_results = self._rerank(fts_results, query)
        if self.score == "relevance":
-            fts_results = fts_results.drop_columns(["score"])
+            fts_results = fts_results.drop_columns(["_score"])

        fts_results = fts_results.sort_by([("_relevance_score", "descending")])
        return fts_results
--- a/python/python/lancedb/rerankers/jinaai.py
+++ b/python/python/lancedb/rerankers/jinaai.py
@@ -117,6 +117,6 @@ class JinaReranker(Reranker):
    ):
        result_set = self._rerank(fts_results, query)
        if self.score == "relevance":
-            result_set = result_set.drop_columns(["score"])
+            result_set = result_set.drop_columns(["_score"])

        return result_set
--- a/python/python/lancedb/rerankers/linear_combination.py
+++ b/python/python/lancedb/rerankers/linear_combination.py
@@ -69,12 +69,12 @@ class LinearCombinationReranker(Reranker):
            vi = vector_list[i]
            fj = fts_list[j]
            # invert the fts score from relevance to distance
-            inverted_fts_score = self._invert_score(fj["score"])
+            inverted_fts_score = self._invert_score(fj["_score"])
            if vi["_rowid"] == fj["_rowid"]:
                vi["_relevance_score"] = self._combine_score(
                    vi["_distance"], inverted_fts_score
                )
-                vi["score"] = fj["score"]  # keep the original score
+                vi["_score"] = fj["_score"]  # keep the original score
                combined_list.append(vi)
                i += 1
                j += 1
--- a/python/python/lancedb/rerankers/openai.py
+++ b/python/python/lancedb/rerankers/openai.py
@@ -108,7 +108,7 @@ class OpenaiReranker(Reranker):
    def rerank_fts(self, query: str, fts_results: pa.Table):
        fts_results = self._rerank(fts_results, query)
        if self.score == "relevance":
-            fts_results = fts_results.drop_columns(["score"])
+            fts_results = fts_results.drop_columns(["_score"])

        fts_results = fts_results.sort_by([("_relevance_score", "descending")])

--- a/python/python/lancedb/table.py
+++ b/python/python/lancedb/table.py
@@ -1,15 +1,5 @@
-#  Copyright 2023 LanceDB Developers
-#
-#  Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-#  You may obtain a copy of the License at
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-#  limitations under the License.
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors

 from __future__ import annotations

@@ -59,10 +49,9 @@ from .util import (
 if TYPE_CHECKING:
    import PIL
    from lance.dataset import CleanupStats, ReaderLike
-
    from ._lancedb import Table as LanceDBTable, OptimizeStats
    from .db import LanceDBConnection
-    from .index import BTree, IndexConfig, IvfPq
+    from .index import BTree, IndexConfig, IvfPq, Bitmap, LabelList, FTS


 pd = safe_import_pandas()
@@ -352,6 +341,7 @@ class Table(ABC):
        column: str,
        *,
        replace: bool = True,
+        index_type: Literal["BTREE", "BITMAP", "LABEL_LIST"] = "BTREE",
    ):
        """Create a scalar index on a column.

@@ -401,6 +391,8 @@ class Table(ABC):
            or string column.
        replace : bool, default True
            Replace the existing index if it exists.
+        index_type: Literal["BTREE", "BITMAP", "LABEL_LIST"], default "BTREE"
+            The type of index to create.

        Examples
        --------
@@ -413,6 +405,47 @@ class Table(ABC):
        """
        raise NotImplementedError

+    def create_fts_index(
+        self,
+        field_names: Union[str, List[str]],
+        ordering_field_names: Union[str, List[str]] = None,
+        *,
+        replace: bool = False,
+        writer_heap_size: Optional[int] = 1024 * 1024 * 1024,
+        tokenizer_name: str = "default",
+        use_tantivy: bool = True,
+    ):
+        """Create a full-text search index on the table.
+
+        Warning - this API is highly experimental and is highly likely to change
+        in the future.
+
+        Parameters
+        ----------
+        field_names: str or list of str
+            The name(s) of the field to index.
+            can be only str if use_tantivy=True for now.
+        replace: bool, default False
+            If True, replace the existing index if it exists. Note that this is
+            not yet an atomic operation; the index will be temporarily
+            unavailable while the new index is being created.
+        writer_heap_size: int, default 1GB
+            Only available with use_tantivy=True
+        ordering_field_names:
+            A list of unsigned type fields to index to optionally order
+            results on at search time.
+            only available with use_tantivy=True
+        tokenizer_name: str, default "default"
+            The tokenizer to use for the index. Can be "raw", "default" or the 2 letter
+            language code followed by "_stem". So for english it would be "en_stem".
+            For available languages see: https://docs.rs/tantivy/latest/tantivy/tokenizer/enum.Language.html
+            only available with use_tantivy=True for now
+        use_tantivy: bool, default True
+            If True, use the legacy full-text search implementation based on tantivy.
+            If False, use the new full-text search implementation based on lance-index.
+        """
+        raise NotImplementedError
+
    @abstractmethod
    def add(
        self,
@@ -511,6 +544,8 @@ class Table(ABC):
        query: Optional[Union[VEC, str, "PIL.Image.Image", Tuple]] = None,
        vector_column_name: Optional[str] = None,
        query_type: str = "auto",
+        ordering_field_name: Optional[str] = None,
+        fts_columns: Union[str, List[str]] = None,
    ) -> LanceQueryBuilder:
        """Create a search query to find the nearest neighbors
        of the given query vector. We currently support [vector search][search]
@@ -807,6 +842,18 @@ class Table(ABC):
            The names of the columns to drop.
        """

+    @cached_property
+    def _dataset_uri(self) -> str:
+        return _table_uri(self._conn.uri, self.name)
+
+    def _get_fts_index_path(self) -> Tuple[str, pa_fs.FileSystem, bool]:
+        if get_uri_scheme(self._dataset_uri) != "file":
+            return ("", None, False)
+        path = join_uri(self._dataset_uri, "_indices", "fts")
+        fs, path = fs_from_uri(path)
+        index_exists = fs.get_file_info(path).type != pa_fs.FileType.NotFound
+        return (path, fs, index_exists)
+

 class _LanceDatasetRef(ABC):
    @property
@@ -946,10 +993,6 @@ class LanceTable(Table):
        # Cacheable since it's deterministic
        return _table_path(self._conn.uri, self.name)

-    @cached_property
-    def _dataset_uri(self) -> str:
-        return _table_uri(self._conn.uri, self.name)
-
    @property
    def _dataset(self) -> LanceDataset:
        return self._ref.dataset
@@ -1188,9 +1231,15 @@ class LanceTable(Table):
            index_cache_size=index_cache_size,
        )

-    def create_scalar_index(self, column: str, *, replace: bool = True):
+    def create_scalar_index(
+        self,
+        column: str,
+        *,
+        replace: bool = True,
+        index_type: Literal["BTREE", "BITMAP", "LABEL_LIST"] = "BTREE",
+    ):
        self._dataset_mut.create_scalar_index(
-            column, index_type="BTREE", replace=replace
+            column, index_type=index_type, replace=replace
        )

    def create_fts_index(
@@ -1201,29 +1250,21 @@ class LanceTable(Table):
        replace: bool = False,
        writer_heap_size: Optional[int] = 1024 * 1024 * 1024,
        tokenizer_name: str = "default",
+        use_tantivy: bool = True,
    ):
-        """Create a full-text search index on the table.
+        if not use_tantivy:
+            if not isinstance(field_names, str):
+                raise ValueError("field_names must be a string when use_tantivy=False")
+            # delete the existing legacy index if it exists
+            if replace:
+                path, fs, exist = self._get_fts_index_path()
+                if exist:
+                    fs.delete_dir(path)
+            self._dataset_mut.create_scalar_index(
+                field_names, index_type="INVERTED", replace=replace
+            )
+            return

-        Warning - this API is highly experimental and is highly likely to change
-        in the future.
-
-        Parameters
-        ----------
-        field_names: str or list of str
-            The name(s) of the field to index.
-        replace: bool, default False
-            If True, replace the existing index if it exists. Note that this is
-            not yet an atomic operation; the index will be temporarily
-            unavailable while the new index is being created.
-        writer_heap_size: int, default 1GB
-        ordering_field_names:
-            A list of unsigned type fields to index to optionally order
-            results on at search time
-        tokenizer_name: str, default "default"
-            The tokenizer to use for the index. Can be "raw", "default" or the 2 letter
-            language code followed by "_stem". So for english it would be "en_stem".
-            For available languages see: https://docs.rs/tantivy/latest/tantivy/tokenizer/enum.Language.html
-        """
        from .fts import create_index, populate_index

        if isinstance(field_names, str):
@@ -1232,9 +1273,8 @@ class LanceTable(Table):
        if isinstance(ordering_field_names, str):
            ordering_field_names = [ordering_field_names]

-        fs, path = fs_from_uri(self._get_fts_index_path())
-        index_exists = fs.get_file_info(path).type != pa_fs.FileType.NotFound
-        if index_exists:
+        path, fs, exist = self._get_fts_index_path()
+        if exist:
            if not replace:
                raise ValueError("Index already exists. Use replace=True to overwrite.")
            fs.delete_dir(path)
@@ -1245,7 +1285,7 @@ class LanceTable(Table):
            )

        index = create_index(
-            self._get_fts_index_path(),
+            path,
            field_names,
            ordering_fields=ordering_field_names,
            tokenizer_name=tokenizer_name,
@@ -1258,13 +1298,6 @@ class LanceTable(Table):
            writer_heap_size=writer_heap_size,
        )

-    def _get_fts_index_path(self):
-        if get_uri_scheme(self._dataset_uri) != "file":
-            raise NotImplementedError(
-                "Full-text search is not supported on object stores."
-            )
-        return join_uri(self._dataset_uri, "_indices", "tantivy")
-
    def add(
        self,
        data: DATA,
@@ -1392,6 +1425,7 @@ class LanceTable(Table):
        vector_column_name: Optional[str] = None,
        query_type: str = "auto",
        ordering_field_name: Optional[str] = None,
+        fts_columns: Union[str, List[str]] = None,
    ) -> LanceQueryBuilder:
        """Create a search query to find the nearest neighbors
        of the given query vector. We currently support [vector search][search]
@@ -1446,6 +1480,10 @@ class LanceTable(Table):
            or raise an error if no corresponding embedding function is found.
            If the `query` is a string, then the query type is "vector" if the
            table has embedding functions, else the query type is "fts"
+        fts_columns: str or list of str, default None
+            The column(s) to search in for full-text search.
+            If None then the search is performed on all indexed columns.
+            For now, only one column can be searched at a time.

        Returns
        -------
@@ -1455,14 +1493,11 @@ class LanceTable(Table):
            and also the "_distance" column which is the distance between the query
            vector and the returned vector.
        """
-        if vector_column_name is None and query is not None:
+        if vector_column_name is None and query is not None and query_type != "fts":
            try:
                vector_column_name = inf_vector_column_query(self.schema)
            except Exception as e:
-                if query_type == "fts":
-                    vector_column_name = ""
-                else:
-                    raise e
+                raise e

        return LanceQueryBuilder.create(
            self,
@@ -1653,18 +1688,23 @@ class LanceTable(Table):
        self, query: Query, batch_size: Optional[int] = None
    ) -> pa.RecordBatchReader:
        ds = self.to_lance()
-        return ds.scanner(
-            columns=query.columns,
-            filter=query.filter,
-            prefilter=query.prefilter,
-            nearest={
+        nearest = None
+        if len(query.vector) > 0:
+            nearest = {
                "column": query.vector_column,
                "q": query.vector,
                "k": query.k,
                "metric": query.metric,
                "nprobes": query.nprobes,
                "refine_factor": query.refine_factor,
-            },
+            }
+        return ds.scanner(
+            columns=query.columns,
+            limit=query.k,
+            filter=query.filter,
+            prefilter=query.prefilter,
+            nearest=nearest,
+            full_text_query=query.full_text_query,
            with_row_id=query.with_row_id,
            batch_size=batch_size,
        ).to_reader()
@@ -2088,7 +2128,7 @@ class AsyncTable:
        column: str,
        *,
        replace: Optional[bool] = None,
-        config: Optional[Union[IvfPq, BTree]] = None,
+        config: Optional[Union[IvfPq, BTree, Bitmap, LabelList, FTS]] = None,
    ):
        """Create an index to speed up queries

@@ -2413,7 +2453,10 @@ class AsyncTable:
        await self._inner.restore()

    async def optimize(
-        self, *, cleanup_older_than: Optional[timedelta] = None
+        self,
+        *,
+        cleanup_older_than: Optional[timedelta] = None,
+        delete_unverified: bool = False,
    ) -> OptimizeStats:
        """
        Optimize the on-disk data and indices for better performance.
@@ -2432,6 +2475,11 @@ class AsyncTable:
            All files belonging to versions older than this will be removed.  Set
            to 0 days to remove all versions except the latest.  The latest version
            is never removed.
+        delete_unverified: bool, default False
+            Files leftover from a failed transaction may appear to be part of an
+            in-progress operation (e.g. appending new data) and these files will not
+            be deleted unless they are at least 7 days old. If delete_unverified is True
+            then these files will be deleted regardless of their age.

        Experimental API
        ----------------
@@ -2453,7 +2501,7 @@ class AsyncTable:
        """
        if cleanup_older_than is not None:
            cleanup_older_than = round(cleanup_older_than.total_seconds() * 1000)
-        return await self._inner.optimize(cleanup_older_than)
+        return await self._inner.optimize(cleanup_older_than, delete_unverified)

    async def list_indices(self) -> IndexConfig:
        """
--- a/python/python/tests/test_db.py
+++ b/python/python/tests/test_db.py
@@ -22,7 +22,8 @@ import pytest
 from lancedb.pydantic import LanceModel, Vector


-def test_basic(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_basic(tmp_path, use_tantivy):
    db = lancedb.connect(tmp_path)

    assert db.uri == str(tmp_path)
@@ -55,7 +56,7 @@ def test_basic(tmp_path):
    assert len(rs) == 1
    assert rs["item"].iloc[0] == "foo"

-    table.create_fts_index(["item"])
+    table.create_fts_index("item", use_tantivy=use_tantivy)
    rs = table.search("bar", query_type="fts").to_pandas()
    assert len(rs) == 1
    assert rs["item"].iloc[0] == "bar"
--- a/python/python/tests/test_fts.py
+++ b/python/python/tests/test_fts.py
@@ -15,6 +15,7 @@ import random
 from unittest import mock

 import lancedb as ldb
+from lancedb.index import FTS
 import numpy as np
 import pandas as pd
 import pytest
@@ -60,6 +61,43 @@ def table(tmp_path) -> ldb.table.LanceTable:
    return table


+@pytest.fixture
+async def async_table(tmp_path) -> ldb.table.AsyncTable:
+    db = await ldb.connect_async(tmp_path)
+    vectors = [np.random.randn(128) for _ in range(100)]
+
+    nouns = ("puppy", "car", "rabbit", "girl", "monkey")
+    verbs = ("runs", "hits", "jumps", "drives", "barfs")
+    adv = ("crazily.", "dutifully.", "foolishly.", "merrily.", "occasionally.")
+    adj = ("adorable", "clueless", "dirty", "odd", "stupid")
+    text = [
+        " ".join(
+            [
+                nouns[random.randrange(0, 5)],
+                verbs[random.randrange(0, 5)],
+                adv[random.randrange(0, 5)],
+                adj[random.randrange(0, 5)],
+            ]
+        )
+        for _ in range(100)
+    ]
+    count = [random.randint(1, 10000) for _ in range(100)]
+    table = await db.create_table(
+        "test",
+        data=pd.DataFrame(
+            {
+                "vector": vectors,
+                "id": [i % 2 for i in range(100)],
+                "text": text,
+                "text2": text,
+                "nested": [{"text": t} for t in text],
+                "count": count,
+            }
+        ),
+    )
+    return table
+
+
 def test_create_index(tmp_path):
    index = ldb.fts.create_index(str(tmp_path / "index"), ["text"])
    assert isinstance(index, tantivy.Index)
@@ -74,7 +112,12 @@ def test_create_index_with_stemming(tmp_path, table):
    assert os.path.exists(str(tmp_path / "index"))

    # Check stemming by running tokenizer on non empty table
-    table.create_fts_index("text", tokenizer_name="en_stem")
+    table.create_fts_index("text", tokenizer_name="en_stem", use_tantivy=True)
+
+
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_create_inverted_index(table, use_tantivy):
+    table.create_fts_index("text", use_tantivy=use_tantivy)


 def test_populate_index(tmp_path, table):
@@ -86,14 +129,27 @@ def test_search_index(tmp_path, table):
    index = ldb.fts.create_index(str(tmp_path / "index"), ["text"])
    ldb.fts.populate_index(index, table, ["text"])
    index.reload()
-    results = ldb.fts.search_index(index, query="puppy", limit=10)
+    results = ldb.fts.search_index(index, query="puppy", limit=5)
    assert len(results) == 2
-    assert len(results[0]) == 10  # row_ids
-    assert len(results[1]) == 10  # _distance
+    assert len(results[0]) == 5  # row_ids
+    assert len(results[1]) == 5  # _score
+
+
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_search_fts(table, use_tantivy):
+    table.create_fts_index("text", use_tantivy=use_tantivy)
+    results = table.search("puppy").limit(5).to_list()
+    assert len(results) == 5
+
+
+async def test_search_fts_async(async_table):
+    await async_table.create_index("text", config=FTS())
+    results = await async_table.query().nearest_to_text("puppy").limit(5).to_list()
+    assert len(results) == 5


 def test_search_ordering_field_index_table(tmp_path, table):
-    table.create_fts_index("text", ordering_field_names=["count"])
+    table.create_fts_index("text", ordering_field_names=["count"], use_tantivy=True)
    rows = (
        table.search("puppy", ordering_field_name="count")
        .limit(20)
@@ -113,11 +169,11 @@ def test_search_ordering_field_index(tmp_path, table):
    ldb.fts.populate_index(index, table, ["text"], ordering_fields=["count"])
    index.reload()
    results = ldb.fts.search_index(
-        index, query="puppy", limit=10, ordering_field="count"
+        index, query="puppy", limit=5, ordering_field="count"
    )
    assert len(results) == 2
-    assert len(results[0]) == 10  # row_ids
-    assert len(results[1]) == 10  # _distance
+    assert len(results[0]) == 5  # row_ids
+    assert len(results[1]) == 5  # _distance
    rows = table.to_lance().take(results[0]).to_pylist()

    for r in rows:
@@ -125,10 +181,11 @@ def test_search_ordering_field_index(tmp_path, table):
    assert sorted(rows, key=lambda x: x["count"], reverse=True) == rows


-def test_create_index_from_table(tmp_path, table):
-    table.create_fts_index("text")
-    df = table.search("puppy").limit(10).select(["text"]).to_pandas()
-    assert len(df) <= 10
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_create_index_from_table(tmp_path, table, use_tantivy):
+    table.create_fts_index("text", use_tantivy=use_tantivy)
+    df = table.search("puppy").limit(5).select(["text"]).to_pandas()
+    assert len(df) <= 5
    assert "text" in df.columns

    # Check whether it can be updated
@@ -145,36 +202,37 @@ def test_create_index_from_table(tmp_path, table):
        ]
    )

-    with pytest.raises(ValueError, match="already exists"):
-        table.create_fts_index("text")
+    with pytest.raises(Exception, match="already exists"):
+        table.create_fts_index("text", use_tantivy=use_tantivy)

-    table.create_fts_index("text", replace=True)
+    table.create_fts_index("text", replace=True, use_tantivy=use_tantivy)
    assert len(table.search("gorilla").limit(1).to_pandas()) == 1


 def test_create_index_multiple_columns(tmp_path, table):
-    table.create_fts_index(["text", "text2"])
-    df = table.search("puppy").limit(10).to_pandas()
-    assert len(df) == 10
+    table.create_fts_index(["text", "text2"], use_tantivy=True)
+    df = table.search("puppy").limit(5).to_pandas()
+    assert len(df) == 5
    assert "text" in df.columns
    assert "text2" in df.columns


 def test_empty_rs(tmp_path, table, mocker):
-    table.create_fts_index(["text", "text2"])
+    table.create_fts_index(["text", "text2"], use_tantivy=True)
    mocker.patch("lancedb.fts.search_index", return_value=([], []))
-    df = table.search("puppy").limit(10).to_pandas()
+    df = table.search("puppy").limit(5).to_pandas()
    assert len(df) == 0


 def test_nested_schema(tmp_path, table):
-    table.create_fts_index("nested.text")
-    rs = table.search("puppy").limit(10).to_list()
-    assert len(rs) == 10
+    table.create_fts_index("nested.text", use_tantivy=True)
+    rs = table.search("puppy").limit(5).to_list()
+    assert len(rs) == 5


-def test_search_index_with_filter(table):
-    table.create_fts_index("text")
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_search_index_with_filter(table, use_tantivy):
+    table.create_fts_index("text", use_tantivy=use_tantivy)
    orig_import = __import__

    def import_mock(name, *args):
@@ -186,7 +244,7 @@ def test_search_index_with_filter(table):
    with mock.patch("builtins.__import__", side_effect=import_mock):
        rs = table.search("puppy").where("id=1").limit(10)
        # test schema
-        assert rs.to_arrow().drop("score").schema.equals(table.schema)
+        assert rs.to_arrow().drop("_score").schema.equals(table.schema)

        rs = rs.to_list()
        for r in rs:
@@ -204,7 +262,8 @@ def test_search_index_with_filter(table):
        assert r["_rowid"] is not None


-def test_null_input(table):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_null_input(table, use_tantivy):
    table.add(
        [
            {
@@ -217,12 +276,12 @@ def test_null_input(table):
            }
        ]
    )
-    table.create_fts_index("text")
+    table.create_fts_index("text", use_tantivy=use_tantivy)


 def test_syntax(table):
    # https://github.com/lancedb/lancedb/issues/769
-    table.create_fts_index("text")
+    table.create_fts_index("text", use_tantivy=True)
    with pytest.raises(ValueError, match="Syntax Error"):
        table.search("they could have been dogs OR").limit(10).to_list()

--- a/python/python/tests/test_index.py
+++ b/python/python/tests/test_index.py
@@ -1,10 +1,14 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright The LanceDB Authors
+
 from datetime import timedelta
+import random

 import pyarrow as pa
 import pytest
 import pytest_asyncio
 from lancedb import AsyncConnection, AsyncTable, connect_async
-from lancedb.index import BTree, IvfPq
+from lancedb.index import BTree, IvfPq, Bitmap, LabelList


@pytest_asyncio.fixture
@@ -25,8 +29,11 @@ NROWS = 256
 async def some_table(db_async):
    data = pa.Table.from_pydict(
        {
-            "id": list(range(256)),
+            "id": list(range(NROWS)),
            "vector": sample_fixed_size_list_array(NROWS, DIM),
+            "tags": [
+                [f"tag{random.randint(0, 8)}" for _ in range(2)] for _ in range(NROWS)
+            ],
        }
    )
    return await db_async.create_table(
@@ -53,6 +60,22 @@ async def test_create_scalar_index(some_table: AsyncTable):
    await some_table.create_index("id", config=BTree())


+@pytest.mark.asyncio
+async def test_create_bitmap_index(some_table: AsyncTable):
+    await some_table.create_index("id", config=Bitmap())
+    # TODO: Fix via https://github.com/lancedb/lance/issues/2039
+    # indices = await some_table.list_indices()
+    # assert str(indices) == '[Index(Bitmap, columns=["id"])]'
+
+
+@pytest.mark.asyncio
+async def test_create_label_list_index(some_table: AsyncTable):
+    await some_table.create_index("tags", config=LabelList())
+    # TODO: Fix via https://github.com/lancedb/lance/issues/2039
+    # indices = await some_table.list_indices()
+    # assert str(indices) == '[Index(LabelList, columns=["id"])]'
+
+
@pytest.mark.asyncio
 async def test_create_vector_index(some_table: AsyncTable):
    # Can create
--- a/python/python/tests/test_query.py
+++ b/python/python/tests/test_query.py
@@ -354,3 +354,11 @@ async def test_query_camelcase_async(tmp_path):

    result = await table.query().select(["camelCase"]).to_arrow()
    assert result == pa.table({"camelCase": pa.array([1, 2])})
+
+
+@pytest.mark.asyncio
+async def test_query_to_list_async(table_async: AsyncTable):
+    list = await table_async.query().to_list()
+    assert len(list) == 2
+    assert list[0]["vector"] == [1, 2]
+    assert list[1]["vector"] == [3, 4]
--- a/python/python/tests/test_rerankers.py
+++ b/python/python/tests/test_rerankers.py
@@ -22,7 +22,7 @@ from lancedb.table import LanceTable
 pytest.importorskip("lancedb.fts")


-def get_test_table(tmp_path):
+def get_test_table(tmp_path, use_tantivy):
    db = lancedb.connect(tmp_path)
    # Create a LanceDB table schema with a vector and a text column
    emb = EmbeddingFunctionRegistry.get_instance().get("test")()
@@ -89,7 +89,7 @@ def get_test_table(tmp_path):
    )

    # Create a fts index
-    table.create_fts_index("text")
+    table.create_fts_index("text", use_tantivy=use_tantivy)

    return table, MyTable

@@ -174,8 +174,8 @@ def _run_test_reranker(reranker, table, query, query_vector, schema):
    assert len(result) == 20 and result == result_arrow


-def _run_test_hybrid_reranker(reranker, tmp_path):
-    table, schema = get_test_table(tmp_path)
+def _run_test_hybrid_reranker(reranker, tmp_path, use_tantivy):
+    table, schema = get_test_table(tmp_path, use_tantivy)
    # The default reranker
    result1 = (
        table.search(
@@ -221,46 +221,52 @@ def _run_test_hybrid_reranker(reranker, tmp_path):
    )


-def test_linear_combination(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_linear_combination(tmp_path, use_tantivy):
    reranker = LinearCombinationReranker()
-    _run_test_hybrid_reranker(reranker, tmp_path)
+    _run_test_hybrid_reranker(reranker, tmp_path, use_tantivy)


-def test_rrf_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_rrf_reranker(tmp_path, use_tantivy):
    reranker = RRFReranker()
-    _run_test_hybrid_reranker(reranker, tmp_path)
+    _run_test_hybrid_reranker(reranker, tmp_path, use_tantivy)


@pytest.mark.skipif(
    os.environ.get("COHERE_API_KEY") is None, reason="COHERE_API_KEY not set"
 )
-def test_cohere_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_cohere_reranker(tmp_path, use_tantivy):
    pytest.importorskip("cohere")
    reranker = CohereReranker()
-    table, schema = get_test_table(tmp_path)
+    table, schema = get_test_table(tmp_path, use_tantivy)
    _run_test_reranker(reranker, table, "single player experience", None, schema)


-def test_cross_encoder_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_cross_encoder_reranker(tmp_path, use_tantivy):
    pytest.importorskip("sentence_transformers")
    reranker = CrossEncoderReranker()
-    table, schema = get_test_table(tmp_path)
+    table, schema = get_test_table(tmp_path, use_tantivy)
    _run_test_reranker(reranker, table, "single player experience", None, schema)


-def test_colbert_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_colbert_reranker(tmp_path, use_tantivy):
    pytest.importorskip("transformers")
    reranker = ColbertReranker()
-    table, schema = get_test_table(tmp_path)
+    table, schema = get_test_table(tmp_path, use_tantivy)
    _run_test_reranker(reranker, table, "single player experience", None, schema)


@pytest.mark.skipif(
    os.environ.get("OPENAI_API_KEY") is None, reason="OPENAI_API_KEY not set"
 )
-def test_openai_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_openai_reranker(tmp_path, use_tantivy):
    pytest.importorskip("openai")
-    table, schema = get_test_table(tmp_path)
+    table, schema = get_test_table(tmp_path, use_tantivy)
    reranker = OpenaiReranker()
    _run_test_reranker(reranker, table, "single player experience", None, schema)

@@ -268,8 +274,9 @@ def test_openai_reranker(tmp_path):
@pytest.mark.skipif(
    os.environ.get("JINA_API_KEY") is None, reason="JINA_API_KEY not set"
 )
-def test_jina_reranker(tmp_path):
+@pytest.mark.parametrize("use_tantivy", [True, False])
+def test_jina_reranker(tmp_path, use_tantivy):
    pytest.importorskip("jina")
-    table, schema = get_test_table(tmp_path)
+    table, schema = get_test_table(tmp_path, use_tantivy)
    reranker = JinaReranker()
    _run_test_reranker(reranker, table, "single player experience", None, schema)
--- a/python/python/tests/test_s3.py
+++ b/python/python/tests/test_s3.py
@@ -251,7 +251,8 @@ def test_s3_dynamodb_sync(s3_bucket: str, commit_table: str, monkeypatch):

    # FTS indices should error since they are not supported yet.
    with pytest.raises(
-        NotImplementedError, match="Full-text search is not supported on object stores."
+        NotImplementedError,
+        match="Full-text search is only supported on the local filesystem",
    ):
        table.create_fts_index("x")

--- a/python/python/tests/test_table.py
+++ b/python/python/tests/test_table.py
@@ -8,6 +8,7 @@ from pathlib import Path
 from time import sleep
 from typing import List
 from unittest.mock import PropertyMock, patch
+import os

 import lance
 import lancedb
@@ -27,7 +28,7 @@ from pydantic import BaseModel

 class MockDB:
    def __init__(self, uri: Path):
-        self.uri = uri
+        self.uri = str(uri)
        self.read_consistency_interval = None

    @functools.cached_property
@@ -1052,3 +1053,25 @@ async def test_optimize(db_async: AsyncConnection):
    assert stats.prune.old_versions_removed == 3

    assert await table.query().to_arrow() == pa.table({"x": [[1], [2]]})
+
+
+@pytest.mark.asyncio
+async def test_optimize_delete_unverified(db_async: AsyncConnection, tmp_path):
+    table = await db_async.create_table(
+        "test",
+        data=[{"x": [1]}],
+    )
+    await table.add(
+        data=[
+            {"x": [2]},
+        ],
+    )
+    version = await table.version()
+    path = tmp_path / "test.lance" / "_versions" / f"{version - 1}.manifest"
+    os.remove(path)
+    stats = await table.optimize(delete_unverified=False)
+    assert stats.prune.old_versions_removed == 0
+    stats = await table.optimize(
+        cleanup_older_than=timedelta(seconds=0), delete_unverified=True
+    )
+    assert stats.prune.old_versions_removed == 2
--- a/python/src/index.rs
+++ b/python/src/index.rs
@@ -84,6 +84,20 @@ impl Index {
            inner: Mutex::new(Some(LanceDbIndex::BTree(BTreeIndexBuilder::default()))),
        })
    }
+
+    #[staticmethod]
+    pub fn bitmap() -> PyResult<Self> {
+        Ok(Self {
+            inner: Mutex::new(Some(LanceDbIndex::Bitmap(Default::default()))),
+        })
+    }
+
+    #[staticmethod]
+    pub fn label_list() -> PyResult<Self> {
+        Ok(Self {
+            inner: Mutex::new(Some(LanceDbIndex::LabelList(Default::default()))),
+        })
+    }
 }

 #[pyclass(get_all)]
--- a/python/src/query.rs
+++ b/python/src/query.rs
@@ -15,17 +15,20 @@
 use arrow::array::make_array;
 use arrow::array::ArrayData;
 use arrow::pyarrow::FromPyArrow;
+use lancedb::index::scalar::FullTextSearchQuery;
 use lancedb::query::QueryExecutionOptions;
 use lancedb::query::{
    ExecutableQuery, Query as LanceDbQuery, QueryBase, Select, VectorQuery as LanceDbVectorQuery,
 };
 use pyo3::exceptions::PyRuntimeError;
-use pyo3::pyclass;
+use pyo3::prelude::{PyAnyMethods, PyDictMethods};
 use pyo3::pymethods;
+use pyo3::types::PyDict;
 use pyo3::Bound;
 use pyo3::PyAny;
 use pyo3::PyRef;
 use pyo3::PyResult;
+use pyo3::{pyclass, PyErr};
 use pyo3_asyncio_0_21::tokio::future_into_py;

 use crate::arrow::RecordBatchStream;
@@ -68,6 +71,24 @@ impl Query {
        Ok(VectorQuery { inner })
    }

+    pub fn nearest_to_text(&mut self, query: Bound<'_, PyDict>) -> PyResult<()> {
+        let query_text = query
+            .get_item("query")?
+            .ok_or(PyErr::new::<PyRuntimeError, _>(
+                "Query text is required for nearest_to_text",
+            ))?
+            .extract::<String>()?;
+        let columns = query
+            .get_item("columns")?
+            .map(|columns| columns.extract::<Vec<String>>())
+            .transpose()?;
+
+        let fts_query = FullTextSearchQuery::new(query_text).columns(columns);
+        self.inner = self.inner.clone().full_text_search(fts_query);
+
+        Ok(())
+    }
+
    pub fn execute(
        self_: PyRef<'_, Self>,
        max_batch_length: Option<u32>,
--- a/python/src/table.rs
+++ b/python/src/table.rs
@@ -248,6 +248,7 @@ impl Table {
    pub fn optimize(
        self_: PyRef<'_, Self>,
        cleanup_since_ms: Option<u64>,
+        delete_unverified: Option<bool>,
    ) -> PyResult<Bound<'_, PyAny>> {
        let inner = self_.inner_ref()?.clone();
        let older_than = if let Some(ms) = cleanup_since_ms {
@@ -275,7 +276,7 @@ impl Table {
            let prune_stats = inner
                .optimize(OptimizeAction::Prune {
                    older_than,
-                    delete_unverified: None,
+                    delete_unverified,
                    error_if_tagged_old_versions: None,
                })
                .await
--- a/rust/ffi/node/Cargo.toml
+++ b/rust/ffi/node/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-node"
-version = "0.8.0"
+version = "0.10.0-beta.0"
 description = "Serverless, low-latency vector database for AI applications"
 license.workspace = true
 edition.workspace = true
--- a/rust/lancedb/Cargo.toml
+++ b/rust/lancedb/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.8.0"
+version = "0.10.0-beta.0"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
@@ -56,6 +56,7 @@ tokenizers = { version = "0.19.1", optional = true }
 [dev-dependencies]
 tempfile = "3.5.0"
 rand = { version = "0.8.3", features = ["small_rng"] }
+random_word = { version = "0.4.3", features = ["en"] }
 uuid = { version = "1.7.0", features = ["v4"] }
 walkdir = "2"
 aws-sdk-dynamodb = { version = "1.38.0" }
--- a/rust/lancedb/examples/full_text_search.rs
+++ b/rust/lancedb/examples/full_text_search.rs
@@ -0,0 +1,114 @@
+// Copyright 2024 Lance Developers.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::sync::Arc;
+
+use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, RecordBatchReader, StringArray};
+use arrow_schema::{DataType, Field, Schema};
+
+use futures::TryStreamExt;
+use lance_index::scalar::FullTextSearchQuery;
+use lancedb::connection::Connection;
+use lancedb::index::scalar::FtsIndexBuilder;
+use lancedb::index::Index;
+use lancedb::query::{ExecutableQuery, QueryBase};
+use lancedb::{connect, Result, Table};
+use rand::random;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    if std::path::Path::new("data").exists() {
+        std::fs::remove_dir_all("data").unwrap();
+    }
+    let uri = "data/sample-lancedb";
+    let db = connect(uri).execute().await?;
+    let tbl = create_table(&db).await?;
+
+    create_index(&tbl).await?;
+    search_index(&tbl).await?;
+    Ok(())
+}
+
+fn create_some_records() -> Result<Box<dyn RecordBatchReader + Send>> {
+    const TOTAL: usize = 1000;
+
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("id", DataType::Int32, false),
+        Field::new("doc", DataType::Utf8, true),
+    ]));
+
+    let words = random_word::all(random_word::Lang::En)
+        .iter()
+        .step_by(1024)
+        .take(500)
+        .map(|w| *w)
+        .collect::<Vec<_>>();
+    let n_terms = 3;
+    let batches = RecordBatchIterator::new(
+        vec![RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(Int32Array::from_iter_values(0..TOTAL as i32)),
+                Arc::new(StringArray::from_iter_values((0..TOTAL).map(|_| {
+                    (0..n_terms)
+                        .map(|_| words[random::<usize>() % words.len()])
+                        .collect::<Vec<_>>()
+                        .join(" ")
+                }))),
+            ],
+        )
+        .unwrap()]
+        .into_iter()
+        .map(Ok),
+        schema.clone(),
+    );
+    Ok(Box::new(batches))
+}
+
+async fn create_table(db: &Connection) -> Result<Table> {
+    let initial_data: Box<dyn RecordBatchReader + Send> = create_some_records()?;
+    let tbl = db.create_table("my_table", initial_data).execute().await?;
+    Ok(tbl)
+}
+
+async fn create_index(table: &Table) -> Result<()> {
+    table
+        .create_index(&["doc"], Index::FTS(FtsIndexBuilder::default()))
+        .execute()
+        .await?;
+    Ok(())
+}
+
+async fn search_index(table: &Table) -> Result<()> {
+    let words = random_word::all(random_word::Lang::En)
+        .iter()
+        .step_by(1024)
+        .take(500)
+        .map(|w| *w)
+        .collect::<Vec<_>>();
+    let query = words[0].to_owned();
+    println!("Searching for: {}", query);
+
+    let mut results = table
+        .query()
+        .full_text_search(FullTextSearchQuery::new(words[0].to_owned()))
+        .select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
+        .limit(10)
+        .execute()
+        .await?;
+    while let Some(batch) = results.try_next().await? {
+        println!("{:?}", batch);
+    }
+    Ok(())
+}
--- a/rust/lancedb/src/connection.rs
+++ b/rust/lancedb/src/connection.rs
@@ -1217,7 +1217,7 @@ mod tests {

        let tbl = db
            .create_table("v2_test", make_data())
-            .use_legacy_format(false)
+            .data_storage_version(LanceFileVersion::Stable)
            .execute()
            .await
            .unwrap();
--- a/rust/lancedb/src/index.rs
+++ b/rust/lancedb/src/index.rs
@@ -14,24 +14,54 @@

 use std::sync::Arc;

+use scalar::FtsIndexBuilder;
 use serde::Deserialize;
 use serde_with::skip_serializing_none;

 use crate::{table::TableInternal, Result};

 use self::{
-    scalar::BTreeIndexBuilder,
+    scalar::{BTreeIndexBuilder, BitmapIndexBuilder, LabelListIndexBuilder},
    vector::{IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder},
 };

 pub mod scalar;
 pub mod vector;

+/// Supported index types.
 pub enum Index {
    Auto,
+    /// A `BTree` index is an sorted index on scalar columns.
+    /// This index is good for scalar columns with mostly distinct values and does best when
+    /// the query is highly selective. It can apply to numeric, temporal, and string columns.
+    ///
+    /// BTree index is useful to answer queries with
+    /// equality (`=`), inequality (`>`, `>=`, `<`, `<=`),and range queries.
+    ///
+    /// This is the default index type for scalar columns.
    BTree(BTreeIndexBuilder),
+
+    /// A `Bitmap` index stores a bitmap for each distinct value in the column for every row.
+    ///
+    /// This index works best for low-cardinality columns,
+    /// where the number of unique values is small (i.e., less than a few hundreds).
+    Bitmap(BitmapIndexBuilder),
+
+    /// [LabelListIndexBuilder] is a scalar index that can be used on `List<T>` columns to
+    /// support queries with `array_contains_all` and `array_contains_any`
+    /// using an underlying bitmap index.
+    LabelList(LabelListIndexBuilder),
+
+    /// Full text search index using bm25.
+    FTS(FtsIndexBuilder),
+
+    /// IVF index with Product Quantization
    IvfPq(IvfPqIndexBuilder),
+
+    /// IVF-HNSW index with Product Quantization
    IvfHnswPq(IvfHnswPqIndexBuilder),
+
+    /// IVF-HNSW index with Scalar Quantization
    IvfHnswSq(IvfHnswSqIndexBuilder),
 }

@@ -72,10 +102,14 @@ impl IndexBuilder {

 #[derive(Debug, Clone, PartialEq)]
 pub enum IndexType {
+    // Vector
    IvfPq,
    IvfHnswPq,
    IvfHnswSq,
+    // Scalar
    BTree,
+    Bitmap,
+    LabelList,
 }

 /// A description of an index currently configured on a column
--- a/rust/lancedb/src/index/scalar.rs
+++ b/rust/lancedb/src/index/scalar.rs
@@ -28,3 +28,32 @@
 pub struct BTreeIndexBuilder {}

 impl BTreeIndexBuilder {}
+
+/// Builder for a Bitmap index.
+///
+/// It is a scalar index that stores a bitmap for each possible value
+///
+/// This index works best for low-cardinality (i.e., less than 1000 unique values) columns,
+/// where the number of unique values is small.
+/// The bitmap stores a list of row ids where the value is present.
+#[derive(Debug, Clone, Default)]
+pub struct BitmapIndexBuilder {}
+
+/// Builder for LabelList index.
+///
+/// [LabeListIndexBuilder] is a scalar index that can be used on `List<T>` columns to
+/// support queries with `array_contains_all` and `array_contains_any`
+/// using an underlying bitmap index.
+///
+#[derive(Debug, Clone, Default)]
+pub struct LabelListIndexBuilder {}
+
+/// Builder for a full text search index
+///
+/// A full text search index is an index on a string column that allows for full text search
+#[derive(Debug, Clone, Default)]
+pub struct FtsIndexBuilder {}
+
+impl FtsIndexBuilder {}
+
+pub use lance_index::scalar::FullTextSearchQuery;
--- a/rust/lancedb/src/lib.rs
+++ b/rust/lancedb/src/lib.rs
@@ -13,7 +13,7 @@
 // limitations under the License.

 //! [LanceDB](https://github.com/lancedb/lancedb) is an open-source database for vector-search built with persistent storage,
-//! which greatly simplifies retrevial, filtering and management of embeddings.
+//! which greatly simplifies retrieval, filtering and management of embeddings.
 //!
 //! The key features of LanceDB include:
 //! - Production-scale vector search with no servers to manage.
@@ -133,6 +133,13 @@
 //!
 //! #### Create vector index (IVF_PQ)
 //!
+//! LanceDB is capable to automatically create appropriate indices based on the data types
+//! of the columns. For example,
+//!
+//! * If a column has a data type of `FixedSizeList<Float16/Float32>`,
+//!   LanceDB will create a `IVF-PQ` vector index with default parameters.
+//! * Otherwise, it creates a `BTree` index by default.
+//!
 //! ```no_run
 //! # use std::sync::Arc;
 //! # use arrow_array::{FixedSizeListArray, types::Float32Type, RecordBatch,
@@ -150,7 +157,10 @@
 //! # });
 //! ```
 //!
-//! #### Open table and run search
+//!
+//! User can also specify the index type explicitly, see [`Table::create_index`].
+//!
+//! #### Open table and search
 //!
 //! ```rust
 //! # use std::sync::Arc;
--- a/rust/lancedb/src/query.rs
+++ b/rust/lancedb/src/query.rs
@@ -21,6 +21,7 @@ use datafusion_physical_plan::ExecutionPlan;
 use half::f16;
 use lance::dataset::scanner::DatasetRecordBatchStream;
 use lance_datafusion::exec::execute_plan;
+use lance_index::scalar::FullTextSearchQuery;

 use crate::arrow::SendableRecordBatchStream;
 use crate::error::{Error, Result};
@@ -351,6 +352,17 @@ pub trait QueryBase {
    /// on the filter column(s).
    fn only_if(self, filter: impl AsRef<str>) -> Self;

+    /// Perform a full text search on the table.
+    ///
+    /// The results will be returned in order of BM25 scores.
+    ///
+    /// This method is only valid on tables that have a full text search index.
+    ///
+    /// ```ignore
+    /// query.full_text_search(FullTextSearchQuery::new("hello world"))
+    /// ```
+    fn full_text_search(self, query: FullTextSearchQuery) -> Self;
+
    /// Return only the specified columns.
    ///
    /// By default a query will return all columns from the table.  However, this can have
@@ -401,6 +413,11 @@ impl<T: HasQuery> QueryBase for T {
        self
    }

+    fn full_text_search(mut self, query: FullTextSearchQuery) -> Self {
+        self.mut_query().full_text_search = Some(query);
+        self
+    }
+
    fn select(mut self, select: Select) -> Self {
        self.mut_query().select = select;
        self
@@ -502,8 +519,13 @@ pub struct Query {

    /// limit the number of rows to return.
    pub(crate) limit: Option<usize>,
+
    /// Apply filter to the returned rows.
    pub(crate) filter: Option<String>,
+
+    /// Perform a full text search on the table.
+    pub(crate) full_text_search: Option<FullTextSearchQuery>,
+
    /// Select column projection.
    pub(crate) select: Select,

@@ -520,6 +542,7 @@ impl Query {
            parent,
            limit: None,
            filter: None,
+            full_text_search: None,
            select: Select::All,
            fast_search: false,
        }
--- a/rust/lancedb/src/table.rs
+++ b/rust/lancedb/src/table.rs
@@ -573,7 +573,8 @@ impl Table {
    /// There are a variety of indices available.  They are described more in
    /// [`crate::index::Index`].  The simplest thing to do is to use `index::Index::Auto` which
    /// will attempt to create the most useful index based on the column type and column
-    /// statistics.
+    /// statistics. `BTree` index is created by default for numeric, temporal, and
+    /// string columns.
    ///
    /// Once an index is created it will remain until the data is overwritten (e.g. an
    /// add operation with mode overwrite) or the indexed column is dropped.
@@ -607,10 +608,21 @@ impl Table {
    ///     .await
    ///     .unwrap();
    /// # let tbl = db.open_table("idx_test").execute().await.unwrap();
+    /// // Create IVF PQ index on the "vector" column by default.
    /// tbl.create_index(&["vector"], Index::Auto)
    ///    .execute()
    ///    .await
    ///    .unwrap();
+    /// // Create a BTree index on the "id" column.
+    /// tbl.create_index(&["id"], Index::Auto)
+    ///     .execute()
+    ///     .await
+    ///     .unwrap();
+    /// // Create a LabelList index on the "tags" column.
+    /// tbl.create_index(&["tags"], Index::LabelList(Default::default()))
+    ///     .execute()
+    ///     .await
+    ///     .unwrap();
    /// # });
    /// ```
    pub fn create_index(&self, columns: &[impl AsRef<str>], index: Index) -> IndexBuilder {
@@ -1054,6 +1066,24 @@ impl NativeTable {
            )
    }

+    fn supported_bitmap_data_type(dtype: &DataType) -> bool {
+        dtype.is_integer() || matches!(dtype, DataType::Utf8)
+    }
+
+    fn supported_label_list_data_type(dtype: &DataType) -> bool {
+        match dtype {
+            DataType::List(field) => Self::supported_bitmap_data_type(field.data_type()),
+            DataType::FixedSizeList(field, _) => {
+                Self::supported_bitmap_data_type(field.data_type())
+            }
+            _ => false,
+        }
+    }
+
+    fn supported_fts_data_type(dtype: &DataType) -> bool {
+        matches!(dtype, DataType::Utf8 | DataType::LargeUtf8)
+    }
+
    fn supported_vector_data_type(dtype: &DataType) -> bool {
        match dtype {
            DataType::FixedSizeList(inner, _) => DataType::is_floating(inner.data_type()),
@@ -1512,6 +1542,87 @@ impl NativeTable {
        let lance_idx_params = lance_index::scalar::ScalarIndexParams {
            force_index_type: Some(lance_index::scalar::ScalarIndexType::BTree),
        };
+        dataset
+            .create_index(
+                &[field.name()],
+                IndexType::BTree,
+                None,
+                &lance_idx_params,
+                opts.replace,
+            )
+            .await?;
+        Ok(())
+    }
+
+    async fn create_bitmap_index(&self, field: &Field, opts: IndexBuilder) -> Result<()> {
+        if !Self::supported_bitmap_data_type(field.data_type()) {
+            return Err(Error::Schema {
+                message: format!(
+                    "A Bitmap index cannot be created on the field `{}` which has data type {}",
+                    field.name(),
+                    field.data_type()
+                ),
+            });
+        }
+
+        let mut dataset = self.dataset.get_mut().await?;
+        let lance_idx_params = lance_index::scalar::ScalarIndexParams {
+            force_index_type: Some(lance_index::scalar::ScalarIndexType::Bitmap),
+        };
+        dataset
+            .create_index(
+                &[field.name()],
+                IndexType::Bitmap,
+                None,
+                &lance_idx_params,
+                opts.replace,
+            )
+            .await?;
+        Ok(())
+    }
+
+    async fn create_label_list_index(&self, field: &Field, opts: IndexBuilder) -> Result<()> {
+        if !Self::supported_label_list_data_type(field.data_type()) {
+            return Err(Error::Schema {
+                message: format!(
+                    "A LabelList index cannot be created on the field `{}` which has data type {}",
+                    field.name(),
+                    field.data_type()
+                ),
+            });
+        }
+
+        let mut dataset = self.dataset.get_mut().await?;
+        let lance_idx_params = lance_index::scalar::ScalarIndexParams {
+            force_index_type: Some(lance_index::scalar::ScalarIndexType::LabelList),
+        };
+        dataset
+            .create_index(
+                &[field.name()],
+                IndexType::LabelList,
+                None,
+                &lance_idx_params,
+                opts.replace,
+            )
+            .await?;
+        Ok(())
+    }
+
+    async fn create_fts_index(&self, field: &Field, opts: IndexBuilder) -> Result<()> {
+        if !Self::supported_fts_data_type(field.data_type()) {
+            return Err(Error::Schema {
+                message: format!(
+                    "A FTS index cannot be created on the field `{}` which has data type {}",
+                    field.name(),
+                    field.data_type()
+                ),
+            });
+        }
+
+        let mut dataset = self.dataset.get_mut().await?;
+        let lance_idx_params = lance_index::scalar::ScalarIndexParams {
+            force_index_type: Some(lance_index::scalar::ScalarIndexType::Inverted),
+        };
        dataset
            .create_index(
                &[field.name()],
@@ -1659,6 +1770,9 @@ impl TableInternal for NativeTable {
        match opts.index {
            Index::Auto => self.create_auto_index(field, opts).await,
            Index::BTree(_) => self.create_btree_index(field, opts).await,
+            Index::Bitmap(_) => self.create_bitmap_index(field, opts).await,
+            Index::LabelList(_) => self.create_label_list_index(field, opts).await,
+            Index::FTS(_) => self.create_fts_index(field, opts).await,
            Index::IvfPq(ivf_pq) => self.create_ivf_pq_index(ivf_pq, field, opts.replace).await,
            Index::IvfHnswPq(ivf_hnsw_pq) => {
                self.create_ivf_hnsw_pq_index(ivf_hnsw_pq, field, opts.replace)
@@ -1789,6 +1903,10 @@ impl TableInternal for NativeTable {
            scanner.filter(filter)?;
        }

+        if let Some(fts) = &query.base.full_text_search {
+            scanner.full_text_search(fts.clone())?;
+        }
+
        if let Some(refine_factor) = query.refine_factor {
            scanner.refine(refine_factor);
        }
@@ -1977,6 +2095,7 @@ mod tests {
    use std::time::Duration;

    use arrow_array::{
+        builder::{ListBuilder, StringBuilder},
        Array, BooleanArray, Date32Array, FixedSizeListArray, Float32Array, Float64Array,
        Int32Array, Int64Array, LargeStringArray, RecordBatch, RecordBatchIterator,
        RecordBatchReader, StringArray, TimestampMillisecondArray, TimestampNanosecondArray,
@@ -1986,17 +2105,17 @@ mod tests {
    use arrow_schema::{DataType, Field, Schema, TimeUnit};
    use futures::TryStreamExt;
    use lance::dataset::{Dataset, WriteMode};
+    use lance::index::DatasetIndexInternalExt;
    use lance::io::{ObjectStoreParams, WrappingObjectStore};
    use rand::Rng;
    use tempfile::tempdir;

+    use super::*;
    use crate::connect;
    use crate::connection::ConnectBuilder;
    use crate::index::scalar::BTreeIndexBuilder;
    use crate::query::{ExecutableQuery, QueryBase};

-    use super::*;
-
    #[tokio::test]
    async fn test_open() {
        let tmp_dir = tempdir().unwrap();
@@ -2961,6 +3080,151 @@ mod tests {
        );
    }

+    #[tokio::test]
+    async fn test_create_bitmap_index() {
+        let tmp_dir = tempdir().unwrap();
+        let uri = tmp_dir.path().to_str().unwrap();
+
+        let conn = ConnectBuilder::new(uri).execute().await.unwrap();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new("category", DataType::Utf8, true),
+        ]));
+
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(Int32Array::from_iter_values(0..100)),
+                Arc::new(StringArray::from_iter_values(
+                    (0..100).map(|i| format!("category_{}", i % 5)),
+                )),
+            ],
+        )
+        .unwrap();
+
+        let table = conn
+            .create_table(
+                "test_bitmap",
+                RecordBatchIterator::new(vec![Ok(batch.clone())], batch.schema()),
+            )
+            .execute()
+            .await
+            .unwrap();
+
+        // Create bitmap index on the "category" column
+        table
+            .create_index(&["category"], Index::Bitmap(Default::default()))
+            .execute()
+            .await
+            .unwrap();
+
+        // Verify the index was created
+        let index_configs = table.list_indices().await.unwrap();
+        assert_eq!(index_configs.len(), 1);
+        let index = index_configs.into_iter().next().unwrap();
+        // TODO: Fix via https://github.com/lancedb/lance/issues/2039
+        // assert_eq!(index.index_type, crate::index::IndexType::Bitmap);
+        assert_eq!(index.columns, vec!["category".to_string()]);
+
+        // For now, just open the index to verify its type
+        let lance_dataset = table.as_native().unwrap().dataset.get().await.unwrap();
+        let indices = lance_dataset
+            .load_indices_by_name(&index.name)
+            .await
+            .unwrap();
+        let index_meta = &indices[0];
+        let idx = lance_dataset
+            .open_scalar_index("category", &index_meta.uuid.to_string())
+            .await
+            .unwrap();
+        assert_eq!(idx.index_type(), IndexType::Bitmap);
+    }
+
+    #[tokio::test]
+    async fn test_create_label_list_index() {
+        let tmp_dir = tempdir().unwrap();
+        let uri = tmp_dir.path().to_str().unwrap();
+
+        let conn = ConnectBuilder::new(uri).execute().await.unwrap();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new(
+                "tags",
+                DataType::List(Field::new("item", DataType::Utf8, true).into()),
+                true,
+            ),
+        ]));
+
+        const TAGS: [&str; 3] = ["cat", "dog", "fish"];
+
+        let values_builder = StringBuilder::new();
+        let mut builder = ListBuilder::new(values_builder);
+        for i in 0..120 {
+            builder.values().append_value(TAGS[i % 3].to_string());
+            if i % 3 == 0 {
+                builder.append(true)
+            }
+        }
+        let tags = Arc::new(builder.finish());
+
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![Arc::new(Int32Array::from_iter_values(0..40)), tags],
+        )
+        .unwrap();
+
+        let table = conn
+            .create_table(
+                "test_bitmap",
+                RecordBatchIterator::new(vec![Ok(batch.clone())], batch.schema()),
+            )
+            .execute()
+            .await
+            .unwrap();
+
+        // Can not create btree or bitmap index on list column
+        assert!(table
+            .create_index(&["tags"], Index::BTree(Default::default()))
+            .execute()
+            .await
+            .is_err());
+        assert!(table
+            .create_index(&["tags"], Index::Bitmap(Default::default()))
+            .execute()
+            .await
+            .is_err());
+
+        // Create bitmap index on the "category" column
+        table
+            .create_index(&["tags"], Index::LabelList(Default::default()))
+            .execute()
+            .await
+            .unwrap();
+
+        // Verify the index was created
+        let index_configs = table.list_indices().await.unwrap();
+        assert_eq!(index_configs.len(), 1);
+        let index = index_configs.into_iter().next().unwrap();
+        // TODO: Fix via https://github.com/lancedb/lance/issues/2039
+        // assert_eq!(index.index_type, crate::index::IndexType::LabelList);
+        assert_eq!(index.columns, vec!["tags".to_string()]);
+
+        // For now, just open the index to verify its type
+        let lance_dataset = table.as_native().unwrap().dataset.get().await.unwrap();
+        let indices = lance_dataset
+            .load_indices_by_name(&index.name)
+            .await
+            .unwrap();
+        let index_meta = &indices[0];
+        let idx = lance_dataset
+            .open_scalar_index("tags", &index_meta.uuid.to_string())
+            .await
+            .unwrap();
+        assert_eq!(idx.index_type(), IndexType::LabelList);
+    }
+
    #[tokio::test]
    async fn test_read_consistency_interval() {
        let intervals = vec![
Author	SHA1	Message	Date
Lance Release	89bcc1b2e7	Bump version: 0.13.0-beta.0 → 0.13.0-beta.1	2024-08-23 13:56:30 +00:00
rahuljo	6ad5553eca	docs: add dlt-lancedb integration page (#1551 ) Co-authored-by: Akela Drissner-Schmid <32450038+akelad@users.noreply.github.com>	2024-08-22 15:18:49 +05:30
Gagan Bhullar	6eb7ccfdee	fix: rerank attribute unknown (#1554 ) PR fixes #1550	2024-08-22 11:46:36 +05:30
Rithik Kumar	758c82858f	docs: add AI agent example (#1553 ) before: ![Screenshot 2024-08-21 225014](https://github.com/user-attachments/assets/e5b05586-87c5-4739-a4df-2d6cd0704ba5) After: ![Screenshot 2024-08-21 225029](https://github.com/user-attachments/assets/504959db-f560-49b2-9492-557e9846a793) --------- Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com>	2024-08-22 00:54:05 +05:30
Rithik Kumar	0cbc9cd551	docs: add evaluation example (#1552 ) before: ![Screenshot 2024-08-21 194228](https://github.com/user-attachments/assets/68d96658-7579-4934-85af-e8c898b64660) After: ![Screenshot 2024-08-21 195258](https://github.com/user-attachments/assets/81ddb9cd-cb93-47fc-a121-ff82701fd11f) --------- Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com>	2024-08-21 20:37:04 +05:30
Ayush Chaurasia	7d65dd97cf	chore(python): update Colbert architecture and minor improvements (#1547 ) - Update ColBertReranker architecture: The current implementation doesn't use the right arch. This PR uses the implementation in Rerankers library. Fixes https://github.com/lancedb/lancedb/issues/1546 Benchmark diff (hit rate): Hybrid - 91 vs 87 reranked vector - 85 vs 80 - Reranking in FTS is basically disabled in main after last week's FTS updates. I think there's no blocker in supporting that? - Allow overriding accelerators: Most transformer based Rerankers and Embedding automatically select device. This PR allows overriding those settings by passing `device`. Fixes: https://github.com/lancedb/lancedb/issues/1487 --------- Co-authored-by: BubbleCal <bubble-cal@outlook.com>	2024-08-21 12:26:52 +05:30
Ayush Chaurasia	85bb7e54e4	docs: missing griffe dependency for mkdocs deployment (#1545 )	2024-08-19 07:48:23 +05:30
Rithik Kumar	21014cab45	docs: add chatbot example and improve quality of other examples (#1544 )	2024-08-17 12:35:33 +05:30
Lei Xu	5857cb4c6e	docs: add a section to describe scalar index (#1495 )	2024-08-16 18:48:29 -07:00
Rithik Kumar	09ce6c5bb5	docs: add vector search example (#1543 )	2024-08-16 21:30:45 +05:30
BubbleCal	0fa50775d6	feat: support to query/index FTS on RemoteTable/AsyncTable (#1537 ) Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-16 12:01:05 +08:00
Gagan Bhullar	20faa4424b	feat(python): add delete unverified parameter (#1542 ) PR fixes #1527	2024-08-15 09:01:32 -07:00
BubbleCal	b624fc59eb	docs: add `create_fts_index` doc in Python API Reference (#1533 ) resolve #1313 --------- Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-15 11:35:16 +08:00
Gagan Bhullar	d2caa5e202	feat(nodejs): add delete unverified (#1530 ) PR fixes part of #1527	2024-08-14 08:53:53 -07:00
BubbleCal	501817cfac	chore: bump the required python version to 3.9 (#1541 ) Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-14 08:44:31 -07:00
Ryan Green	b3daa25f46	feat: allow new scalar index types to be created in remote table (#1538 )	2024-08-13 16:05:42 -02:30
Matt Basta	6008a8257b	fix: remove native.d.ts from .npmignore (#1531 ) This removes the type definitions for a number of important TypeScript interfaces from `.npmignore` so that the package is not incorrectly typed `any` in a number of places. --- Presently the `opts` argument to `lancedb.connect` is typed `any`, even though it shouldn't be. <img width="560" alt="image" src="https://github.com/user-attachments/assets/5c974ce8-5a59-44a1-935d-cbb808f0ea24"> Clicking into the type definitions for the published package, it has the correct type signature: <img width="831" alt="image" src="https://github.com/user-attachments/assets/6e39a519-13ff-4ca8-95ae-85538ac59d5d"> However, `ConnectionOptions` is imported from `native.js` (along with a number of other imports a bit further down): <img width="384" alt="image" src="https://github.com/user-attachments/assets/10c1b055-ae78-4088-922e-2816af64c23c"> This is not otherwise an issue, except that the type definitions for `native.js` are not included in the published package: <img width="217" alt="image" src="https://github.com/user-attachments/assets/f15cd3b6-a8de-4011-9fa2-391858da20ec"> I haven't compiled the Rust code and run the build script, but I strongly suspect that disincluding the type definitions in `.npmignore` is ultimately the root cause here.	2024-08-13 10:06:15 -07:00
Lance Release	aaff43d304	Updating package-lock.json	2024-08-12 19:48:18 +00:00
Lance Release	d4c3a8ca87	Bump version: 0.9.0 → 0.10.0-beta.0	2024-08-12 19:48:02 +00:00
Lance Release	ff5bbfdd4c	Bump version: 0.12.0 → 0.13.0-beta.0	2024-08-12 19:47:57 +00:00
Lei Xu	694ca30c7c	feat(nodejs): add bitmap and label list index types in nodejs (#1532 )	2024-08-11 12:06:02 -07:00
Lei Xu	b2317c904d	feat: create bitmap and label list scalar index using python async api (#1529 ) * Expose `bitmap` and `LabelList` scalar index type via Rust and Async Python API * Add documents	2024-08-11 09:16:11 -07:00
BubbleCal	613f3063b9	chore: upgrade lance to 0.16.1 (#1524 ) Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-09 19:18:05 +08:00
BubbleCal	5d2cd7fb2e	chore: upgrade object_store to 0.10.2 (#1523 ) To use the same version with lance Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-09 12:03:46 +08:00
Ayush Chaurasia	a88e9bb134	docs: add lancedb embedding fcn on cloud docs (#1521 )	2024-08-09 07:21:04 +05:30
Gagan Bhullar	9c1adff426	feat(python): add to_list to async api (#1520 ) PR fixes #1517	2024-08-08 11:45:20 -07:00
BubbleCal	f9d5fa88a1	feat!: migrate FTS from tantivy to lance-index (#1483 ) Lance now supports FTS, so add it into lancedb Python, TypeScript and Rust SDKs. For Python, we still use tantivy based FTS by default because the lance FTS index now misses some features of tantivy. For Python: - Support to create lance based FTS index - Support to specify columns for full text search (only available for lance based FTS index) For TypeScript: - Change the search method so that it can accept both string and vector - Support full text search For Rust - Support full text search The others: - Update the FTS doc BREAKING CHANGE: - for Python, this renames the attached score column of FTS from "score" to "_score", this could be a breaking change for users that rely the scores --------- Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-08-08 15:33:15 +08:00
Lance Release	4db554eea5	Updating package-lock.json	2024-08-07 20:56:12 +00:00
Lance Release	101066788d	Bump version: 0.9.0-beta.0 → 0.9.0	2024-08-07 20:55:53 +00:00
Lance Release	c4135d9d30	Bump version: 0.8.0 → 0.9.0-beta.0	2024-08-07 20:55:52 +00:00