mirror of
https://github.com/lancedb/lancedb.git
synced 2026-01-07 12:22:59 +00:00
docs: Updates and refactor (#683)
This PR makes incremental changes to the documentation. * Closes #697 * Closes #698 - [x] Add dark mode - [x] Fix headers in navbar - [x] Add `extra.css` to customize navbar styles - [x] Customize fonts for prose/code blocks, navbar and admonitions - [x] Inspect all admonition boxes (remove redundant dropdowns) and improve clarity and readability - [x] Ensure that all images in the docs have white background (not transparent) to be viewable in dark mode - [x] Improve code formatting in code blocks to make them consistent with autoformatters (eslint/ruff) - [x] Add bolder weight to h1 headers - [x] Add diagram showing the difference between embedded (OSS) and serverless (Cloud) - [x] Fix [Creating an empty table](https://lancedb.github.io/lancedb/guides/tables/#creating-empty-table) section: right now, the subheaders are not clickable. - [x] In critical data ingestion methods like `table.add` (among others), the type signature often does not match the actual code - [x] Proof-read each documentation section and rewrite as necessary to provide more context, use cases, and explanations so it reads less like reference documentation. This is especially important for CRUD and search sections since those are so central to the user experience. - [x] The section for [Adding data](https://lancedb.github.io/lancedb/guides/tables/#adding-to-a-table) only shows examples for pandas and iterables. We should include pydantic models, arrow tables, etc. - [x] Add conceptual tutorial for IVF-PQ index - [x] Clearly separate vector search, FTS and filtering sections so that these are easier to find - [x] Add docs on refine factor to explain its importance for recall. Closes #716 - [x] Add an FAQ page showing answers to commonly asked questions about LanceDB. Closes #746 - [x] Add simple polars example to the integrations section. Closes #756 and closes #153 - [ ] Add basic docs for the Rust API (more detailed API docs can come later). Closes #781 - [x] Add a section on the various storage options on local vs. cloud (S3, EBS, EFS, local disk, etc.) and the tradeoffs involved. Closes #782 - [x] Revamp filtering docs: add pre-filtering examples and redo headers and update content for SQL filters. Closes #783 and closes #784. - [x] Add docs for data management: compaction, cleaning up old versions and incremental indexing. Closes #785 - [ ] Add a benchmark section that also discusses some best practices. Closes #787 --------- Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com> Co-authored-by: Will Jones <willjones127@gmail.com>
This commit is contained in:
Prashanth Rao
committed by
Weston Pace
Weston Pace
parent
33ab68c790
commit
4d5d748acd
@@ -1,10 +1,12 @@
|
||||
# Basic LanceDB Functionality
|
||||
# Quick start
|
||||
|
||||
We'll cover the basics of using LanceDB on your local machine in this section.
|
||||
!!! info "LanceDB can be run in a number of ways:"
|
||||
|
||||
??? info "LanceDB runs embedded on your backend application, so there is no need to run a separate server."
|
||||
* Embedded within an existing backend (like your Django, Flask, Node.js or FastAPI application)
|
||||
* Connected to directly from a client application like a Jupyter notebook for analytical workloads
|
||||
* Deployed as a remote serverless database
|
||||
|
||||
<img src="../assets/lancedb_embedded_explanation.png" width="650px" />
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -46,46 +48,39 @@ We'll cover the basics of using LanceDB on your local machine in this section.
|
||||
## How to create a table
|
||||
|
||||
=== "Python"
|
||||
```python
|
||||
tbl = db.create_table("my_table",
|
||||
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
|
||||
```
|
||||
```python
|
||||
tbl = db.create_table("my_table",
|
||||
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
|
||||
```
|
||||
|
||||
If the table already exists, LanceDB will raise an error by default.
|
||||
If you want to overwrite the table, you can pass in `mode="overwrite"`
|
||||
to the `create_table` method.
|
||||
If the table already exists, LanceDB will raise an error by default.
|
||||
If you want to overwrite the table, you can pass in `mode="overwrite"`
|
||||
to the `create_table` method.
|
||||
|
||||
You can also pass in a pandas DataFrame directly:
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.DataFrame([{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
|
||||
tbl = db.create_table("table_from_df", data=df)
|
||||
```
|
||||
|
||||
!!! warning
|
||||
|
||||
If the table already exists, LanceDB will raise an error by default.
|
||||
If you want to make sure you overwrite the table, pass in `mode="overwrite"`
|
||||
to the `createTable` function.
|
||||
You can also pass in a pandas DataFrame directly:
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.DataFrame([{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
|
||||
tbl = db.create_table("table_from_df", data=df)
|
||||
```
|
||||
|
||||
=== "Javascript"
|
||||
```javascript
|
||||
const tb = await db.createTable(
|
||||
```javascript
|
||||
const tb = await db.createTable(
|
||||
"myTable",
|
||||
[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
|
||||
```
|
||||
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}]
|
||||
)
|
||||
```
|
||||
|
||||
!!! warning
|
||||
If the table already exists, LanceDB will raise an error by default.
|
||||
If you want to overwrite the table, you can pass in `mode="overwrite"`
|
||||
to the `createTable` function.
|
||||
|
||||
If the table already exists, LanceDB will raise an error by default.
|
||||
If you want to overwrite the table, you can pass in `"overwrite"`
|
||||
to the `createTable` function like this: `await con.createTable(tableName, data, { writeMode: WriteMode.Overwrite })`
|
||||
|
||||
|
||||
??? info "Under the hood, LanceDB is converting the input data into an Apache Arrow table and persisting it to disk in [Lance format](https://www.github.com/lancedb/lance)."
|
||||
!!! info "Under the hood, LanceDB is converting the input data into an Apache Arrow table and persisting it to disk in [Lance format](https://www.github.com/lancedb/lance)."
|
||||
|
||||
### Creating an empty table
|
||||
|
||||
@@ -134,7 +129,7 @@ After a table has been created, you can always add more data to it using
|
||||
|
||||
# Option 1: Add a list of dicts to a table
|
||||
data = [{"vector": [1.3, 1.4], "item": "fizz", "price": 100.0},
|
||||
{"vector": [9.5, 56.2], "item": "buzz", "price": 200.0}]
|
||||
{"vector": [9.5, 56.2], "item": "buzz", "price": 200.0}]
|
||||
tbl.add(data)
|
||||
|
||||
# Option 2: Add a pandas DataFrame to a table
|
||||
@@ -145,7 +140,7 @@ After a table has been created, you can always add more data to it using
|
||||
=== "Javascript"
|
||||
```javascript
|
||||
await tbl.add([{vector: [1.3, 1.4], item: "fizz", price: 100.0},
|
||||
{vector: [9.5, 56.2], item: "buzz", price: 200.0}])
|
||||
{vector: [9.5, 56.2], item: "buzz", price: 200.0}])
|
||||
```
|
||||
|
||||
## How to search for (approximate) nearest neighbors
|
||||
@@ -214,20 +209,22 @@ Use the `drop_table()` method on the database to remove a table.
|
||||
This permanently removes the table and is not recoverable, unlike deleting rows.
|
||||
If the table does not exist an exception is raised.
|
||||
|
||||
!!! note "Bundling `vectordb` apps with Webpack"
|
||||
|
||||
If you're using the `vectordb` module in JavaScript, since LanceDB contains a prebuilt Node binary, you must configure `next.config.js` to exclude it from webpack. This is required for both using Next.js and deploying a LanceDB app on Vercel.
|
||||
|
||||
```javascript
|
||||
/** @type {import('next').NextConfig} */
|
||||
module.exports = ({
|
||||
webpack(config) {
|
||||
config.externals.push({ vectordb: 'vectordb' })
|
||||
return config;
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## What's next
|
||||
|
||||
This section covered the very basics of the LanceDB API.
|
||||
LanceDB supports many additional features when creating indices to speed up search and options for search.
|
||||
These are contained in the next section of the documentation.
|
||||
This section covered the very basics of using LanceDB. If you're learning about vector databases for the first time, you may want to read the page on [indexing](concepts/index_ivfpq.md) to get familiar with the concepts.
|
||||
|
||||
## Note: Bundling vectorDB apps with webpack
|
||||
Since LanceDB contains a prebuilt Node binary, you must configure `next.config.js` to exclude it from webpack. This is required for both using Next.js and deploying on Vercel.
|
||||
```javascript
|
||||
/** @type {import('next').NextConfig} */
|
||||
module.exports = ({
|
||||
webpack(config) {
|
||||
config.externals.push({ vectordb: 'vectordb' })
|
||||
return config;
|
||||
}
|
||||
})
|
||||
```
|
||||
If you've already worked with other vector databases, you may want to read the [guides](guides/tables.md) to learn how to work with LanceDB in more detail.
|
||||
|
||||
Reference in New Issue
Block a user