rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-01-09 05:12:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(python): async-sync feature parity on Table (#1914)

Browse Source 
### Changes to sync API
* Updated `LanceTable` and `LanceDBConnection` reprs
* Add `storage_options`, `data_storage_version`, and
`enable_v2_manifest_paths` to sync create table API.
* Add `storage_options` to `open_table` in sync API.
* Add `list_indices()` and `index_stats()` to sync API
* `create_table()` will now create only 1 version when data is passed.
Previously it would always create two versions: 1 to create an empty
table and 1 to add data to it.

### Changes to async API
* Add `embedding_functions` to async `create_table()` API.
* Added `head()` to async API

### Refactors
* Refactor index parameters into dataclasses so they are easier to use
from Python
* Moved most tests to use an in-memory DB so we don't need to create so
many temp directories

Closes #1792
Closes #1932

---------

Co-authored-by: Weston Pace <weston.pace@gmail.com>
This commit is contained in:
Will Jones
2024-12-13 12:56:44 -08:00
committed by GitHub
parent d83e5a0208
commit 980aa70e2d
23 changed files with 1296 additions and 1324 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/src/basic.md
View File

@@ -141,14 +141,6 @@ recommend switching to stable releases.
--8<-- "python/python/tests/docs/test_basic.py:connect_async"
```
!!! note "Asynchronous Python API"
The asynchronous Python API is new and has some slight differences compared
to the synchronous API. Feel free to start using the asynchronous version.
Once all features have migrated we will start to move the synchronous API to
use the same syntax as the asynchronous API. To help with this migration we
have created a [migration guide](migration.md) detailing the differences.
=== "Typescript[^1]"
=== "@lancedb/lancedb"

79
docs/src/migration.md
View File

@@ -1,81 +1,14 @@
# Rust-backed Client Migration Guide
In an effort to ensure all clients have the same set of capabilities we have begun migrating the
python and node clients onto a common Rust base library. In python, this new client is part of
the same lancedb package, exposed as an asynchronous client. Once the asynchronous client has
reached full functionality we will begin migrating the synchronous library to be a thin wrapper
around the asynchronous client.
In an effort to ensure all clients have the same set of capabilities we have
migrated the Python and Node clients onto a common Rust base library. In Python,
both the synchronous and asynchronous clients are based on this implementation.
In Node, the new client is available as `@lancedb/lancedb`, which replaces
the existing `vectordb` package.
This guide describes the differences between the two APIs and will hopefully assist users
This guide describes the differences between the two Node APIs and will hopefully assist users
that would like to migrate to the new API.
## Python
### Closeable Connections
The Connection now has a `close` method. You can call this when
you are done with the connection to eagerly free resources. Currently
this is limited to freeing/closing the HTTP connection for remote
connections. In the future we may add caching or other resources to
native connections so this is probably a good practice even if you
aren't using remote connections.
In addition, the connection can be used as a context manager which may
be a more convenient way to ensure the connection is closed.
```python
import lancedb
async def my_async_fn():
with await lancedb.connect_async("my_uri") as db:
print(await db.table_names())
```
It is not mandatory to call the `close` method. If you do not call it
then the connection will be closed when the object is garbage collected.
### Closeable Table
The Table now also has a `close` method, similar to the connection. This
can be used to eagerly free the cache used by a Table object. Similar to
the connection, it can be used as a context manager and it is not mandatory
to call the `close` method.
#### Changes to Table APIs
- Previously `Table.schema` was a property. Now it is an async method.
- The method `Table.__len__` was removed and `len(table)` will no longer
work. Use `Table.count_rows` instead.
#### Creating Indices
The `Table.create_index` method is now used for creating both vector indices
and scalar indices. It currently requires a column name to be specified (the
column to index). Vector index defaults are now smarter and scale better with
the size of the data.
To specify index configuration details you will need to specify which kind of
index you are using.
#### Querying
The `Table.search` method has been renamed to `AsyncTable.vector_search` for
clarity.
### Features not yet supported
The following features are not yet supported by the asynchronous API. However,
we plan to support them soon.
- You cannot specify an embedding function when creating or opening a table.
You must calculate embeddings yourself if using the asynchronous API
- The merge insert operation is not supported in the asynchronous API
- Cleanup / compact / optimize indices are not supported in the asynchronous API
- add / alter columns is not supported in the asynchronous API
- The asynchronous API does not yet support any full text search or reranking
search
- Remote connections to LanceDb Cloud are not yet supported.
- The method Table.head is not yet supported.
## TypeScript/JavaScript
For JS/TS users, we offer a brand new SDK [@lancedb/lancedb](https://www.npmjs.com/package/@lancedb/lancedb)

2
docs/src/python/python.md
View File

@@ -47,6 +47,8 @@ is also an [asynchronous API client](#connections-asynchronous).
::: lancedb.embeddings.registry.EmbeddingFunctionRegistry
::: lancedb.embeddings.base.EmbeddingFunctionConfig
::: lancedb.embeddings.base.EmbeddingFunction
::: lancedb.embeddings.base.TextEmbeddingFunction