Todo:
- [x] add proper documentation
- [x] add unit tests
- [x] better handling of the registry**1
- [x] allow user defined registry**2
**1 The python implementation just uses a global registry so it makes
things a bit easier. I attached it to the db/connection to prevent
future conflicts if running multiple connections/databases. I mostly
modeled the registry & pattern off of datafusion's
[FunctionRegistry](https://docs.rs/datafusion/latest/datafusion/execution/trait.FunctionRegistry.html).
**2 Ideally, the user should be able to provide it's own registry
entirely, but currently it just uses an in memory registry by default
(_which isn't configurable_)
`rust/lancedb/examples/embedding_registry.rs` provides a thorough
example of expected usage.
---
Some additional notes:
This does not provide any of the out of box functionality that the
python registry does.
_i.e there are no built-in embedding functions._
You can think of this as the ground work for adding those built in
functions, So while this is part of
https://github.com/lancedb/lancedb/issues/994, it does not yet offer
feature parity.
https://github.com/lancedb/lancedb/issues/1266#event-12703166915
This happens because openai API errors out with None values. The current
log level didn't really print out the msg on screen. Changed the log
level to warning, which better suits this case.
Also, retry loop can be disabled by setting `max_retries=0` (I'm not
sure if we should also set this as the default behaviour as hitting api
rate is quite common when ingesting large corpus)
```
func = get_registry().get("openai").create(max_retries=0)
````
It's confusing to users that keyword arguments from the async API like
`storage_options` are accepted by `connect()`, but don't do anything. We
should error if unknown arguments are passed instead.
This was already configurable in the rust API but it wasn't actually
being passed down to the underlying dataset. I added this option to both
the async python API and the new nodejs API.
I also added this option to the synchronous python API.
I did not add the option to vectordb.
Fixes issue where we would throw `Either data or schema needs to
defined` when passing `data` to `createTable` as a property of the first
argument (an object).
```ts
await db.createTable({
name: 'table1',
data,
schema
})
```
The rust implementation of the remote client is not yet ready. This is
understandably confusing for users since it is enabled by default. This
PR disables it by default. We can re-enable it when we are ready (even
then it is not clear this is something that should be a default
feature).
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
1. added rename_table fn to enable dashboard to rename a table
2. added index_type and distance_type (for vector index) to index_stats
so that more detailed data can be shown on the table page.
Lance Release
Bert
QianZhu
Aman Kishore
Ryan Green
Will Jones
Cory Grinstead
Ben Poulson
Ayush Chaurasia
Nehil Jain
Ikko Eltociear Ashimine
Rohit Rastogi
asmith26
Weston Pace
Lei Xu
Alex Kohler
Rob Meng
Prashanth Rao