1. Support persistent embedding function so users can just search using
query string
2. Add fixed size list conversion for multiple vector columns
3. Add support for empty query (just apply select/where/limit).
4. Refactor and simplify some of the data prep code
---------
Co-authored-by: Chang She <chang@lancedb.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Combine delete and append to make a temporary update feature that is
only enabled for the local python lancedb.
The reason why this is temporary is because it first has to load the
data that matches the where clause into memory, which is technical
unbounded.
---------
Co-authored-by: Chang She <chang@lancedb.com>
This reverts commit 87e9a0250f.
I triggered the nodejs release commit GHA by mistake. Reverting it.
The tag will be removed manually.
Co-authored-by: Chang She <chang@lancedb.com>
Previously if you needed to add a column to a table you'd have to
rewrite the whole table. Instead,
we use the merge functionality from Lance format
to incrementally add columns from another table
or dataframe.
---------
Co-authored-by: Chang She <chang@lancedb.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Previously the temporary restore feature required copying data. The new
feature in pylance does not.
---------
Co-authored-by: Chang She <chang@lancedb.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
- Supports creating an empty table as long as an Arrow Schema is provided
- Supports creating a table from an Arrow Table (can be passed as data)
- Simplified some Arrow code in the TS/FFI side
- removed createTableArrow method, it was never documented / tested.
Previously when logging the `LocalConnection` and `LocalTable` classes,
we would expose the aws creds inside them. This PR changes the stored
creds to a anonymous function to hide the creds
It improves the UX as iterators can be of any type supported by the
table (plus recordbatch) & there is no separate requirement.
Also expands the test cases for pydantic & arrow schema.
If this is looks good I'll update the docs.
Example usage:
```
class Content(LanceModel):
vector: vector(2)
item: str
price: float
def make_batches():
for _ in range(5):
yield from [
# pandas
pd.DataFrame({
"vector": [[3.1, 4.1], [1, 1]],
"item": ["foo", "bar"],
"price": [10.0, 20.0],
}),
# pylist
[
{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0},
],
# recordbatch
pa.RecordBatch.from_arrays(
[
pa.array([[3.1, 4.1], [5.9, 26.5]], pa.list_(pa.float32(), 2)),
pa.array(["foo", "bar"]),
pa.array([10.0, 20.0]),
],
["vector", "item", "price"],
),
# pydantic list
[
Content(vector=[3.1, 4.1], item="foo", price=10.0),
Content(vector=[5.9, 26.5], item="bar", price=20.0),
]]
db = lancedb.connect("db")
tbl = db.create_table("tabley", make_batches(), schema=Content, mode="overwrite")
tbl.add(make_batches())
```
Same should with arrow schema.
---------
Co-authored-by: Weston Pace <weston.pace@gmail.com>
This adds LanceTable.restore as a temporary feature. It reads data from
a previous version and creates
a new snapshot version using that data. This makes the version writeable
unlike checkout. This should be replaced once the feature is implemented
in pylance.
Co-authored-by: Chang She <chang@lancedb.com>
BREAKING CHANGE: The `score` column has been renamed to `_distance` to
more accurately describe the semantics (smaller means closer / better).
---------
Co-authored-by: Lei Xu <lei@lancedb.com>
#416 Fixed.
added drop_database() method . This deletes all the tables from the
database with a single command.
---------
Signed-off-by: Ashis Kumar Naik <ashishami2002@gmail.com>
- Callers / SDKs are responsible for keeping track of the last version of the Table
- Remove the mpsc from Table and make all Table operations non-blocking
Saves users from having to explicitly call
`LanceModel.to_arrow_schema()` when creating an empty table.
See new docs for full details.
---------
Co-authored-by: Chang She <chang@lancedb.com>
Chang She
Ayush Chaurasia
Will Jones
Lance Release
QianZhu
Tevin Wang
Jai
Rob Meng
gsilvestrin
Ashis Kumar Naik
Lei Xu
Weston Pace