mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-02 03:28:41 +00:00
feat: accept multiple on columns for merge insert on remote tables (#4102)
Merge insert has always taken a list of columns to match on, and local tables have always joined on all of them. Remote tables did not: any list longer than one was rejected with `MergeInsertBuilder only supports a single 'on' column`, so a composite-key upsert was impossible against LanceDB Cloud and Enterprise from Rust, Python or TypeScript. The remote request now carries `on` as a list and sends it as one repeated query parameter per column — `?on=shard_key&on=id`. That is how the lance-namespace spec encodes an array-valued `on`, so the server receives a composite key in the shape it expects. A single column still serializes to `?on=id`, exactly what clients sent before, so existing callers are unaffected. A column repeated within `on` is now rejected client-side rather than sent for the server to reject with a 400. No binding changes were needed: `Table.merge_insert` in Python and `Table.mergeInsert` in TypeScript already accepted a list, it just could not reach a remote table. Both gain a test for composite keys, and the doc comments now say what passing several columns means. Part of [ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the). ## Example ```python table.merge_insert(["shard_key", "id"]) \ .when_matched_update_all() \ .when_not_matched_insert_all() \ .execute(new_data) ``` A row whose `id` matches an existing row but whose `shard_key` differs is an insert, not an update. ## Not included Java. Java callers reach merge insert through `org.lance.namespace.LanceNamespace`, whose `MergeInsertIntoTableRequest.on` is a single string until lance-namespace 0.12 ([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363), [lance#8915](https://github.com/lance-format/lance/pull/8915)). There is nothing in this repo's Java SDK to change until the `lance-core` pin can move. Sending more than one column requires a server that accepts the repeated parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571)); an older server returns a 400 rather than silently merging on one column. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1547,7 +1547,9 @@ class Table(ABC):
|
||||
on: Union[str, Iterable[str]]
|
||||
A column (or columns) to join on. This is how records from the
|
||||
source table and target table are matched. Typically this is some
|
||||
kind of key or id column.
|
||||
kind of key or id column. Passing several columns matches on the
|
||||
composite key: a source row updates a target row only when it
|
||||
agrees on every one of them.
|
||||
|
||||
Examples
|
||||
--------
|
||||
@@ -5701,7 +5703,9 @@ class AsyncTable:
|
||||
on: Union[str, Iterable[str]]
|
||||
A column (or columns) to join on. This is how records from the
|
||||
source table and target table are matched. Typically this is some
|
||||
kind of key or id column.
|
||||
kind of key or id column. Passing several columns matches on the
|
||||
composite key: a source row updates a target row only when it
|
||||
agrees on every one of them.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
@@ -2682,6 +2682,43 @@ def test_merge_insert(mem_db: DBConnection):
|
||||
)
|
||||
|
||||
|
||||
def test_merge_insert_composite_key(mem_db: DBConnection):
|
||||
table = mem_db.create_table(
|
||||
"my_table",
|
||||
data=pa.table(
|
||||
{
|
||||
"shard": ["a", "a", "b"],
|
||||
"id": [1, 2, 1],
|
||||
"val": ["x", "y", "z"],
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
# ("a", 1) matches an existing row and updates it. ("b", 2) agrees with an
|
||||
# existing row on each key column separately but on neither pair, so it is
|
||||
# an insert.
|
||||
new_data = pa.table({"shard": ["a", "b"], "id": [1, 2], "val": ["X", "W"]})
|
||||
res = (
|
||||
table.merge_insert(["shard", "id"])
|
||||
.when_matched_update_all()
|
||||
.when_not_matched_insert_all()
|
||||
.execute(new_data)
|
||||
)
|
||||
assert res.num_updated_rows == 1
|
||||
assert res.num_inserted_rows == 1
|
||||
|
||||
expected = pa.table(
|
||||
{
|
||||
"shard": ["a", "a", "b", "b"],
|
||||
"id": [1, 2, 1, 2],
|
||||
"val": ["X", "y", "z", "W"],
|
||||
}
|
||||
)
|
||||
assert table.to_arrow().sort_by([("shard", "ascending"), ("id", "ascending")]) == (
|
||||
expected
|
||||
)
|
||||
|
||||
|
||||
def test_merge_insert_nullable_pandas_into_pydantic_schema(mem_db: DBConnection):
|
||||
# Regression test for https://github.com/lancedb/lancedb/issues/2366
|
||||
pd = pytest.importorskip("pandas")
|
||||
|
||||
Reference in New Issue
Block a user