feat: accept multiple on columns for merge insert on remote tables (#4102)

Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.

The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.

No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.

Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).

## Example

```python
table.merge_insert(["shard_key", "id"]) \
    .when_matched_update_all() \
    .when_not_matched_insert_all() \
    .execute(new_data)
```

A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.

## Not included

Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.

Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-09-01 16:32:04 -07:00
committed by GitHub
parent 9a1ffb9e02
commit d2ca0ce0ab
8 changed files with 196 additions and 12 deletions
+6 -2
View File
@@ -1547,7 +1547,9 @@ class Table(ABC):
on: Union[str, Iterable[str]]
A column (or columns) to join on. This is how records from the
source table and target table are matched. Typically this is some
kind of key or id column.
kind of key or id column. Passing several columns matches on the
composite key: a source row updates a target row only when it
agrees on every one of them.
Examples
--------
@@ -5701,7 +5703,9 @@ class AsyncTable:
on: Union[str, Iterable[str]]
A column (or columns) to join on. This is how records from the
source table and target table are matched. Typically this is some
kind of key or id column.
kind of key or id column. Passing several columns matches on the
composite key: a source row updates a target row only when it
agrees on every one of them.
Examples
--------
+37
View File
@@ -2682,6 +2682,43 @@ def test_merge_insert(mem_db: DBConnection):
)
def test_merge_insert_composite_key(mem_db: DBConnection):
table = mem_db.create_table(
"my_table",
data=pa.table(
{
"shard": ["a", "a", "b"],
"id": [1, 2, 1],
"val": ["x", "y", "z"],
}
),
)
# ("a", 1) matches an existing row and updates it. ("b", 2) agrees with an
# existing row on each key column separately but on neither pair, so it is
# an insert.
new_data = pa.table({"shard": ["a", "b"], "id": [1, 2], "val": ["X", "W"]})
res = (
table.merge_insert(["shard", "id"])
.when_matched_update_all()
.when_not_matched_insert_all()
.execute(new_data)
)
assert res.num_updated_rows == 1
assert res.num_inserted_rows == 1
expected = pa.table(
{
"shard": ["a", "a", "b", "b"],
"id": [1, 2, 1, 2],
"val": ["X", "y", "z", "W"],
}
)
assert table.to_arrow().sort_by([("shard", "ascending"), ("id", "ascending")]) == (
expected
)
def test_merge_insert_nullable_pandas_into_pydantic_schema(mem_db: DBConnection):
# Regression test for https://github.com/lancedb/lancedb/issues/2366
pd = pytest.importorskip("pandas")