feat: accept multiple on columns for merge insert on remote tables (#4102)

Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.

The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.

No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.

Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).

## Example

```python
table.merge_insert(["shard_key", "id"]) \
    .when_matched_update_all() \
    .when_not_matched_insert_all() \
    .execute(new_data)
```

A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.

## Not included

Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.

Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-09-01 16:32:04 -07:00
committed by GitHub
parent 9a1ffb9e02
commit d2ca0ce0ab
8 changed files with 196 additions and 12 deletions
+34 -1
View File
@@ -737,11 +737,12 @@ it("should query documents with LangChain PDF metadata", async () => {
describe("merge insert", () => {
let tmpDir: tmp.DirResult;
let conn: Connection;
let table: Table;
beforeEach(async () => {
tmpDir = tmp.dirSync({ unsafeCleanup: true });
const conn = await connect(tmpDir.name);
conn = await connect(tmpDir.name);
table = await conn.createTable("some_table", [
{ a: 1, b: "a" },
@@ -779,6 +780,38 @@ describe("merge insert", () => {
expect(result.map((row) => ({ ...row }))).toEqual(expected);
});
test("upsert on a composite key", async () => {
const composite = await conn.createTable("composite", [
{ shard: "a", id: 1, val: "x" },
{ shard: "a", id: 2, val: "y" },
{ shard: "b", id: 1, val: "z" },
]);
// ("a", 1) matches an existing row and updates it. ("b", 2) agrees with an
// existing row on each key column separately but on neither pair, so it is
// an insert.
const mergeInsertRes = await composite
.mergeInsert(["shard", "id"])
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute([
{ shard: "a", id: 1, val: "X" },
{ shard: "b", id: 2, val: "W" },
]);
expect(mergeInsertRes.numUpdatedRows).toBe(1);
expect(mergeInsertRes.numInsertedRows).toBe(1);
const result = (await composite.toArrow())
.toArray()
.sort((a, b) => a.shard.localeCompare(b.shard) || a.id - b.id);
expect(result.map((row) => ({ ...row }))).toEqual([
{ shard: "a", id: 1, val: "X" },
{ shard: "a", id: 2, val: "y" },
{ shard: "b", id: 1, val: "z" },
{ shard: "b", id: 2, val: "W" },
]);
});
test("conditional update", async () => {
const newData = [
{ a: 2, b: "x" },