mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-03 20:18:54 +00:00
feat: accept multiple on columns for merge insert on remote tables (#4102)
Merge insert has always taken a list of columns to match on, and local tables have always joined on all of them. Remote tables did not: any list longer than one was rejected with `MergeInsertBuilder only supports a single 'on' column`, so a composite-key upsert was impossible against LanceDB Cloud and Enterprise from Rust, Python or TypeScript. The remote request now carries `on` as a list and sends it as one repeated query parameter per column — `?on=shard_key&on=id`. That is how the lance-namespace spec encodes an array-valued `on`, so the server receives a composite key in the shape it expects. A single column still serializes to `?on=id`, exactly what clients sent before, so existing callers are unaffected. A column repeated within `on` is now rejected client-side rather than sent for the server to reject with a 400. No binding changes were needed: `Table.merge_insert` in Python and `Table.mergeInsert` in TypeScript already accepted a list, it just could not reach a remote table. Both gain a test for composite keys, and the doc comments now say what passing several columns means. Part of [ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the). ## Example ```python table.merge_insert(["shard_key", "id"]) \ .when_matched_update_all() \ .when_not_matched_insert_all() \ .execute(new_data) ``` A row whose `id` matches an existing row but whose `shard_key` differs is an insert, not an update. ## Not included Java. Java callers reach merge insert through `org.lance.namespace.LanceNamespace`, whose `MergeInsertIntoTableRequest.on` is a single string until lance-namespace 0.12 ([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363), [lance#8915](https://github.com/lance-format/lance/pull/8915)). There is nothing in this repo's Java SDK to change until the `lance-core` pin can move. Sending more than one column requires a server that accepts the repeated parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571)); an older server returns a 400 rather than silently merging on one column. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -737,11 +737,12 @@ it("should query documents with LangChain PDF metadata", async () => {
|
||||
|
||||
describe("merge insert", () => {
|
||||
let tmpDir: tmp.DirResult;
|
||||
let conn: Connection;
|
||||
let table: Table;
|
||||
|
||||
beforeEach(async () => {
|
||||
tmpDir = tmp.dirSync({ unsafeCleanup: true });
|
||||
const conn = await connect(tmpDir.name);
|
||||
conn = await connect(tmpDir.name);
|
||||
|
||||
table = await conn.createTable("some_table", [
|
||||
{ a: 1, b: "a" },
|
||||
@@ -779,6 +780,38 @@ describe("merge insert", () => {
|
||||
|
||||
expect(result.map((row) => ({ ...row }))).toEqual(expected);
|
||||
});
|
||||
test("upsert on a composite key", async () => {
|
||||
const composite = await conn.createTable("composite", [
|
||||
{ shard: "a", id: 1, val: "x" },
|
||||
{ shard: "a", id: 2, val: "y" },
|
||||
{ shard: "b", id: 1, val: "z" },
|
||||
]);
|
||||
|
||||
// ("a", 1) matches an existing row and updates it. ("b", 2) agrees with an
|
||||
// existing row on each key column separately but on neither pair, so it is
|
||||
// an insert.
|
||||
const mergeInsertRes = await composite
|
||||
.mergeInsert(["shard", "id"])
|
||||
.whenMatchedUpdateAll()
|
||||
.whenNotMatchedInsertAll()
|
||||
.execute([
|
||||
{ shard: "a", id: 1, val: "X" },
|
||||
{ shard: "b", id: 2, val: "W" },
|
||||
]);
|
||||
expect(mergeInsertRes.numUpdatedRows).toBe(1);
|
||||
expect(mergeInsertRes.numInsertedRows).toBe(1);
|
||||
|
||||
const result = (await composite.toArrow())
|
||||
.toArray()
|
||||
.sort((a, b) => a.shard.localeCompare(b.shard) || a.id - b.id);
|
||||
|
||||
expect(result.map((row) => ({ ...row }))).toEqual([
|
||||
{ shard: "a", id: 1, val: "X" },
|
||||
{ shard: "a", id: 2, val: "y" },
|
||||
{ shard: "b", id: 1, val: "z" },
|
||||
{ shard: "b", id: 2, val: "W" },
|
||||
]);
|
||||
});
|
||||
test("conditional update", async () => {
|
||||
const newData = [
|
||||
{ a: 2, b: "x" },
|
||||
|
||||
Reference in New Issue
Block a user