## Summary
- Add an issue-specific regression for appending generated embeddings to
an empty table with a non-nullable vector field.
- Verify the custom embedding function produces the declared Float64
vectors and both appended rows are readable.
## Root cause
In v0.4.19, records without a vector value were materialized against the
explicit schema before embeddings were inserted. Apache Arrow inferred
the generated batch vector field as nullable while the table retained
the user-provided non-nullable field, then rejected the mismatched
schemas.
The current conversion path excludes the generated field from the
initial record conversion and realigns the completed batch to the stored
schema after embedding, but the reported empty-table append sequence
lacked permanent regression coverage.
## Validation
- `pnpm exec biome format --write __test__/embedding.test.ts`
- `pnpm lint-ci`
- `pnpm test -- --runInBand __test__/embedding.test.ts` (12 passed, 1
skipped integration test)
- `pnpm build`
- `pnpm run docs`
Fixes#1281
<!-- lance-gatekeeper-fix:v1 agent=6b7270aeb92e6b6c6f5b45022fa83f6a
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
- Fixes issue where passing `{ vector: undefined }` with an embedding
function threw "Found field not in schema" error instead of calling the
embedding function like `null` or omitted fields.
**Changes:**
- Modified `rowPathsAndValues` to skip undefined values during schema
inference
- Added test case verifying undefined, null, and omitted vector fields
all work correctly
**Before:** `{ vector: undefined }` → Error
**After:** `{ vector: undefined }` → Calls embedding function
Closes#2647
BREAKING CHANGE: embedding function implementations in Node need to now
call `resolveVariables()` in their constructors and should **not**
implement `toJSON()`.
This tries to address the handling of secrets. In Node, they are
currently lost. In Python, they are currently leaked into the table
schema metadata.
This PR introduces an in-memory variable store on the function registry.
It also allows embedding function definitions to label certain config
values as "sensitive", and the preprocessing logic will raise an error
if users try to pass in hard-coded values.
Closes#2110Closes#521
---------
Co-authored-by: Weston Pace <weston.pace@gmail.com>
* `createTable()` now saves embeddings in the schema metadata.
Previously, it would drop them. (`createEmptyTable()` was already tested
and worked.)
* `mergeInsert()` now uses embeddings.
Fixes#2066
lancedb-gatefixer[bot]
Neha Prasad
Will Jones
Cory Grinstead