Builds on #4359 (which added the Arrow 19-21 test aliases) and makes the
package genuinely work across the whole peer range:
* Resolve the library's own `apache-arrow` to 21.2.0 (devDependency) so
the main CI jobs build, type-check and test on the newest release, and
widen the peer range to `>=15.0.0 <22.0.0` instead of an exact pin.
* Add a CI job that re-installs with `apache-arrow` resolved to 19.0.1 and
20.0.0 and runs the suite, covering what a consumer on those versions
gets.
* Arrow 21 overrides `Symbol.hasInstance` on Schema/Field/Table/RecordBatch/
Data so `instanceof` matches objects from another copy of the library,
but not on DataType. The sanitizer's short-circuits therefore returned
foreign schemas unchanged and Arrow's builders then failed with
"Unrecognized type 'NONE'". Check the real prototype chain instead.
* Sanitize the types added in Arrow 20/21 (LargeList, Utf8View, BinaryView,
IntervalMonthDayNano), looked up dynamically so the library still compiles
and runs on older Arrow releases, with a clear error when the installed
Arrow lacks the type. Carry `variadicBuffers` through `Data`.
* Loosen `FieldLike` so a branded DataType from another Arrow copy still
type-checks; export the new `DataTypeLike`.
* Share the Arrow version list between test files, exercise embedding
function types per version in the registry tests, and add sanitize and
end-to-end coverage for the new column types.
* Enable `skipLibCheck`: flatbuffers 25 (via Arrow 21) ships declarations
that need TypeScript 5.7+.
* Record the new dev dependencies in the third-party license inventory.
Co-Authored-By: Vincent Koc <vincentkoc@ieee.org>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two bugs in Node's reading of the embedding_functions schema metadata.
First, parseFunctions keyed its result map by function name, so a table
whose metadata configures the same function for two vector columns came
back with only the last one. It now keys by the vector column, the
convention Python's parser already uses.
Second, Node could not read metadata written by the Python bindings at
all, which spell the keys snake_case: configs parsed with both columns
undefined, breaking embedding application on add() and leaving only
query-side embedding working. The parse now accepts both spellings.
Both fixes land in one shared parser used by every reader --
parseFunctions and the makeArrowTable schema validator, which had its
own private camelCase-only parse -- so the wire contract cannot fork
between entry points. A config naming no source or vector column is an
error at the boundary rather than a default downstream, as are two
configs claiming one column. The "vector" fallback remains only on the
optional field of user-supplied configs.
Breaking: parseFunctions is exported and its map keys change from
function name to vector column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
BREAKING CHANGE: embedding function implementations in Node need to now
call `resolveVariables()` in their constructors and should **not**
implement `toJSON()`.
This tries to address the handling of secrets. In Node, they are
currently lost. In Python, they are currently leaked into the table
schema metadata.
This PR introduces an in-memory variable store on the function registry.
It also allows embedding function definitions to label certain config
values as "sensitive", and the preprocessing logic will raise an error
if users try to pass in hard-coded values.
Closes#2110Closes#521
---------
Co-authored-by: Weston Pace <weston.pace@gmail.com>
previously if you tried to install both vectordb and @lancedb/lancedb,
you would get a peer dependency issue due to `vectordb` requiring
`14.0.2` and `@lancedb/lancedb` requiring `15.0.0`. now
`@lancedb/lancedb` should just work with any arrow version 13-17
previously the `registry` would return `undefined | EmbeddingFunction`
even for built in functions such as "openai"
now it'll return the correct type for `getRegistry().get("openai")
as well as pass in the correct options type to `create`
### before
```ts
const options: {model: 'not-a-real-model'}
// this'd compile just fine, but result in runtime error
const openai: EmbeddingFunction | undefined = getRegistry().get("openai").create(options)
// this'd also compile fine
const openai: EmbeddingFunction | undefined = getRegistry().get("openai").create({MODEL: ''})
```
### after
```ts
const options: {model: 'not-a-real-model'}
const openai: OpenAIEmbeddingFunction = getRegistry().get("openai").create(options)
// Type '"not-a-real-model"' is not assignable to type '"text-embedding-ada-002" | "text-embedding-3-large" | "text-embedding-3-small" | undefined'
```
while adding some more docs & examples for the new js sdk, i ran across
a few compatibility issues when using different arrow versions. This
should fix those issues.
Wyatt AltandClaude Opus 5
Will JonesandWeston Pace
Cory Grinstead