Builds on #4359 (which added the Arrow 19-21 test aliases) and makes the
package genuinely work across the whole peer range:
* Resolve the library's own `apache-arrow` to 21.2.0 (devDependency) so
the main CI jobs build, type-check and test on the newest release, and
widen the peer range to `>=15.0.0 <22.0.0` instead of an exact pin.
* Add a CI job that re-installs with `apache-arrow` resolved to 19.0.1 and
20.0.0 and runs the suite, covering what a consumer on those versions
gets.
* Arrow 21 overrides `Symbol.hasInstance` on Schema/Field/Table/RecordBatch/
Data so `instanceof` matches objects from another copy of the library,
but not on DataType. The sanitizer's short-circuits therefore returned
foreign schemas unchanged and Arrow's builders then failed with
"Unrecognized type 'NONE'". Check the real prototype chain instead.
* Sanitize the types added in Arrow 20/21 (LargeList, Utf8View, BinaryView,
IntervalMonthDayNano), looked up dynamically so the library still compiles
and runs on older Arrow releases, with a clear error when the installed
Arrow lacks the type. Carry `variadicBuffers` through `Data`.
* Loosen `FieldLike` so a branded DataType from another Arrow copy still
type-checks; export the new `DataTypeLike`.
* Share the Arrow version list between test files, exercise embedding
function types per version in the registry tests, and add sanitize and
end-to-end coverage for the new column types.
* Enable `skipLibCheck`: flatbuffers 25 (via Arrow 21) ships declarations
that need TypeScript 5.7+.
* Record the new dev dependencies in the third-party license inventory.
Co-Authored-By: Vincent Koc <vincentkoc@ieee.org>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
- reconstruct foreign Arrow Map schemas from their single sanitized
entries field
- reject malformed Map types with anything other than one child
- preserve the complete Map schema and `keysSorted` value through
empty-table creation and IPC round trips across Arrow 15–18
## Testing
- `./node_modules/.bin/jest --runInBand __test__/arrow.test.ts
__test__/sanitize.test.ts`
- `pnpm lint`
- `pnpm build`
- `pnpm run docs`
Fixes#2337
The [`FieldLike` type in
arrow.ts](https://github.com/lancedb/lancedb/blob/5ec12c99713e28a8fa550547cc9e7536da063f6e/nodejs/lancedb/arrow.ts#L71-L78)
can have a `type: string` property, but before this change, actually
trying to create a table that has a schema that specifies field types by
name results in an error:
```
Error: Expected a Type but object was null/undefined
```
This change adds support for mapping some type name strings to arrow
`DataType`s, so that passing `FieldLike`s with a `type: string` property
to `sanitizeField` does not throw an error.
The type names that can be passed are upper/lowercase variations of the
keys of the `constructorsByTypeName` object. This does not support
mapping types that need parameters, such as timestamps which need
timezones.
With this, it is possible to create empty tables from `SchemaLike`
objects without instantiating arrow types, e.g.:
```
import { SchemaLike } from "../lancedb/arrow"
// ...
const schemaLike = {
fields: [
{
name: "id",
type: "int64",
nullable: true,
},
{
name: "vector",
type: "float64",
nullable: true,
},
],
// ...
} satisfies SchemaLike;
const table = await con.createEmptyTable("test", schemaLike);
```
This change also makes `FieldLike.nullable` required since the `sanitizeField` function throws if it is undefined.
kid
Tom LaMarre