Thanks @vincentkoc for starting this! I was just looking into this
myself. I started with your PR and added:
- make arrow 21 the default
- widen the peer range to <22 so if they release 21.3 that'll be
included
- have to do skipLibCheck in tsconfig.json, because of flatbuffers 25
which needs TypeScript 5.7
- fix override in Symbol.hasInstance
- support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView,
IntervalMonthDayNano)
- arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20
(adds 2 sec)
- put the list of arrow types in one place
- update NODEJS_THIRD_PARTY_LICENSES.md
- regenerated typescript docs (sorry about reorderings, one time cost)
- reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome.
git diff -w shows 110 lines edited)
- fixed Sliced Utf8View / BinaryView serialization error, by just
converting them to Utf8 and Binary (as our rust code would anyway)
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
- reconstruct foreign Arrow Map schemas from their single sanitized
entries field
- reject malformed Map types with anything other than one child
- preserve the complete Map schema and `keysSorted` value through
empty-table creation and IPC round trips across Arrow 15–18
## Testing
- `./node_modules/.bin/jest --runInBand __test__/arrow.test.ts
__test__/sanitize.test.ts`
- `pnpm lint`
- `pnpm build`
- `pnpm run docs`
Fixes#2337
The [`FieldLike` type in
arrow.ts](https://github.com/lancedb/lancedb/blob/5ec12c99713e28a8fa550547cc9e7536da063f6e/nodejs/lancedb/arrow.ts#L71-L78)
can have a `type: string` property, but before this change, actually
trying to create a table that has a schema that specifies field types by
name results in an error:
```
Error: Expected a Type but object was null/undefined
```
This change adds support for mapping some type name strings to arrow
`DataType`s, so that passing `FieldLike`s with a `type: string` property
to `sanitizeField` does not throw an error.
The type names that can be passed are upper/lowercase variations of the
keys of the `constructorsByTypeName` object. This does not support
mapping types that need parameters, such as timestamps which need
timezones.
With this, it is possible to create empty tables from `SchemaLike`
objects without instantiating arrow types, e.g.:
```
import { SchemaLike } from "../lancedb/arrow"
// ...
const schemaLike = {
fields: [
{
name: "id",
type: "int64",
nullable: true,
},
{
name: "vector",
type: "float64",
nullable: true,
},
],
// ...
} satisfies SchemaLike;
const table = await con.createEmptyTable("test", schemaLike);
```
This change also makes `FieldLike.nullable` required since the `sanitizeField` function throws if it is undefined.
kid
Tom LaMarre