rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-01-07 12:22:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
02d31ee412a1bfcba5d2bba36d6f811396db8077
lancedb/docs/src/js/globals.md
Weston Pace 5a19cf15a6 feat: a utility for creating "permutation views" (#2552) 
I'm working on a lancedb version of pytorch data loading (and hopefully
addressing https://github.com/lancedb/lance/issues/3727).

However, rather than rely on pytorch for everything I'm moving some of
the things that pytorch does into rust. This gives us more control over
data loading (e.g. using shards or a hash-based split) and it allows
permutations to be persistent. In particular I hope to be able to:

* Create a persistent permutation
* This permutation can handle splits, filtering, shuffling, and sharding
* Create a rust data loader that can read a permutation (one or more
splits), or a subset of a permutation (for DDP)
* Create a python data loader that delegates to the rust data loader

Eventually create integrations for other data loading libraries,
including rust & node
2025-10-09 18:07:31 -07:00

4.2 KiB
Raw Blame History

@lancedb/lancedb • Docs

@lancedb/lancedb

Namespaces

Enumerations

Classes

Interfaces

Type Aliases

Functions