rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2025-12-27 23:12:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
451411b1a166beecdf28400fff7cc4a95095137a
lancedb/docs/src/js/interfaces/SplitSequentialOptions.md
Weston Pace 5a19cf15a6 feat: a utility for creating "permutation views" (#2552) 
I'm working on a lancedb version of pytorch data loading (and hopefully
addressing https://github.com/lancedb/lance/issues/3727).

However, rather than rely on pytorch for everything I'm moving some of
the things that pytorch does into rust. This gives us more control over
data loading (e.g. using shards or a hash-based split) and it allows
permutations to be persistent. In particular I hope to be able to:

* Create a persistent permutation
* This permutation can handle splits, filtering, shuffling, and sharding
* Create a rust data loader that can read a permutation (one or more
splits), or a subset of a permutation (for DDP)
* Create a python data loader that delegates to the rust data loader

Eventually create integrations for other data loading libraries,
including rust & node
2025-10-09 18:07:31 -07:00

326 B
Raw Blame History

@lancedb/lancedb • Docs

@lancedb/lancedb / SplitSequentialOptions

Interface: SplitSequentialOptions

Properties

counts?

optional counts: number[];

fixed?

optional fixed: number;

ratios?

optional ratios: number[];