mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-21 04:25:53 +00:00
Weston Pace
5a19cf15a6
I'm working on a lancedb version of pytorch data loading (and hopefully addressing https://github.com/lancedb/lance/issues/3727). However, rather than rely on pytorch for everything I'm moving some of the things that pytorch does into rust. This gives us more control over data loading (e.g. using shards or a hash-based split) and it allows permutations to be persistent. In particular I hope to be able to: * Create a persistent permutation * This permutation can handle splits, filtering, shuffling, and sharding * Create a rust data loader that can read a permutation (one or more splits), or a subset of a permutation (for DDP) * Create a python data loader that delegates to the rust data loader Eventually create integrations for other data loading libraries, including rust & node
24 lines
256 B
Markdown
24 lines
256 B
Markdown
[**@lancedb/lancedb**](../README.md) • **Docs**
|
|
|
|
***
|
|
|
|
[@lancedb/lancedb](../globals.md) / ShuffleOptions
|
|
|
|
# Interface: ShuffleOptions
|
|
|
|
## Properties
|
|
|
|
### clumpSize?
|
|
|
|
```ts
|
|
optional clumpSize: number;
|
|
```
|
|
|
|
***
|
|
|
|
### seed?
|
|
|
|
```ts
|
|
optional seed: number;
|
|
```
|