mirror of
https://github.com/lancedb/lancedb.git
synced 2026-01-13 23:32:57 +00:00
5a19cf15a6
I'm working on a lancedb version of pytorch data loading (and hopefully addressing https://github.com/lancedb/lance/issues/3727). However, rather than rely on pytorch for everything I'm moving some of the things that pytorch does into rust. This gives us more control over data loading (e.g. using shards or a hash-based split) and it allows permutations to be persistent. In particular I hope to be able to: * Create a persistent permutation * This permutation can handle splits, filtering, shuffling, and sharding * Create a rust data loader that can read a permutation (one or more splits), or a subset of a permutation (for DDP) * Create a python data loader that delegates to the rust data loader Eventually create integrations for other data loading libraries, including rust & node
32 lines
326 B
Markdown
32 lines
326 B
Markdown
[**@lancedb/lancedb**](../README.md) • **Docs**
|
|
|
|
***
|
|
|
|
[@lancedb/lancedb](../globals.md) / SplitSequentialOptions
|
|
|
|
# Interface: SplitSequentialOptions
|
|
|
|
## Properties
|
|
|
|
### counts?
|
|
|
|
```ts
|
|
optional counts: number[];
|
|
```
|
|
|
|
***
|
|
|
|
### fixed?
|
|
|
|
```ts
|
|
optional fixed: number;
|
|
```
|
|
|
|
***
|
|
|
|
### ratios?
|
|
|
|
```ts
|
|
optional ratios: number[];
|
|
```
|