mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
68 lines
2.6 KiB
Plaintext
68 lines
2.6 KiB
Plaintext
---
|
|
title: "Observability"
|
|
sidebarTitle: "Observability"
|
|
description: "Pipeline counters for diagnosing StreamingDataset bottlenecks."
|
|
icon: gauge
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
Optimizing data loader performance can be tricky, as runtime bottlenecks are often difficult to locate across various stages of the I/O and CPU pipelines. To help distinguish bottlenecks, [`StreamingDataset`](/training/data-loading) exposes several pipeline counters: `raw_queue_depth` shows the number of loaded rows waiting to be transformed,
|
|
and `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and
|
|
`consumed_rows` show how far the current iterator has progressed.
|
|
|
|
If the `prefetch_queue_depth` is consistently zero but the `raw_queue_depth` is not, then you likely have a CPU
|
|
transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation.
|
|
This can often be done by batching the compute work. If both `prefetch_queue_depth` and `raw_queue_depth` are
|
|
consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped
|
|
shuffling could help.
|
|
|
|
<CodeGroup>
|
|
```py Python icon=Python
|
|
import threading, time
|
|
|
|
from lancedb.streaming import StreamingDataset
|
|
|
|
ds = StreamingDataset(table, shuffle_seed=42)
|
|
|
|
def log_pipeline_health():
|
|
while True:
|
|
print(
|
|
f"unscanned={ds.unscanned_rows} "
|
|
f"raw={ds.raw_queue_depth} "
|
|
f"cooked={ds.prefetch_queue_depth} "
|
|
f"consumed={ds.consumed_rows}"
|
|
)
|
|
time.sleep(1.0)
|
|
|
|
monitor = threading.Thread(target=log_pipeline_health, daemon=True)
|
|
monitor.start()
|
|
|
|
for sample in ds:
|
|
train_step(sample)
|
|
|
|
print(
|
|
f"bytes loaded: {ds.bytes_loaded} "
|
|
f"fetch time: {ds.fetch_time:.2f}s "
|
|
f"transform time: {ds.transform_time:.2f}s"
|
|
)
|
|
```
|
|
</CodeGroup>
|
|
|
|
`bytes_loaded` measures raw Arrow buffer bytes before transformation. `bytes_loaded`, `fetch_time`, and `transform_time`
|
|
are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times
|
|
can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters
|
|
reflect the latest iterator snapshot.
|
|
|
|
## Next steps {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="PyTorch Data Loader" icon="boxes-stacked" href="/training/data-loading">
|
|
Stream, shuffle and transform training data with StreamingDataset.
|
|
</Card>
|
|
<Card title="Checkpointing" icon="floppy-disk" href="/training/checkpointing">
|
|
Save progress so a failed run can pick back up.
|
|
</Card>
|
|
</CardGroup>
|