rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-26 13:36:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(python): open lazy blob files from SQL descriptors (#4668)

Browse Source 
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.

### Python experience

```python
rows = db.execute_query("""
    SELECT f.timestamp, blob_descriptor(v.video) AS video
    FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()

videos = db.open_table("videos")
for row in rows.to_pylist():
    if row["video"] is not None:
        with videos.open_blob_file(row["video"]) as video:
            frame = extract_frame(video, row["timestamp"])
```

The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.

The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:

```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
    if video is not None:
        with video:
            frame = extract_frame(video, row["timestamp"])
```

Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.

For an async table:

```python
async for video in async_videos.open_blob_files(rows["video"]):
    if video is not None:
        with video:
            payload = await video.aread()
```

### Execution

The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.

Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.

Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.

Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
This commit is contained in:
Jack Ye
2026-10-25 02:31:21 -07:00
committed by GitHub
parent 9af7757053
commit dcfef7c5ab
13 changed files with 674 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rust/lancedb/src/blob.rs
+110 -1
View File
@@ -20,11 +20,120 @@ use arrow_schema::{DataType, Field, Schema};
use lance::dataset::{BlobRangeRequest as LanceBlobRangeRequest, Dataset, WriteParams};
use lance_arrow::FieldExt;
use lance_file::version::{ConcreteFileVersion, LanceFileVersion};
use lance_io::object_store::ObjectStore;
use lance_io::object_store::{ObjectStore, uri_to_url};
use object_store::path::Path;
use serde::{Deserialize, Serialize};
use crate::error::{Error, Result};
/// A stored Blob V2 descriptor with the source needed to reopen its snapshot.
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct BlobDescriptor {
/// Canonical Blob V2 storage kind (0 through 5).
pub kind: u8,
/// Stored offset within the backing object.
pub position: u64,
/// Stored length; zero on an external blob can mean unspecified.
pub size: u64,
/// Stored sidecar or base identifier.
pub blob_id: u32,
/// Stored external URI or managed path.
pub blob_uri: String,
/// Source identity captured by the query.
pub source: BlobDescriptorSource,
}
/// The pinned source of a descriptor returned by SQL.
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct BlobDescriptorSource {
/// URI of the dataset on the source branch.
pub uri: String,
/// Exact table version read by the query.
pub version: u64,
/// Commit timestamp in microseconds, distinguishing recreated tables.
pub committed_at_micros: u64,
/// Source branch; `None` identifies main.
pub branch: Option<String>,
/// Blob column path in the source table.
pub column: String,
/// Row ID or row address, as specified by `row_is_address`.
pub row_reference: u64,
/// Whether the row reference is a physical address.
pub row_is_address: bool,
}
impl BlobDescriptor {
pub(crate) fn validate(&self) -> Result<()> {
if self.kind > 5
|| self.source.version == 0
|| self.source.uri.is_empty()
|| self.source.column.is_empty()
{
return Err(Error::InvalidInput {
message: "invalid stored blob descriptor or missing pinned source".into(),
});
}
Ok(())
}
}
pub(crate) async fn open_native_blob_file(
dataset: &Dataset,
descriptor: &BlobDescriptor,
) -> Result<BlobFile> {
descriptor.validate()?;
let source = &descriptor.source;
let expected_uri = dataset
.branch_location()
.find_branch(source.branch.as_deref())?
.uri;
if uri_to_url(&expected_uri)? != uri_to_url(&source.uri)? {
return Err(Error::InvalidInput {
message: "blob descriptor belongs to a different table".into(),
});
}
let dataset = Arc::new(
dataset
.checkout_version(lance::dataset::refs::Ref::Version(
source.branch.clone(),
Some(source.version),
))
.await?,
);
if uri_to_url(dataset.uri())? != uri_to_url(&source.uri)? {
return Err(Error::InvalidInput {
message: "blob descriptor belongs to a different table".into(),
});
}
if dataset.manifest().timestamp_nanos / 1_000 != u128::from(source.committed_at_micros) {
return Err(Error::InvalidInput {
message: "blob descriptor source snapshot has been replaced".into(),
});
}
ensure_blob_v2_column(dataset.schema(), &source.column)?;
let mut files = if source.row_is_address {
dataset
.take_blobs_by_addresses(&[source.row_reference], &source.column)
.await?
} else {
dataset
.take_blobs(&[source.row_reference], &source.column)
.await?
};
if files.len() != 1 {
return Err(Error::InvalidInput {
message: "blob descriptor does not resolve to one stored value".into(),
});
}
files
.pop()
.flatten()
.map(Into::into)
.ok_or_else(|| Error::InvalidInput {
message: "non-null blob descriptor resolved to a null value".into(),
})
}
/// Seekable handle for one blob value, backed by local storage or a remote
/// HTTP byte-range endpoint.
#[derive(Debug)]
rust/lancedb/src/remote/client.rs
+5
View File
@@ -809,6 +809,11 @@ impl<S: HttpSend> RestfulLanceDbClient<S> {
self.client.get(full_uri)
}
pub fn head(&self, uri: &str) -> RequestBuilder {
let full_uri = format!("{}{}", self.host, uri);
self.client.head(full_uri)
}
pub fn post(&self, uri: &str) -> RequestBuilder {
let full_uri = format!("{}{}", self.host, uri);
self.client.post(full_uri)
rust/lancedb/src/remote/table.rs
+4
View File
@@ -2927,6 +2927,10 @@ impl<S: HttpSend> BaseTable for RemoteTable<S> {
self.fetch_blob_files_impl(column, row_ids).await
}
async fn open_blob_file(&self, descriptor: &crate::blob::BlobDescriptor) -> Result<BlobFile> {
self.open_blob_file_impl(descriptor).await
}
async fn explain_plan(&self, query: &AnyQuery, verbose: bool) -> Result<String> {
if let AnyQuery::Query(request) = query
&& let Some(offsets) = &request.take_offsets
rust/lancedb/src/remote/table/blobs.rs
+176 -7
View File
@@ -21,7 +21,7 @@ use reqwest::{Response, StatusCode, header};
use tokio::sync::{Mutex, OwnedSemaphorePermit};
use crate::Error;
use crate::blob::{BlobFile, BlobRangeRequest};
use crate::blob::{BlobDescriptor, BlobDescriptorSource, BlobFile, BlobRangeRequest};
use crate::error::Result;
use crate::query::{QueryFilter, QueryRequest, Select};
use crate::remote::client::{HttpSend, RequestResultExt, RestfulLanceDbClient};
@@ -49,6 +49,7 @@ fn is_fetch_blobs_byte_limit_error(error: &Error) -> bool {
enum RangeRequestMode {
SizeProbe,
DataRead,
MetadataRead,
}
#[async_trait::async_trait]
@@ -86,6 +87,9 @@ struct TableBlobRangeRequester<S: HttpSend> {
path: String,
version: Option<u64>,
branch: Option<String>,
source_uri: Option<String>,
source_commit_micros: Option<u64>,
row_is_address: bool,
freshness: Arc<std::sync::Mutex<FreshnessState>>,
parent_freshness: Arc<std::sync::Mutex<FreshnessState>>,
parent_freshness_request: FreshnessHeaders,
@@ -109,15 +113,27 @@ impl<S: HttpSend> BlobRangeRequester for TableBlobRangeRequester<S> {
.expect("blob request semaphore is never closed");
let freshness_request =
freshness_headers_snapshot(&self.freshness, self.read_consistency_interval);
let mut request = freshness_request
.apply(self.client.get(&self.path))
.header(header::RANGE, range_header);
let mut request = if matches!(mode, RangeRequestMode::MetadataRead) {
freshness_request.apply(self.client.head(&self.path))
} else {
freshness_request
.apply(self.client.get(&self.path))
.header(header::RANGE, range_header)
};
if let Some(version) = version.or(self.version) {
request = request.query(&[("version", version)]);
}
if let Some(branch) = &self.branch {
request = request.query(&[("branch", branch)]);
}
if let Some(source_uri) = &self.source_uri {
request = request
.query(&[("source_uri", source_uri)])
.query(&[("row_is_address", self.row_is_address)]);
}
if let Some(timestamp) = self.source_commit_micros {
request = request.query(&[("source_commit_micros", timestamp)]);
}
let (request_id, response) = self.client.send_with_retry(request, None, true).await?;
// Preserve 416 size-probe responses so the caller can detect empty blobs.
if response.status() == StatusCode::RANGE_NOT_SATISFIABLE
@@ -131,8 +147,10 @@ impl<S: HttpSend> BlobRangeRequester for TableBlobRangeRequester<S> {
}
let response = self.client.check_response(&request_id, response).await?;
freshness_request.observe_headers(&self.freshness, response.headers());
self.parent_freshness_request
.observe_headers(&self.parent_freshness, response.headers());
if self.source_uri.is_none() {
self.parent_freshness_request
.observe_headers(&self.parent_freshness, response.headers());
}
Ok(BlobRangeResponse {
request_id,
response,
@@ -630,6 +648,51 @@ impl<S: HttpSend> RemoteTable<S> {
Ok(files)
}
pub(super) async fn open_blob_file_impl(
&self,
descriptor: &BlobDescriptor,
) -> Result<BlobFile> {
descriptor.validate()?;
self.ensure_blob_files_supported()?;
let source = &descriptor.source;
let snapshot = ReadSnapshot {
version: Some(source.version),
freshness_state: FreshnessState {
pinned: true,
..Default::default()
},
..Default::default()
};
let requester = self.blob_range_requester_with_source(
&source.column,
source.row_reference,
&snapshot,
Some(source),
);
let size = if descriptor.kind == 3 && descriptor.size == 0 {
let metadata = requester
.request_range("", RangeRequestMode::MetadataRead, Some(source.version))
.await?;
if metadata.response.status() != StatusCode::OK {
return Err(Error::InvalidInput {
message: "non-null blob descriptor did not resolve to a stored value".into(),
});
}
metadata
.response
.headers()
.get(header::CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<u64>().ok())
.ok_or_else(|| Error::InvalidInput {
message: "blob metadata response has no valid Content-Length".into(),
})?
} else {
descriptor.size
};
Ok(RemoteBlobFile::new(requester, size, Some(source.version)).into())
}
/// Probe each row once for its size.
///
/// Serves callers that must not issue a query, such as a single blob that
@@ -793,6 +856,16 @@ impl<S: HttpSend> RemoteTable<S> {
column: &str,
row_id: u64,
read_snapshot: &ReadSnapshot,
) -> Arc<dyn BlobRangeRequester> {
self.blob_range_requester_with_source(column, row_id, read_snapshot, None)
}
fn blob_range_requester_with_source(
&self,
column: &str,
row_id: u64,
read_snapshot: &ReadSnapshot,
source: Option<&BlobDescriptorSource>,
) -> Arc<dyn BlobRangeRequester> {
let path = format!(
"/v1/table/{}/blob/{}/{row_id}/bytes",
@@ -803,7 +876,10 @@ impl<S: HttpSend> RemoteTable<S> {
client: self.client.clone(),
path,
version: read_snapshot.version,
branch: self.branch.clone(),
branch: source.map_or_else(|| self.branch.clone(), |source| source.branch.clone()),
source_uri: source.map(|source| source.uri.clone()),
source_commit_micros: source.map(|source| source.committed_at_micros),
row_is_address: source.is_some_and(|source| source.row_is_address),
freshness: Arc::new(std::sync::Mutex::new(read_snapshot.freshness_state)),
parent_freshness: self.freshness.clone(),
parent_freshness_request: read_snapshot.freshness,
@@ -1024,6 +1100,99 @@ mod tests {
.unwrap()
}
#[tokio::test]
async fn open_descriptor_files_are_lazy_and_pin_ranges_for_every_storage_kind() {
for kind in 0..=5 {
let requests = Arc::new(StdMutex::new(Vec::new()));
let captured = requests.clone();
let table = RemoteTable::new_mock(
"videos".to_string(),
move |request| {
for name in [
"x-lancedb-min-version",
"x-lancedb-min-read-version",
"x-lancedb-min-timestamp",
] {
assert!(request.headers().get(name).is_none());
}
assert_eq!(
request.url().path(),
"/v1/table/videos/blob/video/12884901889/bytes"
);
let parameters = request
.url()
.query_pairs()
.into_owned()
.collect::<HashMap<_, _>>();
assert_eq!(parameters["version"], "7");
assert_eq!(parameters["source_commit_micros"], "1234");
assert_eq!(parameters["branch"], "dev");
assert_eq!(parameters["row_is_address"], "true");
assert_eq!(
parameters["source_uri"],
"s3://bucket/videos.lance/tree/dev"
);
captured.lock().unwrap().push(request.method().clone());
if request.method() == reqwest::Method::HEAD {
assert!(request.headers().get(header::RANGE).is_none());
http::Response::builder()
.status(200)
.header(header::CONTENT_LENGTH, PAYLOAD.len().to_string())
.body(Vec::new())
.unwrap()
} else {
range_response(&request, PAYLOAD)
}
},
Some(Version::new(0, 5, 0)),
);
{
let mut freshness = table.freshness.lock().unwrap();
freshness.min_version = Some(99);
freshness.min_read_version = Some(2);
}
let descriptor = BlobDescriptor {
kind,
position: 0,
size: if kind == 3 { 0 } else { PAYLOAD.len() as u64 },
blob_id: 0,
blob_uri: String::new(),
source: BlobDescriptorSource {
uri: "s3://bucket/videos.lance/tree/dev".into(),
version: 7,
committed_at_micros: 1234,
branch: Some("dev".into()),
column: "video".into(),
row_reference: 12884901889,
row_is_address: true,
},
};
let file = table.open_blob_file(&descriptor).await.unwrap();
let expected = if kind == 3 {
vec![reqwest::Method::HEAD]
} else {
vec![]
};
assert_eq!(*requests.lock().unwrap(), expected);
assert_eq!(file.size(), PAYLOAD.len() as u64);
file.seek(4).await.unwrap();
assert_eq!(file.read_up_to(5).await.unwrap().as_ref(), &PAYLOAD[4..9]);
assert_eq!(file.tell().await.unwrap(), 9);
assert_eq!(table.freshness.lock().unwrap().min_read_version, Some(2));
assert_eq!(file.read_range(0..3).await.unwrap().as_ref(), &PAYLOAD[..3]);
assert_eq!(file.tell().await.unwrap(), 9);
assert_eq!(
requests
.lock()
.unwrap()
.iter()
.filter(|method| **method == reqwest::Method::GET)
.count(),
2
);
}
}
fn mock_remote_blob_table(
requests: Arc<StdMutex<Vec<String>>>,
) -> RemoteTable<crate::remote::client::test_utils::MockSender> {
rust/lancedb/src/table.rs
+32 -1
View File
@@ -3,7 +3,7 @@
//! LanceDB Table APIs
use crate::blob::BlobFile;
use crate::blob::{BlobDescriptor, BlobFile};
use arrow_array::{LargeBinaryArray, RecordBatch, RecordBatchReader};
use arrow_schema::{Schema, SchemaRef};
use async_trait::async_trait;
@@ -796,6 +796,12 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
message: "fetch_blob_files is not supported on this table type".into(),
})
}
/// Open a descriptor at its pinned source version. See [`Table::open_blob_file`].
async fn open_blob_file(&self, _descriptor: &BlobDescriptor) -> Result<BlobFile> {
Err(Error::NotSupported {
message: "open_blob_file is not supported on this table type".into(),
})
}
/// Gets the table tag manager.
async fn tags(&self) -> Result<Box<dyn Tags + '_>>;
/// Optimize the dataset.
@@ -1377,6 +1383,26 @@ impl Table {
self.inner.fetch_blob_files(column.as_ref(), row_ids).await
}
/// Open a seekable file from a SQL `blob_descriptor` result.
///
/// The descriptor carries its table version, branch, column and row identity.
/// Opening does not materialize the payload or change this table's checkout.
/// Remote reads use the table's authenticated HTTP byte-range endpoint.
///
/// ```
/// use lancedb::{Table, blob::BlobDescriptor};
///
/// # async fn read_header(videos: &Table, descriptor: &BlobDescriptor) -> lancedb::Result<()> {
/// let video = videos.open_blob_file(descriptor).await?;
/// let header = video.read_range(0..video.size().min(4096)).await?;
/// video.close().await?;
/// # Ok(())
/// # }
/// ```
pub async fn open_blob_file(&self, descriptor: &BlobDescriptor) -> Result<BlobFile> {
self.inner.open_blob_file(descriptor).await
}
/// Insert new records into this Table
///
/// # Arguments
@@ -3764,6 +3790,11 @@ impl BaseTable for NativeTable {
crate::blob::take_blob_files_aligned(&dataset, column, row_ids).await
}
async fn open_blob_file(&self, descriptor: &BlobDescriptor) -> Result<BlobFile> {
let dataset = self.dataset.get().await?;
crate::blob::open_native_blob_file(&dataset, descriptor).await
}
/// Delete rows from the table
async fn delete(&self, predicate: Predicate<'_>) -> Result<DeleteResult> {
let result = delete::execute_delete(self, predicate).await?;