mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
9bb2be56a86a377cf0961ae835d3f5a58df172e1
3001
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Jonathan M HsiehandClaude Opus 5
|
9bb2be56a8 |
fix: leave name length to the service, and check the delimiter on the config
Three review items.
The Secret character set no longer carries a length bound. `{1,255}` was the
only client-side length rule for any object -- tables, namespaces, Functions
and materialized views have never had one -- and it disagreed with the service,
which allows 512 bytes. A bound here could only be wrong in one direction or
the other: shorter refuses names the catalog would hold, longer says nothing.
How long a name may be is the service's to decide, as it already is everywhere
else.
`id_delimiter` is checked through `ClientConfig::validate` now, called as a
precondition before a client is built from the config rather than from inside
the struct literal that consumes it. A configuration mistake is reported where
it was made. The method is public so a caller assembling a config can ask the
same question without connecting.
It also refuses the exact relative segments. The root namespace is addressed by
the bare delimiter, so `id_delimiter="."` makes a root listing
`/v1/namespace/./secret/list`, which resolves to `/v1/namespace/secret/list` --
a different route, and one that exists. Only the exact values are refused: `.x.`
and `...` remain usable, for the same reason `...` is a usable name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
8b8479dc21 |
fix: narrow the relative-segment rule, and check the delimiter itself
Two corrections from review, both verified against the URL parser rather than
reasoned about.
The rule refused any component of periods alone. URL parsing resolves only `.`
and `..`:
"." -> /v1/function/drop collapses
".." -> /v1/drop collapses
"%2e" -> /v1/function/drop collapses
"..." -> /v1/function/.../drop addressable
"...." -> /v1/function/..../drop addressable
So `...` and longer runs are ordinary segments that address perfectly well, and
refusing them would have made a table, namespace, Secret or Function that works
today stop working on upgrade -- for a hazard it does not have. The predicate is
now exactly `.` and `..`, and the percent-encoded spellings of them, since
resolution happens after decoding. A test pins `...`, `....`, `a.`, `.a` and
`a..b` as addressable so the rule cannot silently widen again.
Second, refusing a component that *contains* the delimiter does not cover a
delimiter that is itself path syntax -- there, neither component contains one:
id_delimiter = "/" drop_secret("openai", ["prod"]) -> /v1/secret/prod/openai/drop
which is one segment too many and matches no route at all. The delimiter is
written into the path raw, because it is what separates the components, so it
cannot be escaped the way they can. It is now checked where the client is
built: not empty, and free of `/`, `?`, `#`, `%` and `\`. An empty delimiter
fails from the other side -- it joins `["a", "b"]` into `ab`, which no split
recovers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
912195cee2 |
feat: check table, namespace and Function identifiers the same way
Extends the Secret component rules to the other two objects that join a name
into a route, because the failures are the same and were reachable from the
public Rust API for all three.
drop_table("../jobs", &[]) -> /v1/jobs/drop/
drop_table("a$b", &[]) -> /v1/table/a$b/drop/
drop_table("b", &["a"]) -> /v1/table/a$b/drop/ the same URL
The first leaves the table route entirely. The second and third are one URL
for two identities -- and `a$b` is not a legal table name, so what the caller
asked for silently becomes a different table that may exist. Neither is an
error the service could return: it is asked about the wrong object, or not
asked at all.
The dot-segment rule moves into `validate_table_name` and
`validate_namespace_name`, so the local path in `database/listing.rs` and the
service -- which imports both -- inherit it with the client. `.` and `..` pass
every character test and still cannot address anything: they are resolved as
relative path segments, and after percent-decoding, so `%2E%2E` is resolved
exactly as `..` is.
`build_table_identifier` and `build_namespace_identifier` become fallible and
validate each component before joining. That covers every remote table route
from one place rather than each call site. Both also refuse a component holding
the configured delimiter, which is the rule that cannot live in a shared
validator: the default `$` is outside every charset here, but a configured
delimiter may be any string and the readable choices -- `.`, `-`, `_` -- are
all inside them.
Functions need only the dot-segment rule. Their identifier is percent-encoded,
which covers the rest, and they have no namespace to join -- so there is no
boundary a delimiter could erase. `..` is unreserved, so it survives encoding
untouched and would post a registration body to `/v1/create`.
One test changed rather than being added to. `test_table_names_with_nested_namespace`
had its mock return `ns1$ns2$table1` from a namespace listing; the service
returns bare names there, as the sibling test and the service's own tests both
show. The client joins the namespace back on to build each table's identifier,
so the joined form was being joined twice into `ns1$ns2$ns1$ns2$table1`. Nothing
caught it because nothing validated the result.
Tests: a dot-only or delimiter-bearing table name, namespace segment and
Function name are each refused before a request is built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
7898699208 |
feat(secrets): check the identifier where it is built, not after it is split
A Secret is addressed as `POST /v1/secret/{id}/<action>`, where `{id}` is the
namespace path and the name joined by a delimiter. The client performs that
join; the service can only judge the components the split produced. So a
component carrying path or delimiter syntax does not arrive as bad input for
the service to refuse -- it arrives as a different, well-formed identity.
`create_secret("a$b", value)` read as the namespace `a` and the name `b`: a
Secret that may already exist, whose credential an alter would overwrite. The
service never refused it because it was never asked about `a$b`. `"../jobs"`
did not reach the service at all, taking a body holding a credential to
whatever route the normalized path resolved to.
`Connection` now checks each component before anything is joined:
alphanumerics, `_`, `-` and `.`, at most 255 bytes, and not periods alone --
a component of only periods is read as a relative path after decoding, so no
spelling of it addresses anything. `RemoteDatabase` adds the one rule that
depends on configuration it owns: a component may not contain the delimiter it
is about to be joined with. The default `$` is outside the character set, so
that fires only for a configured delimiter, where the readable choices -- `.`,
`-`, `_` -- are all inside it.
This is not a second opinion on whether a name is acceptable. It is what makes
the service's opinion reachable, and it belongs to whoever builds the URL:
- The join happens here, and the join is what decides identity.
- It is one definition, not two. The service already imports this crate's
`validate_table_name` and `validate_namespace`; the Secret rule is placed
beside them so it can be imported the same way rather than restated.
- It covers every binding. Python and Node wrap this crate, and today the
checks are scattered -- Secrets validated in Python, Functions at
registration, tables in exactly one method, Rust nowhere.
- `validate_table_name` is already called on the local path in
`database/listing.rs`. The remote path is the inconsistent one.
Percent-encoding stays as a second line. Every admitted character is
unreserved, so an accepted component encodes to itself and the route reads as
before -- `/v1/secret/prod$vision_2$openai-prod.v1/drop`.
Tests: an illegal component never reaches the transport; a dot-only component
is refused as a name and as a segment; a configured delimiter is refused inside
a component while the same two names as separate segments are the ordinary
case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
516bb586a1 |
fix(secrets): make the identifier join reversible and admit the listing route
Two gaps in the route boundary, both found in review. Percent-encoding a component does not make the join reversible. It leaves the unreserved characters alone, and `.`, `-` and `_` are both unreserved and plausible delimiters -- so under `id_delimiter="."` the one component `prod.vision` and the two components `prod` and `vision` join to the same string, and two distinct Secrets share one route. An alter or a drop then reaches whichever exists. The delimiter is now escaped inside each component, which is what keeps the boundaries the caller drew. The default `$` is not unreserved, so it was already escaped inside a component and only ever appeared raw as the separator; nothing on that path changes shape. The listing route took a namespace identifier without the admission the Secret identifier gets, so `list_secrets(&[".."])` resolved to `/v1/secret/list` and left the route it was built for. Both identifiers now go through one `require_addressable`, which is also why it is no longer named for Secrets alone. Tests: distinct identities stay distinct under a configured delimiter, and a dot-only namespace is refused before a request is built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
fa23de3e65 |
fix(secrets): keep a Secret name from choosing the route it travels to
Addressing a Secret by a path segment makes its name part of the URL's
structure, not just its content. The name arrives from a public Rust API that
does no validation -- validation is the service's -- so a name carrying path
syntax decided which handler received the request. `create_secret("../jobs",
value)` resolved to `/v1/jobs/create` and delivered a body holding a live
credential to a route with none of this one's body suppression.
Each component of an identifier is now percent-encoded before the delimiter
joins them. Per component, not over the joined string: the delimiter stays a
delimiter, and a component cannot become two. Every character a Secret name and
a namespace segment may hold is unreserved, so an identifier the service would
accept passes through byte-identical and reads like the table and Function
identifiers beside it -- `/v1/secret/prod$openai-prod.v1/drop`. One the service
would refuse is merely made to arrive, so that it can be refused. This is not
name validation; it is what makes the service's validation reachable.
Encoding cannot cover a dot-only component. `.` and `..` are unreserved, so
nothing in them needs escaping character by character, but URL parsing resolves
dot segments after percent-decoding -- `%2E%2E` is a double-dot segment as
surely as `..` is, and no spelling of one reaches a route. Those are refused
before a request is built, with the client reporting that it cannot express the
request rather than sending one that resolves elsewhere.
Tables and Functions join their identifiers unencoded and have the same gap:
`drop_table("../jobs", &[])` produces `/v1/jobs/drop/` today. That is filed
separately rather than widened into this change, which is already the first
route to carry a credential in its body.
Tests: the two reproducers from review are kept as regression tests -- a name
cannot leave `/v1/secret/`, and two distinct identities cannot share one URL --
alongside one that pins the ordinary case as unencoded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
7c25676d05 |
style(secrets): apply rustfmt to the describe_secret request
`cargo fmt --all -- --check` is a CI gate and this line did not satisfy it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
c18aabf8b1 |
docs(secrets): name both halves of the binding forward-compatibility contract
The `SecretBinding` doc said an unknown delivery mode decodes rather than failing the FunctionVersion, but named only the `Unrecognized` variant. That is half the mechanism: the variant gives the wire somewhere to land, and `#[non_exhaustive]` is what denies callers an exhaustive match, so a mode added later arrives as a case they were already required to handle rather than as a break. Both are load-bearing and neither works alone -- without the attribute a caller could match `Env` and `Unrecognized` and still be broken by a third variant -- so the doc now says so, the same pairing `PythonRuntimeSpec` uses for runtimes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
9d26cc42cf |
docs(secrets): say why the code is as it is, not what it replaced
Comments that explained a decision by naming what came before it read as
changelog rather than rationale, and they age badly: a reader who never saw the
earlier shape learns nothing from being told it existed. This is a new surface,
so every comment here now states the property that holds today.
- `RemoteCreateFunctionRequest` said a literal "is why `secret_bindings`
silently stopped reaching the service". It now says what the struct buys:
the compiler decides what reaches the service, and a new field on the
registration request is a build error here until it is handled.
- Wire-omission comments that appealed to clients predating Secrets or
namespaces now state the rule itself -- a binding states a namespace only
when it has one, a body carries a binding list only when there is one.
- `BodyLogging::Allowed` described logging "as every request did before
Secrets existed", which is really the default: a body is diagnostic unless
the call that built it says otherwise.
- `a_version_without_bindings_keeps_the_original_wire_shape` named a shape no
longer worth referencing. Both the Rust and Python tests now name the
property they check -- absent decodes as an empty list, and an empty list
serializes back to absent.
Comments that cite the Table and Function conventions are kept. Those describe
what the code sits beside today, which is the reason a reader needs.
Tests: lancedb secret unit tests 7, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, 93 Python tests across both slices.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
6b77670e5c |
refactor(secrets): address Secrets the way every other object is addressed
Secret CRUD used plural collection routes with the identity in the body, which is the one shape in the HTTP surface that does not match Table and Function CRUD. #4166 moved Functions onto `<object>/{id}/<action>`; this does the same for Secrets, so there is one route grammar rather than one per object type. POST /v1/secrets/create -> POST /v1/secret/{id}/create POST /v1/secrets/alter -> POST /v1/secret/{id}/alter POST /v1/secrets/describe -> POST /v1/secret/{id}/describe POST /v1/secrets/drop -> POST /v1/secret/{id}/drop POST /v1/secrets/list -> GET /v1/namespace/{id}/secret/list `{id}` is the namespace path and the name joined with the client's configured delimiter, built by the same helper a table identifier is. The name and the namespace therefore leave every request body: create and alter carry the value alone, drop and describe carry nothing, and listing moves its `page_token` to a query parameter. `add_namespace_path` goes with them -- a root Secret is addressed by its bare name, so there is no longer an empty field to omit. Identifiers are written into the path as-is rather than percent-encoded. The charset a Secret name and a namespace segment admit already excludes everything that would need escaping, and a route that reads like the ones beside it is worth more than encoding against a charset that does not exist yet. If that charset widens, this is the place that has to widen with it. Listing takes a real namespace rather than rejecting a non-root one the way the Function route does: Secrets are addressable inside nested namespaces, so the namespace identifier is the whole point of the route. `build_secret_identifier` is a wrapper over `build_table_identifier` rather than a second implementation. The grammar belongs to the namespace spec rather than to tables, and a call site should read as the thing it addresses. Also removes `_DECLARED_SECRET`, which has been unreferenced since client-side binding validation was dropped, and whose pattern was the environment variable grammar rather than a Secret name's. Tests: lancedb secret unit tests 7, `first_class_function_slice1` 9, `first_class_function_slice2` 3, 93 Python tests across both slices. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
dddc94474f |
feat(secrets): address Secrets by namespace path
A Secret is identified by a namespace path plus a name, and resolution is
exact: one under `["prod"]` is not visible from `["prod", "vision"]` and never
falls back to a parent.
The binding names it as a `SecretReference` of `{name, namespace_path}` rather
than one joined string. A joined id needs a delimiter, and a delimiter has to be
excluded from every name and every segment forever, agreed on by both sides, and
re-agreed each time either side grows a new way to be configured --
`ClientConfig.id_delimiter` is exactly that, configurable and advertised on
every request, so an id joined on a fixed `$` would contradict a delimiter the
same request declares. Naming the parts costs one object and settles it.
Every verb takes `namespace_path` keyword-only, defaulting to the root. On
`EnvVarSecret` the same argument is `secret_namespace_path`, matching
`secret_name`: a binding names two things, and the unqualified word reads as
the variable's namespace rather than the Secret's. It pins the path at
construction, so a worker resolves the Secret it was handed rather than
re-resolving against its own default.
The five PyO3 secret methods take `Option<Vec<String>>` with a `None` default,
as every other namespace-taking method on `Connection` already does. They were
the only ones requiring the argument positionally, so the keyword form their
neighbours accept did not work on them.
A root path is omitted from the request body rather than sent empty, so a root
request is byte-identical to one from a client that predates this, and a root
binding to one written before namespaces existed. Tests pin both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
fc68417397 |
refactor(secrets): one binding list with a kind, and split the secret writes
`secret_env_bindings` took the general noun for one delivery mode. A binding is
the concept; how it arrives is a property of one. `secret_bindings` is a list of
`SecretBinding`, tagged by `kind`, so a later mode is a variant rather than a
sibling field -- and the rules that are per-Function, like how many Secrets it
may bind, stay answerable from one place.
The cost of one field is that an unknown kind is a decode error unless it is
caught. It is caught, the way `PythonRuntimeSpec` catches an unknown runtime:
`Unrecognized { kind }`, `#[non_exhaustive]`, and the payload dropped rather
than retained because the client does not proxy catalog values. A test pins it
-- a `file` binding from a newer server decodes, reports its kind, and
round-trips as its discriminator without failing the version around it.
A list has no key order to inherit, and the list is in the version hash, so
`bind_secrets` sorts it: a caller's argument order is not part of what a
Function is.
The Secret is named under `secret_ref`, not `secret`: the service scans Job
payloads for credential-shaped keys and refuses one called `secret` whatever it
holds. That guard is worth more blunt than argued with.
`EnvVarSecret` names it the same way for the same reason: the field is
`secret_name`, not `secret`. In a binding both halves need saying -- which
variable, which Secret -- and the bare noun reads as the credential rather than
the name of one, which is the misreading the type exists to prevent.
`create_secret` and `alter_secret` also stop sharing a request shape. They are
different operations to the service -- one refuses an existing name, the other
requires it -- and either may grow a field the other has no meaning for. What
they share is posting a body that must not be logged, which is a function.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
|
||
|
Jonathan M HsiehandClaude Opus 5
|
7e55977dc7 |
refactor(secrets): epoch timestamps, a named list request, and unit tests
`SecretInfo` carries `created_at_millis` / `updated_at_millis` as `i64` rather than RFC 3339 strings, matching `created_at_millis` elsewhere in the platform. A caller comparing two timestamps no longer parses anything, and the PyO3 layer hands Python integers instead of stringifying them. `updated_at_millis` is the only observable that a rotation landed -- no API returns a credential -- so it is worth being a number a caller can compare. `RemoteListSecretsRequest` is declared alongside its response instead of being an inline object built field by field, so a reader of one finds the other. `function.rs` gains unit tests for the pure parts that had none: canonical JSON sorts keys at every depth and leaves array order alone -- it is what the version hash is taken over, so both matter -- floats are refused at any depth, and the unknown-key check inspects only the level it is handed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
841edf1316 |
fix(secrets): drop an import left behind by the removed validation tests
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
7d8bff1227 |
refactor(secrets): give Secrets their own module, leave validation to the service
Three things from review. `SecretInfo` moves to `lancedb::secrets`. It was in `database` because that is where the trait lives, not because it belongs to a database's shape; a new object should land in its own module and the Secret verbs will follow it. Binding and value validation come out of the client. The service owns those rules -- it owns the runtime the names land in and the store the values go to -- and a copy here could disagree with it without either side noticing. What the client was checking, the service already rejects: the per-Function cap, the environment-variable grammar, disjointness from `runtime.env`, and the value size. The cost is a round trip before the error, and the error is the service's own words rather than a paraphrase that can drift. The tests follow the rule rather than the check: what the client guarantees is that the envelope it sends is the envelope the caller wrote, so that is what is asserted now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
d81310fa96 |
fix(secrets): admit periods in Secret names and namespace segments
LanceDB namespace and table names are `[A-Za-z0-9_.-]`, so excluding periods here put Secrets out of reach inside any namespace a user already has one in -- unreachable to create, alter, describe, drop or bind, with no way to recover but renaming the namespace. The rule is that set and nothing further. No positional constraint rides along: the established grammar says nothing about which character comes first, so a segment may begin with `_`, `-` or `.` today, and a Secret has to be nameable wherever a namespace already is. A narrower rule would reintroduce the same unaddressability it is here to remove, one character class over. Names and path segments follow one rule, and the client matches the service. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
ecf44f3967 |
fix(secrets): complete the binding and value contract at the Rust boundary
`validate` checked only the binding count, so a caller reaching the low-level entry point past `bind_secrets` could register an environment variable name the runtime cannot deliver, or bind a Secret to a name `runtime.env` already sets -- which would resolve by delivery order, with a value visible in the Function's record and a value that is not. Secret values are bounded here too, at the limit the service enforces, so an oversized credential is refused before a request body is built rather than after it has been serialized and uploaded. Sandbox-reserved names are deliberately still the service's alone: that list belongs to the runtime that owns it, and a copy here would drift from it silently. Both gate reproducers now fail closed with no request reaching the service. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
7db75a7cdb |
fix(secrets): keep credentials out of the client's own debug log
`log_request` logs any JSON body verbatim at debug, and Python and Node both wire that logger to `LANCEDB_LOG`. `create_secret` posts the value in its body, so ordinary SDK debug logging wrote the credential to application logs. The comment on `write_secret` reasoned correctly about proxy traces and access logs and missed the logger in this process. Redaction cannot live in the value model: the logger sees the serialized body, where the credential is already plaintext bytes. So the request says whether its body may be logged -- `send_suppressing_body` for the one whose body is the credential -- and `log_request` obeys rather than deciding. The transport cannot tell a credential from any other payload, and a list of routes there would have to be kept in step with endpoints declared elsewhere. The same debug line prints the request's Debug, which prints headers, so the API key was in every debug line of every request regardless of route. Marking the header value sensitive is what stops that. The end-to-end regression fails without this: the log carried `,"value":"SECRET_VALUE_SENTINEL"}` verbatim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
1a959990ca |
refactor(secrets): name the binding field for its delivery mode, cap in Rust
`secret_bindings` read as the general record of what a Function needs, but a `Map<String, String>` keyed on environment-variable name can only ever hold an env-delivered binding: an accessor binding has no variable to key on. Naming the field for its delivery mode leaves a later mode a sibling field rather than a tagged value type, which would break a field already in the identity hash. The per-Function cap moves from `bind_secrets` to `Connection::create_function_async`, above the backend dispatch, so it holds for every language surface rather than only the one that validates first. The low-level PyO3 entry point reached the wire past the Python check; it no longer does. Also corrects a doc comment claiming registration fails when a bound Secret is absent. Existence is first answered at `add_columns`, by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6 |
||
|
Jonathan M HsiehandClaude Opus 5
|
1f97e81e54 |
feat(secrets): named Secrets and EnvVarSecret bindings
Adds the client half of database-scoped named Secrets: a Secret is a name and an opaque value stored by the service, and a Function binds one to the environment variable its library already reads. - `db.create_secret` / `alter_secret` / `list_secrets` / `describe_secret` / `drop_secret` on sync, async and remote connections, with the pyo3 binding and the Rust client behind them. There is no read API by construction rather than by policy: no code path returns a stored credential, and `describe_secret` answers with metadata only. - `EnvVarSecret(secret=..., env_variable=...)` pairs a Secret with the variable it arrives in. It is a pure local constructor -- it contacts no server, so it cannot fail on a Secret that does not exist -- and it exists so a bare string in that position, which would be a credential, is a TypeError rather than a plausible-looking mistake that reads identically in a diff. - `create_function(..., secrets=[...])` carries the bindings as `secret_bindings`, a map from variable name to Secret name. The value never travels: it is resolved by the service when the Function runs, which is what lets a rotation reach columns pinned to an older FunctionVersion. The UDF body is unchanged and stays portable -- it reads `OPENAI_API_KEY` the way it always did, and the binding is what puts a value there. Squashed: the original three commits were a first design plus a rewrite of it, so their sequence describes an interface that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XE1UwYKsgbb3USBfkqCE6v |
||
 Jack Ye
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
 Wyatt Alt
|
2d4622491e |
fix: keep computed-column freshness across branches and clones (#4188)
A branch or shallow clone recomputed every computed column on its first refresh: the input signature carried each data file's raw base id, which the clone commit rewrites, and the signature sidecar was looked up under the branch's own tree, where nothing was ever written. A file is now identified by where its store resolves it, so the source's own root and a clone's registered reference to that root sign the same while different bases stay distinct; compaction products are matched the same way. Sidecars live in one `_computed/` at the table root shared by main and its branches; a shallow clone reads through the base it was cloned from and keeps a copy. Pruning collects references across every branch, and the staleness walk treats the versions a branch does not hold as unknown. Stamps written before this change no longer match, so the first refresh after upgrading recomputes once; a deep clone, which copies no sidecar, still starts over. |
||
 Jack YeandXuanwo
|
2f88b71c21 |
feat: add persistent OAuth token cache and session APIs (#4182)
Stacked on #4173 (diff includes it until that merges; will rebase after). Addresses the token-cache part of [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). Adds an explicit, opt-in persistent OAuth token cache shared by Rust, Python, and Node clients, plus `login` / `status` / `logout` session APIs, so short-lived processes (CLIs, scripts, notebooks) reuse one session instead of restarting a browser or device flow on every start. - **Opt-in and minimal**: existing callers stay memory-only and lazy. Only refresh tokens are persisted (never access tokens, never client secrets), so there are no local token-expiry decisions to get wrong when clocks move. Each process start performs one silent refresh grant. - **Hardened file backend**: private directory (`0700`), per-record files (`0600`), owner validation, symlink rejection, and atomic `rename` replacement. Corrupt, truncated, unknown-version, or permission-invalid records fail with actionable errors naming the file. Native keyring backends were evaluated (keyring crate routes Linux through D-Bus/zbus: heavy deps, headless/CI flakiness) and are deferred; the file store is the explicit opt-in, not a downgrade from a keyring. - **Cache key**: SHA-256 of the canonical identity (issuer, client ID, sorted/de-duplicated scopes, flow, public/confidential), so no secret appears in a filename and distinct identities never collide. Versioned record schema (`version: 1`). One record per identity: last login wins, documented. - **Cross-process rotation locking**: per-key `fs4` file lock (`flock` / `LockFileEx`) around the refresh critical section — acquire, reread the durable record, refresh exactly once, atomically store the rotated refresh token, release. The OS releases locks on process death, so crashes cannot strand stale locks. Only confirmed `invalid_grant`/`invalid_token` deletes a record and reauthenticates; transport, 5xx, 429, and parse failures retain it. - **Session APIs**: `OAuthSession::login/status/logout` in Rust, `lancedb.remote.OAuthSession` (async) in Python, `OAuthSession` class in Node. `status` returns non-secret metadata only. `logout` removes only the local credential — provider revocation (RFC 7009) is a deliberate follow-up, and local logout never terminates browser SSO. Azure managed identity is rejected for persistence (machine identity stays in memory); client credentials have nothing refreshable to persist and stay memory-only. - No CLI binary exists in this repo, so this ships library APIs plus doc examples in all three languages. Tests: Rust unit + mock-IdP integration (cache-key canonicalization/separation, record versioning/corruption/truncation/symlink/owner/perms, lock serialization + release, two concurrent providers proving no `invalid_grant` and correct rotation, transient-failure retention, `invalid_grant` delete + reauthenticate, login/status/logout lifecycle, client-credentials no-op, IMDS rejection, secret redaction); Python lifecycle + a true two-subprocess cross-process reuse test (second process refreshes once, never hits the device endpoint); Node lifecycle + device-flow login test. Local builds were skipped in development; CI validates all bindings. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
575286922b |
fix: redact authentication headers from debug logs (#4179)
Prevent bearer tokens, API keys, cookies, and proxy credentials from appearing in request debug output through a reusable `redact_sensitive_headers` utility. The utility is applied after default/configured header construction and dynamic header merging, so it also covers retry requests. Invalid dynamic-header values are no longer echoed, and regression coverage verifies that credentials are redacted while ordinary diagnostic headers remain visible. Extracted from the OAuth discussion in #4173, following [Colin's review](https://github.com/lancedb/lancedb/pull/4173#issuecomment-5674048100). |
||
 dependabot[bot]andXuanwo
|
09e5418943 |
build(deps): bump the rust-minor-patch group across 1 directory with 5 updates (#4178)
Bumps the rust-minor-patch group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [uuid](https://github.com/uuid-rs/uuid) | `1.26.0` | `1.26.1` | | [serde_with](https://github.com/jonasbb/serde_with) | `3.22.0` | `3.23.0` | | [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.4.8` | `1.6.3` | | [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.3` | `3.6.5` | | [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.1` | `2.4.2` | Updates `uuid` from 1.26.0 to 1.26.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/uuid-rs/uuid/releases">uuid's releases</a>.</em></p> <blockquote> <h2>v1.26.1</h2> <h2>What's Changed</h2> <ul> <li>Seat the v7 counter below the version nibble by <a href="https://github.com/lenamonj"><code>@lenamonj</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> <li>Don't panic in overflowing Timestamp to SystemTime conversion by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/909">uuid-rs/uuid#909</a></li> <li>Prepare for 1.26.1 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/910">uuid-rs/uuid#910</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/lenamonj"><code>@lenamonj</code></a> made their first contribution in <a href="https://redirect.github.com/uuid-rs/uuid/pull/907">uuid-rs/uuid#907</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/uuid-rs/uuid/commit/9f927126c89892ddfed6cd2f92df16852f3f9aa6"><code>9f92712</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/910">#910</a> from uuid-rs/cargo/v1.26.1</li> <li><a href="https://github.com/uuid-rs/uuid/commit/d4df8f0cd9f461b4ef493254420052ffa5ce6277"><code>d4df8f0</code></a> prepare for 1.26.1 release</li> <li><a href="https://github.com/uuid-rs/uuid/commit/5613f2357c1fc06afc5fffd98e96da2ccf25a608"><code>5613f23</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/909">#909</a> from uuid-rs/fix/ts-conversion-overflow</li> <li><a href="https://github.com/uuid-rs/uuid/commit/fda00eba938383d242bad33143c8af73227f2a2c"><code>fda00eb</code></a> don't panic in overflowing Timestamp to SystemTime conversion</li> <li><a href="https://github.com/uuid-rs/uuid/commit/c82e88ca184e4ab83ce6b4ac0be33d32a0b9c3c4"><code>c82e88c</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/907">#907</a> from lenamonj/v7-counter-placement</li> <li><a href="https://github.com/uuid-rs/uuid/commit/ac065a6c17389a67dc6e98ef5f9a2e1d7ad4a670"><code>ac065a6</code></a> Align the counter diagram</li> <li><a href="https://github.com/uuid-rs/uuid/commit/34ec10208d813672928c299146dfe4c18cedcec7"><code>34ec102</code></a> Seat the v7 counter below the version nibble</li> <li>See full diff in <a href="https://github.com/uuid-rs/uuid/compare/v1.26.0...v1.26.1">compare view</a></li> </ul> </details> <br /> Updates `serde_with` from 3.22.0 to 3.23.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/jonasbb/serde_with/releases">serde_with's releases</a>.</em></p> <blockquote> <h2>serde_with v3.23.0</h2> <h3>Changed</h3> <ul> <li>Update <code>syn</code> and <code>darling</code> dependencies to use <code>syn</code> v3 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/992">#992</a>)</li> <li>Update dev-dependencies to newer versions (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li>Update <code>base64</code> to a newer version. This should not have any API change, but some error messages might change. (<a href="https://redirect.github.com/jonasbb/serde_with/issues/993">#993</a>)</li> <li><code>serde_as</code> can now parse <code>cfg_attr(true, ...)</code> and <code>cfg_attr(false, ...)</code> (<a href="https://redirect.github.com/jonasbb/serde_with/issues/995">#995</a>) <code>true</code>/<code>false</code> are new literals as of Rust 1.88 but need to be parsed explicitly with the <code>syn</code> types. This is used when emitting <code>schemars</code> annotations.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jonasbb/serde_with/commit/ea5dfdc6fd1b4732188519e871e2fd2a8fe49f88"><code>ea5dfdc</code></a> Bump version to v3.23.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1005">#1005</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e52e85b1db55e4dc6305b6b19c2710d6a8b2d433"><code>e52e85b</code></a> Bump version to v3.23.0</li> <li><a href="https://github.com/jonasbb/serde_with/commit/39955b6954796d963d02d2f28e0a22087c48fcd4"><code>39955b6</code></a> Bump rmp dev-dependency (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1004">#1004</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/b51e5fb877c642c47da36c296c222e1d172d75da"><code>b51e5fb</code></a> Bump rmp dev-dependency</li> <li><a href="https://github.com/jonasbb/serde_with/commit/ef598c6683d591851e09cac3a4bd767c93bacb9e"><code>ef598c6</code></a> Use setup-rust-toolchain v2 instead of v1 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1003">#1003</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/9ea2658f45e763369f18078d80ea3b109d491de8"><code>9ea2658</code></a> Fix cargo lint about workspace lints being inherited in the test crate</li> <li><a href="https://github.com/jonasbb/serde_with/commit/e5ad81af7721bb770772e8e28de32437f3ff217f"><code>e5ad81a</code></a> Use setup-rust-toolchain v2 instead of v1</li> <li><a href="https://github.com/jonasbb/serde_with/commit/81001414f153d18fc1aca6c2f52990c557f90471"><code>8100141</code></a> Bump the github-actions group across 1 directory with 2 updates (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1002">#1002</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/940f4a62a418784e23026953db8a8fed577bff6d"><code>940f4a6</code></a> Bump jsonschema from 0.49.8 to 0.52.0 (<a href="https://redirect.github.com/jonasbb/serde_with/issues/1001">#1001</a>)</li> <li><a href="https://github.com/jonasbb/serde_with/commit/6f42b94d7bb71b004d0e591a26fbf8d900878c91"><code>6f42b94</code></a> Bump the github-actions group across 1 directory with 2 updates</li> <li>Additional commits viewable in <a href="https://github.com/jonasbb/serde_with/compare/v3.22.0...v3.23.0">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-types` from 1.4.8 to 1.6.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `napi-derive` from 3.6.3 to 3.6.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-derive's releases</a>.</em></p> <blockquote> <h2>napi-derive-v3.6.5</h2> <h3>Other</h3> <ul> <li>update Cargo.toml dependencies</li> </ul> <h2>napi-derive-v3.6.4</h2> <h3>Fixed</h3> <ul> <li><em>(deps)</em> update rust crate convert_case to 0.12 (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3469">#3469</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/1492b220d5ad01807b2dcbd250e8383f9d738311"><code>1492b22</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3502">#3502</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/828983aab9b93f516275625f89905fffca459780"><code>828983a</code></a> fix(napi): return errors from the serde deserializer for unexpected JS value ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/606b14238b8d82578705161c0b0d6b6f4b7c2556"><code>606b142</code></a> fix(napi): validate wrapped payload provenance in Object::unwrap/remove_wrapp...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a75d89f85c0a9a0ffd02b8bb11a2c0897b72a2ae"><code>a75d89f</code></a> fix(napi): point from_external slices at the engine-owned copy after finalize...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/e414c8c21924a4c47bf1c8c5d6ad244dac479578"><code>e414c8c</code></a> fix(deps): update dependency obug to v3 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3499">#3499</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/a5fedde2597dc3ac17c7d422f158adce5fec59bf"><code>a5fedde</code></a> chore(deps): update release-plz/action action to v0.5.136 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3505">#3505</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.3...napi-derive-v3.6.5">compare view</a></li> </ul> </details> <br /> Updates `napi-build` from 2.4.1 to 2.4.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/napi-rs/napi-rs/releases">napi-build's releases</a>.</em></p> <blockquote> <h2>napi-build-v2.4.2</h2> <h3>Fixed</h3> <ul> <li><em>(cli,build)</em> make wasm32-wasip1-threads link with wasi-sdk 34 and Rust nightly (<a href="https://redirect.github.com/napi-rs/napi-rs/pull/3492">#3492</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/napi-rs/napi-rs/commit/31c27a1676a7c4b317f4e144e0a9cb94e8354143"><code>31c27a1</code></a> chore: release (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3470">#3470</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/7e3f293e2d6a3032eabfe51ff38bcaa82d342a2f"><code>7e3f293</code></a> chore(release): publish</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/f772ee0aabf4dbb27250a8df47a73463e8f78cf4"><code>f772ee0</code></a> fix(cli): align generated file formats (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3501">#3501</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1cf5ec573637d96dddb77b1a0ae6aaf316739bd0"><code>1cf5ec5</code></a> fix(cli): use accessible WASI preopen root on Android (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3485">#3485</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/cf9245d0b367a8fc2b3c1209269832b9f076de32"><code>cf9245d</code></a> chore(deps): update vitest monorepo to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3481">#3481</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/78fa3ad79d38b52e7c49f9a86ba2e57f46805d1b"><code>78fa3ad</code></a> fix(macro): recognize fully-qualified napi::Env as the special Env parameter ...</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/37283c75a1f1f20d166a8f9f641ada43d04d24a5"><code>37283c7</code></a> chore(deps): lock file maintenance (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3474">#3474</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/1932d2f42198cb9f79704aba66302a0956590c48"><code>1932d2f</code></a> chore(deps): update release-plz/action action to v0.5.135 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3500">#3500</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/0238e8b1b8fcafa2261d4d1b96e7568dd0fb1fc3"><code>0238e8b</code></a> fix(deps): update dependency js-yaml to v5 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3344">#3344</a>)</li> <li><a href="https://github.com/napi-rs/napi-rs/commit/9ed9d3509e1c2cab2cd89dcff04963adaad4cb7d"><code>9ed9d35</code></a> chore(deps): update dependency electron to v44 (<a href="https://redirect.github.com/napi-rs/napi-rs/issues/3468">#3468</a>)</li> <li>Additional commits viewable in <a href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.1...napi-build-v2.4.2">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 
|
21ecafe9ae |
chore: update lance dependency to v13.0.0-beta.1 (#4183)
Update the Rust workspace Lance dependencies and Java lance-core from v12.0.0-beta.18 to [v13.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.1), and refresh Cargo.lock; no compatibility fixes were required. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. Also fixes the flaky Node test `when optimizing a dataset › cleanups old versions` that failed the NPM Publish Linux test jobs on this PR. The test captured `new Date()` (millisecond precision) in the same millisecond as the last commit, while Lance compares version timestamps at nanosecond precision, so that version was not pruned. The test now waits for the clock to tick to the next millisecond before taking the cutoff. This is a pre-existing flake on `main` since #4160, unrelated to the Lance upgrade. --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Jack YeandXuanwo
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 lancedb-gatefixer[bot]andXuanwo
|
ffe94a65a1 |
fix(node): preserve optimize cleanup timestamp (#4160)
<!-- lance-gatekeeper-fix:v1 agent=278cb095b2e5d69051442bc254d803a6 generation=1 --> ## Summary - pass the TypeScript `cleanupOlderThan` date to the native binding as an unchanged epoch timestamp - prune with Lance's absolute `before_timestamp` policy so dispatch and compaction time cannot move the cutoff - retain versions created after the supplied cutoff and document that behavior - add boundary and end-to-end regression coverage ## Root cause The TypeScript layer converted the absolute date into an elapsed duration before calling native optimize. Lance converted that duration back into a timestamp only after compaction, which silently advanced the requested cutoff and made the cleanup count depend on a millisecond timing boundary. ## Validation - `cargo fmt --all` - `cargo clippy --quiet --features remote --tests --examples -p lancedb -p lancedb-nodejs` - `pnpm build` - `pnpm lint` - `pnpm run docs` - `pnpm test __test__/table.test.ts --runInBand` (309 passed) Fixes #4159 --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jack Ye
|
37771fd4fc |
fix(deps): update rustls and cap aws-smithy-types to unbreak CI (#4177)
Two upstream dependency releases broke CI on `main`. Both fixes are dependency constraints, so they ride together. ## `deny` — RUSTSEC-2026-0285 rustls 0.23.40 accepts TLS 1.3 handshake messages sent at the wrong encryption level ([advisory](https://rustsec.org/advisories/RUSTSEC-2026-0285)), patched in 0.23.45. rustls 0.23.45 requires `aws-lc-rs >= 1.18`, which the nodejs crate pinned to `=1.16.3`, so this also bumps that pin and its `aws-lc-sys` companion to `=1.18.1` / `=0.45.0`. The pin comment already calls for periodic updates on security patches. The workspace's other rustls (0.21.12) is below the advisory's affected range (`unaffected = ["< 0.23.13"]`). ## `build-no-lock` — aws-smithy-types 1.7.0 `aws-smithy-types` 1.7.0 and `aws-smithy-json` 0.64.0 both released 2026-09-14. 1.7.0 made `Document` `non_exhaustive`, which `aws-smithy-json` 0.63 does not compile against: ``` error[E0004]: non-exhaustive patterns: `&_` not covered --> aws-smithy-json-0.63.0/src/serialize.rs:36:15 note: `aws_smithy_types::Document` defined here --> aws-smithy-types-1.7.0/src/document/mod.rs:91:1 ``` Every `aws-sdk-*` crate moved to `aws-smithy-json ^0.64`, but `aws-config` 1.12.0 still requires `^0.63`, so a lockfile-free resolve pairs json 0.63.0 with types 1.7.0 and fails. This caps `aws-smithy-types` below 1.7 as a constraint-only dev-dependency, matching the existing `aws-smithy-runtime` entry. Revert once `aws-config` moves to `aws-smithy-json` 0.64. Note this break is not specific to this PR — `build-no-lock` fails the same way on unrelated branches (e.g. `jon/secrets-client-api` run 34903587364), which passed it hours earlier. ## Verification Resolution only, no local build: - Locked resolve unchanged: `aws-smithy-types` stays 1.4.8; the only `Cargo.lock` delta from the cap is the new dev-dep edge. - Fresh resolve (`rm Cargo.lock`): `aws-smithy-json` 0.63.0 with `aws-smithy-types` 1.6.3, `aws-sdk-*` one release back, `rustls` 0.23.45 retained. |
||
 Lance Release
|
c44b192334 | Bump version: 0.39.0-beta.7 → 0.39.0-beta.8 | ||
|
Wyatt Alt
|
7575c2597a |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
|
Wyatt Alt
|
0665575a76 |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
 陈志谦andXuanwo
|
0113cee489 |
docs(embeddings): correct the documented max_retries default (#4145)
`lancedb/embeddings/utils.py` documents `max_retries` with "(default is 10)" — the signature default is `7`. Docs-only; conventional title per the contribution guide. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
6bb64c3edb |
fix: reject repeated job pagination tokens (#4139)
Fixes #4138 `RemoteDatabase::list_jobs` followed every returned pagination token without remembering previously seen values. A server-side token cycle therefore caused repeated requests and duplicate accumulation until the 100-page safeguard returned partial results as a success. This change tracks non-empty job-list page tokens and returns an HTTP-context error as soon as a token repeats, matching the existing `list_functions` behavior. A mock-handler regression test verifies a repeated `loop` token is rejected after two requests. Validation: - `cargo test --quiet --features remote -p lancedb test_list_jobs` - `cargo fmt --all` - `cargo check --quiet --features remote --tests --examples` <!-- lance-gatekeeper-fix:v1 agent=ddd56389737a405d32cf4f43c7696032 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
255da8a854 |
fix(python): resolve native job types in API docs (#4170)
Native job metadata types report `builtins` as their module, so Griffe cannot resolve the public `lancedb.job` re-exports and the Python API reference build fails. Set the PyO3 module metadata for `JobInfo`, `JobDescription`, and `JobFailureInfo`, and cover import resolution in the existing package metadata tests. Reproduced the failure and validated the fix with the docs CI toolchain (`griffe==0.49.0`, `mkdocstrings==0.25.2`, and `mkdocstrings-python==1.10.9`). After rebuilding the native extension, the full `PYTHONPATH=. mkdocs build` succeeds and all three classes and their public members appear in the generated reference. |
||
|
Lance Release
|
ec410e015a | Bump version: 0.39.0-beta.6 → 0.39.0-beta.7 | ||
|
Jack Ye
|
9fe10c7362 |
fix(remote): align Function CRUD routes (#4166)
Align the experimental Function HTTP transport with the equivalent Table
CRUD API shape. This is an intentional breaking change to the
experimental Function routes; public Rust and Python APIs remain
unchanged.
## Route comparison
| Operation | Function before | Function after | Equivalent Table API |
| --- | --- | --- | --- |
| Create | `POST /v1/functions/create` | `POST /v1/function/{id}/create`
| `POST /v1/table/{id}/create` |
| Describe | `POST /v1/functions/describe` | `POST
/v1/function/{id}/describe` | `POST /v1/table/{id}/describe` |
| List | `POST /v1/functions/list` | `GET
/v1/namespace/{id}/function/list` | `GET /v1/namespace/{id}/table/list`
|
| Drop | `POST /v1/functions/drop` | `POST /v1/function/{id}/drop` |
`POST /v1/table/{id}/drop` |
## Contract details
- Create, describe, and drop use a singular resource path. Their `{id}`
path parameter is the URL-encoded Function name, and the duplicate
Function identifier is removed from each request body.
- Create continues to accept `202 Accepted`.
- List changes from a POST with a JSON body to a namespace-scoped GET.
Its `{id}` path parameter is the namespace identifier rather than a
Function name.
- Functions do not support nested namespaces yet, so the client lists
against the root namespace identifier (`$` with the default delimiter).
A non-root namespace is rejected.
- The optional list filter is named `name`. `limit`, `page_token`, and
`include_definition` remain available as query parameters.
- The paginated list response shape is unchanged.
|
||
|
LanceDB Robot
|
b8f0048b5a |
chore: update lance dependency to v12.0.0-beta.18 (#4164)
Update the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core to [v12.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.18). Fix redundant visibility declarations in the Node.js Rust blob helpers required by the workspace Clippy check. Validated with workspace Clippy (all features and tests, warnings denied), cargo fmt, pnpm build, and 13 targeted Node.js blob tests. |
||
 Drew
|
6702e3fec1 |
feat(node): add blob v2 fetch and field helpers (#4155)
this PR blob v2 field helpers and reads to the Node SDK.
`blob()` marks a field as blob v2 and lets you set the storage
thresholds. Inputs can be bytes, a URI, or a data/uri struct.
Queries return descriptors. `fetchBlobs()` reads the bytes by row ID,
and `fetchBlobFiles()` gives you lazy handles for full or range reads.
`blobColumns()` lists the blob fields, including nested ones.
Fetch uses the table’s current checkout. It preserves order, duplicates,
and nulls. Holding row IDs across compaction still requires stable row
IDs.
```javascript
const db = await connect("./data");
const video = await readFile("clip.mp4");
const table = await db.createTable(
"videos",
[{ id: 1n, video }],
{
schema: new Schema([
new Field("id", new Int64()),
blob("video"),
]),
},
);
const rows = await table.query().select(["id"]).withRowId().toArray();
const rowIds = rows.map((row) => row._rowid as bigint);
const bytes = await table.fetchBlobs("video", rowIds);
const [handle] = await table.fetchBlobFiles("video", rowIds);
const header = await handle!.readRange(0n, 65536n);
```
### Testing
- cover input validation, thresholds, nested fields, fetch ordering,
nulls, and range reads.
|
||
|
LanceDB Robot
|
e0bd4b5fa1 |
chore: update lance dependency to v12.0.0-beta.17 (#4162)
Update the Rust workspace Lance dependencies and Java lance-core to [v12.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.17). Align object_store to 0.14.1 for compatibility with Lance and refresh the Cargo lockfile, including the required reqsign updates. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings` and `cargo fmt --all --quiet`. |
||
|
dependabot[bot]
|
13f9dd630b |
build(deps): bump prost from 0.14.3 to 0.14.4 in the rust-minor-patch group (#4135)
Bumps the rust-minor-patch group with 1 update: [prost](https://github.com/tokio-rs/prost). Updates `prost` from 0.14.3 to 0.14.4 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md">prost's changelog</a>.</em></p> <blockquote> <h1>Prost version 0.14.4</h1> <p><em>PROST!</em> is a <a href="https://protobuf.dev/">Protocol Buffers</a> implementation for the <a href="https://www.rust-lang.org/">Rust Language</a>. <code>prost</code> generates simple, idiomatic Rust code from <code>proto2</code> and <code>proto3</code> files.</p> <h3>🚀 Features</h3> <ul> <li><em>(prost-derive)</em> Make is_valid a constant function (<a href="https://redirect.github.com/tokio-rs/prost/issues/1401">#1401</a>)</li> <li>Increase MSRV to 1.85 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1428">#1428</a>)</li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>Use Display instead of Debug for generated enumeration attributes (<a href="https://redirect.github.com/tokio-rs/prost/issues/1419">#1419</a>)</li> <li><em>(prost-derive)</em> Return error for invalid enumeration default identifiers (<a href="https://redirect.github.com/tokio-rs/prost/issues/1426">#1426</a>)</li> <li><em>(build)</em> Grab binary path from cargo (<a href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li> <li><em>(build)</em> Fix C++ build on GCC 15 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1395">#1395</a>)</li> </ul> <h3>📚 Documentation</h3> <ul> <li>Add example for <code>decode_length_delimiter</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1311">#1311</a>)</li> <li>Update protobuf-src example to avoid unsafe set_var</li> </ul> <h3>🧪 Testing</h3> <ul> <li>Test derive Eq behavior (<a href="https://redirect.github.com/tokio-rs/prost/issues/1422">#1422</a>)</li> <li><em>(groups)</em> Actually construct <code>NestedGroup</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li> </ul> <h3>💼 Dependencies</h3> <ul> <li><em>(deps)</em> Update criterion requirement from 0.7 to 0.8 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1374">#1374</a>)</li> <li><em>(deps)</em> Remove <code>getrandom@0.4.1</code> from build-dependencies (<a href="https://redirect.github.com/tokio-rs/prost/issues/1400">#1400</a>)</li> <li><em>(deps)</em> Update rand requirement from 0.9 to 0.10 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1397">#1397</a>)</li> <li><em>(deps)</em> Bump actions/upload-artifact from 6 to 7 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li> <li><em>(deps)</em> Update <code>cargo clippy</code> to 1.89 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li> <li><em>(deps)</em> Update <code>cargo clippy</code> to 1.91 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li> <li><em>(deps)</em> Update and improve nix devshell (<a href="https://redirect.github.com/tokio-rs/prost/issues/1393">#1393</a>)</li> </ul> <h3>🎨 Styling</h3> <ul> <li>Prevent needless borrow (<a href="https://redirect.github.com/tokio-rs/prost/issues/1404">#1404</a>)</li> <li>Use <code>std::hint::black_box()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1403">#1403</a>)</li> <li>Use variables directly in <code>format!()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li> <li>Remove explicit <code>.into_iter()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li> <li>Run clippy on benches (<a href="https://redirect.github.com/tokio-rs/prost/issues/1405">#1405</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tokio-rs/prost/commit/13646cde7eab75c81b3047767aa0a86e7dbecf12"><code>13646cd</code></a> chore: Release version 0.14.4 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1437">#1437</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/dad79d5c8e3549d93ebe6f6c723bb42928d805d8"><code>dad79d5</code></a> fix(prost-derive): return error for invalid enumeration default identifiers (...</li> <li><a href="https://github.com/tokio-rs/prost/commit/b0b6c93e3aac89df28690a4967a8bbe93ec95391"><code>b0b6c93</code></a> ci: Update <code>cargo clippy</code> to 1.91 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/32cfffbc494f2faf461cab85e04a42412484c0e4"><code>32cfffb</code></a> style: remove explicit <code>.into_iter()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/2710efdb9978d9c75fb19b0b092a369a2d385b55"><code>2710efd</code></a> ci: Update <code>cargo clippy</code> to 1.89 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/18ea4e42bbc307d33d65e05ad47b3c45623c0500"><code>18ea4e4</code></a> style: use variables directly in <code>format!()</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/2821bd1d8c20137c83ead4db39f8e1da00b4e854"><code>2821bd1</code></a> build(deps): bump actions/upload-artifact from 6 to 7 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/3ce3b39f9206b5e3bbe34c6e1aa69fe3c53f0924"><code>3ce3b39</code></a> test(groups): Actually construct <code>NestedGroup</code> (<a href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/8776405574b3ba0a0fe96ada8799ac8bc61ceb3e"><code>8776405</code></a> docs: Update changelog for version 0.14.3 (<a href="https://redirect.github.com/tokio-rs/prost/issues/1431">#1431</a>)</li> <li><a href="https://github.com/tokio-rs/prost/commit/33d3ef18c008da13e862d7e7674d751ab2776360"><code>33d3ef1</code></a> build: Grab binary path from cargo (<a href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li> <li>Additional commits viewable in <a href="https://github.com/tokio-rs/prost/compare/v0.14.3...v0.14.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
LanceDB Robot
|
bc4497b21a |
chore: update lance dependency to v12.0.0-beta.16 (#4156)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v12.0.0-beta.15 to [v12.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.16). No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. |
||
 Will JonesandClaude Sonnet 5
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Madan Kumar
|
577fb48376 |
fix(python): apply offset when combining async hybrid results (#4028)
Fixes #4027 ## Summary `AsyncHybridQuery` (`table.query().nearest_to(...).nearest_to_text(...)`) paginates incorrectly when `.offset()` is used: the second page repeats rows from the first page and silently drops others. `offset()` on a hybrid query pushes the offset down into *both* sub-queries (`HybridQuery::offset` in `python/src/query.rs` forwards to `inner_vec` and `inner_fts`), so each sub-query independently skips its own first `offset` rows before the results are fused. `AsyncHybridQuery.to_batches` then called `_combine_hybrid_results(..., limit=self._inner.get_limit())` without an `offset`, so the reranked table was sliced starting at position 0 and the sub-query limits were never raised to cover the skipped prefix. On the 4-row fixture in `test_hybrid_query.py`, with `_rowid` ordering `[3, 0, 2, 1]`: | query | before | after | | --- | --- | --- | | `.limit(2)` | `[0, 3]` | `[0, 3]` | | `.offset(2).limit(2)` | `[3, 1]` | `[2, 1]` | Row `3` was returned on both pages and row `2` was never returned at all. This is the async counterpart of #3769 (`Fixes #3765`), which fixed the same bug in the synchronous `LanceHybridQueryBuilder`. #3765 explicitly deferred the async path; this PR closes that gap and reuses the `offset` parameter that #3769 already added to `_combine_hybrid_results`. The synchronous path is unaffected — it was fixed in #3769. ## Changes `python/python/lancedb/query.py`, `AsyncHybridQuery.to_batches`: - Each sub-query now fetches `limit + offset` rows and its own offset is reset to 0, so the fused result contains the full prefix the window is sliced out of. - The combined, reranked table is sliced with `offset=` instead of always starting at 0. Both halves are needed: raising the sub-query limits without the final slice still returns page 1, and slicing without raising the limits still misses rows. `nodejs` has no equivalent hybrid combine path, so there is no SDK parity gap here. ## Test plan - [x] New regression test `test_async_hybrid_query_offset` in `python/python/tests/test_hybrid_query.py`, mirroring the sync `test_hybrid_query_offset`. It asserts the offset window is a suffix of the un-offset result *and* that page 1 + page 2 together cover every row exactly once (a row-count-only assertion would pass even with duplicates). - [x] `pytest python/tests/test_hybrid_query.py` — 16 passed - [x] `pytest python/tests/test_rerankers.py` — 9 passed, 11 skipped - [x] `pytest python/tests/test_query.py` — 86 passed - [x] `pytest --doctest-modules python/lancedb/query.py` — 13 passed - [x] `ruff format --check` / `ruff check` — clean --- ## Scope, after review @lancedb-gatekeeper raised three points. Two were mine and are fixed in `04d07c2`; the third is deliberately left alone and I'd like a maintainer's call on it. **Fixed — effective limit was read from the FTS child only.** `HybridQuery::get_limit()` (`python/src/query.rs:1159`) returns `self.inner_fts.inner.current_request().limit`, so an FTS-first hybrid with no explicit `.limit()` yielded `None`, skipped the widening branch and passed `limit=None` to the combiner — returning the union of both candidate lists instead of the documented default of 10. The limit is now derived from both children with a `DEFAULT_HYBRID_LIMIT = 10` fallback, so construction order no longer matters. **Fixed — `explain_plan()` / `analyze_plan()` described a different query than the one that ran.** Both built their children straight from `self._inner`, bypassing the limit/offset rewrite in `to_batches`, and reported `skip=2, fetch=2` while execution used `skip=0, fetch=4`. Child preparation now lives in one `_create_child_queries()` helper used by all three. > **Visible change to `explain_plan()` output:** because the plan is now built from the real execution children, which carry `with_row_id()`, the two `ProjectionExec` lines gain a `_rowid` column. The doctest is updated to match. This is the diagnostic becoming truthful rather than the assertion being weakened — it is still an exact-match comparison. **Not fixed here — RRF candidate-pool invariance.** Widening each sub-query to `limit + offset` does change the candidate pool between page requests, so the fused ranking can shift and pagination can still repeat rows. That's a real problem, but it is exactly what the merged sync path does today: ```python # LanceHybridQueryBuilder (sync), merged in #3769 sub_query_limit = self._limit + (self._offset or 0) ``` Making the pool invariant means choosing a contract — a fixed candidate pool, or an explicit cursor — and that ought to apply to sync and async together rather than letting the two paths diverge. I've asked in the review thread which way you'd prefer, and I'm happy to do it here or in a follow-up covering both paths. So, to be precise about what this PR delivers: it makes `.offset()` take effect on the async hybrid path and makes the diagnostics honest. It does not make hybrid pagination stable across pages under reranking — that needs the contract decision above. |
||
|
陈志谦
|
c7b051aff7 |
docs: fix spelling typos across python package docstrings (#4146)
Six files carried spelling typos in user-visible docstrings: - `table.py` (×3) + `remote/table.py`: "The **targetted** vector to search for" → "targeted" - `query.py`: "pa.Array **wouln't** be allowed" → "wouldn't" - `embeddings/gte.py`: "mlx package **insalled**" → "installed" - `rerankers/base.py`: "This is **inteded**" → "intended" - `index.py`: "dimension **divded** by 8" → "divided" Docstrings only. |
||
|
Lance Release
|
2e205ac9bb | Bump version: 0.39.0-beta.5 → 0.39.0-beta.6 | ||
|
LanceDB RobotandJack Ye
|
3e3878b223 |
chore: update lance dependency to v12.0.0-beta.15 (#4143)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v12.0.0-beta.14 to [v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15). No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
Lance Release
|
19fb665c76 | Bump version: 0.39.0-beta.4 → 0.39.0-beta.5 | ||
|
Wyatt AltandClaude Fable 5.1
|
1f95398c34 |
feat: function columns on materialized views (#4119)
A view could not carry a column it does not compute: the definition planned every output as a SQL expression, and the refresh engine treated any commit it did not make as drift and rebuilt. Servers fill such columns on tables with a separate job, as computed columns bound to a registered function, and want the same column on a view. This lets a declaration add computed columns, placed at their positions in the select list and validated by the existing computed-column contract, with the view created in one commit. Refresh writes those columns NULL on every path and never reads them, so a rewritten row comes back unfilled, and a commit that rewrites only computed columns is recognised as a fill rather than drift, so the next refresh carries on incrementally. A source column a computed column reads without the view projecting it is held as an internal projection, so the select list stays the view's column list. Nothing in the stored definition changes; an older reader fails closed on the schema check. Two smaller changes ride along because the feature needs them: an identity projection keeps its source column's nullability, with the schema check accepting a nullable physical field for a non-null planned one so existing views keep refreshing; and `prepare_declaration` takes `Option` projections, so an empty list declares no projection rather than `SELECT *`. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Lance Release
|
0111a72dc3 | Bump version: 0.39.0-beta.3 → 0.39.0-beta.4 |