mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-21 18:13:57 +00:00
feat(python): bundle Function package resources (#4987)
A package shipped through `code=` currently loses its non-Python files.
This breaks vendored packages that read JSON configuration, manifests,
prompts, or binary resources after deployment. Include regular package
resources in a separate base64-encoded `python_bundle.resources` map,
preserving exact bytes and relative paths. Both source and resources
contribute to the artifact digest. UTF-8 Python source bytes retain
their original line endings as well, so manifests that hash vendor code
continue to match.
For example, `@function(code=[my_package])` can use
`importlib.resources.files("my_package").joinpath("models.json").read_text()`
on the worker. Hidden files and caches are excluded; symlinks and unsafe
paths warn instead of following files outside the package. Source-only
bundles retain their existing wire form. Resource bundles require
matching server support.
Adds an isolated-import regression with a vendored subpackage, JSON
metadata, binary data, SHA256 verification, preserved CRLF bytes, and
resource-only version changes. Local builds and tests are disabled by
request. Ruff formatting and linting run locally; CI runs the tests.
This commit is contained in:
@@ -1420,3 +1420,15 @@ name; omitting it allows matching reads of any revision. Prefixes must be concre
|
||||
paths without wildcards or traversal. This integration does not make Hugging Face
|
||||
a writable database location for `ATTACH DATABASE`. Dataset SQL sources select a
|
||||
matching integration automatically, with no integration argument in the query.
|
||||
|
||||
|
||||
### Function package resources
|
||||
|
||||
`@function(code=[my_package])` includes the package’s Python modules and data
|
||||
files, including vendored subpackages. Resource paths and bytes are preserved,
|
||||
so `importlib.resources` and package integrity checks work after deployment.
|
||||
Changing a resource changes the Function artifact digest. This requires a server
|
||||
that accepts `python_bundle.resources`.
|
||||
|
||||
Caches, hidden paths, symlinks and unsafe paths are excluded. Use `pip=` or
|
||||
`conda=` for platform-specific dependencies and `models=` for large model snapshots.
|
||||
|
||||
@@ -499,13 +499,20 @@ def caption(jpeg: bytes) -> str:
|
||||
</CodeGroup>
|
||||
|
||||
Each module in `code` is captured and shipped the same way the function
|
||||
itself is: as source, snapshotted at registration. Only `.py` source files are
|
||||
included; package data such as JSON, YAML, prompt, or tokenizer files is not
|
||||
copied. `@function` warns before registration when it skips such files from a
|
||||
package in `code`, naming the affected paths. To use a package that includes
|
||||
the required resources, install it with `pip=` or `conda=` and remove that
|
||||
package from `code`; otherwise the Python-only copied package shadows the
|
||||
installed distribution.
|
||||
itself is: as source, snapshotted at registration. Packages also include regular
|
||||
data files such as JSON, YAML, prompts, and tokenizer files, including resources
|
||||
inside vendored subpackages. Resource paths and bytes are preserved for
|
||||
`importlib.resources` and package integrity checks. Changing a resource changes
|
||||
the Function artifact's identity. Caches, hidden paths, symlinks, and unsafe
|
||||
paths are excluded.
|
||||
|
||||
Package resources require matching SDK and server support. Upgrade query nodes
|
||||
and Function image-building workers before registering resource bundles; older
|
||||
servers reject them. Older SDKs ship only Python source and warn when excluding
|
||||
package data. On deployments without resource-bundle support, install the
|
||||
package with `pip=` or `conda=` and omit it from `code` so the copied package does
|
||||
not shadow the installed distribution. Use environment dependencies for
|
||||
platform-specific packages and `models=` for large model snapshots.
|
||||
|
||||
## Drop it {#drop}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user