mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
5ad2de91a2
* feat(pipelines): ingestion (EL) templates + docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review nits — draft collision guard, template-mode selection reset, invariant test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lead the insert menu with ingestion templates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipelines): ingestion story as docs-only — drop editor template UI The insert-menu template section mixed two selection grammars in one popover and confused more than it helped. The three E2E-verified example pipelines now live verbatim in docs/pipeline-ingestion.md; the Python bare-string S3 key fix in pipelineTemplates.ts stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sdk): bare string S3 keys in py/ts clients + asset parsers A plain string passed where an S3Object is expected is now a bare key in the default storage — previously the py client silently degraded it to s3="" (auto-generated key) and both asset parsers canonicalized it without the leading slash, splitting lineage. parseS3Object moves to s3Types.ts so it is unit-testable without the generated services. The pipeline template fix from the earlier commit is superseded (bare strings are the supported spelling again); docs examples flipped to bare keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(sdk): enforce s3:// URIs for string S3Object params Bare strings now raise/throw with a hint pointing at the s3:///<key> spelling instead of being treated as keys (previous commit) or silently degrading to an empty key (original behavior). One string spelling everywhere: SDK calls, // on annotations, and DuckDB SQL all use s3:///<key>. TS regains the s3://-template-literal type; the asset parsers record no asset for a bare string (the call can only error); templates emit the URI form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(pipelines): move ingestion (EL) guide to windmilldocs, keep design constraints User-facing how-to (engine choice, cursor recipes, schema drift, worked examples) moves to windmilldocs core_concepts/63_pipelines (windmilldocs#1462); the repo keeps only the design constraints future feature work must not break, as a section of ducklake-materialization.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: regenerate system prompts after parse_s3_object docstring change Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): reject empty-key s3 URIs; align asset parsers with the runtime rule Addresses CI review: s3:/// and s3://bucket/ now raise (an empty key would fall back to the auto-generated-key path the strict contract exists to prevent); the asset parsers' string branch applies the same valid-URI-with-non-empty-key rule so no R/W edge is recorded for a call that can only error (the generic URI-literal scan still records ambiguous access-None assets, by design); comments rephrased as current constraints per AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
66 lines
2.1 KiB
TypeScript
66 lines
2.1 KiB
TypeScript
/**
|
|
* S3 object representation, either as a URI string or a record object
|
|
*/
|
|
export type S3Object = S3ObjectURI | S3ObjectRecord;
|
|
|
|
/**
|
|
* S3 object URI in the format `s3://storage/key` (`s3:///key` targets the
|
|
* workspace default storage)
|
|
*/
|
|
export type S3ObjectURI = `s3://${string}/${string}`;
|
|
|
|
/**
|
|
* S3 object record with file key, optional storage identifier, and optional presigned token
|
|
*/
|
|
export type S3ObjectRecord = {
|
|
/** File key/path in S3 bucket */
|
|
s3: string;
|
|
/** Storage backend identifier */
|
|
storage?: string;
|
|
/** Presigned URL query string for public access */
|
|
presigned?: string;
|
|
};
|
|
|
|
/**
|
|
* S3 client configuration settings for Deno S3 light client
|
|
*/
|
|
export type DenoS3LightClientSettings = {
|
|
/** S3 endpoint URL */
|
|
endPoint: string;
|
|
/** AWS region */
|
|
region: string;
|
|
/** Bucket name */
|
|
bucket?: string;
|
|
/** Use HTTPS connection */
|
|
useSSL?: boolean;
|
|
/** AWS access key */
|
|
accessKey?: string;
|
|
/** AWS secret key */
|
|
secretKey?: string;
|
|
/** Use path-style URLs instead of virtual-hosted style */
|
|
pathStyle?: boolean;
|
|
};
|
|
|
|
/**
|
|
* Parse an S3 object from URI string or record format
|
|
* @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key`
|
|
* for the default storage) or record. Any other string throws rather than
|
|
* falling back to an auto-generated key: an auto key is requested by
|
|
* omitting the object, and a fallback would silently misplace the upload
|
|
* on any typo.
|
|
* @returns S3 object record with storage and s3 key
|
|
*/
|
|
export function parseS3Object(s3Object: S3Object): S3ObjectRecord {
|
|
if (typeof s3Object === "object") return s3Object;
|
|
const match = s3Object.match(/^s3:\/\/([^/]*)\/(.+)$/);
|
|
if (match) return { storage: match[1] || undefined, s3: match[2] };
|
|
if (s3Object.startsWith("s3://")) {
|
|
throw new Error(
|
|
`Invalid s3 object URI ${JSON.stringify(s3Object)}: expected s3://<storage>/<key> with a non-empty key (s3:///<key> for the default storage)`
|
|
);
|
|
}
|
|
throw new Error(
|
|
`Invalid s3 object ${JSON.stringify(s3Object)}: expected an s3://<storage>/<key> URI (e.g. "s3:///${s3Object}" for key "${s3Object}" in the default storage) or { s3: <key> }`
|
|
);
|
|
}
|