mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
feat(apps): authorize deployed-app S3 reads on-behalf of the author for logged-in viewers (#10048)
* feat(apps): authorize deployed-app S3 reads on-behalf of the author for logged-in viewers A logged-in user viewing a deployed app now reads S3 files (rich result, table/image/PDF preview, CSV export, download, metadata) the same way an anonymous viewer already does: on-behalf of the app author per the app policy's execution_mode, gated by an app-provenance check — instead of against the viewer's own S3 permissions. This aligns S3 with every other thing an app does (scripts, flows, resources all already run on-behalf of the author) and lets an operator who lacks folder S3 permission still see data rendered inside the app. The raw job_helpers/* S3 API stays viewer-scoped: a viewer who lacks folder permission is still denied there. Only which endpoint the app frontend uses for logged-in deployed viewers changes. Backend: - Add app-scoped, provenance-gated apps_u/* variants for all S3 display ops (download_s3_file already existed; add download_s3_parquet_file_as_csv, load_file_metadata, load_file_preview, load_parquet_preview, load_csv_preview, load_table_count). Each routes through one shared helper (app_s3_on_behalf_and_provenance) that scope-confines an app embed token, resolves the on-behalf identity, and runs the provenance gate ONCE before dispatching to the EE *_internal S3 helpers. - Close the confused-deputy hole in check_if_allowed_to_access_s3_file_from_app: the unconditional Ok() bypass for a logged-in, non-embed session now only applies in viewer execution mode (where the on-behalf identity IS the viewer, so the viewer's own permissions still bound the read downstream). Author-mode reads (anonymous/publisher) always enforce provenance, for anonymous and logged-in viewers alike, so a viewer cannot launder the author's S3 permissions with an arbitrary file_key. Frontend: - Route the deployed-app view through apps_u/* using the app-viewer isEditor signal instead of login state (the old $userStore proxy wrongly sent logged-in deployed viewers to the viewer-scoped job_helpers API). Editor and preview keep viewer identity via job_helpers. execution_mode: viewer remains the escape hatch for per-viewer S3 enforcement. Fixes provenance-gated S3 display for logged-in operators on deployed apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(backend): document cargo features, restarting the dev backend, and filesystem object storage The dev backend runs `cargo watch --features quickjs` by default, which omits S3, EE, MCP, and non-JS runtimes — feature-gated routes then 404 or return a "requires <feature>" stub at runtime. Add a backend/CLAUDE.md section that: - explains that you must restart the backend with the appropriate features to exercise gated functionality, with the pid/cwd-scoped restart recipe (never pkill target/debug/windmill) and the PORT=$BACKEND_PORT gotcha; - documents what each commonly-toggled feature gate does (private, enterprise, license, parquet, duckdb, language runtimes, mcp, trigger kinds, no_auth) plus common combinations; - documents using the built-in FilesystemStorage large-file storage for dev workspace object storage (hidden from the UI dropdown; set via edit_large_file_storage_config), including the advanced_permissions shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): don't flatten inner query in app-scoped S3 preview routes axum's `Query` uses `serde_urlencoded`, which cannot deserialize the typed (numeric/bool) fields of a `#[serde(flatten)]`-ed struct and 400s on `limit` / `offset` ("invalid type: string, expected u32"). The app-scoped load_csv_preview / load_parquet_preview / load_table_count routes flattened LoadPreviewQuery / LoadCountQuery, so their previews were broken. Restate the fields directly on the outer query structs (with an into_inner() to rebuild the inner query) and extend the CE OSS stub to match. Also bumps ee-repo-ref.txt for the companion EE csv-separator panic fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address CI review — nested DisplayResult routing, byte-range contract, docs, tests - [P1] Thread `appPath` into the nested `DisplayResult`s (render_all children and the expanded-result drawer) so logged-in deployed viewers route nested/expanded S3 tables, images, PDFs, and downloads through `apps_u/*` too, not job_helpers. - [P2] Mark `read_bytes_from`/`read_bytes_length` required on the `apps_u/load_file_preview` route (they are non-optional in LoadFilePreviewQuery), and mirror the full query shape in the CE OSS stub so the byte-range contract is enforced identically on CE and EE. - [P2] Fix the backend retrigger command in backend/CLAUDE.md: cargo watch runs from `backend/`, so `touch README.md` (not `backend/README.md`). - [P2] Trim app_s3_onbehalf.rs comments per AGENTS.md (state the invariant once, no drafting-history narration). - Extend the integration test to cover the table-count, csv-preview (numeric limit/offset deserialization), and file-preview (byte-range required) routes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(apps): tighten S3 provenance-gate comments per AGENTS.md Consolidate the viewer-mode / author-mode rationale to ≤4 lines at each branch of the gate, and drop the repeated explanation from the shared app_s3_on_behalf_and_provenance doc comment (which now just states what the helper does). No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to f292a1040da6a667ce7c22abf63ec0debfdd480f This commit updates the EE repository reference after PR #657 was merged in windmill-ee-private. Previous ee-repo-ref: a582389084eb363997cb5e8053f29220e0d3eaec New ee-repo-ref: f292a1040da6a667ce7c22abf63ec0debfdd480f Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
@@ -18,6 +18,86 @@
|
||||
- **Running data pipelines (DuckLake) from source**: see the section below — a plain build
|
||||
advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy.
|
||||
|
||||
## Cargo features & running the dev backend
|
||||
|
||||
The dev backend runs under `cargo watch` and is launched by default with **only
|
||||
`--features quickjs`** (see the tmux backend pane). That baseline compiles fast but
|
||||
**deliberately omits most functionality** — notably S3/object storage, the S3 proxy, all
|
||||
EE code, MCP, and every non-JS language runtime. A running server never gains a feature you
|
||||
didn't compile in: feature-gated routes 404 or return a `"requires <feature>"` stub. So if
|
||||
you touch code behind a feature gate, or need to *exercise* such a feature at runtime, you
|
||||
MUST **restart the backend with the appropriate features** for what you're working on.
|
||||
|
||||
### Restarting the dev backend with the right features
|
||||
|
||||
The backend runs in tmux pane 1 as `cargo watch -x "run --features <…>"`. To restart it with a
|
||||
different feature set — scope kills by pid/cwd, **never** `pkill -f target/debug/windmill` (it
|
||||
kills every sibling worktree's backend):
|
||||
|
||||
1. Stop the current run: `tmux send-keys -t <pane1> C-c`, then kill *this worktree's*
|
||||
`cargo-watch` pid (find it via `/proc/<pid>/cwd`).
|
||||
2. Relaunch in the same pane so it inherits the shell's `DATABASE_URL` etc.; the pane env's
|
||||
`PORT` may be stale, so set it explicitly:
|
||||
```bash
|
||||
export PORT=$BACKEND_PORT
|
||||
cargo watch -x "run --features enterprise,private,parquet,quickjs"
|
||||
```
|
||||
3. Wait for `health check completed` in the pane before hitting the API.
|
||||
|
||||
cargo-watch only re-runs on a file change, so after an idle/failed run `touch README.md` (from
|
||||
`backend/`, where the watch runs) is a cheap retrigger (touching a `.rs` forces a full rebuild).
|
||||
|
||||
### What each feature gate does (the ones you'll actually toggle)
|
||||
|
||||
`backend/Cargo.toml` `[features]` is the source of truth; this is the practical dev map. Combine
|
||||
only what you need — build time scales with the set.
|
||||
|
||||
| Feature | Enables | Need it for |
|
||||
|---|---|---|
|
||||
| `quickjs` | Embedded JS engine for inline JS eval (the default dev baseline). | Keep in every dev set. |
|
||||
| `private` | Compiles the `*_ee.rs` files (symlinked from `windmill-ee-private`). Gates **all** EE code, including the real S3 helpers, the S3 proxy, and advanced S3 permission checks. | Any EE code path, S3/object storage. |
|
||||
| `enterprise` | EE business logic (autoscaling, SAML hooks, advanced S3 rule **enforcement**, WAP, forks, …). Pulls in `license`. | Running EE features. Advanced S3 permission rules only take effect with this. |
|
||||
| `license` | License-key/plan plumbing (`LICENSE_KEY`). Pulled in by `enterprise`. Having the feature compiled does **not** require a license *key* at runtime — CE defaults to a free plan and most EE paths still run keyless. | License-gated behavior. |
|
||||
| `parquet` | S3/object-storage support: the `job_helpers/*` and `apps_u/*` S3 endpoints, parquet/CSV preview, workspace large-file storage. Without it those routes return `"requires parquet"`. | Anything touching S3/object storage or datasets. |
|
||||
| `duckdb` | DuckDB script executor (also needs the FFI dylib — see above). | DuckDB scripts, DuckLake. |
|
||||
| `python` `rust` `php` `java` `ruby` `csharp` `nu` `deno_core` `mysql` `mssql` `bigquery` `oracledb` `rlang` | Each enables that language/DB runtime for job execution. | Running jobs in that language. |
|
||||
| `mcp` | MCP gateway routes (baseline `quickjs` does NOT include it → MCP routes 404). | MCP work. |
|
||||
| `websocket` `http_trigger` `kafka` `nats` `mqtt_trigger` `sqs_trigger` `gcp_trigger` `azure_trigger` `postgres_trigger` `native_trigger` | Each native trigger kind; none on by default (creating one 404s without its feature). | Working on / exercising that trigger. |
|
||||
| `no_auth` | Treats every request as an admin superadmin (`CLOUD_HOSTED`-guarded). | Local auth-free experiments only. |
|
||||
|
||||
Convenience bundles (`ce`, `ee`, `oss`, …) exist in `[features]` but are heavy — prefer the
|
||||
minimal explicit set for dev.
|
||||
|
||||
**Common combinations** (run from `backend/`):
|
||||
|
||||
| Goal | `--features` |
|
||||
|---|---|
|
||||
| Plain dev baseline (JS eval only) | `quickjs` |
|
||||
| S3 / object storage / datasets (CE) | `quickjs,private,parquet` |
|
||||
| S3 + EE (advanced S3 rules, on-behalf app reads, WAP, forks) | `quickjs,enterprise,private,parquet` |
|
||||
| DuckLake / DuckDB (CE) | `quickjs,duckdb,parquet,private` (+ build the FFI) |
|
||||
| + Python jobs | append `,python` |
|
||||
|
||||
## Workspace object storage in dev — use the local filesystem
|
||||
|
||||
For a dev workspace you don't need MinIO/S3: use the built-in **`FilesystemStorage`** large-file
|
||||
storage (a root path on local disk). It is intentionally hidden from the settings-UI storage
|
||||
dropdown (dev-only), so set it via the API. Requires the backend built with `parquet` (+ `private`
|
||||
for the real S3 helpers, + `enterprise` if you want advanced permission rules enforced):
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/w/<ws>/workspaces/edit_large_file_storage_config" \
|
||||
-H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
|
||||
-d '{"large_file_storage":{"type":"FilesystemStorage","root_path":"/abs/writable/dir",
|
||||
"public_resource":false,"advanced_permissions":null,"secondary_storage":{}}}'
|
||||
```
|
||||
|
||||
Optional `advanced_permissions` (EE) is a list of `{"pattern":"<glob>","allow":"read[,write,delete,list]"}`
|
||||
rules: admins bypass them, non-admins are confined to matching grants. Uploads/reads then flow
|
||||
through the normal `job_helpers/*` (viewer-scoped) and `apps_u/*` (app-author on-behalf) S3
|
||||
endpoints. Caveat: direct DuckDB access rejects filesystem stores (`"Filesystem is not supported
|
||||
in DuckDB"`) — DuckLake/datatable go through the S3 proxy instead, which works.
|
||||
|
||||
## Running data pipelines (DuckLake) from source
|
||||
|
||||
DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A
|
||||
|
||||
@@ -1 +1 @@
|
||||
9bc5dfb9ce73a2d9b981a1de86eea6aa26688b79
|
||||
f292a1040da6a667ce7c22abf63ec0debfdd480f
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
//! Deployed-app S3 reads authorize on-behalf of the app author and are confined
|
||||
//! to app provenance (declared keys or recent job outputs): a viewer cannot read
|
||||
//! an arbitrary `file_key` as the author. Requires the `parquet` feature — the
|
||||
//! real `apps_u/*` S3 handlers are gated on it.
|
||||
//!
|
||||
//! `base` fixture: test-user (admin, SECRET_TOKEN); test-user-2 (non-admin,
|
||||
//! SECRET_TOKEN_2, no S3 folder permission).
|
||||
#![cfg(feature = "parquet")]
|
||||
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
const ADMIN_TOKEN: &str = "SECRET_TOKEN";
|
||||
const USER_TOKEN: &str = "SECRET_TOKEN_2";
|
||||
const APP: &str = "u/test-user/s3onbehalf";
|
||||
const DECLARED: &str = "provenance/allowed.csv";
|
||||
const NON_PROVENANCE: &str = "evil/secret.csv";
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", format!("Bearer {}", token))
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_deployed_app_s3_onbehalf_provenance(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
let ws = format!("http://localhost:{port}/api/w/test-workspace");
|
||||
|
||||
// `on_behalf_of` is auto-set to the creator (admin) for an anonymous app, so
|
||||
// the app reads S3 as that author; `DECLARED` is the only allowlisted key.
|
||||
let resp = authed(client().post(format!("{ws}/apps/create")), ADMIN_TOKEN)
|
||||
.json(&json!({
|
||||
"path": APP,
|
||||
"summary": "s3 onbehalf test",
|
||||
"value": {},
|
||||
"policy": {
|
||||
"execution_mode": "anonymous",
|
||||
"triggerables": {},
|
||||
"allowed_s3_keys": [{ "s3_path": DECLARED }]
|
||||
}
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 201, "app create: {}", resp.text().await?);
|
||||
|
||||
// GET an app-scoped S3 route as `token`. No workspace storage is configured,
|
||||
// so a request that clears the provenance gate fails later at the storage
|
||||
// lookup (or the CE OSS stub), never with "File restricted" — which is what
|
||||
// lets these assertions distinguish "gate passed" from "gate denied".
|
||||
let get = |route: &str, token: &'static str| {
|
||||
let url = format!("{ws}/apps_u/{route}");
|
||||
authed(client().get(url), token).send()
|
||||
};
|
||||
let denied = |body: &str| body.contains("File restricted");
|
||||
|
||||
// download_s3_file: author-on-behalf allowed for the declared key, denied for
|
||||
// a key the app never declared (the confused-deputy guard).
|
||||
let body = get(&format!("download_s3_file/{APP}?s3={DECLARED}"), USER_TOKEN)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(!denied(&body), "declared key must clear the gate: {body}");
|
||||
let body = get(
|
||||
&format!("download_s3_file/{APP}?s3={NON_PROVENANCE}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(denied(&body), "non-provenance key must be denied: {body}");
|
||||
|
||||
// load_table_count and load_csv_preview enforce the same gate. The preview's
|
||||
// numeric `limit`/`offset` must deserialize (regression: a flattened query
|
||||
// struct 400s on them under serde_urlencoded).
|
||||
let body = get(
|
||||
&format!("load_table_count/{APP}?file_key={DECLARED}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"table_count declared key must clear the gate: {body}"
|
||||
);
|
||||
let body = get(
|
||||
&format!("load_table_count/{APP}?file_key={NON_PROVENANCE}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
denied(&body),
|
||||
"table_count non-provenance key must be denied: {body}"
|
||||
);
|
||||
|
||||
let resp = get(
|
||||
&format!("load_csv_preview/{APP}?file_key={DECLARED}&limit=5&offset=0"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let body = resp.text().await?;
|
||||
assert_ne!(status, 400, "numeric limit/offset must deserialize: {body}");
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"csv_preview declared key must clear the gate: {body}"
|
||||
);
|
||||
|
||||
// load_file_preview: `read_bytes_from` / `read_bytes_length` are required.
|
||||
let resp = get(
|
||||
&format!("load_file_preview/{APP}?file_key={DECLARED}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
400,
|
||||
"file_preview without byte range must 400: {}",
|
||||
resp.text().await?
|
||||
);
|
||||
let body = get(
|
||||
&format!(
|
||||
"load_file_preview/{APP}?file_key={DECLARED}&read_bytes_from=0&read_bytes_length=4096"
|
||||
),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"file_preview declared key must clear the gate: {body}"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -12121,6 +12121,247 @@ paths:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/apps_u/load_file_metadata/{path}:
|
||||
get:
|
||||
summary: Load metadata of an s3 file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadFileMetadata
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: FileMetadata
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/WindmillFileMetadata"
|
||||
|
||||
/w/{workspace}/apps_u/load_file_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of an s3 file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadFilePreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: file_size_in_bytes
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
- name: file_mime_type
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_separator
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_has_header
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: read_bytes_from
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
- name: read_bytes_length
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: FilePreview
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/WindmillFilePreview"
|
||||
|
||||
/w/{workspace}/apps_u/load_parquet_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of a parquet file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadParquetPreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: offset
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: sort_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: sort_desc
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Parquet Preview
|
||||
content:
|
||||
application/json: {}
|
||||
|
||||
/w/{workspace}/apps_u/load_csv_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of a csv file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadCsvPreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: offset
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: sort_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: sort_desc
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_separator
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Csv Preview
|
||||
content:
|
||||
application/json: {}
|
||||
|
||||
/w/{workspace}/apps_u/load_table_count/{path}:
|
||||
get:
|
||||
summary: Load the table row count on-behalf of the app author (deployed app)
|
||||
operationId: appLoadTableCount
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Table count
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
count:
|
||||
type: number
|
||||
|
||||
/w/{workspace}/apps_u/download_s3_parquet_file_as_csv/{path}:
|
||||
get:
|
||||
summary: Download a parquet s3 file as csv on-behalf of the app author (deployed app)
|
||||
operationId: appDownloadS3ParquetFileAsCsv
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: The downloaded file
|
||||
content:
|
||||
text/csv:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/jobs/run/f/{path}:
|
||||
post:
|
||||
summary: run flow by path
|
||||
|
||||
@@ -26,6 +26,7 @@ use crate::{
|
||||
job_helpers_oss::{
|
||||
download_s3_file_internal, get_random_file_name, get_s3_resource,
|
||||
get_workspace_s3_resource_and_check_paths, upload_file_from_req, DownloadFileQuery,
|
||||
LoadCountQuery, LoadFileMetadataQuery, LoadFilePreviewQuery, LoadPreviewQuery,
|
||||
},
|
||||
users::fetch_api_authed_from_permissioned_as,
|
||||
};
|
||||
@@ -140,6 +141,18 @@ pub fn unauthed_service() -> Router {
|
||||
.route("/upload_s3_file/{*path}", post(upload_s3_file_from_app))
|
||||
.route("/delete_s3_file", delete(delete_s3_file_from_app))
|
||||
.route("/download_s3_file/{*path}", get(download_s3_file_from_app))
|
||||
.route(
|
||||
"/download_s3_parquet_file_as_csv/{*path}",
|
||||
get(app_download_s3_parquet_file_as_csv),
|
||||
)
|
||||
.route("/load_file_metadata/{*path}", get(app_load_file_metadata))
|
||||
.route("/load_file_preview/{*path}", get(app_load_file_preview))
|
||||
.route("/load_table_count/{*path}", get(app_load_table_count))
|
||||
.route(
|
||||
"/load_parquet_preview/{*path}",
|
||||
get(app_load_parquet_preview),
|
||||
)
|
||||
.route("/load_csv_preview/{*path}", get(app_load_csv_preview))
|
||||
.route("/public_app/{secret}", get(get_public_app_by_secret))
|
||||
.route("/embed_token/{secret}", get(get_app_embed_token))
|
||||
.route("/public_resource/{*path}", get(get_public_resource))
|
||||
@@ -3825,8 +3838,9 @@ async fn check_if_allowed_to_access_s3_file_from_app(
|
||||
path: &str,
|
||||
policy: &Policy,
|
||||
) -> Result<()> {
|
||||
// if anonymous, check that the file was the result of an app script ran by an anonymous user in the last 3 hours
|
||||
// otherwise, if logged in, allow any file (TODO: change that when we implement better s3 policy)
|
||||
let is_app_embed = opt_authed.as_ref().is_some_and(|authed| {
|
||||
windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref())
|
||||
});
|
||||
|
||||
if file_query.sig.is_some() {
|
||||
#[cfg(feature = "private")]
|
||||
@@ -3846,19 +3860,17 @@ async fn check_if_allowed_to_access_s3_file_from_app(
|
||||
return Err(Error::InternalErr(
|
||||
"Internal error: signature validation is not supported in open source mode".to_string(),
|
||||
));
|
||||
} else if opt_authed.as_ref().is_some_and(|authed| {
|
||||
!windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref())
|
||||
}) {
|
||||
// A normal logged-in caller (editor / full session) may fetch any file they
|
||||
// can reach. An app embed token also carries an identity but represents
|
||||
// untrusted app JS, so it falls through to the allowlist below instead of
|
||||
// this bypass — otherwise the app could read arbitrary S3 keys the
|
||||
// viewer/on-behalf identity can see, beyond its own declared keys/outputs.
|
||||
} else if matches!(policy.execution_mode, ExecutionMode::Viewer) && !is_app_embed {
|
||||
// Viewer mode: the on-behalf identity IS the viewer, so the downstream
|
||||
// get_workspace_s3_resource_and_check_paths already bounds the read by
|
||||
// their own perms — no provenance gate (it would over-restrict). Embed
|
||||
// tokens are excluded (untrusted app JS stays confined below).
|
||||
Ok(())
|
||||
} else {
|
||||
// Anonymous viewer, or an app embed token: confine to the app's declared S3
|
||||
// keys, or files produced by THIS app's own component runs. The producing
|
||||
// identity is the embed viewer for a token, else `anonymous`.
|
||||
// Author-mode (Anonymous/Publisher) or embed token: confine to the app's
|
||||
// declared keys or files it recently produced. Without this gate a
|
||||
// logged-in viewer could launder the author's S3 perms via an arbitrary
|
||||
// file_key (confused deputy). Producing identity = caller else `anonymous`.
|
||||
let creator = opt_authed
|
||||
.as_ref()
|
||||
.map(|authed| authed.username.clone())
|
||||
@@ -3983,6 +3995,258 @@ async fn download_s3_file_from_app(
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
fn app_s3_file_query(s3: String, storage: Option<String>) -> AppS3FileQuery {
|
||||
AppS3FileQuery {
|
||||
s3,
|
||||
storage,
|
||||
sig: None,
|
||||
#[cfg(feature = "private")]
|
||||
exp: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared entry for every app-scoped (`apps_u/*`) S3 display op: scope-confine an
|
||||
/// app embed token, resolve the on-behalf identity per `execution_mode`, then run
|
||||
/// the provenance gate (`check_if_allowed_to_access_s3_file_from_app`) once before
|
||||
/// dispatching to the S3 helpers.
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_s3_on_behalf_and_provenance(
|
||||
db: &DB,
|
||||
path: &str,
|
||||
w_id: &str,
|
||||
opt_authed: &Option<ApiAuthed>,
|
||||
file_query: &AppS3FileQuery,
|
||||
) -> Result<crate::db::OptJobAuthed> {
|
||||
if let Some(authed) = opt_authed.as_ref() {
|
||||
check_scopes(authed, || format!("apps:read:{}", path))?;
|
||||
}
|
||||
let (on_behalf_authed, policy) =
|
||||
get_on_behalf_authed_from_app(db, path, w_id, opt_authed, None).await?;
|
||||
check_if_allowed_to_access_s3_file_from_app(db, opt_authed, file_query, w_id, path, &policy)
|
||||
.await?;
|
||||
Ok(crate::db::OptJobAuthed { authed: on_behalf_authed, job_id: None })
|
||||
}
|
||||
|
||||
// The app-scoped display ops carry the app path in the URL and everything else
|
||||
// (file_key + op args) in the query, so they avoid a second `{*path}` wildcard.
|
||||
// `LoadCountQuery` / `LoadPreviewQuery` don't include the file key (it's a path
|
||||
// param on the raw `job_helpers/*` route), so restate their fields here with the
|
||||
// file key added. Do NOT `#[serde(flatten)]` the inner struct: axum's `Query`
|
||||
// uses `serde_urlencoded`, which cannot deserialize a flattened field's typed
|
||||
// (numeric/bool) values and 400s on `limit`/`offset` — the fields must be
|
||||
// declared directly on the outer struct.
|
||||
#[cfg(feature = "parquet")]
|
||||
#[derive(Deserialize)]
|
||||
struct AppLoadCountQuery {
|
||||
file_key: String,
|
||||
search_col: Option<String>,
|
||||
search_term: Option<String>,
|
||||
storage: Option<String>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
impl AppLoadCountQuery {
|
||||
fn into_inner(self) -> (String, LoadCountQuery) {
|
||||
(
|
||||
self.file_key,
|
||||
LoadCountQuery {
|
||||
search_col: self.search_col,
|
||||
search_term: self.search_term,
|
||||
storage: self.storage,
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
#[derive(Deserialize)]
|
||||
struct AppLoadPreviewQuery {
|
||||
file_key: String,
|
||||
limit: Option<u32>,
|
||||
offset: Option<i64>,
|
||||
sort_col: Option<String>,
|
||||
sort_desc: Option<bool>,
|
||||
search_col: Option<String>,
|
||||
search_term: Option<String>,
|
||||
storage: Option<String>,
|
||||
csv_separator: Option<String>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
impl AppLoadPreviewQuery {
|
||||
fn into_inner(self) -> (String, LoadPreviewQuery) {
|
||||
(
|
||||
self.file_key,
|
||||
LoadPreviewQuery {
|
||||
limit: self.limit,
|
||||
offset: self.offset,
|
||||
sort_col: self.sort_col,
|
||||
sort_desc: self.sort_desc,
|
||||
search_col: self.search_col,
|
||||
search_term: self.search_term,
|
||||
storage: self.storage,
|
||||
csv_separator: self.csv_separator,
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_download_s3_parquet_file_as_csv(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<DownloadFileQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
crate::job_helpers_oss::download_s3_parquet_file_as_csv_internal(
|
||||
job_authed,
|
||||
&db,
|
||||
None,
|
||||
&w_id,
|
||||
DownloadFileQuery {
|
||||
file_key: query.file_key,
|
||||
s3_resource_path: None,
|
||||
storage: query.storage,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_file_metadata(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<LoadFileMetadataQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_file_metadata_internal(job_authed, &db, &w_id, query).await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_file_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<LoadFilePreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_file_preview_internal(job_authed, &db, &w_id, query).await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_table_count(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadCountQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_table_count_internal(job_authed, &db, &w_id, file_key, inner)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_parquet_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadPreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp = crate::job_helpers_oss::load_preview_internal(
|
||||
job_authed, &db, &w_id, file_key, inner, true,
|
||||
)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_csv_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadPreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp = crate::job_helpers_oss::load_preview_internal(
|
||||
job_authed, &db, &w_id, file_key, inner, false,
|
||||
)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_download_s3_parquet_file_as_csv() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_file_metadata() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_file_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_table_count() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_parquet_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_csv_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
fn get_on_behalf_of(policy: &Policy) -> Result<(String, String)> {
|
||||
let permissioned_as = policy
|
||||
.on_behalf_of
|
||||
|
||||
@@ -217,6 +217,130 @@ pub struct DeleteS3FileQuery {
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
// Stubs for the app-scoped S3 display ops (mirrors the EE `*_internal` helpers +
|
||||
// their query/response structs). Only compiled for a CE build with `parquet` but
|
||||
// without `private`; the real implementations live in `job_helpers_ee.rs`.
|
||||
#[cfg(all(feature = "parquet", not(feature = "private")))]
|
||||
mod app_s3_display_stubs {
|
||||
use super::*;
|
||||
use serde::Serialize;
|
||||
use serde_json::value::RawValue;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadFileMetadataQuery {
|
||||
pub file_key: String,
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct LoadFileMetadataResponse {}
|
||||
|
||||
// Mirror the EE query's required/optional fields so the CE build enforces the
|
||||
// same query contract (e.g. the mandatory byte range) at the extraction layer.
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadFilePreviewQuery {
|
||||
pub storage: Option<String>,
|
||||
pub file_key: String,
|
||||
pub file_size_in_bytes: Option<u64>,
|
||||
pub file_mime_type: Option<String>,
|
||||
pub csv_separator: Option<String>,
|
||||
pub csv_has_header: Option<bool>,
|
||||
pub read_bytes_from: u64,
|
||||
pub read_bytes_length: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct LoadFilePreviewResponse {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadCountQuery {
|
||||
pub search_col: Option<String>,
|
||||
pub search_term: Option<String>,
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct TableCount {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadPreviewQuery {
|
||||
pub limit: Option<u32>,
|
||||
pub offset: Option<i64>,
|
||||
pub sort_col: Option<String>,
|
||||
pub sort_desc: Option<bool>,
|
||||
pub search_col: Option<String>,
|
||||
pub search_term: Option<String>,
|
||||
pub storage: Option<String>,
|
||||
pub csv_separator: Option<String>,
|
||||
}
|
||||
|
||||
pub async fn load_file_metadata_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_query: LoadFileMetadataQuery,
|
||||
) -> error::Result<LoadFileMetadataResponse> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_file_preview_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_query: LoadFilePreviewQuery,
|
||||
) -> error::Result<LoadFilePreviewResponse> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_table_count_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_file_key: String,
|
||||
_query: LoadCountQuery,
|
||||
) -> error::Result<TableCount> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_preview_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_file_key: String,
|
||||
_query: LoadPreviewQuery,
|
||||
_is_parquet: bool,
|
||||
) -> error::Result<Box<RawValue>> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn download_s3_parquet_file_as_csv_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_user_db: Option<UserDB>,
|
||||
_w_id: &str,
|
||||
_query: DownloadFileQuery,
|
||||
) -> error::Result<Response> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "parquet", not(feature = "private")))]
|
||||
pub use app_s3_display_stubs::*;
|
||||
|
||||
#[cfg(not(feature = "private"))]
|
||||
pub async fn get_workspace_s3_resource_and_check_paths<'c>(
|
||||
_db_with_opt_authed: &DbWithOptAuthed<'c, ApiAuthed>,
|
||||
|
||||
@@ -135,6 +135,25 @@
|
||||
let enableHtml = $state(false)
|
||||
let s3FileDisplayRawMode = $state(false)
|
||||
|
||||
// Build the image/PDF source URL for an S3 object. When `appPath` is set
|
||||
// (deployed app view) the read is authorized on-behalf of the app author via
|
||||
// the provenance-gated `apps_u/download_s3_file/{appPath}` endpoint; otherwise
|
||||
// (editor/preview) it uses the viewer-scoped `job_helpers/load_image_preview`.
|
||||
function s3DisplayUrl(s3object: { s3: string; storage?: string; presigned?: string }): string {
|
||||
const endpoint = appPath
|
||||
? `apps_u/download_s3_file/${appPath}`
|
||||
: 'job_helpers/load_image_preview'
|
||||
const keyParam = appPath ? 's3' : 'file_key'
|
||||
let url = `/api/w/${workspaceId}/${endpoint}?${keyParam}=${encodeURIComponent(s3object.s3)}`
|
||||
if (s3object.storage) {
|
||||
url += `&storage=${s3object.storage}`
|
||||
}
|
||||
if (appPath && s3object.presigned) {
|
||||
url += `&${s3object.presigned}`
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
function isTableRow(result: any): boolean {
|
||||
return Array.isArray(result) && result.every((x) => Array.isArray(x))
|
||||
}
|
||||
@@ -677,6 +696,7 @@
|
||||
{jobId}
|
||||
{nodeId}
|
||||
{workspaceId}
|
||||
{appPath}
|
||||
forceJson={globalForceJson}
|
||||
hideAsJson={true}
|
||||
/>
|
||||
@@ -1032,48 +1052,26 @@
|
||||
{/if}
|
||||
</div>
|
||||
{#if typeof s3object?.s3 === 'string'}
|
||||
{#if !appPath && (s3object?.s3?.endsWith('.parquet') || s3object?.s3?.endsWith('.csv'))}
|
||||
{#if s3object?.s3?.endsWith('.parquet') || s3object?.s3?.endsWith('.csv')}
|
||||
{#key s3object.s3}
|
||||
<ParqetTableRenderer
|
||||
disable_download={s3object?.disable_download}
|
||||
{workspaceId}
|
||||
{appPath}
|
||||
s3resource={s3object?.s3}
|
||||
storage={s3object?.storage}
|
||||
/>
|
||||
{/key}
|
||||
{:else if s3object?.s3?.endsWith('.png') || s3object?.s3?.endsWith('.jpeg') || s3object?.s3?.endsWith('.jpg') || s3object?.s3?.endsWith('.webp')}
|
||||
<div class="h-full mt-2">
|
||||
<img
|
||||
alt="preview rendered"
|
||||
class="w-auto h-full"
|
||||
src="{`/api/w/${workspaceId}/${
|
||||
appPath
|
||||
? 'apps_u/download_s3_file/' + appPath
|
||||
: 'job_helpers/load_image_preview'
|
||||
}?${appPath ? 's3' : 'file_key'}=${encodeURIComponent(s3object.s3)}` +
|
||||
(s3object.storage ? `&storage=${s3object.storage}` : '')}{appPath &&
|
||||
s3object.presigned
|
||||
? `&${s3object.presigned}`
|
||||
: ''}"
|
||||
/>
|
||||
<img alt="preview rendered" class="w-auto h-full" src={s3DisplayUrl(s3object)} />
|
||||
</div>
|
||||
{:else if s3object?.s3?.endsWith('.pdf')}
|
||||
<div class="h-96 mt-2 border">
|
||||
{#await import('$lib/components/display/PdfViewer.svelte')}
|
||||
<Loader2 class="animate-spin" />
|
||||
{:then Module}
|
||||
<Module.default
|
||||
allowFullscreen
|
||||
source="{`/api/w/${workspaceId}/${
|
||||
appPath
|
||||
? 'apps_u/download_s3_file/' + appPath
|
||||
: 'job_helpers/load_image_preview'
|
||||
}?${appPath ? 's3' : 'file_key'}=${encodeURIComponent(s3object.s3)}` +
|
||||
(s3object.storage ? `&storage=${s3object.storage}` : '')}{appPath &&
|
||||
s3object.presigned
|
||||
? `&${s3object.presigned}`
|
||||
: ''}"
|
||||
/>
|
||||
<Module.default allowFullscreen source={s3DisplayUrl(s3object)} />
|
||||
{/await}
|
||||
</div>
|
||||
{/if}
|
||||
@@ -1115,6 +1113,7 @@
|
||||
<ParqetTableRenderer
|
||||
disable_download={s3object?.disable_download}
|
||||
{workspaceId}
|
||||
{appPath}
|
||||
s3resource={s3object?.s3}
|
||||
storage={s3object?.storage}
|
||||
/>{:else}
|
||||
@@ -1132,9 +1131,7 @@
|
||||
<img
|
||||
alt="preview rendered"
|
||||
class="w-auto h-full"
|
||||
src={`/api/w/${workspaceId}/job_helpers/load_image_preview?file_key=${encodeURIComponent(
|
||||
s3object.s3
|
||||
)}` + (s3object.storage ? `&storage=${s3object.storage}` : '')}
|
||||
src={s3DisplayUrl(s3object)}
|
||||
/>
|
||||
</div>
|
||||
{:else}
|
||||
@@ -1151,12 +1148,7 @@
|
||||
{#await import('$lib/components/display/PdfViewer.svelte')}
|
||||
<Loader2 class="animate-spin" />
|
||||
{:then Module}
|
||||
<Module.default
|
||||
allowFullscreen
|
||||
source={`/api/w/${workspaceId}/job_helpers/load_image_preview?file_key=${encodeURIComponent(
|
||||
s3object.s3
|
||||
)}` + (s3object.storage ? `&storage=${s3object.storage}` : '')}
|
||||
/>
|
||||
<Module.default allowFullscreen source={s3DisplayUrl(s3object)} />
|
||||
{/await}
|
||||
</div>
|
||||
{/if}
|
||||
@@ -1292,6 +1284,7 @@
|
||||
{jobId}
|
||||
{nodeId}
|
||||
{workspaceId}
|
||||
{appPath}
|
||||
{hideAsJson}
|
||||
{forceJson}
|
||||
disableExpand={true}
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
import 'ag-grid-community/styles/ag-theme-alpine.css'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import DarkModeObserver from './DarkModeObserver.svelte'
|
||||
import { HelpersService } from '$lib/gen'
|
||||
import { AppService, HelpersService } from '$lib/gen'
|
||||
import { base } from '$lib/base'
|
||||
import { downloadViaClient, shouldDownloadViaClient } from '$lib/utils/downloadFile'
|
||||
import { enterpriseLicense, workspaceStore } from '$lib/stores'
|
||||
@@ -22,9 +22,61 @@
|
||||
storage: string | undefined
|
||||
workspaceId: string | undefined
|
||||
disable_download?: boolean
|
||||
// When set (deployed app view), read the file on-behalf of the app author
|
||||
// through the app-scoped, provenance-gated `apps_u/*` endpoints instead of
|
||||
// the viewer-scoped `job_helpers/*` API. Undefined in the editor/preview.
|
||||
appPath?: string | undefined
|
||||
}
|
||||
|
||||
let { s3resource, storage, workspaceId, disable_download = false }: Props = $props()
|
||||
let {
|
||||
s3resource,
|
||||
storage,
|
||||
workspaceId,
|
||||
disable_download = false,
|
||||
appPath = undefined
|
||||
}: Props = $props()
|
||||
|
||||
// Route the parquet/csv read through the app-scoped endpoints when `appPath`
|
||||
// is set, else the viewer-scoped helpers. Same request/response shape either
|
||||
// way — the only difference is which identity authorizes the S3 read.
|
||||
function loadRowCount(searchCol: string | undefined, searchTerm: string | undefined) {
|
||||
const workspace = workspaceId ?? $workspaceStore!
|
||||
return appPath
|
||||
? AppService.appLoadTableCount({
|
||||
workspace,
|
||||
path: appPath,
|
||||
fileKey: s3resource,
|
||||
searchCol,
|
||||
searchTerm,
|
||||
storage
|
||||
})
|
||||
: HelpersService.loadTableRowCount({
|
||||
workspace,
|
||||
path: s3resource,
|
||||
searchCol,
|
||||
searchTerm,
|
||||
storage
|
||||
})
|
||||
}
|
||||
|
||||
function loadChunk(args: {
|
||||
offset?: number
|
||||
limit?: number
|
||||
sortCol?: string
|
||||
sortDesc?: boolean
|
||||
searchCol?: string
|
||||
searchTerm?: string
|
||||
csvSeparator?: string
|
||||
}) {
|
||||
const workspace = workspaceId ?? $workspaceStore!
|
||||
const csv = s3resource.endsWith('.csv')
|
||||
if (appPath) {
|
||||
const data = { workspace, path: appPath, fileKey: s3resource, storage, ...args }
|
||||
return csv ? AppService.appLoadCsvPreview(data) : AppService.appLoadParquetPreview(data)
|
||||
}
|
||||
const data = { workspace, path: s3resource, storage, ...args }
|
||||
return csv ? HelpersService.loadCsvPreview(data) : HelpersService.loadParquetPreview(data)
|
||||
}
|
||||
|
||||
let lastSearch: string | undefined = undefined
|
||||
|
||||
@@ -40,34 +92,20 @@
|
||||
const newSearch = searchCol ? searchCol + searchTerm : undefined
|
||||
if (!nbRows || lastSearch != newSearch) {
|
||||
nbRows = undefined
|
||||
const res = await HelpersService.loadTableRowCount({
|
||||
workspace: workspaceId ?? $workspaceStore!,
|
||||
path: s3resource,
|
||||
searchCol: searchCol,
|
||||
storage,
|
||||
searchTerm
|
||||
})
|
||||
const res = await loadRowCount(searchCol, searchTerm)
|
||||
nbRows = res.count
|
||||
lastSearch = newSearch
|
||||
}
|
||||
|
||||
const requestBody = {
|
||||
workspace: workspaceId ?? $workspaceStore!,
|
||||
path: s3resource,
|
||||
const res = (await loadChunk({
|
||||
offset: params.startRow,
|
||||
limit: params.endRow - params.startRow,
|
||||
sortCol: params.sortModel?.[0]?.colId,
|
||||
sortDesc: params.sortModel?.[0]?.sort == 'desc',
|
||||
searchCol,
|
||||
searchTerm,
|
||||
storage: storage,
|
||||
csvSeparator: csv ? csvSeparatorChar : undefined
|
||||
}
|
||||
const res = (
|
||||
csv
|
||||
? await HelpersService.loadCsvPreview(requestBody)
|
||||
: await HelpersService.loadParquetPreview(requestBody)
|
||||
) as any
|
||||
})) as any
|
||||
for (let i = 0; i < res.rows.length; i++) {
|
||||
res.rows[i]['__index'] = i + params.startRow
|
||||
if (!$enterpriseLicense) {
|
||||
@@ -110,20 +148,10 @@
|
||||
try {
|
||||
const csv = s3resource.endsWith('.csv')
|
||||
|
||||
const res = csv
|
||||
? await HelpersService.loadCsvPreview({
|
||||
workspace: $workspaceStore!,
|
||||
path: s3resource,
|
||||
limit: 0,
|
||||
storage: storage,
|
||||
csvSeparator: csvSeparatorChar
|
||||
})
|
||||
: await HelpersService.loadParquetPreview({
|
||||
workspace: $workspaceStore!,
|
||||
path: s3resource,
|
||||
limit: 0,
|
||||
storage: storage
|
||||
})
|
||||
const res = (await loadChunk({
|
||||
limit: 0,
|
||||
csvSeparator: csv ? csvSeparatorChar : undefined
|
||||
})) as any
|
||||
|
||||
createGrid(
|
||||
eGui,
|
||||
@@ -201,14 +229,15 @@
|
||||
</div>
|
||||
{/if}
|
||||
{#if !disable_download && !s3resource.endsWith('.csv')}
|
||||
{@const csvApiPath = `/w/${workspaceId}/job_helpers/download_s3_parquet_file_as_csv?file_key=${encodeURIComponent(s3resource)}${storage ? `&storage=${storage}` : ''}`}
|
||||
{@const csvApiPath = appPath
|
||||
? `/w/${workspaceId}/apps_u/download_s3_parquet_file_as_csv/${appPath}?file_key=${encodeURIComponent(s3resource)}${storage ? `&storage=${storage}` : ''}`
|
||||
: `/w/${workspaceId}/job_helpers/download_s3_parquet_file_as_csv?file_key=${encodeURIComponent(s3resource)}${storage ? `&storage=${storage}` : ''}`}
|
||||
{@const csvName = (s3resource.split('/').pop() ?? 'download') + '.csv'}
|
||||
{#if shouldDownloadViaClient()}
|
||||
<button
|
||||
class="text-secondary w-full text-right underline text-2xs whitespace-nowrap"
|
||||
onclick={() => downloadViaClient(csvApiPath, csvName)}
|
||||
><div class="flex flex-row-reverse gap-2 items-center"
|
||||
><Download size={12} /> CSV</div
|
||||
><div class="flex flex-row-reverse gap-2 items-center"><Download size={12} /> CSV</div
|
||||
></button
|
||||
>
|
||||
{:else}
|
||||
@@ -216,9 +245,7 @@
|
||||
target="_blank"
|
||||
href="{base}/api{csvApiPath}"
|
||||
class="text-secondary w-full text-right underline text-2xs whitespace-nowrap"
|
||||
><div class="flex flex-row-reverse gap-2 items-center"
|
||||
><Download size={12} /> CSV</div
|
||||
></a
|
||||
><div class="flex flex-row-reverse gap-2 items-center"><Download size={12} /> CSV</div></a
|
||||
>
|
||||
{/if}
|
||||
{/if}
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
import ResolveStyle from '../helpers/ResolveStyle.svelte'
|
||||
import { components } from '../../editor/component'
|
||||
import ResolveConfig from '../helpers/ResolveConfig.svelte'
|
||||
import { userStore } from '$lib/stores'
|
||||
|
||||
interface Props {
|
||||
id: string
|
||||
@@ -37,13 +36,16 @@
|
||||
}: Props = $props()
|
||||
|
||||
const requireHtmlApproval = getContext<boolean | undefined>(IS_APP_PUBLIC_CONTEXT_KEY)
|
||||
const { app, worldStore, componentControl, workspace, appPath } =
|
||||
const { app, worldStore, componentControl, workspace, appPath, isEditor } =
|
||||
getContext<AppViewerContext>('AppViewerContext')
|
||||
|
||||
let result: any = $state(undefined)
|
||||
|
||||
const resolvedConfig = $state(
|
||||
initConfig(components['displaycomponent'].initialData.configuration, untrack(() => configuration))
|
||||
initConfig(
|
||||
components['displaycomponent'].initialData.configuration,
|
||||
untrack(() => configuration)
|
||||
)
|
||||
)
|
||||
|
||||
$componentControl[untrack(() => id)] = {
|
||||
@@ -52,12 +54,21 @@
|
||||
}
|
||||
}
|
||||
|
||||
const outputs = initOutput($worldStore, untrack(() => id), {
|
||||
result: undefined,
|
||||
loading: false
|
||||
})
|
||||
const outputs = initOutput(
|
||||
$worldStore,
|
||||
untrack(() => id),
|
||||
{
|
||||
result: undefined,
|
||||
loading: false
|
||||
}
|
||||
)
|
||||
|
||||
let css = $state(initCss($app.css?.displaycomponent, untrack(() => customCss)))
|
||||
let css = $state(
|
||||
initCss(
|
||||
$app.css?.displaycomponent,
|
||||
untrack(() => customCss)
|
||||
)
|
||||
)
|
||||
let loading = $state(false)
|
||||
</script>
|
||||
|
||||
@@ -119,7 +130,7 @@
|
||||
{result_stream}
|
||||
{requireHtmlApproval}
|
||||
disableExpand={resolvedConfig?.hideDetails}
|
||||
appPath={$userStore ? undefined : $appPath}
|
||||
appPath={isEditor ? undefined : $appPath}
|
||||
forceJson={resolvedConfig?.forceJson}
|
||||
/>
|
||||
</div>
|
||||
|
||||
+22
-10
@@ -16,7 +16,6 @@
|
||||
import ResolveStyle from '../helpers/ResolveStyle.svelte'
|
||||
import InitializeComponent from '../helpers/InitializeComponent.svelte'
|
||||
import DisplayResult from '$lib/components/DisplayResult.svelte'
|
||||
import { userStore } from '$lib/stores'
|
||||
|
||||
interface Props {
|
||||
id: string
|
||||
@@ -34,22 +33,35 @@
|
||||
render
|
||||
}: Props = $props()
|
||||
|
||||
const { app, worldStore, workspace, appPath } = getContext<AppViewerContext>('AppViewerContext')
|
||||
const { app, worldStore, workspace, appPath, isEditor } =
|
||||
getContext<AppViewerContext>('AppViewerContext')
|
||||
const requireHtmlApproval = getContext<boolean | undefined>(IS_APP_PUBLIC_CONTEXT_KEY)
|
||||
|
||||
let resolvedConfig = $state(
|
||||
initConfig(components['jobiddisplaycomponent'].initialData.configuration, untrack(() => configuration))
|
||||
initConfig(
|
||||
components['jobiddisplaycomponent'].initialData.configuration,
|
||||
untrack(() => configuration)
|
||||
)
|
||||
)
|
||||
|
||||
const outputs = initOutput($worldStore, untrack(() => id), {
|
||||
result: undefined,
|
||||
loading: false,
|
||||
jobId: undefined as string | undefined
|
||||
})
|
||||
const outputs = initOutput(
|
||||
$worldStore,
|
||||
untrack(() => id),
|
||||
{
|
||||
result: undefined,
|
||||
loading: false,
|
||||
jobId: undefined as string | undefined
|
||||
}
|
||||
)
|
||||
|
||||
initializing = false
|
||||
|
||||
let css = $state(initCss($app.css?.jobiddisplaycomponent, untrack(() => customCss)))
|
||||
let css = $state(
|
||||
initCss(
|
||||
$app.css?.jobiddisplaycomponent,
|
||||
untrack(() => customCss)
|
||||
)
|
||||
)
|
||||
|
||||
let jobLoader: JobLoader | undefined = $state(undefined)
|
||||
let testIsLoading: boolean = $state(false)
|
||||
@@ -137,7 +149,7 @@
|
||||
{result}
|
||||
{requireHtmlApproval}
|
||||
disableExpand={resolvedConfig?.hideDetails}
|
||||
appPath={$userStore ? undefined : $appPath}
|
||||
appPath={isEditor ? undefined : $appPath}
|
||||
forceJson={resolvedConfig?.forceJson}
|
||||
/>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user