Commit Graph

71 Commits

Author SHA1 Message Date
Diego Imbert e68bf83303 fix: fix parse_postgres_uri not decoding password (#7157)
* nit overscroll auto

* decode password in parse_postgres_url

* don't encode host in pg_executor

* nit
2025-11-17 11:29:44 +00:00
Diego Imbert 6ecd4954b2 SQL result_collection flag (#7041)
* Return array of results duckdb

* Migration script to add result_collection=legacy

* migration script fixes

* app_version_lite not necessary

* New annotations macro that supports custom types

* pass unit tests

* nit style

* result_collection almost works for postgres

* fix last_statement

* frontend suggesitons

* fix column_order making columns disappear

* added version check for duckdb FFI lib to avoid crashes when changing FFI interface on Windows agent workers

* result_collection for duckdb

* Correct legacy behavior in DuckDB

* mysql result_collection

* mssql collection_strategy

* result_collection for oracle

* snowflake result_collection

* fix errors

* mistake, .clone() causing deadlock

* fix frontend errors on oracle / mssql

* fix legacy behavior on mssql

* add exception catch in migration

* missing app_script update

* change cache layout to avoid worker pulling old scripts
2025-11-12 15:32:07 +00:00
hugocasa 7b5a02be4b feat: add support for contextual vars in SQL (#6791) 2025-10-10 18:52:24 +00:00
Alexander Petric 592c3067ed fix: support pg jsonb array (#6788) 2025-10-09 21:01:43 +00:00
Diego Imbert 8aafb63d51 Timestamp input in db manager (#6754)
* nullable by default

* unrelated nits

* native timezone input postgres

* Fix timestamptz NaiveDateTime conversion is pg_executor

* remove consolelog
2025-10-06 15:10:17 +00:00
Diego Imbert 4d8f1dfb76 NULL Toggle in InsertRow drawer (#6729)
* NULL toggle in InsertRow

* fix long type parsing in postgres

* nits

* graphite catch

* lazy_static

* support for time/timestamp/tz long forms in pg parser

* graphite suggestion
2025-10-02 10:10:53 +00:00
Diego Imbert d5e9213fd5 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel e15b94692d feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
Diego Imbert e88954a217 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Diego Imbert 25bbdba1d1 feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel c76bc4a9cc fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel 947c91809d feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
Ruben Fiszel 123b4405ed perf: optimize number of queries needed for job run (#5504)
* optPerf

* update sqlx

* update sqlx
2025-03-24 20:12:26 +01:00
wendrul 71c54024d3 feat: unsafe parameters for sql queries (table names, column names) (#5488)
* Make schema validation struct

Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).

* Make other validator builder

* Fail dependency job like with lockfile failing for schema validator

* Add last types + tests

* Remove unused dependency

* fix typos

* Migration ID was colliding with another, changed it manually

* Add Oneof + other fixes

* fix: cache for querying scripts correclty handles ScriptMetadata

* Add cache for schema validation from main arg sig

* Prepare sqlx

* Remove default features

* Feature flags

* WIP: unsafe sql params for sql langauges

* Fix down migration table name

* cleanup: put validation logic inside a function

* Refactor to cache the should_validate boolean

Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.

Also pass the schema when avaialble to construct the schema validator

* Add other job kinds to u8 cache key just in case

* Change sql languages to all get arguments as Values instead of RawValue

* Only cache if not preview

* Add last sql languages and some CI fixes

* Rename after typo on `sanitized`

* Finish rename

* Remove unused import

* Fix wrong test

* Add newly published regex parser version

* Remove default features from cargo.toml

* Change to a cleaner syntax for the interpolated args

* Update republished parser
2025-03-20 20:02:25 +01:00
Ruben Fiszel aaa4cde349 feat: filter by worker + backend perf opt (#5489)
* all

* all

* all

* refactor

* sqlx

* ref

* impr

* fix

* fix

* fix

* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel 1dd9ca10fc pg nits 2025-03-14 12:13:12 +01:00
Ruben Fiszel f726b3af88 fix: avoid lock contention for native workers on cached connection (#5481) 2025-03-14 11:58:03 +01:00
Ruben Fiszel 628ae617f7 feat: docker runtime 2024-12-09 18:02:18 +01:00
HugoCasa 5ef105b833 fix: revert bool to text support in pg (#4727) 2024-11-16 10:01:16 +01:00
HugoCasa 3939f42f5c unbreak nullable inputs in pg (#4726) 2024-11-15 22:27:28 +01:00
HugoCasa 8734df138c unbreak ::text for json in pg (#4722)
* unbreak ::text for json in pg

* better error handling
2024-11-15 18:35:21 +01:00
HugoCasa 921e3217c6 improve array params handling in postgres (#4605) 2024-10-30 14:44:42 +01:00
Ruben Fiszel 85de44cb93 feat: autoscaling v0 (#4593)
* all

* all

all

all

all

all

all

all

all

all

* all

* all

* nits

* nits
2024-10-28 00:47:19 +01:00
pyranota c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa 97cdafac86 feat: add return_last_result annotation to sql (#4443) 2024-09-25 18:11:11 +02:00
Ruben Fiszel fc6e480e56 fix: postgres scripts that take longer than 20s do not timeout anymore 2024-09-18 14:32:16 +02:00
Ruben Fiszel bef1bcb2a4 fix: timeout in pg executor on postgresql connection after 20s 2024-09-12 18:49:43 +02:00
HugoCasa 8942e80ebb fix: handle time with tz col type in pg (#4239) 2024-08-14 13:25:56 +02:00
HugoCasa 2b9b4eedc3 feat: multi sql statement with pg fix (#4134)
* Revert "Revert "feat: multi statement sql (#4104)" (#4133)"

This reverts commit c578f05e2d.

* fix: sort pg args
2024-07-26 09:23:39 +02:00
Ruben Fiszel c578f05e2d Revert "feat: multi statement sql (#4104)" (#4133)
This reverts commit 5bc0e96171.
2024-07-26 01:24:03 +02:00
HugoCasa 5bc0e96171 feat: multi statement sql (#4104)
* feat: multi statement pg

* fix: add other flavors

* feat: make pg params start at 1 and sequential

* fix: improve sql statement parsing

* add tests

* fix: allow no semi in last statement

* fix: merge conflict

* fix: minor improvement

* fix: parser version
2024-07-25 20:46:04 +02:00
HugoCasa 6da5e15903 feat: add support for bytea in pg (#3926)
* feat: add support for bytea in pg

* fix: editor nits
2024-06-18 17:38:46 +02:00
Ruben Fiszel 6a332bc30f fix: make all error strings more verbose 2024-05-25 22:15:15 +02:00
Ruben Fiszel 75730d410d rm go bin if cloud hosted 2024-04-26 13:42:24 +02:00
HugoCasa e8905b2734 feat: pg add json support (#3620) 2024-04-26 13:28:15 +02:00
Ruben Fiszel 8cb1e4846c increase pg size limit for cloud 2024-04-21 10:20:50 +02:00
Ruben Fiszel da3ded6196 fix: measure memory usage on postgres scripts 2024-04-18 01:05:16 +02:00
HugoCasa c2d18a6d6f feat keep sql columns ordering (#3444)
* feat: keep pg columns ordering

* feat: pass column order through flow status

* feat: sql ordering for all sql langs

* fix: sqlx build on mac + rename to _metadata

* Update JobPreview.svelte

* Update +page.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update JobPreview.svelte

* fix: build

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-03-22 10:34:32 +01:00
HugoCasa c0104641f0 feat: add bit support in pg (#3407) 2024-03-14 10:40:20 +01:00
HugoCasa c03c73797b fix: pg timstamptz param (#3364) 2024-03-06 12:50:01 +01:00
HugoCasa 1ee45a1f7d feat: allow to pin database in sql scripts (#3304) 2024-02-27 19:59:52 +01:00
Ruben Fiszel d414e9fff9 refactor future wrapper for queries 2024-02-19 22:11:45 +01:00
Ruben Fiszel 37e80ca1a4 fix(psql): add mem peak to postgresql 2024-02-19 20:51:09 +01:00
Ruben Fiszel fa9d5af10e respect query timeout for pg queries 2024-02-19 20:11:28 +01:00
Ruben Fiszel 52b2b5048d fix: prevent postgres timeout abuse 2024-02-19 19:59:43 +01:00
HugoCasa af1b4c0dde fix: add timestamp array support in pg (#3229) 2024-02-15 17:18:41 +01:00
HugoCasa d7834f3694 fix: pg coerce nb to string (#3127) 2024-02-01 14:08:14 +01:00
Faton Ramadani 35756cf411 feat(frontend): DB Explorer (#2892)
* feat(frontend): Make table cell editable

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): clean up

* fix(frontend): extract db schemas

* feat(frontend): v0 done

* feat(frontend): v0 done

* feat(frontend): v0 done

* feat(frontend): fix insert

* feat(frontend): fix insert

* feat(frontend): remove temp data

* feat(frontend): align insert button to the right

* feat(frontend): rework columns

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): fix jsonb display

* feat(frontend): fix reloading issues

* feat(frontend): fix reloading issues

* feat(frontend): fix reacticity issue

* feat(frontend): fix insert

* feat(frontend): support nullable default value

* feat(frontend): fix build

* update

* update

* db studio v0

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-01-06 17:27:15 +01:00
HugoCasa 3cfcf28c57 fix: postgres ssl mode (#2861) 2023-12-15 15:15:55 +01:00