Files
windmill/frontend/src/lib/utils.test.ts
T
Diego Imbert 0c0c2f62a5 feat: type-checked data tables v0 (#7381)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* stash

* sql asset parser parses most s3 literals

* nit

* Detect attach + handle returning RW

* detect assets used with dot notation

* detect implicit access with USE dl; syntax

* Add assets as unknown if var was never used

* Support default ducklake/datatable main in parser

* ignore asset parsing errors in frontend (avoid flow layout shift)

* super weird duplication (merge conflict ?)

* nits

* fix duckdb parser detecting too much as asset when RW ctx is unknown

* fix transparent assets btn

* missing arg

* nit styling

* asset parser specific table parsing

* fix resource specific table parsing

* More concise asset display in flows + better icons

* fix assets page filtering out resources with added table

* Fix frontend to support specific table assets

* Open DB Manager to specific table

* Specific table parser in Python and TS + unit tests

* Fix UPDATE setting access to None

* fix flow edge rendering on top of output picker

* python parser fix var override bug

* add ts test

* fix compilation

* sqlx prepare

* update parsers version

* fix missing schema key onDelete

* Grant permission to create schemas in custom instance databases

* Update pg query to return empty schemas

* Create schema

* Select nits

* support schemas in sql parser

* ts parser handle schema with sql parser result

* detect .schema() syntax

* detect schema syntax in python

* support .schema() in ts and py SDK

* open db manager to specific schema

* support reassignment in ts parser

* nit better unitest

* : syntax in ts

* datatable:schema syntax in python

* fix client py

* nit select dropdown darkmode

* object | null fetchOne

* ts client nits

* parse_sql_client_name fn

* getImportWmillTsStatement refactor in EditorBar

* text to json() in python client

* update parser versions

* pkg lock

* Sql query details in TS asset parser

* code transformation with type parameter in Editor

* Custom Language Worker, code substition works !

* Error marker mapping works

* hover info is correct

* completions work correctly

* other overrides

* type inference kinda works

* Position mapping tests

* refactor prepare_queries

* Refactor PgDatabase to share common code

* Pgdatabase in prepare_queries

* TokioPgConnection refactor

* refactor prepare_queries

* type parameter to sql function

* Fix deadlock

* nit fix

* Fix worker async call freezing because of svelte Proxy

* Force worker to recompute when we set queries

* nit refactor

* nits console logs

* wait that ts worker initialize

* monaco change file version

* update diagnostics

* Refactor for errors

* Show SQL errors in Monaco

* improve sdk

* cleaning refactor + MapResource + usePreparedAssetSqlQueries

* Fixes

* Fix error position mapping

* cache in typescript worker

* fix insert no values

* don't inject type if already present

* Support schema in prepare queries

* update parsers

* ChangeOnDeepInequality

* inferAsset ScriptEditor usage refactor

* sql query typecheck work in flow editor

* Assets and SQL Query check in Raw App Inline Editor

* pkg lock

* Fix DatatableSqlTemplateFunction nit

* prepare query schema nit

* duplicate diagnostics

* nit getScriptVersion mock

* Reprepare queries when switching workspaces

* nit fix

* nit fix

* fetch_one_scalar and execute in python client

* limit pg_connections

* -- prepare flag in postgres

* skip serializing

* fix destructuring undefined

* Prepare queries in workers instead of backend

* nit

* Execute search_path instructions normally

* nit fix

* Fix SET search_path issue in prepare

* only support preparing single-statement queries for now

* update parsers

* safety

* better remove_comments

* Fix getQueryStmtCountHeuristic

* getQueryStmtCountHeuristic tests

* comment out failing tests

* Fix getQueryStmtCountHeuristic impl

* only datatable
2025-12-20 08:51:07 +00:00

286 lines
9.2 KiB
TypeScript

import { describe, it, expect } from 'vitest'
import { getQueryStmtCountHeuristic } from './utils'
describe('getQueryStmtCountHeuristic', () => {
describe('basic statements', () => {
it('should count a single statement without semicolon', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users')).toBe(1)
})
it('should count a single statement with semicolon', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users;')).toBe(1)
})
it('should count multiple statements', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users; SELECT * FROM posts;')).toBe(2)
})
it('should handle empty string as implicit statement', () => {
expect(getQueryStmtCountHeuristic('')).toBe(0)
})
it('should handle whitespace only as implicit statement', () => {
expect(getQueryStmtCountHeuristic(' \n\t ')).toBe(0)
})
it('should handle statement with trailing whitespace', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users; \n ')).toBe(1)
})
it('should count three statements', () => {
expect(
getQueryStmtCountHeuristic('SELECT 1; UPDATE users SET name = "test"; DELETE FROM logs;')
).toBe(3)
})
})
describe('single-quoted strings', () => {
it('should ignore semicolon in single-quoted string', () => {
expect(getQueryStmtCountHeuristic("SELECT 'hello;world' FROM users")).toBe(1)
})
it('should handle escaped single quote', () => {
expect(getQueryStmtCountHeuristic("SELECT 'it''s working' FROM users")).toBe(1)
})
it('should handle multiple escaped single quotes', () => {
expect(getQueryStmtCountHeuristic("SELECT 'it''s a ''test''' FROM users")).toBe(1)
})
it('should handle semicolon in escaped quote context', () => {
expect(getQueryStmtCountHeuristic("SELECT 'val''s;data' FROM users")).toBe(1)
})
it('should handle multiple statements with single-quoted strings', () => {
expect(
getQueryStmtCountHeuristic("SELECT 'hello;' FROM users; SELECT 'world;' FROM posts")
).toBe(2)
})
it('should handle adjacent escaped quotes', () => {
expect(getQueryStmtCountHeuristic("SELECT '''' FROM users")).toBe(1)
})
})
describe('double-quoted strings', () => {
it('should ignore semicolon in double-quoted string', () => {
expect(getQueryStmtCountHeuristic('SELECT "hello;world" FROM users')).toBe(1)
})
it('should handle escaped double quote', () => {
expect(getQueryStmtCountHeuristic('SELECT "it""s working" FROM users')).toBe(1)
})
it('should handle multiple escaped double quotes', () => {
expect(getQueryStmtCountHeuristic('SELECT "it""s a ""test""" FROM users')).toBe(1)
})
it('should handle semicolon in escaped quote context', () => {
expect(getQueryStmtCountHeuristic('SELECT "val""s;data" FROM users')).toBe(1)
})
it('should handle adjacent escaped quotes', () => {
expect(getQueryStmtCountHeuristic('SELECT """" FROM users')).toBe(1)
})
})
describe('mixed quotes', () => {
it('should handle both single and double quotes in same query', () => {
expect(getQueryStmtCountHeuristic('SELECT "col;1", \'val;2\' FROM users')).toBe(1)
})
it('should handle single quote inside double quote', () => {
expect(getQueryStmtCountHeuristic('SELECT "it\'s;ok" FROM users')).toBe(1)
})
it('should handle double quote inside single quote', () => {
expect(getQueryStmtCountHeuristic('SELECT \'"quoted";value\' FROM users')).toBe(1)
})
it('should handle complex nested scenarios', () => {
expect(
getQueryStmtCountHeuristic('SELECT "a\'b;c", \'d"e;f\', g FROM users; DELETE FROM logs')
).toBe(2)
})
})
describe('line comments', () => {
it('should ignore semicolon in line comment', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 -- comment;here\nFROM users')).toBe(1)
})
it('should handle multiple line comments', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 -- comment;1\n-- comment;2\nFROM users')).toBe(1)
})
it('should handle line comment at end of query', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users -- final;comment')).toBe(1)
})
it('should handle line comment with semicolon at end', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users; -- comment;here')).toBe(1)
})
it('should count statements after line comment', () => {
expect(getQueryStmtCountHeuristic('SELECT 1; -- comment\nSELECT 2')).toBe(2)
})
it('should handle line comment without newline at end (stays in comment state)', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 FROM users -- comment;here')).toBe(1)
})
it('should handle dashes inside string not as comment', () => {
expect(getQueryStmtCountHeuristic("SELECT '--not;comment' FROM users")).toBe(1)
})
it('should handle single dash (not a comment)', () => {
expect(getQueryStmtCountHeuristic('SELECT a-b FROM users')).toBe(1)
})
})
describe('block comments', () => {
it('should ignore semicolon in block comment', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 /* comment;here */ FROM users')).toBe(1)
})
it('should handle multiline block comment', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 /* comment;\nacross;\nlines */ FROM users')).toBe(
1
)
})
it('should handle multiple block comments', () => {
expect(
getQueryStmtCountHeuristic(
'SELECT /* c1;here */ 1, /* c2;here */ 2 FROM /* c3;here */ users'
)
).toBe(1)
})
it('should handle nested-looking block comments', () => {
expect(
getQueryStmtCountHeuristic('SELECT 1 /* outer /* inner;here */ still */ FROM users')
).toBe(1)
})
it('should handle block comment at end', () => {
expect(getQueryStmtCountHeuristic('SELECT * FROM users /* final;comment */')).toBe(1)
})
it('should count statements after block comment', () => {
expect(getQueryStmtCountHeuristic('SELECT 1; /* comment */ SELECT 2')).toBe(2)
})
it('should handle block comment markers in string', () => {
expect(getQueryStmtCountHeuristic("SELECT '/* not;comment */' FROM users")).toBe(1)
})
it('should handle unclosed block comment (stays in comment state)', () => {
expect(getQueryStmtCountHeuristic('SELECT 1 /* unclosed;comment')).toBe(1)
})
it('should handle slash-star inside string not as comment', () => {
expect(getQueryStmtCountHeuristic("SELECT '/*;not comment' FROM users")).toBe(1)
})
})
describe('mixed comments and strings', () => {
it('should handle comment after string', () => {
expect(getQueryStmtCountHeuristic("SELECT 'value;1' -- comment;2\nFROM users")).toBe(1)
})
it('should handle string after comment', () => {
expect(getQueryStmtCountHeuristic("SELECT 1 -- comment\n, 'value;here' FROM users")).toBe(1)
})
it('should handle complex mix', () => {
expect(
getQueryStmtCountHeuristic(
'SELECT "col;1" /* comment;1 */ , \'val;2\' -- comment;2\nFROM users'
)
).toBe(1)
})
it('should handle quotes in comments', () => {
expect(getQueryStmtCountHeuristic("SELECT 1 /* it's;working */ FROM users")).toBe(1)
})
it('should handle comment markers in string', () => {
expect(getQueryStmtCountHeuristic("SELECT '--/*;*/' FROM users -- real;comment")).toBe(1)
})
})
describe('edge cases', () => {
it('should handle only semicolons', () => {
expect(getQueryStmtCountHeuristic(';;;')).toBe(3)
})
it('should handle semicolons with whitespace', () => {
expect(getQueryStmtCountHeuristic(' ; ; ; ')).toBe(3)
})
it('should handle query ending with multiple semicolons', () => {
expect(getQueryStmtCountHeuristic('SELECT 1;;')).toBe(2)
})
it('should handle unclosed single quote (stays in quote state)', () => {
expect(getQueryStmtCountHeuristic("SELECT 'unclosed")).toBe(1)
})
it('should handle unclosed double quote (stays in quote state)', () => {
expect(getQueryStmtCountHeuristic('SELECT "unclosed')).toBe(1)
})
it('should handle unclosed line comment (no newline at end)', () => {
expect(getQueryStmtCountHeuristic('SELECT 1; -- comment;here')).toBe(1)
})
it('should handle all types together', () => {
expect(
getQueryStmtCountHeuristic(
`SELECT "col;1", 'val;2' FROM users WHERE x = 1; -- comment;here
/* block;comment */ UPDATE posts SET title = 'new;title';
DELETE FROM logs -- final;cleanup`
)
).toBe(3)
})
it('should handle empty statements', () => {
expect(getQueryStmtCountHeuristic(';')).toBe(1)
})
it('should handle consecutive escaped quotes at statement boundary', () => {
expect(getQueryStmtCountHeuristic("SELECT '''';")).toBe(1)
})
it('should handle alternating quote types', () => {
expect(getQueryStmtCountHeuristic('SELECT "a", \'b\', "c", \'d\' FROM users')).toBe(1)
})
it('should handle very long string with semicolons', () => {
const longString = ';'.repeat(1000)
expect(getQueryStmtCountHeuristic(`SELECT '${longString}' FROM users`)).toBe(1)
})
it('should handle realistic multiline query', () => {
const query = `
SELECT
id,
name,
"user;email" -- email column
FROM users
WHERE status = 'active;pending'
AND created_at > '2024-01-01';
/* Update user preferences */
UPDATE user_settings
SET theme = 'dark;mode'
WHERE user_id IN (SELECT id FROM users WHERE name LIKE '%test%');
-- Cleanup old data
DELETE FROM logs WHERE timestamp < NOW() - INTERVAL '30 days'
`
expect(getQueryStmtCountHeuristic(query)).toBe(3)
})
})
})