discord9
0185a65905
feat(flow): refill flow task def(Part 2) ( #5317 )
...
* feat: refill task def
* chore: per review
* chore: after rebase
2025-01-15 08:48:54 +00:00
discord9
66f63ae981
feat: more than one flow workers ( #5315 )
...
* feat: more workers
* feat: use round robin
* refactor: per review
* refactor: per bot review
* chore: per review
* docs: example
* docs: update config.md
* docs: update
* chore: per review
* refactor: set workers to cpu/2.max(1)
* fix: flow config in standalone mode
* test: fix config test
* docs: update docs&opt name
* chore: update config.md
* refactor: per review, sanitize at top
* chore: per review
* chore: config.md
2025-01-14 12:28:18 +00:00
discord9
95b20592ac
fix: handle insert default value ( #5307 )
...
* fix: handle flow inserts with default values
* test: sqlness
* chore: typo
* chore: newline
* feat(WIP): impure default filler
* feat: fill impure default values
* test: add test for default fill impure
* feat: check for impure
* fix: also handle stmt to region
* refactor: per review
* refactor: per review
* chore: rebase fix
* chore: clippy
* chore: per review
2025-01-14 09:06:53 +00:00
discord9
0ee41339aa
feat(flow): flow refill state (Part 1) ( #5295 )
...
* feat(flow): (Part 1) refill utils
* chore: after rebase fix
* chore: more rebase
* rm refill.rs to reduce pr size
* chore: simpler args
* refactor: per review
* docs: more explain for instant requests
* refactor: per review
2025-01-08 06:45:56 +00:00
yihong
369b59c84a
fix: drop unused dep using udeps to minial the size ( #5301 )
...
* fix: drop unused dep using udeps to minial the size
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
* fix: adress comments fix the problem
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
2025-01-08 06:36:22 +00:00
discord9
353c8230db
refactor: flow replace check&better error msg ( #5277 )
...
* chore: better error msg
* chore eof newline
* refactor: move replace check to flow worker
* chore: add ctx to insert flow failure
* chore: Update src/flow/src/adapter/flownode_impl.rs
* test: add order by for deterministic
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2025-01-03 10:59:58 +00:00
discord9
d20b592fe8
fix: flow handle reordered inserts ( #5275 )
...
* fix: reorder correct schema
* tests: reorder insert handled correctly
* chore: rm unused
* refactor: per review
* chore: more comment
* chore: per review
2025-01-03 06:25:39 +00:00
discord9
4d6fe31fff
fix(flow): flow's table schema cache ( #5251 )
...
* fix: flow schema cache
* refactor: location for `to_meta_err`
* chore: endfile emptyline
* chore: review(partially)
* chore: per review
* refactor: per review
* refactor: per review
2025-01-02 10:33:23 +00:00
discord9
d18c8b5e16
chore: typo ( #5265 )
...
* fix: a typo
* chore: even more typos
2025-01-02 03:17:53 +00:00
discord9
5ba8bd09fb
fix: flow compare null values ( #5234 )
...
* fix: flow compare null values
* fix: fix again ck ty before cmp
* chore: rm comment
* fix: handle null
* chore: typo
* docs: update comment
* refactor: per review
* tests: more sqlness
* tests: sqlness not show create table
2024-12-25 15:31:27 +00:00
discord9
abf34b845c
feat(flow): check sink table mismatch on flow creation ( #5112 )
...
* tests: more mismatch errors
* feat: check sink table schema if exists&prompt nice err msg
* chore: rm unused variant
* chore: fmt
* chore: cargo clippy
* feat: check schema on create
* feat: better err msg when mismatch
* tests: fix a schema mismatch
* todo: create sink table
* feat: create sink table
* fix: find time index
* tests: auto created sink table
* fix: remove empty keys
* refactor: per review
* chore: fmt
* test: sqlness
* chore: after rebase
2024-12-25 13:42:37 +00:00
Ruihang Xia
4051be4214
feat: add some critical metrics to flownode ( #5235 )
...
* feat: add some critical metrics to flownode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-12-25 10:57:21 +00:00
discord9
a4d61bcaf1
fix(flow): batch builder with type ( #5195 )
...
* fix: typed builder
* chore: clippy
* chore: rename
* fix: unit tests
* refactor: per review
2024-12-19 09:16:56 +00:00
discord9
2d6f63a504
feat: show flow's mem usage in INFORMATION_SCHEMA.FLOWS ( #4890 )
...
* feat: add flow mem size to sys table
* chore: rm dup def
* chore: remove unused variant
* chore: minor refactor
* refactor: per review
2024-12-19 08:24:04 +00:00
Ruihang Xia
548e1988ab
refactor: remove unused symbols ( #5193 )
...
chore: remove unused symbols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-12-18 11:24:43 +00:00
shuiyisong
9d7fea902e
chore: remove unused dep ( #5163 )
...
* chore: remove unused dep
* chore: remove more unused dep
2024-12-16 06:17:27 +00:00
discord9
d931389a4c
fix(flow): minor fix about count(*)&sink keyword ( #5061 )
...
* fix: SiNk
* fix: sink&count(*)
* tests: SiNk
* refactor: per review
2024-11-29 03:06:27 +00:00
discord9
4d8fe29ea8
feat: CREATE OR REPLACE FLOW ( #5001 )
...
* feat: Replace flow
* refactor: better show create flow&tests: better check
* tests: sqlness result update
* tests: unit test for update
* refactor: cmp with raw bytes
* refactor: rename
* refactor: per review
2024-11-19 08:44:57 +00:00
discord9
a70b4d7eba
chore: update greptime-proto to e1070a ( #4992 )
...
* chore: update protot to e1070a
* fix: update proto usage
2024-11-14 11:59:00 +00:00
discord9
35898f0b2e
test: more sqlness tests for flow ( #4988 )
...
* tests: more flow testcase
* tests(WIP): more tests
* tests: more flow tests
* test: wired regex for sqlness
* refactor: put blog&example to two files
2024-11-14 07:40:14 +00:00
LFC
2e9737c01d
refactor: pass LogicalPlan to promql execution interceptor ( #4937 )
2024-11-05 08:49:05 +00:00
dennis zhuang
7a4276c24a
fix: typo ( #4931 )
...
fix/database-base-ttl:
Fix typos in comments and function names across multiple modules
- Correct spelling of 'parallelism' in region_server, engine, and scan_region modules
- Amend typo in TODO comment from 'persisent' to 'persistent' in server module
- Update incorrect test query from 'versiona' to 'version' in federated module tests
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2024-11-04 01:56:11 +00:00
LFC
fd8eba36a8
refactor: make use of the "pre_execute" in sql execution interceptor ( #4875 )
...
* feat: dynamic definition of plugin options
* rebase
* revert
* fix ci
2024-10-30 09:16:46 +00:00
Lei, HUANG
e328c7067c
chore: udapte Rust toolchain to 2024-10-19 ( #4857 )
...
* update rust toolchain
* change toolchain to 2024-10-17
* fix: clippy
* fix: ut
* bump shadow-rs
* fix: use nightly-2024-10-19
* fix: clippy
* chore/udapte-toolchain-2024-10-17: Update DEV_BUILDER_IMAGE_TAG to 2024-10-19-a5c00e85-20241024184445 in Makefile
2024-10-25 00:23:32 +00:00
discord9
ff38abde2e
chore: better column schema check for flow ( #4855 )
...
* chore: better column schema check for flow
* chore: better msg
* tests: clean up after tests
* chore: better msg
* chore: per review
* tests: sqlness
2024-10-24 09:43:32 +00:00
Yingwen
2f2b4b306c
feat!: implement interval type by multiple structs ( #4772 )
...
* define structs and methods
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: re-implement interval types in time crate
* feat: use new
* feat: interval value
* feat: query crate interval
* feat: pg and mysql interval
* chore: remove unused imports
* chore: remove commented codes
* feat: make flow compile but may not work
* feat: flow datetime
* test: fix some tests
* test: fix some flow tests(WIP)
* chore: some fix test&docs
* fix: change interval order
* chore: remove unused codes
* chore: fix cilppy
* chore: now signature change
* chore: remove todo
* feat: update error message
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: discord9 <discord9@163.com >
2024-10-14 03:09:03 +00:00
zyy17
e39a9e6feb
feat: add StatementStatistics for slow query logging implementation ( #4719 )
...
* feat: log slow query
* feat: log slow query for sql
* refactor: add slow query logging options
* ci: fix errors
* feat: add StatementStatistics
* chore: revert modification of servers crate
* docs: update config docs
* fix: clippy errors
2024-09-30 03:26:50 +00:00
Ruihang Xia
f5cf25b0db
refactor: remove DfPlan wrapper ( #4733 )
...
* refactor: remove DfPlan wrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test assertion
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-19 12:29:33 +00:00
discord9
a3d567f0c9
perf(flow): use batch mode for flow ( #4599 )
...
* generic bundle trait
* feat: impl get/let
* fix: drop batch
* test: tumble batch
* feat: use batch eval flow
* fix: div use arrow::div not mul
* perf: not append batch
* perf: use bool mask for reduce
* perf: tiny opt
* perf: refactor slow path
* feat: opt if then
* fix: WIP
* perf: if then
* chore: use trace instead
* fix: reduce missing non-first batch
* perf: flow if then using interleave
* docs: add TODO
* perf: remove unnecessary eq
* chore: remove unused import
* fix: run_available no longer loop forever
* feat: blocking on high input buf
* chore: increase threhold
* chore: after rebase
* chore: per review
* chore: per review
* fix: allow empty values in reduce&test
* tests: more flow doc example tests
* chore: per review
* chore: per review
2024-09-11 03:31:52 +00:00
Ruihang Xia
93f202694c
refactor: remove unused error variants ( #4666 )
...
* add python script
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix all negative cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* setup CI
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add license header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-03 13:19:38 +00:00
discord9
8c8499ce53
perf(flow): Map&Reduce Operator use batch to reduce alloc ( #4567 )
...
* feat: partial impl mfp
* feat: eval batch inner
* chore: fmt
* feat: mfp eval_batch
* WIP
* feat: Collection generic over row&Batch
* feat: render source batch
* chore: chore
* feat: render mfp batch
* feat: render reduce batch(WIP)
* feat(WIP): render reduce
* feat: reduce batch
* feat: render sink batch
* feat: render constant batch
* chore: error handling& mfp batch test
* test: mfp batch
* chore: rm import
* test: render reduce batch
* chore: add TODO
* chore: per bot review
* refactor: per review
* chore: cmt
* chore: rename
* docs: update no panic
2024-08-29 07:28:13 +00:00
discord9
9f9c1dab60
feat(flow): use DataFusion's optimizer ( #4489 )
...
* feat: use datafusion optimization
refactor: mv `sql_to_flow_plan` elsewhere
feat(WIP): use df optimization
WIP analyzer rule
feat(WIP): avg expander
fix: transform avg expander
fix: avg expand
feat: names from substrait
fix: avg rewrite
test: update `test_avg`&`test_avg_group_by`
test: fix `test_sum`
test: fix some tests
chore: remove unused flow plan transform
feat: tumble expander
test: update tests
* chore: clippy
* fix: tumble lose `group expr`
* test: sqlness test update
* test: rm unused cast
* test: simplify sqlness
* refactor: per review
* chore: after rebase
* fix: remove a outdated test
* test: add comment
* fix: report error when not literal
* chore: update sqlness test after rebase
* refactor: per review
2024-08-29 02:52:00 +00:00
Weny Xu
25cd61b310
chore: upgrade toolchain to nightly-2024-08-07 ( #4549 )
...
* chore: upgrade toolchain to `nightly-2024-08-07`
* chore(ci): upgrade toolchain
* fix: fix unit test
2024-08-22 11:02:18 +00:00
discord9
2c3fccb516
feat(flow): add eval_batch for ScalarExpr ( #4551 )
...
* refactor: better perf flow
* feat(WIP): batching proc
* feat: UnaryFunc::eval_batch untested
* feat: BinaryFunc::eval_batch untested
* feat: VariadicFunc::eval_batch un tested
* feat: literal eval_batch
* refactor: move DfScalarFunc to separate file
* chore: remove unused imports
* feat: eval_batch df func&ifthen
* chore: remove unused file
* refactor: use Batch type
* chore: remove unused
* chore: remove a done TODO
* refactor: per review
* chore: import
* refactor: eval_batch if then
* chore: typo
2024-08-14 11:29:30 +00:00
discord9
cbb06cd0c6
feat(flow): add some metrics ( #4539 )
...
* feat: add some metrics
* fix: tmp rate limiter
* feat: add task count metrics
* refactor: use bounded channel anyway
* refactor: better metrics
2024-08-14 03:23:49 +00:00
discord9
e6cc4df8c8
feat: flow recreate on reboot ( #4509 )
...
* feat: flow reboot clean
* refactor: per review
* refactor: per review
* test: sqlness flow reboot
2024-08-05 13:57:48 +00:00
Ruihang Xia
7daf24c47f
feat: remove dedicated runtime for grpc, mysql and pg protocols ( #4436 )
...
* feat: remove dedicated runtime for grpc, mysql and pg protocols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove other runtimes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* spawn compact task into compact_runtime
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refine naming
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/servers/tests/mysql/mysql_server_test.rs
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* turnoff fuzz test matrix fail fast option
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update rt config for ci tests
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-07-30 06:17:58 +00:00
discord9
021ec7b6ac
feat(flow): flush_flow function ( #4416 )
...
* refactor: df err variant
* WIP
* chore: update proto version
* chore: revert mistaken rust-toolchain
* feat(WIP): added FlowService to QueryEngine
* refactor: move flow service to operator
* refactor: flush use flow name not id
* refactor: use full path in macro
* feat: flush flow
* feat: impl flush flow
* chore: remove unused
* chore: meaninful response
* chore: remove unused
* chore: clippy
* fix: flush_flow with proper blocking
* test: sqlness tests added back for flow
* test: better predicate for flush_flow
* refactor: rwlock
* fix: flush lock
* fix: flush lock write then drop
* test: add a new flow sqlness test
* fix: sqlness testcase
* chore: style
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-07-26 23:04:13 +00:00
discord9
9fa9156bde
feat: FLOWS table in information_schema&SHOW FLOWS ( #4386 )
...
* feat(WIP): flow info table
refactor: better err handling&log
feat: add flow metadata to info schema provider
feat(WIP): info_schema.flows
feat: info_schema.flows table
* fix: err after rebase
* fix: wrong comparsion op
* feat: SHOW FLOWS&tests
* refactor: per review
* chore: unused
* refactor: json error
* chore: per review
* test: sqlness
* chore: rm inline error
* refactor: per review
2024-07-19 09:29:36 +00:00
discord9
2b912d93fb
feat: flow perf&fix df func call ( #4347 )
...
* feat: flow perf&fix df func call
feat: make source sender `send` non-blocking
feat: better control of flow worker freq
feat: support transform interval
fix: const folding df func args&tests
tests: update cast const fold
chore: adjust flow work's freq
refactor: batch split
feat: adaptive run freq flow worker&check for errors
chore: better debug log
* refactor: per review
* chore: per zc's review
* chore: per bot review
* chore: remove some `TODO` completed
* docs: add comments for a test
2024-07-15 09:20:04 +00:00
discord9
1ddf19d886
feat: flownode use Inserter to write to database ( #4323 )
...
* feat: use `Inserter` as Frontend
* fix: enable procedure in flownode
* docs: remove `frontend_addr` opts
* chore: rm fe addr in test runner
* refactor: int test also use inserter invoker
* feat: flow shutdown&refactor: remove `Frontendinvoker`
* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`
* refactor: per review
* refactor: remove a layer of box
* fix: standalone use `node_manager`
* fix: remove a `Arc` cycle
2024-07-09 10:44:22 +00:00
Zhenchi
00308218b3
feat(fulltext_index): allow enable full-text index in SQL and gRPC way ( #4310 )
...
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typo
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: test_fulltext_intm_path
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: explicitly build column options
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: fix error msg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-08 20:18:48 +00:00
discord9
09fff24ac4
feat: make flow distributed work&tests ( #4256 )
...
feat: flownode frontend client&test
feat: Frontend Client
feat: set frontend invoker for flownode
feat: set frontend invoker for flownode
chore: test script
WIP: test flow distributed
feat: hard coded demo
docs: flownode example toml
feat: add flownode support in runner
docs: comments for node
chore: after rebase
docs: add a todo
tests: move flow tests to common
fix: flownode sqlness dist test
chore: per review
docs: make
fix: make doc
2024-07-05 14:46:44 +00:00
discord9
c21e969329
fix: call df_func with literal ( #4265 )
...
* fix: call df_func with literal
* chore: rm dbg log forget to remove
2024-07-05 06:21:22 +00:00
Jeremyhi
ddf2e6a3c0
feat: provide a simple way to create metaclient ( #4257 )
...
* feat: provide a simple way to create metaclient
* chore: minor refactor using metaclient
* chore: minor refactor using metaclient
2024-07-03 08:11:55 +00:00
discord9
f035a7c79c
feat: flow cli for distributed ( #4226 )
...
* feat(WIP): add FlownodeInstance for flow cli
* feat(WIP): cli
* feat: add merge opts func
* refactor: move server&error to src dir
* feat: flownode cli build
* feat: add `flownode` subcmd to cli
* refactor: per review
* refactor!: BREAKING remove alias `metasrv-addr`
* chore: after rebase
* feat: cache invalide flownode cache
* chore: small refactor per review
* chore: fix a typo
* feat!: revert breaking change
* chore: per review
* refactor: not accept `metasrv-addr` only for flownode
2024-07-01 09:56:15 +00:00
Jeremyhi
214fd38f69
feat: add build info for flow heartbeat task ( #4228 )
...
* chore: refactor load region stats
* feat: add build info for flow heartbeat
2024-07-01 03:19:25 +00:00
discord9
b6585e3581
refactor(flow): make from_substrait_* async& worker handle refactor ( #4210 )
...
* refactor: use oneshot to receive result
* refactor: make from_substrait_* async
* refacrot: remove serde for plan&expr
2024-06-27 17:17:46 +00:00
dennis zhuang
a779cb36ec
fix: wrong frontend registration address ( #4199 )
...
* fix: frontend registration address is wrong, #4186
* fix: license header
* chore: adds hostname to frontend grpc
* fix: forgot run make config-docs
* chore: warn when using bind_addr
* fix: flow node heartbeat carrying address
2024-06-26 06:13:07 +00:00
discord9
a44fe627ce
feat: heartbeat task&peer lookup in proc ( #4179 )
...
* feat: herat beat task
* feat: use real flow peer allocator when building
* feat: add peer look up in ddl context
* fix: drop flow test
* refactor: per review(WIP)
* refactor: not check if is alive
* refactor: per review
* refactor: remove useless `reset`
* refactor: per bot advices
* refactor: alive peer
* chore: bot review
2024-06-24 15:06:33 +00:00
