discord9
ea49f8a5c4
feat(flow): make write path faster with shared lock ( #4073 )
...
* feat(WIP): make write faster
* feat: read lock on fast path
* chore: per review
2024-05-31 06:50:22 +00:00
dennis zhuang
24612f62dd
feat: querying from view works ( #3952 )
...
* feat: querying from view works
* feat: use MemoryCatalogProviderList instead of DummyCatalogList
* refactor: revert src/query/src/dummy_catalog.rs
* chore: clean code
* fix: make clippy happy
* fix: toml format
* fix: sqlness
* fix: forgot files
* fix: make sqlness happy
* test: table source, serializer and decoder
* fix: fail to decode plan because of invalid table names
* test: adds more sqlness test for view
* chore: remove unused errors
* fix: comments
* fix: typo
* fix: invalidate view info cache after creating view successfully
* chore: apply suggestion
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: apply suggestion
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: compile error after rebeasing
* chore: style
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: don't export table_name in common-meta
* chore: change ViewInfo::new signature
* docs: leave a TODO for optimize param
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-30 21:45:56 +00:00
discord9
7de336f087
fix(flow): correctness bugs ( #4018 )
...
* fix: optional args of tumble
* fix(WIP): choose
* feat: rename default ts to GREPTIME_TIMESTAMP
* fix: default timestamp name
* fix: reorder write requests
* fix: expire state
* fix: test of tumble
* fix: send buf clear
* fix: ts<start time correct window
* fix: window_start when ts<start_time
* mend
* fix: range begin>range end
* refactor: per reviews
* feat!: ts placeholder rename to __ts_placeholder
* refactor: better condition
* tests(WIP): func sig choose
* tests(WIP): tumble func
* feat: make run_available optional blocking
* tests: tumble transform
* chore: clippy
* fix?: lagged missing data
* fix: flow source break on empty chnl
2024-05-30 03:49:11 +00:00
Ruihang Xia
1de17aec74
feat: change EXPIRE WHEN to EXPIRE AFTER ( #4002 )
...
* feat: change EXPIRE WHEN to EXPIRE AFTER
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change remaining
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename create_if_not_exist to create_if_not_exists
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* parse interval expr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
2024-05-27 04:05:55 +00:00
discord9
533ada70ca
chore: remove a dbg! forget to remove ( #3990 )
...
* chore: remove a dbg! forget to remove
* remove other dbg! and add lint
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 feature
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-20 08:34:47 +00:00
discord9
3477fde0e5
feat(flow): tumble window func ( #3968 )
...
* feat(WIP): tumble window rewrite parser
* tests: tumble func
* feat: add `update_at` column for all flow output
* chore: cleanup per review
* fix: update_at not as time index
* fix: demo tumble
* fix: tests&tumble signature&accept both ts&datetime
* refactor: update_at now ts millis type
* chore: per review advices
2024-05-17 12:10:28 +00:00
discord9
a45017ad71
feat(flow): expire arrange according to time_index type ( #3956 )
...
* feat: render_reduce's arrangement expire after time passed
* feat: set expire when create flow
2024-05-16 11:41:03 +00:00
discord9
0d9e71b653
feat(flow): flow node manager ( #3954 )
...
* feat(flow): flow node manager
feat(flow): render src/sink
feat(flow): flow node manager in standalone
fix?: higher run freq
chore: remove abunant error enum variant
fix: run with higher freq if insert more
chore: fix after rebase
chore: typos
* chore(WIP): per review
* chore: per review
2024-05-16 11:37:14 +00:00
discord9
93f178f3ad
feat(flow): avg func rewrite to sum/count ( #3955 )
...
* feat(WIP): parse avg
* feat: RelationType::apply_mfp no need expr typs
* feat: avg&tests
* fix(WIP): avg eval
* fix: sum ret correct type
* chore: typos
2024-05-16 10:03:56 +00:00
discord9
97eb196699
feat(flow): query table schema&refactor ( #3943 )
...
* feat: get table info
* feat: remove new&unwrap
* chore: per PR advices
* chore: per review
2024-05-15 08:35:12 +00:00
discord9
15d7b9755e
feat(flow): flow worker ( #3934 )
...
* feat: flow worker
* chore: fix after cherry pick
* refactor: error handling
* refactor: error handling
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: merge origin/main
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
2024-05-14 16:09:25 +00:00
discord9
4ca7ac7632
feat(flow): add types for every plan enum variant ( #3938 )
...
* feat: Plan with types
* chore: per review advices
2024-05-14 10:51:37 +00:00
discord9
be1eb4efb7
feat(flow): render source/sink ( #3903 )
...
* feat(flow): render src/sink
* chore: add empty impl
* chore: typos
* refactor: according to review(WIP)
* refactor: reexport df_sbustrait&use to_sub_plan
* fix: add implict location to error enum
* fix: error handling unwrap query_ctx
2024-05-13 11:58:02 +00:00
Ruihang Xia
115c74791d
build(deps): bump snafu to 0.8 ( #3911 )
...
* change Cargo.toml
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle alias in script engine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-10 13:36:25 +00:00
discord9
2d0f493040
feat(flow): render reduce ( #3769 )
...
* feat: render reduce
* fix: distinct input dedup&change per review
* chore: typos
* fix: eval_distinct_core&per review
* chore: typos
* docs: more comment on accums internal
* chore: per review
* chore: rebase to main
2024-04-25 09:35:12 +00:00
Zhenchi
659d34a170
refactor(flow): refine comments and code ( #3785 )
...
* refactor(flow): refine comments and code
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* doc: description of the properties of removed keys
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: `get`'s fast path for cur val
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: discord9 <discord9@163.com >
2024-04-24 07:09:53 +00:00
discord9
d4b814f698
refactor(flow): split render.rs and other minor changes ( #3750 )
...
* refactor: split render.rs
* chore: per review
* chore: per review
* chore: docs explain `apply_updates` behavior
* chore: per review
2024-04-22 09:48:09 +00:00
LFC
314f2704d4
build(deps): update datafusion to latest and arrow to 51.0 ( #3661 )
...
* chore: update datafusion
* update sqlness case of time.sql
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: adjust range query partition
* fix: hisogram incorrect result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: ignore filter pushdown temporarily
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: update limit sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: histogram with wrong distribution
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: update negative ordinal sqlness case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: bump df to cd7a00b
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve conflicts
* ignore test_range_filter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix promql exec panic
* fix "select count(*)" exec error
* re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed
* fix: range query schema error
* update sqlness results
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve conflicts
* update datafusion, again
* fix pyo3 compile error, and update some sqlness results
* update decimal sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: promql literal
* fix udaf tests
* fix filter pushdown sqlness tests
* fix?: test_cast
* fix: rspy test fail due to datafusion `sin` signature change
* rebase main to see if there are any failed tests
* debug ci
* debug ci
* debug ci
* enforce input partition
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* debug ci
* fix ci
* fix ci
* debug ci
* debug ci
* debug ci
* fix sqlness
* feat: do not return error while creating a filter
* chore: remove array from error
* chore: replace todo with unimplemented
* Update src/flow/clippy.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: WUJingdi <taylor-lagrange@qq.com >
Co-authored-by: discord9 <discord9@163.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-18 12:07:18 +00:00
discord9
db329f6c80
feat(flow): transform substrait SELECT&WHERE&GROUP BY to Flow Plan ( #3690 )
...
* feat: transofrm substrait SELECT&WHERE&GROUP BY to Flow Plan
* chore: reexport from common/substrait
* feat: use datafusion Aggr Func to map to Flow aggr func
* chore: remove unwrap&split literal
* refactor: split transform.rs into smaller files
* feat: apply optimize for variadic fn
* refactor: split unit test
* chore: per review
2024-04-12 07:38:42 +00:00
discord9
c00c1d95ee
chore(flow): more comments&lint ( #3680 )
...
* chore: more comments&lint
* chore: per review
* chore: remove abundant dep
2024-04-10 03:31:22 +00:00
discord9
ea9367f371
refactor(flow): func spec api&use Error not EvalError in mfp ( #3657 )
...
* refactor: func's specialization& use Error not EvalError
* docs: some pub item
* chore: typo
* docs: add comments for every pub item
* chore: per review
* chore: per reveiw&derive Copy
* chore: per review&test for binary fn spec
* docs: comment explain how binary func spec works
* chore: minor style change
* fix: Error not EvalError
2024-04-09 02:32:02 +00:00
discord9
28fd0dc276
feat(flow): render map&related tests ( #3581 )
...
* feat: render map&related tests
* chore: license header
* chore: update Cargo.lock&remove unused
* refactor: rename ComputeState to DataflowState
* chore: use org fork
* chore: fix typos
* chore: per review
* chore: more explain to use `VecDeque` in err collector
* chore: typos
* chore: more comment on `Plan::Let`
* chore: typos
* refactor mfp rendering
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: update `now` in closure
* feat: use insert_local
* chore: remove unused
* chore: per review
* chore: fmt comment
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
2024-04-08 11:36:07 +00:00
dimbtp
dd18d8c97b
build(deps): remove some unused dependencies ( #3582 )
...
* build(deps): remove some unused dependencies
* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
discord9
9f020aa414
fix(flow): Arrange get range with batch unaligned ( #3552 )
...
* fix: Arrange get range with batch unaligned
* chore: per review
* refactor: sort at apply_updates
2024-03-22 04:08:37 +00:00
discord9
2c115bc22a
feat(flow): shared in-memory state for dataflow operator ( #3508 )
...
* feat: Arrangement shared state
* feat: arrange&tests
* docs: detailed&tests for get
* chore: license
* refactor: opt out ts expr&tests: internal ts
* docs: remove some TODOs
* feat: use smallvec size of 2
* refactor: per review
* chore: per review
* chore: per review
* chore: remove reduant clone
* feat: return max expire time&docs: more explain cur expire config
2024-03-19 10:03:05 +00:00
discord9
58bd065c6b
feat(flow): plan def ( #3490 )
...
* feat: plan def
* chore: add license
* docs: remove TODO done
* chore: add derive Ord
2024-03-12 10:59:07 +00:00
discord9
7639c227ca
feat(flow): accumlator for aggr func ( #3396 )
...
* feat: Accumlator trait
* feat: add `OrdValue` accum&use enum_dispatch
* test: more accum test
* feat: eval aggr funcs
* chore: refactor test&fmt clippy
* refactor: less verbose
* test: more tests
* refactor: better err handling&use OrdValue for Count
* refactor: ignore null&more tests for error handle
* refactor: OrdValue accum
* chore: extract null check
* refactor: def&use fn signature
* chore: use extra cond with match guard
* chore: per review
2024-03-12 02:09:27 +00:00
gcmutator
21ff3620be
chore: remove repetitive words ( #3469 )
...
remove repetitive words
Signed-off-by: gcmutator <329964069@qq.com >
2024-03-09 04:18:47 +00:00
discord9
dbb1ce1a9b
feat(flow): impl for MapFilterProject ( #3359 )
...
* feat: mfp impls
* fix: after rebase
* test: temporal filter mfp
* refactor: more comments&test
* test: permute
* fix: check input len when eval
* refactor: err handle&docs: more explain graph
* docs: better flowchart map,filter,project
* refactor: visit_* falliable
* chore: better temp lint allow
* fix: permute partially
* chore: remove duplicated checks
* docs: more explain&tests for clarity
* refactor: use ensure! instead
2024-02-27 08:13:55 +00:00
SteveLauC
e9a2b0a9ee
chore: use workspace-wide lints ( #3352 )
...
* chore: use workspace-wide lints
* respond to review
2024-02-22 01:01:10 +00:00
discord9
860b1e9d9e
feat(flow): impl ScalarExpr&Scalar Function ( #3283 )
...
* feat: impl for ScalarExpr
* feat: plain functions
* refactor: simpler trait bound&tests
* chore: remove unused imports
* chore: fmt
* refactor: early ret on first error
* refactor: remove abunant match arm
* chore: per review
* doc: `support` fn
* chore: per review more
* chore: more per review
* fix: extract_bound
* chore: per review
* refactor: reduce nest
2024-02-21 12:53:16 +00:00
discord9
c62c67cf18
feat: Basic Definitions for Expression&Functions for Dataflow ( #3267 )
...
* added expression&func
* fix: EvalError derive&imports
* chore: add header
* feat: variadic func
* chore: minor adjust
* feat: accum
* feat: use accum for eval func
* feat: montonic min/max as accumulative
* feat: support min/max Date&DateTime
* chore: fix compile error&add test(WIP)
* test: sum, count, min, max
* feat: remove trait impl for EvalError
* chore: remove all impl retain only type definitions
* refactor: nest datatypes errors
* fix: remove `.build()`
* fix: not derive Clone
* docs: add comment for types
* feat: more func&remove `CurrentDatabase`
2024-02-05 07:49:34 +00:00
discord9
0a9361a63c
feat: basic types for Dataflow Framework ( #3186 )
...
* feat: basic types for Dataflow Framework
* docs: license header
* chore: add name in todo, remove deprecated code
* chore: typo
* test: linear&repr unit test
* refactor: avoid mod.rs
* feat: Relation Type
* feat: unmat funcs
* feat: simple temporal filter(sliding window)
* refactor: impl Snafu for EvalError
* feat: cast as type
* feat: temporal filter
* feat: time index in RelationType
* refactor: move EvalError to expr
* refactor: error handling for func
* chore: fmt&comment
* make EvalError pub again
* refactor: move ScalarExpr to scalar.rs
* refactor: remove mod.rs for relation
* chore: slim down PR size
* chore: license header
* chore: per review
* chore: more fix per review
* chore: even more fix per review
* chore: fmt
* chore: fmt
* feat: impl From/Into ProtoRow instead
* chore: use cast not cast_with_opt&`Key` struct
* chore: new_unchecked
* feat: `Key::subset_of` method
* chore: toml in order
2024-01-30 07:48:22 +00:00
