Ruihang Xia
cde5a36f5e
feat: precise filter for mito parquet reader ( #3178 )
...
* impl SimpleFilterEvaluator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* time index and field filter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finish parquet filter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove empty Batch
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix fmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update metric
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use projected schema from batch
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* correct naming
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unnecessary error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-18 06:59:48 +00:00
niebayes
63205907fb
refactor: introduce common-wal to aggregate wal stuff ( #3171 )
...
* refactor: aggregate wal configs
* refactor: move wal options to common-wal
* chore: slim Cargo.toml
* fix: add missing crates
* fix: format
* chore: update comments
* chore: add testing feature gate for test_util
* fix: apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix: apply suggestions from code review
* fix: compiling
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-18 03:49:37 +00:00
Wei
3d7d2fdb4a
feat: auto config cache size according to memory size ( #3165 )
...
* feat: auto config cache and buffer size according to mem size
* feat: utils
* refactor: add util function to common config
* refactor: check cgroups
* refactor: code
* fix: test
* fix: test
* chore: cr comment
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* chore: remove default comment
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
2024-01-17 14:35:35 +00:00
LFC
3cfd60e139
refactor: expose region edit in mito engine ( #3179 )
...
* refactor: expose region edit in mito engine
* feat: add a method for editing region directly
* fix: resolve PR comments
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: resolve PR comments
* fix: resolve PR comments
* fix: resolve PR comments
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-17 14:25:08 +00:00
shuiyisong
a29b9f71be
chore: carry metrics in flight metadata from datanode to frontend ( #3113 )
...
* chore: carry metrics in flight metadata from datanode to frontend
* chore: fix typo
* fix: ignore metric flight message on client
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: add cr comment
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: add cr comment
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update proto
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-17 11:38:03 +00:00
shuiyisong
ae160c2def
fix: change back GREPTIME_DB_HEADER_NAME header key name ( #3184 )
...
fix: change back dbname header key
2024-01-17 09:30:32 +00:00
Ruihang Xia
fbd0197794
refactor: remove TableEngine ( #3181 )
...
* refactor: remove TableEngine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/table/src/table_reference.rs
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
2024-01-17 04:18:37 +00:00
dennis zhuang
204b9433b8
feat: adds date_format function ( #3167 )
...
* feat: adds date_format function
* fix: compile error
* chore: use system timezone for FunctionContext and EvalContext
* test: as_formatted_string
* test: sqlness test
* chore: rename function
2024-01-17 03:24:40 +00:00
JeremyHi
c6c4ea5e64
feat: tables stream with CatalogManager ( #3180 )
...
* feat: add tables for CatalogManager
* feat: replace table with tables
* Update src/catalog/src/information_schema/columns.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/catalog/src/information_schema/columns.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/catalog/src/information_schema/tables.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/catalog/src/information_schema/tables.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* feat: tables for MemoryCatalogManager
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-01-17 02:31:15 +00:00
JeremyHi
c2edaffa5c
feat: let tables API return a stream ( #3170 )
2024-01-15 12:36:39 +00:00
Wei
07de65d2ac
test: engine with write cache ( #3163 )
...
* feat: write cache test for engine
* chore: unused
* chore: comment
* refactor: super to crate
* chore: cr comment
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: clippy
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 10:02:53 +00:00
Ning Sun
1294d6f6e1
feat: upgrade pgwire to 0.19 ( #3157 )
...
* feat: upgrade pgwire to 0.19
* fix: update pgwire to 0.19.1
2024-01-15 09:14:08 +00:00
Zhenchi
6f07d69155
feat(mito): enable inverted index ( #3158 )
...
* feat(mito): enable inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* accidentally resolved the incorrect filtering issue within the Metric Engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
* Update src/mito2/src/test_util/scheduler_util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: format -> join_dir
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move intermediate_manager from arg of write_and_upload_sst to field of WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: add IndexerBuidler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 09:08:07 +00:00
WU Jingdi
816d94892c
feat: support HTTP&gRPC&pg set timezone ( #3125 )
...
* feat: support HTTP&gRPC&pg set timezone
* chore: fix code advice
* chore: fix code advice
2024-01-15 06:29:31 +00:00
LFC
93f28c2a37
refactor: make grpc service able to be added dynamically ( #3160 )
2024-01-15 04:33:27 +00:00
Eugene Tolbakov
ca4d690424
feat: add modulo function ( #3147 )
...
* feat: add modulo function
* fix: address CR feedback
2024-01-13 00:24:25 +00:00
Weny Xu
75975adcb6
fix: fix tests failed on windows ( #3155 )
...
* fix: fix tests failed on windows
* feat: add comments
* Update src/object-store/src/util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-12 09:12:11 +00:00
Ruihang Xia
527e523a38
fix: handle non-identical time index and field column in PromQL set operation ( #3145 )
...
* handle different field columns
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix and/unless on different time index
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-12 06:27:03 +00:00
Weny Xu
bf88b3b4a0
fix: fix store all wal options ( #3149 )
...
* fix: fix store all wal options
* fix: incorrect updating DatanodeTable value
2024-01-12 04:48:14 +00:00
Weny Xu
bf96ce3049
fix: print detailed error ( #3146 )
2024-01-12 04:02:32 +00:00
Weny Xu
430ffe0e28
fix(kafka): overwrite the EntryId with Offset while consuming records ( #3148 )
...
* fix(kafka): overwrite the EntryId with Offset while consuming the KafkaRecords
* fix: temporarily workaround of incorrect entry Id
2024-01-12 03:46:17 +00:00
Zhenchi
c1190bae7b
feat(mito): support write cache for index file ( #3144 )
...
* feat(mito): support write cache for index file
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: merge main
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-01-12 02:40:56 +00:00
Ruihang Xia
0882da4d01
feat: support PromQL operations over the same metric ( #3124 )
...
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update ut cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove deadcode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-11 23:07:17 +00:00
Wei
8ec1e42754
feat: read data from write cache ( #3128 )
...
* feat: read from write cache
* chore: add read ranges test
* fix: use get instead of contains_key
* chore: clippy
* chore: cr comment
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: with_label_values
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-11 12:06:28 +00:00
Ruihang Xia
09b3c7029b
feat: handle drop request for metric table ( #3136 )
...
* handle drop request
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* adjust procedure manager
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add create table sqlness test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* insert/query metric table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* address CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/common/meta/src/kv_backend.rs
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reuse region option for metadata region
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tweak variable name
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-11 09:38:43 +00:00
Zhenchi
fd8fb641fd
feat(parquet): introduce inverted index applier to reader ( #3130 )
...
* feat(parquet): introduce inverted index applier to reader
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: purger removes index file
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add TODO for escape route
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add TODO for escape route
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* Update src/mito2/src/sst/parquet/reader.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* feat: min-max index to prune row groups filtered by inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: file_meta.inverted_index_available -> file_meta.available_indexes
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add TODO for leveraging WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix fmt
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: misset available indexes
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add index file size
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: use smallvec to reduce heap allocation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: add index size to disk usage
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-11 08:04:59 +00:00
Weny Xu
312e8e824e
fix: save code in debug_assert! ( #3137 )
...
fix: save code in debug_assert!
2024-01-11 06:07:08 +00:00
Wei
29a7f301df
feat: write and upload sst ( #3106 )
...
* feat: write and upload sst file
* refactor: unit test
* cr comment
* chore: typos
* chore: cr comment
* chore: conflict
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* chore: fmt
* chore: style
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-11 02:34:16 +00:00
LFC
51a3fbc7bf
refactor: change how frontend grpc services are orchestrated ( #3134 )
2024-01-11 02:26:44 +00:00
Lanqing Yang
d521bc9dc5
chore: impl KvBackend for MetaPeerClient ( #3076 )
2024-01-10 14:16:03 +00:00
Weny Xu
7fad4e8356
fix: incorrect parsing broker_endpoints env variable ( #3135 )
2024-01-10 13:59:49 +00:00
Ning Sun
b6033f62cd
refactor: implement version as built-in function and use fixed mysql version ( #3133 )
...
* refactor: implement version as built-in function
* test: add sqlness test for version()
2024-01-10 11:04:18 +00:00
dennis zhuang
fd3f23ea15
feat: adds runtime_metrics ( #3127 )
...
* feat: adds runtime_metrics
* fix: comment
* feat: refactor metrics table
* chore: ensure build_info and runtime_metrics only avaiable in greptime catalog
* feat: adds timestamp column
2024-01-10 10:51:30 +00:00
niebayes
1b0e39a7f2
chore: stop exposing num_partitions ( #3132 )
2024-01-10 10:45:18 +00:00
Weny Xu
3ab370265a
feat: expose the region migration replay_timeout argument ( #3129 )
...
* feat: expose region migration args
* fix: fix ci
2024-01-10 09:47:59 +00:00
Weny Xu
ec8266b969
refactor: refactor the locks in the procedure ( #3126 )
...
* feat: add lock key
* refactor: procedure lock keys
* chore: apply suggestions from CR
2024-01-10 09:46:39 +00:00
Zhenchi
490312bf57
fix: unstable time record test ( #3131 )
...
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-01-10 09:41:52 +00:00
Ning Sun
1fc168bf6a
feat: update our cross schema check to cross catalog ( #3123 )
2024-01-09 09:38:48 +00:00
Zhenchi
db98484796
feat(inverted_index): introduce SstIndexCreator ( #3107 )
...
* feat(inverted_index): introduce SstIndexCreator
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: tiny polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: distinguish intermediate store and index store
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: move comment as doc comment
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: column id as index name
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-01-09 09:24:16 +00:00
Ruihang Xia
7d0d2163d2
fix: expose unsupported datatype error on mysql protocol ( #3121 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-09 09:13:53 +00:00
niebayes
a0a31c8acc
chore(remote_wal): remove topic alias ( #3120 )
...
chore: remove topic alias
2024-01-09 07:35:02 +00:00
tison
0db1861452
chore(python): Print Python interpreter version ( #3118 )
...
* chore(pyo3_backend): Print bundle Python interpreter version
Signed-off-by: tison <wander4096@gmail.com >
* print RustPython interpreter version on init
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-01-09 07:04:23 +00:00
Wei
225ae953d1
feat: add parquet metadata to cache ( #3097 )
...
* feat: parquet metadata to sst meta cache
* chore: clippy
* refactor: move code to access_layer
* chore: clone()
2024-01-09 07:00:42 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
fys
6e860bc0fd
feat: support grpc for otlp trace and metrics ( #3105 )
...
* feat: add grpc support for otlp trace and metrics
* cr: add some comment
* fix: ut
* fix: cr
2024-01-09 05:01:48 +00:00
Yingwen
8bd4a36136
feat(mito): Init the write cache in datanode ( #3100 )
...
* feat: add builder to build cache manager
* refactor: make MitoEngine::new async
* refactor: refactor object store creation
* refactor: add helper fn to attaches layers
* feat: fn to build fs store
* feat: add write cache to engine
* feat: config write cache
* style: fix clippy
* test: fix test
* feat: add warning
* chore: add experimental prefix to configs
* test: fix config test
* test: test weighted size
* feat: add switch to enable write cache
* fix: update cache stats by using get
* style: use then
2024-01-09 04:40:22 +00:00
Ruihang Xia
af0c4c068a
feat: support PromQL function vector ( #3036 )
...
* produce vector plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* work with OR
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply review sugg
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move common const strings to common_query
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add comment for GREPTIME_COUNT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-09 03:44:00 +00:00
tison
316d843482
feat: support CSV format in sql HTTP API ( #3062 )
...
* chore: fix typo
Signed-off-by: tison <wander4096@gmail.com >
* add csv format
Signed-off-by: tison <wander4096@gmail.com >
* flatten response
Signed-off-by: tison <wander4096@gmail.com >
* more flatten response
Signed-off-by: tison <wander4096@gmail.com >
* add CSV format
Signed-off-by: tison <wander4096@gmail.com >
* format InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format ErrorResponse
Signed-off-by: tison <wander4096@gmail.com >
* propagate ErrorResponse to InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format GreptimedbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format CsvResponse
Signed-off-by: tison <wander4096@gmail.com >
* impl IntoResponse for QueryResponse
Signed-off-by: tison <wander4096@gmail.com >
* promql
Signed-off-by: tison <wander4096@gmail.com >
* sql
Signed-off-by: tison <wander4096@gmail.com >
* compile
Signed-off-by: tison <wander4096@gmail.com >
* fixup aide
Signed-off-by: tison <wander4096@gmail.com >
* clear debt
Signed-off-by: tison <wander4096@gmail.com >
* fixup UT test_recordbatches_conversion
Signed-off-by: tison <wander4096@gmail.com >
* fixup IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup more IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup test-integration cases
Signed-off-by: tison <wander4096@gmail.com >
* update comment
Signed-off-by: tison <wander4096@gmail.com >
* fixup deserialize and most query < 1ms
Signed-off-by: tison <wander4096@gmail.com >
* fixup auth tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup and align X-GreptimeDB headers
Signed-off-by: tison <wander4096@gmail.com >
* fixup compile
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-01-08 10:54:27 +00:00
niebayes
8c58d3f85b
test(remote_wal): add unit tests for kafka remote wal ( #2993 )
...
* test: add unit tests
* feat: introduce kafka runtime backed by testcontainers
* test: add test for kafka runtime
* fix: format
* chore: make kafka image ready to be used
* feat: add entry builder
* tmp
* test: add unit tests for client manager
* test: add some unit tests for kafka log store
* chore: resolve some todos
* chore: resolve some todos
* test: add unit tests for kafka log store
* chore: add deprecate develop branch warning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tmp: ready to move unit tests to an indie dir
* test: update unit tests for client manager
* test: add unit tests for meta srv remote wal
* fix: license
* fix: test
* refactor: kafka image
* doc: add doc example for kafka image
* chore: migrate kafka image to an indie PR
* fix: CR
* fix: CR
* fix: test
* fix: CR
* fix: update Cargo.toml
* fix: CR
* feat: skip test if no endpoints env
* fix: format
* test: rewrite parallel test with barrier
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-08 10:48:11 +00:00
