yuanbohan
44280f7c9d
feat(otlp): initial OTLP trace support ( #2627 )
...
* feat: otlp tracing framework via http
* feat: otlp trace transformer plugin
* feat: successfully write traces into db
* chore: plugin to parse request
* test: helper functions
* feat: parse_request_to_spans function
* chore: remove implicite calling parse in PraceParser
* chore: fix clippy
* chore: add TODO marker for span fields
* refactor TraceParser trait
* refactor TraceParser trait
* table_name method in OTLP TraceParser trait
* fix: approximate row, column count
* chore: function signature without row
* chore: do not clone by moving span.kind upper
* docs for parse and to_grpc_insert_requests
---------
Co-authored-by: fys <fengys1996@gmail.com >
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
2023-10-23 06:37:43 +00:00
Baasit
346b57cf10
feat: row protocol support for opentsdb ( #2623 )
...
* feat: opentsdb row protocol
* fix: added commnets for num of rows and failure if output is not of affecetd rows
* fix: added extra 1 to number of columns
* fix: avoided cloning datapoints, took ownership instead
* fix: avoided cloning datapoints, took ownership instead
* fix: changed vecotr slice to vector
* fix: remove clone
* fix: combined datapoints and requests with zip instead of enumerating
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-43-183.us-east-2.compute.internal >
2023-10-20 06:25:59 +00:00
Yun Chen
f859932745
fix: convert to ReadableSize & Durations ( #2594 )
...
* fix: convert to ReadableSize & Durations
* fix: change more grpc sender/recv message size to ReadableSize
fix: format
fix: cargo fmt
fix: change cmd test to use durations
fix: revert metaclient change
fix: convert default fields in meta client options
fix: human serde meta client durations
* fix: remove milisecond postfix in heartbeat option
* fix: humantime serde on heartbeat
* fix: update config example
* fix: update integration test config
* fix: address pr comments
* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
Weny Xu
cc83764331
fix: check table exists before allocating table id ( #2546 )
...
* fix: check table exists before allocating table_id
* chore: apply suggestions from CR
2023-10-09 11:40:10 +00:00
Weny Xu
8cc862ff8a
refactor: refactor cache invalidator ( #2540 )
2023-10-09 08:19:18 +00:00
shuiyisong
007f7ba03c
refactor: extract plugins crate ( #2487 )
...
* chore: move frontend plugins fn
* chore: move datanode plugins to fn
* chore: add opt plugins
* chore: add plugins to meta-srv
* chore: setup meta plugins, wait for router extension
* chore: try use configurator for grpc too
* chore: minor fix fmt
* chore: minor fix fmt
* chore: add start meta_srv for hook
* chore: merge develop
* chore: minor fix
* chore: replace Arc<Plugins> with PluginsRef
* chore: fix header
* chore: remove empty file
* chore: modify comments
* chore: remove PluginsRef type alias
* chore: remove `OptPlugins`
2023-10-09 04:54:27 +00:00
Zhenchi
201acd152d
fix: missing file engine with default options ( #2519 )
...
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-28 10:25:12 +00:00
Yingwen
a6116bb866
feat(mito): Add cache manager ( #2488 )
...
* feat: add cache manager
* feat: add cache to reader builder
* feat: add AsyncFileReaderCache
* feat: Impl AsyncFileReaderCache
* chore: move moka dep to workspace
* feat: add moka cache to the manager
* feat: implement parquet meta cache
* test: test cache manager
* feat: consider vec size
* style: fix clippy
* test: fix config api test
* feat: divide cache
* test: test disabling meta cache
* test: fix config api test
* feat: remove meta cache if file is purged
2023-09-26 11:46:19 +00:00
Ruihang Xia
515ce825bd
feat: stack trace style debug print for error ( #2489 )
...
* impl macro stack_trace_debug
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* manually mark external error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use debug print
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify the error and warn macro
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ut
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add docs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* replace snafu backtrace with location
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-26 11:23:21 +00:00
Zhenchi
0bf26642a4
feat: re-support query engine execute dml ( #2484 )
...
* feat: re-support query engine execute dml
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: remove region_number in InsertRequest
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add doc comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-26 08:37:04 +00:00
Ning Sun
ffa729cdf5
feat: implement storage for OTLP histogram ( #2282 )
...
* feat: implement new histogram data model
* feat: use prometheus table format for histogram
* refactor: remove duplicated code
* fix: histogram tag column
* fix: use accumulated count in buckets
* refactor: using row based protocol for otlp WIP
* refactor: use row based writer for otlp.
Also updated row writer for owned keys
* refactor: use row writers for otlp
* test: add integration tests for histogram
* refactor: change le column name
2023-09-23 07:59:14 +00:00
LinFeng
621c6f371b
feat: limit grpc message size ( #2459 )
...
* feat: add two grpc config options
Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size
* test: add integration tests for message size limit
2023-09-22 02:07:46 +00:00
Weny Xu
580d11b1e1
refactor: unify table metadata cache invalidator ( #2449 )
...
* refactor: unify table metadata cache invalidator
* chore: apply from suggestions
2023-09-21 03:45:49 +00:00
shuiyisong
20f4f7971a
refactor: remove source and location in snafu display ( #2428 )
...
* refactor: remove source pt 1
* refactor: remove source pt 2
* refactor: remove source pt 3
* refactor: remove location pt 1
* refactor: remove location pt 2
* chore: remove rustc files
* chore: fix error case
* chore: fix test
* chore: fix test
* chore: fix cr issue
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
---------
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
2023-09-21 02:55:24 +00:00
Zhenchi
deac284973
refactor: RegionRequestHandler -> RegionQueryHandler ( #2439 )
...
* refactor: RegionRequestHandler -> RegionQueryHandler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: rename FrontendRegionQueryHandler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: privte RegionInvoker
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-19 08:19:58 +00:00
dennis zhuang
5566f34bd1
feat: make scripts table work again ( #2420 )
...
* feat: make scripts table work again
* chore: typo
* fix: license header
* Update src/table/src/metadata.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: cr comments
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-18 11:43:21 +00:00
Zhenchi
3a39215f11
feat: migrate file engine from table to reigon ( #2365 )
...
* feat: migrate file engine from table to reigon
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/file-engine/src/engine.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* feat: specify ts index for file engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: handle time index for external table
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: some integration testsg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add file schema and table schema compatibility
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: compatible file schema to region schema
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add error msg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: simplify close
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: implement set_writable
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: tests-integration compilation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local >
2023-09-18 08:02:43 +00:00
JeremyHi
542e863ecc
fix: missing datanode id on keep lease ( #2415 )
2023-09-17 07:57:17 +00:00
JeremyHi
98a40bae95
feat!: unify naming with options ( #2416 )
2023-09-17 07:24:57 +00:00
JeremyHi
342a6d071f
feat: heartbeat request with header ( #2412 )
...
* feat: heartbeat request with header
* chore: frontend send heartbeat with a longer interval
2023-09-16 09:56:41 +00:00
dennis zhuang
627c5b7419
feat: move table operations from frontend to operator crate ( #2411 )
...
* feat: move table operations from frontend to operator crate
* chore: blank line
* fix: toml format
* chore: move constants
2023-09-16 07:58:45 +00:00
LFC
a688760563
fix: validate partition columns ( #2393 )
...
fix: partition column must belong to primary keys or equals to time index
2023-09-15 10:07:32 +00:00
LFC
4b13c88752
fix: resolve more integration tests ( #2406 )
...
* fix: resolve more integration tests
* Update tests-integration/tests/http.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-09-15 09:43:16 +00:00
dennis zhuang
43e3c94fd1
refactor: catalog managers ( #2405 )
...
* feat: rename catalog::local to catalog::memory
* refactor: catalog managers
* chore: license header
2023-09-15 08:48:14 +00:00
LFC
fe5679e77e
refactor: remove table ident ( #2368 )
...
* refactor:
1. remove TableIdent, use TableId directly
2. use the latest greptime-proto
3. independently invalidate table id cache and table name cache
* rebase
* fix: resolve PR comments
* fix: resolve PR comments
2023-09-15 05:14:40 +00:00
JeremyHi
d1adb915bf
feat: set readonly first when deregister region ( #2391 )
...
* feat: set readonly first when deregister region
* revert distxxx
2023-09-14 12:12:38 +00:00
Yingwen
da54a0c139
fix: alter table procedure forgets to update next column id ( #2385 )
...
* feat: add more info to error messages
* feat: store next column id in procedure
* fix: update next column id for table info
* test: fix add col test
* chore: remove location from invalid request error
* test: update test
* test: fix test
2023-09-14 02:06:57 +00:00
Weny Xu
93f3048f4f
refactor: migrate OpenDal to 0.39 ( #2383 )
...
* chore: bump opendal to 7d552
* refactor: migrate OpenDal to 0.39
* chore: apply suggestions from CR
2023-09-13 12:43:53 +00:00
LFC
d08b05c963
fix: make test-integration able to compile ( #2384 )
...
* fix: make test-integration able to compile
* chore: fmt toml
---------
Co-authored-by: WenyXu <wenymedia@gmail.com >
2023-09-13 12:42:46 +00:00
JeremyHi
f76aa278fd
feat: atomic metadata ( #2366 )
...
* feat: atomic creating metadata
* chore: exist exists
* chore: license header
* chore: weny never say that
* feat: add put_conditionally to kv_backend
2023-09-13 10:51:05 +00:00
dennis zhuang
7448e975c2
chore: change error messages ( #2379 )
...
* chore: change error messages
* chore: remove locaton in table not found error msg
2023-09-13 08:21:03 +00:00
Ruihang Xia
1ad5f6e5d5
refactor: system tables in FrontendCatalogManager ( #2358 )
...
* rename method names
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove system table, table engine, register/deregister
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add system catalog
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* run nextest
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* some documents
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: fix clippy
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: WenyXu <wenymedia@gmail.com >
2023-09-12 07:57:15 -05:00
Weny Xu
912341e4fa
fix: fix start issues under standalone mode ( #2352 )
...
* fix: fix standalone starts
* chore: bump raft-engine to 571462e
* refactor: remove MetadataService
2023-09-12 07:57:15 -05:00
Zhenchi
4af126eb1b
feat: consolidate Insert request related partitioning and distributed processing operations into Inserter ( #2346 )
...
* refactor: RegionRequest as param of RegionRequestHandler.handle
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: partition insert & delete reqs for both standalone and distributed mode
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: nit change
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: wrong function nameg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: do request in inserter & deleter
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: remove RegionRequestHandler.handle
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: rename table_creator
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: nit change
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: nit change
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-12 07:57:15 -05:00
LFC
fe954b78a2
refactor: system tables in new region server ( #2344 )
...
refactor: inverse the dependency between system tables and catalog manager
2023-09-12 07:57:15 -05:00
Weny Xu
a4604afde5
refactor: rename NEXT_TABLE_ROUTE_PREFIX to TABLE_ROUTE_PREFIX ( #2348 )
...
* refactor: rename NEXT_TABLE_ROUTE_PREFIX to TABLE_ROUTE_PREFIX
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Weny Xu
f386329e29
refactor: introduce DdlTaskExecutor and refactor statement executor ( #2341 )
...
* feat: add kv store option
* refactor: refactor statement executor
* refactor: refactor standalone table creator
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor
* fix: fix RegionDistribution
* feat: build standalone
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Ruihang Xia
6215f124f7
refactor: remove datanode instance ( #2342 )
...
* pass nextest
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove deadcode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename region_alive_keepers
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
LFC
1d83c942a9
refactor: script table creation ( #2340 )
...
* refactor:
1. remove method `register_system_table` from CatalogManager
2. the creation of ScriptTable (as a system table) is removed from CatalogManager. Instead, the ScriptTable is created when Frontend instance is starting; and is created by calling Frontend instance's grpc handler.
* rebase
2023-09-12 07:57:15 -05:00
Zhenchi
dac6b2e80a
feat(frontend): migrate delete to region server ( #2329 )
...
* feat(frontend): migrate delete to region server
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add more check and do trim columns
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: RegionRequestHandler.handle retrun AffectedRows
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-12 07:57:15 -05:00
Ruihang Xia
eb7116ab56
feat: read/write works in distributed mode 🎉 ( #2327 )
...
* add do_get method to RegionRequestHandler
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move RegionRequestHandler to client crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use RegionRequestHandler in MergeScan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* minor fix
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
Zhenchi
5f7d48f107
feat(frontend): reorg insert converters and introduce stmt_to_region ( #2324 )
...
* feat(frontend): reorg insert converters and introduce stmt_to_region
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: shorten import path
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: add check for column count
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-12 07:57:15 -05:00
LFC
711e27d9fa
feat: distributed alter table in region server ( #2311 )
...
* feat: distributed alter table in region server
* rebase
2023-09-12 07:57:15 -05:00
Weny Xu
922e342b63
refactor: refactor ddl manager ( #2306 )
...
* refactor: refactor ddl manager
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Zhenchi
7dde9ce3ce
feat(frontend): migrate insert to region server ( #2318 )
...
* feat(frontend): migrate insert to region server
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move converter to Inserter
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: rename convert function
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: add span id
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: compilation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* retrigger action
* retrigger action
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-12 07:57:15 -05:00
dennis zhuang
a3d5931fca
feat: unify all protocol options ( #2316 )
...
* feat: unify all protocol options
* feat: adds enable to example configs
* chore: style
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-09-12 07:57:15 -05:00
Niwaka
3504d8254e
fix: unused table options ( #2267 )
...
* fix: unused table options keys
* refactor: simplify validate table options
* chore: Add newlines
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-09-12 07:57:15 -05:00
Weny Xu
4818887e38
refactor: refactor DistInstance ( #2305 )
2023-09-12 07:57:15 -05:00
Zhenchi
b42d343ae6
feat(frontend): unify column inserter and row inserter ( #2293 )
...
* refactor: unify inserter
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat(frontend): unify column inserter and row inserter
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: remove redundant clone
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: move empty check ahead
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add more logs
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: leading license
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* adjust indent
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-12 07:57:15 -05:00
Ruihang Xia
718246ea1a
feat: implement heartbeat for region server ( #2279 )
...
* retrieve region stats from region server
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement heartbeat handler
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with region server
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* disable non-unit test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement heartbeat task
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
