LFC
711e27d9fa
feat: distributed alter table in region server ( #2311 )
...
* feat: distributed alter table in region server
* rebase
2023-09-12 07:57:15 -05:00
Weny Xu
922e342b63
refactor: refactor ddl manager ( #2306 )
...
* refactor: refactor ddl manager
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1
feat: start datanode with config ( #2312 )
...
* remove memory-catalog and procedure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* derive serde for MitoConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with configs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove dir in WalConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add rename field attr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add stupid duplicated mito config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove wrong import
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wired compile error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
dennis zhuang
38697e0c4d
feat: build http client for cloud object storage ( #2314 )
...
* feat: build http client for s3/oss/azblob storages
* chore: style
* fix: test
* fix: cargo toml fmt
2023-09-12 07:57:15 -05:00
Niwaka
3504d8254e
fix: unused table options ( #2267 )
...
* fix: unused table options keys
* refactor: simplify validate table options
* chore: Add newlines
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-09-12 07:57:15 -05:00
Yingwen
648b2ae293
feat(mito): Flush region ( #2291 )
...
* chore: call handle_flush_request
* feat: alias SchedulerRef and clean scheduler on drop
* feat: add scheduler to workers
* feat: remove RegionMemtableStats
* feat: pick regions to flush
* feat: add more fields to region flush task
* feat: smallvec workspace dep
* feat: Use list to hold immutable memtables
* feat: flush job wip
* feat: use access layer to read write sst
* feat: flush memtables to l0
* feat: write manifest
* feat: schedule next flush on success
* feat: schedule flush on success and failure
* feat: add purger to region
* feat: apply edit after flush
* feat: collect stats for SSTs
* feat: manual flush
* test: test flush and fix manifest test
* feat: remove flush scheduler job limit
* fix: typo
* style: clippy
* feat: clean flushed files on failure
* chore: address CR comment
* refactor: Use put_rows
* feat: Clean flush scheduler on drop
* feat: remove region flush status on drop and close
* chore: address CR comment
2023-09-12 07:57:15 -05:00
Weny Xu
fa5e3b94d3
refactor: refactor ddl procedure ( #2304 )
2023-09-12 07:57:15 -05:00
Weny Xu
c839ed271c
refactor: refactor: ddl context ( #2301 )
...
* refactor: refactor: ddl context
* refactor: remove unused code
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
JeremyHi
4d2cae4174
refactor: inactive node manager ( #2300 )
...
refactor: use region_id instead of table®ion_num in InactiveNodeManager
2023-09-12 07:57:15 -05:00
Lei, HUANG
9691d19601
feat: impl kv backend for raft engine ( #2280 )
...
* feat: kv backend on raft-engine
* feat: raft-engine kvbackend
* fix: toml
* fix: some review comments
* chore: optimize delete
* fix: lift lock in batch_delete
2023-09-12 07:57:15 -05:00
JeremyHi
fa542f6e93
feat: new heartbeat ( #2299 )
2023-09-12 07:57:15 -05:00
Lei, HUANG
86d56f71ef
fix: flume bug ( #2298 )
...
fix: flume
2023-09-12 07:57:15 -05:00
Ruihang Xia
46d171d341
chore: bump greptime-proto to replace region_dir ( #2290 )
...
chore: bump greptime-proto to replace region_dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
Ruihang Xia
718246ea1a
feat: implement heartbeat for region server ( #2279 )
...
* retrieve region stats from region server
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement heartbeat handler
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with region server
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* disable non-unit test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement heartbeat task
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
JeremyHi
58d07e0e62
feat: v04 rm unused exprs ( #2285 )
...
* feat: rm compact and flush exprs
* refactor: continue to rm compact and flush
2023-09-12 07:57:15 -05:00
shuiyisong
5df4d44761
feat: schema level opts ( #2283 )
...
* chore: update proto
* chore: add try from for schema name value
* chore: merge schema opts to table opts while creating table
* chore: use table ttl opts first
* chore: add unit test
* chore: update proto version
2023-08-30 08:11:08 +00:00
LFC
ef75e8f7c3
feat: create distributed Mito2 table ( #2246 )
...
* feat: create distributed Mito2 table
* rebase develop
2023-08-28 12:07:52 +00:00
Ruihang Xia
6b8cf0bbf0
feat: impl region engine for mito ( #2269 )
...
* update proto
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* convert request
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update proto
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* import result convertor
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename symbols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-28 09:24:12 +00:00
JeremyHi
63b22b2403
feat: prometheus row inserter ( #2263 )
...
* feat: prometheus row inserter
* chore: add unit test
* refactor: to row_insert_requests
* chore: typo
* chore: alloc row by TableData
* chore: by review comment
2023-08-28 03:22:23 +00:00
dennis zhuang
5160838d04
chore: change version to 0.4.0-nightly ( #2258 )
...
* chore: change version to 0.4.0-nightly
* fix: test
2023-08-25 09:44:39 +00:00
Ruihang Xia
b633a16667
feat: apply rewriter to subquery exprs ( #2245 )
...
* apply rewriter to subquery exprs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* workaround for datafusion's check
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add sqlness test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change time index type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-24 11:48:04 +00:00
JeremyHi
7746e5b172
feat: dist row inserter ( #2231 )
...
* feat: fronend row inserter
* feat: row splitter
chore: row splitter's unit test
* feat: RowDistInserter
* feat: make influxdb line protocol using row-based protocol
* Update src/partition/src/row_splitter.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/frontend/src/instance/distributed/row_inserter.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by review comment
* Update src/frontend/src/instance/distributed/row_inserter.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* chore: by comment
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: LFC <bayinamine@gmail.com >
2023-08-24 06:58:05 +00:00
Bamboo1
4dbc32f532
refactor: remove associate type in scheduler to simplify it #2153 ( #2194 )
...
* feature: add a simple scheduler using flume
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: only use a sender rather clone many senders
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: use select to avoid loop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: add parameters in new function to build the flume capacity and number of receivers
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: add countdownlatch test concurrency
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: add barrier replacing countdownlatch to test concurrency and add wait all tasks finished in stop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add some document annotation
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add license header
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add Cargo.lock
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: Cargo.toml format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: delete println in test
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: add error handle
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: fix error handle and add test scheduler stop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: spelling mistake
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: wait all tasks finished
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add todo which need wrap Future returned by send_async
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: remove unnessary sleep in test
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: resolve some conflicts
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: resolve conversation
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: modify the function of schedule to synchronize and drop sender after stopping scheduler
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
---------
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
2023-08-23 06:28:00 +00:00
Ruihang Xia
18250c4803
feat: implement Flight and gRPC services for RegionServer ( #2226 )
...
* extract FlightCraft trait
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* split service handler in GrpcServer
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* left grpc server implement
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start region server if configured
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-22 13:30:09 +00:00
WU Jingdi
b411769de6
feat: Implement a basical range select query ( #2138 )
...
* feat: Implement a basical range select query
* chore: support any timestamp type & CR fix
2023-08-22 03:07:14 +00:00
niebayes
e5f4ca2dab
feat: streaming do_get ( #2171 )
...
* feat: rewrite do_get for streaming get flight data
* feat: rewrite do_get call stack but leave the async stream adapter not modified yet
* feat: rewrite the async stream adapter to accept greptime record batch stream
* fix: resolve some PR comments
* feat: rewrite tests to adapt to the streaming do_get
* feat: add unit tests for streaming do_get
* feat: rewrite timer metric of merge scan
* remove unhelpful unit tests for streaming do_get
* add a new metric timer for merge scan and fix some test errors
* rewrite mysql writer to write query results in a streaming manner
* fix: fix fmt errors
* fix: rewrite sqlness runner to take into account the streaming do_get
* fix: fix toml format errors
* fix: resolve some PR comments
* fix: resolve some PR comments
* fix: refactor do_get to increase readability
* fix: refactor mysql try_write_one to increase readability
2023-08-22 02:54:05 +00:00
Ruihang Xia
4aaf6aa51b
feat: implement query API for RegionServer ( #2197 )
...
* some initial change
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl dummy structs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* decode and send logical plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement table scan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add some comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-17 11:02:31 +00:00
JeremyHi
6cd7319d67
refactor: grpc insert ( #2188 )
...
* feat: interval type for row protocol
* feat: minor refactor grpc insert
* Update src/common/grpc-expr/src/util.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: by comment
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-16 11:25:25 +00:00
Ruihang Xia
4c3169431b
feat: move region metadata to store-api ( #2181 )
...
* add metadata & handle_read
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move metadata to store-api
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* dep aquamarine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove deadcode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove temporary code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/store-api/Cargo.toml
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* remove old mod
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2023-08-16 04:18:26 +00:00
Niwaka
a8f2e4468d
feat: handle multiple grpc deletes ( #2150 )
...
* feat: handle multiple grpc deletes
* fix: make DistDeleter::grpc_delete return usize
* fix: remove backtrace from MissingTimeIndexColumn
* fix: avoid using unwrap in PartitionRuleManager::split_delete_request
* fix: simplify MissingTimeIndexColumn
2023-08-15 08:22:46 +00:00
Ruihang Xia
2168970814
feat: define region server and related requests ( #2160 )
...
* define region server and related requests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fill request body
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change mito2's request type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: bump greptime-proto to d9167cab (row insert/delete)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove name_to_index
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* address cr comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finilise
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-15 06:27:27 +00:00
Lei, HUANG
e924b44e83
refactor: KeyValues return ValueRef ( #2170 )
...
* refactor: KeyValues return ValueRef
* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.
* feat: define min value of OpType as a const
* fix: toml format
2023-08-14 14:51:13 +00:00
Lei, HUANG
55b5df9c51
feat: row wise converter ( #2162 )
...
* feat: impl mem-comparable encoding for timestamp
* fix: test cases
* impl time series encode/decoder
* fix: merge unsupported match arms
* fix: clippy
* chore: big number delimiter
* feat: encode timestamps as i64
* fix: remove useless error variant
2023-08-14 07:13:39 +00:00
LFC
606b489d53
feat: redact secrets in sql when logging ( #2141 )
2023-08-14 06:40:00 +00:00
Weny Xu
d0b3607633
feat: add table route manager and upgrade tool ( #2145 )
...
* feat: add table route manager and upgrade tool
* test: add table route manager tests
* feat: add new TableRouteValue struct
* chore: apply suggestions from CR
* refactor: change HashMap to BTreeMap
* feat: add version to TableRouteValue
2023-08-14 04:19:44 +00:00
shuiyisong
7f51141ed0
refactor: auth crate ( #2148 )
...
* chore: move user_info to auth crate
* chore: temp commit before resolving tests compile error
* chore: fix compile issue
* chore: minor fix
* chore: tmp save
* chore: change user_info to trait
* chore: minor change & use auth result user info in pg session setup
* chore: add as_any to user_info
* chore: rename user_info
* chore: remove ice file
* chore: add permission checker
* chore: add grpc permission check
* chore: add session spawn user_info to query_ctx
* chore: minor update
* chore: add permission checker to sql handler & temp save
* chore: add permission checker to prometheus handler
* chore: add permission checker to opentsdb handler
* chore: add permission checker to other handlers
* chore: add test
* chore: add user_info setting on http entrance
* chore: fix toml
* chore: remove box in permission req
* chore: cr issue
* chore: cr issue
2023-08-14 02:51:26 +00:00
Ruihang Xia
6d64e1c296
feat(mito): checkpoint for mito2 ( #2142 )
...
* basic impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* adjust dir structure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix styles
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* sort result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* downgrade log level
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply CR sugg.
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add region id to log
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-13 09:26:01 +00:00
谢政
b62e643e92
build: update protobuf-build to support apple silicon ( #2143 )
...
* build: update protobuf-build to support apple silicon
* build: Update src/log-store/Cargo.toml
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* build: update the Cargo.lock too
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-12 03:31:51 +00:00
dennis zhuang
6f40128058
feat!: enable telemetry by default ( #2137 )
...
* feat: remove greptimedb-telemetry feature
* feat: adds enable_telemetry option to metasrv and datanode
* refactor: move data_home from file config to storage config
* feat: store the installation uuid into datanode and metasrv working home
* fix: cargo toml fmt
* test: ignore region failver test when using local fle storage
* test: ignore telemetry reporter in test mode
* feat: print warning log when enabling telemetry
* chore: the telemetry doc link
* chore: remove enable_telemetry from datanode example config file
* refactor: rename GREPTIMEDB_TELEMETRY_CLIENT_REQUEST_TIMEOUT
* chore: rename print_warn_log to print_anonymous_usage_data_disclaimer
2023-08-11 14:50:40 +00:00
LFC
5c19913a91
build: on windows ( #2054 )
...
* build on windows
* rebase develop
* fix: resolve PR comments
2023-08-10 08:08:37 +00:00
WU Jingdi
b7319fe2b1
feat: Support RangeSelect LogicalPlan rewrite ( #2058 )
...
* feat: Support RangeSelect LogicalPlan rewrite
* chore: fix code advice
* fix: change format of range_fn
* chore: optimize project plan rewrite
* chore: fix code advice
2023-08-10 02:53:20 +00:00
Ning Sun
d18eb18b32
feat: use server inferenced types on statement describe ( #2032 )
...
* feat: use server inferenced types on statement describe
* feat: add support for server inferenced type
* feat: allow parameter type inferencing
* chore: update comments
* fix: lint issue
* style: comfort rustfmt
* Update src/servers/src/postgres/types.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-08-09 02:57:56 +00:00
Zou Wei
d5cadeeec3
feat: conversion between interval and gRPC ( #2064 )
...
* feat: support grpc for interval type
* chore: add unit test cases.
* chore: cargo clippy
* chore: modify greptime-proto version
* chore: cr comment.
* chore: cargo fmt
* refactor: convert function.
2023-08-07 06:22:04 +00:00
Yingwen
cb4dd89754
feat(mito): Implement mito2 Wal ( #2103 )
...
* feat: define wal struct
* feat: Implement Wal read/write
* feat: obsolete wal
* test: test wal
* refactor: use try_stream and remove async from scan
2023-08-04 11:04:25 +00:00
Ning Sun
93cd4ab89d
ci: require cargo.lock up to date ( #2094 )
2023-08-04 02:59:01 +00:00
Yingwen
e5663a075f
feat(mito): preparation to implementing write ( #2085 )
...
* refactor: move request mod
* feat: add mutation
* feat: add handle_write mod
* feat: one mutation at a time
* feat: handle write requests
* feat: validate schema
* refactor: move schema check to write request
* feat: add convert value
* feat: fill default values
* chore: remove comments
* feat: remove code
* feat: remove code
* feat: buf requests
* style: fix clippy
* refactor: rename check functions
* chore: fix compile error
* chore: Revert "feat: remove code"
This reverts commit 6516597540 .
* chore: Revert "feat: remove code"
This reverts commit 5f2b790a01 .
* chore: upgrade greptime-proto
* chore: Update comment
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-08-04 02:53:02 +00:00
fys
dda922507f
feat: impl pubsub in metasrv ( #2045 )
...
* feat: impl pubsub
* add test_subscriber_disconnect unit test
* chore: cr
* cr
* cr
2023-08-03 03:56:43 +00:00
Ning Sun
7a27ef8d11
fix: remove openssl from reqwest and use rustls instead ( #2081 )
...
* fix: remove openssl from reqwest and use rustls instead
* fix: correct server url
* style: fix toml format
2023-08-02 10:23:21 +00:00
zyy17
be8f243c64
chore: update Cargo.lock ( #2068 )
2023-08-02 15:23:16 +08:00
JeremyHi
fd7eb87a52
refactor: common semantic-type ( #2057 )
2023-08-01 11:18:05 +00:00