rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-06 13:22:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,441 Commits 106 Branches 226 Tags
e5f4ca2dab7fae0b0682efce397a3f469a68f6a7
Commit Graph

1441 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
niebayes
e5f4ca2dab feat: streaming do_get (#2171) 
* feat: rewrite do_get for streaming get flight data

* feat: rewrite do_get call stack but leave the async stream adapter not modified yet

* feat: rewrite the async stream adapter to accept greptime record batch stream

* fix: resolve some PR comments

* feat: rewrite tests to adapt to the streaming do_get

* feat: add unit tests for streaming do_get

* feat: rewrite timer metric of merge scan

* remove unhelpful unit tests for streaming do_get

* add a new metric timer for merge scan and fix some test errors

* rewrite mysql writer to write query results in a streaming manner

* fix: fix fmt errors

* fix: rewrite sqlness runner to take into account the streaming do_get

* fix: fix toml format errors

* fix: resolve some PR comments

* fix: resolve some PR comments

* fix: refactor do_get to increase readability

* fix: refactor mysql try_write_one to increase readability
 2023-08-22 02:54:05 +00:00
Weny Xu
5b7b2cf77d fix: fix ddl client can not update leader addr (#2205) 
* fix: fix ddl client can not update leader addr

* chore: apply suggestions from CR

* feat: add message to context

* fix: only retry if unavailable or deadline exceeded

* chore: apply suggestions from CR
 2023-08-21 13:57:29 +00:00
shuiyisong
9352649f22 chore: add table region key to delete in upgrade tool (#2214) 2023-08-21 08:16:10 +00:00
shuiyisong
c5f507c20e fix: add user_info extension to prom_store handler (#2212) 
chore: add user_info extention to prom_store auth
 2023-08-21 04:55:34 +00:00
JeremyHi
033b650d0d feat: row write protocol (#2189) 
* feat: datanode's row insrter

* refactor: ExprFactory

* feat: row inserter in standalon mode

* chore: minor refactor

* feat: influxdb line protocol's row protocol

* chore: minor refactor

* improve: avoid to use too many string

* no longer async

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: do not check empty data

* chore: by review comment

* chore: by comment

* chore: by review comment

* chore: by review comment

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
v0.4.0-nightly-20230821 		2023-08-19 13:08:44 +00:00
dennis zhuang
272f649b22 fix: some TODO in sqlness cases and refactor meta-client error (#2207) 
* fix: some TODO in sqlness cases and refactor meta-client error

* fix: delete tests/cases/standalone/alter/drop_col_not_null_next.output
 2023-08-18 10:09:11 +00:00
Ruihang Xia
3150f4b22e fix: specify input ordering and distribution for prom plan (#2204) 
* fix: specify input ordering and distribution for prom plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-18 09:45:46 +00:00
Weny Xu
e1ce1d86a1 refactor: unite key serialization method (#2195) 2023-08-18 09:42:19 +00:00
ZonaHe
b8595e1960 feat: update dashboard to v0.3.1 (#2192) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2023-08-18 09:42:18 +00:00
shuiyisong
61e6656fea fix: auth in prometheus gateway service (#2206) 
* fix: auth in prometheus gateway service

* chore: remove unused code

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-18 09:41:38 +00:00
Ruihang Xia
1bbec75f5b fix: skip partition clause in show create table (#2200) 
* fix: skip partition clause in show create table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-18 09:10:31 +00:00
Zhenchi
8d6a2d0b59 refactor: apply numbers to ThinTable (#2202) 
* refactor: apply numbers to `ThinTable`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: tiny polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: unused import

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-08-18 03:11:37 +00:00
Weny Xu
177036475a fix: support to copy from parquet with typecast (#2201) 2023-08-18 03:09:54 +00:00
Zhenchi
87a730658a refactor: add ThinTable to proxy tables from infoschema (#2193) 
* refactor: add thin table to proxy tables in info_schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix(catalog): fix typo in DataSourceAdapter struct name

* fix: remove redundant Send + Sync

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor(catalog): rename DataSourceAdapter to InformationTableDataSource

* feat(catalog): add ThinTableAdapter for adapting ThinTable to Table interface

* rebase develop

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: default impl for table_type of InformationTable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: filter_pushdown as table field

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove explicit type declaration

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-08-17 15:19:14 +00:00
JeremyHi
b67e5bbf70 fix: invalid err msg (#2196) 2023-08-17 11:12:35 +00:00
Ruihang Xia
4aaf6aa51b feat: implement query API for RegionServer (#2197) 
* some initial change

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl dummy structs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* decode and send logical plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement table scan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add some comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-17 11:02:31 +00:00
Weny Xu
6e6ff5a606 refactor: update table metadata in single txn (#2172) 
* refactor: table-metadata-manager

* feat: remove comparing when deleting metadata

* fix: fix comment typos

* chore: apply suggestions from CR

* test: add tests for updating DatanodeTable

* fix: fix clippy

* chore: apply suggestions from CR

* refactor: improve update table route tests

* refactor: return Txn instead of TxnRequest

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: update table metadata in single txn

* feat: check table exists before drop table executing

* test: add tests for table metadata manager

* refactor: remove table region manager

* chore: apply suggestions from CR

* feat: add bench program

* chore: apply suggestions from CR
 2023-08-17 06:29:19 +00:00
Yingwen
4ba12155fe feat(mito): Implement SST format for mito2 (#2178) 
* chore: update comment

* feat: stream writer takes arrow's types

* feat: Define Batch struct

* feat: arrow_schema_to_store

* refactor: rename

* feat: write parquet in new format with tsids

* feat: reader support projection

* feat: Impl read compat

* refactor: rename SchemaCompat to CompatRecordBatch

* feat: changing sst format

* feat: make it compile

* feat: remove tsid and some structs

* feat: from_sst_record_batch wip

* chore: push array

* chore: wip

* feat: decode batches from RecordBatch

* feat: reader converts record batches

* feat: remove compat mod

* chore: remove some codes

* feat: sort fields by column id

* test: test to_sst_arrow_schema

* feat: do not sort fields

* test: more test helpers

* feat: simplify projection

* fix: projection indices is incorrect

* refactor: define write/read format

* test: test write format

* test: test projection

* test: test convert record batch

* feat: remove unused errors

* refactor: wrap get_field_batch_columns

* chore: clippy

* chore: fix clippy

* feat: build arrow schema from region meta in ReadFormat

* feat: initialize the parquet reader at `build()`

* chore: fix typo
 2023-08-17 06:25:50 +00:00
Weny Xu
832e5dcfd7 chore: remove allow-unused (#2184) 2023-08-17 03:15:12 +00:00
shuiyisong
d45ee8b42a chore: fix collect region stat on non-base table (#2190) 2023-08-17 02:13:49 +00:00
JeremyHi
6cd7319d67 refactor: grpc insert (#2188) 
* feat: interval type for row protocol

* feat: minor refactor grpc insert

* Update src/common/grpc-expr/src/util.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: by comment

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-16 11:25:25 +00:00
Yingwen
bb062003ef ci: fallback to run_id to avoid cancelling other jobs (#2186) 
ci: fallback to run id to avoid cancelling other jobs
 2023-08-16 09:24:17 +00:00
Weny Xu
8ea1763033 refactor: refactor table metadata manager (#2159) 
* refactor: table-metadata-manager

* feat: remove comparing when deleting metadata

* fix: fix comment typos

* chore: apply suggestions from CR

* test: add tests for updating DatanodeTable

* fix: fix clippy

* chore: apply suggestions from CR

* refactor: improve update table route tests

* refactor: return Txn instead of TxnRequest

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2023-08-16 06:43:03 +00:00
Zhenchi
1afe96e397 refactor: prevent dist table from invoking scan (#2179) 
* refactor: prevent dist table from invoking `scan`

* refactor: reorg code

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-08-16 04:43:33 +00:00
Ruihang Xia
814c599029 ci: cancel in-progress actions on new commit (#2182) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-16 04:21:14 +00:00
Ruihang Xia
4c3169431b feat: move region metadata to store-api (#2181) 
* add metadata & handle_read

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move metadata to store-api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* dep aquamarine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove temporary code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/store-api/Cargo.toml

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* remove old mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-08-16 04:18:26 +00:00
sh2
202540823f refactor!: move prometheus routes to default http server (#2005) 
* move prometheus routes to default http server

Signed-off-by: sh2 <shawnhxh@outlook.com>

* fix ci test and remove the server logic of prometheus

* remove unused import and prometheus relevant code

* fix ci: rustfmt and test

* fix ci: silly fmt

* fix ci: silly silly fmt

* change `/prom_store` back to `/prometheus`

* remove unused variable

---------

Signed-off-by: sh2 <shawnhxh@outlook.com>
 2023-08-16 03:21:14 +00:00
dennis zhuang
0967678a51 feat: don't enable telemetry for debug building (#2177) 2023-08-16 01:53:11 +00:00
shuiyisong
c8cde704cf chore: minor auth crate change (#2176) 
* chore: pub auth_mysql

* chore: pub all error

* chore: remove back to error

* chore: wrap failed permission check result to err

* chore: minor change
 2023-08-15 10:49:22 +00:00
JeremyHi
24dc827ff9 feat: grpc handler result (#2107) 
* feat: grpc handler inner result

* feat: ext header, x-greptime-err-code, x-greptime-err-msg

* fix: sqlness case

* chore: by comment

* fix: convert status to Error
 2023-08-15 09:34:00 +00:00
Weny Xu
f5e44ba4cf docs: rfc of update metadata in single txn (#2165) 
* docs: rfc of update metadata in single txn

* chore: apply suggestion from CR
 2023-08-15 17:44:07 +08:00
zyy17
32c3ac4fcf refactor: improve the image building performance (#2175) 
* refactor: use '--output type=local' in 'build-greptime-by-buildx' target to reduce unnecessary 'docker cp'"

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve the image building performance

* ci: release centos dev builder

* ci: use 'make build-by-dev-builder' to improve docker build performance

* refactor: add 'which' command in centos

* fix: add 'OUTPUT_DIR' to fix 'make docker-image-buildx' error

* fix: fix incorrect dockerfile path

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: remove configure-aws-credentials action and use env variables

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* ci: update slack notification prompt

* refactor: clean up the target directory before building artifacts of centos7

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2023-08-15 09:28:09 +00:00
Niwaka
a8f2e4468d feat: handle multiple grpc deletes (#2150) 
* feat: handle multiple grpc deletes

* fix: make DistDeleter::grpc_delete return usize

* fix: remove backtrace from MissingTimeIndexColumn

* fix: avoid using unwrap in PartitionRuleManager::split_delete_request

* fix: simplify MissingTimeIndexColumn
 2023-08-15 08:22:46 +00:00
Yingwen
d4565c0a94 feat(mito): Defines the read Batch struct for mito2 (#2174) 
* feat: define batch

* feat: define Batch struct

* feat: stream writer takes arrow's types

* feat: make it compile

* feat: use uint64vector and uint8vector

* feat: add timestamps and primary key
 2023-08-15 06:39:21 +00:00
Ruihang Xia
2168970814 feat: define region server and related requests (#2160) 
* define region server and related requests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fill request body

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change mito2's request type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: bump greptime-proto to d9167cab (row insert/delete)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove name_to_index

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* address cr comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finilise

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-15 06:27:27 +00:00
Weny Xu
69a2036cee feat!: add deserializer for Partition (#2169) 
* feat!: add deserializer for Partition

* fix: fix tests
 2023-08-15 03:36:58 +00:00
Lei, HUANG
e924b44e83 refactor: KeyValues return ValueRef (#2170) 
* refactor: KeyValues return ValueRef

* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.

* feat: define min value of OpType as a const

* fix: toml format
 2023-08-14 14:51:13 +00:00
Yingwen
768239eb49 fix: panic on truncate table in distributed mode (#2173) 2023-08-14 14:20:20 +00:00
Ning Sun
f3157df190 fix: normalize otlp string keys (#2168) 2023-08-14 09:39:54 +00:00
dennis zhuang
b353bd20db fix: print_anonymous_usage_data_disclaimer at wrong place (#2167) 2023-08-14 08:01:10 +00:00
Lei, HUANG
55b5df9c51 feat: row wise converter (#2162) 
* feat: impl mem-comparable encoding for timestamp

* fix: test cases

* impl time series encode/decoder

* fix: merge unsupported match arms

* fix: clippy

* chore: big number delimiter

* feat: encode timestamps as i64

* fix: remove useless error variant
 2023-08-14 07:13:39 +00:00
Ruihang Xia
393047a541 feat: implement metric for MergeScanExec (#2166) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-14 07:10:45 +00:00
LFC
606b489d53 feat: redact secrets in sql when logging (#2141) 2023-08-14 06:40:00 +00:00
Weny Xu
d0b3607633 feat: add table route manager and upgrade tool (#2145) 
* feat: add table route manager and upgrade tool

* test: add table route manager tests

* feat: add new TableRouteValue struct

* chore: apply suggestions from CR

* refactor: change HashMap to BTreeMap

* feat: add version to TableRouteValue
 2023-08-14 04:19:44 +00:00
Weny Xu
5b012a1f67 feat!: switch to new catalog/schema key (#2140) 
* feat!: switch to new catalog/schema key

* chore: apply suggestions from CR
 2023-08-14 03:08:43 +00:00
Ruihang Xia
f6b53984da fix(metasrv)!: do not overwrite boolean options unconditionally (#2161) 
* fix: do not overwrite boolean options unconditionally

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness start command

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-14 03:04:54 +00:00
shuiyisong
7f51141ed0 refactor: auth crate (#2148) 
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
 2023-08-14 02:51:26 +00:00
Ruihang Xia
6d64e1c296 feat(mito): checkpoint for mito2 (#2142) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust dir structure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* downgrade log level

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply CR sugg.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add region id to log

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
v0.4.0-nightly-20230814 		2023-08-13 09:26:01 +00:00
Yingwen
e6090a8d5b feat(mito): Write wal and memtable (#2135) 
* feat: hold wal entry in RegionWriteCtx

* feat: entry id and commited sequence

* feat: write to wal

* feat: write memtable

* feat: fill missing columns

* feat: validate write request

* feat: more validation to write request

* chore: fix typos

* feat: remove init and validate rows in new()

* style: fix clippy
 2023-08-12 07:44:44 +00:00
谢政
b62e643e92 build: update protobuf-build to support apple silicon (#2143) 
* build: update protobuf-build to support apple silicon

* build: Update src/log-store/Cargo.toml

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* build: update the Cargo.lock too

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-12 03:31:51 +00:00