Commit Graph

964 Commits

Author SHA1 Message Date
Ning Sun
ec1b95c250 docs: add play section (#1528)
* docs: add play section

* Update README.md

Co-authored-by: xiaomin tang <xtang@users.noreply.github.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: xiaomin tang <xtang@users.noreply.github.com>
2023-05-08 14:26:22 +08:00
gitccl
fbf1ddd006 feat: open catalogs and schemas in parallel (#1527)
* feat: open catalogs and schemas in parallel

* fix: code review
2023-05-08 10:34:30 +08:00
Ning Sun
d679cfcb53 feat: add semantic_type to information_schema.columns (#1530) 2023-05-06 15:48:37 +08:00
discord9
2c82ded975 feat: table metrics (#1469)
* feat: Statistic

* add todo

* fmt: cargo fmt

* feat: some simple impl for MemTable

* chore: a try on adding statistics

* Update src/table/src/stats.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* docs: fix typo

* newlines unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-06 14:59:49 +08:00
Ruihang Xia
d4f3f617e4 chore(toolchain): update rust-toolchain to 2023-05-03 (#1524)
* chore(toolchain): update rust-toolchain to 2023-05-03

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update workflow yaml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-05-06 11:34:09 +08:00
Ruihang Xia
6fe117d7d5 fix: vector and matrix in Prometheus use different field (#1520)
* fix empty tag

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix result type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* make it work

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-05-05 15:54:26 +08:00
Ning Sun
b0ab641602 feat: add catalog/schema/table count as catalog metrics (#1499)
* feat: add catalog/schema/table count as catalog metrics

* test: add integration tests for catalog metrics
2023-05-05 05:54:12 +00:00
Huaijin
224ec9bd25 fix: wrong max_table_id log in remote catalog manager (#1516)
* fix: wrong max_table_id log in remote catalog manager

* chore: update link in CONTRIBUTING.md

* chore: add a new const MAX_SYS_TABLE_ID
2023-05-05 03:39:45 +00:00
Niwaka
d86b3386dc fix: incorrect show create table output (#1514)
* fix: incorrect show create table output

* feat: change CreateTable's Display if table is external

* feat: change CreateTable's Display if table is external
2023-05-05 11:29:09 +08:00
Lei, HUANG
c8301feed7 fix: respect MySQL timestamp format (#1510) 2023-05-04 18:57:38 +08:00
dennis zhuang
b1920c41a4 fix: object store cache bug (#1482)
* feat: use streaming read instead of reading whole file

* feat: enable atomic writing for object store file caching

* fix: recover existing keys from local cache

* test: recovering keys from local file cache for LruCachePolicy

* Update src/datanode/src/instance.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr comments

* feat: md5 hash caching path

* fix: test

* fix: read cache

* Update src/object-store/src/cache_policy.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2023-05-04 18:25:40 +08:00
Weny Xu
c471007edd feat: support to copy table from/to CSV and JSON format file (#1475)
* refactor: refactor copy from executor

* feat: support to copy from CSV and JSON format files

* feat: support to copy table to the CSV and JSON format file

* test: add tests copy from/to

* chore: apply suggestions from CR
2023-05-04 17:20:28 +08:00
Yingwen
2818f466d3 feat: Log error in GreptimeRequestHandler (#1507)
* feat(common-error): Add should_log_error

* feat(servers): log error in grpc handler
2023-05-04 15:48:38 +08:00
JeremyHi
d7a906e0bd feat: metasrv mailbox (#1481)
* refactor: id first in pusher_key

* feat: is_acceptable for multi roles

* feat: mailbox

* fix: channel for mailbox

* feat: impl mailbox via heartbeat

* chore: add unit test for mailbox

* chore: by cr

* chore: typo

* chore: refactor the mailbox API

* chore: br cr

* chore: check timeout interval to 10ms

* chore: add response header
2023-05-04 15:42:43 +08:00
Ning Sun
6e1bb9e458 feat: add support for information_schema.columns (#1500)
* feat: add support for information_schema.columns

* feat: remove information_schema from its view

* Update src/catalog/src/information_schema.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: error on table data type

* test: correct sqlness test for information schema

* test: add information_schema.columns sqlness tests

---------

Co-authored-by: LFC <bayinamine@gmail.com>
2023-05-04 14:29:38 +08:00
Ning Sun
494ad570c5 feat: update pgwire to 0.14 (#1504) 2023-05-04 14:24:26 +08:00
Vanish
12d59e6341 chore: remove redundant code. (#1502) 2023-05-04 14:20:26 +08:00
Yingwen
479ef9d379 fix: checkpoint GC task also deletes the file with the last version (#1491)
* test(storage): use assert_eq to check scan result

* feat(storage): Add more info to manifest log

* feat: Avoid error log when unable to delete

* fix: The manifest gc task should delete files <= last_version

* feat(storage): Don't log if the error kind is not found

* feat: Add keep_last_checkpoint option
2023-05-04 14:18:38 +08:00
Niwaka
93ffe1ff33 feat: improve and distinguish different errors for IllegalInsertData (#1503)
* feat: improve and distinguish different errors for IllegalInsertData

* feat: change error code for UnexpectedValuesLength and ColumnAlreadyExists

* chore: improve readability of error message
2023-05-04 12:36:24 +08:00
Niwaka
d461328238 fix: insert distributed table if partition column has default value (#1498)
* fix: insert distributed table if partition column has default value

* Address review

* address review

* address review

* chore: introduce assert_columns

---------

Co-authored-by: WenyXu <wenymedia@gmail.com>
2023-05-02 20:50:02 +08:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501)
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
2023-05-01 20:54:54 +08:00
Ning Sun
7dbac89000 feat: add metrics for protocol interfaces (#1495)
* feat: add metrics for various interfaces

* feat: add db label for protocols

* feat: add postgres protocol metrics

* feat: add metrics for grpcs apis

* feat: add auth failure counter for mysql/pg

* fix: add db label to grpc prometheus interface

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: add error code for auth failure counter

* fix: use schema as dbname when catalog is default

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-04-28 23:42:35 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489)
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
2023-04-28 22:12:57 +08:00
Yingwen
51be35a7b1 feat(mito): Combine the original and procedure's implementation (#1468)
* fix(mito): Add metrics to mito DDL procedure

* feat(mito): Use procedure's implementation to create table

* feat(mito): Use procedure's implementation to alter table

* feat(mito): Use procedure's implementation to drop table

* style(mito): Fix clippy

* test(mito): Fix tests

* feat(mito): Add TableCreator

* feat(mito): update alter table procedure

* fix(mito): alter procedure create alter op first

* feat(mito): Combine alter table code

* fix(mito): Fix deadlock

* feat(mito): Simplify drop table procedure
2023-04-28 11:48:52 +08:00
Lei, HUANG
9e4887f29f fix: disable dashboard (#1494) 2023-04-27 22:55:15 +08:00
yuanbohan
cca34aa914 chore: upgrade promql-parser version (#1484) 2023-04-27 13:10:15 +00:00
Ruihang Xia
0ac50632aa feat: use server time if it's not specified (#1480)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-27 20:54:26 +08:00
Yingwen
b1f7ad097a test: Fix s3 region in test (#1493) 2023-04-27 12:25:20 +00:00
Weny Xu
a77a4a4bd1 fix: add s3 region info (#1492)
fix: add region info
2023-04-27 19:13:01 +08:00
Weny Xu
47f1cbaaed fix: add s3 region info (#1486) 2023-04-27 17:35:34 +08:00
Yingwen
8e3c3cbc40 build: Download assets to cargo output dir (#1476)
* build: Download assets to cargo output dir

Also remove the output from the build script and only print the output
on failure

* chore: Update src/servers/build.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* build: replace pushd by cd

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-27 17:09:10 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444)
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
2023-04-27 15:30:04 +08:00
Ruihang Xia
939a51aea9 feat: adopt REPLACE interceptor and quit all processes on exit (#1478)
* bump version and update test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* quit all processes on drop

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update tests/runner/src/env.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2023-04-27 07:16:41 +00:00
Weny Xu
bf35620904 refactor: refactor BufferedWriter (#1439)
* feat: implement ApproximateBufWriter

* refactor: refactor BufferedWriter

* refactor: remove ApproximateBufWriter

* fix: fix losing pending writes issue

* chore: fmt

* chore: remove unused import

* chore: rename method name

* feat: return written row count

* chore: apply suggestions from CR

* fix: fix counting the bytes_written twice issue
2023-04-27 14:45:33 +08:00
Weny Xu
09f55e3cd8 chore: remove info log (#1483) 2023-04-27 14:05:22 +08:00
dennis zhuang
b88d8e5b82 feat: bump opendal to 0.33 (#1479) 2023-04-27 12:13:18 +08:00
Weny Xu
a709a5c842 feat: support to create parquet format external table (#1463)
* feat: support parquet format external table

* Update src/file-table-engine/src/error.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-26 16:45:37 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454)
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
2023-04-26 08:36:40 +00:00
discord9
ef4e473e6d fix: recompile&register scripts as UDF on reboot (#1421)
* fixme: recompile somewhere else

* feat: re-compile&re-register all scripts in table

* fix: allow empty scripts table

* chore: add non-blocking somewhere

* chore: PR advices

* chore: more PR advices

* style: remove useless join

* style: remove redunent code

* refactor: use `bg` runtime instead

* style: cargo fmt
2023-04-26 16:30:58 +08:00
Ning Sun
1a245f35b9 feat: improve metrics and log level (#1470)
* refactor: tune log and metrics for meta/frontend

* feat: add panic counter
2023-04-26 13:13:40 +08:00
dennis zhuang
8d8a480dc1 fix: object store caching bug, #1466 (#1467)
* fix: object store caching bug, #1466

* fix: forgot to add S3WithCache tests
2023-04-25 21:48:51 +08:00
Lei, HUANG
197c34bc17 fix: grpc client keepalive (#1461)
fix: grpc keepalive
2023-04-25 20:07:57 +08:00
Ruihang Xia
4d9afee8ef chore(deps): update substrait dep in client (#1453)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-25 16:21:59 +08:00
Weny Xu
7f14d40798 test: add tests for external table (#1460) 2023-04-25 15:14:46 +08:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448)
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
2023-04-25 12:04:02 +08:00
Lei, HUANG
92c0808766 fix: frontend opt should respect http addr in config file when no com… (#1456)
* fix: frontend opt should respect http addr in config file when no command options is given

* refactor: command line options should be Option<bool>

* fix: ci
2023-04-25 03:43:42 +00:00
Ruihang Xia
f9ea6b63bf feat: impl instant query and add tests (#1452)
* feat: impl instant query and add tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-25 11:08:14 +08:00
fys
2287db7ff7 fix: execute sql query in another catalog (#1457) 2023-04-25 10:30:35 +08:00
shuiyisong
69acf32914 chore: add len() to Bytes and StringBytes (#1455)
* chore: add `len()` to Bytes and StringBytes

* chore: add `len()` to Bytes and StringBytes
2023-04-25 10:18:41 +08:00
Ruihang Xia
b9db2cfd83 fix: support restart sqlness in distributed mode (#1443)
* fix: support restart sqlness in distributed mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move alter_table case to common dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* is_standalone flag

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: LFC <bayinamine@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: LFC <bayinamine@gmail.com>
2023-04-24 19:36:12 +08:00