Commit Graph

950 Commits

Author SHA1 Message Date
Ning Sun
6e1bb9e458 feat: add support for information_schema.columns (#1500)
* feat: add support for information_schema.columns

* feat: remove information_schema from its view

* Update src/catalog/src/information_schema.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: error on table data type

* test: correct sqlness test for information schema

* test: add information_schema.columns sqlness tests

---------

Co-authored-by: LFC <bayinamine@gmail.com>
2023-05-04 14:29:38 +08:00
Ning Sun
494ad570c5 feat: update pgwire to 0.14 (#1504) 2023-05-04 14:24:26 +08:00
Vanish
12d59e6341 chore: remove redundant code. (#1502) 2023-05-04 14:20:26 +08:00
Yingwen
479ef9d379 fix: checkpoint GC task also deletes the file with the last version (#1491)
* test(storage): use assert_eq to check scan result

* feat(storage): Add more info to manifest log

* feat: Avoid error log when unable to delete

* fix: The manifest gc task should delete files <= last_version

* feat(storage): Don't log if the error kind is not found

* feat: Add keep_last_checkpoint option
2023-05-04 14:18:38 +08:00
Niwaka
93ffe1ff33 feat: improve and distinguish different errors for IllegalInsertData (#1503)
* feat: improve and distinguish different errors for IllegalInsertData

* feat: change error code for UnexpectedValuesLength and ColumnAlreadyExists

* chore: improve readability of error message
2023-05-04 12:36:24 +08:00
Niwaka
d461328238 fix: insert distributed table if partition column has default value (#1498)
* fix: insert distributed table if partition column has default value

* Address review

* address review

* address review

* chore: introduce assert_columns

---------

Co-authored-by: WenyXu <wenymedia@gmail.com>
2023-05-02 20:50:02 +08:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501)
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
2023-05-01 20:54:54 +08:00
Ning Sun
7dbac89000 feat: add metrics for protocol interfaces (#1495)
* feat: add metrics for various interfaces

* feat: add db label for protocols

* feat: add postgres protocol metrics

* feat: add metrics for grpcs apis

* feat: add auth failure counter for mysql/pg

* fix: add db label to grpc prometheus interface

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: add error code for auth failure counter

* fix: use schema as dbname when catalog is default

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-04-28 23:42:35 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489)
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
2023-04-28 22:12:57 +08:00
Yingwen
51be35a7b1 feat(mito): Combine the original and procedure's implementation (#1468)
* fix(mito): Add metrics to mito DDL procedure

* feat(mito): Use procedure's implementation to create table

* feat(mito): Use procedure's implementation to alter table

* feat(mito): Use procedure's implementation to drop table

* style(mito): Fix clippy

* test(mito): Fix tests

* feat(mito): Add TableCreator

* feat(mito): update alter table procedure

* fix(mito): alter procedure create alter op first

* feat(mito): Combine alter table code

* fix(mito): Fix deadlock

* feat(mito): Simplify drop table procedure
2023-04-28 11:48:52 +08:00
Lei, HUANG
9e4887f29f fix: disable dashboard (#1494) 2023-04-27 22:55:15 +08:00
yuanbohan
cca34aa914 chore: upgrade promql-parser version (#1484) 2023-04-27 13:10:15 +00:00
Ruihang Xia
0ac50632aa feat: use server time if it's not specified (#1480)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-27 20:54:26 +08:00
Yingwen
b1f7ad097a test: Fix s3 region in test (#1493) 2023-04-27 12:25:20 +00:00
Weny Xu
a77a4a4bd1 fix: add s3 region info (#1492)
fix: add region info
2023-04-27 19:13:01 +08:00
Weny Xu
47f1cbaaed fix: add s3 region info (#1486) 2023-04-27 17:35:34 +08:00
Yingwen
8e3c3cbc40 build: Download assets to cargo output dir (#1476)
* build: Download assets to cargo output dir

Also remove the output from the build script and only print the output
on failure

* chore: Update src/servers/build.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* build: replace pushd by cd

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-27 17:09:10 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444)
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
2023-04-27 15:30:04 +08:00
Ruihang Xia
939a51aea9 feat: adopt REPLACE interceptor and quit all processes on exit (#1478)
* bump version and update test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* quit all processes on drop

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update tests/runner/src/env.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2023-04-27 07:16:41 +00:00
Weny Xu
bf35620904 refactor: refactor BufferedWriter (#1439)
* feat: implement ApproximateBufWriter

* refactor: refactor BufferedWriter

* refactor: remove ApproximateBufWriter

* fix: fix losing pending writes issue

* chore: fmt

* chore: remove unused import

* chore: rename method name

* feat: return written row count

* chore: apply suggestions from CR

* fix: fix counting the bytes_written twice issue
2023-04-27 14:45:33 +08:00
Weny Xu
09f55e3cd8 chore: remove info log (#1483) 2023-04-27 14:05:22 +08:00
dennis zhuang
b88d8e5b82 feat: bump opendal to 0.33 (#1479) 2023-04-27 12:13:18 +08:00
Weny Xu
a709a5c842 feat: support to create parquet format external table (#1463)
* feat: support parquet format external table

* Update src/file-table-engine/src/error.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-26 16:45:37 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454)
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
2023-04-26 08:36:40 +00:00
discord9
ef4e473e6d fix: recompile&register scripts as UDF on reboot (#1421)
* fixme: recompile somewhere else

* feat: re-compile&re-register all scripts in table

* fix: allow empty scripts table

* chore: add non-blocking somewhere

* chore: PR advices

* chore: more PR advices

* style: remove useless join

* style: remove redunent code

* refactor: use `bg` runtime instead

* style: cargo fmt
2023-04-26 16:30:58 +08:00
Ning Sun
1a245f35b9 feat: improve metrics and log level (#1470)
* refactor: tune log and metrics for meta/frontend

* feat: add panic counter
2023-04-26 13:13:40 +08:00
dennis zhuang
8d8a480dc1 fix: object store caching bug, #1466 (#1467)
* fix: object store caching bug, #1466

* fix: forgot to add S3WithCache tests
2023-04-25 21:48:51 +08:00
Lei, HUANG
197c34bc17 fix: grpc client keepalive (#1461)
fix: grpc keepalive
2023-04-25 20:07:57 +08:00
Ruihang Xia
4d9afee8ef chore(deps): update substrait dep in client (#1453)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-25 16:21:59 +08:00
Weny Xu
7f14d40798 test: add tests for external table (#1460) 2023-04-25 15:14:46 +08:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448)
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
2023-04-25 12:04:02 +08:00
Lei, HUANG
92c0808766 fix: frontend opt should respect http addr in config file when no com… (#1456)
* fix: frontend opt should respect http addr in config file when no command options is given

* refactor: command line options should be Option<bool>

* fix: ci
2023-04-25 03:43:42 +00:00
Ruihang Xia
f9ea6b63bf feat: impl instant query and add tests (#1452)
* feat: impl instant query and add tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-25 11:08:14 +08:00
fys
2287db7ff7 fix: execute sql query in another catalog (#1457) 2023-04-25 10:30:35 +08:00
shuiyisong
69acf32914 chore: add len() to Bytes and StringBytes (#1455)
* chore: add `len()` to Bytes and StringBytes

* chore: add `len()` to Bytes and StringBytes
2023-04-25 10:18:41 +08:00
Ruihang Xia
b9db2cfd83 fix: support restart sqlness in distributed mode (#1443)
* fix: support restart sqlness in distributed mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move alter_table case to common dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* is_standalone flag

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: LFC <bayinamine@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: LFC <bayinamine@gmail.com>
2023-04-24 19:36:12 +08:00
JeremyHi
6d247f73fd fix: add log on leader stepdown (#1450) 2023-04-24 19:16:57 +08:00
Ruihang Xia
2cf828da3c feat: implement Prometheus-compatible API in gRPC (#1449)
* update greptime-proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove duplicate delete enum

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl handler and service

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-24 18:03:48 +08:00
Weny Xu
f2167663b2 feat: support to create external table (#1372)
* feat: support to create external table

* chore: apply suggestions from CR

* test: add create external table without ts type

* chore: apply suggestions from CR

* fix: fix import typo

* refactor: move consts to table crate

* chore: apply suggestions from CR

* refactor: rename create_table_schema
2023-04-24 14:43:12 +08:00
LFC
17daf4cdff feat: support "delete" in distributed mode (#1441)
* feat: support "delete" in distributed mode

* fix: resolve PR comments
2023-04-24 12:07:50 +08:00
shuiyisong
7c6754d03e feat: meter write request (#1447)
* chore: add write meter

* chore: update meter macro

* chore: update meter framework url to https
2023-04-24 11:42:06 +08:00
zyy17
e64fea3a15 ci: upgrade nightly release tag from v0.2.0 to v0.3.0 (#1446) 2023-04-24 11:04:39 +08:00
Weny Xu
22b5a94d02 feat: support creating the physical plan for JSON and CSV files (#1424)
* feat: support creating the physical plan for JSON and CSV files

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor(file-table-engine): use datasource Format instead
2023-04-24 10:17:11 +08:00
Weny Xu
d374859e24 refactor: replace Copy Format with datasource Format (#1435)
* refactor: replace Copy Format with datasource Format

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2023-04-23 08:31:54 +00:00
Ning Sun
c5dba29f9e refactor: remove redundant plugins argument (#1436) 2023-04-23 12:39:46 +08:00
Hao
9f442dedf9 chore: fix some typo and add deriv to plan in promql (#1438) 2023-04-23 12:21:25 +08:00
Ruihang Xia
5d77ed00bb test: add basic cases for distributed TQL (#1437)
* test: add basic cases for distributed TQL

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* drop table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-23 03:34:42 +00:00
Zheming Li
c75845c570 fix: wrong next column in manifest (#1440)
Signed-off-by: Zheming Li <nkdudu@126.com>
2023-04-23 11:25:38 +08:00
Yingwen
1ee9ad4ca1 feat: manage multiple engine procedure in the engine manager (#1434)
* feat(table): Add engine procedure to engine manager

* feat(datanode): Get engine procedure from engine manager

* feat(table-procedure): Add source error to SubprocedureFailed

* test: Enable procedure in tests and pass all tests

* style(table-procedure): Fix clippy
2023-04-23 10:04:09 +08:00
Weny Xu
f2cc912c87 feat: implement ParquetFileReaderFactory (#1423)
* feat: implement ParquetFileReaderFactory

* refactor: use LazyParquetFileReader instead

* chore: apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2023-04-21 13:40:58 +08:00