* feat: mysql and pg server support tls
* chore: replace opensrv-mysql to original
* chore: TlsOption is required but supply default value
* feat: mysql server support force tls
* chore: move TlsOption to servers
* test: mysql server disable / prefer / required tls mode
* test: pg server disable / prefer / required tls mode
* chore: add doc and remove no used code
* chore: add TODO and restore cargo linker config
* deps: Bump OpenDAL to v0.21.1
Signed-off-by: Xuanwo <github@xuanwo.io>
* Avoid using raw types when not needed
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
* refactor: options and sample configurations
* chore: newline at end of file
* chore: format code
* chore: remove comment and set sample configurations to default values
* chore: use single quoted string in sample configuration files
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend
* wip: start frontend in datanode
* wip: migrate create database to frontend
* wip: impl alter table
* fix: CR comments
* feat: add table id and region ids field to CreateExpr
* chore: rebase develop
* refactor: frontend catalog should set from datanode
* feat: gRPC AddColumn request support add multi columns
* wip: move create table and create-on-insertion to frontend
* wip: error handling
* fix: some unit tests
* fix: all unit tests
* chore: merge develop
* feat: add create/alter-on-insertion to dist_insert/sql_dist_insert
* fix: add region number/catalog/schema to InsertExpr
* feat: add handle_create_table/handle_create_database...
* fix: remove catalog from insert expr
* fix: CR comments
* fix: when running in standalone mode, mysql opts and postgres opts should pass to frontend so that auctually running service can change the port to listen on
* refactor: add a standalone subcommand, move frontend start stuff to cmd package
* chore: optimize create table failure logs
* docs: change readme
* docs: update readme
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend
* wip: start frontend in datanode
* wip: migrate create database to frontend
* wip: impl alter table
* fix: CR comments
* add line_writer and convert insert_stmt to InsertRequest
* support convert influxdb line protocol to InsertRequest
* support convert opentsdb to InsertRequest
* cr
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: databode lease
* feat: remote catalog (#356)
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: datanode heartbeat (#355)
* feat: add heartbeat task to instance
* feat: add node_id datanode opts
* fix: use real node id in heartbeat and meta client
* feat: distribute table in frontend
* test: distribute read demo
* test: distribute read demo
* test: distribute read demo
* add write spliter
* fix: node id changed to u64
* feat: datanode uses remote catalog implementation
* dist insert integrate table
* feat: specify region ids on creating table (#359)
* fix: compiling issues
* feat: datanode lease (#354)
* Some glue code about dist_insert
* fix: correctly wrap string value with quotes
* feat: create route
* feat: frontend catalog (#362)
* feat: integrate catalog to frontend
* feat: preserve partition rule on create
* fix: print tables on start
* chore: log in create route
* test: distribute read demo
* feat: support metasrv addr command line options
* feat: optimize DataNodeInstance creation (#368)
* chore: remove unnecessary changes
* chore: revert changes to src/api
* chore: revert changes to src/datanode/src/server.rs
* chore: remove opendal backend
* chore: optimize imports
* chore: revert changes to instance and region ids
* refactor: MetaKvBackend range
* fix: remove some wrap
* refactor: initiation of catalog
* feat: add region id to create table request and add heartbeat task to datanode instance
* fix: fix auto reconnect for heartbeat task
* chore: change TableValue::region_numbers to vec<u32>.
* fix: some tests
* fix: avoid concurrently start Heartbeat task by compare_exchange
* feat: refactor catalog key and values, separate table info into two kinds of keys
* feat: bump table id from metasrv
* fix: compare and set table id
* chore: merge develop
* fix: use integer serialization instead of string serialization
Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: databode lease
* feat: remote catalog (#356)
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: datanode heartbeat (#355)
* feat: add heartbeat task to instance
* feat: add node_id datanode opts
* fix: use real node id in heartbeat and meta client
* feat: distribute table in frontend
* test: distribute read demo
* test: distribute read demo
* test: distribute read demo
* add write spliter
* fix: node id changed to u64
* feat: datanode uses remote catalog implementation
* dist insert integrate table
* feat: specify region ids on creating table (#359)
* fix: compiling issues
* feat: datanode lease (#354)
* Some glue code about dist_insert
* fix: correctly wrap string value with quotes
* feat: create route
* feat: frontend catalog (#362)
* feat: integrate catalog to frontend
* feat: preserve partition rule on create
* fix: print tables on start
* chore: log in create route
* test: distribute read demo
* feat: support metasrv addr command line options
* feat: optimize DataNodeInstance creation (#368)
* chore: remove unnecessary changes
* chore: revert changes to src/api
* chore: revert changes to src/datanode/src/server.rs
* chore: remove opendal backend
* chore: optimize imports
* chore: revert changes to instance and region ids
* refactor: MetaKvBackend range
* fix: remove some wrap
* refactor: initiation of catalog
* feat: add region id to create table request and add heartbeat task to datanode instance
* fix: fix auto reconnect for heartbeat task
* chore: change TableValue::region_numbers to vec<u32>.
* fix: some tests
* fix: avoid concurrently start Heartbeat task by compare_exchange
* fix: some cr comments
* fix: fix unit tests
Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: databode lease
* feat: remote catalog (#356)
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: datanode heartbeat (#355)
* feat: add heartbeat task to instance
* feat: add node_id datanode opts
* fix: use real node id in heartbeat and meta client
* feat: distribute table in frontend
* test: distribute read demo
* test: distribute read demo
* test: distribute read demo
* add write spliter
* fix: node id changed to u64
* feat: datanode uses remote catalog implementation
* dist insert integrate table
* feat: specify region ids on creating table (#359)
* fix: compiling issues
* feat: datanode lease (#354)
* Some glue code about dist_insert
* fix: correctly wrap string value with quotes
* feat: create route
* feat: frontend catalog (#362)
* feat: integrate catalog to frontend
* feat: preserve partition rule on create
* fix: print tables on start
* chore: log in create route
* test: distribute read demo
* feat: support metasrv addr command line options
* feat: optimize DataNodeInstance creation (#368)
* chore: remove unnecessary changes
* chore: revert changes to src/api
* chore: revert changes to src/datanode/src/server.rs
* chore: remove opendal backend
* chore: optimize imports
* chore: revert changes to instance and region ids
* refactor: MetaKvBackend range
* fix: remove some wrap
* refactor: initiation of catalog
* fix: next range request start key
* fix: mock delete range
* refactor: simplify range response handling
Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* chore: meta mock
* chore: refacor datanode selector
* chore: create route mock test
* chore: add mock module
* chore: memory store for test
* chore: mock meta for test
* chore: ensure memorysotre has the same behavious with etcd
* chore: replace tokio lock to parking_lot
This patch changes output for our http SQL API and prepare it for our SQL editor development. Changes includes:
- includes aide for OAS 3.1 openapi documents, available at /v1/private/api.json
- simplified some of http handlers return type, use string or json directly
- created new HttpRecordsOutput type to hide internals of RecordBatch from end-user. It also tuned data structure to be friendly for application to consume
- updated response struct to use code for success or detailed error code
Residual issue #366
* feat: allow http post for our sql http api
* feat: update our http api and attempt to add openapi spec support
* test: correct test against new handler apis
* refactor: rename rows to records
* refactor: removed HttpResponse completely
* feat: add information to our openapi docs
* feat: add docs for sql interface response
* refactor: use struct to represent query so we can doc it via aide
* refactor: use arc wrapped api
* feat: add redoc UI support
* Update src/servers/src/http.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* Update src/servers/src/http.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* fix: address review comments
* test: update integration tests for new api output
* refactor: make prometheus http apis compatible with recent changes
* refactor: get schema from stream
* test: add test for recordbatch to json serialization
* test: add todo for a test to be fixed later
* Revert "test: add todo for a test to be fixed later"
This reverts commit a5a50c7afb.
* fix: Revert "refactor: get schema from stream"
This reverts commit 945b685556.
* chore: add todo for pending issue #366
* chore: remove fixed server url in openapi docs
* feat: include error_code in json response
* refactor: use code over success field in json response
Co-authored-by: LFC <bayinamine@gmail.com>
* feat: scaffold for prometheus protocol handler
* feat: impl remote write and read for prometheus
* chore: make label matchers working in remote reading
* chore: case senstive regexp matching for labers and tweak restful api
* test: prometheus test
* test: adds test for prometheus handler and http server
* fix: typo in comment
* refactor: move snappy_compress and snappy_decompress
* fix: by code review
* fix: collect_timeseries_ids
* fix: timestamp and value column's value may be null
