* refactor: dependency, from frontend depends on datanode to datanode depends on frontend
* wip: start frontend in datanode
* wip: migrate create database to frontend
* wip: impl alter table
* fix: CR comments
* feat: move time index metadata from schema into field
* chore: remove useless code
* test: test select with column alias
* fix: conflicts with develop branch
* test: add test
* test: order by timestamp to ensure query results order
* fix: comment
* add line_writer and convert insert_stmt to InsertRequest
* support convert influxdb line protocol to InsertRequest
* support convert opentsdb to InsertRequest
* cr
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: databode lease
* feat: remote catalog (#356)
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: datanode heartbeat (#355)
* feat: add heartbeat task to instance
* feat: add node_id datanode opts
* fix: use real node id in heartbeat and meta client
* feat: distribute table in frontend
* test: distribute read demo
* test: distribute read demo
* test: distribute read demo
* add write spliter
* fix: node id changed to u64
* feat: datanode uses remote catalog implementation
* dist insert integrate table
* feat: specify region ids on creating table (#359)
* fix: compiling issues
* feat: datanode lease (#354)
* Some glue code about dist_insert
* fix: correctly wrap string value with quotes
* feat: create route
* feat: frontend catalog (#362)
* feat: integrate catalog to frontend
* feat: preserve partition rule on create
* fix: print tables on start
* chore: log in create route
* test: distribute read demo
* feat: support metasrv addr command line options
* feat: optimize DataNodeInstance creation (#368)
* chore: remove unnecessary changes
* chore: revert changes to src/api
* chore: revert changes to src/datanode/src/server.rs
* chore: remove opendal backend
* chore: optimize imports
* chore: revert changes to instance and region ids
* refactor: MetaKvBackend range
* fix: remove some wrap
* refactor: initiation of catalog
* feat: add region id to create table request and add heartbeat task to datanode instance
* fix: fix auto reconnect for heartbeat task
* chore: change TableValue::region_numbers to vec<u32>.
* fix: some tests
* fix: avoid concurrently start Heartbeat task by compare_exchange
* fix: some cr comments
* fix: fix unit tests
Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* fix: Fix TestGuard being dropped before grpc test starts
* feat: Let start and shutdown takes immutable reference to self
Also implement shutdown for GrpcServer
* feat: Implement shutdown for HttpServer
* style: Fix clippy
* chore: Add name to AlreadyStarted error
This patch changes output for our http SQL API and prepare it for our SQL editor development. Changes includes:
- includes aide for OAS 3.1 openapi documents, available at /v1/private/api.json
- simplified some of http handlers return type, use string or json directly
- created new HttpRecordsOutput type to hide internals of RecordBatch from end-user. It also tuned data structure to be friendly for application to consume
- updated response struct to use code for success or detailed error code
Residual issue #366
* feat: allow http post for our sql http api
* feat: update our http api and attempt to add openapi spec support
* test: correct test against new handler apis
* refactor: rename rows to records
* refactor: removed HttpResponse completely
* feat: add information to our openapi docs
* feat: add docs for sql interface response
* refactor: use struct to represent query so we can doc it via aide
* refactor: use arc wrapped api
* feat: add redoc UI support
* Update src/servers/src/http.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* Update src/servers/src/http.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* fix: address review comments
* test: update integration tests for new api output
* refactor: make prometheus http apis compatible with recent changes
* refactor: get schema from stream
* test: add test for recordbatch to json serialization
* test: add todo for a test to be fixed later
* Revert "test: add todo for a test to be fixed later"
This reverts commit a5a50c7afb.
* fix: Revert "refactor: get schema from stream"
This reverts commit 945b685556.
* chore: add todo for pending issue #366
* chore: remove fixed server url in openapi docs
* feat: include error_code in json response
* refactor: use code over success field in json response
Co-authored-by: LFC <bayinamine@gmail.com>
* feat: scaffold for prometheus protocol handler
* feat: impl remote write and read for prometheus
* chore: make label matchers working in remote reading
* chore: case senstive regexp matching for labers and tweak restful api
* test: prometheus test
* test: adds test for prometheus handler and http server
* fix: typo in comment
* refactor: move snappy_compress and snappy_decompress
* fix: by code review
* fix: collect_timeseries_ids
* fix: timestamp and value column's value may be null
* refactor: add table_info method for Table trait
* feat: add table_info method to Table trait
* test: add more unit test
* fix: impl table_info for SystemTable
* test: fix failing test
* feat: Adds ColumnDefaultConstraint::create_default_vector
ColumnDefaultConstraint::create_default_vector is ported from
MitoTable::try_get_column_default_constraint_vector.
* refactor: Replace try_get_column_default_constraint_vector by create_default_vector
* style: Remove unnecessary map_err in MitoTable::insert
* feat: Adds compat_write
For column in `dest_schema` but not in `write_batch`, this method would insert a
vector with default value to the `write_batch`. If there are columns not in
`dest_schema`, an error would be returned.
* chore: Add info log to RegionInner::alter
* feat(storage): RegionImpl::write support request with old version
* feat: Add nullable check when creating default value
* feat: Validate nullable and default value
* chore: Modify PutOperation comments
* chore: Make ColumnDescriptor::is_nullable readonly and validate name
* feat: Use CompatWrite trait to replace campat::compat_write method
Adds a CompactWrite trait to support padding columns to WriteBatch:
- The WriteBatch and PutData implements this trait
- Fix the issue that WriteBatch::schema is not updated to the
schema after compat
- Also validate the created column when adding to PutData
The WriteBatch would also pad default value to missing columns in
PutData, so the memtable inserter don't need to manually check whether
the column is nullable and then insert a NullVector. All WriteBatch is
ensured to have all columns defined by the schema in its PutData.
* feat: Validate constraint by ColumnDefaultConstraint::validate()
The ColumnDefaultConstraint::validate() would also ensure the default
value has the same data type as the column's.
* feat: Use NullVector for null columns
* fix: Fix BinaryType returns wrong logical_type_id
* fix: Fix tests and revert NullVector for null columns
NullVector doesn't support custom logical type make it hard to
encode/decode, which also cause the arrow/protobuf codec of write batch
fail.
* fix: create_default_vector use replicate to create vector with default value
This would fix the test_codec_with_none_column_protobuf test, as we need
to downcast the vector to construct the protobuf values.
* test: add tests for column default constraints
* test: Add tests for CompatWrite trait impl
* test: Test write region with old schema
* fix(storage): Fix replay() applies metadata too early
The committed sequence of the RegionChange action is the sequence of the
last entry that use the old metadata (schema). During replay, we should
apply the new metadata after we see an entry that has sequence greater
than (not equals to) the `RegionChange::committed_sequence`
Also remove duplicate `set_committed_sequence()` call in
persist_manifest_version()
* chore: Removes some unreachable codes
Also add more comments to document codes in these files
* refactor: Refactor MitoTable::insert
Return error if we could not create a default vector for given column,
instead of ignoring the error
* chore: Fix incorrect comments
* chore: Fix typo in error message
* feat: frontend instance
* no need to carry column length in `Column` proto
* add more tests
* rebase develop
* create a new variant with already provisioned RecordBatches in Output
* resolve code review comments
* new frontend instance does not connect datanode grpc
* add more tests
* add more tests
* rebase develop
Co-authored-by: luofucong <luofucong@greptime.com>
