* chore: Remove unused MutationExtra
* refactor(storage): Refactor Mutation and Payload
Change Mutation from enum to a struct that holds op type and record
batches so the encoder don't need to convert the mutation into record
batch. Now The Payload is no more an enum, it just holds the data, to
be serialized to the WAL, of the WriteBatch. The encoder and decoder
now deal with the Payload instead of the WriteBatch, so we could hold
more information not necessary to be stored to the WAL in the
WriteBatch.
This commit also merge variants in write_batch::Error to storage::Error
as some variants of them denote the same error.
* test(storage): Pass all tests in storage
* chore: Remove unused codes then format codes
* test(storage): Fix test_put_unknown_column test
* style(storage): Fix clippy
* chore: Remove some unused codes
* chore: Rebase upstream and fix clippy
* chore(storage): Remove unused codes
* chore(storage): Update comments
* feat: Remove PayloadType from wal.proto
* chore: Address CR comments
* chore: Remove unused write_batch.proto
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: use Substrait logical plan to query data from Datanode in Frontend in distributed mode
* fix: resolve PR comments
* fix: resolve PR comments
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
* fix: table conflicts in different database, #483
* feat: support db query param in prometheus remoting read/write
* feat: support db query param in influxdb line protocol
* fix: make schema_name work in gRPC
* fix: table data path
* fix: table manifest dir
* feat: adds opendal logging layer to object store
* Update src/frontend/src/instance.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* Update src/frontend/src/instance.rs
Co-authored-by: LFC <bayinamine@gmail.com>
* Update src/servers/src/line_writer.rs
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
* Update src/servers/src/line_writer.rs
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
* fix: compile error
* ci: use larger runner for running coverage
* fix: address already in use in test
Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend
* wip: start frontend in datanode
* wip: migrate create database to frontend
* wip: impl alter table
* fix: CR comments
* feat: add table id and region ids field to CreateExpr
* chore: rebase develop
* refactor: frontend catalog should set from datanode
* feat: gRPC AddColumn request support add multi columns
* wip: move create table and create-on-insertion to frontend
* wip: error handling
* fix: some unit tests
* fix: all unit tests
* chore: merge develop
* feat: add create/alter-on-insertion to dist_insert/sql_dist_insert
* fix: add region number/catalog/schema to InsertExpr
* feat: add handle_create_table/handle_create_database...
* fix: remove catalog from insert expr
* fix: CR comments
* fix: when running in standalone mode, mysql opts and postgres opts should pass to frontend so that auctually running service can change the port to listen on
* refactor: add a standalone subcommand, move frontend start stuff to cmd package
* chore: optimize create table failure logs
* docs: change readme
* docs: update readme
* feat: move time index metadata from schema into field
* chore: remove useless code
* test: test select with column alias
* fix: conflicts with develop branch
* test: add test
* test: order by timestamp to ensure query results order
* fix: comment
* feat: Support removing columns from mito table
Implements drop column for mito table engine, and adjusts the execution
order of altering table, persists the table manifest first, then alter
the schema of the region.
* feat(storage): Remove duplicate table_info() impl
Table already provides a table_info() now, some downcast in tests are
also no longer needed.
* test: Add tests for add/remove columns
* style(table): Fix clippy
* fix: Find timestamp index by its column name
Previous implementation updates the timestamp index too early, which
would cause the index check that compare the index to remove with
timestamp index failed.
* chore: Remove generated comment in Cargo.toml
* chore: Rename alter to builder_with_alter_kind
* refactor: Alloc new column from TableMeta
* style: Fix clippy
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: databode lease
* feat: remote catalog (#356)
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* refactor: merge refactor/catalog-crate
* feat: table key with version
* feat: impl KvBackend for MetaClient
* fix: integrate metaclient
* fix: catalog use local table info as baseline
* fix: sync metsrv
* fix: wip
* fix: update remote catalog on register and deregister
* refactor: CatalogProvider
* refactor: CatalogManager
* fix: catalog key filtering
* fix: pass some test
* refactor: catalog iterating
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* chore: merge catalog crate
* fix: adapt to recent meta-client api change
* feat: datanode heartbeat (#355)
* feat: add heartbeat task to instance
* feat: add node_id datanode opts
* fix: use real node id in heartbeat and meta client
* feat: distribute table in frontend
* test: distribute read demo
* test: distribute read demo
* test: distribute read demo
* add write spliter
* fix: node id changed to u64
* feat: datanode uses remote catalog implementation
* dist insert integrate table
* feat: specify region ids on creating table (#359)
* fix: compiling issues
* feat: datanode lease (#354)
* Some glue code about dist_insert
* fix: correctly wrap string value with quotes
* feat: create route
* feat: frontend catalog (#362)
* feat: integrate catalog to frontend
* feat: preserve partition rule on create
* fix: print tables on start
* chore: log in create route
* test: distribute read demo
* feat: support metasrv addr command line options
* feat: optimize DataNodeInstance creation (#368)
* chore: remove unnecessary changes
* chore: revert changes to src/api
* chore: revert changes to src/datanode/src/server.rs
* chore: remove opendal backend
* chore: optimize imports
* chore: revert changes to instance and region ids
* refactor: MetaKvBackend range
* fix: remove some wrap
* refactor: initiation of catalog
* feat: add region id to create table request and add heartbeat task to datanode instance
* fix: fix auto reconnect for heartbeat task
* chore: change TableValue::region_numbers to vec<u32>.
* fix: some tests
* fix: avoid concurrently start Heartbeat task by compare_exchange
* fix: some cr comments
* fix: fix unit tests
Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* refactor: Serialize Schema/TableMeta/TableInfo to raw structs
* test: Add tests for raw struct conversion
* style: Fix clippy
* refactor: SchemaBuilder::timestamp_index takes Option<usize>
So caller could chain the timestamp_index method call where there is no
timestamp index.
* style(datatypes): Chains SchemaBuilder method calls
* feat(storage): Implement skeleton of ReadResolver
ReadResolver is used to resolve difference between schemas
* feat(storage): Add user_column_end to ReadResover
* feat(storage): Implement Batch::batch_from_parts
Used to construct Batch from parts according to the schema that user
expects to read.
* feat(storage): Compat memtable schema
* feat(storage): Compat parquet file schema
* fix(storage): ReadResolver supports projection under same schema version
Now ReadResolver takes ProjectedSchemaRef as dest schema, and checks
whether a value column is needed by the schema after projection.
* feat(storage): Check whether columns are same columns
is_source_column_readable() takes ColumnMetadata instead of
ColumnSchema, and compares their column id to check whether they are
same columns.
* refactor(storage): Use row_key_end/user_column_end in source_schema
Rename ReadResolver::is_needed to ReadResolver::is_source_needed, and
remove row_key_end/user_column_end from ReadResolver, since they should
be same as source_schema's
* chore(storage): Remove unused codes
* test(storage): Add tests for the resolver
* feat(storage): Returns error on different source and dest column names
* style(storage): Fix clippy
* refactor: Rename ReadResolver to ReadAdapter
* chore(table): Removed unused comment
* refactor: rename to is_source_column_compatible
* chore: refactor dir for local catalog manager
* refactor: CatalogProvider returns Result
* refactor: SchemaProvider returns Result
* feat: add kv operations to remote catalog
* chore: refactor some code
* feat: impl catalog initialization
* feat: add register table and register system table function
* refactor: add table_info method for Table trait
* chore: add some tests
* chore: add register schema test
* chore: fix build issue after rebase onto develop
* refactor: mock to separate file
* build: failed to compile
* fix: use a container struct to bridge KvBackend and Accessor trait
* feat: upgrade opendal to 0.17
* test: add more tests
* chore: add catalog name and schema name to table info
* chore: add catalog name and schema name to table info
* chore: rebase onto develop
* refactor: common-catalog crate
* refactor: remove remote catalog related files
* fix: compilation
* feat: add table version to TableKey
* feat: add node id to TableValue
* fix: some CR comments
* chore: change async fn create_expr_to_request to sync
* fix: add backtrace to errors
* fix: code style
* fix: CatalogManager::table also requires both catalog_name and schema_name
* chore: merge develop
* refactor: return PhysicalPlan in Table trait's scan method, to support partitioned execution in Frontend's distribute read
* refactor: pub use necessary DataFusion types
* refactor: replace old "PhysicalPlan" and its adapters
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
* refactor: add table_info method for Table trait
* feat: add table_info method to Table trait
* test: add more unit test
* fix: impl table_info for SystemTable
* test: fix failing test
* wip add predicate definition
* fix value move
* implement predicate and prune
* impl filter push down in chunk reader
* add more expr tests
* chore: rebase develop
* fix: unit test
* fix: field name/index lookup when building pruning stats
* chore: add some meaningless test
* fix: remove unnecessary extern crate
* fix: use datatypes::schema::SchemaRef
* feat: impl scripts table and /execute restful api
* fix: test failures
* fix: test failures
* feat: impl /run_script API
* refactor: rename run_script api to run-script and test script manager
* fix: remove println
* refactor: error mod
* refactor: by CR comments
* feat: rebase develop and change timestamp/gmt_crated/gmt_modified type to timestamp
* refactor: use assert_eq instread of assert
* doc: fix comment in Script#execute function
* feat: implement alter table
* Currently we have no plans to support altering the primary keys (maybe never), so removed the related codes.
* make `alter` a trait function in table
* address other CR comments
* cleanup
* rebase develop
* resolve code review comments
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: upgrade rust to nightly-2022-07-14
* style: Fix some clippy warnings
* style: clippy fix
* style: fix clippy
* style: Fix clippy
Some PartialEq warnings have been work around using cfg_attr test
* feat: Implement Eq and PartialEq for PrimitiveType
* chore: Remove unnecessary allow
* chore: Remove usage of cfg_attr for PartialEq
* catalog manager allocates table id
* rebase develop
* add some tests
* add some more test
* fix some cr comments
* insert into system catalog
* use slice pattern to simplify code
* add optional dependencies
* add sql-to-request test
* successfully recover
* fix unit tests
* rebase develop
* add some tests
* fix some cr comments
* fix some cr comments
* add a lock to CatalogManager
* feat: add gmt_created and gmt_modified columns to system catalog table
* feat: impl TableManifest and refactor table engine, object store etc.
* feat: persist table metadata when creating it
* fix: remove unused file src/storage/src/manifest/impl.rs
* feat: impl recover table info from manifest
* test: add open table test and table manifest test
* fix: resolve CR problems
* fix: compile error and remove region id
* doc: describe parent_dir
* fix: address CR problems
* fix: typo
* Revert "fix: compile error and remove region id"
This reverts commit c14c250f8a.
* fix: compile error and generate region id by table_id and region number
Implement catalog manager that provides a vision of all existing tables while instance start. Current implementation is based on local table engine, all catalog info is stored in an system catalog table.
* feat: memtable backed by DataFusion to ease testing
* move test utility codes out of src folder
* Implement our own MemTable because DataFusion's MemTable does not support limit; and replace the original testing numbers table.
* fix: address PR comments
* fix: "testutil" -> "test-util"
* roll back "NumbersTable"
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: Add `open_table()` method to `TableEngine`
* feat: Implements MitoEngine::open_table()
For simplicity, this implementation just use the table name as region
name, and using that name to open a region for that table. It also
introduce a mutex to avoid opening the same table simultaneously.
* refactor: Shorten generic param name
Use `S` instead of `Store` for `MitoEngine`.
* test: Mock storage engine for table engine test
Add a `MockEngine` to mock the storage engine, so that testing the mito
table engine can sometimes use the mocked storage.
* test: Add open table test
Also remove `storage::gen_region_name` method, and always use table name
as default region name, so the table engine can open the table created
by `create_table()`.
* chore: Add open table log
* Impl TableEngine, bridge to storage
* Impl sql handler to process insert sql
* fix: minor changes and typo
* test: add datanode test
* test: add table-engine test
* fix: code style
* refactor: split out insert mod from sql and minor changes by CR
* refactor: replace with_context with context
* chore(store-api): Fix typo in region comments
* feat(storage): Init storage crate
* feat(store-api): Make some method async
* feat(storage): Blank StorageEngine implementation
* feat(storage): StorageEngine returns owned SchemaRef
* feat: pub use arrow in datatypes
* feat(store-api): Implement RegionMetadata
* feat(storage): Impl create region in memory.
* chore(object-store): Format cargo toml
* chore(storage): Log on region created
* feat: Impl CowCell
* feat: Store id to cf meta mapping
* refactor: Refactor version and rename it to VersionControl
* feat: Impl write batch for put, refactor column family
* feat(storage): Skeleton of writing to memtable
* refactor(storage): MemTable returns MemTableSchema
* feat: Add ColumnSchema and conversion between schema and arrow's schema
* feat: Validate put data
* feat: Valid schema of write batch
* feat: insert memtable WIP
* feat: Impl Inserter for memtable
* feat(datatypes): Implement Eq/Ord for Value
feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes
test: Test Value::from()
* feat: Define BTreeMemTable
* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.
* feat: Impl BTreeMemTable::write()
* refactor: Remove useless ColumnFamilyHandle now
* chore: Clean comment
* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value
* test(storage): Add tests for WriteBatch
* chore: Fix clippy
* feat: Add builder for RowKey/ColumnFamilyDescriptor
* test: Add test for metadata
* chore: Fix clippy
* test: Add test for region and engine
* chore: Fix clippy
* chore: Address CR comment
