refactor: catalog crate (#415)

* chore: refactor dir for local catalog manager * refactor: CatalogProvider returns Result * refactor: SchemaProvider returns Result * feat: add kv operations to remote catalog * chore: refactor some code * feat: impl catalog initialization * feat: add register table and register system table function * refactor: add table_info method for Table trait * chore: add some tests * chore: add register schema test * chore: fix build issue after rebase onto develop * refactor: mock to separate file * build: failed to compile * fix: use a container struct to bridge KvBackend and Accessor trait * feat: upgrade opendal to 0.17 * test: add more tests * chore: add catalog name and schema name to table info * chore: add catalog name and schema name to table info * chore: rebase onto develop * refactor: common-catalog crate * chore: refactor dir for local catalog manager * refactor: CatalogProvider returns Result * refactor: SchemaProvider returns Result * feat: add kv operations to remote catalog * chore: refactor some code * feat: impl catalog initialization * feat: add register table and register system table function * refactor: add table_info method for Table trait * chore: add some tests * chore: add register schema test * chore: fix build issue after rebase onto develop * refactor: mock to separate file * build: failed to compile * fix: use a container struct to bridge KvBackend and Accessor trait * feat: upgrade opendal to 0.17 * test: add more tests * chore: add catalog name and schema name to table info * chore: add catalog name and schema name to table info * chore: rebase onto develop * refactor: common-catalog crate * refactor: remove remote catalog related files * fix: compilation * feat: add table version to TableKey * feat: add node id to TableValue * fix: some CR comments * chore: change async fn create_expr_to_request to sync * fix: add backtrace to errors * fix: code style * refactor: merge refactor/catalog-crate * feat: table key with version * feat: impl KvBackend for MetaClient * fix: integrate metaclient * fix: catalog use local table info as baseline * fix: sync metsrv * fix: wip * fix: update remote catalog on register and deregister * refactor: CatalogProvider * refactor: CatalogManager * fix: catalog key filtering * fix: pass some test * refactor: catalog iterating * fix: CatalogManager::table also requires both catalog_name and schema_name * chore: merge develop * chore: merge catalog crate * fix: adapt to recent meta-client api change * feat: databode lease * feat: remote catalog (#356) * chore: refactor dir for local catalog manager * refactor: CatalogProvider returns Result * refactor: SchemaProvider returns Result * feat: add kv operations to remote catalog * chore: refactor some code * feat: impl catalog initialization * feat: add register table and register system table function * refactor: add table_info method for Table trait * chore: add some tests * chore: add register schema test * chore: fix build issue after rebase onto develop * refactor: mock to separate file * build: failed to compile * fix: use a container struct to bridge KvBackend and Accessor trait * feat: upgrade opendal to 0.17 * test: add more tests * chore: add catalog name and schema name to table info * chore: add catalog name and schema name to table info * chore: rebase onto develop * refactor: common-catalog crate * chore: refactor dir for local catalog manager * refactor: CatalogProvider returns Result * refactor: SchemaProvider returns Result * feat: add kv operations to remote catalog * chore: refactor some code * feat: impl catalog initialization * feat: add register table and register system table function * refactor: add table_info method for Table trait * chore: add some tests * chore: add register schema test * chore: fix build issue after rebase onto develop * refactor: mock to separate file * build: failed to compile * fix: use a container struct to bridge KvBackend and Accessor trait * feat: upgrade opendal to 0.17 * test: add more tests * chore: add catalog name and schema name to table info * chore: add catalog name and schema name to table info * chore: rebase onto develop * refactor: common-catalog crate * refactor: remove remote catalog related files * fix: compilation * feat: add table version to TableKey * feat: add node id to TableValue * fix: some CR comments * chore: change async fn create_expr_to_request to sync * fix: add backtrace to errors * fix: code style * refactor: merge refactor/catalog-crate * feat: table key with version * feat: impl KvBackend for MetaClient * fix: integrate metaclient * fix: catalog use local table info as baseline * fix: sync metsrv * fix: wip * fix: update remote catalog on register and deregister * refactor: CatalogProvider * refactor: CatalogManager * fix: catalog key filtering * fix: pass some test * refactor: catalog iterating * fix: CatalogManager::table also requires both catalog_name and schema_name * chore: merge develop * chore: merge catalog crate * fix: adapt to recent meta-client api change * feat: datanode heartbeat (#355) * feat: add heartbeat task to instance * feat: add node_id datanode opts * fix: use real node id in heartbeat and meta client * feat: distribute table in frontend * test: distribute read demo * test: distribute read demo * test: distribute read demo * add write spliter * fix: node id changed to u64 * feat: datanode uses remote catalog implementation * dist insert integrate table * feat: specify region ids on creating table (#359) * fix: compiling issues * feat: datanode lease (#354) * Some glue code about dist_insert * fix: correctly wrap string value with quotes * feat: create route * feat: frontend catalog (#362) * feat: integrate catalog to frontend * feat: preserve partition rule on create * fix: print tables on start * chore: log in create route * test: distribute read demo * feat: support metasrv addr command line options * feat: optimize DataNodeInstance creation (#368) * chore: remove unnecessary changes * chore: revert changes to src/api * chore: revert changes to src/datanode/src/server.rs * chore: remove opendal backend * chore: optimize imports * chore: revert changes to instance and region ids * refactor: MetaKvBackend range * fix: remove some wrap * refactor: initiation of catalog * feat: add region id to create table request and add heartbeat task to datanode instance * fix: fix auto reconnect for heartbeat task * chore: change TableValue::region_numbers to vec<u32>. * fix: some tests * fix: avoid concurrently start Heartbeat task by compare_exchange * feat: refactor catalog key and values, separate table info into two kinds of keys * feat: bump table id from metasrv * fix: compare and set table id * chore: merge develop * fix: use integer serialization instead of string serialization Co-authored-by: jiachun <jiachun_fjc@163.com> Co-authored-by: luofucong <luofucong@greptime.com> Co-authored-by: fys <1113014250@qq.com> Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2026-05-31 04:10:38 +00:00 · 2022-11-08 20:40:16 +08:00
parent dfd4b10493
commit 2c9bcbe885
16 changed files with 424 additions and 197 deletions
--- a/src/datanode/src/instance/grpc.rs
+++ b/src/datanode/src/instance/grpc.rs
@@ -65,7 +65,11 @@ impl Instance {
        insert_batches: &[InsertBatch],
    ) -> Result<()> {
        // Create table automatically, build schema from data.
-        let table_id = self.catalog_manager.next_table_id();
+        let table_id = self
+            .catalog_manager
+            .next_table_id()
+            .await
+            .context(CatalogSnafu)?;
        let create_table_request = insert::build_create_table_request(
            catalog_name,
            schema_name,
--- a/src/datanode/src/instance/sql.rs
+++ b/src/datanode/src/instance/sql.rs
@@ -10,7 +10,7 @@ use servers::query_handler::SqlQueryHandler;
 use snafu::prelude::*;
 use sql::statements::statement::Statement;

-use crate::error::{ExecuteSqlSnafu, Result};
+use crate::error::{CatalogSnafu, ExecuteSqlSnafu, Result};
 use crate::instance::Instance;
 use crate::metric;
 use crate::sql::SqlRequest;
@@ -49,7 +49,11 @@ impl Instance {
            }

            Statement::Create(c) => {
-                let table_id = self.catalog_manager.next_table_id();
+                let table_id = self
+                    .catalog_manager
+                    .next_table_id()
+                    .await
+                    .context(CatalogSnafu)?;
                let _engine_name = c.engine.clone();
                // TODO(hl): Select table engine by engine_name

--- a/src/datanode/src/server/grpc/ddl.rs
+++ b/src/datanode/src/server/grpc/ddl.rs
@@ -11,14 +11,14 @@ use futures::TryFutureExt;
 use snafu::prelude::*;
 use table::requests::{AddColumnRequest, AlterKind, AlterTableRequest, CreateTableRequest};

-use crate::error::{self, ColumnDefaultConstraintSnafu, MissingFieldSnafu, Result};
+use crate::error::{self, CatalogSnafu, ColumnDefaultConstraintSnafu, MissingFieldSnafu, Result};
 use crate::instance::Instance;
 use crate::server::grpc::handler::AdminResultBuilder;
 use crate::sql::SqlRequest;

 impl Instance {
    pub(crate) async fn handle_create(&self, expr: CreateExpr) -> AdminResult {
-        let request = self.create_expr_to_request(expr);
+        let request = self.create_expr_to_request(expr).await;
        let result = futures::future::ready(request)
            .and_then(|request| self.sql_handler().execute(SqlRequest::Create(request)))
            .await;
@@ -63,7 +63,7 @@ impl Instance {
        }
    }

-    fn create_expr_to_request(&self, expr: CreateExpr) -> Result<CreateTableRequest> {
+    async fn create_expr_to_request(&self, expr: CreateExpr) -> Result<CreateTableRequest> {
        let schema = create_table_schema(&expr)?;

        let primary_key_indices = expr
@@ -76,8 +76,6 @@ impl Instance {
            })
            .collect::<Result<Vec<usize>>>()?;

-        let table_id = self.catalog_manager().next_table_id();
-
        let catalog_name = expr
            .catalog_name
            .unwrap_or_else(|| DEFAULT_CATALOG_NAME.to_string());
@@ -85,6 +83,12 @@ impl Instance {
            .schema_name
            .unwrap_or_else(|| DEFAULT_SCHEMA_NAME.to_string());

+        let table_id = self
+            .catalog_manager()
+            .next_table_id()
+            .await
+            .context(CatalogSnafu)?;
+
        let region_id = expr
            .table_options
            .get(&"region_id".to_string())
@@ -196,7 +200,7 @@ mod tests {
        instance.start().await.unwrap();

        let expr = testing_create_expr();
-        let request = instance.create_expr_to_request(expr).unwrap();
+        let request = instance.create_expr_to_request(expr).await.unwrap();
        assert_eq!(request.id, common_catalog::consts::MIN_USER_TABLE_ID);
        assert_eq!(request.catalog_name, "greptime".to_string());
        assert_eq!(request.schema_name, "public".to_string());
@@ -208,7 +212,7 @@ mod tests {

        let mut expr = testing_create_expr();
        expr.primary_keys = vec!["host".to_string(), "not-exist-column".to_string()];
-        let result = instance.create_expr_to_request(expr);
+        let result = instance.create_expr_to_request(expr).await;
        assert!(result.is_err());
        assert!(result
            .unwrap_err()