* fix: carry not recordbatch result in FlightData, to allow executing SQLs other than selection in new GRPC interface
* Update src/datanode/src/instance/flight/stream.rs
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* chore: Remove unused MutationExtra
* refactor(storage): Refactor Mutation and Payload
Change Mutation from enum to a struct that holds op type and record
batches so the encoder don't need to convert the mutation into record
batch. Now The Payload is no more an enum, it just holds the data, to
be serialized to the WAL, of the WriteBatch. The encoder and decoder
now deal with the Payload instead of the WriteBatch, so we could hold
more information not necessary to be stored to the WAL in the
WriteBatch.
This commit also merge variants in write_batch::Error to storage::Error
as some variants of them denote the same error.
* test(storage): Pass all tests in storage
* chore: Remove unused codes then format codes
* test(storage): Fix test_put_unknown_column test
* style(storage): Fix clippy
* chore: Remove some unused codes
* chore: Rebase upstream and fix clippy
* chore(storage): Remove unused codes
* chore(storage): Update comments
* feat: Remove PayloadType from wal.proto
* chore: Address CR comments
* chore: Remove unused write_batch.proto
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: move time index metadata from schema into field
* chore: remove useless code
* test: test select with column alias
* fix: conflicts with develop branch
* test: add test
* test: order by timestamp to ensure query results order
* fix: comment
* chore: Update StoreSchema comment
* feat: Add metadata to ColumnSchema
* feat: Impl conversion between ColumnMetadata and ColumnSchema
We could use this feature to store the ColumnMetadata as arrow's
Schema, since the ColumnSchema could be further converted to an arrow
schema. Then we could use ColumnMetadata in StoreSchema, which contains
more information, especially the column id.
* feat(storage): Merge schema::Error to metadata::Error
To avoid cyclic dependency of two Errors
* feat(storage): Store ColumnMetadata in StoreSchema
* feat(storage): Use StoreSchemaRef to avoid cloning the whole StoreSchema struct
* test(storage): Fix test_store_schema
* feat(datatypes): Return error on duplicate meta key
* chore: Address CR comments
* feat: Adds ColumnDefaultConstraint::create_default_vector
ColumnDefaultConstraint::create_default_vector is ported from
MitoTable::try_get_column_default_constraint_vector.
* refactor: Replace try_get_column_default_constraint_vector by create_default_vector
* style: Remove unnecessary map_err in MitoTable::insert
* feat: Adds compat_write
For column in `dest_schema` but not in `write_batch`, this method would insert a
vector with default value to the `write_batch`. If there are columns not in
`dest_schema`, an error would be returned.
* chore: Add info log to RegionInner::alter
* feat(storage): RegionImpl::write support request with old version
* feat: Add nullable check when creating default value
* feat: Validate nullable and default value
* chore: Modify PutOperation comments
* chore: Make ColumnDescriptor::is_nullable readonly and validate name
* feat: Use CompatWrite trait to replace campat::compat_write method
Adds a CompactWrite trait to support padding columns to WriteBatch:
- The WriteBatch and PutData implements this trait
- Fix the issue that WriteBatch::schema is not updated to the
schema after compat
- Also validate the created column when adding to PutData
The WriteBatch would also pad default value to missing columns in
PutData, so the memtable inserter don't need to manually check whether
the column is nullable and then insert a NullVector. All WriteBatch is
ensured to have all columns defined by the schema in its PutData.
* feat: Validate constraint by ColumnDefaultConstraint::validate()
The ColumnDefaultConstraint::validate() would also ensure the default
value has the same data type as the column's.
* feat: Use NullVector for null columns
* fix: Fix BinaryType returns wrong logical_type_id
* fix: Fix tests and revert NullVector for null columns
NullVector doesn't support custom logical type make it hard to
encode/decode, which also cause the arrow/protobuf codec of write batch
fail.
* fix: create_default_vector use replicate to create vector with default value
This would fix the test_codec_with_none_column_protobuf test, as we need
to downcast the vector to construct the protobuf values.
* test: add tests for column default constraints
* test: Add tests for CompatWrite trait impl
* test: Test write region with old schema
* fix(storage): Fix replay() applies metadata too early
The committed sequence of the RegionChange action is the sequence of the
last entry that use the old metadata (schema). During replay, we should
apply the new metadata after we see an entry that has sequence greater
than (not equals to) the `RegionChange::committed_sequence`
Also remove duplicate `set_committed_sequence()` call in
persist_manifest_version()
* chore: Removes some unreachable codes
Also add more comments to document codes in these files
* refactor: Refactor MitoTable::insert
Return error if we could not create a default vector for given column,
instead of ignoring the error
* chore: Fix incorrect comments
* chore: Fix typo in error message
* feat: adds commited_sequence to RegionChange action, #281
* refactor: saving protocol action when writer version is changed
* feat: recover all region medata in manifest and replay them when replaying WAL, #282
* refactor: minor change and test recovering metadata after altering table schema
* fix: write wrong min_reader_version into manifest for region
* refactor: move up DataRow
* refactor: by CR comments
* test: assert recovered metadata
* refactor: by CR comments
* fix: comment
* feat: Change signature of the Region::alter method
* refactor: Add builders for ColumnsMetadata and ColumnFamiliesMetadata
* feat: Support altering the region metadata
Altering the region metadata is done in a copy-write fashion:
1. Convert the `RegionMetadata` into `RegionDescriptor` which is more
convenient to mutate
2. Apply the `AlterOperation` to the `RegionDescriptor`. This would
mutate the descriptor in-place
3. Create a `RegionMetadataBuilder` from the descriptor, bump the
version and then build the new metadata
* feat: Implement altering table using the new Region::alter api
* refactor: Replaced wal name by region id
Region id is cheaper to clone than name
* chore: Remove pub(crate) of build_xxxx in engine mod
* style: fix clippy
* test: Add tests for AlterOperation and RegionMetadata::alter
* chore: ColumnsMetadataBuilder methods return &mut Self
* wip add predicate definition
* fix value move
* implement predicate and prune
* impl filter push down in chunk reader
* add more expr tests
* chore: rebase develop
* fix: unit test
* fix: field name/index lookup when building pruning stats
* chore: add some meaningless test
* fix: remove unnecessary extern crate
* fix: use datatypes::schema::SchemaRef
* feat: implement alter table
* Currently we have no plans to support altering the primary keys (maybe never), so removed the related codes.
* make `alter` a trait function in table
* address other CR comments
* cleanup
* rebase develop
* resolve code review comments
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: upgrade rust to nightly-2022-07-14
* style: Fix some clippy warnings
* style: clippy fix
* style: fix clippy
* style: Fix clippy
Some PartialEq warnings have been work around using cfg_attr test
* feat: Implement Eq and PartialEq for PrimitiveType
* chore: Remove unnecessary allow
* chore: Remove usage of cfg_attr for PartialEq
* feat: save create table schema and respect user defined columns order when querying, close#179
* fix: address CR problems
* refactor: use with_context with ProjectedColumnNotFoundSnafu
* feat: Add projected schema
* feat: Use projected schema to read sst
* feat: Use vector of column to implement Batch
* feat: Use projected schema to convert batch to chunk
* feat: Add no_projection() to build ProjectedSchema
* feat: Memtable supports projection
The btree memtable use `is_needed()` to filter unneeded value columns,
then use `ProjectedSchema::batch_from_parts()` to construct
batch, so it don't need to known the layout of internal columns.
* test: Add tests for ProjectedSchema
* test: Add tests for ProjectedSchema
Also returns error if the `projected_columns` used to build the
`ProjectedSchema` is empty.
* test: Add test for memtable projection
* feat: Table pass projection to storage engine
* fix: Use timestamp column name as schema metadata
This fix the issue that the metadata refer to the wrong timestamp column
if datafusion reorder the fields of the arrow schema.
* fix: Fix projected schema not passed to memtable
* feat: Add tests for region projection
* chore: fix clippy
* test: Add test for unordered projection
* chore: Move projected_schema to ReadOptions
Also fix some typo
* refactor: Merge RowKeyMetadata into ColumnsMetadata
Now RowKeyMetadata and ColumnsMetadata are almost always being used together, no need
to separate them into two structs. Now they are combined into the single
ColumnsMetadata struct.
chore: Make some fields of metadata private
feat: Replace schema in RegionMetadata by RegionSchema
The internal schema of a region should have the knownledge about all
internal columns that are reserved and used by the storage engine, such as
sequence, value type. So we introduce the `RegionSchema`, and it would
holds a `SchemaRef` that only contains the columns that user could see.
feat: Value derives Serialize and supports converting into json value
feat: Add version to schema
The schema version has an initial value 0 and would bump each time the
schema being altered.
feat: Adds internal columns to region metadata
Introduce the concept of reserved columns and internal columns.
Reserved columns are columns that their names, ids are reserved by the storage
engine, and could not be used by the user. Reserved columns usually have
special usage. Reserved columns expect the version columns are also
called internal columns (though the version could also be thought as a
special kind of internal column), are not visible to user, such as our
internal sequence, value_type columns.
The RegionMetadataBuilder always push internal columns used by the
engine to the columns in metadata. Internal columns are all stored
behind all user columns in the columns vector.
To avoid column id collision, the id reserved for columns has the most
significant bit set to 1. And the RegionMetadataBuilder would check the
uniqueness of the column id.
chore: Rebase develop and fix compile error
feat: add internal schema to region schema
feat: Add SchemaBuilder to build Schema
feat: Store row key end in region schema metadata
Also move the arrow schema construction to region::schema mod
feat: Add SstSchema
refactor: Replace MemtableSchema by RegionSchema
Now when writing sst files, we could use the arrow schema from our sst
schema, which contains the internal columns.
feat: Use SstSchema to read parquet
Adds user_column_end to metadata. When reading parquet file,
converts the arrow schema into SstSchema, then uses the row_key_end
and user_column_end to find out row key parts, value parts and internal
columns, instead of using the timestamp index, which may yields
incorrect index if we don't put the timestamp at the end of row key.
Move conversion from Batch to arrow Chunk to SstSchema, so SST mod doesn't
need to care the order of key, value and internal columns.
test: Add test for Value to serde_json::Value
feat: Add RawRegionMetadata to persist RegionMetadata
test: Add test to RegionSchema
fix: Fix clippy
To fix clippy::enum_clike_unportable_variant lint, define the column id
offset in ReservedColumnType and compute the final column id in
ReservedColumnId's const method
refactor: Move batch/chunk conversion to SstSchema
The parquet ChunkStream now holds the SstSchema and use its method to
convert Chunk into Batch.
chore: Address CR comment
Also add a test for pushing internal column to RegionMetadataBuilder
chore: Address CR comment
chore: Use bitwise or to compute column id
* chore: Address CR comment
* catalog manager allocates table id
* rebase develop
* add some tests
* add some more test
* fix some cr comments
* insert into system catalog
* use slice pattern to simplify code
* add optional dependencies
* add sql-to-request test
* successfully recover
* fix unit tests
* rebase develop
* add some tests
* fix some cr comments
* fix some cr comments
* add a lock to CatalogManager
* feat: add gmt_created and gmt_modified columns to system catalog table
* feat: impl TableManifest and refactor table engine, object store etc.
* feat: persist table metadata when creating it
* fix: remove unused file src/storage/src/manifest/impl.rs
* feat: impl recover table info from manifest
* test: add open table test and table manifest test
* fix: resolve CR problems
* fix: compile error and remove region id
* doc: describe parent_dir
* fix: address CR problems
* fix: typo
* Revert "fix: compile error and remove region id"
This reverts commit c14c250f8a.
* fix: compile error and generate region id by table_id and region number
* add pwrite
* write
* fix write
* error handling in write thread
* wrap some LogFile field to state field
* remove some unwraps
* reStructure some code
* implement file chunk
* composite chunk decode
* add test for chunk stream
* fix buffer test
* remove some useless code
* add test for read_at and file_chunk_stream
* use bounded channel to implement back pressure
* reimplement entry read and decoding
* add some doc
* clean some code
* use Sender::blocking_send to replace manually spawn
* support synchronous file chunk stream
* remove useless clone
* remove set_offset from Entry trait
* cr: fix some comments
* fix: add peek methods for Buffer
* add test for read at the middle of file
* fix some minor issues on comments
* rebase on to develop
* add peek_to_slice and read_to_slice
* initialize file chunk on heap
* fix some comments in CR
* respect entry id set outside LogStore
* fix unit test
* Update src/log-store/src/fs/file.rs
Co-authored-by: evenyag <realevenyag@gmail.com>
* fix some cr comments
Co-authored-by: evenyag <realevenyag@gmail.com>
* refactor(storage): Add region id and name to metadata
Add region id and name to `RegionMetadata`, simplify input arguments of
`RegionImpl::create()` and `RegionImpl::new()` method, since id and name
are already in metadata/version.
To avoid an atomic load of `Version` each time we access the region
id/name, we still store a copy of id/name in `SharedData`.
* chore: Remove todo in OpenOptions
Create region if missing when opening the region would be hard to
implement, since sometimes we may don't known the exact region schema user
would like to have.
* refactor: Make id and name of region readonly
By making `id` and `name` fields of `SharedData` and `RegionMetadata`
private and only exposing a pub getter.
* feat: Add `open_table()` method to `TableEngine`
* feat: Implements MitoEngine::open_table()
For simplicity, this implementation just use the table name as region
name, and using that name to open a region for that table. It also
introduce a mutex to avoid opening the same table simultaneously.
* refactor: Shorten generic param name
Use `S` instead of `Store` for `MitoEngine`.
* test: Mock storage engine for table engine test
Add a `MockEngine` to mock the storage engine, so that testing the mito
table engine can sometimes use the mocked storage.
* test: Add open table test
Also remove `storage::gen_region_name` method, and always use table name
as default region name, so the table engine can open the table created
by `create_table()`.
* chore: Add open table log
* feat: impl recovering version from manifest for region
* refactor: rename try_apply_edit to replay_edit
* fix: remove println
* fix: address CR problems
* feat: remove Metadata in manifest trait and update region manifest state after recovering
* refactor: Move fields in SharedData to EngineInner
Since `SharedData` isn't shared now, we move all its fields to
EngineInner, and remove the `SharedData` struct, also remove the
unused config field.
* feat: Store RegionSlot in engine's region map
A `RegionSlot` has three possible state:
- Opening
- Creating
- Ready (Holds the `RegionImpl`)
Also use the `RegionSlot` as a placeholder in the region map to indicate
the region is opening/creating, so another open/create request will
fail immediately. The `SlotGuard` is used to clean the slot if we failed
to create/open the region.
* feat: Add a blank method `RegionImpl::open`
* feat: Remove MetadataId from Manifest
Now metadata id of manifest is unused, also unnecessary as we have
manifest dir to build the manifest, but constructing the manifest
still needs a passing region id as argument, which is unavailable
during opening region. So we remove the metadata id from manifest so
`region_store_config()` don't need region id as input anymore
* feat: Remove region id from logstore::Namespace and Wal
This is necessary for implementing open, since we don't have region
id this time, but we need to build Wal and its logstore namespace. Now
this is ok as id is not actually used by logstore.
* feat: Setup `open_region` code skeleton
* fix: Fix write stall blocks flush applying version
refactor: Use store config to help constructing Region
chore: Address CR comments
* feat: adds manifest protocol supporting and refactor region metadata protocol
feat: ignore sqlparser log
refactor: PREV_VERSION_KEY constant
refactor: minor change for checking readable/writable
fix: address CR problems
refactor: use binary literal
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add buffer abstraction and rewrite entry encode/decode process
* add some tests
* remove pad.rst
* fix some comments
* fix comments
* remove mmap mod
* feat: Bytes type implementation switch to bytes::Bytes
* fix: use Bytes::from(String) and Bytes::from(Vec<u8>)
* feat: add new method to Entry trait
* feat: Maintain last sequence in VersionControl
* refactor(recordbatch): Replace `Arc<Schema>` by SchemaRef
* feat: Memtable support filter rows with invisible sequence
* feat: snapshot wip
* feat: Implement scan for SnapshotImpl
* test: Add a test that simply puts and scans a region
* chore: Fix clippy
* fix(memtable): Fix memtable returning duplicate keys
* test(memtable): Add sequence visibility test
* test: Add ValueType test
* chore: Address cr comments
* fix: Fix value is not storing but adding to committed sequence
* Impl TableEngine, bridge to storage
* Impl sql handler to process insert sql
* fix: minor changes and typo
* test: add datanode test
* test: add table-engine test
* fix: code style
* refactor: split out insert mod from sql and minor changes by CR
* refactor: replace with_context with context
* add log store impl
* add some test
* delete failing test
* fix: concurrent close issue
* feat: use arcswap to replace unsafe AtomicPtr
* fix: use lock to protect rolling procedure.
fix: use try_recv to replace poll_recv on appender task.
* chores: 1. use direct tmp dir instead of creating TempDir instance; 2. inline some short function; 3. rename some structs; 4. optimize namespace to arc wrapper inner struct.
