* feat: add take index method for VectorOp
* chore: make clippy happy
* chore: make clippy happy
* chore: improve the code
* chore: improve the code
* chore: add take null test
* chore: fix clippy
* 1. Reimplement Eq for Timestamp
2. Add and/or for GenericRange
* feat: extract time range from filters
* feat: select sst files according to time range
* fix: clippy
* fix: empty value in range
* fix: some cr comments
* fix: return optional timestamp range
* fix: cr comments
If the table has a non-null column, we need to use default value instead
of null to fill the value columns in the record batch for deletion.
Otherwise, we can't create the record batch since the schema check
doesn't allow null in the non-null column.
* docs: Fix incorrect comment of Vector::only_null
* feat: Add delete to WriteRequest and WriteBatch
* feat: Filter deleted rows
* fix: Fix panic after reopening engine
This is detected by adding a reopen step to the delete test for region.
* fix: Fix OpType::min_type()
* test: Add delete absent key test
* chore: Address CR comments
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: use Substrait logical plan to query data from Datanode in Frontend in distributed mode
* fix: resolve PR comments
* fix: resolve PR comments
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
* fix: dedup should not mark element as unneeded
It should only mark element as selected, because some column of
different rows may have same value.
* refactor: Rename dedup to find_unique
As the original `dedup` method only mark bitmap to true when it finds
the element is unique, so `find_unique` is more appropriate for its
name.
* test: Renew bitmap in test_batch_find_unique
* chore: Update comments
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend
* wip: start frontend in datanode
* wip: migrate create database to frontend
* wip: impl alter table
* fix: CR comments
* feat: move time index metadata from schema into field
* chore: remove useless code
* test: test select with column alias
* fix: conflicts with develop branch
* test: add test
* test: order by timestamp to ensure query results order
* fix: comment
* feat: supports list array in arrow_array_get
* feat: supports string and list type conversions in python coprocessor
* test: add test cases for returning list in coprocessor
* fix: Fix int64 type not considered in DEFAULT CURRENT_TIMESTAMP() constraint
Also avoid using `ConstantVector` in default constraint, as other user
may try to downcast it to a concrete type, and sometimes may forget to
check whether it is a constant vector.
* test: Add test for writing default value
* refactor: Serialize Schema/TableMeta/TableInfo to raw structs
* test: Add tests for raw struct conversion
* style: Fix clippy
* refactor: SchemaBuilder::timestamp_index takes Option<usize>
So caller could chain the timestamp_index method call where there is no
timestamp index.
* style(datatypes): Chains SchemaBuilder method calls
* feat: align_bucket support i64 and timestamp values
* feat: add Int64 to timestamp
* feat: support query i64 timestamp vector
* test: fix failling tests
* refactor: simplify some code
* fix: CR comments and add insert and query test for i64 timestamp column
* chore: Update StoreSchema comment
* feat: Add metadata to ColumnSchema
* feat: Impl conversion between ColumnMetadata and ColumnSchema
We could use this feature to store the ColumnMetadata as arrow's
Schema, since the ColumnSchema could be further converted to an arrow
schema. Then we could use ColumnMetadata in StoreSchema, which contains
more information, especially the column id.
* feat(storage): Merge schema::Error to metadata::Error
To avoid cyclic dependency of two Errors
* feat(storage): Store ColumnMetadata in StoreSchema
* feat(storage): Use StoreSchemaRef to avoid cloning the whole StoreSchema struct
* test(storage): Fix test_store_schema
* feat(datatypes): Return error on duplicate meta key
* chore: Address CR comments
* feat: Adds ColumnDefaultConstraint::create_default_vector
ColumnDefaultConstraint::create_default_vector is ported from
MitoTable::try_get_column_default_constraint_vector.
* refactor: Replace try_get_column_default_constraint_vector by create_default_vector
* style: Remove unnecessary map_err in MitoTable::insert
* feat: Adds compat_write
For column in `dest_schema` but not in `write_batch`, this method would insert a
vector with default value to the `write_batch`. If there are columns not in
`dest_schema`, an error would be returned.
* chore: Add info log to RegionInner::alter
* feat(storage): RegionImpl::write support request with old version
* feat: Add nullable check when creating default value
* feat: Validate nullable and default value
* chore: Modify PutOperation comments
* chore: Make ColumnDescriptor::is_nullable readonly and validate name
* feat: Use CompatWrite trait to replace campat::compat_write method
Adds a CompactWrite trait to support padding columns to WriteBatch:
- The WriteBatch and PutData implements this trait
- Fix the issue that WriteBatch::schema is not updated to the
schema after compat
- Also validate the created column when adding to PutData
The WriteBatch would also pad default value to missing columns in
PutData, so the memtable inserter don't need to manually check whether
the column is nullable and then insert a NullVector. All WriteBatch is
ensured to have all columns defined by the schema in its PutData.
* feat: Validate constraint by ColumnDefaultConstraint::validate()
The ColumnDefaultConstraint::validate() would also ensure the default
value has the same data type as the column's.
* feat: Use NullVector for null columns
* fix: Fix BinaryType returns wrong logical_type_id
* fix: Fix tests and revert NullVector for null columns
NullVector doesn't support custom logical type make it hard to
encode/decode, which also cause the arrow/protobuf codec of write batch
fail.
* fix: create_default_vector use replicate to create vector with default value
This would fix the test_codec_with_none_column_protobuf test, as we need
to downcast the vector to construct the protobuf values.
* test: add tests for column default constraints
* test: Add tests for CompatWrite trait impl
* test: Test write region with old schema
* fix(storage): Fix replay() applies metadata too early
The committed sequence of the RegionChange action is the sequence of the
last entry that use the old metadata (schema). During replay, we should
apply the new metadata after we see an entry that has sequence greater
than (not equals to) the `RegionChange::committed_sequence`
Also remove duplicate `set_committed_sequence()` call in
persist_manifest_version()
* chore: Removes some unreachable codes
Also add more comments to document codes in these files
* refactor: Refactor MitoTable::insert
Return error if we could not create a default vector for given column,
instead of ignoring the error
* chore: Fix incorrect comments
* chore: Fix typo in error message
* feat: add type conversion optimizer
* feat: add expr rewrite logical plan optimizer
* chore: add some doc
* fix: unit test
* fix: time zone issue in unit tests
* chore: add more tests
* fix: some CR comments
* chore: rebase develop
* chore: fix unit tests
* fix: unit test use timestamp with time zone
* chore: add more tests
* feat: Handle empty NullVector in replicate_null
* chore: Rename ChunkReaderImpl::sst_reader to batch_reader
* feat: dedup reader wip
* feat: Add BatchOp
Add BatchOp to support dedup/filter Batch and implement BatchOp for
ProjectedSchema.
Moves compare_row_of_batch to BatchOp::compare_row.
* feat: Allow Batch has empty columns
* feat: Implement DedupReader
Also add From<MutableBitmap> for BooleanVector
* test: Test dedup reader
Fix issue that compare_row compare by full key not row key
* chore: Add comments to BatchOp
* feat: Dedup results from merge reader
* test: Test merge read after flush
* test: Test merge read after flush and reopen
* test: Test replicate empty NullVector
* test: Add tests for `ProjectedSchema::dedup/filter`
* feat: Filter empty batch in DedepReader
Also fix clippy warnings and refactor some codes
