mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-05-27 10:20:38 +00:00
c06e04afbbeecbc54fe42c075a3d4ae563efdf06
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yingwen
|
f8500e54c1 |
refactor: Remove PutOperation and Simplify WriteRequest API (#775)
* chore: Remove unused MutationExtra * refactor(storage): Refactor Mutation and Payload Change Mutation from enum to a struct that holds op type and record batches so the encoder don't need to convert the mutation into record batch. Now The Payload is no more an enum, it just holds the data, to be serialized to the WAL, of the WriteBatch. The encoder and decoder now deal with the Payload instead of the WriteBatch, so we could hold more information not necessary to be stored to the WAL in the WriteBatch. This commit also merge variants in write_batch::Error to storage::Error as some variants of them denote the same error. * test(storage): Pass all tests in storage * chore: Remove unused codes then format codes * test(storage): Fix test_put_unknown_column test * style(storage): Fix clippy * chore: Remove some unused codes * chore: Rebase upstream and fix clippy * chore(storage): Remove unused codes * chore(storage): Update comments * feat: Remove PayloadType from wal.proto * chore: Address CR comments * chore: Remove unused write_batch.proto |
||
Lei, HUANG
|
0653301754 |
feat: replace arrow2 with official implementation 🎉 (#753)
* chore: kick off. change datafusion/arrow/parquet to target version
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* chore: replace one last datafusion dep
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* feat: arrow_array switch to arrow
* chore: update dep of binary vector
* chore: fix wrong merge commit
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* feat: Switch to datatypes2
* feat: Make recordbatch compile
* chore: sort Cargo.toml
* feat: Fix common::recordbatch compiler errors
* feat: Fix recordbatch test compiling issue
* fix: api crate (#708)
* fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp
* fix: revert changes in datatypes2
* fix: helper
* chore: delete datatypes based on arrow2
* feat: Fix some compiler errors in common::query (#710)
* feat: Fix some compiler errors in common::query
* feat: test_collect use vectors api
* fix: common-query subcrate (#712)
* fix: record batch adapter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix error enum
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: Fix common::query compiler errors (#713)
* feat: Move conversion to ScalarValue to value.rs
* fix: Fix common::query compiler errors
This commit also make InnerError pub(crate)
* feat: Implements diff accumulator using WrapperType (#715)
* feat: Remove usage of opaque error from common::recordbatch
* feat: Remove opaque error from common::query
* feat: Fix diff compiler errors
Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.
* feat: Remove LargestType from NativeType trait
* chore: Update comments
* feat: Restrict Scalar::RefType of WrapperType to itself
Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType
* chore: Address CR comments
* chore: Format codes
* fix: fix compile error for mean/polyval/pow/interp ops
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* Revert "fix: fix compile error for mean/polyval/pow/interp ops"
This reverts commit
|
||
dennis zhuang
|
74ea529d1a |
feat: move time index metadata from schema into field (#444)
* feat: move time index metadata from schema into field * chore: remove useless code * test: test select with column alias * fix: conflicts with develop branch * test: add test * test: order by timestamp to ensure query results order * fix: comment |
||
|
Yingwen
|
a457c49d99 |
refactor: Remove column_null_mask in MutationExtra (#314)
* refactor: Remove column_null_mask in MutationExtra MutationExtra::column_null_mask is no longer needed as we could ensure there is no missing column in WriteBatch. * feat(storage): Remove MutationExtra Just stores MutationType in the WalHeader, no longer needs MutationExtra |
||
|
Lei, Huang
|
9366e77407 |
feat: impl timestamp type, value and vectors (#226)
* wip: impl timestamp data type * add timestamp vectors * adapt to recent changes to vector module * fix all unit test * rebase develop * fix slice * change default time unit to millisecond * add more tests * fix some CR comments * fix some CR comments * fix clippy * fix some cr comments * fix some CR comments * fix some CR comments * remove time unit in LogicalTypeId::Timestamp |
||
egg
|
8d51ad3429 |
feat: write_batch proto codec (#122)
* feat: protobuf codec * chore: minor fix * chore: beatify the macro code * chore: minor fix * chore: by cr * chore: by cr and impl wal with proto * bugfix: invalid num_rows for multi put_data in mutations Co-authored-by: jiachun <jiachun_fjc@163.com> |
||
|
evenyag
|
bf5975ca3e |
feat: Prototype of the storage engine (#107)
* feat: memtable flush (#63) * wip: memtable flush * optimize schema conversion * remove unnecessary import * add parquet file verfication * add backtrace to error * chore: upgrade opendal to 0.9 and fixed some problems * rename error * fix: error description Co-authored-by: Dennis Zhuang <killme2008@gmail.com> * feat: region manifest service (#57) * feat: adds Manifest API * feat: impl region manifest service * refactor: by CR comments * fix: storage error mod test * fix: tweak storage cargo * fix: tweak storage cargo * refactor: by CR comments * refactor: rename current_version * feat: add wal writer (#60) * feat: add Wal * upgrade engine for wal * fix: unit test for wal * feat: wal into region * fix: unix test * fix clippy * chore: by cr * chore: by cr * chore: prevent test data polution * chore: by cr * minor fix * chore: by cr * feat: Implement flush (#65) * feat: Flush framework - feat: Add id to memtable - refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet - feat: Freeze memtable - feat: Trigger flush - feat: Background job pool - feat: flush job - feat: Sst access layer - feat: Custom Deserialize for StringBytes - feat: Use RegionWriter to apply file metas - feat: Apply version edit - chore: Remove unused imports refactor: Use ParquetWriter to replace FlushTask refactor: FsAccessLayer takes object store as param chore: Remove todo from doc comments feat: Move wal to WriterContext chore: Fix clippy chore: Add backtrace to WriteWal error * feat: adds manifest to region and refactor sst/manifest dir config (#72) * feat: adds manifest to region and refactor sst/manifest dir with EngineConfig * refactor: ensure path ends with '/' in ManifestLogStorage * fix: style * refactor: normalize storage directory path and minor changes by CR * refactor: doesn't need slash any more * feat: Implement apply_edit() and add timestamp index to schema (#73) * feat: Implement VersionControl::apply_edit() * feat: Add timestamp index to schema * feat: Implement Schema::timestamp_column() * feat: persist region metadata to manifest (#74) * feat: persist metadata when creating region or sst files * fix: revert FileMeta comment * feat: resolve todo * fix: clippy warning * fix: revert files_to_remove type in RegionEdit * feat: impl SizeBasedStrategy for flush (#76) * feat: impl SizeBasedStrategy for flush * doc: get_mutable_limitation * fix: code style and comment * feat: align timestamp (#75) * feat: align timestamps in write batch * fix cr comments * fix timestamp overflow * simplify overflow check * fix cr comments * fix clippy issues * test: Fix region tests (comment out some unsupported tests) (#82) * feat: flush job (#80) * feat: flush job * fix cr comments * move file name instead of clone * comment log file test (#84) * feat: improve MemtableVersion (#78) * feat: improve MemtableVersion * feat: remove flushed immutable memtables and test MemtableVersion * refactor: by CR comments * refactor: clone kv in iterator * fix: clippy warning * refactor: Make BatchIterator supertrait of Iterator (#85) * refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83) * feat: Insert multiple memtables by time range (#77) * feat: memtable::Inserter supports insert multiple memtables by time range * chore: Update timestamp comment * test: Add tests for Inserter * test: Fix region tests (comment out some unsupported tests) * refactor: align_timestamp() use TimestampMillis::aligned_by_bucket() * chore: rename aligned_by_bucket to align_by_bucket * fix: Fix compile errors * fix: sst and manifest dir (#86) * Set RowKeyDescriptor::enable_version_column to false by default * feat: Implement write stall (#90) * feat: Implement write stall * chore: Update comments * feat: Support reading multiple memtables (#93) * feat: Support reading multiple memtables * test: uncomment tests rely on snapshot read * feat: wal format (#70) * feat: wal codec * chore: minor fix * chore: comment * chore: by cr * chore: write_batch_codec mod * chore: by cr * chore: upgrade proto * chore: by cr * fix failing test * fix failing test * feat: manifest to wal (#100) * feat: write manifest to wal * chore: sequence into wal * chore: by cr * chore: by cr * refactor: create log store (#104) Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: fariygirl <clickmetoday@163.com> Co-authored-by: Jiachun Feng <jiachun_feng@proton.me> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com> * chore: Fix clippy Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Jiachun Feng <jiachun_feng@proton.me> Co-authored-by: fariygirl <clickmetoday@163.com> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com> |