evenyag
03e965954a
feat: implement read framework ( #108 )
...
* feat: implement read framework
feat: chunk reader builder
refactor: rename BatchIteratorPtr to BoxedBatchIterator
feat: BatchReader to read batch from ssts
feat: Add a ConcatReader to concat sst readers
test: Add tests for concat reader
chore: Fix clippy
* feat: implement SST parquet reader (#109 )
* feat: implement parquet sst reader
* chores: fix some CR comments
* gst
* fix sst writer flush issue
* feat: Implement FsAccessLayer::read_sst
* fix: remove lifetime from ChunkStream
* refactor: Store file name in FileMeta
- Store file name instead of path (`region-name/file-name`) in FileMeta.
- `AccessLayer::read()` takes file name instead of path, so the read/write api are consistent
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-28 11:46:51 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
evenyag
4171173b76
feat: Support creating in memory region and writing to memtable ( #40 )
...
* chore(store-api): Fix typo in region comments
* feat(storage): Init storage crate
* feat(store-api): Make some method async
* feat(storage): Blank StorageEngine implementation
* feat(storage): StorageEngine returns owned SchemaRef
* feat: pub use arrow in datatypes
* feat(store-api): Implement RegionMetadata
* feat(storage): Impl create region in memory.
* chore(object-store): Format cargo toml
* chore(storage): Log on region created
* feat: Impl CowCell
* feat: Store id to cf meta mapping
* refactor: Refactor version and rename it to VersionControl
* feat: Impl write batch for put, refactor column family
* feat(storage): Skeleton of writing to memtable
* refactor(storage): MemTable returns MemTableSchema
* feat: Add ColumnSchema and conversion between schema and arrow's schema
* feat: Validate put data
* feat: Valid schema of write batch
* feat: insert memtable WIP
* feat: Impl Inserter for memtable
* feat(datatypes): Implement Eq/Ord for Value
feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes
test: Test Value::from()
* feat: Define BTreeMemTable
* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.
* feat: Impl BTreeMemTable::write()
* refactor: Remove useless ColumnFamilyHandle now
* chore: Clean comment
* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value
* test(storage): Add tests for WriteBatch
* chore: Fix clippy
* feat: Add builder for RowKey/ColumnFamilyDescriptor
* test: Add test for metadata
* chore: Fix clippy
* test: Add test for region and engine
* chore: Fix clippy
* chore: Address CR comment
2022-06-09 16:50:02 +08:00
dennis zhuang
a819b6bce5
feat: impl object-store based on opendal ( #27 )
...
* feat: impl object-store based on opendal
* feat: adds s3 test
* feat: export Layer
* feat: pass s3 access key/secret by env vars
* fix: logging when s3 test is running
2022-05-17 15:58:27 +08:00
evenyag
ced9a7c97f
chore: Create datanode, datatypes, log-store, object-store, query, table crates
2022-04-20 16:34:22 +08:00
