* feat: change table options from string map to a struct, add ttl and write_buffer_size
* fix: also pass table options to table meta
* feat: pass table options when opening/creating regions
* fix: CR comments
* feat: trigger compaction on flush
* chore: rebase develop
* feat: add config item max_file_in_level0 and remove compaction_after_flush
* fix: cr comments
* chore: add unit test to cover Timestamp::new_inclusive
* fix: workaround to fix future is not Sync
* fix: future is not sync
* fix: some cr comments
* refactor:replace another axum-test-helper branch
* refactor: upgrade opendal version
* refactor: use cursor for file buffer
* refactor:remove native-tls in mysql_async
* refactor: use async block and pipeline for newer opendal api
* chore: update Cargo.lock
* chore: update dependencies
* docs: removed openssl from build requirement
* fix: call close on pipe writer to flush reader for parquet streamer
* refactor: remove redundant return
* chore: use pinned revision for our forked mysql_async
* style: avoid wild-card import in test code
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com>
* style: use chained call for builder
Co-authored-by: liangxingjian <965662709@qq.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
* feat: impl TableManifest and refactor table engine, object store etc.
* feat: persist table metadata when creating it
* fix: remove unused file src/storage/src/manifest/impl.rs
* feat: impl recover table info from manifest
* test: add open table test and table manifest test
* fix: resolve CR problems
* fix: compile error and remove region id
* doc: describe parent_dir
* fix: address CR problems
* fix: typo
* Revert "fix: compile error and remove region id"
This reverts commit c14c250f8a.
* fix: compile error and generate region id by table_id and region number
* refactor(storage): Add region id and name to metadata
Add region id and name to `RegionMetadata`, simplify input arguments of
`RegionImpl::create()` and `RegionImpl::new()` method, since id and name
are already in metadata/version.
To avoid an atomic load of `Version` each time we access the region
id/name, we still store a copy of id/name in `SharedData`.
* chore: Remove todo in OpenOptions
Create region if missing when opening the region would be hard to
implement, since sometimes we may don't known the exact region schema user
would like to have.
* refactor: Make id and name of region readonly
By making `id` and `name` fields of `SharedData` and `RegionMetadata`
private and only exposing a pub getter.
* feat: Implements RegionWriter::replay()
Refactors `preprocess_write()`, wraps time ranges calculation and
memtable creation to `prepare_memtables()` so these logic can be reused
by `WriterInner::replay()`. Then implements `WriterInner::replay()`
which reads write batch from wal and inserts it into memtables.
* feat: Use sequence in request as committed sequence
Also checks that sequence should increase monotonically and returns
error if found sequence decreases
* chore: Remove OpenOptions param from RegionWriter::replay
* test: Add region reopen tests
refactor(storage): Rename read_write test mod to basic
refactor(storage): Move common region test logic to TesterBase
Let read/write Tester and flush Tester share the same TesterBase struct,
which implements common operations like put/full_scan.
* feat: Constructs RegionImpl in open()
Constructs RegionImpl after replay in `RegionImpl::open()`
* feat: Adds RegionImpl::create()
Adds `RegionImpl::create()` method to persist region metadata to
manifest, then create the RegionImpl instance, so the storage engine
just invoke `RegionImpl::create()` instead of `RegionImpl::new()` to
create the region instance, and don't need to update manifest after
creating region instance anymore. Now `RegionImpl::new()` need to takes
version instead of metadata as input.
This change is also a necessary part to pass the region open test, since
to open a region, need to persist something to manifest first.
* feat: Pass region open test
Use LocalFileLogStore for region test since NoopLogStore won't persist
data to the file system.
Create dir in `LocalFileLogStore::open` if it is not exist, so we don't
need to create the dir before using the logstore.
To pass the test, we always recover from flushed_sequence and use
`req_sequence + 1` as last sequence.
* test: Test reopen region multiple times
* chore: Address CR comments
Add more info to replay log and add an assert to check committed
sequence after reopen.
* refactor: Add cfg(test) to Version::new()
Remove `VersionControl::new()`, and add `#[cfg(test)]` to
`Version::new()` as it is only used by tests.
* refactor: Move fields in SharedData to EngineInner
Since `SharedData` isn't shared now, we move all its fields to
EngineInner, and remove the `SharedData` struct, also remove the
unused config field.
* feat: Store RegionSlot in engine's region map
A `RegionSlot` has three possible state:
- Opening
- Creating
- Ready (Holds the `RegionImpl`)
Also use the `RegionSlot` as a placeholder in the region map to indicate
the region is opening/creating, so another open/create request will
fail immediately. The `SlotGuard` is used to clean the slot if we failed
to create/open the region.
* feat: Add a blank method `RegionImpl::open`
* feat: Remove MetadataId from Manifest
Now metadata id of manifest is unused, also unnecessary as we have
manifest dir to build the manifest, but constructing the manifest
still needs a passing region id as argument, which is unavailable
during opening region. So we remove the metadata id from manifest so
`region_store_config()` don't need region id as input anymore
* feat: Remove region id from logstore::Namespace and Wal
This is necessary for implementing open, since we don't have region
id this time, but we need to build Wal and its logstore namespace. Now
this is ok as id is not actually used by logstore.
* feat: Setup `open_region` code skeleton
* fix: Fix write stall blocks flush applying version
refactor: Use store config to help constructing Region
chore: Address CR comments
* feat: adds manifest protocol supporting and refactor region metadata protocol
feat: ignore sqlparser log
refactor: PREV_VERSION_KEY constant
refactor: minor change for checking readable/writable
fix: address CR problems
refactor: use binary literal
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* chore(store-api): Fix typo in region comments
* feat(storage): Init storage crate
* feat(store-api): Make some method async
* feat(storage): Blank StorageEngine implementation
* feat(storage): StorageEngine returns owned SchemaRef
* feat: pub use arrow in datatypes
* feat(store-api): Implement RegionMetadata
* feat(storage): Impl create region in memory.
* chore(object-store): Format cargo toml
* chore(storage): Log on region created
* feat: Impl CowCell
* feat: Store id to cf meta mapping
* refactor: Refactor version and rename it to VersionControl
* feat: Impl write batch for put, refactor column family
* feat(storage): Skeleton of writing to memtable
* refactor(storage): MemTable returns MemTableSchema
* feat: Add ColumnSchema and conversion between schema and arrow's schema
* feat: Validate put data
* feat: Valid schema of write batch
* feat: insert memtable WIP
* feat: Impl Inserter for memtable
* feat(datatypes): Implement Eq/Ord for Value
feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes
test: Test Value::from()
* feat: Define BTreeMemTable
* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.
* feat: Impl BTreeMemTable::write()
* refactor: Remove useless ColumnFamilyHandle now
* chore: Clean comment
* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value
* test(storage): Add tests for WriteBatch
* chore: Fix clippy
* feat: Add builder for RowKey/ColumnFamilyDescriptor
* test: Add test for metadata
* chore: Fix clippy
* test: Add test for region and engine
* chore: Fix clippy
* chore: Address CR comment
