* fix: carry not recordbatch result in FlightData, to allow executing SQLs other than selection in new GRPC interface
* Update src/datanode/src/instance/flight/stream.rs
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com>
The log store use start sequence instead of file start id to filter
log stream. Add more tests about flush, including flush empty memtable
and reopen after flush
* feat: upgrade rust to nightly-2022-07-14
* style: Fix some clippy warnings
* style: clippy fix
* style: fix clippy
* style: Fix clippy
Some PartialEq warnings have been work around using cfg_attr test
* feat: Implement Eq and PartialEq for PrimitiveType
* chore: Remove unnecessary allow
* chore: Remove usage of cfg_attr for PartialEq
Use `error!(e; xxx)` pattern so we could get backtrace in error log.
Also use BoxedError as error source of ExecuteQuery instead of String,
so we could carry backtrace and other info in it.
* catalog manager allocates table id
* rebase develop
* add some tests
* add some more test
* fix some cr comments
* insert into system catalog
* use slice pattern to simplify code
* add optional dependencies
* add sql-to-request test
* successfully recover
* fix unit tests
* rebase develop
* add some tests
* fix some cr comments
* fix some cr comments
* add a lock to CatalogManager
* feat: add gmt_created and gmt_modified columns to system catalog table
* add pwrite
* write
* fix write
* error handling in write thread
* wrap some LogFile field to state field
* remove some unwraps
* reStructure some code
* implement file chunk
* composite chunk decode
* add test for chunk stream
* fix buffer test
* remove some useless code
* add test for read_at and file_chunk_stream
* use bounded channel to implement back pressure
* reimplement entry read and decoding
* add some doc
* clean some code
* use Sender::blocking_send to replace manually spawn
* support synchronous file chunk stream
* remove useless clone
* remove set_offset from Entry trait
* cr: fix some comments
* fix: add peek methods for Buffer
* add test for read at the middle of file
* fix some minor issues on comments
* rebase on to develop
* add peek_to_slice and read_to_slice
* initialize file chunk on heap
* fix some comments in CR
* respect entry id set outside LogStore
* fix unit test
* Update src/log-store/src/fs/file.rs
Co-authored-by: evenyag <realevenyag@gmail.com>
* fix some cr comments
Co-authored-by: evenyag <realevenyag@gmail.com>
* feat: Implements RegionWriter::replay()
Refactors `preprocess_write()`, wraps time ranges calculation and
memtable creation to `prepare_memtables()` so these logic can be reused
by `WriterInner::replay()`. Then implements `WriterInner::replay()`
which reads write batch from wal and inserts it into memtables.
* feat: Use sequence in request as committed sequence
Also checks that sequence should increase monotonically and returns
error if found sequence decreases
* chore: Remove OpenOptions param from RegionWriter::replay
* test: Add region reopen tests
refactor(storage): Rename read_write test mod to basic
refactor(storage): Move common region test logic to TesterBase
Let read/write Tester and flush Tester share the same TesterBase struct,
which implements common operations like put/full_scan.
* feat: Constructs RegionImpl in open()
Constructs RegionImpl after replay in `RegionImpl::open()`
* feat: Adds RegionImpl::create()
Adds `RegionImpl::create()` method to persist region metadata to
manifest, then create the RegionImpl instance, so the storage engine
just invoke `RegionImpl::create()` instead of `RegionImpl::new()` to
create the region instance, and don't need to update manifest after
creating region instance anymore. Now `RegionImpl::new()` need to takes
version instead of metadata as input.
This change is also a necessary part to pass the region open test, since
to open a region, need to persist something to manifest first.
* feat: Pass region open test
Use LocalFileLogStore for region test since NoopLogStore won't persist
data to the file system.
Create dir in `LocalFileLogStore::open` if it is not exist, so we don't
need to create the dir before using the logstore.
To pass the test, we always recover from flushed_sequence and use
`req_sequence + 1` as last sequence.
* test: Test reopen region multiple times
* chore: Address CR comments
Add more info to replay log and add an assert to check committed
sequence after reopen.
* refactor: Add cfg(test) to Version::new()
Remove `VersionControl::new()`, and add `#[cfg(test)]` to
`Version::new()` as it is only used by tests.
* refactor: Move fields in SharedData to EngineInner
Since `SharedData` isn't shared now, we move all its fields to
EngineInner, and remove the `SharedData` struct, also remove the
unused config field.
* feat: Store RegionSlot in engine's region map
A `RegionSlot` has three possible state:
- Opening
- Creating
- Ready (Holds the `RegionImpl`)
Also use the `RegionSlot` as a placeholder in the region map to indicate
the region is opening/creating, so another open/create request will
fail immediately. The `SlotGuard` is used to clean the slot if we failed
to create/open the region.
* feat: Add a blank method `RegionImpl::open`
* feat: Remove MetadataId from Manifest
Now metadata id of manifest is unused, also unnecessary as we have
manifest dir to build the manifest, but constructing the manifest
still needs a passing region id as argument, which is unavailable
during opening region. So we remove the metadata id from manifest so
`region_store_config()` don't need region id as input anymore
* feat: Remove region id from logstore::Namespace and Wal
This is necessary for implementing open, since we don't have region
id this time, but we need to build Wal and its logstore namespace. Now
this is ok as id is not actually used by logstore.
* feat: Setup `open_region` code skeleton
* feat: UDAF implementation backed by DataFusion.
Directly Transplant DataFusion's UDAF related structs, traits and functions, like `AggregateUDF`, `Accumulator` or `create_udaf` etc.
Implement median UDAF on top of it and used in unit testing.
Refs: #61
* feat: UDAF made generically
Refs: #61
* fix: cargo fmt
* fix: use prelude
* fix: uniform the name
* fix: move maybe commonly used functions together
* fix: make comments more clear
* fix: resolve conversations in CR
* fix: store input types in AccumulatorCreator, and use ScalarVector's iterator
* feat: introducing List value and List datatype
* refactor: use ArcSwap instead of Mutext
* refactor: shorten some namings
* refactor: move median UDAF out of tests
* refactor: rename
* feat: aggregate function registry
* fix: make `Value` satisfy ordering again
* fix: clippy warnings
* doc: add "how to write aggregate function"
* fix: address PR comments
* fix: trying to get rid of unwraps
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: add buffer abstraction and rewrite entry encode/decode process
* add some tests
* remove pad.rst
* fix some comments
* fix comments
* remove mmap mod
* feat: Bytes type implementation switch to bytes::Bytes
* fix: use Bytes::from(String) and Bytes::from(Vec<u8>)
* feat: add new method to Entry trait
* add log store impl
* add some test
* delete failing test
* fix: concurrent close issue
* feat: use arcswap to replace unsafe AtomicPtr
* fix: use lock to protect rolling procedure.
fix: use try_recv to replace poll_recv on appender task.
* chores: 1. use direct tmp dir instead of creating TempDir instance; 2. inline some short function; 3. rename some structs; 4. optimize namespace to arc wrapper inner struct.
