dennis zhuang
7c55783e53
feat!: reorganize the storage layout ( #1609 )
...
* feat: adds data_home to DataOptions
* refactor: split out object store stuffs from datanode instance
* feat: move data_home into FileConfig
* refactor: object storage layers
* feat: adds datanode path to procedure paths
* feat: temp commit
* refactor: clean code
* fix: forgot files
* fix: forgot files
* Update src/common/test-util/src/ports.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update tests/runner/src/env.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: compile error
* chore: cr comments
* fix: dependencies order in cargo
* fix: data path in test
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-05-23 13:58:26 +08:00
localhost
34c7f78861
chore: add configurator to http server ( #1488 )
...
* chore: add configurator params to start server fun
* chore: update plugins type
---------
Co-authored-by: paomian <qtang@greptime.com >
2023-05-08 10:55:03 +00:00
Vanish
12d59e6341
chore: remove redundant code. ( #1502 )
2023-05-04 14:20:26 +08:00
shuiyisong
69acf32914
chore: add len() to Bytes and StringBytes ( #1455 )
...
* chore: add `len()` to Bytes and StringBytes
* chore: add `len()` to Bytes and StringBytes
2023-04-25 10:18:41 +08:00
Ruihang Xia
da66138e80
refactor(error): remove backtrace, and introduce call-site location for debugging ( #1329 )
...
* wip: global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unneeded tests of errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ErrorExt trait implementator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-04-06 04:06:00 +00:00
Lei, HUANG
0253136333
feat: buffered parquet writer ( #1263 )
...
* wip: use
* rebase develop
* chore: fix typos
* feat: replace export parquet writer with buffered writer
* fix: some cr comments
* feat: add sst_write_buffer_size config item to config how many bytes to buffer before flush to underlying storage
* chore: reabse onto develop
2023-04-01 17:21:19 +08:00
LFC
11d45e2918
refactor: upgrade DataFusion, Arrow and Sqlparser ( #1074 )
...
* refactor: upgrade DataFusion, Arrow and Sqlparser
* fix: resolve PR comments
2023-02-27 22:20:08 +08:00
shuiyisong
9989a8c192
fix: check full table name during logical plan creation ( #948 )
2023-02-09 17:23:28 +08:00
Lei, HUANG
90fcaa8487
feat: expose wal config ( #852 )
...
* feat: wal config
* fix: use human-readable string in wal config
* feat: copy ReadableSize and humanize size config items in toml files
* fix: clippy
2023-01-10 16:07:26 +08:00
Lei, HUANG
8ffc078f88
fix: license header ( #815 )
2023-01-03 15:09:49 +08:00
Ruihang Xia
26a3e93ca7
chore: util workspace deps in more places ( #792 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-12-27 16:26:59 +08:00
LFC
dc52a51576
chore: upgrade to Arrow 29.0 and use workspace package and dependencies ( #782 )
...
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com >
2022-12-23 14:28:37 +08:00
Ruihang Xia
6fb413ae50
ci: add toml format linter ( #706 )
...
* chore: run taplo format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ci: add workflow to check toml
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rerun formatter with ident to 4 spaces
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update check command
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-12-05 20:03:10 +08:00
SSebo
68c2de8e45
feat: mysql and pg server support tls ( #641 )
...
* feat: mysql and pg server support tls
* chore: replace opensrv-mysql to original
* chore: TlsOption is required but supply default value
* feat: mysql server support force tls
* chore: move TlsOption to servers
* test: mysql server disable / prefer / required tls mode
* test: pg server disable / prefer / required tls mode
* chore: add doc and remove no used code
* chore: add TODO and restore cargo linker config
2022-11-30 12:46:15 +08:00
Yingwen
0791c65149
refactor: replace some usage of MutableBitmap by BitVec ( #610 )
2022-11-21 17:36:53 +08:00
Ruihang Xia
7ba512980a
chore: add APACHE-2.0 license header ( #518 )
...
* feat: add license checker workflow
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix existing header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* specify license for internal sub-crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 18:05:46 +08:00
fys
ba93aa83f2
chore: replace bitvec impl ( #214 )
...
* chore: replace bitvec impl
* chore: reduce one copy of nullmask
* chore: move bitvec to common_base
2022-08-31 14:13:36 +08:00
fys
34133fae5a
feat: impl select (grpc) ( #138 )
...
* SelectExpr: change to oneof expr
* Convert between Vec<u8> and SelectResult
* Chore: use encode_to_vec and decode, instead of encode_length_delimited_to_vec and decode_length_delimited
* Chore: move bitset into separate file
* Grpc select impl
2022-08-15 18:31:47 +08:00
Lei, Huang
d141fbc674
fix: log store write and read ( #97 )
...
* add pwrite
* write
* fix write
* error handling in write thread
* wrap some LogFile field to state field
* remove some unwraps
* reStructure some code
* implement file chunk
* composite chunk decode
* add test for chunk stream
* fix buffer test
* remove some useless code
* add test for read_at and file_chunk_stream
* use bounded channel to implement back pressure
* reimplement entry read and decoding
* add some doc
* clean some code
* use Sender::blocking_send to replace manually spawn
* support synchronous file chunk stream
* remove useless clone
* remove set_offset from Entry trait
* cr: fix some comments
* fix: add peek methods for Buffer
* add test for read at the middle of file
* fix some minor issues on comments
* rebase on to develop
* add peek_to_slice and read_to_slice
* initialize file chunk on heap
* fix some comments in CR
* respect entry id set outside LogStore
* fix unit test
* Update src/log-store/src/fs/file.rs
Co-authored-by: evenyag <realevenyag@gmail.com >
* fix some cr comments
Co-authored-by: evenyag <realevenyag@gmail.com >
2022-08-10 11:16:04 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
Lei, Huang
008f62afc1
feat: buffer abstraction ( #51 )
...
* feat: add buffer abstraction and rewrite entry encode/decode process
* add some tests
* remove pad.rst
* fix some comments
* fix comments
* remove mmap mod
* feat: Bytes type implementation switch to bytes::Bytes
* fix: use Bytes::from(String) and Bytes::from(Vec<u8>)
* feat: add new method to Entry trait
2022-07-04 14:08:23 +08:00
evenyag
8fe577649f
feat: Constructing Bytes/StringBytes from Vec<u8>/&[u8]/String/&str ( #41 )
2022-06-08 14:25:25 +08:00
dennis zhuang
f7136819fc
function crate and scalars function ( #39 )
...
* feat: adds scalars mod and enhance vectors
* temp commit
* fix compile error
* Impl pow function with new udf framework
* Adds common-function crate and impl scalars function
* fix: remove used code
* test: adds test for function crate and refactor vectors
* fix: fmt style
* feat: impl numpy.clip function
* feat: improve clip function returning int64 type when arguments do not have float type
* feat: adds more test for vectors
* feat: adds replicate method test for primitive vector
* fix: by code review
* feat: clip function returns uint64 when all arguments type are unsigned
* refactor: improve vectors#only_null
* fix: clippy warning
* fix: clippy warning
* fix: clip should return float64 when arguments have both signed and unsigned types
2022-06-08 13:15:22 +08:00
dennis zhuang
a2331366f6
feat: adds adds register_udf api to query engine and refactor datatypes ( #34 )
...
* feat: adds ColumnarValue and refactor vectors
* fix: ConcreteDataType compile error
* feat:adds udf/function mods
* feat: adds test for common_query crate
* feat: adds register_udf api to query engine
* feat: adds common_query::error test
* refactor: by CR comments
* refactor: adds impl_new_concret_type_functions! macro to reduce boilerplate codes
* fix: typo
2022-05-24 16:50:56 +08:00
evenyag
b505c87e8d
refactor: Rename base to common-base
2022-04-26 11:48:22 +08:00
evenyag
49081d818c
refactor: Divide common crate into small crates
2022-04-25 17:03:05 +08:00
