Yingwen
a7df5a7c9a
fix(mito): incorrect field index in ProjectionMapper ( #2388 )
...
* chore: update todo comments
* test: add test for projection
* fix: panics when projecting fields
* chore: remove todos
2023-09-14 04:15:15 +00:00
Yingwen
d4ae8a6fed
feat(mito): Add writable flag to region ( #2349 )
...
* feat: add writable flag to region.
* refactor: rename MitoEngine to MitoEngine::scanner
* feat: add set_writable() to RegionEngine
* feat: check whether region is writable
* feat: make set_writable sync
* test: test set_writable
* docs: update comments
* feat: send result on compaction failure
* refactor: wrap output sender in new type
* feat: on failure
* refactor: use get_region_or/writable_region_or
* refactor: remove send_result
* feat: notify waiters on flush scheduler drop
* test: fix tests
* fix: only alter writable region
2023-09-14 02:45:30 +00:00
Yingwen
da54a0c139
fix: alter table procedure forgets to update next column id ( #2385 )
...
* feat: add more info to error messages
* feat: store next column id in procedure
* fix: update next column id for table info
* test: fix add col test
* chore: remove location from invalid request error
* test: update test
* test: fix test
2023-09-14 02:06:57 +00:00
JeremyHi
80c5d52015
feat: stop region server ( #2356 )
...
* feat: stop region server
* fix: close region first
2023-09-12 07:57:15 -05:00
Yingwen
606ee43f1d
feat(mito): Implement skeleton for alteration ( #2343 )
...
* feat: impl handle_alter wip
* refactor: move send_result to worker.rs
* feat: skeleton for handle_alter_request
* feat: write requests should wait for alteration
* feat: define alter request
* chore: no warnings
* fix: remove memtables after flush
* chore: update comments and impl add_write_request_to_pending
* feat: add schema version to RegionMetadata
* feat: impl alter_schema/can_alter_directly
* chore: use send_result
* test: pull next_batch again
* feat: convert pb AlterRequest to RegionAlterRequest
* feat: validate alter request
* feat: validate request and alter metadata
* feat: allow none location
* test: test alter
* fix: recover files and flushed entry id from manifest
* test: test alter
* chore: change comments and variables
* chore: fix compiler errors
* feat: add is_empty() to MemtableVersion
* test: fix metadata alter test
* fix: Compaction picker doesn't notify waiters if it returns None
* chore: address CR comments
* test: add tests for alter request
* refactor: use send_result
2023-09-12 07:57:15 -05:00
Lei, HUANG
3331e3158c
feat(mito2): compaction ( #2317 )
...
* feat: compaction component
* feat: mito2 compaction
* Avoid building time range predicates when merge SST files since in TWCS we don't enforce strict time window.
* fix: some CR comments
* minor: change CompactionRequest::senders to an option
* chore: handle compaction finish error
* feat: integrate compaction into region worker
* chore: rebase upstream
* fix: Some CR comments
* chore: Apply suggestions from code review
* style: fix clippy
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-09-12 07:57:15 -05:00
LFC
711e27d9fa
feat: distributed alter table in region server ( #2311 )
...
* feat: distributed alter table in region server
* rebase
2023-09-12 07:57:15 -05:00
Ruihang Xia
46d171d341
chore: bump greptime-proto to replace region_dir ( #2290 )
...
chore: bump greptime-proto to replace region_dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
Ruihang Xia
6b8cf0bbf0
feat: impl region engine for mito ( #2269 )
...
* update proto
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* convert request
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update proto
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* import result convertor
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename symbols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-28 09:24:12 +00:00
Yingwen
e2522dff21
feat(mito): Skeleton for scanning a region ( #2230 )
...
* feat: define stream builder
* feat: scan region wip
* feat: create SeqScan in ScanRegion
* feat: scanner
* feat: engine handles scan request
* feat: map projection index to column id
* feat: Impl record batch stream
* refactor: change BatchConverter to ProjectionMapper
* feat: add column_ids to mapper
* feat: implement SeqScan::build()
* chore: fix typo
* docs: add mermaid for ScanRegion
* style: fix clippy
* test: fix record batch test
* fix: update sequence and entry id
* test: test query
* feat: address CR comment
* chore: address CR comments
* chore: Update src/mito2/src/read/scan_region.rs
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2023-08-28 06:59:31 +00:00
Yingwen
cd3755c615
feat(mito): Support handling RegionWriteRequest ( #2218 )
...
* feat: convert region request to worker write request
* chore: remove unused codes
* test: fix tests compiler errors
* chore: remove create/close/open request from worker requests
* chore: add comment
* chore: fix typo
2023-08-22 11:16:00 +00:00
Lei, HUANG
be1e13c713
feat(mito2): time series memtable ( #2208 )
...
* feat: time series memtable
* feat: add some test
* fix: some clippy warnings
* chore: some rustdoc
* refactor: test
* fix: remove useless functions
* feat: add config for TimeSeriesMemtable
* chore: some optimize
* refactor: remove bucketing
* refactor: avoid cloing RegionMetadataRef across all Series; make initial_builder_capacity a const; sort batch only by timestamp and sequence
2023-08-22 08:40:46 +00:00
Zhenchi
87a730658a
refactor: add ThinTable to proxy tables from infoschema ( #2193 )
...
* refactor: add thin table to proxy tables in info_schema
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix(catalog): fix typo in DataSourceAdapter struct name
* fix: remove redundant Send + Sync
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor(catalog): rename DataSourceAdapter to InformationTableDataSource
* feat(catalog): add ThinTableAdapter for adapting ThinTable to Table interface
* rebase develop
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: default impl for table_type of InformationTable
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: filter_pushdown as table field
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: remove explicit type declaration
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-08-17 15:19:14 +00:00
Ruihang Xia
4aaf6aa51b
feat: implement query API for RegionServer ( #2197 )
...
* some initial change
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl dummy structs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* decode and send logical plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement table scan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add some comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-17 11:02:31 +00:00
Yingwen
4ba12155fe
feat(mito): Implement SST format for mito2 ( #2178 )
...
* chore: update comment
* feat: stream writer takes arrow's types
* feat: Define Batch struct
* feat: arrow_schema_to_store
* refactor: rename
* feat: write parquet in new format with tsids
* feat: reader support projection
* feat: Impl read compat
* refactor: rename SchemaCompat to CompatRecordBatch
* feat: changing sst format
* feat: make it compile
* feat: remove tsid and some structs
* feat: from_sst_record_batch wip
* chore: push array
* chore: wip
* feat: decode batches from RecordBatch
* feat: reader converts record batches
* feat: remove compat mod
* chore: remove some codes
* feat: sort fields by column id
* test: test to_sst_arrow_schema
* feat: do not sort fields
* test: more test helpers
* feat: simplify projection
* fix: projection indices is incorrect
* refactor: define write/read format
* test: test write format
* test: test projection
* test: test convert record batch
* feat: remove unused errors
* refactor: wrap get_field_batch_columns
* chore: clippy
* chore: fix clippy
* feat: build arrow schema from region meta in ReadFormat
* feat: initialize the parquet reader at `build()`
* chore: fix typo
2023-08-17 06:25:50 +00:00
Ruihang Xia
4c3169431b
feat: move region metadata to store-api ( #2181 )
...
* add metadata & handle_read
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move metadata to store-api
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* dep aquamarine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove deadcode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove temporary code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/store-api/Cargo.toml
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* remove old mod
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2023-08-16 04:18:26 +00:00
Ruihang Xia
2168970814
feat: define region server and related requests ( #2160 )
...
* define region server and related requests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fill request body
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change mito2's request type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: bump greptime-proto to d9167cab (row insert/delete)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove name_to_index
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* address cr comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finilise
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-15 06:27:27 +00:00
Lei, HUANG
e924b44e83
refactor: KeyValues return ValueRef ( #2170 )
...
* refactor: KeyValues return ValueRef
* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.
* feat: define min value of OpType as a const
* fix: toml format
2023-08-14 14:51:13 +00:00
Ning Sun
c39de9072f
refactor: use workspace dependencies for internal modules ( #2119 )
...
* refactor: use workspace dependencies for internal modules
* fix: resolve issue with mock module in datanode
* refactor: update test modules
2023-08-08 11:02:34 +00:00
LFC
46fa3eb629
chore: upgrade rust toolchain to latest nightly ( #2049 )
...
* chore: upgrade rust toolchain to latest nightly
* rebase develop
* update rust toolchain in ci
2023-08-08 07:17:51 +00:00
Vanish
cf7e8c9142
feat: truncate region ( #2097 )
...
* feat: impl truncate region
* test: test truncate region
* chore: typo
* refactor: table truncate
* chore: remove useless changes
* chore: reset version
* fix: wait for flush task to complete
* fix: clippy
* chore: remove useless changes
* CR
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/engine.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/engine.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/region.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/region/tests/truncate.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/region/tests/truncate.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/region/writer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* CR
* Update src/storage/src/engine.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/storage/src/manifest/region.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-08-04 12:26:25 +00:00
Ruihang Xia
9718aa17c9
feat: define region group and sequence ( #2100 )
...
* define region group
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* define region sequence
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* check partition number
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* test region seq and group
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-04 09:08:07 +00:00
Yingwen
4d5ecb54c5
feat(mito): Implement open for RegionManifestManager ( #2036 )
...
* feat: file purger trait
* feat: Implement open for RegionManifestManager
* feat: remove RegionVersion
* feat: Use RwLock
* chore: remove AtomicManifestVersion
* feat: Remove unused error
* feat: store meta action
* chore: update comment
2023-07-31 10:04:22 +00:00
shuiyisong
9c1f0234de
refactor: query context ( #2022 )
...
* chore: refactor query_context
* chore: remove use statement
* chore: make query_context build return arc
* fix: sqlness test
* fix: cr issue
* fix: use unwrap or else
2023-07-25 06:11:34 +00:00
Ruihang Xia
8bea853954
refactor(mito2): implement RegionManifestManager ( #1984 )
...
* finilise manager and related API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl manifest initialize and update
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more test and utils
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-07-18 14:03:35 +00:00
Lei, HUANG
97cfa3d6c9
feat: support append entries from multiple regions at a time ( #1959 )
...
* feat: support append entries from multiple regions at a time
* chore: add some tests
* fix: false postive mutable_key warning
* fix: append_batch api
* fix: remove unused clippy allows
2023-07-14 09:57:17 +00:00
Ben Baoyi
9ff7670adf
refactor:remove common_error::prelude ( #1946 )
...
* feat:Remove common_error::prelude
* fix merge error
* cr comment
* fix error
2023-07-13 10:36:36 +08:00
Lei, HUANG
a7ea3bbc16
feat: manual compact api ( #1912 )
...
* merge develop
* chore: merge develop
* fix: some cr commentx
* fix: cr comments
2023-07-11 04:00:39 +00:00
Yingwen
f731193ddc
refactor: Define RegionId as a new type ( #1903 )
...
* refactor: Define RegionId as a new type
* chore: use into
* feat: custom debug print for region id
fix: test_show_create_table
2023-07-07 21:26:03 +08:00
Lei, HUANG
3b6f70cde3
feat: initial twcs impl ( #1851 )
...
* feat: initial twcs impl
* chore: rename SimplePicker to LeveledPicker
* rename some structs
* Remove Compaction strategy
* make compaction picker a trait object
* make compaction picker configurable for every region
* chore: add some test for ttl
* add some tests
* fix: some style issues in cr
* feat: enable twcs when creating tables
* feat: allow config time window when creating tables
* fix: some cr comments
2023-07-04 16:42:27 +08:00
Vanish
563f6e05e2
feat: remove all the manifests in drop_region. ( #1834 )
...
* feat: drop_region delete manifest file
* chore: remove redundant code
* chore: fmt
* chore: clippy
* chore: clippy
* feat: support delete_all in manifest.
* chore:CR
* test: test_drop_basic, test_drop_reopen
* chore: cr
* fix: typo
* chore: cr
2023-06-30 17:42:11 +08:00
LFC
fcff66e039
chore: deny unused results ( #1825 )
...
* chore: deny unused results
* rebase
2023-06-27 15:33:53 +08:00
Lei, HUANG
7961de25ad
feat: persist compaction time window ( #1757 )
...
* feat: persist compaction time window
* refactor: remove useless compaction window fields
* chore: revert some useless change
* fix: some CR comments
* fix: comment out unstable sqlness test
* revert commented sqlness
2023-06-13 10:15:42 +08:00
Ruihang Xia
2615718999
feat: merge scan for distributed execution ( #1660 )
...
* generate exec plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move DatanodeClients to client crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wip MergeScanExec::to_stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix default catalog
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix expand order of new stage
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* move sqlness cases contains plan out of common dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refactor information schema to allow duplicated scan call
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: ignore two cases due to substrait
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reorganise sqlness common cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typos
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* redact round robin partition number
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
* skip tranforming projection
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* revert common/order
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/query/src/dist_plan/merge_scan.rs
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore region failover IT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result again and again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* unignore some tests about projection
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* enable failover tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-06-02 06:42:54 +00:00
Ruihang Xia
e5b6f8654a
feat: optimizer rule to pass expected output ordering hint ( #1675 )
...
* move type convertsion rule into optimizer dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement order_hint rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* it works!
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use column name instead
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* accomplish test case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update lock file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-06-02 03:43:51 +00:00
Weny Xu
0460f3ae30
test: add write test for region failover ( #1673 )
...
* test: add write test for region failover
* test: add written data assertion after failover
* test: support more storage types
2023-05-31 15:42:00 +08:00
Ruihang Xia
b27c569ae0
refactor: add scan_to_stream() to Table trait to postpone the stream generation ( #1639 )
...
* add scan_to_stream to Table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl parquet stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reorganise adapters
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement scan_to_stream for mito table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add location info
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: table scan
* UT pass
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl project record batch
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix information schema
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve CR comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove one todo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix errors generated by merge commit
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add output_ordering method to record batch stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* enhance error types
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2023-05-29 20:03:47 +08:00
Weny Xu
9f67ad8bce
fix: fix doesn't release closed regions issue ( #1596 )
...
* fix: fix close region issue
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* chore: apply suggestion from CR
* refactor: remove close method from Region trait
* chore: remove PartialEq from CloseTableResult
2023-05-23 11:40:12 +08:00
Yingwen
6dbceb1ad5
feat: Trigger flush based on global write buffer size ( #1585 )
...
* feat(storage): Add AllocTracker
* feat(storage): flush request wip
* feat(storage): support global write buffer size
* fix(storage): Test and fix size based strategy
* test(storage): Test AllocTracker
* test(storage): Test pick_by_write_buffer_full
* docs: Add flush config example
* test(storage): Test schedule_engine_flush
* feat(storage): Add metrics for write buffer size
* chore(flush): Add log when triggering flush by global buffer
* chore(storage): track allocation in update_stats
2023-05-22 19:00:30 +08:00
Vanish
8764ce7845
feat: add delete WAL in drop_region ( #1577 )
...
* feat: add delete WAL in drop_region
* chore: fix typo err.
* feat: mark all SSTs deleted and remove the region from StorageEngine's region map.
* test: add test_drop_region for StorageEngine.
* chore: make clippy happy
* fix: fix conflict
* chore: CR.
* chore: CR
* chore: fix clippy
* fix: temp file life time
2023-05-18 18:02:34 +08:00
Yingwen
8d54d40b21
feat: Add FlushPicker to flush regions periodically ( #1559 )
...
* feat: Add FlushPicker
* feat(storage): Add close to StorageEngine
* style(storage): fix clippy
* feat(storage): Close regions in StorageEngine::close
* chore(storage): Clear requests on scheduler stop
* test(storage): Test flush picker
* feat(storage): Add metrics for auto flush
* feat(storage): Add flush reason and record it in metrics
* feat: Expose flush config
docs(config): Update config example
* refactor(storage): Run auto flush task in FlushScheduler
* refactor(storage): Add FlushItem trait to make FlushPicker easy to test
2023-05-15 17:29:28 +08:00
Lei, HUANG
cfcfc72681
refactor: remove version column ( #1576 )
2023-05-15 11:03:37 +08:00
discord9
2c82ded975
feat: table metrics ( #1469 )
...
* feat: Statistic
* add todo
* fmt: cargo fmt
* feat: some simple impl for MemTable
* chore: a try on adding statistics
* Update src/table/src/stats.rs
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* docs: fix typo
* newlines unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-06 14:59:49 +08:00
Ruihang Xia
d4f3f617e4
chore(toolchain): update rust-toolchain to 2023-05-03 ( #1524 )
...
* chore(toolchain): update rust-toolchain to 2023-05-03
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update workflow yaml
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-05-06 11:34:09 +08:00
Yingwen
479ef9d379
fix: checkpoint GC task also deletes the file with the last version ( #1491 )
...
* test(storage): use assert_eq to check scan result
* feat(storage): Add more info to manifest log
* feat: Avoid error log when unable to delete
* fix: The manifest gc task should delete files <= last_version
* feat(storage): Don't log if the error kind is not found
* feat: Add keep_last_checkpoint option
2023-05-04 14:18:38 +08:00
dennis zhuang
2a9f482bc7
feat: show create table ( #1336 )
...
* temp commit
* feat: impl Display for CreateTable statement
* feat: impl show create table for standalone
* fix: forgot show.rs
* feat: clean code
* fix: typo
* feat: impl show create table for distributed
* test: add show create table sqlness test
* fix: typo
* fix: sqlness tests
* feat: render partition rules for distributed table
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/sql/src/statements/create.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* fix: compile error
* fix: missing column comments and extra table options
* test: add show create table test
* test: add show create table test
* chore: timestamp precision
* fix: test
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-04-21 11:37:16 +08:00
Ruihang Xia
da66138e80
refactor(error): remove backtrace, and introduce call-site location for debugging ( #1329 )
...
* wip: global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unneeded tests of errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ErrorExt trait implementator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-04-06 04:06:00 +00:00
Eugene Tolbakov
59f7630000
feat: initial changes for compaction_time_window field support ( #1083 )
...
* feat(compaction_time_window): initial changes for compaction_time_window field support
* feat(compaction_time_window): move PickerContext creation
* feat(compaction_time_window): update region descriptor, fix formatting
* feat(compaction_time_window): add minor enhancements
* feat(compaction_time_window): fix failing test
* feat(compaction_time_window): return an error instead silently skip for the user provided compaction_time_window
* feat(compaction_time_window): add TODO reminder
2023-04-06 10:32:41 +08:00
dennis zhuang
563adbabe9
feat!: improve region manifest service ( #1268 )
...
* feat: try to use batch delete in ManifestLogStorage
* feat: clean temp dir when startup with file backend
* refactor: export region manifest checkpoint actions magin and refactor storage options
* feat: purge unused manifest and checkpoint files by repeat gc task
* chore: debug deleted logs
* feat: adds RepeatedTask and refactor all gc tasks
* chore: clean code
* feat: export gc_duration to manifest config
* test: assert gc works
* fix: typo
* Update src/common/runtime/src/error.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* fix: format
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* chore: by CR comments
* fix: serde default for StorageConfig
* chore: remove compaction config in StandaloneOptions
---------
Co-authored-by: LFC <bayinamine@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-03-31 10:42:00 +08:00
dennis zhuang
4f15b26b28
feat: region manifest checkpoint ( #1202 )
...
* chore: adds log when manifest protocol is changed
* chore: refactor region manifest
* temp commit
* feat: impl region manifest checkpoint
* feat: recover region version from manifest snapshot
* test: adds region snapshot test
* test: region manifest checkpoint
* test: alter region with manifest checkpoint
* fix: revert storage api
* feat: delete old snapshot
* refactor: manifest log storage
* Update src/storage/src/version.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/checkpoint.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/region.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/storage/src/manifest/region.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: by CR comments
* refactor: by CR comments
* fix: typo
* chore: tweak start_version
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-03-27 11:15:52 +08:00
