Weny Xu
7ad248d6f6
fix(config): enable file engine by default ( #4345 )
...
* fix: enable file engine by default
* fix: fix tests
2024-07-11 17:35:18 +00:00
Zhenchi
a710676d06
feat(fulltext_index): integrate full-text indexer with sst writer ( #4302 )
...
* feat(fulltext_index): integrate full-text indexer with sst writer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: delay building puffin writer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: indexer test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add abort on empty indexer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: indicates default mode
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: introduce "auto" and "unlimited" as mem threshold
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* doc: comment about push empty string
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-07 04:10:19 +00:00
Zhenchi
65c9fbbd2f
feat(fulltext_index): integrate puffin manager with inverted index applier ( #4266 )
...
* feat(fulltext_index): integrate puffin manager with inverted index applier
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: get rid of unexpected not found from write cache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: move create_dir_all to BoundedStager::new
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: update config.md
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: unify directories
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: silent remove
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: config docs
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: auxiliary -> aux
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-04 06:18:58 +00:00
Jeremyhi
df0fff2f2c
feat(servers): make http timeout and body limit optional ( #4217 )
...
* feat(servers): make http timeout and body limit optional
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: make config-docs
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-26 06:14:14 +00:00
Weny Xu
ea2d067cf1
feat: implement the OrderedBatchProducer ( #4134 )
...
* feat: implement the `OrderedBatchProducer`
* test: add test of cancel safety
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: simplify the `BackgroundProducerWorker`
* feat: implement the OrderedBatchProducer v2
* refactor: switch to `OrderedBatchProducer`
* chore: rename to `MAX_FLUSH_QUEUE_SIZE`
* refactor: switch to `OrderedBatchProducerV2`
* refactor: remove `OrderedBatchProducerV1`
* test: add tests
* refactor: make config configurable
* refactor: minor refactor
* chore: remove unused code
* chore: remove `benchmarks` crate
* chore: update config doc
* chore: remove unused comment
* refactor: refactor client registry
* refactor: rename `max_batch_size` to `max_batch_bytes`
* chore: use constant value
* chore: ensure serialized meta < ESTIMATED_META_SIZE
* chore: apply suggestions from CR
* chore: remove the `CHANNEL_SIZE`
* chore: apply suggestions from CR
* fix: ensure serialized meta < ESTIMATED_META_SIZE
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-06-18 07:20:01 +00:00
Ning Sun
70d113a355
feat: update default size of bgworkers, add hbworkers ( #4129 )
...
* feat: update default size of bgworkers, add hbworkers
* feat: update frontend heartbeat as well
* chore: update sample config files and default settings
* chore: update config docs
* Revert "chore: update config docs"
This reverts commit 8107f4c120f5ae701c8d
2024-06-18 06:18:37 +00:00
LFC
c0aed1d267
feat: set global runtime size by config file ( #4063 )
...
* set global runtime size
* fix: resolve PR comments
* fix: log the whole option
* fix ci
* debug ci
* debug ci
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-04 10:03:33 +00:00
taobo
a3a2c8d063
feat: Add TLS support for gRPC service ( #3957 )
...
* feat: Add tls support for grpc service
* feat: add integration test
* fix: integration test
* fix: revert by suggestion
* fix: typos
* fix: optimize code
* fix: optimize code
* docs: update configs
2024-05-23 19:00:16 +00:00
shuiyisong
9e1af79637
chore: add ttl to write_cache ( #4010 )
...
* chore: add ttl to write_cache
* chore: update test & add example config
* chore: fix typo
* chore: fix typo
* chore: fix typo
2024-05-22 06:50:12 +00:00
zyy17
60eb5de3f1
refactor: add tracing options in xOptions ( #3919 )
...
* refactor: add tracing options in {DatanodeOptions, FrontendOptions, MetasrvOptions, StandaloneOptions}
* ci: fix integration error
* refactor: minor modification for initialization of tracing options
2024-05-13 05:16:50 +00:00
shuiyisong
6e9e8fad26
refactor!: remove opentsdb tcp server ( #3828 )
...
* refactor: remove opentsdb tcp server
* refactor: remove config and add test
* refactor: update docs and remove unused code
2024-05-06 06:42:05 +00:00
Yingwen
d2e081c1f9
docs: update memtable config example ( #3712 )
2024-04-16 07:26:20 +00:00
zyy17
8060c81e1d
refactor: use toml2docs to generate config docs ( #3704 )
...
* refactor: use toml2docs to generate config docs
* ci: add docs check in 'check-typos-and-docs'
2024-04-15 09:08:32 +00:00
Lei, HUANG
183fccbbd6
chore: remove global_ttl config ( #3673 )
...
* chore: remove global_ttl config
* fix: clippy
2024-04-09 02:00:50 +00:00
Yingwen
39b69f1e3b
refactor!: Renames the new memtable to PartitionTreeMemtable ( #3547 )
...
* refactor: rename mod merge_tree to partition_tree
* refactor: rename merge_tree
* refactor: change merge tree comment
* refactor: rename merge tree struct
* refactor: memtable options
2024-03-20 06:40:41 +00:00
Lei, HUANG
7183fa198c
refactor: make MergeTreeMemtable the default choice ( #3430 )
...
* refactor: make MergeTreeMemtable the default choice
* refactor: reformat
* chore: add doc to config
2024-03-05 10:00:08 +00:00
WU Jingdi
53f2a5846c
feat: support tracing rule sampler ( #3405 )
...
* feat: support tracing rule sampler
* chore: simplify code
2024-03-05 15:40:02 +08:00
Ning Sun
d4a54a085b
feat: add configuration for tls watch option ( #3395 )
...
* feat: add configuration for tls watch option
* test: sleep longer to ensure async task run
* test: update config api integration test
* refactor: rename function
2024-03-01 03:49:54 +00:00
Ruihang Xia
31787f4bfd
feat!: switch prom remote write to metric engine ( #3198 )
...
* feat: switch prom remote write to metric engine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* read physical table name from url
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove physical table from header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix merge error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add with_metric_engine option to config remote write behavior
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* check parameter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add specific config param
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* default with_metric_engine to true
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update UT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 14:48:28 +00:00
shuiyisong
2bf4b08a6b
chore: change default factor to compute memory size ( #3211 )
...
* chore: change default factor to compute memory size
* chore: update doc
* chore: update comment in example config
* chore: extract factor to const and update comments
* chore: update comment by cr suggestion
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 09:03:29 +00:00
niebayes
63205907fb
refactor: introduce common-wal to aggregate wal stuff ( #3171 )
...
* refactor: aggregate wal configs
* refactor: move wal options to common-wal
* chore: slim Cargo.toml
* fix: add missing crates
* fix: format
* chore: update comments
* chore: add testing feature gate for test_util
* fix: apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix: apply suggestions from code review
* fix: compiling
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-18 03:49:37 +00:00
Zhenchi
6f07d69155
feat(mito): enable inverted index ( #3158 )
...
* feat(mito): enable inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* accidentally resolved the incorrect filtering issue within the Metric Engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
* Update src/mito2/src/test_util/scheduler_util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: format -> join_dir
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move intermediate_manager from arg of write_and_upload_sst to field of WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: add IndexerBuidler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 09:08:07 +00:00
niebayes
1b0e39a7f2
chore: stop exposing num_partitions ( #3132 )
2024-01-10 10:45:18 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
niebayes
78303639db
feat(remote_wal): split an entry if it's too large ( #3092 )
...
* feat: split an entry if it's too large
* chore: rewrite check records
* test: add some unit tests for record
* chore: rewrite entry splitting
* chore: add unit tests for build records
* chore: add more unit tests for record
* chore: rewrite encdec of record
* revert: ignored test
* fix: set limit for max_batch_size
* fix: clippy
* chore: remove heavy logging
* fix: CR
* fix: properly terminate
* fix: CR
* fix: compiling
* fix: sqlness
* fix: CR
* fix: license
* fix: license
2024-01-05 12:41:43 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
Weny Xu
5653389063
feat!: correct the kafka config option ( #3065 )
...
* feat: correct the kafka config option
* refactor: rewrite the verbose comments
2024-01-02 07:31:37 +00:00
WU Jingdi
d1ee1ba56a
feat: support set timezone in db ( #2992 )
...
* feat: support set timezone in db
* chore: fix ci
* chore: fix code advice
* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
niebayes
d061bf3d07
feat(remote_wal): introduce kafka remote wal ( #3001 )
...
* feat: integrate remote wal to standalone
* fix: test
* chore: ready to debug kafka remote wal
* fix: test
* chore: add some logs for remote wal
* chore: add logs for topic manager
* fix: properly terminate stream consumer
* fix: properly handle TopicAlreadyExists error
* fix: parse config file error
* fix: properly handle last entry id
* chore: prepare for merge
* fix: test
* fix: typo
* fix: set replication_factor properly
* fix: CR
* test: tmp for test
* Revert "test: tmp for test"
This reverts commit 093a3e0038
2023-12-26 12:35:24 +00:00
LFC
196c06db14
feat: make logging to stdout configurable ( #3003 )
...
* feat: make logging to stdout configurable
* fix sqlness
* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
niebayes
bab198ae68
feat(remote_wal): impl kafka log store ( #2971 )
...
* feat: introduce client manager
* chore: add errors for client manager
* chore: add record utils
* chore: impl kafka log store
* chore: build kafka log store upon starting datanode
* chore: update comments for kafka log store
* chore: add a todo for getting entry offset
* fix: typo
* chore: remove unused
* chore: update comments
* fix: typo
* fix: resolve some review conversations
* chore: move commonly referenced crates to workspace Cargo.toml
* fix: style
* fix: style
* chore: unify topic name prefix
* chore: make backoff config configurable by users
* chore: properly use backoff config in wal config
* refactor: read/write of kafka log store
* fix: typo
* fix: typo
* fix: resolve review conversations
2023-12-25 09:21:52 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
09aa4b72a5
chore: update storage config example ( #2887 )
...
chore: update config example
2023-12-07 03:18:56 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
WU Jingdi
0badb3715e
feat: support sample ratio in trace ( #2809 )
...
* feat: support sample ratio in trace
* chore: fix code advice
2023-11-27 06:46:46 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f
feat(mito): implements row group level page cache ( #2688 )
...
* feat: add page cache
* docs: update mito config toml
* feat: impl CachedPageReader
* feat: use cache reader to read row group
* feat: do not fetch data if we have pages in cache
* chore: return if nothing to fetch
* feat: enlarge page cache size
* test: test write read parquet
* test: test cache
* docs: update comments
* test: fix config api test
* feat: cache metrics
* feat: change default page cache size
* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
ZhangJian He
d9e7b898a3
feat: add walconfig dir back ( #2606 )
...
Signed-off-by: ZhangJian He <shoothzj@gmail.com >
2023-10-16 11:26:06 +00:00
Ruihang Xia
62bcb45787
feat!: change config name from kv_store to metadata_store ( #2585 )
...
featchange config name from kv_store to metadata_store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-12 06:55:09 +00:00
dennis zhuang
0a9972aa9a
fix: cache capacity unit in sample config ( #2575 )
2023-10-11 11:02:39 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
JeremyHi
98a40bae95
feat!: unify naming with options ( #2416 )
2023-09-17 07:24:57 +00:00
Weny Xu
f386329e29
refactor: introduce DdlTaskExecutor and refactor statement executor ( #2341 )
...
* feat: add kv store option
* refactor: refactor statement executor
* refactor: refactor standalone table creator
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor
* fix: fix RegionDistribution
* feat: build standalone
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1
feat: start datanode with config ( #2312 )
...
* remove memory-catalog and procedure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* derive serde for MitoConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with configs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove dir in WalConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add rename field attr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add stupid duplicated mito config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove wrong import
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wired compile error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
dennis zhuang
a3d5931fca
feat: unify all protocol options ( #2316 )
...
* feat: unify all protocol options
* feat: adds enable to example configs
* chore: style
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-09-12 07:57:15 -05:00
