zyy17
b1ef327bac
refactor: remove MixOptions and use StandaloneOptions only ( #3893 )
...
* refactor: remove MixOptions and use StandaloneOptions only
* refactor: refactor code by code review comments
1. Use '&self' in frontend_options() and datanode_options();
2. Remove unused 'clone()';
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* ci: fix integration error
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com >
2024-05-10 16:08:34 +00:00
Ruihang Xia
a6a702de4e
feat: support querying field column names in Prometheus HTTP API ( #3880 )
...
* feat: support querying field column names in Prometheus HTTP API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use tables stream API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-08 12:18:20 +00:00
Ruihang Xia
71c1c7ca24
fix: return metric name instead of query in Prometheus /series API ( #3864 )
...
* fix: return metric name instead of query in Prometheus /series API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* omit non-tag columns from result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-07 09:07:16 +00:00
shuiyisong
6e9e8fad26
refactor!: remove opentsdb tcp server ( #3828 )
...
* refactor: remove opentsdb tcp server
* refactor: remove config and add test
* refactor: update docs and remove unused code
2024-05-06 06:42:05 +00:00
Ruihang Xia
1272bc9afc
fix: post process result on query full column name of prom labels API ( #3793 )
...
* fix: post process result on query full column name of prom labels API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* only preserve tag column
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-24 09:38:03 +00:00
Ruihang Xia
f764fd5847
fix: consider both db param and extended db header in Prometheus HTTP API ( #3776 )
...
* fix: consider both db param and extended db header in Prometheus HTTP API
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove debug code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-23 07:13:05 +00:00
irenjj
7a04bfe50a
feat: add strict mode to validate protocol strings ( #3638 )
...
* feat: add strict mode to validate protocol strings
* hotfix: fix test
* fix: fix return pair and test param
* test: add test for utf-8 validation
* fix: cargo check
* Update src/servers/src/prom_row_builder.rs
Co-authored-by: Eugene Tolbakov <ev.tolbakov@gmail.com >
* fix: fix param of without_strict_mode
* fix: change field name in HttpOptions
* fix: replace if else with match
* fix: replace all strict_mode with is_stirct_mode
* fix: fix test_config_api
* fix: fix bench, add vm handshake, catch error
---------
Co-authored-by: Eugene Tolbakov <ev.tolbakov@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-15 07:53:48 +00:00
tison
5d739932c0
chore: remove TODO that has been done ( #3683 )
...
This TODO is done by https://github.com/GreptimeTeam/greptimedb/pull/3473 .
2024-04-09 22:55:55 +00:00
Ruihang Xia
50bea2f107
feat: treat all number types as field candidates ( #3670 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-09 03:28:21 +00:00
tison
883b7fce96
refactor: bundle the lightweight axum test client ( #3669 )
...
* refactor: bundle the lightweight axum test client
Signed-off-by: tison <wander4096@gmail.com >
* address comments
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-04-09 02:33:26 +00:00
Ning Sun
b32e0bba9c
feat: adding victoriametrics remote write ( #3641 )
...
* feat: adding victoria metrics remote write
* test: add e2e tests for prom and vm remote writes
2024-04-07 07:09:21 +00:00
Ruihang Xia
77cc7216af
feat: support 2+2 and /status/buildinfo ( #3604 )
...
* feat: implement buildinfo endpoint
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refactor prom result struct
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add more integration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format toml file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/servers/src/http/prometheus_resp.rs
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-03-29 06:31:39 +00:00
Ruihang Xia
06a90527a3
fix: adjust status code to http error code map ( #3601 )
...
* fix: adjust status code to http error code map
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update integration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-03-28 08:54:51 +00:00
Yingwen
641592644d
feat: support per table memtable options ( #3524 )
...
* feat: add memtable builder to region
* refactor: rename memtable_builder in worker to default_memtable_builder
* fix: return error instead of using default compaction options
Support deserializing memtable and compaction options from the option
map
* feat: optional memtable options
* feat: add MemtableBuilderProvider to create builders
* feat: change default memtable and skip deserializing dedup
* chore: update test and comment
* chore: test invalid type
* feat: metric engine use new memtable manually
* feat: expose more memtable configs
* feat: add memtable options to valid option list
* test: add test
* test: sqlness test
* chore: serde workspace
* chore: remove comments
2024-03-19 08:50:10 +00:00
Yingwen
06dcd0f6ed
fix: freeze data buffer in shard ( #3468 )
...
* feat: call freeze if the active data buffer in a shard is full
* chore: more metrics
* chore: print metrics
* chore: enlarge freeze threshold
* test: test freeze
* test: fix config test
2024-03-11 14:51:06 +00:00
crwen
aa125a50f9
refactor: make http api returns non-200 status code ( #3473 )
...
* refactor: make http api returns non-200 status code
* recover some code
2024-03-11 03:38:36 +00:00
Lei, HUANG
7183fa198c
refactor: make MergeTreeMemtable the default choice ( #3430 )
...
* refactor: make MergeTreeMemtable the default choice
* refactor: reformat
* chore: add doc to config
2024-03-05 10:00:08 +00:00
Yingwen
49157868f9
feat: Correct server metrics and add more metrics for scan ( #3426 )
...
* feat: drop timer on stream terminated
* refactor: combine metrics into a histogram vec
* refactor: frontend grpc metrics
* feat: add metrics middleware layer to grpc server
* refactor: move http metrics layer to metrics mod
* feat: bucket for grpc/http elapsed
* feat: remove duplicate metrics
* style: fix cilppy
* fix: incorrect bucket of promql series
* feat: more metrics for mito
* feat: convert cost
* test: fix metrics test
2024-03-04 10:15:10 +00:00
Ning Sun
d4a54a085b
feat: add configuration for tls watch option ( #3395 )
...
* feat: add configuration for tls watch option
* test: sleep longer to ensure async task run
* test: update config api integration test
* refactor: rename function
2024-03-01 03:49:54 +00:00
Ning Sun
1bc4f25de2
feat: http sql api return schema on empty resultset ( #3237 )
...
* feat: return schema on empty resultset
* refactor: make schema a required field in http output
* test: update integration test and add schema output
2024-01-25 06:44:28 +00:00
Weny Xu
19413eb345
refactor!: rename initialize_region_in_background to init_regions_in_background ( #3216 )
...
refactor!: change initialize_region_in_background to init_regions_in_background
2024-01-23 04:33:48 +00:00
Ruihang Xia
31787f4bfd
feat!: switch prom remote write to metric engine ( #3198 )
...
* feat: switch prom remote write to metric engine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* read physical table name from url
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove physical table from header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix merge error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add with_metric_engine option to config remote write behavior
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* check parameter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add specific config param
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* default with_metric_engine to true
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update UT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 14:48:28 +00:00
Zhenchi
2e4c48ae7a
fix(index): S3 EntityTooSmall error ( #3192 )
...
* fix(index): S3 `EntityTooSmall` error
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: config api
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-01-19 02:57:07 +00:00
Wei
3d7d2fdb4a
feat: auto config cache size according to memory size ( #3165 )
...
* feat: auto config cache and buffer size according to mem size
* feat: utils
* refactor: add util function to common config
* refactor: check cgroups
* refactor: code
* fix: test
* fix: test
* chore: cr comment
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* chore: remove default comment
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
2024-01-17 14:35:35 +00:00
Zhenchi
6f07d69155
feat(mito): enable inverted index ( #3158 )
...
* feat(mito): enable inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* accidentally resolved the incorrect filtering issue within the Metric Engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
* Update src/mito2/src/test_util/scheduler_util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: format -> join_dir
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move intermediate_manager from arg of write_and_upload_sst to field of WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: add IndexerBuidler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 09:08:07 +00:00
WU Jingdi
816d94892c
feat: support HTTP&gRPC&pg set timezone ( #3125 )
...
* feat: support HTTP&gRPC&pg set timezone
* chore: fix code advice
* chore: fix code advice
2024-01-15 06:29:31 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
Yingwen
8bd4a36136
feat(mito): Init the write cache in datanode ( #3100 )
...
* feat: add builder to build cache manager
* refactor: make MitoEngine::new async
* refactor: refactor object store creation
* refactor: add helper fn to attaches layers
* feat: fn to build fs store
* feat: add write cache to engine
* feat: config write cache
* style: fix clippy
* test: fix test
* feat: add warning
* chore: add experimental prefix to configs
* test: fix config test
* test: test weighted size
* feat: add switch to enable write cache
* fix: update cache stats by using get
* style: use then
2024-01-09 04:40:22 +00:00
tison
316d843482
feat: support CSV format in sql HTTP API ( #3062 )
...
* chore: fix typo
Signed-off-by: tison <wander4096@gmail.com >
* add csv format
Signed-off-by: tison <wander4096@gmail.com >
* flatten response
Signed-off-by: tison <wander4096@gmail.com >
* more flatten response
Signed-off-by: tison <wander4096@gmail.com >
* add CSV format
Signed-off-by: tison <wander4096@gmail.com >
* format InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format ErrorResponse
Signed-off-by: tison <wander4096@gmail.com >
* propagate ErrorResponse to InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format GreptimedbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format CsvResponse
Signed-off-by: tison <wander4096@gmail.com >
* impl IntoResponse for QueryResponse
Signed-off-by: tison <wander4096@gmail.com >
* promql
Signed-off-by: tison <wander4096@gmail.com >
* sql
Signed-off-by: tison <wander4096@gmail.com >
* compile
Signed-off-by: tison <wander4096@gmail.com >
* fixup aide
Signed-off-by: tison <wander4096@gmail.com >
* clear debt
Signed-off-by: tison <wander4096@gmail.com >
* fixup UT test_recordbatches_conversion
Signed-off-by: tison <wander4096@gmail.com >
* fixup IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup more IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup test-integration cases
Signed-off-by: tison <wander4096@gmail.com >
* update comment
Signed-off-by: tison <wander4096@gmail.com >
* fixup deserialize and most query < 1ms
Signed-off-by: tison <wander4096@gmail.com >
* fixup auth tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup and align X-GreptimeDB headers
Signed-off-by: tison <wander4096@gmail.com >
* fixup compile
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-01-08 10:54:27 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
LFC
196c06db14
feat: make logging to stdout configurable ( #3003 )
...
* feat: make logging to stdout configurable
* fix sqlness
* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
JeremyHi
99dda93f0e
feat: sql with influxdb v1 result format ( #2917 )
...
* feat: sql with influxdb v1 result format
* chore: add unit tests
* feat: minor refactor
* chore: by comment
* chore; u128 to u64 since serde can't deser u128 in enum
* chore: by comment
* chore: apply suggestion
* chore: revert suggestion
* chore: try again
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-13 16:15:37 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f
feat(mito): implements row group level page cache ( #2688 )
...
* feat: add page cache
* docs: update mito config toml
* feat: impl CachedPageReader
* feat: use cache reader to read row group
* feat: do not fetch data if we have pages in cache
* chore: return if nothing to fetch
* feat: enlarge page cache size
* test: test write read parquet
* test: test cache
* docs: update comments
* test: fix config api test
* feat: cache metrics
* feat: change default page cache size
* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074
fix!: improve user experience on setting compression type ( #2765 )
...
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type
* Update src/common/datasource/src/compression.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-17 06:27:13 +00:00
Ruihang Xia
6599bb5a46
feat: do not require worker's number to be power of 2 ( #2732 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-13 09:04:44 +00:00
tison
f5eede4ce1
feat: support prometheus format_query endpoint ( #2731 )
...
* feat: support prometheus format_query endpoint
Signed-off-by: tison <wander4096@gmail.com >
* add test
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-11-10 10:33:28 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
Niwaka
06d273b75a
fix: GET: /config return all configurations when running standalone ( #2630 )
...
* fix: config api return all configurations when running standalone
* chore: follow same style
* fix: avoid panic
2023-11-08 03:22:19 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Yun Chen
f859932745
fix: convert to ReadableSize & Durations ( #2594 )
...
* fix: convert to ReadableSize & Durations
* fix: change more grpc sender/recv message size to ReadableSize
fix: format
fix: cargo fmt
fix: change cmd test to use durations
fix: revert metaclient change
fix: convert default fields in meta client options
fix: human serde meta client durations
* fix: remove milisecond postfix in heartbeat option
* fix: humantime serde on heartbeat
* fix: update config example
* fix: update integration test config
* fix: address pr comments
* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
