rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-14 01:02:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,127 Commits 105 Branches 227 Tags
fc6300a2ba760dd5b9db2e2458e7d34d5f7c61ba
Commit Graph

216 Commits

Author SHA1 Message Date
Ruihang Xia
7ae0e150e5 feat: support altering multiple logical table in one remote write request (#6137) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-05-20 11:22:38 +00:00
localhost
c2e3c3d398 chore: Add more data format support to the pipeline dryrun api. (#6115) 
* chore: supporting more data type for pipeline dryrun API

* chore: add docs for parse_dryrun_data

* chore: fix by pr comment

* chore: add user-friendly error message

* chore: change EventPayloadResolver content_type field type from owner to ref

* Apply suggestions from code review

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
 2025-05-20 03:29:28 +00:00
Zhenchi
400229c384 feat: introduce index result cache (#6110) 
* feat: introduce index result cache

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/mito2/src/sst/index/inverted_index/applier/builder.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* optimize selector_len

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-05-20 01:45:42 +00:00
shuiyisong
a0d89c9ed1 feat: Prometheus remote write with pipeline (#5981) 
* chore: update nightly version

* chore: sort lint lines

* chore: minor fix

* chore: update nix

* chore: update toolchain to 2024-04-14

* chore: update toolchain to 2024-04-15

* chore: remove unnecessory test

* chore: do not assert oid in sqlness test

* chore: fix margin issue

* chore: fix cr issues

* chore: fix cr issues

* chore: add pipelie handler to prom state

* chore: add prom series processor to merge function

* chore: add run pipeline in decode

* chore: add channel to pipeline ctx

* chore: add pipeline info to remote wirte hander

* chore: minor update

* chore: minor update

* chore: add test

* chore: add comment

* refactor: simplify identity pipeline params

* fix: test

* refactor: remove is_prometheus

---------

Co-authored-by: Ning Sun <sunning@greptime.com>
 2025-05-19 08:00:59 +00:00
zyy17
01076069a3 chore: modify default slow_query.threshold from 5s to 30s (#6107) 
chore: modify slow_query.threshold from 5s to 30s
 2025-05-15 20:16:13 +00:00
zyy17
14b655ea57 refactor: add SlowQueryRecorder to record slow query in system table and refactor slow query options (#6008) 
* refactor: add common-slow-query crate

* refactor: refine the naming

* chore: fix clippy

* chore: fix typo

* chore: sperate SlowQueryOptions From Logging

* chore: fix clippy

* chore: fix ci

* chore: refine the code

* chore: update config example

* refactor: use drop() to end the slow query timer

* refactor: move common-slow-query to frontend crate

* chore: polish some code

* refactor: code review

* refactor: add promql_range/promql_step/promql_start/promql_end fields in slow_queries

* refactor: add build_slow_query_logger()

* refactor: turn on slow query on frontend by default
 2025-05-15 04:18:48 +00:00
zyy17
50df275097 fix!: disable append mode in trace services table (#6066) 
fix: disable append mode in trace services table and make 'service_name' as primary key
 2025-05-09 19:06:51 +00:00
dennis zhuang
fbf50c594e fix: csv format escaping (#6061) 
* fix: csv format escaping

* chore: change status code

* fix: crate version
 2025-05-08 05:52:20 +00:00
shuiyisong
56f31d5933 feat(pipeline): select processor (#6019) 
* feat: support auto transform

* refactor: replace hashbrown with ahash

* refactor: params of run identity pipeline

* refactor: minor update

* test: add test for auto transform

* feat: add select processor

* test: select processor

* chore: use include and exclude for key

* fix: typos

* chore: address CR comment

* chore: typo

* chore: typo

* chore: address CR comment

* chore: use with_context
 2025-05-07 03:40:11 +00:00
Yingwen
07e84a28a3 fix: do not add projection to cast timestamp in label_values (#6040) 
* fix: do not add projection for cast

Use cast to build time filter directly instead of adding a projection,
which will cause column not found

* feat: cast before creating plan
 2025-05-06 23:47:41 +00:00
shuiyisong
a706edbb73 feat(pipeline): auto transform (#6013) 
* feat: support auto transform

* refactor: replace hashbrown with ahash

* refactor: params of run identity pipeline

* refactor: minor update

* test: add test for auto transform

* chore: fix cr issues
 2025-04-30 07:40:26 +00:00
Ning Sun
bcefc6b83f feat: add format support for promql http api (not prometheus) (#5939) 
* feat: add format support for promql http api (not prometheus)

* test: add csv format test
 2025-04-22 08:10:35 +00:00
Ruihang Xia
e4556ce12b fix: label values potential panic (#5921) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-04-17 14:01:21 +00:00
Lin Yihai
7274ceba30 feat: Add query pipeline http api (#5819) 
* feat(pipeline): add query pipeline http api.

* chore(pipeline): rename get pipepile method

* refactor(pipeline): Also insert string piple  into cache after inserting into table.

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
 2025-04-16 10:17:20 +00:00
zyy17
7b13376239 refactor: add partition_rules_for_uuid() (#5743) 
* refactor: add partition_rules_for_uuid()

* refactor: support up to 65536 partitions for partition_rules_for_uuid()
 2025-04-15 06:46:31 +00:00
Zhenchi
8d485e9be0 feat: support altering fulltext backend (#5896) 
* feat: add `greptime_index_type` to `information_schema.key_column_usage`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: show create

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-04-15 06:36:06 +00:00
Ning Sun
be837ddc24 test: add tests to ensure nested data structure for identity pipeline (#5888) 2025-04-14 03:13:46 +00:00
LFC
311727939d chore: update datafusion family (#5814) 2025-04-09 02:20:55 +00:00
zyy17
ee4fe9d273 refactor: improve performance for Jaeger APIs (#5838) 
* refactor: improve jaeger '/api/services' performance by adding the trace services table

* chore: refine some logic

* chore: compatible v0

* test: add integration test

* chore: expand default limit from 100 to 2000

* test: fix integration test

* refactor: make trace service table configurable

* refactor: use a timestamp(2100-01-01 00:00:00) as large as possible

* refactor: use '<trace_table>_services' as trace services table name
 2025-04-08 02:28:06 +00:00
zyy17
cf1440fc32 refactor: add time range for jager get operations API (#5791) 
* refactor: add default time range for jager get operations API

* refactor: use desc order for timestamp colomn

* chore: modify http header name
 2025-04-07 09:07:31 +00:00
Ning Sun
f2907bb009 refactor!: make pipeline a required parameter when ingesting trace (#5828) 
* feat: make pipeline a required header for trace

* test: add test case without pipeline
 2025-04-07 06:18:17 +00:00
fys
33c9fb737c refactor: remove mode option in configuration files (#5809) 
* refactor: remove mode option in configuration files

* chore: remove mode in configuration file

* remvoe mode field in FlownodeOptions

* add comment for test

* update config.md

* remove mode field in standalone options

* fix: ci
 2025-04-01 07:14:10 +00:00
Weny Xu
d3a60d8821 feat: add limit for the number of running procedures (#5793) 
* refactor: remove unused `messages`

* feat: introduce running procedure num limit

* feat: update config

* chore: apply suggestions from CR

* feat: impl `status_code` for `log-store` crate
 2025-03-31 06:14:21 +00:00
shuiyisong
bef45ed0e8 feat(pipeline): support table name suffix templating in pipeline (#5775) 
* chore: add table name template in pipeline yaml

* chore: implement apply function and add simple test

* chore: add comment and integration test

* chore: minor update

* fix: typos

* chore: change to table suffix

* chore: update comment and test

* chore: change name to table_suffix
 2025-03-28 18:12:46 +00:00
Yingwen
737558ef53 fix: support __name__ matcher in label values (#5773) 2025-03-28 02:18:59 +00:00
shuiyisong
c77ce958a3 chore: support custom time index selector for identity pipeline (#5750) 
* chore: minor refactor

* chore: minor refactor

* chore: support custom ts for identity pipeline

* chore: fix clippy

* chore: minor refactor & update tests

* chore: use ref on identity pipeline param
 2025-03-24 04:27:22 +00:00
zyy17
a19441bed8 refactor: remove trace id from primary key in opentelemetry_traces table (#5733) 
* refactor: remove trace id in primary key

* refactor: remove trace id in primary key in v0 model

* refactor: add span id in v1

* fix: integration test
 2025-03-19 06:17:58 +00:00
Ning Sun
1ab4ddab8d feat: update pipeline header name to x-greptime-pipeline-name (#5710) 
* feat: update pipeline header name to x-greptime-pipeline-name

* refactor: update string_value_from_header
 2025-03-18 02:39:54 +00:00
Ning Sun
9e63018198 feat: disable http timeout (#5721) 
* feat: update to disable http timeout by default

* feat: make http timeout default to 0

* test: correct test case

* chore: generate new config doc

* test: correct tests
 2025-03-18 01:18:56 +00:00
Ning Sun
2260782c12 refactor: update jaeger api implementation for new trace modeling (#5655) 
* refactor: update jaeger api implementation

* test: add tests for v1 data model

* feat: customize trace table name

* fix: update column requirements to use Column type instead of String

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: lint fix

* refactor: accumulate resource attributes for v1

* fix: add empty check for additional string

* feat: add table option to mark data model version

* fix: do not overwrite all tags

* feat: use table option to mark table data model version and process accordingly

* chore: update comments to reflect query changes

* feat: use header for jaeger table name

* feat: update index for service_name, drop index for span_name

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: zyy17 <zyylsxm@gmail.com>
 2025-03-17 07:31:32 +00:00
shuiyisong
fcb898e9a4 chore: support inverted index in pipeline (#5700) 
chore: rebase main
 2025-03-13 08:30:29 +00:00
Ning Sun
8fa2fdfc42 feat: make empty parent_span_id null for v1 (#5690) 2025-03-13 07:48:15 +00:00
shuiyisong
4dc1a1d60f chore: support tag in transform (#5701) 
chore: support tag in transform to specify tag
 2025-03-13 07:27:12 +00:00
Yingwen
25645a3303 feat: expose virtual_host_style config for s3 storage (#5696) 
* feat: expose enable_virtual_host_style for s3 storage

* docs: update examples

* test: fix config test
 2025-03-12 13:46:56 +00:00
Weny Xu
0bd322a078 perf(prom): optimize label values query (#5653) 
perf: optimize label values query
 2025-03-10 13:20:47 +00:00
shuiyisong
448e588fa7 chore: improve /v1/jaeger/api/trace/{trace_id}'s resp (#5663) 
* chore: improve jaeger trace api resp

* chore: fix timestamp type

* chore: fix timestamp type

* chore: complete more fields

* chore: change to microseconds

* chore: add empty check & span status code

* chore: minor update

* chore: update test
 2025-03-07 04:31:42 +00:00
Ning Sun
37f8341963 feat: opentelemetry trace new data modeling (#5622) 
* feat: include trace v1 encoding

* feat: add trace ingestion in inserter

* feat: add partition rules and index for trace_id

* chore: format

* chore: fmt

* fix: issue introduced with merge

* feat: adjust index and add integration test for v1

* refactor: remove comment key

* fix: update default value of skip index granularity

* fix: update default value of skip index granularity

* refactor: rename some functions

* feat: remove skipping index from span_id

* refactor: made span_id part of primary key for potential dedup purpose

* feat: move the special attribute resource_attribute.service.name to top level

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
 2025-03-05 04:08:52 +00:00
shuiyisong
31f29d8a77 chore: support specifying skipping index in pipeline (#5635) 
* chore: support setting skipping index in pipeline

* chore: fix typo key

* chore: add test

* chore: fix typo
 2025-03-03 18:37:13 +00:00
Ruihang Xia
d69e93b91a feat: support to generate json output for explain analyze in http api (#5567) 
* impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/servers/src/http/hints.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor: with FORMAT option for explain format

* lift some well-known metrics

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
 2025-02-21 05:13:09 +00:00
shuiyisong
53b25c04a2 chore: support Loki's structured metadata for ingestion (#5541) 
* chore: support loki's structured metadata

* test: update test

* chore: revert some code change

* chore: address CR comment
 2025-02-19 16:44:26 +00:00
Zhenchi
421e38c481 feat: allow purging a given puffin file in staging area (#5558) 
* feat: purge a given puffin file in staging area

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish log

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* ttl set to 2d

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: expose staging_ttl to index config

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* use `invalidate_entries_if` instead of maintaining map

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* run_pending_tasks after purging

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
 2025-02-19 08:58:30 +00:00
zyy17
8ebf454bc1 fix(jaeger): return error when no tracing table (#5539) 
fix: return error when no tracing table
 2025-02-14 06:20:56 +00:00
Weny Xu
63d5a69a31 fix(query_range): skip data field on errors (#5520) 
* fix: skip serializing PrometheusResponse when None

* fix: fix unit test

* chore: clippy
 2025-02-13 04:32:24 +00:00
zyy17
954310f917 feat: implement Jaeger query APIs (#5452) 
* feat: implement jaeger query api

* test: add some unit tests

* test: add integration tests for jaeger query APIs

* refactor: parse tags from url parameters

* refactor: support to query traces by tags

* refactor: add limit parameter

* refactor: add jaeger query api metrics

* chore: add some comment docs and default limit value

* test: add more unit tests

* docs: add jaeger options in config docs

* refactor: code review

* wip

* refactor: use datafusion's dataframe APIs to query traces

* refactor: code review

* chore: format test cases

* refactor: add check_schema()

* chore: fix clippy errors and rename function name

* refactor: throw error when covert start_time and duration error

* chore: modify incorrect request type name

* chore: remove unecessary serde rename

* refactor: add some important comments

* refactor: add SPAN_KIND_PREFIX

* refactor: code review
 2025-02-12 23:36:38 +00:00
Ruihang Xia
e22aa819be feat: support server-side keep-alive for mysql and pg protocols (#5496) 
* feat: support server-side keep-alive for mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config.md

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config to use humantime for keep-alive configuration

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: Update socket2 dependency

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-11 19:22:10 +00:00
Ning Sun
480b05c590 feat: pipeline dispatcher part 2: execution (#5409) 
* fmt: correct format

* test: add negative tests

* feat: Add pipeline dispatching and execution output handling

* refactor: Enhance ingest function to correctly process original data values

custom table names during pipeline execution while optimizing the management of
transformed rows and multiple dispatched pipelines

* refactor: call greptime_identity with intermediate values

* fix: typo

* test: port tests to refactored apis

* refactor: adapt dryrun api call

* refactor: move pipeline execution code to a separated module

* refactor: update otlp pipeline execution path

* fmt: format imports

* fix: compilation

* fix: resolve residual issues

* refactor: address review comments

* chore: use btreemap as pipeline intermediate status trait modify

* refactor: update dispatcher to accept BTreeMap

* refactor: update identity pipeline

* refactor: use new input for pipeline

* chore: wip

* refactor: use updated prepare api

* refactor: improve error and header name

* feat: port flatten to new api

* chore: update pipeline api

* chore: fix transform and some pipeline test

* refactor: reimplement cmcd

* refactor: update csv processor

* fmt: update format

* chore: fix regex and dissect processor

* chore: fix test

* test: add integration test for http pipeline

* refactor: improve regex pipeline

* refactor: improve required field check

* refactor: rename table_part to table_suffix

* fix: resolve merge issue

---------

Co-authored-by: paomian <xpaomian@gmail.com>
 2025-02-08 09:01:54 +00:00
Ruihang Xia
0de0fd80b0 feat: move pipelines to the first-class endpoint (#5480) 
* feat: move pipelines to the first-class endpoint

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change endpoints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* prefix path with /

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update integration result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-08 03:46:31 +00:00
ozewr
0e4bd59fac build: Update Loki proto (#5484) 
* build: mv loki-api to loki-proto

* fmt: fmt toml

* fix: loki-proto using rev

---------

Co-authored-by: wangrui <wangrui@baihai.ai>
 2025-02-07 09:09:39 +00:00
shuiyisong
88c3d331a1 refactor: otlp logs insertion (#5479) 
* chore: add test for selector overlapping

* refactor: simplify otlp logs insertion

* fix: use layered extracted value array

* fix: wrong len

* chore: minor renaming and update

* chore: rename

* fix: clippy

* fix: typos

* chore: update test

* chore: address CR comment & update meter-deps version
 2025-02-07 07:21:20 +00:00
Weny Xu
c80d2a3222 fix: introduce gc task for metadata store (#5461) 
* fix: introduce gc task for metadata kvbackend

* refactor: refine KvbackendConfig

* chore: apply suggestions from CR
 2025-02-06 12:12:43 +00:00