shuiyisong
dec8c52b18
feat(pipeline): support Loki API ( #6390 )
...
* chore: use schema_info
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* refactor: abstract loki item generator
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: introduce middle item
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* feat: introduce pipeline in loki api
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* test: add tests
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: minor update
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: minor update
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: update prefix and test
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: change recursion to loop
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* fix: cr issue
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
2025-06-28 01:01:08 +00:00
shuiyisong
7cd6be41ce
feat(pipeline): introduce pipeline doc version 2 for combine-transform ( #6360 )
...
* chore: init commit of pipeline doc version v2
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: remove unused code
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: remove unused code
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: add test
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: add test
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: compatible with v1 to remain field in the map during transform
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* refactor: pipeline.exec_mut
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* fix: typo
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: change from v2 to 2 in version setting
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
2025-06-22 00:58:36 +00:00
localhost
53c4fd478e
chore: add skip error for pipeline skip error log ( #6318 )
...
* chore: wip
Signed-off-by: paomian <xpaomian@gmail.com >
* chore: add skip error for pipeline skip error log
Signed-off-by: paomian <xpaomian@gmail.com >
* chore: add test and check timestamp must be non null
Signed-off-by: paomian <xpaomian@gmail.com >
* chore: fix test
* chore: fix by pr comment
* fix: typo
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Signed-off-by: paomian <xpaomian@gmail.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-06-17 06:44:11 +00:00
shuiyisong
079daf5db9
feat: support special labels parsing in prom remote write ( #6302 )
...
* feat: support special labels parsing in prom remote write
* chore: change __schema__ to __database__
* fix: test
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* fix: remove the missing type alias
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
* chore: update cr issue
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com >
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2025-06-17 03:20:33 +00:00
localhost
c10b8f8474
chore: add failover cache for pipeline table ( #6284 )
...
* chore: add second level cache for pipeline table
* chore: change pipeline failover cache name
* chore: add counter metrics for get pipeline operate
2025-06-12 03:15:02 +00:00
shuiyisong
041b683a8d
refactor: remove PipelineMap and use Value instead ( #6278 )
...
* refactor: remove pipeline_map and use value instead
* chore: remove unused comments
* chore: move error to illegal state
2025-06-11 17:02:32 +00:00
shuiyisong
538b5abaae
chore: support table suffix in hint ( #6223 )
...
* feat: pipeline recognize hints from exec
* chore: rename and add test
* chore: minor improve
* chore: rename and add comments
* fix: typos
* feat: add initial impl for vrl processor
* chore: update processors to allow vrl process
* feat: pipeline recognize hints from exec
* chore: rename and add test
* chore: minor improve
* chore: rename and add comments
* fix: typos
* chore: remove unnecessory clone fn
* chore: group metrics
* chore: use struct in transform output enum
* test: add test for vrl
* fix: leaked conflicts
* chore: merge branch code & add check in compile
* fix: check condition
* fix: check auto-transform timeindex
* chore: support table_suffix in hint
* chore: add test for table suffix in vrl hint
* refactor: change context_opt to a struct
2025-06-06 20:39:10 +00:00
shuiyisong
a2b3ad77df
feat(pipeline): vrl processor ( #6205 )
...
* feat: pipeline recognize hints from exec
* chore: rename and add test
* chore: minor improve
* chore: rename and add comments
* fix: typos
* feat: add initial impl for vrl processor
* chore: update processors to allow vrl process
* feat: pipeline recognize hints from exec
* chore: rename and add test
* chore: minor improve
* chore: rename and add comments
* fix: typos
* chore: remove unnecessory clone fn
* chore: group metrics
* chore: use struct in transform output enum
* test: add test for vrl
* fix: leaked conflicts
* chore: merge branch code & add check in compile
* fix: check condition
2025-06-06 16:35:19 +00:00
Yingwen
ee4f830be6
fix: do not accommodate fields for multi-value protocol ( #6237 )
2025-06-04 01:10:52 +00:00
shuiyisong
69975f1f71
feat: pipeline with insert options ( #6192 )
...
* feat: pipeline recognize hints from exec
* chore: rename and add test
* chore: minor improve
* chore: rename and add comments
* fix: typos
* chore: remove unnecessory clone fn
* chore: group metrics
* chore: use struct in transform output enum
* chore: update hint prefix
2025-06-03 18:46:48 +00:00
shuiyisong
af6cf999c1
chore: shared pipeline under same catalog with compatibility ( #6143 )
...
* chore: support shared pipeline under catalog with compatibility
* test: add test for cross schema ref
* chore: use empty string schema by default
* chore: remove unwrap in the patch
* fix: df check
2025-05-30 07:19:32 +00:00
Ruihang Xia
300262562b
feat: accommodate default column name with pre-created table schema ( #6126 )
...
* refactor: prepare_mocked_backend
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* modify request in place
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply to influx line protocol
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* return on empty alter expr list
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* expose to other write paths
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-05-20 07:22:13 +00:00
localhost
c2e3c3d398
chore: Add more data format support to the pipeline dryrun api. ( #6115 )
...
* chore: supporting more data type for pipeline dryrun API
* chore: add docs for parse_dryrun_data
* chore: fix by pr comment
* chore: add user-friendly error message
* chore: change EventPayloadResolver content_type field type from owner to ref
* Apply suggestions from code review
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
---------
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
2025-05-20 03:29:28 +00:00
shuiyisong
a0d89c9ed1
feat: Prometheus remote write with pipeline ( #5981 )
...
* chore: update nightly version
* chore: sort lint lines
* chore: minor fix
* chore: update nix
* chore: update toolchain to 2024-04-14
* chore: update toolchain to 2024-04-15
* chore: remove unnecessory test
* chore: do not assert oid in sqlness test
* chore: fix margin issue
* chore: fix cr issues
* chore: fix cr issues
* chore: add pipelie handler to prom state
* chore: add prom series processor to merge function
* chore: add run pipeline in decode
* chore: add channel to pipeline ctx
* chore: add pipeline info to remote wirte hander
* chore: minor update
* chore: minor update
* chore: add test
* chore: add comment
* refactor: simplify identity pipeline params
* fix: test
* refactor: remove is_prometheus
---------
Co-authored-by: Ning Sun <sunning@greptime.com >
2025-05-19 08:00:59 +00:00
shuiyisong
b442414422
chore: support rename syntax in field ( #6065 )
...
* chore: support rename syntax in field
* test: rename in transform
2025-05-09 00:12:23 +00:00
dennis zhuang
fbf50c594e
fix: csv format escaping ( #6061 )
...
* fix: csv format escaping
* chore: change status code
* fix: crate version
2025-05-08 05:52:20 +00:00
shuiyisong
56f31d5933
feat(pipeline): select processor ( #6019 )
...
* feat: support auto transform
* refactor: replace hashbrown with ahash
* refactor: params of run identity pipeline
* refactor: minor update
* test: add test for auto transform
* feat: add select processor
* test: select processor
* chore: use include and exclude for key
* fix: typos
* chore: address CR comment
* chore: typo
* chore: typo
* chore: address CR comment
* chore: use with_context
2025-05-07 03:40:11 +00:00
shuiyisong
a706edbb73
feat(pipeline): auto transform ( #6013 )
...
* feat: support auto transform
* refactor: replace hashbrown with ahash
* refactor: params of run identity pipeline
* refactor: minor update
* test: add test for auto transform
* chore: fix cr issues
2025-04-30 07:40:26 +00:00
shuiyisong
3c943be189
chore: update rust toolchain ( #5818 )
...
* chore: update nightly version
* chore: sort lint lines
* chore: minor fix
* chore: update nix
* chore: update toolchain to 2024-04-14
* chore: update toolchain to 2024-04-15
* chore: remove unnecessory test
* chore: do not assert oid in sqlness test
* chore: fix margin issue
* chore: fix cr issues
* chore: fix cr issues
---------
Co-authored-by: Ning Sun <sunning@greptime.com >
2025-04-27 09:02:36 +00:00
shuiyisong
5c07f0dec7
refactor: run_pipeline parameters ( #5954 )
...
* refactor: simplify run_pipeline params
* refactor: remove unnecessory function wrap
2025-04-22 11:34:19 +00:00
shuiyisong
3b8c6d5ce3
chore: use once_cell to avoid parse everytime in pipeline exec ( #5943 )
...
* chore: use once_cell to avoid parse everytime
* chore: remove pub on options
2025-04-21 12:55:48 +00:00
shuiyisong
9df493988b
fix: wrong error msg in pipeline ( #5937 )
2025-04-21 04:05:46 +00:00
shuiyisong
fdab5d198e
feat: add json parse processor ( #5910 )
...
* feat: add json parse processor
* chore: support parse to arr
2025-04-16 10:37:07 +00:00
Lin Yihai
7274ceba30
feat: Add query pipeline http api ( #5819 )
...
* feat(pipeline): add query pipeline http api.
* chore(pipeline): rename get pipepile method
* refactor(pipeline): Also insert string piple into cache after inserting into table.
---------
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
2025-04-16 10:17:20 +00:00
Ruihang Xia
c26e165887
refactor: check and fix super import ( #5846 )
...
* refactor: check and fix super import
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add to makefile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change dir
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-08 11:48:52 +00:00
Ning Sun
f2907bb009
refactor!: make pipeline a required parameter when ingesting trace ( #5828 )
...
* feat: make pipeline a required header for trace
* test: add test case without pipeline
2025-04-07 06:18:17 +00:00
shuiyisong
f9221e9e66
perf: introduce simd_json for parsing ndjson ( #5794 )
...
* perf: introduce simd_json for parsing ndjson
* fix: some tests
* fix: some tests
* fix: es test case
* chore: use `as_bytes_mut()`
* chore: remove unnecessary `to_string`
* chore: add safety comment
2025-04-01 08:17:26 +00:00
shuiyisong
bef45ed0e8
feat(pipeline): support table name suffix templating in pipeline ( #5775 )
...
* chore: add table name template in pipeline yaml
* chore: implement apply function and add simple test
* chore: add comment and integration test
* chore: minor update
* fix: typos
* chore: change to table suffix
* chore: update comment and test
* chore: change name to table_suffix
2025-03-28 18:12:46 +00:00
shuiyisong
f13a43647a
chore: remove Transformer trait ( #5772 )
...
* chore: remove transformer trait
* chore: remove unnecessory generic
2025-03-26 02:53:30 +00:00
shuiyisong
c77ce958a3
chore: support custom time index selector for identity pipeline ( #5750 )
...
* chore: minor refactor
* chore: minor refactor
* chore: support custom ts for identity pipeline
* chore: fix clippy
* chore: minor refactor & update tests
* chore: use ref on identity pipeline param
2025-03-24 04:27:22 +00:00
shuiyisong
2431cd3bdf
chore: merge error files under pipeline crate ( #5738 )
2025-03-19 09:55:51 +00:00
Ning Sun
2260782c12
refactor: update jaeger api implementation for new trace modeling ( #5655 )
...
* refactor: update jaeger api implementation
* test: add tests for v1 data model
* feat: customize trace table name
* fix: update column requirements to use Column type instead of String
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: lint fix
* refactor: accumulate resource attributes for v1
* fix: add empty check for additional string
* feat: add table option to mark data model version
* fix: do not overwrite all tags
* feat: use table option to mark table data model version and process accordingly
* chore: update comments to reflect query changes
* feat: use header for jaeger table name
* feat: update index for service_name, drop index for span_name
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: zyy17 <zyylsxm@gmail.com >
2025-03-17 07:31:32 +00:00
shuiyisong
2bfdae4f8f
feat: add simple extract processor ( #5688 )
...
* feat: add simple extract processor
* chore: add test
* chore: add license header
* chore: minor update
2025-03-13 09:19:58 +00:00
shuiyisong
fcb898e9a4
chore: support inverted index in pipeline ( #5700 )
...
chore: rebase main
2025-03-13 08:30:29 +00:00
shuiyisong
4dc1a1d60f
chore: support tag in transform ( #5701 )
...
chore: support tag in transform to specify tag
2025-03-13 07:27:12 +00:00
localhost
c14aa176b5
chore: impl ref and ref_mut for json like ( #5679 )
...
* chore: impl ref and ref_mut for json like
* chore: add code source
2025-03-10 10:43:15 +00:00
shuiyisong
31f29d8a77
chore: support specifying skipping index in pipeline ( #5635 )
...
* chore: support setting skipping index in pipeline
* chore: fix typo key
* chore: add test
* chore: fix typo
2025-03-03 18:37:13 +00:00
Ning Sun
c9d70e0e28
refactor: add pipeline concept to OTLP traces and remove OTLP over gRPC ( #5605 )
2025-02-27 14:01:45 +00:00
Ning Sun
81da18e5df
refactor: use global type alias for pipeline input ( #5568 )
...
* refactor: use global type alias for pipeline input
* fmt: reformat
2025-02-19 10:41:33 +00:00
yihong
224b1d15cd
chore: use the same version of chrono-tz ( #5523 )
...
* fix: use the same version of chrono-tz
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
* fix: address comments
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
2025-02-13 17:23:29 +00:00
localhost
beb9c0a797
chore: set now as timestamp field default value ( #5502 )
...
* chore: set now as timestamp field default value
* chore: import pipeline default value
2025-02-11 17:41:44 +00:00
Ning Sun
480b05c590
feat: pipeline dispatcher part 2: execution ( #5409 )
...
* fmt: correct format
* test: add negative tests
* feat: Add pipeline dispatching and execution output handling
* refactor: Enhance ingest function to correctly process original data values
custom table names during pipeline execution while optimizing the management of
transformed rows and multiple dispatched pipelines
* refactor: call greptime_identity with intermediate values
* fix: typo
* test: port tests to refactored apis
* refactor: adapt dryrun api call
* refactor: move pipeline execution code to a separated module
* refactor: update otlp pipeline execution path
* fmt: format imports
* fix: compilation
* fix: resolve residual issues
* refactor: address review comments
* chore: use btreemap as pipeline intermediate status trait modify
* refactor: update dispatcher to accept BTreeMap
* refactor: update identity pipeline
* refactor: use new input for pipeline
* chore: wip
* refactor: use updated prepare api
* refactor: improve error and header name
* feat: port flatten to new api
* chore: update pipeline api
* chore: fix transform and some pipeline test
* refactor: reimplement cmcd
* refactor: update csv processor
* fmt: update format
* chore: fix regex and dissect processor
* chore: fix test
* test: add integration test for http pipeline
* refactor: improve regex pipeline
* refactor: improve required field check
* refactor: rename table_part to table_suffix
* fix: resolve merge issue
---------
Co-authored-by: paomian <xpaomian@gmail.com >
2025-02-08 09:01:54 +00:00
shuiyisong
88c3d331a1
refactor: otlp logs insertion ( #5479 )
...
* chore: add test for selector overlapping
* refactor: simplify otlp logs insertion
* fix: use layered extracted value array
* fix: wrong len
* chore: minor renaming and update
* chore: rename
* fix: clippy
* fix: typos
* chore: update test
* chore: address CR comment & update meter-deps version
2025-02-07 07:21:20 +00:00
shuiyisong
d870987a65
chore: update toolchain to 2024-12-25 ( #5430 )
...
* chore: update toolchain to 2024-12-25
* chore: fix clippy
* feat: update flakes
* chore: remove `rerun-if-changed` for now
* chore: update shadow-rs
* fix: clippy
* chore: update version in DEV_BUILDER_IMAGE_TAG
---------
Co-authored-by: Ning Sun <sunning@greptime.com >
2025-01-24 09:30:54 +00:00
zyy17
1eb4b8ed4f
refactor: support to flatten json object in greptime_identity pipeline ( #5358 )
...
* refactor: support to flatten json object in greptime_identity pipeline
* refactor: add GreptimeIdentityPipelineParams to configure greptime_identity pipeline
* refactor: pass greptime identity pipeline params by one header kv
* refactor: code review
* refactor: make pipeline params more general for all internal pipelines
* chore: remove axum deps from pipeline
* fix: clippy errors
* chore: fix and add test
* test: adopt api change for test client
---------
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
Co-authored-by: Ning Sun <sunng@protonmail.com >
Co-authored-by: Ning Sun <sunning@greptime.com >
2025-01-23 08:50:50 +00:00
Ning Sun
d73815ba84
feat: pipeline dispatch part 1, add definition ( #5359 )
...
* feat: add dispatcher definition
* feat: add dispatcher element in pipelien definition
* fmt: correct format
* test: add negative tests
* fmt: fix format
* refactor: replace consts
* feat: add tostring for dispatcher
* refactor: remove to_string which is actually debug
* Update src/pipeline/src/dispatcher.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2025-01-18 11:47:17 +00:00
yihong
52697a9e66
fix: maybe double free from static str in Snafu ( #5383 )
...
Signed-off-by: yihong0618 <zouzou0208@gmail.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2025-01-17 08:24:26 +00:00
Ruihang Xia
86bd54194a
feat: digest pipeline processor ( #5323 )
...
* feat: basic impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add document
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply code review comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
* follow the naming master
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-01-16 09:46:37 +00:00
Ruihang Xia
bd37e086c2
fix: improve error handling in pipeline and log in log store ( #5357 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-01-14 13:58:30 +00:00
Ning Sun
04708f10aa
feat(pipeline): allow coerce timestamp from integer values ( #5270 )
...
* feat: allow coerce timestamp from integer values
* test: add tests for parsing integer values
2025-01-06 08:53:09 +00:00
