Lei, HUANG
e328c7067c
chore: udapte Rust toolchain to 2024-10-19 ( #4857 )
...
* update rust toolchain
* change toolchain to 2024-10-17
* fix: clippy
* fix: ut
* bump shadow-rs
* fix: use nightly-2024-10-19
* fix: clippy
* chore/udapte-toolchain-2024-10-17: Update DEV_BUILDER_IMAGE_TAG to 2024-10-19-a5c00e85-20241024184445 in Makefile
2024-10-25 00:23:32 +00:00
localhost
fcde0a4874
feat: Add functionality to the Opentelemetry write interface to extract fields from attr to top-level data. ( #4859 )
...
* chore: add otlp select
* chore: change otlp select
* chore: remove json path
* chore: format toml
* chore: change opentelemetry extract keys header name
* chore: add some doc and remove useless code and lib
* chore: make clippy happy
* chore: fix by pr comment
* chore: fix by pr comment
* chore: opentelemetry logs select key change some type default semantic type
2024-10-24 05:55:57 +00:00
localhost
4622412dfe
feat: add API to write OpenTelemetry logs to GreptimeDB ( #4755 )
...
* chore: otlp logs api
* feat: add API to write OpenTelemetry logs to GreptimeDB
* chore: fix test data schema error
* chore: modify the underlying data structure of the pipeline value map type from hashmap to btremap to keep key order
* chore: fix by pr comment
* chore: resolve conflicts and add some test
* chore: remove useless error
* chore: change otlp header name
* chore: fmt code
* chore: fix integration test for otlp log write api
* chore: fix by pr comment
* chore: set otlp body with fulltext default
2024-10-16 04:36:08 +00:00
shuiyisong
5f0a83b2b1
fix: ts conversion during transform phase ( #4790 )
...
* fix: allow ts conversion during transform phase
* chore: replace `unimplemented` with snafu
2024-10-08 17:54:44 +00:00
localhost
71a66d15f7
chore: add json write ( #4744 )
...
* chore: add json write
* chore: add test for write json log api
* chore: enhancement of Error Handling
* chore: fix by pr comment
* chore: fix by pr comment
* chore: enhancement of error content and add some doc
2024-10-08 12:11:09 +00:00
localhost
627a326273
refactor: Change the error type in the pipeline crate from String to Error ( #4763 )
...
* chore: in process
* chore: change pipeline crate error type
* chore: improve event error
* chore: fix by pr comment
* chore: use snafu context replace ok_or_else
* refactor: update snafu usage
---------
Co-authored-by: Ning Sun <sunng@protonmail.com >
2024-09-25 19:32:34 +00:00
Ruihang Xia
f5cf25b0db
refactor: remove DfPlan wrapper ( #4733 )
...
* refactor: remove DfPlan wrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test assertion
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-19 12:29:33 +00:00
shuiyisong
52d627e37d
chore: add log ingest interceptor ( #4734 )
...
* chore: add log ingest interceptor
* chore: rename
* chore: update interceptor signature
2024-09-19 05:14:47 +00:00
localhost
36b1bafbf0
fix: pipeline dissert error is returned directly to the user, instead of printing a warn log ( #4709 )
...
* fix: pipeline dissert error is returned directly to the user, instead of printing a warn log
* chore: add more test for pipeline
2024-09-12 18:21:05 +00:00
JohnsonLee
a8477e4142
fix: table resolving logic related to pg_catalog ( #4580 )
...
* fix: table resolving logic related to pg_catalog
refer to
https://github.com/GreptimeTeam/greptimedb/issues/3560#issuecomment-2287794348
and #4543
* refactor: remove CatalogProtocol type
* fix: sqlness
* fix: forbid create database pg_catalog with mysql client
* refactor: use QueryContext as arguments rather than Channel
* refactor: pass None as default behaviour in information_schema
* test: fix test
2024-09-09 00:47:59 +00:00
localhost
67d95d2088
refactor!: add processor builder and transform buidler ( #4571 )
...
* chore: add processor builder and transform buidler
* chore: in process
* chore: intermediate state from hashmap to vector in pipeline
* chore: remove useless code and rename some struct
* chore: fix typos
* chore: format code
* chore: add error handling and optimize code readability
* chore: fix typos
* chore: remove useless code
* chore: add some doc
* chore: fix by pr commit
* chore: remove useless code and change struct name
* chore: modify the location of the find_key_index function.
2024-09-06 07:51:08 +00:00
localhost
bf16422cee
fix: pipeline prepare loop break detects a conditional error ( #4593 )
2024-08-21 06:20:09 +00:00
localhost
202c730363
perf: Optimizing pipeline performance ( #4390 )
...
* chore: improve pipeline performance
* chore: use arc to improve time type
* chore: improve pipeline coerce
* chore: add vec refactor
* chore: add vec pp
* chore: improve pipeline
* inprocess
* chore: set log ingester use new pipeline
* chore: fix some error by pr comment
* chore: fix typo
* chore: use enum_dispatch to simplify code
* chore: some minor fix
* chore: format code
* chore: update by pr comment
* chore: fix typo
* chore: make clippy happy
* chore: fix by pr comment
* chore: remove epoch and date process add new timestamp process
* chore: add more test for pipeline
* chore: restore epoch and date processor
* chore: compatibility issue
* chore: fix by pr comment
* chore: move the evaluation out of the loop
* chore: fix by pr comment
* chore: fix dissect output key filter
* chore: fix transform output greptime value has order error
* chore: keep pipeline transform output order
* chore: revert tests
* chore: simplify pipeline prepare implementation
* chore: add test for timestamp pipelin processor
* chore: make clippy happy
* chore: replace is_some check to match
---------
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
2024-08-13 11:32:04 +00:00
shuiyisong
3b701d8f5e
test: more on processors ( #4493 )
...
* test: add date test
* test: add epoch test
* test: add letter test and complete some others
* test: add urlencoding test
* chore: typo
2024-08-04 08:29:31 +00:00
shuiyisong
8ab6136d1c
chore: support pattern as pipeline key name ( #4368 )
...
* chore: add pattern to processor key name
* fix: typo
* refactor: test
2024-07-18 03:32:26 +00:00
shuiyisong
67dfdd6c61
feat: support text/plain format for log ingestion ( #4300 )
...
* feat: support text/plain format of log input
* refactor: pipeline query and delete using dataframe api
* chore: minor refactor
* refactor: skip jsonify when processing plan/text
* refactor: support array(string) as pipeline engine input
2024-07-12 09:17:15 +00:00
Zhenchi
00308218b3
feat(fulltext_index): allow enable full-text index in SQL and gRPC way ( #4310 )
...
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typo
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: test_fulltext_intm_path
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: explicitly build column options
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: fix error msg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-08 20:18:48 +00:00
shuiyisong
849e0b9249
feat: delete pipeline ( #4156 )
...
* feat: add delete for pipeline
* chore: remove unused code
* refactor: delete pipeline
* chore: add pipeline management api metrics
* chore: minor cr issues
* chore: add unit test
* chore: fix cr issue
* fix: test
* chore: add `GreptimedbManageResponse`
* fix: typo
* fix: typo
2024-07-05 06:23:49 +00:00
shuiyisong
69bb7ded6a
fix: enable space string in yaml value ( #4286 )
...
* fix: enable space string in yaml value
* fix: typo
2024-07-05 03:39:26 +00:00
Ning Sun
11cf9c827e
feat: dbeaver mysql compatibility, use statement and information_schema.tables ( #4218 )
...
* feat: add more placeholder field in information_schema.tables
* feat: make schema modifiable for use statement
* chore: add todo items
* fix: resolve lint issues after data type changes
* chore: update sqlness results
* refactor: patch for select database is no longer needed
* test: align tests and data types
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fix: use canonicalize_identifier for database name
* feat: add all columns for information_schema.tables
* test: remove vairables from sqlness results
* feat: add to_string impl for table options
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-07-03 01:31:13 +00:00
localhost
48a0f39b19
chore: enhance add pipeline http api return data ( #4167 )
...
* chore: enhance add pipeline http api return data
* chore: replaceing hard code header value
2024-06-20 02:19:31 +00:00
shuiyisong
cb657ae51e
feat(pipeline): join processor ( #4158 )
...
* feat: add join processor
* test: add join simple test
* chore: fix header
* chore: update commit
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* test: add more join test
* chore: fix lint
* chore: update comment
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-18 05:00:34 +00:00
yuanbohan
0fc18b6865
feat(pipeline): gsub prosessor ( #4121 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* gsub prosessor
* chore: add test
* chore: update commit
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: paomian <xpaomian@gmail.com >
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-17 07:57:47 +00:00
yuanbohan
0aceebf0a3
feat(pipeline): transform support on_failure ( #4123 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* transform on_failure
* chore: add test
* chore: move test to a separate file
* chore: add comment
---------
Co-authored-by: paomian <xpaomian@gmail.com >
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
2024-06-17 06:56:31 +00:00
localhost
01e3a24cf7
feat: log ingestion support ( #4014 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* chore: fix format issue
* fix: make the LogicalPlan of a query pipeline sorted in desc order
* chore: remove some debug log
* chore: replacing hashmap cache with moak
* chore: fix by pr commit
* chore: fix toml format issue
* chore: update Cargo.lock
* chore: fix by pr commit
* chore: fix some issue by pr commit
* chore: add more doc for pipeline version
---------
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
2024-06-14 17:03:30 +00:00
shuiyisong
2ade511f26
feat: introduce pipeline crate ( #4109 )
...
* chore: introduce pipeline crate
* chore: fix typo
2024-06-05 17:23:25 +00:00
