Ning Sun 480b05c590 feat: pipeline dispatcher part 2: execution (#5409)
* fmt: correct format

* test: add negative tests

* feat: Add pipeline dispatching and execution output handling

* refactor: Enhance ingest function to correctly process original data values

custom table names during pipeline execution while optimizing the management of
transformed rows and multiple dispatched pipelines

* refactor: call greptime_identity with intermediate values

* fix: typo

* test: port tests to refactored apis

* refactor: adapt dryrun api call

* refactor: move pipeline execution code to a separated module

* refactor: update otlp pipeline execution path

* fmt: format imports

* fix: compilation

* fix: resolve residual issues

* refactor: address review comments

* chore: use btreemap as pipeline intermediate status trait modify

* refactor: update dispatcher to accept BTreeMap

* refactor: update identity pipeline

* refactor: use new input for pipeline

* chore: wip

* refactor: use updated prepare api

* refactor: improve error and header name

* feat: port flatten to new api

* chore: update pipeline api

* chore: fix transform and some pipeline test

* refactor: reimplement cmcd

* refactor: update csv processor

* fmt: update format

* chore: fix regex and dissect processor

* chore: fix test

* test: add integration test for http pipeline

* refactor: improve regex pipeline

* refactor: improve required field check

* refactor: rename table_part to table_suffix

* fix: resolve merge issue

---------

Co-authored-by: paomian <xpaomian@gmail.com>
2025-02-08 09:01:54 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2025-02-07 11:43:40 +00:00
2023-11-09 10:38:12 +00:00
2023-03-28 19:14:29 +08:00

GreptimeDB Logo

Unified & Cost-Effective Time Series Database for Metrics, Logs, and Events

Introduction

GreptimeDB is an open-source unified & cost-effective time-series database for Metrics, Logs, and Events (also Traces in plan). You can gain real-time insights from Edge to Cloud at Any Scale.

Why GreptimeDB

Our core developers have been building time-series data platforms for years. Based on our best practices, GreptimeDB was born to give you:

  • Unified Processing of Metrics, Logs, and Events

    GreptimeDB unifies time series data processing by treating all data - whether metrics, logs, or events - as timestamped events with context. Users can analyze this data using either SQL or PromQL and leverage stream processing (Flow) to enable continuous aggregation. Read more.

  • Cloud-native Distributed Database

    Built for Kubernetes. GreptimeDB achieves seamless scalability with its cloud-native architecture of separated compute and storage, built on object storage (AWS S3, Azure Blob Storage, etc.) while enabling cross-cloud deployment through a unified data access layer.

  • Performance and Cost-effective

    Written in pure Rust for superior performance and reliability. GreptimeDB features a distributed query engine with intelligent indexing to handle high cardinality data efficiently. Its optimized columnar storage achieves 50x cost efficiency on cloud object storage through advanced compression. Benchmark reports.

  • Cloud-Edge Collaboration

    GreptimeDB seamlessly operates across cloud and edge (ARM/Android/Linux), providing consistent APIs and control plane for unified data management and efficient synchronization. Learn how to run on Android.

  • Multi-protocol Ingestion, SQL & PromQL Ready

    Widely adopted database protocols and APIs, including MySQL, PostgreSQL, InfluxDB, OpenTelemetry, Loki and Prometheus, etc. Effortless Adoption & Seamless Migration. Supported Protocols Overview.

For more detailed info please read Why GreptimeDB.

Try GreptimeDB

1. Live Demo

Try out the features of GreptimeDB right from your browser.

2. GreptimeCloud

Start instantly with a free cluster.

3. Docker Image

To install GreptimeDB locally, the recommended way is via Docker:

docker pull greptime/greptimedb

Start a GreptimeDB container with:

docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb:/tmp/greptimedb" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --rpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Access the dashboard via http://localhost:4000/dashboard.

Read more about Installation on docs.

Getting Started

Build

Check the prerequisite:

  • Rust toolchain (nightly)
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials, including gcc/g++/autoconf and glibc library (eg. libc6-dev on Ubuntu and glibc-devel on Fedora)
  • Python toolchain (optional): Required only if using some test scripts.

Build GreptimeDB binary:

make

Run a standalone server:

cargo run -- standalone start

Tools & Extensions

Kubernetes

Dashboard

SDK

Grafana Dashboard

Our official Grafana dashboard for monitoring GreptimeDB is available at grafana directory.

Project Status

GreptimeDB is currently in Beta. We are targeting GA (General Availability) with v1.0 release by Early 2025.

While in Beta, GreptimeDB is already:

  • Being used in production by early adopters
  • Actively maintained with regular releases, about version number
  • Suitable for testing and evaluation

For production use, we recommend using the latest stable release.

Community

Our core team is thrilled to see you participate in any ways you like. When you are stuck, try to ask for help by filling an issue with a detailed description of what you were trying to do and what went wrong. If you have any questions or if you would like to get involved in our community, please check out:

In addition, you may:

Commercial Support

If you are running GreptimeDB OSS in your organization, we offer additional enterprise add-ons, installation services, training, and consulting. Contact us and we will reach out to you with more detail of our commercial license.

License

GreptimeDB uses the Apache License 2.0 to strike a balance between open contributions and allowing you to use the software however you want.

Contributing

Please refer to contribution guidelines and internal concepts docs for more information.

Acknowledgement

Special thanks to all the contributors who have propelled GreptimeDB forward. For a complete list of contributors, please refer to AUTHOR.md.

Description
Languages
Rust 99.6%