evenyag e697ba975b feat: Implement dedup and filter for vectors (#245)
* feat: Dedup vector

* refactor: Re-export Date/DateTime/Timestamp

* refactor: Named field for ListValueRef::Ref

Use field val instead of tuple for variant ListValueRef::Ref to keep
consistence with ListValueRef::Indexed

* feat: Implement ScalarVector for ListVector

Also implements ScalarVectorBuilder for ListVectorBuilder, Scalar for
ListValue and ScalarRef for ListValueRef

* test: Add tests for ScalarVector implementation of ListVector

* feat: Implement dedup using match_scalar_vector

* refactor: Move dedup func to individual mod

* chore: Update ListValueRef comments

* refactor: Move replicate to VectorOp

Move compute operations to VectorOp trait and acts as an super trait of
Vector. So we could later put dedup/filter methods to VectorOp trait,
avoid to define too many methods in Vector trait.

* refactor: Move scalar bounds to PrimitiveElement

Move Scalar and ScalarRef trait bounds to PrimitiveElement, so for each
native type which implements PrimitiveElement, its PrimitiveVector
always implements ScalarVector, so we could use it as ScalarVector
without adding additional trait bounds

* refactor: Move dedup to VectorOp

Remove compute mod and move dedup logic to operations::dedup

* feat: Implement VectorOp::filter

* test: Move replicate test of primitive to replicate.rs

* test: Add more replicate tests

* test: Add tests for dedup and filter

Also fix NullVector::dedup and ConstantVector::dedup

* style: fix clippy

* chore: Remove unused scalar.rs

* test: Add more tests for VectorOp and fix failed tests

Also fix TimestampVector eq not implemented.

* chore: Address CR comments

* chore: mention vector should be sorted in comment

* refactor: slice the vector directly in replicate_primitive_with_type
2022-09-19 14:05:02 +08:00
2022-09-14 16:06:59 +08:00
2022-07-07 15:54:34 +08:00

GreptimeDB

codecov

GreptimeDB: the next-generation hybrid timeseries/analytics processing database in the cloud.

Getting Started

Prerequisites

To compile GreptimeDB from source, you'll need the following:

  • Rust
  • Protobuf
  • OpenSSL

Rust

The easiest way to install Rust is to use rustup, which will check our rust-toolchain file and install correct Rust version for you.

Protobuf

protoc is required for compiling .proto files. protobuf is available from major package manager on macos and linux distributions. You can find an installation instructions here.

OpenSSL

For Ubuntu:

sudo apt install libssl-dev

For RedHat-based: Fedora, Oracle Linux, etc:

sudo dnf install openssl-devel

For macOS:

brew install openssl

Build the Docker Image

docker build --network host -f docker/Dockerfile -t greptimedb .

Usage

Start Datanode

// Start datanode with default options.
cargo run -- datanode start

OR

// Start datanode with `http-addr` option.
cargo run -- datanode start --http-addr=0.0.0.0:9999

OR

// Start datanode with `log-dir` and `log-level` options.
cargo run -- --log-dir=logs --log-level=debug datanode start

Start datanode with config file:

cargo run -- --log-dir=logs --log-level=debug datanode start -c ./config/datanode.example.toml

Start datanode by runing docker container:

docker run -p 3000:3000 \
-p 3001:3001 \
-p 3306:3306 \
greptimedb

Start Frontend

Frontend should connect to Datanode, so Datanode must have been started at first!

// Connects to local Datanode at its default GRPC port: 3001

// Start Frontend with default options.
cargo run -- frontend start

OR

// Start Frontend with `mysql-addr` option.
cargo run -- frontend start --mysql-addr=0.0.0.0:9999

OR

// Start datanode with `log-dir` and `log-level` options.
cargo run -- --log-dir=logs --log-level=debug frontend start

Start datanode with config file:

cargo run -- --log-dir=logs --log-level=debug frontend start -c ./config/frontend.example.toml

SQL Operations

  1. Connecting DB by mysql client:

    # The datanode listen on port 3306 by default.
    mysql -h 127.0.0.1 -P 3306
    
  2. Create table:

    CREATE TABLE monitor (
      host STRING,
      ts TIMESTAMP,
      cpu DOUBLE DEFAULT 0,
      memory DOUBLE,
      TIME INDEX (ts),
      PRIMARY KEY(ts,host)) ENGINE=mito WITH(regions=1);
    
  3. Insert data:

    INSERT INTO monitor(host, cpu, memory, ts) VALUES ('host1', 66.6, 1024, 1660897955);
    INSERT INTO monitor(host, cpu, memory, ts) VALUES ('host2', 77.7, 2048, 1660897956);
    INSERT INTO monitor(host, cpu, memory, ts) VALUES ('host3', 88.8, 4096, 1660897957);
    
  4. Query data:

    mysql> SELECT * FROM monitor;
    +-------+------------+------+--------+
    | host  | ts         | cpu  | memory |
    +-------+------------+------+--------+
    | host1 | 1660897955 | 66.6 |   1024 |
    | host2 | 1660897956 | 77.7 |   2048 |
    | host3 | 1660897957 | 88.8 |   4096 |
    +-------+------------+------+--------+
    3 rows in set (0.01 sec)
    

    You can delete your data by removing /tmp/greptimedb.

Contribute

  1. Install rust
  2. Install pre-commit for run hooks on every commit automatically such as cargo fmt etc.
$ pip install pre-commit

or

$ brew install pre-commit
$
  1. Install the git hook scripts:
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

$ pre-commit install --hook-type commit-msg
pre-commit installed at .git/hooks/commit-msg

$ pre-commit install --hook-type pre-push
pre-commit installed at .git/hooks/pre-pus

now pre-commit will run automatically on git commit.

  1. Check out branch from develop and make your contribution. Follow the style guide. Create a PR when you are ready, feel free and have fun!
Description
Languages
Rust 99.6%