mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-17 00:07:00 +00:00
Merge remote-tracking branch 'upstream/main' into feat/adjust-row-group
Signed-off-by: evenyag <realevenyag@gmail.com> # Conflicts: # src/mito2/src/engine/flush_test.rs # src/store-api/src/region_request.rs
This commit is contained in:
@@ -0,0 +1,368 @@
|
||||
---
|
||||
name: greptimedb-development-docker-image
|
||||
description: Builds a development-only GreptimeDB Docker image from a local debug binary for local-cluster testing, and optionally pushes it to a development registry. Use when the user asks to package, build, tag, publish, or cross-build a non-release GreptimeDB or GreptimeDB Enterprise image for debugging.
|
||||
compatibility: Requires Docker with Buildx for cross-platform builds; Podman is supported for native-platform builds.
|
||||
metadata:
|
||||
protocols: docker buildx podman
|
||||
platforms: linux-amd64 linux-arm64
|
||||
---
|
||||
|
||||
# GreptimeDB Development Docker Image
|
||||
|
||||
## Goal
|
||||
|
||||
Package a locally built GreptimeDB binary into a **development-only** Docker
|
||||
image for debugging and local-cluster testing, optionally push it to a
|
||||
development registry, and retain the non-secret image settings in `.env` for
|
||||
the next build. It is **not** a release-image workflow and must not be used to
|
||||
publish a production or release artifact.
|
||||
|
||||
This skill follows the documented development-image build procedure:
|
||||
|
||||
- Build `greptime` with Cargo's `nightly` profile.
|
||||
- Copy the target binary into the Docker build context as `./greptime`.
|
||||
- Build the supplied `Dockerfile`, which uses Ubuntu 24.04 only as the runtime
|
||||
base image and exposes the `greptime` binary as its entrypoint.
|
||||
|
||||
## Inputs
|
||||
|
||||
Collect or discover the following before any build or push:
|
||||
|
||||
| Input | Discovery and rule |
|
||||
| --- | --- |
|
||||
| Source repository | Treat the directory in which this skill is invoked as the default workspace. Include it as an editable field in the batch configuration; do not ask a separate confirmation. Do not assume open-source versus Enterprise. |
|
||||
| Edition and binary | Default to `greptime`; use the binary name the user requests for Enterprise builds. The copied Docker-context filename must always be `greptime`. |
|
||||
| Cargo profile and binary source | Include this in the batch configuration. Default to rebuilding `nightly`; reuse an existing binary only after the user explicitly accepts that its freshness is unverified. |
|
||||
| Target platform | Preselect `linux/amd64` unless Docker's server is `linux/arm64`, then preselect `linux/arm64`. Let the user override it in the batch configuration. Ubuntu 24.04 is the runtime base image, not a target-platform choice. Each image has exactly one target platform, but it may differ from the host platform. |
|
||||
| Build mode | Ask whether the user wants a locally loadable debug image or a registry push. |
|
||||
| Registry/repository | Read `IMAGE_REGISTRY` and `IMAGE_REPOSITORY` only from the selected workspace's `.env` and prefill them in the batch configuration. Example: `registry.example.com/team` + `greptimedb-dev`. |
|
||||
| Tag | Read `IMAGE_TAG` only from the selected workspace's `.env` and prefill it in the batch configuration. When the tag exists in the selected registry, preselect an incremented version. |
|
||||
|
||||
Use the image reference `${IMAGE_REGISTRY}/${IMAGE_REPOSITORY}:${IMAGE_TAG}`.
|
||||
If the registry is intentionally empty, omit its slash rather than producing a
|
||||
leading slash.
|
||||
|
||||
## Interactive Workflow
|
||||
|
||||
Use the platform's interactive prompt component for every question, selection,
|
||||
and confirmation. Do not ask an open-ended text question when a single-choice,
|
||||
multi-select, or confirmation component can represent the decision. If the
|
||||
platform does not provide an interactive component, use the equivalent numbered
|
||||
or lettered prompt below and wait for input before continuing.
|
||||
|
||||
### Batch configuration
|
||||
|
||||
Minimize user round trips: run the collector first against the invocation
|
||||
directory, then use one interactive form or batched prompt to collect workspace,
|
||||
edition/binary, profile/reuse-or-rebuild choice, target platform, build mode,
|
||||
registry/repository, tag, and whether to inspect the configured registry tag.
|
||||
Use the invocation directory as the editable workspace default. Prefill
|
||||
collector values and mark recommended defaults. If the user changes workspace,
|
||||
rerun the collector for that workspace without asking another configuration
|
||||
question. Only ask a follow-up if information is missing or invalid, or a safety
|
||||
gate is required. Treat unchanged prefilled values as accepted.
|
||||
|
||||
Keep separate confirmation components only for elevated privileges: `sudo`,
|
||||
package installation, and privileged QEMU setup. Use one final confirmation for
|
||||
all non-privileged selected work.
|
||||
|
||||
Always offer **Cancel** for an action that can write state, build, push, or
|
||||
require elevated privileges. Display the relevant preview before the user
|
||||
confirms it: source/binary path, `file` architecture result, platform(s), image
|
||||
reference, build mode, and non-secret `.env` values.
|
||||
|
||||
### Batch configuration fields
|
||||
|
||||
Use single-choice controls inside the one batch form for mutually exclusive
|
||||
choices:
|
||||
|
||||
| Decision | Required choices |
|
||||
| --- | --- |
|
||||
| Cargo output | Rebuild `nightly` (recommended), reuse an existing binary only with a freshness-unverified acknowledgement, choose a profile/binary |
|
||||
| Target platform | `linux/amd64` (default unless Docker server is arm64), `linux/arm64` (default when Docker server is arm64). The selected image may be cross-built with Buildx. |
|
||||
| Build mode | Load locally (recommended), push to development registry |
|
||||
| Registry/repository | Use `.env` defaults, enter new values |
|
||||
| Tag | Use proposed increment when the configured tag exists, enter a version |
|
||||
|
||||
When push is selected and the image configuration is complete, automatically
|
||||
run the read-only registry-tag check. It may return `unknown`; do not treat that
|
||||
as a missing tag.
|
||||
|
||||
Use one final confirmation for all selected non-privileged operations:
|
||||
|
||||
- persist changed non-secret image settings to `.env`;
|
||||
- run the selected Cargo build or reuse the resolved binary;
|
||||
- prepare the Docker context;
|
||||
- build, verify, and, when selected, push the image.
|
||||
|
||||
The final confirmation must clearly state: **“This creates a development and
|
||||
local-cluster test image, not a release or production artifact.”**
|
||||
|
||||
For a real multi-select only, use lettered choices (`[A]`, `[B]`) and accept
|
||||
`all`, `none`, or `cancel`; otherwise use single-choice components.
|
||||
|
||||
### Preflight context collection
|
||||
|
||||
Before asking the profile, registry, or tag questions, run the bundled,
|
||||
read-only collector. It works on macOS and Linux, does not invoke `cargo build`,
|
||||
and emits JSON that can be shown or summarized to the user:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/collect_context.py \
|
||||
--source <source-checkout> \
|
||||
--bin greptime \
|
||||
--profile nightly \
|
||||
[--target <rust-target>]
|
||||
```
|
||||
|
||||
Use its report to identify:
|
||||
|
||||
- missing `IMAGE_REGISTRY`, `IMAGE_REPOSITORY`, or `IMAGE_TAG` values in the
|
||||
selected workspace's `.env`;
|
||||
- Cargo binary targets available in the workspace and whether the requested
|
||||
`--bin` exists;
|
||||
- the resolved Cargo target directory, expected profile output path, and whether
|
||||
that binary already exists; and
|
||||
- macOS/Linux host architecture plus Docker's server platform when Docker is
|
||||
available.
|
||||
|
||||
When push is selected and image configuration is complete, rerun the collector
|
||||
with `--check-registry-tag` and all selected image values. It uses the selected
|
||||
engine's read-only manifest inspection and returns `exists` or `unknown`;
|
||||
`unknown` includes an unavailable registry, an absent tag, or missing
|
||||
authentication and must not be treated as an absent tag.
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/collect_context.py \
|
||||
--source <source-checkout> \
|
||||
--bin <binary> \
|
||||
--profile <profile> \
|
||||
--engine <docker|podman> \
|
||||
--registry <registry> \
|
||||
--repository <repository> \
|
||||
--tag <tag> \
|
||||
--check-registry-tag
|
||||
```
|
||||
|
||||
When the status is `exists` and the tag ends in a number, preselect the
|
||||
collector's `candidate_tag` as the next development tag. The user must still
|
||||
confirm it before `.env` is updated or an image is built. Never use automatic
|
||||
incrementing as permission to overwrite or push an existing image.
|
||||
|
||||
If Cargo metadata or the requested binary target is unavailable, stop before
|
||||
building and ask the user to correct the source path or binary target.
|
||||
|
||||
The profile/binary choice is a field in the batch configuration, not a separate
|
||||
question. Its default is the existing `nightly` output when present:
|
||||
|
||||
```text
|
||||
Question: Which Cargo build output should package this image?
|
||||
Options:
|
||||
- Rebuild nightly (Recommended): run Cargo with `--profile nightly`, then use its output.
|
||||
- Reuse nightly output: use the existing output only after showing its path, modification time, size, and matching platform; freshness is unverified.
|
||||
- Choose profile or binary: provide a different Cargo profile or an explicit binary path.
|
||||
```
|
||||
|
||||
Do not infer that a binary under `target/debug`, `target/release`, or another
|
||||
profile is suitable. For an existing binary, show its resolved path and run
|
||||
`file` before asking for final build confirmation.
|
||||
|
||||
## Environment State
|
||||
|
||||
The image configuration lives only in the selected workspace's `.env`. The
|
||||
collector reads only the following keys and never reads or displays other `.env`
|
||||
values:
|
||||
|
||||
```dotenv
|
||||
IMAGE_REGISTRY=registry.example.com/team
|
||||
IMAGE_REPOSITORY=greptimedb-dev
|
||||
IMAGE_TAG=dev-001
|
||||
```
|
||||
|
||||
1. Read `.env` if it exists. Ignore blank lines and `#` comments.
|
||||
2. Show the discovered registry, repository, and proposed tag to the user.
|
||||
3. Include registry/repository and tag in the batch configuration even when
|
||||
values already exist; saved values are defaults, not authorization to reuse
|
||||
them.
|
||||
4. Compare the selected configuration with the three current managed values. If
|
||||
all match, skip `.env` confirmation and do not update the file. Otherwise,
|
||||
preview only the changed `IMAGE_*` values, request confirmation, and update
|
||||
the file while preserving unrelated entries and comments.
|
||||
5. Never put registry credentials, access tokens, passwords, or `docker login`
|
||||
output in `.env`, build commands, logs, or responses. Ask the user to log in
|
||||
themselves if a push needs authentication.
|
||||
|
||||
For a repeated build, offer an incremented tag before asking. Use
|
||||
`scripts/next_image_tag.py --tag <existing-tag>` to calculate it. It increments
|
||||
the final numeric component while preserving zero padding:
|
||||
|
||||
```text
|
||||
weny-2025-0715-01 -> weny-2025-0715-02
|
||||
v0.1.4 -> v0.1.5
|
||||
debug-009 -> debug-010
|
||||
```
|
||||
|
||||
If the saved tag has no trailing number, ask the user for the next tag rather
|
||||
than inventing a versioning convention. Do not overwrite an existing image tag
|
||||
without explicit user confirmation.
|
||||
|
||||
Only if a managed value is missing or differs, persist the values with the
|
||||
bundled helper after confirmation instead of hand-editing the file:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/update_image_env.py \
|
||||
--env <source-checkout>/.env \
|
||||
--registry <registry> \
|
||||
--repository <repository> \
|
||||
--tag <tag>
|
||||
```
|
||||
|
||||
## Build Procedure
|
||||
|
||||
### 1. Inspect the host and container tooling
|
||||
|
||||
Run these checks and report the selected path:
|
||||
|
||||
```bash
|
||||
uname -s
|
||||
uname -m
|
||||
docker version --format '{{.Server.Os}}/{{.Server.Arch}}'
|
||||
docker buildx version
|
||||
docker buildx inspect --bootstrap
|
||||
```
|
||||
|
||||
For a native-platform build, Podman can be used if Docker is unavailable. For
|
||||
any non-native request, require Docker Buildx. Do not silently fall back to a
|
||||
native build when the requested image platform differs.
|
||||
|
||||
If the requested target is unavailable, stop and ask the user to configure a
|
||||
Buildx builder and cross-compilation toolchain externally. Do not automate
|
||||
privileged QEMU or binfmt setup, and never run an unpinned privileged image.
|
||||
|
||||
### 2. Build the executable for the image platform
|
||||
|
||||
From the source checkout, build the requested binary. For open-source amd64:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> --package cmd --bin greptime --profile nightly
|
||||
```
|
||||
|
||||
For open-source arm64:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> \
|
||||
--package cmd \
|
||||
--bin greptime \
|
||||
--profile nightly \
|
||||
--target aarch64-unknown-linux-gnu
|
||||
```
|
||||
|
||||
For Enterprise, use the user-provided executable target, for example:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> --bin greptime-ent-cloud --profile nightly
|
||||
```
|
||||
|
||||
For a user-selected profile, replace `nightly` in the helper call and resolve
|
||||
the binary under `target/<profile>/<binary>` (or
|
||||
`target/<rust-target>/<profile>/<binary>` for cross-compilation). Reuse the
|
||||
existing target file only when the interactive profile question selected reuse;
|
||||
otherwise invoke the helper to rebuild it.
|
||||
|
||||
Do not use a binary compiled for the host architecture in an image intended for
|
||||
another architecture. Determine the binary path from the Cargo target and
|
||||
profile, then copy it into the Docker context as `greptime`:
|
||||
|
||||
```bash
|
||||
cp <source-target-binary> <build-context>/greptime
|
||||
```
|
||||
|
||||
Verify it before building the image with `file <build-context>/greptime`; its
|
||||
reported architecture must match the requested target platform.
|
||||
|
||||
Create a fresh isolated build context (never use the repository root) and use
|
||||
the bundled preparation script. The helper rejects an existing Dockerfile or
|
||||
binary output to prevent accidental context reuse:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/prepare_context.sh \
|
||||
--context "$(mktemp -d)" \
|
||||
--binary <source-target-binary> \
|
||||
--dockerfile <skill-dir>/assets/Dockerfile \
|
||||
--platform <platform>
|
||||
```
|
||||
|
||||
### 3. Build or push the image
|
||||
|
||||
Run the command only after the user confirms the final image reference and
|
||||
whether it should be pushed.
|
||||
|
||||
Use the bundled build helper rather than spelling out individual Docker or
|
||||
Podman commands. It follows the existing scripts' Docker-first/Podman-fallback
|
||||
behavior, uses Buildx for a non-native Docker request, and accepts exactly one
|
||||
target platform per image:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_image.sh \
|
||||
--context <build-context> \
|
||||
--image <image-reference> \
|
||||
--platform <platform> \
|
||||
--mode local|push
|
||||
```
|
||||
|
||||
`--mode local` builds a native image locally or uses `docker buildx --load` for
|
||||
a non-native image. `--mode push` pushes the selected single-platform image.
|
||||
The helper never pushes in `local` mode.
|
||||
|
||||
This is intentionally a runtime image built from a precompiled binary, not a
|
||||
multi-stage Dockerfile. Ubuntu 24.04 is the runtime base image only; select one
|
||||
target platform separately as `linux/amd64` or `linux/arm64`. Only introduce a
|
||||
multi-stage Dockerfile if the user asks to compile within Docker or the local
|
||||
Rust toolchain cannot build the required target. Keep the final Ubuntu 24.04
|
||||
runtime stage and existing entrypoint unless the user asks to change runtime
|
||||
behavior.
|
||||
|
||||
At the final confirmation, repeat that the selected reference is a
|
||||
**development/local-cluster test image**, not a release artifact. If the user
|
||||
requests a release or production image, stop and direct them to the release
|
||||
process instead.
|
||||
|
||||
### 4. Verify the result
|
||||
|
||||
For a local image, use the selected engine to inspect its architecture and
|
||||
start it with `--help`:
|
||||
|
||||
```bash
|
||||
docker image inspect <image-reference> --format '{{.Os}}/{{.Architecture}}'
|
||||
docker run --rm <image-reference> --help
|
||||
```
|
||||
|
||||
For Podman, use `podman image inspect <image-reference>` and
|
||||
`podman run --rm <image-reference> --help`. For a pushed Podman image, use
|
||||
`podman manifest inspect <image-reference>`.
|
||||
|
||||
For a pushed image, inspect its remote manifest:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect <image-reference>
|
||||
```
|
||||
|
||||
Report the final image reference, target platform, source binary path, and
|
||||
whether the image was loaded locally or pushed.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- This skill is only for development and local-cluster testing. Never present
|
||||
its image as a production, release, or officially published artifact.
|
||||
- Never push by default. Local debug builds should use a local tag and load the
|
||||
image unless the user explicitly requests a push.
|
||||
- Do not run `docker login`, privileged QEMU setup, package installation, or
|
||||
overwrite a tag without confirmation.
|
||||
- Do not delete existing images, builders, binaries, or `.env` entries as part
|
||||
of this workflow.
|
||||
- Stop and explain if Docker's server is unavailable, Buildx cannot support the
|
||||
requested platform, or the copied binary architecture does not match the
|
||||
image platform.
|
||||
@@ -0,0 +1,15 @@
|
||||
FROM ubuntu:24.04
|
||||
|
||||
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
curl \
|
||||
unzip \
|
||||
mysql-client \
|
||||
wget \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY ./greptime /greptime/bin/
|
||||
|
||||
ENV PATH=/greptime/bin/:$PATH
|
||||
|
||||
ENTRYPOINT ["greptime"]
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Validate that a binary is an executable ELF for a requested Linux platform."""
|
||||
|
||||
import argparse
|
||||
import stat
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
PLATFORM_MACHINES = {"linux/amd64": 0x3E, "linux/arm64": 0xB7}
|
||||
|
||||
|
||||
def validate_platform(platform: str) -> int:
|
||||
try:
|
||||
return PLATFORM_MACHINES[platform]
|
||||
except KeyError as error:
|
||||
raise ValueError("platform must be linux/amd64 or linux/arm64") from error
|
||||
|
||||
|
||||
def inspect_elf(path: Path) -> int:
|
||||
executable_bits = stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
|
||||
if path.is_symlink() or not path.is_file() or not (path.stat().st_mode & executable_bits):
|
||||
raise ValueError("binary must be an executable regular file")
|
||||
try:
|
||||
with path.open("rb") as handle:
|
||||
header = handle.read(64)
|
||||
except OSError as error:
|
||||
raise ValueError(f"cannot read binary: {error}") from error
|
||||
if len(header) != 64 or header[:4] != b"\x7fELF":
|
||||
raise ValueError("binary is not ELF")
|
||||
if header[4] != 2 or header[5] != 1 or header[6] != 1:
|
||||
raise ValueError("binary must be a 64-bit little-endian ELF")
|
||||
if header[7] not in (0, 3):
|
||||
raise ValueError("binary must use the System V or Linux ELF ABI")
|
||||
elf_type, machine, version = struct.unpack_from("<HHI", header, 16)
|
||||
header_size = struct.unpack_from("<H", header, 52)[0]
|
||||
if elf_type not in (2, 3) or version != 1 or header_size != 64:
|
||||
raise ValueError("binary has an unsupported ELF executable header")
|
||||
return machine
|
||||
|
||||
|
||||
def validate_binary(path: Path, platform: str) -> None:
|
||||
expected = validate_platform(platform)
|
||||
actual = inspect_elf(path)
|
||||
if actual != expected:
|
||||
raise ValueError("binary architecture does not match requested platform")
|
||||
|
||||
|
||||
def main(argv: Optional[list[str]] = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--binary", required=True, type=Path)
|
||||
parser.add_argument("--platform", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
validate_binary(args.binary, args.platform)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --source PATH --bin NAME --profile PROFILE [--target RUST_TARGET] [--package PACKAGE] [--features FEATURES]\n' "$0" >&2
|
||||
}
|
||||
|
||||
source_dir=""
|
||||
binary_name=""
|
||||
profile=""
|
||||
target=""
|
||||
package=""
|
||||
features=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--source|--bin|--profile|--target|--package|--features)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
case "$1" in
|
||||
--source) source_dir="$2" ;;
|
||||
--bin) binary_name="$2" ;;
|
||||
--profile) profile="$2" ;;
|
||||
--target) target="$2" ;;
|
||||
--package) package="$2" ;;
|
||||
--features) features="$2" ;;
|
||||
esac
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$source_dir" || -z "$binary_name" || -z "$profile" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ! -f "$source_dir/Cargo.toml" ]]; then
|
||||
printf 'error: not a Cargo workspace: %s\n' "$source_dir" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
command=(cargo build --locked --bin "$binary_name" --profile "$profile")
|
||||
if [[ -n "$package" ]]; then
|
||||
command+=(--package "$package")
|
||||
fi
|
||||
if [[ -n "$features" ]]; then
|
||||
command+=(--features "$features")
|
||||
fi
|
||||
if [[ -n "$target" ]]; then
|
||||
command+=(--target "$target")
|
||||
fi
|
||||
|
||||
printf 'building %s with profile %s\n' "$binary_name" "$profile"
|
||||
(
|
||||
cd "$source_dir"
|
||||
exec "${command[@]}"
|
||||
)
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --context PATH --image REFERENCE --platform PLATFORM --mode local|push [--dockerfile PATH] [--engine auto|docker|podman] [--allow-local-tag-overwrite]\n' "$0" >&2
|
||||
}
|
||||
|
||||
context=""
|
||||
image=""
|
||||
platform=""
|
||||
mode=""
|
||||
dockerfile=""
|
||||
engine="auto"
|
||||
allow_local_tag_overwrite=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--context|--image|--platform|--mode|--dockerfile|--engine)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
case "$1" in
|
||||
--context) context="$2" ;;
|
||||
--image) image="$2" ;;
|
||||
--platform) platform="$2" ;;
|
||||
--mode) mode="$2" ;;
|
||||
--dockerfile) dockerfile="$2" ;;
|
||||
--engine) engine="$2" ;;
|
||||
esac
|
||||
shift 2
|
||||
;;
|
||||
--allow-local-tag-overwrite)
|
||||
allow_local_tag_overwrite=true
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$context" || -z "$image" || -z "$platform" || -z "$mode" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "$mode" != "local" && "$mode" != "push" ]]; then
|
||||
printf 'error: --mode must be local or push\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "$engine" != "auto" && "$engine" != "docker" && "$engine" != "podman" ]]; then
|
||||
printf 'error: --engine must be auto, docker, or podman\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
dockerfile="${dockerfile:-$context/Dockerfile}"
|
||||
if [[ ! -f "$dockerfile" ]]; then
|
||||
printf 'error: Dockerfile does not exist: %s\n' "$dockerfile" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$platform" == *,* ]]; then
|
||||
printf 'error: only one target platform is supported per image build\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
case "$platform" in
|
||||
linux/amd64|linux/arm64) ;;
|
||||
*)
|
||||
printf 'error: unsupported platform: %s\n' "$platform" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
python3 "$script_dir/binary_platform.py" --binary "$context/greptime" --platform "$platform"
|
||||
|
||||
check_collision() {
|
||||
local runtime="$1"
|
||||
if [[ "$allow_local_tag_overwrite" == true ]]; then
|
||||
return 0
|
||||
fi
|
||||
if [[ "$runtime" == docker ]] && docker image inspect "$image" >/dev/null 2>&1; then
|
||||
printf 'error: local image tag exists: %s (pass --allow-local-tag-overwrite to replace it)\n' "$image" >&2
|
||||
exit 3
|
||||
fi
|
||||
if [[ "$runtime" == podman ]] && podman image exists "$image"; then
|
||||
printf 'error: local image tag exists: %s (pass --allow-local-tag-overwrite to replace it)\n' "$image" >&2
|
||||
exit 3
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "$engine" != podman ]] && command -v docker >/dev/null 2>&1 && server_platform="$(docker version --format '{{.Server.Os}}/{{.Server.Arch}}' 2>/dev/null)"; then
|
||||
if [[ "$platform" == "$server_platform" ]]; then
|
||||
check_collision docker
|
||||
docker build --platform "$platform" -f "$dockerfile" -t "$image" "$context"
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec docker push "$image"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec docker buildx build --platform "$platform" --push -f "$dockerfile" -t "$image" "$context"
|
||||
fi
|
||||
check_collision docker
|
||||
exec docker buildx build --platform "$platform" --load -f "$dockerfile" -t "$image" "$context"
|
||||
fi
|
||||
|
||||
if [[ "$engine" != docker ]] && command -v podman >/dev/null 2>&1; then
|
||||
native_platform="$(podman info --format '{{.Host.Os}}/{{.Host.Arch}}' 2>/dev/null || true)"
|
||||
case "$native_platform" in
|
||||
linux/amd64|linux/arm64) ;;
|
||||
*)
|
||||
printf 'error: Podman is unavailable or has unsupported platform: %s\n' "$native_platform" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
if [[ "$platform" != "$native_platform" ]]; then
|
||||
printf 'error: Docker Buildx is required for non-native platform %s\n' "$platform" >&2
|
||||
exit 1
|
||||
fi
|
||||
check_collision podman
|
||||
podman build -f "$dockerfile" -t "$image" "$context"
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec podman push "$image"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf 'error: Docker or Podman is required\n' >&2
|
||||
exit 1
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Collect read-only build context for a GreptimeDB development image."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, cast
|
||||
|
||||
from next_image_tag import increment_tag
|
||||
from binary_platform import PLATFORM_MACHINES, inspect_elf
|
||||
from image_config import image_reference, parse_env_file, validate_values
|
||||
|
||||
|
||||
ENV_KEYS = ("IMAGE_REGISTRY", "IMAGE_REPOSITORY", "IMAGE_TAG")
|
||||
PROFILE_DIRECTORIES = {"dev": "debug", "test": "debug", "bench": "release"}
|
||||
PLATFORM_ARCHITECTURES = {
|
||||
"x86_64": "linux/amd64",
|
||||
"amd64": "linux/amd64",
|
||||
"aarch64": "linux/arm64",
|
||||
"arm64": "linux/arm64",
|
||||
}
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source", required=True, type=Path)
|
||||
parser.add_argument("--bin", dest="binary", default="greptime")
|
||||
parser.add_argument("--profile", default="nightly")
|
||||
parser.add_argument("--target", help="Rust target triple for cross-compilation")
|
||||
parser.add_argument("--package", help="Cargo package containing the selected binary")
|
||||
parser.add_argument("--platform", choices=("linux/amd64", "linux/arm64"))
|
||||
parser.add_argument("--engine", choices=("auto", "docker", "podman"), default="auto")
|
||||
parser.add_argument("--registry")
|
||||
parser.add_argument("--repository")
|
||||
parser.add_argument("--tag")
|
||||
parser.add_argument(
|
||||
"--check-registry-tag",
|
||||
action="store_true",
|
||||
help="inspect the configured image tag with Docker without modifying it",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def native_platform(architecture: str) -> Optional[str]:
|
||||
return PLATFORM_ARCHITECTURES.get(architecture.lower())
|
||||
|
||||
|
||||
def parse_env(env_path: Path) -> dict[str, object]:
|
||||
values = parse_env_file(env_path)
|
||||
missing = [key for key in ("IMAGE_REPOSITORY", "IMAGE_TAG") if not values.get(key)]
|
||||
try:
|
||||
if not missing:
|
||||
validate_values(values)
|
||||
error = None
|
||||
except ValueError as exc:
|
||||
error = str(exc)
|
||||
return {"path": str(env_path), "exists": env_path.is_file(), "values": values, "missing": missing, "error": error}
|
||||
|
||||
|
||||
def cargo_metadata(source: Path) -> dict[str, Any]:
|
||||
command = ["cargo", "metadata", "--locked", "--format-version=1", "--no-deps"]
|
||||
try:
|
||||
result = subprocess.run(command, cwd=source, text=True, capture_output=True, check=True)
|
||||
except FileNotFoundError as error:
|
||||
raise RuntimeError("cargo is not installed or is not on PATH") from error
|
||||
except subprocess.CalledProcessError as error:
|
||||
detail = error.stderr.strip() or error.stdout.strip()
|
||||
raise RuntimeError(f"cargo metadata failed: {detail}") from error
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def binary_targets(metadata: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
targets = []
|
||||
for package in metadata["packages"]:
|
||||
for target in package["targets"]:
|
||||
if "bin" in target["kind"]:
|
||||
targets.append({
|
||||
"package": package["name"],
|
||||
"name": target["name"],
|
||||
"path": target["src_path"],
|
||||
"required_features": target.get("required-features", []),
|
||||
})
|
||||
return sorted(targets, key=lambda target: (target["name"], target["package"]))
|
||||
|
||||
|
||||
def expected_binary(target_dir: Path, binary: str, profile: str, rust_target: Optional[str]) -> Path:
|
||||
profile_directory = PROFILE_DIRECTORIES.get(profile, profile)
|
||||
output_dir = target_dir / rust_target if rust_target else target_dir
|
||||
return output_dir / profile_directory / binary
|
||||
|
||||
|
||||
def engine_status(engine: str) -> dict[str, object]:
|
||||
if engine == "docker":
|
||||
command = ["docker", "version", "--format", "{{.Server.Os}}/{{.Server.Arch}}"]
|
||||
else:
|
||||
command = ["podman", "info", "--format", "{{.Host.Os}}/{{.Host.Arch}}"]
|
||||
try:
|
||||
result = subprocess.run(command, text=True, capture_output=True, check=True)
|
||||
except (FileNotFoundError, subprocess.CalledProcessError):
|
||||
return {"available": False, "server_platform": None}
|
||||
return {"available": True, "server_platform": result.stdout.strip()}
|
||||
|
||||
|
||||
def registry_tag_status(values: dict[str, str], enabled: bool, engine: str, source: str) -> dict[str, object]:
|
||||
if not enabled:
|
||||
return {"checked": False, "status": "not_checked", "candidate_tag": None}
|
||||
|
||||
try:
|
||||
reference = image_reference(values)
|
||||
except ValueError:
|
||||
return {"checked": False, "status": "not_configured", "candidate_tag": None, "configuration_source": source}
|
||||
command = ["docker", "buildx", "imagetools", "inspect", reference] if engine == "docker" else ["podman", "manifest", "inspect", reference]
|
||||
try:
|
||||
subprocess.run(command, text=True, capture_output=True, check=True)
|
||||
except FileNotFoundError:
|
||||
return {"checked": True, "status": "unknown", "candidate_tag": None, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
except subprocess.CalledProcessError:
|
||||
return {"checked": True, "status": "unknown", "candidate_tag": None, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
|
||||
try:
|
||||
candidate_tag = increment_tag(values["IMAGE_TAG"])
|
||||
except ValueError:
|
||||
candidate_tag = None
|
||||
return {"checked": True, "status": "exists", "candidate_tag": candidate_tag, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
|
||||
|
||||
def collect_report(args: argparse.Namespace) -> dict[str, object]:
|
||||
source = args.source.expanduser().resolve()
|
||||
if not (source / "Cargo.toml").is_file():
|
||||
raise RuntimeError(f"not a Cargo workspace: {source}")
|
||||
|
||||
metadata = cargo_metadata(source)
|
||||
targets = binary_targets(metadata)
|
||||
target_dir = Path(metadata["target_directory"])
|
||||
expected = expected_binary(target_dir, args.binary, args.profile, args.target)
|
||||
host_architecture = platform.machine()
|
||||
selected_target = next((target for target in targets if target["name"] == args.binary), None)
|
||||
environment = parse_env(source / ".env")
|
||||
supplied_values = (args.registry, args.repository, args.tag)
|
||||
if any(value is not None for value in supplied_values) and not all(value is not None for value in supplied_values):
|
||||
raise RuntimeError("--registry, --repository, and --tag must be supplied together")
|
||||
values: dict[str, str] = {"IMAGE_REGISTRY": args.registry or "", "IMAGE_REPOSITORY": args.repository or "", "IMAGE_TAG": args.tag or ""} if all(value is not None for value in supplied_values) else cast(dict[str, str], environment["values"])
|
||||
configuration_source = "arguments" if all(value is not None for value in supplied_values) else "environment"
|
||||
docker = engine_status("docker")
|
||||
podman = engine_status("podman")
|
||||
engine = args.engine if args.engine != "auto" else "docker" if docker["available"] else "podman"
|
||||
binary_details: dict[str, object] = {"platform": None, "error": None}
|
||||
if expected.is_file():
|
||||
try:
|
||||
machine = inspect_elf(expected)
|
||||
binary_details["platform"] = next(
|
||||
(name for name, expected_machine in PLATFORM_MACHINES.items() if expected_machine == machine),
|
||||
None,
|
||||
)
|
||||
if binary_details["platform"] is None:
|
||||
binary_details["error"] = f"binary has unsupported ELF machine: {machine}"
|
||||
except ValueError as exc:
|
||||
binary_details["error"] = str(exc)
|
||||
return {
|
||||
"host": {"os": platform.system(), "arch": host_architecture, "native_platform": native_platform(host_architecture)},
|
||||
"docker": docker,
|
||||
"podman": podman,
|
||||
"environment": environment,
|
||||
"registry_tag": registry_tag_status(values, args.check_registry_tag, engine, configuration_source),
|
||||
"cargo": {"workspace_root": metadata["workspace_root"], "target_dir": str(target_dir), "binaries": targets},
|
||||
"selected_binary": {"name": args.binary, "package": args.package, "cargo_target_exists": selected_target is not None, "target": selected_target},
|
||||
"expected_binary": {"profile": args.profile, "rust_target": args.target, "path": str(expected), "exists": expected.is_file(), "executable": os.access(expected, os.X_OK), "platform": binary_details["platform"], "requested_platform": args.platform, "platform_matches": binary_details["platform"] == args.platform if args.platform and binary_details["platform"] else None, "validation_error": binary_details["error"], "mtime": datetime.fromtimestamp(expected.stat().st_mtime, timezone.utc).isoformat() if expected.exists() else None, "freshness": "unverified"},
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
print(json.dumps(collect_report(parse_arguments()), indent=2, sort_keys=True))
|
||||
except RuntimeError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Bounded parsing and safe updating of the image settings in a .env file."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Mapping
|
||||
|
||||
MANAGED_KEYS = ("IMAGE_REGISTRY", "IMAGE_REPOSITORY", "IMAGE_TAG")
|
||||
_REGISTRY = re.compile(r"[a-z0-9][a-z0-9.-]*(?::[0-9]+)?(?:/[a-z0-9][a-z0-9._-]*)*")
|
||||
_REPOSITORY = re.compile(r"[a-z0-9][a-z0-9._-]*(?:/[a-z0-9][a-z0-9._-]*)*")
|
||||
_TAG = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}")
|
||||
|
||||
|
||||
def _value(text: str) -> str:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return ""
|
||||
quote = text[0] if text[0] in "'\"" else None
|
||||
if quote:
|
||||
end = text.find(quote, 1)
|
||||
if end < 0 or text[end + 1 :].strip() and not text[end + 1 :].lstrip().startswith("#"):
|
||||
raise ValueError("unmatched quote or trailing value syntax")
|
||||
return text[1:end]
|
||||
# A comment is inline only when introduced by whitespace. This keeps
|
||||
# valid values such as an image tag containing '#'.
|
||||
match = re.search(r"\s+#", text)
|
||||
if match:
|
||||
text = text[: match.start()].rstrip()
|
||||
if any(character in text for character in "'\";$`"):
|
||||
raise ValueError("unsupported shell syntax in .env value")
|
||||
return text
|
||||
|
||||
|
||||
def parse_env(text: str) -> dict[str, str]:
|
||||
"""Parse the three managed keys, without validating their values."""
|
||||
lines = text.splitlines()
|
||||
result: dict[str, str] = {}
|
||||
for number, original in enumerate(lines, 1):
|
||||
line = original.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
export = re.match(r"export[ \t]+", line)
|
||||
if export:
|
||||
line = line[export.end() :].lstrip()
|
||||
key, separator, raw = line.partition("=")
|
||||
if not separator:
|
||||
continue
|
||||
key = key.strip()
|
||||
if key in MANAGED_KEYS:
|
||||
try:
|
||||
result[key] = _value(raw)
|
||||
except ValueError as error:
|
||||
raise ValueError(f"line {number}: {error}") from error
|
||||
return result
|
||||
|
||||
|
||||
def parse_env_file(path: Path) -> dict[str, str]:
|
||||
return parse_env(path.read_text() if path.exists() else "")
|
||||
|
||||
|
||||
def normalize_values(values: Mapping[str, str]) -> dict[str, str]:
|
||||
return {
|
||||
"IMAGE_REGISTRY": values.get("IMAGE_REGISTRY", ""),
|
||||
"IMAGE_REPOSITORY": values.get("IMAGE_REPOSITORY", ""),
|
||||
"IMAGE_TAG": values.get("IMAGE_TAG", ""),
|
||||
}
|
||||
|
||||
|
||||
def validate_values(values: Mapping[str, str]) -> None:
|
||||
values = normalize_values(values)
|
||||
missing = [key for key in ("IMAGE_REPOSITORY", "IMAGE_TAG") if not values.get(key)]
|
||||
if missing:
|
||||
raise ValueError("missing managed values: " + ", ".join(missing))
|
||||
registry = values.get("IMAGE_REGISTRY", "")
|
||||
if registry and (
|
||||
not _REGISTRY.fullmatch(registry)
|
||||
):
|
||||
raise ValueError("invalid registry")
|
||||
if registry:
|
||||
host = registry.split("/", 1)[0]
|
||||
if host != "localhost" and "." not in host and ":" not in host:
|
||||
raise ValueError("ambiguous registry; use an FQDN, explicit port, or localhost")
|
||||
if not _REPOSITORY.fullmatch(values["IMAGE_REPOSITORY"]):
|
||||
raise ValueError("invalid repository")
|
||||
if not _TAG.fullmatch(values["IMAGE_TAG"]):
|
||||
raise ValueError("invalid tag")
|
||||
if any(any(character.isspace() for character in value) for value in values.values()):
|
||||
raise ValueError("managed values cannot contain whitespace")
|
||||
|
||||
|
||||
def image_reference(values: Mapping[str, str]) -> str:
|
||||
values = normalize_values(values)
|
||||
validate_values(values)
|
||||
prefix = values.get("IMAGE_REGISTRY", "")
|
||||
repository = values["IMAGE_REPOSITORY"]
|
||||
return f"{prefix}/{repository}:{values['IMAGE_TAG']}" if prefix else f"{repository}:{values['IMAGE_TAG']}"
|
||||
|
||||
|
||||
def update_env(path: Path, values: Mapping[str, str]) -> bool:
|
||||
"""Atomically update managed entries; return whether the file changed."""
|
||||
values = normalize_values(values)
|
||||
validate_values(values)
|
||||
if path.is_symlink():
|
||||
raise ValueError("refusing to update symlink .env")
|
||||
old = path.read_text() if path.exists() else ""
|
||||
parse_env(old) # detect malformed managed syntax before changing anything
|
||||
lines = old.splitlines()
|
||||
output: list[str] = []
|
||||
written: set[str] = set()
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
export = re.match(r"export[ \t]+", stripped)
|
||||
candidate = stripped[export.end() :].lstrip() if export else stripped
|
||||
key = candidate.split("=", 1)[0].strip() if "=" in candidate else ""
|
||||
if key in MANAGED_KEYS:
|
||||
if key in written:
|
||||
continue
|
||||
output.append(f"{key}={values[key]}")
|
||||
written.add(key)
|
||||
else:
|
||||
output.append(line)
|
||||
output.extend(f"{key}={values.get(key, '')}" for key in MANAGED_KEYS if key not in written)
|
||||
new = "\n".join(output) + "\n"
|
||||
if old == new:
|
||||
return False
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
mode = os.stat(path).st_mode & 0o777 if path.exists() else 0o600
|
||||
fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
|
||||
try:
|
||||
os.fchmod(fd, mode)
|
||||
with os.fdopen(fd, "w") as handle:
|
||||
handle.write(new)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
return True
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Print a tag with its final numeric component incremented."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def increment_tag(tag: str) -> str:
|
||||
match = re.search(r"(\d+)$", tag)
|
||||
if match is None:
|
||||
raise ValueError("tag must end with a numeric version component")
|
||||
|
||||
number = match.group(1)
|
||||
return f"{tag[:match.start()]}{int(number) + 1:0{len(number)}d}"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Increment the final numeric component of a Docker image tag."
|
||||
)
|
||||
parser.add_argument("--tag", required=True, help="existing Docker image tag")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
print(increment_tag(args.tag))
|
||||
except ValueError as error:
|
||||
print(f"error: {error}: {args.tag!r}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --context PATH --binary PATH --dockerfile PATH --platform PLATFORM\n' "$0" >&2
|
||||
}
|
||||
|
||||
context=""
|
||||
binary=""
|
||||
dockerfile=""
|
||||
platform=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--context)
|
||||
context="$2"
|
||||
shift 2
|
||||
;;
|
||||
--binary)
|
||||
binary="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dockerfile)
|
||||
dockerfile="$2"
|
||||
shift 2
|
||||
;;
|
||||
--platform)
|
||||
platform="$2"
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$context" || -z "$binary" || -z "$dockerfile" || -z "$platform" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
python3 "$script_dir/binary_platform.py" --binary "$binary" --platform "$platform"
|
||||
|
||||
if [[ ! -f "$dockerfile" ]]; then
|
||||
printf 'error: Dockerfile template does not exist: %s\n' "$dockerfile" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -e "$context" && ( -L "$context" || -n "$(ls -A "$context")" ) ]]; then
|
||||
printf 'error: build context must be a new or empty directory: %s\n' "$context" >&2
|
||||
exit 1
|
||||
fi
|
||||
mkdir -p "$context"
|
||||
if [[ -e "$context/greptime" || -e "$context/Dockerfile" || -L "$context/greptime" || -L "$context/Dockerfile" ]]; then
|
||||
printf 'error: build context already contains output files: %s\n' "$context" >&2
|
||||
exit 1
|
||||
fi
|
||||
cp "$binary" "$context/greptime"
|
||||
cp "$dockerfile" "$context/Dockerfile"
|
||||
printf 'created Dockerfile: %s\n' "$context/Dockerfile"
|
||||
|
||||
file "$context/greptime"
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Create or update non-secret GreptimeDB image settings in a .env file."""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from image_config import image_reference, normalize_values, update_env, validate_values
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--env", type=Path)
|
||||
parser.add_argument("--registry", required=True)
|
||||
parser.add_argument("--repository", required=True)
|
||||
parser.add_argument("--tag", required=True)
|
||||
parser.add_argument("--validate-only", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
values = normalize_values({"IMAGE_REGISTRY": args.registry, "IMAGE_REPOSITORY": args.repository, "IMAGE_TAG": args.tag})
|
||||
validate_values(values)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.validate_only:
|
||||
print(image_reference(values))
|
||||
return 0
|
||||
if args.env is None:
|
||||
parser.error("--env is required unless --validate-only is set")
|
||||
if update_env(args.env, values):
|
||||
print(f"updated {args.env}")
|
||||
else:
|
||||
print(f"unchanged {args.env}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import importlib
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
binary_platform = importlib.import_module("binary_platform")
|
||||
|
||||
|
||||
def elf(machine: int, *, bits: int = 2, endian: int = 1, abi: int = 3, elf_type: int = 3) -> bytes:
|
||||
header = bytearray(64)
|
||||
header[:4] = b"\x7fELF"
|
||||
header[4:8] = bytes((bits, endian, 1, abi))
|
||||
struct.pack_into("<H", header, 16, elf_type)
|
||||
struct.pack_into("<H" if endian == 1 else ">H", header, 18, machine)
|
||||
struct.pack_into("<I", header, 20, 1)
|
||||
struct.pack_into("<H", header, 52, 64)
|
||||
return bytes(header)
|
||||
|
||||
|
||||
class BinaryPlatformTest(unittest.TestCase):
|
||||
def make_binary(self, content: bytes) -> Path:
|
||||
path = Path(self.directory.name) / "greptime"
|
||||
path.write_bytes(content)
|
||||
path.chmod(0o755)
|
||||
return path
|
||||
|
||||
def setUp(self):
|
||||
self.directory = tempfile.TemporaryDirectory()
|
||||
|
||||
def tearDown(self):
|
||||
self.directory.cleanup()
|
||||
|
||||
def test_matching_architectures(self):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "linux/amd64")
|
||||
binary_platform.validate_binary(self.make_binary(elf(0xB7)), "linux/arm64")
|
||||
|
||||
def test_rejects_bad_executable_elf_and_platform(self):
|
||||
path = self.make_binary(elf(0x3E))
|
||||
path.chmod(0o644)
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
path = self.make_binary(b"not an elf")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
path = self.make_binary(elf(0x3E, bits=1))
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E, abi=9)), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E, elf_type=1)), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(b"\x7fELF"), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "darwin/amd64")
|
||||
|
||||
def test_rejects_wrong_architecture_and_symlink(self):
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "linux/arm64")
|
||||
target = self.make_binary(elf(0x3E))
|
||||
link = Path(self.directory.name) / "link"
|
||||
link.symlink_to(target)
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(link, "linux/amd64")
|
||||
|
||||
def test_cli_success_and_failures(self):
|
||||
path = self.make_binary(elf(0x3E))
|
||||
assert binary_platform.__file__ is not None
|
||||
script = Path(binary_platform.__file__)
|
||||
ok = subprocess.run([sys.executable, str(script), "--binary", str(path), "--platform", "linux/amd64"], capture_output=True, text=True)
|
||||
self.assertEqual(ok.returncode, 0, ok.stderr)
|
||||
bad = subprocess.run([sys.executable, str(script), "--binary", str(path), "--platform", "linux/arm64"], capture_output=True, text=True)
|
||||
self.assertEqual(bad.returncode, 2)
|
||||
missing = subprocess.run([sys.executable, str(script), "--binary", str(path) + ".missing", "--platform", "linux/amd64"], capture_output=True, text=True)
|
||||
self.assertEqual(missing.returncode, 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import importlib
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
image_config = importlib.import_module("image_config")
|
||||
|
||||
|
||||
VALUES = {"IMAGE_REGISTRY": "registry.example.com/team", "IMAGE_REPOSITORY": "greptime-dev", "IMAGE_TAG": "dev-001"}
|
||||
|
||||
|
||||
class ImageConfigTest(unittest.TestCase):
|
||||
def test_parse_supported_bounded_dotenv(self):
|
||||
text = """\n# comment\n export IMAGE_REGISTRY = \"registry.example.com/team\" # note\nIMAGE_REPOSITORY='greptime-dev'\nIMAGE_TAG=dev-001 # safe comment\nOTHER=$HOME\n"""
|
||||
self.assertEqual(image_config.parse_env(text), VALUES)
|
||||
|
||||
def test_parser_does_not_validate_values(self):
|
||||
self.assertEqual(image_config.parse_env("IMAGE_TAG=not valid\n"), {"IMAGE_TAG": "not valid"})
|
||||
|
||||
def test_parser_rejects_unmatched_and_unsupported_syntax(self):
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.parse_env('IMAGE_TAG="unterminated\n')
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.parse_env("IMAGE_TAG=$(date)\n")
|
||||
|
||||
def test_validation_and_reference(self):
|
||||
image_config.validate_values(VALUES)
|
||||
self.assertEqual(image_config.image_reference(VALUES), "registry.example.com/team/greptime-dev:dev-001")
|
||||
local = dict(VALUES, IMAGE_REGISTRY="")
|
||||
self.assertEqual(image_config.image_reference(local), "greptime-dev:dev-001")
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="localhost"))
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="example.com"))
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="registry/team"))
|
||||
|
||||
def test_atomic_update_preserves_mode_and_canonicalizes_duplicates(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("# keep\nIMAGE_TAG=old\nIMAGE_TAG=duplicate\nOTHER=value\n")
|
||||
path.chmod(0o640)
|
||||
self.assertTrue(image_config.update_env(path, VALUES))
|
||||
self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o640)
|
||||
self.assertEqual(path.read_text(), "# keep\nIMAGE_TAG=dev-001\nOTHER=value\nIMAGE_REGISTRY=registry.example.com/team\nIMAGE_REPOSITORY=greptime-dev\n")
|
||||
self.assertFalse(image_config.update_env(path, VALUES))
|
||||
|
||||
def test_update_rejects_symlink(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
target = Path(directory) / "target"
|
||||
target.write_text("IMAGE_TAG=old\n")
|
||||
link = Path(directory) / ".env"
|
||||
link.symlink_to(target)
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.update_env(link, VALUES)
|
||||
self.assertEqual(target.read_text(), "IMAGE_TAG=old\n")
|
||||
|
||||
def test_update_clears_omitted_registry_and_uses_private_new_file_mode(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("IMAGE_REGISTRY=registry.example.com\nIMAGE_REPOSITORY=old\nIMAGE_TAG=old\n")
|
||||
values = {"IMAGE_REPOSITORY": "greptime-dev", "IMAGE_TAG": "dev-001"}
|
||||
image_config.update_env(path, values)
|
||||
self.assertIn("IMAGE_REGISTRY=\n", path.read_text())
|
||||
new_path = Path(directory) / "new.env"
|
||||
image_config.update_env(new_path, VALUES)
|
||||
self.assertEqual(stat.S_IMODE(new_path.stat().st_mode), 0o600)
|
||||
|
||||
def test_failed_replace_keeps_original_and_removes_temporary_file(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("OTHER=value\n")
|
||||
with mock.patch("image_config.os.replace", side_effect=OSError("no space")):
|
||||
with self.assertRaises(OSError):
|
||||
image_config.update_env(path, VALUES)
|
||||
self.assertEqual(path.read_text(), "OTHER=value\n")
|
||||
self.assertEqual(list(Path(directory).glob("..env.*")), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -3,6 +3,19 @@ FROM ghcr.io/actions/actions-runner@sha256:08c30b0a7105f64bddfc485d2487a22aa0393
|
||||
|
||||
USER root
|
||||
|
||||
ARG SCCACHE_VERSION=0.16.0
|
||||
ARG SCCACHE_SHA256=aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35588
|
||||
ARG RUSTUP_INIT_VERSION=1.29.0
|
||||
ARG RUSTUP_INIT_TARGET=x86_64-unknown-linux-gnu
|
||||
ARG RUSTUP_INIT_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10
|
||||
ARG RUST_TOOLCHAIN=nightly-2026-03-21
|
||||
|
||||
ENV RUSTUP_HOME=/opt/rustup \
|
||||
CARGO_HOME=/opt/cargo \
|
||||
RUSTUP_TOOLCHAIN=${RUST_TOOLCHAIN} \
|
||||
RUSTUP_AUTO_INSTALL=0 \
|
||||
PATH=/opt/cargo/bin:${PATH}
|
||||
|
||||
RUN apt-get update \
|
||||
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
@@ -30,4 +43,42 @@ RUN apt-get update \
|
||||
|
||||
COPY --from=uv /uv /uvx /usr/local/bin/
|
||||
|
||||
RUN curl --fail --location --silent --show-error \
|
||||
--output /tmp/sccache.tar.gz \
|
||||
"https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
|
||||
&& echo "${SCCACHE_SHA256} /tmp/sccache.tar.gz" | sha256sum --check --status - \
|
||||
&& mkdir --parents /tmp/sccache \
|
||||
&& tar --extract --gzip --file /tmp/sccache.tar.gz \
|
||||
--directory /tmp/sccache --strip-components=1 \
|
||||
"sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl/sccache" \
|
||||
&& install --mode=0755 /tmp/sccache/sccache /usr/local/bin/sccache \
|
||||
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache
|
||||
|
||||
RUN curl --fail --location --silent --show-error \
|
||||
--output /tmp/rustup-init \
|
||||
"https://static.rust-lang.org/rustup/archive/${RUSTUP_INIT_VERSION}/${RUSTUP_INIT_TARGET}/rustup-init" \
|
||||
&& echo "${RUSTUP_INIT_SHA256} /tmp/rustup-init" | sha256sum --check --status - \
|
||||
&& chmod 0755 /tmp/rustup-init \
|
||||
&& /tmp/rustup-init -y --profile minimal --default-toolchain "${RUST_TOOLCHAIN}" --no-modify-path \
|
||||
&& rm -f /tmp/rustup-init \
|
||||
&& chown -R root:root /opt/rustup /opt/cargo \
|
||||
&& chmod -R go-w /opt/rustup /opt/cargo \
|
||||
&& find /opt/rustup /opt/cargo -type d -exec chmod 0755 {} + \
|
||||
&& find /opt/rustup /opt/cargo -type f -exec chmod a+r {} +
|
||||
|
||||
USER runner
|
||||
|
||||
RUN set -eu \
|
||||
&& test "$(id -u)" = "1001" \
|
||||
&& temporary_cargo_home="$(mktemp --directory)" \
|
||||
&& trap 'rm -rf "${temporary_cargo_home}"' 0 \
|
||||
&& export CARGO_HOME="${temporary_cargo_home}" \
|
||||
&& test "${RUSTUP_AUTO_INSTALL}" = "0" \
|
||||
&& rustup --version \
|
||||
&& cargo --version \
|
||||
&& rustc --version \
|
||||
&& active_toolchain="$(rustup show active-toolchain)" \
|
||||
&& printf 'Active toolchain: %s\n' "${active_toolchain}" \
|
||||
&& case "${active_toolchain}" in "${RUST_TOOLCHAIN}-x86_64-unknown-linux-gnu"|"${RUST_TOOLCHAIN}-x86_64-unknown-linux-gnu "*) ;; *) exit 1;; esac \
|
||||
&& test ! -w /opt/rustup \
|
||||
&& test ! -w /opt/cargo/bin
|
||||
|
||||
@@ -1,21 +1,14 @@
|
||||
# Query regression self-hosted runners
|
||||
|
||||
The `Query Regression` workflow targets self-hosted GitHub Actions runners via
|
||||
runner labels or ARC runner scale set names:
|
||||
The `Query Regression` workflow uses the dedicated ARC runner scale set
|
||||
`perf-regression-8-cores`. ARC runner Pods run in the target Kubernetes cluster
|
||||
and connect outbound to GitHub. The live scale set is currently **paused**:
|
||||
`minRunners=0`, `maxRunners=0`, and no runner Pods. Do not resume it without
|
||||
explicit approval.
|
||||
|
||||
- `perf-regression-8-cores`
|
||||
## Prerequisites and trust admission
|
||||
|
||||
The name intentionally avoids generic labels such as `ubuntu-22.04-8-cores`,
|
||||
which may already be used by GitHub-hosted larger runners or other runner pools.
|
||||
|
||||
For Kubernetes-based runners, ARC runner pods run inside the target Kubernetes
|
||||
cluster and connect outbound to GitHub; GitHub can then dispatch jobs whose
|
||||
`runs-on` value matches the runner scale set name or runner labels.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install the ARC scale set controller in the Kubernetes cluster if it is not
|
||||
already installed:
|
||||
Install the ARC scale set controller if it is not already installed:
|
||||
|
||||
```bash
|
||||
helm upgrade --install arc \
|
||||
@@ -25,8 +18,8 @@ helm upgrade --install arc \
|
||||
--version 0.14.2
|
||||
```
|
||||
|
||||
Create the GitHub App or PAT secret in the namespace that will host the runner
|
||||
scale sets. Prefer a GitHub App with access limited to `GreptimeTeam/greptimedb`.
|
||||
Create the GitHub App secret in the runner namespace. Prefer an App limited to
|
||||
`GreptimeTeam/greptimedb`:
|
||||
|
||||
```bash
|
||||
kubectl -n arc-runners create secret generic greptimedb-arc-github-app \
|
||||
@@ -35,14 +28,46 @@ kubectl -n arc-runners create secret generic greptimedb-arc-github-app \
|
||||
--from-file=github_app_private_key=<private-key.pem>
|
||||
```
|
||||
|
||||
The values files in this directory reference that secret by name.
|
||||
The values files here reference that secret by name.
|
||||
|
||||
## Build the runner image
|
||||
A maintainer applying the `query-regression` label is **trust admission for
|
||||
that exact PR revision**. The admitted job may use this scale set's dedicated,
|
||||
writable persistent cache. `pull_request: labeled` is the only PR trigger: the
|
||||
label event snapshots its merge, head, and base SHAs. A queued job fetches that
|
||||
immutable event merge SHA directly, verifies it is a two-parent merge whose
|
||||
parents include the snapshotted head exactly once, and uses its other parent as
|
||||
the actual base build revision. The snapshotted event base is retained for audit
|
||||
only, so a difference from the merge's non-head parent is not a failure. The job
|
||||
never follows a newer mutable PR merge ref. An unavailable event merge, or one
|
||||
that does not contain exactly one snapshotted head parent, fails closed. A later
|
||||
PR head change does not retarget an already queued run: it may execute only its
|
||||
previously trusted event revision if that revision remains fetchable. To run the
|
||||
new revision, the maintainer must review it, remove the label, and re-add
|
||||
`query-regression`; cancel the old run if it is no longer wanted. An existing
|
||||
label does not automatically rerun the benchmark.
|
||||
|
||||
Use a derived ARC runner image instead of the minimal upstream runner image. The
|
||||
image keeps the official `/home/runner/run.sh` entrypoint layout and adds the
|
||||
tools this workflow expects to be present on a normal CI host, including `wget`,
|
||||
`uv`, C/C++ build tools, OpenSSL headers, and protobuf tooling.
|
||||
Admission does not relax runner hardening or GitHub permissions. Keep
|
||||
service-account token mounting disabled; do not mount host paths, the Docker
|
||||
socket, kubeconfig, or long-lived credentials. The runner and cache initializer
|
||||
use UID/GID 1001, disallow privilege escalation, drop all capabilities, and use
|
||||
the RuntimeDefault seccomp profile. Keep GitHub tokens least-privilege and
|
||||
review workflow changes before admission. Where the CNI supports it, restrict
|
||||
egress to required GitHub Actions, artifact/cache, Rust/crate/toolchain, DNS,
|
||||
and image-registry endpoints; block unrelated cluster services, private ranges,
|
||||
and metadata endpoints unless a case requires them.
|
||||
|
||||
### Office routing prerequisite
|
||||
|
||||
Direct split routing through `.2` is an **external office-network prerequisite**.
|
||||
The gateway at `192.168.50.2` must route GitHub Actions, GitHub content,
|
||||
artifact/cache, crates.io, Rust toolchain, and image-registry traffic directly
|
||||
rather than through the VPN. Neither this repository nor Kubernetes configures
|
||||
that route. Verify it with the responsible network operator before any canary.
|
||||
|
||||
## Runner image and workflow tools
|
||||
|
||||
Build and push the derived runner image; it preserves the official
|
||||
`/home/runner/run.sh` entrypoint and supplies CI tools needed at runtime:
|
||||
|
||||
```bash
|
||||
docker build \
|
||||
@@ -53,100 +78,219 @@ docker build \
|
||||
docker push greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest
|
||||
```
|
||||
|
||||
Deploy the runner image by digest rather than by a mutable tag. Update
|
||||
`values-8-cores.yaml` after pushing a rebuilt image.
|
||||
Deploy by digest, not mutable tag, by updating `values-8-cores.yaml` after a
|
||||
rebuild. If the registry is private, use a dedicated read-only pull secret only
|
||||
as `imagePullSecrets`; never expose registry credentials to runner containers.
|
||||
Both digest-pinned init and runner containers use `IfNotPresent`: the immutable
|
||||
digest makes a cached image safe and avoids adding a registry dependency to every
|
||||
runner startup.
|
||||
|
||||
The workflow still runs setup actions for pinned Rust and `uv` behavior. `mold`
|
||||
is installed in the image and selected through `CARGO_BUILD_RUSTFLAGS`, so jobs
|
||||
do not need privileged package installation at runtime.
|
||||
Before builds, the workflow asserts UID/GID 1001 and exact image tool versions:
|
||||
`libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.30.0`, `Python 3.12.3`, `sccache
|
||||
0.16.0`, root-owned `rustup 1.29.0`, and the image-baked
|
||||
`nightly-2026-03-21` Rust toolchain. Rustup, Cargo, and Rustc must resolve from
|
||||
`/opt/cargo/bin`; the runner cannot write `/opt/rustup` or `/opt/cargo/bin`.
|
||||
`actions-rust-lang/setup-rust-toolchain@v1` is intentionally removed. The
|
||||
workflow sets its warning-denying mold `RUSTFLAGS` directly, disables automatic
|
||||
Rustup installation, and performs no runtime toolchain downloads.
|
||||
|
||||
Decide whether the registry repository is public or private. Public pull access
|
||||
avoids distributing registry credentials to the runner namespace. If the image
|
||||
must be private, create a dedicated read-only image-pull secret for this image
|
||||
and attach it only as `imagePullSecrets`; do not expose registry credentials to
|
||||
runner containers.
|
||||
The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`, or
|
||||
runtime Rust setup: the image establishes immutable executable state and the PVC
|
||||
supplies only reusable Cargo data. Do not reintroduce those actions unless the
|
||||
corresponding cache or image contract changes.
|
||||
|
||||
## Install the query-regression scale set
|
||||
## Capacity and persistent cache
|
||||
|
||||
Install the runner scale set. The Helm release name and `runnerScaleSetName`
|
||||
should match the `runs-on` value used by the workflow.
|
||||
`values-8-cores.yaml` is normal operation: `minRunners=0`, `maxRunners=1`.
|
||||
`values-paused.yaml` is the mandatory pause overlay: `minRunners=0`,
|
||||
`maxRunners=0`. The job uses group `query-regression-persistent-cache-v1`,
|
||||
`queue: max`, and `cancel-in-progress: false`; admitted jobs queue rather than
|
||||
replacing older pending jobs. During maintenance, cancel admitted queued runs as
|
||||
well as pausing ARC. Runner Pods have `activeDeadlineSeconds=12600`.
|
||||
|
||||
The cache claim `query-regression-build-cache` is a nominal 600Gi `local-path`
|
||||
PVC in `arc-runners`. It is `ReadWriteOnce`; `local-path` uses
|
||||
WaitForFirstConsumer binding and Delete reclaim behavior, produces a
|
||||
node-affine local PV, is non-expandable, and the 600Gi request is not a hard
|
||||
storage quota. The runner's `minipc-3` selector is its only consumer candidate.
|
||||
|
||||
The initializer mounts the PVC root at `/cache`, creates and write-tests these
|
||||
versioned subpaths as non-root UID/GID 1001, and the runner mounts them as:
|
||||
|
||||
| Persistent state | PVC subpath | Runner mount |
|
||||
| --- | --- | --- |
|
||||
| Ephemeral Cargo home | `emptyDir` | `/home/runner/.cargo` |
|
||||
| Cargo registry data | `cargo-registry-v1` | `/home/runner/.cargo/registry` |
|
||||
| Cargo Git data | `cargo-git-v1` | `/home/runner/.cargo/git` |
|
||||
| Cargo target | `query-regression-target-v1` | `/home/runner/query-regression-target` |
|
||||
| Cache metadata | `meta-v1` | `/home/runner/query-regression-cache-meta` |
|
||||
| sccache local disk cache | `sccache-v1` | `/home/runner/.cache/sccache` |
|
||||
| Immutable Rust toolchain | image-owned | `/opt/rustup`, `/opt/cargo/bin` |
|
||||
|
||||
The Pod security context uses UID/GID and `fsGroup` 1001 with
|
||||
`fsGroupChangePolicy: OnRootMismatch`; no privileged `chown` or raw `hostPath`
|
||||
is used. `CARGO_HOME` is a per-Pod `emptyDir`; only its nested `registry` and
|
||||
`git` mounts are persistent. `RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are
|
||||
image-owned immutable paths, while `CARGO_TARGET_DIR`, cache metadata, and
|
||||
`SCCACHE_DIR` are persistent absolute paths. The runner sets
|
||||
`RUSTC_WRAPPER=/usr/local/bin/sccache`,
|
||||
`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=40G`, and
|
||||
`CARGO_INCREMENTAL=0`. sccache uses its local PVC disk backend and self-evicts
|
||||
at 40G; do not add runtime downloads, object storage, or a shared backend.
|
||||
|
||||
The repository's `.cargo/config.toml` remains a trusted per-revision build input.
|
||||
In contrast, `$CARGO_HOME/config*`, credentials, installed bins, and Cargo
|
||||
metadata outside the persistent `registry` and `git` data mounts are ephemeral
|
||||
and cannot survive to another Pod.
|
||||
|
||||
The local disk backend has a one-server constraint. `maxRunners=1` and the
|
||||
unchanged `query-regression-persistent-cache-v1` workflow concurrency group
|
||||
serialize runs; do not increase runner capacity or relax that serialization
|
||||
while this backend is in use. Base and candidate builds share the target; Cargo
|
||||
fingerprints invalidate source and dependency changes. The workflow records the
|
||||
sccache version and relevant environment in the target ABI marker, starts and
|
||||
zeros sccache after cache and toolchain checks, shows initial/base/candidate
|
||||
statistics, and resets statistics between base and candidate builds.
|
||||
|
||||
### Disk preflight and cleanup contract
|
||||
|
||||
Before applying or unpausing, verify the backing filesystem on `minipc-3` has
|
||||
at least 900GiB free. The current local-path provisioner source is
|
||||
`/opt/local-path-provisioner`; measure the filesystem containing it:
|
||||
|
||||
```bash
|
||||
df -PB1G /opt/local-path-provisioner
|
||||
```
|
||||
|
||||
The workflow reports `du`, `df -P`, human-readable free space, and inode
|
||||
availability before builds and in an always-run report. Its cleanup is narrow
|
||||
and non-destructive:
|
||||
|
||||
- warn at target size 400GiB; at 450GiB clear only the complete target root;
|
||||
- warn at Cargo registry-plus-Git data size 60GiB; at 80GiB remove only
|
||||
`registry/src` and `git/checkouts`, then abort if that persistent data remains
|
||||
at least 80GiB;
|
||||
- below 300GiB backing free space, clear the complete target root first,
|
||||
remeasure, then remove only those Cargo extracted trees and checkouts; abort
|
||||
if free space is still below 300GiB;
|
||||
- never automatically remove Cargo registry cache/index, Git database, the
|
||||
image-owned Cargo bin or Rustup toolchain, cache metadata, the self-evicting
|
||||
sccache directory, or the PVC.
|
||||
|
||||
The target clear uses fixed absolute roots and removes all entries, including
|
||||
dotfiles. After migration validation, remove obsolete `cargo-home-v1` and
|
||||
`rustup-home-v1` only in explicit maintenance while ARC is 0/0 and no runner Pod
|
||||
exists; they are not mounted by the current configuration.
|
||||
|
||||
## Deploy and pause safely
|
||||
|
||||
First verify external `.2` routing and the disk preflight. Apply the PVC; while
|
||||
the scale set is paused, it is expected to remain `Pending` because
|
||||
WaitForFirstConsumer has no scheduled runner:
|
||||
|
||||
```bash
|
||||
kubectl apply --dry-run=server \
|
||||
-f .github/runner-scale-sets/query-regression/cache-pvc.yaml
|
||||
kubectl apply -f .github/runner-scale-sets/query-regression/cache-pvc.yaml
|
||||
```
|
||||
|
||||
Render normal and paused configurations. Normal values are always first; the
|
||||
pause overlay is always last:
|
||||
|
||||
```bash
|
||||
helm template perf-regression-8-cores \
|
||||
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
|
||||
--namespace arc-runners --version 0.14.2 \
|
||||
--set controllerServiceAccount.name=arc-gha-rs-controller \
|
||||
--set controllerServiceAccount.namespace=arc-systems \
|
||||
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
|
||||
|
||||
helm template perf-regression-8-cores \
|
||||
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
|
||||
--namespace arc-runners --version 0.14.2 \
|
||||
--set controllerServiceAccount.name=arc-gha-rs-controller \
|
||||
--set controllerServiceAccount.namespace=arc-systems \
|
||||
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
|
||||
-f .github/runner-scale-sets/query-regression/values-paused.yaml
|
||||
```
|
||||
|
||||
The **first post-merge Helm deployment must reconcile the release in paused
|
||||
mode**. Keep the pause overlay last:
|
||||
|
||||
```bash
|
||||
# First post-merge deployment and every return to paused mode: 0/0.
|
||||
helm upgrade --install perf-regression-8-cores \
|
||||
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
|
||||
--namespace arc-runners --create-namespace --version 0.14.2 \
|
||||
--reset-values --wait \
|
||||
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
|
||||
-f .github/runner-scale-sets/query-regression/values-paused.yaml
|
||||
|
||||
# Expect 0/0 and no runner resources before considering normal mode.
|
||||
kubectl -n arc-runners get autoscalingrunnerset perf-regression-8-cores \
|
||||
-o jsonpath='{.spec.minRunners}{"/"}{.spec.maxRunners}{"\n"}'
|
||||
kubectl -n arc-runners get ephemeralrunners,pods \
|
||||
-l actions.github.com/scale-set-name=perf-regression-8-cores
|
||||
```
|
||||
|
||||
Only after that verification and separate explicit approval, apply normal 0/1
|
||||
operation without the pause overlay:
|
||||
|
||||
```bash
|
||||
helm upgrade --install perf-regression-8-cores \
|
||||
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
|
||||
--namespace arc-runners \
|
||||
--create-namespace \
|
||||
--version 0.14.2 \
|
||||
--namespace arc-runners --create-namespace --version 0.14.2 \
|
||||
--reset-values --wait \
|
||||
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
|
||||
```
|
||||
|
||||
Check registration and pods:
|
||||
Do not use bare `helm rollback`, `--atomic`, or `--reuse-values`: a stored
|
||||
revision can restore nonzero runner capacity. Inspect rendered manifests for
|
||||
capacity, the `minipc-3` selector, cache claim and mounts, initializer,
|
||||
security context, and resources. After approved normal mode receives its first
|
||||
canary, the PVC binds to `minipc-3`.
|
||||
|
||||
```bash
|
||||
kubectl -n arc-runners get pods
|
||||
kubectl -n arc-runners get autoscalingrunnersets
|
||||
```
|
||||
For local-PV node loss, cache recovery is intentionally disposable: return to
|
||||
0/0, recreate the PVC on a healthy node, cold-fill it, and run a new canary.
|
||||
|
||||
The scale sets should also appear under repository Actions runner settings.
|
||||
## Canary and rollback
|
||||
|
||||
## Security notes for fork PRs
|
||||
With explicit approval, run two identical `workflow_dispatch` canaries on
|
||||
`perf-regression-8-cores`, using immutable full base and candidate commit SHAs
|
||||
and `cargo_profile=nightly`. The first is the cold fill; the second verifies warm
|
||||
reuse. Record the workflow's base/candidate build elapsed logs and cache
|
||||
ABI-marker output, initial/base/candidate sccache statistics, and cache report.
|
||||
Confirm the image tool contract (root-owned Rustup/Cargo paths, baked nightly,
|
||||
and non-writable `/opt` roots) and that the ephemeral Cargo home contains only
|
||||
the mounted registry/Git data before Cargo creates per-Pod state.
|
||||
Obtain dependency and tool network byte counters from the
|
||||
identified `.2` counter source, filtered to `minipc-3` and the dependency/tool
|
||||
destinations.
|
||||
|
||||
Maintainer-approved fork PRs can run on self-hosted runners. Approval only lets
|
||||
the workflow execute; it does not make the fork code trusted.
|
||||
Accept the canary only when all of the following hold:
|
||||
|
||||
The `query-regression` label is the explicit trigger for PR runs. Updating a PR
|
||||
does not rerun the benchmark automatically; remove and re-add the label after
|
||||
reviewing the updated changes.
|
||||
- exactly one runner Pod runs on `minipc-3`, and both jobs use the same bound PV;
|
||||
- UID/GID 1001 cache mounts are writable; the warm run does not invalidate the
|
||||
target or bulk-redownload crates or toolchains; sccache reports separate base
|
||||
and candidate build statistics without server or cache-path errors; immutable
|
||||
Rustup/Cargo roots remain non-writable and only registry/Git data persists;
|
||||
- warm base build time is at most 50% of cold base build time;
|
||||
- warm dependency/tool network bytes are at most 10% of cold fill bytes;
|
||||
- cache sizes remain below soft watermarks, node free space remains at least
|
||||
300GiB, and the benchmark is correct without TLS EOFs or timeouts;
|
||||
- `.2` confirms this traffic is outside VPN accounting.
|
||||
|
||||
These runners execute PR code and should be treated as untrusted execution
|
||||
capacity:
|
||||
Immediately return to 0/0 after either canary unless ongoing normal operation
|
||||
has been explicitly approved; return immediately on any traffic, cache, disk,
|
||||
TLS, or correctness failure. To roll back, use the paused Helm upgrade above,
|
||||
or another explicit `helm upgrade --install` with known-good values followed by
|
||||
`values-paused.yaml`, `--reset-values`, and `--wait`. Do not delete the PVC
|
||||
automatically; preserve it for diagnosis unless intentionally discarding cache.
|
||||
|
||||
- keep them isolated from sensitive internal services unless explicitly required;
|
||||
- do not mount host paths, Docker socket, kubeconfig, or long-lived credentials;
|
||||
- use ephemeral runner pods and no shared work directory with trusted jobs;
|
||||
- disable service account token mounting in runner pods unless Kubernetes API
|
||||
access is intentionally required;
|
||||
- use a runner image whose default user is non-root, disable privilege
|
||||
escalation, drop Linux capabilities, and use the runtime-default seccomp
|
||||
profile;
|
||||
- keep GitHub tokens least-privilege and rely on normal `pull_request` behavior
|
||||
for fork PRs, where repository secrets are withheld and `GITHUB_TOKEN` is
|
||||
read-only;
|
||||
- review fork workflow changes before approving the run.
|
||||
## Future optional phases
|
||||
|
||||
If stronger isolation is required, install a separate runner group/namespace just
|
||||
for query-regression PR workloads and restrict repository/workflow access to the
|
||||
`Query Regression` workflow.
|
||||
|
||||
Use namespace or cluster network policy to restrict runner egress where the CNI
|
||||
supports the needed controls. Query-regression runners need outbound access to
|
||||
GitHub Actions services, GitHub artifact/cache endpoints, Rust/crate/toolchain
|
||||
endpoints, DNS, and the configured image registries. Block access to unrelated
|
||||
cluster services, private network ranges, and cloud metadata endpoints unless a
|
||||
case explicitly needs them.
|
||||
|
||||
## Build cache
|
||||
|
||||
The workflow builds base and candidate in the same job, the same source path,
|
||||
and a shared `CARGO_TARGET_DIR`. It checks out the base ref into `src`, builds
|
||||
and copies the base binary aside, then resets that same `src` checkout to the
|
||||
candidate ref before building the candidate binary. Keeping the workspace path
|
||||
stable improves Cargo incremental reuse for local workspace crates compared with
|
||||
building separate `base-src` and `candidate-src` checkouts.
|
||||
|
||||
The workflow also uses the GitHub Actions Rust cache for restore-only cache
|
||||
reuse. PR and dispatch runs do not save cache entries. Refresh shared caches from
|
||||
trusted maintenance workflows only.
|
||||
|
||||
The ARC values in this directory do not configure a cross-run runner-local
|
||||
compiler cache. If cross-run compile time still dominates the benchmark,
|
||||
deployers may add one of the following at the runner infrastructure layer:
|
||||
|
||||
- a custom runner image with Rust tooling, `mold`, and `sccache` preinstalled;
|
||||
- `RUSTC_WRAPPER=sccache` plus a shared `sccache` backend such as object storage,
|
||||
Redis, or a Kubernetes storage class that is safe for concurrent runner pods;
|
||||
- separate cache namespaces/buckets for untrusted PR code and trusted branches to
|
||||
avoid cache poisoning across trust boundaries.
|
||||
|
||||
Do not mount a shared writable host path, Docker socket, kubeconfig, or other
|
||||
privileged credentials into runners that execute PR code.
|
||||
The current phase uses a digest-pinned image with sccache 0.16.0 and no shared
|
||||
cache service. Optional follow-ups are an image additionally seeded with the
|
||||
exact Rust toolchain and `cargo fetch --locked`; or an internal read/write
|
||||
sccache backend or Cargo/Git mirror. Evaluate them only if persistent PVC reuse
|
||||
is insufficient.
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: query-regression-build-cache
|
||||
namespace: arc-runners
|
||||
labels:
|
||||
app.kubernetes.io/part-of: perf-regression-8-cores
|
||||
spec:
|
||||
volumeMode: Filesystem
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 600Gi
|
||||
@@ -3,21 +3,96 @@ githubConfigSecret: greptimedb-arc-github-app
|
||||
|
||||
runnerScaleSetName: "perf-regression-8-cores"
|
||||
minRunners: 0
|
||||
maxRunners: 4
|
||||
maxRunners: 1
|
||||
|
||||
template:
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
activeDeadlineSeconds: 11400
|
||||
activeDeadlineSeconds: 12600
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: minipc-3
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1001
|
||||
runAsGroup: 1001
|
||||
fsGroup: 1001
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumes:
|
||||
- name: cargo-home
|
||||
emptyDir: {}
|
||||
- name: build-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: query-regression-build-cache
|
||||
initContainers:
|
||||
- name: initialize-build-cache
|
||||
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:7022b551771dc82ec4c493f0c3db6ee3050ececd336598690a2165240ebe953e
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
umask 0002
|
||||
for directory in cargo-registry-v1 cargo-git-v1 query-regression-target-v1 meta-v1 sccache-v1; do
|
||||
cache_directory="/cache/${directory}"
|
||||
mkdir -p "${cache_directory}"
|
||||
test -w "${cache_directory}"
|
||||
probe_file="${cache_directory}/.write-probe"
|
||||
: > "${probe_file}"
|
||||
test -f "${probe_file}"
|
||||
rm "${probe_file}"
|
||||
done
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1001
|
||||
runAsGroup: 1001
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
ephemeral-storage: 1Gi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
ephemeral-storage: 1Gi
|
||||
volumeMounts:
|
||||
- name: build-cache
|
||||
mountPath: /cache
|
||||
containers:
|
||||
- name: runner
|
||||
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:108b7357532e5cf3329853373f561b0c674e954d6307b53d37075df5a189b0b2
|
||||
imagePullPolicy: Always
|
||||
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:7022b551771dc82ec4c493f0c3db6ee3050ececd336598690a2165240ebe953e
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/home/runner/run.sh"]
|
||||
env:
|
||||
- name: CARGO_HOME
|
||||
value: /home/runner/.cargo
|
||||
- name: RUSTUP_HOME
|
||||
value: /opt/rustup
|
||||
- name: RUSTUP_TOOLCHAIN
|
||||
value: nightly-2026-03-21
|
||||
- name: RUSTUP_AUTO_INSTALL
|
||||
value: "0"
|
||||
- name: CARGO_TARGET_DIR
|
||||
value: /home/runner/query-regression-target
|
||||
- name: QUERY_REGRESSION_CACHE_META
|
||||
value: /home/runner/query-regression-cache-meta
|
||||
- name: RUSTC_WRAPPER
|
||||
value: /usr/local/bin/sccache
|
||||
- name: SCCACHE_DIR
|
||||
value: /home/runner/.cache/sccache
|
||||
- name: SCCACHE_CACHE_SIZE
|
||||
value: 40G
|
||||
- name: CARGO_INCREMENTAL
|
||||
value: "0"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1001
|
||||
runAsGroup: 1001
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
@@ -31,3 +106,21 @@ template:
|
||||
cpu: "8"
|
||||
memory: 24Gi
|
||||
ephemeral-storage: 80Gi
|
||||
volumeMounts:
|
||||
- name: cargo-home
|
||||
mountPath: /home/runner/.cargo
|
||||
- name: build-cache
|
||||
mountPath: /home/runner/.cargo/registry
|
||||
subPath: cargo-registry-v1
|
||||
- name: build-cache
|
||||
mountPath: /home/runner/.cargo/git
|
||||
subPath: cargo-git-v1
|
||||
- name: build-cache
|
||||
mountPath: /home/runner/query-regression-target
|
||||
subPath: query-regression-target-v1
|
||||
- name: build-cache
|
||||
mountPath: /home/runner/query-regression-cache-meta
|
||||
subPath: meta-v1
|
||||
- name: build-cache
|
||||
mountPath: /home/runner/.cache/sccache
|
||||
subPath: sccache-v1
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
minRunners: 0
|
||||
maxRunners: 0
|
||||
@@ -241,8 +241,10 @@ module.exports = async function validateQueryRegressionComment({ github, context
|
||||
'> Rendered by a trusted workflow from JSON artifacts produced by the query-regression run. Results from untrusted PR code are advisory until reviewed.',
|
||||
'',
|
||||
`- **Workflow run:** ${serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${expectedRunId}`,
|
||||
`- **Base SHA:** \`${text(metadata.base_sha)}\``,
|
||||
`- **Built base SHA:** \`${text(metadata.built_base_sha)}\``,
|
||||
`- **Event base SHA:** \`${text(metadata.event_base_sha)}\``,
|
||||
`- **Head SHA:** \`${text(metadata.head_sha)}\``,
|
||||
`- **Candidate merge SHA:** \`${text(metadata.candidate_sha)}\``,
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
|
||||
@@ -31,7 +31,10 @@ def main() -> int:
|
||||
metadata = {
|
||||
"pr_number": int(os.environ["PR_NUMBER"]),
|
||||
"head_sha": os.environ["HEAD_SHA"],
|
||||
"base_sha": os.environ["BASE_SHA"],
|
||||
"base_sha": os.environ["EVENT_BASE_SHA"],
|
||||
"event_base_sha": os.environ["EVENT_BASE_SHA"],
|
||||
"built_base_sha": os.environ["VERIFIED_BASE_SHA"],
|
||||
"candidate_sha": os.environ["VERIFIED_CANDIDATE_SHA"],
|
||||
"head_repo": os.environ["HEAD_REPO"],
|
||||
"base_repo": os.environ["BASE_REPO"],
|
||||
"run_id": int(os.environ["RUN_ID"]),
|
||||
|
||||
@@ -10,14 +10,19 @@ on:
|
||||
description: The nightly build workflow run id to download GreptimeDB artifacts from
|
||||
required: true
|
||||
type: string
|
||||
download_dataset:
|
||||
description: Download the JSONBench dataset before running the benchmark
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
actions: read
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
group: ${{ github.workflow }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
resolve-artifact:
|
||||
@@ -115,6 +120,34 @@ jobs:
|
||||
security-group-id: ${{ vars.EC2_RUNNER_SECURITY_GROUP_ID }}
|
||||
subnet-id: ${{ vars.EC2_RUNNER_SUBNET_ID }}
|
||||
|
||||
- name: Attach JSONBench data volume
|
||||
if: ${{ steps.start-linux-runner.outputs.ec2-instance-id != '' && inputs.download_dataset != true }}
|
||||
shell: bash
|
||||
env:
|
||||
INSTANCE_ID: ${{ steps.start-linux-runner.outputs.ec2-instance-id }}
|
||||
VOLUME_ID: ${{ secrets.JSONBENCH_DATA_VOLUME_ID }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "${VOLUME_ID}" ]]; then
|
||||
echo "JSONBENCH_DATA_VOLUME_ID is not configured"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Waiting for JSONBench data volume to become available"
|
||||
aws ec2 wait volume-available --volume-ids "${VOLUME_ID}"
|
||||
|
||||
echo "Attaching JSONBench data volume to EC2 instance ${INSTANCE_ID} as /dev/sdf"
|
||||
aws ec2 attach-volume \
|
||||
--instance-id "${INSTANCE_ID}" \
|
||||
--volume-id "${VOLUME_ID}" \
|
||||
--device /dev/sdf \
|
||||
> /dev/null
|
||||
|
||||
echo "Waiting for JSONBench data volume attachment to complete"
|
||||
aws ec2 wait volume-in-use --volume-ids "${VOLUME_ID}"
|
||||
echo "JSONBench data volume is attached to EC2 instance ${INSTANCE_ID}"
|
||||
|
||||
jsonbench:
|
||||
name: Run JSONBench
|
||||
if: ${{ github.repository == 'GreptimeTeam/greptimedb' && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
@@ -123,7 +156,37 @@ jobs:
|
||||
timeout-minutes: 120
|
||||
env:
|
||||
JSONBENCH_OUTPUT_PREFIX: _linux-${{ needs.resolve-artifact.outputs.artifact-arch }}
|
||||
JSONBENCH_DATASET: 3
|
||||
JSONBENCH_DATA_DIR: /root/data/bluesky
|
||||
steps:
|
||||
- name: Mount JSONBench data volume
|
||||
if: ${{ needs.allocate-runner.outputs.linux-ec2-runner-instance-id != '' && inputs.download_dataset != true }}
|
||||
shell: bash
|
||||
env:
|
||||
VOLUME_ID: ${{ secrets.JSONBENCH_DATA_VOLUME_ID }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
volume_serial="${VOLUME_ID//-/}"
|
||||
device=""
|
||||
echo "Waiting for the JSONBench data volume block device"
|
||||
for _ in {1..30}; do
|
||||
device=$(lsblk -dpno NAME,SERIAL | awk -v serial="${volume_serial}" '$2 == serial { print $1; exit }')
|
||||
[[ -n "${device}" ]] && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [[ -z "${device}" ]]; then
|
||||
echo "Cannot find the JSONBench data volume block device"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Resolved JSONBench data volume to ${device}"
|
||||
mkdir -p "${JSONBENCH_DATA_DIR}"
|
||||
echo "Mounting ${device} read-only at ${JSONBENCH_DATA_DIR}"
|
||||
mount -o ro "${device}" "${JSONBENCH_DATA_DIR}"
|
||||
echo "JSONBench data volume is mounted at ${JSONBENCH_DATA_DIR}"
|
||||
|
||||
- name: Download GreptimeDB nightly artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
@@ -142,26 +205,33 @@ jobs:
|
||||
chmod +x ./greptime
|
||||
rm -rf greptimedb-artifact "${{ needs.resolve-artifact.outputs.artifact-name }}"
|
||||
|
||||
- name: Run JSONBench
|
||||
env:
|
||||
# TODO(LFC): Change to "3" (100m) when JSON2 ingestion performance is optimized.
|
||||
JSONBENCH_DATASET: 2
|
||||
- name: Clone JSONBench
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
export JSONBENCH_DATA_DIR="/root/data/bluesky"
|
||||
echo "Use JSONBench data directory ${JSONBENCH_DATA_DIR}"
|
||||
echo "${JSONBENCH_DATASET}" > jsonbench.dataset
|
||||
|
||||
echo "Cloning JSONBench"
|
||||
git clone --branch greptimedb-new-json --depth 1 https://github.com/GreptimeTeam/JSONBench.git JSONBench
|
||||
|
||||
- name: Download JSONBench dataset
|
||||
if: ${{ inputs.download_dataset == true }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "Downloading JSONBench dataset choice ${JSONBENCH_DATASET} to ${JSONBENCH_DATA_DIR}"
|
||||
mkdir -p "${JSONBENCH_DATA_DIR}"
|
||||
printf "${JSONBENCH_DATASET}\n" | ./JSONBench/download_data.sh
|
||||
|
||||
- name: Run JSONBench
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "Use JSONBench data directory ${JSONBENCH_DATA_DIR}"
|
||||
echo "${JSONBENCH_DATASET}" > jsonbench.dataset
|
||||
|
||||
downloaded_files=$(find "${JSONBENCH_DATA_DIR}" -type f | wc -l)
|
||||
echo "Downloaded JSONBench dataset files: ${downloaded_files}"
|
||||
echo "Available JSONBench dataset files: ${downloaded_files}"
|
||||
|
||||
export GREPTIMEDB_STANDALONE__WAL__DIR=greptimedb_data/wal
|
||||
export GREPTIMEDB_STANDALONE__STORAGE__DATA_HOME=greptimedb_data
|
||||
|
||||
@@ -45,10 +45,6 @@ on:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
query-regression:
|
||||
if: >-
|
||||
@@ -59,12 +55,29 @@ jobs:
|
||||
(github.event.action != 'labeled' || github.event.label.name == 'query-regression')) }}
|
||||
runs-on: ${{ github.event_name == 'workflow_dispatch' && inputs.runner || 'perf-regression-8-cores' }}
|
||||
timeout-minutes: 180
|
||||
concurrency:
|
||||
group: query-regression-persistent-cache-v1
|
||||
queue: max
|
||||
cancel-in-progress: false
|
||||
env:
|
||||
CARGO_PROFILE: ${{ github.event_name == 'pull_request' && 'nightly' || inputs.cargo_profile }}
|
||||
CARGO_TARGET_DIR: ${{ github.workspace }}/query-regression-target
|
||||
CARGO_BUILD_RUSTFLAGS: "-C link-arg=-fuse-ld=mold"
|
||||
CARGO_HOME: /home/runner/.cargo
|
||||
RUSTUP_HOME: /opt/rustup
|
||||
RUSTUP_TOOLCHAIN: nightly-2026-03-21
|
||||
RUSTUP_AUTO_INSTALL: "0"
|
||||
CARGO_TARGET_DIR: /home/runner/query-regression-target
|
||||
QUERY_REGRESSION_CACHE_META: /home/runner/query-regression-cache-meta
|
||||
RUSTC_WRAPPER: /usr/local/bin/sccache
|
||||
SCCACHE_DIR: /home/runner/.cache/sccache
|
||||
SCCACHE_CACHE_SIZE: 40G
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold
|
||||
QUERY_REGRESSION_CACHE_EPOCH: "1"
|
||||
BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || inputs.base_ref }}
|
||||
CANDIDATE_REF: ${{ github.event_name == 'pull_request' && format('refs/pull/{0}/merge', github.event.pull_request.number) || inputs.candidate_ref || github.ref }}
|
||||
CANDIDATE_REF: ${{ github.event_name == 'workflow_dispatch' && (inputs.candidate_ref || github.ref) || '' }}
|
||||
EVENT_MERGE_SHA: ${{ github.event_name == 'pull_request' && github.sha || '' }}
|
||||
EVENT_HEAD_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || '' }}
|
||||
EVENT_BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }}
|
||||
steps:
|
||||
- name: Checkout base source
|
||||
uses: actions/checkout@v4
|
||||
@@ -82,28 +95,338 @@ jobs:
|
||||
cp src/.github/scripts/query-regression-summary.py \
|
||||
query-regression-trusted-scripts/query-regression-summary.py
|
||||
|
||||
- uses: arduino/setup-protoc@v3
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Resolve immutable candidate and build base
|
||||
working-directory: src
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache: false
|
||||
fail_closed() {
|
||||
printf '%s\n' \
|
||||
"PR changed or merge result unavailable; maintainer must remove and re-add query-regression after reviewing current revision. $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
- uses: astral-sh/setup-uv@v5
|
||||
is_full_sha() {
|
||||
[[ "$1" =~ ^[[:xdigit:]]{40}$ ]]
|
||||
}
|
||||
|
||||
- name: Rust cache
|
||||
uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: src
|
||||
shared-key: query-regression
|
||||
cache-all-crates: "true"
|
||||
save-if: false
|
||||
initially_checked_out_base_sha="$(git rev-parse --verify HEAD)"
|
||||
is_full_sha "${initially_checked_out_base_sha}" || {
|
||||
printf 'Could not resolve initially checked-out BASE_REF commit.\n' >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
|
||||
for identity in EVENT_MERGE_SHA EVENT_HEAD_SHA EVENT_BASE_SHA; do
|
||||
value="${!identity-}"
|
||||
is_full_sha "${value}" || fail_closed "Missing or invalid ${identity}."
|
||||
done
|
||||
|
||||
if ! git fetch --no-tags --prune --depth=2 origin "${EVENT_MERGE_SHA}"; then
|
||||
fail_closed "Could not fetch event merge SHA ${EVENT_MERGE_SHA}."
|
||||
fi
|
||||
fetched_sha="$(git rev-parse --verify FETCH_HEAD 2>/dev/null)" \
|
||||
|| fail_closed "Fetched event merge SHA could not be resolved."
|
||||
[[ "${fetched_sha,,}" == "${EVENT_MERGE_SHA,,}" ]] \
|
||||
|| fail_closed "FETCH_HEAD ${fetched_sha} does not match event merge SHA ${EVENT_MERGE_SHA}."
|
||||
|
||||
read -r -a parents <<< "$(git show -s --format=%P FETCH_HEAD)"
|
||||
(( ${#parents[@]} == 2 )) \
|
||||
|| fail_closed "Event merge SHA ${EVENT_MERGE_SHA} is not a two-parent merge commit."
|
||||
head_parent_count=0
|
||||
for parent in "${parents[@]}"; do
|
||||
if [[ "${parent,,}" == "${EVENT_HEAD_SHA,,}" ]]; then
|
||||
((head_parent_count += 1))
|
||||
else
|
||||
VERIFIED_BASE_SHA="${parent,,}"
|
||||
fi
|
||||
done
|
||||
(( head_parent_count == 1 )) \
|
||||
|| fail_closed "Event merge SHA must have exactly one EVENT_HEAD_SHA parent."
|
||||
is_full_sha "${VERIFIED_BASE_SHA-}" \
|
||||
|| fail_closed "Event merge SHA has no non-head parent for the base build."
|
||||
VERIFIED_CANDIDATE_SHA="${fetched_sha,,}"
|
||||
else
|
||||
git fetch --no-tags --prune --depth=1 origin "${CANDIDATE_REF}"
|
||||
VERIFIED_CANDIDATE_SHA="$(git rev-parse --verify FETCH_HEAD)"
|
||||
VERIFIED_BASE_SHA="${initially_checked_out_base_sha}"
|
||||
fi
|
||||
|
||||
is_full_sha "${VERIFIED_CANDIDATE_SHA}" || {
|
||||
printf 'Could not resolve candidate SHA.\n' >&2
|
||||
exit 1
|
||||
}
|
||||
is_full_sha "${VERIFIED_BASE_SHA}" || {
|
||||
printf 'Could not resolve build base SHA.\n' >&2
|
||||
exit 1
|
||||
}
|
||||
printf 'VERIFIED_CANDIDATE_SHA=%s\n' "${VERIFIED_CANDIDATE_SHA,,}" >> "${GITHUB_ENV}"
|
||||
printf 'VERIFIED_BASE_SHA=%s\n' "${VERIFIED_BASE_SHA,,}" >> "${GITHUB_ENV}"
|
||||
printf 'Verified candidate SHA: %s\n' "${VERIFIED_CANDIDATE_SHA,,}"
|
||||
printf 'Verified build base SHA: %s\n' "${VERIFIED_BASE_SHA,,}"
|
||||
printf -- '- Verified candidate SHA: `%s`; built base SHA: `%s`\n' \
|
||||
"${VERIFIED_CANDIDATE_SHA,,}" "${VERIFIED_BASE_SHA,,}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
git reset --hard "${VERIFIED_BASE_SHA}"
|
||||
git clean -ffdx
|
||||
|
||||
- name: Verify runner image tools
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ "$(id -u)" == "1001" ]]
|
||||
[[ "$(id -g)" == "1001" ]]
|
||||
[[ "$(protoc --version)" == "libprotoc 3.21.12" ]]
|
||||
[[ "$(uv --version)" =~ ^uv[[:space:]]0\.11\.26([[:space:]]|$) ]]
|
||||
mold_version="$(mold --version)"
|
||||
[[ "${mold_version}" =~ ^mold[[:space:]]2\.30\.0([[:space:]]|$) ]]
|
||||
[[ "$(python3 --version)" == "Python 3.12.3" ]]
|
||||
sccache_version="$(sccache --version)"
|
||||
[[ "${sccache_version}" =~ ^sccache[[:space:]]0\.16\.0([[:space:]]|$) ]]
|
||||
[[ "$(command -v rustup)" == "/opt/cargo/bin/rustup" ]]
|
||||
[[ "$(command -v cargo)" == "/opt/cargo/bin/cargo" ]]
|
||||
[[ "$(command -v rustc)" == "/opt/cargo/bin/rustc" ]]
|
||||
[[ "$(rustup --version)" =~ ^rustup[[:space:]]1\.29\.0([[:space:]]|$) ]]
|
||||
cargo_version="$(cargo --version)"
|
||||
[[ "${cargo_version}" =~ ^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
|
||||
rustc_version="$(rustc --version)"
|
||||
[[ "${rustc_version}" =~ ^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
|
||||
active_toolchain="$(rustup show active-toolchain)"
|
||||
[[ "${active_toolchain}" =~ ^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$) ]]
|
||||
[[ "${RUSTUP_HOME}" == "/opt/rustup" ]]
|
||||
[[ "${RUSTUP_TOOLCHAIN}" == "nightly-2026-03-21" ]]
|
||||
[[ "${RUSTUP_AUTO_INSTALL}" == "0" ]]
|
||||
test -r /opt/rustup && test -x /opt/rustup
|
||||
test ! -w /opt/rustup
|
||||
test ! -w /opt/cargo/bin
|
||||
for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do
|
||||
test ! -e "${CARGO_HOME}/${entry}"
|
||||
done
|
||||
mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git"
|
||||
|
||||
- name: Prepare persistent query regression cache
|
||||
shell: bash
|
||||
working-directory: src
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
readonly EXPECTED_CARGO_HOME="/home/runner/.cargo"
|
||||
readonly EXPECTED_CARGO_REGISTRY="/home/runner/.cargo/registry"
|
||||
readonly EXPECTED_CARGO_GIT="/home/runner/.cargo/git"
|
||||
readonly EXPECTED_RUSTUP_HOME="/opt/rustup"
|
||||
readonly EXPECTED_TARGET_DIR="/home/runner/query-regression-target"
|
||||
readonly EXPECTED_CACHE_META="/home/runner/query-regression-cache-meta"
|
||||
readonly EXPECTED_SCCACHE_DIR="/home/runner/.cache/sccache"
|
||||
readonly EXPECTED_RUSTC_WRAPPER="/usr/local/bin/sccache"
|
||||
readonly RUNNER_IMAGE_DIGEST="sha256:7022b551771dc82ec4c493f0c3db6ee3050ececd336598690a2165240ebe953e"
|
||||
readonly RUNNER_IMAGE_EPOCH="3"
|
||||
|
||||
require_expected_root() {
|
||||
local name="$1"
|
||||
local actual="$2"
|
||||
local expected="$3"
|
||||
[[ "${actual}" == /* && "${actual}" == "${expected}" ]] || {
|
||||
printf 'Refusing unexpected %s root: %s\n' "${name}" "${actual}" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
clear_directory() {
|
||||
local root="$1"
|
||||
[[ "${root}" == /* && "${root}" != "/" ]] || {
|
||||
printf 'Refusing to clear non-absolute or root path: %s\n' "${root}" >&2
|
||||
exit 1
|
||||
}
|
||||
case "${root}" in
|
||||
"${EXPECTED_TARGET_DIR}"|"${EXPECTED_TARGET_DIR}/debug"|"${EXPECTED_TARGET_DIR}/release"|"${EXPECTED_TARGET_DIR}/nightly") ;;
|
||||
*)
|
||||
printf 'Refusing to clear unexpected cache path: %s\n' "${root}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
find "${root}" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
|
||||
}
|
||||
|
||||
clear_cargo_extracted_trees() {
|
||||
require_expected_root CARGO_REGISTRY "${CARGO_HOME}/registry" "${EXPECTED_CARGO_REGISTRY}"
|
||||
require_expected_root CARGO_GIT "${CARGO_HOME}/git" "${EXPECTED_CARGO_GIT}"
|
||||
rm -rf -- "${CARGO_HOME}/registry/src" "${CARGO_HOME}/git/checkouts"
|
||||
}
|
||||
|
||||
cargo_size_kib() {
|
||||
du -sk -- "${CARGO_HOME}/registry" "${CARGO_HOME}/git" | awk '{ total += $1 } END { print total }'
|
||||
}
|
||||
|
||||
target_size_kib() {
|
||||
du -sk -- "${CARGO_TARGET_DIR}" | cut -f1
|
||||
}
|
||||
|
||||
free_kib() {
|
||||
df -Pk "${CARGO_TARGET_DIR}" | awk 'NR == 2 { print $4 }'
|
||||
}
|
||||
|
||||
report_cache_usage() {
|
||||
du -sh -- "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \
|
||||
"${RUSTUP_HOME}" "${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}"
|
||||
df -P "${CARGO_TARGET_DIR}"
|
||||
df -hP "${CARGO_TARGET_DIR}"
|
||||
df -Pi "${CARGO_TARGET_DIR}"
|
||||
}
|
||||
|
||||
require_expected_root CARGO_HOME "${CARGO_HOME}" "${EXPECTED_CARGO_HOME}"
|
||||
require_expected_root CARGO_REGISTRY "${CARGO_HOME}/registry" "${EXPECTED_CARGO_REGISTRY}"
|
||||
require_expected_root CARGO_GIT "${CARGO_HOME}/git" "${EXPECTED_CARGO_GIT}"
|
||||
require_expected_root RUSTUP_HOME "${RUSTUP_HOME}" "${EXPECTED_RUSTUP_HOME}"
|
||||
require_expected_root CARGO_TARGET_DIR "${CARGO_TARGET_DIR}" "${EXPECTED_TARGET_DIR}"
|
||||
require_expected_root QUERY_REGRESSION_CACHE_META "${QUERY_REGRESSION_CACHE_META}" "${EXPECTED_CACHE_META}"
|
||||
require_expected_root SCCACHE_DIR "${SCCACHE_DIR}" "${EXPECTED_SCCACHE_DIR}"
|
||||
[[ "${RUSTC_WRAPPER}" == "${EXPECTED_RUSTC_WRAPPER}" ]] || {
|
||||
printf 'Refusing unexpected RUSTC_WRAPPER: %s\n' "${RUSTC_WRAPPER}" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "${SCCACHE_CACHE_SIZE}" == "40G" ]] || {
|
||||
printf 'Refusing unexpected SCCACHE_CACHE_SIZE: %s\n' "${SCCACHE_CACHE_SIZE}" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "${CARGO_INCREMENTAL}" == "0" ]] || {
|
||||
printf 'Refusing unexpected CARGO_INCREMENTAL: %s\n' "${CARGO_INCREMENTAL}" >&2
|
||||
exit 1
|
||||
}
|
||||
test -r "${RUSTUP_HOME}"
|
||||
test -x "${RUSTUP_HOME}"
|
||||
test ! -w "${RUSTUP_HOME}"
|
||||
mkdir -p "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \
|
||||
"${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}"
|
||||
for root in "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \
|
||||
"${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}"; do
|
||||
test -w "${root}"
|
||||
touch "${root}/.query-regression-write-test"
|
||||
rm -f "${root}/.query-regression-write-test"
|
||||
done
|
||||
|
||||
case "${CARGO_PROFILE}" in
|
||||
dev) target_profile="debug" ;;
|
||||
release|nightly) target_profile="${CARGO_PROFILE}" ;;
|
||||
*) printf 'Unsupported CARGO_PROFILE: %s\n' "${CARGO_PROFILE}" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
rustc_vv="$(rustc -vV)"
|
||||
if cargo -Vv >/dev/null 2>&1; then
|
||||
cargo_vv="$(cargo -Vv)"
|
||||
else
|
||||
cargo_vv="$(cargo -V)"
|
||||
fi
|
||||
host_triple="$(printf '%s\n' "${rustc_vv}" | grep '^host: ' | cut -d' ' -f2-)"
|
||||
cc_version="$(cc --version)"
|
||||
mold_version="$(mold --version | grep -m1 .)"
|
||||
protoc_version="$(protoc --version)"
|
||||
sccache_version="$(sccache --version)"
|
||||
rustup_version="$(rustup --version)"
|
||||
active_toolchain="$(rustup show active-toolchain)"
|
||||
rustup_path="$(command -v rustup)"
|
||||
cargo_path="$(command -v cargo)"
|
||||
rustc_path="$(command -v rustc)"
|
||||
profile_target_dir="${CARGO_TARGET_DIR}/${target_profile}"
|
||||
marker="${QUERY_REGRESSION_CACHE_META}/target-${target_profile}.abi"
|
||||
new_marker="$(mktemp "${QUERY_REGRESSION_CACHE_META}/.${target_profile}.abi.XXXXXX")"
|
||||
{
|
||||
printf 'cache_epoch=%s\n' "${QUERY_REGRESSION_CACHE_EPOCH}"
|
||||
printf 'runner_image_digest=%s\n' "${RUNNER_IMAGE_DIGEST}"
|
||||
printf 'runner_image_epoch=%s\n' "${RUNNER_IMAGE_EPOCH}"
|
||||
printf 'host_triple=%s\n' "${host_triple}"
|
||||
printf 'architecture=%s\n' "$(uname -m)"
|
||||
printf 'cargo_profile=%s\n' "${CARGO_PROFILE}"
|
||||
printf 'target_profile=%s\n' "${target_profile}"
|
||||
printf 'RUSTFLAGS=%s\n' "${RUSTFLAGS-}"
|
||||
printf 'CARGO_ENCODED_RUSTFLAGS=%s\n' "${CARGO_ENCODED_RUSTFLAGS-}"
|
||||
printf 'CARGO_INCREMENTAL=%s\n' "${CARGO_INCREMENTAL-}"
|
||||
printf 'RUSTC_WRAPPER=%s\n' "${RUSTC_WRAPPER-}"
|
||||
printf 'SCCACHE_DIR=%s\n' "${SCCACHE_DIR-}"
|
||||
printf 'SCCACHE_CACHE_SIZE=%s\n' "${SCCACHE_CACHE_SIZE-}"
|
||||
printf 'RUSTUP_HOME=%s\n' "${RUSTUP_HOME-}"
|
||||
printf 'RUSTUP_TOOLCHAIN=%s\n' "${RUSTUP_TOOLCHAIN-}"
|
||||
printf 'RUSTUP_AUTO_INSTALL=%s\n' "${RUSTUP_AUTO_INSTALL-}"
|
||||
printf 'PATH=%s\n' "${PATH-}"
|
||||
printf '%s\n' "rustc_vV=${rustc_vv}"
|
||||
printf '%s\n' "cargo_vV=${cargo_vv}"
|
||||
printf '%s\n' "cc_version=${cc_version}"
|
||||
printf '%s\n' "mold_version=${mold_version}"
|
||||
printf '%s\n' "protoc_version=${protoc_version}"
|
||||
printf '%s\n' "sccache_version=${sccache_version}"
|
||||
printf '%s\n' "rustup_version=${rustup_version}"
|
||||
printf '%s\n' "active_toolchain=${active_toolchain}"
|
||||
printf '%s\n' "rustup_path=${rustup_path}"
|
||||
printf '%s\n' "cargo_path=${cargo_path}"
|
||||
printf '%s\n' "rustc_path=${rustc_path}"
|
||||
} > "${new_marker}"
|
||||
|
||||
if ! cmp -s "${marker}" "${new_marker}"; then
|
||||
printf 'Cache ABI marker mismatch; invalidating target profile: %s\n' "${target_profile}"
|
||||
mkdir -p "${profile_target_dir}"
|
||||
clear_directory "${profile_target_dir}"
|
||||
else
|
||||
printf 'Cache ABI marker match; reusing target profile: %s\n' "${target_profile}"
|
||||
fi
|
||||
mv -f "${new_marker}" "${marker}"
|
||||
|
||||
report_cache_usage
|
||||
|
||||
target_size="$(target_size_kib)"
|
||||
if (( target_size >= 400 * 1024 * 1024 )); then
|
||||
printf 'Warning: target cache is at least 400 GiB (%s KiB)\n' "${target_size}" >&2
|
||||
fi
|
||||
if (( target_size >= 450 * 1024 * 1024 )); then
|
||||
printf 'Target cache is at least 450 GiB; clearing complete target root\n' >&2
|
||||
clear_directory "${CARGO_TARGET_DIR}"
|
||||
fi
|
||||
|
||||
cargo_size="$(cargo_size_kib)"
|
||||
if (( cargo_size >= 60 * 1024 * 1024 )); then
|
||||
printf 'Warning: Cargo cache is at least 60 GiB (%s KiB)\n' "${cargo_size}" >&2
|
||||
fi
|
||||
if (( cargo_size >= 80 * 1024 * 1024 )); then
|
||||
printf 'Cargo cache is at least 80 GiB; removing extracted sources and checkouts\n' >&2
|
||||
clear_cargo_extracted_trees
|
||||
cargo_size="$(cargo_size_kib)"
|
||||
if (( cargo_size >= 80 * 1024 * 1024 )); then
|
||||
printf 'Cargo cache remains at least 80 GiB after cleanup (%s KiB)\n' "${cargo_size}" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
free_space="$(free_kib)"
|
||||
if (( free_space < 300 * 1024 * 1024 )); then
|
||||
printf 'Free space is below 300 GiB; clearing complete target root\n' >&2
|
||||
clear_directory "${CARGO_TARGET_DIR}"
|
||||
free_space="$(free_kib)"
|
||||
if (( free_space < 300 * 1024 * 1024 )); then
|
||||
printf 'Free space remains below 300 GiB; removing Cargo extracted sources and checkouts\n' >&2
|
||||
clear_cargo_extracted_trees
|
||||
free_space="$(free_kib)"
|
||||
if (( free_space < 300 * 1024 * 1024 )); then
|
||||
printf 'Free space remains below 300 GiB after cleanup (%s KiB)\n' "${free_space}" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
report_cache_usage
|
||||
|
||||
sccache --start-server
|
||||
sccache --zero-stats
|
||||
sccache --show-stats
|
||||
|
||||
- name: Build base greptime
|
||||
working-directory: src
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SECONDS=0
|
||||
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
|
||||
base_build_elapsed="${SECONDS}"
|
||||
printf 'Base greptime cargo build elapsed: %s seconds\n' "${base_build_elapsed}"
|
||||
printf -- '- Base greptime cargo build: %s seconds\n' "${base_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
sccache --show-stats
|
||||
sccache --zero-stats
|
||||
target_dir="${CARGO_PROFILE}"
|
||||
if [[ "${CARGO_PROFILE}" == "dev" ]]; then
|
||||
target_dir="debug"
|
||||
@@ -114,9 +437,18 @@ jobs:
|
||||
|
||||
- name: Switch source to candidate
|
||||
working-directory: src
|
||||
shell: bash
|
||||
run: |
|
||||
git fetch --no-tags --prune --depth=1 origin "${CANDIDATE_REF}"
|
||||
git reset --hard FETCH_HEAD
|
||||
set -euo pipefail
|
||||
[[ "${VERIFIED_CANDIDATE_SHA}" =~ ^[[:xdigit:]]{40}$ ]] || {
|
||||
printf 'Verified candidate SHA is missing or invalid.\n' >&2
|
||||
exit 1
|
||||
}
|
||||
git cat-file -e "${VERIFIED_CANDIDATE_SHA}^{commit}" || {
|
||||
printf 'Verified candidate SHA is unavailable in the local repository.\n' >&2
|
||||
exit 1
|
||||
}
|
||||
git reset --hard "${VERIFIED_CANDIDATE_SHA}"
|
||||
git clean -ffdx
|
||||
|
||||
- name: Test query regression runner lifecycle
|
||||
@@ -126,8 +458,14 @@ jobs:
|
||||
- name: Build candidate greptime and fixture generators
|
||||
working-directory: src
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SECONDS=0
|
||||
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
|
||||
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools
|
||||
candidate_build_elapsed="${SECONDS}"
|
||||
printf 'Candidate greptime and fixture generator cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
|
||||
printf -- '- Candidate greptime and fixture generator cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
sccache --show-stats
|
||||
target_dir="${CARGO_PROFILE}"
|
||||
if [[ "${CARGO_PROFILE}" == "dev" ]]; then
|
||||
target_dir="debug"
|
||||
@@ -192,6 +530,46 @@ jobs:
|
||||
if-no-files-found: warn
|
||||
retention-days: 7
|
||||
|
||||
- name: Report persistent cache usage
|
||||
if: ${{ always() }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
for root in \
|
||||
/home/runner/.cargo \
|
||||
/home/runner/.cargo/registry \
|
||||
/home/runner/.cargo/git \
|
||||
/opt/rustup \
|
||||
/home/runner/query-regression-target \
|
||||
/home/runner/query-regression-cache-meta \
|
||||
/home/runner/.cache/sccache; do
|
||||
if [[ -e "${root}" ]]; then
|
||||
du -sh -- "${root}"
|
||||
else
|
||||
printf 'Missing cache path (report only): %s\n' "${root}"
|
||||
fi
|
||||
done
|
||||
if [[ -e /opt/rustup ]]; then
|
||||
if [[ -w /opt/rustup ]]; then
|
||||
printf 'Immutable Rustup path is unexpectedly writable: /opt/rustup\n' >&2
|
||||
else
|
||||
printf 'Immutable Rustup path is read-only to runner: /opt/rustup\n'
|
||||
fi
|
||||
fi
|
||||
report_root=/home/runner/query-regression-target
|
||||
if [[ ! -e "${report_root}" ]]; then
|
||||
report_root=/home/runner
|
||||
printf 'Target cache path is missing; reporting its filesystem through %s\n' "${report_root}"
|
||||
fi
|
||||
df -P "${report_root}"
|
||||
df -hP "${report_root}"
|
||||
df -Pi "${report_root}"
|
||||
if command -v sccache >/dev/null 2>&1; then
|
||||
sccache --show-stats || printf 'Unable to show sccache statistics (report only)\n' >&2
|
||||
else
|
||||
printf 'sccache is unavailable (report only)\n' >&2
|
||||
fi
|
||||
|
||||
- name: Fail on regression failure
|
||||
if: ${{ steps.run.outputs.status != '0' }}
|
||||
run: exit 1
|
||||
|
||||
@@ -77,8 +77,10 @@ change-coupling points, and gotchas:
|
||||
2. `make clippy`
|
||||
3. `make test` (or `cargo nextest run`)
|
||||
4. `make check-udeps` (run `make fix-udeps` if it reports unused dependencies).
|
||||
5. If you changed sample config under `config/`, run `make config-docs` (needs
|
||||
Docker) and commit the regenerated `config/config.md`.
|
||||
5. If you added or changed a public configuration option, update the applicable
|
||||
example TOMLs, configuration-loading and serialized-config snapshot tests,
|
||||
and related user-facing documentation. Run `make config-docs` (needs Docker)
|
||||
and commit the regenerated `config/config.md`.
|
||||
6. If you changed a persisted or wire format, add a compatibility test case (see
|
||||
`.agents/architecture-invariants.md`).
|
||||
7. Use a conventional-commit title, sign off commits (`git commit -s`), and sign
|
||||
|
||||
Generated
+5
-1
@@ -948,11 +948,14 @@ dependencies = [
|
||||
"common-test-util",
|
||||
"digest 0.10.7",
|
||||
"hex",
|
||||
"hmac 0.12.1",
|
||||
"pbkdf2",
|
||||
"rand 0.9.4",
|
||||
"sha1 0.10.6",
|
||||
"sha2 0.10.9",
|
||||
"snafu 0.8.6",
|
||||
"sql",
|
||||
"stringprep",
|
||||
"subtle",
|
||||
"tokio",
|
||||
]
|
||||
@@ -5938,7 +5941,7 @@ dependencies = [
|
||||
[[package]]
|
||||
name = "greptime-proto"
|
||||
version = "0.1.0"
|
||||
source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=4a69c9e75240a76afeb0796798085813669288f8#4a69c9e75240a76afeb0796798085813669288f8"
|
||||
source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=c6f92ee9e12ee7bd5ee3c0956b1c9caa39be3f76#c6f92ee9e12ee7bd5ee3c0956b1c9caa39be3f76"
|
||||
dependencies = [
|
||||
"prost 0.14.1",
|
||||
"prost-types 0.14.1",
|
||||
@@ -11020,6 +11023,7 @@ dependencies = [
|
||||
"bytemuck",
|
||||
"common-error",
|
||||
"common-macro",
|
||||
"common-query",
|
||||
"common-recordbatch",
|
||||
"common-telemetry",
|
||||
"criterion 0.7.0",
|
||||
|
||||
+2
-1
@@ -158,7 +158,7 @@ fs2 = "0.4"
|
||||
fst = "0.4.7"
|
||||
futures = "0.3"
|
||||
futures-util = "0.3"
|
||||
greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "4a69c9e75240a76afeb0796798085813669288f8" }
|
||||
greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "c6f92ee9e12ee7bd5ee3c0956b1c9caa39be3f76" }
|
||||
hex = "0.4"
|
||||
http = "1"
|
||||
humantime = "2.1"
|
||||
@@ -242,6 +242,7 @@ sqlx = { version = "0.8", default-features = false, features = [
|
||||
"tls-rustls-aws-lc-rs",
|
||||
"rust_decimal",
|
||||
] }
|
||||
stringprep = "0.1"
|
||||
strum = { version = "0.27", features = ["derive"] }
|
||||
sysinfo = "0.33"
|
||||
tempfile = "3"
|
||||
|
||||
+7
-7
@@ -15,7 +15,7 @@
|
||||
| `default_timezone` | String | Unset | The default timezone of the server. |
|
||||
| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index and value columns. |
|
||||
| `auto_create_table` | Bool | `true` | Server-side global switch for auto table creation on write.<br/>When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. |
|
||||
| `user_provider` | String | Unset | The user provider for authentication.<br/>Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"<br/>Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",<br/>"mysql_native_password:<hex_sha1_sha1_password>"<br/>"pbkdf2_sha256" protects passwords at rest but is not compatible with mysql_native_password. |
|
||||
| `user_provider` | String | Unset | The user provider for authentication.<br/>Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"<br/>Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",<br/>"mysql_native_password:<hex_sha1_sha1_password>",<br/>"pg_scram_sha256:<iterations>:<hex_salt>:<hex_stored_key>:<hex_server_key>"<br/>"pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest.<br/>"mysql_native_password" is MySQL-specific and falls back to cleartext password auth for PostgreSQL.<br/>For PostgreSQL SCRAM, keep all users on the default iteration count (4096) and salt length (16);<br/>a non-default "pbkdf2_sha256"/"pg_scram_sha256" iteration count or salt length is observable in the<br/>SCRAM handshake and weakens resistance to username enumeration. |
|
||||
| `max_in_flight_write_bytes` | String | Unset | Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight).<br/>Set to 0 to disable the limit. Default: "0" (unlimited) |
|
||||
| `write_bytes_exhausted_policy` | String | Unset | Policy when write bytes quota is exhausted.<br/>Options: "wait" (default, 10s timeout), "wait(<duration>)" (e.g., "wait(30s)"), "fail" |
|
||||
| `init_regions_in_background` | Bool | `false` | Initialize all regions in the background during the startup.<br/>By default, it provides services after all regions have been initialized. |
|
||||
@@ -157,7 +157,8 @@
|
||||
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
|
||||
| `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
|
||||
| `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`. |
|
||||
| `region_engine.mito.sst_meta_cache_size` | String | Auto | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB. |
|
||||
| `region_engine.mito.default_region_write_buffer_size` | String | `0` | Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`. |
|
||||
| `region_engine.mito.sst_meta_cache_size` | String | Auto | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory with a max limitation of 512MB. |
|
||||
| `region_engine.mito.vector_cache_size` | String | Auto | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
|
||||
| `region_engine.mito.page_cache_size` | String | Auto | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory. |
|
||||
| `region_engine.mito.selector_result_cache_size` | String | Auto | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
|
||||
@@ -205,7 +206,6 @@
|
||||
| `region_engine.mito.bloom_filter_index.mem_threshold_on_create` | String | `auto` | Memory threshold for bloom filter creation.<br/>- `auto`: automatically determine the threshold based on the system memory size (default)<br/>- `unlimited`: no memory limit<br/>- `[size]` e.g. `64MB`: fixed memory threshold |
|
||||
| `region_engine.file` | -- | -- | Enable the file engine. |
|
||||
| `region_engine.metric` | -- | -- | Metric engine options. |
|
||||
| `region_engine.metric.sparse_primary_key_encoding` | Bool | `true` | Whether to use sparse primary key encoding. |
|
||||
| `logging` | -- | -- | The logging options. |
|
||||
| `logging.dir` | String | `./greptimedb_data/logs` | The directory to store the log files. If set to empty, logs will not be written to files. |
|
||||
| `logging.level` | String | Unset | The log level. Can be `info`/`debug`/`warn`/`error`. |
|
||||
@@ -239,7 +239,7 @@
|
||||
| `default_timezone` | String | Unset | The default timezone of the server. |
|
||||
| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index and value columns. |
|
||||
| `auto_create_table` | Bool | `true` | Server-side global switch for auto table creation on write.<br/>When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. |
|
||||
| `user_provider` | String | Unset | The user provider for authentication.<br/>Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"<br/>Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",<br/>"mysql_native_password:<hex_sha1_sha1_password>"<br/>"pbkdf2_sha256" protects passwords at rest but is not compatible with mysql_native_password. |
|
||||
| `user_provider` | String | Unset | The user provider for authentication.<br/>Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"<br/>Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",<br/>"mysql_native_password:<hex_sha1_sha1_password>",<br/>"pg_scram_sha256:<iterations>:<hex_salt>:<hex_stored_key>:<hex_server_key>"<br/>"pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest.<br/>"mysql_native_password" is MySQL-specific and falls back to cleartext password auth for PostgreSQL.<br/>For PostgreSQL SCRAM, keep all users on the default iteration count (4096) and salt length (16);<br/>a non-default "pbkdf2_sha256"/"pg_scram_sha256" iteration count or salt length is observable in the<br/>SCRAM handshake and weakens resistance to username enumeration. |
|
||||
| `max_in_flight_write_bytes` | String | Unset | Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight).<br/>Set to 0 to disable the limit. Default: "0" (unlimited) |
|
||||
| `write_bytes_exhausted_policy` | String | Unset | Policy when write bytes quota is exhausted.<br/>Options: "wait" (default, 10s timeout), "wait(<duration>)" (e.g., "wait(30s)"), "fail" |
|
||||
| `runtime` | -- | -- | The runtime options. |
|
||||
@@ -514,7 +514,7 @@
|
||||
| `wal.timeout` | String | `3s` | The timeout for kafka client.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.max_batch_bytes` | String | `1MB` | The max size of a single producer batch.<br/>Warning: Kafka has a default limit of 1MB per message in a topic.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.consumer_wait_timeout` | String | `100ms` | The consumer wait timeout.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.create_index` | Bool | `true` | Whether to enable WAL index creation.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.create_index` | Bool | `false` | Whether to enable WAL index creation.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.dump_index_interval` | String | `60s` | The interval for dumping WAL indexes.<br/>**It's only used when the provider is `kafka`**. |
|
||||
| `wal.overwrite_entry_start_id` | Bool | `false` | Ignore missing entries during read WAL.<br/>**It's only used when the provider is `kafka`**.<br/><br/>This option ensures that when Kafka messages are deleted, the system<br/>can still successfully replay memtable data without throwing an<br/>out-of-range error.<br/>However, enabling this option might lead to unexpected data loss,<br/>as the system will skip over missing entries instead of treating<br/>them as critical errors. |
|
||||
| `query` | -- | -- | The query engine options. |
|
||||
@@ -560,7 +560,8 @@
|
||||
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
|
||||
| `region_engine.mito.global_write_buffer_size` | String | Auto | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
|
||||
| `region_engine.mito.global_write_buffer_reject_size` | String | Auto | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
|
||||
| `region_engine.mito.sst_meta_cache_size` | String | Auto | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB. |
|
||||
| `region_engine.mito.default_region_write_buffer_size` | String | `0` | Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`. |
|
||||
| `region_engine.mito.sst_meta_cache_size` | String | Auto | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory with a max limitation of 512MB. |
|
||||
| `region_engine.mito.vector_cache_size` | String | Auto | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
|
||||
| `region_engine.mito.page_cache_size` | String | Auto | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory. |
|
||||
| `region_engine.mito.selector_result_cache_size` | String | Auto | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
|
||||
@@ -612,7 +613,6 @@
|
||||
| `region_engine.mito.gc.unknown_file_lingering_time` | String | `1d` | Lingering time before deleting unknown files (files with undetermined expel time).<br/>Only applies during full file listing GC.<br/>This uses the object's last-modified timestamp as a heuristic (strict less-than comparison);<br/>do not configure this value too small in production to avoid deleting pre-manifest files<br/>from in-progress compaction or flush.<br/>For active/open regions, an unknown file is deleted only if its object last-modified time exceeds this TTL.<br/>If the object store does not provide a last-modified timestamp, the file is conservatively kept.<br/>For dropped regions, unknown files are deleted immediately. |
|
||||
| `region_engine.file` | -- | -- | Enable the file engine. |
|
||||
| `region_engine.metric` | -- | -- | Metric engine options. |
|
||||
| `region_engine.metric.sparse_primary_key_encoding` | Bool | `true` | Whether to use sparse primary key encoding. |
|
||||
| `logging` | -- | -- | The logging options. |
|
||||
| `logging.dir` | String | `./greptimedb_data/logs` | The directory to store the log files. If set to empty, logs will not be written to files. |
|
||||
| `logging.level` | String | Unset | The log level. Can be `info`/`debug`/`warn`/`error`. |
|
||||
|
||||
@@ -185,7 +185,7 @@ consumer_wait_timeout = "100ms"
|
||||
|
||||
## Whether to enable WAL index creation.
|
||||
## **It's only used when the provider is `kafka`**.
|
||||
create_index = true
|
||||
create_index = false
|
||||
|
||||
## The interval for dumping WAL indexes.
|
||||
## **It's only used when the provider is `kafka`**.
|
||||
@@ -468,10 +468,13 @@ auto_flush_interval = "1h"
|
||||
## @toml2docs:none-default="Auto"
|
||||
#+ global_write_buffer_reject_size = "2GB"
|
||||
|
||||
## Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`.
|
||||
#+ default_region_write_buffer_size = "0"
|
||||
|
||||
## Cache size for SST metadata. Setting it to 0 to disable the cache.
|
||||
## If not set, it's default to 1/32 of OS memory with a max limitation of 128MB.
|
||||
## If not set, it's default to 1/8 of OS memory with a max limitation of 512MB.
|
||||
## @toml2docs:none-default="Auto"
|
||||
#+ sst_meta_cache_size = "128MB"
|
||||
#+ sst_meta_cache_size = "512MB"
|
||||
|
||||
## Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.
|
||||
## If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.
|
||||
@@ -697,8 +700,6 @@ unknown_file_lingering_time = "1d"
|
||||
[[region_engine]]
|
||||
## Metric engine options.
|
||||
[region_engine.metric]
|
||||
## Whether to use sparse primary key encoding.
|
||||
sparse_primary_key_encoding = true
|
||||
|
||||
## The logging options.
|
||||
[logging]
|
||||
|
||||
@@ -13,8 +13,13 @@ default_column_prefix = "greptime"
|
||||
## The user provider for authentication.
|
||||
## Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"
|
||||
## Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",
|
||||
## "mysql_native_password:<hex_sha1_sha1_password>"
|
||||
## "pbkdf2_sha256" protects passwords at rest but is not compatible with mysql_native_password.
|
||||
## "mysql_native_password:<hex_sha1_sha1_password>",
|
||||
## "pg_scram_sha256:<iterations>:<hex_salt>:<hex_stored_key>:<hex_server_key>"
|
||||
## "pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest.
|
||||
## "mysql_native_password" is MySQL-specific and falls back to cleartext password auth for PostgreSQL.
|
||||
## For PostgreSQL SCRAM, keep all users on the default iteration count (4096) and salt length (16);
|
||||
## a non-default "pbkdf2_sha256"/"pg_scram_sha256" iteration count or salt length is observable in the
|
||||
## SCRAM handshake and weakens resistance to username enumeration.
|
||||
## @toml2docs:none-default
|
||||
#+ user_provider = "static_user_provider:file:/path/to/users"
|
||||
|
||||
|
||||
@@ -13,8 +13,13 @@ default_column_prefix = "greptime"
|
||||
## The user provider for authentication.
|
||||
## Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"
|
||||
## Password verifier formats: "plain:<password>", "pbkdf2_sha256:<iterations>:<hex_salt>:<hex_hash>",
|
||||
## "mysql_native_password:<hex_sha1_sha1_password>"
|
||||
## "pbkdf2_sha256" protects passwords at rest but is not compatible with mysql_native_password.
|
||||
## "mysql_native_password:<hex_sha1_sha1_password>",
|
||||
## "pg_scram_sha256:<iterations>:<hex_salt>:<hex_stored_key>:<hex_server_key>"
|
||||
## "pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest.
|
||||
## "mysql_native_password" is MySQL-specific and falls back to cleartext password auth for PostgreSQL.
|
||||
## For PostgreSQL SCRAM, keep all users on the default iteration count (4096) and salt length (16);
|
||||
## a non-default "pbkdf2_sha256"/"pg_scram_sha256" iteration count or salt length is observable in the
|
||||
## SCRAM handshake and weakens resistance to username enumeration.
|
||||
## @toml2docs:none-default
|
||||
#+ user_provider = "static_user_provider:file:/path/to/users"
|
||||
|
||||
@@ -597,10 +602,13 @@ auto_flush_interval = "1h"
|
||||
## @toml2docs:none-default="Auto"
|
||||
#+ global_write_buffer_reject_size = "2GB"
|
||||
|
||||
## Default write buffer size for each region. Regions stall at this size and reject writes at twice this size. Setting it to 0 disables both limits unless the table specifies `write_buffer_size`.
|
||||
#+ default_region_write_buffer_size = "0"
|
||||
|
||||
## Cache size for SST metadata. Setting it to 0 to disable the cache.
|
||||
## If not set, it's default to 1/32 of OS memory with a max limitation of 128MB.
|
||||
## If not set, it's default to 1/8 of OS memory with a max limitation of 512MB.
|
||||
## @toml2docs:none-default="Auto"
|
||||
#+ sst_meta_cache_size = "128MB"
|
||||
#+ sst_meta_cache_size = "512MB"
|
||||
|
||||
## Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.
|
||||
## If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.
|
||||
@@ -807,8 +815,6 @@ mem_threshold_on_create = "auto"
|
||||
[[region_engine]]
|
||||
## Metric engine options.
|
||||
[region_engine.metric]
|
||||
## Whether to use sparse primary key encoding.
|
||||
sparse_primary_key_encoding = true
|
||||
|
||||
## The logging options.
|
||||
[logging]
|
||||
|
||||
+85
-25
@@ -20,7 +20,7 @@ use common_decimal::decimal128::{DECIMAL128_DEFAULT_SCALE, DECIMAL128_MAX_PRECIS
|
||||
use common_time::time::Time;
|
||||
use common_time::timestamp::TimeUnit;
|
||||
use common_time::{Date, IntervalDayTime, IntervalMonthDayNano, IntervalYearMonth, Timestamp};
|
||||
use datatypes::json::value::{JsonNumber, JsonValue, JsonValueRef, JsonVariant};
|
||||
use datatypes::json::value::{JsonNumber, JsonValue, JsonValueRef, JsonVariant, JsonVariantRef};
|
||||
use datatypes::prelude::{ConcreteDataType, ValueRef};
|
||||
use datatypes::types::json_type::JsonNativeType;
|
||||
use datatypes::types::{
|
||||
@@ -931,33 +931,59 @@ pub fn encode_json_value(value: JsonValue) -> v1::JsonValue {
|
||||
}
|
||||
|
||||
fn decode_json_value(value: &v1::JsonValue) -> JsonValueRef<'_> {
|
||||
let (variant, json_type) = decode_json_value_parts(value);
|
||||
JsonValueRef::new_with_type(variant, json_type)
|
||||
}
|
||||
|
||||
fn decode_json_value_parts(value: &v1::JsonValue) -> (JsonVariantRef<'_>, JsonNativeType) {
|
||||
let Some(value) = &value.value else {
|
||||
return JsonValueRef::null();
|
||||
return (JsonVariantRef::Null, JsonNativeType::Null);
|
||||
};
|
||||
match value {
|
||||
json_value::Value::Boolean(x) => (*x).into(),
|
||||
json_value::Value::Int(x) => (*x).into(),
|
||||
json_value::Value::Uint(x) => (*x).into(),
|
||||
json_value::Value::Float(x) => (*x).into(),
|
||||
json_value::Value::Str(x) => (x.as_str()).into(),
|
||||
json_value::Value::Array(array) => array
|
||||
.items
|
||||
.iter()
|
||||
.map(|x| decode_json_value(x).into_variant())
|
||||
.collect::<Vec<_>>()
|
||||
.into(),
|
||||
json_value::Value::Object(x) => x
|
||||
.entries
|
||||
.iter()
|
||||
.filter_map(|entry| {
|
||||
entry
|
||||
.value
|
||||
.as_ref()
|
||||
.map(|v| (entry.key.as_str(), decode_json_value(v).into_variant()))
|
||||
})
|
||||
.collect::<BTreeMap<_, _>>()
|
||||
.into(),
|
||||
json_value::Value::Variant(x) => x.as_slice().into(),
|
||||
json_value::Value::Boolean(x) => (JsonVariantRef::Bool(*x), JsonNativeType::Bool),
|
||||
json_value::Value::Int(x) => ((*x).into(), JsonNativeType::i64()),
|
||||
json_value::Value::Uint(x) => ((*x).into(), JsonNativeType::u64()),
|
||||
json_value::Value::Float(x) => ((*x).into(), JsonNativeType::f64()),
|
||||
json_value::Value::Str(x) => (x.as_str().into(), JsonNativeType::String),
|
||||
json_value::Value::Array(array) => {
|
||||
let mut variants = Vec::with_capacity(array.items.len());
|
||||
let mut item_type = JsonNativeType::Null;
|
||||
for item in &array.items {
|
||||
let (variant, ty) = decode_json_value_parts(item);
|
||||
variants.push(variant);
|
||||
if !matches!(item_type, JsonNativeType::Variant) {
|
||||
item_type.merge(&ty);
|
||||
}
|
||||
}
|
||||
(
|
||||
JsonVariantRef::Array(variants),
|
||||
JsonNativeType::Array(Box::new(item_type)),
|
||||
)
|
||||
}
|
||||
json_value::Value::Object(object) => {
|
||||
let mut variants = Vec::with_capacity(object.entries.len());
|
||||
let mut fields = Vec::with_capacity(object.entries.len());
|
||||
for entry in &object.entries {
|
||||
let Some(value) = &entry.value else {
|
||||
continue;
|
||||
};
|
||||
let (variant, json_type) = decode_json_value_parts(value);
|
||||
variants.push((entry.key.as_str(), variant));
|
||||
fields.push((entry.key.clone(), json_type));
|
||||
}
|
||||
let variants = variants.into_iter().collect::<BTreeMap<_, _>>();
|
||||
let fields = fields.into_iter().collect::<BTreeMap<_, _>>();
|
||||
let json_type = if fields.is_empty() {
|
||||
JsonNativeType::Null
|
||||
} else {
|
||||
JsonNativeType::Object(fields)
|
||||
};
|
||||
(JsonVariantRef::Object(variants), json_type)
|
||||
}
|
||||
json_value::Value::Variant(x) => (
|
||||
JsonVariantRef::Variant(x.as_slice()),
|
||||
JsonNativeType::Variant,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1854,15 +1880,40 @@ mod tests {
|
||||
]
|
||||
}))
|
||||
);
|
||||
assert_eq!(
|
||||
JsonNativeType::Array(Box::new(JsonNativeType::i64())),
|
||||
decode_json_value_parts(&proto).1
|
||||
);
|
||||
let value = decode_json_value(&proto);
|
||||
assert_eq!(json.as_ref(), value);
|
||||
|
||||
let proto = v1::JsonValue {
|
||||
value: Some(json_value::Value::Array(JsonList {
|
||||
items: vec![
|
||||
v1::JsonValue {
|
||||
value: Some(json_value::Value::Int(1)),
|
||||
},
|
||||
v1::JsonValue {
|
||||
value: Some(json_value::Value::Float(2.0)),
|
||||
},
|
||||
],
|
||||
})),
|
||||
};
|
||||
assert_eq!(
|
||||
JsonNativeType::Array(Box::new(JsonNativeType::f64())),
|
||||
decode_json_value_parts(&proto).1
|
||||
);
|
||||
|
||||
let json: JsonValue = [(); 0].into();
|
||||
let proto = encode_json_value(json.clone());
|
||||
assert_eq!(
|
||||
proto.value,
|
||||
Some(json_value::Value::Array(JsonList { items: vec![] }))
|
||||
);
|
||||
assert_eq!(
|
||||
JsonNativeType::Array(Box::new(JsonNativeType::Null)),
|
||||
decode_json_value_parts(&proto).1
|
||||
);
|
||||
let value = decode_json_value(&proto);
|
||||
assert_eq!(json.as_ref(), value);
|
||||
|
||||
@@ -1893,6 +1944,14 @@ mod tests {
|
||||
]
|
||||
}))
|
||||
);
|
||||
assert_eq!(
|
||||
JsonNativeType::Object(JsonObjectType::from([
|
||||
("k1".to_string(), JsonNativeType::i64()),
|
||||
("k2".to_string(), JsonNativeType::i64()),
|
||||
("k3".to_string(), JsonNativeType::i64()),
|
||||
])),
|
||||
decode_json_value_parts(&proto).1
|
||||
);
|
||||
let value = decode_json_value(&proto);
|
||||
assert_eq!(json.as_ref(), value);
|
||||
|
||||
@@ -1902,6 +1961,7 @@ mod tests {
|
||||
proto.value,
|
||||
Some(json_value::Value::Object(JsonObject { entries: vec![] }))
|
||||
);
|
||||
assert_eq!(JsonNativeType::Null, decode_json_value_parts(&proto).1);
|
||||
let value = decode_json_value(&proto);
|
||||
assert_eq!(json.as_ref(), value);
|
||||
|
||||
|
||||
@@ -21,11 +21,14 @@ common-macro.workspace = true
|
||||
common-telemetry.workspace = true
|
||||
digest = "0.10"
|
||||
hex.workspace = true
|
||||
hmac = "0.12"
|
||||
pbkdf2 = "0.12"
|
||||
rand.workspace = true
|
||||
sha1 = "0.10"
|
||||
sha2 = "0.10"
|
||||
snafu.workspace = true
|
||||
sql.workspace = true
|
||||
stringprep.workspace = true
|
||||
subtle = "2.6"
|
||||
tokio.workspace = true
|
||||
|
||||
|
||||
+275
-1
@@ -12,14 +12,16 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
use common_base::secrets::SecretString;
|
||||
use digest::Digest;
|
||||
use hmac::{Hmac, Mac};
|
||||
use pbkdf2::pbkdf2_hmac;
|
||||
use sha1::Sha1;
|
||||
use sha2::Sha256;
|
||||
use snafu::{OptionExt, ensure};
|
||||
use subtle::ConstantTimeEq;
|
||||
|
||||
use crate::error::{IllegalParamSnafu, InvalidConfigSnafu, Result, UserPasswordMismatchSnafu};
|
||||
use crate::user_info::DefaultUserInfo;
|
||||
@@ -31,9 +33,17 @@ use crate::{UserInfoRef, UserProviderRef};
|
||||
|
||||
pub(crate) const DEFAULT_USERNAME: &str = "greptime";
|
||||
pub const DEFAULT_PBKDF2_SHA256_ITERATIONS: u32 = 4096;
|
||||
pub const DEFAULT_PBKDF2_SHA256_SALT_LEN: usize = 16;
|
||||
pub const PBKDF2_SHA256_HASH_LEN: usize = 32;
|
||||
pub const MAX_PBKDF2_SHA256_ITERATIONS: u32 = 1_000_000;
|
||||
pub const MAX_PBKDF2_SHA256_SALT_LEN: usize = 1024;
|
||||
pub const PG_SCRAM_SHA256_KEY_LEN: usize = 32;
|
||||
|
||||
type HmacSha256 = Hmac<Sha256>;
|
||||
|
||||
/// Process-wide secret used to derive a stable mock salt for unknown users, so
|
||||
/// the SCRAM `server-first-message` can't be used to enumerate usernames.
|
||||
static PG_SCRAM_MOCK_SECRET: LazyLock<[u8; PG_SCRAM_SHA256_KEY_LEN]> = LazyLock::new(rand::random);
|
||||
|
||||
/// construct a [`UserInfo`](crate::user_info::UserInfo) impl with name
|
||||
/// use default username `greptime` if None is provided
|
||||
@@ -200,6 +210,212 @@ pub fn format_pbkdf2_sha256_password_verifier(
|
||||
))
|
||||
}
|
||||
|
||||
#[derive(Clone, PartialEq, Eq)]
|
||||
pub struct PgScramSha256Verifier {
|
||||
iterations: u32,
|
||||
salt: Vec<u8>,
|
||||
stored_key: Vec<u8>,
|
||||
server_key: Vec<u8>,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for PgScramSha256Verifier {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("PgScramSha256Verifier")
|
||||
.field("iterations", &self.iterations)
|
||||
.field("salt", &"<REDACTED>")
|
||||
.field("stored_key", &"<REDACTED>")
|
||||
.field("server_key", &"<REDACTED>")
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl PgScramSha256Verifier {
|
||||
pub fn new(
|
||||
iterations: u32,
|
||||
salt: Vec<u8>,
|
||||
stored_key: Vec<u8>,
|
||||
server_key: Vec<u8>,
|
||||
) -> Result<Self> {
|
||||
ensure!(
|
||||
iterations > 0 && iterations <= MAX_PBKDF2_SHA256_ITERATIONS,
|
||||
IllegalParamSnafu {
|
||||
msg: format!(
|
||||
"pg_scram_sha256 iterations must be in 1..={}",
|
||||
MAX_PBKDF2_SHA256_ITERATIONS
|
||||
)
|
||||
}
|
||||
);
|
||||
ensure!(
|
||||
!salt.is_empty() && salt.len() <= MAX_PBKDF2_SHA256_SALT_LEN,
|
||||
IllegalParamSnafu {
|
||||
msg: format!(
|
||||
"pg_scram_sha256 salt length must be in 1..={}",
|
||||
MAX_PBKDF2_SHA256_SALT_LEN
|
||||
)
|
||||
}
|
||||
);
|
||||
ensure!(
|
||||
stored_key.len() == PG_SCRAM_SHA256_KEY_LEN,
|
||||
IllegalParamSnafu {
|
||||
msg: "pg_scram_sha256 stored key must be 32 bytes"
|
||||
}
|
||||
);
|
||||
ensure!(
|
||||
server_key.len() == PG_SCRAM_SHA256_KEY_LEN,
|
||||
IllegalParamSnafu {
|
||||
msg: "pg_scram_sha256 server key must be 32 bytes"
|
||||
}
|
||||
);
|
||||
|
||||
Ok(Self {
|
||||
iterations,
|
||||
salt,
|
||||
stored_key,
|
||||
server_key,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn from_password(password: &[u8], salt: &[u8], iterations: u32) -> Result<Self> {
|
||||
let salted_password = pg_scram_sha256_salted_password(password, salt, iterations)?;
|
||||
Self::from_salted_password(salted_password, salt.to_vec(), iterations)
|
||||
}
|
||||
|
||||
pub fn from_salted_password(
|
||||
salted_password: Vec<u8>,
|
||||
salt: Vec<u8>,
|
||||
iterations: u32,
|
||||
) -> Result<Self> {
|
||||
ensure!(
|
||||
salted_password.len() == PG_SCRAM_SHA256_KEY_LEN,
|
||||
IllegalParamSnafu {
|
||||
msg: "pg_scram_sha256 salted password must be 32 bytes"
|
||||
}
|
||||
);
|
||||
let client_key = hmac_sha256(&salted_password, b"Client Key");
|
||||
let stored_key = sha256(&client_key);
|
||||
let server_key = hmac_sha256(&salted_password, b"Server Key");
|
||||
Self::new(iterations, salt, stored_key, server_key)
|
||||
}
|
||||
|
||||
/// Builds a throwaway verifier for an unknown user. The salt is derived
|
||||
/// deterministically from the username and a process-wide secret, so the
|
||||
/// SCRAM `server-first-message` (salt + iterations) stays stable per
|
||||
/// username and indistinguishable from a real user across reconnects. No
|
||||
/// PBKDF2 is run (avoids a CPU-exhaustion DoS keyed on unknown usernames),
|
||||
/// and the random keys guarantee the client proof never matches.
|
||||
pub fn mock_for_unknown_user(username: &[u8]) -> Self {
|
||||
let mock_salt = hmac_sha256(PG_SCRAM_MOCK_SECRET.as_slice(), username);
|
||||
Self {
|
||||
iterations: DEFAULT_PBKDF2_SHA256_ITERATIONS,
|
||||
salt: mock_salt[..DEFAULT_PBKDF2_SHA256_SALT_LEN].to_vec(),
|
||||
stored_key: rand::random::<[u8; PG_SCRAM_SHA256_KEY_LEN]>().to_vec(),
|
||||
server_key: rand::random::<[u8; PG_SCRAM_SHA256_KEY_LEN]>().to_vec(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn iterations(&self) -> u32 {
|
||||
self.iterations
|
||||
}
|
||||
|
||||
pub fn salt(&self) -> &[u8] {
|
||||
&self.salt
|
||||
}
|
||||
|
||||
pub fn verify_plain_password(&self, password: &[u8]) -> Result<bool> {
|
||||
let salted_password =
|
||||
pg_scram_sha256_salted_password(password, &self.salt, self.iterations)?;
|
||||
let client_key = hmac_sha256(&salted_password, b"Client Key");
|
||||
let stored_key = sha256(&client_key);
|
||||
Ok(self.stored_key.ct_eq(&stored_key).into())
|
||||
}
|
||||
|
||||
pub fn verify_client_proof(
|
||||
&self,
|
||||
auth_message: &[u8],
|
||||
client_proof: &[u8],
|
||||
) -> Result<Option<Vec<u8>>> {
|
||||
if client_proof.len() != PG_SCRAM_SHA256_KEY_LEN {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let client_signature = hmac_sha256(&self.stored_key, auth_message);
|
||||
let client_key = xor(client_proof, &client_signature);
|
||||
let stored_key = sha256(&client_key);
|
||||
if self.stored_key.ct_eq(&stored_key).into() {
|
||||
Ok(Some(hmac_sha256(&self.server_key, auth_message)))
|
||||
} else {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn format_pg_scram_sha256_password_verifier(
|
||||
password: &[u8],
|
||||
salt: &[u8],
|
||||
iterations: u32,
|
||||
) -> Result<String> {
|
||||
let verifier = PgScramSha256Verifier::from_password(password, salt, iterations)?;
|
||||
Ok(format!(
|
||||
"pg_scram_sha256:{iterations}:{}:{}:{}",
|
||||
hex::encode(verifier.salt),
|
||||
hex::encode(verifier.stored_key),
|
||||
hex::encode(verifier.server_key)
|
||||
))
|
||||
}
|
||||
|
||||
fn pg_scram_sha256_salted_password(
|
||||
password: &[u8],
|
||||
salt: &[u8],
|
||||
iterations: u32,
|
||||
) -> Result<Vec<u8>> {
|
||||
ensure!(
|
||||
iterations > 0 && iterations <= MAX_PBKDF2_SHA256_ITERATIONS,
|
||||
IllegalParamSnafu {
|
||||
msg: format!(
|
||||
"pg_scram_sha256 iterations must be in 1..={}",
|
||||
MAX_PBKDF2_SHA256_ITERATIONS
|
||||
)
|
||||
}
|
||||
);
|
||||
ensure!(
|
||||
!salt.is_empty() && salt.len() <= MAX_PBKDF2_SHA256_SALT_LEN,
|
||||
IllegalParamSnafu {
|
||||
msg: format!(
|
||||
"pg_scram_sha256 salt length must be in 1..={}",
|
||||
MAX_PBKDF2_SHA256_SALT_LEN
|
||||
)
|
||||
}
|
||||
);
|
||||
|
||||
let prepared_password = std::str::from_utf8(password)
|
||||
.ok()
|
||||
.and_then(|password| stringprep::saslprep(password).ok());
|
||||
let password = prepared_password
|
||||
.as_deref()
|
||||
.map(str::as_bytes)
|
||||
.unwrap_or(password);
|
||||
|
||||
let mut salted_password = [0u8; PG_SCRAM_SHA256_KEY_LEN];
|
||||
pbkdf2_hmac::<Sha256>(password, salt, iterations, &mut salted_password);
|
||||
Ok(salted_password.to_vec())
|
||||
}
|
||||
|
||||
fn hmac_sha256(key: &[u8], msg: &[u8]) -> Vec<u8> {
|
||||
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC accepts any key length");
|
||||
mac.update(msg);
|
||||
mac.finalize().into_bytes().to_vec()
|
||||
}
|
||||
|
||||
fn sha256(data: &[u8]) -> Vec<u8> {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(data);
|
||||
hasher.finalize().to_vec()
|
||||
}
|
||||
|
||||
fn xor(lhs: &[u8], rhs: &[u8]) -> Vec<u8> {
|
||||
lhs.iter().zip(rhs).map(|(l, r)| l ^ r).collect()
|
||||
}
|
||||
|
||||
fn sha1_two(input_1: &[u8], input_2: &[u8]) -> Vec<u8> {
|
||||
let mut hasher = Sha1::new();
|
||||
hasher.update(input_1);
|
||||
@@ -274,4 +490,62 @@ mod tests {
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_format_pg_scram_sha256_password_verifier() {
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier("password".as_bytes(), b"salt", 4096).unwrap();
|
||||
assert_eq!(
|
||||
"pg_scram_sha256:4096:73616c74:945e1c466fc9932efadc23781edc5d1e78d5e10f005933652af1a6105154f084:b9bf0e811b1fb6793671c0cc3adedf7c75cd72291191092ad65878c5a02aad2c",
|
||||
verifier
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pg_scram_sha256_applies_saslprep() {
|
||||
let normalized = PgScramSha256Verifier::from_password(b"pass word", b"salt", 4096).unwrap();
|
||||
let non_breaking_space =
|
||||
PgScramSha256Verifier::from_password("pass\u{00a0}word".as_bytes(), b"salt", 4096)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(normalized, non_breaking_space);
|
||||
assert!(
|
||||
non_breaking_space
|
||||
.verify_plain_password(b"pass word")
|
||||
.unwrap()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pg_scram_sha256_uses_original_bytes_when_saslprep_is_not_possible() {
|
||||
let invalid_utf8 = b"password\xff";
|
||||
let prohibited = b"password\x07";
|
||||
|
||||
for password in [invalid_utf8.as_slice(), prohibited.as_slice()] {
|
||||
let verifier = PgScramSha256Verifier::from_password(password, b"salt", 4096).unwrap();
|
||||
assert!(verifier.verify_plain_password(password).unwrap());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mock_verifier_is_stable_per_username() {
|
||||
let alice = PgScramSha256Verifier::mock_for_unknown_user(b"alice");
|
||||
let alice_again = PgScramSha256Verifier::mock_for_unknown_user(b"alice");
|
||||
let bob = PgScramSha256Verifier::mock_for_unknown_user(b"bob");
|
||||
|
||||
// Same username must yield the same salt and iterations across reconnects,
|
||||
// otherwise the SCRAM server-first message leaks user (non-)existence.
|
||||
assert_eq!(alice.salt(), alice_again.salt());
|
||||
assert_ne!(alice.salt(), bob.salt());
|
||||
assert_eq!(alice.iterations(), DEFAULT_PBKDF2_SHA256_ITERATIONS);
|
||||
assert_eq!(alice.salt().len(), DEFAULT_PBKDF2_SHA256_SALT_LEN);
|
||||
|
||||
// The mock verifier must never accept a client proof.
|
||||
assert!(
|
||||
alice
|
||||
.verify_client_proof(b"auth-message", &[0u8; PG_SCRAM_SHA256_KEY_LEN])
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+6
-5
@@ -22,16 +22,17 @@ mod user_provider;
|
||||
pub mod tests;
|
||||
|
||||
pub use common::{
|
||||
DEFAULT_PBKDF2_SHA256_ITERATIONS, HashedPassword, Identity, MAX_PBKDF2_SHA256_ITERATIONS,
|
||||
MAX_PBKDF2_SHA256_SALT_LEN, PBKDF2_SHA256_HASH_LEN, Password, auth_mysql,
|
||||
DEFAULT_PBKDF2_SHA256_ITERATIONS, DEFAULT_PBKDF2_SHA256_SALT_LEN, HashedPassword, Identity,
|
||||
MAX_PBKDF2_SHA256_ITERATIONS, MAX_PBKDF2_SHA256_SALT_LEN, PBKDF2_SHA256_HASH_LEN,
|
||||
PG_SCRAM_SHA256_KEY_LEN, Password, PgScramSha256Verifier, auth_mysql,
|
||||
format_mysql_native_password_verifier, format_pbkdf2_sha256_password_verifier,
|
||||
mysql_native_password_hash, static_user_provider_from_option, user_provider_from_option,
|
||||
userinfo_by_name,
|
||||
format_pg_scram_sha256_password_verifier, mysql_native_password_hash,
|
||||
static_user_provider_from_option, user_provider_from_option, userinfo_by_name,
|
||||
};
|
||||
pub use permission::{DefaultPermissionChecker, PermissionChecker, PermissionReq, PermissionResp};
|
||||
pub use user_info::UserInfo;
|
||||
pub use user_provider::UserProvider;
|
||||
pub use user_provider::static_user_provider::StaticUserProvider;
|
||||
pub use user_provider::{PgAuthInfo, UserProvider};
|
||||
|
||||
/// pub type alias
|
||||
pub type UserInfoRef = std::sync::Arc<dyn UserInfo>;
|
||||
|
||||
@@ -36,7 +36,11 @@ pub enum PermissionReq<'a> {
|
||||
PromStoreRead,
|
||||
Otlp,
|
||||
LogWrite,
|
||||
BulkInsert,
|
||||
BulkInsert {
|
||||
catalog: &'a str,
|
||||
schema: &'a str,
|
||||
table: &'a str,
|
||||
},
|
||||
}
|
||||
|
||||
impl<'a> PermissionReq<'a> {
|
||||
@@ -57,7 +61,7 @@ impl<'a> PermissionReq<'a> {
|
||||
| PermissionReq::PromStoreWrite
|
||||
| PermissionReq::Otlp
|
||||
| PermissionReq::LogWrite
|
||||
| PermissionReq::BulkInsert => false,
|
||||
| PermissionReq::BulkInsert { .. } => false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,6 +83,15 @@ pub trait PermissionChecker: Send + Sync {
|
||||
user_info: UserInfoRef,
|
||||
req: PermissionReq,
|
||||
) -> Result<PermissionResp>;
|
||||
|
||||
fn check_permission_with_context(
|
||||
&self,
|
||||
user_info: UserInfoRef,
|
||||
req: PermissionReq,
|
||||
_current_schema: Option<&str>,
|
||||
) -> Result<PermissionResp> {
|
||||
self.check_permission(user_info, req)
|
||||
}
|
||||
}
|
||||
|
||||
impl PermissionChecker for Option<&PermissionCheckerRef> {
|
||||
@@ -86,13 +99,24 @@ impl PermissionChecker for Option<&PermissionCheckerRef> {
|
||||
&self,
|
||||
user_info: UserInfoRef,
|
||||
req: PermissionReq,
|
||||
) -> Result<PermissionResp> {
|
||||
self.check_permission_with_context(user_info, req, None)
|
||||
}
|
||||
|
||||
fn check_permission_with_context(
|
||||
&self,
|
||||
user_info: UserInfoRef,
|
||||
req: PermissionReq,
|
||||
current_schema: Option<&str>,
|
||||
) -> Result<PermissionResp> {
|
||||
match self {
|
||||
Some(checker) => match checker.check_permission(user_info, req) {
|
||||
Ok(PermissionResp::Reject) => PermissionDeniedSnafu.fail(),
|
||||
Ok(PermissionResp::Allow) => Ok(PermissionResp::Allow),
|
||||
Err(e) => Err(e),
|
||||
},
|
||||
Some(checker) => {
|
||||
match checker.check_permission_with_context(user_info, req, current_schema) {
|
||||
Ok(PermissionResp::Reject) => PermissionDeniedSnafu.fail(),
|
||||
Ok(PermissionResp::Allow) => Ok(PermissionResp::Allow),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
None => Ok(PermissionResp::Allow),
|
||||
}
|
||||
}
|
||||
@@ -209,4 +233,15 @@ mod tests {
|
||||
|
||||
assert!(req.is_write());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_bulk_insert_is_write_request() {
|
||||
let req = PermissionReq::BulkInsert {
|
||||
catalog: "greptime",
|
||||
schema: "public",
|
||||
table: "metrics",
|
||||
};
|
||||
|
||||
assert!(req.is_write());
|
||||
}
|
||||
}
|
||||
|
||||
+336
-15
@@ -28,8 +28,9 @@ use snafu::{OptionExt, ResultExt, ensure};
|
||||
use subtle::ConstantTimeEq;
|
||||
|
||||
use crate::common::{
|
||||
Identity, MAX_PBKDF2_SHA256_ITERATIONS, MAX_PBKDF2_SHA256_SALT_LEN, PBKDF2_SHA256_HASH_LEN,
|
||||
Password, auth_mysql_with_hash_stage_2,
|
||||
DEFAULT_PBKDF2_SHA256_SALT_LEN, Identity, MAX_PBKDF2_SHA256_ITERATIONS,
|
||||
MAX_PBKDF2_SHA256_SALT_LEN, PBKDF2_SHA256_HASH_LEN, PG_SCRAM_SHA256_KEY_LEN, Password,
|
||||
PgScramSha256Verifier, auth_mysql_with_hash_stage_2,
|
||||
};
|
||||
use crate::error::{
|
||||
IllegalParamSnafu, InvalidConfigSnafu, IoSnafu, Result, UnsupportedPasswordTypeSnafu,
|
||||
@@ -64,15 +65,35 @@ pub trait UserProvider: Send + Sync {
|
||||
Ok(user_info)
|
||||
}
|
||||
|
||||
async fn postgres_auth_info(&self, _id: Identity<'_>) -> Result<PgAuthInfo> {
|
||||
Ok(PgAuthInfo::Cleartext)
|
||||
}
|
||||
|
||||
/// Returns whether this user provider implementation is backed by an external system.
|
||||
fn external(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, PartialEq, Eq)]
|
||||
pub enum PgAuthInfo {
|
||||
ScramSha256 {
|
||||
verifier: PgScramSha256Verifier,
|
||||
user_info: Option<UserInfoRef>,
|
||||
},
|
||||
Cleartext,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) enum PasswordVerifier {
|
||||
PlainText(String),
|
||||
PlainText {
|
||||
password: String,
|
||||
/// SCRAM verifier derived once at load time. Precomputing it keeps the
|
||||
/// Postgres SCRAM `server-first-message` (stable salt, fixed iterations)
|
||||
/// and per-connection cost indistinguishable from stored-hash and
|
||||
/// unknown users, instead of running PBKDF2 with a fresh salt on every
|
||||
/// connection.
|
||||
scram: PgScramSha256Verifier,
|
||||
},
|
||||
Pbkdf2Sha256 {
|
||||
iterations: u32,
|
||||
salt: Vec<u8>,
|
||||
@@ -81,12 +102,46 @@ pub(crate) enum PasswordVerifier {
|
||||
MysqlNativePassword {
|
||||
hash_stage_2: Vec<u8>,
|
||||
},
|
||||
PgScramSha256(PgScramSha256Verifier),
|
||||
}
|
||||
|
||||
impl PartialEq for PasswordVerifier {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
// The precomputed SCRAM verifier is a cache derived from the password, so
|
||||
// two plaintext verifiers are equal iff their passwords match.
|
||||
match (self, other) {
|
||||
(
|
||||
PasswordVerifier::PlainText { password: a, .. },
|
||||
PasswordVerifier::PlainText { password: b, .. },
|
||||
) => a == b,
|
||||
(
|
||||
PasswordVerifier::Pbkdf2Sha256 {
|
||||
iterations: i1,
|
||||
salt: s1,
|
||||
hash: h1,
|
||||
},
|
||||
PasswordVerifier::Pbkdf2Sha256 {
|
||||
iterations: i2,
|
||||
salt: s2,
|
||||
hash: h2,
|
||||
},
|
||||
) => i1 == i2 && s1 == s2 && h1 == h2,
|
||||
(
|
||||
PasswordVerifier::MysqlNativePassword { hash_stage_2: a },
|
||||
PasswordVerifier::MysqlNativePassword { hash_stage_2: b },
|
||||
) => a == b,
|
||||
(PasswordVerifier::PgScramSha256(a), PasswordVerifier::PgScramSha256(b)) => a == b,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for PasswordVerifier {}
|
||||
|
||||
impl fmt::Debug for PasswordVerifier {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
PasswordVerifier::PlainText(_) => {
|
||||
PasswordVerifier::PlainText { .. } => {
|
||||
f.debug_tuple("PlainText").field(&"<REDACTED>").finish()
|
||||
}
|
||||
PasswordVerifier::Pbkdf2Sha256 { iterations, .. } => f
|
||||
@@ -99,14 +154,33 @@ impl fmt::Debug for PasswordVerifier {
|
||||
.debug_struct("MysqlNativePassword")
|
||||
.field("hash_stage_2", &"<REDACTED>")
|
||||
.finish(),
|
||||
PasswordVerifier::PgScramSha256(_) => f
|
||||
.debug_struct("PgScramSha256")
|
||||
.field("verifier", &"<REDACTED>")
|
||||
.finish(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl PasswordVerifier {
|
||||
/// Builds a plaintext verifier, precomputing a stable SCRAM verifier so the
|
||||
/// Postgres SCRAM handshake for this user carries a stable salt and runs no
|
||||
/// per-connection PBKDF2. Uses [`DEFAULT_PBKDF2_SHA256_ITERATIONS`] to match
|
||||
/// the mock verifier handed to unknown users.
|
||||
fn plain_text(password: String) -> Option<Self> {
|
||||
let salt = rand::random::<[u8; DEFAULT_PBKDF2_SHA256_SALT_LEN]>();
|
||||
let scram = PgScramSha256Verifier::from_password(
|
||||
password.as_bytes(),
|
||||
&salt,
|
||||
crate::DEFAULT_PBKDF2_SHA256_ITERATIONS,
|
||||
)
|
||||
.ok()?;
|
||||
Some(Self::PlainText { password, scram })
|
||||
}
|
||||
|
||||
fn parse(input: &str) -> Option<Self> {
|
||||
if let Some(password) = input.strip_prefix("plain:") {
|
||||
return Some(Self::PlainText(password.to_string()));
|
||||
return Self::plain_text(password.to_string());
|
||||
}
|
||||
|
||||
if let Some(verifier) = input.strip_prefix("pbkdf2_sha256:") {
|
||||
@@ -140,14 +214,50 @@ impl PasswordVerifier {
|
||||
return Some(Self::MysqlNativePassword { hash_stage_2 });
|
||||
}
|
||||
|
||||
Some(Self::PlainText(input.to_string()))
|
||||
if let Some(verifier) = input.strip_prefix("pg_scram_sha256:") {
|
||||
let mut parts = verifier.split(':');
|
||||
let iterations = parts.next()?.parse::<u32>().ok()?;
|
||||
let salt = hex::decode(parts.next()?).ok()?;
|
||||
let stored_key = hex::decode(parts.next()?).ok()?;
|
||||
let server_key = hex::decode(parts.next()?).ok()?;
|
||||
if parts.next().is_some()
|
||||
|| stored_key.len() != PG_SCRAM_SHA256_KEY_LEN
|
||||
|| server_key.len() != PG_SCRAM_SHA256_KEY_LEN
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
return PgScramSha256Verifier::new(iterations, salt, stored_key, server_key)
|
||||
.ok()
|
||||
.map(Self::PgScramSha256);
|
||||
}
|
||||
|
||||
Self::plain_text(input.to_string())
|
||||
}
|
||||
|
||||
fn supports_pg_scram_sha256(&self) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
PasswordVerifier::PlainText { .. } | PasswordVerifier::PgScramSha256(_)
|
||||
)
|
||||
}
|
||||
|
||||
fn to_pg_scram_sha256_verifier(&self) -> Option<PgScramSha256Verifier> {
|
||||
match self {
|
||||
PasswordVerifier::PlainText { scram, .. } => Some(scram.clone()),
|
||||
// Legacy PBKDF2 verifiers were derived without SASLprep, and the
|
||||
// original password is unavailable to normalize them safely.
|
||||
PasswordVerifier::Pbkdf2Sha256 { .. } => None,
|
||||
PasswordVerifier::PgScramSha256(verifier) => Some(verifier.clone()),
|
||||
PasswordVerifier::MysqlNativePassword { .. } => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn verify_plain_text(&self, password: &str) -> bool {
|
||||
match self {
|
||||
PasswordVerifier::PlainText(expected) => {
|
||||
expected.as_bytes().ct_eq(password.as_bytes()).into()
|
||||
}
|
||||
PasswordVerifier::PlainText {
|
||||
password: expected, ..
|
||||
} => expected.as_bytes().ct_eq(password.as_bytes()).into(),
|
||||
PasswordVerifier::Pbkdf2Sha256 {
|
||||
iterations,
|
||||
salt,
|
||||
@@ -161,6 +271,9 @@ impl PasswordVerifier {
|
||||
hash.as_slice().ct_eq(&actual[..]).into()
|
||||
}
|
||||
PasswordVerifier::MysqlNativePassword { .. } => false,
|
||||
PasswordVerifier::PgScramSha256(verifier) => verifier
|
||||
.verify_plain_password(password.as_bytes())
|
||||
.unwrap_or(false),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -171,7 +284,7 @@ impl PasswordVerifier {
|
||||
username: &str,
|
||||
) -> Result<()> {
|
||||
match self {
|
||||
PasswordVerifier::PlainText(password) => {
|
||||
PasswordVerifier::PlainText { password, .. } => {
|
||||
auth_mysql(auth_data, salt, username, password.as_bytes())
|
||||
}
|
||||
PasswordVerifier::MysqlNativePassword { hash_stage_2 } => {
|
||||
@@ -181,6 +294,10 @@ impl PasswordVerifier {
|
||||
password_type: "mysql_native_password_with_pbkdf2_sha256_verifier",
|
||||
}
|
||||
.fail(),
|
||||
PasswordVerifier::PgScramSha256(_) => UnsupportedPasswordTypeSnafu {
|
||||
password_type: "mysql_native_password_with_pg_scram_sha256_verifier",
|
||||
}
|
||||
.fail(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -243,6 +360,7 @@ fn load_credential_from_file(filepath: &str) -> Result<UserInfoMap> {
|
||||
/// - `plain:<password>`
|
||||
/// - `pbkdf2_sha256:<iterations>:<hex-encoded-salt>:<hex-encoded-hash>`
|
||||
/// - `mysql_native_password:<hex-encoded-sha1-sha1-password>`
|
||||
/// - `pg_scram_sha256:<iterations>:<hex-encoded-salt>:<hex-encoded-stored-key>:<hex-encoded-server-key>`
|
||||
pub(crate) fn parse_credential_line(
|
||||
line: &str,
|
||||
) -> Option<(String, (PasswordVerifier, PermissionMode))> {
|
||||
@@ -263,6 +381,53 @@ pub(crate) fn parse_credential_line(
|
||||
Some((username.to_string(), (verifier, permission_mode)))
|
||||
}
|
||||
|
||||
pub(crate) fn postgres_auth_info_with_credential(
|
||||
users: &UserInfoMap,
|
||||
input_id: Identity<'_>,
|
||||
) -> Result<PgAuthInfo> {
|
||||
match input_id {
|
||||
Identity::UserId(username, _) => {
|
||||
ensure!(
|
||||
!username.is_empty(),
|
||||
IllegalParamSnafu {
|
||||
msg: "blank username"
|
||||
}
|
||||
);
|
||||
|
||||
if !users
|
||||
.values()
|
||||
.all(|(verifier, _)| verifier.supports_pg_scram_sha256())
|
||||
{
|
||||
// PostgreSQL chooses one auth method during startup. Selecting it
|
||||
// per username would expose user or verifier existence.
|
||||
return Ok(PgAuthInfo::Cleartext);
|
||||
}
|
||||
|
||||
if let Some((verifier, permission_mode)) = users.get(username) {
|
||||
if let Some(verifier) = verifier.to_pg_scram_sha256_verifier() {
|
||||
return Ok(PgAuthInfo::ScramSha256 {
|
||||
verifier,
|
||||
user_info: Some(DefaultUserInfo::with_name_and_permission(
|
||||
username,
|
||||
*permission_mode,
|
||||
)),
|
||||
});
|
||||
}
|
||||
|
||||
return Ok(PgAuthInfo::Cleartext);
|
||||
}
|
||||
|
||||
// Unknown user: hand back a deterministic mock verifier so the SCRAM
|
||||
// handshake is indistinguishable from a real user, without running
|
||||
// PBKDF2 or leaking existence through an unstable salt.
|
||||
Ok(PgAuthInfo::ScramSha256 {
|
||||
verifier: PgScramSha256Verifier::mock_for_unknown_user(username.as_bytes()),
|
||||
user_info: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn authenticate_with_credential(
|
||||
users: &UserInfoMap,
|
||||
input_id: Identity<'_>,
|
||||
@@ -317,10 +482,10 @@ mod tests {
|
||||
use sha1::Sha1;
|
||||
|
||||
use super::*;
|
||||
use crate::common::mysql_native_password_hash;
|
||||
use crate::common::{format_pg_scram_sha256_password_verifier, mysql_native_password_hash};
|
||||
|
||||
fn plain(password: &str) -> PasswordVerifier {
|
||||
PasswordVerifier::PlainText(password.to_string())
|
||||
PasswordVerifier::plain_text(password.to_string()).unwrap()
|
||||
}
|
||||
|
||||
fn sha1_one(data: &[u8]) -> Vec<u8> {
|
||||
@@ -484,6 +649,23 @@ mod tests {
|
||||
let result = parse_credential_line("user=mysql_native_password:abcd");
|
||||
assert_eq!(result, None);
|
||||
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier(b"password", b"salt", 4096).unwrap();
|
||||
let result = parse_credential_line(&format!("user={verifier}"));
|
||||
assert!(matches!(
|
||||
result,
|
||||
Some((
|
||||
_,
|
||||
(
|
||||
PasswordVerifier::PgScramSha256(PgScramSha256Verifier { .. }),
|
||||
PermissionMode::ReadWrite
|
||||
)
|
||||
))
|
||||
));
|
||||
|
||||
let result = parse_credential_line("user=pg_scram_sha256:4096:73616c74:abcd:abcd");
|
||||
assert_eq!(result, None);
|
||||
|
||||
// Invalid format - no equals sign
|
||||
let result = parse_credential_line("invalid_line");
|
||||
assert_eq!(result, None);
|
||||
@@ -541,7 +723,7 @@ mod tests {
|
||||
let users = HashMap::from([(
|
||||
"user".to_string(),
|
||||
(
|
||||
PasswordVerifier::PlainText(password.to_string()),
|
||||
PasswordVerifier::plain_text(password.to_string()).unwrap(),
|
||||
PermissionMode::default(),
|
||||
),
|
||||
)]);
|
||||
@@ -585,9 +767,148 @@ mod tests {
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_authenticate_with_pg_scram_sha256_verifier_plain_text() {
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier(b"password", b"salt", 4096).unwrap();
|
||||
let (_, user) = parse_credential_line(&format!("user={verifier}")).unwrap();
|
||||
let users = HashMap::from([("user".to_string(), user)]);
|
||||
|
||||
let result = authenticate_with_credential(
|
||||
&users,
|
||||
Identity::UserId("user", None),
|
||||
Password::PlainText("password".to_string().into()),
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
|
||||
let result = authenticate_with_credential(
|
||||
&users,
|
||||
Identity::UserId("user", None),
|
||||
Password::PlainText("wrong".to_string().into()),
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plain_text_scram_verifier_is_stable() {
|
||||
let verifier = PasswordVerifier::plain_text("password".to_string()).unwrap();
|
||||
let first = verifier.to_pg_scram_sha256_verifier().unwrap();
|
||||
let second = verifier.to_pg_scram_sha256_verifier().unwrap();
|
||||
|
||||
// A plaintext-backed user must present a stable salt and fixed iterations
|
||||
// across connections without re-running PBKDF2, otherwise the SCRAM
|
||||
// server-first message (and its timing) leaks that the user exists.
|
||||
assert_eq!(first.salt(), second.salt());
|
||||
assert_eq!(first.salt().len(), DEFAULT_PBKDF2_SHA256_SALT_LEN);
|
||||
assert_eq!(first.iterations(), crate::DEFAULT_PBKDF2_SHA256_ITERATIONS);
|
||||
|
||||
// The derived verifier must still accept the real password.
|
||||
assert!(first.verify_plain_password(b"password").unwrap());
|
||||
assert!(!first.verify_plain_password(b"wrong").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_postgres_auth_info_uses_scram_for_unknown_user() {
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier(b"password", b"salt", 4096).unwrap();
|
||||
let (_, user) = parse_credential_line(&format!("user={verifier}")).unwrap();
|
||||
let users = HashMap::from([("user".to_string(), user)]);
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("unknown", None)).unwrap();
|
||||
assert!(matches!(
|
||||
auth_info,
|
||||
PgAuthInfo::ScramSha256 {
|
||||
user_info: None,
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_postgres_auth_info_falls_back_to_cleartext() {
|
||||
let hash_stage_2 = mysql_native_password_hash("password".as_bytes());
|
||||
let users = HashMap::from([(
|
||||
"user".to_string(),
|
||||
(
|
||||
PasswordVerifier::MysqlNativePassword { hash_stage_2 },
|
||||
PermissionMode::default(),
|
||||
),
|
||||
)]);
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("user", None)).unwrap();
|
||||
assert!(matches!(auth_info, PgAuthInfo::Cleartext));
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("unknown", None)).unwrap();
|
||||
assert!(matches!(auth_info, PgAuthInfo::Cleartext));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_postgres_auth_info_with_pbkdf2_falls_back_to_cleartext() {
|
||||
let iterations = 4096;
|
||||
let salt = b"salt";
|
||||
let password = "pass\u{00a0}word";
|
||||
let mut hash = [0u8; PBKDF2_SHA256_HASH_LEN];
|
||||
pbkdf2_hmac::<Sha256>(password.as_bytes(), salt, iterations, &mut hash);
|
||||
let users = HashMap::from([(
|
||||
"user".to_string(),
|
||||
(
|
||||
PasswordVerifier::Pbkdf2Sha256 {
|
||||
iterations,
|
||||
salt: salt.to_vec(),
|
||||
hash: hash.to_vec(),
|
||||
},
|
||||
PermissionMode::default(),
|
||||
),
|
||||
)]);
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("user", None)).unwrap();
|
||||
assert!(matches!(auth_info, PgAuthInfo::Cleartext));
|
||||
|
||||
assert!(
|
||||
authenticate_with_credential(
|
||||
&users,
|
||||
Identity::UserId("user", None),
|
||||
Password::PlainText(password.to_string().into()),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_postgres_auth_info_mixed_verifiers_fall_back_to_cleartext() {
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier(b"password", b"salt", 4096).unwrap();
|
||||
let (_, scram_user) = parse_credential_line(&format!("scram={verifier}")).unwrap();
|
||||
let mysql_user = (
|
||||
PasswordVerifier::MysqlNativePassword {
|
||||
hash_stage_2: mysql_native_password_hash("password".as_bytes()),
|
||||
},
|
||||
PermissionMode::default(),
|
||||
);
|
||||
let users = HashMap::from([
|
||||
("scram".to_string(), scram_user),
|
||||
("mysql".to_string(), mysql_user),
|
||||
]);
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("scram", None)).unwrap();
|
||||
assert!(matches!(auth_info, PgAuthInfo::Cleartext));
|
||||
|
||||
let auth_info =
|
||||
postgres_auth_info_with_credential(&users, Identity::UserId("unknown", None)).unwrap();
|
||||
assert!(matches!(auth_info, PgAuthInfo::Cleartext));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_password_verifier_debug_redacts_secrets() {
|
||||
let debug = format!("{:?}", PasswordVerifier::PlainText("secret".to_string()));
|
||||
let debug = format!(
|
||||
"{:?}",
|
||||
PasswordVerifier::plain_text("secret".to_string()).unwrap()
|
||||
);
|
||||
assert!(debug.contains("<REDACTED>"));
|
||||
assert!(!debug.contains("secret"));
|
||||
|
||||
|
||||
@@ -17,7 +17,8 @@ use snafu::OptionExt;
|
||||
|
||||
use crate::error::{InvalidConfigSnafu, Result};
|
||||
use crate::user_provider::{
|
||||
UserInfoMap, authenticate_with_credential, load_credential_from_file, parse_credential_line,
|
||||
PgAuthInfo, UserInfoMap, authenticate_with_credential, load_credential_from_file,
|
||||
parse_credential_line, postgres_auth_info_with_credential,
|
||||
};
|
||||
use crate::{Identity, Password, UserInfoRef, UserProvider};
|
||||
|
||||
@@ -67,6 +68,10 @@ impl UserProvider for StaticUserProvider {
|
||||
authenticate_with_credential(&self.users, id, pwd)
|
||||
}
|
||||
|
||||
async fn postgres_auth_info(&self, id: Identity<'_>) -> Result<PgAuthInfo> {
|
||||
postgres_auth_info_with_credential(&self.users, id)
|
||||
}
|
||||
|
||||
async fn authorize(
|
||||
&self,
|
||||
_catalog: &str,
|
||||
|
||||
@@ -20,7 +20,10 @@ use common_telemetry::{info, warn};
|
||||
use snafu::ResultExt;
|
||||
|
||||
use crate::error::{FileWatchSnafu, Result};
|
||||
use crate::user_provider::{UserInfoMap, authenticate_with_credential, load_credential_from_file};
|
||||
use crate::user_provider::{
|
||||
PgAuthInfo, UserInfoMap, authenticate_with_credential, load_credential_from_file,
|
||||
postgres_auth_info_with_credential,
|
||||
};
|
||||
use crate::{Identity, Password, UserInfoRef, UserProvider};
|
||||
|
||||
pub(crate) const WATCH_FILE_USER_PROVIDER: &str = "watch_file_user_provider";
|
||||
@@ -83,6 +86,11 @@ impl UserProvider for WatchFileUserProvider {
|
||||
authenticate_with_credential(&users, id, password)
|
||||
}
|
||||
|
||||
async fn postgres_auth_info(&self, id: Identity<'_>) -> Result<PgAuthInfo> {
|
||||
let users = self.users.lock().expect("users credential must be valid");
|
||||
postgres_auth_info_with_credential(&users, id)
|
||||
}
|
||||
|
||||
async fn authorize(&self, _: &str, _: &str, _: &UserInfoRef) -> Result<()> {
|
||||
// default allow all
|
||||
Ok(())
|
||||
|
||||
+40
-5
@@ -17,8 +17,9 @@
|
||||
use std::io;
|
||||
|
||||
use auth::{
|
||||
DEFAULT_PBKDF2_SHA256_ITERATIONS, MAX_PBKDF2_SHA256_SALT_LEN,
|
||||
DEFAULT_PBKDF2_SHA256_ITERATIONS, DEFAULT_PBKDF2_SHA256_SALT_LEN, MAX_PBKDF2_SHA256_SALT_LEN,
|
||||
format_mysql_native_password_verifier, format_pbkdf2_sha256_password_verifier,
|
||||
format_pg_scram_sha256_password_verifier,
|
||||
};
|
||||
use clap::{ArgGroup, Parser, Subcommand, ValueEnum};
|
||||
use rand::RngCore;
|
||||
@@ -65,15 +66,15 @@ pub struct HashPasswordCommand {
|
||||
#[clap(long)]
|
||||
password_stdin: bool,
|
||||
|
||||
/// PBKDF2-SHA256 iteration count.
|
||||
/// PBKDF2-SHA256 / SCRAM-SHA-256 iteration count.
|
||||
#[clap(long, default_value_t = DEFAULT_PBKDF2_SHA256_ITERATIONS)]
|
||||
iterations: u32,
|
||||
|
||||
/// PBKDF2-SHA256 random salt length in bytes.
|
||||
#[clap(long, default_value_t = 16)]
|
||||
/// PBKDF2-SHA256 / SCRAM-SHA-256 random salt length in bytes.
|
||||
#[clap(long, default_value_t = DEFAULT_PBKDF2_SHA256_SALT_LEN)]
|
||||
salt_len: usize,
|
||||
|
||||
/// PBKDF2-SHA256 salt as hex. Mainly useful for deterministic automation.
|
||||
/// PBKDF2-SHA256 / SCRAM-SHA-256 salt as hex. Mainly useful for deterministic automation.
|
||||
#[clap(long)]
|
||||
salt_hex: Option<String>,
|
||||
}
|
||||
@@ -83,6 +84,7 @@ pub struct HashPasswordCommand {
|
||||
enum PasswordFormat {
|
||||
Pbkdf2Sha256,
|
||||
MysqlNativePassword,
|
||||
PgScramSha256,
|
||||
}
|
||||
|
||||
impl HashPasswordCommand {
|
||||
@@ -98,6 +100,16 @@ impl HashPasswordCommand {
|
||||
PasswordFormat::MysqlNativePassword => {
|
||||
format_mysql_native_password_verifier(password.as_bytes())
|
||||
}
|
||||
PasswordFormat::PgScramSha256 => {
|
||||
let salt = self.pbkdf2_salt()?;
|
||||
format_pg_scram_sha256_password_verifier(
|
||||
password.as_bytes(),
|
||||
&salt,
|
||||
self.iterations,
|
||||
)
|
||||
.map_err(common_error::ext::BoxedError::new)
|
||||
.context(error::OtherSnafu)?
|
||||
}
|
||||
};
|
||||
|
||||
println!("{verifier}");
|
||||
@@ -186,6 +198,29 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_hash_password_command_with_pg_scram_sha256() {
|
||||
let cmd = HashPasswordCommand {
|
||||
format: PasswordFormat::PgScramSha256,
|
||||
password: Some("password".to_string()),
|
||||
password_stdin: false,
|
||||
iterations: 4096,
|
||||
salt_len: 16,
|
||||
salt_hex: Some("73616c74".to_string()),
|
||||
};
|
||||
|
||||
let password = cmd.read_password().unwrap();
|
||||
let salt = cmd.pbkdf2_salt().unwrap();
|
||||
let verifier =
|
||||
format_pg_scram_sha256_password_verifier(password.as_bytes(), &salt, cmd.iterations)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
"pg_scram_sha256:4096:73616c74:945e1c466fc9932efadc23781edc5d1e78d5e10f005933652af1a6105154f084:b9bf0e811b1fb6793671c0cc3adedf7c75cd72291191092ad65878c5a02aad2c",
|
||||
verifier
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_hash_password_command_with_mysql_native_password() {
|
||||
let verifier = format_mysql_native_password_verifier("123456".as_bytes());
|
||||
|
||||
@@ -12,13 +12,16 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::io::Write;
|
||||
use std::time::Duration;
|
||||
|
||||
use cmd::options::GreptimeOptions;
|
||||
use common_base::memory_limit::MemoryLimit;
|
||||
use common_base::readable_size::ReadableSize;
|
||||
use common_config::{Configurable, DEFAULT_DATA_HOME, ENV_VAR_SEP};
|
||||
use common_options::datanode::{ClientOptions, DatanodeClientOptions};
|
||||
use common_telemetry::logging::{DEFAULT_LOGGING_DIR, DEFAULT_OTLP_HTTP_ENDPOINT, LoggingOptions};
|
||||
use common_test_util::temp_dir::create_named_temp_file;
|
||||
use common_wal::config::DatanodeWalConfig;
|
||||
use common_wal::config::raft_engine::RaftEngineConfig;
|
||||
use datanode::config::{DatanodeOptions, RegionEngineConfig, StorageConfig};
|
||||
@@ -90,13 +93,13 @@ fn test_load_datanode_example_config() {
|
||||
region_engine: vec![
|
||||
RegionEngineConfig::Mito(MitoConfig {
|
||||
auto_flush_interval: Duration::from_secs(3600),
|
||||
default_region_write_buffer_size: ReadableSize::mb(0),
|
||||
write_cache_ttl: Some(Duration::from_secs(60 * 60 * 8)),
|
||||
scan_memory_limit: MemoryLimit::Unlimited,
|
||||
..Default::default()
|
||||
}),
|
||||
RegionEngineConfig::File(FileEngineConfig {}),
|
||||
RegionEngineConfig::Metric(MetricEngineConfig {
|
||||
sparse_primary_key_encoding: true,
|
||||
flush_metadata_region_interval: Duration::from_secs(30),
|
||||
}),
|
||||
],
|
||||
@@ -299,13 +302,13 @@ fn test_load_standalone_example_config() {
|
||||
region_engine: vec![
|
||||
RegionEngineConfig::Mito(MitoConfig {
|
||||
auto_flush_interval: Duration::from_secs(3600),
|
||||
default_region_write_buffer_size: ReadableSize::mb(0),
|
||||
write_cache_ttl: Some(Duration::from_secs(60 * 60 * 8)),
|
||||
scan_memory_limit: MemoryLimit::Unlimited,
|
||||
..Default::default()
|
||||
}),
|
||||
RegionEngineConfig::File(FileEngineConfig {}),
|
||||
RegionEngineConfig::Metric(MetricEngineConfig {
|
||||
sparse_primary_key_encoding: true,
|
||||
flush_metadata_region_interval: Duration::from_secs(30),
|
||||
}),
|
||||
],
|
||||
@@ -383,3 +386,42 @@ fn test_load_heartbeat_env_vars_from_env() {
|
||||
similar_asserts::assert_eq!(standalone.component.heartbeat_env_vars, expected);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_load_metric_config_with_removed_sparse_primary_key_encoding() {
|
||||
// The `sparse_primary_key_encoding` option was removed from the metric
|
||||
// engine config and is now always-on. Existing user configs that still set
|
||||
// it must be tolerated (ignored), not rejected.
|
||||
let mut file = create_named_temp_file();
|
||||
let toml_str = r#"
|
||||
node_id = 42
|
||||
|
||||
[meta_client]
|
||||
metasrv_addrs = ["127.0.0.1:3002"]
|
||||
|
||||
[[region_engine]]
|
||||
[region_engine.metric]
|
||||
sparse_primary_key_encoding = false
|
||||
flush_metadata_region_interval = "30s"
|
||||
"#;
|
||||
write!(file, "{}", toml_str).unwrap();
|
||||
|
||||
let options =
|
||||
GreptimeOptions::<DatanodeOptions>::load_layered_options(file.path().to_str(), "")
|
||||
.expect("config with removed 'sparse_primary_key_encoding' should load without error");
|
||||
|
||||
// The unknown option is ignored; known metric engine options are still parsed/applied.
|
||||
let metric = options
|
||||
.component
|
||||
.region_engine
|
||||
.iter()
|
||||
.find_map(|c| match c {
|
||||
RegionEngineConfig::Metric(c) => Some(c),
|
||||
_ => None,
|
||||
})
|
||||
.expect("metric engine config should be present");
|
||||
assert_eq!(
|
||||
metric.flush_metadata_region_interval,
|
||||
Duration::from_secs(30)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -179,6 +179,7 @@ impl DropTableProcedure {
|
||||
&self.context.node_manager,
|
||||
&self.context.leader_region_registry,
|
||||
&self.data.physical_region_routes,
|
||||
true,
|
||||
)
|
||||
.await?;
|
||||
self.context
|
||||
|
||||
@@ -15,8 +15,8 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use api::v1::region::{
|
||||
CloseRequest as PbCloseRegionRequest, DropRequest as PbDropRegionRequest, RegionRequest,
|
||||
RegionRequestHeader, region_request,
|
||||
CleanUpRequest as PbCleanUpRequest, CloseRequest as PbCloseRegionRequest,
|
||||
DropRequest as PbDropRegionRequest, RegionRequest, RegionRequestHeader, region_request,
|
||||
};
|
||||
use common_error::ext::ErrorExt;
|
||||
use common_error::status_code::StatusCode;
|
||||
@@ -26,12 +26,14 @@ use common_wal::options::WalOptions;
|
||||
use futures::future::join_all;
|
||||
use snafu::ensure;
|
||||
use store_api::storage::{RegionId, RegionNumber};
|
||||
use table::metadata::TableId;
|
||||
use table::metadata::{TableId, TableInfo};
|
||||
use table::table_name::TableName;
|
||||
|
||||
use crate::cache_invalidator::Context;
|
||||
use crate::ddl::DdlContext;
|
||||
use crate::ddl::utils::{add_peer_context_if_needed, convert_region_routes_to_detecting_regions};
|
||||
use crate::ddl::utils::{
|
||||
add_peer_context_if_needed, convert_region_routes_to_detecting_regions, region_storage_path,
|
||||
};
|
||||
use crate::ddl::{CreateRequestBuilder, DdlContext, build_template_from_raw_table_info};
|
||||
use crate::error::{self, Result};
|
||||
use crate::instruction::CacheIdent;
|
||||
use crate::key::table_name::TableNameKey;
|
||||
@@ -313,6 +315,7 @@ impl DropTableExecutor {
|
||||
}),
|
||||
body: Some(region_request::Body::Close(PbCloseRegionRequest {
|
||||
region_id: region_id.as_u64(),
|
||||
flush_on_close: false,
|
||||
})),
|
||||
};
|
||||
|
||||
@@ -347,43 +350,53 @@ impl DropTableExecutor {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Closes all table regions on datanodes without deleting region files or metadata tombstones.
|
||||
pub async fn on_close_regions(
|
||||
/// Cleans leader regions on datanodes without reopening them as live regions.
|
||||
pub async fn on_cleanup_regions_offline(
|
||||
&self,
|
||||
node_manager: &NodeManagerRef,
|
||||
leader_region_registry: &LeaderRegionRegistryRef,
|
||||
table_info: &TableInfo,
|
||||
region_routes: &[RegionRoute],
|
||||
region_wal_options: &HashMap<RegionNumber, WalOptions>,
|
||||
) -> Result<()> {
|
||||
let table_id = self.table_id;
|
||||
let mut seen_peer_ids = HashSet::new();
|
||||
let peers = find_leaders(region_routes)
|
||||
.into_iter()
|
||||
.chain(find_followers(region_routes))
|
||||
.filter(|peer| seen_peer_ids.insert(peer.id));
|
||||
let mut close_region_tasks = Vec::new();
|
||||
let template = build_template_from_raw_table_info(table_info)?;
|
||||
let builder = CreateRequestBuilder::new(template, None);
|
||||
let storage_path = region_storage_path(&self.table.catalog_name, &self.table.schema_name);
|
||||
|
||||
for datanode in peers {
|
||||
let leaders = find_leaders(region_routes);
|
||||
let mut cleanup_region_tasks = Vec::with_capacity(leaders.len());
|
||||
let table_id = self.table_id;
|
||||
for datanode in leaders {
|
||||
let requester = node_manager.datanode(&datanode).await;
|
||||
let region_ids = find_leader_regions(region_routes, &datanode)
|
||||
.into_iter()
|
||||
.chain(find_follower_regions(region_routes, &datanode))
|
||||
.map(|region_number| RegionId::new(table_id, region_number));
|
||||
let regions = find_leader_regions(region_routes, &datanode);
|
||||
let region_ids = regions
|
||||
.iter()
|
||||
.map(|region_number| RegionId::new(table_id, *region_number))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
for region_id in region_ids {
|
||||
debug!("Closing region {region_id} on Datanode {datanode:?}");
|
||||
debug!("Cleaning region {region_id} offline on Datanode {datanode:?}");
|
||||
let create_request = builder.build_one(
|
||||
region_id,
|
||||
storage_path.clone(),
|
||||
region_wal_options,
|
||||
&HashMap::new(),
|
||||
)?;
|
||||
let request = RegionRequest {
|
||||
header: Some(RegionRequestHeader {
|
||||
tracing_context: TracingContext::from_current_span().to_w3c(),
|
||||
..Default::default()
|
||||
}),
|
||||
body: Some(region_request::Body::Close(PbCloseRegionRequest {
|
||||
region_id: region_id.as_u64(),
|
||||
body: Some(region_request::Body::CleanUp(PbCleanUpRequest {
|
||||
region_id: create_request.region_id,
|
||||
engine: create_request.engine,
|
||||
path: create_request.path,
|
||||
options: create_request.options,
|
||||
})),
|
||||
};
|
||||
|
||||
let datanode = datanode.clone();
|
||||
let requester = requester.clone();
|
||||
close_region_tasks.push(async move {
|
||||
cleanup_region_tasks.push(async move {
|
||||
if let Err(err) = requester.handle(request).await
|
||||
&& err.status_code() != StatusCode::RegionNotFound
|
||||
{
|
||||
@@ -394,6 +407,84 @@ impl DropTableExecutor {
|
||||
}
|
||||
}
|
||||
|
||||
join_all(cleanup_region_tasks)
|
||||
.await
|
||||
.into_iter()
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
|
||||
let region_ids = operating_leader_regions(region_routes);
|
||||
leader_region_registry.batch_delete(region_ids.into_iter().map(|(region_id, _)| region_id));
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Closes all table regions on datanodes without deleting region files or metadata tombstones.
|
||||
/// When `flush_leaders_on_close` is set, only leader regions are flushed before close.
|
||||
pub async fn on_close_regions(
|
||||
&self,
|
||||
node_manager: &NodeManagerRef,
|
||||
leader_region_registry: &LeaderRegionRegistryRef,
|
||||
region_routes: &[RegionRoute],
|
||||
flush_leaders_on_close: bool,
|
||||
) -> Result<()> {
|
||||
let table_id = self.table_id;
|
||||
let mut seen_peer_ids = HashSet::new();
|
||||
let peers = find_leaders(region_routes)
|
||||
.into_iter()
|
||||
.chain(find_followers(region_routes))
|
||||
.filter(|peer| seen_peer_ids.insert(peer.id));
|
||||
let close_region_tasks = peers.map(|datanode| {
|
||||
let region_ids = find_leader_regions(region_routes, &datanode)
|
||||
.into_iter()
|
||||
.map(|region_number| {
|
||||
(
|
||||
RegionId::new(table_id, region_number),
|
||||
flush_leaders_on_close,
|
||||
)
|
||||
})
|
||||
.chain(
|
||||
find_follower_regions(region_routes, &datanode)
|
||||
.into_iter()
|
||||
.map(|region_number| (RegionId::new(table_id, region_number), false)),
|
||||
)
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
async move {
|
||||
let requester = node_manager.datanode(&datanode).await;
|
||||
let close_region_tasks =
|
||||
region_ids.into_iter().map(|(region_id, flush_on_close)| {
|
||||
debug!("Closing region {region_id} on Datanode {datanode:?}");
|
||||
let request = RegionRequest {
|
||||
header: Some(RegionRequestHeader {
|
||||
tracing_context: TracingContext::from_current_span().to_w3c(),
|
||||
..Default::default()
|
||||
}),
|
||||
body: Some(region_request::Body::Close(PbCloseRegionRequest {
|
||||
region_id: region_id.as_u64(),
|
||||
flush_on_close,
|
||||
})),
|
||||
};
|
||||
|
||||
let datanode = datanode.clone();
|
||||
let requester = requester.clone();
|
||||
async move {
|
||||
if let Err(err) = requester.handle(request).await
|
||||
&& err.status_code() != StatusCode::RegionNotFound
|
||||
{
|
||||
return Err(add_peer_context_if_needed(datanode)(err));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
});
|
||||
|
||||
join_all(close_region_tasks)
|
||||
.await
|
||||
.into_iter()
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
Ok(())
|
||||
}
|
||||
});
|
||||
|
||||
join_all(close_region_tasks)
|
||||
.await
|
||||
.into_iter()
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use common_catalog::consts::FILE_ENGINE;
|
||||
use common_catalog::format_full_table_name;
|
||||
use snafu::{OptionExt, ensure};
|
||||
use store_api::metric_engine_consts::METRIC_ENGINE_NAME;
|
||||
@@ -61,6 +62,12 @@ impl DropTableProcedure {
|
||||
|
||||
if physical_table_id == self.data.table_id() {
|
||||
let engine = table_info_value.table_info.meta.engine;
|
||||
ensure!(
|
||||
!(self.context.soft_drop_enabled && engine == FILE_ENGINE),
|
||||
error::UnsupportedSnafu {
|
||||
operation: "soft-dropping file-engine tables".to_string()
|
||||
}
|
||||
);
|
||||
// rollback only if dropping the metric physical table fails
|
||||
self.data.allow_rollback = engine.as_str() == METRIC_ENGINE_NAME;
|
||||
|
||||
|
||||
@@ -29,7 +29,6 @@ use table::table_name::TableName;
|
||||
|
||||
use crate::ddl::DdlContext;
|
||||
use crate::ddl::drop_table::executor::DropTableExecutor;
|
||||
use crate::ddl::undrop_table::open_regions_ignore_region_not_found;
|
||||
use crate::ddl::utils::{
|
||||
convert_region_routes_to_detecting_regions, is_metric_engine_logical_table,
|
||||
map_to_procedure_error,
|
||||
@@ -84,22 +83,6 @@ impl PurgeDroppedTableProcedure {
|
||||
self.data.table_info = Some(dropped_table.table_info_value.table_info);
|
||||
self.data.table_route_value = Some(dropped_table.table_route_value);
|
||||
self.data.region_wal_options = dropped_table.region_wal_options;
|
||||
self.data.state = PurgeDroppedTableState::OpenRegions;
|
||||
Ok(Status::executing(true))
|
||||
}
|
||||
|
||||
async fn on_open_regions(&mut self) -> Result<Status> {
|
||||
if let Some(region_routes) = self.data.physical_region_routes() {
|
||||
open_regions_ignore_region_not_found(
|
||||
&self.context,
|
||||
self.data.table_id(),
|
||||
self.data.table_name(),
|
||||
self.data.table_info(),
|
||||
region_routes,
|
||||
&self.data.region_wal_options,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
self.data.state = PurgeDroppedTableState::DropRegions;
|
||||
Ok(Status::executing(true))
|
||||
}
|
||||
@@ -107,13 +90,12 @@ impl PurgeDroppedTableProcedure {
|
||||
async fn on_drop_regions(&mut self) -> Result<Status> {
|
||||
if let Some(region_routes) = self.data.physical_region_routes() {
|
||||
self.executor()
|
||||
.on_drop_regions(
|
||||
.on_cleanup_regions_offline(
|
||||
&self.context.node_manager,
|
||||
&self.context.leader_region_registry,
|
||||
self.data.table_info(),
|
||||
region_routes,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
&self.data.region_wal_options,
|
||||
)
|
||||
.await?;
|
||||
self.context
|
||||
@@ -157,7 +139,6 @@ impl Procedure for PurgeDroppedTableProcedure {
|
||||
async fn execute(&mut self, _: &ProcedureContext) -> ProcedureResult<Status> {
|
||||
match self.data.state {
|
||||
PurgeDroppedTableState::Prepare => self.on_prepare().await,
|
||||
PurgeDroppedTableState::OpenRegions => self.on_open_regions().await,
|
||||
PurgeDroppedTableState::DropRegions => self.on_drop_regions().await,
|
||||
PurgeDroppedTableState::DeleteTombstone => self.on_delete_tombstone().await,
|
||||
}
|
||||
@@ -206,14 +187,14 @@ impl PurgeDroppedTableData {
|
||||
self.table_name.as_ref().unwrap()
|
||||
}
|
||||
|
||||
fn table_info(&self) -> &TableInfo {
|
||||
self.table_info.as_ref().unwrap()
|
||||
}
|
||||
|
||||
fn table_route_value(&self) -> &TableRouteValue {
|
||||
self.table_route_value.as_ref().unwrap()
|
||||
}
|
||||
|
||||
fn table_info(&self) -> &TableInfo {
|
||||
self.table_info.as_ref().unwrap()
|
||||
}
|
||||
|
||||
fn physical_region_routes(&self) -> Option<&[RegionRoute]> {
|
||||
match self.table_route_value() {
|
||||
TableRouteValue::Physical(route) => Some(&route.region_routes),
|
||||
@@ -225,7 +206,6 @@ impl PurgeDroppedTableData {
|
||||
#[derive(Debug, Serialize, Deserialize, AsRefStr, PartialEq)]
|
||||
enum PurgeDroppedTableState {
|
||||
Prepare,
|
||||
OpenRegions,
|
||||
DropRegions,
|
||||
DeleteTombstone,
|
||||
}
|
||||
|
||||
@@ -14,19 +14,17 @@
|
||||
|
||||
use std::assert_matches;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::{Arc, Mutex as StdMutex};
|
||||
use std::sync::Arc;
|
||||
|
||||
use api::region::RegionResponse;
|
||||
use api::v1::region::{RegionRequest, region_request};
|
||||
use async_trait::async_trait;
|
||||
use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME};
|
||||
use common_error::ext::{BoxedError, ErrorExt, StackError};
|
||||
use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME, FILE_ENGINE};
|
||||
use common_error::ext::ErrorExt;
|
||||
use common_error::status_code::StatusCode;
|
||||
use common_procedure::Procedure;
|
||||
use common_procedure::{Procedure, StringKey};
|
||||
use common_procedure_test::{
|
||||
execute_procedure_until, execute_procedure_until_done, new_test_procedure_context,
|
||||
};
|
||||
use snafu::ResultExt;
|
||||
use store_api::region_engine::RegionRole;
|
||||
use store_api::storage::RegionId;
|
||||
use table::metadata::TableId;
|
||||
@@ -44,7 +42,7 @@ use crate::ddl::test_util::{
|
||||
};
|
||||
use crate::ddl::undrop_table::UndropTableProcedure;
|
||||
use crate::ddl::{DetectingRegion, RegionFailureDetectorController, TableMetadata};
|
||||
use crate::error::{self, Error};
|
||||
use crate::error::Error;
|
||||
use crate::key::table_name::TableNameKey;
|
||||
use crate::key::table_route::TableRouteValue;
|
||||
use crate::kv_backend::memory::MemoryKvBackend;
|
||||
@@ -293,16 +291,16 @@ async fn test_soft_drop_closes_regions_and_keeps_tombstone() {
|
||||
let Some(region_request::Body::Close(req)) = request.body else {
|
||||
unreachable!();
|
||||
};
|
||||
requests.push((peer.id, req.region_id));
|
||||
requests.push((peer.id, req.region_id, req.flush_on_close));
|
||||
}
|
||||
requests.sort_unstable();
|
||||
assert_eq!(
|
||||
requests,
|
||||
vec![
|
||||
(1, RegionId::new(table_id, 1).as_u64()),
|
||||
(1, RegionId::new(table_id, 2).as_u64()),
|
||||
(2, RegionId::new(table_id, 1).as_u64()),
|
||||
(2, RegionId::new(table_id, 2).as_u64()),
|
||||
(1, RegionId::new(table_id, 1).as_u64(), true),
|
||||
(1, RegionId::new(table_id, 2).as_u64(), false),
|
||||
(2, RegionId::new(table_id, 1).as_u64(), false),
|
||||
(2, RegionId::new(table_id, 2).as_u64(), true),
|
||||
]
|
||||
);
|
||||
assert!(rx.try_recv().is_err());
|
||||
@@ -741,6 +739,34 @@ async fn test_soft_drop_metric_logical_table_fails() {
|
||||
assert_eq!(err.status_code(), StatusCode::Unsupported);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_soft_drop_file_engine_table_fails() {
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler));
|
||||
let mut ddl_context = new_ddl_context(node_manager);
|
||||
ddl_context.soft_drop_enabled = true;
|
||||
let table_id = 1024;
|
||||
let table_name = "foo";
|
||||
let mut task = test_create_table_task(table_name, table_id);
|
||||
task.table_info.meta.engine = FILE_ENGINE.to_string();
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.create_table_metadata(
|
||||
task.table_info.clone(),
|
||||
TableRouteValue::physical(vec![]),
|
||||
HashMap::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mut procedure = DropTableProcedure::new(
|
||||
new_drop_table_task(table_name, table_id, false),
|
||||
ddl_context,
|
||||
);
|
||||
let err = procedure.on_prepare().await.unwrap_err();
|
||||
|
||||
assert_eq!(err.status_code(), StatusCode::Unsupported);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_undrop_metric_logical_table_fails() {
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler));
|
||||
@@ -872,6 +898,174 @@ async fn test_undrop_table_fails_when_live_name_is_created_after_prepare() {
|
||||
assert_eq!(live_table.table_id(), live_table_id);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_undrop_table_closes_opened_regions_when_restore_metadata_races_with_create() {
|
||||
let (tx, mut rx) = mpsc::channel(8);
|
||||
let datanode_handler = DatanodeWatcher::new(tx);
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(datanode_handler));
|
||||
let detector_controller = Arc::new(RecordingRegionFailureDetectorController::default());
|
||||
let mut ddl_context = new_ddl_context(node_manager);
|
||||
ddl_context.soft_drop_enabled = true;
|
||||
ddl_context.region_failure_detector_controller = detector_controller.clone();
|
||||
let dropped_table_id = 1024;
|
||||
let live_table_id = 1025;
|
||||
let table_name = "foo";
|
||||
let task = test_create_table_task(table_name, dropped_table_id);
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.create_table_metadata(
|
||||
task.table_info.clone(),
|
||||
TableRouteValue::physical(vec![RegionRoute {
|
||||
region: Region::new_test(RegionId::new(dropped_table_id, 1)),
|
||||
leader_peer: Some(Peer::empty(1)),
|
||||
follower_peers: vec![Peer::empty(2)],
|
||||
leader_state: None,
|
||||
leader_down_since: None,
|
||||
write_route_policy: None,
|
||||
}]),
|
||||
HashMap::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut drop_procedure = DropTableProcedure::new(
|
||||
new_drop_table_task(table_name, dropped_table_id, false),
|
||||
ddl_context.clone(),
|
||||
);
|
||||
execute_procedure_until_done(&mut drop_procedure).await;
|
||||
while rx.try_recv().is_ok() {}
|
||||
detector_controller.clear().await;
|
||||
|
||||
let mut procedure =
|
||||
UndropTableProcedure::new(new_undrop_table_task(dropped_table_id), ddl_context.clone());
|
||||
let ctx = new_test_procedure_context();
|
||||
procedure.execute(&ctx).await.unwrap();
|
||||
procedure.execute(&ctx).await.unwrap();
|
||||
|
||||
let mut opened_regions = HashSet::new();
|
||||
for _ in 0..2 {
|
||||
let (peer, request) = rx.try_recv().unwrap();
|
||||
let Some(region_request::Body::Open(req)) = request.body else {
|
||||
unreachable!();
|
||||
};
|
||||
opened_regions.insert((peer.id, req.region_id));
|
||||
}
|
||||
assert_eq!(
|
||||
opened_regions,
|
||||
HashSet::from([
|
||||
(1, RegionId::new(dropped_table_id, 1).as_u64()),
|
||||
(2, RegionId::new(dropped_table_id, 1).as_u64()),
|
||||
])
|
||||
);
|
||||
assert_eq!(
|
||||
detector_controller.registered().await,
|
||||
vec![(1, RegionId::new(dropped_table_id, 1))]
|
||||
);
|
||||
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.create_table_metadata(
|
||||
test_create_table_task(table_name, live_table_id).table_info,
|
||||
TableRouteValue::physical(vec![]),
|
||||
HashMap::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let err = procedure.execute(&ctx).await.unwrap_err();
|
||||
assert_eq!(err.status_code(), StatusCode::TableAlreadyExists);
|
||||
|
||||
let mut closed_regions = HashSet::new();
|
||||
for _ in 0..2 {
|
||||
let (peer, request) = rx.try_recv().unwrap();
|
||||
let Some(region_request::Body::Close(req)) = request.body else {
|
||||
unreachable!();
|
||||
};
|
||||
closed_regions.insert((peer.id, req.region_id));
|
||||
}
|
||||
assert_eq!(
|
||||
closed_regions,
|
||||
HashSet::from([
|
||||
(1, RegionId::new(dropped_table_id, 1).as_u64()),
|
||||
(2, RegionId::new(dropped_table_id, 1).as_u64()),
|
||||
])
|
||||
);
|
||||
assert!(rx.try_recv().is_err());
|
||||
assert_eq!(
|
||||
detector_controller.deregistered().await,
|
||||
vec![(1, RegionId::new(dropped_table_id, 1))]
|
||||
);
|
||||
|
||||
let live_table = ddl_context
|
||||
.table_metadata_manager
|
||||
.table_name_manager()
|
||||
.get(TableNameKey::new(
|
||||
DEFAULT_CATALOG_NAME,
|
||||
DEFAULT_SCHEMA_NAME,
|
||||
table_name,
|
||||
))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(live_table.table_id(), live_table_id);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_undrop_table_lock_key_includes_original_table_name_before_prepare() {
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler));
|
||||
let mut ddl_context = new_ddl_context(node_manager);
|
||||
ddl_context.soft_drop_enabled = true;
|
||||
let table_id = 1024;
|
||||
let table_name = "foo";
|
||||
let task = test_create_table_task(table_name, table_id);
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.create_table_metadata(
|
||||
task.table_info.clone(),
|
||||
TableRouteValue::physical(vec![]),
|
||||
HashMap::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut drop_procedure = DropTableProcedure::new(
|
||||
new_drop_table_task(table_name, table_id, false),
|
||||
ddl_context.clone(),
|
||||
);
|
||||
execute_procedure_until_done(&mut drop_procedure).await;
|
||||
|
||||
let original_table_name = ddl_context
|
||||
.table_metadata_manager
|
||||
.get_dropped_table_by_id(table_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.table_name;
|
||||
let procedure = UndropTableProcedure::new_with_original_table_name(
|
||||
new_undrop_table_task(table_id),
|
||||
ddl_context,
|
||||
Some(original_table_name),
|
||||
);
|
||||
|
||||
let keys = procedure
|
||||
.lock_key()
|
||||
.keys_to_lock()
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
assert!(
|
||||
keys.iter().any(|key| matches!(
|
||||
key,
|
||||
StringKey::Exclusive(key) if key == "__table_name_lock/greptime.public.foo"
|
||||
)),
|
||||
"undrop lock keys should include the original table name: {keys:?}"
|
||||
);
|
||||
assert!(
|
||||
keys.iter().any(|key| matches!(
|
||||
key,
|
||||
StringKey::Exclusive(key) if key == "__table_lock/1024"
|
||||
)),
|
||||
"undrop lock keys should include the dropped table id: {keys:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_undrop_table_replayed_restore_metadata_is_idempotent() {
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler));
|
||||
@@ -921,7 +1115,7 @@ async fn test_undrop_table_replayed_restore_metadata_is_idempotent() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_purge_dropped_table_drops_regions_and_deletes_tombstone() {
|
||||
async fn test_purge_dropped_table_cleans_regions_offline_and_deletes_tombstone() {
|
||||
let (tx, mut rx) = mpsc::channel(8);
|
||||
let datanode_handler = DatanodeWatcher::new(tx);
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(datanode_handler));
|
||||
@@ -963,15 +1157,16 @@ async fn test_purge_dropped_table_drops_regions_and_deletes_tombstone() {
|
||||
execute_procedure_until_done(&mut procedure).await;
|
||||
|
||||
let mut requests = Vec::new();
|
||||
for _ in 0..4 {
|
||||
for _ in 0..1 {
|
||||
let (peer, request) = rx.try_recv().unwrap();
|
||||
requests.push((peer.id, request.body.unwrap()));
|
||||
}
|
||||
requests.sort_unstable_by_key(|(peer_id, _)| *peer_id);
|
||||
assert_matches!(requests[0].1, region_request::Body::Open(_));
|
||||
assert_matches!(requests[1].1, region_request::Body::Drop(_));
|
||||
assert_matches!(requests[2].1, region_request::Body::Open(_));
|
||||
assert_matches!(requests[3].1, region_request::Body::Close(_));
|
||||
let region_request::Body::CleanUp(req) = &requests[0].1 else {
|
||||
unreachable!();
|
||||
};
|
||||
assert_eq!(requests[0].0, 1);
|
||||
assert_eq!(req.region_id, RegionId::new(table_id, 1).as_u64());
|
||||
assert!(rx.try_recv().is_err());
|
||||
assert!(
|
||||
ddl_context
|
||||
@@ -1050,119 +1245,13 @@ async fn test_purge_dropped_table_by_id_selects_tombstone_when_live_table_exists
|
||||
assert_eq!(live_table.table_id(), live_table_id);
|
||||
|
||||
let (_, request) = rx.try_recv().unwrap();
|
||||
let Some(region_request::Body::Open(req)) = request.body else {
|
||||
unreachable!();
|
||||
};
|
||||
assert_eq!(req.region_id, RegionId::new(dropped_table_id, 1).as_u64());
|
||||
|
||||
let (_, request) = rx.try_recv().unwrap();
|
||||
let Some(region_request::Body::Drop(req)) = request.body else {
|
||||
let Some(region_request::Body::CleanUp(req)) = request.body else {
|
||||
unreachable!();
|
||||
};
|
||||
assert_eq!(req.region_id, RegionId::new(dropped_table_id, 1).as_u64());
|
||||
assert!(rx.try_recv().is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_purge_dropped_table_replayed_open_regions_ignores_dropped_regions() {
|
||||
let (tx, mut rx) = mpsc::channel(8);
|
||||
let dropped_regions = Arc::new(StdMutex::new(HashSet::new()));
|
||||
let datanode_handler = DatanodeWatcher::new(tx).with_handler({
|
||||
let dropped_regions = dropped_regions.clone();
|
||||
move |_peer, request| {
|
||||
let Some(body) = request.body.as_ref() else {
|
||||
return Ok(RegionResponse::new(0));
|
||||
};
|
||||
match body {
|
||||
region_request::Body::Open(req)
|
||||
if dropped_regions.lock().unwrap().contains(&req.region_id) =>
|
||||
{
|
||||
Err::<RegionResponse, _>(BoxedError::new(MockRegionNotFoundError))
|
||||
.context(error::ExternalSnafu)
|
||||
}
|
||||
region_request::Body::Drop(req) => {
|
||||
dropped_regions.lock().unwrap().insert(req.region_id);
|
||||
Ok(RegionResponse::new(0))
|
||||
}
|
||||
_ => Ok(RegionResponse::new(0)),
|
||||
}
|
||||
}
|
||||
});
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(datanode_handler));
|
||||
let mut ddl_context = new_ddl_context(node_manager);
|
||||
ddl_context.soft_drop_enabled = true;
|
||||
let table_id = 1024;
|
||||
let table_name = "foo";
|
||||
let task = test_create_table_task(table_name, table_id);
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.create_table_metadata(
|
||||
task.table_info.clone(),
|
||||
TableRouteValue::physical(vec![RegionRoute {
|
||||
region: Region::new_test(RegionId::new(table_id, 1)),
|
||||
leader_peer: Some(Peer::empty(1)),
|
||||
follower_peers: vec![],
|
||||
leader_state: None,
|
||||
leader_down_since: None,
|
||||
write_route_policy: None,
|
||||
}]),
|
||||
HashMap::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut drop_procedure = DropTableProcedure::new(
|
||||
new_drop_table_task(table_name, table_id, false),
|
||||
ddl_context.clone(),
|
||||
);
|
||||
execute_procedure_until_done(&mut drop_procedure).await;
|
||||
while rx.try_recv().is_ok() {}
|
||||
|
||||
let mut procedure = PurgeDroppedTableProcedure::new(
|
||||
new_purge_dropped_table_task(table_id),
|
||||
ddl_context.clone(),
|
||||
);
|
||||
let ctx = new_test_procedure_context();
|
||||
procedure.execute(&ctx).await.unwrap();
|
||||
let open_regions_data = procedure.dump().unwrap();
|
||||
procedure.execute(&ctx).await.unwrap();
|
||||
procedure.execute(&ctx).await.unwrap();
|
||||
|
||||
let mut replayed =
|
||||
PurgeDroppedTableProcedure::from_json(&open_regions_data, ddl_context.clone()).unwrap();
|
||||
execute_procedure_until_done(&mut replayed).await;
|
||||
|
||||
assert!(
|
||||
ddl_context
|
||||
.table_metadata_manager
|
||||
.get_dropped_table(&drop_procedure.data.task.table_name())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[derive(Debug, snafu::Snafu)]
|
||||
#[snafu(display("mock region not found"))]
|
||||
struct MockRegionNotFoundError;
|
||||
|
||||
impl StackError for MockRegionNotFoundError {
|
||||
fn debug_fmt(&self, _: usize, _: &mut Vec<String>) {}
|
||||
|
||||
fn next(&self) -> Option<&dyn StackError> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
impl ErrorExt for MockRegionNotFoundError {
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
|
||||
fn status_code(&self) -> StatusCode {
|
||||
StatusCode::RegionNotFound
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_on_rollback() {
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler));
|
||||
|
||||
@@ -18,13 +18,12 @@ use api::v1::region::{
|
||||
OpenRequest as PbOpenRegionRequest, RegionRequest, RegionRequestHeader, region_request,
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
use common_error::ext::ErrorExt;
|
||||
use common_error::status_code::StatusCode;
|
||||
use common_procedure::error::{FromJsonSnafu, ToJsonSnafu};
|
||||
use common_procedure::{
|
||||
Context as ProcedureContext, LockKey, Procedure, Result as ProcedureResult, Status,
|
||||
};
|
||||
use common_telemetry::tracing_context::TracingContext;
|
||||
use common_telemetry::warn;
|
||||
use common_wal::options::WalOptions;
|
||||
use futures::future::join_all;
|
||||
use serde::{Deserialize, Serialize};
|
||||
@@ -34,6 +33,7 @@ use strum::AsRefStr;
|
||||
use table::metadata::TableId;
|
||||
use table::table_name::TableName;
|
||||
|
||||
use crate::ddl::drop_table::executor::DropTableExecutor;
|
||||
use crate::ddl::utils::{
|
||||
add_peer_context_if_needed, convert_region_routes_to_detecting_regions,
|
||||
is_metric_engine_logical_table, map_to_procedure_error, region_storage_path,
|
||||
@@ -43,7 +43,7 @@ use crate::error::{self, Result};
|
||||
use crate::instruction::CacheIdent;
|
||||
use crate::key::table_name::TableNameKey;
|
||||
use crate::key::table_route::TableRouteValue;
|
||||
use crate::lock_key::TableLock;
|
||||
use crate::lock_key::{CatalogLock, SchemaLock, TableLock, TableNameLock};
|
||||
use crate::rpc::ddl::UndropTableTask;
|
||||
use crate::rpc::router::{
|
||||
RegionRoute, find_follower_regions, find_followers, find_leader_regions, find_leaders,
|
||||
@@ -58,10 +58,17 @@ impl UndropTableProcedure {
|
||||
pub const TYPE_NAME: &'static str = "metasrv-procedure::UndropTable";
|
||||
|
||||
pub fn new(task: UndropTableTask, context: DdlContext) -> Self {
|
||||
Self {
|
||||
context,
|
||||
data: UndropTableData::new(task),
|
||||
}
|
||||
Self::new_with_original_table_name(task, context, None)
|
||||
}
|
||||
|
||||
pub(crate) fn new_with_original_table_name(
|
||||
task: UndropTableTask,
|
||||
context: DdlContext,
|
||||
table_name: Option<TableName>,
|
||||
) -> Self {
|
||||
let mut data = UndropTableData::new(task);
|
||||
data.table_name = table_name;
|
||||
Self { context, data }
|
||||
}
|
||||
|
||||
pub fn from_json(json: &str, context: DdlContext) -> ProcedureResult<Self> {
|
||||
@@ -108,29 +115,101 @@ impl UndropTableProcedure {
|
||||
}
|
||||
|
||||
async fn on_restore_metadata(&mut self) -> Result<Status> {
|
||||
self.ensure_table_name_loaded().await?;
|
||||
let table_name = self.data.table_name().clone();
|
||||
let table_route_value = self.data.table_route_value();
|
||||
self.context
|
||||
if let Err(err) = self
|
||||
.context
|
||||
.table_metadata_manager
|
||||
.restore_table_metadata(
|
||||
self.data.task.table_id,
|
||||
self.data.table_name(),
|
||||
&table_name,
|
||||
table_route_value,
|
||||
&self.data.region_wal_options,
|
||||
)
|
||||
.await
|
||||
.map_err(|err| match err {
|
||||
error::Error::TombstoneTargetAlreadyExists { .. } => {
|
||||
error::TableAlreadyExistsSnafu {
|
||||
table_name: self.data.table_name().to_string(),
|
||||
}
|
||||
.build()
|
||||
}
|
||||
err => err,
|
||||
})?;
|
||||
{
|
||||
let should_cleanup_opened_regions = !err.is_retry_later();
|
||||
let err = self.map_restore_metadata_error(err);
|
||||
if should_cleanup_opened_regions
|
||||
&& let Err(cleanup_err) = self.cleanup_opened_regions_after_restore_failure().await
|
||||
{
|
||||
warn!(
|
||||
cleanup_err;
|
||||
"Failed to close opened regions after undrop metadata restore failure, table_id: {}",
|
||||
self.data.task.table_id
|
||||
);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
self.data.state = UndropTableState::InvalidateTableCache;
|
||||
Ok(Status::executing(true))
|
||||
}
|
||||
|
||||
async fn ensure_table_name_loaded(&mut self) -> Result<()> {
|
||||
if self.data.table_name.is_some() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let dropped_table = self
|
||||
.context
|
||||
.table_metadata_manager
|
||||
.get_dropped_table_by_id(self.data.task.table_id)
|
||||
.await?
|
||||
.with_context(|| error::TableNotFoundSnafu {
|
||||
table_name: self.data.task.table_id.to_string(),
|
||||
})?;
|
||||
self.data.table_name = Some(dropped_table.table_name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn map_restore_metadata_error(&self, err: error::Error) -> error::Error {
|
||||
match err {
|
||||
error::Error::TombstoneTargetAlreadyExists { .. } => {
|
||||
let table_name = self
|
||||
.data
|
||||
.table_name
|
||||
.as_ref()
|
||||
.map(ToString::to_string)
|
||||
.unwrap_or_else(|| self.data.task.table_id.to_string());
|
||||
error::TableAlreadyExistsSnafu { table_name }.build()
|
||||
}
|
||||
err => err,
|
||||
}
|
||||
}
|
||||
|
||||
async fn cleanup_opened_regions_after_restore_failure(&self) -> Result<()> {
|
||||
let Some(table_route_value) = self.data.table_route_value.as_ref() else {
|
||||
return Ok(());
|
||||
};
|
||||
let TableRouteValue::Physical(route) = table_route_value else {
|
||||
return Ok(());
|
||||
};
|
||||
let region_routes = route.region_routes.clone();
|
||||
let close_result = if let Some(table_name) = self.data.table_name.as_ref() {
|
||||
let executor =
|
||||
DropTableExecutor::new(table_name.clone(), self.data.task.table_id, false);
|
||||
|
||||
executor
|
||||
.on_close_regions(
|
||||
&self.context.node_manager,
|
||||
&self.context.leader_region_registry,
|
||||
®ion_routes,
|
||||
false,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
self.context
|
||||
.deregister_failure_detectors(convert_region_routes_to_detecting_regions(
|
||||
®ion_routes,
|
||||
))
|
||||
.await;
|
||||
close_result?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn on_open_regions(&mut self) -> Result<Status> {
|
||||
self.ensure_live_table_not_exists().await?;
|
||||
let TableRouteValue::Physical(route) = self.data.table_route_value() else {
|
||||
@@ -213,7 +292,22 @@ impl Procedure for UndropTableProcedure {
|
||||
}
|
||||
|
||||
fn lock_key(&self) -> LockKey {
|
||||
LockKey::new(vec![TableLock::Write(self.data.task.table_id).into()])
|
||||
let mut lock_key = Vec::new();
|
||||
if let Some(table_name) = &self.data.table_name {
|
||||
lock_key.push(CatalogLock::Read(&table_name.catalog_name).into());
|
||||
lock_key
|
||||
.push(SchemaLock::read(&table_name.catalog_name, &table_name.schema_name).into());
|
||||
lock_key.push(
|
||||
TableNameLock::new(
|
||||
&table_name.catalog_name,
|
||||
&table_name.schema_name,
|
||||
&table_name.table_name,
|
||||
)
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
lock_key.push(TableLock::Write(self.data.task.table_id).into());
|
||||
LockKey::new(lock_key)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -232,27 +326,6 @@ pub(crate) async fn open_regions(
|
||||
table_info,
|
||||
region_routes,
|
||||
region_wal_options,
|
||||
false,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn open_regions_ignore_region_not_found(
|
||||
context: &DdlContext,
|
||||
table_id: TableId,
|
||||
table_name: &TableName,
|
||||
table_info: &table::metadata::TableInfo,
|
||||
region_routes: &[RegionRoute],
|
||||
region_wal_options: &HashMap<RegionNumber, WalOptions>,
|
||||
) -> Result<()> {
|
||||
open_regions_inner(
|
||||
context,
|
||||
table_id,
|
||||
table_name,
|
||||
table_info,
|
||||
region_routes,
|
||||
region_wal_options,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -264,7 +337,6 @@ async fn open_regions_inner(
|
||||
table_info: &table::metadata::TableInfo,
|
||||
region_routes: &[RegionRoute],
|
||||
region_wal_options: &HashMap<RegionNumber, WalOptions>,
|
||||
ignore_region_not_found: bool,
|
||||
) -> Result<()> {
|
||||
let template = build_template_from_raw_table_info(table_info)?;
|
||||
let builder = CreateRequestBuilder::new(template, None);
|
||||
@@ -303,9 +375,7 @@ async fn open_regions_inner(
|
||||
let datanode = datanode.clone();
|
||||
let requester = requester.clone();
|
||||
tasks.push(async move {
|
||||
if let Err(err) = requester.handle(request).await
|
||||
&& !(ignore_region_not_found && err.status_code() == StatusCode::RegionNotFound)
|
||||
{
|
||||
if let Err(err) = requester.handle(request).await {
|
||||
return Err(add_peer_context_if_needed(datanode)(err));
|
||||
}
|
||||
Ok(())
|
||||
@@ -366,3 +436,130 @@ enum UndropTableState {
|
||||
OpenRegions,
|
||||
InvalidateTableCache,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use api::region::RegionResponse;
|
||||
use api::v1::region::region_request;
|
||||
use async_trait::async_trait;
|
||||
use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME};
|
||||
use common_error::ext::ErrorExt;
|
||||
use common_error::status_code::StatusCode;
|
||||
use store_api::storage::RegionId;
|
||||
use table::table_name::TableName;
|
||||
use tokio::sync::{Mutex, mpsc};
|
||||
|
||||
use super::*;
|
||||
use crate::ddl::test_util::datanode_handler::DatanodeWatcher;
|
||||
use crate::ddl::{DetectingRegion, RegionFailureDetectorController};
|
||||
use crate::peer::Peer;
|
||||
use crate::rpc::router::{Region, RegionRoute};
|
||||
use crate::test_util::{MockDatanodeManager, new_ddl_context};
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingRegionFailureDetectorController {
|
||||
deregistered: Mutex<Vec<DetectingRegion>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl RegionFailureDetectorController for RecordingRegionFailureDetectorController {
|
||||
async fn register_failure_detectors(&self, _detecting_regions: Vec<DetectingRegion>) {}
|
||||
|
||||
async fn reset_failure_detectors(&self, _detecting_regions: Vec<DetectingRegion>) {}
|
||||
|
||||
async fn deregister_failure_detectors(&self, detecting_regions: Vec<DetectingRegion>) {
|
||||
self.deregistered.lock().await.extend(detecting_regions);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_map_restore_metadata_error_without_table_name() {
|
||||
let context = new_ddl_context(Arc::new(MockDatanodeManager::new(())));
|
||||
let procedure = UndropTableProcedure::new(UndropTableTask { table_id: 42 }, context);
|
||||
|
||||
let err = procedure.map_restore_metadata_error(
|
||||
error::TombstoneTargetAlreadyExistsSnafu {
|
||||
key: "table-name-key".to_string(),
|
||||
}
|
||||
.build(),
|
||||
);
|
||||
|
||||
assert_eq!(StatusCode::TableAlreadyExists, err.status_code());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_cleanup_opened_regions_without_table_name_deregisters_detectors() {
|
||||
let detector_controller = Arc::new(RecordingRegionFailureDetectorController::default());
|
||||
let mut context = new_ddl_context(Arc::new(MockDatanodeManager::new(())));
|
||||
context.region_failure_detector_controller = detector_controller.clone();
|
||||
|
||||
let table_id = 1024;
|
||||
let region_id = RegionId::new(table_id, 1);
|
||||
let mut procedure = UndropTableProcedure::new(UndropTableTask { table_id }, context);
|
||||
procedure.data.table_route_value = Some(TableRouteValue::physical(vec![RegionRoute {
|
||||
region: Region::new_test(region_id),
|
||||
leader_peer: Some(Peer::empty(1)),
|
||||
follower_peers: vec![],
|
||||
leader_state: None,
|
||||
leader_down_since: None,
|
||||
write_route_policy: None,
|
||||
}]));
|
||||
|
||||
procedure
|
||||
.cleanup_opened_regions_after_restore_failure()
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
detector_controller.deregistered.lock().await.as_slice(),
|
||||
&[(1, region_id)]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_cleanup_opened_regions_deregisters_detectors_when_close_fails() {
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let datanode_handler = DatanodeWatcher::new(tx).with_handler(|_, request| {
|
||||
if matches!(request.body, Some(region_request::Body::Close(_))) {
|
||||
return error::UnexpectedSnafu {
|
||||
err_msg: "mock close error".to_string(),
|
||||
}
|
||||
.fail();
|
||||
}
|
||||
Ok(RegionResponse::new(0))
|
||||
});
|
||||
let detector_controller = Arc::new(RecordingRegionFailureDetectorController::default());
|
||||
let mut context = new_ddl_context(Arc::new(MockDatanodeManager::new(datanode_handler)));
|
||||
context.region_failure_detector_controller = detector_controller.clone();
|
||||
|
||||
let table_id = 1024;
|
||||
let region_id = RegionId::new(table_id, 1);
|
||||
let mut procedure = UndropTableProcedure::new(UndropTableTask { table_id }, context);
|
||||
procedure.data.table_name = Some(TableName::new(
|
||||
DEFAULT_CATALOG_NAME,
|
||||
DEFAULT_SCHEMA_NAME,
|
||||
"foo",
|
||||
));
|
||||
procedure.data.table_route_value = Some(TableRouteValue::physical(vec![RegionRoute {
|
||||
region: Region::new_test(region_id),
|
||||
leader_peer: Some(Peer::empty(1)),
|
||||
follower_peers: vec![],
|
||||
leader_state: None,
|
||||
leader_down_since: None,
|
||||
write_route_policy: None,
|
||||
}]));
|
||||
|
||||
let err = procedure
|
||||
.cleanup_opened_regions_after_restore_failure()
|
||||
.await
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(StatusCode::Unexpected, err.status_code());
|
||||
assert_eq!(
|
||||
detector_controller.deregistered.lock().await.as_slice(),
|
||||
&[(1, region_id)]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -480,7 +480,19 @@ impl DdlManager {
|
||||
undrop_table_task: UndropTableTask,
|
||||
) -> Result<(ProcedureId, Option<Output>)> {
|
||||
let context = self.create_context();
|
||||
let procedure = UndropTableProcedure::new(undrop_table_task, context);
|
||||
let original_table_name = context
|
||||
.table_metadata_manager
|
||||
.get_dropped_table_by_id(undrop_table_task.table_id)
|
||||
.await?
|
||||
.with_context(|| TableNotFoundSnafu {
|
||||
table_name: undrop_table_task.table_id.to_string(),
|
||||
})?
|
||||
.table_name;
|
||||
let procedure = UndropTableProcedure::new_with_original_table_name(
|
||||
undrop_table_task,
|
||||
context,
|
||||
Some(original_table_name),
|
||||
);
|
||||
let procedure_with_id = ProcedureWithId::with_random_id(Box::new(procedure));
|
||||
|
||||
self.execute_procedure_and_wait(procedure_with_id).await
|
||||
@@ -1204,7 +1216,8 @@ mod tests {
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use common_error::ext::BoxedError;
|
||||
use common_error::ext::{BoxedError, ErrorExt};
|
||||
use common_error::status_code::StatusCode;
|
||||
use common_procedure::local::LocalManager;
|
||||
use common_procedure::test_util::InMemoryPoisonStore;
|
||||
use common_procedure::{BoxedProcedure, ProcedureManagerRef};
|
||||
@@ -1228,6 +1241,7 @@ mod tests {
|
||||
use crate::peer::Peer;
|
||||
use crate::region_keeper::MemoryRegionKeeper;
|
||||
use crate::region_registry::LeaderRegionRegistry;
|
||||
use crate::rpc::ddl::UndropTableTask;
|
||||
use crate::sequence::SequenceBuilder;
|
||||
use crate::state_store::KvStateStore;
|
||||
use crate::wal_provider::WalProvider;
|
||||
@@ -1326,4 +1340,55 @@ mod tests {
|
||||
assert!(procedure_manager.contains_loader(loader));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_submit_undrop_missing_tombstone_returns_table_not_found_directly() {
|
||||
let kv_backend = Arc::new(MemoryKvBackend::new());
|
||||
let table_metadata_manager = Arc::new(TableMetadataManager::new(kv_backend.clone()));
|
||||
let table_metadata_allocator = Arc::new(TableMetadataAllocator::new(
|
||||
Arc::new(SequenceBuilder::new("test", kv_backend.clone()).build()),
|
||||
Arc::new(WalProvider::default()),
|
||||
));
|
||||
let flow_metadata_manager = Arc::new(FlowMetadataManager::new(kv_backend.clone()));
|
||||
let flow_metadata_allocator = Arc::new(FlowMetadataAllocator::with_noop_peer_allocator(
|
||||
Arc::new(SequenceBuilder::new("flow-test", kv_backend.clone()).build()),
|
||||
));
|
||||
|
||||
let state_store = Arc::new(KvStateStore::new(kv_backend.clone()));
|
||||
let poison_manager = Arc::new(InMemoryPoisonStore::default());
|
||||
let procedure_manager = Arc::new(LocalManager::new(
|
||||
Default::default(),
|
||||
state_store,
|
||||
poison_manager,
|
||||
None,
|
||||
None,
|
||||
));
|
||||
|
||||
let ddl_manager = DdlManager::try_new(
|
||||
DdlContext {
|
||||
node_manager: Arc::new(DummyDatanodeManager),
|
||||
cache_invalidator: Arc::new(DummyCacheInvalidator),
|
||||
table_metadata_manager,
|
||||
table_metadata_allocator,
|
||||
flow_metadata_manager,
|
||||
flow_metadata_allocator,
|
||||
memory_region_keeper: Arc::new(MemoryRegionKeeper::default()),
|
||||
leader_region_registry: Arc::new(LeaderRegionRegistry::default()),
|
||||
region_failure_detector_controller: Arc::new(NoopRegionFailureDetectorControl),
|
||||
soft_drop_enabled: true,
|
||||
},
|
||||
procedure_manager,
|
||||
Arc::new(DummyRepartitionProcedureFactory),
|
||||
true,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let err = ddl_manager
|
||||
.submit_undrop_table_task(UndropTableTask { table_id: 1024 })
|
||||
.await
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.status_code(), StatusCode::TableNotFound);
|
||||
assert!(matches!(err, crate::error::Error::TableNotFound { .. }));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,7 @@ pub mod error;
|
||||
pub mod logical_plan;
|
||||
pub mod native_histogram;
|
||||
pub mod prelude;
|
||||
pub mod prometheus;
|
||||
pub mod request;
|
||||
pub mod stream;
|
||||
#[cfg(any(test, feature = "testing"))]
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
/// Canonical Prometheus stale-marker NaN bit pattern.
|
||||
pub const PROMETHEUS_STALE_NAN_BITS: u64 = 0x7ff0_0000_0000_0002;
|
||||
|
||||
/// Returns whether `value` is the canonical Prometheus stale-marker NaN.
|
||||
#[inline]
|
||||
pub fn is_prometheus_stale_nan(value: f64) -> bool {
|
||||
value.to_bits() == PROMETHEUS_STALE_NAN_BITS
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn recognizes_only_the_canonical_stale_marker() {
|
||||
assert!(is_prometheus_stale_nan(f64::from_bits(
|
||||
0x7ff0_0000_0000_0002
|
||||
)));
|
||||
assert!(!is_prometheus_stale_nan(f64::from_bits(
|
||||
0x7ff8_0000_0000_0000
|
||||
)));
|
||||
}
|
||||
}
|
||||
@@ -34,3 +34,7 @@ tokio.workspace = true
|
||||
[[bench]]
|
||||
name = "iter_record_batch_rows"
|
||||
harness = false
|
||||
|
||||
[[bench]]
|
||||
name = "memory_accounting"
|
||||
harness = false
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::hint::black_box;
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_recordbatch::RecordBatch;
|
||||
use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
|
||||
use datatypes::data_type::ConcreteDataType;
|
||||
use datatypes::schema::{ColumnSchema, Schema};
|
||||
use datatypes::vectors::{Int32Vector, StringVector, UInt64Vector, VectorRef};
|
||||
|
||||
const LARGE_ROWS: usize = 8_192;
|
||||
|
||||
fn batch(columns: Vec<(&str, ConcreteDataType, bool, VectorRef)>) -> RecordBatch {
|
||||
let schema = Arc::new(Schema::new(
|
||||
columns
|
||||
.iter()
|
||||
.map(|(name, data_type, nullable, _)| {
|
||||
ColumnSchema::new(*name, data_type.clone(), *nullable)
|
||||
})
|
||||
.collect(),
|
||||
));
|
||||
let vectors: Vec<VectorRef> = columns
|
||||
.into_iter()
|
||||
.map(|(_, _, _, vector)| vector)
|
||||
.collect();
|
||||
RecordBatch::new(schema, vectors).unwrap()
|
||||
}
|
||||
|
||||
fn primitive_string_batch(rows: usize) -> RecordBatch {
|
||||
let strings = (0..rows)
|
||||
.map(|row| (row % 5 != 0).then(|| format!("value-{row:08}-with-payload")))
|
||||
.collect::<Vec<_>>();
|
||||
batch(vec![
|
||||
(
|
||||
"i32",
|
||||
ConcreteDataType::int32_datatype(),
|
||||
false,
|
||||
Arc::new(Int32Vector::from_slice(
|
||||
(0..rows).map(|value| value as i32).collect::<Vec<_>>(),
|
||||
)),
|
||||
),
|
||||
(
|
||||
"u64",
|
||||
ConcreteDataType::uint64_datatype(),
|
||||
false,
|
||||
Arc::new(UInt64Vector::from_slice(
|
||||
(0..rows).map(|value| value as u64).collect::<Vec<_>>(),
|
||||
)),
|
||||
),
|
||||
(
|
||||
"string",
|
||||
ConcreteDataType::string_datatype(),
|
||||
true,
|
||||
Arc::new(StringVector::from(strings)),
|
||||
),
|
||||
])
|
||||
}
|
||||
|
||||
fn bench_one_call(c: &mut Criterion) {
|
||||
let mut group = c.benchmark_group("memory_accounting/one_call");
|
||||
for rows in [128, LARGE_ROWS] {
|
||||
let batch = primitive_string_batch(rows);
|
||||
group.bench_function(BenchmarkId::new("buffer", rows), |b| {
|
||||
b.iter(|| black_box(black_box(&batch).buffer_memory_size()))
|
||||
});
|
||||
group.bench_function(BenchmarkId::new("logical", rows), |b| {
|
||||
b.iter(|| black_box(black_box(&batch).logical_slice_memory_size()))
|
||||
});
|
||||
}
|
||||
group.finish();
|
||||
}
|
||||
|
||||
fn bench_shared_buffer_slices(c: &mut Criterion) {
|
||||
let backing = primitive_string_batch(LARGE_ROWS);
|
||||
let mut group = c.benchmark_group("memory_accounting/shared_buffer_slices");
|
||||
for slice_len in [1, 16] {
|
||||
let slices = (0..512)
|
||||
.map(|index| backing.slice(index * slice_len, slice_len).unwrap())
|
||||
.collect::<Vec<_>>();
|
||||
group.throughput(Throughput::Elements(slices.len() as u64));
|
||||
group.bench_function(BenchmarkId::new("buffer", slice_len), |b| {
|
||||
b.iter(|| {
|
||||
let total = black_box(&slices)
|
||||
.iter()
|
||||
.map(RecordBatch::buffer_memory_size)
|
||||
.fold(0usize, usize::saturating_add);
|
||||
black_box(total)
|
||||
})
|
||||
});
|
||||
group.bench_function(BenchmarkId::new("logical", slice_len), |b| {
|
||||
b.iter(|| {
|
||||
let total = black_box(&slices)
|
||||
.iter()
|
||||
.map(RecordBatch::logical_slice_memory_size)
|
||||
.fold(0usize, usize::saturating_add);
|
||||
black_box(total)
|
||||
})
|
||||
});
|
||||
}
|
||||
group.finish();
|
||||
}
|
||||
|
||||
criterion_group!(benches, bench_one_call, bench_shared_buffer_slices);
|
||||
criterion_main!(benches);
|
||||
@@ -769,7 +769,7 @@ impl MemoryTrackedStream {
|
||||
batch: RecordBatch,
|
||||
cx: &mut Context<'_>,
|
||||
) -> Poll<Option<Result<RecordBatch>>> {
|
||||
let additional = batch.buffer_memory_size();
|
||||
let additional = batch.logical_slice_memory_size();
|
||||
let tracker = self.ready_tracker_mut();
|
||||
|
||||
if let Err(error) = tracker.try_track(additional) {
|
||||
@@ -860,6 +860,35 @@ mod tests {
|
||||
as usize
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_memory_tracked_stream_charges_logical_slice_size() {
|
||||
let schema = Arc::new(Schema::new(vec![ColumnSchema::new(
|
||||
"payload",
|
||||
ConcreteDataType::string_datatype(),
|
||||
false,
|
||||
)]));
|
||||
let payloads: Vec<_> = (0..1024)
|
||||
.map(|value| format!("payload-{value:04}"))
|
||||
.collect();
|
||||
let batch = RecordBatch::new(schema, vec![Arc::new(StringVector::from(payloads)) as _])
|
||||
.unwrap()
|
||||
.slice(512, 1)
|
||||
.unwrap();
|
||||
let expected_bytes = aligned_tracked_bytes(batch.logical_slice_memory_size());
|
||||
assert!(expected_bytes < aligned_tracked_bytes(batch.buffer_memory_size()));
|
||||
let tracker = QueryMemoryTracker::builder(MB, OnExhaustedPolicy::Fail).build();
|
||||
let mut stream = MemoryTrackedStream::new(
|
||||
RecordBatches::try_new(batch.schema.clone(), vec![batch])
|
||||
.unwrap()
|
||||
.as_stream(),
|
||||
tracker.clone(),
|
||||
);
|
||||
|
||||
stream.next().await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(tracker.current(), expected_bytes);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recordbatches_try_from_columns() {
|
||||
let schema = Arc::new(Schema::new(vec![ColumnSchema::new(
|
||||
@@ -1052,7 +1081,7 @@ mod tests {
|
||||
})
|
||||
.build();
|
||||
let batch = large_string_batch(700 * 1024);
|
||||
let expected_bytes = aligned_tracked_bytes(batch.buffer_memory_size());
|
||||
let expected_bytes = aligned_tracked_bytes(batch.logical_slice_memory_size());
|
||||
|
||||
let mut stream1 = MemoryTrackedStream::new(
|
||||
RecordBatches::try_new(batch.schema.clone(), vec![batch.clone()])
|
||||
|
||||
@@ -269,6 +269,28 @@ impl RecordBatch {
|
||||
.sum()
|
||||
}
|
||||
|
||||
/// Returns the logical memory size of this batch's array slices.
|
||||
///
|
||||
/// This sums Arrow's logical visible slice buffers rather than the capacity of their shared
|
||||
/// backing buffers. View out-of-line payloads and nested custom payloads are not separately
|
||||
/// traversed or accounted. It is not an exact measure of live physical memory. If Arrow cannot
|
||||
/// calculate a slice's size, the full buffer size is used conservatively.
|
||||
///
|
||||
/// Mito's current scan paths do not produce top-level View arrays. If they do in the future,
|
||||
/// their out-of-line payload accounting must be reassessed here.
|
||||
pub fn logical_slice_memory_size(&self) -> usize {
|
||||
self.df_record_batch
|
||||
.columns()
|
||||
.iter()
|
||||
.fold(0, |total, array| {
|
||||
let array_size = array
|
||||
.to_data()
|
||||
.get_slice_memory_size()
|
||||
.unwrap_or_else(|_| array.get_buffer_memory_size());
|
||||
total.saturating_add(array_size)
|
||||
})
|
||||
}
|
||||
|
||||
/// Iterate the values as strings in the column at index `i`.
|
||||
///
|
||||
/// Note that if the underlying array is not a valid GreptimeDB vector, an empty iterator is
|
||||
@@ -382,13 +404,14 @@ fn maybe_align_json_array_with_schema(
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use datatypes::arrow::array::{AsArray, BinaryArray, UInt32Array};
|
||||
use datatypes::arrow::array::{
|
||||
AsArray, BinaryArray, StringArray, StringViewArray, UInt32Array,
|
||||
};
|
||||
use datatypes::arrow::datatypes::{DataType, Field, Schema as ArrowSchema, UInt32Type};
|
||||
use datatypes::arrow_array::StringArray;
|
||||
use datatypes::data_type::ConcreteDataType;
|
||||
use datatypes::extension::json::{JsonExtensionType, JsonMetadata};
|
||||
use datatypes::schema::{ColumnSchema, Schema};
|
||||
use datatypes::vectors::{StringVector, UInt32Vector};
|
||||
use datatypes::vectors::{BinaryVector, StringVector, UInt32Vector};
|
||||
|
||||
use super::*;
|
||||
|
||||
@@ -475,6 +498,97 @@ mod tests {
|
||||
assert!(recordbatch.slice(1, 5).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_logical_slice_memory_size_for_visible_primitive_string_binary_slices() {
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
ColumnSchema::new("numbers", ConcreteDataType::uint32_datatype(), false),
|
||||
ColumnSchema::new("strings", ConcreteDataType::string_datatype(), true),
|
||||
ColumnSchema::new("binary", ConcreteDataType::binary_datatype(), true),
|
||||
]));
|
||||
let numbers: Vec<_> = (0..1024).collect();
|
||||
let strings = (0..1024)
|
||||
.map(|value| (value % 3 != 0).then(|| format!("value-{value}")))
|
||||
.collect::<Vec<_>>();
|
||||
let binary = (0_u32..1024)
|
||||
.map(|value| (value % 3 != 0).then(|| value.to_le_bytes().to_vec()))
|
||||
.collect::<Vec<_>>();
|
||||
let columns: Vec<VectorRef> = vec![
|
||||
Arc::new(UInt32Vector::from_slice(numbers)),
|
||||
Arc::new(StringVector::from(strings)),
|
||||
Arc::new(BinaryVector::from(binary)),
|
||||
];
|
||||
let batch = RecordBatch::new(schema, columns).unwrap();
|
||||
let slice = batch.slice(511, 3).unwrap();
|
||||
|
||||
assert!(slice.columns().iter().any(|column| column.null_count() > 0));
|
||||
assert!(slice.logical_slice_memory_size() < slice.buffer_memory_size());
|
||||
assert_eq!(
|
||||
slice.logical_slice_memory_size(),
|
||||
slice
|
||||
.columns()
|
||||
.iter()
|
||||
.map(|column| column.to_data().get_slice_memory_size().unwrap())
|
||||
.sum::<usize>()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_logical_slice_memory_size_for_many_shared_buffer_slices() {
|
||||
let schema = Arc::new(Schema::new(vec![ColumnSchema::new(
|
||||
"strings",
|
||||
ConcreteDataType::string_datatype(),
|
||||
false,
|
||||
)]));
|
||||
let strings = (0..1024)
|
||||
.map(|value| format!("shared-value-{value}"))
|
||||
.collect::<Vec<_>>();
|
||||
let backing = RecordBatch::new(
|
||||
schema,
|
||||
vec![Arc::new(StringVector::from(strings)) as VectorRef],
|
||||
)
|
||||
.unwrap();
|
||||
let slices = (0..128)
|
||||
.map(|index| backing.slice(index * 4, 2).unwrap())
|
||||
.collect::<Vec<_>>();
|
||||
let logical_total = slices
|
||||
.iter()
|
||||
.map(RecordBatch::logical_slice_memory_size)
|
||||
.sum::<usize>();
|
||||
let buffer_total = slices
|
||||
.iter()
|
||||
.map(RecordBatch::buffer_memory_size)
|
||||
.sum::<usize>();
|
||||
|
||||
assert!(logical_total < buffer_total);
|
||||
assert!(slices.iter().all(|slice| {
|
||||
slice.logical_slice_memory_size()
|
||||
== slice.column(0).to_data().get_slice_memory_size().unwrap()
|
||||
}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_logical_slice_memory_size_uses_arrow_slice_scope_for_views() {
|
||||
let schema = Arc::new(Schema::new(vec![ColumnSchema::new(
|
||||
"strings",
|
||||
ConcreteDataType::utf8_view_datatype(),
|
||||
false,
|
||||
)]));
|
||||
let columns: Vec<VectorRef> =
|
||||
vec![Arc::new(StringVector::from(StringViewArray::from(vec![
|
||||
"unrelated backing payload",
|
||||
"visible string view payload",
|
||||
])))];
|
||||
let batch = RecordBatch::new(schema, columns)
|
||||
.unwrap()
|
||||
.slice(1, 1)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
batch.column(0).to_data().get_slice_memory_size().unwrap(),
|
||||
batch.logical_slice_memory_size()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_merge_record_batch() {
|
||||
let column_schemas = vec![
|
||||
|
||||
@@ -198,6 +198,10 @@ impl Date {
|
||||
pub fn negative(&self) -> Self {
|
||||
Self(-self.0)
|
||||
}
|
||||
|
||||
pub fn checked_negative(&self) -> Option<Self> {
|
||||
self.0.checked_neg().map(Self)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -97,6 +97,11 @@ impl Duration {
|
||||
self.value = -self.value;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn checked_negative(mut self) -> Option<Self> {
|
||||
self.value = self.value.checked_neg()?;
|
||||
Some(self)
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert i64 to Duration Type.
|
||||
|
||||
@@ -80,6 +80,10 @@ impl IntervalYearMonth {
|
||||
Self::new(-self.months)
|
||||
}
|
||||
|
||||
pub fn checked_negative(&self) -> Option<Self> {
|
||||
self.months.checked_neg().map(Self::new)
|
||||
}
|
||||
|
||||
pub fn to_iso8601_string(&self) -> String {
|
||||
IntervalFormat::from(*self).to_iso8601_string()
|
||||
}
|
||||
@@ -157,6 +161,13 @@ impl IntervalDayTime {
|
||||
Self::new(-self.days, -self.milliseconds)
|
||||
}
|
||||
|
||||
pub fn checked_negative(&self) -> Option<Self> {
|
||||
Some(Self::new(
|
||||
self.days.checked_neg()?,
|
||||
self.milliseconds.checked_neg()?,
|
||||
))
|
||||
}
|
||||
|
||||
pub fn to_iso8601_string(&self) -> String {
|
||||
IntervalFormat::from(*self).to_iso8601_string()
|
||||
}
|
||||
@@ -278,6 +289,14 @@ impl IntervalMonthDayNano {
|
||||
Self::new(-self.months, -self.days, -self.nanoseconds)
|
||||
}
|
||||
|
||||
pub fn checked_negative(&self) -> Option<Self> {
|
||||
Some(Self::new(
|
||||
self.months.checked_neg()?,
|
||||
self.days.checked_neg()?,
|
||||
self.nanoseconds.checked_neg()?,
|
||||
))
|
||||
}
|
||||
|
||||
pub fn to_iso8601_string(&self) -> String {
|
||||
IntervalFormat::from(*self).to_iso8601_string()
|
||||
}
|
||||
|
||||
@@ -145,6 +145,11 @@ impl Time {
|
||||
self.value = -self.value;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn checked_negative(mut self) -> Option<Self> {
|
||||
self.value = self.value.checked_neg()?;
|
||||
Some(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<i64> for Time {
|
||||
|
||||
@@ -484,6 +484,11 @@ impl Timestamp {
|
||||
self.value = -self.value;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn checked_negative(mut self) -> Option<Self> {
|
||||
self.value = self.value.checked_neg()?;
|
||||
Some(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl Timestamp {
|
||||
|
||||
@@ -60,7 +60,7 @@ impl Default for DatanodeKafkaConfig {
|
||||
consumer_wait_timeout: Duration::from_millis(100),
|
||||
kafka_topic: KafkaTopicConfig::default(),
|
||||
auto_create_topics: true,
|
||||
create_index: true,
|
||||
create_index: false,
|
||||
dump_index_interval: Duration::from_secs(60),
|
||||
topic_latest_offset_fetch_interval: DEFAULT_TOPIC_LATEST_OFFSET_FETCH_INTERVAL,
|
||||
overwrite_entry_start_id: false,
|
||||
|
||||
@@ -153,7 +153,7 @@ impl RegionAliveKeeper {
|
||||
|
||||
async fn close_staled_region(&self, region_id: RegionId) {
|
||||
info!("Closing staled region: {region_id}");
|
||||
let request = RegionRequest::Close(RegionCloseRequest {});
|
||||
let request = RegionRequest::Close(RegionCloseRequest::default());
|
||||
if let Err(e) = self.region_server.handle_request(region_id, request).await
|
||||
&& e.status_code() != StatusCode::RegionNotFound
|
||||
{
|
||||
|
||||
@@ -287,6 +287,18 @@ pub enum Error {
|
||||
location: Location,
|
||||
},
|
||||
|
||||
#[snafu(display(
|
||||
"GC configuration mismatch: metasrv.gc.enable={}, datanode.region_engine.mito.gc.enable={}",
|
||||
metasrv_gc_enabled,
|
||||
datanode_gc_enabled,
|
||||
))]
|
||||
GcConfigMismatch {
|
||||
metasrv_gc_enabled: bool,
|
||||
datanode_gc_enabled: bool,
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
},
|
||||
|
||||
#[snafu(display("Unsupported output type, expected: {}", expected))]
|
||||
UnsupportedOutput {
|
||||
expected: String,
|
||||
@@ -456,6 +468,7 @@ impl ErrorExt for Error {
|
||||
| ColumnNoneDefaultValue { .. }
|
||||
| MissingRequiredField { .. }
|
||||
| RegionEngineNotFound { .. }
|
||||
| GcConfigMismatch { .. }
|
||||
| ParseAddr { .. }
|
||||
| TomlFormat { .. }
|
||||
| BuildDatanode { .. } => StatusCode::InvalidArguments,
|
||||
|
||||
@@ -121,6 +121,7 @@ impl HeartbeatTask {
|
||||
|
||||
pub async fn create_streams(
|
||||
meta_client: &MetaClient,
|
||||
local_gc_enabled: bool,
|
||||
running: Arc<AtomicBool>,
|
||||
handler_executor: HeartbeatResponseHandlerExecutorRef,
|
||||
mailbox: MailboxRef,
|
||||
@@ -129,6 +130,13 @@ impl HeartbeatTask {
|
||||
) -> Result<(HeartbeatSender, HeartbeatConfig)> {
|
||||
let client_id = meta_client.id();
|
||||
let (tx, mut rx, config) = meta_client.heartbeat().await.context(MetaClientInitSnafu)?;
|
||||
if config.gc_enabled != local_gc_enabled {
|
||||
return error::GcConfigMismatchSnafu {
|
||||
metasrv_gc_enabled: config.gc_enabled,
|
||||
datanode_gc_enabled: local_gc_enabled,
|
||||
}
|
||||
.fail();
|
||||
}
|
||||
|
||||
let mut last_received_lease = Instant::now();
|
||||
|
||||
@@ -220,9 +228,17 @@ impl HeartbeatTask {
|
||||
let mailbox = Arc::new(HeartbeatMailbox::new(outgoing_tx));
|
||||
|
||||
let quit_signal = Arc::new(Notify::new());
|
||||
let local_gc_enabled = self
|
||||
.region_server
|
||||
.mito_engine()
|
||||
.context(RegionEngineNotFoundSnafu { name: "mito" })?
|
||||
.mito_config()
|
||||
.gc
|
||||
.enable;
|
||||
|
||||
let (mut tx, config) = Self::create_streams(
|
||||
&meta_client,
|
||||
local_gc_enabled,
|
||||
running.clone(),
|
||||
handler_executor.clone(),
|
||||
mailbox.clone(),
|
||||
@@ -368,6 +384,7 @@ impl HeartbeatTask {
|
||||
error!(e; "Failed to send heartbeat to metasrv");
|
||||
match Self::create_streams(
|
||||
&meta_client,
|
||||
local_gc_enabled,
|
||||
running.clone(),
|
||||
handler_executor.clone(),
|
||||
mailbox.clone(),
|
||||
|
||||
@@ -522,7 +522,10 @@ mod tests {
|
||||
.unwrap();
|
||||
|
||||
region_server
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut heartbeat_env = HeartbeatResponseTestEnv::new();
|
||||
|
||||
@@ -40,8 +40,10 @@ impl InstructionHandler for CloseRegionsHandler {
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let futs = region_ids.iter().map(|region_id| {
|
||||
ctx.region_server
|
||||
.handle_request(*region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
ctx.region_server.handle_request(
|
||||
*region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
});
|
||||
|
||||
let results = join_all(futs).await;
|
||||
|
||||
@@ -169,11 +169,17 @@ mod tests {
|
||||
.await
|
||||
.unwrap();
|
||||
region_server
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
region_server
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
|
||||
@@ -1214,6 +1214,24 @@ impl RegionServerInner {
|
||||
})?
|
||||
.clone(),
|
||||
},
|
||||
RegionChange::OfflineCleanup(attribute) => match current_region_status {
|
||||
Some(status) => match status.clone() {
|
||||
RegionEngineWithStatus::Registering(_)
|
||||
| RegionEngineWithStatus::Deregistering(_)
|
||||
| RegionEngineWithStatus::Ready(_) => {
|
||||
return error::RegionBusySnafu { region_id }.fail();
|
||||
}
|
||||
},
|
||||
None => self
|
||||
.engines
|
||||
.read()
|
||||
.unwrap()
|
||||
.get(attribute.engine())
|
||||
.with_context(|| RegionEngineNotFoundSnafu {
|
||||
name: attribute.engine(),
|
||||
})?
|
||||
.clone(),
|
||||
},
|
||||
RegionChange::Deregisters => match current_region_status {
|
||||
Some(status) => match status.clone() {
|
||||
RegionEngineWithStatus::Registering(_) => {
|
||||
@@ -1559,6 +1577,10 @@ impl RegionServerInner {
|
||||
let attribute = parse_region_attribute(&open.engine, &open.options)?;
|
||||
RegionChange::Register(attribute)
|
||||
}
|
||||
RegionRequest::CleanUp(clean_up) => {
|
||||
let attribute = parse_region_attribute(&clean_up.engine, &clean_up.options)?;
|
||||
RegionChange::OfflineCleanup(attribute)
|
||||
}
|
||||
RegionRequest::Close(_) | RegionRequest::Drop(_) => RegionChange::Deregisters,
|
||||
RegionRequest::Put(_) | RegionRequest::Delete(_) | RegionRequest::BulkInserts(_) => {
|
||||
RegionChange::Ingest
|
||||
@@ -1678,7 +1700,7 @@ impl RegionServerInner {
|
||||
region_change: RegionChange,
|
||||
) {
|
||||
match region_change {
|
||||
RegionChange::None | RegionChange::Ingest => {}
|
||||
RegionChange::None | RegionChange::Ingest | RegionChange::OfflineCleanup(_) => {}
|
||||
RegionChange::Register(_) => {
|
||||
self.region_map.remove(®ion_id);
|
||||
}
|
||||
@@ -1698,7 +1720,7 @@ impl RegionServerInner {
|
||||
) -> Result<()> {
|
||||
let engine_type = engine.name();
|
||||
match region_change {
|
||||
RegionChange::None | RegionChange::Ingest => {}
|
||||
RegionChange::None | RegionChange::Ingest | RegionChange::OfflineCleanup(_) => {}
|
||||
RegionChange::Register(attribute) => {
|
||||
info!(
|
||||
"Region {region_id} is registered to engine {}",
|
||||
@@ -1852,7 +1874,10 @@ impl RegionServerInner {
|
||||
|
||||
for (region_id, engine) in regions {
|
||||
let closed = engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await;
|
||||
match closed {
|
||||
Ok(_) => debug!("Region {region_id} is closed"),
|
||||
@@ -1880,6 +1905,7 @@ impl RegionServerInner {
|
||||
enum RegionChange {
|
||||
None,
|
||||
Register(RegionAttribute),
|
||||
OfflineCleanup(RegionAttribute),
|
||||
Deregisters,
|
||||
Catchup,
|
||||
Ingest,
|
||||
@@ -1944,8 +1970,8 @@ mod tests {
|
||||
use store_api::metadata::{ColumnMetadata, RegionMetadata, RegionMetadataBuilder};
|
||||
use store_api::region_engine::RegionEngine;
|
||||
use store_api::region_request::{
|
||||
PathType, RegionCompactRequest, RegionDeleteRequest, RegionDropRequest, RegionOpenRequest,
|
||||
RegionPutRequest, RegionTruncateRequest,
|
||||
PathType, RegionCleanUpRequest, RegionCompactRequest, RegionDeleteRequest,
|
||||
RegionDropRequest, RegionOpenRequest, RegionPutRequest, RegionTruncateRequest,
|
||||
};
|
||||
use store_api::storage::RegionId;
|
||||
|
||||
@@ -2116,6 +2142,77 @@ mod tests {
|
||||
assert!(pinned.as_ref().get_ref().metrics().is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_offline_cleanup_does_not_register_region() {
|
||||
let mut mock_region_server = mock_region_server();
|
||||
let (engine, mut receiver) = MockRegionEngine::new(MITO_ENGINE_NAME);
|
||||
mock_region_server.register_engine(engine);
|
||||
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let response = mock_region_server
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::CleanUp(RegionCleanUpRequest {
|
||||
engine: MITO_ENGINE_NAME.to_string(),
|
||||
table_dir: String::new(),
|
||||
path_type: PathType::Bare,
|
||||
options: HashMap::new(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(response.affected_rows, 0);
|
||||
let (handled_region_id, handled_request) = receiver.try_recv().unwrap();
|
||||
assert_eq!(handled_region_id, region_id);
|
||||
assert_matches!(handled_request, RegionRequest::CleanUp(_));
|
||||
assert!(
|
||||
mock_region_server
|
||||
.inner
|
||||
.region_map
|
||||
.get(®ion_id)
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_offline_cleanup_rejects_registered_region() {
|
||||
let mut mock_region_server = mock_region_server();
|
||||
let (engine, mut receiver) = MockRegionEngine::new(MITO_ENGINE_NAME);
|
||||
mock_region_server.register_engine(engine.clone());
|
||||
|
||||
let region_id = RegionId::new(1, 1);
|
||||
mock_region_server
|
||||
.inner
|
||||
.region_map
|
||||
.insert(region_id, RegionEngineWithStatus::Ready(engine));
|
||||
|
||||
let err = mock_region_server
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::CleanUp(RegionCleanUpRequest {
|
||||
engine: MITO_ENGINE_NAME.to_string(),
|
||||
table_dir: String::new(),
|
||||
path_type: PathType::Bare,
|
||||
options: HashMap::new(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.status_code(), StatusCode::RegionBusy);
|
||||
assert!(receiver.try_recv().is_err());
|
||||
assert!(matches!(
|
||||
mock_region_server
|
||||
.inner
|
||||
.region_map
|
||||
.get(®ion_id)
|
||||
.unwrap()
|
||||
.clone(),
|
||||
RegionEngineWithStatus::Ready(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_region_registering() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
@@ -2218,7 +2315,10 @@ mod tests {
|
||||
);
|
||||
|
||||
let response = mock_region_server
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.affected_rows, 0);
|
||||
@@ -2524,6 +2624,43 @@ mod tests {
|
||||
assert_matches!(current_engine, CurrentEngine::Engine(_));
|
||||
}),
|
||||
},
|
||||
// RegionChange::OfflineCleanup
|
||||
CurrentEngineTest {
|
||||
region_id,
|
||||
current_region_status: None,
|
||||
region_change: RegionChange::OfflineCleanup(RegionAttribute::Mito),
|
||||
assert: Box::new(|result| {
|
||||
let current_engine = result.unwrap();
|
||||
assert_matches!(current_engine, CurrentEngine::Engine(_));
|
||||
}),
|
||||
},
|
||||
CurrentEngineTest {
|
||||
region_id,
|
||||
current_region_status: Some(RegionEngineWithStatus::Registering(engine.clone())),
|
||||
region_change: RegionChange::OfflineCleanup(RegionAttribute::Mito),
|
||||
assert: Box::new(|result| {
|
||||
let err = result.unwrap_err();
|
||||
assert_eq!(err.status_code(), StatusCode::RegionBusy);
|
||||
}),
|
||||
},
|
||||
CurrentEngineTest {
|
||||
region_id,
|
||||
current_region_status: Some(RegionEngineWithStatus::Deregistering(engine.clone())),
|
||||
region_change: RegionChange::OfflineCleanup(RegionAttribute::Mito),
|
||||
assert: Box::new(|result| {
|
||||
let err = result.unwrap_err();
|
||||
assert_eq!(err.status_code(), StatusCode::RegionBusy);
|
||||
}),
|
||||
},
|
||||
CurrentEngineTest {
|
||||
region_id,
|
||||
current_region_status: Some(RegionEngineWithStatus::Ready(engine.clone())),
|
||||
region_change: RegionChange::OfflineCleanup(RegionAttribute::Mito),
|
||||
assert: Box::new(|result| {
|
||||
let err = result.unwrap_err();
|
||||
assert_eq!(err.status_code(), StatusCode::RegionBusy);
|
||||
}),
|
||||
},
|
||||
];
|
||||
|
||||
for test in tests {
|
||||
|
||||
@@ -317,13 +317,6 @@ impl JsonValue {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn new_with(json_variant: JsonVariant, json_type: Arc<JsonNativeType>) -> Self {
|
||||
Self {
|
||||
json_type: OnceLock::from(json_type),
|
||||
json_variant,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn data_type(&self) -> ConcreteDataType {
|
||||
ConcreteDataType::json2(self.json_type().clone())
|
||||
}
|
||||
@@ -383,10 +376,6 @@ impl JsonValue {
|
||||
self.json_variant
|
||||
}
|
||||
|
||||
pub(crate) fn variant(&self) -> &JsonVariant {
|
||||
&self.json_variant
|
||||
}
|
||||
|
||||
pub(crate) fn into_value(self) -> Value {
|
||||
fn helper(v: JsonVariant) -> Value {
|
||||
match v {
|
||||
@@ -828,6 +817,15 @@ impl<'a> JsonValueRef<'a> {
|
||||
().into()
|
||||
}
|
||||
|
||||
/// Creates a JSON value reference with its precomputed native type.
|
||||
/// The native type must describe `json_variant` exactly.
|
||||
pub fn new_with_type(json_variant: JsonVariantRef<'a>, json_type: JsonNativeType) -> Self {
|
||||
Self {
|
||||
json_type: OnceLock::from(Arc::new(json_type)),
|
||||
json_variant,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn data_type(&self) -> ConcreteDataType {
|
||||
ConcreteDataType::json2(self.json_type().as_ref().clone())
|
||||
}
|
||||
|
||||
+54
-11
@@ -543,20 +543,20 @@ impl Value {
|
||||
None
|
||||
}
|
||||
}
|
||||
Value::Int8(x) => Some(Value::Int8(-*x)),
|
||||
Value::Int16(x) => Some(Value::Int16(-*x)),
|
||||
Value::Int32(x) => Some(Value::Int32(-*x)),
|
||||
Value::Int64(x) => Some(Value::Int64(-*x)),
|
||||
Value::Int8(x) => x.checked_neg().map(Value::Int8),
|
||||
Value::Int16(x) => x.checked_neg().map(Value::Int16),
|
||||
Value::Int32(x) => x.checked_neg().map(Value::Int32),
|
||||
Value::Int64(x) => x.checked_neg().map(Value::Int64),
|
||||
Value::Float32(x) => Some(Value::Float32(-*x)),
|
||||
Value::Float64(x) => Some(Value::Float64(-*x)),
|
||||
Value::Decimal128(x) => Some(Value::Decimal128(x.negative())),
|
||||
Value::Date(x) => Some(Value::Date(x.negative())),
|
||||
Value::Timestamp(x) => Some(Value::Timestamp(x.negative())),
|
||||
Value::Time(x) => Some(Value::Time(x.negative())),
|
||||
Value::Duration(x) => Some(Value::Duration(x.negative())),
|
||||
Value::IntervalYearMonth(x) => Some(Value::IntervalYearMonth(x.negative())),
|
||||
Value::IntervalDayTime(x) => Some(Value::IntervalDayTime(x.negative())),
|
||||
Value::IntervalMonthDayNano(x) => Some(Value::IntervalMonthDayNano(x.negative())),
|
||||
Value::Date(x) => x.checked_negative().map(Value::Date),
|
||||
Value::Timestamp(x) => x.checked_negative().map(Value::Timestamp),
|
||||
Value::Time(x) => x.checked_negative().map(Value::Time),
|
||||
Value::Duration(x) => x.checked_negative().map(Value::Duration),
|
||||
Value::IntervalYearMonth(x) => x.checked_negative().map(Value::IntervalYearMonth),
|
||||
Value::IntervalDayTime(x) => x.checked_negative().map(Value::IntervalDayTime),
|
||||
Value::IntervalMonthDayNano(x) => x.checked_negative().map(Value::IntervalMonthDayNano),
|
||||
|
||||
Value::Binary(_)
|
||||
| Value::String(_)
|
||||
@@ -1744,6 +1744,49 @@ pub(crate) mod tests {
|
||||
use crate::types::json_type::{JsonNativeType, JsonObjectType};
|
||||
use crate::vectors::ListVectorBuilder;
|
||||
|
||||
#[test]
|
||||
fn test_try_negative_overflow() {
|
||||
// Negating a MIN value overflows, so try_negative returns None instead
|
||||
// of panicking (consistent with the unsigned arms).
|
||||
assert_eq!(Value::Int8(i8::MIN).try_negative(), None);
|
||||
assert_eq!(Value::Int16(i16::MIN).try_negative(), None);
|
||||
assert_eq!(Value::Int32(i32::MIN).try_negative(), None);
|
||||
assert_eq!(Value::Int64(i64::MIN).try_negative(), None);
|
||||
assert_eq!(
|
||||
Value::Timestamp(Timestamp::new_nanosecond(i64::MIN)).try_negative(),
|
||||
None
|
||||
);
|
||||
assert_eq!(Value::Date(Date::new(i32::MIN)).try_negative(), None);
|
||||
assert_eq!(
|
||||
Value::Time(Time::new_nanosecond(i64::MIN)).try_negative(),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
Value::Duration(Duration::new_nanosecond(i64::MIN)).try_negative(),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
Value::IntervalYearMonth(IntervalYearMonth::new(i32::MIN)).try_negative(),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
Value::IntervalDayTime(IntervalDayTime::new(i32::MIN, i32::MIN)).try_negative(),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
Value::IntervalMonthDayNano(IntervalMonthDayNano::new(i32::MIN, i32::MIN, i64::MIN))
|
||||
.try_negative(),
|
||||
None
|
||||
);
|
||||
|
||||
// Non-MIN values still negate.
|
||||
assert_eq!(Value::Int64(5).try_negative(), Some(Value::Int64(-5)));
|
||||
assert_eq!(
|
||||
Value::Timestamp(Timestamp::new_nanosecond(5)).try_negative(),
|
||||
Some(Value::Timestamp(Timestamp::new_nanosecond(-5)))
|
||||
);
|
||||
}
|
||||
|
||||
pub(crate) fn build_struct_type() -> StructType {
|
||||
StructType::new(Arc::new(vec![
|
||||
StructField::new("id".to_string(), ConcreteDataType::int32_datatype(), false),
|
||||
|
||||
@@ -19,17 +19,17 @@ use arrow_schema::DataType;
|
||||
|
||||
use crate::data_type::ConcreteDataType;
|
||||
use crate::error::{Result, TryFromValueSnafu, UnexpectedSnafu, UnsupportedOperationSnafu};
|
||||
use crate::json::value::{JsonNumber, JsonValue, JsonVariant};
|
||||
use crate::json::value::{JsonNumber, JsonVariant, encode_json_variant};
|
||||
use crate::prelude::{ValueRef, Vector, VectorRef};
|
||||
use crate::types::StructType;
|
||||
use crate::types::json_type::{JsonNativeType, is_include};
|
||||
use crate::value::{ListValueRef, StructValueRef};
|
||||
use crate::value::{ListValue, StructValue, StructValueRef, Value};
|
||||
use crate::vectors::{MutableVector, StructVectorBuilder};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct JsonVectorBuilder {
|
||||
merged_type: JsonNativeType,
|
||||
values: Vec<JsonValue>,
|
||||
values: Vec<JsonVariant>,
|
||||
}
|
||||
|
||||
impl JsonVectorBuilder {
|
||||
@@ -56,23 +56,22 @@ impl JsonVectorBuilder {
|
||||
|
||||
let mut builder =
|
||||
StructVectorBuilder::with_type_and_capacity(struct_type.clone(), self.values.len());
|
||||
for value in self.values.iter_mut() {
|
||||
if value.is_null() {
|
||||
for value in std::mem::take(&mut self.values) {
|
||||
if matches!(&value, JsonVariant::Null) {
|
||||
builder.push_null();
|
||||
continue;
|
||||
}
|
||||
value.try_align(&self.merged_type)?;
|
||||
let value_ref = json_variant_to_struct_value_ref(value.variant(), struct_type.clone())?;
|
||||
builder.push_struct_value_ref(value_ref)?;
|
||||
let value = json_variant_into_struct_value(value, struct_type.clone())?;
|
||||
builder.push_struct_value_ref(StructValueRef::Ref(&value))?;
|
||||
}
|
||||
Ok(builder.to_vector())
|
||||
}
|
||||
}
|
||||
|
||||
fn json_variant_to_struct_value_ref(
|
||||
value: &JsonVariant,
|
||||
fn json_variant_into_struct_value(
|
||||
value: JsonVariant,
|
||||
struct_type: StructType,
|
||||
) -> Result<StructValueRef<'_>> {
|
||||
) -> Result<StructValue> {
|
||||
let JsonVariant::Object(object) = value else {
|
||||
return TryFromValueSnafu {
|
||||
reason: format!("expected json object value, got {value:?}"),
|
||||
@@ -80,63 +79,87 @@ fn json_variant_to_struct_value_ref(
|
||||
.fail();
|
||||
};
|
||||
|
||||
let values = struct_type
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|field| {
|
||||
object
|
||||
.get(field.name())
|
||||
.map(|v| json_variant_to_value_ref(v, field.data_type()))
|
||||
.unwrap_or(Ok(ValueRef::Null))
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
let mut entries = object.into_iter();
|
||||
let mut entry = entries.next();
|
||||
let mut values = Vec::with_capacity(struct_type.fields().len());
|
||||
for field in struct_type.fields().iter() {
|
||||
let value = match entry.take() {
|
||||
Some((name, value)) if name == field.name() => {
|
||||
entry = entries.next();
|
||||
json_variant_into_value(value, field.data_type())?
|
||||
}
|
||||
Some((name, _)) if name.as_str() < field.name() => {
|
||||
return TryFromValueSnafu {
|
||||
reason: format!("field {name} is missing from merged JSON type"),
|
||||
}
|
||||
.fail();
|
||||
}
|
||||
next => {
|
||||
entry = next;
|
||||
Value::Null
|
||||
}
|
||||
};
|
||||
values.push(value);
|
||||
}
|
||||
if let Some((name, _)) = entry {
|
||||
return TryFromValueSnafu {
|
||||
reason: format!("field {name} is missing from merged JSON type"),
|
||||
}
|
||||
.fail();
|
||||
}
|
||||
|
||||
Ok(StructValueRef::RefList {
|
||||
val: values,
|
||||
fields: struct_type,
|
||||
})
|
||||
Ok(StructValue::new(values, struct_type))
|
||||
}
|
||||
|
||||
fn json_variant_to_value_ref<'a>(
|
||||
value: &'a JsonVariant,
|
||||
expected_type: &ConcreteDataType,
|
||||
) -> Result<ValueRef<'a>> {
|
||||
let value = match value {
|
||||
JsonVariant::Null => ValueRef::Null,
|
||||
JsonVariant::Bool(x) => ValueRef::Boolean(*x),
|
||||
JsonVariant::Number(x) => match x {
|
||||
JsonNumber::PosInt(i) => ValueRef::UInt64(*i),
|
||||
JsonNumber::NegInt(i) => ValueRef::Int64(*i),
|
||||
JsonNumber::Float(f) => ValueRef::Float64(*f),
|
||||
},
|
||||
JsonVariant::String(x) => ValueRef::String(x),
|
||||
JsonVariant::Array(array) => {
|
||||
let item_type = match expected_type {
|
||||
ConcreteDataType::List(list_type) => list_type.item_type().clone(),
|
||||
_ => ConcreteDataType::null_datatype(),
|
||||
};
|
||||
let values = array
|
||||
.iter()
|
||||
.map(|v| json_variant_to_value_ref(v, &item_type))
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
ValueRef::List(ListValueRef::RefList {
|
||||
val: values,
|
||||
item_datatype: Arc::new(item_type),
|
||||
})
|
||||
fn json_variant_into_value(value: JsonVariant, expected_type: &ConcreteDataType) -> Result<Value> {
|
||||
let value = match (value, expected_type) {
|
||||
(JsonVariant::Null, _) | (_, ConcreteDataType::Null(_)) => Value::Null,
|
||||
(JsonVariant::Bool(x), ConcreteDataType::Boolean(_)) => Value::Boolean(x),
|
||||
(JsonVariant::Number(JsonNumber::PosInt(x)), ConcreteDataType::UInt64(_)) => {
|
||||
Value::UInt64(x)
|
||||
}
|
||||
JsonVariant::Object(_) => {
|
||||
let ConcreteDataType::Struct(struct_type) = expected_type else {
|
||||
(JsonVariant::Number(x), ConcreteDataType::Int64(_)) => {
|
||||
let x = match x {
|
||||
JsonNumber::PosInt(x) => i64::try_from(x).ok(),
|
||||
JsonNumber::NegInt(x) => Some(x),
|
||||
JsonNumber::Float(_) => None,
|
||||
};
|
||||
let Some(x) = x else {
|
||||
return TryFromValueSnafu {
|
||||
reason: format!("expected struct type, got {expected_type}"),
|
||||
reason: format!("unable to convert {x:?} to Int64"),
|
||||
}
|
||||
.fail();
|
||||
};
|
||||
ValueRef::Struct(json_variant_to_struct_value_ref(
|
||||
value,
|
||||
struct_type.clone(),
|
||||
)?)
|
||||
Value::Int64(x)
|
||||
}
|
||||
(JsonVariant::Number(JsonNumber::PosInt(x)), ConcreteDataType::Float64(_)) => {
|
||||
Value::Float64((x as f64).into())
|
||||
}
|
||||
(JsonVariant::Number(JsonNumber::NegInt(x)), ConcreteDataType::Float64(_)) => {
|
||||
Value::Float64((x as f64).into())
|
||||
}
|
||||
(JsonVariant::Number(JsonNumber::Float(x)), ConcreteDataType::Float64(_)) => {
|
||||
Value::Float64(x)
|
||||
}
|
||||
(JsonVariant::String(x), ConcreteDataType::String(_)) => Value::String(x.into()),
|
||||
(JsonVariant::Array(array), ConcreteDataType::List(list_type)) => {
|
||||
let item_type = list_type.item_type().clone();
|
||||
let values = array
|
||||
.into_iter()
|
||||
.map(|v| json_variant_into_value(v, &item_type))
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
Value::List(ListValue::new(values, Arc::new(item_type)))
|
||||
}
|
||||
(value @ JsonVariant::Object(_), ConcreteDataType::Struct(struct_type)) => {
|
||||
Value::Struct(json_variant_into_struct_value(value, struct_type.clone())?)
|
||||
}
|
||||
(value, ConcreteDataType::Binary(_)) => Value::from(encode_json_variant(value)?),
|
||||
(value, expected_type) => {
|
||||
return TryFromValueSnafu {
|
||||
reason: format!("unable to convert json value {value:?} to {expected_type}"),
|
||||
}
|
||||
.fail();
|
||||
}
|
||||
JsonVariant::Variant(x) => ValueRef::Binary(x),
|
||||
};
|
||||
Ok(value)
|
||||
}
|
||||
@@ -185,13 +208,12 @@ impl MutableVector for JsonVectorBuilder {
|
||||
self.merged_type.merge(json_type);
|
||||
}
|
||||
|
||||
let value = JsonValue::new_with(JsonVariant::from(value.variant()), value.json_type());
|
||||
self.values.push(value);
|
||||
self.values.push(JsonVariant::from(value.variant()));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn push_null(&mut self) {
|
||||
self.values.push(JsonValue::null())
|
||||
self.values.push(JsonVariant::Null)
|
||||
}
|
||||
|
||||
fn extend_slice_of(&mut self, _: &dyn Vector, _: usize, _: usize) -> Result<()> {
|
||||
@@ -227,7 +249,7 @@ mod tests {
|
||||
}
|
||||
|
||||
// Object inputs should merge into a superset schema, preserve null rows,
|
||||
// and align conflicting nested values into Variant payloads.
|
||||
// and project conflicting nested values into Variant payloads.
|
||||
let mut builder = JsonVectorBuilder::new(JsonNativeType::Object(Default::default()), 3);
|
||||
let first = parse_json_value(r#"{"id":1,"payload":{"name":"foo"}}"#);
|
||||
let second = parse_json_value(r#"{"id":2,"extra":true,"payload":"raw"}"#);
|
||||
@@ -332,7 +354,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_json_variant_to_struct_value_ref() -> Result<()> {
|
||||
fn test_json_variant_into_struct_value() -> Result<()> {
|
||||
let item_type =
|
||||
ConcreteDataType::struct_datatype(StructType::new(Arc::new(vec![StructField::new(
|
||||
"id".to_string(),
|
||||
@@ -370,8 +392,10 @@ mod tests {
|
||||
JsonVariant::from([("name", JsonVariant::from("foo"))]),
|
||||
),
|
||||
]);
|
||||
let value_ref = json_variant_to_struct_value_ref(&variant, struct_type.clone())?;
|
||||
let value = value_ref.to_value();
|
||||
let value = Value::Struct(json_variant_into_struct_value(
|
||||
variant,
|
||||
struct_type.clone(),
|
||||
)?);
|
||||
|
||||
assert_eq!(
|
||||
value,
|
||||
|
||||
+21
-1
@@ -76,6 +76,23 @@ pub(crate) mod table_source;
|
||||
use crate::FrontendInvoker;
|
||||
use crate::error::Error;
|
||||
|
||||
fn expire_after_secs_to_millis(expire_after_secs: i64) -> Result<repr::Duration, Error> {
|
||||
ensure!(
|
||||
expire_after_secs >= 0,
|
||||
InvalidQuerySnafu {
|
||||
reason: format!("EXPIRE AFTER must be non-negative, got {expire_after_secs} seconds"),
|
||||
}
|
||||
);
|
||||
|
||||
expire_after_secs
|
||||
.checked_mul(1_000)
|
||||
.with_context(|| InvalidQuerySnafu {
|
||||
reason: format!(
|
||||
"EXPIRE AFTER value {expire_after_secs} seconds cannot be represented in milliseconds"
|
||||
),
|
||||
})
|
||||
}
|
||||
|
||||
// `GREPTIME_TIMESTAMP` is not used to distinguish when table is created automatically by flow
|
||||
pub const AUTO_CREATED_PLACEHOLDER_TS_COL: &str = "__ts_placeholder";
|
||||
|
||||
@@ -740,7 +757,7 @@ impl StreamingEngine {
|
||||
source_table_ids,
|
||||
create_if_not_exists,
|
||||
or_replace,
|
||||
expire_after,
|
||||
expire_after: expire_after_secs,
|
||||
eval_interval: _,
|
||||
comment,
|
||||
sql,
|
||||
@@ -748,6 +765,9 @@ impl StreamingEngine {
|
||||
query_ctx,
|
||||
..
|
||||
} = args;
|
||||
let expire_after = expire_after_secs
|
||||
.map(expire_after_secs_to_millis)
|
||||
.transpose()?;
|
||||
|
||||
let mut node_ctx = self.node_context.write().await;
|
||||
// assign global id to source and sink table
|
||||
|
||||
@@ -85,7 +85,10 @@ impl StreamingEngine {
|
||||
}
|
||||
}
|
||||
|
||||
let expire_after = info.expire_after();
|
||||
let expire_after = info
|
||||
.expire_after()
|
||||
.map(super::expire_after_secs_to_millis)
|
||||
.transpose()?;
|
||||
// TODO(discord9): better way to get last point
|
||||
let now = self.tick_manager.tick();
|
||||
let plan = self
|
||||
|
||||
@@ -61,3 +61,25 @@ pub fn new_test_table_info_with_name<I: IntoIterator<Item = u32>>(
|
||||
///
|
||||
/// containing several default table info and schema
|
||||
fn mock_harness_flow_node_manager() {}
|
||||
|
||||
#[test]
|
||||
fn test_expire_after_secs_to_millis() {
|
||||
assert_eq!(expire_after_secs_to_millis(300).unwrap(), 300_000);
|
||||
assert_eq!(expire_after_secs_to_millis(0).unwrap(), 0);
|
||||
|
||||
let max_secs = i64::MAX / 1_000;
|
||||
assert_eq!(
|
||||
expire_after_secs_to_millis(max_secs).unwrap(),
|
||||
max_secs * 1_000
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_expire_after_secs_to_millis_invalid() {
|
||||
let invalid_values = [i64::MAX / 1_000 + 1, -1];
|
||||
|
||||
for invalid_secs in invalid_values {
|
||||
let error = expire_after_secs_to_millis(invalid_secs).unwrap_err();
|
||||
assert!(matches!(error, Error::InvalidQuery { .. }));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -34,6 +34,7 @@ pub struct CreateFlowArgs {
|
||||
pub source_table_ids: Vec<TableId>,
|
||||
pub create_if_not_exists: bool,
|
||||
pub or_replace: bool,
|
||||
/// Duration in seconds.
|
||||
pub expire_after: Option<i64>,
|
||||
pub eval_interval: Option<i64>,
|
||||
pub comment: Option<String>,
|
||||
|
||||
@@ -237,6 +237,7 @@ mod tests {
|
||||
is_handshake.then_some(api::v1::meta::HeartbeatConfig {
|
||||
heartbeat_interval_ms,
|
||||
retry_interval_ms: heartbeat_interval_ms,
|
||||
gc_enabled: false,
|
||||
});
|
||||
is_handshake = false;
|
||||
let response = HeartbeatResponse {
|
||||
|
||||
@@ -611,7 +611,11 @@ impl Instance {
|
||||
let checker_ref = self.plugins.get::<PermissionCheckerRef>();
|
||||
checker_ref
|
||||
.as_ref()
|
||||
.check_permission(query_ctx.current_user(), PermissionReq::SqlStatement(&stmt))
|
||||
.check_permission_with_context(
|
||||
query_ctx.current_user(),
|
||||
PermissionReq::SqlStatement(&stmt),
|
||||
Some(&query_ctx.current_schema()),
|
||||
)
|
||||
.context(PermissionSnafu)?;
|
||||
check_permission(self.plugins.clone(), &stmt, &query_ctx)?;
|
||||
let catalog_name = query_ctx.current_catalog().to_string();
|
||||
@@ -672,9 +676,10 @@ impl Instance {
|
||||
let mut results = Vec::with_capacity(stmts.len());
|
||||
for stmt in stmts {
|
||||
if let Err(e) = checker
|
||||
.check_permission(
|
||||
.check_permission_with_context(
|
||||
query_ctx.current_user(),
|
||||
PermissionReq::SqlStatement(&stmt),
|
||||
Some(&query_ctx.current_schema()),
|
||||
)
|
||||
.context(PermissionSnafu)
|
||||
{
|
||||
@@ -848,7 +853,11 @@ impl Instance {
|
||||
self.plugins
|
||||
.get::<PermissionCheckerRef>()
|
||||
.as_ref()
|
||||
.check_permission(query_ctx.current_user(), PermissionReq::SqlStatement(&stmt))
|
||||
.check_permission_with_context(
|
||||
query_ctx.current_user(),
|
||||
PermissionReq::SqlStatement(&stmt),
|
||||
Some(&query_ctx.current_schema()),
|
||||
)
|
||||
.context(PermissionSnafu)?;
|
||||
|
||||
let plan = self
|
||||
|
||||
@@ -71,7 +71,11 @@ impl GrpcQueryHandler for Instance {
|
||||
self.plugins
|
||||
.get::<PermissionCheckerRef>()
|
||||
.as_ref()
|
||||
.check_permission(ctx.current_user(), PermissionReq::GrpcRequest(&request))
|
||||
.check_permission_with_context(
|
||||
ctx.current_user(),
|
||||
PermissionReq::GrpcRequest(&request),
|
||||
Some(&ctx.current_schema()),
|
||||
)
|
||||
.context(PermissionSnafu)?;
|
||||
|
||||
let output = match request {
|
||||
@@ -356,7 +360,14 @@ impl Instance {
|
||||
plugins
|
||||
.get::<PermissionCheckerRef>()
|
||||
.as_ref()
|
||||
.check_permission(ctx.current_user(), PermissionReq::BulkInsert)
|
||||
.check_permission(
|
||||
ctx.current_user(),
|
||||
PermissionReq::BulkInsert {
|
||||
catalog: &table_name.catalog_name,
|
||||
schema: &table_name.schema_name,
|
||||
table: &table_name.table_name,
|
||||
},
|
||||
)
|
||||
.context(PermissionSnafu)?;
|
||||
|
||||
// Resolve table reference
|
||||
|
||||
@@ -214,9 +214,9 @@ impl TimeFilter {
|
||||
(Some(start), Some(end), _) => {
|
||||
// Both 'start' and 'end' are provided
|
||||
let (start, _) = Self::parse_datetime(start)?;
|
||||
let (end, _) = Self::parse_datetime(end)?;
|
||||
let (end, inferred_end) = Self::parse_datetime(end)?;
|
||||
start_dt = Some(start);
|
||||
end_dt = Some(end);
|
||||
end_dt = Some(inferred_end.unwrap_or(end));
|
||||
}
|
||||
(Some(start), None, Some(span)) => {
|
||||
let (start, _) = Self::parse_datetime(start)?;
|
||||
@@ -276,41 +276,39 @@ impl TimeFilter {
|
||||
if let Ok(dt) = DateTime::parse_from_rfc3339(s) {
|
||||
Ok((dt.with_timezone(&Utc), None))
|
||||
} else {
|
||||
let formats = ["%Y-%m-%d", "%Y-%m", "%Y"];
|
||||
for format in &formats {
|
||||
if let Ok(naive_date) = NaiveDate::parse_from_str(s, format) {
|
||||
let start = Utc.from_utc_datetime(
|
||||
&naive_date.and_time(NaiveTime::from_hms_opt(0, 0, 0).unwrap()),
|
||||
);
|
||||
let end = match *format {
|
||||
"%Y-%m-%d" => start + Duration::days(1),
|
||||
"%Y-%m" => {
|
||||
let next_month = if naive_date.month() == 12 {
|
||||
NaiveDate::from_ymd_opt(naive_date.year() + 1, 1, 1).unwrap()
|
||||
} else {
|
||||
NaiveDate::from_ymd_opt(
|
||||
naive_date.year(),
|
||||
naive_date.month() + 1,
|
||||
1,
|
||||
)
|
||||
.unwrap()
|
||||
};
|
||||
Utc.from_utc_datetime(&next_month.and_hms_opt(0, 0, 0).unwrap())
|
||||
}
|
||||
"%Y" => {
|
||||
let next_year =
|
||||
NaiveDate::from_ymd_opt(naive_date.year() + 1, 1, 1).unwrap();
|
||||
Utc.from_utc_datetime(&next_year.and_hms_opt(0, 0, 0).unwrap())
|
||||
}
|
||||
_ => unreachable!(),
|
||||
};
|
||||
return Ok((start, Some(end)));
|
||||
let (naive_date, format) = if let Ok(date) = NaiveDate::parse_from_str(s, "%Y-%m-%d") {
|
||||
(date, "%Y-%m-%d")
|
||||
} else if let Ok(date) = NaiveDate::parse_from_str(&format!("{s}-01"), "%Y-%m-%d") {
|
||||
(date, "%Y-%m")
|
||||
} else if let Ok(date) = NaiveDate::parse_from_str(&format!("{s}-01-01"), "%Y-%m-%d") {
|
||||
(date, "%Y")
|
||||
} else {
|
||||
return Err(InvalidDateFormatSnafu {
|
||||
input: s.to_string(),
|
||||
}
|
||||
}
|
||||
Err(InvalidDateFormatSnafu {
|
||||
input: s.to_string(),
|
||||
}
|
||||
.build())
|
||||
.build());
|
||||
};
|
||||
|
||||
let start = Utc
|
||||
.from_utc_datetime(&naive_date.and_time(NaiveTime::from_hms_opt(0, 0, 0).unwrap()));
|
||||
let end = match format {
|
||||
"%Y-%m-%d" => start + Duration::days(1),
|
||||
"%Y-%m" => {
|
||||
let next_month = if naive_date.month() == 12 {
|
||||
NaiveDate::from_ymd_opt(naive_date.year() + 1, 1, 1).unwrap()
|
||||
} else {
|
||||
NaiveDate::from_ymd_opt(naive_date.year(), naive_date.month() + 1, 1)
|
||||
.unwrap()
|
||||
};
|
||||
Utc.from_utc_datetime(&next_month.and_hms_opt(0, 0, 0).unwrap())
|
||||
}
|
||||
"%Y" => {
|
||||
let next_year = NaiveDate::from_ymd_opt(naive_date.year() + 1, 1, 1).unwrap();
|
||||
Utc.from_utc_datetime(&next_year.and_hms_opt(0, 0, 0).unwrap())
|
||||
}
|
||||
_ => unreachable!(),
|
||||
};
|
||||
Ok((start, Some(end)))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -545,4 +543,44 @@ mod tests {
|
||||
let result = tf.canonicalize();
|
||||
assert!(matches!(result, Err(Error::EndBeforeStart { .. })));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_canonicalize_date_end() {
|
||||
let mut date_end = TimeFilter {
|
||||
start: Some("2023-10-01".to_string()),
|
||||
end: Some("2023-10-02".to_string()),
|
||||
span: None,
|
||||
};
|
||||
date_end.canonicalize().unwrap();
|
||||
assert_eq!(date_end.end.as_deref(), Some("2023-10-03T00:00:00+00:00"));
|
||||
|
||||
let mut timestamp_end = TimeFilter {
|
||||
start: Some("2023-10-01".to_string()),
|
||||
end: Some("2023-10-02T12:34:56Z".to_string()),
|
||||
span: None,
|
||||
};
|
||||
timestamp_end.canonicalize().unwrap();
|
||||
assert_eq!(
|
||||
timestamp_end.end.as_deref(),
|
||||
Some("2023-10-02T12:34:56+00:00")
|
||||
);
|
||||
|
||||
let mut month = TimeFilter {
|
||||
start: Some("2023-10".to_string()),
|
||||
end: Some("2023-10".to_string()),
|
||||
span: None,
|
||||
};
|
||||
month.canonicalize().unwrap();
|
||||
assert_eq!(month.start.as_deref(), Some("2023-10-01T00:00:00+00:00"));
|
||||
assert_eq!(month.end.as_deref(), Some("2023-11-01T00:00:00+00:00"));
|
||||
|
||||
let mut year = TimeFilter {
|
||||
start: Some("2023".to_string()),
|
||||
end: Some("2023".to_string()),
|
||||
span: None,
|
||||
};
|
||||
year.canonicalize().unwrap();
|
||||
assert_eq!(year.start.as_deref(), Some("2023-01-01T00:00:00+00:00"));
|
||||
assert_eq!(year.end.as_deref(), Some("2024-01-01T00:00:00+00:00"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -511,6 +511,10 @@ impl LogStore for KafkaLogStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn obsolete_all(&self, _provider: &Provider, _region_id: RegionId) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Returns the highest entry id of the specified topic in remote WAL.
|
||||
fn latest_entry_id(&self, provider: &Provider) -> Result<EntryId> {
|
||||
let provider = provider
|
||||
|
||||
@@ -227,13 +227,17 @@ pub fn remaining_entries(
|
||||
/// - Emits a [RecordType::Full] type record immediately.
|
||||
///
|
||||
/// For type of [Entry::MultiplePart] Entry:
|
||||
/// - Emits a complete or incomplete [Entry] while the next same [RegionId] record arrives.
|
||||
/// - Emits a complete [Entry] immediately when a [RecordType::Last] record arrives with
|
||||
/// buffered parts.
|
||||
/// - Emits an incomplete [Entry] when a new [RecordType::First] record replaces
|
||||
/// buffered incomplete records for the same [RegionId].
|
||||
///
|
||||
/// **Incomplete Entry:**
|
||||
/// If the records arrive in the following order, it emits **the incomplete [Entry]** when the next record arrives.
|
||||
/// - **[RecordType::First], [RecordType::Middle]**, [RecordType::First]
|
||||
/// - **[RecordType::Middle]**, [RecordType::First]
|
||||
/// - **[RecordType::Last]**
|
||||
/// **Discarded records:**
|
||||
/// - A standalone [RecordType::First] record is discarded when another [RecordType::First]
|
||||
/// record with the same payload for the same [RegionId] arrives.
|
||||
/// - A [RecordType::Last] record without buffered parts is discarded.
|
||||
///
|
||||
/// A trailing incomplete entry is emitted by [`remaining_entries`].
|
||||
pub(crate) fn maybe_emit_entry(
|
||||
provider: &Arc<KafkaProvider>,
|
||||
record: Record,
|
||||
@@ -244,7 +248,16 @@ pub(crate) fn maybe_emit_entry(
|
||||
RecordType::Full => entry = Some(convert_to_naive_entry(provider.clone(), record)),
|
||||
RecordType::First => {
|
||||
let region_id = record.meta.ns.region_id.into();
|
||||
let duplicate = buffered_records.get(®ion_id).is_some_and(|records| {
|
||||
records.len() == 1
|
||||
&& records[0].meta.tp == RecordType::First
|
||||
&& records[0].data == record.data
|
||||
});
|
||||
if let Some(records) = buffered_records.insert(region_id, vec![record]) {
|
||||
if duplicate {
|
||||
// A duplicate standalone First cannot form a complete entry.
|
||||
return Ok(None);
|
||||
}
|
||||
// Incomplete entry
|
||||
entry = Some(convert_to_multiple_entry(
|
||||
provider.clone(),
|
||||
@@ -261,6 +274,11 @@ pub(crate) fn maybe_emit_entry(
|
||||
if !records.is_empty() {
|
||||
// Safety: the records are guaranteed not empty if the key exists.
|
||||
let last_record = records.last().unwrap();
|
||||
if matches!(last_record.meta.tp, RecordType::Middle(last_seq) if last_seq == seq)
|
||||
&& last_record.data == record.data
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
let legal = match last_record.meta.tp {
|
||||
// Legal if this record follows a First record.
|
||||
RecordType::First => seq == 1,
|
||||
@@ -290,14 +308,10 @@ pub(crate) fn maybe_emit_entry(
|
||||
provider.clone(),
|
||||
region_id,
|
||||
records,
|
||||
))
|
||||
));
|
||||
} else {
|
||||
// Incomplete entry
|
||||
entry = Some(convert_to_multiple_entry(
|
||||
provider.clone(),
|
||||
region_id,
|
||||
vec![record],
|
||||
))
|
||||
// Intentionally discard a Last record without buffered parts. It is either a
|
||||
// duplicate Last record or the tail of an incomplete multipart entry.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -359,7 +373,16 @@ mod tests {
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
let record = new_test_record(RecordType::First, 2, region_id.as_u64(), vec![2; 100]);
|
||||
let record = new_test_record(RecordType::First, 2, region_id.as_u64(), vec![1; 100]);
|
||||
// An identical First is a duplicate and does not emit an incomplete entry.
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(buffer[®ion_id][0].meta.entry_id, 2);
|
||||
|
||||
let record = new_test_record(RecordType::First, 3, region_id.as_u64(), vec![2; 100]);
|
||||
let incomplete_entry = maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
@@ -369,29 +392,21 @@ mod tests {
|
||||
Entry::MultiplePart(MultiplePartEntry {
|
||||
provider: Provider::Kafka(provider.clone()),
|
||||
region_id,
|
||||
entry_id: 1,
|
||||
entry_id: 2,
|
||||
headers: vec![MultiplePartHeader::First],
|
||||
parts: vec![vec![1; 100]],
|
||||
})
|
||||
);
|
||||
|
||||
// `Last` overwrite `None`
|
||||
// `Last` after `None`
|
||||
let mut buffer = HashMap::new();
|
||||
let record = new_test_record(RecordType::Last, 1, region_id.as_u64(), vec![1; 100]);
|
||||
let incomplete_entry = maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
incomplete_entry,
|
||||
Entry::MultiplePart(MultiplePartEntry {
|
||||
provider: Provider::Kafka(provider.clone()),
|
||||
region_id,
|
||||
entry_id: 1,
|
||||
headers: vec![MultiplePartHeader::Last],
|
||||
parts: vec![vec![1; 100]],
|
||||
})
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert!(buffer.is_empty());
|
||||
|
||||
// `First` overwrite `Middle(0)`
|
||||
let mut buffer = HashMap::new();
|
||||
@@ -451,6 +466,99 @@ mod tests {
|
||||
assert_matches!(err, error::Error::IllegalSequence { .. });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_maybe_emit_entry_deduplicates_multipart_records() {
|
||||
let provider = Arc::new(KafkaProvider::new("my_topic".to_string()));
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let mut buffer = HashMap::new();
|
||||
|
||||
for record in [
|
||||
new_test_record(RecordType::First, 1, region_id.as_u64(), vec![1; 100]),
|
||||
new_test_record(RecordType::Middle(1), 1, region_id.as_u64(), vec![2; 100]),
|
||||
new_test_record(RecordType::Middle(1), 1, region_id.as_u64(), vec![2; 100]),
|
||||
] {
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
let last = new_test_record(RecordType::Last, 1, region_id.as_u64(), vec![3; 100]);
|
||||
let entry = maybe_emit_entry(&provider, last, &mut buffer)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
entry,
|
||||
Entry::MultiplePart(MultiplePartEntry {
|
||||
provider: Provider::Kafka(provider.clone()),
|
||||
region_id,
|
||||
entry_id: 1,
|
||||
headers: vec![
|
||||
MultiplePartHeader::First,
|
||||
MultiplePartHeader::Middle(1),
|
||||
MultiplePartHeader::Last,
|
||||
],
|
||||
parts: vec![vec![1; 100], vec![2; 100], vec![3; 100]],
|
||||
})
|
||||
);
|
||||
let duplicate = new_test_record(RecordType::Last, 1, region_id.as_u64(), vec![3; 100]);
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, duplicate, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_maybe_emit_entry_rejects_conflicting_duplicate_middle_record() {
|
||||
let provider = Arc::new(KafkaProvider::new("my_topic".to_string()));
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let mut buffer = HashMap::new();
|
||||
|
||||
for record in [
|
||||
new_test_record(RecordType::First, 1, region_id.as_u64(), vec![1; 100]),
|
||||
new_test_record(RecordType::Middle(1), 1, region_id.as_u64(), vec![2; 100]),
|
||||
] {
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, record, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
let duplicate = new_test_record(RecordType::Middle(1), 1, region_id.as_u64(), vec![3; 100]);
|
||||
let err = maybe_emit_entry(&provider, duplicate, &mut buffer).unwrap_err();
|
||||
|
||||
assert_matches!(err, error::Error::IllegalSequence { .. });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_maybe_emit_entry_preserves_multipart_entry_order_before_full_record() {
|
||||
let provider = Arc::new(KafkaProvider::new("my_topic".to_string()));
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let mut buffer = HashMap::new();
|
||||
|
||||
let first = new_test_record(RecordType::First, 1, region_id.as_u64(), vec![1; 100]);
|
||||
assert!(
|
||||
maybe_emit_entry(&provider, first, &mut buffer)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
let last = new_test_record(RecordType::Last, 2, region_id.as_u64(), vec![2; 100]);
|
||||
let multipart_entry = maybe_emit_entry(&provider, last, &mut buffer)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
let full = new_test_record(RecordType::Full, 3, region_id.as_u64(), vec![3; 100]);
|
||||
let full_entry = maybe_emit_entry(&provider, full, &mut buffer)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(multipart_entry.entry_id(), 2);
|
||||
assert_eq!(full_entry.entry_id(), 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_meta_size() {
|
||||
let meta = RecordMeta {
|
||||
|
||||
@@ -86,6 +86,10 @@ impl LogStore for NoopLogStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn obsolete_all(&self, _provider: &Provider, _region_id: RegionId) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn latest_entry_id(&self, _provider: &Provider) -> Result<EntryId> {
|
||||
Ok(0)
|
||||
}
|
||||
|
||||
@@ -484,6 +484,12 @@ impl LogStore for RaftEngineLogStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn obsolete_all(&self, provider: &Provider, region_id: RegionId) -> Result<()> {
|
||||
let latest_entry_id = self.latest_entry_id(provider)?;
|
||||
self.obsolete(provider, region_id, latest_entry_id).await?;
|
||||
self.delete_namespace(provider).await
|
||||
}
|
||||
|
||||
fn latest_entry_id(&self, provider: &Provider) -> Result<EntryId> {
|
||||
let ns = provider
|
||||
.as_raft_engine_provider()
|
||||
@@ -576,6 +582,38 @@ mod tests {
|
||||
assert!(logstore.list_namespaces().await.unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_obsolete_all_removes_entries_and_namespace() {
|
||||
let dir = create_temp_dir("raft-engine-logstore-test");
|
||||
let logstore = RaftEngineLogStore::try_new(
|
||||
dir.path().to_str().unwrap().to_string(),
|
||||
&RaftEngineConfig::default(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let provider = Provider::raft_engine_provider(region_id.as_u64());
|
||||
logstore.create_namespace(&provider).await.unwrap();
|
||||
for entry_id in 1..=3 {
|
||||
logstore
|
||||
.append(
|
||||
EntryImpl::create(
|
||||
entry_id,
|
||||
region_id.as_u64(),
|
||||
entry_id.to_string().into_bytes(),
|
||||
)
|
||||
.into(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
logstore.obsolete_all(&provider, region_id).await.unwrap();
|
||||
|
||||
assert_eq!(0, logstore.latest_entry_id(&provider).unwrap());
|
||||
assert!(logstore.list_namespaces().await.unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_append_and_read() {
|
||||
let dir = create_temp_dir("raft-engine-logstore-test");
|
||||
|
||||
@@ -39,6 +39,7 @@ use crate::error::{InvalidResponseHeaderSnafu, Result};
|
||||
pub struct HeartbeatConfig {
|
||||
pub interval: Duration,
|
||||
pub retry_interval: Duration,
|
||||
pub gc_enabled: bool,
|
||||
}
|
||||
|
||||
impl Default for HeartbeatConfig {
|
||||
@@ -46,6 +47,7 @@ impl Default for HeartbeatConfig {
|
||||
Self {
|
||||
interval: BASE_HEARTBEAT_INTERVAL,
|
||||
retry_interval: BASE_HEARTBEAT_INTERVAL,
|
||||
gc_enabled: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -54,8 +56,8 @@ impl fmt::Display for HeartbeatConfig {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
write!(
|
||||
f,
|
||||
"interval={:?}, retry={:?}",
|
||||
self.interval, self.retry_interval
|
||||
"interval={:?}, retry={:?}, gc_enabled={}",
|
||||
self.interval, self.retry_interval, self.gc_enabled
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -68,6 +70,7 @@ impl HeartbeatConfig {
|
||||
Self {
|
||||
interval: Duration::from_millis(cfg.heartbeat_interval_ms),
|
||||
retry_interval: Duration::from_millis(cfg.retry_interval_ms),
|
||||
gc_enabled: cfg.gc_enabled,
|
||||
}
|
||||
} else {
|
||||
let fallback = Self::default();
|
||||
@@ -259,7 +262,6 @@ impl Inner {
|
||||
.map_err(error::Error::from)?
|
||||
.context(error::CreateHeartbeatStreamSnafu)?;
|
||||
|
||||
// Extract heartbeat configuration from handshake response
|
||||
let config = HeartbeatConfig::from_response(&res);
|
||||
|
||||
info!(
|
||||
|
||||
@@ -21,8 +21,8 @@ use std::time::{Duration, Instant};
|
||||
|
||||
use api::v1::meta::mailbox_message::Payload;
|
||||
use api::v1::meta::{
|
||||
HeartbeatRequest, HeartbeatResponse, MailboxMessage, PROTOCOL_VERSION, RegionLease,
|
||||
ResponseHeader, Role,
|
||||
HeartbeatConfig, HeartbeatRequest, HeartbeatResponse, MailboxMessage, PROTOCOL_VERSION,
|
||||
RegionLease, ResponseHeader, Role,
|
||||
};
|
||||
use check_leader_handler::CheckLeaderHandler;
|
||||
use collect_cluster_info_handler::{
|
||||
@@ -387,7 +387,8 @@ impl HeartbeatHandlerGroup {
|
||||
|
||||
// Populate heartbeat_config during handshake
|
||||
let heartbeat_config = if is_handshake {
|
||||
let config = ctx.heartbeat_options_for(role).into();
|
||||
let mut config: HeartbeatConfig = ctx.heartbeat_options_for(role).into();
|
||||
config.gc_enabled = ctx.gc_enabled;
|
||||
|
||||
info!(
|
||||
"Handshake with {:?} node, sending config: {:?}",
|
||||
|
||||
@@ -92,6 +92,7 @@ impl TestEnv {
|
||||
leader_region_registry: self.leader_region_registry.clone(),
|
||||
topic_stats_registry: self.topic_stats_registry.clone(),
|
||||
heartbeat_interval: BASE_HEARTBEAT_INTERVAL,
|
||||
gc_enabled: false,
|
||||
is_handshake: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -175,6 +175,7 @@ impl From<HeartbeatOptions> for HeartbeatConfig {
|
||||
Self {
|
||||
heartbeat_interval_ms: opts.interval.as_millis() as u64,
|
||||
retry_interval_ms: opts.retry_interval.as_millis() as u64,
|
||||
gc_enabled: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -438,6 +439,7 @@ pub struct Context {
|
||||
pub leader_region_registry: LeaderRegionRegistryRef,
|
||||
pub topic_stats_registry: TopicStatsRegistryRef,
|
||||
pub heartbeat_interval: Duration,
|
||||
pub gc_enabled: bool,
|
||||
pub is_handshake: bool,
|
||||
}
|
||||
|
||||
@@ -932,6 +934,7 @@ impl Metasrv {
|
||||
leader_region_registry,
|
||||
topic_stats_registry,
|
||||
heartbeat_interval: self.options().heartbeat_interval,
|
||||
gc_enabled: self.options().gc.enable,
|
||||
is_handshake: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,7 +28,9 @@ use common_meta::ddl::table_meta::{TableMetadataAllocator, TableMetadataAllocato
|
||||
use common_meta::ddl::{
|
||||
DdlContext, NoopRegionFailureDetectorControl, RegionFailureDetectorControllerRef,
|
||||
};
|
||||
use common_meta::ddl_manager::{DdlManager, DdlManagerConfiguratorRef};
|
||||
use common_meta::ddl_manager::{
|
||||
DdlManager, DdlManagerConfiguratorRef, RepartitionProcedureFactoryRef,
|
||||
};
|
||||
use common_meta::distributed_time_constants::default_distributed_time_constants;
|
||||
use common_meta::key::TableMetadataManager;
|
||||
use common_meta::key::flow::FlowMetadataManager;
|
||||
@@ -70,7 +72,9 @@ use crate::metasrv::{
|
||||
use crate::peer::MetasrvPeerAllocator;
|
||||
use crate::procedure::region_migration::DefaultContextFactory;
|
||||
use crate::procedure::region_migration::manager::RegionMigrationManager;
|
||||
use crate::procedure::repartition::DefaultRepartitionProcedureFactory;
|
||||
use crate::procedure::repartition::{
|
||||
DefaultRepartitionProcedureFactory, GcDisabledRepartitionProcedureFactory,
|
||||
};
|
||||
use crate::procedure::wal_prune::Context as WalPruneContext;
|
||||
use crate::procedure::wal_prune::manager::{WalPruneManager, WalPruneTicker};
|
||||
use crate::region::flush_trigger::RegionFlushTrigger;
|
||||
@@ -428,10 +432,17 @@ impl MetasrvBuilder {
|
||||
soft_drop_enabled: ddl_soft_drop_enabled(&options),
|
||||
};
|
||||
let procedure_manager_c = procedure_manager.clone();
|
||||
let repartition_procedure_factory = Arc::new(DefaultRepartitionProcedureFactory::new(
|
||||
mailbox.clone(),
|
||||
options.grpc.server_addr.clone(),
|
||||
));
|
||||
let repartition_procedure_factory: RepartitionProcedureFactoryRef = if options.gc.enable {
|
||||
Arc::new(DefaultRepartitionProcedureFactory::new(
|
||||
mailbox.clone(),
|
||||
options.grpc.server_addr.clone(),
|
||||
))
|
||||
} else {
|
||||
Arc::new(GcDisabledRepartitionProcedureFactory::new(
|
||||
mailbox.clone(),
|
||||
options.grpc.server_addr.clone(),
|
||||
))
|
||||
};
|
||||
let ddl_manager = DdlManager::try_new(
|
||||
ddl_context,
|
||||
procedure_manager_c,
|
||||
|
||||
@@ -808,6 +808,50 @@ impl DefaultRepartitionProcedureFactory {
|
||||
}
|
||||
}
|
||||
|
||||
/// Rejects new repartition requests when metasrv GC is disabled.
|
||||
///
|
||||
/// Procedure loaders are still delegated to the enabled factory so procedures
|
||||
/// persisted before a metasrv restart can be recovered.
|
||||
pub struct GcDisabledRepartitionProcedureFactory {
|
||||
enabled_factory: DefaultRepartitionProcedureFactory,
|
||||
}
|
||||
|
||||
impl GcDisabledRepartitionProcedureFactory {
|
||||
pub fn new(mailbox: MailboxRef, server_addr: String) -> Self {
|
||||
Self {
|
||||
enabled_factory: DefaultRepartitionProcedureFactory::new(mailbox, server_addr),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl RepartitionProcedureFactory for GcDisabledRepartitionProcedureFactory {
|
||||
fn create(
|
||||
&self,
|
||||
_ddl_ctx: &DdlContext,
|
||||
_table_name: TableName,
|
||||
_table_id: TableId,
|
||||
_source: RepartitionSource,
|
||||
_to_exprs: Vec<String>,
|
||||
_timeout: Option<Duration>,
|
||||
) -> std::result::Result<BoxedProcedure, BoxedError> {
|
||||
Err(BoxedError::new(
|
||||
error::InvalidArgumentsSnafu {
|
||||
err_msg: "Repartition requires metasrv GC to be enabled".to_string(),
|
||||
}
|
||||
.build(),
|
||||
))
|
||||
}
|
||||
|
||||
fn register_loaders(
|
||||
&self,
|
||||
ddl_ctx: &DdlContext,
|
||||
procedure_manager: &ProcedureManagerRef,
|
||||
) -> std::result::Result<(), BoxedError> {
|
||||
self.enabled_factory
|
||||
.register_loaders(ddl_ctx, procedure_manager)
|
||||
}
|
||||
}
|
||||
|
||||
impl RepartitionProcedureFactory for DefaultRepartitionProcedureFactory {
|
||||
fn create(
|
||||
&self,
|
||||
@@ -919,7 +963,7 @@ mod tests {
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use common_error::ext::BoxedError;
|
||||
use common_error::ext::{BoxedError, ErrorExt};
|
||||
use common_error::mock::MockError;
|
||||
use common_error::status_code::StatusCode;
|
||||
use common_meta::ddl::test_util::datanode_handler::{
|
||||
@@ -1050,6 +1094,37 @@ mod tests {
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_gc_disabled_factory_rejects_repartition() {
|
||||
let env = TestingEnv::new();
|
||||
let node_manager = Arc::new(MockDatanodeManager::new(UnexpectedErrorDatanodeHandler));
|
||||
let ddl_ctx = env.ddl_context(node_manager);
|
||||
let factory = GcDisabledRepartitionProcedureFactory::new(
|
||||
env.mailbox_ctx.mailbox().clone(),
|
||||
env.server_addr.clone(),
|
||||
);
|
||||
|
||||
let err = factory
|
||||
.create(
|
||||
&ddl_ctx,
|
||||
TableName::new("test_catalog", "test_schema", "test_table"),
|
||||
1024,
|
||||
RepartitionSource::Unpartitioned {
|
||||
partition_columns: vec![],
|
||||
},
|
||||
vec![],
|
||||
None,
|
||||
)
|
||||
.err()
|
||||
.expect("GC-disabled factory must reject repartition");
|
||||
|
||||
assert_eq!(StatusCode::InvalidArguments, err.status_code());
|
||||
assert_eq!(
|
||||
"Invalid arguments: Repartition requires metasrv GC to be enabled",
|
||||
err.to_string()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filter_allocated_region_routes() {
|
||||
let table_id = 1024;
|
||||
|
||||
@@ -23,9 +23,6 @@ pub(crate) const DEFAULT_FLUSH_METADATA_REGION_INTERVAL: Duration = Duration::fr
|
||||
/// Configuration for the metric engine.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct EngineConfig {
|
||||
/// Whether to use sparse primary key encoding.
|
||||
#[serde(default = "EngineConfig::default_sparse_primary_key_encoding")]
|
||||
pub sparse_primary_key_encoding: bool,
|
||||
/// The flush interval of the metadata region.
|
||||
#[serde(
|
||||
with = "humantime_serde",
|
||||
@@ -38,7 +35,6 @@ impl Default for EngineConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
flush_metadata_region_interval: DEFAULT_FLUSH_METADATA_REGION_INTERVAL,
|
||||
sparse_primary_key_encoding: Self::default_sparse_primary_key_encoding(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -48,10 +44,6 @@ impl EngineConfig {
|
||||
DEFAULT_FLUSH_METADATA_REGION_INTERVAL
|
||||
}
|
||||
|
||||
fn default_sparse_primary_key_encoding() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// Sanitizes the configuration.
|
||||
pub fn sanitize(&mut self) {
|
||||
if self.flush_metadata_region_interval.is_zero() {
|
||||
|
||||
@@ -237,6 +237,9 @@ impl RegionEngine for MetricEngine {
|
||||
}
|
||||
RegionRequest::Drop(drop) => self.inner.drop_region(region_id, drop).await,
|
||||
RegionRequest::Open(open) => self.inner.open_region(region_id, open).await,
|
||||
RegionRequest::CleanUp(clean_up) => {
|
||||
self.inner.clean_up_region(region_id, clean_up).await
|
||||
}
|
||||
RegionRequest::Close(close) => self.inner.close_region(region_id, close).await,
|
||||
RegionRequest::Alter(alter) => {
|
||||
self.inner
|
||||
@@ -476,7 +479,6 @@ impl MetricEngine {
|
||||
metadata_region,
|
||||
data_region,
|
||||
state: state.clone(),
|
||||
config,
|
||||
row_modifier: RowModifier::default(),
|
||||
flush_task: RepeatedTask::new(
|
||||
flush_interval,
|
||||
@@ -556,11 +558,6 @@ impl MetricEngine {
|
||||
) -> Result<common_recordbatch::SendableRecordBatchStream, BoxedError> {
|
||||
self.inner.scan_to_stream(region_id, request).await
|
||||
}
|
||||
|
||||
/// Returns the configuration of the engine.
|
||||
pub fn config(&self) -> &EngineConfig {
|
||||
&self.inner.config
|
||||
}
|
||||
}
|
||||
|
||||
struct MetricEngineInner {
|
||||
@@ -568,7 +565,6 @@ struct MetricEngineInner {
|
||||
metadata_region: MetadataRegion,
|
||||
data_region: DataRegion,
|
||||
state: Arc<RwLock<MetricEngineState>>,
|
||||
config: EngineConfig,
|
||||
row_modifier: RowModifier,
|
||||
flush_task: RepeatedTask<Error>,
|
||||
}
|
||||
@@ -585,8 +581,8 @@ mod test {
|
||||
use store_api::metric_engine_consts::PHYSICAL_TABLE_METADATA_KEY;
|
||||
use store_api::mito_engine_options::WAL_OPTIONS_KEY;
|
||||
use store_api::region_request::{
|
||||
PathType, RegionCloseRequest, RegionDropRequest, RegionFlushRequest, RegionOpenRequest,
|
||||
RegionRequest,
|
||||
PathType, RegionCleanUpRequest, RegionCloseRequest, RegionDropRequest, RegionFlushRequest,
|
||||
RegionOpenRequest, RegionRequest,
|
||||
};
|
||||
|
||||
use super::*;
|
||||
@@ -604,7 +600,7 @@ mod test {
|
||||
engine
|
||||
.handle_request(
|
||||
physical_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {}),
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -632,7 +628,7 @@ mod test {
|
||||
engine
|
||||
.handle_request(
|
||||
nonexistent_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {}),
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -656,6 +652,58 @@ mod test {
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_offline_cleanup_physical_region() {
|
||||
let env = TestEnv::new().await;
|
||||
env.init_metric_region().await;
|
||||
let engine = env.metric();
|
||||
let mito = env.mito();
|
||||
let physical_region_id = env.default_physical_region_id();
|
||||
let metadata_region_id = crate::utils::to_metadata_region_id(physical_region_id);
|
||||
let data_region_id = crate::utils::to_data_region_id(physical_region_id);
|
||||
|
||||
engine
|
||||
.handle_request(
|
||||
physical_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let object_store = env.get_object_store().unwrap();
|
||||
let metadata_region_dir = region_dir_from_table_dir(
|
||||
&TestEnv::default_table_dir(),
|
||||
metadata_region_id,
|
||||
PathType::Metadata,
|
||||
);
|
||||
let data_region_dir = region_dir_from_table_dir(
|
||||
&TestEnv::default_table_dir(),
|
||||
data_region_id,
|
||||
PathType::Data,
|
||||
);
|
||||
assert!(object_store.exists(&metadata_region_dir).await.unwrap());
|
||||
assert!(object_store.exists(&data_region_dir).await.unwrap());
|
||||
|
||||
let physical_region_option = [(PHYSICAL_TABLE_METADATA_KEY.to_string(), String::new())]
|
||||
.into_iter()
|
||||
.collect();
|
||||
let clean_up_request = RegionCleanUpRequest {
|
||||
engine: METRIC_ENGINE_NAME.to_string(),
|
||||
table_dir: TestEnv::default_table_dir(),
|
||||
path_type: PathType::Bare,
|
||||
options: physical_region_option,
|
||||
};
|
||||
engine
|
||||
.handle_request(physical_region_id, RegionRequest::CleanUp(clean_up_request))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(!mito.is_region_exists(metadata_region_id));
|
||||
assert!(!mito.is_region_exists(data_region_id));
|
||||
assert!(!object_store.exists(&metadata_region_dir).await.unwrap());
|
||||
assert!(!object_store.exists(&data_region_dir).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_role() {
|
||||
let env = TestEnv::new().await;
|
||||
@@ -736,7 +784,7 @@ mod test {
|
||||
metric_engine
|
||||
.handle_request(
|
||||
physical_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {}),
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -828,7 +876,10 @@ mod test {
|
||||
// Closes all regions
|
||||
for region_id in logical_region_ids.iter().chain(physical_region_ids.iter()) {
|
||||
metric_engine
|
||||
.handle_request(*region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
*region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
@@ -29,7 +29,7 @@ impl MetricEngineInner {
|
||||
pub async fn close_region(
|
||||
&self,
|
||||
region_id: RegionId,
|
||||
_req: RegionCloseRequest,
|
||||
req: RegionCloseRequest,
|
||||
) -> Result<AffectedRows> {
|
||||
let data_region_id = utils::to_data_region_id(region_id);
|
||||
if self
|
||||
@@ -38,7 +38,8 @@ impl MetricEngineInner {
|
||||
.unwrap()
|
||||
.exist_physical_region(data_region_id)
|
||||
{
|
||||
self.close_physical_region(data_region_id).await?;
|
||||
self.close_physical_region(data_region_id, req.flush_on_close)
|
||||
.await?;
|
||||
self.state
|
||||
.write()
|
||||
.unwrap()
|
||||
@@ -59,18 +60,25 @@ impl MetricEngineInner {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn close_physical_region(&self, region_id: RegionId) -> Result<AffectedRows> {
|
||||
pub(crate) async fn close_physical_region(
|
||||
&self,
|
||||
region_id: RegionId,
|
||||
flush_on_close: bool,
|
||||
) -> Result<AffectedRows> {
|
||||
let data_region_id = utils::to_data_region_id(region_id);
|
||||
let metadata_region_id = utils::to_metadata_region_id(region_id);
|
||||
|
||||
self.mito
|
||||
.handle_request(data_region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
data_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest { flush_on_close }),
|
||||
)
|
||||
.await
|
||||
.with_context(|_| CloseMitoRegionSnafu { region_id })?;
|
||||
self.mito
|
||||
.handle_request(
|
||||
metadata_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {}),
|
||||
RegionRequest::Close(RegionCloseRequest { flush_on_close }),
|
||||
)
|
||||
.await
|
||||
.with_context(|_| CloseMitoRegionSnafu { region_id })?;
|
||||
|
||||
@@ -554,10 +554,7 @@ impl MetricEngineInner {
|
||||
data_region_request.primary_key = primary_key;
|
||||
|
||||
// set data region options
|
||||
set_data_region_options(
|
||||
&mut data_region_request.options,
|
||||
self.config.sparse_primary_key_encoding,
|
||||
);
|
||||
set_data_region_options(&mut data_region_request.options);
|
||||
|
||||
data_region_request
|
||||
}
|
||||
|
||||
@@ -22,14 +22,17 @@ use datafusion::common::HashMap;
|
||||
use mito2::engine::MITO_ENGINE_NAME;
|
||||
use snafu::{OptionExt, ResultExt};
|
||||
use store_api::region_engine::{BatchResponses, RegionEngine};
|
||||
use store_api::region_request::{AffectedRows, PathType, RegionOpenRequest, ReplayCheckpoint};
|
||||
use store_api::region_request::{
|
||||
AffectedRows, PathType, RegionCleanUpRequest, RegionOpenRequest, RegionRequest,
|
||||
ReplayCheckpoint,
|
||||
};
|
||||
use store_api::storage::RegionId;
|
||||
|
||||
use crate::engine::MetricEngineInner;
|
||||
use crate::engine::create::region_options_for_metadata_region;
|
||||
use crate::engine::options::{PhysicalRegionOptions, set_data_region_options};
|
||||
use crate::error::{
|
||||
BatchOpenMitoRegionSnafu, NoOpenRegionResultSnafu, OpenMitoRegionSnafu,
|
||||
BatchOpenMitoRegionSnafu, CleanUpMitoRegionSnafu, NoOpenRegionResultSnafu, OpenMitoRegionSnafu,
|
||||
PhysicalRegionNotFoundSnafu, Result,
|
||||
};
|
||||
use crate::metrics::{LOGICAL_REGION_COUNT, PHYSICAL_REGION_COUNT};
|
||||
@@ -105,7 +108,7 @@ impl MetricEngineInner {
|
||||
utils::to_metadata_region_id(physical_region_id),
|
||||
utils::to_data_region_id(physical_region_id)
|
||||
);
|
||||
if let Err(err) = self.close_physical_region(physical_region_id).await {
|
||||
if let Err(err) = self.close_physical_region(physical_region_id, false).await {
|
||||
error!(err; "Failed to close physical region {}", physical_region_id);
|
||||
}
|
||||
}
|
||||
@@ -200,6 +203,91 @@ impl MetricEngineInner {
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn clean_up_region(
|
||||
&self,
|
||||
region_id: RegionId,
|
||||
request: RegionCleanUpRequest,
|
||||
) -> Result<AffectedRows> {
|
||||
if request.is_physical_table() {
|
||||
self.cleanup_physical_region_offline(region_id, request)
|
||||
.await
|
||||
} else {
|
||||
Ok(0)
|
||||
}
|
||||
}
|
||||
|
||||
async fn cleanup_physical_region_offline(
|
||||
&self,
|
||||
region_id: RegionId,
|
||||
request: RegionCleanUpRequest,
|
||||
) -> Result<AffectedRows> {
|
||||
let metadata_region_id = utils::to_metadata_region_id(region_id);
|
||||
let data_region_id = utils::to_data_region_id(region_id);
|
||||
let (clean_up_metadata_region_request, clean_up_data_region_request) =
|
||||
self.transform_clean_up_physical_region_request(request);
|
||||
let _ = self
|
||||
.mito
|
||||
.handle_request(
|
||||
metadata_region_id,
|
||||
RegionRequest::CleanUp(clean_up_metadata_region_request),
|
||||
)
|
||||
.await
|
||||
.context(CleanUpMitoRegionSnafu {
|
||||
region_type: "metadata",
|
||||
})?;
|
||||
let data_region_response = self
|
||||
.mito
|
||||
.handle_request(
|
||||
data_region_id,
|
||||
RegionRequest::CleanUp(clean_up_data_region_request),
|
||||
)
|
||||
.await
|
||||
.context(CleanUpMitoRegionSnafu {
|
||||
region_type: "data",
|
||||
})?;
|
||||
|
||||
if self.state.read().unwrap().exist_physical_region(region_id) {
|
||||
self.state
|
||||
.write()
|
||||
.unwrap()
|
||||
.remove_physical_region(region_id)?;
|
||||
PHYSICAL_REGION_COUNT.dec();
|
||||
}
|
||||
|
||||
Ok(data_region_response.affected_rows)
|
||||
}
|
||||
|
||||
/// Transform the cleanup request to metadata region and data region cleanup requests.
|
||||
///
|
||||
/// Returns:
|
||||
/// - The cleanup request for metadata region.
|
||||
/// - The cleanup request for data region.
|
||||
fn transform_clean_up_physical_region_request(
|
||||
&self,
|
||||
request: RegionCleanUpRequest,
|
||||
) -> (RegionCleanUpRequest, RegionCleanUpRequest) {
|
||||
let clean_up_metadata_region_request = RegionCleanUpRequest {
|
||||
table_dir: request.table_dir.clone(),
|
||||
path_type: PathType::Metadata,
|
||||
options: region_options_for_metadata_region(&request.options),
|
||||
engine: MITO_ENGINE_NAME.to_string(),
|
||||
};
|
||||
|
||||
let mut data_region_options = request.options;
|
||||
set_data_region_options(&mut data_region_options);
|
||||
let clean_up_data_region_request = RegionCleanUpRequest {
|
||||
table_dir: request.table_dir,
|
||||
path_type: PathType::Data,
|
||||
options: data_region_options,
|
||||
engine: MITO_ENGINE_NAME.to_string(),
|
||||
};
|
||||
|
||||
(
|
||||
clean_up_metadata_region_request,
|
||||
clean_up_data_region_request,
|
||||
)
|
||||
}
|
||||
|
||||
/// Transform the open request to open metadata region and data region.
|
||||
///
|
||||
/// Returns:
|
||||
@@ -226,10 +314,7 @@ impl MetricEngineInner {
|
||||
};
|
||||
|
||||
let mut data_region_options = request.options;
|
||||
set_data_region_options(
|
||||
&mut data_region_options,
|
||||
self.config.sparse_primary_key_encoding,
|
||||
);
|
||||
set_data_region_options(&mut data_region_options);
|
||||
let open_data_region_request = RegionOpenRequest {
|
||||
table_dir: request.table_dir.clone(),
|
||||
path_type: PathType::Data,
|
||||
|
||||
@@ -60,10 +60,7 @@ pub enum IndexOptions {
|
||||
}
|
||||
|
||||
/// Sets data region specific options.
|
||||
pub fn set_data_region_options(
|
||||
options: &mut HashMap<String, String>,
|
||||
sparse_primary_key_encoding_if_absent: bool,
|
||||
) {
|
||||
pub fn set_data_region_options(options: &mut HashMap<String, String>) {
|
||||
options.remove(METRIC_ENGINE_INDEX_TYPE_OPTION);
|
||||
options.remove(METRIC_ENGINE_INDEX_SKIPPING_INDEX_GRANULARITY_OPTION);
|
||||
options.remove(METRIC_ENGINE_INDEX_SKIPPING_INDEX_FALSE_POSITIVE_RATE_OPTION);
|
||||
@@ -83,11 +80,11 @@ pub fn set_data_region_options(
|
||||
options.insert("sst_format".to_string(), "flat".to_string());
|
||||
|
||||
// Decide the top-level primary key encoding: caller-supplied top-level key wins,
|
||||
// then extracted legacy value, then the `sparse` default if requested.
|
||||
// then extracted legacy value, then the `sparse` default.
|
||||
if !options.contains_key(PRIMARY_KEY_ENCODING) {
|
||||
if let Some(encoding) = legacy_encoding {
|
||||
options.insert(PRIMARY_KEY_ENCODING.to_string(), encoding);
|
||||
} else if sparse_primary_key_encoding_if_absent {
|
||||
} else {
|
||||
options.insert(PRIMARY_KEY_ENCODING.to_string(), "sparse".to_string());
|
||||
}
|
||||
}
|
||||
@@ -175,7 +172,7 @@ mod tests {
|
||||
METRIC_ENGINE_INDEX_SKIPPING_INDEX_FALSE_POSITIVE_RATE_OPTION.to_string(),
|
||||
"0.01".to_string(),
|
||||
);
|
||||
set_data_region_options(&mut options, false);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
for key in [
|
||||
METRIC_ENGINE_INDEX_TYPE_OPTION,
|
||||
@@ -227,7 +224,7 @@ mod tests {
|
||||
fn test_set_data_region_options_default_compaction_time_window() {
|
||||
// Test that default time window is set when not specified
|
||||
let mut options = HashMap::new();
|
||||
set_data_region_options(&mut options, false);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
assert_eq!(options.get("memtable.type"), Some(&"bulk".to_string()));
|
||||
assert_eq!(options.get("sst_format"), Some(&"flat".to_string()));
|
||||
@@ -241,7 +238,7 @@ mod tests {
|
||||
#[test]
|
||||
fn test_set_data_region_options_sparse_primary_key_encoding() {
|
||||
let mut options = HashMap::new();
|
||||
set_data_region_options(&mut options, true);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
assert_eq!(options.get("memtable.type"), Some(&"bulk".to_string()));
|
||||
assert_eq!(options.get("sst_format"), Some(&"flat".to_string()));
|
||||
@@ -264,7 +261,7 @@ mod tests {
|
||||
"memtable.partition_tree.index_max_keys_per_shard".to_string(),
|
||||
"2048".to_string(),
|
||||
);
|
||||
set_data_region_options(&mut options, false);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
assert_eq!(options.get("memtable.type"), Some(&"bulk".to_string()));
|
||||
assert_eq!(options.get("sst_format"), Some(&"flat".to_string()));
|
||||
@@ -282,7 +279,7 @@ mod tests {
|
||||
let mut options = HashMap::new();
|
||||
options.insert(PRIMARY_KEY_ENCODING.to_string(), "dense".to_string());
|
||||
// Sparse flag is on but caller already specified dense.
|
||||
set_data_region_options(&mut options, true);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
assert_eq!(
|
||||
options.get(PRIMARY_KEY_ENCODING),
|
||||
@@ -296,7 +293,7 @@ mod tests {
|
||||
let mut options = HashMap::new();
|
||||
options.insert(TWCS_TIME_WINDOW.to_string(), "2h".to_string());
|
||||
options.insert(COMPACTION_TYPE.to_string(), "twcs".to_string());
|
||||
set_data_region_options(&mut options, false);
|
||||
set_data_region_options(&mut options);
|
||||
|
||||
// User's time window should be preserved
|
||||
assert_eq!(options.get(TWCS_TIME_WINDOW), Some(&"2h".to_string()));
|
||||
|
||||
@@ -304,7 +304,7 @@ mod tests {
|
||||
metric_engine
|
||||
.handle_request(
|
||||
target_physical_region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {}),
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
@@ -43,6 +43,14 @@ pub enum Error {
|
||||
location: Location,
|
||||
},
|
||||
|
||||
#[snafu(display("Failed to clean up mito region, region type: {}", region_type))]
|
||||
CleanUpMitoRegion {
|
||||
region_type: String,
|
||||
source: BoxedError,
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
},
|
||||
|
||||
#[snafu(display("Failed to batch open mito region"))]
|
||||
BatchOpenMitoRegion {
|
||||
source: BoxedError,
|
||||
@@ -438,6 +446,7 @@ impl ErrorExt for Error {
|
||||
|
||||
CreateMitoRegion { source, .. }
|
||||
| OpenMitoRegion { source, .. }
|
||||
| CleanUpMitoRegion { source, .. }
|
||||
| CloseMitoRegion { source, .. }
|
||||
| MitoReadOperation { source, .. }
|
||||
| MitoWriteOperation { source, .. }
|
||||
@@ -471,6 +480,7 @@ impl ErrorExt for Error {
|
||||
match self {
|
||||
CreateMitoRegion { source, .. }
|
||||
| OpenMitoRegion { source, .. }
|
||||
| CleanUpMitoRegion { source, .. }
|
||||
| BatchOpenMitoRegion { source, .. }
|
||||
| BatchCatchupMitoRegion { source, .. }
|
||||
| CloseMitoRegion { source, .. }
|
||||
|
||||
@@ -97,7 +97,6 @@ mod tests {
|
||||
"test_flush_metadata_region_task",
|
||||
EngineConfig {
|
||||
flush_metadata_region_interval: Duration::from_millis(10),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
@@ -124,7 +123,6 @@ mod tests {
|
||||
"test_flush_metadata_region_task_with_long_interval",
|
||||
EngineConfig {
|
||||
flush_metadata_region_interval: Duration::from_secs(60),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
@@ -145,18 +143,12 @@ mod tests {
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_flush_metadata_region_sanitize() {
|
||||
let env = TestEnv::with_prefix_and_config(
|
||||
"test_flush_metadata_region_sanitize",
|
||||
EngineConfig {
|
||||
flush_metadata_region_interval: Duration::from_secs(0),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let metric = env.metric();
|
||||
let config = metric.config();
|
||||
#[test]
|
||||
fn test_flush_metadata_region_sanitize() {
|
||||
let mut config = EngineConfig {
|
||||
flush_metadata_region_interval: Duration::from_secs(0),
|
||||
};
|
||||
config.sanitize();
|
||||
assert_eq!(
|
||||
config.flush_metadata_region_interval,
|
||||
DEFAULT_FLUSH_METADATA_REGION_INTERVAL
|
||||
|
||||
@@ -64,7 +64,9 @@ use crate::error::{
|
||||
RegionClosedSnafu, RegionDroppedSnafu, RegionTruncatedSnafu, RemoteCompactionSnafu, Result,
|
||||
TimeRangePredicateOverflowSnafu, TimeoutSnafu,
|
||||
};
|
||||
use crate::metrics::{COMPACTION_STAGE_ELAPSED, INFLIGHT_COMPACTION_COUNT};
|
||||
use crate::metrics::{
|
||||
COMPACTION_MEMORY_REJECTED, COMPACTION_STAGE_ELAPSED, INFLIGHT_COMPACTION_COUNT,
|
||||
};
|
||||
use crate::read::FlatSource;
|
||||
use crate::read::flat_projection::FlatProjectionMapper;
|
||||
use crate::read::read_columns::ReadColumns;
|
||||
@@ -613,8 +615,21 @@ impl CompactionScheduler {
|
||||
waiters
|
||||
};
|
||||
|
||||
// Create a local compaction task.
|
||||
// Check whether this local compaction can ever fit before submitting it.
|
||||
let estimated_bytes = estimate_compaction_bytes(&picker_output);
|
||||
if let Some(limit_bytes) = self.exceeds_compaction_memory_limit(estimated_bytes) {
|
||||
COMPACTION_MEMORY_REJECTED
|
||||
.with_label_values(&["oversized"])
|
||||
.inc();
|
||||
warn!(
|
||||
"Skip compaction for region {} because estimated memory {} bytes exceeds compaction memory limit {} bytes",
|
||||
region_id, estimated_bytes, limit_bytes,
|
||||
);
|
||||
for waiter in waiters {
|
||||
waiter.send(Ok(0));
|
||||
}
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let cancel_handle = Arc::new(CancellationHandle::default());
|
||||
let state = LocalCompactionState::new(cancel_handle.clone());
|
||||
@@ -652,6 +667,15 @@ impl CompactionScheduler {
|
||||
)
|
||||
}
|
||||
|
||||
fn exceeds_compaction_memory_limit(&self, estimated_bytes: u64) -> Option<u64> {
|
||||
let limit_bytes = self.memory_manager.limit_bytes();
|
||||
if limit_bytes > 0 && estimated_bytes > limit_bytes {
|
||||
Some(limit_bytes)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn remove_region_on_failure(&mut self, region_id: RegionId, err: Arc<Error>) {
|
||||
// Remove this region.
|
||||
let Some(status) = self.region_status.remove(®ion_id) else {
|
||||
@@ -1551,6 +1575,65 @@ mod tests {
|
||||
assert!(scheduler.region_status.contains_key(®ion_id));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_schedule_compaction_skips_task_exceeding_memory_limit() {
|
||||
let job_scheduler = Arc::new(VecScheduler::default());
|
||||
let env = SchedulerEnv::new().await.scheduler(job_scheduler.clone());
|
||||
let (tx, _rx) = mpsc::channel(4);
|
||||
let mut scheduler = env.mock_compaction_scheduler(tx);
|
||||
scheduler.memory_manager = Arc::new(new_compaction_memory_manager(1024 * 1024));
|
||||
|
||||
let mut builder = VersionControlBuilder::new();
|
||||
let region_id = builder.region_id();
|
||||
let end = 1000 * 1000;
|
||||
let version_control = Arc::new(
|
||||
builder
|
||||
.push_l0_file_with_max_row_group_size(0, end, 1024 * 1024)
|
||||
.push_l0_file_with_max_row_group_size(10, end, 1024 * 1024)
|
||||
.push_l0_file_with_max_row_group_size(50, end, 1024 * 1024)
|
||||
.push_l0_file_with_max_row_group_size(80, end, 1024 * 1024)
|
||||
.push_l0_file_with_max_row_group_size(90, end, 1024 * 1024)
|
||||
.build(),
|
||||
);
|
||||
let manifest_ctx = env
|
||||
.mock_manifest_context(version_control.current().version.metadata.clone())
|
||||
.await;
|
||||
let (schema_metadata_manager, kv_backend) = mock_schema_metadata_manager();
|
||||
schema_metadata_manager
|
||||
.register_region_table_info(
|
||||
region_id.table_id(),
|
||||
"test_table",
|
||||
"test_catalog",
|
||||
"test_schema",
|
||||
None,
|
||||
kv_backend,
|
||||
)
|
||||
.await;
|
||||
let (output_tx, output_rx) = oneshot::channel();
|
||||
let rejected = COMPACTION_MEMORY_REJECTED.with_label_values(&["oversized"]);
|
||||
let rejected_before = rejected.get();
|
||||
|
||||
let scheduled = scheduler
|
||||
.schedule_compaction(
|
||||
region_id,
|
||||
Options::Regular(Default::default()),
|
||||
&version_control,
|
||||
&env.access_layer,
|
||||
OptionOutputTx::from(output_tx),
|
||||
&manifest_ctx,
|
||||
schema_metadata_manager,
|
||||
1,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(!scheduled);
|
||||
assert_eq!(output_rx.await.unwrap().unwrap(), 0);
|
||||
assert_eq!(rejected_before + 1, rejected.get());
|
||||
assert_eq!(0, job_scheduler.num_jobs());
|
||||
assert!(!scheduler.region_status.contains_key(®ion_id));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_schedule_on_finished() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
|
||||
@@ -255,6 +255,7 @@ mod tests {
|
||||
sst_format: None,
|
||||
max_row_group_row_count: None,
|
||||
primary_key_encoding: None,
|
||||
write_buffer_size: None,
|
||||
},
|
||||
compaction_time_window: None,
|
||||
}
|
||||
|
||||
+36
-4
@@ -38,7 +38,11 @@ pub(crate) const DEFAULT_MAX_CONCURRENT_SCAN_FILES: usize = 384;
|
||||
// Use `1/GLOBAL_WRITE_BUFFER_SIZE_FACTOR` of OS memory as global write buffer size in default mode
|
||||
const GLOBAL_WRITE_BUFFER_SIZE_FACTOR: u64 = 8;
|
||||
/// Use `1/SST_META_CACHE_SIZE_FACTOR` of OS memory size as SST meta cache size in default mode
|
||||
const SST_META_CACHE_SIZE_FACTOR: u64 = 32;
|
||||
const SST_META_CACHE_SIZE_FACTOR: u64 = 8;
|
||||
/// Use `1/PREFILTER_RESULT_CACHE_SIZE_FACTOR` of OS memory size as prefilter result cache size in default mode
|
||||
const PREFILTER_RESULT_CACHE_SIZE_FACTOR: u64 = 32;
|
||||
/// Use `1/INDEX_METADATA_CACHE_SIZE_FACTOR` of OS memory size as index metadata cache size in default mode
|
||||
const INDEX_METADATA_CACHE_SIZE_FACTOR: u64 = 32;
|
||||
/// Use `1/MEM_CACHE_SIZE_FACTOR` of OS memory size as mem cache size in default mode
|
||||
const MEM_CACHE_SIZE_FACTOR: u64 = 16;
|
||||
/// Use `1/PAGE_CACHE_SIZE_FACTOR` of OS memory size as page cache size in default mode
|
||||
@@ -105,6 +109,10 @@ pub struct MitoConfig {
|
||||
pub global_write_buffer_size: ReadableSize,
|
||||
/// Global write buffer size threshold to reject write requests.
|
||||
pub global_write_buffer_reject_size: ReadableSize,
|
||||
/// Default write buffer size for each region. Regions stall at this size and
|
||||
/// reject writes at twice this size. Setting it to 0 disables both limits
|
||||
/// unless the table specifies `write_buffer_size`.
|
||||
pub default_region_write_buffer_size: ReadableSize,
|
||||
|
||||
// Cache configs:
|
||||
/// Cache size for SST metadata. Setting it to 0 to disable the cache.
|
||||
@@ -202,6 +210,7 @@ impl Default for MitoConfig {
|
||||
auto_flush_interval: Duration::from_secs(30 * 60),
|
||||
global_write_buffer_size: ReadableSize::gb(1),
|
||||
global_write_buffer_reject_size: ReadableSize::gb(2),
|
||||
default_region_write_buffer_size: ReadableSize::mb(0),
|
||||
sst_meta_cache_size: ReadableSize::mb(128),
|
||||
vector_cache_size: ReadableSize::mb(512),
|
||||
page_cache_size: ReadableSize::mb(512),
|
||||
@@ -321,9 +330,14 @@ impl MitoConfig {
|
||||
);
|
||||
// Use 2x of global write buffer size as global write buffer reject size.
|
||||
let global_write_buffer_reject_size = global_write_buffer_size * 2;
|
||||
// shouldn't be greater than 128MB in default mode.
|
||||
// Page-index-bearing SST metadata can be much larger than footers alone.
|
||||
// Keep the auto-sized default bounded, but allow a larger warm working set.
|
||||
let sst_meta_cache_size = cmp::min(
|
||||
sys_memory / SST_META_CACHE_SIZE_FACTOR,
|
||||
ReadableSize::mb(512),
|
||||
);
|
||||
let prefilter_result_cache_size = cmp::min(
|
||||
sys_memory / PREFILTER_RESULT_CACHE_SIZE_FACTOR,
|
||||
ReadableSize::mb(128),
|
||||
);
|
||||
// shouldn't be greater than 512MB in default mode.
|
||||
@@ -338,7 +352,7 @@ impl MitoConfig {
|
||||
self.selector_result_cache_size = mem_cache_size;
|
||||
self.range_result_cache_size = mem_cache_size;
|
||||
// Use a smaller cache size because prefilter result usually should be small.
|
||||
self.prefilter_result_cache_size = sst_meta_cache_size;
|
||||
self.prefilter_result_cache_size = prefilter_result_cache_size;
|
||||
|
||||
self.index.adjust_buffer_and_cache_size(sys_memory);
|
||||
}
|
||||
@@ -359,6 +373,24 @@ impl MitoConfig {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_adjust_sst_metadata_and_prefilter_cache_caps_independently() {
|
||||
let mut config = MitoConfig::default();
|
||||
|
||||
config.adjust_buffer_and_cache_size(ReadableSize::gb(1));
|
||||
assert_eq!(ReadableSize::mb(128), config.sst_meta_cache_size);
|
||||
assert_eq!(ReadableSize::mb(32), config.prefilter_result_cache_size);
|
||||
|
||||
config.adjust_buffer_and_cache_size(ReadableSize::gb(64));
|
||||
assert_eq!(ReadableSize::mb(512), config.sst_meta_cache_size);
|
||||
assert_eq!(ReadableSize::mb(128), config.prefilter_result_cache_size);
|
||||
}
|
||||
}
|
||||
|
||||
/// Index build mode.
|
||||
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
@@ -466,7 +498,7 @@ impl IndexConfig {
|
||||
self.content_cache_size = cmp::min(self.content_cache_size, cache_size);
|
||||
|
||||
let metadata_cache_size = cmp::min(
|
||||
sys_memory / SST_META_CACHE_SIZE_FACTOR,
|
||||
sys_memory / INDEX_METADATA_CACHE_SIZE_FACTOR,
|
||||
ReadableSize::mb(64),
|
||||
);
|
||||
self.metadata_cache_size = cmp::min(self.metadata_cache_size, metadata_cache_size);
|
||||
|
||||
@@ -498,7 +498,10 @@ async fn test_catchup_with_manifest_update(factory: Option<LogStoreFactory>) {
|
||||
|
||||
async fn close_region(engine: &MitoEngine, region_id: RegionId) {
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
@@ -15,15 +15,20 @@
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::Ordering;
|
||||
|
||||
use api::v1::Rows;
|
||||
use common_base::Plugins;
|
||||
use common_recordbatch::RecordBatches;
|
||||
use store_api::region_engine::RegionEngine;
|
||||
use store_api::region_request::{RegionCloseRequest, RegionRequest};
|
||||
use store_api::storage::RegionId;
|
||||
use store_api::storage::{RegionId, ScanRequest};
|
||||
|
||||
use crate::config::MitoConfig;
|
||||
use crate::engine::flush_test::MockRegionHook;
|
||||
use crate::engine::listener::AlterFlushListener;
|
||||
use crate::engine::region_hook::RegionHookRef;
|
||||
use crate::test_util::{CreateRequestBuilder, TestEnv};
|
||||
use crate::test_util::{
|
||||
CreateRequestBuilder, TestEnv, build_rows, flush_region, put_rows, rows_schema,
|
||||
};
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_engine_close_region() {
|
||||
@@ -43,7 +48,10 @@ async fn test_engine_close_region_with_format(flat_format: bool) {
|
||||
let region_id = RegionId::new(1, 1);
|
||||
// It's okay to close a region doesn't exist.
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
@@ -55,14 +63,20 @@ async fn test_engine_close_region_with_format(flat_format: bool) {
|
||||
|
||||
// Close the created region.
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!engine.is_region_exists(region_id));
|
||||
|
||||
// It's okay to close this region again.
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
@@ -93,7 +107,10 @@ async fn test_region_hook_on_close() {
|
||||
assert_eq!(hook.dropped_count.load(Ordering::Relaxed), 0);
|
||||
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
@@ -103,3 +120,160 @@ async fn test_region_hook_on_close() {
|
||||
assert_eq!(hook.dropped_count.load(Ordering::Relaxed), 0);
|
||||
assert_eq!(hook.files_removed_count.load(Ordering::Relaxed), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_engine_close_region_flush_on_close() {
|
||||
let mut env = TestEnv::with_prefix("close-flush-on-close").await;
|
||||
let engine = env.create_engine(MitoConfig::default()).await;
|
||||
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let request = CreateRequestBuilder::new().build();
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Create(request.clone()))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
put_rows(
|
||||
&engine,
|
||||
region_id,
|
||||
Rows {
|
||||
schema: rows_schema(&request),
|
||||
rows: build_rows(0, 3),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
!engine
|
||||
.get_region(region_id)
|
||||
.unwrap()
|
||||
.version()
|
||||
.memtables
|
||||
.is_empty()
|
||||
);
|
||||
|
||||
engine
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {
|
||||
flush_on_close: true,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!engine.is_region_exists(region_id));
|
||||
|
||||
engine
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Open(store_api::region_request::RegionOpenRequest {
|
||||
engine: String::new(),
|
||||
table_dir: request.table_dir.clone(),
|
||||
path_type: store_api::region_request::PathType::Bare,
|
||||
options: request.options.clone(),
|
||||
skip_wal_replay: true,
|
||||
checkpoint: None,
|
||||
requirements: Default::default(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let stream = engine
|
||||
.scan_to_stream(region_id, ScanRequest::default())
|
||||
.await
|
||||
.unwrap();
|
||||
let batches = RecordBatches::try_collect(stream).await.unwrap();
|
||||
assert_eq!(3, batches.iter().map(|b| b.num_rows()).sum::<usize>());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_engine_close_region_flush_on_close_while_flushing() {
|
||||
let mut env = TestEnv::with_prefix("close-flush-on-close-while-flushing").await;
|
||||
let listener = Arc::new(AlterFlushListener::default());
|
||||
let engine = env
|
||||
.create_engine_with(MitoConfig::default(), None, Some(listener.clone()), None)
|
||||
.await;
|
||||
|
||||
let region_id = RegionId::new(1, 1);
|
||||
let request = CreateRequestBuilder::new().build();
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Create(request.clone()))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
put_rows(
|
||||
&engine,
|
||||
region_id,
|
||||
Rows {
|
||||
schema: rows_schema(&request),
|
||||
rows: build_rows(0, 3),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
let flush_engine = engine.clone();
|
||||
let flush_task = tokio::spawn(async move {
|
||||
flush_region(&flush_engine, region_id, None).await;
|
||||
});
|
||||
listener.wait_flush_begin().await;
|
||||
|
||||
put_rows(
|
||||
&engine,
|
||||
region_id,
|
||||
Rows {
|
||||
schema: rows_schema(&request),
|
||||
rows: build_rows(3, 6),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
let request_count = listener.request_count();
|
||||
let close_engine = engine.clone();
|
||||
let close_task = tokio::spawn(async move {
|
||||
close_engine
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest {
|
||||
flush_on_close: true,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
listener.wait_request_count(request_count + 1).await;
|
||||
|
||||
let second_flush_listener = listener.clone();
|
||||
let second_flush_task = tokio::spawn(async move {
|
||||
second_flush_listener.wait_flush_begin().await;
|
||||
second_flush_listener.wake_flush();
|
||||
});
|
||||
listener.wake_flush();
|
||||
|
||||
flush_task.await.unwrap();
|
||||
close_task.await.unwrap();
|
||||
second_flush_task.abort();
|
||||
assert!(!engine.is_region_exists(region_id));
|
||||
|
||||
engine
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Open(store_api::region_request::RegionOpenRequest {
|
||||
engine: String::new(),
|
||||
table_dir: request.table_dir,
|
||||
path_type: store_api::region_request::PathType::Bare,
|
||||
options: request.options,
|
||||
skip_wal_replay: true,
|
||||
checkpoint: None,
|
||||
requirements: Default::default(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let stream = engine
|
||||
.scan_to_stream(region_id, ScanRequest::default())
|
||||
.await
|
||||
.unwrap();
|
||||
let batches = RecordBatches::try_collect(stream).await.unwrap();
|
||||
assert_eq!(6, batches.iter().map(|b| b.num_rows()).sum::<usize>());
|
||||
}
|
||||
|
||||
@@ -105,7 +105,10 @@ async fn test_engine_create_close_create_region_with_format(flat_format: bool) {
|
||||
.unwrap();
|
||||
// Close the region.
|
||||
engine
|
||||
.handle_request(region_id, RegionRequest::Close(RegionCloseRequest {}))
|
||||
.handle_request(
|
||||
region_id,
|
||||
RegionRequest::Close(RegionCloseRequest::default()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
// Create the same region id again.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user