mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-08-18 12:08:22 +00:00
chore: add development Docker image skill (#8477)
* chore: add development Docker image skill Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden development image skill Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: address development image review feedback Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: add image tag script license header Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>
This commit is contained in:
Weny Xu
GitHub
@@ -0,0 +1,368 @@
|
||||
---
|
||||
name: greptimedb-development-docker-image
|
||||
description: Builds a development-only GreptimeDB Docker image from a local debug binary for local-cluster testing, and optionally pushes it to a development registry. Use when the user asks to package, build, tag, publish, or cross-build a non-release GreptimeDB or GreptimeDB Enterprise image for debugging.
|
||||
compatibility: Requires Docker with Buildx for cross-platform builds; Podman is supported for native-platform builds.
|
||||
metadata:
|
||||
protocols: docker buildx podman
|
||||
platforms: linux-amd64 linux-arm64
|
||||
---
|
||||
|
||||
# GreptimeDB Development Docker Image
|
||||
|
||||
## Goal
|
||||
|
||||
Package a locally built GreptimeDB binary into a **development-only** Docker
|
||||
image for debugging and local-cluster testing, optionally push it to a
|
||||
development registry, and retain the non-secret image settings in `.env` for
|
||||
the next build. It is **not** a release-image workflow and must not be used to
|
||||
publish a production or release artifact.
|
||||
|
||||
This skill follows the documented development-image build procedure:
|
||||
|
||||
- Build `greptime` with Cargo's `nightly` profile.
|
||||
- Copy the target binary into the Docker build context as `./greptime`.
|
||||
- Build the supplied `Dockerfile`, which uses Ubuntu 24.04 only as the runtime
|
||||
base image and exposes the `greptime` binary as its entrypoint.
|
||||
|
||||
## Inputs
|
||||
|
||||
Collect or discover the following before any build or push:
|
||||
|
||||
| Input | Discovery and rule |
|
||||
| --- | --- |
|
||||
| Source repository | Treat the directory in which this skill is invoked as the default workspace. Include it as an editable field in the batch configuration; do not ask a separate confirmation. Do not assume open-source versus Enterprise. |
|
||||
| Edition and binary | Default to `greptime`; use the binary name the user requests for Enterprise builds. The copied Docker-context filename must always be `greptime`. |
|
||||
| Cargo profile and binary source | Include this in the batch configuration. Default to rebuilding `nightly`; reuse an existing binary only after the user explicitly accepts that its freshness is unverified. |
|
||||
| Target platform | Preselect `linux/amd64` unless Docker's server is `linux/arm64`, then preselect `linux/arm64`. Let the user override it in the batch configuration. Ubuntu 24.04 is the runtime base image, not a target-platform choice. Each image has exactly one target platform, but it may differ from the host platform. |
|
||||
| Build mode | Ask whether the user wants a locally loadable debug image or a registry push. |
|
||||
| Registry/repository | Read `IMAGE_REGISTRY` and `IMAGE_REPOSITORY` only from the selected workspace's `.env` and prefill them in the batch configuration. Example: `registry.example.com/team` + `greptimedb-dev`. |
|
||||
| Tag | Read `IMAGE_TAG` only from the selected workspace's `.env` and prefill it in the batch configuration. When the tag exists in the selected registry, preselect an incremented version. |
|
||||
|
||||
Use the image reference `${IMAGE_REGISTRY}/${IMAGE_REPOSITORY}:${IMAGE_TAG}`.
|
||||
If the registry is intentionally empty, omit its slash rather than producing a
|
||||
leading slash.
|
||||
|
||||
## Interactive Workflow
|
||||
|
||||
Use the platform's interactive prompt component for every question, selection,
|
||||
and confirmation. Do not ask an open-ended text question when a single-choice,
|
||||
multi-select, or confirmation component can represent the decision. If the
|
||||
platform does not provide an interactive component, use the equivalent numbered
|
||||
or lettered prompt below and wait for input before continuing.
|
||||
|
||||
### Batch configuration
|
||||
|
||||
Minimize user round trips: run the collector first against the invocation
|
||||
directory, then use one interactive form or batched prompt to collect workspace,
|
||||
edition/binary, profile/reuse-or-rebuild choice, target platform, build mode,
|
||||
registry/repository, tag, and whether to inspect the configured registry tag.
|
||||
Use the invocation directory as the editable workspace default. Prefill
|
||||
collector values and mark recommended defaults. If the user changes workspace,
|
||||
rerun the collector for that workspace without asking another configuration
|
||||
question. Only ask a follow-up if information is missing or invalid, or a safety
|
||||
gate is required. Treat unchanged prefilled values as accepted.
|
||||
|
||||
Keep separate confirmation components only for elevated privileges: `sudo`,
|
||||
package installation, and privileged QEMU setup. Use one final confirmation for
|
||||
all non-privileged selected work.
|
||||
|
||||
Always offer **Cancel** for an action that can write state, build, push, or
|
||||
require elevated privileges. Display the relevant preview before the user
|
||||
confirms it: source/binary path, `file` architecture result, platform(s), image
|
||||
reference, build mode, and non-secret `.env` values.
|
||||
|
||||
### Batch configuration fields
|
||||
|
||||
Use single-choice controls inside the one batch form for mutually exclusive
|
||||
choices:
|
||||
|
||||
| Decision | Required choices |
|
||||
| --- | --- |
|
||||
| Cargo output | Rebuild `nightly` (recommended), reuse an existing binary only with a freshness-unverified acknowledgement, choose a profile/binary |
|
||||
| Target platform | `linux/amd64` (default unless Docker server is arm64), `linux/arm64` (default when Docker server is arm64). The selected image may be cross-built with Buildx. |
|
||||
| Build mode | Load locally (recommended), push to development registry |
|
||||
| Registry/repository | Use `.env` defaults, enter new values |
|
||||
| Tag | Use proposed increment when the configured tag exists, enter a version |
|
||||
|
||||
When push is selected and the image configuration is complete, automatically
|
||||
run the read-only registry-tag check. It may return `unknown`; do not treat that
|
||||
as a missing tag.
|
||||
|
||||
Use one final confirmation for all selected non-privileged operations:
|
||||
|
||||
- persist changed non-secret image settings to `.env`;
|
||||
- run the selected Cargo build or reuse the resolved binary;
|
||||
- prepare the Docker context;
|
||||
- build, verify, and, when selected, push the image.
|
||||
|
||||
The final confirmation must clearly state: **“This creates a development and
|
||||
local-cluster test image, not a release or production artifact.”**
|
||||
|
||||
For a real multi-select only, use lettered choices (`[A]`, `[B]`) and accept
|
||||
`all`, `none`, or `cancel`; otherwise use single-choice components.
|
||||
|
||||
### Preflight context collection
|
||||
|
||||
Before asking the profile, registry, or tag questions, run the bundled,
|
||||
read-only collector. It works on macOS and Linux, does not invoke `cargo build`,
|
||||
and emits JSON that can be shown or summarized to the user:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/collect_context.py \
|
||||
--source <source-checkout> \
|
||||
--bin greptime \
|
||||
--profile nightly \
|
||||
[--target <rust-target>]
|
||||
```
|
||||
|
||||
Use its report to identify:
|
||||
|
||||
- missing `IMAGE_REGISTRY`, `IMAGE_REPOSITORY`, or `IMAGE_TAG` values in the
|
||||
selected workspace's `.env`;
|
||||
- Cargo binary targets available in the workspace and whether the requested
|
||||
`--bin` exists;
|
||||
- the resolved Cargo target directory, expected profile output path, and whether
|
||||
that binary already exists; and
|
||||
- macOS/Linux host architecture plus Docker's server platform when Docker is
|
||||
available.
|
||||
|
||||
When push is selected and image configuration is complete, rerun the collector
|
||||
with `--check-registry-tag` and all selected image values. It uses the selected
|
||||
engine's read-only manifest inspection and returns `exists` or `unknown`;
|
||||
`unknown` includes an unavailable registry, an absent tag, or missing
|
||||
authentication and must not be treated as an absent tag.
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/collect_context.py \
|
||||
--source <source-checkout> \
|
||||
--bin <binary> \
|
||||
--profile <profile> \
|
||||
--engine <docker|podman> \
|
||||
--registry <registry> \
|
||||
--repository <repository> \
|
||||
--tag <tag> \
|
||||
--check-registry-tag
|
||||
```
|
||||
|
||||
When the status is `exists` and the tag ends in a number, preselect the
|
||||
collector's `candidate_tag` as the next development tag. The user must still
|
||||
confirm it before `.env` is updated or an image is built. Never use automatic
|
||||
incrementing as permission to overwrite or push an existing image.
|
||||
|
||||
If Cargo metadata or the requested binary target is unavailable, stop before
|
||||
building and ask the user to correct the source path or binary target.
|
||||
|
||||
The profile/binary choice is a field in the batch configuration, not a separate
|
||||
question. Its default is the existing `nightly` output when present:
|
||||
|
||||
```text
|
||||
Question: Which Cargo build output should package this image?
|
||||
Options:
|
||||
- Rebuild nightly (Recommended): run Cargo with `--profile nightly`, then use its output.
|
||||
- Reuse nightly output: use the existing output only after showing its path, modification time, size, and matching platform; freshness is unverified.
|
||||
- Choose profile or binary: provide a different Cargo profile or an explicit binary path.
|
||||
```
|
||||
|
||||
Do not infer that a binary under `target/debug`, `target/release`, or another
|
||||
profile is suitable. For an existing binary, show its resolved path and run
|
||||
`file` before asking for final build confirmation.
|
||||
|
||||
## Environment State
|
||||
|
||||
The image configuration lives only in the selected workspace's `.env`. The
|
||||
collector reads only the following keys and never reads or displays other `.env`
|
||||
values:
|
||||
|
||||
```dotenv
|
||||
IMAGE_REGISTRY=registry.example.com/team
|
||||
IMAGE_REPOSITORY=greptimedb-dev
|
||||
IMAGE_TAG=dev-001
|
||||
```
|
||||
|
||||
1. Read `.env` if it exists. Ignore blank lines and `#` comments.
|
||||
2. Show the discovered registry, repository, and proposed tag to the user.
|
||||
3. Include registry/repository and tag in the batch configuration even when
|
||||
values already exist; saved values are defaults, not authorization to reuse
|
||||
them.
|
||||
4. Compare the selected configuration with the three current managed values. If
|
||||
all match, skip `.env` confirmation and do not update the file. Otherwise,
|
||||
preview only the changed `IMAGE_*` values, request confirmation, and update
|
||||
the file while preserving unrelated entries and comments.
|
||||
5. Never put registry credentials, access tokens, passwords, or `docker login`
|
||||
output in `.env`, build commands, logs, or responses. Ask the user to log in
|
||||
themselves if a push needs authentication.
|
||||
|
||||
For a repeated build, offer an incremented tag before asking. Use
|
||||
`scripts/next_image_tag.py --tag <existing-tag>` to calculate it. It increments
|
||||
the final numeric component while preserving zero padding:
|
||||
|
||||
```text
|
||||
weny-2025-0715-01 -> weny-2025-0715-02
|
||||
v0.1.4 -> v0.1.5
|
||||
debug-009 -> debug-010
|
||||
```
|
||||
|
||||
If the saved tag has no trailing number, ask the user for the next tag rather
|
||||
than inventing a versioning convention. Do not overwrite an existing image tag
|
||||
without explicit user confirmation.
|
||||
|
||||
Only if a managed value is missing or differs, persist the values with the
|
||||
bundled helper after confirmation instead of hand-editing the file:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/update_image_env.py \
|
||||
--env <source-checkout>/.env \
|
||||
--registry <registry> \
|
||||
--repository <repository> \
|
||||
--tag <tag>
|
||||
```
|
||||
|
||||
## Build Procedure
|
||||
|
||||
### 1. Inspect the host and container tooling
|
||||
|
||||
Run these checks and report the selected path:
|
||||
|
||||
```bash
|
||||
uname -s
|
||||
uname -m
|
||||
docker version --format '{{.Server.Os}}/{{.Server.Arch}}'
|
||||
docker buildx version
|
||||
docker buildx inspect --bootstrap
|
||||
```
|
||||
|
||||
For a native-platform build, Podman can be used if Docker is unavailable. For
|
||||
any non-native request, require Docker Buildx. Do not silently fall back to a
|
||||
native build when the requested image platform differs.
|
||||
|
||||
If the requested target is unavailable, stop and ask the user to configure a
|
||||
Buildx builder and cross-compilation toolchain externally. Do not automate
|
||||
privileged QEMU or binfmt setup, and never run an unpinned privileged image.
|
||||
|
||||
### 2. Build the executable for the image platform
|
||||
|
||||
From the source checkout, build the requested binary. For open-source amd64:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> --package cmd --bin greptime --profile nightly
|
||||
```
|
||||
|
||||
For open-source arm64:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> \
|
||||
--package cmd \
|
||||
--bin greptime \
|
||||
--profile nightly \
|
||||
--target aarch64-unknown-linux-gnu
|
||||
```
|
||||
|
||||
For Enterprise, use the user-provided executable target, for example:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_binary.sh \
|
||||
--source <source-checkout> --bin greptime-ent-cloud --profile nightly
|
||||
```
|
||||
|
||||
For a user-selected profile, replace `nightly` in the helper call and resolve
|
||||
the binary under `target/<profile>/<binary>` (or
|
||||
`target/<rust-target>/<profile>/<binary>` for cross-compilation). Reuse the
|
||||
existing target file only when the interactive profile question selected reuse;
|
||||
otherwise invoke the helper to rebuild it.
|
||||
|
||||
Do not use a binary compiled for the host architecture in an image intended for
|
||||
another architecture. Determine the binary path from the Cargo target and
|
||||
profile, then copy it into the Docker context as `greptime`:
|
||||
|
||||
```bash
|
||||
cp <source-target-binary> <build-context>/greptime
|
||||
```
|
||||
|
||||
Verify it before building the image with `file <build-context>/greptime`; its
|
||||
reported architecture must match the requested target platform.
|
||||
|
||||
Create a fresh isolated build context (never use the repository root) and use
|
||||
the bundled preparation script. The helper rejects an existing Dockerfile or
|
||||
binary output to prevent accidental context reuse:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/prepare_context.sh \
|
||||
--context "$(mktemp -d)" \
|
||||
--binary <source-target-binary> \
|
||||
--dockerfile <skill-dir>/assets/Dockerfile \
|
||||
--platform <platform>
|
||||
```
|
||||
|
||||
### 3. Build or push the image
|
||||
|
||||
Run the command only after the user confirms the final image reference and
|
||||
whether it should be pushed.
|
||||
|
||||
Use the bundled build helper rather than spelling out individual Docker or
|
||||
Podman commands. It follows the existing scripts' Docker-first/Podman-fallback
|
||||
behavior, uses Buildx for a non-native Docker request, and accepts exactly one
|
||||
target platform per image:
|
||||
|
||||
```bash
|
||||
<skill-dir>/scripts/build_image.sh \
|
||||
--context <build-context> \
|
||||
--image <image-reference> \
|
||||
--platform <platform> \
|
||||
--mode local|push
|
||||
```
|
||||
|
||||
`--mode local` builds a native image locally or uses `docker buildx --load` for
|
||||
a non-native image. `--mode push` pushes the selected single-platform image.
|
||||
The helper never pushes in `local` mode.
|
||||
|
||||
This is intentionally a runtime image built from a precompiled binary, not a
|
||||
multi-stage Dockerfile. Ubuntu 24.04 is the runtime base image only; select one
|
||||
target platform separately as `linux/amd64` or `linux/arm64`. Only introduce a
|
||||
multi-stage Dockerfile if the user asks to compile within Docker or the local
|
||||
Rust toolchain cannot build the required target. Keep the final Ubuntu 24.04
|
||||
runtime stage and existing entrypoint unless the user asks to change runtime
|
||||
behavior.
|
||||
|
||||
At the final confirmation, repeat that the selected reference is a
|
||||
**development/local-cluster test image**, not a release artifact. If the user
|
||||
requests a release or production image, stop and direct them to the release
|
||||
process instead.
|
||||
|
||||
### 4. Verify the result
|
||||
|
||||
For a local image, use the selected engine to inspect its architecture and
|
||||
start it with `--help`:
|
||||
|
||||
```bash
|
||||
docker image inspect <image-reference> --format '{{.Os}}/{{.Architecture}}'
|
||||
docker run --rm <image-reference> --help
|
||||
```
|
||||
|
||||
For Podman, use `podman image inspect <image-reference>` and
|
||||
`podman run --rm <image-reference> --help`. For a pushed Podman image, use
|
||||
`podman manifest inspect <image-reference>`.
|
||||
|
||||
For a pushed image, inspect its remote manifest:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect <image-reference>
|
||||
```
|
||||
|
||||
Report the final image reference, target platform, source binary path, and
|
||||
whether the image was loaded locally or pushed.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- This skill is only for development and local-cluster testing. Never present
|
||||
its image as a production, release, or officially published artifact.
|
||||
- Never push by default. Local debug builds should use a local tag and load the
|
||||
image unless the user explicitly requests a push.
|
||||
- Do not run `docker login`, privileged QEMU setup, package installation, or
|
||||
overwrite a tag without confirmation.
|
||||
- Do not delete existing images, builders, binaries, or `.env` entries as part
|
||||
of this workflow.
|
||||
- Stop and explain if Docker's server is unavailable, Buildx cannot support the
|
||||
requested platform, or the copied binary architecture does not match the
|
||||
image platform.
|
||||
@@ -0,0 +1,14 @@
|
||||
FROM ubuntu:24.04
|
||||
|
||||
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
unzip \
|
||||
mysql-client \
|
||||
wget \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY ./greptime /greptime/bin/
|
||||
|
||||
ENV PATH=/greptime/bin/:$PATH
|
||||
|
||||
ENTRYPOINT ["greptime"]
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Validate that a binary is an executable ELF for a requested Linux platform."""
|
||||
|
||||
import argparse
|
||||
import stat
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
PLATFORM_MACHINES = {"linux/amd64": 0x3E, "linux/arm64": 0xB7}
|
||||
|
||||
|
||||
def validate_platform(platform: str) -> int:
|
||||
try:
|
||||
return PLATFORM_MACHINES[platform]
|
||||
except KeyError as error:
|
||||
raise ValueError("platform must be linux/amd64 or linux/arm64") from error
|
||||
|
||||
|
||||
def inspect_elf(path: Path) -> int:
|
||||
executable_bits = stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
|
||||
if path.is_symlink() or not path.is_file() or not (path.stat().st_mode & executable_bits):
|
||||
raise ValueError("binary must be an executable regular file")
|
||||
try:
|
||||
with path.open("rb") as handle:
|
||||
header = handle.read(64)
|
||||
except OSError as error:
|
||||
raise ValueError(f"cannot read binary: {error}") from error
|
||||
if len(header) != 64 or header[:4] != b"\x7fELF":
|
||||
raise ValueError("binary is not ELF")
|
||||
if header[4] != 2 or header[5] != 1 or header[6] != 1:
|
||||
raise ValueError("binary must be a 64-bit little-endian ELF")
|
||||
if header[7] not in (0, 3):
|
||||
raise ValueError("binary must use the System V or Linux ELF ABI")
|
||||
elf_type, machine, version = struct.unpack_from("<HHI", header, 16)
|
||||
header_size = struct.unpack_from("<H", header, 52)[0]
|
||||
if elf_type not in (2, 3) or version != 1 or header_size != 64:
|
||||
raise ValueError("binary has an unsupported ELF executable header")
|
||||
return machine
|
||||
|
||||
|
||||
def validate_binary(path: Path, platform: str) -> None:
|
||||
expected = validate_platform(platform)
|
||||
actual = inspect_elf(path)
|
||||
if actual != expected:
|
||||
raise ValueError("binary architecture does not match requested platform")
|
||||
|
||||
|
||||
def main(argv: Optional[list[str]] = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--binary", required=True, type=Path)
|
||||
parser.add_argument("--platform", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
validate_binary(args.binary, args.platform)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --source PATH --bin NAME --profile PROFILE [--target RUST_TARGET] [--package PACKAGE] [--features FEATURES]\n' "$0" >&2
|
||||
}
|
||||
|
||||
source_dir=""
|
||||
binary_name=""
|
||||
profile=""
|
||||
target=""
|
||||
package=""
|
||||
features=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--source|--bin|--profile|--target|--package|--features)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
case "$1" in
|
||||
--source) source_dir="$2" ;;
|
||||
--bin) binary_name="$2" ;;
|
||||
--profile) profile="$2" ;;
|
||||
--target) target="$2" ;;
|
||||
--package) package="$2" ;;
|
||||
--features) features="$2" ;;
|
||||
esac
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$source_dir" || -z "$binary_name" || -z "$profile" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ! -f "$source_dir/Cargo.toml" ]]; then
|
||||
printf 'error: not a Cargo workspace: %s\n' "$source_dir" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
command=(cargo build --locked --bin "$binary_name" --profile "$profile")
|
||||
if [[ -n "$package" ]]; then
|
||||
command+=(--package "$package")
|
||||
fi
|
||||
if [[ -n "$features" ]]; then
|
||||
command+=(--features "$features")
|
||||
fi
|
||||
if [[ -n "$target" ]]; then
|
||||
command+=(--target "$target")
|
||||
fi
|
||||
|
||||
printf 'building %s with profile %s\n' "$binary_name" "$profile"
|
||||
(
|
||||
cd "$source_dir"
|
||||
exec "${command[@]}"
|
||||
)
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --context PATH --image REFERENCE --platform PLATFORM --mode local|push [--dockerfile PATH] [--engine auto|docker|podman] [--allow-local-tag-overwrite]\n' "$0" >&2
|
||||
}
|
||||
|
||||
context=""
|
||||
image=""
|
||||
platform=""
|
||||
mode=""
|
||||
dockerfile=""
|
||||
engine="auto"
|
||||
allow_local_tag_overwrite=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--context|--image|--platform|--mode|--dockerfile|--engine)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
case "$1" in
|
||||
--context) context="$2" ;;
|
||||
--image) image="$2" ;;
|
||||
--platform) platform="$2" ;;
|
||||
--mode) mode="$2" ;;
|
||||
--dockerfile) dockerfile="$2" ;;
|
||||
--engine) engine="$2" ;;
|
||||
esac
|
||||
shift 2
|
||||
;;
|
||||
--allow-local-tag-overwrite)
|
||||
allow_local_tag_overwrite=true
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$context" || -z "$image" || -z "$platform" || -z "$mode" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "$mode" != "local" && "$mode" != "push" ]]; then
|
||||
printf 'error: --mode must be local or push\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "$engine" != "auto" && "$engine" != "docker" && "$engine" != "podman" ]]; then
|
||||
printf 'error: --engine must be auto, docker, or podman\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
dockerfile="${dockerfile:-$context/Dockerfile}"
|
||||
if [[ ! -f "$dockerfile" ]]; then
|
||||
printf 'error: Dockerfile does not exist: %s\n' "$dockerfile" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$platform" == *,* ]]; then
|
||||
printf 'error: only one target platform is supported per image build\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
case "$platform" in
|
||||
linux/amd64|linux/arm64) ;;
|
||||
*)
|
||||
printf 'error: unsupported platform: %s\n' "$platform" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
python3 "$script_dir/binary_platform.py" --binary "$context/greptime" --platform "$platform"
|
||||
|
||||
check_collision() {
|
||||
local runtime="$1"
|
||||
if [[ "$allow_local_tag_overwrite" == true ]]; then
|
||||
return 0
|
||||
fi
|
||||
if [[ "$runtime" == docker ]] && docker image inspect "$image" >/dev/null 2>&1; then
|
||||
printf 'error: local image tag exists: %s (pass --allow-local-tag-overwrite to replace it)\n' "$image" >&2
|
||||
exit 3
|
||||
fi
|
||||
if [[ "$runtime" == podman ]] && podman image exists "$image"; then
|
||||
printf 'error: local image tag exists: %s (pass --allow-local-tag-overwrite to replace it)\n' "$image" >&2
|
||||
exit 3
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "$engine" != podman ]] && command -v docker >/dev/null 2>&1 && server_platform="$(docker version --format '{{.Server.Os}}/{{.Server.Arch}}' 2>/dev/null)"; then
|
||||
if [[ "$platform" == "$server_platform" ]]; then
|
||||
check_collision docker
|
||||
docker build --platform "$platform" -f "$dockerfile" -t "$image" "$context"
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec docker push "$image"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec docker buildx build --platform "$platform" --push -f "$dockerfile" -t "$image" "$context"
|
||||
fi
|
||||
check_collision docker
|
||||
exec docker buildx build --platform "$platform" --load -f "$dockerfile" -t "$image" "$context"
|
||||
fi
|
||||
|
||||
if [[ "$engine" != docker ]] && command -v podman >/dev/null 2>&1; then
|
||||
native_platform="$(podman info --format '{{.Host.Os}}/{{.Host.Arch}}' 2>/dev/null || true)"
|
||||
case "$native_platform" in
|
||||
linux/amd64|linux/arm64) ;;
|
||||
*)
|
||||
printf 'error: Podman is unavailable or has unsupported platform: %s\n' "$native_platform" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
if [[ "$platform" != "$native_platform" ]]; then
|
||||
printf 'error: Docker Buildx is required for non-native platform %s\n' "$platform" >&2
|
||||
exit 1
|
||||
fi
|
||||
check_collision podman
|
||||
podman build -f "$dockerfile" -t "$image" "$context"
|
||||
if [[ "$mode" == "push" ]]; then
|
||||
exec podman push "$image"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf 'error: Docker or Podman is required\n' >&2
|
||||
exit 1
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Collect read-only build context for a GreptimeDB development image."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, cast
|
||||
|
||||
from next_image_tag import increment_tag
|
||||
from binary_platform import PLATFORM_MACHINES, inspect_elf
|
||||
from image_config import image_reference, parse_env_file, validate_values
|
||||
|
||||
|
||||
ENV_KEYS = ("IMAGE_REGISTRY", "IMAGE_REPOSITORY", "IMAGE_TAG")
|
||||
PROFILE_DIRECTORIES = {"dev": "debug", "test": "debug", "bench": "release"}
|
||||
PLATFORM_ARCHITECTURES = {
|
||||
"x86_64": "linux/amd64",
|
||||
"amd64": "linux/amd64",
|
||||
"aarch64": "linux/arm64",
|
||||
"arm64": "linux/arm64",
|
||||
}
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source", required=True, type=Path)
|
||||
parser.add_argument("--bin", dest="binary", default="greptime")
|
||||
parser.add_argument("--profile", default="nightly")
|
||||
parser.add_argument("--target", help="Rust target triple for cross-compilation")
|
||||
parser.add_argument("--package", help="Cargo package containing the selected binary")
|
||||
parser.add_argument("--platform", choices=("linux/amd64", "linux/arm64"))
|
||||
parser.add_argument("--engine", choices=("auto", "docker", "podman"), default="auto")
|
||||
parser.add_argument("--registry")
|
||||
parser.add_argument("--repository")
|
||||
parser.add_argument("--tag")
|
||||
parser.add_argument(
|
||||
"--check-registry-tag",
|
||||
action="store_true",
|
||||
help="inspect the configured image tag with Docker without modifying it",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def native_platform(architecture: str) -> Optional[str]:
|
||||
return PLATFORM_ARCHITECTURES.get(architecture.lower())
|
||||
|
||||
|
||||
def parse_env(env_path: Path) -> dict[str, object]:
|
||||
values = parse_env_file(env_path)
|
||||
missing = [key for key in ("IMAGE_REPOSITORY", "IMAGE_TAG") if not values.get(key)]
|
||||
try:
|
||||
if not missing:
|
||||
validate_values(values)
|
||||
error = None
|
||||
except ValueError as exc:
|
||||
error = str(exc)
|
||||
return {"path": str(env_path), "exists": env_path.is_file(), "values": values, "missing": missing, "error": error}
|
||||
|
||||
|
||||
def cargo_metadata(source: Path) -> dict[str, Any]:
|
||||
command = ["cargo", "metadata", "--locked", "--format-version=1", "--no-deps"]
|
||||
try:
|
||||
result = subprocess.run(command, cwd=source, text=True, capture_output=True, check=True)
|
||||
except FileNotFoundError as error:
|
||||
raise RuntimeError("cargo is not installed or is not on PATH") from error
|
||||
except subprocess.CalledProcessError as error:
|
||||
detail = error.stderr.strip() or error.stdout.strip()
|
||||
raise RuntimeError(f"cargo metadata failed: {detail}") from error
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def binary_targets(metadata: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
targets = []
|
||||
for package in metadata["packages"]:
|
||||
for target in package["targets"]:
|
||||
if "bin" in target["kind"]:
|
||||
targets.append({
|
||||
"package": package["name"],
|
||||
"name": target["name"],
|
||||
"path": target["src_path"],
|
||||
"required_features": target.get("required-features", []),
|
||||
})
|
||||
return sorted(targets, key=lambda target: (target["name"], target["package"]))
|
||||
|
||||
|
||||
def expected_binary(target_dir: Path, binary: str, profile: str, rust_target: Optional[str]) -> Path:
|
||||
profile_directory = PROFILE_DIRECTORIES.get(profile, profile)
|
||||
output_dir = target_dir / rust_target if rust_target else target_dir
|
||||
return output_dir / profile_directory / binary
|
||||
|
||||
|
||||
def engine_status(engine: str) -> dict[str, object]:
|
||||
if engine == "docker":
|
||||
command = ["docker", "version", "--format", "{{.Server.Os}}/{{.Server.Arch}}"]
|
||||
else:
|
||||
command = ["podman", "info", "--format", "{{.Host.Os}}/{{.Host.Arch}}"]
|
||||
try:
|
||||
result = subprocess.run(command, text=True, capture_output=True, check=True)
|
||||
except (FileNotFoundError, subprocess.CalledProcessError):
|
||||
return {"available": False, "server_platform": None}
|
||||
return {"available": True, "server_platform": result.stdout.strip()}
|
||||
|
||||
|
||||
def registry_tag_status(values: dict[str, str], enabled: bool, engine: str, source: str) -> dict[str, object]:
|
||||
if not enabled:
|
||||
return {"checked": False, "status": "not_checked", "candidate_tag": None}
|
||||
|
||||
try:
|
||||
reference = image_reference(values)
|
||||
except ValueError:
|
||||
return {"checked": False, "status": "not_configured", "candidate_tag": None, "configuration_source": source}
|
||||
command = ["docker", "buildx", "imagetools", "inspect", reference] if engine == "docker" else ["podman", "manifest", "inspect", reference]
|
||||
try:
|
||||
subprocess.run(command, text=True, capture_output=True, check=True)
|
||||
except FileNotFoundError:
|
||||
return {"checked": True, "status": "unknown", "candidate_tag": None, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
except subprocess.CalledProcessError:
|
||||
return {"checked": True, "status": "unknown", "candidate_tag": None, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
|
||||
try:
|
||||
candidate_tag = increment_tag(values["IMAGE_TAG"])
|
||||
except ValueError:
|
||||
candidate_tag = None
|
||||
return {"checked": True, "status": "exists", "candidate_tag": candidate_tag, "reference": reference, "engine": engine, "configuration_source": source}
|
||||
|
||||
|
||||
def collect_report(args: argparse.Namespace) -> dict[str, object]:
|
||||
source = args.source.expanduser().resolve()
|
||||
if not (source / "Cargo.toml").is_file():
|
||||
raise RuntimeError(f"not a Cargo workspace: {source}")
|
||||
|
||||
metadata = cargo_metadata(source)
|
||||
targets = binary_targets(metadata)
|
||||
target_dir = Path(metadata["target_directory"])
|
||||
expected = expected_binary(target_dir, args.binary, args.profile, args.target)
|
||||
host_architecture = platform.machine()
|
||||
selected_target = next((target for target in targets if target["name"] == args.binary), None)
|
||||
environment = parse_env(source / ".env")
|
||||
supplied_values = (args.registry, args.repository, args.tag)
|
||||
if any(value is not None for value in supplied_values) and not all(value is not None for value in supplied_values):
|
||||
raise RuntimeError("--registry, --repository, and --tag must be supplied together")
|
||||
values: dict[str, str] = {"IMAGE_REGISTRY": args.registry or "", "IMAGE_REPOSITORY": args.repository or "", "IMAGE_TAG": args.tag or ""} if all(value is not None for value in supplied_values) else cast(dict[str, str], environment["values"])
|
||||
configuration_source = "arguments" if all(value is not None for value in supplied_values) else "environment"
|
||||
docker = engine_status("docker")
|
||||
podman = engine_status("podman")
|
||||
engine = args.engine if args.engine != "auto" else "docker" if docker["available"] else "podman"
|
||||
binary_details: dict[str, object] = {"platform": None, "error": None}
|
||||
if expected.is_file():
|
||||
try:
|
||||
machine = inspect_elf(expected)
|
||||
binary_details["platform"] = next(
|
||||
(name for name, expected_machine in PLATFORM_MACHINES.items() if expected_machine == machine),
|
||||
None,
|
||||
)
|
||||
if binary_details["platform"] is None:
|
||||
binary_details["error"] = f"binary has unsupported ELF machine: {machine}"
|
||||
except ValueError as exc:
|
||||
binary_details["error"] = str(exc)
|
||||
return {
|
||||
"host": {"os": platform.system(), "arch": host_architecture, "native_platform": native_platform(host_architecture)},
|
||||
"docker": docker,
|
||||
"podman": podman,
|
||||
"environment": environment,
|
||||
"registry_tag": registry_tag_status(values, args.check_registry_tag, engine, configuration_source),
|
||||
"cargo": {"workspace_root": metadata["workspace_root"], "target_dir": str(target_dir), "binaries": targets},
|
||||
"selected_binary": {"name": args.binary, "package": args.package, "cargo_target_exists": selected_target is not None, "target": selected_target},
|
||||
"expected_binary": {"profile": args.profile, "rust_target": args.target, "path": str(expected), "exists": expected.is_file(), "executable": os.access(expected, os.X_OK), "platform": binary_details["platform"], "requested_platform": args.platform, "platform_matches": binary_details["platform"] == args.platform if args.platform and binary_details["platform"] else None, "validation_error": binary_details["error"], "mtime": datetime.fromtimestamp(expected.stat().st_mtime, timezone.utc).isoformat() if expected.exists() else None, "freshness": "unverified"},
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
print(json.dumps(collect_report(parse_arguments()), indent=2, sort_keys=True))
|
||||
except RuntimeError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Bounded parsing and safe updating of the image settings in a .env file."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Mapping
|
||||
|
||||
MANAGED_KEYS = ("IMAGE_REGISTRY", "IMAGE_REPOSITORY", "IMAGE_TAG")
|
||||
_REGISTRY = re.compile(r"[a-z0-9][a-z0-9.-]*(?::[0-9]+)?(?:/[a-z0-9][a-z0-9._-]*)*")
|
||||
_REPOSITORY = re.compile(r"[a-z0-9][a-z0-9._-]*(?:/[a-z0-9][a-z0-9._-]*)*")
|
||||
_TAG = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}")
|
||||
|
||||
|
||||
def _value(text: str) -> str:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return ""
|
||||
quote = text[0] if text[0] in "'\"" else None
|
||||
if quote:
|
||||
end = text.find(quote, 1)
|
||||
if end < 0 or text[end + 1 :].strip() and not text[end + 1 :].lstrip().startswith("#"):
|
||||
raise ValueError("unmatched quote or trailing value syntax")
|
||||
return text[1:end]
|
||||
# A comment is inline only when introduced by whitespace. This keeps
|
||||
# valid values such as an image tag containing '#'.
|
||||
match = re.search(r"\s+#", text)
|
||||
if match:
|
||||
text = text[: match.start()].rstrip()
|
||||
if any(character in text for character in "'\";$`"):
|
||||
raise ValueError("unsupported shell syntax in .env value")
|
||||
return text
|
||||
|
||||
|
||||
def parse_env(text: str) -> dict[str, str]:
|
||||
"""Parse the three managed keys, without validating their values."""
|
||||
lines = text.splitlines()
|
||||
result: dict[str, str] = {}
|
||||
for number, original in enumerate(lines, 1):
|
||||
line = original.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
export = re.match(r"export[ \t]+", line)
|
||||
if export:
|
||||
line = line[export.end() :].lstrip()
|
||||
key, separator, raw = line.partition("=")
|
||||
if not separator:
|
||||
continue
|
||||
key = key.strip()
|
||||
if key in MANAGED_KEYS:
|
||||
try:
|
||||
result[key] = _value(raw)
|
||||
except ValueError as error:
|
||||
raise ValueError(f"line {number}: {error}") from error
|
||||
return result
|
||||
|
||||
|
||||
def parse_env_file(path: Path) -> dict[str, str]:
|
||||
return parse_env(path.read_text() if path.exists() else "")
|
||||
|
||||
|
||||
def normalize_values(values: Mapping[str, str]) -> dict[str, str]:
|
||||
return {
|
||||
"IMAGE_REGISTRY": values.get("IMAGE_REGISTRY", ""),
|
||||
"IMAGE_REPOSITORY": values.get("IMAGE_REPOSITORY", ""),
|
||||
"IMAGE_TAG": values.get("IMAGE_TAG", ""),
|
||||
}
|
||||
|
||||
|
||||
def validate_values(values: Mapping[str, str]) -> None:
|
||||
values = normalize_values(values)
|
||||
missing = [key for key in ("IMAGE_REPOSITORY", "IMAGE_TAG") if not values.get(key)]
|
||||
if missing:
|
||||
raise ValueError("missing managed values: " + ", ".join(missing))
|
||||
registry = values.get("IMAGE_REGISTRY", "")
|
||||
if registry and (
|
||||
not _REGISTRY.fullmatch(registry)
|
||||
):
|
||||
raise ValueError("invalid registry")
|
||||
if registry:
|
||||
host = registry.split("/", 1)[0]
|
||||
if host != "localhost" and "." not in host and ":" not in host:
|
||||
raise ValueError("ambiguous registry; use an FQDN, explicit port, or localhost")
|
||||
if not _REPOSITORY.fullmatch(values["IMAGE_REPOSITORY"]):
|
||||
raise ValueError("invalid repository")
|
||||
if not _TAG.fullmatch(values["IMAGE_TAG"]):
|
||||
raise ValueError("invalid tag")
|
||||
if any(any(character.isspace() for character in value) for value in values.values()):
|
||||
raise ValueError("managed values cannot contain whitespace")
|
||||
|
||||
|
||||
def image_reference(values: Mapping[str, str]) -> str:
|
||||
values = normalize_values(values)
|
||||
validate_values(values)
|
||||
prefix = values.get("IMAGE_REGISTRY", "")
|
||||
repository = values["IMAGE_REPOSITORY"]
|
||||
return f"{prefix}/{repository}:{values['IMAGE_TAG']}" if prefix else f"{repository}:{values['IMAGE_TAG']}"
|
||||
|
||||
|
||||
def update_env(path: Path, values: Mapping[str, str]) -> bool:
|
||||
"""Atomically update managed entries; return whether the file changed."""
|
||||
values = normalize_values(values)
|
||||
validate_values(values)
|
||||
if path.is_symlink():
|
||||
raise ValueError("refusing to update symlink .env")
|
||||
old = path.read_text() if path.exists() else ""
|
||||
parse_env(old) # detect malformed managed syntax before changing anything
|
||||
lines = old.splitlines()
|
||||
output: list[str] = []
|
||||
written: set[str] = set()
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
export = re.match(r"export[ \t]+", stripped)
|
||||
candidate = stripped[export.end() :].lstrip() if export else stripped
|
||||
key = candidate.split("=", 1)[0].strip() if "=" in candidate else ""
|
||||
if key in MANAGED_KEYS:
|
||||
if key in written:
|
||||
continue
|
||||
output.append(f"{key}={values[key]}")
|
||||
written.add(key)
|
||||
else:
|
||||
output.append(line)
|
||||
output.extend(f"{key}={values.get(key, '')}" for key in MANAGED_KEYS if key not in written)
|
||||
new = "\n".join(output) + "\n"
|
||||
if old == new:
|
||||
return False
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
mode = os.stat(path).st_mode & 0o777 if path.exists() else 0o600
|
||||
fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
|
||||
try:
|
||||
os.fchmod(fd, mode)
|
||||
with os.fdopen(fd, "w") as handle:
|
||||
handle.write(new)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
return True
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Print a tag with its final numeric component incremented."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def increment_tag(tag: str) -> str:
|
||||
match = re.search(r"(\d+)$", tag)
|
||||
if match is None:
|
||||
raise ValueError("tag must end with a numeric version component")
|
||||
|
||||
number = match.group(1)
|
||||
return f"{tag[:match.start()]}{int(number) + 1:0{len(number)}d}"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Increment the final numeric component of a Docker image tag."
|
||||
)
|
||||
parser.add_argument("--tag", required=True, help="existing Docker image tag")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
print(increment_tag(args.tag))
|
||||
except ValueError as error:
|
||||
print(f"error: {error}: {args.tag!r}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s --context PATH --binary PATH --dockerfile PATH --platform PLATFORM\n' "$0" >&2
|
||||
}
|
||||
|
||||
context=""
|
||||
binary=""
|
||||
dockerfile=""
|
||||
platform=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--context)
|
||||
context="$2"
|
||||
shift 2
|
||||
;;
|
||||
--binary)
|
||||
binary="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dockerfile)
|
||||
dockerfile="$2"
|
||||
shift 2
|
||||
;;
|
||||
--platform)
|
||||
platform="$2"
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$context" || -z "$binary" || -z "$dockerfile" || -z "$platform" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
python3 "$script_dir/binary_platform.py" --binary "$binary" --platform "$platform"
|
||||
|
||||
if [[ ! -f "$dockerfile" ]]; then
|
||||
printf 'error: Dockerfile template does not exist: %s\n' "$dockerfile" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -e "$context" && ( -L "$context" || -n "$(ls -A "$context")" ) ]]; then
|
||||
printf 'error: build context must be a new or empty directory: %s\n' "$context" >&2
|
||||
exit 1
|
||||
fi
|
||||
mkdir -p "$context"
|
||||
if [[ -e "$context/greptime" || -e "$context/Dockerfile" || -L "$context/greptime" || -L "$context/Dockerfile" ]]; then
|
||||
printf 'error: build context already contains output files: %s\n' "$context" >&2
|
||||
exit 1
|
||||
fi
|
||||
cp "$binary" "$context/greptime"
|
||||
cp "$dockerfile" "$context/Dockerfile"
|
||||
printf 'created Dockerfile: %s\n' "$context/Dockerfile"
|
||||
|
||||
file "$context/greptime"
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Create or update non-secret GreptimeDB image settings in a .env file."""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from image_config import image_reference, normalize_values, update_env, validate_values
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--env", type=Path)
|
||||
parser.add_argument("--registry", required=True)
|
||||
parser.add_argument("--repository", required=True)
|
||||
parser.add_argument("--tag", required=True)
|
||||
parser.add_argument("--validate-only", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
values = normalize_values({"IMAGE_REGISTRY": args.registry, "IMAGE_REPOSITORY": args.repository, "IMAGE_TAG": args.tag})
|
||||
validate_values(values)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.validate_only:
|
||||
print(image_reference(values))
|
||||
return 0
|
||||
if args.env is None:
|
||||
parser.error("--env is required unless --validate-only is set")
|
||||
if update_env(args.env, values):
|
||||
print(f"updated {args.env}")
|
||||
else:
|
||||
print(f"unchanged {args.env}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import importlib
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
binary_platform = importlib.import_module("binary_platform")
|
||||
|
||||
|
||||
def elf(machine: int, *, bits: int = 2, endian: int = 1, abi: int = 3, elf_type: int = 3) -> bytes:
|
||||
header = bytearray(64)
|
||||
header[:4] = b"\x7fELF"
|
||||
header[4:8] = bytes((bits, endian, 1, abi))
|
||||
struct.pack_into("<H", header, 16, elf_type)
|
||||
struct.pack_into("<H" if endian == 1 else ">H", header, 18, machine)
|
||||
struct.pack_into("<I", header, 20, 1)
|
||||
struct.pack_into("<H", header, 52, 64)
|
||||
return bytes(header)
|
||||
|
||||
|
||||
class BinaryPlatformTest(unittest.TestCase):
|
||||
def make_binary(self, content: bytes) -> Path:
|
||||
path = Path(self.directory.name) / "greptime"
|
||||
path.write_bytes(content)
|
||||
path.chmod(0o755)
|
||||
return path
|
||||
|
||||
def setUp(self):
|
||||
self.directory = tempfile.TemporaryDirectory()
|
||||
|
||||
def tearDown(self):
|
||||
self.directory.cleanup()
|
||||
|
||||
def test_matching_architectures(self):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "linux/amd64")
|
||||
binary_platform.validate_binary(self.make_binary(elf(0xB7)), "linux/arm64")
|
||||
|
||||
def test_rejects_bad_executable_elf_and_platform(self):
|
||||
path = self.make_binary(elf(0x3E))
|
||||
path.chmod(0o644)
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
path = self.make_binary(b"not an elf")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
path = self.make_binary(elf(0x3E, bits=1))
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(path, "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E, abi=9)), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E, elf_type=1)), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(b"\x7fELF"), "linux/amd64")
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "darwin/amd64")
|
||||
|
||||
def test_rejects_wrong_architecture_and_symlink(self):
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(self.make_binary(elf(0x3E)), "linux/arm64")
|
||||
target = self.make_binary(elf(0x3E))
|
||||
link = Path(self.directory.name) / "link"
|
||||
link.symlink_to(target)
|
||||
with self.assertRaises(ValueError):
|
||||
binary_platform.validate_binary(link, "linux/amd64")
|
||||
|
||||
def test_cli_success_and_failures(self):
|
||||
path = self.make_binary(elf(0x3E))
|
||||
assert binary_platform.__file__ is not None
|
||||
script = Path(binary_platform.__file__)
|
||||
ok = subprocess.run([sys.executable, str(script), "--binary", str(path), "--platform", "linux/amd64"], capture_output=True, text=True)
|
||||
self.assertEqual(ok.returncode, 0, ok.stderr)
|
||||
bad = subprocess.run([sys.executable, str(script), "--binary", str(path), "--platform", "linux/arm64"], capture_output=True, text=True)
|
||||
self.assertEqual(bad.returncode, 2)
|
||||
missing = subprocess.run([sys.executable, str(script), "--binary", str(path) + ".missing", "--platform", "linux/amd64"], capture_output=True, text=True)
|
||||
self.assertEqual(missing.returncode, 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2023 Greptime Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import importlib
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
image_config = importlib.import_module("image_config")
|
||||
|
||||
|
||||
VALUES = {"IMAGE_REGISTRY": "registry.example.com/team", "IMAGE_REPOSITORY": "greptime-dev", "IMAGE_TAG": "dev-001"}
|
||||
|
||||
|
||||
class ImageConfigTest(unittest.TestCase):
|
||||
def test_parse_supported_bounded_dotenv(self):
|
||||
text = """\n# comment\n export IMAGE_REGISTRY = \"registry.example.com/team\" # note\nIMAGE_REPOSITORY='greptime-dev'\nIMAGE_TAG=dev-001 # safe comment\nOTHER=$HOME\n"""
|
||||
self.assertEqual(image_config.parse_env(text), VALUES)
|
||||
|
||||
def test_parser_does_not_validate_values(self):
|
||||
self.assertEqual(image_config.parse_env("IMAGE_TAG=not valid\n"), {"IMAGE_TAG": "not valid"})
|
||||
|
||||
def test_parser_rejects_unmatched_and_unsupported_syntax(self):
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.parse_env('IMAGE_TAG="unterminated\n')
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.parse_env("IMAGE_TAG=$(date)\n")
|
||||
|
||||
def test_validation_and_reference(self):
|
||||
image_config.validate_values(VALUES)
|
||||
self.assertEqual(image_config.image_reference(VALUES), "registry.example.com/team/greptime-dev:dev-001")
|
||||
local = dict(VALUES, IMAGE_REGISTRY="")
|
||||
self.assertEqual(image_config.image_reference(local), "greptime-dev:dev-001")
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="localhost"))
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="example.com"))
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.validate_values(dict(VALUES, IMAGE_REGISTRY="registry/team"))
|
||||
|
||||
def test_atomic_update_preserves_mode_and_canonicalizes_duplicates(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("# keep\nIMAGE_TAG=old\nIMAGE_TAG=duplicate\nOTHER=value\n")
|
||||
path.chmod(0o640)
|
||||
self.assertTrue(image_config.update_env(path, VALUES))
|
||||
self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o640)
|
||||
self.assertEqual(path.read_text(), "# keep\nIMAGE_TAG=dev-001\nOTHER=value\nIMAGE_REGISTRY=registry.example.com/team\nIMAGE_REPOSITORY=greptime-dev\n")
|
||||
self.assertFalse(image_config.update_env(path, VALUES))
|
||||
|
||||
def test_update_rejects_symlink(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
target = Path(directory) / "target"
|
||||
target.write_text("IMAGE_TAG=old\n")
|
||||
link = Path(directory) / ".env"
|
||||
link.symlink_to(target)
|
||||
with self.assertRaises(ValueError):
|
||||
image_config.update_env(link, VALUES)
|
||||
self.assertEqual(target.read_text(), "IMAGE_TAG=old\n")
|
||||
|
||||
def test_update_clears_omitted_registry_and_uses_private_new_file_mode(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("IMAGE_REGISTRY=registry.example.com\nIMAGE_REPOSITORY=old\nIMAGE_TAG=old\n")
|
||||
values = {"IMAGE_REPOSITORY": "greptime-dev", "IMAGE_TAG": "dev-001"}
|
||||
image_config.update_env(path, values)
|
||||
self.assertIn("IMAGE_REGISTRY=\n", path.read_text())
|
||||
new_path = Path(directory) / "new.env"
|
||||
image_config.update_env(new_path, VALUES)
|
||||
self.assertEqual(stat.S_IMODE(new_path.stat().st_mode), 0o600)
|
||||
|
||||
def test_failed_replace_keeps_original_and_removes_temporary_file(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / ".env"
|
||||
path.write_text("OTHER=value\n")
|
||||
with mock.patch("image_config.os.replace", side_effect=OSError("no space")):
|
||||
with self.assertRaises(OSError):
|
||||
image_config.update_env(path, VALUES)
|
||||
self.assertEqual(path.read_text(), "OTHER=value\n")
|
||||
self.assertEqual(list(Path(directory).glob("..env.*")), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user