diff --git a/.agents/skills/greptimedb-release-note/SKILL.md b/.agents/skills/greptimedb-release-note/SKILL.md index 026ad90cd8d..461c5149d5a 100644 --- a/.agents/skills/greptimedb-release-note/SKILL.md +++ b/.agents/skills/greptimedb-release-note/SKILL.md @@ -39,6 +39,12 @@ Double-check with the user. ## 3. Pick the range and generate +For every release type, use the release tag's commit as `` when the tag exists. +Resolve the exact tag from the remote (use the peeled `^{}` commit for annotated tags) +and exclude commits added to the release branch afterward. Before tagging, pin the +intended release commit and record its SHA. Pass that SHA as `--target` when creating +the release, then verify the resulting tag before publishing the docs copy. + Key topology fact: **patch tags** (`v1.0.1`, `v1.0.2`) live on `release/vX.Y` and are **NOT** ancestors of `main` (they are cherry-picks with different SHAs). Up to v1.1 the `.0` tag was the point where `release/vX.Y` left `main`; from v1.2 the branch is cut before the `.0`, and @@ -51,8 +57,8 @@ left `main`, minus what any published `vX.(Y-1).*` release (pre-releases, `.0`, already shipped. The line's own pre-releases (`vX.Y.0-beta.*`) are not subtracted; the `.0` note covers the whole minor. -Base = the **previous minor tag** (e.g. `v1.1.0`); tip = the tagged `vX.Y.0` commit, resolved as -in "Pre-release" below. Split the range at the point where `release/vX.Y` left `main`: +Base = the **previous minor tag** (e.g. `v1.1.0`); tip = the release commit resolved above. +Split the range at the point where `release/vX.Y` left `main`: ``` bp=$(git merge-base /release/vX.Y /main) ``` @@ -80,14 +86,14 @@ small Python script is the reliable way — match `pull/)` on lines starting The previous patch tag is an ancestor of the release branch, so a plain range works **when the cherry-picks land as individual commits**: ``` -GITHUB_TOKEN=$(gh auth token) git cliff .. --tag vX.Y.Z -o ... +GITHUB_TOKEN=$(gh auth token) git cliff .. --tag vX.Y.Z -o ... ``` No extra subtraction (the branch only contains the new cherry-picks). **Common case — squashed pick commit.** Patch branches are often a single squashed commit like `chore: pick fixes and bump version to vX.Y.Z (#NNNN)`, so the plain range above yields just that one bullet. First **find the picked PRs**: read the squash commit message -(`git log -1 --format=%B `) — each `* fix: ... (#NN)` line is a picked PR. +(`git log -1 --format=%B `) — each `* fix: ... (#NN)` line is a picked PR. Then pick one of two ways to build the changelog (a patch is usually only a few PRs — **ask the user which they prefer**): @@ -131,16 +137,6 @@ Either source alone misses PRs: a release body can omit a backport that its tag backport commit can name only the backport PR. ### Pre-release (`X.Y.Z-{alpha,beta,rc}.N` after an earlier pre-release) -**The tip is the tagged commit, not the release branch tip.** Resolve it from the remote and -write this SHA in the docs PR body (§8) as what the note was checked against: -``` -git ls-remote --tags 'vX.Y.Z-beta.N*' -``` -Use the `refs/tags/vX.Y.Z-beta.N` line (its `^{}` line if the tag is annotated), not the -build-suffixed tags it also lists. PRs that land on `release/vX.Y` after the tag -(`git log --oneline ../release/vX.Y`) belong to the next release, even if they are -already merged when you write the note. - The range `..` usually mixes two kinds of commits: individual backports (`... [Backport release/vX.Y] (#NNNN)`) and squashed pick commits (`chore: backport selected changes for ... (#NNNN)`). In both, the original PRs are the `(#NN)` at the end of body lines; @@ -156,8 +152,8 @@ done | tr -dc '0-9\n' | sort -un List the bullets under the **original** PRs (title and author of the PR merged to `main`), not the backport PRs. Build them with patch (a) or (b) above; for (b), fetch `gh api repos/GreptimeTeam/greptimedb/pulls/ --jq '{number,title,author:.user.login,merged_at}'`, -group by the `cliff.toml` parsers (first match wins), order by `merged_at`, and put `!` titles -under Breaking changes (§6). +group by the `cliff.toml` parsers (first match wins), order by `merged_at`, and review `!` titles +for user-visible compatibility impact (§6) before assigning a category. A squashed pick PR can also contain release-only commits that have no PR. List them: ``` @@ -179,6 +175,10 @@ them and add the user-facing ones by hand: ``` git log --format=%s | grep -E '\(#[0-9]+\)$' | grep -vE '^[a-z]+(\([^)]*\))?!?: ' ``` +Review generated breaking-change bullets using the same user-impact criteria as §6. Move +internal-only entries to ordinary groups, preserving genuine incompatibilities, PR links, +and authors. + Use only the groups in `cliff.toml`. `git cliff` prints a `### Build` section for `build:` commits because no parser matches them; move a user-facing `build(deps)` fix to Bug Fixes and the rest to Miscellaneous Tasks. Keep each bullet list free of blank lines; a blank line turns it into a @@ -217,45 +217,18 @@ line (ask the user; some prefer to keep it). ## 6. Human-curated sections (insert after the `Release date:` line) -**Style.** Follow the Google developer documentation style guide: sentence-case headings, active voice, present -tense for behavior, American spelling, serial comma, code font for identifiers, bold only for UI -names. Keep it short: -- Intro: two sentences at most, what the release covers and its main theme. -- Highlight: one paragraph of up to four sentences, plus an example if it earns one (see below). -- Upgrade note or breaking change: one bullet, one or two sentences: what changes, who is - affected, what to do. -- Every sentence states a fact the reader acts on or checks. Delete sentences that restate the - heading or the previous sentence, transitions, and summaries. -- No marketing words (`significantly`, `powerful`, `seamless`, `robust`, `enhanced`), no vague - verbs (`supports`, `handles`, `improves`; say what happens), no `several`/`many` when the number - is known. Keep the source's hedges; don't add new ones. +Write concise, factual English. Rank changes by their impact and explain what changed, +why it matters, and any required action. Verify claims against the release's PRs, code, +and documentation; preserve their conditions and limitations. -Example: -> Bad: This release significantly enhances write reliability by introducing a robust retry -> mechanism that seamlessly handles transient failures. -> -> Good: In distributed clusters, the frontend retries a failed write up to three times with -> exponential backoff. Set `max_retries = 0` to disable retries. - -Mirror past release bodies (`gh release view v1.0.0 --repo GreptimeTeam/greptimedb`): -- **Short intro** (see Style). -- **`### 👍 Highlights`** — *few, deep* highlights, ranked by user impact. Fixes for silently - wrong query results, lost or invisible data, and security issues rank with new features and - can be highlights. - - Write a fix as its symptom and the affected deployments or usage, not the PR title. Example: - "with no traffic, `errors / requests > 0.05` was true, so the alert fired". - - State whether a feature is on by default. If an example config uses the default values, say - so. - - Give an example (SQL / TOML config) only where it helps readers act. Don't pick highlights, - or the example, by what is easy to demonstrate. - - One highlight covers one change or one user-visible problem. Don't merge unrelated PRs into - one entry because they touch the same component. - - Sources: the PR body's sections on impact, compatibility, and behavior changes, plus the - code and config it touches, not the title alone. Verify examples against the related docs - PR/page in the docs repo (`GreptimeTeam/docs`, often checked out locally), - `docs/reference/sql/*.md`, `config/*.example.toml`, `config/config.md`. - Do **not** mention implementation details, tiny features, or unfinished/experimental-but-not-ready - features. Always have the user review and edit the highlights; iterate. +- **Short intro** — summarize the release in one or two sentences. +- **`### 👍 Highlights`** — select the most consequential changes, including new features, + correctness, security, performance, and substantial architectural improvements. Internal changes can + be highlights when their technical significance or benefit is supported by evidence; + change size alone is not a reason to feature them. Explain enough of the mechanism to + make the benefit clear, without turning the note into an implementation walkthrough. + Keep related changes together and distinguish shipped benefits from future possibilities. + Include usage examples only when needed, and verify them against the released version. - **Dashboard** subsection — don't just bump the version. Read the dashboard PRs in the bundled release (`gh release view --repo GreptimeTeam/dashboard`; then the linked PRs) and describe the user-facing change. Use the names the UI shows, not the PR titles; check them in @@ -266,22 +239,18 @@ Mirror past release bodies (`gh release view v1.0.0 --repo GreptimeTeam/greptime ``` Screenshots need absolute `https://` URLs: the body is copied to the GitHub release, where relative and site-root (`/img/...`) paths don't resolve. -- **`### Upgrade notes`** and **`### Breaking changes`** — check every PR in the release for: - `!` titles; `Compatibility`, `Behavior change`, `Breaking` paragraphs in the body (often a - plain `Compatibility:` line, not a heading); unchecked "API changes are backward compatible" / - "Schema or data changes are backward compatible" boxes; new config options and their defaults; - changed defaults; persisted format changes; rolling-upgrade order; new hard limits. These - signals set reading order, not a filter: a limit can sit in plain prose with every box - checked, so read the body of every user-facing `feat`/`fix` PR. Examples of what is easy to - miss: a 512 MiB cap on decompressed request bodies (#9264); a default step for stepless - subqueries (#9481). - - **Upgrade notes**: steps operators take — upgrade order, index rebuilds, settings to review. - - **Breaking changes**: behavior that used to work and now errors or returns different - results. Put the `!` PR bullets here too, after these entries (see `v1.3.0-beta.1`). +- **`### Upgrade and compatibility notes`** — distinguish internal changes from changes + visible to database users and operators. Include only user-visible compatibility changes + and required upgrade actions; state who is affected, what changes, and what to do. + Read PR bodies to establish impact: `!` titles and compatibility checkboxes alone do not + determine whether a change belongs here. Internal changes without upgrade impact belong in + highlights or the ordinary changelog, according to significance. Preserve PR links and + contributor attribution. Omit empty sections and avoid repeating information across + curated sections. - **Benchmark results** (optional) — state the baseline version (often the previous formal - release, not the previous beta), data size, cache state, and hardware. Don't attribute results - to single PRs. Leave out results the benchmark marks as not for publication, and smoke-level - runs. + release, not the previous beta), workload, data size, cache state, and hardware. Attribute results + to individual changes only when the measurements isolate their effect. Leave out results the + benchmark marks as not for publication, and smoke-level runs. ## 7. Output @@ -323,7 +292,7 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**. (`.github/pull_request_template.md`) from the checkout used for the release. Fill in every required section; do not rely on hard-coded section names from an older template. The checklist items are the author's to confirm: check the ones you did, and for one - that doesn't apply, check it and say why. For pre-releases, state the tagged SHA (§3) under + that doesn't apply, check it and say why. State the release SHA (§3) under Verification. - **Build the site before opening the PR.** A new worktree has no `node_modules`. Install with the pnpm version pinned in `package.json` (`packageManager`), as CI does: diff --git a/.agents/skills/greptimedb-release/SKILL.md b/.agents/skills/greptimedb-release/SKILL.md index 215493519ec..9efb60e646c 100644 --- a/.agents/skills/greptimedb-release/SKILL.md +++ b/.agents/skills/greptimedb-release/SKILL.md @@ -62,11 +62,12 @@ publishing. **Do NOT pre-create the tag.** Creating the release creates the tag and fires the tag-push CI (`.github/workflows/release.yml`) that builds all binaries (**~hours**). +Use the exact commit SHA verified for the changelog as ``. Confirm with the user, then: ``` gh release create vX.Y.Z \ --repo GreptimeTeam/greptimedb \ - --target release/vX.Y \ + --target \ --title "Release vX.Y.Z" \ --notes-file CHANGELOG-vX.Y.Z.md \ --prerelease @@ -78,7 +79,8 @@ gh release create vX.Y.Z \ ## 4. Open the docs release-note PR (do not wait for CI) -Right after triggering the release, open the docs draft PR (see the docs section of the +Verify that the remote tag resolves to the verified release SHA (peel annotated tags); +stop if it differs. Then open the docs draft PR (see the docs section of the `greptimedb-release-note` skill). It only needs the finalized changelog. Then **delete the local `CHANGELOG-vX.Y.Z.md`** (its content now lives in the release body diff --git a/.github/actions/deploy-greptimedb/action.yml b/.github/actions/deploy-greptimedb/action.yml index b18830ef8d7..634cae85dcd 100644 --- a/.github/actions/deploy-greptimedb/action.yml +++ b/.github/actions/deploy-greptimedb/action.yml @@ -19,6 +19,12 @@ inputs: runs: using: composite steps: + # Ubuntu 26.04 does not apply the runner's /etc/sysctl.conf at boot. + # https://github.com/actions/runner-images/issues/15108 + - name: Configure inotify limits for kind + shell: bash + run: sudo sysctl -w fs.inotify.max_user_instances=1280 fs.inotify.max_user_watches=655360 + - name: Deploy GreptimeDB by Helm shell: bash env: @@ -28,4 +34,5 @@ runs: AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }} AWS_SECRET_ACCESS_KEY: ${{ inputs.aws-secret-access-key }} run: | - ./.github/scripts/deploy-greptimedb.sh + # Composite steps do not support timeout-minutes. + timeout --kill-after=30s 30m ./.github/scripts/deploy-greptimedb.sh diff --git a/.github/actions/sqlness-test/action.yml b/.github/actions/sqlness-test/action.yml index 004f2b40b16..255a175f7a0 100644 --- a/.github/actions/sqlness-test/action.yml +++ b/.github/actions/sqlness-test/action.yml @@ -58,11 +58,19 @@ runs: run: | aws s3 rm s3://${{ inputs.aws-ci-test-bucket }}/${{ inputs.data-root }} --recursive - name: Export kind logs - if: failure() + if: failure() || cancelled() shell: bash - run: kind export logs /tmp/kind + run: | + # Bound the entire collection while leaving the upload step available. + timeout --kill-after=30s 5m bash -e -o pipefail <<'EOF' + mkdir -p /tmp/kind + kubectl --request-timeout=30s get pods -A -o wide > /tmp/kind/pods.txt 2>&1 || true + kubectl --request-timeout=30s get events -A --sort-by=.metadata.creationTimestamp > /tmp/kind/events.txt 2>&1 || true + kubectl --request-timeout=30s describe pods -A > /tmp/kind/pod-descriptions.txt 2>&1 || true + kind export logs /tmp/kind + EOF - name: Upload logs - if: failure() + if: failure() || cancelled() uses: actions/upload-artifact@v4 with: name: kind-logs diff --git a/.github/scripts/deploy-greptimedb.sh b/.github/scripts/deploy-greptimedb.sh index 92ca440dbd9..9747edc7796 100755 --- a/.github/scripts/deploy-greptimedb.sh +++ b/.github/scripts/deploy-greptimedb.sh @@ -50,7 +50,7 @@ function deploy_etcd_cluster() { -n "$namespace" # Wait for etcd cluster to be ready. - kubectl rollout status statefulset/etcd -n "$namespace" + kubectl rollout status statefulset/etcd -n "$namespace" --timeout=5m } # Deploy greptimedb-operator. @@ -81,16 +81,8 @@ function deploy_greptimedb_cluster() { -n "$install_namespace" # Wait for greptimedb cluster to be ready. - while true; do - PHASE=$(kubectl -n "$install_namespace" get gtc "$cluster_name" -o jsonpath='{.status.clusterPhase}') - if [ "$PHASE" == "Running" ]; then - echo "Cluster is ready" - break - else - echo "Cluster is not ready yet: Current phase: $PHASE" - sleep 5 # wait for 5 seconds before check again. - fi - done + kubectl -n "$install_namespace" wait "gtc/$cluster_name" \ + --for=jsonpath='{.status.clusterPhase}'=Running --timeout=5m # Expose greptimedb cluster to local access and check if port-forward is available. kubectl -n "$install_namespace" port-forward svc/"$cluster_name"-frontend \ @@ -141,16 +133,8 @@ function deploy_greptimedb_cluster_with_s3_storage() { --set objectStorage.credentials.secretAccessKey="$AWS_SECRET_ACCESS_KEY" # Wait for greptimedb cluster to be ready. - while true; do - PHASE=$(kubectl -n "$install_namespace" get gtc "$cluster_name" -o jsonpath='{.status.clusterPhase}') - if [ "$PHASE" == "Running" ]; then - echo "Cluster is ready" - break - else - echo "Cluster is not ready yet: Current phase: $PHASE" - sleep 5 # wait for 5 seconds before check again. - fi - done + kubectl -n "$install_namespace" wait "gtc/$cluster_name" \ + --for=jsonpath='{.status.clusterPhase}'=Running --timeout=5m # Expose greptimedb cluster to local access. kubectl -n "$install_namespace" port-forward svc/"$cluster_name"-frontend \ @@ -168,8 +152,8 @@ function deploy_standalone_greptimedb() { --set image.tag="$GREPTIMEDB_IMAGE_TAG" \ -n "$DEFAULT_INSTALL_NAMESPACE" - # Wait for etcd cluster to be ready. - kubectl rollout status statefulset/greptimedb-standalone -n "$DEFAULT_INSTALL_NAMESPACE" + # Wait for standalone GreptimeDB to be ready. + kubectl rollout status statefulset/greptimedb-standalone -n "$DEFAULT_INSTALL_NAMESPACE" --timeout=5m # Expose greptimedb to local access. kubectl -n "$DEFAULT_INSTALL_NAMESPACE" port-forward svc/greptimedb-standalone \ diff --git a/.github/workflows/nightly-ci.yml b/.github/workflows/nightly-ci.yml index 2b6862bb827..fe4e89cb59d 100644 --- a/.github/workflows/nightly-ci.yml +++ b/.github/workflows/nightly-ci.yml @@ -14,6 +14,7 @@ jobs: name: Run sqlness test if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} runs-on: ubuntu-latest + timeout-minutes: 45 steps: - name: Checkout uses: actions/checkout@v4 @@ -33,7 +34,7 @@ jobs: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - name: Upload sqlness logs - if: failure() + if: failure() || cancelled() uses: actions/upload-artifact@v4 with: name: sqlness-logs-kind diff --git a/Cargo.lock b/Cargo.lock index b420850ce68..fd31d88c4dc 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2544,6 +2544,7 @@ dependencies = [ "mito-codec", "nalgebra", "num-traits", + "partition", "paste", "pretty_assertions", "regex", @@ -2822,6 +2823,7 @@ dependencies = [ "common-time", "datafusion", "datafusion-common", + "datafusion-datasource", "datafusion-expr", "datafusion-proto", "datatypes", @@ -3779,7 +3781,7 @@ checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" [[package]] name = "datafusion" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -3831,7 +3833,7 @@ dependencies = [ [[package]] name = "datafusion-catalog" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3855,7 +3857,7 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3878,7 +3880,7 @@ dependencies = [ [[package]] name = "datafusion-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-ipc 59.2.0", @@ -3904,7 +3906,7 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "futures", "log", @@ -3914,7 +3916,7 @@ dependencies = [ [[package]] name = "datafusion-datasource" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-compression", @@ -3950,7 +3952,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-ipc 59.2.0", @@ -3974,7 +3976,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-trait", @@ -3997,7 +3999,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-trait", @@ -4020,7 +4022,7 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4052,12 +4054,12 @@ dependencies = [ [[package]] name = "datafusion-doc" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" [[package]] name = "datafusion-execution" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-buffer 59.2.0", @@ -4082,7 +4084,7 @@ dependencies = [ [[package]] name = "datafusion-expr" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4106,7 +4108,7 @@ dependencies = [ [[package]] name = "datafusion-expr-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4117,7 +4119,7 @@ dependencies = [ [[package]] name = "datafusion-functions" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-buffer 59.2.0", @@ -4148,7 +4150,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4168,7 +4170,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4179,7 +4181,7 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-ord 59.2.0", @@ -4203,7 +4205,7 @@ dependencies = [ [[package]] name = "datafusion-functions-table" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "async-trait", @@ -4218,7 +4220,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4234,7 +4236,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -4243,7 +4245,7 @@ dependencies = [ [[package]] name = "datafusion-macros" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "datafusion-doc", "quote", @@ -4253,7 +4255,7 @@ dependencies = [ [[package]] name = "datafusion-optimizer" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "chrono", @@ -4301,7 +4303,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-schema 59.2.0", @@ -4324,7 +4326,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4338,7 +4340,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "chrono", @@ -4355,7 +4357,7 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4374,7 +4376,7 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "arrow-data 59.2.0", @@ -4411,7 +4413,7 @@ dependencies = [ [[package]] name = "datafusion-proto" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-catalog", @@ -4437,7 +4439,7 @@ dependencies = [ [[package]] name = "datafusion-proto-common" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4447,7 +4449,7 @@ dependencies = [ [[package]] name = "datafusion-proto-models" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "datafusion-common", "datafusion-proto-common", @@ -4457,7 +4459,7 @@ dependencies = [ [[package]] name = "datafusion-pruning" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "datafusion-common", @@ -4472,7 +4474,7 @@ dependencies = [ [[package]] name = "datafusion-session" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow-schema 59.2.0", "async-trait", @@ -4486,7 +4488,7 @@ dependencies = [ [[package]] name = "datafusion-sql" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "arrow 59.2.0", "bigdecimal 0.4.8", @@ -4505,7 +4507,7 @@ dependencies = [ [[package]] name = "datafusion-substrait" version = "55.1.0" -source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=70f84c5bf2511d40ee4a4036fa086ba36f207a85#70f84c5bf2511d40ee4a4036fa086ba36f207a85" +source = "git+https://github.com/GreptimeTeam/datafusion.git?rev=638a77fb504e86f85ef6053c552bbee1ed814fb9#638a77fb504e86f85ef6053c552bbee1ed814fb9" dependencies = [ "async-recursion", "async-trait", @@ -6180,7 +6182,7 @@ dependencies = [ [[package]] name = "greptime-proto" version = "0.1.0" -source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=5fc04135bde7f30adea63c33a53533f69fe194b2#5fc04135bde7f30adea63c33a53533f69fe194b2" +source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=c46f7ee9c82ff90f5762ae192fcf8bb2b0d6046f#c46f7ee9c82ff90f5762ae192fcf8bb2b0d6046f" dependencies = [ "prost 0.14.1", "prost-types 0.14.1", @@ -7267,6 +7269,17 @@ version = "4.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0d762194228a2f1c11063e46e32e5acb96e66e906382b9eb5441f2e0504bbd5a" +[[package]] +name = "intern-arc" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2fe5735fd955e619f30a5257c8ce5234354cbe72d8a04a24414c9004125f02b2" +dependencies = [ + "loom", + "once_cell", + "parking_lot 0.12.4", +] + [[package]] name = "inventory" version = "0.3.20" @@ -8720,6 +8733,7 @@ dependencies = [ "object-store", "partition", "prometheus 0.14.0", + "rstest", "serde", "serde_json", "smallvec", @@ -8855,6 +8869,7 @@ dependencies = [ "humantime", "humantime-serde", "index", + "intern-arc", "itertools 0.14.0", "lazy_static", "log-store", @@ -9752,6 +9767,9 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "parking_lot_core 0.9.11", +] [[package]] name = "once_cell_polyfill" diff --git a/Cargo.toml b/Cargo.toml index 7429f0bf634..ea8f556548e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -164,7 +164,7 @@ fs2 = "0.4" fst = "0.4.7" futures = "0.3" futures-util = "0.3" -greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "5fc04135bde7f30adea63c33a53533f69fe194b2" } +greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "c46f7ee9c82ff90f5762ae192fcf8bb2b0d6046f" } hdfs-native = "0.14.6" hex = "0.4" hostname = "0.4.0" @@ -175,6 +175,7 @@ hyper = "1.1" hyper-util = "0.1" icu_properties = "2.0.1" indicatif = "0.17" +intern-arc = "0.6.2" itertools = "0.14" jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "5a6472858d860ac034075c17253f41617bdfa956", default-features = false, features = [ "databend", @@ -374,22 +375,22 @@ rev = "f743c3a5fe8c1f57363eac5866c356b7d91f9b39" [patch.crates-io] substrait = { git = "https://github.com/GreptimeTeam/substrait-rs.git", rev = "91ec978b0649417ad3da8390e7a515baec723b1b" } -datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-execution = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } -datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "70f84c5bf2511d40ee4a4036fa086ba36f207a85" } +datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-execution = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } +datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" } datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" } arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" } sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "9e9019bb1c7040ed956f654e39378dd42ab17884" } diff --git a/config/config.md b/config/config.md index 03029ae1fdc..0151d505092 100644 --- a/config/config.md +++ b/config/config.md @@ -13,11 +13,12 @@ | Key | Type | Default | Descriptions | | --- | -----| ------- | ----------- | | `default_timezone` | String | Unset | The default timezone of the server. | -| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, and native histogram columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | +| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | | `auto_create_table` | Bool | `true` | Server-side global switch for auto table creation on write.
When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. | +| `auto_create_physical_table_timestamp_unit` | String | Unset | Time index unit for physical metric tables created by auto table creation
(Prometheus remote write, OTLP metrics, and other metric-engine ingestion).
New logical tables created under such a physical table inherit its unit.
Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". | | `user_provider` | String | Unset | The user provider for authentication.
Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"
Password verifier formats: "plain:", "pbkdf2_sha256:::",
"mysql_native_password:",
"pg_scram_sha256::::"
"pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest, but such users cannot log in over MySQL:
`static_user_provider` and `watch_file_user_provider` only negotiate `mysql_native_password`, which cannot verify these formats.
"mysql_native_password" is MySQL-specific and cannot authenticate over PostgreSQL at all.
PostgreSQL SCRAM only covers "plain" and "pg_scram_sha256" users; if any user is "pbkdf2_sha256" or
"mysql_native_password", PostgreSQL falls back to cleartext password auth for every user.
For "pg_scram_sha256" users, keep the default iteration count (4096) and salt length (16): both are
observable in the SCRAM server-first message, and non-default values weaken resistance to username
enumeration. | -| `max_in_flight_write_bytes` | String | 0 | Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight).
Set to 0 to disable the limit. Default: "0" (unlimited) | -| `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.
Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail" | +| `max_in_flight_write_bytes` | String | 0 | Shared quota for concurrent write request bodies and messages (HTTP, gRPC, Flight)
plus retained decoder memory and estimated PB-to-Arrow conversion memory for ordinary-table batching.
A batch must fit together with its request's existing reservations; only additional memory is acquired.
Set to 0 to disable the limit. Default: "0" (unlimited) | +| `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.
Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail"
Table-batch conversion never waits for additional memory. Under a wait policy,
temporary contention returns a retryable error; requests that cannot fit are rejected. | | `init_regions_in_background` | Bool | `false` | Initialize all regions in the background during the startup.
By default, it provides services after all regions have been initialized. | | `init_regions_parallelism` | Integer | `16` | Parallelism of initializing regions. | | `max_concurrent_queries` | Integer | `0` | The maximum concurrent region-read streams. Zero means unlimited.
A single query can consume multiple permits. | @@ -293,11 +294,12 @@ | Key | Type | Default | Descriptions | | --- | -----| ------- | ----------- | | `default_timezone` | String | Unset | The default timezone of the server. | -| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, and native histogram columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | +| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | | `auto_create_table` | Bool | `true` | Server-side global switch for auto table creation on write.
When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. | +| `auto_create_physical_table_timestamp_unit` | String | Unset | Time index unit for physical metric tables created by auto table creation
(Prometheus remote write, OTLP metrics, and other metric-engine ingestion).
New logical tables created under such a physical table inherit its unit.
Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". | | `user_provider` | String | Unset | The user provider for authentication.
Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd"
Password verifier formats: "plain:", "pbkdf2_sha256:::",
"mysql_native_password:",
"pg_scram_sha256::::"
"pbkdf2_sha256" and "pg_scram_sha256" protect passwords at rest, but such users cannot log in over MySQL:
`static_user_provider` and `watch_file_user_provider` only negotiate `mysql_native_password`, which cannot verify these formats.
"mysql_native_password" is MySQL-specific and cannot authenticate over PostgreSQL at all.
PostgreSQL SCRAM only covers "plain" and "pg_scram_sha256" users; if any user is "pbkdf2_sha256" or
"mysql_native_password", PostgreSQL falls back to cleartext password auth for every user.
For "pg_scram_sha256" users, keep the default iteration count (4096) and salt length (16): both are
observable in the SCRAM server-first message, and non-default values weaken resistance to username
enumeration. | -| `max_in_flight_write_bytes` | String | 0 | Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight).
Set to 0 to disable the limit. Default: "0" (unlimited) | -| `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.
Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail" | +| `max_in_flight_write_bytes` | String | 0 | Shared quota for concurrent write request bodies and messages (HTTP, gRPC, Flight)
plus retained decoder memory and estimated PB-to-Arrow conversion memory for ordinary-table batching.
A batch must fit together with its request's existing reservations; only additional memory is acquired.
Set to 0 to disable the limit. Default: "0" (unlimited) | +| `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted.
Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail"
Table-batch conversion never waits for additional memory. Under a wait policy,
temporary contention returns a retryable error; requests that cannot fit are rejected. | | `runtime` | -- | -- | The runtime options. | | `runtime.global_rt_size` | Integer | Auto | The number of threads to execute the runtime for global read operations.
Defaults to max(num_cpus, 2). | | `runtime.compact_rt_size` | Integer | Auto | The number of threads to execute compact operations.
Defaults to max(num_cpus / 2, 2). | @@ -558,7 +560,7 @@ | Key | Type | Default | Descriptions | | --- | -----| ------- | ----------- | | `node_id` | Integer | Unset | The datanode identifier and should be unique in the cluster. | -| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, and native histogram columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | +| `default_column_prefix` | String | Unset | The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns.
Legacy OTLP summary columns keep their historical `greptime_` prefix. | | `require_lease_before_startup` | Bool | `false` | Start services after regions have obtained leases.
It will block the datanode start if it can't receive leases in the heartbeat from metasrv. | | `init_regions_in_background` | Bool | `false` | Initialize all regions in the background during the startup.
By default, it provides services after all regions have been initialized. | | `init_regions_parallelism` | Integer | `16` | Parallelism of initializing regions. | diff --git a/config/datanode.example.toml b/config/datanode.example.toml index 698bc0e9600..bbf7f284ea3 100644 --- a/config/datanode.example.toml +++ b/config/datanode.example.toml @@ -2,7 +2,7 @@ ## @toml2docs:none-default node_id = 42 -## The default column prefix for auto-created time index, value, and native histogram columns. +## The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns. ## Legacy OTLP summary columns keep their historical `greptime_` prefix. ## @toml2docs:none-default default_column_prefix = "greptime" diff --git a/config/frontend.example.toml b/config/frontend.example.toml index 6fac16b4b6a..f3be55fcaa0 100644 --- a/config/frontend.example.toml +++ b/config/frontend.example.toml @@ -2,7 +2,7 @@ ## @toml2docs:none-default default_timezone = "UTC" -## The default column prefix for auto-created time index, value, and native histogram columns. +## The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns. ## Legacy OTLP summary columns keep their historical `greptime_` prefix. ## @toml2docs:none-default default_column_prefix = "greptime" @@ -11,6 +11,13 @@ default_column_prefix = "greptime" ## When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. #+ auto_create_table = true +## Time index unit for physical metric tables created by auto table creation +## (Prometheus remote write, OTLP metrics, and other metric-engine ingestion). +## New logical tables created under such a physical table inherit its unit. +## Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". +## @toml2docs:none-default +#+ auto_create_physical_table_timestamp_unit = "Millisecond" + ## The user provider for authentication. ## Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd" ## Password verifier formats: "plain:", "pbkdf2_sha256:::", @@ -27,13 +34,17 @@ default_column_prefix = "greptime" ## @toml2docs:none-default #+ user_provider = "static_user_provider:file:/path/to/users" -## Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight). +## Shared quota for concurrent write request bodies and messages (HTTP, gRPC, Flight) +## plus retained decoder memory and estimated PB-to-Arrow conversion memory for ordinary-table batching. +## A batch must fit together with its request's existing reservations; only additional memory is acquired. ## Set to 0 to disable the limit. Default: "0" (unlimited) ## @toml2docs:none-default="0" #+ max_in_flight_write_bytes = "1GB" ## Policy when write bytes quota is exhausted. ## Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail" +## Table-batch conversion never waits for additional memory. Under a wait policy, +## temporary contention returns a retryable error; requests that cannot fit are rejected. #+ write_bytes_exhausted_policy = "wait" ## The runtime options. diff --git a/config/standalone.example.toml b/config/standalone.example.toml index ef5dff43dd9..f34a881af3d 100644 --- a/config/standalone.example.toml +++ b/config/standalone.example.toml @@ -2,7 +2,7 @@ ## @toml2docs:none-default default_timezone = "UTC" -## The default column prefix for auto-created time index, value, and native histogram columns. +## The default column prefix for auto-created time index, value, native histogram, and exemplar-label columns. ## Legacy OTLP summary columns keep their historical `greptime_` prefix. ## @toml2docs:none-default default_column_prefix = "greptime" @@ -11,6 +11,13 @@ default_column_prefix = "greptime" ## When `false`, a missing table is never auto-created even if the request sets the `auto_create_table` hint to `true`. Default: `true`. #+ auto_create_table = true +## Time index unit for physical metric tables created by auto table creation +## (Prometheus remote write, OTLP metrics, and other metric-engine ingestion). +## New logical tables created under such a physical table inherit its unit. +## Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". +## @toml2docs:none-default +#+ auto_create_physical_table_timestamp_unit = "Millisecond" + ## The user provider for authentication. ## Examples: "static_user_provider:file:/path/to/users", "static_user_provider:cmd:greptime_user=greptime_pwd" ## Password verifier formats: "plain:", "pbkdf2_sha256:::", @@ -27,13 +34,17 @@ default_column_prefix = "greptime" ## @toml2docs:none-default #+ user_provider = "static_user_provider:file:/path/to/users" -## Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight). +## Shared quota for concurrent write request bodies and messages (HTTP, gRPC, Flight) +## plus retained decoder memory and estimated PB-to-Arrow conversion memory for ordinary-table batching. +## A batch must fit together with its request's existing reservations; only additional memory is acquired. ## Set to 0 to disable the limit. Default: "0" (unlimited) ## @toml2docs:none-default="0" #+ max_in_flight_write_bytes = "1GB" ## Policy when write bytes quota is exhausted. ## Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail" +## Table-batch conversion never waits for additional memory. Under a wait policy, +## temporary contention returns a retryable error; requests that cannot fit are rejected. #+ write_bytes_exhausted_policy = "wait" ## Initialize all regions in the background during the startup. diff --git a/docs/rfcs/2026-09-06-object-store-wal.md b/docs/rfcs/2026-09-06-object-store-wal.md index d3e38bb5a26..c94abc073aa 100644 --- a/docs/rfcs/2026-09-06-object-store-wal.md +++ b/docs/rfcs/2026-09-06-object-store-wal.md @@ -31,7 +31,7 @@ Object storage is already provisioned, already durable across zones, and already # Non-goals - Distributed mode. The provider is rejected when a datanode is controlled by a metasrv; the metasrv side has no allocation logic for it yet. -- Persistence of the obsolete watermarks that flushes and truncates move. Mito re-establishes each of them when it opens a region, which is what garbage collection reasons from. Only the release of a region that is never opened again, after a drop or an offline cleanup, is persisted, as described under *Entry ids*. +- Persistence of the obsolete watermarks that flushes and truncates move. Mito re-establishes each of them when it opens a region, which is what garbage collection reasons from. A close persists a bounded release through the region's safe recovery watermark; a drop or offline cleanup persists the permanent release described under *Entry ids*. Ordinary flush and truncate watermarks are not persisted by the store. - Matching Raft Engine's write latency. The provider trades latency for the absence of local durable WAL state. - Making standalone metadata independent of local disk. The metadata store is unchanged by this RFC. @@ -106,7 +106,7 @@ Two rules keep every new id of a region above every id it already has, on a pref The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move for a prefix that holds the region's history, since every watermark Mito hands the store then names an object below the next sequence. A watermark above the newest object holding entries that the store found means the prefix does not hold that history: the history check under *Log store* refuses such an open before Mito calls `obsolete`, and the floor moves only when `accept_missing_history` accepted the loss. Otherwise the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced. -Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after a release in full, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after every release and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes the watermark of every region Mito opens again, and the boundary advances on the same rule. A region that is never opened again needs the release described below. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, or its own sequence when the recovered chain holds no segment, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing. +Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after a release in full, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after every release and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes the watermark of every region Mito opens again, and the boundary advances on the same rule. Closing a region while the WAL is running queues its safe watermark through the bounded release described below. Once the record is durable, recovery retains that progress even if the region is not reopened; closes after WAL shutdown retain extra history instead. A region that is dropped or cleaned up offline uses a permanent release. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, or its own sequence when the recovered chain holds no segment, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing. Every present object with a sequence below the durable boundary is deletable, on the chain or not: the objects the catalog indexes, the objects recovery listed but did not index, the objects of an earlier epoch that the start object moved past, and the sequences of batches that failed in this run, whose objects may or may not exist. A collection starts after a successful `obsolete`, after a release and after an indexing that raised the durable boundary. Deletes run in the background, at most four at a time and in no particular order, and never hold up admission, sealing, uploads or acknowledgements; an indexed object is unindexed only after its delete succeeded, and the WAL holes recorded for it are dropped with it. Every deleted object is counted in `greptime_logstore_object_store_wal_deleted_objects_total`; a failed delete is logged, counted in `greptime_logstore_object_store_wal_failed_deletes_total` and retried at the next collection, and the object stays indexed meanwhile. An object deleted but not yet unindexed when the process crashes is simply absent at the next recovery, which lists the prefix. Nothing is collected once `stop` has begun or while the store is poisoned; a delete in flight when `stop` begins runs to completion before `stop` returns, and the actor settles every attempt it abandons when it exits. @@ -143,22 +143,28 @@ The mode also needs a barrier that the durable mode gets for free. An entry id t | Situation | Sequence | Waiters of the batch | Store | | --- | --- | --- | --- | | create succeeds, or identical retry | advances | acknowledged in sequence order | healthy | -| transient I/O error, `durable` mode | consumed and never reused, nor are the sequences of the other failed batches | this and every batch that is not indexed fail, including those whose objects were created; a create still in flight completes as an orphan | healthy: the next batch links to the last indexed object | +| transient I/O error, `durable` mode | consumed and never reused, nor are the sequences of the other failed batches | this and every batch that is not indexed fail, including those whose objects were created; a create still in flight completes as an orphan | healthy: the next batch links to the last indexed object; retained release records back off before a release-only retry | | an object of an earlier epoch holds the sequence, `durable` mode | consumed and never reused | as for a transient I/O error | healthy | -| transient I/O error, `enqueued` mode | unchanged | already acknowledged, except a release, which waits for the repeat | healthy: the create is repeated after a short delay | -| I/O error that requires intervention, `enqueued` mode | unchanged | already acknowledged, except a release, which fails with a retryable error while its record stays in the backlog; later appends and releases are rejected with that error until a create succeeds | failing: the lowest pending create is repeated alone with backoff; the first create that succeeds ends it | -| any I/O error, `enqueued` mode after `stop` began | unchanged | already acknowledged, except a release, which fails with the stopped error | the backlog is dropped and `stop` reports the loss | +| transient I/O error, `enqueued` mode | unchanged | already acknowledged, except a permanent release, which waits for the repeat | healthy: the create is repeated after a short delay | +| I/O error that requires intervention, `enqueued` mode | unchanged | already acknowledged, except a permanent release, which fails with a retryable error while its record stays in the backlog; later appends and permanent releases are rejected with that error until a create succeeds | failing: the lowest pending create is repeated alone with backoff; the first create that succeeds ends it | +| any I/O error, `enqueued` mode after `stop` began | unchanged | already acknowledged, except a permanent release, which fails with the stopped error | the backlog is dropped and `stop` reports the loss | | an object of the same or a later epoch holds the sequence, or any conflicting object in the `enqueued` mode | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned | | encoding or catalog error | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned | | create succeeds at the last representable sequence | cannot advance | acknowledged | poisoned: no later batch can be allocated a sequence, every later append fails | | the start object meets an object of an earlier epoch | moves to the next sequence | none admitted yet | construction continues | | the start object meets an object of the same or a later epoch, or its create has an unknown outcome | unchanged | none admitted yet | construction fails; an unknown outcome is retryable | -A release is a waiter of the batch that carries its record in both modes: it returns when the batch is indexed and fails whenever the batch fails. A store cannot be constructed on a prefix whose largest object already carries the maximum sequence. A poisoned store fails every `LogStore` operation except `stop` with the same terminal error; in particular `obsolete` can no longer move a watermark. In the `durable` mode transient errors are not retried inside the store, but a failed create is reconciled once: the store immediately reads the key back, accepts the batch if the object exists with identical bytes, treats it as a conflict if the bytes differ, and reports the transient error only when the key is absent or that read fails too. So a lost response alone does not fail the append. When the append does fail, neither the store nor Mito keeps the batch: it surfaces through Mito to the caller of the write, and a retry is a new write by the caller under a new sequence. If the create had in fact succeeded and both its response and the read-back were lost, or its request lands only later, the object is an orphan as soon as a later object is acknowledged. Retrying the batch itself inside the store in the `durable` mode would only hide the latency and would hold writes up for as long as an outage lasts; the `enqueued` mode repeats the create with the same bytes, so a create that succeeded without a response is accepted as an identical retry. +A permanent release (`release_region` or `obsolete_all`) is a waiter of the batch that carries its record in both modes: it returns when the batch is indexed and fails whenever the batch fails. A store cannot be constructed on a prefix whose largest object already carries the maximum sequence. A poisoned store fails every `LogStore` operation except `stop` with the same terminal error; in particular `obsolete` can no longer move a watermark. In the `durable` mode a failed data batch is not retried inside the store, but a failed create is reconciled once: the store immediately reads the key back, accepts the batch if the object exists with identical bytes, treats it as a conflict if the bytes differ, and reports the transient error only when the key is absent or that read fails too. So a lost response alone does not fail the append. When the append does fail, neither the store nor Mito keeps the batch: it surfaces through Mito to the caller of the write, and a retry is a new write by the caller under a new sequence. If the create had in fact succeeded and both its response and the read-back were lost, or its request lands only later, the object is an orphan as soon as a later object is acknowledged. Retrying the batch itself inside the store in the `durable` mode would only hide the latency and would hold writes up for as long as an outage lasts; the `enqueued` mode repeats the create with the same bytes, so a create that succeeded without a response is accepted as an identical retry. -**Stopping.** `stop` is idempotent and awaits the actor. Once stop has begun, queued appends are not admitted, no timer or size triggered seal starts, and appends after stop, including empty ones, and releases fail with the stopped error. In the `durable` mode the release records of failed batches and of the batches dropped here are sealed into one more object, which is created before `stop` returns; `stop` reports the error when that create fails too. In the `durable` mode the open batch and every sealed batch whose create has not started are dropped: their waiters receive the stopped error. A create that is already in flight runs to completion; if it succeeds while its batch is still pending, the batch is indexed in order and acknowledged. A transient failure after stop began fails every batch that is not indexed, as it does while the store runs, and those batches receive the stopped error; a create of such a batch that completes later is an orphan and acknowledges nothing. A create that finds a conflicting object of the same or a later epoch after stop began poisons the store as it would while the store runs, but the waiters of the batches that are not indexed receive the stopped error, the conflict is kept only as the terminal error, and `stop` returns successfully once the creates in flight have completed, unless the poisoning drops release records, which `stop` reports by returning the conflict. In the `enqueued` mode the held-back appends receive the stopped error, the open batch is sealed and the whole backlog is uploaded before `stop` returns, by a failing store too; any object store failure in that phase drops the backlog and `stop` returns the error, as described under *Acknowledgement*. A durability wait still pending when the actor exits receives the stopped error, and a delete already in flight runs to completion before `stop` returns. +**Bounded releases on close.** `LogStore::release_entries(provider, region_id, entry_id)` queues an inclusive safe recovery watermark. Mito stops the region first, then computes that watermark from one Version snapshot with `cap_wal_release`: the minimum of `flushed_entry_id` and the `after` bounds of this node's pending tails. Both ordinary close and successful flush-on-close use this path; close-time flush eligibility is unchanged. Other WAL providers default to a no-op. In `enqueued` mode the actor clamps both the in-memory and persisted release watermark to the region's durable entry ID at admission, as `obsolete` does. The sequence floor still uses the requested ID so later appends cannot reuse IDs covered by the request. -**Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so the store persists no watermark of a region Mito opens again. The release of a region, which `obsolete_all` and `release_region` make, is persisted as described under *Entry ids*. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region. +The actor fences later entry IDs above the requested watermark, updates its in-memory obsolete state, and coalesces pending records by region using the maximum watermark. It acknowledges admission without sealing or waiting for an upload. The next sealed batch carries those records alongside ordinary entries; a timer tick can seal a release-only batch. Failed records remain pending for a later create. While the store is failing or its sealed-batch count, unpersisted bytes or age reaches a backlog threshold, bounded releases are still admitted and coalesced, but timer ticks defer release-only sealing. In `durable` mode any rollback that leaves pending release records starts exponential backoff for the next release-only seal, including a failed data batch that carried releases. The first delay is 100 ms; repeated failures double it up to 30 seconds. The failed sequences remain consumed, and the retained records are coalesced into a later batch. Indexing a batch carrying release records resets the delay to 100 ms. Ordinary data batches and explicit permanent releases can carry pending records before the delay expires; shutdown also seals them despite the delay and backlog thresholds. A rollback that retains releases logs its error and the next release-only retry delay on the first failure, then at most once per 60 seconds until a batch carrying releases is indexed. Shutdown processes pending records using the existing acknowledgement-mode shutdown rules. A crash before persistence may lose reclamation progress and retain extra history. The WAL stops before regions close during shutdown. Bounded hints submitted after stopping begins are ignored successfully, without adding records, warnings or failure counts; their progress must be re-established if the region reopens. A genuine admission failure while the store is running does not fail close: it is logged and increments `greptime_mito_wal_release_admission_failure_total`. The persisted-record metric continues to count persistence, not admission. + +Bounded releases reuse the existing release segment and object format. The release segment's footer entry ID is derived from the sequence of the object carrying it, not from the watermarks in its payload. Recovery includes that segment in `history_end`. Thus a later release of a lower-watermark region still carries history evidence past an earlier release that collection removed. Empty start objects do not provide that evidence, and the existing location check is unchanged. Physical deletion continues to use only the durable reclaim boundary. + +**Stopping.** `stop` is idempotent and awaits the actor. Once stop has begun, queued appends are not admitted, no timer or size triggered seal starts, and appends after stop, including empty ones, and permanent releases fail with the stopped error. Later bounded releases are ignored successfully and retain extra history. In the `durable` mode pending bounded releases and the release records of failed batches and of the batches dropped here are sealed into one more object, which is created before `stop` returns; `stop` reports the error when that create fails too. In the `durable` mode the open batch and every sealed batch whose create has not started are dropped: their waiters receive the stopped error. A create that is already in flight runs to completion; if it succeeds while its batch is still pending, the batch is indexed in order and acknowledged. A transient failure after stop began fails every batch that is not indexed, as it does while the store runs, and those batches receive the stopped error; a create of such a batch that completes later is an orphan and acknowledges nothing. A create that finds a conflicting object of the same or a later epoch after stop began poisons the store as it would while the store runs, but the waiters of the batches that are not indexed receive the stopped error, the conflict is kept only as the terminal error, and `stop` returns successfully once the creates in flight have completed, unless the poisoning drops release records, which `stop` reports by returning the conflict. In the `enqueued` mode the held-back appends receive the stopped error, the open batch is sealed and the whole backlog is uploaded before `stop` returns, by a failing store too; any object store failure in that phase drops the backlog and `stop` returns the error, as described under *Acknowledgement*. A durability wait still pending when the actor exits receives the stopped error, and a delete already in flight runs to completion before `stop` returns. + +**Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so the store persists no ordinary flush or truncate watermark, but does persist the safe watermark queued when a region closes. The release of a region, which `obsolete_all` and `release_region` make, is persisted as described under *Entry ids*. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region. **History check.** A region persists its prefix, not the object store that holds the prefix, so pointing `storage_provider`, or the bucket or root of the provider it names, at another location, or deleting the objects under the prefix, would let every region open against a prefix without its history, replay nothing and lose its acknowledged writes that were not flushed. Two invariants of a prefix expose such a location without any persisted state of their own. A prefix that a store opened is never empty afterwards, unless the store withdrew its start object as described below: the store writes its start object before it admits an append, and collection deletes only objects below the durable boundary, the boundary of the last indexed object, which is at or below that object's own sequence, since a larger one fails the header check. And a watermark a region persisted names an object at or below the newest object holding entries that any later recovery of the prefix indexes: the entry it names was durable and indexed when the watermark was persisted, as described under *Acknowledgement*; no durable boundary passes the newest object holding entries, since a batch carries at most the sequence after the last indexed object and a start object carries the boundary recovery found, so collection keeps that object and recovery indexes it on the chain of the tip. The segment of release records counts as entries: it never holds the reclaim boundary, so an object holding only release records can be the one that collection keeps while it deletes every older object. Objects without segments, start objects and the like, do not move it. `LogStore::check_region_history(provider, entry_id)` therefore reports the history of the provider's region as missing when recovery found no object under the prefix, or when `entry_id` names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through `entry_id`, as it does for a region written since the store opened, or the region's history starts in the store: Mito calls `LogStore::start_region_history` when it creates a region, before it persists the region's manifest, and an open that accepted a missing history records the region the same way, in memory, so the region reopens on the same store without any entry and is counted once. Both methods default to accepting and to doing nothing, so Raft Engine and Kafka are unchanged. Mito calls it before it replays an existing region from its persisted state, on open, on catch-up and when a create finds the region already present, with the larger of the manifest's `flushed_entry_id` and `truncated_entry_id`; a region whose options skip the WAL keeps no history there and is not checked, nor is a region with the noop provider. A missing history fails the open, and such a create, which does not fall back to creating the region over its manifest, with an `InvalidArguments` error that names the region, the prefix, the configured storage provider with its scheme, bucket and root, and the two ways forward below; in standalone a region that fails to open stops startup. The check runs before Mito calls `obsolete` with the watermark, so the sequence floor never hides a missing history. @@ -180,7 +186,7 @@ Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provid The scalar options are validated before any store or engine exists: the prefix must be non-empty, relative and free of empty or `..` components; `flush_interval` at least 10 ms; `max_batch_bytes`, `max_unpersisted_bytes` and `max_unpersisted_age` positive. `storage_provider` is resolved later, once the object store manager has been built and before the log store and the Mito engine are: empty selects the default store, otherwise the name must match a configured provider. Failures in both stages are `InvalidArguments` naming the field, and both happen before any WAL object is touched. The datanode requires a node id to derive the node prefix it opens the store under, and the standalone bootstrap derives the same prefix for the WAL options it allocates. -`DatanodeBuilder` keeps the store handle from the moment it is created and stops it if a later build step fails. On success the handle moves into `Datanode`, whose shutdown stops the store after the region server. Shutdown of the datanode and of the standalone instance is best-effort and ordered: every step runs, the first error is returned, so a failing region engine does not leave the WAL actor alive. +`DatanodeBuilder` keeps the store handle from the moment it is created and stops it if a later build step fails. On success the handle moves into `Datanode`, whose shutdown stops the store before the region server, so pending WAL durability waits settle before regions close. Shutdown of the datanode and of the standalone instance is best-effort and ordered: every step runs, the first error is returned, so a failing region engine does not leave the WAL actor alive. ## Configuration @@ -340,9 +346,9 @@ Simpler catalog, but every region then produces its own timer-driven object, so - Write latency under the `durable` mode is the time until the batch seals (up to `flush_interval`, 100 ms by default), plus waiting for every earlier object, plus one PUT. A serial writer waits for the next seal, so its extra latency over Raft Engine is the interval minus the base insert cost modulo the interval, and concurrency does not close the gap: measured within one session on a laptop it was about 23% of throughput at concurrency 1 and about 28% at concurrency 8, and on S3 against Raft Engine on a local NVMe store the `durable` mode reached 31% of Raft Engine's throughput at concurrency 1 and 54% at concurrency 8, a serial writer being bound to one batch per interval, while the `enqueued` mode matched it, see *Write path measurements* and *Cloud write path measurements*. - Every object costs a request. Timer-triggered sealing creates at most one object per interval per node, ten per second at the default; the `enqueued` mode can seal earlier when the unpersisted backlog reaches a threshold, and the batch seals on size only when the admitted bytes reach `max_batch_bytes` before the next tick. Under sustained load on S3 the timer ceiling was reached, about 600 creates per minute per node, about $0.18 per node-hour at the S3 standard list price. -- Garbage collection keeps every object at or above the reclaim boundary, so a region that is written but rarely flushed, or never flushed, holds the boundary at its first object and keeps every later object the prefix shares with other regions; recovery time grows with the number of retained objects. A prefix that receives no appends cannot persist a higher boundary, so an idle prefix is collected only after its next object; a release seals that object itself. +- Garbage collection keeps every object at or above the reclaim boundary, so a region that is written but rarely flushed, or never flushed, holds the boundary at its first object and keeps every later object the prefix shares with other regions; recovery time grows with the number of retained objects. An idle prefix advances its durable boundary when another object is sealed. A permanent release seals immediately; a pending bounded release lets a timer tick seal a release-only object even without appends. - A dropped region stays unreleased, and holds the reclaim boundary for good, after a crash between the deletion of its table's metadata and a durable release record, or after a crash that follows a failed release before any later create succeeds; `indexed_objects` then only grows. The first window is a gap of the drop procedure for every WAL provider, issue #9577. See *Entry ids*. -- A release costs one object of its own and waits for every object sealed before it, in the `enqueued` mode as well, so dropping a region waits for the uploads ahead of its release record; when they fail, or the store is stopped or poisoned, the drop succeeds all the same and logs the failure; a stopped or poisoned store leaves the region unreleased. +- A permanent release costs one object of its own and waits for every object sealed before it, in the `enqueued` mode as well, so dropping a region waits for the uploads ahead of its release record; when they fail, or the store is stopped or poisoned, the drop succeeds all the same and logs the failure; a stopped or poisoned store leaves the region unreleased. - Every open of the store writes one start object without segments, one extra PUT per open, and every object carries a 54-byte header. Orphans, the objects of failed batches that were stored anyway, hold storage together with the rows Mito wrote again after those failures until garbage collection deletes them. - Object-sequence-major entry ids consume 20 bits per object for positions, which caps a region at about a million entries per object; the batch seals when the cap is reached. Sequence numbers are 44 bits wide as a result, enough for one object per second for half a million years. diff --git a/docs/rfcs/2026-11-10-object-store-metadata-leader-election.md b/docs/rfcs/2026-11-10-object-store-metadata-leader-election.md new file mode 100644 index 00000000000..8d55201c49f --- /dev/null +++ b/docs/rfcs/2026-11-10-object-store-metadata-leader-election.md @@ -0,0 +1,547 @@ +--- +Feature Name: Object Store Metadata Backend — Authority and Leader Election +Tracking Issue: TBD +Date: 2026-11-10 +Author: Alorun +Status: Draft for discussion +--- + +# Summary +Add an object-store-backed authority and leader election mechanism to GreptimeDB's Metasrv. Strongly consistent single-object reads and atomic conditional replacements maintain one fixed **Authority Head**, stored as versioned JSON in `head.json`. Its contents record the elected owner, logical epoch, diagnostic lease deadline, and write sequence. OpenDAL `Metadata` from the same object version supplies the condition for `if_not_changed(&metadata)`. Every metadata write that changes authoritative state must pass through this Head. The Head has three responsibilities: + +- It identifies the elected owner and supplies a routing hint; the owner may already have stepped down locally. +- It carries the fencing epoch that invalidates an old Leader. +- It provides the single serialization point for later metadata commits. + +This RFC defines the Authority and Leader Election contract for the first phase. A separate batching RFC will define metadata commit semantics; it will be added later. Storage layout and read semantics will be specified separately and will reuse the Head defined here. + +# Motivation +Metasrv currently uses etcd, MySQL, or PostgreSQL as the metadata authority and leader-election backend. The data path already relies on object storage, so a bucket-backed deployment still needs an independently operated metadata service. Replacing the metadata service with object storage is not a simple key-value translation. The current metadata workflow depends on several properties: + +- A write must be serialized with other related writes. +- A Leader that has lost its term must not commit later. +- A failed request must not be reported as rejected while its outcome is unknown. +- A new Leader must be able to take over after the previous Leader fails. + +The central design choice is to make the Leader the single writer of authoritative metadata. Conditional object versions reject stale updates, and the epoch fences stale terms. With this backend, every authoritative metadata write is routed through the Leader, so failover pauses metadata writes. Renewal costs and failover estimates are below; the full storage requirements are in *Object-store semantics*. + +# Goals + +- Elect exactly one effective Leader among concurrently starting nodes. +- Allow a surviving node to take over after the previous Leader expires. +- Prevent an old Leader from modifying the Head after a takeover. +- Anchor leases to the store's modification timestamp, with a bounded candidate clock offset. +- Handle uncertain conditional writes without blindly retrying them. +- Serialize Head mutations on each node so renewal and commit do not race locally. +- Preserve the existing Metasrv `Election` trait where possible. +- Provide a stable authority interface for the later phase-one RFCs. +- Reuse the existing OpenDAL object-store abstraction without extending OpenDAL. +- Make the design testable against MinIO and the cloud object stores in scope. + +# Non-goals + +- The batching and immutable batch format. +- Transaction commit semantics and commit-tail comparisons. +- B-link Tree nodes, manifests, splitting, and the read path. +- Checkpointing, garbage collection, and materialization. +- Quorum replication and follower-local consistent reads. +- A migration path from etcd, MySQL, or PostgreSQL. +- Running the new backend and an existing backend at the same time. +- Changing the persisted format of existing metadata keys. +- Supporting standalone Metasrv through this election mechanism. + +Keeping a KV service would retain the independent service dependency, while a routing-only Head would still need another component for fencing and commit serialization. The WAL-style alternative advances history by conditionally creating monotonically named immutable objects and needs rules for discovering the latest valid record and handling concurrent publication. This RFC renews a fixed Head so `owner`, `epoch`, and store-side `last_modified` can be read from one version, requiring atomic replacement conditioned on the version read. + +# Phase boundary +The first phase is divided into three related RFCs: + +| RFC | Responsibility | Relation to the Head | +| --- | --- | --- | +| Authority and election | Leader, epoch, lease, fencing | Defines and owns the Head authority | +| Batching and immutable batch | Batch construction and commit | Publishes committed batches through the Head | +| B-link Tree layout | Nodes, manifests, and reads | Publishes checkpoint roots through the Head | + +This RFC defines the shared authority contract. It does not implement the `KvBackend` write methods because their visibility semantics belong to the future batching RFC. Later RFCs may extend the Head payload with planned fields such as: + +- `committed_sequence` +- `last_batch` +- `checkpoint_root` + +The election fields and their meanings are fixed by this RFC. + +# Renewal costs and failover estimates + +Renewal shares the serialized Head writer with later metadata commits. Each renewal requires a conditional replacement and enough time to obtain coherent response metadata or resolve an uncertain result. Renewal priority and request latency must leave sufficient time within the local lease; commit throughput and batching requirements belong to the future batching RFC. + +Failover time includes the remaining store-side lease, the clock guard, the campaign interval with jitter, and the storage requests needed to read and replace the Head. A ten-second lease alone is not a ten-second failover bound. Storage outages or a delayed write that advances `last_modified` can delay takeover further. + +# Design overview +The design separates three questions: + +- **Who may write?** The current epoch, Head ownership, active local Leader state, and local deadline decide this. +- **When is a write visible?** The Head commit performed by the batching layer decides this. +- **Where does data live?** Immutable batches and B-link Tree objects decide this. + +The Authority Head is rewritten in place. Every mutation checks a confirmed Head state and passes its corresponding OpenDAL `Metadata` to `if_not_changed`. +The high-level sequence is: + +1. An operator explicitly initializes an empty Head in a new namespace. +2. Nodes register themselves as candidates. +3. Candidates read the Head and campaign when the lease is absent or expired. +4. Exactly one candidate conditionally replaces the previously read Head version. +5. The winner becomes Leader and receives the new epoch as its fencing token. +6. The Leader renews the Head through the same serialized writer used by commits. +7. Renewal failure follows the local retirement transition in *Renewal*; uncertain results follow *Outcomes*. +8. Candidates continue campaigning under the takeover rules. + +The election decision is made against the Head state. Candidate listing is never used to decide who may write. + +# Authority Head + +## Object path +The Head uses a fixed object path: + +```text +//authority/head.json +``` + +`root` comes from the selected storage configuration. When clusters share a root, each must explicitly set a non-empty, unique, and stable `store_key_prefix`. An empty prefix is permitted only when the cluster exclusively owns that root. If two clusters resolve to the same Head path, either may take over the other's Head after lease expiry, and later commits may enter the same chain. First-phase isolation relies on deployment owners assigning paths correctly, using the existing configuration without a separate cluster identifier or Authority prefix option. + +Paths in this RFC include the storage root for clarity; calls to the opened operator use relative paths and do not add the root again. The Head path is reserved for Authority updates. + +## Payload +The first phase fixes the following logical payload: + +```rust +struct AuthorityHead { + format_version: u32, + owner: Option, + epoch: u64, + lease_deadline_ms: u64, + write_seq: u64, +} + +struct NodeIdentity { + addr: String, + incarnation_id: Uuid, +} +``` + +The initial Head has `owner = null`, `epoch = 0`, and `write_seq = 0`: + +```json +{ + "format_version": 1, + "owner": null, + "epoch": 0, + "lease_deadline_ms": 0, + "write_seq": 0 +} +``` + +A version-1 Head after takeover contains a non-empty owner: + +```json +{ + "format_version": 1, + "owner": { + "addr": "127.0.0.1:3002", + "incarnation_id": "550e8400-e29b-41d4-a716-446655440000" + }, + "epoch": 1, + "lease_deadline_ms": 1794528010000, + "write_seq": 1 +} +``` + +`addr` is the Metasrv gRPC address. `incarnation_id` is a new UUID generated once at each process start, stable for that process's lifetime, and serialized as a canonical hyphenated UUID string. Ownership checks compare both fields, so restarting at the same address does not inherit a term. `lease_deadline_ms` is diagnostic only. Every successful Head mutation increments `write_seq`; both `epoch` and `write_seq` use checked increments and reject overflow without writing. + +## Encoding +Serialize the Head to `head.json` with `serde_json` and top-level `format_version: 1`; v1 readers ignore unknown fields in both Head and nested `NodeIdentity`, while validating required fields and types (`owner` must be present even when `null`). Only optional fields that old readers can ignore and old writers can discard on rewrite without affecting correctness may be added to v1. Adding required fields or changing ownership, lease, fencing, commit visibility, or other protocol semantics requires a higher `format_version`; malformed JSON and unsupported versions are rejected. + +# Object-store semantics + +## Object-store capabilities + +Admission requires strong single-object read-after-write consistency and atomic replacement conditioned on the version that was read. Head contents and the complete OpenDAL `Metadata`, including the condition's version identifier and `last_modified`, must describe the same object version. Separate metadata and content requests must be conditionally bound or retried to obtain a coherent state. Reads must not regress to superseded versions. Listing consistency is not required for election or commit decisions. + +| Capability | Purpose | +| --- | --- | +| Strong single-object read | Read the Authority Head and resolve uncertain writes | +| `if_not_exists(true)` | Initialize the Authority Head | +| `if_not_changed(&metadata)` backed by atomic replacement | Takeover, renewal, resignation, and later commits | +| Backend condition identifier in `Metadata` | Identify the version used for replacement and read-back comparison | +| `last_modified` | Determine lease expiration | +| Unconditional write | Register candidate nodes | +| Object listing | Discover candidate nodes | +| Object deletion | Clean up temporary probes; candidate cleanup additionally requires a safe conditional delete | + +First-phase qualification covers S3-compatible backends, including MinIO, Azure Blob, and GCS. Each actual endpoint must pass the startup probe through the production Head read/write path and configured OpenDAL layers. S3 and Azure use ETag conditions; GCS uses the generation exposed as `Metadata::version()`. GCS must pass real-endpoint generation-path tests before it is enabled; that validation has not been performed for this RFC revision. + +Alibaba OSS server-side `PutObject` rejects `If-Match` and `If-None-Match` with `NotImplemented` (`0017-00000245`); see [Alibaba Cloud's documentation](https://www.alibabacloud.com/help/en/oss/user-guide/0017-00000245). OSS is therefore not supported by the current Head protocol, which requires atomic replacement conditioned on the version read. The local filesystem and other backends without the same atomic condition, coherent metadata, or strong-read guarantees are also excluded; declared flags or read-then-unconditional-write emulation do not satisfy admission. + +The current OpenDAL write interfaces for HDFS and MySQL also lack the required conditional replacement; the MySQL restriction applies to the adapter. HDFS `append` only appends bytes and cannot replace this version-checked Head update: using it would require a separate election protocol, and append support varies by backend. + +GreptimeDB already provides object-store access through OpenDAL. The Metasrv adapter owns JSON encoding, coherent Head reads, conditional-write handling, election, fencing, candidate registration, and capability validation. The functional probe verifies the endpoint and layers against the same admission contract. + +## Deployment requirements + +The bucket is the storage container holding the Head object; on Azure Blob the corresponding resource is a container. The epoch is a logical term stored inside the Head, independent of bucket configuration or object-version IDs. All nodes must use the same selected store, root, and `store_key_prefix`. + +The Authority prefix must be excluded from lifecycle expiration and storage-class transition rules. Administrative deletion, rollback to an older version, or replacement outside this protocol can destroy the authority history and requires operator recovery. + +In a versioned S3 bucket, `If-None-Match: *` can succeed when the current version is a delete marker, even if older Head versions exist. A successful conditional create therefore does not prove that the namespace has never been initialized. See the [S3 conditional-write behavior](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-write-behavior). Explicit initialization is restricted to a namespace the operator has established is new. + +## Conditional writes + +The design uses two conditional write forms: + +| Operation | Condition | Purpose | +| --- | --- | --- | +| Create if absent | `write_with(...).if_not_exists(true)` | Initialize the Head and immutable objects | +| Replace if unchanged | `write_with(...).if_not_changed(&metadata)` | Takeover, renewal, resignation, and later commits | + +Use the `Metadata` paired with the decision's Head contents for every takeover, renewal, and resignation. OpenDAL lowers `if_not_changed` to a supported native version-match condition, or an ETag match when that is the supported primitive. The condition must be atomic with publication, including final multipart or staged-upload completion. A `stat` followed by an unconditional write is not acceptable. Operators or layers that ignore conditions are rejected, and transparent retries must not replay Head mutations. Storage branches use OpenDAL `ErrorKind`. + +## Object version + +Preserve the complete OpenDAL `Metadata` associated with the observed Head path and contents. The identifier selected for conditional writes is opaque: compare it for equality during read-back, without interpreting or ordering it. In OpenDAL 0.59.4, S3 and Azure select `Metadata::etag()`, while GCS selects the object generation in `Metadata::version()`. These physical identifiers differ from the logical epoch; renewal keeps the epoch while changing the conditional version. + +For S3, retain ETags exactly as returned, including surrounding quotes, when OpenDAL passes them to `If-Match`. Writing identical contents may reuse an ETag, so incrementing `write_seq` must change the Head contents on every mutation. Bucket version IDs are not substituted for the ETag condition. Versioned-bucket delete-marker behavior is covered in *Deployment requirements*. + +## Store timestamp and lease + +Candidates compare their wall clocks with `last_modified` from the Head's OpenDAL `Metadata`, using the complete rule in *Expiration test*. Deployments must keep candidate-to-store clock offsets within the stated bound. The Leader uses a monotonic clock for its local deadline, as specified in *Renewal*. The diagnostic `lease_deadline_ms` written by a node is never a lease authority. + +## Capability probe + +After opening the selected storage and applying its configured layers, and before campaigning, check admission status and the declared primitives: `write_with_if_not_exists` plus `write_with_if_version_match` or `write_with_if_match` with a usable metadata identifier. `if_not_changed` has no independent capability flag. The provider's configuration name is not its backend type. Then issue real requests against a unique temporary key under the same `probes/` naming convention as the object-store WAL: + +```text +//probes/.json +``` + +1. Create distinct contents with `if_not_exists(true)`. Read them through the production Head read path; require coherent contents, a usable conditional version identifier, and non-empty `last_modified` in the returned `Metadata`. +2. Repeat the conditional create on the same key with different contents. Accept `ConditionNotMatch` or `AlreadyExists`, then read back and verify the original contents and conditional identifier are unchanged. +3. Replace it with different contents using `if_not_changed(¤t_metadata)`. Require success and read back coherent new contents and metadata, with a changed conditional identifier and available `last_modified`. +4. Attempt another replacement with the old metadata. Require `ConditionNotMatch`, then read back and verify the latest contents and identifier are unchanged. + +Probe payloads use the version-1 Head codec, with distinct contents produced through `write_seq`. The probe uses the production Head read/write path and OpenDAL layers; for GCS it must exercise generation-based conditions. Ignored conditions, missing capabilities, incoherent or incomplete metadata, or an unresolved result fail startup. Also require unconditional writes, listing, and probe deletion. Clean up the temporary object on both success and failure, and report cleanup failures. Probe keys are isolated from `authority/` and candidate records; a probe must never create or modify the Head. Flags alone cannot prove these semantics. + +# Initialization +Head creation is an explicit operator operation for a new namespace. Add a planned Metasrv subcommand following the existing `greptime metasrv --config-file` CLI style: + +```sh +greptime metasrv init --config-file metasrv.toml +``` + +This proposed command opens the configured store, runs the isolated capability probe, and creates the version-1 empty Head with `if_not_exists(true)`. It does not campaign. Concurrent initializers that receive `ConditionNotMatch` or `AlreadyExists` read and validate the existing Head and report it as already initialized; they never reset its epoch or contents. Other conflicts require read-back and re-evaluation. An uncertain create is read back without blindly resending it. Creation and takeover remain separate operations: + +| Property | Initialization | Takeover | +| --- | --- | --- | +| Owner | Empty | Current candidate | +| Epoch | `0` | Previous epoch plus one | +| Condition | Object absent | Metadata from the previously read Head version | +| Lease | Not established | Established by this write | + +The initial empty Head has no lease. Normal `greptime metasrv start --config-file metasrv.toml` requires an existing, valid Head. If the Head is missing at startup or runtime, stop election and writes and report an explicit error naming its path: `Authority Head is missing; operator recovery is required`. Do not invoke initialization as a fallback. + +Recovery requires fencing or stopping existing writers, checking the configured store and namespace, and inspecting retained object versions or backups for the last valid Head. Restore the authority history and its referenced metadata from a consistent recovery point before restarting nodes. A missing current object, including an S3 delete marker, is insufficient evidence of a new namespace. If history cannot be recovered, any destructive reset is a separate recovery decision outside this initialization command. + +# Leader election + +## Takeover +A candidate repeatedly attempts takeover: + +```text +read Head +if owner is empty or lease is expired: + write owner = candidate + write epoch = old_epoch + 1 + write_seq = old_write_seq + 1 + replace using if_not_changed(&metadata) from that Head read +else: + wait with jitter +``` + +The lease check and the conditional replacement belong to one authority operation. If two candidates use metadata from the same Head version, only one replacement succeeds, and the epoch increases exactly once. The loser rereads and re-evaluates. Candidate listing is never a vote or quorum. A takeover increments the epoch even after resignation; epoch zero represents an initialized Head with no election term. + +The attempt has a provisional monotonic deadline anchored to its send time plus the lease duration. It grants leadership only when its result is confirmed before that deadline. Uncertain takeover and recovery from a retired term follow *Outcomes*; a late result cannot activate an expired attempt. + +## Expiration test +A Head with no owner can be taken over immediately. Otherwise, a candidate considers its lease expired only when: + +```text +wall_now > last_modified + lease_duration + clock_guard +clock_guard = max_clock_skew + timestamp_resolution +``` + +`last_modified` comes from the same `Metadata` as the Head version used for the decision. `wall_now` is the candidate's local wall clock in the store timestamp's time domain. The initial internal allowances are one second for the maximum candidate-to-store clock offset and one second for timestamp resolution. Deployments must keep clock offsets within that bound; the lease timing guarantee does not hold outside it. The resolution allowance must cover the supported backend's timestamp precision, verified during implementation. + +This expiration rule determines when takeover is permitted. Routing uses a separate, earlier cutoff defined in *Existing Election integration*. CAS arbitrates competing updates to the same Head version. If renewal replaces that version first, the candidate must reread and repeat the expiration check. If takeover wins, the old Leader's conditional update fails. CAS does not replace the clock assumptions or validate a candidate's expiration calculation. + +## Renewal +Use the independent object-store election constant `OBJECT_STORE_META_LEASE_SECS: u64 = 10`, allowing time for storage requests and result read-back without changing existing KV election leases. Default renewal runs at half this lease, every five seconds, through the shared Head writer. Renewal keeps `owner` and `epoch` unchanged and increments `write_seq`. A successful PUT refreshes the store-side `last_modified`. Sample the monotonic clock when sending the request and calculate the proposed deadline: + +```text +proposed_deadline = sent_at + lease_duration +``` + +While the request is pending, retain the existing local deadline. Install the proposed deadline only on a successful renewal response received before that deadline, while local leadership remains active and this node still owns the term. Anchoring at send time avoids counting response latency as additional lease time. + +On the first renewal failure or unknown result, immediately clear the local Leader flag, stop admitting new Leader writes, retire the local term, and emit one `StepDown`. Existing event handlers close heartbeat streams and stop or switch background tasks. Read-back may continue within the original deadline solely to determine whether the PUT applied. Late confirmation and service recovery follow *Outcomes*. + +Reaching the local deadline also retires an active term; repeated failures or deadline expiry after an earlier `StepDown` do not emit it again. The future batching RFC will define whether later batch commits also count as local lease renewal. + +## Resignation +A Leader stops admitting new writes when it begins resignation, clears `owner`, retains the epoch, and increments `write_seq` through a fenced Head update. Confirmed resignation ends the local term. An uncertain result follows the same read-back and deadline rules as other mutations. The next takeover increments the epoch and starts a new term. + +# Fencing and comparison + +## Fencing token +A Leader receives the current epoch as its fencing token: + +```rust +struct FencingToken { + epoch: u64, +} +``` + +The token carries the logical term. The conditional version identifier in `Metadata` can change during renewal without changing the term or a queued commit's business state. Later commit callers also supply a business precondition, such as the committed tail against which a batch was prepared. + +## Version and term checks + +Every successful Head mutation changes its contents and conditional version. Takeover also changes the owner and epoch, invalidating the old writer's metadata condition and logical token. Before a fenced update: + +1. Obtain coherent Head contents and their complete `Metadata`, by reading them or using the writer's latest confirmed state. +2. Check that the token epoch equals the Head epoch. +3. Check that the owner is this node's process identity, local Leader state is active, the term has not been retired, and the local deadline has not passed. +4. Check the caller's business precondition. +5. Apply the mutation with `if_not_changed(&state.metadata)` for that checked state. + +A failed condition requires Head read-back and re-evaluation of the term, owner, deadline, and applicable preconditions, or recovery if the Head is missing, as specified in *Error handling*. An old-epoch operation is rejected even with fresh metadata. A stale metadata condition is rejected by the store even if the caller has not observed the new epoch. All authoritative publication passes through these checks. + +# Head write serialization +Each node uses one writer to serialize all Head updates. Renewal may take priority over queued operations so a busy queue does not starve the lease. Subsequent operations recheck the confirmed Head and metadata, local leadership, deadline, and business preconditions. Renewal alone does not require rebuilding a prepared batch; a moved commit tail does. An unknown outcome pauses subsequent updates. Renewal failures follow *Renewal*, and queued operations for a retired term remain rejected under *Outcomes*. Entry points are defined in *Authority interface*. + +# Uncertain writes + +## Problem +A conditional write can apply before its response is lost, or can remain in flight and apply after the client has timed out. A client timeout does not cancel server-side execution. The protocol preserves the distinction between a Head mutation's outcome, current ownership, and a historical client transaction's outcome. + +## Read-back procedure +After a timeout or a write error that may follow persistence, pause new Head updates and apply the transition in *Renewal* when appropriate. Read the Head with backoff within the original local deadline solely to resolve the PUT outcome; the unresolved request does not extend that deadline. A takeover attempt uses its provisional deadline. Never blindly resend the original write. + +Compare the read-back identifier selected by the backend for conditional writes with the pre-write identifier, then confirm application only from evidence unique to this operation. Check the complete expected state, including epoch, `write_seq`, and process identity where applicable. An owner address, diagnostic deadline, or empty owner alone is insufficient. A later Head may no longer contain enough evidence to establish an overwritten mutation's history. + +## Outcomes + +| Read-back result | Interpretation | +| --- | --- | +| Conditional version identifier unchanged, before or after client timeout | Still unknown; the original request may apply later | +| Evidence uniquely identifies this mutation | Confirmed applied; service recovery is handled separately below | +| Higher epoch or changed owner | Immediately abandon the current term; the original outcome remains unknown without independent evidence | +| Conditional version identifier changed without unique evidence | Still unknown; the mutation may have applied and been superseded | +| Read-back fails | Retry read-back with backoff within the original deadline | +| Original local deadline reached before confirmation | Retire the term if still active, without duplicating `StepDown`, and preserve the operation result as unknown | + +A late success response or successful read-back can establish only that the PUT applied. It cannot extend or reactivate a retired term, even before the original deadline, and queued writes for that term remain rejected. Restoring Leader services and background tasks requires winning a higher epoch through normal takeover and emitting `Elected`. If a late PUT refreshes `last_modified`, candidates use that new timestamp in the unchanged expiration rule. + +Once a legal takeover replaces the observed version, a delayed request with the old metadata condition cannot modify the Head. This fences future execution without proving whether it applied earlier. Historical commit outcomes need independent evidence, such as operation IDs and durable records in the published batch chain, to be defined by the future batching RFC. + +# Candidate registration +Candidate registration is separate from the Authority Head. Each node owns one object: + +```text +//candidates/.json +``` + +The JSON record contains the existing `MetasrvNodeInfo` and a unique refresh identifier. The existing candidate methods and their return types remain unchanged; callers receive `MetasrvNodeInfo`. A node unconditionally overwrites its own path so a restart at the same address can replace the previous record. Refresh runs at an internal interval and retries temporary errors with backoff until stopped. Each refresh attempt uses a fresh identifier, ensuring distinguishable contents and versions even when the node information is unchanged. + +Freshness is judged from the store's `last_modified` and the internal candidate TTL. `all_candidates()` lists records and filters stale entries. Candidate records serve display and peer discovery only; they never decide takeover, epoch assignment, fencing, or commit ordering. Duplicate addresses remain a configuration error; process identities must remain distinct for Authority ownership. + +Periodically clean up stale candidate records using an atomic conditional delete bound to the same version whose timestamp was checked. A refresh after the scan must cause the stale delete to fail and preserve the refreshed record. An unconditional delete after a freshness check is unsafe. On S3, identical-content refreshes could reuse an ETag, which is why refresh identifiers must change. Verify each backend's conditional-delete semantics, including versioned buckets, during implementation. Where safe deletion is unavailable, retain filtering and defer deletion to offline cleanup with refreshers stopped. + +# Authority interface +The authority exposes a small interface to the election and later metadata layers: + +```rust +trait Authority { + async fn head(&self) -> Result; + async fn takeover(&self) -> Result; + async fn mutate( + &self, + token: &FencingToken, + precondition: Precondition, + mutation: Mutation, + ) -> Result; +} +``` + +All update entry points use the same local writer: + +| Entry point | Requirement | Effect | +| --- | --- | --- | +| `takeover()` | Empty owner or expired lease | Establish ownership and increment epoch and write sequence | +| `mutate(..., Mutation::Renew)` | Current token, owner, active local leadership, and local deadline | Refresh the lease and increment write sequence | +| `mutate(..., Mutation::Resign)` | Current token, owner, active local leadership, and local deadline | Clear owner and increment write sequence | + +Takeover creates the first token of a term and is not a `Mutation` variant. Later RFCs may add `Commit` with a commit-tail precondition and `PublishCheckpoint` through the same writer. + +`AuthorityState` contains: + +```rust +struct AuthorityState { + head: AuthorityHead, + metadata: opendal::Metadata, +} +``` + +`head()` returns contents and complete metadata from one coherent object version. Every update entry point uses that metadata with `if_not_changed`; lease formulas use its `last_modified`. An active writer may use its latest confirmed state for the next conditional update. Result resolution follows *Outcomes*. Followers and leader discovery read the Head from the store. + +# Existing Election integration +Keep the existing `Election` trait and routing flow. `campaign()` maintains the term using *Renewal*, and result resolution and service recovery follow *Outcomes*. `is_leader()` is local state that can lag the store. Request admission checks it, the active term, and the local deadline; Head updates also enforce fencing and preconditions. + +Confirmed ownership loss, reset, and resignation retire an active term through the same existing event handlers. Remember retired terms locally even while the Head still names this node. `Elected` and `StepDown` occur only on actual transitions, and the infancy flag remains one-shot. Standalone Metasrv does not use this backend. + +## Routing hints + +`leader()` reads the Head and returns an owner address that the caller may try to contact. `LeaderValue` continues to carry the gRPC address. Return `ElectionNoLeader` when the owner is empty, the routing deadline has been reached, or this node knows the observed term has retired. The caller stops routing the current write request on this error. + +Followers use a separate, conservative wall-clock routing deadline: + +```text +routing_guard = max_clock_skew + timestamp_resolution +route_deadline = last_modified + lease_duration - routing_guard +return a hint only if wall_now < route_deadline +``` + +The clock and precision allowances shorten the hint's lifetime; takeover still adds `clock_guard` under *Expiration test*. Sample `wall_now` after reading the Head, so read latency consumes the routing window rather than extending it from response receipt. With the proposed ten-second lease and two-second allowances, a Head last modified at second zero stops yielding hints at second eight, while takeover is permitted only after second twelve. A lookup at second eleven returns `ElectionNoLeader` while candidates still wait. + +A delayed PUT may place `last_modified` later than the owner's local send time. An owner may also step down early without changing the Head. Followers that do not know this may return its address within the routing window, and forwarding delay can further age the hint. The receiving node must check active local leadership, the current term, and its local deadline and reject writes for a retired or expired term. The first phase uses Head-based hints without an owner-confirmation RPC in `leader()`; an address is not proof that the receiver will accept a write. + +# Configuration +Add `BackendImpl::ExperimentalObjectStore`, serialized as `experimental_object_store`, to the existing Metasrv backend selection. Extend Metasrv options with the shared storage configuration and `storage_provider`, following the object-store WAL's provider resolution: + +```toml +backend = "experimental_object_store" +store_key_prefix = "cluster-a" +# Empty selects the default store configured in [storage]. +storage_provider = "" + +[storage] +type = "S3" +bucket = "example-metasrv-bucket" +root = "metasrv" +``` + +An empty `storage_provider` selects the default `[storage]` store. A non-empty value is matched case-insensitively to the configured names in `[storage]` and `[[storage.providers]]`, using the same rule as `wal_object_store` in `src/datanode/src/datanode.rs` and `ObjectStoreManager::find`. Unknown names fail configuration validation. Validate the resolved backend type and operator capabilities, rather than comparing the provider name with `s3` or `azblob`. + +For this example, the Head is `metasrv/cluster-a/authority/head.json`, candidates are under `metasrv/cluster-a/candidates/`, and probes are under `metasrv/cluster-a/probes/`. `store_key_prefix` is the only cluster namespace setting; its shared-root isolation rules are defined in *Object path*. + +Election timings follow the repository's internal-constant convention. `OBJECT_STORE_META_LEASE_SECS` and its half-lease renewal interval are defined in *Renewal*, independently of existing KV election timing. Campaigning uses a 500 ms interval with jitter. Clock allowances remain in *Expiration test*. Candidate TTL and refresh reuse the existing 600-second lease and half-lease refresh interval. These are internal constants, not additional configuration options. + +# Error handling +Storage branches use OpenDAL `ErrorKind` and its retry metadata. Renewal failures first follow *Renewal*; unknown results and service recovery follow *Outcomes*: + +| OpenDAL result | Action | +| --- | --- | +| `ConditionNotMatch` | Read the Head; if present, recheck the term, owner, deadline, and operation preconditions against the new state; if missing, stop election and writes and enter operator recovery | +| `Conflict` | Read back and re-evaluate contention; retryability alone does not justify replaying the mutation | +| `NotFound` on a Head read | Stop and enter operator recovery; never initialize automatically | +| `Unexpected` or a temporary write error where persistence is uncertain | Pause updates and use the uncertain-write procedure | +| Temporary read-back failure | Retry the read with backoff within the original local deadline | +| `Unsupported`, `ConfigInvalid`, or `PermissionDenied` | Report the capability, configuration, or permission failure; fail startup or the operation | + +A client timeout follows the uncertain-write procedure; it is not a separate OpenDAL error kind. A missing target during conditional replacement is a failed condition, not proof that a new Head should be created: only the follow-up read establishes absence. Business-precondition failures return the corresponding conflict for the caller to rebuild and re-evaluate. Preserve unresolved operations as unknown at the API boundary, rather than inviting duplicate execution through an ordinary rejection. + +# Metrics and observability +Existing backend-labelled Metasrv metrics cover ordinary request and transaction latency. Add Head write latency and outcome, conditional-write conflicts, uncertain writes and read-back attempts, takeover count and current epoch, renewal failures, late successes, forced step-downs, observed timestamp skew, and stale candidate and cleanup-failure counts. These distinguish storage latency, clock-bound violations, lease churn, and candidate accumulation. Clock-skew metrics do not remove the deployment's bounded-clock requirement. + +# Compatibility and backup +The Authority Head is a persisted JSON format covered by the repository compatibility framework. Freeze both version-1 examples, including the non-empty `NodeIdentity`, and verify deterministic `serde_json` encoding and decoding under the rules in [Encoding](#encoding). + +A consistent backup contains the Head and, once later RFCs add them, the published batch chain reachable from `last_batch` and the required tree objects. Candidate records and probes are excluded. The Head and referenced metadata must come from one consistent recovery point. Restoring a backup is an operator recovery action with existing writers fenced, not initialization of an empty Head. + +# Failure scenarios + +This table defines the expected outcomes. The validation plan references scenario IDs instead of repeating them; retirement and service recovery use the contracts in *Renewal* and *Outcomes*. + +| ID | Scenario | Expected behavior | +| --- | --- | --- | +| F1 | Concurrent initialization in a new namespace | One create succeeds; other initializers validate the existing Head without resetting it | +| F2 | Two candidates take over from the same observed version | Exactly one wins and the epoch increases once | +| F3 | Leader stops renewing | A candidate takes over after the guarded expiration of the latest store timestamp | +| F4 | First renewal failure or unknown result | Perform the immediate local retirement and service shutdown defined in *Renewal* | +| F5 | Old Leader uses stale metadata or an old epoch | The store rejects the stale condition; logical checks reject the old epoch even with fresh metadata | +| F6 | Renewal races with takeover on the same version | Exactly one conditional replacement succeeds | +| F7 | A write applies but its response is lost | Unique read-back evidence confirms application; otherwise preserve the unknown result | +| F8 | A timed-out renewal applies or is confirmed later | Apply *Outcomes* even before the original deadline; candidates honor any refreshed store timestamp | +| F9 | The former Leader wins a higher epoch | Service recovery follows *Outcomes* through the existing event handlers | +| F10 | Routing has expired while takeover is not yet permitted | Return `ElectionNoLeader` and stop routing the current write request | +| F11 | A routing hint names a node that stepped down early | An uninformed follower may return the hint before its cutoff; the receiver rejects the write, and a node aware of retirement returns `ElectionNoLeader` | +| F12 | Takeover precedes a delayed old-term request | The old metadata condition cannot replace the new Head | +| F13 | Head is missing at startup or runtime, including behind an S3 delete marker | Stop election and writes and require operator recovery; never initialize automatically | +| F14 | Head is deleted between observation and conditional replacement | `ConditionNotMatch` triggers read-back: recheck term, owner, and preconditions if present, or follow F13 if missing | +| F15 | Candidate refresh fails temporarily or a node restarts | Refresh retries with backoff; records become stale or are replaced by distinguishable refreshes | +| F16 | Candidate refresh races with stale cleanup | Safe conditional deletion preserves the refreshed record; otherwise use the offline-cleanup fallback | +| F17 | Multipart or staged publication uses stale metadata | Final publication rejects the condition and leaves the current Head unchanged | +| F18 | Leader resigns | Owner becomes empty; the next takeover starts a higher epoch | +| F19 | Startup probe encounters ignored conditions or missing metadata/capabilities | Fail startup, clean up the unique probe key, and never modify the Head | +| F20 | A process restarts at the same address | Its new incarnation does not inherit the previous process's ownership | + +# Validation plan + +## Format and deterministic protocol tests + +Freeze both version-1 JSON examples and cover deterministic serialization, required fields (including `owner`, `addr`, and `incarnation_id`), UUID encoding and validation, malformed JSON, invalid types, and unsupported versions. Add checked-overflow cases for both counters. Use injected wall and monotonic clocks and coherent mock `Metadata`; in-memory timestamps do not qualify a backend's real semantics. + +Add three explicit compatibility cases: + +- A v1 reader accepts an otherwise valid Head with an unknown top-level field and decodes the known fields unchanged. +- A v1 reader accepts an unknown field inside a non-empty `owner` and decodes its `NodeIdentity` unchanged. +- A v1 reader rejects a Head with `format_version: 2` even when the rest of its body follows the v1 structure. + +Unit and in-process tests cover the following scenarios, using the outcomes in [Failure scenarios](#failure-scenarios): + +| Scenarios | Test-layer-specific checks | +| --- | --- | +| F1, F19 | Exercise both `ConditionNotMatch` and `AlreadyExists` on repeated create; inject ignored conditions, incomplete or mismatched metadata, and probe cleanup failures. Check path isolation under `probes/` | +| F2, F5, F6, F18, F20 | Synchronize candidates on one snapshot; test stale metadata separately from a stale epoch with current metadata. Reuse an address with a new UUID, and keep each process UUID stable across operations | +| F3, F4, F9–F11 | Inject exact routing/takeover cutoffs, clock offsets, and Head-read delay. Observe leadership flags, event counts, heartbeat closure, and background-task callbacks through the existing handlers | +| F13, F14 | Delete between read and replacement; drive both present and missing follow-up reads and inspect the re-evaluated term, owner, and operation preconditions | +| F15, F16 | Keep node information unchanged across refreshes, advance refresh identifiers, pause cleanup after its scan, and inject transient refresh errors | + +Also check writer serialization, renewal priority over queued operations, queue rejection after retirement, and business-precondition re-evaluation against confirmed metadata. Mock both native version-match and ETag-match lowering, including unusable condition metadata; do not add an `if_not_changed` capability flag. + +## Failure injection + +Cover F4, F7–F9, and F12 by controlling persistence and response delivery independently: lose a response after persistence, hold a request until after client timeout, return successful read-back before and after the original deadline, and release an old request after takeover. Pause processes and interleave queued operations. Record the original deadline and event-handler ordering to check the referenced outcomes without relying on sleeps or eventual flags alone. Include late successful takeover confirmation against its provisional deadline. + +## Real-backend qualification + +Qualify S3-compatible endpoints including MinIO, Azure Blob, and GCS through the production Head read/write path and configured OpenDAL layers. Run the scenario groups below with real requests; mock tests alone do not enable a backend. + +- **F2, F5, F6, F17, F19:** verify coherent contents, conditional identifier, and `last_modified` from one version. Test current and stale metadata at the actual publication point, including forced S3 multipart completion and Azure block-list publication. Check completed-write visibility, timestamp precision, and that retry layers do not replay unresolved Head writes. +- **GCS qualification for F2, F5, F6, F17, F19:** use a real endpoint to verify `Metadata::version()` contains the observed generation and `if_not_changed` selects the generation condition, including any staged publication path. Apply the same admission criteria as other backends. **Status: not yet tested for this RFC revision; enable only after these tests pass.** +- **F1, F13, F14, F19:** verify repeated-create error normalization and read-back. A missing conditional-replacement target must map to `ConditionNotMatch`; only a subsequent Head read determines absence. In a versioned S3 bucket, test a delete marker on a disposable key and inspect retained history. Verify S3 ETag quotes survive round trips unchanged. +- **F15, F16:** check candidate listing metadata and each backend's actual conditional-delete capability and behavior, including versioned storage and refresh/delete interleavings. +- **F4, F7–F12:** repeat fault-injection schedules with real request delays, routing cutoffs, and event handlers. Measure renewal request and read-back latency under queued load to validate the margin provided by the ten-second lease and five-second renewal interval. + +Future OSS support requires evaluating a separate election protocol, such as one based on immutable records. + +# Implementation plan +The work is split into reviewable changes: + +- Add Authority types, persisted process identities, versioned JSON encoding, and both compatibility fixtures. +- Add capability-based backend admission and the isolated `probes/` runtime checks, including GCS generation-path qualification before enablement. +- Add coherent Head/Metadata reads, `if_not_changed` mutations, explicit initialization CLI, and missing-Head recovery errors. +- Add takeover, epoch fencing, the independent `OBJECT_STORE_META_LEASE_SECS` constant, renewal, local deadlines, and resignation. Validate request and read-back timing margins at the five-second renewal interval within the ten-second lease. +- Add the single-writer task and uncertain-write read-back. +- Add retrying candidate refresh, safe stale cleanup, and the existing `Election` integration. +- Add backend selection and storage options, metrics, and real-backend qualification with failure injection. + +The future batching RFC adds commit-tail mutations without changing the election contract. The B-link Tree RFC adds checkpoint publication through the same writer. + +# Future work +The future batching RFC will define immutable batches, operation identifiers, commit-tail preconditions, and client-visible commit confirmation. The B-link Tree RFC will define nodes, manifests, checkpoint roots, and read semantics. A later phase may add checkpoint garbage collection, quorum replication, and follower-local consistent reads. diff --git a/merge-main.log b/merge-main.log deleted file mode 100644 index ec2af961861..00000000000 --- a/merge-main.log +++ /dev/null @@ -1,10 +0,0 @@ -Auto-merging src/flow/src/batching_mode/frontend_client.rs -Auto-merging src/flow/src/lib.rs -CONFLICT (content): Merge conflict in src/flow/src/lib.rs -Auto-merging src/flow/src/server.rs -CONFLICT (content): Merge conflict in src/flow/src/server.rs -Auto-merging src/operator/src/statement/ddl.rs -Recorded preimage for 'src/flow/src/lib.rs' -Recorded preimage for 'src/flow/src/server.rs' -Automatic merge failed; fix conflicts and then commit the result. -exit=1 diff --git a/scripts/fetch-dashboard-assets.sh b/scripts/fetch-dashboard-assets.sh index 3ccebb95218..a2055f9033e 100755 --- a/scripts/fetch-dashboard-assets.sh +++ b/scripts/fetch-dashboard-assets.sh @@ -27,7 +27,7 @@ function retry_fetch() { local filename=$2 local auth_args=() - if [[ -n "$DASHBOARD_GITHUB_TOKEN" ]]; then + if [[ -n "$DASHBOARD_GITHUB_TOKEN" && "$url" == https://github.com/* ]]; then auth_args=( -H "Authorization: Bearer ${DASHBOARD_GITHUB_TOKEN}" -H "Accept: application/octet-stream" diff --git a/src/cli/src/bench.rs b/src/cli/src/bench.rs index eb559316527..14d58738fef 100644 --- a/src/cli/src/bench.rs +++ b/src/cli/src/bench.rs @@ -21,6 +21,8 @@ use async_trait::async_trait; use clap::Parser; use common_error::ext::BoxedError; use common_meta::key::{TableMetadataManager, TableMetadataManagerRef}; +#[cfg(any(feature = "pg_kvbackend", feature = "mysql_kvbackend"))] +use common_meta::kv_backend::KvBackendRef; use common_meta::kv_backend::etcd::EtcdStore; use common_meta::kv_backend::memory::MemoryKvBackend; #[cfg(feature = "mysql_kvbackend")] @@ -86,7 +88,7 @@ impl BenchTableMetadataCommand { }; #[cfg(feature = "pg_kvbackend")] - let kv_backend = if let Some(postgres_addr) = &self.postgres_addr { + let kv_backend: KvBackendRef = if let Some(postgres_addr) = &self.postgres_addr { info!("Using postgres as kv backend"); PgStore::with_url(postgres_addr, "greptime_metakv", 128) .await @@ -96,7 +98,7 @@ impl BenchTableMetadataCommand { }; #[cfg(feature = "mysql_kvbackend")] - let kv_backend = if let Some(mysql_addr) = &self.mysql_addr { + let kv_backend: KvBackendRef = if let Some(mysql_addr) = &self.mysql_addr { info!("Using mysql as kv backend"); MySqlStore::with_url(mysql_addr, "greptime_metakv", 128) .await diff --git a/src/cli/src/common/object_store.rs b/src/cli/src/common/object_store.rs index e671e6a534b..dc7e8adcb75 100644 --- a/src/cli/src/common/object_store.rs +++ b/src/cli/src/common/object_store.rs @@ -209,7 +209,7 @@ wrap_with_clap_prefix! { #[doc = "The endpoint of the object store."] endpoint: String = Default::default(), #[doc = "The SAS token of the object store."] - sas_token: Option, + sas_token: Option, } } diff --git a/src/cli/src/common/store.rs b/src/cli/src/common/store.rs index c3b8e477ed1..17418440b44 100644 --- a/src/cli/src/common/store.rs +++ b/src/cli/src/common/store.rs @@ -151,7 +151,7 @@ impl StoreConfig { &self.sanitize_store_addrs(), self.backend ); - let kvbackend = match self.backend { + let kvbackend: Result = match self.backend { BackendImpl::EtcdStore => { let tls_config = self.tls_config(); let etcd_client = create_etcd_client_with_tls( diff --git a/src/cli/src/data/export_v2/data.rs b/src/cli/src/data/export_v2/data.rs index dac094c0dda..e9242adcdac 100644 --- a/src/cli/src/data/export_v2/data.rs +++ b/src/cli/src/data/export_v2/data.rs @@ -368,7 +368,7 @@ fn build_gcs_connection( fn build_azblob_connection(storage: &ObjectStoreConfig) -> (String, Vec>) { let account_name = expose_optional_secret(&storage.azblob.azblob_account_name); let account_key = expose_optional_secret(&storage.azblob.azblob_account_key); - let sas_token = storage.azblob.azblob_sas_token.clone(); + let sas_token = expose_optional_secret(&storage.azblob.azblob_sas_token); let mut options = Vec::new(); if let Some(account_name) = &account_name { @@ -390,7 +390,7 @@ fn build_azblob_connection(storage: &ObjectStoreConfig) -> (String, Vec) -> Option { fn mask_secrets(sql: &str, secrets: &[Option]) -> String { let mut masked = sql.to_string(); + let mut secrets: Vec<_> = secrets + .iter() + .flatten() + .filter(|secret| !secret.is_empty()) + .collect(); + secrets.sort_by_key(|secret| std::cmp::Reverse(secret.len())); for secret in secrets { - if let Some(secret) = secret - && !secret.is_empty() - { - let escaped = escape_sql_literal(secret); - if escaped != *secret { - masked = masked.replace(&escaped, "[REDACTED]"); - } - masked = masked.replace(secret, "[REDACTED]"); + let escaped = escape_sql_literal(secret); + if escaped != *secret { + masked = masked.replace(&escaped, "[REDACTED]"); } + masked = masked.replace(secret, "[REDACTED]"); } masked } @@ -518,9 +520,9 @@ mod tests { fn test_build_azblob_connection_redacts_sas_token() { let storage = ObjectStoreConfig { azblob: PrefixedAzblobConnection { - azblob_account_name: Some(SecretString::from("account".to_string())), - azblob_account_key: Some(SecretString::from("key".to_string())), - azblob_sas_token: Some("sig=secret-token".to_string()), + azblob_account_name: Some(SecretString::from("sig".to_string())), + azblob_account_key: Some(SecretString::from("prefix_sig_CANARY_KEY".to_string())), + azblob_sas_token: Some("sv=2024-11-04&sig=CANARY_SIGNATURE".to_string().into()), ..Default::default() }, ..Default::default() @@ -529,16 +531,18 @@ mod tests { let (connection, secrets) = build_azblob_connection(&storage); let masked = mask_secrets(&connection, &secrets); - assert!(connection.contains("SAS_TOKEN='sig=secret-token'")); + assert!(connection.contains("SAS_TOKEN='sv=2024-11-04&sig=CANARY_SIGNATURE'")); assert!(masked.contains("SAS_TOKEN='[REDACTED]'")); - assert!(!masked.contains("sig=secret-token")); + assert!(!masked.contains("CANARY_SIGNATURE")); + assert!(masked.contains("ACCOUNT_KEY='[REDACTED]'")); + assert!(!masked.contains("CANARY_KEY")); } #[test] fn test_mask_secrets_redacts_sql_escaped_literals() { let sql = "COPY DATABASE \"greptime\".\"public\" TO 's3://bucket' CONNECTION (SECRET='ab''cd');"; - let masked = mask_secrets(sql, &[Some("ab'cd".to_string())]); + let masked = mask_secrets(sql, &[Some("ab".to_string()), Some("ab'cd".to_string())]); assert!(!masked.contains("ab'cd")); assert!(!masked.contains("ab''cd")); diff --git a/src/cli/src/data/storage_export.rs b/src/cli/src/data/storage_export.rs index 01bf252bbf5..c3222630981 100644 --- a/src/cli/src/data/storage_export.rs +++ b/src/cli/src/data/storage_export.rs @@ -41,10 +41,10 @@ fn format_root_path(root: &str) -> String { /// Helper function to mask multiple secrets in a string. fn mask_secrets(mut sql: String, secrets: &[&str]) -> String { + let mut secrets: Vec<_> = secrets.iter().filter(|secret| !secret.is_empty()).collect(); + secrets.sort_by_key(|secret| std::cmp::Reverse(secret.len())); for secret in secrets { - if !secret.is_empty() { - sql = sql.replace(secret, "[REDACTED]"); - } + sql = sql.replace(*secret, "[REDACTED]"); } sql } @@ -293,7 +293,7 @@ impl StorageExport for AzblobBackend { ]; if let Some(sas_token) = &self.config.azblob_sas_token { - connection_options.push(format!("SAS_TOKEN='{}'", sas_token)); + connection_options.push(format!("SAS_TOKEN='{}'", sas_token.expose_secret())); } let connection_str = format!(" CONNECTION ({})", connection_options.join(", ")); @@ -313,6 +313,7 @@ impl StorageExport for AzblobBackend { mask_secrets( sql.to_string(), &[ + expose_optional_secret(&self.config.azblob_sas_token), expose_optional_secret(&self.config.azblob_account_name), expose_optional_secret(&self.config.azblob_account_key), ], @@ -367,3 +368,27 @@ impl StorageType { !matches!(self, StorageType::Fs(_)) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_azblob_export_masks_sas_token() { + let backend = AzblobBackend { + config: PrefixedAzblobConnection { + azblob_account_name: Some("sig".to_string().into()), + azblob_account_key: Some("prefix_sig_CANARY_KEY".to_string().into()), + azblob_sas_token: Some("sig=secret-token".to_string().into()), + ..Default::default() + }, + }; + let (_, connection) = backend.get_storage_path("greptime", "public"); + assert!(connection.contains("SAS_TOKEN='sig=secret-token'")); + let masked = backend.mask_sensitive_info(&connection); + assert!(masked.contains("SAS_TOKEN='[REDACTED]'")); + assert!(masked.contains("ACCOUNT_KEY='[REDACTED]'")); + assert!(!masked.contains("CANARY_KEY")); + assert!(!masked.contains("sig=secret-token")); + } +} diff --git a/src/cmd/src/bin/query_perf_fixture/direct_sst.rs b/src/cmd/src/bin/query_perf_fixture/direct_sst.rs index fc004c2af13..ca50192c21d 100644 --- a/src/cmd/src/bin/query_perf_fixture/direct_sst.rs +++ b/src/cmd/src/bin/query_perf_fixture/direct_sst.rs @@ -34,7 +34,7 @@ use mito2::config::IndexConfig; use mito2::manifest::action::{RegionCheckpoint, RegionManifest, RemovedFilesRecord}; use mito2::memtable::sort_primary_key_record_batch; use mito2::read::FlatSource; -use mito2::sst::file::{FileMeta, RegionFileId}; +use mito2::sst::file::{FileMeta, FileMetaRef, RegionFileId}; use mito2::sst::index::{Indexer, IndexerBuilder}; use mito2::sst::parquet::writer::ParquetWriter; use mito2::sst::parquet::{SstInfo, WriteOptions}; @@ -490,7 +490,7 @@ pub(super) async fn run_direct_sst(args: DirectArgs) { remap_sst_file(&obj_store_dir, &table_dir, region_id, info.file_id, file_id); files.insert( file_id, - file_meta_from_sst_info(&info, region_id, file_id, sequence), + FileMetaRef::from(file_meta_from_sst_info(&info, region_id, file_id, sequence)), ); } } diff --git a/src/cmd/src/datanode/sst_replace.rs b/src/cmd/src/datanode/sst_replace.rs index d72a12a6002..e358c016e34 100644 --- a/src/cmd/src/datanode/sst_replace.rs +++ b/src/cmd/src/datanode/sst_replace.rs @@ -20,6 +20,7 @@ use mito2::manifest::action::{RegionEdit, RegionMetaAction, RegionMetaActionList use mito2::manifest::manager::{RegionManifestManager, RegionManifestOptions}; use mito2::region::ManifestStats; use mito2::sst::file::RegionFileId; +use mito2::sst::file_metadata::FileMetadataInterner; use mito2::sst::location::{region_dir_from_table_dir, sst_file_path}; use object_store::ObjectStore; use parquet::file::FOOTER_SIZE; @@ -94,11 +95,16 @@ impl SstReplaceCommand { .unwrap_or_else(|| vec![PathType::Bare, PathType::Data, PathType::Metadata]); let mut found = None; let mut existing_manifests = Vec::new(); + let file_metadata_interner = FileMetadataInterner::new(); for candidate in candidates { let stats = ManifestStats::default(); let region_dir = region_dir_from_table_dir(&self.table_dir, region_id, candidate); - let manifest_opts = - RegionManifestOptions::new(&mito_config, ®ion_dir, &object_store); + let manifest_opts = RegionManifestOptions::new( + &mito_config, + ®ion_dir, + &object_store, + file_metadata_interner.clone(), + ); let Some(manifest_manager) = RegionManifestManager::open(manifest_opts, &stats) .await .map_err(|e| { @@ -161,7 +167,7 @@ impl SstReplaceCommand { let new_num_row_groups = parquet_meta.num_row_groups() as u64; validate_replacement(&old_file, new_num_rows, new_num_row_groups)?; - let mut new_file = old_file.clone(); + let mut new_file = old_file.as_ref().clone(); new_file.file_size = new_file_size; new_file.num_rows = new_num_rows; new_file.num_row_groups = new_num_row_groups; diff --git a/src/cmd/src/frontend.rs b/src/cmd/src/frontend.rs index b9aa76f0f49..4348d55799d 100644 --- a/src/cmd/src/frontend.rs +++ b/src/cmd/src/frontend.rs @@ -464,7 +464,9 @@ impl StartCommand { ) .context(error::BuildCacheRegistrySnafu)?; - if let Some(plugin_cache_builder) = plugins::frontend::configure_cache_registry(&plugins) { + if let Some(plugin_cache_builder) = + plugins::frontend::configure_cache_registry(&plugins, cached_meta_backend.clone()) + { layered_cache_builder = layered_cache_builder.add_cache_registry(plugin_cache_builder.build()); } diff --git a/src/cmd/tests/load_config_test.rs b/src/cmd/tests/load_config_test.rs index 4a393c1d393..4f2a68ba988 100644 --- a/src/cmd/tests/load_config_test.rs +++ b/src/cmd/tests/load_config_test.rs @@ -25,6 +25,7 @@ use common_options::datanode::{ClientOptions, DatanodeClientOptions}; use common_runtime::global::RuntimeOptions; use common_telemetry::logging::{DEFAULT_LOGGING_DIR, DEFAULT_OTLP_HTTP_ENDPOINT, LoggingOptions}; use common_test_util::temp_dir::create_named_temp_file; +use common_time::timestamp::TimeUnit; use common_wal::config::DatanodeWalConfig; use common_wal::config::raft_engine::RaftEngineConfig; use datanode::config::{DatanodeOptions, RegionEngineConfig, StorageConfig}; @@ -249,6 +250,7 @@ fn test_load_frontend_example_config() { default_timezone: Some("UTC".to_string()), default_column_prefix: Some("greptime".to_string()), auto_create_table: true, + auto_create_physical_table_timestamp_unit: TimeUnit::Millisecond, meta_client: Some(MetaClientOptions { metasrv_addrs: vec!["127.0.0.1:3002".to_string()], timeout: Duration::from_secs(3), @@ -430,6 +432,7 @@ fn test_load_standalone_example_config() { default_timezone: Some("UTC".to_string()), default_column_prefix: Some("greptime".to_string()), auto_create_table: true, + auto_create_physical_table_timestamp_unit: TimeUnit::Millisecond, wal: DatanodeWalConfig::RaftEngine(RaftEngineConfig { dir: Some(format!("{}/{}", DEFAULT_DATA_HOME, WAL_DIR)), sync_period: Some(Duration::from_secs(5)), @@ -622,6 +625,7 @@ where "storage.access_key_secret", "storage.account_name", "storage.account_key", + "storage.sas_token", "storage.credential_path", "storage.credential", "region_engine.mito.inverted_index.intermediate_path", diff --git a/src/common/base/src/cancellation.rs b/src/common/base/src/cancellation.rs index 94aa872834c..d08a1ba7b03 100644 --- a/src/common/base/src/cancellation.rs +++ b/src/common/base/src/cancellation.rs @@ -104,6 +104,8 @@ where #[derive(Copy, Clone, Debug)] pub struct Cancelled; +impl std::error::Error for Cancelled {} + impl Display for Cancelled { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { write!(f, "Future has been cancelled") diff --git a/src/common/cache/src/lib.rs b/src/common/cache/src/lib.rs index 7cd68b2c81b..3e4ad6811e4 100644 --- a/src/common/cache/src/lib.rs +++ b/src/common/cache/src/lib.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! Shared memory cache primitives with native asynchronous load deduplication. +//! Shared memory caches. mod cache; mod sync_cache; diff --git a/src/common/function/Cargo.toml b/src/common/function/Cargo.toml index fbb7c173e76..4fa7eb67860 100644 --- a/src/common/function/Cargo.toml +++ b/src/common/function/Cargo.toml @@ -53,6 +53,7 @@ mito-codec.workspace = true nalgebra.workspace = true num-traits = "0.2" paste.workspace = true +partition.workspace = true regex.workspace = true reqwest = { workspace = true, optional = true } s2 = { version = "0.0.12", optional = true } diff --git a/src/common/function/src/function_registry.rs b/src/common/function/src/function_registry.rs index 8657636fe6b..027a576a51f 100644 --- a/src/common/function/src/function_registry.rs +++ b/src/common/function/src/function_registry.rs @@ -36,6 +36,10 @@ use crate::scalars::avg_calc::AvgCalcFunction; use crate::scalars::date::DateFunction; use crate::scalars::expression::ExpressionFunction; use crate::scalars::hll_count::HllCalcFunction; +use crate::scalars::iceberg_hash::{ + IcebergHashBoundsFunction, IcebergHashBreakpointsFunction, IcebergHashFunction, + IcebergHashRangesSqlFunction, +}; use crate::scalars::ip::IpFunctions; use crate::scalars::json::JsonFunction; use crate::scalars::matches::MatchesFunction; @@ -226,6 +230,12 @@ pub static FUNCTION_REGISTRY: LazyLock> = LazyLock::new(|| WelfordStddevFunction::register(&function_registry); DecodePrimaryKeyFunction::register(&function_registry); + // Hash used by `iceberg_hash` partition rules, plus its breakpoint helpers. + IcebergHashFunction::register(&function_registry); + IcebergHashBoundsFunction::register(&function_registry); + IcebergHashBreakpointsFunction::register(&function_registry); + IcebergHashRangesSqlFunction::register(&function_registry); + // Full text search function MatchesFunction::register(&function_registry); MatchesTermFunction::register(&function_registry); diff --git a/src/common/function/src/permission.rs b/src/common/function/src/permission.rs index 1a8392a0585..f3f5c6731a4 100644 --- a/src/common/function/src/permission.rs +++ b/src/common/function/src/permission.rs @@ -27,6 +27,17 @@ pub fn ensure_admin_privilege( func_ctx: &FunctionContext, fn_name: &str, ) -> datafusion_common::Result<()> { + // Database-less MySQL sessions have no authorized default scope. Admin targets + // are runtime function arguments, so frontend relation checks cannot authorize + // them, even when qualified. Require database selection before invoking a handler. + if func_ctx.query_ctx.channel() == session::context::Channel::Mysql + && func_ctx.query_ctx.current_schema().is_empty() + { + return Err(DataFusionError::External(Box::new(BoxedError::new( + session::SessionError::NoDatabaseSelected {}, + )))); + } + let admin_stmt = Statement::Admin(Admin::Func(sql::ast::Function { name: ObjectName::from(vec![Ident::new(fn_name)]), uses_odbc_syntax: false, diff --git a/src/common/function/src/scalars.rs b/src/common/function/src/scalars.rs index 4bc4c17f5bb..b65fce554d8 100644 --- a/src/common/function/src/scalars.rs +++ b/src/common/function/src/scalars.rs @@ -29,6 +29,7 @@ pub mod vector; pub(crate) mod avg_calc; pub(crate) mod hll_count; +pub(crate) mod iceberg_hash; pub mod ip; #[cfg(test)] pub(crate) mod test; diff --git a/src/common/function/src/scalars/iceberg_hash.rs b/src/common/function/src/scalars/iceberg_hash.rs new file mode 100644 index 00000000000..987b979477b --- /dev/null +++ b/src/common/function/src/scalars/iceberg_hash.rs @@ -0,0 +1,1247 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! SQL forms of the hash used by `iceberg_hash` partition rules. +//! +//! `iceberg_hash(col)` is the same function partition rules evaluate, exposed so +//! that users can measure a distribution or verify which bucket a value falls +//! into. It must never diverge from the rule form: both are built from +//! [`partition::function::iceberg_hash_udf`]. +//! +//! `iceberg_hash_bounds(n)` prints the breakpoints of an equal-width n-bucket +//! split, ready to paste into `PARTITION ON COLUMNS`. It only formats the +//! numbers; it does not create or change partitions. +//! +//! `iceberg_hash_breakpoints(v1, v2, ...)` hashes the given constant values +//! (any order, all from one hash encoding: strings, or integers) and prints them +//! sorted, for building buckets around known key values instead of around equal +//! widths. +//! +//! `iceberg_hash_ranges_sql(column, n)` renders the whole rule list for +//! `PARTITION ON COLUMNS` from an equal-width split, so it can be pasted +//! without expanding the breakpoints by hand. The column name is quoted, so +//! names that are not plain lowercase identifiers round-trip. + +use std::fmt; + +use datafusion::arrow::array::ArrayRef; +use datafusion::arrow::datatypes::DataType; +use datafusion_common::{DataFusionError, ScalarValue}; +use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, Signature, Volatility}; +use partition::function::{IcebergHash, iceberg_hash_udf}; + +use crate::function::Function; +use crate::function_registry::FunctionRegistry; + +/// `iceberg_hash_bounds` hard limit. Each region carries its own WAL, memtable, +/// manifest and compaction stream, so four-digit bucket counts are already far +/// beyond any practical table. +const MAX_BUCKETS: i64 = 1024; + +/// `iceberg_hash(col)`: registers the UDF partition rules already plan. +/// +/// The raw [`iceberg_hash_udf`] is registered instead of a `Function` wrapper: +/// a wrapper is a different `ScalarUDFImpl` type, and DataFusion compares +/// expressions by concrete type, so `iceberg_hash(host)` from SQL would not +/// compare equal to the expression `Operand::IcebergHash::to_logical_expr` +/// builds. One impl type also keeps a batch from constructing a fresh +/// `ScalarUDF`. +pub(crate) struct IcebergHashFunction; + +impl IcebergHashFunction { + pub fn register(registry: &FunctionRegistry) { + registry.register(iceberg_hash_udf()); + } +} + +/// `iceberg_hash_bounds(n)`: the `n - 1` equal-width breakpoints of the hash +/// space, as comma separated `INT32` literals. +#[derive(Debug)] +pub(crate) struct IcebergHashBoundsFunction { + signature: Signature, +} + +impl Default for IcebergHashBoundsFunction { + fn default() -> Self { + Self { + signature: Signature::any(1, Volatility::Immutable), + } + } +} + +impl IcebergHashBoundsFunction { + pub fn register(registry: &FunctionRegistry) { + registry.register_scalar(Self::default()); + } + + /// Returns the bucket count a constant scalar names, or an error naming + /// the rejected input and the calling function. + fn bucket_count(name: &str, scalar: &ScalarValue) -> datafusion_common::Result { + let count = match scalar { + ScalarValue::Int64(Some(v)) => *v, + ScalarValue::Int32(Some(v)) => i64::from(*v), + ScalarValue::Int16(Some(v)) => i64::from(*v), + ScalarValue::Int8(Some(v)) => i64::from(*v), + ScalarValue::UInt64(Some(v)) => i64::try_from(*v).map_err(|_| { + DataFusionError::Execution(format!("{name} bucket count is too large: {v}")) + })?, + ScalarValue::UInt32(Some(v)) => i64::from(*v), + ScalarValue::UInt16(Some(v)) => i64::from(*v), + ScalarValue::UInt8(Some(v)) => i64::from(*v), + other => { + return Err(DataFusionError::Execution(format!( + "{name} expects an integer bucket count, got {other:?}" + ))); + } + }; + Ok(count) + } + + /// Returns the `n - 1` equal-width breakpoints a constant bucket count + /// names, or an error when the count is not a constant integer in + /// `[2, MAX_BUCKETS]`. Shared with [`IcebergHashRangesSqlFunction`], which + /// passes its own name so errors name the function the user called. + fn breakpoints_of(name: &str, scalar: &ScalarValue) -> datafusion_common::Result> { + equal_split_breakpoints(name, Self::bucket_count(name, scalar)?) + } +} + +impl Function for IcebergHashBoundsFunction { + fn name(&self) -> &str { + "iceberg_hash_bounds" + } + + fn return_type(&self, _input_types: &[DataType]) -> datafusion_common::Result { + Ok(DataType::Utf8) + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn invoke_with_args( + &self, + args: ScalarFunctionArgs, + ) -> datafusion_common::Result { + let [arg] = args.args.as_slice() else { + return Err(DataFusionError::Execution(format!( + "{} expects one argument, got {}", + self.name(), + args.args.len() + ))); + }; + let value = match arg { + ColumnarValue::Scalar(scalar) => scalar, + ColumnarValue::Array(_) => { + return Err(DataFusionError::Execution(format!( + "{} expects a constant bucket count, not a column", + self.name() + ))); + } + }; + if value.is_null() { + return Ok(ColumnarValue::Scalar(ScalarValue::Utf8(None))); + } + Ok(ColumnarValue::Scalar(ScalarValue::Utf8(Some( + Self::breakpoints_of(self.name(), value)? + .iter() + .map(i32::to_string) + .collect::>() + .join(", "), + )))) + } +} + +impl fmt::Display for IcebergHashBoundsFunction { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("ICEBERG_HASH_BOUNDS") + } +} + +/// `iceberg_hash_breakpoints(v1, v2, ...)`: the sorted, comma separated hash +/// values of the given constant values, as `INT32` literals. +/// +/// The values do not need to be sorted; the output is. Every value must be a +/// constant from one hash encoding, exactly like the values a partition rule +/// compares the hashed column against: all strings, or all integers. Mixing a +/// string with an integer is rejected because the two hash differently, so the +/// result would not describe the target column. +/// +/// NULL values hash to NULL and can never match a comparison bound, so they are +/// rejected rather than silently dropped; an empty argument list is rejected +/// too. +#[derive(Debug)] +pub(crate) struct IcebergHashBreakpointsFunction { + signature: Signature, +} + +impl Default for IcebergHashBreakpointsFunction { + fn default() -> Self { + Self { + signature: Signature::variadic_any(Volatility::Immutable), + } + } +} + +impl IcebergHashBreakpointsFunction { + pub fn register(registry: &FunctionRegistry) { + registry.register_scalar(Self::default()); + } + + /// Hashes the scalar arguments into sorted breakpoints, or an error when + /// the arguments cannot form a usable breakpoint list. + fn breakpoints(&self, args: &ScalarFunctionArgs) -> datafusion_common::Result { + if args.args.is_empty() { + return Err(DataFusionError::Execution(format!( + "{} expects at least one value, got none", + self.name() + ))); + } + let mut domain: Option = None; + let mut scalars: Vec = Vec::with_capacity(args.args.len()); + for (position, arg) in args.args.iter().enumerate() { + let scalar = match arg { + ColumnarValue::Scalar(scalar) => scalar, + ColumnarValue::Array(_) => { + return Err(DataFusionError::Execution(format!( + "{} expects constant values, not a column (argument {})", + self.name(), + position + 1 + ))); + } + }; + let hashed = if scalar.is_null() { + // A NULL hash can never match a comparison bound, so a NULL + // value cannot be a breakpoint. Reject it instead of silently + // dropping it: the user asked for that value's breakpoint. + return Err(DataFusionError::Execution(format!( + "{} cannot use NULL as a breakpoint (argument {}): NULL rows fall in the lowest bucket", + self.name(), + position + 1 + ))); + } else { + let value_domain = HashDomain::of(scalar).ok_or_else(|| { + DataFusionError::Execution(format!( + "{} expects string or integer values, argument {} is {scalar:?}", + self.name(), + position + 1 + )) + })?; + match domain { + Some(expected) if expected != value_domain => { + return Err(DataFusionError::Execution(format!( + "{} expects values with one hash encoding, argument {} is {} but earlier values are {}", + self.name(), + position + 1, + value_domain, + expected + ))); + } + Some(_) => {} + None => domain = Some(value_domain), + } + into_hashed_scalar(self.name(), position + 1, scalar)? + }; + scalars.push(hashed); + } + scalars.sort_unstable(); + scalars.dedup(); + Ok(scalars + .iter() + .map(i32::to_string) + .collect::>() + .join(", ")) + } +} + +impl Function for IcebergHashBreakpointsFunction { + fn name(&self) -> &str { + "iceberg_hash_breakpoints" + } + + fn return_type(&self, _input_types: &[DataType]) -> datafusion_common::Result { + Ok(DataType::Utf8) + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn invoke_with_args( + &self, + args: ScalarFunctionArgs, + ) -> datafusion_common::Result { + Ok(ColumnarValue::Scalar(ScalarValue::Utf8(Some( + self.breakpoints(&args)?, + )))) + } +} + +impl fmt::Display for IcebergHashBreakpointsFunction { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("ICEBERG_HASH_BREAKPOINTS") + } +} + +/// `iceberg_hash_ranges_sql(column, n)`: the ready-to-paste rule list for +/// `PARTITION ON COLUMNS`, built from an equal-width `n`-bucket split. +/// +/// The first argument is the column name as a constant string, the second the +/// constant bucket count accepted by [`IcebergHashBoundsFunction`]. The output +/// is the complete expression list: `iceberg_hash(c) < b1`, +/// `iceberg_hash(c) >= b1 AND iceberg_hash(c) < b2`, ..., one rule per bucket, +/// with the same breakpoints `iceberg_hash_bounds(n)` prints. +/// +/// It only renders text; it does not create or change any partition. +#[derive(Debug)] +pub(crate) struct IcebergHashRangesSqlFunction { + signature: Signature, +} + +impl Default for IcebergHashRangesSqlFunction { + fn default() -> Self { + Self { + signature: Signature::any(2, Volatility::Immutable), + } + } +} + +impl IcebergHashRangesSqlFunction { + pub fn register(registry: &FunctionRegistry) { + registry.register_scalar(Self::default()); + } + + /// Renders the rule list from the hashed column name and the bucket count. + fn ranges_sql(&self, args: &ScalarFunctionArgs) -> datafusion_common::Result> { + let [column_name, buckets] = args.args.as_slice() else { + return Err(DataFusionError::Execution(format!( + "{} expects (column, buckets), got {} arguments", + self.name(), + args.args.len() + ))); + }; + let (name, buckets) = match (column_name, buckets) { + (ColumnarValue::Scalar(name), ColumnarValue::Scalar(buckets)) => (name, buckets), + (ColumnarValue::Array(_), _) | (_, ColumnarValue::Array(_)) => { + return Err(DataFusionError::Execution(format!( + "{} expects constant arguments (column name, bucket count)", + self.name() + ))); + } + }; + if name.is_null() || buckets.is_null() { + return Ok(None); + } + // Every string representation is a valid column name: a literal is + // `Utf8`, but a user can hand over `Utf8View` explicitly (for example + // `arrow_cast('host', 'Utf8View')`), and `Signature::any` does not + // convert it back. + let column = match name.try_as_str() { + Some(Some(column)) => column, + _ => { + return Err(DataFusionError::Execution(format!( + "{} expects a column name string, got {name:?}", + self.name() + ))); + } + }; + if column.is_empty() { + return Err(DataFusionError::Execution(format!( + "{} expects a non-empty column name", + self.name() + ))); + } + let bounds = IcebergHashBoundsFunction::breakpoints_of(self.name(), buckets)?; + // `column` is a raw name, not SQL text: quote it so names with dashes, + // spaces, dots, quotes or upper case stay a single identifier when the + // rendered rules are pasted into `PARTITION ON COLUMNS`. + let column = quoted_identifier(column); + let mut rules = Vec::with_capacity(bounds.len() + 1); + for (index, bound) in bounds.iter().copied().enumerate() { + if index == 0 { + // First rule: only the upper bound; NULL rows land here. + rules.push(format!("iceberg_hash({column}) < {bound}")); + } else { + let lower = bounds[index - 1]; + rules.push(format!( + "iceberg_hash({column}) >= {lower} AND iceberg_hash({column}) < {bound}" + )); + } + } + // Last rule: only the lower bound. + rules.push(format!( + "iceberg_hash({column}) >= {}", + bounds[bounds.len() - 1] + )); + Ok(Some(rules.join(",\n"))) + } +} + +impl Function for IcebergHashRangesSqlFunction { + fn name(&self) -> &str { + "iceberg_hash_ranges_sql" + } + + fn return_type(&self, _input_types: &[DataType]) -> datafusion_common::Result { + Ok(DataType::Utf8) + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn invoke_with_args( + &self, + args: ScalarFunctionArgs, + ) -> datafusion_common::Result { + Ok(ColumnarValue::Scalar(ScalarValue::Utf8( + self.ranges_sql(&args)?, + ))) + } +} + +impl fmt::Display for IcebergHashRangesSqlFunction { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("ICEBERG_HASH_RANGES_SQL") + } +} + +/// The hash encoding a constant goes through. Integer widths share one +/// encoding (`INT32`, `UINT8`, ... all hash as Iceberg longs), while strings use +/// their UTF-8 bytes, so mixing the two produces breakpoints that do not +/// describe a single column. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum HashDomain { + String, + Integer, +} + +impl HashDomain { + fn of(scalar: &ScalarValue) -> Option { + match scalar { + ScalarValue::Utf8(_) | ScalarValue::LargeUtf8(_) | ScalarValue::Utf8View(_) => { + Some(Self::String) + } + ScalarValue::Int8(_) + | ScalarValue::Int16(_) + | ScalarValue::Int32(_) + | ScalarValue::Int64(_) + | ScalarValue::UInt8(_) + | ScalarValue::UInt16(_) + | ScalarValue::UInt32(_) + | ScalarValue::UInt64(_) => Some(Self::Integer), + _ => None, + } + } +} + +impl fmt::Display for HashDomain { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::String => f.write_str("string"), + Self::Integer => f.write_str("integer"), + } + } +} + +/// Quotes a column name as a SQL identifier: wrap it in double quotes and +/// double every embedded quote. +fn quoted_identifier(column: &str) -> String { + format!("\"{}\"", column.replace('"', "\"\"")) +} + +/// Hashes one constant by evaluating it through the shared array form, so the +/// value goes through exactly the code a partition rule evaluates. +fn into_hashed_scalar( + name: &str, + position: usize, + scalar: &ScalarValue, +) -> datafusion_common::Result { + let array: ArrayRef = scalar.to_array()?; + // Hash through the array form a partition rule evaluates, one value per row. + let hashed = IcebergHash::evaluate_array(array.as_ref()).map_err(|error| { + DataFusionError::Execution(format!("{name} argument {position}: {error}")) + })?; + let value = ScalarValue::try_from_array(&hashed, 0)?; + match value { + ScalarValue::Int32(Some(hashed)) => Ok(hashed), + ScalarValue::Int32(None) => Err(DataFusionError::Execution(format!( + "{name} argument {position}: NULL hash" + ))), + other => Err(DataFusionError::Execution(format!( + "{name} argument {position}: unexpected hash {other:?}" + ))), + } +} + +/// Returns the `n - 1` equal-width breakpoints of `buckets` over `INT32`, +/// following `bi = i32::MIN + floor(i * 2^32 / n)`. +fn equal_split_breakpoints(name: &str, buckets: i64) -> datafusion_common::Result> { + if !(2..=MAX_BUCKETS).contains(&buckets) { + return Err(DataFusionError::Execution(format!( + "{name} expects a bucket count in [2, {MAX_BUCKETS}], got {buckets}" + ))); + } + + Ok((1..buckets) + .map(|i| { + let offset = ((i as i128) << 32) / buckets as i128; + i32::MIN as i64 + offset as i64 + }) + .map(|bound| bound as i32) + .collect()) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use datafusion::arrow::array::{Array, AsArray, Int64Array}; + use datafusion::arrow::datatypes::{DataType, Field, Int32Type}; + use datatypes::value::Value; + + use super::*; + use crate::function_registry::FUNCTION_REGISTRY; + + fn invoke( + function: &impl Function, + args: Vec, + rows: usize, + return_type: DataType, + ) -> ColumnarValue { + function + .invoke_with_args(ScalarFunctionArgs { + args, + arg_fields: vec![], + number_rows: rows, + return_field: Arc::new(Field::new("result", return_type, true)), + config_options: Arc::new(Default::default()), + }) + .unwrap() + } + + fn invoke_udf( + udf: &datafusion::logical_expr::ScalarUDF, + args: Vec, + rows: usize, + return_type: DataType, + ) -> ColumnarValue { + udf.invoke_with_args(ScalarFunctionArgs { + args, + arg_fields: vec![], + number_rows: rows, + return_field: Arc::new(Field::new("result", return_type, true)), + config_options: Arc::new(Default::default()), + }) + .unwrap() + } + + fn into_scalar(value: ColumnarValue) -> ScalarValue { + match value { + ColumnarValue::Scalar(scalar) => scalar, + ColumnarValue::Array(_) => panic!("expected a scalar result"), + } + } + + #[test] + fn test_hash_matches_partition_rule() { + let udf = iceberg_hash_udf(); + assert_eq!(udf.name(), "iceberg_hash"); + assert_eq!(udf.return_type(&[DataType::Utf8]).unwrap(), DataType::Int32); + + // Iceberg table spec, appendix B. + let hashed = invoke_udf( + &udf, + vec![ColumnarValue::Scalar(ScalarValue::Utf8(Some( + "iceberg".to_string(), + )))], + 1, + DataType::Int32, + ); + assert_eq!(into_scalar(hashed), ScalarValue::Int32(Some(1210000089))); + + // An untyped NULL reaches the UDF as ScalarValue::Null. + let untyped_null = invoke_udf( + &udf, + vec![ColumnarValue::Scalar(ScalarValue::Null)], + 1, + DataType::Int32, + ); + assert_eq!(into_scalar(untyped_null), ScalarValue::Int32(None)); + + // Integers hash as Iceberg longs, and NULL hashes to NULL. + let array = invoke_udf( + &udf, + vec![ColumnarValue::Array(Arc::new(Int64Array::from(vec![ + Some(34), + None, + ])))], + 2, + DataType::Int32, + ) + .to_array(2) + .unwrap(); + let array = array.as_primitive::(); + assert_eq!(array.value(0), 2017239379); + assert!(array.is_null(1)); + } + + #[test] + fn test_bounds_equal_split() { + let function = IcebergHashBoundsFunction::default(); + assert_eq!(function.name(), "iceberg_hash_bounds"); + assert_eq!( + function.return_type(&[DataType::Int64]).unwrap(), + DataType::Utf8 + ); + + let cases = [ + (2, "0"), + (3, "-715827883, 715827882"), + (4, "-1073741824, 0, 1073741824"), + ( + 8, + "-1610612736, -1073741824, -536870912, 0, 536870912, \ + 1073741824, 1610612736", + ), + ]; + for (buckets, expected) in cases { + let bounds = invoke( + &function, + vec![ColumnarValue::Scalar(ScalarValue::Int64(Some(buckets)))], + 1, + DataType::Utf8, + ); + assert_eq!( + into_scalar(bounds), + ScalarValue::Utf8(Some(expected.to_string())), + "buckets: {buckets}" + ); + } + + // Every integer width is a constant count; small widths must not be + // rejected as non-integer input. + for buckets in [ + ScalarValue::Int8(Some(4)), + ScalarValue::Int16(Some(4)), + ScalarValue::UInt8(Some(4)), + ScalarValue::UInt16(Some(4)), + ScalarValue::UInt32(Some(4)), + ScalarValue::UInt64(Some(4)), + ] { + let label = format!("{buckets:?}"); + let bounds = invoke( + &function, + vec![ColumnarValue::Scalar(buckets)], + 1, + DataType::Utf8, + ); + assert_eq!( + into_scalar(bounds), + ScalarValue::Utf8(Some("-1073741824, 0, 1073741824".to_string())), + "scalar: {label}" + ); + } + } + + #[test] + fn test_breakpoints_sorts_and_dedups() { + let function = IcebergHashBreakpointsFunction::default(); + assert_eq!(function.name(), "iceberg_hash_breakpoints"); + assert_eq!( + function.return_type(&[DataType::Utf8]).unwrap(), + DataType::Utf8 + ); + + // Unsorted strings: 'iceberg', 'a', and 'iceberg' again (dedup). + let bounds = invoke( + &function, + vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("iceberg".to_string()))), + ColumnarValue::Scalar(ScalarValue::Utf8(Some("a".to_string()))), + ColumnarValue::Scalar(ScalarValue::Utf8(Some("iceberg".to_string()))), + ], + 1, + DataType::Utf8, + ); + let iceberg = IcebergHash::evaluate_value(&Value::String("iceberg".into())) + .unwrap() + .as_i64() + .unwrap() as i32; + let a = IcebergHash::evaluate_value(&Value::String("a".into())) + .unwrap() + .as_i64() + .unwrap() as i32; + let expected = match (iceberg, a) { + (min, max) if min < max => format!("{min}, {max}"), + (min, max) => format!("{max}, {min}"), + }; + assert_eq!(into_scalar(bounds), ScalarValue::Utf8(Some(expected))); + + // Integer widths share one encoding, so they mix. + let bounds = invoke( + &function, + vec![ + ColumnarValue::Scalar(ScalarValue::Int64(Some(34))), + ColumnarValue::Scalar(ScalarValue::Int32(Some(-1))), + ColumnarValue::Scalar(ScalarValue::UInt8(Some(0))), + ], + 1, + DataType::Utf8, + ); + assert_eq!( + into_scalar(bounds), + ScalarValue::Utf8(Some(format!( + "{}, {}, {}", + IcebergHash::evaluate_value(&Value::Int32(-1)) + .unwrap() + .as_i64() + .unwrap() as i32, + IcebergHash::evaluate_value(&Value::UInt8(0)) + .unwrap() + .as_i64() + .unwrap() as i32, + IcebergHash::evaluate_value(&Value::Int64(34)) + .unwrap() + .as_i64() + .unwrap() as i32, + ))) + ); + } + + #[test] + fn test_breakpoints_rejects_invalid_input() { + let function = IcebergHashBreakpointsFunction::default(); + let call = |args: Vec| { + function.invoke_with_args(ScalarFunctionArgs { + args, + arg_fields: vec![], + number_rows: 1, + return_field: Arc::new(Field::new("result", DataType::Utf8, true)), + config_options: Arc::new(Default::default()), + }) + }; + + // Empty list cannot form a breakpoint list. + let err = call(vec![]).unwrap_err().to_string(); + assert!(err.contains("at least one value"), "{err}"); + + // NULL can never be a breakpoint: it falls in the lowest bucket. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("a".to_string()))), + ColumnarValue::Scalar(ScalarValue::Null), + ]) + .unwrap_err() + .to_string(); + assert!(err.contains("cannot use NULL as a breakpoint"), "{err}"); + + // A column cannot be hashed at planning time. + let err = call(vec![ColumnarValue::Array(Arc::new(Int64Array::from( + vec![34_i64], + )))]) + .unwrap_err() + .to_string(); + assert!(err.contains("constant values, not a column"), "{err}"); + + // Unsupported value type is named explicitly. + let err = call(vec![ColumnarValue::Scalar(ScalarValue::Float64(Some(1.0)))]) + .unwrap_err() + .to_string(); + assert!(err.contains("expects string or integer values"), "{err}"); + + // A string and an integer hash through different encodings, so mixing + // them cannot describe one column. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("iceberg".to_string()))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(34))), + ]) + .unwrap_err() + .to_string(); + assert!( + err.contains("one hash encoding") + && err.contains("argument 2 is integer") + && err.contains("earlier values are string"), + "{err}" + ); + } + + #[test] + fn test_bounds_rejects_invalid_input() { + let function = IcebergHashBoundsFunction::default(); + let call = |arg: ColumnarValue| { + function.invoke_with_args(ScalarFunctionArgs { + args: vec![arg], + arg_fields: vec![], + number_rows: 1, + return_field: Arc::new(Field::new("result", DataType::Utf8, true)), + config_options: Arc::new(Default::default()), + }) + }; + + for buckets in [i64::MIN, -1, 0, 1, MAX_BUCKETS + 1] { + let err = call(ColumnarValue::Scalar(ScalarValue::Int64(Some(buckets)))) + .unwrap_err() + .to_string(); + assert!( + err.contains("bucket count in [2, 1024]"), + "unexpected error for {buckets}: {err}" + ); + } + + let err = call(ColumnarValue::Scalar(ScalarValue::Utf8(Some( + "4".to_string(), + )))) + .unwrap_err() + .to_string(); + assert!(err.contains("integer bucket count"), "{err}"); + + let err = call(ColumnarValue::Array(Arc::new(Int64Array::from(vec![ + 4_i64, + ])))) + .unwrap_err() + .to_string(); + assert!(err.contains("constant bucket count"), "{err}"); + + // A NULL bucket count propagates instead of failing. + let null = call(ColumnarValue::Scalar(ScalarValue::Int64(None))).unwrap(); + assert_eq!(into_scalar(null), ScalarValue::Utf8(None)); + } + + #[test] + fn test_ranges_sql_renders_rule_list() { + let function = IcebergHashRangesSqlFunction::default(); + assert_eq!(function.name(), "iceberg_hash_ranges_sql"); + assert_eq!( + function + .return_type(&[DataType::Utf8, DataType::Int64]) + .unwrap(), + DataType::Utf8 + ); + + let render = |name: &str, buckets: i64| { + let rendered = invoke( + &function, + vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some(name.to_string()))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(buckets))), + ], + 1, + DataType::Utf8, + ); + into_scalar(rendered) + }; + + let four_buckets = [ + "iceberg_hash(\"host\") < -1073741824", + "iceberg_hash(\"host\") >= -1073741824 AND iceberg_hash(\"host\") < 0", + "iceberg_hash(\"host\") >= 0 AND iceberg_hash(\"host\") < 1073741824", + "iceberg_hash(\"host\") >= 1073741824", + ] + .join(",\n"); + assert_eq!(render("host", 4), ScalarValue::Utf8(Some(four_buckets))); + assert_eq!( + render("tenant", 2), + ScalarValue::Utf8(Some( + "iceberg_hash(\"tenant\") < 0,\niceberg_hash(\"tenant\") >= 0".to_string() + )) + ); + + // Names that are not plain lowercase identifiers must stay one + // identifier: dashes, spaces and embedded quotes are quoted/escaped. + assert_eq!( + render("host-name", 2), + ScalarValue::Utf8(Some( + "iceberg_hash(\"host-name\") < 0,\niceberg_hash(\"host-name\") >= 0".to_string() + )) + ); + assert_eq!( + render("he\"llo", 2), + ScalarValue::Utf8(Some( + "iceberg_hash(\"he\"\"llo\") < 0,\niceberg_hash(\"he\"\"llo\") >= 0".to_string() + )) + ); + + // A column name can arrive in any Arrow string representation. An + // explicit `Utf8View` (for example from `arrow_cast`) must render + // exactly like a plain literal instead of being rejected as non-string. + for name in [ + ScalarValue::LargeUtf8(Some("host-name".to_string())), + ScalarValue::Utf8View(Some("host-name".to_string())), + ] { + let rendered = invoke( + &function, + vec![ + ColumnarValue::Scalar(name), + ColumnarValue::Scalar(ScalarValue::Int64(Some(2))), + ], + 1, + DataType::Utf8, + ); + assert_eq!( + into_scalar(rendered), + ScalarValue::Utf8(Some( + "iceberg_hash(\"host-name\") < 0,\niceberg_hash(\"host-name\") >= 0" + .to_string() + )) + ); + } + + // A NULL name or bucket count propagates. + let call = |args: Vec| { + let rendered = function + .invoke_with_args(ScalarFunctionArgs { + args, + arg_fields: vec![], + number_rows: 1, + return_field: Arc::new(Field::new("result", DataType::Utf8, true)), + config_options: Arc::new(Default::default()), + }) + .unwrap(); + into_scalar(rendered) + }; + assert_eq!( + call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(None)), + ColumnarValue::Scalar(ScalarValue::Int64(Some(4))), + ]), + ScalarValue::Utf8(None) + ); + assert_eq!( + call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("host".to_string()))), + ColumnarValue::Scalar(ScalarValue::Int64(None)), + ]), + ScalarValue::Utf8(None) + ); + } + + /// Rendered rules must be valid partition expressions for the quoted + /// column, including names that need quoting. + #[test] + fn test_ranges_sql_output_parses_as_partition_rule() { + use sql::dialect::GreptimeDbDialect; + use sql::parser::{ParseOptions, ParserContext}; + + for column in ["host", "host-name", "HOST", "he\"llo", "a b"] { + let rendered = into_scalar(invoke( + &IcebergHashRangesSqlFunction::default(), + vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some(column.to_string()))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(4))), + ], + 1, + DataType::Utf8, + )); + let ScalarValue::Utf8(Some(rules)) = rendered else { + panic!("expected rendered rules for {column}"); + }; + let quoted = quoted_identifier(column); + let ddl = format!( + "CREATE TABLE t ({quoted} STRING, ts TIMESTAMP TIME INDEX, PRIMARY KEY ({quoted})) \ + PARTITION ON COLUMNS ({quoted}) ({rules})" + ); + let statements = ParserContext::create_with_dialect( + &ddl, + &GreptimeDbDialect {}, + ParseOptions::default(), + ) + .unwrap_or_else(|error| { + panic!("generated rules must parse for {column}: {error}\n{ddl}") + }); + assert_eq!(1, statements.len(), "{column}"); + } + } + + #[test] + fn test_ranges_sql_rejects_invalid_input() { + let function = IcebergHashRangesSqlFunction::default(); + let call = |args: Vec| { + function + .invoke_with_args(ScalarFunctionArgs { + args, + arg_fields: vec![], + number_rows: 1, + return_field: Arc::new(Field::new("result", DataType::Utf8, true)), + config_options: Arc::new(Default::default()), + }) + .unwrap_err() + .to_string() + }; + + // Bucket count outside the accepted range names the shared limit and + // the function the user actually called. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("host".to_string()))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(1))), + ]); + assert!(err.contains("bucket count in [2, 1024]"), "{err}"); + assert!( + err.contains("iceberg_hash_ranges_sql expects"), + "error must name the caller: {err}" + ); + + // Non-integer bucket count names the shared type error. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some("host".to_string()))), + ColumnarValue::Scalar(ScalarValue::Utf8(Some("4".to_string()))), + ]); + assert!(err.contains("integer bucket count"), "{err}"); + assert!( + err.contains("iceberg_hash_ranges_sql expects"), + "error must name the caller: {err}" + ); + + // Non-string column name. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Int64(Some(1))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(4))), + ]); + assert!(err.contains("column name string"), "{err}"); + + // Empty column name. + let err = call(vec![ + ColumnarValue::Scalar(ScalarValue::Utf8(Some(String::new()))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(4))), + ]); + assert!(err.contains("non-empty column name"), "{err}"); + + // Columns cannot be hashed at planning time. + let err = call(vec![ + ColumnarValue::Array(Arc::new(Int64Array::from(vec![1_i64]))), + ColumnarValue::Scalar(ScalarValue::Int64(Some(4))), + ]); + assert!(err.contains("constant arguments"), "{err}"); + + // Wrong arity. + let err = call(vec![ColumnarValue::Scalar(ScalarValue::Int64(Some(4)))]); + assert!(err.contains("expects (column, buckets)"), "{err}"); + } + + /// The SQL function must be the same `ScalarUDFImpl` the partition planner + /// builds, so a query expression compares equal to a rule expression. + #[test] + fn test_registered_hash_matches_partition_rule_expression() { + use datafusion::prelude::col; + + let sql_expr = FUNCTION_REGISTRY + .get_function("iceberg_hash") + .expect("iceberg_hash is registered") + .provide(crate::function::FunctionContext::default()) + .call(vec![col("host")]); + // The exact expression `Operand::IcebergHash` builds while planning a + // partition rule. + let rule_expr = IcebergHash::new("host").to_logical_expr(); + assert_eq!( + rule_expr, sql_expr, + "the registered iceberg_hash must be the UDF partition rules plan" + ); + } + + /// Functions that share a signature must stay distinct `ScalarUDF`s: + /// DataFusion compares expressions structurally, and treating `isnull` and + /// `iceberg_hash` as the same function corrupted `GROUP BY` planning + /// (`rebase_expr` matched `isnull(x)` against the grouped + /// `iceberg_hash(x)`). + #[test] + fn test_same_signature_functions_stay_distinct() { + use std::collections::HashSet; + use std::collections::hash_map::DefaultHasher; + use std::hash::{Hash, Hasher}; + + use datafusion::prelude::col; + + let provide = |name: &str| { + FUNCTION_REGISTRY + .get_function(name) + .unwrap_or_else(|| panic!("{name} is not registered")) + .provide(crate::function::FunctionContext::default()) + }; + let isnull = provide("isnull"); + let hash = provide("iceberg_hash"); + let bounds = provide("iceberg_hash_bounds"); + + // All three share `Signature::any(1, Immutable)` but are not equal. + assert!(isnull != hash); + assert!(isnull != bounds); + assert!(hash != bounds); + assert_ne!( + isnull.call(vec![col("x")]), + hash.call(vec![col("x")]), + "isnull(x) and iceberg_hash(x) must not compare equal" + ); + + let hash_of = |udf: &datafusion::logical_expr::ScalarUDF| { + let mut hasher = DefaultHasher::new(); + udf.hash(&mut hasher); + hasher.finish() + }; + // Equal objects keep hashing equally, so the registry still finds them. + let hash_again = provide("iceberg_hash"); + assert_eq!(hash, hash_again); + assert_eq!(hash_of(&hash), hash_of(&hash_again)); + // Distinct functions remain separable keys in hash-based containers. + let udfs: HashSet<_> = [isnull, hash, bounds].into_iter().collect(); + assert_eq!(3, udfs.len()); + } + + /// `isnull` and `iceberg_hash` share a signature; grouping by one while + /// selecting the other must still plan (regression for the identity bug). + #[tokio::test] + async fn test_group_by_with_same_signature_functions() { + use datafusion::arrow::array::{ArrayRef, Int64Array}; + use datafusion::arrow::record_batch::RecordBatch; + use datafusion::prelude::SessionContext; + + use crate::function::FunctionContext; + + let ctx = SessionContext::new(); + for name in ["isnull", "iceberg_hash"] { + let udf = FUNCTION_REGISTRY + .get_function(name) + .unwrap() + .provide(FunctionContext::default()); + ctx.register_udf(udf); + } + ctx.register_batch( + "t", + RecordBatch::try_from_iter([( + "x", + Arc::new(Int64Array::from(vec![Some(34), None])) as ArrayRef, + )]) + .unwrap(), + ) + .unwrap(); + + for sql in [ + "SELECT isnull(x) FROM t GROUP BY x, iceberg_hash(x)", + "SELECT iceberg_hash(x) FROM t GROUP BY x, isnull(x)", + ] { + let batches = ctx + .sql(sql) + .await + .unwrap_or_else(|error| panic!("{sql} must plan: {error}")) + .collect() + .await + .unwrap_or_else(|error| panic!("{sql} must execute: {error}")); + assert_eq!( + 2, + batches.iter().map(|batch| batch.num_rows()).sum::(), + "{sql}" + ); + } + } + + /// Both functions must resolve and run through a plain SQL session, not just + /// through the [Function] trait. + #[tokio::test] + async fn test_functions_run_through_sql() { + use datafusion::prelude::SessionContext; + + use crate::function::FunctionContext; + + // Register what the registry actually exposes, which also pins the + // `iceberg_hash` registration to the raw partition-rule UDF. + let ctx = SessionContext::new(); + for name in [ + "iceberg_hash", + "iceberg_hash_bounds", + "iceberg_hash_breakpoints", + "iceberg_hash_ranges_sql", + ] { + let udf = FUNCTION_REGISTRY + .get_function(name) + .unwrap_or_else(|| panic!("{name} is not registered")) + .provide(FunctionContext::default()); + ctx.register_udf(udf); + } + + let batches = ctx + .sql( + "SELECT iceberg_hash('iceberg') AS h, iceberg_hash(34) AS i, \ + iceberg_hash(NULL) AS n, iceberg_hash_bounds(4) AS b, \ + iceberg_hash_bounds(CAST(4 AS SMALLINT)) AS sb, \ + iceberg_hash_breakpoints('iceberg', 'a') AS bp, \ + iceberg_hash_ranges_sql('host-name', 2) AS rs, \ + iceberg_hash_ranges_sql(arrow_cast('host-name', 'Utf8View'), 2) AS rsv", + ) + .await + .unwrap() + .collect() + .await + .unwrap(); + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 1); + assert_eq!( + batches[0].column(0).as_primitive::().value(0), + 1210000089 + ); + assert_eq!( + batches[0].column(1).as_primitive::().value(0), + 2017239379 + ); + assert!(batches[0].column(2).as_primitive::().is_null(0)); + assert_eq!( + batches[0].column(3).as_string::().value(0), + "-1073741824, 0, 1073741824" + ); + assert_eq!( + batches[0].column(4).as_string::().value(0), + "-1073741824, 0, 1073741824" + ); + // Sorted hash values of the two strings, computed by the shared hash. + let iceberg = IcebergHash::evaluate_value(&Value::String("iceberg".into())) + .unwrap() + .as_i64() + .unwrap() as i32; + let a = IcebergHash::evaluate_value(&Value::String("a".into())) + .unwrap() + .as_i64() + .unwrap() as i32; + let expected_breakpoints = if iceberg < a { + format!("{iceberg}, {a}") + } else { + format!("{a}, {iceberg}") + }; + assert_eq!( + batches[0].column(5).as_string::().value(0), + expected_breakpoints + ); + // Ready-to-paste rule list, with the column name quoted. + let rendered_rules = "iceberg_hash(\"host-name\") < 0,\niceberg_hash(\"host-name\") >= 0"; + assert_eq!( + batches[0].column(6).as_string::().value(0), + rendered_rules + ); + // An explicit `Utf8View` column name renders the same rules. + assert_eq!( + batches[0].column(7).as_string::().value(0), + rendered_rules + ); + } + + #[test] + fn test_hash_functions_are_registered() { + for name in [ + "iceberg_hash", + "iceberg_hash_bounds", + "iceberg_hash_breakpoints", + "iceberg_hash_ranges_sql", + ] { + assert!( + FUNCTION_REGISTRY.get_function(name).is_some(), + "{name} is not registered" + ); + } + } +} diff --git a/src/common/function/src/scalars/udf.rs b/src/common/function/src/scalars/udf.rs index 48c59ac230a..151982ef581 100644 --- a/src/common/function/src/scalars/udf.rs +++ b/src/common/function/src/scalars/udf.rs @@ -35,7 +35,13 @@ impl Debug for ScalarUdf { impl PartialEq for ScalarUdf { fn eq(&self, other: &Self) -> bool { - self.function.signature() == other.function.signature() + // Identity first: many unrelated functions share one signature (for + // example `Signature::any(1, Immutable)`), and DataFusion compares + // expressions (and therefore `ScalarUDF`s) structurally, so matching on + // the signature alone makes different functions interchangeable and + // corrupts planning (see `rebase_expr`). + self.function.name() == other.function.name() + && self.function.signature() == other.function.signature() } } @@ -43,7 +49,8 @@ impl Eq for ScalarUdf {} impl Hash for ScalarUdf { fn hash(&self, state: &mut H) { - self.function.signature().hash(state) + self.function.name().hash(state); + self.function.signature().hash(state); } } diff --git a/src/common/function/src/scalars/vector/impl_conv.rs b/src/common/function/src/scalars/vector/impl_conv.rs index 00d23cedf7f..d46f2b12606 100644 --- a/src/common/function/src/scalars/vector/impl_conv.rs +++ b/src/common/function/src/scalars/vector/impl_conv.rs @@ -43,7 +43,9 @@ pub fn binlit_as_veclit(bytes: &[u8]) -> Result> { .fail(); } - if cfg!(target_endian = "little") { + if cfg!(target_endian = "little") && bytes.as_ptr().align_offset(align_of::()) == 0 { + // The length is a multiple of four, the pointer is aligned, and every f32 bit + // pattern is valid. The borrowed slice cannot outlive the input bytes. Ok(unsafe { let vec = std::slice::from_raw_parts( bytes.as_ptr() as *const f32, @@ -123,6 +125,29 @@ mod tests { assert!(result.is_err()); } + #[test] + fn test_binlit_as_veclit_alignment() { + #[repr(align(4))] + struct Aligned([u8; 16]); + let mut storage = Aligned([0; 16]); + let values = [1.25, -2.5, 3.0]; + let bytes = veclit_to_binlit(&values); + for offset in 0..4 { + storage.0[offset..offset + bytes.len()].copy_from_slice(&bytes); + let result = binlit_as_veclit(&storage.0[offset..offset + bytes.len()]).unwrap(); + assert_eq!(result.as_ref(), values); + assert_eq!( + matches!(result, Cow::Borrowed(_)), + cfg!(target_endian = "little") && offset == 0 + ); + assert!( + binlit_as_veclit(&storage.0[offset..offset]) + .unwrap() + .is_empty() + ); + } + } + #[test] fn test_binlit_as_veclit() { let vec = &[1.0, 2.0, 3.0]; diff --git a/src/common/function/src/system/database.rs b/src/common/function/src/system/database.rs index 1884c5525f0..b9ad7ab7178 100644 --- a/src/common/function/src/system/database.rs +++ b/src/common/function/src/system/database.rs @@ -59,7 +59,9 @@ macro_rules! impl_current_schema_function { let func_ctx = find_function_context(&args)?; let db = func_ctx.query_ctx.current_schema(); - Ok(ColumnarValue::Scalar(ScalarValue::Utf8View(Some(db)))) + Ok(ColumnarValue::Scalar(ScalarValue::Utf8View( + (!db.is_empty()).then_some(db), + ))) } } }; diff --git a/src/common/function/src/system/pg_catalog.rs b/src/common/function/src/system/pg_catalog.rs index 3029361c202..bc311d84164 100644 --- a/src/common/function/src/system/pg_catalog.rs +++ b/src/common/function/src/system/pg_catalog.rs @@ -90,7 +90,9 @@ impl Function for CurrentSchemaFunction { let func_ctx = find_function_context(&args)?; let db = func_ctx.query_ctx.current_schema(); - Ok(ColumnarValue::Scalar(ScalarValue::Utf8View(Some(db)))) + Ok(ColumnarValue::Scalar(ScalarValue::Utf8View( + (!db.is_empty()).then_some(db), + ))) } } @@ -158,11 +160,15 @@ impl Function for CurrentSchemasFunction { &self, args: ScalarFunctionArgs, ) -> datafusion_common::Result { + let func_ctx = find_function_context(&args)?; + let schema = func_ctx.query_ctx.current_schema(); let args = ColumnarValue::values_to_arrays(&args.args)?; let input = as_boolean_array(&args[0]); - // Create a UTF8 array with a single value - let mut values = vec!["public"]; + let mut values = Vec::new(); + if !schema.is_empty() { + values.push(schema.as_str()); + } // include implicit schemas if input.value(0) { values.push(INFORMATION_SCHEMA_NAME); diff --git a/src/common/meta/AGENTS.md b/src/common/meta/AGENTS.md index 69d76d3c5ca..1be24b2ee9a 100644 --- a/src/common/meta/AGENTS.md +++ b/src/common/meta/AGENTS.md @@ -27,6 +27,10 @@ and durable DDL procedures. Metasrv process and service wiring live in invalidation. - Procedure changes must preserve persisted state and `TYPE_NAME`; new procedures need loader registration in `src/common/meta/src/ddl_manager.rs`. +- Database drops persist bounded logical-table batches before deleting metadata. + Replays must finish cache invalidation even after table-name keys are gone; + physical-region removal follows the logical phase. Keep the legacy single-table + executor readable for in-flight procedures from older versions. - Marker-style table options that no region consumes (`greptime.semantic.*`, `repartition.column.hint`) go through `AnnotationFamily` in `src/table/src/requests.rs` — classification, validation, logical-table diff --git a/src/common/meta/src/cache_invalidator.rs b/src/common/meta/src/cache_invalidator.rs index edf84b00b83..5e781e9c7da 100644 --- a/src/common/meta/src/cache_invalidator.rs +++ b/src/common/meta/src/cache_invalidator.rs @@ -173,6 +173,10 @@ where let key = NodeAddressKey::with_flownode(*node_id); self.invalidate_key(&key.to_bytes()).await; } + #[cfg(feature = "enterprise")] + CacheIdent::TriggerId(_) => { + // Trigger keys are owned by enterprise cache invalidators. + } CacheIdent::User(_) => { // User cache invalidation is handled by external // CacheInvalidator implementations. diff --git a/src/common/meta/src/ddl/alter_logical_tables.rs b/src/common/meta/src/ddl/alter_logical_tables.rs index ccf130cec8c..ac70980329a 100644 --- a/src/common/meta/src/ddl/alter_logical_tables.rs +++ b/src/common/meta/src/ddl/alter_logical_tables.rs @@ -24,7 +24,7 @@ use common_procedure::{Context, EventContext, EventTrigger, LockKey, Procedure, use common_telemetry::{debug, error, info, warn}; pub use executor::make_alter_region_request; use serde::{Deserialize, Serialize}; -use snafu::{ResultExt, ensure}; +use snafu::ResultExt; use store_api::metadata::ColumnMetadata; use store_api::metric_engine_consts::ALTER_PHYSICAL_EXTENSION_KEY; use strum::AsRefStr; @@ -40,7 +40,7 @@ use crate::ddl::event::table::{ TableDdlEvent, TableDdlEventType, TableDdlLocator, alter_table_kind_name, }; use crate::ddl::utils::{extract_column_metadatas, map_to_procedure_error, sync_follower_regions}; -use crate::error::{self, Result}; +use crate::error::Result; use crate::instruction::CacheIdent; use crate::key::DeserializedValueWithBytes; use crate::key::table_info::TableInfoValue; @@ -67,7 +67,7 @@ fn build_validator_from_alter_table_data<'a>( .iter() .map(|task| &task.alter_table) .collect::>(); - AlterLogicalTableValidator::new(physical_table_id, alters) + AlterLogicalTableValidator::new(physical_table_id, alters, &data.logical_table_ids) } /// Builds the executor from the [`AlterTablesData`]. @@ -149,26 +149,6 @@ impl AlterLogicalTablesProcedure { ); } - // Locks are fixed at submission. A logical table that resolves to an - // id outside the locked set here was dropped and recreated in between, - // so this procedure holds no lock for it. Procedures restored from - // pre-upgrade state have no recorded ids and keep the old behavior. - if !self.data.logical_table_ids.is_empty() { - for value in &table_info_values { - let table_id = value.get_inner_ref().table_info.ident.table_id; - ensure!( - self.data.logical_table_ids.contains(&table_id), - error::UnexpectedSnafu { - err_msg: format!( - "logical table {} (id {table_id}) is not covered by the \ - procedure locks; retry the statement", - value.get_inner_ref().table_info.name - ), - } - ); - } - } - // Updates the procedure state. retain_unskipped(&mut self.data.tasks, &skip_alter); self.data.physical_table_info = Some(physical_table_info); diff --git a/src/common/meta/src/ddl/alter_logical_tables/validator.rs b/src/common/meta/src/ddl/alter_logical_tables/validator.rs index 7e6c130351f..af0c642d919 100644 --- a/src/common/meta/src/ddl/alter_logical_tables/validator.rs +++ b/src/common/meta/src/ddl/alter_logical_tables/validator.rs @@ -26,7 +26,7 @@ use crate::ddl::utils::table_info::{ }; use crate::error::{ AlterLogicalTablesInvalidArgumentsSnafu, Result, TableInfoNotFoundSnafu, - TableRouteNotFoundSnafu, + TableRouteNotFoundSnafu, UnexpectedSnafu, }; use crate::key::table_info::TableInfoValue; use crate::key::table_route::{PhysicalTableRouteValue, TableRouteManager, TableRouteValue}; @@ -36,13 +36,19 @@ use crate::key::{DeserializedValueWithBytes, TableMetadataManagerRef}; pub struct AlterLogicalTableValidator<'a> { physical_table_id: TableId, alters: Vec<&'a AlterTableExpr>, + expected_table_ids: &'a [TableId], } impl<'a> AlterLogicalTableValidator<'a> { - pub fn new(physical_table_id: TableId, alters: Vec<&'a AlterTableExpr>) -> Self { + pub fn new( + physical_table_id: TableId, + alters: Vec<&'a AlterTableExpr>, + expected_table_ids: &'a [TableId], + ) -> Self { Self { physical_table_id, alters, + expected_table_ids, } } @@ -187,6 +193,14 @@ impl<'a> AlterLogicalTableValidator<'a> { let table_ids = get_all_table_ids_by_names(table_metadata_manager.table_name_manager(), &table_names) .await?; + // Check before skipping completed ALTERs, while the procedure holds the pinned ID locks. + // Old persisted procedures have no pinned IDs and retain their previous behavior. + ensure!( + self.expected_table_ids.is_empty() || self.expected_table_ids == table_ids, + UnexpectedSnafu { + err_msg: "logical tables changed and are not covered by the procedure locks; retry the statement", + } + ); let mut table_info_values = get_all_table_info_values_by_table_ids( table_metadata_manager.table_info_manager(), &table_ids, diff --git a/src/common/meta/src/ddl/create_logical_tables.rs b/src/common/meta/src/ddl/create_logical_tables.rs index 9b231f3c538..1e65a7ef724 100644 --- a/src/common/meta/src/ddl/create_logical_tables.rs +++ b/src/common/meta/src/ddl/create_logical_tables.rs @@ -97,7 +97,7 @@ impl CreateLogicalTablesProcedure { self.check_input_tasks()?; // Sets physical region numbers self.fill_physical_table_info().await?; - // Add partition columns from physical table to logical table schemas + // Add the physical table's user partition columns to logical table schemas. self.merge_partition_columns_into_logical_tables()?; // Checks if the tables exist self.check_tables_already_exist().await?; @@ -325,6 +325,10 @@ pub struct CreateTablesData { physical_table_id: TableId, physical_region_numbers: Vec, physical_columns: Vec, + /// Partition column names of the physical table. + /// + /// Procedures persisted before engine identity columns were filtered may store them + /// here, so the merge re-checks the names instead of trusting this list. physical_partition_columns: Vec, } diff --git a/src/common/meta/src/ddl/create_logical_tables/metadata.rs b/src/common/meta/src/ddl/create_logical_tables/metadata.rs index 22424668946..da9cbc1708f 100644 --- a/src/common/meta/src/ddl/create_logical_tables/metadata.rs +++ b/src/common/meta/src/ddl/create_logical_tables/metadata.rs @@ -18,6 +18,7 @@ use std::sync::Arc; use datatypes::prelude::ConcreteDataType; use datatypes::schema::{ColumnSchema, Schema}; use snafu::OptionExt; +use store_api::metric_engine_consts::is_metric_engine_internal_column; use crate::ddl::create_logical_tables::CreateLogicalTablesProcedure; use crate::error::Result; @@ -82,6 +83,14 @@ impl CreateLogicalTablesProcedure { // Add missing partition columns for partition_column in partition_columns { + // The engine owns its identity columns: they are derived from the logical + // table at write time, so a logical schema must never declare them. Older + // procedures may have persisted an unfiltered partition column list, hence + // filtering here rather than only while collecting it. + if is_metric_engine_internal_column(partition_column) { + continue; + } + if !existing_column_names.contains(partition_column) { let new_column_index = column_schemas.len() + new_columns.len(); diff --git a/src/common/meta/src/ddl/drop_database.rs b/src/common/meta/src/ddl/drop_database.rs index 5bbd229ec5b..d083d4402b0 100644 --- a/src/common/meta/src/ddl/drop_database.rs +++ b/src/common/meta/src/ddl/drop_database.rs @@ -26,7 +26,7 @@ use common_procedure::{ Context as ProcedureContext, EventContext, EventTrigger, LockKey, Procedure, Result as ProcedureResult, Status, }; -use futures::stream::BoxStream; +use futures::stream::{BoxStream, Peekable}; use serde::{Deserialize, Serialize}; use snafu::ResultExt; use tonic::async_trait; @@ -54,12 +54,14 @@ pub(crate) enum DropTableTarget { Physical, } +type TableStream = Peekable>>; + /// Context of [DropDatabaseProcedure] execution. pub(crate) struct DropDatabaseContext { catalog: String, schema: String, drop_if_exists: bool, - tables: Option>>, + tables: Option, retrying: bool, } diff --git a/src/common/meta/src/ddl/drop_database/cursor.rs b/src/common/meta/src/ddl/drop_database/cursor.rs index e060567be58..7e300efde86 100644 --- a/src/common/meta/src/ddl/drop_database/cursor.rs +++ b/src/common/meta/src/ddl/drop_database/cursor.rs @@ -13,21 +13,24 @@ // limitations under the License. use std::any::Any; +use std::collections::HashSet; +use std::pin::Pin; +use common_base::readable_size::ReadableSize; use common_catalog::format_full_table_name; use common_procedure::Status; -use futures::TryStreamExt; +use futures::{StreamExt, TryStreamExt}; use serde::{Deserialize, Serialize}; -use snafu::OptionExt; +use snafu::{OptionExt, ensure}; use table::metadata::{TableId, TableType}; use table::table_name::TableName; use crate::cache_invalidator::Context; use crate::ddl::DdlContext; -use crate::ddl::drop_database::executor::DropDatabaseExecutor; +use crate::ddl::drop_database::executor::{DropDatabaseBatchExecutor, DropDatabaseExecutor}; use crate::ddl::drop_database::metadata::DropDatabaseRemoveMetadata; use crate::ddl::drop_database::{DropDatabaseContext, DropTableTarget, State}; -use crate::error::{Result, TableInfoNotFoundSnafu}; +use crate::error::{Result, TableInfoNotFoundSnafu, UnexpectedSnafu}; use crate::instruction::CacheIdent; use crate::key::table_route::TableRouteValue; @@ -42,6 +45,123 @@ impl DropDatabaseCursor { Self { target } } + /// Collects a bounded page without mutating metadata. Only the executor's batch is durable. + async fn collect_logical_tables( + &mut self, + ddl_ctx: &DdlContext, + ctx: &mut DropDatabaseContext, + ) -> Result<(Box, Status)> { + let page = Self::collect_page(ctx).await?; + if page.is_empty() { + return self.handle_reach_end(ctx); + } + let tables = Self::logical_tables(ddl_ctx, ctx, page).await?; + if tables.is_empty() { + return Ok((Box::new(Self::new(self.target)), Status::executing(false))); + } + Ok(( + Box::new(DropDatabaseBatchExecutor { tables }), + Status::executing(true), + )) + } + + // Leave headroom below etcd's default 1.5 MiB request limit. + const MAX_BATCH_BYTES: usize = ReadableSize::mb(1).as_bytes() as usize; + + fn estimated_table_bytes(catalog: &str, schema: &str, name: &str) -> usize { + // A control byte can expand to 6 JSON bytes, then 7 inside ProcedureMessage's + // JSON string. Round up to 8x and allow 128 bytes for fixed per-table JSON/key overhead. + 128 + 8 * (catalog.len() + schema.len() + name.len()) + } + + async fn collect_page(ctx: &mut DropDatabaseContext) -> Result> { + let mut page = Vec::new(); + let mut bytes: usize = 0; + let tables = ctx.tables.as_mut().unwrap(); + loop { + match Pin::new(&mut *tables).peek().await { + Some(Ok((name, _))) => { + let size = Self::estimated_table_bytes(&ctx.catalog, &ctx.schema, name); + if !page.is_empty() && bytes.saturating_add(size) > Self::MAX_BATCH_BYTES { + break; + } + // A single oversized name must still make progress. Procedure storage + // already splits oversized values; never strand an existing table here. + bytes = bytes.saturating_add(size); + } + Some(Err(_)) => {} // Consume below to propagate the owned error. + None => break, + } + let (name, value) = tables.try_next().await?.unwrap(); + page.push((name, value.table_id())); + } + Ok(page) + } + + async fn logical_tables( + ddl_ctx: &DdlContext, + ctx: &mut DropDatabaseContext, + page: Vec<(String, TableId)>, + ) -> Result> { + let manager = &ddl_ctx.table_metadata_manager; + let ids = page.iter().map(|(_, id)| *id).collect::>(); + let infos = manager.table_info_manager().batch_get(&ids).await?; + let routes = manager + .table_route_manager() + .table_route_storage() + .batch_get(&ids) + .await?; + let physical_ids = routes + .iter() + .filter_map(|route| match route { + Some(TableRouteValue::Logical(route)) => Some(route.physical_table_id()), + _ => None, + }) + .collect::>() + .into_iter() + .collect::>(); + let physical_infos = manager + .table_info_manager() + .batch_get(&physical_ids) + .await?; + let mut tables = Vec::new(); + for ((name, id), route) in page.into_iter().zip(routes) { + let info = infos.get(&id).with_context(|| TableInfoNotFoundSnafu { + table: format_full_table_name(&ctx.catalog, &ctx.schema, &name), + })?; + if info.table_info.table_type == TableType::View { + // Views are handled in the physical phase so collecting a batch has no side effects. + continue; + } + if let Some(TableRouteValue::Logical(route)) = route { + let physical = physical_infos + .get(&route.physical_table_id()) + .with_context(|| TableInfoNotFoundSnafu { + table: route.physical_table_id().to_string(), + })?; + // Metadata-only deletion is safe only when the backing table is dropped with us. + ensure!( + physical.table_info.catalog_name == ctx.catalog + && physical.table_info.schema_name == ctx.schema, + UnexpectedSnafu { + err_msg: format!( + "Logical table {} references physical table {} (id {}) outside the dropped database", + format_full_table_name(&ctx.catalog, &ctx.schema, &name), + format_full_table_name( + &physical.table_info.catalog_name, + &physical.table_info.schema_name, + &physical.table_info.name, + ), + route.physical_table_id(), + ), + } + ); + tables.push((id, TableName::new(&ctx.catalog, &ctx.schema, name))); + } + } + Ok(tables) + } + fn handle_reach_end( &mut self, ctx: &mut DropDatabaseContext, @@ -62,32 +182,12 @@ impl DropDatabaseCursor { async fn handle_table( &mut self, - ddl_ctx: &DdlContext, ctx: &mut DropDatabaseContext, table_name: String, table_id: TableId, table_route_value: TableRouteValue, ) -> Result<(Box, Status)> { match (self.target, table_route_value) { - (DropTableTarget::Logical, TableRouteValue::Logical(route)) => { - let physical_table_id = route.physical_table_id(); - - let (_, table_route) = ddl_ctx - .table_metadata_manager - .table_route_manager() - .get_physical_table_route(physical_table_id) - .await?; - Ok(( - Box::new(DropDatabaseExecutor::new( - table_id, - table_id, - TableName::new(&ctx.catalog, &ctx.schema, &table_name), - table_route.region_routes, - self.target, - )), - Status::executing(true), - )) - } (DropTableTarget::Physical, TableRouteValue::Physical(table_route)) => Ok(( Box::new(DropDatabaseExecutor::new( table_id, @@ -104,7 +204,6 @@ impl DropDatabaseCursor { )), } } - async fn handle_view( &self, ddl_ctx: &DdlContext, @@ -148,50 +247,61 @@ impl State for DropDatabaseCursor { ddl_ctx: &DdlContext, ctx: &mut DropDatabaseContext, ) -> Result<(Box, Status)> { - if ctx.tables.as_deref().is_none() { + if ctx.tables.is_none() { let tables = ddl_ctx .table_metadata_manager .table_name_manager() .tables(&ctx.catalog, &ctx.schema); - ctx.tables = Some(tables); + ctx.tables = Some(tables.peekable()); } - // Safety: must exist - match ctx.tables.as_mut().unwrap().try_next().await? { - Some((table_name, table_name_value)) => { - let table_id = table_name_value.table_id(); + let result = async { + if self.target == DropTableTarget::Logical { + return self.collect_logical_tables(ddl_ctx, ctx).await; + } + // Safety: must exist + match ctx.tables.as_mut().unwrap().try_next().await? { + Some((table_name, table_name_value)) => { + let table_id = table_name_value.table_id(); - let table_info_value = ddl_ctx - .table_metadata_manager - .table_info_manager() - .get(table_id) - .await? - .with_context(|| TableInfoNotFoundSnafu { - table: format_full_table_name(&ctx.catalog, &ctx.schema, &table_name), - })?; + let table_info_value = ddl_ctx + .table_metadata_manager + .table_info_manager() + .get(table_id) + .await? + .with_context(|| TableInfoNotFoundSnafu { + table: format_full_table_name(&ctx.catalog, &ctx.schema, &table_name), + })?; - if table_info_value.table_info.table_type == TableType::View { - return self.handle_view(ddl_ctx, ctx, table_name, table_id).await; - } + if table_info_value.table_info.table_type == TableType::View { + return self.handle_view(ddl_ctx, ctx, table_name, table_id).await; + } - match ddl_ctx - .table_metadata_manager - .table_route_manager() - .table_route_storage() - .get(table_id) - .await? - { - Some(table_route_value) => { - self.handle_table(ddl_ctx, ctx, table_name, table_id, table_route_value) - .await + match ddl_ctx + .table_metadata_manager + .table_route_manager() + .table_route_storage() + .get(table_id) + .await? + { + Some(table_route_value) => { + self.handle_table(ctx, table_name, table_id, table_route_value) + .await + } + None => Ok(( + Box::new(DropDatabaseCursor::new(self.target)), + Status::executing(false), + )), } - None => Ok(( - Box::new(DropDatabaseCursor::new(self.target)), - Status::executing(false), - )), } + None => self.handle_reach_end(ctx), } - None => self.handle_reach_end(ctx), } + .await; + if result.is_err() { + // The entry was consumed but no deletion plan was persisted. Rescan on retry. + ctx.tables.take(); + } + result } fn as_any(&self) -> &dyn Any { @@ -206,12 +316,163 @@ mod tests { use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME}; use crate::ddl::drop_database::cursor::DropDatabaseCursor; - use crate::ddl::drop_database::executor::DropDatabaseExecutor; + use crate::ddl::drop_database::executor::{DropDatabaseBatchExecutor, DropDatabaseExecutor}; use crate::ddl::drop_database::metadata::DropDatabaseRemoveMetadata; use crate::ddl::drop_database::{DropDatabaseContext, DropTableTarget, State}; use crate::ddl::test_util::{create_logical_table, create_physical_table}; use crate::test_util::{MockDatanodeManager, new_ddl_context}; + #[tokio::test] + async fn test_cursor_read_failure_rescans() { + use crate::key::table_name::TableNameValue; + use futures::StreamExt; + for target in [DropTableTarget::Logical, DropTableTarget::Physical] { + let ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(()))); + let physical = create_physical_table(&ddl, "physical").await; + let mut ctx = DropDatabaseContext { + catalog: DEFAULT_CATALOG_NAME.into(), + schema: DEFAULT_SCHEMA_NAME.into(), + drop_if_exists: false, + retrying: false, + tables: Some( + futures::stream::iter(vec![Ok(("missing".into(), TableNameValue::new(99999)))]) + .boxed() + .peekable(), + ), + }; + let mut cursor = DropDatabaseCursor::new(target); + assert!(cursor.next(&ddl, &mut ctx).await.is_err()); + assert!(ctx.tables.is_none()); + let (state, _) = cursor.next(&ddl, &mut ctx).await.unwrap(); + if target == DropTableTarget::Physical { + assert!(state.as_any().is::()); + } else { + assert!(state.as_any().is::()); + } + assert!( + ddl.table_metadata_manager + .table_info_manager() + .get(physical) + .await + .unwrap() + .is_some() + ); + } + } + + #[tokio::test] + async fn test_collect_page_oversized_singleton_makes_progress() { + use crate::key::table_name::TableNameValue; + use futures::StreamExt; + let mut ctx = DropDatabaseContext { + catalog: DEFAULT_CATALOG_NAME.into(), + schema: DEFAULT_SCHEMA_NAME.into(), + drop_if_exists: false, + retrying: false, + tables: Some( + futures::stream::iter(vec![ + Ok(( + "x".repeat(DropDatabaseCursor::MAX_BATCH_BYTES), + TableNameValue::new(1), + )), + Ok(("small".into(), TableNameValue::new(2))), + ]) + .boxed() + .peekable(), + ), + }; + let page = DropDatabaseCursor::collect_page(&mut ctx).await.unwrap(); + assert_eq!(page.len(), 1); + assert_eq!(page[0].1, 1); + assert_eq!( + DropDatabaseCursor::collect_page(&mut ctx).await.unwrap(), + vec![("small".into(), 2)] + ); + assert!( + DropDatabaseCursor::collect_page(&mut ctx) + .await + .unwrap() + .is_empty() + ); + } + + #[tokio::test] + async fn test_collect_page_limits_and_request_headroom() { + use common_procedure::store::ProcedureMessage; + use futures::StreamExt; + use table::table_name::TableName; + + use crate::ddl::drop_database::DropDatabaseProcedure; + use crate::key::TableMetadataManager; + use crate::key::table_name::TableNameValue; + for name in [ + "short".to_string(), + "x".repeat(64 * 1024), + "\"\\\n\u{0001}中文".repeat(4096), + ] { + let ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(()))); + let count = if name.len() < 1024 { 5000 } else { 65 }; + let entries = (0..count) + .map(|id| Ok((name.clone(), TableNameValue::new(id)))) + .collect::>(); + let mut ctx = DropDatabaseContext { + catalog: DEFAULT_CATALOG_NAME.into(), + schema: DEFAULT_SCHEMA_NAME.into(), + drop_if_exists: false, + tables: Some(futures::stream::iter(entries).boxed().peekable()), + retrying: false, + }; + let mut collected = Vec::new(); + loop { + let page = DropDatabaseCursor::collect_page(&mut ctx).await.unwrap(); + if page.is_empty() { + break; + } + let estimated = page + .iter() + .map(|(name, _)| { + DropDatabaseCursor::estimated_table_bytes(&ctx.catalog, &ctx.schema, name) + }) + .sum::(); + assert!(estimated <= DropDatabaseCursor::MAX_BATCH_BYTES); + let tables = page + .iter() + .map(|(name, id)| (*id, TableName::new(&ctx.catalog, &ctx.schema, name))) + .collect::>(); + // Each delete has one bytes field nested in two messages. Allow + // 32 bytes per key for tags and length varints, plus a txn envelope. + let delete_bytes = 64 + + tables + .iter() + .flat_map(|(id, name)| TableMetadataManager::table_identity_keys(*id, name)) + .map(|key| key.len() + 32) + .sum::(); + assert!(delete_bytes <= estimated); + let mut procedure = DropDatabaseProcedure::new( + ctx.catalog.clone(), + ctx.schema.clone(), + false, + ddl.clone(), + ); + procedure.state = Box::new(DropDatabaseBatchExecutor { tables }); + let message = ProcedureMessage { + type_name: DropDatabaseProcedure::TYPE_NAME.into(), + data: common_procedure::Procedure::dump(&procedure).unwrap(), + parent_id: None, + subprocedure_recovery: common_procedure::SubprocedureRecovery::Discard, + step: u32::MAX, + error: None, + context: Default::default(), + }; + let serialized = serde_json::to_vec(&message).unwrap(); + assert!(serialized.len() <= estimated); + assert!(serialized.len() + 512 * 1024 < 1536 * 1024); + collected.extend(page.into_iter().map(|(_, id)| id)); + } + assert_eq!(collected, (0..count).collect::>()); + } + } + #[tokio::test] async fn test_next_without_logical_tables() { let node_manager = Arc::new(MockDatanodeManager::new(())); @@ -267,16 +528,10 @@ mod tests { assert!(status.need_persist()); let executor = state .as_any() - .downcast_ref::() + .downcast_ref::() .unwrap(); - let (_, table_route) = ddl_context - .table_metadata_manager - .table_route_manager() - .get_physical_table_route(physical_table_id) - .await - .unwrap(); - assert_eq!(table_route.region_routes, executor.physical_region_routes); - assert_eq!(executor.target, DropTableTarget::Logical); + assert_eq!(executor.tables.len(), 1); + assert_eq!(executor.tables[0].1.table_name, "metric_0"); } #[tokio::test] diff --git a/src/common/meta/src/ddl/drop_database/executor.rs b/src/common/meta/src/ddl/drop_database/executor.rs index 1c01060c5ed..84d2eafe5fd 100644 --- a/src/common/meta/src/ddl/drop_database/executor.rs +++ b/src/common/meta/src/ddl/drop_database/executor.rs @@ -21,12 +21,14 @@ use snafu::OptionExt; use table::metadata::TableId; use table::table_name::TableName; +use crate::cache_invalidator::Context; use crate::ddl::DdlContext; use crate::ddl::drop_database::cursor::DropDatabaseCursor; use crate::ddl::drop_database::{DropDatabaseContext, DropTableTarget, State}; use crate::ddl::drop_table::executor::DropTableExecutor; use crate::ddl::utils::get_region_wal_options; use crate::error::{self, Result}; +use crate::instruction::CacheIdent; use crate::key::table_route::TableRouteValue; use crate::region_keeper::OperatingRegionGuard; use crate::rpc::router::{RegionRoute, operating_leader_region_roles}; @@ -140,7 +142,7 @@ impl State for DropDatabaseExecutor { &ddl_ctx.leader_region_registry, &self.physical_region_routes, true, - false, + self.target == DropTableTarget::Physical, false, ) .await?; @@ -157,6 +159,54 @@ impl State for DropDatabaseExecutor { } } +/// A durable logical-table deletion batch. Physical regions are removed in the next phase. +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct DropDatabaseBatchExecutor { + pub(crate) tables: Vec<(TableId, TableName)>, +} + +#[async_trait::async_trait] +#[typetag::serde] +impl State for DropDatabaseBatchExecutor { + async fn next( + &mut self, + ddl_ctx: &DdlContext, + _ctx: &mut DropDatabaseContext, + ) -> Result<(Box, Status)> { + ddl_ctx + .table_metadata_manager + .destroy_logical_tables_metadata(&self.tables) + .await?; + let idents = self + .tables + .iter() + .flat_map(|(id, name)| { + [ + CacheIdent::TableId(*id), + CacheIdent::TableName(name.clone()), + ] + }) + .collect::>(); + ddl_ctx + .cache_invalidator + .invalidate( + &Context { + subject: Some("Invalidate logical tables by dropping database".to_string()), + }, + &idents, + ) + .await?; + Ok(( + Box::new(DropDatabaseCursor::new(DropTableTarget::Logical)), + Status::executing(false), + )) + } + + fn as_any(&self) -> &dyn Any { + self + } +} + #[cfg(test)] mod tests { use std::collections::HashSet; @@ -448,10 +498,13 @@ mod tests { state.recover(&ddl_context).unwrap(); - let roles = ddl_context + let operating_regions = ddl_context .memory_region_keeper - .extract_operating_region_roles(7, &HashSet::from([region_id])); - assert_eq!(roles.get(®ion_id), Some(&RegionRole::DowngradingLeader)); + .extract_operating_regions(7, &HashSet::from([region_id])); + assert_eq!( + operating_regions.get(®ion_id).map(|region| region.role), + Some(RegionRole::DowngradingLeader) + ); } #[tokio::test] diff --git a/src/common/meta/src/ddl/tests/alter_logical_tables.rs b/src/common/meta/src/ddl/tests/alter_logical_tables.rs index 619e02bb9b9..4dd915d1505 100644 --- a/src/common/meta/src/ddl/tests/alter_logical_tables.rs +++ b/src/common/meta/src/ddl/tests/alter_logical_tables.rs @@ -713,21 +713,27 @@ async fn test_on_prepare_rejects_table_outside_locked_set() { let phy_id = create_physical_table(&ddl_context, "phy").await; create_logical_table(ddl_context.clone(), phy_id, "table1").await; - let tasks = vec![make_alter_logical_table_add_column_task( - None, - "table1", - vec!["column1".to_string()], - )]; - // The table resolved at prepare time is not in the locked set, as if it // had been dropped and recreated after submission. let stale_locked_id = phy_id + 1000; - let mut procedure = - AlterLogicalTablesProcedure::new(tasks, phy_id, vec![stale_locked_id], ddl_context); - let err = procedure.on_prepare().await.unwrap_err(); - assert!( - err.to_string() - .contains("not covered by the procedure locks"), - "{err}" - ); + // A completed ADD must also reject rebinding, even when no schema changes are needed. + for column in ["column1", "host"] { + let tasks = vec![make_alter_logical_table_add_column_task( + None, + "table1", + vec![column.into()], + )]; + let mut procedure = AlterLogicalTablesProcedure::new( + tasks, + phy_id, + vec![stale_locked_id], + ddl_context.clone(), + ); + let err = procedure.on_prepare().await.unwrap_err(); + assert!( + err.to_string() + .contains("not covered by the procedure locks"), + "{err}" + ); + } } diff --git a/src/common/meta/src/ddl/tests/create_logical_tables.rs b/src/common/meta/src/ddl/tests/create_logical_tables.rs index 89ec486ee03..a74ef63403e 100644 --- a/src/common/meta/src/ddl/tests/create_logical_tables.rs +++ b/src/common/meta/src/ddl/tests/create_logical_tables.rs @@ -23,11 +23,17 @@ use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; use common_procedure::{Context as ProcedureContext, Procedure, ProcedureId, Status}; use common_procedure_test::MockContextProvider; +use datatypes::prelude::ConcreteDataType; +use datatypes::schema::{ColumnSchema, Schema}; use store_api::metadata::ColumnMetadata; -use store_api::metric_engine_consts::{ALTER_PHYSICAL_EXTENSION_KEY, MANIFEST_INFO_EXTENSION_KEY}; +use store_api::metric_engine_consts::{ + ALTER_PHYSICAL_EXTENSION_KEY, DATA_SCHEMA_TABLE_ID_COLUMN_NAME, MANIFEST_INFO_EXTENSION_KEY, + is_metric_engine_internal_column, +}; use store_api::region_engine::RegionManifestInfo; use store_api::storage::RegionId; use store_api::storage::consts::ReservedColumnId; +use table::metadata::TableInfo; use tokio::sync::mpsc; use crate::ddl::TableMetadata; @@ -566,3 +572,136 @@ async fn test_on_submit_create_request() { }) ); } + +/// Appends a column to the table schema and returns its index. +fn push_column( + table_info: &mut TableInfo, + name: &str, + data_type: ConcreteDataType, + is_nullable: bool, +) -> usize { + let mut columns = table_info.meta.schema.column_schemas().to_vec(); + columns.push(ColumnSchema::new(name, data_type, is_nullable)); + table_info.meta.schema = Arc::new(Schema::new(columns)); + table_info.meta.schema.column_schemas().len() - 1 +} + +#[tokio::test] +async fn test_on_prepare_skips_internal_partition_columns() { + let (tx, mut rx) = mpsc::channel(8); + let node_manager = Arc::new(MockDatanodeManager::new(DatanodeWatcher::new(tx))); + let ddl_context = new_ddl_context(node_manager); + + // The physical table owns the engine identity column and has been repartitioned by + // both it and a user tag that no logical table declares. + let mut create_physical_table_task = test_create_physical_table_task("phy_table"); + let table_id_column_index = push_column( + &mut create_physical_table_task.table_info, + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, + ConcreteDataType::uint32_datatype(), + false, + ); + let extra_tag_column_index = push_column( + &mut create_physical_table_task.table_info, + "extra_tag", + ConcreteDataType::string_datatype(), + true, + ); + create_physical_table_task + .table_info + .meta + .partition_key_indices = vec![extra_tag_column_index, table_id_column_index]; + + let TableMetadata { + table_id, + table_route, + .. + } = ddl_context + .table_metadata_allocator + .create(&create_physical_table_task) + .await + .unwrap(); + create_physical_table_task.set_table_id(table_id); + create_physical_table_metadata( + &ddl_context, + create_physical_table_task.table_info.clone(), + TableRouteValue::Physical(table_route), + ) + .await; + + let mut procedure = CreateLogicalTablesProcedure::new( + vec![test_create_logical_table_task("foo")], + table_id, + ddl_context.clone(), + ); + let status = procedure.on_prepare().await.unwrap(); + assert_matches!( + status, + Status::Executing { + persist: true, + clean_poisons: false + } + ); + + let ctx = ProcedureContext { + procedure_id: ProcedureId::random(), + provider: Arc::new(MockContextProvider::default()), + event_context: None, + }; + procedure.execute(&ctx).await.unwrap(); + let status = procedure.execute(&ctx).await.unwrap(); + let table_ids = status.downcast_output_ref::>().unwrap().clone(); + + // The engine rejects a request that occupies a reserved identity column. + let mut creates = None; + while let Ok((_, request)) = rx.try_recv() { + if let Some(region_request::Body::Creates(body)) = request.body { + creates = Some(body); + } + } + let creates = creates.unwrap(); + assert_eq!(creates.requests.len(), 1); + let requested_columns = creates.requests[0] + .column_defs + .iter() + .map(|def| def.column_def.as_ref().unwrap().name.as_str()) + .collect::>(); + assert!( + !requested_columns + .iter() + .any(|name| is_metric_engine_internal_column(name)), + "logical region request must not occupy internal columns: {requested_columns:?}" + ); + assert!(requested_columns.contains(&"extra_tag")); + + let logical_table_info = get_raw_table_info(&ddl_context, table_ids[0]).await; + let column_names = logical_table_info + .meta + .schema + .column_schemas() + .iter() + .map(|column| column.name.as_str()) + .collect::>(); + assert!( + !column_names.contains(&DATA_SCHEMA_TABLE_ID_COLUMN_NAME), + "internal identity column must not be merged into the logical schema: {column_names:?}" + ); + assert!(column_names.contains(&"extra_tag")); + + // The user partition column is still merged as a nullable string tag of the primary key. + let extra_tag_index = logical_table_info + .meta + .schema + .column_index_by_name("extra_tag") + .unwrap(); + assert!( + logical_table_info + .meta + .primary_key_indices + .contains(&extra_tag_index), + "merged partition column must be part of the primary key" + ); + let extra_tag = &logical_table_info.meta.schema.column_schemas()[extra_tag_index]; + assert_eq!(extra_tag.data_type, ConcreteDataType::string_datatype()); + assert!(extra_tag.is_nullable()); +} diff --git a/src/common/meta/src/ddl/tests/create_table.rs b/src/common/meta/src/ddl/tests/create_table.rs index d39346e02da..e1c880fbf32 100644 --- a/src/common/meta/src/ddl/tests/create_table.rs +++ b/src/common/meta/src/ddl/tests/create_table.rs @@ -382,10 +382,13 @@ async fn test_memory_region_keeper_guard_dropped_on_procedure_done() { .memory_region_keeper .contains(datanode_id, region_id) ); - let roles = ddl_context + let operating_regions = ddl_context .memory_region_keeper - .extract_operating_region_roles(datanode_id, &HashSet::from([region_id])); - assert_eq!(roles.get(®ion_id), Some(&RegionRole::Leader)); + .extract_operating_regions(datanode_id, &HashSet::from([region_id])); + assert_eq!( + operating_regions.get(®ion_id).map(|region| region.role), + Some(RegionRole::Leader) + ); execute_procedure_until_done(&mut procedure).await; diff --git a/src/common/meta/src/ddl/tests/drop_database.rs b/src/common/meta/src/ddl/tests/drop_database.rs index 05a70962bf0..049eb82e551 100644 --- a/src/common/meta/src/ddl/tests/drop_database.rs +++ b/src/common/meta/src/ddl/tests/drop_database.rs @@ -22,7 +22,7 @@ use common_procedure_test::{ use futures::TryStreamExt; use crate::ddl::drop_database::DropDatabaseProcedure; -use crate::ddl::drop_database::executor::DropDatabaseExecutor; +use crate::ddl::drop_database::executor::{DropDatabaseBatchExecutor, DropDatabaseExecutor}; use crate::ddl::test_util::datanode_handler::{NaiveDatanodeHandler, RetryErrorDatanodeHandler}; use crate::ddl::test_util::{create_logical_table, create_physical_table}; use crate::key::schema_name::SchemaNameKey; @@ -151,24 +151,15 @@ async fn test_drop_database_recover() { ddl_context.clone(), ); let num_operating_regions = 1; - // Before dropping the logical table + // Recover the persisted logical batch before its metadata is removed. execute_procedure_until(&mut procedure, |p| { - p.state() - .as_any() - .downcast_ref::() - .is_some() + p.state().as_any().is::() }) .await; - // Dump data let data = procedure.dump().unwrap(); - assert_eq!(ddl_context.memory_region_keeper.len(), 0); let mut procedure = DropDatabaseProcedure::from_json(&data, ddl_context.clone()).unwrap(); procedure.recover().unwrap(); - assert_eq!( - ddl_context.memory_region_keeper.len(), - num_operating_regions - ); - ddl_context.memory_region_keeper.clear(); + assert_eq!(ddl_context.memory_region_keeper.len(), 0); // Before dropping the physical table execute_procedure_until(&mut procedure, |p| { p.state() @@ -197,3 +188,304 @@ async fn test_drop_database_recover() { .unwrap(); assert!(tables.is_empty()); } + +#[tokio::test] +async fn test_drop_database_batches_recover_after_metadata_deletion() { + use crate::ddl::test_util::datanode_handler::DatanodeWatcher; + use api::v1::region::region_request::Body; + use store_api::storage::RegionId; + + // A full batch, multiple batches, and a trailing partial batch. + for (count, name_padding, expected_batches) in + [(0, 0, 0), (1, 0, 1), (64, 0, 1), (129, 0, 1), (20, 8192, 2)] + { + let (tx, mut rx) = tokio::sync::mpsc::channel(16); + let ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(DatanodeWatcher::new(tx)))); + ddl.table_metadata_manager + .schema_manager() + .create( + SchemaNameKey::new(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME), + None, + false, + ) + .await + .unwrap(); + let physical = create_physical_table(&ddl, "z_physical").await; + let mut ids = Vec::new(); + for i in 0..count { + create_logical_table( + ddl.clone(), + physical, + &format!("logical_{i:04}{}", "x".repeat(name_padding)), + ) + .await; + } + let mut procedure = DropDatabaseProcedure::new( + DEFAULT_CATALOG_NAME.into(), + DEFAULT_SCHEMA_NAME.into(), + false, + ddl.clone(), + ); + let ctx = ProcedureContext { + procedure_id: ProcedureId::random(), + provider: Arc::new(MockContextProvider::default()), + event_context: None, + }; + let mut batches = 0; + loop { + if let Some(batch) = procedure + .state() + .as_any() + .downcast_ref::() + { + ids.extend(batch.tables.iter().map(|(id, _)| *id)); + // Crash after keys are gone but before cache invalidation and cursor advancement. + let dump = procedure.dump().unwrap(); + ddl.table_metadata_manager + .destroy_logical_tables_metadata(&batch.tables) + .await + .unwrap(); + procedure = DropDatabaseProcedure::from_json(&dump, ddl.clone()).unwrap(); + procedure.recover().unwrap(); + batches += 1; + } + if procedure.execute(&ctx).await.unwrap().is_done() { + break; + } + } + assert_eq!(ids.len(), count); + assert_eq!(batches, expected_batches); + let (_, request) = rx.try_recv().unwrap(); + let Some(Body::Drop(drop)) = request.body else { + panic!("expected physical drop") + }; + assert_eq!(RegionId::from_u64(drop.region_id).table_id(), physical); + assert!(drop.force); + assert!( + rx.try_recv().is_err(), + "logical tables must not send region RPCs" + ); + assert!( + ddl.table_metadata_manager + .table_info_manager() + .batch_get(&ids) + .await + .unwrap() + .is_empty() + ); + assert!( + ddl.table_metadata_manager + .table_name_manager() + .tables(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME) + .try_collect::>() + .await + .unwrap() + .is_empty() + ); + } +} + +#[tokio::test] +async fn test_drop_database_legacy_state_fixture() { + let ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler))); + // Single-table executor state from f8d8fe1071, before logical-table batching. + let fixture = r#"{ + "catalog": "greptime", + "schema": "public", + "drop_if_exists": false, + "state": { + "drop_database_state": "DropDatabaseExecutor", + "table_id": 1025, + "physical_table_id": 1025, + "table_name": { + "catalog_name": "greptime", + "schema_name": "public", + "table_name": "logical" + }, + "physical_region_routes": [], + "target": "Logical" + } +}"#; + let mut procedure = DropDatabaseProcedure::from_json(fixture, ddl).unwrap(); + procedure.recover().unwrap(); + assert!(procedure.state().as_any().is::()); + assert_eq!( + serde_json::from_str::(fixture).unwrap(), + serde_json::from_str::(&procedure.dump().unwrap()).unwrap() + ); +} + +#[tokio::test] +async fn test_drop_database_batch_cache_failure_recovery() { + use crate::cache_invalidator::{CacheInvalidator, Context}; + use crate::instruction::CacheIdent; + use std::sync::Mutex; + use std::sync::atomic::{AtomicBool, Ordering}; + + struct FailingInvalidator { + fail: AtomicBool, + calls: Mutex>>, + } + #[async_trait::async_trait] + impl CacheInvalidator for FailingInvalidator { + async fn invalidate(&self, _: &Context, caches: &[CacheIdent]) -> crate::error::Result<()> { + self.calls.lock().unwrap().push(caches.to_vec()); + if self.fail.swap(false, Ordering::Relaxed) { + return crate::error::UnexpectedSnafu { + err_msg: "injected cache failure", + } + .fail(); + } + Ok(()) + } + fn invalidate_all(&self) -> crate::error::Result<()> { + // This test invalidator owns only call records, not cached metadata. + Ok(()) + } + } + let invalidator = Arc::new(FailingInvalidator { + fail: AtomicBool::new(false), + calls: Mutex::new(Vec::new()), + }); + let mut ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler))); + ddl.cache_invalidator = invalidator.clone(); + ddl.table_metadata_manager + .schema_manager() + .create( + SchemaNameKey::new(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME), + None, + false, + ) + .await + .unwrap(); + let physical = create_physical_table(&ddl, "physical").await; + create_logical_table(ddl.clone(), physical, "logical").await; + let mut procedure = DropDatabaseProcedure::new( + DEFAULT_CATALOG_NAME.into(), + DEFAULT_SCHEMA_NAME.into(), + false, + ddl.clone(), + ); + execute_procedure_until(&mut procedure, |p| { + p.state().as_any().is::() + }) + .await; + let dump = procedure.dump().unwrap(); + invalidator.calls.lock().unwrap().clear(); + invalidator.fail.store(true, Ordering::Relaxed); + let ctx = ProcedureContext { + procedure_id: ProcedureId::random(), + provider: Arc::new(MockContextProvider::default()), + event_context: None, + }; + assert!(procedure.execute(&ctx).await.is_err()); + assert_eq!(procedure.dump().unwrap(), dump); + let mut restored = DropDatabaseProcedure::from_json(&dump, ddl).unwrap(); + restored.recover().unwrap(); + restored.execute(&ctx).await.unwrap(); + let calls = invalidator.calls.lock().unwrap(); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0], calls[1]); + assert_eq!(calls[0].len(), 2); +} + +#[tokio::test] +async fn test_drop_database_mixed_table_order_and_recovery() { + use crate::ddl::test_util::create_table::test_create_table_task; + use crate::ddl::tests::create_view::{test_create_view_task, test_table_names}; + use crate::key::table_route::TableRouteValue; + use itertools::Itertools; + use std::collections::HashMap; + + // Every relative ordering of metric physical, logical, mito and view names. + for names in ["a", "b", "c", "d"].into_iter().permutations(4) { + for recover in [false, true] { + let ddl = new_ddl_context(Arc::new(MockDatanodeManager::new(NaiveDatanodeHandler))); + let manager = &ddl.table_metadata_manager; + manager + .schema_manager() + .create( + SchemaNameKey::new(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME), + None, + false, + ) + .await + .unwrap(); + let physical = create_physical_table(&ddl, names[0]).await; + let logical = create_logical_table(ddl.clone(), physical, names[1]).await; + let mito = test_create_table_task(names[2], 20000); + manager + .create_table_metadata( + mito.table_info, + TableRouteValue::physical(vec![]), + HashMap::new(), + ) + .await + .unwrap(); + let mut view = test_create_view_task(names[3]); + view.view_info.ident.table_id = 20001; + manager + .create_view_metadata( + view.view_info, + view.create_view.logical_plan, + test_table_names(), + vec![], + vec![], + "definition".into(), + ) + .await + .unwrap(); + let mut procedure = DropDatabaseProcedure::new( + DEFAULT_CATALOG_NAME.into(), + DEFAULT_SCHEMA_NAME.into(), + false, + ddl.clone(), + ); + let ctx = ProcedureContext { + procedure_id: ProcedureId::random(), + provider: Arc::new(MockContextProvider::default()), + event_context: None, + }; + let mut done = false; + for _ in 0..40 { + let status = procedure.execute(&ctx).await.unwrap(); + if status.is_done() { + done = true; + break; + } + if recover && status.need_persist() { + let dump = procedure.dump().unwrap(); + drop(procedure); + procedure = DropDatabaseProcedure::from_json(&dump, ddl.clone()).unwrap(); + procedure.recover().unwrap(); + } + } + assert!(done, "{names:?}, recovery={recover}"); + assert!( + manager + .table_name_manager() + .tables(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME) + .try_collect::>() + .await + .unwrap() + .is_empty() + ); + assert!( + manager + .table_info_manager() + .batch_get(&[physical, logical, 20000, 20001]) + .await + .unwrap() + .is_empty() + ); + assert!( + manager + .view_info_manager() + .get(20001) + .await + .unwrap() + .is_none() + ); + } + } +} diff --git a/src/common/meta/src/ddl/tests/drop_table.rs b/src/common/meta/src/ddl/tests/drop_table.rs index a278b951dd5..40505c28fa2 100644 --- a/src/common/meta/src/ddl/tests/drop_table.rs +++ b/src/common/meta/src/ddl/tests/drop_table.rs @@ -2782,10 +2782,13 @@ async fn test_memory_region_keeper_guard_dropped_on_procedure_done() { .memory_region_keeper .contains(datanode_id, region_id) ); - let roles = ddl_context + let operating_regions = ddl_context .memory_region_keeper - .extract_operating_region_roles(datanode_id, &HashSet::from([region_id])); - assert_eq!(roles.get(®ion_id), Some(&RegionRole::Leader)); + .extract_operating_regions(datanode_id, &HashSet::from([region_id])); + assert_eq!( + operating_regions.get(®ion_id).map(|region| region.role), + Some(RegionRole::Leader) + ); execute_procedure_until_done(&mut procedure).await; diff --git a/src/common/meta/src/ddl_manager.rs b/src/common/meta/src/ddl_manager.rs index 5078328443d..87a0ddae91b 100644 --- a/src/common/meta/src/ddl_manager.rs +++ b/src/common/meta/src/ddl_manager.rs @@ -625,13 +625,22 @@ impl DdlManager { &self, alter_table_tasks: Vec, physical_table_id: TableId, + expected_table_ids: Option>, procedure_context: ProcedureContext, ) -> Result<(ProcedureId, Option)> { let context = self.create_context(); // Resolve the logical table ids up front: procedure locks are fixed // at submission, so `lock_key` cannot derive them during `Prepare`. - let logical_table_ids = { + let logical_table_ids = if let Some(ids) = expected_table_ids { + ensure!( + ids.len() == alter_table_tasks.len(), + error::AlterLogicalTablesInvalidArgumentsSnafu { + err_msg: "Expected one pinned table ID per logical ALTER task", + } + ); + ids + } else { let table_refs = alter_table_tasks .iter() .map(|task| task.table_ref()) @@ -1043,8 +1052,13 @@ impl DdlManager { .await } AlterLogicalTables(alter_table_tasks) => { - handle_alter_logical_table_tasks(self, alter_table_tasks, procedure_context) - .await + handle_alter_logical_table_tasks( + self, + alter_table_tasks, + query_context, + procedure_context, + ) + .await } DropLogicalTables(_) => todo!(), CreateDatabase(create_database_task) => { @@ -1598,6 +1612,7 @@ async fn handle_create_trigger_task( async fn handle_alter_logical_table_tasks( ddl_manager: &DdlManager, alter_table_tasks: Vec, + query_context: QueryContext, procedure_context: ProcedureContext, ) -> Result { ensure!( @@ -1616,9 +1631,15 @@ async fn handle_alter_logical_table_tasks( let physical_table_id = utils::get_physical_table_id(ddl_manager.table_metadata_manager(), first_table).await?; let num_logical_tables = alter_table_tasks.len(); + let expected_table_ids = query_context.pinned_logical_table_ids()?; let (id, _) = ddl_manager - .submit_alter_logical_table_tasks(alter_table_tasks, physical_table_id, procedure_context) + .submit_alter_logical_table_tasks( + alter_table_tasks, + physical_table_id, + expected_table_ids, + procedure_context, + ) .await?; info!( @@ -2061,6 +2082,59 @@ mod tests { ) } + #[tokio::test] + async fn test_alter_logical_tables_preserves_caller_pinned_ids() { + use crate::ddl::test_util::alter_table::TestAlterTableExprBuilder; + use crate::ddl::test_util::{create_logical_table, create_physical_table}; + use crate::rpc::ddl::ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY; + + let manager = build_soft_drop_test_ddl_manager(); + manager.register_loaders().unwrap(); + manager.procedure_manager.start().await.unwrap(); + let context = manager.create_context(); + let physical_id = create_physical_table(&context, "physical").await; + let logical_id = create_logical_table(context, physical_id, "logical").await; + let expr: api::v1::AlterTableExpr = TestAlterTableExprBuilder::default() + .table_name("logical") + .add_columns(vec![api::v1::ColumnDef { + name: "host".into(), + data_type: api::v1::ColumnDataType::String as i32, + semantic_type: api::v1::SemanticType::Tag as i32, + is_nullable: true, + ..Default::default() + }]) + .add_if_not_exists(true) + .build() + .unwrap() + .into(); + for (ids, succeeds) in [ + (Some((logical_id + 1000).to_string()), false), + (Some(format!("{logical_id},{logical_id}")), false), + (Some(logical_id.to_string()), true), + (None, true), + ] { + let mut context = QueryContext::default(); + if let Some(ids) = ids { + context + .extensions + .insert(ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY.into(), ids); + } + let result = manager + .submit_ddl_task( + ExecutorContext { + query_context: Some(context), + ..Default::default() + }, + SubmitDdlTaskRequest::new(DdlTask::new_alter_logical_tables(vec![ + expr.clone(), + ])), + ) + .await; + assert_eq!(result.is_ok(), succeeds, "{result:?}"); + } + manager.procedure_manager.stop().await.unwrap(); + } + #[cfg(feature = "enterprise")] #[tokio::test] async fn test_reserved_sequence_range_is_rejected_before_custom_handler() { diff --git a/src/common/meta/src/instruction.rs b/src/common/meta/src/instruction.rs index 0291f3a0946..2fba5a24f48 100644 --- a/src/common/meta/src/instruction.rs +++ b/src/common/meta/src/instruction.rs @@ -388,6 +388,9 @@ pub struct UpgradeRegion { pub replay_entry_id: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub metadata_replay_entry_id: Option, + /// The candidate lease id of the migration's registration of this target region. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub candidate_lease_id: Option, } impl UpgradeRegion { @@ -408,6 +411,9 @@ impl UpgradeRegion { /// The identifier of cache. pub enum CacheIdent { FlowId(FlowId), + /// Indicates a change of trigger metadata; invalidated by enterprise plugins. + #[cfg(feature = "enterprise")] + TriggerId(u64), /// Indicate change of address of flownode. FlowNodeAddressChange(u64), FlowName(FlowName), @@ -1559,6 +1565,7 @@ mod tests { location_id: None, replay_entry_id: None, metadata_replay_entry_id: None, + candidate_lease_id: None, }]); let serialized = serde_json::to_string(&upgrade_region).unwrap(); @@ -1656,10 +1663,27 @@ mod tests { location_id: None, replay_entry_id: None, metadata_replay_entry_id: None, + candidate_lease_id: None, }]); assert_eq!(upgrade_region_instruction, upgrade_region); } + #[test] + fn test_upgrade_region_candidate_lease_id_json() { + let upgrade_region = UpgradeRegion { + region_id: RegionId::new(1024, 1), + replay_timeout: Duration::from_millis(1000), + candidate_lease_id: Some(7), + ..Default::default() + }; + let json = serde_json::to_value(&upgrade_region).unwrap(); + assert_eq!(json["candidate_lease_id"], 7); + assert_eq!( + serde_json::from_value::(json).unwrap(), + upgrade_region + ); + } + #[test] fn test_deserialize_instruction_reply() { // legacy close region reply diff --git a/src/common/meta/src/key.rs b/src/common/meta/src/key.rs index 189c427e9ec..61b171b8bfe 100644 --- a/src/common/meta/src/key.rs +++ b/src/common/meta/src/key.rs @@ -994,6 +994,26 @@ impl TableMetadataManager { Ok(()) } + const TABLE_IDENTITY_KEYS: usize = 4; + + /// Keys shared by physical and logical tables, independent of region placement. + pub(crate) fn table_identity_keys( + table_id: TableId, + table_name: &TableName, + ) -> [Vec; Self::TABLE_IDENTITY_KEYS] { + [ + TableNameKey::new( + &table_name.catalog_name, + &table_name.schema_name, + &table_name.table_name, + ) + .to_bytes(), + TableInfoKey::new(table_id).to_bytes(), + TableRouteKey::new(table_id).to_bytes(), + TableRepartKey::new(table_id).to_bytes(), + ] + } + fn table_metadata_keys( &self, table_id: TableId, @@ -1009,15 +1029,7 @@ impl TableMetadataManager { } else { vec![] }; - let mut keys = Vec::with_capacity(3 + datanode_ids.len()); - let table_name = TableNameKey::new( - &table_name.catalog_name, - &table_name.schema_name, - &table_name.table_name, - ); - let table_info_key = TableInfoKey::new(table_id); - let table_route_key = TableRouteKey::new(table_id); - let table_repart_key = TableRepartKey::new(table_id); + let mut keys = Vec::from(Self::table_identity_keys(table_id, table_name)); let datanode_table_keys = datanode_ids .into_iter() .map(|datanode_id| DatanodeTableKey::new(datanode_id, table_id)) @@ -1029,10 +1041,6 @@ impl TableMetadataManager { .iter() .map(|(region_id, topic)| TopicRegionKey::new(*region_id, topic)) .collect::>(); - keys.push(table_name.to_bytes()); - keys.push(table_info_key.to_bytes()); - keys.push(table_route_key.to_bytes()); - keys.push(table_repart_key.to_bytes()); for key in &datanode_table_keys { keys.push(key.to_bytes()); } @@ -1321,6 +1329,30 @@ impl TableMetadataManager { Ok(()) } + /// Permanently deletes a batch of logical tables. The caller holds their schema write lock + /// and persists the deletion plan before calling, since backend requests may partially succeed. + pub(crate) async fn destroy_logical_tables_metadata( + &self, + tables: &[(TableId, TableName)], + ) -> Result<()> { + let max_ops = self.kv_backend.max_txn_ops().max(1); + let tables_per_request = (max_ops / Self::TABLE_IDENTITY_KEYS).max(1); + // Keep a table's keys together whenever they fit in one backend request. A backend + // limit smaller than a single table still requires splitting, as with single-table drop. + for tables in tables.chunks(tables_per_request) { + let keys = tables + .iter() + .flat_map(|(id, name)| Self::table_identity_keys(*id, name)) + .collect::>(); + for keys in keys.chunks(max_ops) { + self.kv_backend + .batch_delete(BatchDeleteRequest::new().with_keys(keys.to_vec())) + .await?; + } + } + Ok(()) + } + /// Rebuilds dropped table metadata from tombstoned keys. async fn get_dropped_table_metadata( &self, @@ -3094,6 +3126,79 @@ mod tests { ); } + #[tokio::test] + async fn test_destroy_logical_tables_metadata_limits_and_partial_failure() { + use crate::kv_backend::test_util::MockKvBackendBuilder; + use crate::rpc::store::{BatchDeleteRequest, BatchDeleteResponse}; + use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; + + for limit in [1, 3, 127, 128, 129] { + let tables = (0..65) + .map(|id| { + ( + id, + TableName::new("greptime", "public", format!("logical_{id}")), + ) + }) + .collect::>(); + let remaining = Arc::new(Mutex::new( + tables + .iter() + .flat_map(|(id, name)| TableMetadataManager::table_identity_keys(*id, name)) + .collect::>(), + )); + let requests = Arc::new(AtomicUsize::new(0)); + let backend = MockKvBackendBuilder::default() + .max_txn_ops(limit) + .batch_delete_fn({ + let remaining = remaining.clone(); + let requests = requests.clone(); + Arc::new(move |req: BatchDeleteRequest| { + assert!(!req.keys.is_empty()); + assert!(req.keys.len() <= limit); + if limit >= 4 { + assert_eq!(req.keys.len() % 4, 0, "single-table keys stay together"); + } + for key in &req.keys { + remaining.lock().unwrap().remove(key); + } + // Simulate a response lost after the second chunk committed. + if requests.fetch_add(1, Ordering::Relaxed) == 1 { + return crate::error::UnexpectedSnafu { + err_msg: "injected lost response", + } + .fail(); + } + Ok(BatchDeleteResponse::default()) + }) + }) + .build() + .unwrap(); + let manager = TableMetadataManager::new(Arc::new(backend)); + assert!( + manager + .destroy_logical_tables_metadata(&tables) + .await + .is_err() + ); + assert!(!remaining.lock().unwrap().is_empty()); + manager + .destroy_logical_tables_metadata(&tables) + .await + .unwrap(); + assert!(remaining.lock().unwrap().is_empty()); + assert_eq!( + requests.load(Ordering::Relaxed), + 2 + if limit < 4 { + 65_usize * 4_usize.div_ceil(limit) + } else { + 65_usize.div_ceil(limit / 4) + } + ); + } + } + #[tokio::test] async fn test_destroy_table_metadata() { let mem_kv = Arc::new(MemoryKvBackend::default()); diff --git a/src/common/meta/src/kv_backend.rs b/src/common/meta/src/kv_backend.rs index 1baa86f9ef1..171241bcd90 100644 --- a/src/common/meta/src/kv_backend.rs +++ b/src/common/meta/src/kv_backend.rs @@ -48,6 +48,8 @@ pub type KvBackendRef = Arc + Send + Sync>; #[async_trait] pub trait ClosableKvBackend: KvBackend { /// Releases the backend's resources; any later use fails. + /// May wait for in-flight operations to release their resources, so callers must + /// stop consumers first. Closing is terminal, not a pause for role transitions. async fn close(&self) -> Result<(), Error>; } diff --git a/src/common/meta/src/kv_backend/rds.rs b/src/common/meta/src/kv_backend/rds.rs index 1e94741dd8e..cb2dc47ef06 100644 --- a/src/common/meta/src/kv_backend/rds.rs +++ b/src/common/meta/src/kv_backend/rds.rs @@ -26,7 +26,7 @@ use crate::error::{Error, RdsTransactionRetryFailedSnafu, Result, UnexpectedSnaf use crate::kv_backend::txn::{ Compare, Txn as KvTxn, TxnOp, TxnOpResponse, TxnResponse as KvTxnResponse, }; -use crate::kv_backend::{KvBackend, TxnService}; +use crate::kv_backend::{ClosableKvBackend, KvBackend, TxnService}; use crate::metrics::METRIC_META_TXN_REQUEST; use crate::rpc::KeyValue; use crate::rpc::store::{ @@ -121,6 +121,11 @@ pub trait Transaction<'a>: Send + Sync { /// Factory for creating default and transaction query executors. #[async_trait::async_trait] pub trait ExecutorFactory: Send + Sync { + /// Closes the connection pool to new acquisitions. This affects all shared pool handles. + /// MySQL waits for checked-out connections to be returned; PostgreSQL allows them to + /// finish and releases them when returned. Stop consumers before closing the backend. + async fn close(&self); + async fn default_executor(&self) -> Result; async fn txn_executor<'a>(&self, default_executor: &'a mut T) -> Result>; @@ -585,6 +590,20 @@ where } } +#[async_trait::async_trait] +impl ClosableKvBackend for RdsStore +where + R: 'static, + Self: KvQueryExecutor + Send + Sync, + T: Executor + 'static, + S: ExecutorFactory + 'static, +{ + async fn close(&self) -> Result<()> { + self.executor_factory.close().await; + Ok(()) + } +} + /// Checks if the transaction operations are the same type. fn check_txn_ops(txn_ops: &[TxnOp]) -> Result { if txn_ops.is_empty() { diff --git a/src/common/meta/src/kv_backend/rds/mysql.rs b/src/common/meta/src/kv_backend/rds/mysql.rs index c933bb345f6..6c314bc69e4 100644 --- a/src/common/meta/src/kv_backend/rds/mysql.rs +++ b/src/common/meta/src/kv_backend/rds/mysql.rs @@ -27,7 +27,7 @@ use strum::AsRefStr; use crate::error::{ CreateMySqlPoolSnafu, MySqlExecutionSnafu, MySqlTransactionSnafu, Result, UnexpectedSnafu, }; -use crate::kv_backend::KvBackendRef; +use crate::kv_backend::ClosableKvBackendRef; use crate::kv_backend::rds::{ Executor, ExecutorFactory, ExecutorImpl, KvQueryExecutor, RDS_STORE_OP_BATCH_DELETE, RDS_STORE_OP_BATCH_GET, RDS_STORE_OP_BATCH_PUT, RDS_STORE_OP_RANGE_DELETE, @@ -348,6 +348,10 @@ pub struct MySqlExecutorFactory { #[async_trait::async_trait] impl ExecutorFactory for MySqlExecutorFactory { + async fn close(&self) { + self.pool.close().await; + } + async fn default_executor(&self) -> Result { Ok(self.pool.clone()) } @@ -631,8 +635,12 @@ impl MySqlStore { Ok(()) } - /// Create [MySqlStore] impl of [KvBackendRef] from url. - pub async fn with_url(url: &str, table_name: &str, max_txn_ops: usize) -> Result { + /// Create [MySqlStore] impl of [ClosableKvBackendRef] from url. + pub async fn with_url( + url: &str, + table_name: &str, + max_txn_ops: usize, + ) -> Result { ensure_rustls_crypto_provider_installed()?; let pool = MySqlPool::connect(url) .await @@ -640,12 +648,12 @@ impl MySqlStore { Self::with_mysql_pool(pool, table_name, max_txn_ops).await } - /// Create [MySqlStore] impl of [KvBackendRef] from [sqlx::Pool]. + /// Create [MySqlStore] impl of [ClosableKvBackendRef] from [sqlx::Pool]. pub async fn with_mysql_pool( pool: Pool, table_name: &str, max_txn_ops: usize, - ) -> Result { + ) -> Result { // This step ensures the mysql metadata backend is ready to use. // We check if greptime_metakv table exists, and we will create a new table // if it does not exist. @@ -690,6 +698,61 @@ mod tests { use crate::rpc::store::{PutRequest, RangeRequest}; use crate::test_util::test_certs_dir; + #[tokio::test] + async fn test_close_backend() { + let pool = + sqlx::mysql::MySqlPoolOptions::new().connect_lazy_with(MySqlConnectOptions::new()); + let backend: ClosableKvBackendRef = Arc::new(MySqlStore { + max_txn_ops: 128, + txn_retry_count: RDS_STORE_TXN_RETRY_COUNT, + executor_factory: MySqlExecutorFactory { + pool: Arc::new(pool.clone()), + }, + sql_template_set: MySqlTemplateFactory::new("close_test").build(), + _phantom: PhantomData, + }); + let shared_backend = backend.clone(); + assert!(!pool.is_closed()); + backend.close().await.unwrap(); + assert!(pool.is_closed()); + shared_backend.close().await.unwrap(); + assert!(matches!(pool.acquire().await, Err(sqlx::Error::PoolClosed))); + let result = tokio::time::timeout( + std::time::Duration::from_secs(1), + shared_backend.put(crate::rpc::store::PutRequest { + key: b"key".to_vec(), + value: b"value".to_vec(), + prev_kv: false, + }), + ) + .await + .expect("closed backend must reject writes promptly"); + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_close_backend_waits_for_checked_out_connection() { + maybe_skip_mysql_integration_test!(); + let pool = mysql_pool().await.unwrap(); + let table = new_test_table_name("close_held"); + let backend = MySqlStore::with_mysql_pool(pool.clone(), &table, 128) + .await + .unwrap(); + let connection = pool.acquire().await.unwrap(); + let close = backend.close(); + tokio::pin!(close); + + assert!(futures::poll!(close.as_mut()).is_pending()); + assert!(pool.is_closed()); + assert!(matches!(pool.acquire().await, Err(sqlx::Error::PoolClosed))); + drop(connection); + tokio::time::timeout(std::time::Duration::from_secs(5), close) + .await + .expect("close must finish after the connection is returned") + .unwrap(); + backend.close().await.unwrap(); + } + fn new_test_table_name(prefix: &str) -> String { let uuid = Uuid::new_v4().simple().to_string(); let max_prefix_len = 63usize.saturating_sub(uuid.len() + 1); diff --git a/src/common/meta/src/kv_backend/rds/postgres.rs b/src/common/meta/src/kv_backend/rds/postgres.rs index 4a2f8b13564..f48d21a5ab7 100644 --- a/src/common/meta/src/kv_backend/rds/postgres.rs +++ b/src/common/meta/src/kv_backend/rds/postgres.rs @@ -36,7 +36,7 @@ use crate::error::{ CreatePostgresPoolSnafu, GetPostgresConnectionSnafu, LoadTlsCertificateSnafu, PostgresExecutionSnafu, PostgresTlsConfigSnafu, PostgresTransactionSnafu, Result, }; -use crate::kv_backend::KvBackendRef; +use crate::kv_backend::ClosableKvBackendRef; use crate::kv_backend::rds::{ Executor, ExecutorFactory, ExecutorImpl, KvQueryExecutor, RDS_STORE_OP_BATCH_DELETE, RDS_STORE_OP_BATCH_GET, RDS_STORE_OP_BATCH_PUT, RDS_STORE_OP_RANGE_DELETE, @@ -389,6 +389,10 @@ impl PgExecutorFactory { #[async_trait::async_trait] impl ExecutorFactory for PgExecutorFactory { + async fn close(&self) { + self.pool.close(); + } + async fn default_executor(&self) -> Result { self.client().await } @@ -808,7 +812,7 @@ impl KvQueryExecutor for PgStore { } impl PgStore { - /// Create [PgStore] impl of [KvBackendRef] from url with optional TLS support. + /// Create [PgStore] impl of [ClosableKvBackendRef] from url with optional TLS support. /// /// # Arguments /// @@ -821,7 +825,7 @@ impl PgStore { table_name: &str, max_txn_ops: usize, tls_config: Option, - ) -> Result { + ) -> Result { let mut cfg = Config::new(); cfg.url = Some(url.to_string()); @@ -853,19 +857,23 @@ impl PgStore { Self::with_pg_pool(pool, None, table_name, max_txn_ops, false).await } - /// Create [PgStore] impl of [KvBackendRef] from url (backward compatibility). - pub async fn with_url(url: &str, table_name: &str, max_txn_ops: usize) -> Result { + /// Create [PgStore] impl of [ClosableKvBackendRef] from url without TLS. + pub async fn with_url( + url: &str, + table_name: &str, + max_txn_ops: usize, + ) -> Result { Self::with_url_and_tls(url, table_name, max_txn_ops, None).await } - /// Create [PgStore] impl of [KvBackendRef] from [deadpool_postgres::Pool] with optional schema. + /// Create [PgStore] impl of [ClosableKvBackendRef] from [deadpool_postgres::Pool] with optional schema. pub async fn with_pg_pool( pool: Pool, schema_name: Option<&str>, table_name: &str, max_txn_ops: usize, auto_create_schema: bool, - ) -> Result { + ) -> Result { // Ensure the postgres metadata backend is ready to use. let client = match pool.get().await { Ok(client) => client, @@ -925,6 +933,68 @@ mod tests { use crate::test_util::test_certs_dir; use crate::{maybe_skip_postgres_integration_test, maybe_skip_postgres15_integration_test}; + #[tokio::test] + async fn test_close_backend() { + let mut config = Config::new(); + config.host = Some("localhost".to_string()); + config.user = Some("test".to_string()); + config.dbname = Some("test".to_string()); + let pool = config.create_pool(Some(Runtime::Tokio1), NoTls).unwrap(); + let backend: ClosableKvBackendRef = Arc::new(PgStore { + max_txn_ops: 128, + txn_retry_count: RDS_STORE_TXN_RETRY_COUNT, + executor_factory: PgExecutorFactory { pool: pool.clone() }, + sql_template_set: PgSqlTemplateFactory::new(None, "close_test").build(), + _phantom: PhantomData, + }); + let shared_backend = backend.clone(); + assert!(!pool.is_closed()); + backend.close().await.unwrap(); + assert!(pool.is_closed()); + shared_backend.close().await.unwrap(); + assert!(matches!( + pool.get().await, + Err(deadpool::managed::PoolError::Closed) + )); + let result = tokio::time::timeout( + std::time::Duration::from_secs(1), + shared_backend.put(crate::rpc::store::PutRequest { + key: b"key".to_vec(), + value: b"value".to_vec(), + prev_kv: false, + }), + ) + .await + .expect("closed backend must reject writes promptly"); + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_close_backend_with_checked_out_connection() { + maybe_skip_postgres_integration_test!(); + let mut config = Config::new(); + config.url = Some(std::env::var("GT_POSTGRES_ENDPOINTS").unwrap()); + let pool = config.create_pool(Some(Runtime::Tokio1), NoTls).unwrap(); + let backend = PgStore::with_pg_pool(pool.clone(), None, "close_held", 128, false) + .await + .unwrap(); + let connection = pool.get().await.unwrap(); + + tokio::time::timeout(std::time::Duration::from_secs(5), backend.close()) + .await + .expect("PostgreSQL close does not wait for checked-out connections") + .unwrap(); + assert!(pool.is_closed()); + assert!(matches!( + pool.get().await, + Err(deadpool::managed::PoolError::Closed) + )); + // Existing executors can finish even though new acquisitions are rejected. + connection.simple_query("SELECT 1").await.unwrap(); + drop(connection); + backend.close().await.unwrap(); + } + async fn build_pg_kv_backend(table_name: &str) -> Option { let endpoints = std::env::var("GT_POSTGRES_ENDPOINTS").unwrap_or_default(); if endpoints.is_empty() { diff --git a/src/common/meta/src/kv_backend/test_util.rs b/src/common/meta/src/kv_backend/test_util.rs index 589c6c699ef..0f19476eced 100644 --- a/src/common/meta/src/kv_backend/test_util.rs +++ b/src/common/meta/src/kv_backend/test_util.rs @@ -22,8 +22,8 @@ use crate::error::Result; use crate::kv_backend::txn::{Txn, TxnResponse}; use crate::kv_backend::{ BatchDeleteRequest, BatchDeleteResponse, BatchGetRequest, BatchGetResponse, BatchPutRequest, - BatchPutResponse, DeleteRangeRequest, DeleteRangeResponse, KvBackend, PutRequest, PutResponse, - RangeRequest, RangeResponse, TxnService, + BatchPutResponse, DeleteRangeRequest, DeleteRangeResponse, KvBackend, KvBackendRef, PutRequest, + PutResponse, RangeRequest, RangeResponse, TxnService, }; pub type MockFn = Arc Result + Send + Sync>; @@ -133,3 +133,60 @@ impl KvBackend for MockKvBackend { } } } + +/// A [KvBackend] that runs `before_txn` before each transaction of `inner`. +pub struct TxnObservingKvBackend { + /// The kv backend every request is forwarded to. + pub inner: KvBackendRef, + /// Runs before each transaction is forwarded to `inner`. + pub before_txn: Box, +} + +#[async_trait::async_trait] +impl TxnService for TxnObservingKvBackend { + type Error = crate::error::Error; + + async fn txn(&self, txn: Txn) -> Result { + (self.before_txn)(); + self.inner.txn(txn).await + } + + fn max_txn_ops(&self) -> usize { + self.inner.max_txn_ops() + } +} + +#[async_trait::async_trait] +impl KvBackend for TxnObservingKvBackend { + fn name(&self) -> &str { + self.inner.name() + } + + fn as_any(&self) -> &dyn Any { + self + } + + async fn range(&self, req: RangeRequest) -> Result { + self.inner.range(req).await + } + + async fn put(&self, req: PutRequest) -> Result { + self.inner.put(req).await + } + + async fn batch_put(&self, req: BatchPutRequest) -> Result { + self.inner.batch_put(req).await + } + + async fn batch_get(&self, req: BatchGetRequest) -> Result { + self.inner.batch_get(req).await + } + + async fn delete_range(&self, req: DeleteRangeRequest) -> Result { + self.inner.delete_range(req).await + } + + async fn batch_delete(&self, req: BatchDeleteRequest) -> Result { + self.inner.batch_delete(req).await + } +} diff --git a/src/common/meta/src/region_keeper.rs b/src/common/meta/src/region_keeper.rs index aca0b958385..d40e4158ff6 100644 --- a/src/common/meta/src/region_keeper.rs +++ b/src/common/meta/src/region_keeper.rs @@ -21,12 +21,25 @@ use store_api::storage::RegionId; use crate::DatanodeId; +/// An operating region on a datanode. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct OperatingRegion { + /// The role of a lease granted because of this registration, when the region route grants + /// none. + pub role: RegionRole, + /// The random non-zero id of a region migration's registration of its target region. A lease + /// granted because of this registration carries it while the table route exists. `None` for + /// other operations. + pub candidate_lease_id: Option, +} + /// Tracks the operating(i.e., creating, opening, dropping) regions. #[derive(Debug, Clone)] pub struct OperatingRegionGuard { datanode_id: DatanodeId, region_id: RegionId, - inner: Arc>>, + candidate_lease_id: Option, + inner: Arc>>, } impl Drop for OperatingRegionGuard { @@ -41,6 +54,11 @@ impl OperatingRegionGuard { pub fn info(&self) -> (DatanodeId, RegionId) { (self.datanode_id, self.region_id) } + + /// Returns the candidate lease id of a region migration's target region registration. + pub fn candidate_lease_id(&self) -> Option { + self.candidate_lease_id + } } pub type MemoryRegionKeeperRef = Arc; @@ -52,7 +70,7 @@ pub type MemoryRegionKeeperRef = Arc; /// - Tracks the deleting regions after the corresponding metadata is deleted. #[derive(Debug, Clone, Default)] pub struct MemoryRegionKeeper { - inner: Arc>>, + inner: Arc>>, } impl MemoryRegionKeeper { @@ -67,19 +85,32 @@ impl MemoryRegionKeeper { region_id: RegionId, role: RegionRole, ) -> Option { - let mut inner = self.inner.write().unwrap(); + self.register( + datanode_id, + region_id, + OperatingRegion { + role, + candidate_lease_id: None, + }, + ) + } - match inner.entry((datanode_id, region_id)) { - Entry::Occupied(_) => None, - Entry::Vacant(vacant_entry) => { - vacant_entry.insert(role); - Some(OperatingRegionGuard { - datanode_id, - region_id, - inner: self.inner.clone(), - }) - } - } + /// Registers the target region of a region migration as a follower with a new random + /// candidate lease id. Returns [OperatingRegionGuard] if Region(`region_id`) on + /// Peer(`datanode_id`) does not exist. + pub fn register_candidate_region( + &self, + datanode_id: DatanodeId, + region_id: RegionId, + ) -> Option { + self.register( + datanode_id, + region_id, + OperatingRegion { + role: RegionRole::Follower, + candidate_lease_id: Some(new_candidate_lease_id(rand::random)), + }, + ) } /// Returns true if the keeper contains a (`datanoe_id`, `region_id`) tuple. @@ -88,19 +119,19 @@ impl MemoryRegionKeeper { inner.contains_key(&(datanode_id, region_id)) } - /// Extracts all operating regions with roles from `region_ids`. - pub fn extract_operating_region_roles( + /// Extracts all operating regions from `region_ids`. + pub fn extract_operating_regions( &self, datanode_id: DatanodeId, region_ids: &HashSet, - ) -> HashMap { + ) -> HashMap { let inner = self.inner.read().unwrap(); region_ids .iter() .filter_map(|region_id| { inner .get(&(datanode_id, *region_id)) - .map(|role| (*region_id, *role)) + .map(|region| (*region_id, *region)) }) .collect() } @@ -121,6 +152,38 @@ impl MemoryRegionKeeper { let mut inner = self.inner.write().unwrap(); inner.clear(); } + + fn register( + &self, + datanode_id: DatanodeId, + region_id: RegionId, + operating_region: OperatingRegion, + ) -> Option { + let mut inner = self.inner.write().unwrap(); + + match inner.entry((datanode_id, region_id)) { + Entry::Occupied(_) => None, + Entry::Vacant(vacant_entry) => { + vacant_entry.insert(operating_region); + Some(OperatingRegionGuard { + datanode_id, + region_id, + candidate_lease_id: operating_region.candidate_lease_id, + inner: self.inner.clone(), + }) + } + } + } +} + +/// Returns the first non-zero value of `random`, since 0 means no candidate lease id on the wire. +fn new_candidate_lease_id(mut random: impl FnMut() -> u64) -> u64 { + loop { + let candidate_lease_id = random(); + if candidate_lease_id != 0 { + return candidate_lease_id; + } + } } #[cfg(test)] @@ -130,7 +193,7 @@ mod tests { use store_api::region_engine::RegionRole; use store_api::storage::RegionId; - use crate::region_keeper::MemoryRegionKeeper; + use crate::region_keeper::{MemoryRegionKeeper, OperatingRegion, new_candidate_lease_id}; #[test] fn test_opening_region_keeper() { @@ -153,7 +216,7 @@ mod tests { RegionId::from_u64(2), RegionId::from_u64(3), ]); - let output = keeper.extract_operating_region_roles(1, ®ions); + let output = keeper.extract_operating_regions(1, ®ions); assert_eq!(output.len(), 2); assert!(output.contains_key(&RegionId::from_u64(1))); @@ -165,12 +228,24 @@ mod tests { RegionId::from_u64(2), RegionId::from_u64(3), ]); - let output = keeper.extract_operating_region_roles(1, ®ions); + let output = keeper.extract_operating_regions(1, ®ions); assert_eq!( output, HashMap::from([ - (RegionId::from_u64(1), RegionRole::Leader), - (RegionId::from_u64(2), RegionRole::Follower), + ( + RegionId::from_u64(1), + OperatingRegion { + role: RegionRole::Leader, + candidate_lease_id: None, + } + ), + ( + RegionId::from_u64(2), + OperatingRegion { + role: RegionRole::Follower, + candidate_lease_id: None, + } + ), ]) ); assert_eq!(keeper.len(), 2); @@ -182,4 +257,42 @@ mod tests { drop(guard2); assert!(keeper.is_empty()); } + + #[test] + fn test_register_candidate_region_uses_new_candidate_lease_id() { + let keeper = MemoryRegionKeeper::new(); + let region_id = RegionId::from_u64(1); + let regions = HashSet::from([region_id]); + + let guard = keeper + .register_with_role(1, region_id, RegionRole::Follower) + .unwrap(); + assert_eq!(guard.candidate_lease_id(), None); + assert_eq!( + keeper.extract_operating_regions(1, ®ions)[®ion_id].candidate_lease_id, + None + ); + drop(guard); + + let guard = keeper.register_candidate_region(1, region_id).unwrap(); + let first = guard.candidate_lease_id().unwrap(); + assert!(keeper.register_candidate_region(1, region_id).is_none()); + assert_eq!( + keeper.extract_operating_regions(1, ®ions)[®ion_id], + OperatingRegion { + role: RegionRole::Follower, + candidate_lease_id: Some(first), + } + ); + drop(guard); + + let guard = keeper.register_candidate_region(1, region_id).unwrap(); + assert_ne!(guard.candidate_lease_id().unwrap(), first); + } + + #[test] + fn test_new_candidate_lease_id_skips_zero() { + let mut values = [0, 0, 7].into_iter(); + assert_eq!(new_candidate_lease_id(|| values.next().unwrap()), 7); + } } diff --git a/src/common/meta/src/rpc/ddl.rs b/src/common/meta/src/rpc/ddl.rs index 9e88527f935..496060e2f52 100644 --- a/src/common/meta/src/rpc/ddl.rs +++ b/src/common/meta/src/rpc/ddl.rs @@ -68,6 +68,8 @@ use crate::key::table_name::{TableNameKey, TableNameManager}; /// Reserved query-context extension key for the frontend peer address that submitted a DDL request. pub const ORIGIN_FRONTEND_ADDR_EXTENSION_KEY: &str = "__greptime_origin_frontend.addr"; +/// Ordered source table IDs pinned by the caller of an automatic logical-table ALTER. +pub const ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY: &str = "__greptime_alter_logical_tables.ids"; /// Reserved query-context extension key for the authenticated database creator. pub const CREATE_DATABASE_CREATOR_EXTENSION_KEY: &str = "__greptime_create_database.creator"; /// Internal gRPC metadata key for the authenticated database creator. @@ -1707,6 +1709,24 @@ impl QueryContext { &self.extensions } + pub(crate) fn pinned_logical_table_ids(&self) -> Result>> { + self.extensions + .get(ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY) + .map(|ids| { + ids.split(',') + .map(|id| { + id.parse::().map_err(|error| { + error::AlterLogicalTablesInvalidArgumentsSnafu { + err_msg: format!("Invalid pinned logical table ID: {error}"), + } + .build() + }) + }) + .collect() + }) + .transpose() + } + /// Get the channel pub fn channel(&self) -> u8 { self.channel @@ -1891,6 +1911,32 @@ mod tests { use super::{AlterTableTask, CreateTableTask, *}; + #[test] + fn test_pinned_logical_table_ids_wire_compatibility() { + let legacy = include_bytes!( + "../../../../../tests/compatibility/wire/read_preference/legacy_query_context.pb" + ); + let mut context = QueryContext::from(PbQueryContext::decode(legacy.as_slice()).unwrap()); + assert_eq!(context.pinned_logical_table_ids().unwrap(), None); + context.extensions.insert( + "__greptime_alter_logical_tables.ids".into(), + "1025,1024".into(), + ); + let wire = PbQueryContext::from(context.clone()).encode_to_vec(); + let decoded = QueryContext::from(PbQueryContext::decode(wire.as_slice()).unwrap()); + assert_eq!(decoded, context); + assert_eq!( + decoded.pinned_logical_table_ids().unwrap(), + Some(vec![1025, 1024]) + ); + for invalid in ["", "1024,", "1024,no", "-1", "4294967296"] { + context + .extensions + .insert(ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY.into(), invalid.into()); + assert!(context.pinned_logical_table_ids().is_err(), "{invalid}"); + } + } + #[test] fn test_ddl_timeout_secs() { assert_eq!(ddl_timeout_secs(Duration::ZERO), 0); diff --git a/src/common/query/Cargo.toml b/src/common/query/Cargo.toml index 75db66bb12b..028f1ed01ea 100644 --- a/src/common/query/Cargo.toml +++ b/src/common/query/Cargo.toml @@ -5,7 +5,10 @@ edition.workspace = true license.workspace = true [features] -testing = [] +# `datafusion-datasource` is only needed by `testing`, which builds the real hash-join +# fixtures. `futures-util` stays a regular dependency here because the series-frame API +# exposes `futures_util::Stream` in its public signatures. +testing = ["dep:datafusion-datasource"] [lints] workspace = true @@ -22,6 +25,7 @@ common-recordbatch.workspace = true common-time.workspace = true datafusion.workspace = true datafusion-common.workspace = true +datafusion-datasource = { workspace = true, optional = true } datafusion-expr.workspace = true datafusion-proto.workspace = true datatypes.workspace = true @@ -37,6 +41,7 @@ store-api.workspace = true [dev-dependencies] common-base.workspace = true +datafusion-datasource.workspace = true serde_json.workspace = true tokio.workspace = true diff --git a/src/common/query/src/error.rs b/src/common/query/src/error.rs index 511f0a82db3..f5f880e8ced 100644 --- a/src/common/query/src/error.rs +++ b/src/common/query/src/error.rs @@ -218,6 +218,27 @@ pub enum Error { #[snafu(implicit)] location: Location, }, + + #[snafu(display( + "Remote dynamic filter predicate cannot be represented on the wire: {}", + reason + ))] + DynFilterUnsupported { + reason: String, + #[snafu(implicit)] + location: Location, + }, +} + +impl Error { + /// Whether the error reports a dynamic filter predicate with no wire form. + /// + /// Initial capture omits the refused snapshot; ongoing fanout sends an explicit + /// `true` reset to invalidate any stale region-side predicate. Neither path fails + /// the query. + pub fn is_dyn_filter_unsupported(&self) -> bool { + matches!(self, Error::DynFilterUnsupported { .. }) + } } pub type Result = std::result::Result; @@ -250,6 +271,9 @@ impl ErrorExt for Error { | Error::InvalidColumnPrefix { .. } => StatusCode::InvalidArguments, Error::DynFilterPayloadTooLarge { .. } => StatusCode::PlanQuery, + // An unrepresentable dynamic predicate is a transport limitation, not a + // server failure: callers omit the initial snapshot or reset the remote predicate. + Error::DynFilterUnsupported { .. } => StatusCode::Unsupported, Error::ConvertDfRecordBatchStream { source, .. } => source.status_code(), @@ -394,6 +418,15 @@ mod tests { StatusCode::PlanQuery ); + let unsupported_error = DataFusionError::External(Box::new(Error::DynFilterUnsupported { + reason: "no wire form".to_string(), + location: Location::default(), + })); + assert_eq!( + datafusion_status_code::(&unsupported_error, Some(StatusCode::Internal)), + StatusCode::Unsupported + ); + let boundary_error = DataFusionError::Shared(Arc::new(DataFusionError::External(Box::new( BoxedError::new(common_recordbatch::error::Error::PhysicalExpr { diff --git a/src/common/query/src/prelude.rs b/src/common/query/src/prelude.rs index 8d950d77096..beafb83f30b 100644 --- a/src/common/query/src/prelude.rs +++ b/src/common/query/src/prelude.rs @@ -30,6 +30,9 @@ static GREPTIME_VALUE_CELL: OnceCell = OnceCell::new(); /// Default native histogram column name. static GREPTIME_NATIVE_HISTOGRAM_CELL: OnceCell = OnceCell::new(); +/// Default exemplar labels column name. +static GREPTIME_EXEMPLAR_LABELS_CELL: OnceCell = OnceCell::new(); + pub fn set_default_prefix(prefix: Option<&str>) -> Result<()> { // Strip surrounding double quotes as a defensive measure against upstream // sources (scripts, CI, template engines, incorrect shell escaping) that may @@ -46,12 +49,14 @@ pub fn set_default_prefix(prefix: Option<&str>) -> Result<()> { GREPTIME_TIMESTAMP_CELL.get_or_init(|| GREPTIME_TIMESTAMP.to_string()); GREPTIME_VALUE_CELL.get_or_init(|| GREPTIME_VALUE.to_string()); GREPTIME_NATIVE_HISTOGRAM_CELL.get_or_init(|| NATIVE_HISTOGRAM_FIELD.to_string()); + GREPTIME_EXEMPLAR_LABELS_CELL.get_or_init(|| GREPTIME_EXEMPLAR_LABELS.to_string()); } Some(s) if s.trim().is_empty() => { // use "" to disable prefix GREPTIME_TIMESTAMP_CELL.get_or_init(|| "timestamp".to_string()); GREPTIME_VALUE_CELL.get_or_init(|| "value".to_string()); GREPTIME_NATIVE_HISTOGRAM_CELL.get_or_init(|| "native_histogram".to_string()); + GREPTIME_EXEMPLAR_LABELS_CELL.get_or_init(|| "exemplar_labels".to_string()); } Some(x) => { ensure!( @@ -61,6 +66,7 @@ pub fn set_default_prefix(prefix: Option<&str>) -> Result<()> { GREPTIME_TIMESTAMP_CELL.get_or_init(|| format!("{}_timestamp", x)); GREPTIME_VALUE_CELL.get_or_init(|| format!("{}_value", x)); GREPTIME_NATIVE_HISTOGRAM_CELL.get_or_init(|| format!("{}_native_histogram", x)); + GREPTIME_EXEMPLAR_LABELS_CELL.get_or_init(|| format!("{}_exemplar_labels", x)); } } Ok(()) @@ -87,6 +93,14 @@ pub fn greptime_native_histogram() -> &'static str { .map_or(NATIVE_HISTOGRAM_FIELD, String::as_str) } +/// Returns the configured exemplar labels column name, or `greptime_exemplar_labels` if not set. +#[inline] +pub fn greptime_exemplar_labels() -> &'static str { + GREPTIME_EXEMPLAR_LABELS_CELL + .get() + .map_or(GREPTIME_EXEMPLAR_LABELS, String::as_str) +} + /// Default timestamp column name constant for backward compatibility. const GREPTIME_TIMESTAMP: &str = "greptime_timestamp"; /// Default value column name constant for backward compatibility. @@ -94,7 +108,7 @@ const GREPTIME_VALUE: &str = "greptime_value"; /// Query-visible source metric name stored in the exemplar table. pub const EXEMPLAR_METRIC_COLUMN: &str = "greptime_metric"; /// JSON2 object containing exemplar-owned labels. -pub const EXEMPLAR_LABELS_COLUMN: &str = "exemplar_labels"; +const GREPTIME_EXEMPLAR_LABELS: &str = "greptime_exemplar_labels"; /// Stored series label that opts ordinary float samples into raw-delta math. pub const OTLP_AGGREGATION_TEMPORALITY_LABEL: &str = "otlp_aggregation_temporality"; /// Authoritative raw-delta value for [`OTLP_AGGREGATION_TEMPORALITY_LABEL`]. @@ -115,6 +129,7 @@ mod tests { assert_eq!(greptime_timestamp(), "greptime_timestamp"); assert_eq!(greptime_value(), "greptime_value"); assert_eq!(greptime_native_histogram(), "greptime_native_histogram"); + assert_eq!(greptime_exemplar_labels(), "greptime_exemplar_labels"); } #[test] @@ -123,6 +138,7 @@ mod tests { assert_eq!(greptime_timestamp(), "timestamp"); assert_eq!(greptime_value(), "value"); assert_eq!(greptime_native_histogram(), "native_histogram"); + assert_eq!(greptime_exemplar_labels(), "exemplar_labels"); } #[test] @@ -132,6 +148,7 @@ mod tests { assert_eq!(greptime_timestamp(), "timestamp"); assert_eq!(greptime_value(), "value"); assert_eq!(greptime_native_histogram(), "native_histogram"); + assert_eq!(greptime_exemplar_labels(), "exemplar_labels"); } #[test] @@ -140,6 +157,7 @@ mod tests { assert_eq!(greptime_timestamp(), "mydb_timestamp"); assert_eq!(greptime_value(), "mydb_value"); assert_eq!(greptime_native_histogram(), "mydb_native_histogram"); + assert_eq!(greptime_exemplar_labels(), "mydb_exemplar_labels"); } #[test] diff --git a/src/common/query/src/request.rs b/src/common/query/src/request.rs index c682cf81e02..2d68133d283 100644 --- a/src/common/query/src/request.rs +++ b/src/common/query/src/request.rs @@ -14,17 +14,21 @@ mod base64_serde; mod initial_remote_dyn_filter_reg; +#[cfg(any(test, feature = "testing"))] +pub mod remote_membership_test_support; use std::cell::Cell; use std::sync::Arc; use api::v1::region::RegionRequestHeader; +use datafusion::arrow::array::{Array, ArrayRef, StringArray}; use datafusion::execution::TaskContext; -use datafusion::physical_expr::expressions::{Column, InListExpr, lit}; +use datafusion::physical_expr::expressions::{Column, InListExpr, Literal}; +use datafusion::physical_expr::{conjunction, split_conjunction}; use datafusion::physical_plan::joins::HashTableLookupExpr; use datafusion::physical_plan::{ExecutionPlan, PhysicalExpr}; -use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion}; -use datafusion_common::{DataFusionError, Result as DataFusionResult}; +use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion}; +use datafusion_common::{DataFusionError, Result as DataFusionResult, ScalarValue}; use datafusion_expr::LogicalPlan; use datafusion_proto::physical_plan::from_proto::parse_physical_expr; use datafusion_proto::physical_plan::to_proto::{ @@ -34,7 +38,8 @@ use datafusion_proto::physical_plan::{ DefaultPhysicalExtensionCodec, DefaultPhysicalProtoConverter, PhysicalExtensionCodec, PhysicalPlanDecodeContext, PhysicalProtoConverterExtension, }; -use datafusion_proto::protobuf::{PhysicalExprNode, PhysicalPlanNode}; +use datafusion_proto::protobuf::physical_expr_node::ExprType as PhysicalExprNodeType; +use datafusion_proto::protobuf::{PhysicalExprNode, PhysicalInListNode, PhysicalPlanNode}; use prost::Message; use serde::{Deserialize, Serialize}; use snafu::ensure; @@ -45,12 +50,46 @@ pub use self::initial_remote_dyn_filter_reg::{ INITIAL_REMOTE_DYN_FILTER_REGISTRATIONS_EXTENSION_KEY, InitialDynFilterReg, InitialDynFilterRegs, InitialDynFilterSnapshot, }; -use crate::error::{DynFilterPayloadTooLargeSnafu, Error as CommonQueryError}; +use crate::error::{ + DynFilterPayloadTooLargeSnafu, DynFilterUnsupportedSnafu, Error as CommonQueryError, +}; pub const DYN_FILTER_PROTOCOL_VERSION: u32 = 1; /// Shared raw protobuf byte budget for remote dynamic filter payloads. pub const REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES: usize = 512 * 1024; +/// Smallest number of encoded bytes a single `IN` list literal can take. +/// +/// Every list entry is a `PhysicalExprNode` whose `literal` arm holds the +/// [`ScalarValue`] message itself, so the oneof field tag plus its length prefix are +/// already 2 bytes even for an empty `Utf8` value (whose `ScalarValue` body may encode to +/// nothing at all). Protobuf sizes add up across nesting, so a payload of `B` bytes can +/// never hold more than `B / 2` literals. This is the floor the value bound is derived +/// from; it is not an encoder policy constant. +const MIN_ENCODED_LITERAL_BYTES: usize = 2; + +/// Upper bound on how many `IN` list literals `max_payload_bytes` can hold. +/// +/// The count bound is derived from the transport budget with +/// [`MIN_ENCODED_LITERAL_BYTES`] rather than being an arbitrary cap: a larger budget +/// admits proportionally more build keys, and a small budget refuses the membership +/// before a single literal is built. +fn max_membership_values(max_payload_bytes: usize) -> usize { + max_payload_bytes / MIN_ENCODED_LITERAL_BYTES +} + +/// Frame size reported for a payload refused because it does not fit the budget. +/// +/// A refused payload is never materialized, so there is no measured frame length. The +/// error reports the largest lower bound that is known instead: the price of the literals +/// the refused membership would have written (`lower_bound`), or - when nothing about the +/// shape is known - one byte over the budget. Either way the number is genuinely above +/// `max_payload_bytes`; it is a bound, never a measurement, and it is only used for +/// diagnostics. +fn refused_payload_size(lower_bound: usize, max_payload_bytes: usize) -> usize { + lower_bound.max(max_payload_bytes.saturating_add(1)) +} + /// Serialized predicate payload for remote dynamic filter updates. /// /// The payload is tagged in JSON so receivers can reject unsupported encodings @@ -72,20 +111,98 @@ pub enum DynFilterPayload { impl DynFilterPayload { /// Encodes a DataFusion physical expression into a bounded dynamic filter payload. /// - /// Runtime-only hash lookup predicates are degraded to `true` before encoding so - /// serializable min/max bounds around them can still be shipped to remote scans. - /// If the full serializable predicate is still larger than `max_payload_bytes`, large - /// membership predicates (`IN (...)`) are also degraded to `true` as a bounds-only fallback. + /// # Narrow contract: a key list used as a positive pruning hint + /// + /// A runtime [`HashTableLookupExpr`] is written as the existing + /// `PhysicalInListNode` — `key IN (build key, ...)` — built from the lookup's + /// published build-key list, so no new wire variant is needed. The expression tree + /// around it is preserved node for node: `IS NULL`, `NOT`, `CASE`, `AND`, `OR` and + /// every other parent are serialized as-is, and a `NOT` over a lookup therefore + /// becomes `NOT IN`. + /// + /// The membership is a **hint for pruning only**, not a re-statement of the join + /// semantics. For supported positive pruning, the list preserves actual build-key + /// matches; it is not universal Boolean equivalence. `IN`/`NOT IN` propagate + /// `NULL` while a hash lookup can be definite on a `NULL` probe key, and hash + /// presence can admit collisions, so a predicate that inspects the *negation* of a + /// lookup is not universally equivalent. + /// + /// The conversion is decided by the lookup alone (single-column `Utf8`, non-empty, + /// `NULL`-free build keys — see [`HashTableLookupExpr::build_key_membership`]). + /// Exactly one node changes: every [`HashTableLookupExpr`] is written as its key + /// list, wherever it sits in the tree (`NOT lookup` included, which is why it + /// becomes `NOT IN`). No node is ever replaced by a *weaker stand-in* — in + /// particular never by `true` — and no membership is dropped from inside a + /// predicate: when a lookup cannot be written at all, the whole top-level `AND` + /// conjunct that holds it is dropped (see below) or the payload is rejected. + /// + /// # Top-level conjunctive retention + /// + /// If a lookup has no such key list (accessor limitation) or the serialized predicate + /// does not fit `max_payload_bytes`, [`split_conjunction`] splits the original + /// expression and the whole conjuncts that cannot be shipped are dropped. Dropping an + /// `AND` term can only widen a predicate, so the result never drops a row the producer + /// would keep. Which conjuncts go depends on why the payload was refused: + /// + /// - a lookup has no wire form: only conjuncts holding a [`HashTableLookupExpr`] are + /// dropped. A static [`InListExpr`] is representable, so `lookup AND id IN (...)` + /// keeps the list and stays `id IN (...)`; + /// - the payload is over budget: conjuncts holding a [`HashTableLookupExpr`] *or* an + /// [`InListExpr`] are dropped, because a static `IN` list is the node that usually + /// blows the budget. `bounds AND oversized IN (...)` becomes `bounds`. + /// + /// Either way a conjunct is kept or dropped as a whole: a membership nested inside + /// `NOT`, `CASE` or `OR` drops that entire conjunct instead of being rewritten. + /// Retained conjuncts are encoded again under the same budget, and when the first two + /// rules still leave an over-budget payload the second one is applied as well, so a + /// static `IN` list that is too large for the budget is dropped last. + /// + /// # Encoder-level no-substitution rule + /// + /// This encoder never replaces a node in the producer expression with `true`: it + /// serializes the expression, retries after dropping whole top-level conjuncts under + /// the retention rules above, or returns an error. A caller decides what a refused + /// generation means for its receiver. Initial capture keeps the parent filter local; + /// the long-lived query fanout path + /// (`remote_dyn_filter_registry::fanout_snapshot_for_query`) publishes an explicit + /// `true` reset to invalidate a stale region-side predicate. That reset is not a + /// representation of the refused producer expression. + /// + /// The two refusals stay distinct so callers can tell a shape problem from a size + /// problem: + /// + /// - [`CommonQueryError::DynFilterUnsupported`]: no conjunct can be retained because + /// every one of them holds a lookup with no UTF-8 key list (`NOT lookup`, `CASE` / + /// `OR` over such a membership). The wire has no form for the predicate. + /// - [`CommonQueryError::DynFilterPayloadTooLarge`]: the predicate is representable but + /// does not fit `max_payload_bytes` - either a key list itself exceeds the budget, or + /// the retained conjuncts still do not fit. An over-budget key list is never reported + /// as unsupported; it is a size refusal and uses + /// [`ConjunctRetention::OverBudget`]. + /// + /// These refusals do not fail the query in either caller: initial capture leaves the + /// parent filter local, while fanout sends an explicit `true` reset instead of leaving + /// a stale narrower predicate installed. pub fn from_datafusion_expr( expr: &Arc, max_payload_bytes: usize, ) -> DataFusionResult { - match encode_remote_dyn_filter_expr(expr, max_payload_bytes, false) { + match encode_bounded_physical_expr_to_bytes(expr, max_payload_bytes) { Ok(bytes) => Ok(Self::Datafusion(bytes)), + // The predicate cannot be shipped as it stands: either a lookup has no key + // list, or the frame does not fit the budget. Keep the widest *safe* part of + // the predicate instead of publishing an always-true filter. + Err(CommonQueryError::DynFilterUnsupported { .. }) => retain_membership_free_conjuncts( + expr, + max_payload_bytes, + ConjunctRetention::UnrepresentableLookup, + ), Err(CommonQueryError::DynFilterPayloadTooLarge { .. }) => { - encode_remote_dyn_filter_expr(expr, max_payload_bytes, true) - .map(Self::Datafusion) - .map_err(DataFusionError::from) + retain_membership_free_conjuncts( + expr, + max_payload_bytes, + ConjunctRetention::OverBudget, + ) } Err(error) => Err(DataFusionError::from(error)), } @@ -96,6 +213,22 @@ impl DynFilterPayload { /// The decoded expression is validated to ensure column indexes stay within the receiver-side /// schema, column names are consistent (defensive check), and the payload stays within /// `max_payload_bytes`. + /// + /// # Precondition: pruning only, never row filtering + /// + /// The decoded predicate is only a **pruning hint**. The sole production consumer — + /// `Predicate::dyn_filters` (in the `table` crate) and its pruning path, reached by + /// mito2's row-group pruner — uses it to skip containers whose statistics prove that + /// no row can match, so a predicate that is merely *wider* than the producer's costs + /// nothing but a wasted scan. + /// + /// The payload is not a re-statement of the producer predicate: the membership is the + /// join's build-key list, so `IN`/`NOT IN` `NULL` propagation and hash collisions can + /// make it disagree with a runtime lookup, and the encoder drops whole top-level + /// conjuncts it cannot ship (see [`DynFilterPayload::from_datafusion_expr`]). A + /// consumer that *filters rows* would turn every such widening into lost data, so row + /// filtering must not consume this payload unconditionally: it would need to prove the + /// predicate is equivalent for the rows it drops, which this format does not carry. pub fn decode_datafusion_expr( &self, task_ctx: &TaskContext, @@ -129,6 +262,141 @@ fn encode_physical_expr_to_bytes(expr: &Arc) -> DataFusionResu struct BoundedPhysicalExprEncoder { max_payload_bytes: usize, encoded_bytes: Cell, + /// Set when the encoding aborted on a runtime lookup that has no serializable key + /// list. + /// + /// The abort travels out as an ordinary [`DataFusionError`], which would lose the + /// typed dynamic filter error on the way, so the flag is what lets + /// `encode_bounded_physical_expr_to_bytes` rebuild + /// [`CommonQueryError::DynFilterUnsupported`]. No placeholder is ever encoded in its + /// place. + unsupported: Cell, + /// Set to the reported frame size when the encoding aborted because a build-key + /// membership does not fit `max_payload_bytes`. + /// + /// This is a size refusal, not a shape one, so + /// `encode_bounded_physical_expr_to_bytes` rebuilds + /// [`CommonQueryError::DynFilterPayloadTooLarge`] from it instead of reporting the + /// predicate as having no wire form. + over_budget: Cell>, +} + +impl BoundedPhysicalExprEncoder { + fn new(max_payload_bytes: usize) -> Self { + Self { + max_payload_bytes, + encoded_bytes: Cell::new(0), + unsupported: Cell::new(false), + over_budget: Cell::new(None), + } + } + + /// Charges `proto` (including its encoded children) against the payload budget. + /// + /// The accounting is additive over siblings and exact for the top-level node, so + /// no branch can push the payload over budget unnoticed. + fn finish( + &self, + before_children: usize, + proto: PhysicalExprNode, + ) -> DataFusionResult { + let encoded_bytes = before_children.saturating_add(proto.encoded_len()); + self.encoded_bytes.set(encoded_bytes); + validate_payload_size(encoded_bytes, self.max_payload_bytes) + .map_err(DataFusionError::from)?; + Ok(proto) + } + + /// Encodes a lookup as its build-key membership, charging every list literal and + /// node against the payload budget. + /// + /// The decision only looks at the lookup itself: if DataFusion can hand out the + /// single-column `Utf8` build-key list it is written as `key IN (...)`; otherwise the + /// whole payload aborts. A lookup is a real predicate that DataFusion only degrades to + /// `true` in its own proto encoder, so this transport must not write any substitute for + /// it, and the abort keeps the two reasons apart: no key list at all (unsupported) is + /// not the same thing as a key list that exceeds the budget (too large). + fn membership_to_proto( + &self, + lookup: &HashTableLookupExpr, + codec: &dyn PhysicalExtensionCodec, + ) -> DataFusionResult { + let membership = match RemoteMembership::from_lookup(lookup, self.max_payload_bytes) { + Ok(membership) => membership, + // A lookup without a key list has no wire form at all: that is a shape + // refusal, not a size one. + Err(MembershipRefusal::NoKeyList) => return Err(self.abort_unsupported_lookup()), + // The key list exists but does not fit the budget: report it as a size + // refusal so it is not counted as "no wire form". + Err(MembershipRefusal::OverBudget { build_keys }) => { + return Err(self.abort_over_budget_membership(build_keys)); + } + }; + // The count bound is enforced once more before any literal exists, so a build side + // larger than the budget can hold is never materialized. This cannot fail for any + // other reason here: the accessor already guaranteed a non-empty, `NULL`-free + // `Utf8` list, so a refusal is again a size refusal. + let Some(literals) = utf8_list_literals( + &membership.values, + max_membership_values(self.max_payload_bytes), + ) else { + return Err(self.abort_over_budget_membership(membership.values.len())); + }; + + let before_children = self.encoded_bytes.get(); + let mut list = Vec::with_capacity(literals.len()); + for literal in literals { + list.push(self.physical_expr_to_proto(&literal, codec)?); + } + + // The `InList` proto is built by hand instead of going through the generic + // serializer: that path needs an `InListExpr`, whose only constructors + // (`InListExpr::try_new` / `try_new_from_array`) require an input `Schema` to + // resolve the key expression's data type, and this encoder converts a bare + // expression without one. The key is known to be `Utf8` (see + // `HashTableLookupExpr::build_key_membership`), which is exactly what the list + // below writes, so no type check is lost. + let proto = PhysicalExprNode { + expr_id: None, + expr_type: Some(PhysicalExprNodeType::InList(Box::new(PhysicalInListNode { + expr: Some(Box::new( + self.physical_expr_to_proto(&membership.key, codec)?, + )), + list, + negated: false, + }))), + }; + + self.finish(before_children, proto) + } + + /// Aborts the payload because the wire has no form for `lookup`. + /// + /// The returned error only carries the abort; [`Self::unsupported`] is what the caller + /// reads to rebuild the typed dynamic filter error. + fn abort_unsupported_lookup(&self) -> DataFusionError { + self.unsupported.set(true); + DataFusionError::NotImplemented( + "hash table lookup has no UTF-8 build-key membership to encode".to_string(), + ) + } + + /// Aborts the payload because a build-key membership does not fit `max_payload_bytes`. + /// + /// This is a *size* refusal, never a "no wire form" one, so the caller rebuilds + /// [`CommonQueryError::DynFilterPayloadTooLarge`] from [`Self::over_budget`] instead of + /// [`CommonQueryError::DynFilterUnsupported`]. `build_keys` is the number of build keys + /// the refused list would have written; every one of them costs at least + /// [`MIN_ENCODED_LITERAL_BYTES`], so that product is a real lower bound of the frame. + fn abort_over_budget_membership(&self, build_keys: usize) -> DataFusionError { + self.over_budget.set(Some(refused_payload_size( + build_keys.saturating_mul(MIN_ENCODED_LITERAL_BYTES), + self.max_payload_bytes, + ))); + DataFusionError::NotImplemented( + "hash table lookup build-key membership exceeds the payload budget".to_string(), + ) + } } impl PhysicalProtoConverterExtension for BoundedPhysicalExprEncoder { @@ -162,55 +430,268 @@ impl PhysicalProtoConverterExtension for BoundedPhysicalExprEncoder { expr: &Arc, codec: &dyn PhysicalExtensionCodec, ) -> DataFusionResult { + // A real `HashTableLookupExpr` has no wire form: DataFusion's own + // `try_to_proto` degrades it to `true`. Intercept it so the build-key list can + // be written with the existing `InList` node instead. The decision is made by + // the lookup alone; every other node, including `IS NULL`, `NOT`, `CASE`, + // `AND` and `OR`, is serialized exactly as it is. + if let Some(lookup) = expr.downcast_ref::() { + return self.membership_to_proto(lookup, codec); + } + let before_children = self.encoded_bytes.get(); let proto = serialize_physical_expr_with_converter(expr, codec, self)?; // Replace the children's contribution with the whole subtree, including protobuf framing. // Checking each child stops large IN lists before materializing all their Arrow IPC literals. - let encoded_bytes = before_children.saturating_add(proto.encoded_len()); - self.encoded_bytes.set(encoded_bytes); - validate_payload_size(encoded_bytes, self.max_payload_bytes) - .map_err(DataFusionError::from)?; - Ok(proto) + self.finish(before_children, proto) } } +/// Encodes `expr` for the remote wire under `max_payload_bytes`. +/// +/// Returns the typed [`CommonQueryError::DynFilterUnsupported`] when the predicate holds a +/// lookup whose key list cannot be serialized, and +/// [`CommonQueryError::DynFilterPayloadTooLarge`] when the frame (or a build-key +/// membership) does not fit the budget. The two refusals stay distinct: an over-budget +/// membership is a size refusal and is never reported as a predicate without a wire form. +/// The caller answers both with [`retain_membership_free_conjuncts`]; neither is ever +/// replaced with `true`. fn encode_bounded_physical_expr_to_bytes( expr: &Arc, max_payload_bytes: usize, ) -> Result, CommonQueryError> { let codec = DefaultPhysicalExtensionCodec {}; - let encoder = BoundedPhysicalExprEncoder { - max_payload_bytes, - encoded_bytes: Cell::new(0), - }; + let encoder = BoundedPhysicalExprEncoder::new(max_payload_bytes); let proto = encoder.physical_expr_to_proto(expr, &codec); - // Preserve the size error so the caller can retry with bounds only. + // A build-key membership that exceeds the budget is a size refusal even though the + // frame that would have held it is never materialized. + if let Some(payload_size_bytes) = encoder.over_budget.get() { + return Err(DynFilterPayloadTooLargeSnafu { + payload_size_bytes, + max_payload_bytes, + } + .build()); + } + let unsupported = encoder.unsupported.get(); + // The encoder stops at the node that went over budget, so this reports the size + // error unwrapped instead of the wrapped DataFusion error. validate_payload_size(encoder.encoded_bytes.get(), max_payload_bytes)?; + if unsupported { + return Err(DynFilterUnsupportedSnafu { + reason: "the predicate holds a hash table lookup without a UTF-8 build-key \ + membership", + } + .build()); + } Ok(proto.map_err(CommonQueryError::from)?.encode_to_vec()) } -fn encode_remote_dyn_filter_expr( - expr: &Arc, - max_payload_bytes: usize, - bounds_only: bool, -) -> Result, CommonQueryError> { - let expr = portable_remote_dyn_filter_expr(Arc::clone(expr), bounds_only) - .map_err(CommonQueryError::from)?; - encode_bounded_physical_expr_to_bytes(&expr, max_payload_bytes) +/// Which top-level conjuncts [`retain_membership_free_conjuncts`] has to drop. +#[derive(Clone, Copy, Debug)] +enum ConjunctRetention { + /// The payload holds a [`HashTableLookupExpr`] without a wire form, so such a lookup + /// is the only unshippable node: a static [`InListExpr`] is representable and stays. + UnrepresentableLookup, + /// The payload exceeds its budget. A static [`InListExpr`] is the node most likely to + /// be the oversized one, so it is dropped with the lookups. + OverBudget, } -fn portable_remote_dyn_filter_expr( - expr: Arc, - bounds_only: bool, -) -> DataFusionResult> { - expr.transform_up(|node| { - if node.is::() || (bounds_only && node.is::()) { - Ok(Transformed::yes(lit(true))) - } else { - Ok(Transformed::no(node)) +/// Keeps the top-level conjuncts of a predicate that cannot be shipped as it stands. +/// +/// Dropping an `AND` term can only widen a predicate, so the retained payload still never +/// drops a row the producer would keep. `retention` decides which conjuncts have to go. It +/// only widens as far as it has to: when the narrow retention leaves conjuncts that still do +/// not fit `max_payload_bytes`, the [`InListExpr`]-bearing ones are dropped as well, because +/// a static `IN` list is the node most likely to be the oversized one. +/// +/// The refusal keeps the two reasons apart instead of folding them into "no wire form": +/// nothing retained under [`ConjunctRetention::UnrepresentableLookup`] means the wire +/// really has no form for the predicate +/// ([`CommonQueryError::DynFilterUnsupported`]), while nothing retained under +/// [`ConjunctRetention::OverBudget`] (or a retained payload that still does not fit) is a +/// size refusal ([`CommonQueryError::DynFilterPayloadTooLarge`]). Transport never +/// substitutes `true` for either. +fn retain_membership_free_conjuncts( + expr: &Arc, + max_payload_bytes: usize, + retention: ConjunctRetention, +) -> DataFusionResult { + match encode_retained_conjuncts_to_bytes(expr, max_payload_bytes, retention) { + Ok(Some(bytes)) => Ok(DynFilterPayload::Datafusion(bytes)), + // Nothing could be retained. Which refusal that is depends on *why* the conjuncts + // had to go, and the two must not be mixed up: an over-budget predicate is + // representable, it only does not fit. + Ok(None) => Err(match retention { + ConjunctRetention::UnrepresentableLookup => DynFilterUnsupportedSnafu { + reason: "every top-level conjunct of the predicate holds a hash table \ + lookup without a UTF-8 build-key membership, so the wire has no \ + form for it", + } + .build() + .into(), + ConjunctRetention::OverBudget => DynFilterPayloadTooLargeSnafu { + payload_size_bytes: refused_payload_size(0, max_payload_bytes), + max_payload_bytes, + } + .build() + .into(), + }), + Err(CommonQueryError::DynFilterPayloadTooLarge { .. }) + if matches!(retention, ConjunctRetention::UnrepresentableLookup) => + { + retain_membership_free_conjuncts(expr, max_payload_bytes, ConjunctRetention::OverBudget) } - }) - .map(|transformed| transformed.data) + Err(error) => Err(error.into()), + } +} + +/// Encodes the top-level conjuncts of `expr` that `retention` allows to keep. +/// +/// [`split_conjunction`] hands out the top-level `AND` terms of the original +/// expression; a term is dropped as a whole when its subtree holds a node the retention +/// cannot ship (see [`ConjunctRetention`]). Membership nested inside `NOT`, `CASE` or `OR` +/// therefore drops that whole conjunct instead of being rewritten in place, so the payload +/// only ever loses top-level `AND` terms, which can only widen a predicate. +/// +/// The retained terms are encoded again under the same `max_payload_bytes`, so a caller +/// supplied budget still bounds the frame. Returns `Ok(None)` when no conjunct can be +/// retained; the caller decides which typed refusal that is from `retention`, because an +/// empty retention caused by an unrepresentable lookup is not the same thing as an empty +/// retention caused by the budget. +fn encode_retained_conjuncts_to_bytes( + expr: &Arc, + max_payload_bytes: usize, + retention: ConjunctRetention, +) -> Result>, CommonQueryError> { + let mut retained = Vec::new(); + for conjunct in split_conjunction(expr) { + if !retention.drops_conjunct(conjunct)? { + retained.push(Arc::clone(conjunct)); + } + } + if retained.is_empty() { + return Ok(None); + } + encode_bounded_physical_expr_to_bytes(&conjunction(retained), max_payload_bytes).map(Some) +} + +impl ConjunctRetention { + /// Whether the whole subtree of `expr` holds a node this retention cannot ship. + /// + /// This is a "whole conjunct" test on purpose: a conjunct that reaches such a node + /// anywhere is dropped by the retention instead of being rewritten around it. + fn drops_conjunct(self, expr: &Arc) -> DataFusionResult { + let mut found = false; + expr.apply(|node| { + let unshippable = match self { + Self::UnrepresentableLookup => node.is::(), + Self::OverBudget => node.is::() || node.is::(), + }; + if unshippable { + found = true; + Ok(TreeNodeRecursion::Stop) + } else { + Ok(TreeNodeRecursion::Continue) + } + })?; + Ok(found) + } +} + +/// Build-key list of one hash lookup, written as `probe_key IN (value, ...)`. +/// +/// The membership reuses the existing `PhysicalInListNode`, so no new wire variant +/// and no in-payload version tag are needed, and it is charged against the normal +/// payload budget. Only the build-key list itself is consulted here; the wire form +/// never depends on where the lookup sits in the predicate tree. +struct RemoteMembership { + /// Probe-side key expression, reused as the `InList` operand. + key: Arc, + /// Published build key values, tied to the lookup's build lease. + values: ArrayRef, +} + +impl RemoteMembership { + /// Key-list membership of `lookup`, or the reason it cannot be written. + /// + /// The count bound handed to DataFusion is derived from the byte budget (see + /// [`max_membership_values`]) instead of being a policy constant, and the byte bound + /// is the raw payload budget itself, which every build-key buffer has to fit before it + /// can be written as literals. Both are conservative early guards, not the exact + /// serialized frame length: the encoder still charges every literal against the budget + /// node by node. + /// + /// The accessor keeps the value-level limitations — it returns `None` for lookups + /// without a build lease (e.g. a test-only lookup), composite keys, non-`Utf8` keys + /// (integer/float/decimal/dictionary/unknown), and `NULL`-containing or empty build + /// keys — and [`utf8_list_literals`] re-checks everything it materializes. Those are + /// accessor and materialization limitations, not encoder policy. + /// + /// The accessor folds its shape and size checks into one `None`, so a key list that + /// exists but exceeds the bounds is told apart by asking again with the bounds lifted: + /// only a list that exists at all can be refused for its size, and a size refusal must + /// not be reported as a predicate without a wire form (it is + /// [`CommonQueryError::DynFilterPayloadTooLarge`], not + /// [`CommonQueryError::DynFilterUnsupported`]). + fn from_lookup( + lookup: &HashTableLookupExpr, + max_payload_bytes: usize, + ) -> Result { + match lookup + .build_key_membership(max_membership_values(max_payload_bytes), max_payload_bytes) + { + Some(membership) => Ok(Self { + key: Arc::clone(membership.key()), + values: Arc::clone(membership.values()), + }), + None => Err(match lookup.build_key_membership(usize::MAX, usize::MAX) { + Some(unbounded) => MembershipRefusal::OverBudget { + build_keys: unbounded.values().len(), + }, + None => MembershipRefusal::NoKeyList, + }), + } + } +} + +/// Why a lookup's build-key membership cannot be written as `probe_key IN (...)`. +/// +/// The two reasons are answered differently: [`Self::NoKeyList`] means the wire has no +/// form for the node, while [`Self::OverBudget`] means the membership is real but too +/// large for the payload budget, which the encoder reports as a size refusal. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MembershipRefusal { + /// The lookup has no single-column `Utf8` build-key list at all: no build lease, + /// composite key, unsupported key type, or an empty / `NULL`-containing build side. + NoKeyList, + /// The build-key list exists but exceeds the caller's bounds. `build_keys` is how many + /// literals it would have written, measured from the unbounded accessor. + OverBudget { build_keys: usize }, +} + +/// Materializes the `IN` list literals for a build-key membership. +/// +/// This is the authoritative value check: the list is only built for a non-empty, +/// `NULL`-free, non-dictionary `Utf8` array of at most `max_values` entries. `max_values` +/// is checked *before* a [`Literal`] is created for every build key, so an over-large +/// build side is refused instead of being materialized and then measured. The caller has +/// already been through [`RemoteMembership::from_lookup`], so a refusal here can only be +/// the count bound and is reported as a size refusal, not as a missing wire form. +fn utf8_list_literals(values: &ArrayRef, max_values: usize) -> Option>> { + let values = values.as_any().downcast_ref::()?; + if values.is_empty() || values.null_count() != 0 || values.len() > max_values { + return None; + } + values + .iter() + .map(|value| { + value.map(|value| { + Arc::new(Literal::new(ScalarValue::Utf8(Some(value.to_owned())))) + as Arc + }) + }) + .collect() } pub(crate) fn decode_physical_expr_from_bytes( @@ -246,6 +727,78 @@ fn validate_payload_size( Ok(()) } +/// Asserts the predicate is reported as having no wire form instead of becoming `true`. +#[cfg(test)] +fn assert_unsupported(error: DataFusionError) { + assert_typed_dyn_filter_error(error, "DynFilterUnsupported", |error| { + matches!(error, CommonQueryError::DynFilterUnsupported { .. }) + }); +} + +/// Asserts the retained payload still exceeds the caller-supplied budget. +#[cfg(test)] +fn assert_too_large(error: DataFusionError) { + assert_typed_dyn_filter_error(error, "DynFilterPayloadTooLarge", |error| { + matches!(error, CommonQueryError::DynFilterPayloadTooLarge { .. }) + }); +} + +#[cfg(test)] +fn assert_typed_dyn_filter_error( + error: DataFusionError, + expected: &str, + matches_error: impl Fn(&CommonQueryError) -> bool, +) { + let DataFusionError::External(error) = error else { + panic!("expected external common query error, got: {error:?}"); + }; + let typed = error + .downcast_ref::() + .unwrap_or_else(|| panic!("expected external {expected}, got: {error:?}")); + assert!(matches_error(typed), "expected {expected}, got: {error:?}"); +} + +/// Whether `error` reports a dynamic filter predicate the wire cannot represent. +/// +/// Transport never rewrites a producer predicate to `true`, so a producer answers this +/// error by not shipping the producer predicate and keeping the query running. Callers +/// that keep a long-lived region-side predicate treat it as an invalidation trigger: the +/// stale predicate is replaced by an explicit `true` reset (see +/// `remote_dyn_filter_registry`), so a producer that later widens its filter cannot be +/// pruned away by it. +pub fn is_dyn_filter_unsupported(error: &DataFusionError) -> bool { + matches!( + error, + DataFusionError::External(source) + if source + .downcast_ref::() + .is_some_and(CommonQueryError::is_dyn_filter_unsupported) + ) +} + +/// Whether `error` reports a dynamic filter payload that exceeded `max_payload_bytes`. +/// +/// This is the other refusal [`DynFilterPayload::from_datafusion_expr`] reports, next to +/// [`is_dyn_filter_unsupported`]: the predicate exists and is representable, but even the +/// retained top-level conjuncts do not fit the budget. It is *not* a "no wire form" case +/// and must not be reported as one. Like the unsupported case it is answered by not +/// shipping the producer predicate (the query keeps running) and never by narrowing or +/// rewriting it; a caller that keeps a long-lived region predicate publishes the explicit +/// `true` reset instead, so a stale, narrower predicate cannot prune rows the producer +/// still keeps. +pub fn is_dyn_filter_payload_too_large(error: &DataFusionError) -> bool { + matches!( + error, + DataFusionError::External(source) + if source + .downcast_ref::() + .is_some_and(|error| matches!( + error, + CommonQueryError::DynFilterPayloadTooLarge { .. } + )) + ) +} + fn validate_supported_payload_expr(expr: &Arc) -> DataFusionResult<()> { expr.apply(|node| { if node.is::() { @@ -352,14 +905,51 @@ pub struct QueryRequest { pub plan: LogicalPlan, } +/// First node of type `T` anywhere in `expr`. +#[cfg(test)] +fn find_expr(expr: &Arc) -> Option> { + let mut found = None; + expr.apply(|node| { + if node.is::() { + found = Some(Arc::clone(node)); + Ok(TreeNodeRecursion::Stop) + } else { + Ok(TreeNodeRecursion::Continue) + } + }) + .unwrap(); + found +} + +/// Whether `expr` holds a node of type `T` anywhere. +#[cfg(test)] +fn has_expr(expr: &Arc) -> bool { + find_expr::(expr).is_some() +} + +/// Encodes `expr` at `max_payload_bytes` and decodes it against `schema`. This is the +/// round trip a receiver performs, so the tests never assert on the encoder alone. +#[cfg(test)] +fn decode( + expr: &Arc, + schema: &Arc, + max_payload_bytes: usize, +) -> Arc { + DynFilterPayload::from_datafusion_expr(expr, max_payload_bytes) + .unwrap() + .decode_datafusion_expr(&TaskContext::default(), schema, max_payload_bytes) + .unwrap() +} + #[cfg(test)] mod tests { use std::sync::Arc; use base64::Engine; use base64::prelude::BASE64_STANDARD; + use datafusion::arrow::array::{ArrayRef, Int32Array, StringArray}; use datafusion::arrow::datatypes::{DataType, Field, Schema}; - use datafusion::physical_expr::expressions::{BinaryExpr, Column, InListExpr, lit}; + use datafusion::physical_expr::expressions::{BinaryExpr, Column, InListExpr, NotExpr, lit}; use datafusion::physical_plan::expressions::col; use datafusion::physical_plan::joins::join_hash_map::JoinHashMapU32; use datafusion::physical_plan::joins::{HashTableLookupExpr, Map, SeededRandomState}; @@ -507,8 +1097,23 @@ mod tests { assert!(matches!(err, DataFusionError::Plan(_))); } + /// A hand-built lookup has no build lease, hence no key list: the wire cannot + /// represent it, so the encoder aborts unless a membership-free conjunct can be kept. + fn unrepresentable_lookup(key: &Arc) -> Arc { + Arc::new(HashTableLookupExpr::new( + vec![Arc::clone(key)], + SeededRandomState::with_seed(0), + Arc::new(Map::HashMap(Box::new(JoinHashMapU32::with_capacity(0)))), + "hash_lookup".to_string(), + None, + )) as Arc + } + #[test] - fn dyn_filter_payload_hash_lookup_fallback_preserves_bounds() { + fn dyn_filter_payload_unconvertible_lookup_keeps_integer_bounds() { + // `bounds AND lookup` where the lookup has no key list: only the + // membership-bearing conjunct is dropped, so the integer bounds still travel as a + // regular payload instead of an always-true filter or an error. let schema = Arc::new(Schema::new(vec![Field::new( "device_id", DataType::Int32, @@ -520,30 +1125,224 @@ mod tests { Operator::GtEq, lit(10i32), )) as Arc; - let lookup = Arc::new(HashTableLookupExpr::new( - vec![Arc::clone(&device_id)], - SeededRandomState::with_seed(0), - Arc::new(Map::HashMap(Box::new(JoinHashMapU32::with_capacity(0)))), - "hash_lookup".to_string(), - None, + let upper_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::LtEq, + lit(20i32), + )) as Arc; + let bounds = Arc::new(BinaryExpr::new(lower_bound, Operator::And, upper_bound)) + as Arc; + let expr = Arc::new(BinaryExpr::new( + Arc::clone(&bounds), + Operator::And, + unrepresentable_lookup(&device_id), )) as Arc; - let expr = - Arc::new(BinaryExpr::new(lower_bound, Operator::And, lookup)) as Arc; - let payload = DynFilterPayload::from_datafusion_expr(&expr, 1024).unwrap(); - let decoded = payload - .decode_datafusion_expr(&TaskContext::default(), &schema, 1024) - .unwrap(); + // The predicate itself has no wire form: the lookup has no key list. + assert!(matches!( + encode_bounded_physical_expr_to_bytes(&expr, 1024), + Err(CommonQueryError::DynFilterUnsupported { .. }) + )); - assert!(!contains_expr::(&decoded)); - let decoded_display = decoded.to_string(); - assert!(decoded_display.contains("device_id")); - assert!(decoded_display.contains(">=")); - assert!(!decoded_display.contains("hash_lookup")); + // Keeping the membership-free top-level conjuncts turns it into a normal payload. + let decoded = decode(&expr, &schema, 1024); + let display = decoded.to_string(); + assert!( + display.contains(">= 10"), + "lower bound must survive: {display}" + ); + assert!( + display.contains("<= 20"), + "upper bound must survive: {display}" + ); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + // Both bounds are copied verbatim: no `AND` term was rewritten in place. + assert_eq!( + encode_physical_expr_to_bytes(&decoded).unwrap(), + encode_physical_expr_to_bytes(&bounds).unwrap() + ); } #[test] - fn dyn_filter_payload_oversized_inlist_falls_back_to_bounds() { + fn unrepresentable_lookup_drops_only_the_lookup_conjunct() { + // One unsupported lookup (no key list) between the bounds and a static `IN` list + // that is itself perfectly representable: only the lookup-bearing conjunct is + // dropped, so the list survives as a whole and the payload stays as narrow as the + // wire allows. + let schema = Arc::new(Schema::new(vec![Field::new( + "device_id", + DataType::Int32, + false, + )])); + let device_id = col("device_id", &schema).unwrap(); + let lower_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::GtEq, + lit(10i32), + )) as Arc; + let upper_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::LtEq, + lit(20i32), + )) as Arc; + let bounds = Arc::new(BinaryExpr::new( + Arc::clone(&lower_bound), + Operator::And, + Arc::clone(&upper_bound), + )) as Arc; + let in_list = Arc::new( + InListExpr::try_new( + Arc::clone(&device_id), + (10..20).map(lit).collect(), + false, + &schema, + ) + .unwrap(), + ) as Arc; + let expr = Arc::new(BinaryExpr::new( + Arc::new(BinaryExpr::new( + Arc::clone(&bounds), + Operator::And, + unrepresentable_lookup(&device_id), + )) as Arc, + Operator::And, + Arc::clone(&in_list), + )) as Arc; + + let decoded = decode(&expr, &schema, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES); + let display = decoded.to_string(); + assert!(display.contains(">= 10"), "{display}"); + assert!(display.contains("<= 20"), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + // The static `IN` list is representable, so it is kept verbatim. + assert!(has_expr::(&decoded), "{display}"); + assert_eq!( + split_conjunction(&decoded).len(), + 3, + "the two bounds and the static `IN` list survive: {display}" + ); + // The retained conjuncts are the original terms verbatim: the list was not + // rewritten, truncated or moved. + let retained_in_list = find_expr::(&decoded).expect("static IN survives"); + let retained_in_list = retained_in_list.downcast_ref::().unwrap(); + assert!(!retained_in_list.negated()); + assert_eq!(retained_in_list.list().len(), 10); + let retained = conjunction(vec![ + Arc::clone(&lower_bound), + Arc::clone(&upper_bound), + in_list, + ]); + assert_eq!( + encode_physical_expr_to_bytes(&decoded).unwrap(), + encode_physical_expr_to_bytes(&retained).unwrap() + ); + } + + #[test] + fn over_budget_membership_drops_the_oversized_in_list_too() { + // The same shape under a budget that only fits the bounds: keeping the + // representative-but-large `IN` list is not enough, so the retention widens once + // more and drops the `InListExpr`-bearing conjuncts as well. + let schema = Arc::new(Schema::new(vec![Field::new( + "device_id", + DataType::Int32, + false, + )])); + let device_id = col("device_id", &schema).unwrap(); + let lower_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::GtEq, + lit(10i32), + )) as Arc; + let upper_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::LtEq, + lit(20i32), + )) as Arc; + let bounds = Arc::new(BinaryExpr::new( + Arc::clone(&lower_bound), + Operator::And, + Arc::clone(&upper_bound), + )) as Arc; + let in_list = Arc::new( + InListExpr::try_new( + Arc::clone(&device_id), + (10..20).map(lit).collect(), + false, + &schema, + ) + .unwrap(), + ) as Arc; + let expr = Arc::new(BinaryExpr::new( + Arc::new(BinaryExpr::new( + Arc::clone(&bounds), + Operator::And, + unrepresentable_lookup(&device_id), + )) as Arc, + Operator::And, + in_list, + )) as Arc; + + let budget = encode_physical_expr_to_bytes(&bounds).unwrap().len(); + assert!(encode_bounded_physical_expr_to_bytes(&expr, budget).is_err()); + + let decoded = decode(&expr, &schema, budget); + let display = decoded.to_string(); + assert!(display.contains(">= 10"), "{display}"); + assert!(display.contains("<= 20"), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + assert_eq!( + encode_physical_expr_to_bytes(&decoded).unwrap(), + encode_physical_expr_to_bytes(&bounds).unwrap() + ); + } + + #[test] + fn dyn_filter_payload_negated_unconvertible_lookup_is_unsupported() { + // `NOT lookup` is a single top-level conjunct that carries the membership, so + // there is nothing safe to keep. Transport must not encode `NOT true` (or `true`) + // in its place: the predicate is reported as unsupported. + let schema = Arc::new(Schema::new(vec![Field::new( + "device_id", + DataType::Int32, + false, + )])); + let device_id = col("device_id", &schema).unwrap(); + let not_lookup = + Arc::new(NotExpr::new(unrepresentable_lookup(&device_id))) as Arc; + + assert_unsupported(DynFilterPayload::from_datafusion_expr(¬_lookup, 1024).unwrap_err()); + } + + #[test] + fn dyn_filter_payload_membership_under_or_is_unsupported() { + // `bound OR lookup` is a single top-level conjunct that carries the membership: + // rewriting it to `bound OR true` would be nothing but `true`, and keeping only + // the bound would *narrow* the predicate, so it is reported as unsupported. + let schema = Arc::new(Schema::new(vec![Field::new( + "device_id", + DataType::Int32, + false, + )])); + let device_id = col("device_id", &schema).unwrap(); + let lower_bound = Arc::new(BinaryExpr::new( + Arc::clone(&device_id), + Operator::GtEq, + lit(10i32), + )) as Arc; + let expr = Arc::new(BinaryExpr::new( + lower_bound, + Operator::Or, + unrepresentable_lookup(&device_id), + )) as Arc; + + assert_unsupported(DynFilterPayload::from_datafusion_expr(&expr, 1024).unwrap_err()); + } + + #[test] + fn dyn_filter_payload_over_budget_keeps_bounds() { let schema = Arc::new(Schema::new(vec![Field::new( "device_id", DataType::Int32, @@ -560,9 +1359,12 @@ mod tests { Operator::LtEq, lit(8255i32), )) as Arc; - let bounds = Arc::new(BinaryExpr::new(lower_bound, Operator::And, upper_bound)) - as Arc; - let in_list = Arc::new( + let bounds = Arc::new(BinaryExpr::new( + Arc::clone(&lower_bound), + Operator::And, + Arc::clone(&upper_bound), + )) as Arc; + let oversized_in = Arc::new( InListExpr::try_new( Arc::clone(&device_id), (8192..8256).map(lit).collect(), @@ -571,23 +1373,33 @@ mod tests { ) .unwrap(), ) as Arc; - let expr = Arc::new(BinaryExpr::new(Arc::clone(&bounds), Operator::And, in_list)) - as Arc; - let bounds_only = portable_remote_dyn_filter_expr(Arc::clone(&expr), true).unwrap(); - let bounds_only_size = encode_physical_expr_to_bytes(&bounds_only).unwrap().len(); - let full_size = encode_physical_expr_to_bytes(&expr).unwrap().len(); - assert!(full_size > bounds_only_size); + let expr = Arc::new(BinaryExpr::new( + Arc::clone(&bounds), + Operator::And, + oversized_in, + )) as Arc; - let payload = DynFilterPayload::from_datafusion_expr(&expr, bounds_only_size).unwrap(); - let decoded = payload - .decode_datafusion_expr(&TaskContext::default(), &schema, bounds_only_size) - .unwrap(); + // A caller-supplied limit that fits the bounds but not the oversized `IN` list: + // the membership conjunct is dropped and the bounds are encoded under the same + // budget. + let bounds_size = encode_physical_expr_to_bytes(&bounds).unwrap().len(); + assert!(encode_bounded_physical_expr_to_bytes(&expr, bounds_size).is_err()); - assert!(!contains_expr::(&decoded)); - let decoded_display = decoded.to_string(); - assert!(decoded_display.contains("device_id")); - assert!(decoded_display.contains(">=")); - assert!(decoded_display.contains("<=")); + let decoded = decode(&expr, &schema, bounds_size); + let display = decoded.to_string(); + assert!(display.contains(">= 8192"), "{display}"); + assert!(display.contains("<= 8255"), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + assert_eq!( + encode_physical_expr_to_bytes(&decoded).unwrap(), + encode_physical_expr_to_bytes(&bounds).unwrap() + ); + + // A budget that does not even fit the retained bounds is reported as too large: + // nothing that cannot be shipped may be replaced with `true`. + let true_size = encode_physical_expr_to_bytes(&lit(true)).unwrap().len(); + assert!(true_size < bounds_size); + assert_too_large(DynFilterPayload::from_datafusion_expr(&expr, true_size).unwrap_err()); } #[test] @@ -661,17 +1473,1069 @@ mod tests { )); } - fn contains_expr(expr: &Arc) -> bool { - let mut found = false; + #[test] + fn membership_value_bound_really_bounds_what_the_byte_budget_holds() { + // The count bound has to be *real*, not a restatement of its own formula: a + // payload of `B` bytes can never hold more than `B / MIN_ENCODED_LITERAL_BYTES` + // list literals. Prove it by encoding one literal more than the bound and checking + // that the resulting frame really exceeds `B`. + let budget = 512; + let bound = max_membership_values(budget); + assert!(bound > 1, "the bound must leave room for a real list"); + let schema = Schema::new(vec![Field::new("host", DataType::Utf8, true)]); + let minimal_in_list = |values: usize| { + Arc::new( + InListExpr::try_new( + Arc::new(Column::new("host", 0)), + (0..values).map(|_| lit("")).collect(), + false, + &schema, + ) + .unwrap(), + ) as Arc + }; + + let over_bound = encode_physical_expr_to_bytes(&minimal_in_list(bound + 1)).unwrap(); + assert!( + over_bound.len() > budget, + "{} literals must not fit a {budget}-byte budget, but the frame is {} bytes", + bound + 1, + over_bound.len() + ); + // The same payload is refused by the bounded encoder, so the derived bound and the + // encoder agree on where the budget ends. + assert!(matches!( + encode_bounded_physical_expr_to_bytes(&minimal_in_list(bound + 1), budget), + Err(CommonQueryError::DynFilterPayloadTooLarge { .. }) + )); + // The floor the bound is derived from must hold for the cheapest literal there is. + let cheapest = encode_physical_expr_to_bytes(&lit("")).unwrap().len(); + assert!( + cheapest >= MIN_ENCODED_LITERAL_BYTES, + "floor must hold for the cheapest literal, got {cheapest} bytes" + ); + } + + #[test] + fn utf8_list_literals_is_the_authoritative_value_check() { + let values: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c"])); + assert!(utf8_list_literals(&values, 3).is_some()); + // The count is bounded before a single `Literal` is built. + assert!(utf8_list_literals(&values, 2).is_none()); + assert!(utf8_list_literals(&values, 0).is_none()); + // NULL-containing, empty and non-`Utf8` arrays are refused here, not only by the + // `HashTableLookupExpr` accessor. + let with_null: ArrayRef = Arc::new(StringArray::from(vec![Some("a"), None])); + assert!(utf8_list_literals(&with_null, 8).is_none()); + let empty: ArrayRef = Arc::new(StringArray::from(Vec::<&str>::new())); + assert!(utf8_list_literals(&empty, 8).is_none()); + let not_utf8: ArrayRef = Arc::new(Int32Array::from(vec![1, 2])); + assert!(utf8_list_literals(¬_utf8, 8).is_none()); + } +} + +/// Remote dynamic filter membership recovered from *real* DataFusion build leases. +/// +/// The tests above build `HashTableLookupExpr` by hand, which always has +/// `build_lease = None`. These tests run an actual `HashJoinExec` so DataFusion +/// publishes lookups backed by a real build lease, which is the only way to prove that +/// the key list (and the membership-free conjunct retention) actually works. +/// +/// Most remote predicates are checked twice: structurally (what the payload contains) +/// and by **evaluating** the producer predicate and the decoded predicate on the same +/// `NULL`-free rows. Evaluation pins the contract that applies here: the converted +/// membership is the build key list, so on `NULL`-free keys only, the decoded predicate +/// keeps exactly the rows the producer kept. It is a pruning hint, not a universal +/// equivalence — see `not_lookup_becomes_not_in` for the `NULL` caveat. +#[cfg(test)] +mod remote_membership_tests { + use std::collections::BTreeMap; + + use datafusion::arrow::array::{ + ArrayRef, BooleanArray, Float64Array, Int32Array, StringArray, StringDictionaryBuilder, + }; + use datafusion::arrow::datatypes::{DataType, Field, Int32Type, Schema}; + use datafusion::arrow::record_batch::RecordBatch; + use datafusion::physical_expr::expressions::{ + BinaryExpr, CaseExpr, Column, InListExpr, IsNullExpr, Literal, NotExpr, lit, + }; + use datafusion::physical_plan::PhysicalExpr; + use datafusion::physical_plan::joins::{HashExpr, HashTableLookupExpr}; + use datafusion_expr::Operator; + + use super::remote_membership_test_support::{ + RealDynFilter, key, one_column_schema, partitions, run_real_hash_join, utf8, utf8_join, + }; + use super::*; + + fn in_lists(expr: &Arc) -> Vec> { + let mut found = Vec::new(); expr.apply(|node| { - if node.is::() { - found = true; + if node.is::() { + found.push(Arc::clone(node)); + } + Ok(TreeNodeRecursion::Continue) + }) + .unwrap(); + found + } + + fn in_list_values(expr: &Arc) -> Vec { + let in_list = expr.downcast_ref::().unwrap(); + assert!(!in_list.negated()); + in_list + .list() + .iter() + .map(|literal| { + literal + .downcast_ref::() + .unwrap() + .value() + .to_string() + .trim_matches('\'') + .to_string() + }) + .collect() + } + + fn assert_has_bounds(expr: &Arc) { + let display = expr.to_string(); + assert!(display.contains(">="), "missing lower bound: {display}"); + assert!(display.contains("<="), "missing upper bound: {display}"); + } + + fn utf8_batch(schema: &Arc, values: &[&str]) -> RecordBatch { + RecordBatch::try_new(Arc::clone(schema), vec![utf8(values)]).unwrap() + } + + fn selected(expr: &Arc, batch: &RecordBatch) -> Vec { + let array = expr + .evaluate(batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let booleans = array.as_any().downcast_ref::().unwrap(); + (0..batch.num_rows()) + .map(|index| booleans.value(index)) + .collect() + } + + /// Widening check: the decoded predicate keeps every row the producer kept. + /// + /// Only used on `NULL`-free rows, where the build-key membership is exact for the + /// converted node. + fn assert_fail_open( + original: &Arc, + decoded: &Arc, + batch: &RecordBatch, + ) -> (Vec, Vec) { + let original_selected = selected(original, batch); + let decoded_selected = selected(decoded, batch); + for (row, (kept, kept_after)) in original_selected + .iter() + .zip(decoded_selected.iter()) + .enumerate() + { + assert!( + kept_after >= kept, + "decoded predicate dropped row {row}: producer={kept} decoded={kept_after} \ + (decoded predicate: {decoded})" + ); + } + (original_selected, decoded_selected) + } + + /// Branch chosen by the CASE routing expression for every row. + fn routed_branch(case: &CaseExpr, batch: &RecordBatch) -> Vec { + let routing = case.expr().expect("hash routing operand"); + let array = routing + .evaluate(batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let branches = array + .as_any() + .downcast_ref::() + .unwrap(); + (0..batch.num_rows()) + .map(|index| branches.value(index)) + .collect() + } + + fn hash_seed(expr: &Arc) -> u64 { + let mut seed = None; + expr.apply(|node| { + if let Some(hash) = node.downcast_ref::() { + seed = Some(hash.seed()); Ok(TreeNodeRecursion::Stop) } else { Ok(TreeNodeRecursion::Continue) } }) .unwrap(); - found + seed.expect("hash routing expression") + } + + /// `trace_id` values used to evaluate recovered filters. + fn evaluation_batch(schema: &Arc, values: &[String]) -> RecordBatch { + let borrowed = values.iter().map(String::as_str).collect::>(); + utf8_batch(schema, &borrowed) + } + + /// UTF-8 batch that can contain SQL NULLs, for wrapper semantics. + fn nullable_evaluation_batch(schema: &Arc, values: &[Option]) -> RecordBatch { + let array = StringArray::from(values.to_vec()); + RecordBatch::try_new(Arc::clone(schema), vec![Arc::new(array) as ArrayRef]).unwrap() + } + + #[tokio::test] + async fn membership_round_trips_real_utf8_build_lease() { + let schema = one_column_schema("trace_id", DataType::Utf8); + let (build, probe, values) = utf8_join(1); + let trace_id = key("trace_id", 0); + let filter = run_real_hash_join( + 1, + Arc::clone(&schema), + Arc::clone(&schema), + build, + probe, + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + + // The producer predicate really holds a runtime-only lookup... + assert!(has_expr::(&filter.expr)); + + // ...and it is serialized as exact build membership, not as `true`. + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + assert!(!has_expr::(&decoded)); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "expected one recovered InList: {decoded}"); + assert_eq!(in_list_values(&lists[0]), values); + assert_has_bounds(&decoded); + + // Members are selected, non-members are not, nothing else changes. + let mut rows = values.clone(); + rows.push("miss-not-a-member".to_string()); + let batch = evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = + assert_fail_open(&filter.expr, &decoded, &batch); + for row in 0..values.len() { + assert!(original_selected[row], "member row {row} must be selected"); + assert!(decoded_selected[row], "member row {row} must stay selected"); + } + assert!( + !decoded_selected[values.len()], + "non-member must not be selected" + ); + } + + #[tokio::test] + async fn membership_preserves_three_branch_case_structure() { + let schema = one_column_schema("trace_id", DataType::Utf8); + let (build, probe, values) = utf8_join(3); + let trace_id = key("trace_id", 0); + let filter = run_real_hash_join( + 3, + Arc::clone(&schema), + Arc::clone(&schema), + build, + probe, + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + + // Three hash-routed partitions produce a three-branch CASE. + let original_case = filter + .expr + .downcast_ref::() + .expect("real three-branch CASE"); + assert_eq!(original_case.when_then_expr().len(), 3); + + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let case = decoded.downcast_ref::().expect("CASE preserved"); + assert_eq!(case.when_then_expr().len(), 3); + assert!(case.else_expr().is_some()); + + // Routing is untouched: same operand expression and same `HashExpr` seed, so + // the receiver selects the same branch for every row. + assert_eq!( + original_case.expr().unwrap().to_string(), + case.expr().unwrap().to_string() + ); + assert_eq!( + hash_seed(original_case.expr().unwrap()), + hash_seed(case.expr().unwrap()) + ); + + // Branch-specific membership: the WHEN literal names the branch, so the values + // recovered for each branch can be checked individually. + let mut branch_values: BTreeMap> = BTreeMap::new(); + for (when, then) in case.when_then_expr() { + let branch = match when.downcast_ref::().unwrap().value() { + ScalarValue::UInt64(Some(branch)) => *branch, + other => panic!("unexpected CASE routing literal: {other:?}"), + }; + let lists = in_lists(then); + assert_eq!(lists.len(), 1, "each branch keeps its membership: {then}"); + let mut branch_list = in_list_values(&lists[0]); + branch_list.sort(); + assert!( + branch_values.insert(branch, branch_list).is_none(), + "branch {branch} appears twice" + ); + assert_has_bounds(then); + } + assert_eq!(branch_values.len(), 3, "three distinct branches"); + + // Every build key is recovered exactly once, in its own branch. + let mut recovered = branch_values + .values() + .flatten() + .cloned() + .collect::>(); + recovered.sort(); + assert_eq!(recovered, values); + + // Evaluation proves branch-specific routing: a row is selected iff its key is in + // the membership of the branch the routing sends it to. + let mut rows = values.clone(); + rows.extend((0..4).map(|i| format!("miss-{i:02}"))); + let batch = evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = + assert_fail_open(&filter.expr, &decoded, &batch); + let branches = routed_branch(case, &batch); + for (row, value) in rows.iter().enumerate() { + let branch = branches[row]; + let expected = branch_values + .get(&branch) + .is_some_and(|members| members.binary_search(value).is_ok()); + assert_eq!( + decoded_selected[row], expected, + "row {value} routed to branch {branch} must be selected iff it is a \ + member of that branch's list" + ); + // Build members must keep their decision; misses only ever widen. + if expected { + assert!( + original_selected[row], + "producer must select its own build member {value}" + ); + assert!(decoded_selected[row]); + } + } + assert!( + decoded_selected + .iter() + .filter(|selected| **selected) + .count() + >= values.len(), + "every build member must still be selected" + ); + } + + /// Runs a real one-partition `Utf8` hash join over `key_count` distinct keys. + async fn real_join_with_keys(key_count: usize) -> (Arc, RealDynFilter, Vec) { + let schema = one_column_schema("trace_id", DataType::Utf8); + let trace_id = key("trace_id", 0); + let values = (0..key_count) + .map(|i| format!("trace-{i:04}")) + .collect::>(); + let borrowed = values.iter().map(String::as_str).collect::>(); + let build = partitions(&schema, vec![vec![utf8(&borrowed)]]); + let probe = partitions(&schema, vec![vec![utf8(&borrowed)]]); + let filter = run_real_hash_join( + 1, + Arc::clone(&schema), + Arc::clone(&schema), + build, + probe, + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + (schema, filter, values) + } + + #[tokio::test] + async fn bounds_survive_unrepresentable_integer_lookup() { + // Integer keys are not a single-column `Utf8` membership, so this real lookup has + // no wire form. Retention keeps the top-level bounds and drops the lookup + // conjunct, so the decoded predicate only ever gets wider. + let schema = one_column_schema("k", DataType::Int32); + let k = key("k", 0); + let filter = run_real_hash_join( + 1, + Arc::clone(&schema), + Arc::clone(&schema), + partitions( + &schema, + vec![vec![Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef]], + ), + partitions( + &schema, + vec![vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 10])) as ArrayRef + ]], + ), + vec![(Arc::clone(&k), Arc::clone(&k))], + ) + .await; + let lookup = find_expr::(&filter.expr).expect("real lookup"); + + let lower_bound = Arc::new(BinaryExpr::new(Arc::clone(&k), Operator::GtEq, lit(2i32))) + as Arc; + let upper_bound = Arc::new(BinaryExpr::new(Arc::clone(&k), Operator::LtEq, lit(4i32))) + as Arc; + let bounds = Arc::new(BinaryExpr::new(lower_bound, Operator::And, upper_bound)) + as Arc; + let expr = Arc::new(BinaryExpr::new(Arc::clone(&bounds), Operator::And, lookup)) + as Arc; + + let decoded = decode( + &expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let display = decoded.to_string(); + assert!( + display.contains(">= 2"), + "lower bound must survive: {display}" + ); + assert!( + display.contains("<= 4"), + "upper bound must survive: {display}" + ); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + + // The producer and the decoded predicate agree on every row the producer kept and + // only diverge on the non-member `4`, where the retained payload widens. + let batch = RecordBatch::try_new( + Arc::clone(&schema), + vec![Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 10])) as ArrayRef], + ) + .unwrap(); + let (original_selected, decoded_selected) = assert_fail_open(&expr, &decoded, &batch); + assert_eq!( + original_selected, + vec![false, true, true, false, false, false] + ); + assert_eq!( + decoded_selected, + vec![false, true, true, true, false, false] + ); + } + + #[tokio::test] + async fn small_budget_keeps_bounds_and_drops_real_membership() { + // A real `Utf8` lookup next to integer bounds, shipped under a caller-supplied + // limit that fits only the bounds: retention drops the membership conjunct and + // still sends the bounds, which can only widen the producer predicate. + let probe_schema = Arc::new(Schema::new(vec![ + Field::new("trace_id", DataType::Utf8, true), + Field::new("k", DataType::Int32, true), + ])); + let build_schema = one_column_schema("trace_id", DataType::Utf8); + let long = "t".repeat(256); + let values = (0..24) + .map(|i| format!("{long}-{i:02}")) + .collect::>(); + let borrowed = values.iter().map(String::as_str).collect::>(); + let build_key = key("trace_id", 0); + let probe_key = key("trace_id", 0); + let filter = run_real_hash_join( + 1, + Arc::clone(&build_schema), + Arc::clone(&probe_schema), + partitions(&build_schema, vec![vec![utf8(&borrowed)]]), + partitions( + &probe_schema, + vec![vec![ + utf8(&borrowed), + Arc::new(Int32Array::from(vec![0i32; values.len()])) as ArrayRef, + ]], + ), + vec![(Arc::clone(&build_key), Arc::clone(&probe_key))], + ) + .await; + let lookup = find_expr::(&filter.expr).expect("real lookup"); + + let k = key("k", 1); + let lower_bound = Arc::new(BinaryExpr::new(Arc::clone(&k), Operator::GtEq, lit(2i32))) + as Arc; + let upper_bound = Arc::new(BinaryExpr::new(Arc::clone(&k), Operator::LtEq, lit(5i32))) + as Arc; + let bounds = Arc::new(BinaryExpr::new(lower_bound, Operator::And, upper_bound)) + as Arc; + let expr = Arc::new(BinaryExpr::new(Arc::clone(&bounds), Operator::And, lookup)) + as Arc; + + // Only the bounds fit: the membership of 24 long keys is far beyond this budget. + let bounds_size = encode_physical_expr_to_bytes(&bounds).unwrap().len(); + let decoded = decode(&expr, &filter.probe_schema, bounds_size); + let display = decoded.to_string(); + assert!( + display.contains(">= 2"), + "lower bound must survive: {display}" + ); + assert!( + display.contains("<= 5"), + "upper bound must survive: {display}" + ); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + + // Evaluation: a member inside the bounds is kept by both, a member outside the + // bounds is dropped by both, and only the non-member is widened. + let trace_ids = vec![ + values[0].as_str(), + values[1].as_str(), + "miss-not-a-member", + values[2].as_str(), + ]; + let batch = RecordBatch::try_new( + Arc::clone(&probe_schema), + vec![ + utf8(&trace_ids), + Arc::new(Int32Array::from(vec![2i32, 6, 3, 5])) as ArrayRef, + ], + ) + .unwrap(); + let (original_selected, decoded_selected) = assert_fail_open(&expr, &decoded, &batch); + assert_eq!(original_selected, vec![true, false, false, true]); + assert_eq!(decoded_selected, vec![true, false, true, true]); + } + + #[tokio::test] + async fn real_642_key_hash_join_becomes_exact_642_literal_in() { + let (schema, filter, values) = real_join_with_keys(642).await; + assert_eq!(values.len(), 642); + // The producer predicate really holds a runtime-only lookup... + assert!(has_expr::(&filter.expr)); + + // ...and it is written as one literal per build key within the payload budget. + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + assert!(!has_expr::(&decoded)); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "expected one converted InList: {decoded}"); + let mut recovered = in_list_values(&lists[0]); + assert_eq!(recovered.len(), 642, "one literal per build key"); + recovered.sort(); + assert_eq!(recovered, values); + + // The membership selects exactly the build keys. + let mut rows = values.clone(); + rows.push("miss-not-a-member".to_string()); + let batch = evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = + assert_fail_open(&filter.expr, &decoded, &batch); + for row in 0..values.len() { + assert!( + original_selected[row], + "build key row {row} must be selected" + ); + assert!( + decoded_selected[row], + "build key row {row} must stay selected" + ); + } + assert!( + !decoded_selected[values.len()], + "non-member must not be selected" + ); + } + + #[tokio::test] + async fn large_build_side_is_refused_before_materializing_literals() { + // A build side far larger than a small budget can hold must be refused by the + // derived count bound *before* the encoder creates one `Literal` per build key. + let (_, filter, values) = real_join_with_keys(1024).await; + let lookup = find_expr::(&filter.expr).expect("real lookup"); + let lookup = lookup.downcast_ref::().unwrap(); + + // DataFusion itself would hand out every build key ... + let unbounded = lookup + .build_key_membership(usize::MAX, usize::MAX) + .expect("the build lease holds the keys"); + assert_eq!(unbounded.values().len(), values.len()); + + // ... but a budget that only fits the producer's own bounds leaves room for a few + // literals, so the count bound derived from it refuses the list outright. + let bounds = conjunction( + split_conjunction(&filter.expr) + .into_iter() + .filter(|conjunct| !has_expr::(conjunct)) + .cloned() + .collect::>(), + ); + let budget = encode_physical_expr_to_bytes(&bounds).unwrap().len(); + assert!( + max_membership_values(budget) < values.len(), + "the derived count bound must stay below the build side size" + ); + // The refusal is reported as a *size* refusal, not as a predicate without a wire + // form: the key list exists, it just does not fit this budget. + assert!( + matches!( + RemoteMembership::from_lookup(lookup, budget), + Err(MembershipRefusal::OverBudget { build_keys }) if build_keys == values.len() + ), + "an over-large build key list must be refused as over budget" + ); + + // Nothing of the 1024-key membership reaches the payload. + let decoded = decode(&filter.expr, &filter.probe_schema, budget); + assert!(!has_expr::(&decoded), "{decoded}"); + assert!(!has_expr::(&decoded), "{decoded}"); + assert_has_bounds(&decoded); + } + + #[tokio::test] + async fn lookup_or_key_is_null_preserves_wrapper_and_becomes_in_list() { + // The `key IS NULL OR ` wrapper of a null-aware join is an ordinary + // node: it stays in the payload and only the lookup inside it is converted. + let (schema, filter, values) = real_join_with_keys(8).await; + let lookup = find_expr::(&filter.expr).expect("real lookup"); + let trace_id = key("trace_id", 0); + let wrapped = Arc::new(BinaryExpr::new( + Arc::clone(&lookup), + Operator::Or, + Arc::new(IsNullExpr::new(Arc::clone(&trace_id))), + )) as Arc; + + let decoded = decode( + &wrapped, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let display = decoded.to_string(); + assert!( + display.contains("IS NULL"), + "IS NULL wrapper must survive: {display}" + ); + assert!(!display.contains("hash_lookup"), "{display}"); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "expected one converted InList: {display}"); + assert_eq!(in_list_values(&lists[0]), values); + + let mut rows = values + .iter() + .map(|value| Some(value.clone())) + .collect::>(); + rows.push(Some("miss-not-a-member".to_string())); + rows.push(None); + let batch = nullable_evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = assert_fail_open(&wrapped, &decoded, &batch); + for row in 0..values.len() { + assert!(original_selected[row]); + assert!(decoded_selected[row]); + } + assert!( + original_selected[rows.len() - 1], + "producer `lookup OR key IS NULL` keeps a NULL probe" + ); + assert!( + decoded_selected[rows.len() - 1], + "decoded `IN ... OR key IS NULL` must also keep a NULL probe" + ); + } + + #[tokio::test] + async fn null_build_keys_keep_only_the_safe_bounds() { + // The key-list accessor requires `NULL`-free build keys, so a build side with a + // `NULL` key has no membership. This producer shape is `bounds AND lookup`: the + // bounds DataFusion derived from the non-`NULL` keys are retained, and the lookup + // is dropped instead of the predicate being replaced with `true`. + let schema = one_column_schema("trace_id", DataType::Utf8); + let trace_id = key("trace_id", 0); + let build_values: ArrayRef = Arc::new(StringArray::from(vec![ + Some("trace-a"), + None, + Some("trace-b"), + ])); + let filter = run_real_hash_join( + 1, + Arc::clone(&schema), + Arc::clone(&schema), + partitions(&schema, vec![vec![build_values]]), + partitions(&schema, vec![vec![utf8(&["trace-a", "trace-b", "miss"])]]), + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + assert!(has_expr::(&filter.expr)); + let producer = filter.expr.to_string(); + assert!(producer.contains(">= trace-a"), "{producer}"); + assert!(producer.contains("<= trace-b"), "{producer}"); + + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let display = decoded.to_string(); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + assert!(display.contains(">= trace-a"), "{display}"); + assert!(display.contains("<= trace-b"), "{display}"); + assert_ne!(display, "true", "transport must not widen to `true`"); + + // The retained conjuncts are the producer bounds verbatim. + let bounds = split_conjunction(&filter.expr) + .into_iter() + .filter(|conjunct| !has_expr::(conjunct)) + .cloned() + .collect::>(); + assert_eq!( + encode_physical_expr_to_bytes(&decoded).unwrap(), + encode_physical_expr_to_bytes(&conjunction(bounds)).unwrap() + ); + + // Widening only: every build key row the producer kept is still kept. + let rows = ["trace-a", "trace-b", "miss"] + .map(ToString::to_string) + .to_vec(); + let batch = evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = + assert_fail_open(&filter.expr, &decoded, &batch); + assert_eq!(original_selected, vec![true, true, false]); + assert!(decoded_selected[0], "build key row must stay selected"); + assert!(decoded_selected[1], "build key row must stay selected"); + } + + /// `NOT lookup` becomes `NOT (key IN (...))`. + /// + /// This narrow contract is about **positive pruning**: the membership is the build + /// key list of the join, so converting the node in place structurally preserves the + /// negation instead of substituting `true` into it. It is *not* a claim of + /// universal equivalence with the join: `IN`/`NOT IN` are `NULL`-propagating while + /// a hash lookup can be definite on a `NULL` probe key, so a `NULL` key that the + /// lookup keeps can still be dropped by `NOT IN`. The rows below are non-`NULL`, + /// which is the shape the hint is meant for. + #[tokio::test] + async fn not_lookup_becomes_not_in() { + let (schema, filter, values) = real_join_with_keys(8).await; + let lookup = find_expr::(&filter.expr).expect("real lookup"); + let not_lookup = Arc::new(NotExpr::new(Arc::clone(&lookup))) as Arc; + + let decoded = decode( + ¬_lookup, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let display = decoded.to_string(); + assert!( + display.starts_with("NOT "), + "NOT must survive the wire: {display}" + ); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "expected one converted InList: {display}"); + assert_eq!(in_list_values(&lists[0]), values); + + let mut rows = values.clone(); + rows.push("miss-not-a-member".to_string()); + let batch = evaluation_batch(&schema, &rows); + let (original_selected, decoded_selected) = assert_fail_open(¬_lookup, &decoded, &batch); + assert!(original_selected[..values.len()].iter().all(|kept| !*kept)); + assert!( + original_selected[values.len()], + "producer `NOT lookup` keeps the non-member" + ); + assert!(decoded_selected[..values.len()].iter().all(|kept| !*kept)); + assert!( + decoded_selected[values.len()], + "decoded `NOT IN` keeps the non-member" + ); + } + + #[tokio::test] + async fn unsupported_float_composite_and_dictionary_keys_keep_only_safe_conjuncts() { + // Float64 keys: the key-list accessor only exposes a single `Utf8` column. + let float_schema = one_column_schema("k", DataType::Float64); + let float_on = key("k", 0); + let float_filter = run_real_hash_join( + 2, + Arc::clone(&float_schema), + Arc::clone(&float_schema), + partitions( + &float_schema, + vec![ + vec![Arc::new(Float64Array::from(vec![0.0, -0.0])) as ArrayRef], + vec![Arc::new(Float64Array::from(vec![1.5])) as ArrayRef], + ], + ), + partitions( + &float_schema, + vec![ + vec![Arc::new(Float64Array::from(vec![0.0, 2.5])) as ArrayRef], + vec![Arc::new(Float64Array::from(vec![1.5])) as ArrayRef], + ], + ), + vec![(Arc::clone(&float_on), Arc::clone(&float_on))], + ) + .await; + assert!(has_expr::(&float_filter.expr)); + // The partitioned producer publishes a per-partition `CASE`, so the only + // top-level conjunct carries the membership and there is nothing safe to keep: + // the predicate is reported as unsupported instead of becoming `true`. + assert_unsupported( + DynFilterPayload::from_datafusion_expr( + &float_filter.expr, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap_err(), + ); + + // Composite (two-column) keys: not a single-column membership. Both columns + // take part in the hash partitioning, exactly like a real two-key join. + let composite_schema = Arc::new(Schema::new(vec![ + Field::new("trace_id", DataType::Utf8, true), + Field::new("host", DataType::Utf8, true), + ])); + let composite_on = vec![ + (key("trace_id", 0), key("trace_id", 0)), + (key("host", 1), key("host", 1)), + ]; + let composite_filter = run_real_hash_join( + 2, + Arc::clone(&composite_schema), + Arc::clone(&composite_schema), + partitions( + &composite_schema, + vec![ + vec![utf8(&["trace-0", "trace-1"]), utf8(&["host-a", "host-a"])], + vec![utf8(&["trace-2"]), utf8(&["host-b"])], + ], + ), + partitions( + &composite_schema, + vec![ + vec![utf8(&["trace-0", "trace-9"]), utf8(&["host-a", "host-z"])], + vec![utf8(&["trace-2"]), utf8(&["host-b"])], + ], + ), + composite_on, + ) + .await; + assert!(has_expr::(&composite_filter.expr)); + // Same per-partition `CASE` shape as the float case: unsupported, not `true`. + assert_unsupported( + DynFilterPayload::from_datafusion_expr( + &composite_filter.expr, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap_err(), + ); + + // Dictionary keys: not a plain `Utf8` key column. + let dict_type = DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8)); + let dict_schema = one_column_schema("trace_id", dict_type); + let dict_on = key("trace_id", 0); + let mut dict_builder = StringDictionaryBuilder::::new(); + dict_builder.append_value("trace-0"); + dict_builder.append_value("trace-1"); + let dict_values: ArrayRef = Arc::new(dict_builder.finish()); + let dict_filter = run_real_hash_join( + 2, + Arc::clone(&dict_schema), + Arc::clone(&dict_schema), + partitions(&dict_schema, vec![vec![Arc::clone(&dict_values)]]), + partitions(&dict_schema, vec![vec![dict_values]]), + vec![(Arc::clone(&dict_on), Arc::clone(&dict_on))], + ) + .await; + assert!(has_expr::(&dict_filter.expr)); + let decoded = decode( + &dict_filter.expr, + &dict_filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + // Dictionary keys are not a plain `Utf8` key column either. This producer shape + // is a flat `bounds AND lookup`, so retention keeps the bounds conjunct that + // DataFusion itself published and drops only the membership. + let display = decoded.to_string(); + assert!(!has_expr::(&decoded), "{display}"); + assert!(!has_expr::(&decoded), "{display}"); + assert!( + display.contains(">="), + "lower bound must survive: {display}" + ); + assert!( + display.contains("<="), + "upper bound must survive: {display}" + ); + } + + #[tokio::test] + async fn producer_paths_encode_150_and_1024_keys() { + // The two producer shapes that can carry a build key list: the static `IN` list + // pushdown (150 and 1024 are the pushdown limits of the runtime experiment) and + // the hash lookup. Neither is capped by an encoder-side value/byte policy: the + // only bound is the payload budget itself. + for key_count in [150, 1024] { + let (schema, filter, values) = real_join_with_keys(key_count).await; + assert!(has_expr::(&filter.expr)); + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "{key_count} keys: {decoded}"); + assert_eq!(in_list_values(&lists[0]), values); + + // The static `IN` list pushdown of the same size encodes as well. + let in_list = Arc::new( + InListExpr::try_new( + key("trace_id", 0), + values.iter().map(|value| lit(value.clone())).collect(), + false, + &schema, + ) + .unwrap(), + ) as Arc; + let decoded = decode( + &in_list, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + let lists = in_lists(&decoded); + assert_eq!(lists.len(), 1, "{key_count} static keys: {decoded}"); + assert_eq!(in_list_values(&lists[0]), values); + } + } + + #[tokio::test] + async fn over_budget_membership_case_is_a_size_refusal() { + let schema = one_column_schema("trace_id", DataType::Utf8); + // Long values so the membership lists dominate the payload. + let long = "t".repeat(256); + let values = (0..24) + .map(|i| format!("{long}-{i:02}")) + .collect::>(); + let borrowed = values.iter().map(String::as_str).collect::>(); + let build = partitions(&schema, vec![vec![utf8(&borrowed)]]); + let probe = partitions(&schema, vec![vec![utf8(&borrowed)]]); + let trace_id = key("trace_id", 0); + let filter = run_real_hash_join( + 3, + Arc::clone(&schema), + Arc::clone(&schema), + build, + probe, + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + assert!(has_expr::(&filter.expr)); + + // The real payload budget carries all three branch memberships exactly. + let decoded = decode( + &filter.expr, + &filter.probe_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ); + assert_eq!(in_lists(&decoded).len(), 3); + let case = decoded.downcast_ref::().expect("CASE preserved"); + assert_eq!(case.when_then_expr().len(), 3); + + // The whole predicate is one top-level `CASE` conjunct that carries the + // membership, so a budget that only fits `true` has nothing safe to keep: + // membership is never rewritten inside `CASE` and the payload is an explicit + // error instead of an always-true filter. The predicates *are* representable, they + // only do not fit, so the refusal is a size one and not "no wire form". + let true_size = encode_physical_expr_to_bytes(&lit(true)).unwrap().len(); + assert!(true_size < REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES); + assert_too_large( + DynFilterPayload::from_datafusion_expr(&filter.expr, true_size).unwrap_err(), + ); + } + + #[test] + fn over_budget_negation_is_a_size_refusal() { + let schema = Arc::new(Schema::new(vec![Field::new("host", DataType::Utf8, false)])); + let host = Arc::new(Column::new("host", 0)) as Arc; + let long = "h".repeat(256); + let in_list = Arc::new( + InListExpr::try_new( + Arc::clone(&host), + (0..16).map(|i| lit(format!("{long}-{i:02}"))).collect(), + false, + &schema, + ) + .unwrap(), + ) as Arc; + // `NOT (host IN (...))`: dropping the list in place would evaluate `NOT true` + // = false and drop every row, so the whole conjunct is dropped. Nothing is left + // to ship and the predicate itself is representable, so the refusal is a size one. + let not_in = Arc::new(NotExpr::new(Arc::clone(&in_list))) as Arc; + let true_size = encode_physical_expr_to_bytes(&lit(true)).unwrap().len(); + + assert_too_large(DynFilterPayload::from_datafusion_expr(¬_in, true_size).unwrap_err()); + } + + #[tokio::test] + async fn recovered_payload_uses_only_existing_wire_variants() { + let schema = one_column_schema("trace_id", DataType::Utf8); + let (build, probe, values) = utf8_join(3); + let trace_id = key("trace_id", 0); + let filter = run_real_hash_join( + 3, + Arc::clone(&schema), + Arc::clone(&schema), + build, + probe, + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + let max = REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES; + + // The recovered payload decodes with the default extension codec, which rejects + // every expression variant it does not know: a new wire variant would fail here. + let payload = DynFilterPayload::from_datafusion_expr(&filter.expr, max).unwrap(); + let bytes = match &payload { + DynFilterPayload::Datafusion(bytes) => bytes.clone(), + }; + let decoded = payload + .decode_datafusion_expr(&TaskContext::default(), &filter.probe_schema, max) + .unwrap(); + assert_eq!(in_lists(&decoded).len(), 3); + assert_eq!( + hash_seed( + filter + .expr + .downcast_ref::() + .unwrap() + .expr() + .unwrap() + ), + hash_seed(decoded.downcast_ref::().unwrap().expr().unwrap()), + "HashExpr routing seed must survive the wire" + ); + + // Receivers still accept a plain `true` payload: an older sender could publish one, + // and a producer may legitimately publish `true` itself. This transport no longer + // synthesizes `true` as an encoding fallback, so the decode-side compatibility stays + // while the encode-side fallback is gone. + let legacy_bytes = encode_physical_expr_to_bytes(&lit(true)).unwrap(); + assert!(legacy_bytes.len() < bytes.len()); + let legacy_decoded = DynFilterPayload::Datafusion(legacy_bytes) + .decode_datafusion_expr(&TaskContext::default(), &filter.probe_schema, max) + .unwrap(); + assert_eq!(legacy_decoded.to_string(), "true"); + let mut rows = values.clone(); + rows.push("miss-not-a-member".to_string()); + let batch = evaluation_batch(&schema, &rows); + assert_fail_open(&filter.expr, &legacy_decoded, &batch); } } diff --git a/src/common/query/src/request/remote_membership_test_support.rs b/src/common/query/src/request/remote_membership_test_support.rs new file mode 100644 index 00000000000..6bcc16d9b49 --- /dev/null +++ b/src/common/query/src/request/remote_membership_test_support.rs @@ -0,0 +1,194 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Test-only helpers that produce *real* DataFusion dynamic filters. +//! +//! A `HashTableLookupExpr` built by hand always has `build_lease = None`, so it cannot +//! prove that build-side membership recovery works. These helpers run an actual +//! `HashJoinExec`, which publishes lookups backed by real build leases, and return the +//! code of the dynamic filter it produced. +//! +//! The helpers live behind `test`/the `testing` feature so both `common-query` and the +//! distributed-plan tests can exercise the same fixture. + +use std::sync::Arc; + +use datafusion::arrow::array::{ArrayRef, StringArray}; +use datafusion::arrow::datatypes::{DataType, Field, Schema}; +use datafusion::arrow::record_batch::RecordBatch; +use datafusion::execution::TaskContext; +use datafusion::execution::config::SessionConfig; +use datafusion::execution::session_state::SessionStateBuilder; +use datafusion::physical_expr::expressions::{Column, DynamicFilterPhysicalExpr, lit}; +use datafusion::physical_plan::filter::FilterExec; +use datafusion::physical_plan::joins::{HashJoinExec, PartitionMode}; +use datafusion::physical_plan::repartition::RepartitionExec; +use datafusion::physical_plan::{ExecutionPlan, Partitioning, PhysicalExpr}; +use datafusion_common::config::ConfigOptions; +use datafusion_common::{JoinType, NullEquality}; +use datafusion_datasource::memory::MemorySourceConfig; +use futures_util::TryStreamExt; + +/// A real hash-join dynamic filter and the probe schema it is bound to. +pub struct RealDynFilter { + /// The predicate DataFusion published for the join's probe side. + pub expr: Arc, + /// Schema of the probe side the predicate applies to. + pub probe_schema: Arc, +} + +/// Schema with a single nullable column. +pub fn one_column_schema(name: &str, data_type: DataType) -> Arc { + Arc::new(Schema::new(vec![Field::new(name, data_type, true)])) +} + +/// Turns one column vector per partition into one `RecordBatch` per partition. +pub fn partitions(schema: &Arc, columns: Vec>) -> Vec> { + columns + .into_iter() + .map(|columns| vec![RecordBatch::try_new(Arc::clone(schema), columns).unwrap()]) + .collect() +} + +/// A column reference by name and index. +pub fn key(name: &str, index: usize) -> Arc { + Arc::new(Column::new(name, index)) +} + +/// A `Utf8` array. +pub fn utf8(values: &[&str]) -> ArrayRef { + Arc::new(StringArray::from(values.to_vec())) +} + +/// Runs a real `HashJoinExec` and returns the dynamic filter it published. +/// +/// The join is forced onto the `Map` pushdown strategy (the only strategy that +/// produces a runtime lookup) by setting the `IN`-list pushdown limit to zero, and +/// both children are hash-partitioned on the complete join-key vector, which is what +/// DataFusion requires before it publishes a partitioned dynamic filter. +pub async fn run_real_hash_join( + partition_count: usize, + left_schema: Arc, + right_schema: Arc, + build_partitions: Vec>, + probe_partitions: Vec>, + on: Vec<(Arc, Arc)>, +) -> RealDynFilter { + assert!(partition_count >= 1); + + let left_scan: Arc = + MemorySourceConfig::try_new_exec(&build_partitions, Arc::clone(&left_schema), None) + .unwrap(); + let right_scan: Arc = + MemorySourceConfig::try_new_exec(&probe_partitions, Arc::clone(&right_schema), None) + .unwrap(); + + let (left, right_scan) = if partition_count > 1 { + // Hash on the complete join-key vector on both sides, like the real plan does. + let left_partitioning = Partitioning::Hash( + on.iter().map(|(build, _)| Arc::clone(build)).collect(), + partition_count, + ); + let right_partitioning = Partitioning::Hash( + on.iter().map(|(_, probe)| Arc::clone(probe)).collect(), + partition_count, + ); + ( + Arc::new(RepartitionExec::try_new(left_scan, left_partitioning).unwrap()) + as Arc, + Arc::new(RepartitionExec::try_new(right_scan, right_partitioning).unwrap()) + as Arc, + ) + } else { + (left_scan, right_scan) + }; + + let probe_filter = Arc::new(DynamicFilterPhysicalExpr::new( + on.iter().map(|(_, probe)| Arc::clone(probe)).collect(), + lit(true), + )); + let dyn_filter: Arc = Arc::clone(&probe_filter) as _; + // The probe subtree must reference the produced filter, otherwise the join never + // publishes it. + let right: Arc = + Arc::new(FilterExec::try_new(Arc::clone(&dyn_filter), right_scan).unwrap()); + + let join = HashJoinExec::try_new( + left, + right, + on, + None, + &JoinType::Inner, + None, + PartitionMode::Partitioned, + NullEquality::NullEqualsNothing, + false, + ) + .unwrap() + .with_dynamic_filter_expr(Arc::clone(&probe_filter)) + .unwrap(); + let join: Arc = Arc::new(join); + + let mut options = ConfigOptions::default(); + options.optimizer.hash_join_inlist_pushdown_max_size = 0; + let session_config = SessionConfig::from(options); + let state = SessionStateBuilder::new() + .with_config(session_config) + .with_default_features() + .build(); + let task_ctx = Arc::new(TaskContext::from(&state)); + + // Every build partition must report before the filter is finalized, so all + // partition streams have to make progress concurrently. + let streams = (0..partition_count) + .map(|partition| { + let join = Arc::clone(&join); + let task_ctx = Arc::clone(&task_ctx); + async move { + let stream = join.execute(partition, task_ctx).unwrap(); + stream.try_collect::>().await.unwrap(); + } + }) + .collect::>(); + futures_util::future::join_all(streams).await; + + RealDynFilter { + expr: probe_filter.current().unwrap(), + probe_schema: right_schema, + } +} + +/// Build and probe batches for a `Utf8` join over distinct keys. +/// +/// Both sides start as one partition; the fixture hash-repartitions them so every hash +/// partition holds rows and DataFusion produces one filter branch per partition. Also +/// returns every distinct key. +pub fn utf8_join( + partition_count: usize, +) -> (Vec>, Vec>, Vec) { + let schema = one_column_schema("trace_id", DataType::Utf8); + let values = (0..partition_count * 8) + .map(|i| format!("trace-{i:02}")) + .collect::>(); + let borrowed = values.iter().map(String::as_str).collect::>(); + let build = partitions(&schema, vec![vec![utf8(&borrowed)]]); + // Probe rows include every member plus keys that belong to no build side. + let mut probe_values = borrowed.clone(); + let misses = (0..partition_count * 4) + .map(|i| format!("miss-{i:02}")) + .collect::>(); + probe_values.extend(misses.iter().map(String::as_str)); + let probe = partitions(&schema, vec![vec![utf8(&probe_values)]]); + (build, probe, values) +} diff --git a/src/common/recordbatch/src/filter.rs b/src/common/recordbatch/src/filter.rs index 0893d2aa3ad..3d8d7da7051 100644 --- a/src/common/recordbatch/src/filter.rs +++ b/src/common/recordbatch/src/filter.rs @@ -25,7 +25,7 @@ use std::sync::Arc; use datafusion::error::Result as DfResult; use datafusion::logical_expr::{Expr, Literal, Operator, utils}; use datafusion::physical_plan::PhysicalExpr; -use datafusion_common::arrow::array::{ArrayRef, Datum}; +use datafusion_common::arrow::array::{ArrayRef, Datum, Scalar}; use datafusion_common::arrow::buffer::BooleanBuffer; use datafusion_common::cast::{as_boolean_array, as_null_array}; use datafusion_common::{DataFusionError, ScalarValue, internal_err}; @@ -79,6 +79,7 @@ impl MetadataFilter { /// Evaluates supported single-column predicates with precompiled regular expressions. /// Compound predicates preserve SQL null semantics until conversion to a scan mask. +/// Float comparisons treat `-0.0` and `+0.0` as equal, as DataFusion does. #[derive(Debug, Clone)] pub struct SimpleFilterEvaluator { column_name: String, @@ -104,7 +105,9 @@ impl SimpleFilterEvaluator { Some(Self { column_name, predicate: SimplePredicate::Comparison { - literal: value.to_scalar().ok()?, + literal: datafusion_common::utils::normalize_float_zero_scalar(value) + .to_scalar() + .ok()?, op, regex: None, regex_negative: false, @@ -224,11 +227,15 @@ impl SimpleFilterEvaluator { let input = input .to_scalar() .with_context(|_| ToArrowScalarSnafu { v: input.clone() })?; + let input = Scalar::new(datafusion_common::utils::normalize_float_zero( + &input.into_inner(), + )); Ok(self.evaluate_datum(&input, 1)?.value(0)) } pub fn evaluate_array(&self, input: &ArrayRef) -> Result { - self.evaluate_datum(input, input.len()) + let input = datafusion_common::utils::normalize_float_zero(input); + self.evaluate_datum(&input, input.len()) } pub fn evaluate_vector(&self, input: &VectorRef) -> Result { diff --git a/src/common/recordbatch/src/filter/predicate.rs b/src/common/recordbatch/src/filter/predicate.rs index e283633dc02..d9827004b53 100644 --- a/src/common/recordbatch/src/filter/predicate.rs +++ b/src/common/recordbatch/src/filter/predicate.rs @@ -98,7 +98,9 @@ impl SimplePredicate { }; let (regex, regex_negative) = Self::maybe_build_regex(op, literal).ok()?; Some(Self::Comparison { - literal: literal.to_scalar().ok()?, + literal: datafusion_common::utils::normalize_float_zero_scalar(literal.clone()) + .to_scalar() + .ok()?, op, regex, regex_negative, @@ -626,6 +628,103 @@ mod tests { } } + #[tokio::test] + async fn float_zero_predicates_match_datafusion() { + use datafusion::prelude::SessionContext; + use datatypes::arrow::array::{Float32Array, Float64Array, Int32Array}; + + let float64: ArrayRef = Arc::new(Float64Array::from(vec![ + Some(-0.0), + Some(0.0), + Some(1.0), + Some(f64::NAN), + None, + ])); + let float32: ArrayRef = Arc::new(Float32Array::from(vec![ + Some(-0.0), + Some(0.0), + Some(1.0), + Some(f32::NAN), + None, + ])); + for input in [float64, float32] { + let literal = |v: f64| { + if input.data_type() == &DataType::Float32 { + lit(v as f32) + } else { + lit(v) + } + }; + let ctx = SessionContext::new(); + let schema = Arc::new(Schema::new(vec![ + Field::new("v", input.data_type().clone(), true), + Field::new("id", DataType::Int32, false), + ])); + ctx.register_batch( + "floats", + RecordBatch::try_new( + schema, + vec![ + input.clone(), + Arc::new(Int32Array::from_iter_values(0..input.len() as i32)), + ], + ) + .unwrap(), + ) + .unwrap(); + for expr in [ + col("v").eq(literal(0.0)), + col("v").eq(literal(-0.0)), + col("v").not_eq(literal(-0.0)), + col("v").lt(literal(0.0)), + col("v").lt_eq(literal(-0.0)), + col("v").gt(literal(-0.0)), + col("v").gt_eq(literal(0.0)), + literal(0.0).lt_eq(col("v")), + col("v").eq(literal(0.0)).or(col("v").eq(literal(1.0))), + col("v").eq(literal(f64::NAN)), + ] { + let expected = ctx + .table("floats") + .await + .unwrap() + .filter(expr.clone()) + .unwrap() + .collect() + .await + .unwrap() + .iter() + .flat_map(|batch| { + batch + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .values() + .to_vec() + }) + .collect::>(); + let evaluator = SimpleFilterEvaluator::try_new(&expr).unwrap(); + let mask = evaluator.evaluate_array(&input).unwrap(); + let selected = mask + .iter() + .enumerate() + .filter_map(|(i, matched)| matched.then_some(i as i32)) + .collect::>(); + assert_eq!(selected, expected, "{expr}"); + for (i, matched) in mask.iter().enumerate() { + let scalar = ScalarValue::try_from_array(&input, i).unwrap(); + assert_eq!( + evaluator.evaluate_scalar(&scalar).unwrap(), + matched, + "{expr}, row={i}" + ); + } + } + } + } + #[test] fn supported_filter_op() { // equal diff --git a/src/common/telemetry/src/logging.rs b/src/common/telemetry/src/logging.rs index b1430bbd726..491b84f3fab 100644 --- a/src/common/telemetry/src/logging.rs +++ b/src/common/telemetry/src/logging.rs @@ -265,7 +265,7 @@ enum TraceState { } /// The logging options that used to initialize the logger. -#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[derive(Clone, PartialEq, Serialize, Deserialize)] #[serde(default)] pub struct LoggingOptions { /// The directory to store log files. If not set, logs will be written to stdout. @@ -303,13 +303,53 @@ pub struct LoggingOptions { pub otlp_export_protocol: Option, /// Additional HTTP headers for OTLP exporter. - #[serde(skip_serializing_if = "HashMap::is_empty")] + #[serde( + skip_serializing_if = "HashMap::is_empty", + serialize_with = "serialize_otlp_headers" + )] pub otlp_headers: HashMap, /// Whether to enable per-region metrics. pub enable_per_region_metrics: bool, } +impl std::fmt::Debug for LoggingOptions { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LoggingOptions") + .field("dir", &self.dir) + .field("level", &self.level) + .field("log_format", &self.log_format) + .field("max_log_files", &self.max_log_files) + .field("max_log_dir_size", &self.max_log_dir_size) + .field("append_stdout", &self.append_stdout) + .field("enable_file_logging", &self.enable_file_logging) + .field("enable_otlp_tracing", &self.enable_otlp_tracing) + .field("otlp_endpoint", &self.otlp_endpoint) + .field("tracing_sample_ratio", &self.tracing_sample_ratio) + .field("otlp_export_protocol", &self.otlp_export_protocol) + .field("enable_per_region_metrics", &self.enable_per_region_metrics) + .field("otlp_headers", &redacted_otlp_headers(&self.otlp_headers)) + .finish() + } +} + +fn redacted_otlp_headers(headers: &HashMap) -> HashMap<&str, &str> { + headers + .keys() + .map(|key| (key.as_str(), "[REDACTED]")) + .collect() +} + +fn serialize_otlp_headers( + headers: &HashMap, + serializer: S, +) -> std::result::Result +where + S: serde::Serializer, +{ + redacted_otlp_headers(headers).serialize(serializer) +} + /// The protocol of OTLP export. #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "snake_case")] @@ -1261,6 +1301,26 @@ mod tests { assert_eq!(protocol, OtlpExportProtocol::Http); } + #[test] + fn test_otlp_headers_are_redacted() { + let options: LoggingOptions = serde_json::from_value(serde_json::json!({ + "otlp_headers": {"authorization": "Bearer otlp-secret", "x-api-key": "api-secret"} + })) + .unwrap(); + assert_eq!(options.otlp_headers["authorization"], "Bearer otlp-secret"); + assert_eq!(options.otlp_headers["x-api-key"], "api-secret"); + for output in [ + serde_json::to_string(&options).unwrap(), + format!("{options:#?}"), + ] { + assert!(!output.contains("otlp-secret")); + assert!(!output.contains("api-secret")); + assert!(output.contains("authorization")); + assert!(output.contains("x-api-key")); + assert!(output.contains("[REDACTED]")); + } + } + #[test] fn test_logging_options_partial_eq_all_fields() { let base = LoggingOptions::default(); diff --git a/src/common/telemetry/src/logging/file_retention.rs b/src/common/telemetry/src/logging/file_retention.rs index 6ea9846be1a..0a79468851f 100644 --- a/src/common/telemetry/src/logging/file_retention.rs +++ b/src/common/telemetry/src/logging/file_retention.rs @@ -436,6 +436,10 @@ pub(crate) fn build_file_appender( builder = builder.latest_symlink(kind.current_link_name()); } + // FIXME(tokio-rs/tracing#3612): tracing-appender lists the directory to prune old logs + // before creating it, so a missing directory prints a misleading error. If this fails, + // `build` reports it. + let _ = fs::create_dir_all(&opts.dir); let appender = builder.build(&opts.dir).unwrap_or_else(|error| { panic!( "initializing rolling file appender at {} failed: {}", diff --git a/src/common/wal/src/config.rs b/src/common/wal/src/config.rs index 950f833dec9..0d26085452c 100644 --- a/src/common/wal/src/config.rs +++ b/src/common/wal/src/config.rs @@ -374,6 +374,25 @@ mod tests { } } + #[test] + fn test_kafka_password_serialization() { + for kind in ["PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512"] { + let input = format!("type = '{kind}'\nusername = 'alice'\npassword = 'kafka-secret'"); + let config: KafkaClientSaslConfig = toml::from_str(&input).unwrap(); + match &config { + KafkaClientSaslConfig::Plain { password, .. } + | KafkaClientSaslConfig::ScramSha256 { password, .. } + | KafkaClientSaslConfig::ScramSha512 { password, .. } => { + assert_eq!(password, "kafka-secret") + } + } + let serialized = toml::to_string(&config).unwrap(); + assert!(!serialized.contains("kafka-secret")); + assert!(serialized.contains("")); + assert!(serialized.contains("alice")); + } + } + #[test] fn test_kafka_wal_config_debug_redacts_password() { let config = MetasrvWalConfig::Kafka(MetasrvKafkaConfig { diff --git a/src/common/wal/src/config/kafka/common.rs b/src/common/wal/src/config/kafka/common.rs index daa5607e476..e1f00869aac 100644 --- a/src/common/wal/src/config/kafka/common.rs +++ b/src/common/wal/src/config/kafka/common.rs @@ -69,20 +69,30 @@ pub struct KafkaClientSasl { pub enum KafkaClientSaslConfig { Plain { username: String, + #[serde(serialize_with = "serialize_password")] password: String, }, #[serde(rename = "SCRAM-SHA-256")] ScramSha256 { username: String, + #[serde(serialize_with = "serialize_password")] password: String, }, #[serde(rename = "SCRAM-SHA-512")] ScramSha512 { username: String, + #[serde(serialize_with = "serialize_password")] password: String, }, } +fn serialize_password(_: &str, serializer: S) -> std::result::Result +where + S: serde::Serializer, +{ + serializer.serialize_str("") +} + impl fmt::Debug for KafkaClientSaslConfig { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { diff --git a/src/datanode/src/alive_keeper.rs b/src/datanode/src/alive_keeper.rs index 13af76202d5..30e27929a29 100644 --- a/src/datanode/src/alive_keeper.rs +++ b/src/datanode/src/alive_keeper.rs @@ -16,7 +16,7 @@ use std::collections::HashMap; use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; -use api::v1::meta::GrantedRegion; +use api::v1::meta::GrantedRegion as PbGrantedRegion; use async_trait::async_trait; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; @@ -26,7 +26,7 @@ use common_meta::heartbeat::handler::{ }; use common_telemetry::{debug, error, info, trace, warn}; use snafu::OptionExt; -use store_api::region_engine::{RegionRole, SettableRegionRoleState}; +use store_api::region_engine::{GrantedRegion, RegionRole, SettableRegionRoleState}; use store_api::region_request::{RegionCloseRequest, RegionRequest}; use store_api::storage::RegionId; #[cfg(test)] @@ -137,15 +137,22 @@ impl RegionAliveKeeper { /// Renews the lease of regions to `deadline`, granted for a heartbeat sent at `sent_at`. async fn renew_region_leases( &self, - regions: &[GrantedRegion], + regions: &[PbGrantedRegion], sent_at: Instant, deadline: Instant, ) { for region in regions { - let (role, region_id) = (region.role().into(), RegionId::from(region.region_id)); + let region = GrantedRegion::from(region.clone()); + let region_id = region.region_id; if let Some(handle) = self.find_handle(region_id).await { handle - .reset_deadline(role, sent_at, deadline, region.extensions.clone()) + .reset_deadline(ResetLease { + role: region.region_role, + sent_at, + deadline, + candidate_lease_id: region.candidate_lease_id, + extensions: region.extensions, + }) .await; } else { warn!( @@ -301,13 +308,23 @@ enum CountdownCommand { /// 4 * `heartbeat_interval_millis` Start(u64), /// Reset countdown deadline to the given instance. - /// (NextRole, HeartbeatSentAt, Deadline, ExtensionInfo) - Reset((RegionRole, Instant, Instant, HashMap>)), + Reset(ResetLease), /// Returns the current deadline of the countdown task. #[cfg(test)] Deadline(oneshot::Sender), } +/// A region lease received in a heartbeat response. +#[derive(Debug)] +struct ResetLease { + role: RegionRole, + /// When the heartbeat was sent. + sent_at: Instant, + deadline: Instant, + candidate_lease_id: Option, + extensions: HashMap>, +} + pub type CountdownTaskHandlerExtRef = Arc; /// Extension trait for [CountdownTaskHandlerExt] to reset deadline of a region. @@ -319,7 +336,7 @@ pub trait CountdownTaskHandlerExt: Send + Sync { region_id: RegionId, role: RegionRole, deadline: Instant, - extension_info: HashMap>, + extensions: HashMap>, ); } @@ -379,23 +396,8 @@ impl CountdownTaskHandle { None } - async fn reset_deadline( - &self, - role: RegionRole, - sent_at: Instant, - deadline: Instant, - extension_info: HashMap>, - ) { - if let Err(e) = self - .tx - .send(CountdownCommand::Reset(( - role, - sent_at, - deadline, - extension_info, - ))) - .await - { + async fn reset_deadline(&self, lease: ResetLease) { + if let Err(e) = self.tx.send(CountdownCommand::Reset(lease)).await { warn!( "Failed to reset region alive keeper deadline: {e}. \ Maybe the task is stopped due to region been closed." @@ -447,9 +449,9 @@ impl CountdownTask { started = true; } }, - Some(CountdownCommand::Reset((role, sent_at, deadline, extension_info))) => { + Some(CountdownCommand::Reset(ResetLease { role, sent_at, deadline, candidate_lease_id, extensions })) => { let prev_role_leader = self.region_server.is_region_leader(self.region_id).unwrap_or(false); - let applied = match self.region_server.set_region_role_with_lease(self.region_id, role, sent_at.into_std(), deadline.into_std()) { + let applied = match self.region_server.set_region_role_with_lease(self.region_id, role, sent_at.into_std(), deadline.into_std(), candidate_lease_id) { Ok(applied) => applied, Err(err) => { if err.status_code() == StatusCode::RegionNotFound { @@ -480,7 +482,7 @@ impl CountdownTask { self.region_id, role, deadline, - extension_info, + extensions, ).await; } trace!( @@ -538,7 +540,10 @@ mod test { use crate::tests::{MockRegionEngine, mock_region_server}; /// Handles a heartbeat response that grants `regions` for a heartbeat sent now. - async fn handle_lease_response(alive_keeper: &RegionAliveKeeper, regions: Vec) { + async fn handle_lease_response( + alive_keeper: &RegionAliveKeeper, + regions: Vec, + ) { let sent_at = alive_keeper.epoch().elapsed(); handle_lease_response_sent_at(alive_keeper, sent_at, regions).await; } @@ -548,7 +553,7 @@ mod test { async fn handle_lease_response_sent_at( alive_keeper: &RegionAliveKeeper, sent_at: Duration, - regions: Vec, + regions: Vec, ) { let (tx, _rx) = mpsc::channel(1); let response = HeartbeatResponse { @@ -587,9 +592,10 @@ mod test { let alive_keeper = RegionAliveKeeper::new(region_server.clone(), None, Duration::from_millis(100)); alive_keeper.register_region(region_id).await; - let grant = GrantedRegion { + let grant = PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::Leader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }; handle_lease_response(&alive_keeper, vec![grant]).await; @@ -598,6 +604,28 @@ mod test { assert_eq!(0, finalized.load(Ordering::Relaxed)); } + #[tokio::test(flavor = "multi_thread")] + async fn lease_response_passes_candidate_lease_id_to_engine() { + common_telemetry::init_default_ut_logging(); + let region_server = mock_region_server(); + let (engine, _) = MockRegionEngine::new(MITO_ENGINE_NAME); + let region_id = RegionId::new(1024, 1); + region_server.register_test_region(region_id, engine.clone()); + let alive_keeper = + RegionAliveKeeper::new(region_server.clone(), None, Duration::from_millis(100)); + alive_keeper.register_region(region_id).await; + let grant = PbGrantedRegion { + region_id: region_id.as_u64(), + role: api::v1::meta::RegionRole::Follower.into(), + candidate_lease_id: 7, + extensions: HashMap::new(), + }; + handle_lease_response(&alive_keeper, vec![grant]).await; + // The countdown task has handled the grant once it answers. + alive_keeper.deadline(region_id).await.unwrap(); + assert_eq!(*engine.candidate_lease_ids.lock().unwrap(), vec![Some(7)]); + } + #[tokio::test(flavor = "multi_thread")] async fn lease_responses_let_regions_write() { common_telemetry::init_default_ut_logging(); @@ -642,9 +670,10 @@ mod test { // A Leader grant renews the lease of the region. alive_keeper.register_region(region_id).await; - let grant = GrantedRegion { + let grant = PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::Leader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }; handle_lease_response(&alive_keeper, vec![grant.clone()]).await; @@ -716,9 +745,10 @@ mod test { // Renew lease then sleep. alive_keeper .renew_region_leases( - &[GrantedRegion { + &[PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::Leader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }], Instant::now(), @@ -773,12 +803,13 @@ mod test { // Reset deadline. countdown_handle - .reset_deadline( - RegionRole::Leader, - Instant::now(), - Instant::now() + Duration::from_millis(heartbeat_interval_millis * 5), - HashMap::new(), - ) + .reset_deadline(ResetLease { + role: RegionRole::Leader, + sent_at: Instant::now(), + deadline: Instant::now() + Duration::from_millis(heartbeat_interval_millis * 5), + candidate_lease_id: None, + extensions: HashMap::new(), + }) .await; assert!( countdown_handle.deadline().await.unwrap() @@ -821,9 +852,10 @@ mod test { alive_keeper.register_region(region_id).await; alive_keeper .renew_region_leases( - &[GrantedRegion { + &[PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::StagingLeader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }], Instant::now(), @@ -870,9 +902,10 @@ mod test { alive_keeper.register_region(region_id).await; alive_keeper .renew_region_leases( - &[GrantedRegion { + &[PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::Leader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }], Instant::now(), @@ -913,9 +946,10 @@ mod test { alive_keeper.register_region(region_id).await; alive_keeper .renew_region_leases( - &[GrantedRegion { + &[PbGrantedRegion { region_id: region_id.as_u64(), role: api::v1::meta::RegionRole::StagingLeader.into(), + candidate_lease_id: 0, extensions: HashMap::new(), }], Instant::now(), diff --git a/src/datanode/src/heartbeat/handler/flush_region.rs b/src/datanode/src/heartbeat/handler/flush_region.rs index c1f055c0eb3..e31a0a3b7f9 100644 --- a/src/datanode/src/heartbeat/handler/flush_region.rs +++ b/src/datanode/src/heartbeat/handler/flush_region.rs @@ -18,11 +18,11 @@ use common_meta::instruction::{ FlushErrorStrategy, FlushRegionReply, FlushRegions, FlushStrategy, InstructionError, InstructionReply, }; -use common_telemetry::{debug, warn}; +use common_telemetry::{debug, error, warn}; use store_api::region_request::{RegionFlushReason, RegionFlushRequest, RegionRequest}; use store_api::storage::RegionId; -use crate::error::{self, RegionNotFoundSnafu, RegionNotReadySnafu, Result, UnexpectedSnafu}; +use crate::error::{self, RegionNotFoundSnafu, RegionNotReadySnafu, Result}; use crate::heartbeat::handler::{HandlerContext, InstructionHandler}; pub struct FlushRegionsHandler; @@ -125,6 +125,9 @@ impl HandlerContext { match &result { Ok(_) => results.push((region_id, Ok(()))), Err(err) => { + // Log the full error chain here; the reply carries the + // classification and the root cause of the original error. + error!(err; "Failed to flush region, region_id: {}", region_id); results.push((region_id, Err(InstructionError::from_error(err)))); // For fail-fast strategy, abort on first error @@ -179,13 +182,7 @@ impl HandlerContext { } let mut watcher = register_result.into_watcher(); - match self.flush_tasks.wait_until_finish(&mut watcher).await { - Ok(()) => Ok(()), - Err(err) => Err(UnexpectedSnafu { - violated: format!("Flush task failed: {err:?}"), - } - .build()), - } + self.flush_tasks.wait_until_finish(&mut watcher).await } } @@ -193,12 +190,15 @@ impl HandlerContext { mod tests { use std::sync::{Arc, RwLock}; + use common_error::ext::RetryHint; + use common_error::status_code::StatusCode; use common_meta::instruction::{FlushErrorStrategy, FlushRegions}; use common_meta::kv_backend::memory::MemoryKvBackend; use mito2::engine::MITO_ENGINE_NAME; use store_api::storage::RegionId; use super::*; + use crate::error::{RegionBusySnafu, UnexpectedSnafu}; use crate::tests::{MockRegionEngine, mock_region_server}; type FlushedRequests = Arc)>>>; @@ -366,6 +366,100 @@ mod tests { assert!(flush_reply.results[1].1.is_err()); } + /// A task body error used to verify that the reply keeps its classification. + #[derive(Clone, Copy, Debug)] + enum TaskFlushError { + RegionBusy, + Unexpected, + } + + impl TaskFlushError { + fn build(self, region_id: RegionId) -> crate::error::Error { + match self { + Self::RegionBusy => RegionBusySnafu { region_id }.build(), + Self::Unexpected => UnexpectedSnafu { + violated: "flush task boom marker", + } + .build(), + } + } + + fn expected(self) -> (StatusCode, RetryHint) { + match self { + Self::RegionBusy => (StatusCode::RegionBusy, RetryHint::Retryable), + Self::Unexpected => (StatusCode::Unexpected, RetryHint::NonRetryable), + } + } + + fn message_fragment(self, region_id: RegionId) -> String { + match self { + Self::RegionBusy => format!("Region {region_id} is busy"), + Self::Unexpected => "flush task boom marker".to_string(), + } + } + } + + #[tokio::test] + async fn test_handle_flush_region_sync_keeps_task_error_classification() { + let failing_region = RegionId::new(1024, 1); + let succeeding_region = RegionId::new(1024, 2); + + for error in [TaskFlushError::RegionBusy, TaskFlushError::Unexpected] { + let flushed_regions: Arc>> = Arc::new(RwLock::new(Vec::new())); + + let mock_region_server = mock_region_server(); + let flushed_regions_ref = flushed_regions.clone(); + let (mock_engine, _) = + MockRegionEngine::with_custom_apply_fn(MITO_ENGINE_NAME, move |region_engine| { + region_engine.handle_request_mock_fn = + Some(Box::new(move |region_id, _request| { + flushed_regions_ref.write().unwrap().push(region_id); + if region_id == failing_region { + Err(error.build(region_id)) + } else { + Ok(0) + } + })); + }); + mock_region_server.register_test_region(failing_region, mock_engine.clone()); + mock_region_server.register_test_region(succeeding_region, mock_engine); + + let kv_backend = Arc::new(MemoryKvBackend::new()); + let handler_context = HandlerContext::new_for_test(mock_region_server, kv_backend); + let instruction = FlushRegions::sync_batch( + vec![failing_region, succeeding_region], + FlushErrorStrategy::TryAll, + ); + let reply = FlushRegionsHandler + .handle(&handler_context, instruction) + .await + .unwrap() + .expect_flush_regions_reply(); + + assert!(!reply.overall_success, "{error:?}"); + assert_eq!(reply.results.len(), 2, "{error:?}"); + assert_eq!(reply.results[0].0, failing_region, "{error:?}"); + let reply_error = reply.results[0].1.as_ref().unwrap_err(); + let (code, retry_hint) = error.expected(); + assert_eq!(reply_error.code, code, "{error:?}"); + assert_eq!(reply_error.retry_hint, retry_hint, "{error:?}"); + assert!( + reply_error + .message + .contains(&error.message_fragment(failing_region)), + "{error:?}: {}", + reply_error.message + ); + assert_eq!(reply.results[1].0, succeeding_region, "{error:?}"); + assert!(reply.results[1].1.is_ok(), "{error:?}"); + assert_eq!( + *flushed_regions.read().unwrap(), + vec![failing_region, succeeding_region], + "{error:?}" + ); + } + } + #[test] fn test_flush_regions_display() { let region_id = RegionId::new(1024, 1); diff --git a/src/datanode/src/region_server.rs b/src/datanode/src/region_server.rs index 6755510a293..8c961e8c28b 100644 --- a/src/datanode/src/region_server.rs +++ b/src/datanode/src/region_server.rs @@ -498,6 +498,7 @@ impl RegionServer { role: RegionRole, sent_at: std::time::Instant, deadline: std::time::Instant, + candidate_lease_id: Option, ) -> Result { let engine = self .inner @@ -505,7 +506,7 @@ impl RegionServer { .get(®ion_id) .with_context(|| RegionNotFoundSnafu { region_id })?; engine - .set_region_role_with_lease(region_id, role, sent_at, deadline) + .set_region_role_with_lease(region_id, role, sent_at, deadline, candidate_lease_id) .with_context(|_| HandleRegionRequestSnafu { region_id }) } @@ -1865,6 +1866,20 @@ impl RegionServerInner { &self, batch_request: BatchRegionDdlRequest, ) -> Result { + // Drops may remove dependent regions and may partially succeed. Complete each + // request's registration changes before proceeding to the next request. + if let BatchRegionDdlRequest::Drop(requests) = batch_request { + let mut response = RegionResponse::new(0); + for (region_id, request) in requests { + let result = self + .handle_request(region_id, RegionRequest::Drop(request)) + .await?; + response.affected_rows += result.affected_rows; + response.extensions.extend(result.extensions); + } + return Ok(response); + } + let region_changes = match &batch_request { BatchRegionDdlRequest::Create(requests) => requests .iter() @@ -1985,6 +2000,12 @@ impl RegionServerInner { CurrentEngine::EarlyReturn(rows) => return Ok(RegionResponse::new(rows)), }; + let logical_regions = if matches!(region_change, RegionChange::Deregisters) { + self.prepare_logical_regions_for_deregistration(&engine, region_id) + } else { + Vec::new() + }; + // Sets corresponding region status to registering/deregistering before the operation. self.set_region_status_not_ready(region_id, &engine, ®ion_change); @@ -1997,6 +2018,14 @@ impl RegionServerInner { } match result { Ok(result) => { + for (logical_region, _) in logical_regions { + self.set_region_status_ready( + logical_region, + engine.clone(), + RegionChange::Deregisters, + ) + .await?; + } // Sets corresponding region status to ready. self.set_region_status_ready(region_id, engine.clone(), region_change) .await?; @@ -2008,6 +2037,9 @@ impl RegionServerInner { }) } Err(err) => { + for (logical_region, status) in logical_regions { + self.region_map.insert(logical_region, status); + } // Removes the region status if the operation fails. self.unset_region_status(region_id, &engine, region_change); Err(err) @@ -2143,6 +2175,26 @@ impl RegionServerInner { Ok(()) } + /// Captures only existing registrations so failures cannot resurrect closed logical regions. + fn prepare_logical_regions_for_deregistration( + &self, + engine: &RegionEngineRef, + physical_region_id: RegionId, + ) -> Vec<(RegionId, RegionEngineWithStatus)> { + let Some(metric_engine) = engine.as_any().downcast_ref::() else { + return Vec::new(); + }; + let logical_regions = metric_engine + .loaded_logical_regions(physical_region_id) + .into_iter() + .filter_map(|id| self.region_map.get(&id).map(|status| (id, status.clone()))) + .collect::>(); + for (region_id, _) in &logical_regions { + self.set_region_status_not_ready(*region_id, engine, &RegionChange::Deregisters); + } + logical_regions + } + async fn register_logical_regions( &self, engine: &RegionEngineRef, @@ -3725,6 +3777,202 @@ mod tests { } } + #[tokio::test] + async fn test_metric_physical_drop_and_close_unregister_logical_regions() { + for mode in ["drop", "close", "batch_drop"] { + let mut env = TestEnv::with_prefix("region-server-metric-drop").await; + let mito = env.create_engine(MitoConfig::default()).await; + let metric = Arc::new( + MetricEngine::try_new(mito.clone(), metric_engine::config::EngineConfig::default()) + .unwrap(), + ); + let mut server = mock_region_server(); + server.register_engine(metric.clone()); + // Keep a second physical group alive to catch overbroad unregistering. + for table in [10, 20] { + let physical = RegionId::new(table, 0); + let logical = RegionId::new(table + 1, 0); + let mut create = CreateRequestBuilder::new().build(); + create.engine = METRIC_ENGINE_NAME.to_string(); + create.column_metadatas = vec![ + store_api::metadata::ColumnMetadata { + column_id: 0, + semantic_type: SemanticType::Timestamp, + column_schema: ColumnSchema::new( + "greptime_timestamp", + ConcreteDataType::timestamp_millisecond_datatype(), + false, + ), + }, + store_api::metadata::ColumnMetadata { + column_id: 1, + semantic_type: SemanticType::Field, + column_schema: ColumnSchema::new( + "greptime_value", + ConcreteDataType::float64_datatype(), + false, + ), + }, + ]; + create.primary_key.clear(); + create.table_dir = format!("metric-drop/{table}/"); + create.options = HashMap::from([( + store_api::metric_engine_consts::PHYSICAL_TABLE_METADATA_KEY.to_string(), + String::new(), + )]); + server + .handle_request(physical, RegionRequest::Create(create.clone())) + .await + .unwrap(); + create.table_dir = format!("metric-drop/{}/", table + 1); + create.options = HashMap::from([( + store_api::metric_engine_consts::LOGICAL_TABLE_METADATA_KEY.to_string(), + physical.as_u64().to_string(), + )]); + server + .handle_request(logical, RegionRequest::Create(create.clone())) + .await + .unwrap(); + if table == 10 { + let closed = RegionId::new(12, 0); + server + .handle_request(closed, RegionRequest::Create(create.clone())) + .await + .unwrap(); + server + .handle_request(closed, RegionRequest::Close(RegionCloseRequest::default())) + .await + .unwrap(); + } + } + let physical = RegionId::new(10, 0); + let logical = RegionId::new(11, 0); + let mut dependents = metric.loaded_logical_regions(physical); + dependents.sort_unstable(); + assert_eq!(dependents, vec![logical, RegionId::new(12, 0)]); + assert!(metric.loaded_logical_regions(logical).is_empty()); + // Rejection must leave both registrations usable for a subsequent forced drop. + assert!( + server + .handle_request( + physical, + RegionRequest::Drop(RegionDropRequest { + fast_path: false, + force: false, + partial_drop: false, + }) + ) + .await + .is_err() + ); + assert!(!server.inner.region_map.contains_key(&RegionId::new(12, 0))); + for id in [physical, logical] { + assert!(matches!( + server.inner.region_map.get(&id).unwrap().value(), + RegionEngineWithStatus::Ready(_) + )); + } + server + .inner + .region_map + .insert(logical, RegionEngineWithStatus::Registering(metric.clone())); + assert!( + server + .handle_request( + physical, + RegionRequest::Drop(RegionDropRequest { + fast_path: false, + force: false, + partial_drop: false, + }) + ) + .await + .is_err() + ); + assert!(matches!( + server.inner.region_map.get(&logical).unwrap().value(), + RegionEngineWithStatus::Registering(_) + )); + server + .inner + .region_map + .insert(logical, RegionEngineWithStatus::Ready(metric.clone())); + if mode == "batch_drop" { + // First group commits, second rejects. Retrying must not resurrect the first. + for _ in 0..2 { + assert!( + server + .inner + .handle_batch_ddl_requests(BatchRegionDdlRequest::Drop(vec![ + ( + physical, + RegionDropRequest { + fast_path: false, + force: true, + partial_drop: false + } + ), + ( + RegionId::new(20, 0), + RegionDropRequest { + fast_path: false, + force: false, + partial_drop: false + } + ), + ])) + .await + .is_err() + ); + assert!(!server.inner.region_map.contains_key(&physical)); + assert!(!server.inner.region_map.contains_key(&logical)); + } + } + let request = || { + if mode == "close" { + RegionRequest::Close(RegionCloseRequest::default()) + } else { + RegionRequest::Drop(RegionDropRequest { + fast_path: false, + force: true, + partial_drop: false, + }) + } + }; + // A retry of an already completed removal remains harmless. + for _ in 0..2 { + if mode == "batch_drop" { + server + .inner + .handle_batch_ddl_requests(BatchRegionDdlRequest::Drop(vec![( + physical, + RegionDropRequest { + fast_path: false, + force: true, + partial_drop: false, + }, + )])) + .await + .unwrap(); + } else { + server.handle_request(physical, request()).await.unwrap(); + } + } + for id in [physical, logical] { + assert!(!server.inner.region_map.contains_key(&id)); + } + assert!(metric.loaded_logical_regions(physical).is_empty()); + for id in [RegionId::new(20, 0), RegionId::new(21, 0)] { + assert!(server.inner.region_map.contains_key(&id)); + } + assert_eq!( + metric.loaded_logical_regions(RegionId::new(20, 0)), + vec![RegionId::new(21, 0)] + ); + server.stop().await.unwrap(); + } + } + #[tokio::test] async fn test_metric_batch_failure_indices() { let mut env = TestEnv::with_prefix("region-server-metric-results").await; diff --git a/src/datanode/src/tests.rs b/src/datanode/src/tests.rs index a126d3429fd..0ee830af535 100644 --- a/src/datanode/src/tests.rs +++ b/src/datanode/src/tests.rs @@ -13,7 +13,7 @@ // limitations under the License. use std::any::Any; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use std::time::Duration; use api::region::RegionResponse; @@ -135,6 +135,8 @@ pub struct MockRegionEngine { pub(crate) handle_sync_region_mock_fn: Option, pub(crate) handle_stop_mock_fn: Option, pub(crate) mock_role: Option>, + /// The candidate lease ids of the leases the engine received. + pub(crate) candidate_lease_ids: Mutex>>, engine: String, } @@ -152,6 +154,7 @@ impl MockRegionEngine { handle_sync_region_mock_fn: None, handle_stop_mock_fn: None, mock_role: None, + candidate_lease_ids: Mutex::default(), engine: engine.to_string(), }), rx, @@ -174,6 +177,7 @@ impl MockRegionEngine { handle_sync_region_mock_fn: None, handle_stop_mock_fn: None, mock_role: None, + candidate_lease_ids: Mutex::default(), engine: engine.to_string(), }), rx, @@ -196,6 +200,7 @@ impl MockRegionEngine { handle_sync_region_mock_fn: None, handle_stop_mock_fn: None, mock_role: None, + candidate_lease_ids: Mutex::default(), engine: engine.to_string(), }), rx, @@ -219,6 +224,7 @@ impl MockRegionEngine { handle_sync_region_mock_fn: None, handle_stop_mock_fn: None, mock_role: None, + candidate_lease_ids: Mutex::default(), engine: engine.to_string(), }; @@ -295,7 +301,12 @@ impl RegionEngine for MockRegionEngine { role: RegionRole, _sent_at: std::time::Instant, _deadline: std::time::Instant, + candidate_lease_id: Option, ) -> Result { + self.candidate_lease_ids + .lock() + .unwrap() + .push(candidate_lease_id); // A mocked role stays, so a grant of another role is not applied. Ok(self.mock_role.flatten().is_none_or(|mocked| mocked == role)) } diff --git a/src/datatypes/src/json/value.rs b/src/datatypes/src/json/value.rs index 7bcb1ecc2b9..5d115f99441 100644 --- a/src/datatypes/src/json/value.rs +++ b/src/datatypes/src/json/value.rs @@ -379,6 +379,11 @@ impl JsonValue { self.json_variant } + /// Borrows the JSON representation without constructing a tree of value references. + pub fn variant(&self) -> &JsonVariant { + &self.json_variant + } + pub(crate) fn into_value(self) -> Value { fn helper(v: JsonVariant) -> Value { match v { diff --git a/src/file-engine/src/engine.rs b/src/file-engine/src/engine.rs index 98ad7409f43..a49b79fa65e 100644 --- a/src/file-engine/src/engine.rs +++ b/src/file-engine/src/engine.rs @@ -139,6 +139,7 @@ impl RegionEngine for FileRegionEngine { role: RegionRole, _sent_at: Instant, _deadline: Instant, + _candidate_lease_id: Option, ) -> Result { self.set_region_role(region_id, role).map(|_| true) } diff --git a/src/frontend/AGENTS.md b/src/frontend/AGENTS.md index 43ebe894baf..214a6d0fd3e 100644 --- a/src/frontend/AGENTS.md +++ b/src/frontend/AGENTS.md @@ -27,6 +27,7 @@ remote datanodes via `operator`/`client`. | `instance` | `src/frontend/src/instance.rs` | `Instance`: the core handler; implements `SqlQueryHandler`, `PrometheusHandler`, etc. | | `instance/builder` | `src/frontend/src/instance/builder.rs` | `FrontendBuilder` assembles `Instance` from its dependencies | | `instance/grpc` | `src/frontend/src/instance/grpc.rs` | `GrpcQueryHandler`: insert/delete/query/promql over gRPC | +| `instance/schema_selection` | `src/frontend/src/instance/schema_selection.rs` | SQL database selection and authorization for sessions without a selected schema | | `instance/region_query` | `src/frontend/src/instance/region_query.rs` | Routes distributed region reads to datanodes | | `instance/exemplar` | `src/frontend/src/instance/exemplar.rs` | Resolves and authorizes selector branches before bounded exemplar-table scans | | `instance/*` | `src/frontend/src/instance/` | Per-protocol handlers (`influxdb.rs`, `promql.rs`, `otlp/`, `jaeger.rs`, `logs.rs`, `prom_store.rs`, ...) | @@ -43,9 +44,16 @@ including caches owned by pipeline and protocol handlers. - **SQL query** (`instance.rs`): `do_query_inner` handles parsing, interceptors, permission checks, timeout/cancellation, and delegates planning/execution to `StatementExecutor`. Distributed scans enter through `region_query.rs`. + MySQL sessions without a selected database validate explicit scopes before + planning or executing SQL, including prepared statements; unqualified targets + fail without falling back to `public`. - **Exemplar query** (`instance/exemplar.rs`): authorize all selector branches - before calling `PromQueryInterceptor::pre_execute_exemplars` for each resolved - exemplar table, with that table's schema context, before scanning it. + before resolving logical schemas or calling + `PromQueryInterceptor::pre_execute_exemplars`. Exact metric selectors project + the union of current logical-table tags; broad or unresolved selectors retain + all tags. Responses include the stored `__table_id` and `__tsid` as string labels. + The process ticket retains the configured slow-query timer through collection; + its query text includes the expression and inclusive nanosecond scan bounds. - **Logical-table batch DDL** (`instance.rs`): dedicated SQL handler → bounded parse/interception → all-member permissions → operator CREATE preparation → one logical-table batch procedure. Keep pre-submission validation side-effect free. @@ -86,10 +94,12 @@ including caches owned by pipeline and protocol handlers. Reserve exemplar expansion with the transport's memory limiter before building rows; retain the permit through `WriteProgress` until writes complete. Expansion must not wait while the request holds transport memory permits. + Confirm source metric writes before inserting their exemplars; unrelated + metric writes remain concurrent and their completions must still be observed. - **Logical-table batching** (`instance/logical_batcher.rs`): `Services` initializes - one shared batcher for opted-in HTTP Prom and OTLP metric-engine - writes. Downstream routers can initialize it when enabling replacement endpoints. + one shared batcher for opted-in Prom remote-write (HTTP and PostgreSQL COPY) + and OTLP metric-engine writes. Downstream routers can initialize it when enabling replacement endpoints. `Instance::handle_otlp_metric_row_inserts` shares OTLP eligibility and dispatch between converters and falls back for incompatible tables. The schema adapter holds a weak instance reference to avoid an ownership cycle. @@ -99,6 +109,10 @@ including caches owned by pipeline and protocol handlers. for `servers::batcher::table::TablePendingRowsBatcher`. A separate execution-only inserter, with no batcher attached, sends the prepared bulk writes to datanodes, avoiding recursive batching. The batcher handles successful-write Flow notifications. + `FrontendBuilder` creates the write memory limiter shared by conversion admission + and the protocol servers wired by `Services`. Conversion admission includes the + request's live transport and retained decoder reservations in its fit check; + previously queued Arrow batches remain transient contention. ## Public surface @@ -126,6 +140,14 @@ cargo nextest run -p frontend - Keep the frontend/servers split straight: wire format and network live in `servers`; permissions, planning, and routing live here. +- A statement whose tables are known only at execution time (`COPY DATABASE`) + resolves them first, authorizes every resolved table, then executes exactly + the authorized set. Do not authorize such a statement with empty targets. +- Built-in permission action names are a contract with external + `PermissionChecker` implementations. Define each as a constant in + `src/auth/src/permission.rs` and list it in `ALL_ACTIONS`; do not build them + from string literals at call sites. A new action or access-mode requirement + on an existing statement or endpoint is a user-visible permission change. - Standalone vs distributed diverge in datanode access (the `standalone` crate's local `RegionServer` adapter vs `NodeClients` RPC), MetaClient usage, and whether heartbeat matters. The builder defaults local invalidation to its diff --git a/src/frontend/src/frontend.rs b/src/frontend/src/frontend.rs index 7e74d079d97..dc669313a55 100644 --- a/src/frontend/src/frontend.rs +++ b/src/frontend/src/frontend.rs @@ -23,6 +23,7 @@ use common_memory_manager::OnExhaustedPolicy; use common_options::datanode::DatanodeClientOptions; use common_options::memory::MemoryOptions; use common_telemetry::logging::{LoggingOptions, SlowQueryOptions, TracingOptions}; +use common_time::timestamp::TimeUnit; use meta_client::{MetaClientOptions, MetaClientRef}; use pipeline::PipelineOptions; use query::options::QueryOptions; @@ -53,11 +54,20 @@ pub struct FrontendOptions { /// even if a request sets the `auto_create_table` hint to `true`. When `true` /// (default), the per-request hint still applies. Default: `true`. pub auto_create_table: bool, - /// Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight). + /// Time index unit for physical metric tables created by auto table creation + /// (Prometheus remote write, OTLP metrics, and other metric-engine ingestion). + /// New logical tables created under such a physical table inherit its unit. + /// Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". + pub auto_create_physical_table_timestamp_unit: TimeUnit, + /// Shared quota for write bodies/messages, retained decoder memory, and estimated + /// ordinary-table batch conversion memory. A batch must fit alongside its request's + /// existing reservations; only additional memory is acquired. /// Set to 0 to disable the limit. Default: "0" (unlimited) pub max_in_flight_write_bytes: ReadableSize, /// Policy when write bytes quota is exhausted. /// Options: "wait" (default, 10s), "wait()", "fail" + /// Table-batch conversion never waits for additional memory. Under a wait policy, + /// temporary contention returns a retryable error; requests that cannot fit are rejected. pub write_bytes_exhausted_policy: OnExhaustedPolicy, pub http: HttpOptions, pub grpc: GrpcOptions, @@ -105,6 +115,7 @@ impl Default for FrontendOptions { default_timezone: None, default_column_prefix: None, auto_create_table: true, + auto_create_physical_table_timestamp_unit: TimeUnit::Millisecond, max_in_flight_write_bytes: ReadableSize(0), write_bytes_exhausted_policy: OnExhaustedPolicy::default(), http: HttpOptions::default(), diff --git a/src/frontend/src/instance.rs b/src/frontend/src/instance.rs index f19cdf54ec2..973e82d62bf 100644 --- a/src/frontend/src/instance.rs +++ b/src/frontend/src/instance.rs @@ -30,6 +30,7 @@ mod otlp; pub mod prom_store; mod promql; mod region_query; +mod schema_selection; use std::collections::HashSet; use std::pin::Pin; @@ -63,6 +64,7 @@ use common_recordbatch::RecordBatchStreamWrapper; use common_recordbatch::error::StreamTimeoutSnafu; use common_telemetry::logging::SlowQueryOptions; use common_telemetry::{debug, error, tracing}; +use common_time::timestamp::TimeUnit; use datafusion::dataframe::DataFrame; use datafusion::physical_plan::ExecutionPlan; use datafusion_expr::LogicalPlan; @@ -90,6 +92,7 @@ use servers::prometheus_handler::{ ParsedPromQuery, PrometheusHandler, resolve_schema_from_matchers, }; use servers::query_handler::sql::SqlQueryHandler; +use servers::request_memory_limiter::ServerMemoryLimiter; use session::context::{Channel, MetricDiscoveryScope, QueryContextRef}; use session::table_name::table_idents_to_full_name; use snafu::prelude::*; @@ -120,6 +123,7 @@ use crate::stream_wrapper::CancellableStreamWrapper; /// [`servers::query_handler::sql::SqlQueryHandler`], etc. #[derive(Clone)] pub struct Instance { + server_memory_limiter: ServerMemoryLimiter, frontend_peer_addr: String, catalog_manager: CatalogManagerRef, pipeline_operator: Arc, @@ -135,6 +139,9 @@ pub struct Instance { process_manager: ProcessManagerRef, slow_query_options: SlowQueryOptions, influxdb_default_merge_mode: InfluxdbMergeMode, + /// Time index unit for physical metric tables created by auto table creation + /// through the frontend instance (e.g., batched logical-table flushes). + auto_create_physical_table_timestamp_unit: TimeUnit, trace_ingest_chunk_size: usize, trace_aux_cache: otlp::TraceAuxCache, trace_aux_limiter: Option, @@ -143,6 +150,11 @@ pub struct Instance { } impl Instance { + /// Returns the memory pool shared by protocol admission and table-batch conversion. + pub fn server_memory_limiter(&self) -> &ServerMemoryLimiter { + &self.server_memory_limiter + } + pub fn frontend_peer_addr(&self) -> &str { &self.frontend_peer_addr } @@ -401,6 +413,16 @@ impl Instance { } _ => { query_interceptor.pre_execute(Some(&stmt), None, query_ctx.clone())?; + if let Statement::SetVariables(set) = &stmt + && query_ctx.channel() == Channel::Postgres + && let Some(schemas) = set.search_path() + { + return self.set_search_path(schemas, query_ctx).await; + } + if let Statement::Use(schema) = &stmt { + self.authorize_database(query_ctx.current_catalog(), schema, &query_ctx) + .await?; + } if let Statement::Copy(sql::statements::copy::Copy::CopyDatabase( CopyDatabase::From(arg), )) = &stmt @@ -566,6 +588,8 @@ impl Instance { stmt: &Statement, query_ctx: &QueryContextRef, ) -> Result<()> { + self.check_unselected_mysql_database(stmt, query_ctx) + .await?; self.plugins .get::() .as_ref() @@ -787,7 +811,9 @@ impl Instance { } Err(e) => { if e.status_code().should_log_error() { - error!(e; "Failed to execute query: {stmt}"); + error!(e; user = query_ctx.current_user().username(), protocol = %query_ctx.channel(), + catalog = query_ctx.current_catalog(), schema = %query_ctx.current_schema(), + process_id = query_ctx.process_id(), "Failed to execute query: {stmt}"); } else { debug!("Failed to execute query: {stmt}, {e}"); } @@ -842,6 +868,15 @@ impl Instance { query_ctx: QueryContextRef, ) -> Result { ensure!(!self.is_suspended(), error::SuspendedSnafu); + if let Some(stmt) = &stmt { + self.check_unselected_mysql_database(stmt, &query_ctx) + .await?; + } else if query_ctx.channel() == Channel::Mysql && query_ctx.current_schema().is_empty() { + return Err(BoxedError::new( + session::SessionError::NoDatabaseSelected {}, + )) + .context(ExternalSnafu); + } let query_interceptor_opt = self.plugins.get::>(); let query_interceptor = query_interceptor_opt.as_ref(); @@ -998,6 +1033,8 @@ impl Instance { query_ctx: QueryContextRef, ) -> Result> { ensure!(!self.is_suspended(), error::SuspendedSnafu); + self.check_unselected_mysql_database(&stmt, &query_ctx) + .await?; // EXPLAIN / EXPLAIN ANALYZE wrap an inner statement; describe them when the // wrapped statement is something we already plan (so that bind parameters @@ -3738,6 +3775,429 @@ mod tests { Ok(()) } + #[tokio::test] + async fn test_sql_schema_selection_authorizes_before_mutation() -> TestResult<()> { + struct SchemaProvider; + #[async_trait::async_trait] + impl auth::UserProvider for SchemaProvider { + fn name(&self) -> &str { + "schema-test" + } + async fn authenticate( + &self, + _: auth::Identity<'_>, + _: auth::Password<'_>, + ) -> auth::error::Result { + unreachable!("the request already has an authenticated user") + } + async fn authorize( + &self, + _: &str, + schema: &str, + _: &UserInfoRef, + ) -> auth::error::Result<()> { + if schema == "backend_error" { + return auth::error::InternalStateSnafu { + msg: "backend unavailable", + } + .fail(); + } + if matches!(schema, "public" | "allowed" | "allowed_missing" | "admin") { + Ok(()) + } else { + auth::error::PermissionDeniedSnafu.fail() + } + } + } + let plugins = Plugins::new(); + plugins.insert::(Arc::new(SchemaProvider)); + let catalog = + catalog::memory::MemoryCatalogManager::new_with_table(test_table(1024, "source")?); + for schema in ["denied", "allowed", "admin"] { + catalog + .register_schema_sync(catalog::RegisterSchemaRequest { + catalog: "greptime".to_string(), + schema: schema.to_string(), + }) + .unwrap(); + } + let instance = test_instance_with_catalog_manager( + catalog, + test_table(1025, "target")?, + plugins, + None, + Arc::new(NoopProcedureExecutor), + ) + .await?; + for sql in [ + "Use denied", + " USE denied", + "/* switch */ USE denied", + "USE public; Use denied", + "USE missing", + ] { + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + let results = instance.do_query_inner(sql, ctx.clone()).await; + assert_eq!( + StatusCode::PermissionDenied, + results.last().unwrap().as_ref().unwrap_err().status_code(), + "{sql}" + ); + assert_eq!("public", ctx.current_schema()); + } + for sql in ["USE allowed", "SET search_path TO allowed"] { + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + assert!( + instance + .do_query_inner(sql, ctx.clone()) + .await + .remove(0) + .is_ok() + ); + assert_eq!("allowed", ctx.current_schema()); + } + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + let error = instance + .do_query_inner("USE allowed_missing", ctx.clone()) + .await + .remove(0) + .unwrap_err(); + assert_eq!(StatusCode::InvalidArguments, error.status_code()); + assert_eq!("public", ctx.current_schema()); + for (sql, expected) in [ + ("SET search_path TO denied, public", "public"), + ("SET search_path TO DEFAULT", "admin"), + ("SET search_path TO allowed_missing, allowed", "allowed"), + ( + "SET search_path TO denied, allowed_missing, public", + "public", + ), + (r#"SET search_path TO "$user", public"#, "admin"), + (r#"SET "search_path" TO denied, allowed"#, "allowed"), + ("SET SEARCH_PATH TO 'denied'", ""), + ("SET search_path TO allowed_missing", ""), + ("SET search_path TO ''", ""), + ("SET search_path TO allowed; SET search_path TO denied", ""), + ] { + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + ctx.set_current_user(Arc::new(AdminUserInfo)); + let results = instance.do_query_inner(sql, ctx.clone()).await; + assert!( + results.iter().all(|result| matches!( + result, + Ok(Output { + data: OutputData::AffectedRows(0), + .. + }) + )), + "{sql}: {results:?}" + ); + assert_eq!(expected, ctx.current_schema(), "{sql}"); + } + + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + // The default user's schema is absent or unauthorized, so public remains usable. + assert!( + instance + .do_query_inner(r#"SET search_path TO "$user", public"#, ctx.clone()) + .await + .remove(0) + .is_ok() + ); + assert_eq!("public", ctx.current_schema()); + assert!( + instance + .do_query_inner("SET search_path TO DEFAULT", ctx.clone()) + .await + .remove(0) + .is_ok() + ); + assert_eq!("public", ctx.current_schema()); + let error = instance + .do_query_inner("SET search_path TO backend_error, allowed", ctx.clone()) + .await + .remove(0) + .unwrap_err(); + assert_eq!(StatusCode::Unexpected, error.status_code()); + assert_eq!("public", ctx.current_schema()); + Ok(()) + } + + #[tokio::test] + async fn test_mysql_without_selected_database() -> TestResult<()> { + let plugins = Plugins::new(); + plugins.insert::(Arc::new(auth::tests::MockUserProvider::default())); + let instance = test_instance_with_plugins( + test_table(1024, "source")?, + test_table(1025, "target")?, + plugins, + ) + .await?; + let ctx = Arc::new(QueryContext::with_channel("greptime", "", Channel::Mysql)); + for sql in [ + "SELECT * FROM source", + "INSERT INTO source VALUES (1, 0)", + "CREATE TABLE t (ts TIMESTAMP TIME INDEX)", + "SHOW TABLES", + "TRUNCATE TABLE source", + "SHOW CREATE TABLE source", + "COPY source TO '/tmp/out'", + "SELECT flush_table('public.source')", + ] { + let result = instance.do_query_inner(sql, ctx.clone()).await.remove(0); + let error = match result { + Err(error) => Err::<(), _>(BoxedError::new(error)) + .context(servers::error::ExecuteQuerySnafu) + .unwrap_err(), + Ok(output) => { + let OutputData::Stream(stream) = output.data else { + panic!("Expected a query stream: {sql}"); + }; + common_recordbatch::util::collect(stream) + .await + .context(servers::error::CollectRecordbatchSnafu) + .unwrap_err() + } + }; + let (kind, message) = servers::mysql::writer::handle_err(error, ctx.clone()); + assert_eq!(1046, kind as u16, "{sql}: {message}"); + assert!(message.contains("No database selected"), "{message}"); + } + let output = instance + .do_query_inner( + "SELECT 1, DATABASE() IS NULL AS no_database, SCHEMA() IS NULL AS no_schema", + ctx.clone(), + ) + .await + .remove(0) + .unwrap(); + assert_eq!(2, output.data.pretty_print().await.matches("true").count()); + for sql in [ + "SELECT * FROM denied.source", + "SHOW TABLES FROM denied", + "USE denied", + ] { + let error = instance + .do_query_inner(sql, ctx.clone()) + .await + .remove(0) + .unwrap_err(); + assert_eq!( + StatusCode::AccessDenied, + error.status_code(), + "{sql}: {error}" + ); + assert!(ctx.current_schema().is_empty()); + } + for sql in [ + "SELECT * FROM public.source", + "SHOW TABLES FROM public", + "SET time_zone = '+00:00'", + ] { + assert!( + instance + .do_query_inner(sql, ctx.clone()) + .await + .remove(0) + .is_ok(), + "{sql}" + ); + } + let plan = instance + .do_describe_inner(parse_one_sql("SELECT * FROM public.source"), ctx.clone()) + .await + .unwrap() + .unwrap() + .logical_plan; + instance + .do_exec_plan_inner( + plan.clone(), + Some(parse_one_sql("SELECT * FROM public.source")), + ctx.clone(), + ) + .await + .unwrap(); + let error = instance + .do_describe_inner(parse_one_sql("SELECT * FROM denied.source"), ctx.clone()) + .await + .unwrap_err(); + assert_eq!(StatusCode::AccessDenied, error.status_code()); + assert!(ctx.current_schema().is_empty()); + let error = instance + .do_exec_plan_inner(plan.clone(), None, ctx.clone()) + .await + .unwrap_err(); + assert_eq!(StatusCode::DatabaseNotFound, error.status_code()); + let current_user = ctx.current_user(); + ctx.set_current_user(Arc::new(AdminUserInfo)); + let error = instance + .do_exec_plan_inner( + plan, + Some(parse_one_sql("SELECT * FROM public.source")), + ctx.clone(), + ) + .await + .unwrap_err(); + assert_eq!(StatusCode::AccessDenied, error.status_code()); + ctx.set_current_user(current_user); + instance + .do_query_inner("USE public", ctx.clone()) + .await + .remove(0) + .unwrap(); + assert_eq!("public", ctx.current_schema()); + instance + .do_query_inner("SELECT * FROM source", ctx) + .await + .remove(0) + .unwrap(); + Ok(()) + } + + #[tokio::test] + async fn test_empty_search_path_statements() -> TestResult<()> { + let instance = test_instance_with_plugins( + test_table(1024, "source")?, + test_table(1025, "target")?, + Plugins::new(), + ) + .await?; + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + instance + .do_query_inner("SET search_path TO ''", ctx.clone()) + .await + .remove(0) + .unwrap(); + let tables = instance + .do_query_inner("SHOW TABLES", ctx.clone()) + .await + .remove(0) + .unwrap() + .data + .pretty_print() + .await; + assert!( + !tables.contains("source") && !tables.contains("target"), + "{tables}" + ); + let error = instance + .do_query_inner("INSERT INTO source VALUES (1, 0)", ctx.clone()) + .await + .remove(0) + .unwrap_err(); + assert!( + matches!(error, Error::TableOperation { source: operator::error::Error::TableNotFound { ref table_name, .. }, .. } if table_name == "greptime..source"), + "{error:?}" + ); + for sql in [ + "CREATE TABLE no_schema (ts TIMESTAMP TIME INDEX)", + "ALTER TABLE source ADD COLUMN no_schema_column INT", + "CREATE VIEW no_schema_view AS SELECT 1 AS n", + "CREATE FLOW no_schema_flow SINK TO sink AS SELECT * FROM public.source", + "CREATE FLOW no_schema_flow SINK TO public.sink AS SELECT * FROM source", + ] { + let error = instance + .do_query_inner(sql, ctx.clone()) + .await + .remove(0) + .unwrap_err(); + assert!( + matches!(error, Error::TableOperation { source: operator::error::Error::SchemaNotFound { ref schema_info, .. }, .. } if schema_info.is_empty()), + "{sql}: {error:?}" + ); + } + + for sql in ["SELECT * FROM source", "DELETE FROM source"] { + assert!( + instance + .do_query_inner(sql, ctx.clone()) + .await + .remove(0) + .is_err(), + "{sql}" + ); + } + let output = instance + .do_query_inner( + "SELECT current_schema() IS NULL AS empty, current_schemas(false) AS schemas", + ctx.clone(), + ) + .await + .remove(0) + .unwrap(); + let printed = output.data.pretty_print().await; + assert!( + printed.contains("true") && printed.contains("[]"), + "{printed}" + ); + let output = instance + .do_query_inner("DROP TABLE IF EXISTS source", ctx.clone()) + .await + .remove(0) + .unwrap(); + assert!(matches!(output.data, OutputData::AffectedRows(0))); + instance + .do_query_inner("USE public", ctx.clone()) + .await + .remove(0) + .unwrap(); + assert!( + instance + .do_query_inner("SELECT * FROM source", ctx) + .await + .remove(0) + .is_ok() + ); + Ok(()) + } + + #[tokio::test] + async fn test_operator_rejects_search_path_without_frontend() -> TestResult<()> { + let instance = test_instance_with_plugins( + test_table(1024, "source")?, + test_table(1025, "target")?, + Plugins::new(), + ) + .await?; + let ctx = Arc::new(QueryContext::with_channel( + "greptime", + "public", + Channel::Postgres, + )); + let error = instance + .statement_executor + .execute_sql(parse_one_sql("SET search_path TO denied"), ctx.clone()) + .await + .unwrap_err(); + assert!(matches!(error, operator::error::Error::NotSupported { .. })); + assert_eq!("public", ctx.current_schema()); + Ok(()) + } + #[tokio::test] async fn test_write_only_grpc_sql_is_checked_after_parsing() -> TestResult<()> { let plugins = Plugins::new(); @@ -4232,6 +4692,58 @@ mod tests { assert!(!should_track_plan_process(None, &insert_dml_plan())); } + #[tokio::test] + async fn test_exemplar_slow_query() -> TestResult<()> { + use common_catalog::consts::METRICS_EXEMPLARS_TABLE; + use servers::exemplar::query::Query as ExemplarQuery; + + for invalid_destination in [false, true] { + let table_name = if invalid_destination { + METRICS_EXEMPLARS_TABLE + } else { + "source" + }; + let mut instance = test_instance_with_tables( + test_table(1024, table_name)?, + test_table(1025, "target")?, + ) + .await?; + let recorder = Arc::new(RecordingSlowQueryEventRecorder::default()); + instance.slow_query_recorder = recorder.clone(); + instance.slow_query_options.sample_ratio = 1.0; + instance.slow_query_options.record_type = SlowQueriesRecordType::SystemTable; + + for (enabled, threshold, should_record) in [ + (true, Duration::ZERO, true), + (false, Duration::ZERO, false), + (true, Duration::from_secs(3600), false), + ] { + instance.slow_query_options.enable = enabled; + instance.slow_query_options.threshold = threshold; + let query = + ExemplarQuery::parse("source".into(), "-0.001", "0.001", "public").unwrap(); + let result = instance.query_exemplars(query, QueryContext::arc()).await; + if invalid_destination { + assert!(result.is_err()); + } else { + assert!(result.unwrap().is_empty()); + } + + let queries = std::mem::take(&mut *recorder.queries.lock().unwrap()); + let expected = if should_record { + vec!["query_exemplars(source, start_ns=-1000000, end_ns=1999999)"] + } else { + vec![] + }; + assert_eq!( + queries, expected, + "invalid_destination={invalid_destination}, enabled={enabled}, threshold={threshold:?}" + ); + } + } + Ok(()) + } + #[tokio::test] async fn test_insert_select_slow_query() -> TestResult<()> { let mut instance = @@ -5134,6 +5646,43 @@ mod tests { Ok(()) } + #[tokio::test] + async fn test_copy_in_checks_table_permissions() -> TestResult<()> { + use servers::query_handler::CopyInHandler; + + for source in [ + test_table(1024, "denied")?, + test_physical_table(1024, "physical")?, + ] { + let name = source.table_info().name.clone(); + let plugins = Plugins::new(); + plugins.insert::(Arc::new(RejectUnresolvedPermissionChecker)); + let instance = + test_instance_with_plugins(source, test_table(1025, "target")?, plugins).await?; + let ctx = QueryContext::arc(); + assert_permission_denied( + instance + .copy_in_table("greptime", "public", &name, ctx.clone()) + .await, + ); + assert!( + instance + .copy_in_table("greptime", "public", "target", ctx.clone()) + .await + .unwrap() + .is_some() + ); + assert!( + instance + .copy_in_table("greptime", "public", "missing", ctx) + .await + .unwrap() + .is_none() + ); + } + Ok(()) + } + /// COPY FROM STDIN must enforce the same cross-catalog restriction as /// SQL INSERT when `disallow_cross_catalog_query` is configured. #[tokio::test] diff --git a/src/frontend/src/instance/builder.rs b/src/frontend/src/instance/builder.rs index 395d9a8af26..3703c5bedea 100644 --- a/src/frontend/src/instance/builder.rs +++ b/src/frontend/src/instance/builder.rs @@ -53,6 +53,7 @@ use query::QueryEngineFactory; use query::region_query::RegionQueryHandlerFactoryRef; use servers::batcher::table::TablePendingRowsBatcher; use servers::batcher::{BatchingProtocol, pending_rows_batch_sync_enabled}; +use servers::request_memory_limiter::ServerMemoryLimiter; use snafu::{OptionExt, ResultExt}; use crate::error::{self, DataFusionSnafu, ExternalSnafu, Result}; @@ -252,10 +253,17 @@ impl FrontendBuilder { table_flownode_cache.clone(), self.options.auto_create_table, ) + .with_auto_create_physical_table_timestamp_unit( + self.options.auto_create_physical_table_timestamp_unit, + ) .with_write_dispatcher(write_dispatcher.clone()) }; // The execution-only inserter owns no batchers, avoiding an Arc cycle. let bulk_inserter = Arc::new(create_inserter()); + let server_memory_limiter = ServerMemoryLimiter::new( + self.options.max_in_flight_write_bytes.as_bytes(), + self.options.write_bytes_exhausted_policy, + ); let build_batcher = |options: &BatcherOptions| -> Option> { if !options.pending_rows_batching_enabled() || (self.options.prom_store.with_metric_engine @@ -273,6 +281,7 @@ impl FrontendBuilder { options.worker_channel_capacity, options.max_inflight_requests, options.flow_notification_queue_capacity, + server_memory_limiter.clone(), bulk_inserter.clone(), ) .map(|batcher| batcher as Arc) @@ -417,6 +426,7 @@ impl FrontendBuilder { }; Ok(Instance { + server_memory_limiter, logical_batcher: Default::default(), frontend_peer_addr, catalog_manager: self.catalog_manager, @@ -432,6 +442,9 @@ impl FrontendBuilder { process_manager, slow_query_options: self.options.slow_query.clone(), influxdb_default_merge_mode: self.options.influxdb.default_merge_mode, + auto_create_physical_table_timestamp_unit: self + .options + .auto_create_physical_table_timestamp_unit, trace_ingest_chunk_size: self.options.otlp.trace_ingest_chunk_size, trace_aux_cache: TraceAuxCache::new(self.options.otlp.trace_aux_cache_size.as_bytes()), trace_aux_limiter, diff --git a/src/frontend/src/instance/copy_in.rs b/src/frontend/src/instance/copy_in.rs index 6580cc44b9d..6ffa3dfe27f 100644 --- a/src/frontend/src/instance/copy_in.rs +++ b/src/frontend/src/instance/copy_in.rs @@ -14,7 +14,10 @@ use api::v1::RowInsertRequests; use async_trait::async_trait; -use auth::{PermissionChecker, PermissionCheckerRef, PermissionReq}; +use auth::{ + PermissionChecker, PermissionCheckerRef, PermissionReq, PermissionTableTarget, + PermissionTableTargets, +}; use common_error::ext::BoxedError; use common_query::Output; use query::query_engine::options::{QueryOptions, validate_catalog_and_schema}; @@ -67,11 +70,28 @@ impl CopyInHandler for Instance { ) .context(AuthSnafu)?; - self.catalog_manager() + let table_info = self + .catalog_manager() .table(catalog, schema, table, None) .await .map(|table| table.map(|table| table.table_info())) - .map_err(servers::error::Error::from) + .map_err(servers::error::Error::from)?; + let target = PermissionTableTarget::new(catalog, schema, table).with_physical_metric_table( + table_info + .as_ref() + .is_some_and(|info| info.is_physical_table()), + ); + self.check_table_permission( + &query_ctx, + PermissionReq::BulkInsert { + catalog, + schema, + table, + }, + PermissionTableTargets::resolved(vec![target]), + ) + .context(AuthSnafu)?; + Ok(table_info) } async fn copy_in_insert( diff --git a/src/frontend/src/instance/entity_graph.rs b/src/frontend/src/instance/entity_graph.rs index 09c23651bcc..603f10568c3 100644 --- a/src/frontend/src/instance/entity_graph.rs +++ b/src/frontend/src/instance/entity_graph.rs @@ -442,11 +442,10 @@ impl EntityGraphProviderImpl { && (is_trace || !table_declarations.is_empty()) && !self.authorize_sources( query_ctx, - PermissionTableTargets::resolved(vec![PermissionTableTarget::new( - catalog, - &schema, - &table_info.name, - )]), + PermissionTableTargets::resolved(vec![ + PermissionTableTarget::new(catalog, &schema, &table_info.name) + .with_physical_metric_table(table_info.is_physical_table()), + ]), )? { debug!( @@ -617,13 +616,17 @@ impl EntityGraphProviderImpl { else { return Ok(None); }; + let table_info = table.table_info(); if !self.authorize_sources( query_ctx, - PermissionTableTargets::resolved(vec![PermissionTableTarget::new( - catalog, - DEFAULT_PRIVATE_SCHEMA_NAME, - SEMANTIC_RELATIONSHIPS_DECLARED_TABLE_NAME, - )]), + PermissionTableTargets::resolved(vec![ + PermissionTableTarget::new( + catalog, + DEFAULT_PRIVATE_SCHEMA_NAME, + SEMANTIC_RELATIONSHIPS_DECLARED_TABLE_NAME, + ) + .with_physical_metric_table(table_info.is_physical_table()), + ]), )? { debug!( "Excluding declared edges: caller lacks read permission on \ @@ -631,7 +634,6 @@ impl EntityGraphProviderImpl { ); return Ok(None); } - let table_info = table.table_info(); if !declared_relationships_schema_matches(&table_info) { return Err(BoxedError::new( error::InvalidSqlSnafu { @@ -797,6 +799,77 @@ mod tests { use super::*; + struct DenyPhysicalTables; + + impl PermissionChecker for DenyPhysicalTables { + fn check_permission( + &self, + _: auth::UserInfoRef, + _: PermissionReq, + ) -> auth::error::Result { + Ok(auth::PermissionResp::Allow) + } + + fn check_permission_with_table_targets( + &self, + _: auth::UserInfoRef, + _: PermissionReq, + targets: PermissionTableTargets, + ) -> auth::error::Result { + let PermissionTableTargets::Resolved(targets) = targets else { + panic!("entity graph must resolve its sources"); + }; + Ok( + if targets.iter().any(|target| target.is_physical_metric_table) { + auth::PermissionResp::Reject + } else { + auth::PermissionResp::Allow + }, + ) + } + } + + #[tokio::test] + async fn test_enumerate_authorizes_physical_tables() { + for physical in [false, true] { + let mut options = vec![("greptime.semantic.entity.service.id", "service_name")]; + if physical { + options.push((PHYSICAL_TABLE_METADATA_KEY, "")); + } + let info = table_info(&["service_name"], &options); + let batch = common_recordbatch::RecordBatch::new_empty(info.meta.schema.clone()); + let source = table::test_util::MemTable::table("t1", batch); + let table = Arc::new(table::Table::new( + Arc::new(info), + table::metadata::FilterPushDownType::Unsupported, + source.data_source(), + )); + let catalog: Arc = + catalog::memory::MemoryCatalogManager::new_with_table(table); + let engine = query::QueryEngineFactory::new( + catalog.clone(), + None, + None, + None, + None, + false, + query::options::QueryOptions::default(), + ) + .query_engine(); + let provider = EntityGraphProviderImpl::new( + engine, + Arc::downgrade(&catalog), + Some(Arc::new(DenyPhysicalTables)), + ); + let ctx = QueryContext::arc(); + let (sources, _) = provider + .enumerate(DEFAULT_CATALOG_NAME, Some(&ctx)) + .await + .unwrap(); + assert_eq!(sources.len(), usize::from(!physical)); + } + } + fn table_info(columns: &[&str], extra: &[(&str, &str)]) -> TableInfo { let mut column_schemas = vec![ ColumnSchema::new( diff --git a/src/frontend/src/instance/exemplar.rs b/src/frontend/src/instance/exemplar.rs index fc0f4862cff..e3b74388c4f 100644 --- a/src/frontend/src/instance/exemplar.rs +++ b/src/frontend/src/instance/exemplar.rs @@ -16,12 +16,13 @@ use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; use auth::{PermissionTableTarget, PermissionTableTargets}; +use catalog::process_manager::{QueryStatement as CatalogQueryStatement, SlowQueryTimer}; use common_base::cancellation::CancellableFuture; use common_catalog::consts::METRICS_EXEMPLARS_TABLE; use common_error::ext::BoxedError; use common_query::OutputData; use common_query::prelude::{ - EXEMPLAR_LABELS_COLUMN, EXEMPLAR_METRIC_COLUMN, greptime_timestamp, greptime_value, + EXEMPLAR_METRIC_COLUMN, greptime_exemplar_labels, greptime_timestamp, greptime_value, }; use common_query::prometheus::format_prometheus_float; use common_recordbatch::RecordBatch; @@ -43,7 +44,8 @@ use servers::prometheus_handler::PrometheusHandler; use session::context::QueryContextRef; use snafu::{OptionExt, ResultExt, ensure}; use store_api::metric_engine_consts::{ - DATA_SCHEMA_TABLE_ID_COLUMN_NAME, DATA_SCHEMA_TSID_COLUMN_NAME, + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, DATA_SCHEMA_TSID_COLUMN_NAME, LOGICAL_TABLE_METADATA_KEY, + METRIC_ENGINE_NAME, }; use table::metadata::TableInfo; @@ -100,13 +102,27 @@ impl Instance { self.check_query_target_permission(PermissionTableTargets::resolved(targets), &local) .await?; } + let raw_query = format!( + "query_exemplars({}, start_ns={}, end_ns={})", + query.expression, query.bounds.0, query.bounds.1 + ); + let slow_query_timer = self.slow_query_options.enable.then(|| { + SlowQueryTimer::new( + CatalogQueryStatement::Plan(raw_query.clone()), + ctx.current_schema(), + self.slow_query_options.threshold, + self.slow_query_options.sample_ratio, + self.slow_query_options.record_type, + self.slow_query_recorder.clone(), + ) + }); let ticket = self.process_manager.register_query( ctx.current_catalog().to_string(), groups.keys().cloned().collect(), - format!("query_exemplars({})", query.expression), + raw_query, ctx.conn_info().to_string(), Some(ctx.process_id()), - None, + slow_query_timer, ); let cancellation = ticket.cancellation_handle.clone(); let future = async { @@ -144,6 +160,8 @@ impl Instance { .map_err(BoxedError::new) .context(error::ExecuteQuerySnafu)?; let filter = filter(&info, &selectors, query.bounds)?; + let source_tags = self.exemplar_source_tags(&selectors, &local).await?; + let (projection, tags) = exemplar_projection(&info, source_tags.as_ref()); let frame = self .query_engine .read_table(table) @@ -152,6 +170,8 @@ impl Instance { let plan = frame .filter(filter) .context(error::DataFusionSnafu)? + .select(projection) + .context(error::DataFusionSnafu)? .into_unoptimized_plan(); interceptor.pre_execute_exemplars(&query, info.table_ref(), local.clone())?; let output = self @@ -165,7 +185,7 @@ impl Instance { while let Some(batch) = stream.next().await { skipped += collect( batch.context(error::CollectRecordbatchSnafu)?, - &info, + &tags, &mut observations, &mut reservation, )?; @@ -174,7 +194,7 @@ impl Instance { } OutputData::RecordBatches(batches) => { for batch in batches.take() { - skipped += collect(batch, &info, &mut observations, &mut reservation)?; + skipped += collect(batch, &tags, &mut observations, &mut reservation)?; tokio::task::yield_now().await; } } @@ -207,6 +227,61 @@ impl Instance { drop(ticket); result } + + async fn exemplar_source_tags( + &self, + selectors: &[&Selector], + ctx: &QueryContextRef, + ) -> Result>> { + let Some(metrics) = selectors + .iter() + .map(|selector| { + selector.matchers.iter().find_map(|matcher| { + (matcher.name == METRIC_NAME + && matches!(matcher.op, MatchOp::Equal) + && !matcher.value.is_empty()) + .then_some(matcher.value.as_str()) + }) + }) + .collect::>>() + else { + return Ok(None); + }; + let mut tags = BTreeSet::new(); + for metric in metrics { + let Some(table) = self + .catalog_manager + .table( + ctx.current_catalog(), + &ctx.current_schema(), + metric, + Some(ctx), + ) + .await + .map_err(BoxedError::new) + .context(error::ExecuteQuerySnafu)? + else { + return Ok(None); + }; + let info = table.table_info(); + if info.meta.engine != METRIC_ENGINE_NAME + || !info + .meta + .options + .extra_options + .contains_key(LOGICAL_TABLE_METADATA_KEY) + { + return Ok(None); + } + tags.extend( + info.meta + .primary_key_indices + .iter() + .map(|&index| info.meta.schema.column_schemas()[index].name.clone()), + ); + } + Ok(Some(tags)) + } } fn column(name: &str) -> Expr { @@ -310,9 +385,44 @@ fn matcher_filter(matcher: &Matcher, tags: &BTreeSet<&str>) -> Expr { } } +fn exemplar_projection<'a>( + info: &'a TableInfo, + source_tags: Option<&BTreeSet>, +) -> (Vec, Vec<&'a str>) { + // Exact metric queries use current logical schemas, including for historical rows. + let tags = info + .meta + .primary_key_indices + .iter() + .map(|&index| info.meta.schema.column_schemas()[index].name.as_str()) + .filter(|name| { + ![ + EXEMPLAR_METRIC_COLUMN, + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, + DATA_SCHEMA_TSID_COLUMN_NAME, + ] + .contains(name) + && source_tags.is_none_or(|tags| tags.contains(*name)) + }) + .collect::>(); + let projection = [ + EXEMPLAR_METRIC_COLUMN, + greptime_timestamp(), + greptime_value(), + greptime_exemplar_labels(), + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, + DATA_SCHEMA_TSID_COLUMN_NAME, + ] + .into_iter() + .chain(tags.iter().copied()) + .map(column) + .collect(); + (projection, tags) +} + fn collect( batch: RecordBatch, - info: &TableInfo, + tag_names: &[&str], observations: &mut Observations, reservation: &mut MemoryReservation, ) -> Result { @@ -331,6 +441,12 @@ fn collect( let value = Helper::try_into_vector(batch.column(index(greptime_value())?)) .map_err(BoxedError::new) .context(error::ExecuteQuerySnafu)?; + let table_id = Helper::try_into_vector(batch.column(index(DATA_SCHEMA_TABLE_ID_COLUMN_NAME)?)) + .map_err(BoxedError::new) + .context(error::ExecuteQuerySnafu)?; + let tsid = Helper::try_into_vector(batch.column(index(DATA_SCHEMA_TSID_COLUMN_NAME)?)) + .map_err(BoxedError::new) + .context(error::ExecuteQuerySnafu)?; let value_type = |vector: &VectorRef| { let mut data_type = vector.data_type(); // Storage may dictionary-encode tags; Vector::get returns their decoded values. @@ -342,37 +458,30 @@ fn collect( ensure!( value_type(&metric).is_string() && value_type(&time).is_timestamp() - && value_type(&value) == ConcreteDataType::float64_datatype(), + && value_type(&value) == ConcreteDataType::float64_datatype() + && value_type(&table_id) == ConcreteDataType::uint32_datatype() + && value_type(&tsid) == ConcreteDataType::uint64_datatype(), error::UnexpectedResultSnafu { reason: "exemplar scan returned incompatible column types", } ); - let label_index = index(EXEMPLAR_LABELS_COLUMN)?; + let label_index = index(greptime_exemplar_labels())?; let labels = JsonArray::from(batch.column(label_index)) .project_to_v2(schema.field(label_index), &DataType::Binary) .map_err(BoxedError::new) .context(error::ExecuteQuerySnafu)?; let mut tags = Vec::<(&str, VectorRef)>::new(); - for tag in &info.meta.primary_key_indices { - let name = info.meta.schema.column_schemas()[*tag].name.as_str(); - if ![ - EXEMPLAR_METRIC_COLUMN, - DATA_SCHEMA_TABLE_ID_COLUMN_NAME, - DATA_SCHEMA_TSID_COLUMN_NAME, - ] - .contains(&name) - { - let values = Helper::try_into_vector(batch.column(index(name)?)) - .map_err(BoxedError::new) - .context(error::ExecuteQuerySnafu)?; - ensure!( - value_type(&values).is_string(), - error::UnexpectedResultSnafu { - reason: "exemplar scan returned a non-string tag", - } - ); - tags.push((name, values)); - } + for &name in tag_names { + let values = Helper::try_into_vector(batch.column(index(name)?)) + .map_err(BoxedError::new) + .context(error::ExecuteQuerySnafu)?; + ensure!( + value_type(&values).is_string(), + error::UnexpectedResultSnafu { + reason: "exemplar scan returned a non-string tag", + } + ); + tags.push((name, values)); } let mut skipped = 0; 'rows: for row in 0..batch.num_rows() { @@ -393,7 +502,19 @@ fn collect( skipped += 1; continue; }; - let mut series = BTreeMap::from([(METRIC_NAME.to_string(), metric.into_string())]); + let (Value::UInt32(table_id), Value::UInt64(tsid)) = (table_id.get(row), tsid.get(row)) + else { + skipped += 1; + continue; + }; + let mut series = BTreeMap::from([ + (METRIC_NAME.to_string(), metric.into_string()), + ( + DATA_SCHEMA_TABLE_ID_COLUMN_NAME.to_string(), + table_id.to_string(), + ), + (DATA_SCHEMA_TSID_COLUMN_NAME.to_string(), tsid.to_string()), + ]); for (name, values) in &tags { if let Value::String(value) = values.get(row) { series.insert((*name).to_string(), value.into_string()); @@ -449,13 +570,146 @@ fn collect( #[cfg(test)] mod tests { use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool}; + use datatypes::arrow::array::{ + ArrayRef, Float64Array, StringArray, StructArray, TimestampNanosecondArray, UInt32Array, + UInt64Array, + }; + use datatypes::arrow::datatypes::Field; + use datatypes::extension::json::Json2ExtensionType; use datatypes::prelude::ConcreteDataType; use datatypes::schema::{ColumnSchema, Schema}; use datatypes::vectors::Float64Vector; - use table::metadata::{TableInfoBuilder, TableMeta}; + use table::metadata::{TableInfoBuilder, TableMetaBuilder}; use super::*; + #[test] + fn test_exemplar_projection_prunes_tags_and_preserves_source_ids() { + let labels = StructArray::from(vec![( + Arc::new(Field::new("trace_id", DataType::Utf8, false)), + Arc::new(StringArray::from(vec!["trace"; 4])) as ArrayRef, + )]); + let columns: Vec<(&str, ArrayRef)> = vec![ + ( + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, + Arc::new(UInt32Array::from(vec![1, 1, 2, 2])), + ), + ( + DATA_SCHEMA_TSID_COLUMN_NAME, + Arc::new(UInt64Array::from(vec![u64::MAX, u64::MAX, u64::MAX, 2])), + ), + ( + EXEMPLAR_METRIC_COLUMN, + Arc::new(StringArray::from(vec!["metric"; 4])), + ), + ( + "job", + Arc::new(StringArray::from(vec![Some(""), Some(""), Some(""), None])), + ), + ( + greptime_timestamp(), + Arc::new(TimestampNanosecondArray::from(vec![1; 4])), + ), + (greptime_value(), Arc::new(Float64Array::from(vec![1.0; 4]))), + (greptime_exemplar_labels(), Arc::new(labels)), + ( + "unrelated_tag", + Arc::new(StringArray::from(vec!["other"; 4])), + ), + ]; + let schema = Arc::new(Schema::new( + columns + .iter() + .map(|(name, values)| { + let mut column = ColumnSchema::new( + *name, + ConcreteDataType::from_arrow_type(values.data_type()), + true, + ); + if *name == greptime_exemplar_labels() { + column.with_extension_type(&Json2ExtensionType::default()); + } + column + }) + .collect::>(), + )); + let batch = RecordBatch::new( + schema.clone(), + columns + .into_iter() + .map(|(_, array)| Helper::try_into_vector(&array).unwrap()), + ) + .unwrap(); + let info = TableInfoBuilder::new( + METRICS_EXEMPLARS_TABLE, + TableMetaBuilder::empty() + .schema(schema) + .primary_key_indices(vec![0, 1, 2, 3, 7]) + .next_column_id(8) + .build() + .unwrap(), + ) + .build() + .unwrap(); + let source_tags = BTreeSet::from(["job".into(), "sample_only_tag".into()]); + let (projection, tags) = exemplar_projection(&info, Some(&source_tags)); + assert_eq!(tags, ["job"]); + assert_eq!( + projection, + [ + EXEMPLAR_METRIC_COLUMN, + greptime_timestamp(), + greptime_value(), + greptime_exemplar_labels(), + DATA_SCHEMA_TABLE_ID_COLUMN_NAME, + DATA_SCHEMA_TSID_COLUMN_NAME, + "job", + ] + .map(column) + ); + let (full_projection, full_tags) = exemplar_projection(&info, None); + let mut expected_full = projection.clone(); + expected_full.push(column("unrelated_tag")); + assert_eq!(full_projection, expected_full); + assert_eq!(full_tags, ["job", "unrelated_tag"]); + let (no_tag_projection, no_tags) = exemplar_projection(&info, Some(&BTreeSet::new())); + assert_eq!(no_tag_projection, projection[..6]); + assert!(no_tags.is_empty()); + let projected = batch.try_project(&[2, 4, 5, 6, 0, 1, 3]).unwrap(); + let pool: Arc = Arc::new(GreedyMemoryPool::new(1 << 20)); + let mut reservation = MemoryConsumer::new("projection test").register(&pool); + let mut full = Observations::new(); + let mut narrow = Observations::new(); + assert_eq!( + collect(batch, &tags, &mut full, &mut reservation).unwrap(), + 0 + ); + assert_eq!( + collect(projected, &tags, &mut narrow, &mut reservation).unwrap(), + 0 + ); + assert_eq!(full.len(), 3); + assert_eq!(full.values().map(BTreeMap::len).sum::(), 3); + assert_eq!(full, narrow); + for table_id in ["1", "2"] { + assert!(narrow.keys().any(|labels| { + labels[DATA_SCHEMA_TABLE_ID_COLUMN_NAME] == table_id + && labels[DATA_SCHEMA_TSID_COLUMN_NAME] == u64::MAX.to_string() + && labels.get("job").is_some_and(String::is_empty) + })); + } + assert!(narrow.keys().any(|labels| { + labels[DATA_SCHEMA_TABLE_ID_COLUMN_NAME] == "2" + && labels[DATA_SCHEMA_TSID_COLUMN_NAME] == "2" + && !labels.contains_key("job") + })); + assert!( + narrow + .keys() + .all(|labels| !labels.contains_key("unrelated_tag")) + ); + } + #[test] fn test_exemplar_scan_batch_respects_query_memory_budget() { let schema = Arc::new(Schema::new(vec![ColumnSchema::new( @@ -468,13 +722,10 @@ mod tests { vec![Arc::new(Float64Vector::from_vec(vec![1.0; 128])) as VectorRef], ) .unwrap(); - let info = TableInfoBuilder::new(METRICS_EXEMPLARS_TABLE, TableMeta::empty()) - .build() - .unwrap(); let pool: Arc = Arc::new(GreedyMemoryPool::new(1)); let mut reservation = MemoryConsumer::new("exemplar test").register(&pool); let mut observations = Observations::new(); - let error = collect(batch, &info, &mut observations, &mut reservation).unwrap_err(); + let error = collect(batch, &[], &mut observations, &mut reservation).unwrap_err(); assert!(matches!( error, error::Error::DataFusion { diff --git a/src/frontend/src/instance/logical_batcher.rs b/src/frontend/src/instance/logical_batcher.rs index 423f2996cae..3d1d7165179 100644 --- a/src/frontend/src/instance/logical_batcher.rs +++ b/src/frontend/src/instance/logical_batcher.rs @@ -34,7 +34,7 @@ impl Instance { .protocols .iter() .any(|protocol| match protocol { - BatchingProtocol::Prom => options.prom_store.enable, + BatchingProtocol::Prom => options.prom_store.enable || options.postgres.enable, BatchingProtocol::Otlp => options.otlp.enable, _ => false, }); @@ -53,6 +53,7 @@ impl Instance { self.table_flownode_set_cache().clone(), true, Arc::new(LogicalTables(Arc::downgrade(self))), + options.auto_create_physical_table_timestamp_unit, options_batcher.pending_rows_flush_interval, options_batcher.max_batch_rows, options_batcher.max_concurrent_flushes, diff --git a/src/frontend/src/instance/otlp.rs b/src/frontend/src/instance/otlp.rs index 5cc406a45dd..8afc1d99836 100644 --- a/src/frontend/src/instance/otlp.rs +++ b/src/frontend/src/instance/otlp.rs @@ -226,7 +226,10 @@ impl OpenTelemetryProtocolHandler for Instance { }); let mut ctx = ctx.fork(); // Queued and detached writes retain the reservation after caller cancellation. - ctx.set_write_progress(Arc::new(WriteProgress::with_reservation(reservation))); + ctx.set_write_progress(Arc::new(WriteProgress::with_reservation( + reservation.granted_bytes(), + reservation, + ))); Arc::new(ctx) } else { ctx @@ -297,18 +300,34 @@ impl OpenTelemetryProtocolHandler for Instance { .await .map_err(BoxedError::new) .context(error::ExecuteGrpcQuerySnafu); - let source_ready = prepared.is_ok(); - let sources = prepared - .as_ref() - .map(|request| request.source_tables().cloned().collect::>()) - .unwrap_or_default(); - let (metric_output, other_output, exemplar_output) = tokio::join!( + let mut exemplar_output = Ok(Output::new_with_affected_rows(0)); + let (metric_progress, other_output) = tokio::join!( async { - self.inserter - .handle_prepared_metric_inserts(prepared?, confirmed.clone()) + let prepared = prepared?; + let sources = prepared.source_tables().cloned().collect::>(); + let output = self + .inserter + .handle_prepared_metric_inserts(prepared, confirmed.clone()) .await .map_err(BoxedError::new) - .context(error::ExecuteGrpcQuerySnafu) + .context(error::ExecuteGrpcQuerySnafu)?; + outcome.write_cost += output.meta.cost; + // Successful insertion outputs can still carry storage failures. + let progress = output.write_progress().await; + if progress.error.is_none() { + exemplar_output = self + .inserter + .handle_exemplar_inserts( + exemplars, + confirmed.clone(), + &self.statement_executor, + &sources, + ) + .await + .map_err(BoxedError::new) + .context(error::ExecuteGrpcQuerySnafu); + } + Ok::<_, error::Error>(progress) }, async { if other.is_empty() { @@ -322,25 +341,12 @@ impl OpenTelemetryProtocolHandler for Instance { .await } }, - async { - // Failed source preparation must not bind exemplars to a later replacement table. - if !source_ready { - return Ok(Output::new_with_affected_rows(0)); - } - self.inserter - .handle_exemplar_inserts( - exemplars, - confirmed.clone(), - &self.statement_executor, - &sources, - ) - .await - .map_err(BoxedError::new) - .context(error::ExecuteGrpcQuerySnafu) - }, ); - let mut progress = common_query::WriteProgress::default(); - for output in [metric_output, other_output, exemplar_output] { + let mut progress = match metric_progress { + Ok(progress) => progress, + Err(error) => common_query::WriteProgress::failed(BoxedError::new(error)), + }; + for output in [other_output, exemplar_output] { match output { Ok(output) => { outcome.write_cost += output.meta.cost; diff --git a/src/frontend/src/instance/prom_store.rs b/src/frontend/src/instance/prom_store.rs index 963a8ae22ed..4d13d391843 100644 --- a/src/frontend/src/instance/prom_store.rs +++ b/src/frontend/src/instance/prom_store.rs @@ -788,7 +788,9 @@ impl Instance { let default_schema = vec![ api::v1::ColumnSchema { column_name: common_query::prelude::greptime_timestamp().to_string(), - datatype: api::v1::ColumnDataType::TimestampMillisecond as i32, + datatype: api::helper::timestamp_datatype( + self.auto_create_physical_table_timestamp_unit, + ) as i32, semantic_type: api::v1::SemanticType::Timestamp as i32, datatype_extension: None, options: None, diff --git a/src/frontend/src/instance/schema_selection.rs b/src/frontend/src/instance/schema_selection.rs new file mode 100644 index 00000000000..af0dbc33984 --- /dev/null +++ b/src/frontend/src/instance/schema_selection.rs @@ -0,0 +1,218 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashSet; + +use common_error::ext::{BoxedError, ErrorExt}; +use common_error::status_code::StatusCode; +use common_query::Output; +use common_telemetry::{debug, warn}; +use session::context::{Channel, QueryContextRef}; +use session::table_name::table_idents_to_full_name; +use snafu::{IntoError, ResultExt}; +use sql::ast::ObjectNamePartExt; +use sql::statements::copy::{Copy, CopyDatabase}; +use sql::statements::statement::Statement; +use sql::util::extract_tables_from_statement_checked; + +use crate::error::{ + Error, ExternalSnafu, InvalidSqlSnafu, PermissionSnafu, Result, TableOperationSnafu, +}; +use crate::instance::Instance; + +impl Instance { + /// Filters unauthorized candidates before selecting an existing schema. + pub(crate) async fn set_search_path( + &self, + schemas: Vec, + query_ctx: QueryContextRef, + ) -> Result { + let mut authorized = Vec::with_capacity(schemas.len()); + let provider = self.plugins.get::(); + let user = query_ctx.current_user(); + for schema in schemas { + let schema = if schema == "$user" { + user.username().to_string() + } else { + schema + }; + if schema.is_empty() { + continue; + } + if let Some(provider) = &provider { + match provider + .authorize(query_ctx.current_catalog(), &schema, &user) + .await + { + Ok(()) => {} + Err(error) + if matches!( + error.status_code(), + StatusCode::PermissionDenied | StatusCode::AccessDenied + ) => + { + debug!(user = user.username(), protocol = %query_ctx.channel(), + catalog = query_ctx.current_catalog(), schema = %schema, + process_id = query_ctx.process_id(), reason = %error.status_code(), + "Skipping unauthorized search_path candidate"); + continue; + } + Err(error) => return Err(error).context(PermissionSnafu), + } + } + authorized.push(schema); + } + self.statement_executor + .set_search_path(authorized, query_ctx) + .await + .context(TableOperationSnafu) + } + + /// Authorizes an explicit database access without changing the session. + pub(crate) async fn authorize_database( + &self, + catalog: &str, + schema: &str, + query_ctx: &QueryContextRef, + ) -> Result<()> { + let Some(provider) = self.plugins.get::() else { + return Ok(()); + }; + let user = query_ctx.current_user(); + if let Err(error) = provider.authorize(catalog, schema, &user).await { + if matches!( + error.status_code(), + StatusCode::PermissionDenied | StatusCode::AccessDenied + ) { + warn!(user = user.username(), protocol = %query_ctx.channel(), catalog, schema, + previous_schema = %query_ctx.current_schema(), process_id = query_ctx.process_id(), + reason = %error.status_code(), "Database access denied"); + } + return Err(error).context(PermissionSnafu); + } + Ok(()) + } + + /// Checks explicit targets before planning when no default scope is authorized. + pub(crate) async fn check_unselected_mysql_database( + &self, + stmt: &Statement, + query_ctx: &QueryContextRef, + ) -> Result<()> { + if query_ctx.channel() != Channel::Mysql || !query_ctx.current_schema().is_empty() { + return Ok(()); + } + for (catalog, schema) in resolve_mysql_database_scopes(stmt, query_ctx)? { + self.authorize_database(&catalog, &schema, query_ctx) + .await?; + } + Ok(()) + } +} + +fn resolve_mysql_database_scopes( + stmt: &Statement, + query_ctx: &QueryContextRef, +) -> Result> { + let mut stmt = stmt; + while let Statement::Explain(explain) = stmt { + stmt = &explain.statement; + } + let mut scopes = HashSet::new(); + let tables = extract_tables_from_statement_checked(stmt).ok_or_else(no_database_selected)?; + for table in tables { + let (catalog, schema, _) = table_idents_to_full_name(&table, query_ctx) + .map_err(BoxedError::new) + .context(ExternalSnafu)?; + if schema.is_empty() { + return Err(no_database_selected()); + } + scopes.insert((catalog, schema)); + } + if let Some(scope) = resolve_statement_database_scope(stmt, query_ctx)? { + scopes.insert(scope); + } + if scopes.is_empty() + && !matches!( + stmt, + Statement::Query(_) + | Statement::Use(_) + | Statement::SetVariables(_) + | Statement::ShowVariables(_) + | Statement::ShowDatabases(_) + | Statement::ShowCharset(_) + | Statement::ShowCollation(_) + | Statement::ShowStatus(_) + | Statement::ShowProcesslist(_) + | Statement::Kill(_) + ) + { + return Err(no_database_selected()); + } + Ok(scopes) +} + +fn resolve_statement_database_scope( + stmt: &Statement, + query_ctx: &QueryContextRef, +) -> Result> { + let database = match stmt { + Statement::ShowTables(show) => Some(show.database.as_deref()), + Statement::ShowTableStatus(show) => Some(show.database.as_deref()), + Statement::ShowViews(show) => Some(show.database.as_deref()), + _ => None, + }; + if let Some(database) = database { + let schema = database + .filter(|schema| !schema.is_empty()) + .ok_or_else(no_database_selected)?; + return Ok(Some(( + query_ctx.current_catalog().to_string(), + schema.to_string(), + ))); + } + let name = match stmt { + Statement::CreateDatabase(create) => &create.name, + Statement::AlterDatabase(alter) => alter.database_name(), + Statement::DropDatabase(drop) => drop.name(), + Statement::ShowCreateDatabase(show) => &show.database_name, + Statement::Copy(Copy::CopyDatabase(copy)) => match copy { + CopyDatabase::To(args) | CopyDatabase::From(args) => &args.database_name, + }, + _ => return Ok(None), + }; + let scope = match &name.0[..] { + [schema] => ( + query_ctx.current_catalog().to_string(), + schema.to_string_unquoted(), + ), + [catalog, schema] => (catalog.to_string_unquoted(), schema.to_string_unquoted()), + _ => { + return InvalidSqlSnafu { + err_msg: format!("Invalid database name: {name}"), + } + .fail(); + } + }; + if scope.1.is_empty() { + return Err(no_database_selected()); + } + Ok(Some(scope)) +} + +fn no_database_selected() -> Error { + ExternalSnafu.into_error(BoxedError::new( + session::SessionError::NoDatabaseSelected {}, + )) +} diff --git a/src/frontend/src/server.rs b/src/frontend/src/server.rs index 4cfb2c22f15..bc16a4cfa10 100644 --- a/src/frontend/src/server.rs +++ b/src/frontend/src/server.rs @@ -73,11 +73,7 @@ where pub fn new(opts: T, instance: Arc, plugins: Plugins) -> Self { let feopts = opts.clone().into(); instance.init_logical_batcher(&feopts); - // Create server request memory limiter for all server protocols - let server_memory_limiter = ServerMemoryLimiter::new( - feopts.max_in_flight_write_bytes.as_bytes(), - feopts.write_bytes_exhausted_policy, - ); + let server_memory_limiter = instance.server_memory_limiter().clone(); Self { opts, @@ -409,7 +405,13 @@ where if opts.postgres.enable { // Init PosgresSQL Server - let with_metric_engine = effective_prom_store_options(&opts).with_metric_engine; + let prom_store = effective_prom_store_options(&opts); + let pending_rows_batcher = opts + .logical_batcher_options() + .protocols + .contains(&BatchingProtocol::Prom) + .then(|| instance.logical_batcher().cloned()) + .flatten(); let opts = &opts.postgres; let pg_addr = parse_addr(&opts.addr)?; @@ -431,8 +433,15 @@ where Some(self.instance.process_manager().clone()), ) .with_influxdb_handler(instance.clone()) - .with_otlp_handler(instance.clone(), with_metric_engine) + .with_otlp_handler(instance.clone(), prom_store.with_metric_engine) .with_memory_limiter(self.server_memory_limiter.clone()) + .with_prom_remote_write( + instance.clone(), + Some(instance.clone()), + prom_store.with_metric_engine, + prom_store.prom_validation_mode, + pending_rows_batcher, + ) .with_batching_enabled(postgres_batching), ) as Box; @@ -579,6 +588,46 @@ mod tests { ); } + #[tokio::test] + async fn test_prom_logical_batcher_endpoint_selection() { + use crate::service_config::BatcherOptions; + + for (http_prom, postgres, metric_engine, expected) in [ + (false, false, true, false), + (true, false, true, true), + (false, true, true, true), + (true, true, true, true), + (false, true, false, false), + ] { + let mut options = FrontendOptions::default(); + options.prom_store.enable = http_prom; + options.postgres.enable = postgres; + options.prom_store.with_metric_engine = metric_engine; + options.pending_rows_batcher.logical_table = Some(BatcherOptions { + protocols: vec![BatchingProtocol::Prom], + pending_rows_flush_interval: Duration::from_millis(10), + ..Default::default() + }); + let meta_client = Arc::new( + MetaClientBuilder::new(0, Role::Frontend) + .enable_procedure() + .build(), + ); + let instance = Arc::new( + FrontendBuilder::new_test(&options, meta_client) + .try_build() + .await + .unwrap(), + ); + instance.init_logical_batcher(&options); + assert_eq!( + instance.logical_batcher().is_some(), + expected, + "http_prom={http_prom}, postgres={postgres}, metric_engine={metric_engine}" + ); + } + } + #[test] fn test_logical_batcher_http_timeout_and_prom_disable() { use crate::service_config::pending_rows_batcher::BatcherOptions; diff --git a/src/log-store/src/object_store_wal.rs b/src/log-store/src/object_store_wal.rs index 3ef6a166c8e..6a24a756475 100644 --- a/src/log-store/src/object_store_wal.rs +++ b/src/log-store/src/object_store_wal.rs @@ -55,9 +55,12 @@ //! or whose entries are all made obsolete, is released: a release record of //! the region goes into an object sealed at once, as the single entry of a //! segment of a reserved region id, so the object keeps format version 1. -//! Recovery reads those segments of the chain and seeds the obsolete -//! watermarks from them, so a store opened later collects the objects of a -//! region that is never opened again. +//! Closing a region can instead queue a release through a safe entry id; +//! queued records share the next sealed object, including a release-only +//! object while idle. Acceptance does not wait for persistence, so a crash +//! before sealing may retain extra history. Recovery reads release segments +//! of the chain and seeds the obsolete watermarks from them, so a store +//! opened later collects the objects of a region that is never opened again. //! //! Recovery lists the objects, reads and verifies only the header, trailer and //! footer of each, and indexes the footers of the chain in sequence order to diff --git a/src/log-store/src/object_store_wal/index.rs b/src/log-store/src/object_store_wal/index.rs index 7063e1e0449..06217f79ba7 100644 --- a/src/log-store/src/object_store_wal/index.rs +++ b/src/log-store/src/object_store_wal/index.rs @@ -19,7 +19,7 @@ use std::collections::BTreeMap; use std::ops::Range; use bytes::Bytes; -use futures::{Stream, TryStreamExt}; +use futures::{FutureExt, Stream, TryStreamExt}; use snafu::{IntoError, OptionExt, ensure}; use crate::error::{CorruptedWalObjectSnafu, Error, InvalidWalObjectSnafu, Result}; @@ -124,22 +124,42 @@ pub(crate) fn chain_from( /// Fetches and verifies the headers and footers of `objects` as they are /// listed, up to `concurrency` objects at a time, and returns them ordered by -/// object sequence whatever the order the fetches complete in. An object the -/// object store no longer has is returned as vanished; a failed listing or -/// any other failure abandons the remaining fetches. +/// object sequence whatever the order the fetches complete in, see +/// [`fetch_listed`]. pub(crate) async fn fetch_footers( io: &dyn WalObjectRead, objects: impl Stream>, concurrency: usize, ) -> Result { + let (mut objects, vanished) = fetch_listed(objects, concurrency, |object| { + fetch_footer_with_releases(io, object) + }) + .await?; + objects.sort_unstable_by_key(|fetched| fetched.object.object_seq); + Ok(FetchedObjects { objects, vanished }) +} + +/// Fetches `objects` as they are listed with `fetch`, up to `concurrency` +/// objects at a time, and returns what the fetches returned, in the order +/// they complete, and the listed objects the object store no longer had, with +/// the error the fetch met. A failed listing or any other failure abandons the +/// remaining fetches. +pub(crate) async fn fetch_listed( + objects: impl Stream>, + concurrency: usize, + fetch: impl Fn(ListedObject) -> F, +) -> Result<(Vec, Vec<(u64, Error)>)> +where + F: Future>, +{ let outcomes = objects - .map_ok(|object| async move { + .map_ok(|object| { let object_seq = object.object_seq; - match fetch_footer_with_releases(io, object).await { + fetch(object).map(move |fetched| match fetched { Ok(fetched) => Ok(Ok(fetched)), Err(error) if is_not_found(&error) => Ok(Err((object_seq, error))), Err(error) => Err(error), - } + }) }) .try_buffer_unordered(concurrency) .try_collect::>() @@ -152,12 +172,33 @@ pub(crate) async fn fetch_footers( Err(gone) => vanished.push(gone), } } - objects.sort_unstable_by_key(|fetched| fetched.object.object_seq); - Ok(FetchedObjects { objects, vanished }) + Ok((objects, vanished)) } +/// Fails with the error its fetch met if an object vanished at or above +/// `boundary`, below which a caller follows no link to a predecessor. A +/// vanished object below it is left out: the only object there a chain can +/// hold is its tip, whose absence the caller reports. An object at or above +/// it, on the chain or off it, may belong to the chain or decide which chain +/// is read or where it stops. +pub(crate) fn ensure_vanished_below(vanished: Vec<(u64, Error)>, boundary: u64) -> Result<()> { + match vanished + .into_iter() + .find(|(object_seq, _)| *object_seq >= boundary) + { + Some((_, error)) => Err(error), + None => Ok(()), + } +} + +/// Returns whether `error` reports an object the object store does not have, +/// also when a load shared by several callers wraps the error. pub(crate) fn is_not_found(error: &Error) -> bool { - matches!(error, Error::WalObjectStore { error, .. } if error.kind() == object_store::ErrorKind::NotFound) + match error { + Error::WalObjectStore { error, .. } => error.kind() == object_store::ErrorKind::NotFound, + Error::ObjectStoreWal { source, .. } => is_not_found(source), + _ => false, + } } /// Reads and verifies the header and footer without copying release records. diff --git a/src/log-store/src/object_store_wal/reader.rs b/src/log-store/src/object_store_wal/reader.rs index 4f5a1a97469..6d60bdd6e89 100644 --- a/src/log-store/src/object_store_wal/reader.rs +++ b/src/log-store/src/object_store_wal/reader.rs @@ -19,9 +19,10 @@ use std::sync::{Arc, Mutex, PoisonError, Weak}; use async_stream::try_stream; use common_cache::Cache; -use futures::{StreamExt, TryStreamExt}; +use futures::StreamExt; use object_store::ObjectStore; use snafu::{OptionExt, ResultExt, ensure}; +pub use store_api::logstore::WalReadUpperBound; use store_api::logstore::entry::{Entry, NaiveEntry}; use store_api::logstore::provider::Provider; use store_api::logstore::{EntryId, SendableEntryStream}; @@ -46,24 +47,6 @@ use crate::object_store_wal::prefetch; /// keeps its existing, separate concurrency setting. const FOOTER_CONCURRENCY: usize = 16; -/// Upper bound of a remote WAL recovery range. A reported final entry must be -/// present, whereas a LIST boundary need not contain entries of this region. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum WalReadUpperBound { - /// The durable last entry reported by the source on downgrade. - LastEntry(EntryId), - /// A conservative bound obtained by listing the source prefix. - Listed(EntryId), -} - -impl WalReadUpperBound { - fn entry_id(self) -> EntryId { - match self { - Self::LastEntry(id) | Self::Listed(id) => id, - } - } -} - /// A normalized WAL node prefix: `/datanodes//epochs/`. /// Construct it from a persisted source prefix or /// [`ObjectStoreWalConfig::node_prefix`](common_wal::config::object_store::ObjectStoreWalConfig::node_prefix). @@ -230,6 +213,14 @@ impl ObjectStoreWalReader { /// call (including a retry) starts a new LIST; cached indexes never replace /// that LIST. Previously indexed, still-listed immutable objects are reused. /// + /// The store writing the prefix may collect listed objects before their + /// indexes are read. An object gone below the sequence where the read stops + /// following links to predecessors (a reclaim boundary, or the object of + /// `after` when it is higher in the `LastEntry` mode) is skipped, as + /// startup recovery skips it, unless it holds the reported entry. One gone + /// at or above it, on the chain or not, fails the read; a retry lists the + /// prefix again. + /// /// Errors can occur after entries have been yielded. The caller must finish /// the stream successfully before publishing recovery progress. In the /// `LastEntry` mode success also proves that the reported entry was read. @@ -319,8 +310,10 @@ impl ObjectStoreWalReader { } ); } - let objects = futures::stream::iter(listed) - .map(|object| async move { + let (objects, vanished) = index::fetch_listed( + futures::stream::iter(listed.into_iter().map(Ok)), + FOOTER_CONCURRENCY, + |object| async move { let seq = object.object_seq; // Include listing metadata in the key so a load for a changed // object cannot join an older in-flight load of the same sequence. @@ -344,10 +337,10 @@ impl ObjectStoreWalReader { .await .context(ObjectStoreWalSnafu)?; Ok::<_, Error>((seq, cached)) - }) - .buffer_unordered(FOOTER_CONCURRENCY) - .try_collect::>() - .await?; + }, + ) + .await?; + let objects = objects.into_iter().collect::>(); let headers = objects .iter() .map(|(seq, object)| (*seq, &object.header)) @@ -362,6 +355,7 @@ impl ObjectStoreWalReader { .max() .unwrap_or(0) .max(start); + index::ensure_vanished_below(vanished, boundary)?; index::chain_from(&headers, last >> POSITION_BITS, boundary).with_context(|| { CorruptedWalObjectSnafu { reason: format!( @@ -373,6 +367,8 @@ impl ObjectStoreWalReader { } WalReadUpperBound::Listed(_) => { let chain = index::select_chain(&headers); + let boundary = chain.last().map_or(0, |tip| headers[tip].reclaim_boundary); + index::ensure_vanished_below(vanished, boundary)?; let tip = chain.last().with_context(|| CorruptedWalObjectSnafu { reason: format!("no complete WAL chain in {}", self.prefix()), })?; diff --git a/src/log-store/src/object_store_wal/reader/tests.rs b/src/log-store/src/object_store_wal/reader/tests.rs index 2034784dd90..425e0ef7a00 100644 --- a/src/log-store/src/object_store_wal/reader/tests.rs +++ b/src/log-store/src/object_store_wal/reader/tests.rs @@ -19,6 +19,7 @@ use bytes::Bytes; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; use common_wal::config::object_store::ObjectStoreWalConfig; +use futures::TryStreamExt; use object_store::services::Memory; use store_api::logstore::LogStore; @@ -68,6 +69,9 @@ struct ObservedIo { io: ObjectStoreIo, gets: Mutex>, lists: Mutex>, + /// Objects deleted after every `list_from`, as a store collecting them + /// before the reader reads their indexes would. + vanishing: Mutex>, gate: Option>, active: AtomicUsize, peak: AtomicUsize, @@ -112,7 +116,12 @@ impl WalObjectRead for ObservedIo { async fn list_from(&self, seq: u64) -> Result> { self.lists.lock().unwrap().push(seq); - self.io.list_from(seq).await + let listed = self.io.list_from(seq).await?; + let vanishing = self.vanishing.lock().unwrap().clone(); + for seq in vanishing { + self.io.delete(seq).await?; + } + Ok(listed) } fn object_path(&self, seq: u64) -> String { @@ -133,6 +142,7 @@ fn observed(io: ObjectStoreIo, gate: Option>) -> Arc io, gets: Mutex::new(BTreeMap::new()), lists: Mutex::new(Vec::new()), + vanishing: Mutex::new(Vec::new()), gate, active: AtomicUsize::new(0), peak: AtomicUsize::new(0), @@ -876,3 +886,87 @@ async fn test_changed_listing_metadata_refetches_index() { assert_eq!(catalog.region_max_entry_id(region()), Some(entry_id(1, 2))); assert_eq!(observed.gets.lock().unwrap()[&1], before + 1); } + +/// Writes the chain 0..=3, whose objects 2 and 3 carry reclaim boundary 2. +async fn put_reclaimed_chain(io: &ObjectStoreIo) { + put(io, 0, 1, None, 0, &[]).await; + put(io, 1, 1, Some((0, 1)), 0, &[record(1, 1)]).await; + put(io, 2, 1, Some((1, 1)), 2, &[record(2, 1)]).await; + put(io, 3, 1, Some((2, 1)), 2, &[record(3, 1)]).await; +} + +fn both_bounds() -> [WalReadUpperBound; 2] { + [ + WalReadUpperBound::LastEntry(entry_id(3, 1)), + WalReadUpperBound::Listed(entry_id(3, POSITION_LIMIT - 1)), + ] +} + +#[tokio::test] +async fn test_objects_vanished_below_reclaim_boundary_are_skipped() { + for upper in both_bounds() { + let (_, io, observed, reader) = fixture(); + put_reclaimed_chain(&io).await; + *observed.vanishing.lock().unwrap() = vec![0, 1]; + assert_eq!( + collect(&reader, entry_id(1, 1), upper).await.unwrap(), + expected(&[record(2, 1), record(3, 1)]), + "{upper:?}" + ); + } +} + +#[tokio::test] +async fn test_chain_object_vanished_at_reclaim_boundary_fails() { + for upper in both_bounds() { + let (_, io, observed, reader) = fixture(); + put_reclaimed_chain(&io).await; + // Without object 2 a listed read would select the complete chain + // ending at object 1. Its epoch is the newest and its entries end at + // `after`, so unless the vanished object 2 fails the read, it succeeds + // with no entries. + *observed.vanishing.lock().unwrap() = vec![2]; + let error = collect(&reader, entry_id(1, 1), upper).await.unwrap_err(); + assert!(index::is_not_found(&error), "{upper:?}: {error:?}"); + } +} + +#[tokio::test] +async fn test_unreadable_object_below_reclaim_boundary_fails() { + for upper in both_bounds() { + let (store, io, _, reader) = fixture(); + put_reclaimed_chain(&io).await; + // Object 1 is listed in both modes and lies below boundary 2, but a + // corrupted header is not a vanished object. + let mut bytes = io.get(1).await.unwrap().to_vec(); + bytes[0] ^= 1; + store.write(&io.object_path(1), bytes).await.unwrap(); + let error = collect(&reader, entry_id(1, 1), upper).await.unwrap_err(); + let Error::ObjectStoreWal { source, .. } = &error else { + panic!("{upper:?}: {error:?}"); + }; + assert!( + matches!(source.as_ref(), Error::InvalidWalObject { path, .. } if *path == io.object_path(1)), + "{upper:?}: {error:?}" + ); + } +} + +#[tokio::test] +async fn test_listed_range_fails_when_later_epoch_object_vanishes() { + let (_, io, observed, reader) = fixture(); + put(&io, 0, 1, None, 0, &[]).await; + put(&io, 1, 1, Some((0, 1)), 0, &[record(1, 1)]).await; + // The newer epoch's object is off the selected chain. Skipping it would + // hide its epoch from the comparison. + put(&io, 5, 4, Some((3, 4)), 0, &[]).await; + *observed.vanishing.lock().unwrap() = vec![5]; + let error = collect( + &reader, + 0, + WalReadUpperBound::Listed(entry_id(5, POSITION_LIMIT - 1)), + ) + .await + .unwrap_err(); + assert!(index::is_not_found(&error), "{error:?}"); +} diff --git a/src/log-store/src/object_store_wal/store.rs b/src/log-store/src/object_store_wal/store.rs index 6da8721891d..46e76afdec5 100644 --- a/src/log-store/src/object_store_wal/store.rs +++ b/src/log-store/src/object_store_wal/store.rs @@ -66,10 +66,14 @@ use crate::object_store_wal::format::{ ChainLink, EncodedObject, FooterEntry, HEADER_LEN, Header, RELEASE_REGION_ID, Record, Release, decode_header, decode_releases, decode_segment, encode_object, encode_releases, object_len, }; -use crate::object_store_wal::index::{FetchedObject, FetchedObjects, fetch_footers, select_chain}; +use crate::object_store_wal::index::{ + FetchedObject, FetchedObjects, ensure_vanished_below, fetch_footers, select_chain, +}; use crate::object_store_wal::io::{ObjectStoreIo, PutResult, WalObjectIo}; use crate::object_store_wal::prefetch::READ_PREFETCH_BYTES; -use crate::object_store_wal::{history, prefetch}; +use crate::object_store_wal::{ + NodeWalPrefix, ObjectStoreWalReaders, WalReadUpperBound, history, prefetch, +}; const COMMAND_BUFFER: usize = 1024; /// Appends queued for the actor; a caller beyond them waits to send. @@ -77,18 +81,14 @@ const APPEND_BUFFER: usize = 16; const MIN_FLUSH_INTERVAL: Duration = Duration::from_millis(10); const MAX_IN_FLIGHT_CREATES: usize = 4; const MAX_IN_FLIGHT_DELETES: usize = 4; -/// Delay before a create that failed transiently is attempted again in the -/// `enqueued` acknowledgement mode, where no caller is left to retry it. +/// Delay before retrying a transiently failed enqueued create, and the initial +/// delay for retained releases after a durable rollback. const CREATE_RETRY_DELAY: Duration = Duration::from_millis(100); -/// Longest delay from a failed attempt of a backlog that the object store -/// refuses with an error that requires intervention, see [`Failing`], to the -/// next attempt. -/// The delay doubles from [`CREATE_RETRY_DELAY`] after every attempt. At the -/// cap a dead bucket costs four requests a minute, a conditional create and -/// the read that reconciles it every 30 seconds; a longer cap would keep -/// appends rejected longer after the cause is fixed. +/// Maximum backoff for an enqueued backlog requiring intervention or a durable +/// rollback retaining releases. Both double from [`CREATE_RETRY_DELAY`]. +/// The cap bounds recovery delay after storage becomes available. const MAX_FAILING_RETRY_DELAY: Duration = Duration::from_secs(30); -/// How often a store logs that the object store still refuses its backlog. +/// Minimum interval between repeated warnings about upload failures or retained releases. const FAILING_LOG_INTERVAL: Duration = Duration::from_secs(60); /// Number of sealed batches that may wait to be created or indexed. One /// admission can seal two batches, the open batch before an append that would @@ -122,10 +122,17 @@ const RECOVERY_CONCURRENCY: usize = 64; /// error that requires intervention. A segment that a read cannot /// decode is handled as `on_corrupted_segment` says. Objects below the reclaim /// boundary of the last indexed object are deleted in the background. A -/// release of a region seals an object with a release record of the region at +/// permanent release (`release_region` or `obsolete_all`) seals an object at /// once and returns, in both modes, once that object is durable and indexed. +/// Bounded releases acknowledge admission and coalesce until a batch is sealed. pub struct ObjectStoreLogStore { prefix: String, + /// The configuration and generation the prefix comes from. The prefix of every node + /// derives from them, so all nodes under the root share this store's generation. + config: ObjectStoreWalConfig, + generation: u64, + /// Reads the WAL of any node, this node's included, see [`LogStore::read_node`]. + node_readers: ObjectStoreWalReaders, /// The storage the prefix is read from, as configured, for messages. storage: String, ack_mode: AckMode, @@ -278,15 +285,18 @@ impl ObjectStoreLogStore { generation: u64, ) -> Result> { let prefix = config.node_prefix(node_id, generation); - let io = ObjectStoreIo::new(object_store, &prefix)?; + let io = ObjectStoreIo::new(object_store.clone(), &prefix)?; io.probe_conditional_create().await?; - Self::open(Arc::new(io), config, prefix).await + let node_readers = ObjectStoreWalReaders::new(object_store); + Self::open(Arc::new(io), config, prefix, generation, node_readers).await } async fn open( io: Arc, config: &ObjectStoreWalConfig, prefix: String, + generation: u64, + node_readers: ObjectStoreWalReaders, ) -> Result> { ensure!( config.flush_interval >= MIN_FLUSH_INTERVAL, @@ -428,7 +438,9 @@ impl ObjectStoreLogStore { reclaim_boundary: writer_reclaim_boundary, durable_reclaim_boundary: reclaim_boundary, unindexed: unindexed.into_iter().chain(moved_past).collect(), - failed_releases: BTreeMap::new(), + pending_releases: BTreeMap::new(), + release_retry: RetryBackoff::default(), + release_retry_logged_at: None, creating: HashSet::new(), deleting: deleting.clone(), deletes: FuturesUnordered::new(), @@ -456,6 +468,9 @@ impl ObjectStoreLogStore { }; Ok(Arc::new(Self { prefix, + config: config.clone(), + generation, + node_readers, storage, ack_mode: config.ack_mode, on_corrupted_segment: config.on_corrupted_segment, @@ -1055,8 +1070,9 @@ impl LogStore for ObjectStoreLogStore { /// ids under the next sequence the floor cannot move, and the call fails /// with [`Error::WalObjectSequenceUnsettled`] and records neither. The /// watermark is kept in memory; the caller re-establishes it after a - /// restart unless it released the region, see - /// [`release_region`](LogStore::release_region). In the `enqueued` mode + /// restart unless it queued a bounded release on close, see + /// [`release_entries`](LogStore::release_entries), or permanently released + /// the region, see [`release_region`](LogStore::release_region). In the `enqueued` mode /// an `entry_id` this store handed out /// needs no floor: it is never handed out again while the store runs. async fn obsolete( @@ -1113,6 +1129,38 @@ impl LogStore for ObjectStoreLogStore { self.release(region_id, false).await } + async fn release_entries( + &self, + provider: &Provider, + region_id: RegionId, + entry_id: EntryId, + ) -> Result<()> { + self.check_region(provider, region_id)?; + // Shutdown stops the WAL before closing regions. These later hints + // cannot be persisted and safely retain extra history instead. + if self.stopped.load(Ordering::Acquire) { + return Ok(()); + } + let (response, receiver) = oneshot::channel(); + let answered = match self + .command_tx + .send(Command::ReleaseEntries { + region_id, + entry_id, + response, + }) + .await + { + Ok(()) => receiver.await.ok(), + Err(_) => None, + }; + match answered { + Some(result) => result, + None if self.stopped.load(Ordering::Acquire) => Ok(()), + None => ObjectStoreWalStoppedSnafu.fail(), + } + } + fn entry( &self, data: Vec, @@ -1199,6 +1247,22 @@ impl LogStore for ObjectStoreLogStore { started_regions.insert(region_id); Ok(()) } + + async fn read_node( + &self, + node_id: u64, + region_id: RegionId, + after: EntryId, + upper: WalReadUpperBound, + ) -> Result>> { + let prefix = NodeWalPrefix::try_new(&self.config.node_prefix(node_id, self.generation))?; + let reader = self.node_readers.reader(&prefix)?; + let entries = reader.read(region_id, after, upper).await?; + // The stream holds the reader, so concurrent reads of the node share its indexes. + Ok(Some(Box::pin(entries.inspect(move |_| { + let _ = &reader; + })))) + } } type AppendResponse = oneshot::Sender>; @@ -1230,6 +1294,13 @@ enum Command { full: bool, response: oneshot::Sender>, }, + /// Answered once the actor has accepted the bounded release for its next + /// batch. Unlike `Release`, this does not wait for persistence. + ReleaseEntries { + region_id: RegionId, + entry_id: EntryId, + response: oneshot::Sender>, + }, /// Answered once the store keeps its start object, or once the delete /// of the start object it gave up settled. Withdraw { response: oneshot::Sender<()> }, @@ -1242,6 +1313,11 @@ enum Command { }, #[cfg(any(test, feature = "testing"))] Crash { response: oneshot::Sender<()> }, + /// Runs test assertions and controls deadlines on the actor's own task. + #[cfg(test)] + Inspect { + inspect: Box, + }, } /// An append waiting for the object that holds its entries. @@ -1322,6 +1398,34 @@ enum Withdrawal { Settled, } +/// Exponential retry timing shared by failed uploads and retained releases. +struct RetryBackoff { + /// Delay to use when scheduling the next failed attempt. + delay: Duration, + /// Scheduled retry deadline, absent until scheduled or after a probe starts/reset. + at: Option, +} + +impl Default for RetryBackoff { + fn default() -> Self { + Self { + delay: CREATE_RETRY_DELAY, + at: None, + } + } +} + +impl RetryBackoff { + fn schedule(&mut self, now: Instant) { + self.at = Some(now + self.delay); + self.delay = (self.delay * 2).min(MAX_FAILING_RETRY_DELAY); + } + + fn is_waiting(&self, now: Instant) -> bool { + self.at.is_some_and(|at| now < at) + } +} + /// The state of an `enqueued` store whose backlog the object store refuses /// with an error that requires intervention, see [`requires_intervention`], /// such as denied @@ -1333,16 +1437,15 @@ enum Withdrawal { /// flight, after a delay that doubles from [`CREATE_RETRY_DELAY`] up to /// [`MAX_FAILING_RETRY_DELAY`]. Appends are rejected at once with a retryable /// error that carries the last error: they take no entry id and never wait -/// behind the backlog. Releases fail with it too, and their records wait for -/// a create. The first create that succeeds ends the state. +/// behind the backlog. Permanent releases fail with it too, and their records +/// wait for a create. Bounded releases still acknowledge admission and coalesce. +/// The first create that succeeds ends the state. struct Failing { /// The last error a create met, which rejected appends carry. error: Arc, since: Instant, - /// The delay before the next attempt is due once nothing is in flight. - delay: Duration, - /// When the next attempt starts, `None` until it is scheduled. - probe_at: Option, + /// Retry timing for the lowest pending create. + retry: RetryBackoff, failed_creates: u64, rejected_appends: u64, logged_at: Instant, @@ -1352,8 +1455,7 @@ impl Failing { /// Schedules the next attempt after the current delay and doubles the /// delay for the one after it. fn schedule(&mut self) { - self.probe_at = Some(Instant::now() + self.delay); - self.delay = (self.delay * 2).min(MAX_FAILING_RETRY_DELAY); + self.retry.schedule(Instant::now()); } /// Counts an admission the store refuses and returns the error it @@ -1379,10 +1481,10 @@ impl Failing { /// | Situation | Sequence | Waiters | Store | /// | --- | --- | --- | --- | /// | created, or identical retry | advances | acknowledged in order | healthy | -/// | transient error, `durable` mode | never reused: a create that reported an error may have written its object | every batch that is not indexed fails, created ones included; retries are assigned new ids | healthy: the next batch links to the last indexed object, so no chain reaches a failed batch | -/// | transient error, `enqueued` mode | unchanged | already acknowledged, except a release, which waits for the repeat | healthy: the create is repeated with the same bytes after [`CREATE_RETRY_DELAY`], which an identical retry accepts | -/// | error that requires intervention, `enqueued` mode | unchanged | already acknowledged, except a release, which fails while its record stays in the backlog; later appends and releases are rejected with a retryable error | failing, see [`Failing`]: the lowest pending create is repeated alone with backoff, and the first create that succeeds ends it | -/// | object store error, `enqueued` mode after `stop` began | never reused | already acknowledged, except a release, which fails with the stopped error | the backlog is dropped and `stop` reports the error | +/// | transient error, `durable` mode | never reused: a create that reported an error may have written its object | every batch that is not indexed fails, created ones included; retries are assigned new ids | healthy: the next batch links to the last indexed object, so no chain reaches a failed batch; retained releases back off before a release-only retry | +/// | transient error, `enqueued` mode | unchanged | already acknowledged, except a permanent release, which waits for the repeat | healthy: the create is repeated with the same bytes after [`CREATE_RETRY_DELAY`], which an identical retry accepts | +/// | error that requires intervention, `enqueued` mode | unchanged | already acknowledged, except a permanent release, which fails while its record stays in the backlog; later appends and permanent releases are rejected with a retryable error | failing, see [`Failing`]: the lowest pending create is repeated alone with backoff, and the first create that succeeds ends it | +/// | object store error, `enqueued` mode after `stop` began | never reused | already acknowledged, except a permanent release, which fails with the stopped error | the backlog is dropped and `stop` reports the error | /// | an object of an earlier epoch holds the sequence, `durable` mode | never reused | as for a transient error | healthy: that object can never be on the chain | /// | an object of the same or a later epoch holds the sequence, any conflicting object in the `enqueued` mode, encoding or catalog error | unchanged | every batch that is not indexed fails | poisoned | /// | created at the last representable sequence | cannot advance | acknowledged | poisoned: no later batch can be allocated a sequence | @@ -1391,8 +1493,8 @@ impl Failing { /// outcome is ignored: whatever it stored is off the chain. After `stop` began /// nothing is admitted and no create starts, except that the `enqueued` mode /// uploads its backlog and the `durable` mode creates one object with the -/// release records of failed batches; creates in flight run to completion and -/// acknowledge if they succeed. +/// pending bounded releases and release records of failed batches. Creates in +/// flight run to completion and acknowledge if they succeed. /// /// Appends arrive on their own channel. The actor holds one back while the /// sealed batches leave no room for the two an admission can seal, or at a @@ -1465,11 +1567,14 @@ struct Actor { /// Sequences of objects that may be present but are not indexed: the /// objects recovery listed off the chain and the batches that failed. unindexed: BTreeSet, - /// Release records of batches that failed, which the next sealed batch - /// carries again, so a release whose call failed still becomes durable - /// while the store runs. A timer tick seals them when no entry is - /// admitted. - failed_releases: BTreeMap, + /// Accepted release records and records from failed batches, coalesced by + /// region for the next sealed batch. Timer ticks seal them when no entries + /// arrive, after any durable rollback backoff; shutdown drains them in both modes. + pending_releases: BTreeMap, + /// Backoff for release-only sealing after a durable rollback retains records. + release_retry: RetryBackoff, + /// Last warning about retained release records after a durable rollback. + release_retry_logged_at: Option, /// Sequences whose create is in flight. creating: HashSet, deleting: Arc, @@ -1499,7 +1604,7 @@ impl Actor { interval.tick().await; loop { - let probe_at = self.failing.as_ref().and_then(|failing| failing.probe_at); + let probe_at = self.failing.as_ref().and_then(|failing| failing.retry.at); tokio::select! { _ = interval.tick() => { // Nothing starts after stop began; the `enqueued` mode @@ -1530,6 +1635,10 @@ impl Actor { Some(Command::Release { region_id, full, response }) => { self.handle_release(region_id, full, response); } + Some(Command::ReleaseEntries { region_id, entry_id, response }) => { + let result = self.handle_release_entries(region_id, entry_id); + let _ = response.send(result); + } Some(Command::Withdraw { response }) => { self.handle_withdraw(response); } @@ -1546,6 +1655,8 @@ impl Actor { let _ = response.send(()); return; } + #[cfg(test)] + Some(Command::Inspect { inspect }) => inspect(&mut self), // Every sender is gone: the store was dropped without // `stop`. The creates in flight are dropped with the actor. None => return, @@ -1708,18 +1819,27 @@ impl Actor { self.seal(Vec::new()) } - /// Seals the open batch together with `releases` and the release records - /// of failed batches as the object `next_object_seq`, even when the open + /// Seals the open batch together with `releases` and pending release + /// records as the object `next_object_seq`, even when the open /// batch holds no entry, and starts its create when a slot is free. /// Returns whether a batch was sealed. /// - /// A failing store keeps the records of failed batches coalesced instead - /// of sealing them alone behind its backlog on every tick, until `stop` - /// seals them for its last attempt of the backlog. + /// A failing store or one at a backlog threshold keeps pending releases + /// coalesced instead of sealing them alone on every tick. Stop seals them + /// for its last attempt even when the backlog is full. Explicit permanent + /// releases still seal immediately, as their callers wait for persistence. + /// A durable rollback retaining releases also delays release-only sealing. + /// Ordinary entries bypass this delay; indexing a batch carrying releases + /// resets it. fn seal(&mut self, releases: Vec) -> bool { if self.open_batch.is_empty() && releases.is_empty() - && (self.failed_releases.is_empty() || (self.failing.is_some() && self.stop.is_empty())) + && (self.pending_releases.is_empty() + || (self.stop.is_empty() + && (self.failing.is_some() + || self.release_retry.is_waiting(Instant::now()) + || self.sealed.len() >= MAX_SEALED_BATCHES + || self.backlog_at_threshold()))) { return false; } @@ -1752,7 +1872,7 @@ impl Actor { ), reclaim_boundary: self.reclaim_boundary, }; - let mut records = std::mem::take(&mut self.failed_releases); + let mut records = std::mem::take(&mut self.pending_releases); for release in releases { let entry_id = records.entry(release.region_id).or_insert(release.entry_id); *entry_id = (*entry_id).max(release.entry_id); @@ -1820,7 +1940,7 @@ impl Actor { if !self.is_stopped() && let Some(failing) = &mut self.failing { - if self.creates.is_empty() && failing.probe_at.is_none() { + if self.creates.is_empty() && failing.retry.at.is_none() { failing.schedule(); } return; @@ -1850,7 +1970,7 @@ impl Actor { let Some(failing) = &mut self.failing else { return; }; - failing.probe_at = None; + failing.retry.at = None; if self.is_stopped() { return; } @@ -1994,7 +2114,7 @@ impl Actor { failing.since.elapsed(), failing.failed_creates, failing.rejected_appends, - failing.delay + failing.retry.delay ); } } @@ -2012,8 +2132,7 @@ impl Actor { let failing = self.failing.insert(Failing { error: Arc::new(error), since: now, - delay: CREATE_RETRY_DELAY, - probe_at: None, + retry: RetryBackoff::default(), failed_creates: 1, rejected_appends: 0, logged_at: now, @@ -2096,6 +2215,10 @@ impl Actor { for waiter in batch.release_waiters { let _ = waiter.send(Ok(())); } + if !batch.releases.is_empty() { + self.release_retry = RetryBackoff::default(); + self.release_retry_logged_at = None; + } #[cfg(any(test, feature = "testing"))] for waiter in batch.seal_waiters { let _ = waiter.send(Ok(())); @@ -2147,6 +2270,8 @@ impl Actor { /// batch. Waiters of a store that was stopped meanwhile learn that /// instead of the I/O error, like every other entry that never became /// durable. + /// Retained releases, including those from data batches, back off before + /// the next release-only seal. Failures are logged at a limited rate. fn roll_back(&mut self, error: Arc) { let stopped = self.is_stopped(); let failure = || { @@ -2161,6 +2286,22 @@ impl Actor { self.keep_releases(&mut batch); batch.fail(failure); } + if self.ack_mode == AckMode::Durable && !stopped && !self.pending_releases.is_empty() { + let now = Instant::now(); + if self + .release_retry_logged_at + .is_none_or(|at| now.duration_since(at) >= FAILING_LOG_INTERVAL) + { + warn!( + error; + "Object store WAL retains {} release records after a failed create; next release-only retry in {:?}", + self.pending_releases.len(), + self.release_retry.delay + ); + self.release_retry_logged_at = Some(now); + } + self.release_retry.schedule(now); + } self.reset_open_batch(); // The stalled append was never admitted; the sealed batches are gone, // so `settle` admits it, or refuses it once stop began. @@ -2169,7 +2310,7 @@ impl Actor { // its caller has arrived yet or not. So it does release records once // stop began, which no later create carries. if self.ack_mode == AckMode::Enqueued - || (!self.stop.is_empty() && !self.failed_releases.is_empty()) + || (!self.stop.is_empty() && !self.pending_releases.is_empty()) { self.stop_error.get_or_insert(error); } @@ -2186,7 +2327,7 @@ impl Actor { fn keep_release(&mut self, release: Release) { let entry_id = self - .failed_releases + .pending_releases .entry(release.region_id) .or_insert(release.entry_id); *entry_id = (*entry_id).max(release.entry_id); @@ -2202,7 +2343,7 @@ impl Actor { fn poison(&mut self, error: Error) -> Arc { let error = set_terminal(&self.terminal_error, error); let stopped = self.is_stopped(); - let releases_lost = !self.failed_releases.is_empty() + let releases_lost = !self.pending_releases.is_empty() || self.sealed.iter().any(|batch| !batch.releases.is_empty()); let failure = || { if stopped { @@ -2326,7 +2467,7 @@ impl Actor { || !self.creates.is_empty() || self.stalled.is_some() || !self.unindexed.is_empty() - || !self.failed_releases.is_empty() + || !self.pending_releases.is_empty() { let _ = response.send(()); return; @@ -2388,6 +2529,42 @@ impl Actor { let _ = response.send(result); } + /// Accepts a bounded release without sealing an object or waiting for I/O. + /// Raising the sequence floor prevents a later reopen from assigning ids + /// covered by this record. Physical collection still uses the durable + /// reclaim boundary, so an unpersisted record only loses reclamation progress. + fn handle_release_entries(&mut self, region_id: RegionId, entry_id: EntryId) -> Result<()> { + if self.is_stopped() { + return Ok(()); + } + self.raise_sequence_floor(region_id, entry_id)?; + // Like obsolete, preserve entries that are not durable at admission. + // Keep the original sequence floor so a reopen cannot reuse their ids. + let watermark = match self.ack_mode { + AckMode::Durable => entry_id, + AckMode::Enqueued => entry_id.min( + self.catalog + .read() + .unwrap_or_else(PoisonError::into_inner) + .region_max_entry_id(region_id) + .unwrap_or(0), + ), + }; + record_obsolete( + &self.obsolete_entry_ids, + &self.catalog, + region_id, + watermark, + ); + self.keep_release(Release { + region_id, + entry_id: watermark, + }); + self.update_reclaim_boundary(); + self.collect(); + Ok(()) + } + /// Moves the watermark of the region up to the release and seals the /// open batch with a release record of the region; `response` is /// answered once that object is durable and indexed. The batch carries @@ -2618,8 +2795,9 @@ impl Actor { /// Begins stopping. Nothing is admitted from now on and the append held /// back is refused; the `durable` mode drops the open batch and the /// batches whose create has not started, and seals the release records of - /// failed and dropped batches once more; the `enqueued` mode seals its - /// backlog and uploads it, a failing store included. Stop is answered by + /// failed and dropped batches together with pending bounded releases once + /// more; the `enqueued` mode seals its backlog and uploads it, a failing + /// store included. Stop is answered by /// [`finish_stop`](Self::finish_stop) once nothing is in flight. fn handle_stop(&mut self, response: oneshot::Sender>) { self.stop.push(response); @@ -3061,12 +3239,7 @@ fn finish_recovery(fetched: FetchedObjects) -> Result { .context(WalObjectSequenceExhaustedSnafu { last_object_seq })?, }; let reclaim_boundary = tip.map_or(0, |tip| headers[&tip.object_seq].reclaim_boundary); - if let Some((_, error)) = vanished - .into_iter() - .find(|(object_seq, _)| *object_seq >= reclaim_boundary) - { - return Err(error); - } + ensure_vanished_below(vanished, reclaim_boundary)?; let chain = chain.into_iter().collect::>(); let mut catalog = ObjectCatalog::default(); @@ -3303,6 +3476,23 @@ mod tests { store.latest_entry_id(&provider(region_id)).unwrap() } + async fn inspect_actor( + store: &ObjectStoreLogStore, + inspect: impl FnOnce(&mut Actor) -> T + Send + 'static, + ) -> T { + let (response, receiver) = oneshot::channel(); + store + .command_tx + .send(Command::Inspect { + inspect: Box::new(move |actor| { + let _ = response.send(inspect(actor)); + }), + }) + .await + .unwrap(); + timeout(WAIT, receiver).await.unwrap().unwrap() + } + /// The id of the entry at `position` of its region in object `object_seq`. fn id(object_seq: u64, position: u64) -> EntryId { entry_id(object_seq, position) @@ -3408,9 +3598,16 @@ mod tests { io: Arc, config: &ObjectStoreWalConfig, ) -> Arc { - ObjectStoreLogStore::open(io, config, PREFIX.to_string()) - .await - .unwrap() + open_io(io, config).await.unwrap() + } + + /// Opens a store over `io` under [PREFIX], reading other nodes from an empty storage. + async fn open_io( + io: Arc, + config: &ObjectStoreWalConfig, + ) -> Result> { + let node_readers = ObjectStoreWalReaders::new(memory_store()); + ObjectStoreLogStore::open(io, config, PREFIX.to_string(), 0, node_readers).await } async fn open_parking_creates( @@ -4859,6 +5056,9 @@ mod tests { let (append_tx, _) = mpsc::channel(APPEND_BUFFER); let store = ObjectStoreLogStore { prefix: PREFIX.to_string(), + config: eager(), + generation: 0, + node_readers: ObjectStoreWalReaders::new(memory_store()), storage: String::new(), ack_mode: AckMode::Durable, on_corrupted_segment: CorruptedSegmentAction::Skip, @@ -5018,9 +5218,7 @@ mod tests { let (io, mut parked) = ParkedIo::over(object_store.clone()); let fetch = { let io = io.clone(); - tokio::spawn(async move { - ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await - }) + tokio::spawn(async move { open_io(io, &eager()).await }) }; let mut wave = Vec::new(); for _ in 0..RECOVERY_CONCURRENCY { @@ -5231,9 +5429,7 @@ mod tests { corrupt_object(&object_store, &path, |bytes| bytes[HEADER_LEN + 20] ^= 1).await; let (io, reads) = RecordingIo::over(object_store); let object = io.list().await.unwrap().remove(0); - let store = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()) - .await - .unwrap(); + let store = open_io(io, &eager()).await.unwrap(); assert_eq!(1, latest(&store, region(1))); let mut reads = reads.lock().unwrap().clone(); reads.sort_unstable(); @@ -5325,9 +5521,7 @@ mod tests { } } let (io, reads) = RecordingIo::over(object_store); - let store = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()) - .await - .unwrap(); + let store = open_io(io, &eager()).await.unwrap(); for number in [1, 2] { reads.lock().unwrap().clear(); let entries = read_entries(&store, region(number), 11).await; @@ -5725,9 +5919,7 @@ mod tests { let (io, mut parked) = ParkedIo::over(object_store); let mut opening = { let io = io.clone(); - tokio::spawn(async move { - ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await - }) + tokio::spawn(async move { open_io(io, &eager()).await }) }; let store = loop { tokio::select! { @@ -7465,7 +7657,7 @@ mod tests { inner: ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(), late: Mutex::new(Some((1, late))), }); - let result = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await; + let result = open_io(io, &eager()).await; if moves { let store = result.unwrap(); let start = decode_header(&fixtures.get(2).await.unwrap()).unwrap(); @@ -7512,9 +7704,7 @@ mod tests { inner: ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(), late: Mutex::new(Some((start_seq, same))), }); - let error = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()) - .await - .unwrap_err(); + let error = open_io(io, &eager()).await.unwrap_err(); assert!( matches!(&error, Error::UnconfirmedWalEpochStart { epoch: actual, path, .. } if *actual == epoch && path == &fixtures.object_path(start_seq)), @@ -7600,9 +7790,7 @@ mod tests { inner: ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(), lose_next: AtomicBool::new(true), }); - let error = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()) - .await - .unwrap_err(); + let error = open_io(io, &eager()).await.unwrap_err(); assert!( matches!(error, Error::WalObjectStore { .. }), "unexpected error: {error:?}" @@ -8030,18 +8218,17 @@ mod tests { let mut failing = Failing { error: Arc::new(ObjectStoreWalStoppedSnafu.build()), since: now, - delay: CREATE_RETRY_DELAY, - probe_at: None, + retry: RetryBackoff::default(), failed_creates: 1, rejected_appends: 0, logged_at: now, }; let mut delays = Vec::new(); for _ in 0..12 { - let delay = failing.delay; + let delay = failing.retry.delay; let scheduled_at = Instant::now(); failing.schedule(); - assert!(failing.probe_at.unwrap() >= scheduled_at + delay); + assert!(failing.retry.at.unwrap() >= scheduled_at + delay); delays.push(delay.as_millis()); } assert_eq!( @@ -9104,6 +9291,550 @@ mod tests { .copied() } + #[tokio::test] + async fn test_bounded_release_batches_and_preserves_later_entries() { + let _serialized = SERIALIZED.lock().await; + for config in [eager(), enqueued(eager())] { + let object_store = memory_store(); + let store = open(object_store.clone(), &config).await; + let (one, two) = (region(1), region(2)); + store + .append_batch(vec![entry(&store, one, "old")]) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + store + .append_batch(vec![entry(&store, one, "keep")]) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + store.wait_durable(&provider(one), id(2, 1)).await.unwrap(); + let before = collected_object_seqs(&store).await; + for (region_id, entry_id) in [(one, id(1, 1)), (one, 0), (two, 0)] { + timeout( + WAIT, + store.release_entries(&provider(region_id), region_id, entry_id), + ) + .await + .unwrap() + .unwrap(); + } + // Admission never seals an object. Both regions share one release-only batch. + assert_eq!(before, object_seqs(store.io.as_ref()).await); + store.seal_open_batch().await.unwrap(); + assert_eq!( + vec![release(one, id(1, 1)), release(two, 0)], + releases_of(store.io.as_ref(), 3).await + ); + store.stop().await.unwrap(); + let reopened = open(object_store, &config).await; + assert_eq!(Some(id(1, 1)), watermark(&reopened, one)); + assert_eq!( + expected_entries(one, &[(id(2, 1), "keep")]), + read_entries(&reopened, one, 0).await + ); + reopened.stop().await.unwrap(); + } + } + + #[tokio::test] + async fn test_bounded_release_preserves_entries_not_durable_at_admission() { + let _serialized = SERIALIZED.lock().await; + let object_store = memory_store(); + let config = enqueued(manual()); + let store = open(object_store.clone(), &config).await; + let region_id = region(1); + let admitted = append(&store, region_id, "keep") + .await + .unwrap() + .last_entry_ids[®ion_id]; + assert_eq!(id(1, 1), admitted); + assert_eq!(0, latest(&store, region_id)); + store + .release_entries(&provider(region_id), region_id, admitted) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + store.crash().await; + let reopened = open(object_store, &config).await; + assert_eq!( + expected_entries(region_id, &[(admitted, "keep")]), + read_entries(&reopened, region_id, 0).await + ); + reopened.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_bounded_release_timer_seals_without_appends() { + let _serialized = SERIALIZED.lock().await; + let store = open( + memory_store(), + &config(Duration::from_millis(100), u64::MAX), + ) + .await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + timeout(WAIT, async { + loop { + let indexed = store + .catalog + .read() + .unwrap() + .objects_in_order() + .any(|(seq, _)| seq == 1); + if indexed { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + assert_eq!( + vec![release(region_id, 0)], + releases_of(store.io.as_ref(), 1).await + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_bounded_release_fences_future_appends_and_ignores_after_stop() { + let _serialized = SERIALIZED.lock().await; + for (config, released) in [(eager(), id(10, 1)), (enqueued(manual()), 0)] { + let store = open(memory_store(), &config).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, id(10, 1)) + .await + .unwrap(); + store + .append_batch(vec![entry(&store, region_id, "new")]) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + assert_eq!(id(11, 1), latest(&store, region_id)); + assert_eq!( + vec![release(region_id, released)], + releases_of(store.io.as_ref(), 11).await + ); + assert_eq!( + expected_entries(region_id, &[(id(11, 1), "new")]), + read_entries(&store, region_id, 0).await + ); + store.stop().await.unwrap(); + let before = object_seqs(store.io.as_ref()).await; + store + .release_entries(&provider(region_id), region_id, EntryId::MAX) + .await + .unwrap(); + assert_eq!(before, object_seqs(store.io.as_ref()).await); + assert_eq!(Some(released), watermark(&store, region_id)); + } + } + + #[tokio::test] + async fn test_bounded_release_coalesces_behind_full_backlog() { + let _serialized = SERIALIZED.lock().await; + for ack_mode in [AckMode::Durable, AckMode::Enqueued] { + for byte_limit in [1, u64::MAX] { + let config = ObjectStoreWalConfig { + ack_mode, + max_unpersisted_bytes: ReadableSize(byte_limit), + ..manual() + }; + let (store, io, mut parked) = open_parking_creates(memory_store(), &config).await; + let before = METRIC_OBJECT_STORE_WAL_OBJECT_BYTES.get_sample_count(); + for _ in 0..2 * MAX_SEALED_BATCHES { + store + .release_entries(&provider(region(1)), region(1), 0) + .await + .unwrap(); + // Drive the timer's sealing path without waiting for parked I/O. + let (response, _) = oneshot::channel(); + store + .command_tx + .send(Command::Seal { response }) + .await + .unwrap(); + } + // Same-channel admission is a barrier after the final Seal and + // leaves a different region pending for the shutdown drain. + store + .release_entries(&provider(region(2)), region(2), 0) + .await + .unwrap(); + let expected = if byte_limit == 1 { + 1 + } else { + MAX_SEALED_BATCHES as u64 + }; + assert_eq!( + expected, + METRIC_OBJECT_STORE_WAL_OBJECT_BYTES.get_sample_count() - before + ); + assert_eq!(vec![0], object_seqs(io.as_ref()).await); + let mut stop = begin_spawned_stop(&store).await; + io.creates_parked.store(false, Ordering::SeqCst); + // Only already issued PUTs remain parked; later creates run normally. + timeout(WAIT, async { + loop { + tokio::select! { + operation = parked.recv() => { + let (_, release) = operation.unwrap(); + release.send(true).unwrap(); + } + result = &mut stop => break result.unwrap().unwrap(), + } + } + }) + .await + .unwrap(); + let recovered = recover(io.as_ref()).await.unwrap(); + assert_eq!(Some(&0), recovered.released_entry_ids.get(®ion(2))); + } + } + } + + #[test] + fn test_retry_backoff_deadlines_and_cap() { + let mut retry = RetryBackoff::default(); + let mut now = Instant::now(); + assert!(!retry.is_waiting(now)); + for millis in [ + 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 30000, 30000, + ] { + retry.schedule(now); + let deadline = now + Duration::from_millis(millis); + assert_eq!(Some(deadline), retry.at); + assert!(retry.is_waiting(deadline - Duration::from_nanos(1))); + assert!(!retry.is_waiting(deadline)); + now = deadline; + } + } + + #[tokio::test] + async fn test_durable_release_retries_back_off_after_create_failure() { + let _serialized = SERIALIZED.lock().await; + let (store, io, mut parked) = open_parking_creates(memory_store(), &manual()).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + // Manual sealing and actor-owned deadlines make this independent of + // executor delays. Each failure must retain the record and grow the + // delay; no upload may start while its deadline is in the future. + for (index, next_delay) in [200, 400, 800, 1600, 3200, 6400, 12800, 25600, 30000, 30000] + .into_iter() + .enumerate() + { + let seal = { + let store = store.clone(); + tokio::spawn(async move { store.seal_open_batch().await }) + }; + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(index as u64 + 1, seq); + create.send(false).unwrap(); + assert!(timeout(WAIT, seal).await.unwrap().unwrap().is_err()); + let next_seq = inspect_actor(&store, move |actor| { + assert_eq!(Duration::from_millis(next_delay), actor.release_retry.delay); + assert!(actor.release_retry.at.is_some()); + assert!(actor.release_retry_logged_at.is_some()); + assert_eq!(Some(&0), actor.pending_releases.get(®ion_id)); + actor.release_retry.at = Some(Instant::now() + Duration::from_secs(3600)); + actor.next_object_seq + }) + .await; + store.seal_open_batch().await.unwrap(); + inspect_actor(&store, move |actor| { + assert_eq!(next_seq, actor.next_object_seq); + assert!(actor.sealed.is_empty()); + actor.release_retry.at = Some(Instant::now()); + }) + .await; + } + + let seal = { + let store = store.clone(); + tokio::spawn(async move { store.seal_open_batch().await }) + }; + let (seq, create) = next_parked_operation(&mut parked).await; + create.send(true).unwrap(); + timeout(WAIT, seal).await.unwrap().unwrap().unwrap(); + inspect_actor(&store, |actor| { + assert_eq!(CREATE_RETRY_DELAY, actor.release_retry.delay); + assert_eq!(None, actor.release_retry.at); + assert_eq!(None, actor.release_retry_logged_at); + }) + .await; + assert_eq!( + vec![release(region_id, 0)], + releases_of(io.as_ref(), seq).await + ); + + // A fresh failure after successful indexing must start at the initial + // delay, rather than retaining the capped delay from the old outage. + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + let seal = { + let store = store.clone(); + tokio::spawn(async move { store.seal_open_batch().await }) + }; + let (_, create) = next_parked_operation(&mut parked).await; + create.send(false).unwrap(); + assert!(timeout(WAIT, seal).await.unwrap().unwrap().is_err()); + inspect_actor(&store, |actor| { + assert_eq!(CREATE_RETRY_DELAY * 2, actor.release_retry.delay); + actor.release_retry.at = Some(Instant::now() + Duration::from_secs(3600)); + }) + .await; + // Shutdown still drains pending records despite the future deadline. + let stop = begin_spawned_stop(&store).await; + let (_, create) = next_parked_operation(&mut parked).await; + create.send(true).unwrap(); + timeout(WAIT, stop).await.unwrap().unwrap().unwrap(); + } + + #[tokio::test] + async fn test_durable_release_timer_recovers_before_shutdown() { + let _serialized = SERIALIZED.lock().await; + let config = ObjectStoreWalConfig { + flush_interval: MIN_FLUSH_INTERVAL, + ..manual() + }; + let (store, io, mut parked) = open_parking_creates(memory_store(), &config).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(1, seq); + create.send(false).unwrap(); + // Only timer ticks can issue the retry: no manual seal or stop occurs. + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(2, seq); + create.send(true).unwrap(); + timeout(WAIT, async { + while inspect_actor(&store, |actor| actor.last_indexed.object_seq).await < seq { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await + .unwrap(); + assert_eq!( + vec![release(region_id, 0)], + releases_of(io.as_ref(), seq).await + ); + let recovered = recover(io.as_ref()).await.unwrap(); + assert_eq!(Some(&0), recovered.released_entry_ids.get(®ion_id)); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_durable_release_backoff_allows_data_and_permanent_releases() { + let _serialized = SERIALIZED.lock().await; + let (store, io, mut parked) = open_parking_creates(memory_store(), &eager()).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + let seal = { + let store = store.clone(); + tokio::spawn(async move { store.seal_open_batch().await }) + }; + let (_, create) = next_parked_operation(&mut parked).await; + create.send(false).unwrap(); + assert!(timeout(WAIT, seal).await.unwrap().unwrap().is_err()); + inspect_actor(&store, |actor| { + actor.release_retry.at = Some(Instant::now() + Duration::from_secs(3600)); + }) + .await; + + // A data batch can carry the retained release before its deadline. + let append = spawn_append_batch(&store, vec![entry(&store, region_id, "data")]); + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(2, seq); + create.send(false).unwrap(); + assert!(timeout(WAIT, append).await.unwrap().unwrap().is_err()); + inspect_actor(&store, move |actor| { + // Rolling back a data batch carrying a release also grows the + // backoff for the next release-only attempt. + assert_eq!(CREATE_RETRY_DELAY * 4, actor.release_retry.delay); + assert_eq!(Some(&0), actor.pending_releases.get(®ion_id)); + actor.release_retry.at = Some(Instant::now() + Duration::from_secs(3600)); + }) + .await; + + // A permanent release still seals immediately and waits for indexing. + let permanent_release = { + let store = store.clone(); + tokio::spawn(async move { store.obsolete_all(&provider(region_id), region_id).await }) + }; + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(3, seq); + create.send(true).unwrap(); + timeout(WAIT, permanent_release) + .await + .unwrap() + .unwrap() + .unwrap(); + inspect_actor(&store, |actor| { + assert_eq!(CREATE_RETRY_DELAY, actor.release_retry.delay); + assert_eq!(None, actor.release_retry.at); + }) + .await; + assert_eq!( + vec![release(region_id, EntryId::MAX)], + releases_of(io.as_ref(), seq).await + ); + store.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_bounded_release_recovery_uses_maximum_across_objects() { + let _serialized = SERIALIZED.lock().await; + for config in [eager(), enqueued(eager())] { + let object_store = memory_store(); + let store = open(object_store.clone(), &config).await; + let (one, pin) = (region(1), region(2)); + store + .append_batch(vec![entry(&store, one, "old"), entry(&store, pin, "pin")]) + .await + .unwrap(); + store.wait_durable(&provider(one), id(1, 1)).await.unwrap(); + store + .release_entries(&provider(one), one, id(1, 1)) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + store + .append_batch(vec![entry(&store, one, "new")]) + .await + .unwrap(); + store.wait_durable(&provider(one), id(3, 1)).await.unwrap(); + // Higher, then lower, records across objects distinguish max from + // both minimum and last-record-wins recovery. + for watermark in [id(3, 1), id(1, 1)] { + store + .release_entries(&provider(one), one, watermark) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + } + store.stop().await.unwrap(); + let store = open(object_store.clone(), &config).await; + assert_eq!(Some(id(3, 1)), watermark(&store, one)); + assert!(read_entries(&store, one, 0).await.is_empty()); + store.obsolete_all(&provider(one), one).await.unwrap(); + // A later stale bounded hint must not undo a permanent release. + store + .release_entries(&provider(one), one, id(1, 1)) + .await + .unwrap(); + store.seal_open_batch().await.unwrap(); + store.stop().await.unwrap(); + let store = open(object_store, &config).await; + assert_eq!(Some(EntryId::MAX), watermark(&store, one)); + assert_eq!( + expected_entries(pin, &[(id(1, 1), "pin")]), + read_entries(&store, pin, 0).await + ); + store.stop().await.unwrap(); + } + } + + #[tokio::test] + async fn test_bounded_release_shutdown_persists_pending_records() { + let _serialized = SERIALIZED.lock().await; + for config in [manual(), enqueued(manual())] { + let object_store = memory_store(); + let store = open(object_store.clone(), &config).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + assert_eq!(vec![0], object_seqs(store.io.as_ref()).await); + store.stop().await.unwrap(); + let reopened = open(object_store, &config).await; + assert_eq!(Some(0), watermark(&reopened, region_id)); + reopened.stop().await.unwrap(); + } + } + + #[tokio::test] + async fn test_bounded_release_crash_before_sealing_retains_history() { + let _serialized = SERIALIZED.lock().await; + let object_store = memory_store(); + let store = open(object_store.clone(), &eager()).await; + let region_id = region(1); + store + .append_batch(vec![entry(&store, region_id, "keep")]) + .await + .unwrap(); + store + .release_entries(&provider(region_id), region_id, id(1, 1)) + .await + .unwrap(); + store.crash().await; + let reopened = open(object_store, &eager()).await; + assert_eq!(None, watermark(&reopened, region_id)); + assert_eq!( + expected_entries(region_id, &[(id(1, 1), "keep")]), + read_entries(&reopened, region_id, 0).await + ); + reopened.stop().await.unwrap(); + } + + #[tokio::test] + async fn test_bounded_release_retries_failed_upload() { + let _serialized = SERIALIZED.lock().await; + let (store, io, mut parked) = open_parking_creates(memory_store(), &manual()).await; + let region_id = region(1); + store + .release_entries(&provider(region_id), region_id, 0) + .await + .unwrap(); + let seal = { + let store = store.clone(); + tokio::spawn(async move { store.seal_open_batch().await }) + }; + let (seq, create) = next_parked_operation(&mut parked).await; + assert_eq!(1, seq); + create.send(false).unwrap(); + assert!(timeout(WAIT, seal).await.unwrap().unwrap().is_err()); + // Manual batching has no timely timer tick. Drive sealing until the + // durable retry backoff expires and the next create is issued. + let (seq, create) = timeout(WAIT, async { + let mut interval = tokio::time::interval(Duration::from_millis(5)); + loop { + tokio::select! { + operation = parked.recv() => break operation.unwrap(), + _ = interval.tick() => { + let (response, _) = oneshot::channel(); + store.command_tx.send(Command::Seal { response }).await.unwrap(); + } + } + } + }) + .await + .unwrap(); + assert_eq!(2, seq); + create.send(true).unwrap(); + store.stop().await.unwrap(); + assert_eq!( + vec![release(region_id, 0)], + releases_of(io.as_ref(), seq).await + ); + } + #[tokio::test] async fn test_store_release_writes_a_record_of_the_largest_issued_id() { let _serialized = SERIALIZED.lock().await; @@ -11780,7 +12511,7 @@ mod tests { inner: ObjectStoreIo::new(object_store.clone(), PREFIX).unwrap(), late: Mutex::new(Some((1, late))), }); - ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await + open_io(io, &eager()).await } async fn start_object_meets_an_earlier_epoch() { @@ -11894,9 +12625,7 @@ mod tests { lose_next: AtomicBool::new(true), }); let before = Counters::sample(); - let error = ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()) - .await - .unwrap_err(); + let error = open_io(io, &eager()).await.unwrap_err(); assert_start_fails( &fixtures, before, @@ -11944,9 +12673,7 @@ mod tests { ) { let gauges = initialize_indexed_gauges().await; let before = Counters::sample(); - let error = ObjectStoreLogStore::open(io.clone(), &eager(), PREFIX.to_string()) - .await - .unwrap_err(); + let error = open_io(io.clone(), &eager()).await.unwrap_err(); assert!(recognise(&error), "unexpected error: {error:?}"); assert_eq!(retry_hint, error.retry_hint()); assert_eq!(present, object_seqs(io.as_ref()).await.as_slice()); @@ -12150,9 +12877,7 @@ mod tests { let before = Counters::sample(); let opening = { let io = io.clone(); - tokio::spawn(async move { - ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await - }) + tokio::spawn(async move { open_io(io, &eager()).await }) }; let path = object_path(&object_store, vanished); let delete = || object_store.delete(&path); @@ -12867,9 +13592,7 @@ mod tests { let started_at = Instant::now(); let opening = { let io = io.clone(); - tokio::spawn(async move { - ObjectStoreLogStore::open(io, &eager(), PREFIX.to_string()).await - }) + tokio::spawn(async move { open_io(io, &eager()).await }) }; // Every object is short enough to be read whole, and every // such read parks. diff --git a/src/meta-srv/src/bootstrap.rs b/src/meta-srv/src/bootstrap.rs index 700cc5d8ce9..3be37c1a436 100644 --- a/src/meta-srv/src/bootstrap.rs +++ b/src/meta-srv/src/bootstrap.rs @@ -343,7 +343,7 @@ pub async fn metasrv_builder( ) .await?; - let kv_backend = build_postgres_kv_backend( + let kv_backend: KvBackendRef = build_postgres_kv_backend( &opts.store_addrs, Some(cfg), opts.backend_tls.clone(), @@ -364,7 +364,7 @@ pub async fn metasrv_builder( use crate::utils::mysql::{build_mysql_election, build_mysql_kv_backend}; - let kv_backend = build_mysql_kv_backend( + let kv_backend: KvBackendRef = build_mysql_kv_backend( &opts.store_addrs, opts.backend_tls.as_ref(), &opts.meta_table_name, diff --git a/src/meta-srv/src/handler/region_lease_handler.rs b/src/meta-srv/src/handler/region_lease_handler.rs index a859bf18327..4f27b8f48e7 100644 --- a/src/meta-srv/src/handler/region_lease_handler.rs +++ b/src/meta-srv/src/handler/region_lease_handler.rs @@ -40,6 +40,11 @@ pub struct RegionLeaseHandler { pub type CustomizedRegionLeaseRenewerRef = Arc; pub trait CustomizedRegionLeaseRenewer: Send + Sync { + /// Returns the leases to grant for the renewed `regions`. + /// + /// The handler sets the `candidate_lease_id` of each returned lease from the entry of + /// `regions` with the same region id, or to `None` if there is no such entry. The value set + /// by the renewer is ignored. fn renew( &self, ctx: &mut Context, @@ -94,17 +99,31 @@ impl HeartbeatHandler for RegionLeaseHandler { renewed, }) => { let renewed = if let Some(renewer) = &self.customized_region_lease_renewer { + let candidate_lease_ids = renewed + .iter() + .filter_map(|(region_id, region_lease_info)| { + region_lease_info + .candidate_lease_id + .map(|candidate_lease_id| (*region_id, candidate_lease_id)) + }) + .collect::>(); renewer .renew(ctx, renewed) .into_iter() - .map(|region| region.into()) + .map(|region| { + GrantedRegion { + candidate_lease_id: candidate_lease_ids + .get(®ion.region_id) + .copied(), + ..region + } + .into() + }) .collect() } else { renewed - .into_iter() - .map(|(region_id, region_lease_info)| { - GrantedRegion::new(region_id, region_lease_info.role).into() - }) + .into_values() + .map(|region_lease_info| GrantedRegion::from(region_lease_info).into()) .collect::>() }; @@ -288,6 +307,10 @@ mod test { let _guard = opening_region_keeper .register_with_role(follower_peer.id, opening_region_id, RegionRole::Follower) .unwrap(); + let candidate_region_id = RegionId::new(table_id, region_number + 3); + let candidate_guard = opening_region_keeper + .register_candidate_region(follower_peer.id, candidate_region_id) + .unwrap(); let acc = &mut HeartbeatAccumulator::default(); @@ -297,6 +320,7 @@ mod test { new_empty_region_stat(region_id, RegionRole::Follower), new_empty_region_stat(another_region_id, RegionRole::Follower), new_empty_region_stat(opening_region_id, RegionRole::Follower), + new_empty_region_stat(candidate_region_id, RegionRole::Follower), ], ..Default::default() }); @@ -313,6 +337,10 @@ mod test { vec![ GrantedRegion::new(region_id, RegionRole::Follower), GrantedRegion::new(opening_region_id, RegionRole::Follower), + GrantedRegion { + candidate_lease_id: Some(candidate_guard.candidate_lease_id().unwrap()), + ..GrantedRegion::new(candidate_region_id, RegionRole::Follower) + }, ], ); assert_eq!(acc.inactive_region_ids, HashSet::from([another_region_id])); @@ -322,6 +350,106 @@ mod test { ); } + /// Grants every renewed region except `skipped`, plus `extra`, which is not in the heartbeat. + /// Each lease has a candidate lease id the handler must not keep. + struct StubRegionLeaseRenewer { + skipped: RegionId, + extra: RegionId, + } + + impl CustomizedRegionLeaseRenewer for StubRegionLeaseRenewer { + fn renew( + &self, + _ctx: &mut Context, + regions: HashMap, + ) -> Vec { + regions + .into_values() + .filter(|info| info.region_id != self.skipped) + .map(|info| (info.region_id, info.role)) + .chain([(self.extra, RegionRole::Follower)]) + .map(|(region_id, role)| GrantedRegion { + candidate_lease_id: Some(u64::MAX), + ..GrantedRegion::new(region_id, role) + }) + .collect() + } + } + + #[tokio::test] + async fn test_customized_renewer_leases_carry_candidate_lease_id() { + let table_id = 10; + let region_id = RegionId::new(table_id, 1); + let candidate_region_id = RegionId::new(table_id, 2); + let skipped_region_id = RegionId::new(table_id, 3); + let extra_region_id = RegionId::new(table_id, 4); + let leader_peer = Peer::empty(1); + let follower_peer = Peer::empty(2); + + let keeper = new_test_keeper(); + let table_metadata_manager = keeper.table_metadata_manager(); + table_metadata_manager + .create_table_metadata( + new_test_table_info(table_id), + TableRouteValue::physical(vec![RegionRoute { + region: Region::new_test(region_id), + leader_peer: Some(leader_peer), + follower_peers: vec![follower_peer.clone()], + ..Default::default() + }]), + HashMap::default(), + ) + .await + .unwrap(); + + let opening_region_keeper = Arc::new(MemoryRegionKeeper::default()); + let candidate_guard = opening_region_keeper + .register_candidate_region(follower_peer.id, candidate_region_id) + .unwrap(); + let _skipped_guard = opening_region_keeper + .register_candidate_region(follower_peer.id, skipped_region_id) + .unwrap(); + + let handler = RegionLeaseHandler::new( + default_distributed_time_constants().region_lease.as_secs(), + table_metadata_manager.clone(), + opening_region_keeper, + Some(Arc::new(StubRegionLeaseRenewer { + skipped: skipped_region_id, + extra: extra_region_id, + })), + ); + let metasrv = MetasrvBuilder::new().build().await.unwrap(); + let ctx = &mut metasrv.new_ctx(); + let acc = &mut HeartbeatAccumulator::default(); + acc.stat = Some(Stat { + id: follower_peer.id, + region_stats: vec![ + new_empty_region_stat(region_id, RegionRole::Follower), + new_empty_region_stat(candidate_region_id, RegionRole::Follower), + new_empty_region_stat(skipped_region_id, RegionRole::Follower), + ], + ..Default::default() + }); + + handler + .handle(&HeartbeatRequest::default(), ctx, acc) + .await + .unwrap(); + + assert_region_lease( + acc, + vec![ + GrantedRegion::new(region_id, RegionRole::Follower), + GrantedRegion { + candidate_lease_id: Some(candidate_guard.candidate_lease_id().unwrap()), + ..GrantedRegion::new(candidate_region_id, RegionRole::Follower) + }, + GrantedRegion::new(extra_region_id, RegionRole::Follower), + ], + ); + } + #[tokio::test] async fn test_handle_downgradable_leader() { diff --git a/src/meta-srv/src/metasrv.rs b/src/meta-srv/src/metasrv.rs index 2cd84815fb7..f3e4b2dae0c 100644 --- a/src/meta-srv/src/metasrv.rs +++ b/src/meta-srv/src/metasrv.rs @@ -216,6 +216,7 @@ pub struct MetasrvOptions { #[deprecated(note = "Use grpc.server_addr instead")] pub server_addr: String, /// The address of the store, e.g., etcd. + #[serde(serialize_with = "serialize_store_addrs")] pub store_addrs: Vec, /// TLS configuration for kv store backend (PostgreSQL/MySQL) /// Only applicable when using PostgreSQL or MySQL as the metadata store @@ -312,7 +313,7 @@ impl fmt::Debug for MetasrvOptions { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { let mut debug_struct = f.debug_struct("MetasrvOptions"); debug_struct - .field("store_addrs", &self.sanitize_store_addrs()) + .field("store_addrs", &sanitize_store_addrs(&self.store_addrs)) .field("backend_tls", &self.backend_tls) .field("selector", &self.selector) .field("enable_region_failover", &self.enable_region_failover) @@ -421,13 +422,18 @@ impl Configurable for MetasrvOptions { } } -impl MetasrvOptions { - fn sanitize_store_addrs(&self) -> Vec { - self.store_addrs - .iter() - .map(|addr| common_meta::kv_backend::util::sanitize_connection_string(addr)) - .collect() - } +fn sanitize_store_addrs(addrs: &[String]) -> Vec { + addrs + .iter() + .map(|addr| common_meta::kv_backend::util::sanitize_connection_string(addr)) + .collect() +} + +fn serialize_store_addrs(addrs: &[String], serializer: S) -> std::result::Result +where + S: serde::Serializer, +{ + sanitize_store_addrs(addrs).serialize(serializer) } pub struct MetasrvInfo { @@ -1005,6 +1011,19 @@ mod tests { use super::*; use crate::metasrv::MetasrvNodeInfo; + #[test] + fn test_store_addresses_are_redacted() { + let address = "mysql://alice:store-secret@localhost/database?password=query-secret"; + let options: MetasrvOptions = + toml::from_str(&format!("store_addrs = ['{address}']")).unwrap(); + assert_eq!(options.store_addrs, [address]); + for output in [options.to_toml().unwrap(), format!("{options:#?}")] { + assert!(!output.contains("store-secret")); + assert!(!output.contains("query-secret")); + assert!(output.contains("alice")); + } + } + #[test] fn test_deserialize_metasrv_node_info() { let str = r#"{"addr":"127.0.0.1:4002","version":"0.1.0","git_commit":"1234567890","start_time_ms":1715145600}"#; diff --git a/src/meta-srv/src/procedure/region_migration.rs b/src/meta-srv/src/procedure/region_migration.rs index 4f7967dd3c4..5210d3c7154 100644 --- a/src/meta-srv/src/procedure/region_migration.rs +++ b/src/meta-srv/src/procedure/region_migration.rs @@ -330,9 +330,11 @@ pub struct VolatileContext { /// `opening_region_guard` will be set after the /// [OpenCandidateRegion](crate::procedure::region_migration::open_candidate_region::OpenCandidateRegion) step. /// - /// `opening_region_guards` should be consumed after - /// the corresponding [RegionRoute](common_meta::rpc::router::RegionRoute) of the opening region - /// was written into [TableRouteValue](common_meta::key::table_route::TableRouteValue). + /// On success, `opening_region_guards` is consumed after the + /// [RegionRoute](common_meta::rpc::router::RegionRoute) of the opening region is written into + /// the [TableRouteValue](common_meta::key::table_route::TableRouteValue). On a rollback it is + /// consumed before the route is restored, so the target leaves the candidate registration + /// before the source can lead again. opening_region_guards: Vec, /// The deadline of leader region lease. leader_region_lease_deadline: Option, @@ -812,6 +814,9 @@ pub struct RegionMigrationData<'a> { pub(crate) struct RegionMigrationProcedure { state: Box, + /// Declared before `_guards`: a procedure that ends without a rollback (e.g. it fails to + /// record the rollback, or the manager stops) drops the candidate registration in `context` + /// before the region leaves the tracker, so another migration cannot start meanwhile. context: Context, _guards: Vec, } @@ -877,6 +882,8 @@ impl RegionMigrationProcedure { let _timer = METRIC_META_REGION_MIGRATION_EXECUTE .with_label_values(&["rollback"]) .start_timer(); + // Before the source can lead again, the target leaves the candidate registration. + self.context.volatile_ctx.opening_region_guards.clear(); let ctx = &self.context; let to_peer_id = ctx.persistent_ctx.to_peer.id; let table_regions = ctx.persistent_ctx.table_regions(); @@ -1119,6 +1126,27 @@ mod tests { assert_eq!(table_route.region_routes().unwrap()[0].leader_state, None); } + #[tokio::test] + async fn test_rollback_drops_candidate_registration_before_route_write() { + let env = TestingEnv::new(); + let persistent_ctx = new_persistent_context(); + let rollback = CandidateRollback::new(&env, &persistent_ctx).await; + let mut procedure = + RegionMigrationProcedure::new(persistent_ctx, rollback.context_factory, vec![]); + procedure + .context + .volatile_ctx + .opening_region_guards + .push(rollback.guard); + + procedure + .rollback_inner(&new_procedure_context()) + .await + .unwrap(); + + assert_registration_dropped_before_route_write(&rollback.keeper_empty_at_txn); + } + #[test] fn test_lock_key() { let persistent_context = new_persistent_context(); diff --git a/src/meta-srv/src/procedure/region_migration/manager.rs b/src/meta-srv/src/procedure/region_migration/manager.rs index 75bd28d6b9e..19553edb003 100644 --- a/src/meta-srv/src/procedure/region_migration/manager.rs +++ b/src/meta-srv/src/procedure/region_migration/manager.rs @@ -672,7 +672,7 @@ mod test { use tokio::sync::Notify; use super::*; - use crate::procedure::region_migration::test_util::TestingEnv; + use crate::procedure::region_migration::test_util::{TestingEnv, new_persistent_context}; #[derive(Default)] struct SubmissionGate { @@ -1324,4 +1324,47 @@ mod test { assert!(result.procedure_id.is_some()); env.procedure_manager().stop().await.unwrap(); } + + #[test] + fn test_dropped_procedure_releases_candidate_registration_before_tracker() { + let env = TestingEnv::new(); + let keeper = env.opening_region_keeper().clone(); + let tracker = RegionMigrationProcedureTracker::default(); + let region_id = RegionId::new(1024, 1); + let task = RegionMigrationProcedureTask::new( + region_id, + Peer::empty(1), + Peer::empty(2), + Duration::from_secs(1), + RegionMigrationTriggerReason::Manual, + ); + let tracker_guard = tracker.insert_running_procedure(&task).unwrap(); + let mut procedure = RegionMigrationProcedure::new( + new_persistent_context(1, 2, region_id), + env.context_factory(), + vec![tracker_guard], + ); + procedure + .context + .volatile_ctx + .opening_region_guards + .push(keeper.register_candidate_region(2, region_id).unwrap()); + + // While this read lock is held, the tracker entry cannot be removed, so the registration + // has to go first. + let running_procedures = tracker.running_procedures.read().unwrap(); + let dropper = std::thread::spawn(move || drop(procedure)); + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while !keeper.is_empty() { + assert!( + std::time::Instant::now() < deadline, + "the candidate registration outlives the tracker entry" + ); + std::thread::sleep(Duration::from_millis(1)); + } + assert!(running_procedures.contains_key(®ion_id)); + drop(running_procedures); + dropper.join().unwrap(); + assert!(tracker.running_procedures.read().unwrap().is_empty()); + } } diff --git a/src/meta-srv/src/procedure/region_migration/open_candidate_region.rs b/src/meta-srv/src/procedure/region_migration/open_candidate_region.rs index bc21746a046..670baae6875 100644 --- a/src/meta-srv/src/procedure/region_migration/open_candidate_region.rs +++ b/src/meta-srv/src/procedure/region_migration/open_candidate_region.rs @@ -27,7 +27,6 @@ use common_telemetry::info; use common_telemetry::tracing_context::TracingContext; use serde::{Deserialize, Serialize}; use snafu::{OptionExt, ResultExt}; -use store_api::region_engine::RegionRole; use store_api::region_request::RegionRequirements; use tokio::time::Instant; @@ -147,7 +146,7 @@ impl OpenCandidateRegion { // Registers the opening region. let guard = ctx .opening_region_keeper - .register_with_role(candidate.id, *region_id, RegionRole::Follower) + .register_candidate_region(candidate.id, *region_id) .context(error::RegionOperatingRaceSnafu { peer_id: candidate.id, region_id: *region_id, @@ -236,7 +235,7 @@ impl OpenCandidateRegion { #[cfg(test)] mod tests { use std::assert_matches; - use std::collections::HashMap; + use std::collections::{HashMap, HashSet}; use common_catalog::consts::MITO2_ENGINE; use common_error::ext::RetryHint; @@ -247,6 +246,7 @@ mod tests { use common_meta::key::test_utils::new_test_table_info; use common_meta::peer::Peer; use common_meta::rpc::router::{Region, RegionRoute}; + use store_api::region_engine::RegionRole; use store_api::storage::RegionId; use super::*; @@ -623,8 +623,24 @@ mod tests { send_mock_reply(mailbox, rx, |id| Ok(new_open_region_reply(id, true, None))); let procedure_ctx = new_procedure_context(); let (next, _) = state.next(&mut ctx, &procedure_ctx).await.unwrap(); - let vc = ctx.volatile_ctx; - assert_eq!(vc.opening_region_guards[0].info(), (to_peer_id, region_id)); + let candidate_lease_id = { + let vc = &ctx.volatile_ctx; + assert_eq!(vc.opening_region_guards[0].info(), (to_peer_id, region_id)); + vc.opening_region_guards[0].candidate_lease_id() + }; + assert!(candidate_lease_id.is_some_and(|id| id != 0)); + let regions = HashSet::from([region_id]); + let registered = |ctx: &Context| { + ctx.opening_region_keeper + .extract_operating_regions(to_peer_id, ®ions)[®ion_id] + .candidate_lease_id + }; + assert_eq!(registered(&ctx), candidate_lease_id); + + // A retry of the step keeps the registration and its id. + state.next(&mut ctx, &procedure_ctx).await.unwrap(); + assert_eq!(ctx.volatile_ctx.opening_region_guards.len(), 1); + assert_eq!(registered(&ctx), candidate_lease_id); let flush_leader_region = next.as_any().downcast_ref::().unwrap(); assert_matches!(flush_leader_region, PreFlushRegion); diff --git a/src/meta-srv/src/procedure/region_migration/test_util.rs b/src/meta-srv/src/procedure/region_migration/test_util.rs index ecee2eae0c5..49cb0f4b119 100644 --- a/src/meta-srv/src/procedure/region_migration/test_util.rs +++ b/src/meta-srv/src/procedure/region_migration/test_util.rs @@ -14,20 +14,22 @@ use std::assert_matches; use std::collections::HashMap; -use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; use std::time::Duration; use api::v1::meta::MailboxMessage; use common_meta::DatanodeId; use common_meta::ddl::NoopRegionFailureDetectorControl; use common_meta::key::table_route::TableRouteValue; +use common_meta::key::test_utils::new_test_table_info; use common_meta::key::{TableMetadataManager, TableMetadataManagerRef}; use common_meta::kv_backend::KvBackendRef; use common_meta::kv_backend::memory::MemoryKvBackend; +use common_meta::kv_backend::test_util::TxnObservingKvBackend; use common_meta::peer::Peer; -use common_meta::region_keeper::{MemoryRegionKeeper, MemoryRegionKeeperRef}; -use common_meta::rpc::router::RegionRoute; +use common_meta::region_keeper::{MemoryRegionKeeper, MemoryRegionKeeperRef, OperatingRegionGuard}; +use common_meta::rpc::router::{LeaderState, Region, RegionRoute}; use common_meta::sequence::SequenceBuilder; use common_meta::state_store::KvStateStore; use common_procedure::local::{LocalManager, ManagerConfig}; @@ -592,3 +594,74 @@ pub fn new_procedure_context() -> ProcedureContext { event_context: None, } } + +/// A migration about to roll back: its region is led by the source peer in the downgrading state, +/// and registered on the target peer as the migration's candidate. +pub(crate) struct CandidateRollback { + /// Creates contexts on the table metadata whose transactions are observed. + pub(crate) context_factory: DefaultContextFactory, + /// The candidate registration of the region on the target peer. + pub(crate) guard: OperatingRegionGuard, + /// Whether the keeper held no registration before each transaction since the setup. + pub(crate) keeper_empty_at_txn: Arc>>, +} + +impl CandidateRollback { + /// Creates the table of `persistent_ctx` and the candidate registration of its region. + pub(crate) async fn new(env: &TestingEnv, persistent_ctx: &PersistentContext) -> Self { + let (table_metadata_manager, keeper_empty_at_txn) = + table_metadata_manager_observing_keeper(env.opening_region_keeper().clone()); + let region_id = persistent_ctx.region_ids[0]; + let region_routes = vec![RegionRoute { + region: Region::new_test(region_id), + leader_peer: Some(persistent_ctx.from_peer.clone()), + leader_state: Some(LeaderState::Downgrading), + ..Default::default() + }]; + table_metadata_manager + .create_table_metadata( + new_test_table_info(region_id.table_id()), + TableRouteValue::physical(region_routes), + HashMap::default(), + ) + .await + .unwrap(); + keeper_empty_at_txn.lock().unwrap().clear(); + let guard = env + .opening_region_keeper() + .register_candidate_region(persistent_ctx.to_peer.id, region_id) + .unwrap(); + Self { + context_factory: env.context_factory_with_metadata(table_metadata_manager), + guard, + keeper_empty_at_txn, + } + } +} + +/// Asserts that the table metadata was written at least once after the setup, and that the +/// candidate registration was gone before every such write. +pub(crate) fn assert_registration_dropped_before_route_write( + keeper_empty_at_txn: &Mutex>, +) { + let keeper_empty_at_txn = keeper_empty_at_txn.lock().unwrap(); + assert!(!keeper_empty_at_txn.is_empty()); + assert!(keeper_empty_at_txn.iter().all(|empty| *empty)); +} + +/// Returns a table metadata manager that records, before each of its transactions, whether +/// `keeper` holds no registration. +fn table_metadata_manager_observing_keeper( + keeper: MemoryRegionKeeperRef, +) -> (TableMetadataManagerRef, Arc>>) { + let keeper_empty_at_txn = Arc::new(Mutex::new(Vec::new())); + let recorded = keeper_empty_at_txn.clone(); + let kv_backend = Arc::new(TxnObservingKvBackend { + inner: Arc::new(MemoryKvBackend::new()), + before_txn: Box::new(move || recorded.lock().unwrap().push(keeper.is_empty())), + }); + ( + Arc::new(TableMetadataManager::new(kv_backend)), + keeper_empty_at_txn, + ) +} diff --git a/src/meta-srv/src/procedure/region_migration/update_metadata.rs b/src/meta-srv/src/procedure/region_migration/update_metadata.rs index e96a025c5de..42d28c06f6d 100644 --- a/src/meta-srv/src/procedure/region_migration/update_metadata.rs +++ b/src/meta-srv/src/procedure/region_migration/update_metadata.rs @@ -69,6 +69,8 @@ impl State for UpdateMetadata { Ok((Box::new(CloseDowngradedRegion), Status::executing(false))) } UpdateMetadata::Rollback => { + // Before the source can lead again, the target leaves the candidate registration. + ctx.volatile_ctx.opening_region_guards.clear(); self.rollback_downgraded_region(ctx, &procedure_ctx.provider) .await?; diff --git a/src/meta-srv/src/procedure/region_migration/update_metadata/rollback_downgraded_region.rs b/src/meta-srv/src/procedure/region_migration/update_metadata/rollback_downgraded_region.rs index 1c9c12a9bc1..a8f94c3018b 100644 --- a/src/meta-srv/src/procedure/region_migration/update_metadata/rollback_downgraded_region.rs +++ b/src/meta-srv/src/procedure/region_migration/update_metadata/rollback_downgraded_region.rs @@ -85,7 +85,9 @@ mod tests { use crate::error::Error; use crate::procedure::region_migration::migration_abort::RegionMigrationAbort; - use crate::procedure::region_migration::test_util::{self, TestingEnv, new_procedure_context}; + use crate::procedure::region_migration::test_util::{ + self, CandidateRollback, TestingEnv, new_procedure_context, + }; use crate::procedure::region_migration::update_metadata::UpdateMetadata; use crate::procedure::region_migration::{ContextFactory, PersistentContext, State}; @@ -173,4 +175,20 @@ mod tests { table_route.region_routes().unwrap() ); } + + #[tokio::test] + async fn test_next_drops_candidate_registration_before_route_write() { + let env = TestingEnv::new(); + let persistent_context = new_persistent_context(); + let rollback = CandidateRollback::new(&env, &persistent_context).await; + let mut ctx = rollback.context_factory.new_context(persistent_context); + ctx.volatile_ctx.opening_region_guards.push(rollback.guard); + + UpdateMetadata::Rollback + .next(&mut ctx, &new_procedure_context()) + .await + .unwrap(); + + test_util::assert_registration_dropped_before_route_write(&rollback.keeper_empty_at_txn); + } } diff --git a/src/meta-srv/src/procedure/region_migration/upgrade_candidate_region.rs b/src/meta-srv/src/procedure/region_migration/upgrade_candidate_region.rs index dbd870076af..9b879962378 100644 --- a/src/meta-srv/src/procedure/region_migration/upgrade_candidate_region.rs +++ b/src/meta-srv/src/procedure/region_migration/upgrade_candidate_region.rs @@ -154,6 +154,12 @@ impl UpgradeCandidateRegion { .get(®ion_id) .copied(); let checkpoint = replay_checkpoints.get(®ion_id).copied(); + let candidate_lease_id = ctx + .volatile_ctx + .opening_region_guards + .iter() + .find(|guard| guard.info().1 == region_id) + .and_then(|guard| guard.candidate_lease_id()); upgrade_regions.push(UpgradeRegion { region_id, last_entry_id, @@ -162,6 +168,7 @@ impl UpgradeCandidateRegion { location_id: Some(ctx.persistent_ctx.from_peer.id), replay_entry_id: checkpoint.map(|c| c.entry_id), metadata_replay_entry_id: checkpoint.and_then(|c| c.metadata_entry_id), + candidate_lease_id, }); } @@ -466,6 +473,46 @@ mod tests { assert_eq!(upgrade_regions[0].metadata_replay_entry_id, None); } + #[tokio::test] + async fn test_build_upgrade_region_instruction_carries_candidate_lease_id() { + let state = UpgradeCandidateRegion::default(); + let env = TestingEnv::new(); + let mut persistent_context = new_persistent_context(); + let region_ids = vec![RegionId::new(1024, 1), RegionId::new(1024, 2)]; + persistent_context.region_ids = region_ids.clone(); + let mut ctx = env.context_factory().new_context(persistent_context); + prepare_table_metadata(&ctx, RegionWalOptions::default()).await; + let to_peer_id = ctx.persistent_ctx.to_peer.id; + let mut expected = HashMap::new(); + // The guards are stored in the opposite order of the regions. + for region_id in region_ids.iter().rev() { + let guard = env + .opening_region_keeper() + .register_candidate_region(to_peer_id, *region_id) + .unwrap(); + expected.insert(*region_id, guard.candidate_lease_id().unwrap()); + ctx.volatile_ctx.opening_region_guards.push(guard); + } + + // Retries send the same ids. + for _ in 0..2 { + let instruction = state + .build_upgrade_region_instruction(&mut ctx, Duration::from_secs(1)) + .await + .unwrap(); + let Instruction::UpgradeRegions(upgrade_regions) = instruction else { + unreachable!() + }; + assert_eq!(upgrade_regions.len(), 2); + for upgrade_region in upgrade_regions { + assert_eq!( + upgrade_region.candidate_lease_id, + Some(expected[&upgrade_region.region_id]) + ); + } + } + } + #[tokio::test] async fn test_build_upgrade_region_instruction_merges_metric_metadata_pruned_entry_id() { let state = UpgradeCandidateRegion::default(); diff --git a/src/meta-srv/src/procedure/repartition.rs b/src/meta-srv/src/procedure/repartition.rs index 9011a122b90..30aaa93bc98 100644 --- a/src/meta-srv/src/procedure/repartition.rs +++ b/src/meta-srv/src/procedure/repartition.rs @@ -1370,24 +1370,30 @@ mod tests { let _guards = Context::register_operating_regions(&keeper, ®ion_routes).unwrap(); - let leader_roles = - keeper.extract_operating_region_roles(1, &HashSet::from([RegionId::new(1024, 1)])); - let staging_roles = - keeper.extract_operating_region_roles(2, &HashSet::from([RegionId::new(1024, 2)])); - let downgrading_roles = - keeper.extract_operating_region_roles(3, &HashSet::from([RegionId::new(1024, 3)])); + let leader_regions = + keeper.extract_operating_regions(1, &HashSet::from([RegionId::new(1024, 1)])); + let staging_regions = + keeper.extract_operating_regions(2, &HashSet::from([RegionId::new(1024, 2)])); + let downgrading_regions = + keeper.extract_operating_regions(3, &HashSet::from([RegionId::new(1024, 3)])); assert_eq!( - leader_roles.get(&RegionId::new(1024, 1)), - Some(&RegionRole::Leader) + leader_regions + .get(&RegionId::new(1024, 1)) + .map(|region| region.role), + Some(RegionRole::Leader) ); assert_eq!( - staging_roles.get(&RegionId::new(1024, 2)), - Some(&RegionRole::StagingLeader) + staging_regions + .get(&RegionId::new(1024, 2)) + .map(|region| region.role), + Some(RegionRole::StagingLeader) ); assert_eq!( - downgrading_roles.get(&RegionId::new(1024, 3)), - Some(&RegionRole::DowngradingLeader) + downgrading_regions + .get(&RegionId::new(1024, 3)) + .map(|region| region.role), + Some(RegionRole::DowngradingLeader) ); } diff --git a/src/meta-srv/src/region/lease_keeper.rs b/src/meta-srv/src/region/lease_keeper.rs index d4a899d002d..97c22e50a1f 100644 --- a/src/meta-srv/src/region/lease_keeper.rs +++ b/src/meta-srv/src/region/lease_keeper.rs @@ -18,11 +18,11 @@ use std::sync::Arc; use common_meta::DatanodeId; use common_meta::key::TableMetadataManagerRef; use common_meta::key::table_route::TableRouteValue; -use common_meta::region_keeper::MemoryRegionKeeperRef; +use common_meta::region_keeper::{MemoryRegionKeeperRef, OperatingRegion}; use common_meta::rpc::router::RegionRoute; use common_telemetry::{info, warn}; use snafu::ResultExt; -use store_api::region_engine::RegionRole; +use store_api::region_engine::{GrantedRegion, RegionRole}; use store_api::storage::{RegionId, TableId}; use crate::error::{self, Result}; @@ -81,16 +81,22 @@ fn renew_region_lease_via_region_route( } fn renew_region_lease_via_operating_regions( - operating_regions: &HashMap, + operating_regions: &HashMap, datanode_id: DatanodeId, region_id: RegionId, reported_role: RegionRole, + table_route_exists: bool, ) -> Option { // `operating_regions` is filtered by the current datanode in `collect_metadata`, // so looking up by `region_id` is sufficient here. - if let Some(role) = operating_regions.get(®ion_id) { - let region_lease_info = RegionLeaseInfo::operating(region_id, *role); - if *role != reported_role { + if let Some(operating_region) = operating_regions.get(®ion_id) { + let role = operating_region.role; + let mut region_lease_info = RegionLeaseInfo::operating(region_id, role); + // Without the table route, the registration alone must not vouch for a migration. + if table_route_exists { + region_lease_info.candidate_lease_id = operating_region.candidate_lease_id; + } + if role != reported_role { info!( "The region {} on datanode {} is operating with role {:?}, but reported as {:?}", region_id, datanode_id, role, reported_role @@ -112,6 +118,9 @@ pub struct RegionLeaseInfo { pub is_operating: bool, /// The role of region. pub role: RegionRole, + /// The candidate lease id of the region migration that registered this region as its + /// target, if the lease is granted because of that registration and the table route exists. + pub candidate_lease_id: Option, } impl RegionLeaseInfo { @@ -121,6 +130,7 @@ impl RegionLeaseInfo { region_id, is_operating: true, role, + candidate_lease_id: None, } } } @@ -131,6 +141,16 @@ impl From<(RegionId, RegionRole)> for RegionLeaseInfo { region_id, is_operating: false, role, + candidate_lease_id: None, + } + } +} + +impl From for GrantedRegion { + fn from(value: RegionLeaseInfo) -> Self { + GrantedRegion { + candidate_lease_id: value.candidate_lease_id, + ..GrantedRegion::new(value.region_id, value.role) } } } @@ -165,13 +185,14 @@ impl RegionLeaseKeeper { fn renew_region_lease( &self, table_metadata: &HashMap, - operating_regions: &HashMap, + operating_regions: &HashMap, datanode_id: DatanodeId, region_id: RegionId, reported_role: RegionRole, ) -> Option { + let table_route = table_metadata.get(®ion_id.table_id()); // First try to renew via region route - if let Some(table_route) = table_metadata.get(®ion_id.table_id()) + if let Some(table_route) = table_route && let Ok(Some(region_route)) = table_route.region_route(region_id) && let Some(region_lease) = renew_region_lease_via_region_route(®ion_route, datanode_id, region_id) @@ -184,6 +205,7 @@ impl RegionLeaseKeeper { datanode_id, region_id, reported_role, + table_route.is_some(), ) { return Some(region_lease_info); } @@ -200,11 +222,11 @@ impl RegionLeaseKeeper { region_ids: HashSet, ) -> Result<( HashMap, - HashMap, + HashMap, )> { let operating_regions = self .memory_region_keeper - .extract_operating_region_roles(datanode_id, ®ion_ids); + .extract_operating_regions(datanode_id, ®ion_ids); let table_ids = region_ids .into_iter() .map(|region_id| region_id.table_id()) @@ -271,12 +293,13 @@ mod tests { use std::collections::{HashMap, HashSet}; use std::sync::Arc; + use common_meta::DatanodeId; use common_meta::key::TableMetadataManager; use common_meta::key::table_route::{LogicalTableRouteValue, TableRouteValue}; use common_meta::key::test_utils::new_test_table_info; use common_meta::kv_backend::memory::MemoryKvBackend; use common_meta::peer::Peer; - use common_meta::region_keeper::MemoryRegionKeeper; + use common_meta::region_keeper::{MemoryRegionKeeper, OperatingRegion, OperatingRegionGuard}; use common_meta::rpc::router::{LeaderState, Region, RegionRouteBuilder}; use store_api::region_engine::RegionRole; use store_api::storage::RegionId; @@ -414,7 +437,13 @@ mod tests { ); assert_eq!( regions, - HashMap::from([(opening_region_id, RegionRole::Leader)]) + HashMap::from([( + opening_region_id, + OperatingRegion { + role: RegionRole::Leader, + candidate_lease_id: None, + } + )]) ); } @@ -817,4 +846,95 @@ mod tests { )]) ); } + + /// Creates a table whose route has the region led by `leader_peer_id`, and registers the region + /// on `candidate_peer_id` as the target of a migration. + async fn register_candidate_with_route( + keeper: &RegionLeaseKeeper, + region_id: RegionId, + leader_peer_id: DatanodeId, + candidate_peer_id: DatanodeId, + ) -> OperatingRegionGuard { + let region_route = RegionRouteBuilder::default() + .region(Region::new_test(region_id)) + .leader_peer(Peer::empty(leader_peer_id)) + .build() + .unwrap(); + keeper + .table_metadata_manager() + .create_table_metadata( + new_test_table_info(region_id.table_id()), + TableRouteValue::physical(vec![region_route]), + HashMap::default(), + ) + .await + .unwrap(); + keeper + .memory_region_keeper + .register_candidate_region(candidate_peer_id, region_id) + .unwrap() + } + + #[tokio::test] + async fn test_renew_region_lease_from_registration_carries_candidate_lease_id() { + let keeper = new_test_keeper(); + let region_id = RegionId::new(2048, 1); + let guard = register_candidate_with_route(&keeper, region_id, 1, 2).await; + + let RenewRegionLeasesResponse { renewed, .. } = keeper + .renew_region_leases(2, &[(region_id, RegionRole::Follower)]) + .await + .unwrap(); + + let expected = RegionLeaseInfo { + candidate_lease_id: guard.candidate_lease_id(), + ..RegionLeaseInfo::operating(region_id, RegionRole::Follower) + }; + assert!(expected.candidate_lease_id.is_some()); + assert_eq!(renewed, HashMap::from([(region_id, expected)])); + } + + #[tokio::test] + async fn test_renew_region_leases_route_lease_has_no_candidate_lease_id() { + let keeper = new_test_keeper(); + let region_id = RegionId::new(2048, 1); + // The route already names the candidate as the leader. + let _guard = register_candidate_with_route(&keeper, region_id, 2, 2).await; + + let RenewRegionLeasesResponse { renewed, .. } = keeper + .renew_region_leases(2, &[(region_id, RegionRole::Follower)]) + .await + .unwrap(); + + assert_eq!( + renewed, + HashMap::from([( + region_id, + RegionLeaseInfo::from((region_id, RegionRole::Leader)) + )]) + ); + } + + #[tokio::test] + async fn test_renew_region_leases_without_table_route_has_no_candidate_lease_id() { + let keeper = new_test_keeper(); + let region_id = RegionId::new(2048, 1); + let _guard = keeper + .memory_region_keeper + .register_candidate_region(2, region_id) + .unwrap(); + + let RenewRegionLeasesResponse { renewed, .. } = keeper + .renew_region_leases(2, &[(region_id, RegionRole::Follower)]) + .await + .unwrap(); + + assert_eq!( + renewed, + HashMap::from([( + region_id, + RegionLeaseInfo::operating(region_id, RegionRole::Follower) + )]) + ); + } } diff --git a/src/meta-srv/src/utils/mysql.rs b/src/meta-srv/src/utils/mysql.rs index 326f3880f43..5bbfb82357a 100644 --- a/src/meta-srv/src/utils/mysql.rs +++ b/src/meta-srv/src/utils/mysql.rs @@ -14,7 +14,7 @@ use common_meta::election::ElectionRef; use common_meta::election::rds::mysql::{ElectionMysqlClient, MySqlElection}; -use common_meta::kv_backend::KvBackendRef; +use common_meta::kv_backend::ClosableKvBackendRef; use common_meta::kv_backend::rds::MySqlStore; use common_telemetry::info; use servers::tls::{TlsMode, TlsOption}; @@ -88,7 +88,7 @@ pub async fn create_mysql_pool( Ok(pool) } -/// Builds a MySQL-backed metadata [`KvBackendRef`]. +/// Builds a MySQL-backed metadata [`ClosableKvBackendRef`]. /// /// * `store_addrs` - MySQL connection URLs; only the first address is used. /// * `tls_config` - optional TLS settings for the MySQL connection. @@ -99,7 +99,7 @@ pub async fn build_mysql_kv_backend( tls_config: Option<&TlsOption>, table_name: &str, max_txn_ops: usize, -) -> Result { +) -> Result { let pool = create_mysql_pool(store_addrs, tls_config).await?; MySqlStore::with_mysql_pool(pool, table_name, max_txn_ops) .await diff --git a/src/meta-srv/src/utils/postgres.rs b/src/meta-srv/src/utils/postgres.rs index 148a654e2cd..f4bed382237 100644 --- a/src/meta-srv/src/utils/postgres.rs +++ b/src/meta-srv/src/utils/postgres.rs @@ -18,7 +18,7 @@ use std::str::FromStr; use common_error::ext::BoxedError; use common_meta::election::ElectionRef; use common_meta::election::rds::postgres::{ElectionPgClient, PgElection}; -use common_meta::kv_backend::KvBackendRef; +use common_meta::kv_backend::ClosableKvBackendRef; use common_meta::kv_backend::rds::PgStore; use common_meta::kv_backend::rds::postgres::{ TlsMode as PgTlsMode, TlsOption as PgTlsOption, create_postgres_tls_connector, @@ -117,7 +117,7 @@ fn is_unix_socket_url(_: &str) -> bool { false } -/// Builds a Postgres-backed metadata [`KvBackendRef`]. +/// Builds a Postgres-backed metadata [`ClosableKvBackendRef`]. /// /// * `store_addrs` - Postgres connection URLs; only the first address is used. /// * `cfg` - optional deadpool config to customize pool/session behavior. @@ -135,7 +135,7 @@ pub async fn build_postgres_kv_backend( table_name: &str, max_txn_ops: usize, auto_create_schema: bool, -) -> Result { +) -> Result { let pool = create_postgres_pool(store_addrs, cfg, tls_config).await?; PgStore::with_pg_pool( pool, diff --git a/src/metric-engine/AGENTS.md b/src/metric-engine/AGENTS.md index 9c3820a0cc3..5efd43d82f6 100644 --- a/src/metric-engine/AGENTS.md +++ b/src/metric-engine/AGENTS.md @@ -83,6 +83,9 @@ cargo nextest run -p metric-engine write lock and invalidate the cache after the Mito2 write succeeds; preserve that ordering when adding a metadata mutation path. - Always convert ids via `utils::to_data_region_id` / `to_metadata_region_id`. +- `MetricEngine::loaded_logical_regions` exposes physical regions' loaded logical + children. RegionServer captures them before drop/close and unregisters them + after success; physical removal must also clear logical column caches. ## Maintenance contract diff --git a/src/metric-engine/Cargo.toml b/src/metric-engine/Cargo.toml index 0cc39dc5add..89c8ce3daef 100644 --- a/src/metric-engine/Cargo.toml +++ b/src/metric-engine/Cargo.toml @@ -53,6 +53,7 @@ log-store = { workspace = true, features = ["testing"] } mito2 = { workspace = true, features = ["test"] } mur3 = "0.1" partition = { workspace = true } +rstest = { workspace = true } [[bench]] name = "bench_tsid_generator" diff --git a/src/metric-engine/src/data_region.rs b/src/metric-engine/src/data_region.rs index a11b3e57b54..8847307ff14 100644 --- a/src/metric-engine/src/data_region.rs +++ b/src/metric-engine/src/data_region.rs @@ -12,6 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::collections::HashMap; + use api::v1::SemanticType; use common_query::native_histogram::is_native_histogram_value_type; use common_telemetry::{debug, info}; @@ -26,7 +28,7 @@ use store_api::region_request::{ use store_api::storage::consts::ReservedColumnId; use store_api::storage::{ConcreteDataType, RegionId}; -use crate::engine::IndexOptions; +use crate::engine::{IndexOptions, validate_shared_column}; use crate::error::{ AddingFieldColumnSnafu, ColumnTypeMismatchSnafu, ForbiddenPhysicalAlterSnafu, MitoReadOperationSnafu, MitoWriteOperationSnafu, Result, SetSkippingIndexOptionSnafu, @@ -44,7 +46,7 @@ pub struct DataRegion { } impl DataRegion { - pub fn new(mito: MitoEngine) -> Self { + pub(crate) fn new(mito: MitoEngine) -> Self { Self { mito } } @@ -55,10 +57,16 @@ impl DataRegion { /// logical regions. /// /// Invoker don't need to set up or verify the column id. This method will adjust - /// it using underlying schema. + /// it using underlying schema, and reuse the persisted id of a column that the + /// physical region already has. /// /// This method will also set the nullable marker to true. All of those change are applies /// to `columns` in-place. + /// + /// Callers must hold the column-add lock of the physical region, otherwise two + /// concurrent requests could read the same metadata and assign one column id to + /// different columns. Use + /// [MetricEngineInner::add_physical_columns](crate::engine::MetricEngineInner::add_physical_columns). pub async fn add_columns( &self, region_id: RegionId, @@ -72,10 +80,13 @@ impl DataRegion { let region_id = utils::to_data_region_id(region_id); - let num_columns = columns.len(); - let request = self + let Some((new_column_count, request)) = self .assemble_alter_request(region_id, columns, index_options) - .await?; + .await? + else { + // Every requested column already exists. + return Ok(()); + }; let _timer = MITO_DDL_DURATION.start_timer(); @@ -85,26 +96,55 @@ impl DataRegion { .await .context(MitoWriteOperationSnafu)?; - PHYSICAL_COLUMN_COUNT.add(num_columns as _); + PHYSICAL_COLUMN_COUNT.add(new_column_count as _); Ok(()) } /// Generate wrapped [RegionAlterRequest] with given [ColumnMetadata]. - /// This method will modify `columns` in-place. + /// + /// Columns that the physical region already has are not added again: their + /// definitions are validated and their persisted column ids are reused. + /// Returns `None` if every given column already exists. async fn assemble_alter_request( &self, region_id: RegionId, columns: Vec, index_options: IndexOptions, - ) -> Result { - // retrieve underlying version + ) -> Result> { let region_metadata = self .mito .get_metadata(region_id) .await .context(MitoReadOperationSnafu)?; + // The caller derives the columns to add from a cached schema that a cancelled or + // concurrent alter may have left behind, so the names are checked against the + // real metadata instead of being trusted. + let mut seen: HashMap = HashMap::with_capacity(columns.len()); + let mut columns_to_add = Vec::with_capacity(columns.len()); + for column in columns { + let name = column.column_schema.name.clone(); + if let Some(existing) = region_metadata.column_by_name(&name) { + validate_shared_column(region_id, existing, &column)?; + continue; + } + // The batch itself may repeat a column that is not installed yet: the duplicate + // must agree with the first occurrence, and only one of them is added. + if let Some(&index) = seen.get(&name) { + validate_shared_column(region_id, &columns_to_add[index], &column)?; + continue; + } + seen.insert(name, columns_to_add.len()); + columns_to_add.push(column); + } + + if columns_to_add.is_empty() { + debug!("No new columns to add to physical region {region_id:?}"); + return Ok(None); + } + let new_column_count = columns_to_add.len(); + // find the max column id let new_column_id_start = 1 + region_metadata .column_metadatas @@ -120,7 +160,7 @@ impl DataRegion { .unwrap_or(0); // overwrite semantic type - let new_columns = columns + let new_columns = columns_to_add .into_iter() .enumerate() .map(|(delta, mut c)| { @@ -188,7 +228,7 @@ impl DataRegion { }, }); - Ok(alter_request) + Ok(Some((new_column_count, alter_request))) } pub async fn write_data( @@ -259,6 +299,14 @@ mod test { use super::*; use crate::test_util::TestEnv; + fn tag_column(name: &str) -> ColumnMetadata { + ColumnMetadata { + column_id: 0, + semantic_type: SemanticType::Tag, + column_schema: ColumnSchema::new(name, ConcreteDataType::string_datatype(), false), + } + } + #[tokio::test] async fn test_add_columns() { let env = TestEnv::new().await; @@ -345,4 +393,122 @@ mod test { .await; assert!(result.is_err()); } + + // A duplicate column inside one batch must agree with the first occurrence instead of + // being dropped silently. + #[tokio::test] + async fn test_add_columns_rejects_conflicting_duplicates() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let physical_region_id = env.default_physical_region_id(); + let data_region_id = utils::to_data_region_id(physical_region_id); + + let conflicting = ColumnMetadata { + column_id: 0, + semantic_type: SemanticType::Field, + column_schema: ColumnSchema::new("dup", ConcreteDataType::float64_datatype(), false), + }; + let result = env + .data_region() + .add_columns( + physical_region_id, + vec![tag_column("dup"), conflicting], + IndexOptions::Inverted, + ) + .await; + + assert!(result.is_err()); + assert!( + env.mito() + .get_metadata(data_region_id) + .await + .unwrap() + .column_by_name("dup") + .is_none() + ); + } + + // A retry passes the column it added before, because the caller derives the columns to + // add from a cache that a cancelled request may not have refreshed. The column keeps the + // id it was installed with. + #[tokio::test] + async fn test_add_columns_reuses_the_existing_column_id() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let physical_region_id = env.default_physical_region_id(); + let data_region_id = utils::to_data_region_id(physical_region_id); + + env.data_region() + .add_columns( + physical_region_id, + vec![tag_column("tag2")], + IndexOptions::Inverted, + ) + .await + .unwrap(); + let tag2_id = env + .mito() + .get_metadata(data_region_id) + .await + .unwrap() + .column_by_name("tag2") + .unwrap() + .column_id; + + env.data_region() + .add_columns( + physical_region_id, + vec![tag_column("tag2"), tag_column("tag3")], + IndexOptions::Inverted, + ) + .await + .unwrap(); + + let metadata = env.mito().get_metadata(data_region_id).await.unwrap(); + assert_eq!(metadata.column_by_name("tag2").unwrap().column_id, tag2_id); + assert_eq!( + metadata.column_by_name("tag3").unwrap().column_id, + tag2_id + 1 + ); + } + + // Adding only columns that already exist must not submit an ALTER at all. + #[tokio::test] + async fn test_add_columns_skips_the_alter_when_all_columns_exist() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let physical_region_id = env.default_physical_region_id(); + let data_region_id = utils::to_data_region_id(physical_region_id); + + env.data_region() + .add_columns( + physical_region_id, + vec![tag_column("tag2")], + IndexOptions::Inverted, + ) + .await + .unwrap(); + let metadata = env.mito().get_metadata(data_region_id).await.unwrap(); + let version = metadata.schema_version; + + env.data_region() + .add_columns( + physical_region_id, + vec![tag_column("tag2")], + IndexOptions::Inverted, + ) + .await + .unwrap(); + + let metadata = env.mito().get_metadata(data_region_id).await.unwrap(); + assert_eq!(metadata.schema_version, version); + assert_eq!( + metadata + .column_metadatas + .iter() + .filter(|column| column.column_schema.name == "tag2") + .count(), + 1 + ); + } } diff --git a/src/metric-engine/src/engine.rs b/src/metric-engine/src/engine.rs index 376d9f4b564..ea307a21053 100644 --- a/src/metric-engine/src/engine.rs +++ b/src/metric-engine/src/engine.rs @@ -25,7 +25,7 @@ mod put; mod read; mod region_metadata; mod staging; -mod state; +pub(crate) mod state; mod sync; use std::any::Any; @@ -59,8 +59,8 @@ use store_api::storage::{RegionId, ScanRequest, SequenceNumber}; use crate::config::EngineConfig; use crate::data_region::DataRegion; use crate::error::{ - self, Error, Result, StartRepeatedTaskSnafu, UnsupportedRegionRequestSnafu, - UnsupportedRemapManifestsRequestSnafu, + self, Error, PhysicalRegionNotFoundSnafu, Result, StartRepeatedTaskSnafu, + UnsupportedRegionRequestSnafu, UnsupportedRemapManifestsRequestSnafu, }; use crate::metadata_region::MetadataRegion; use crate::repeated_task::FlushMetadataRegionTask; @@ -390,20 +390,34 @@ impl RegionEngine for MetricEngine { role: RegionRole, sent_at: Instant, deadline: Instant, + candidate_lease_id: Option, ) -> Result { // The metadata region goes first. If it ignores the grant, the data region, whose role the // engine reports, must not take it either. A region that is not found is ignored. - for x in [ - utils::to_metadata_region_id(region_id), - utils::to_data_region_id(region_id), - ] { - match self - .inner - .mito - .set_region_role_with_lease(x, role, sent_at, deadline) - { + let metadata_region_id = utils::to_metadata_region_id(region_id); + let data_region_id = utils::to_data_region_id(region_id); + for x in [metadata_region_id, data_region_id] { + match self.inner.mito.set_region_role_with_lease( + x, + role, + sent_at, + deadline, + candidate_lease_id, + ) { Ok(true) => {} - Ok(false) => return Ok(false), + Ok(false) => { + // If the metadata region ignored the grant, the data region is not handed it, so + // record it for the data region. If the data region ignored it, the data region + // has already recorded it. + if x == metadata_region_id { + self.inner.mito.record_granted_lease( + data_region_id, + sent_at, + candidate_lease_id, + ); + } + return Ok(false); + } Err(e) if e.status_code() != StatusCode::RegionNotFound => return Err(e), Err(_) => {} } @@ -567,6 +581,20 @@ impl MetricEngine { self.inner.put_regions_batch(requests).await } + /// Returns the physical region's logical regions from the loaded engine state. + /// Unlike `logical_regions`, this does not read the metadata region and remains usable + /// while a partially completed drop or close is being retried. + pub fn loaded_logical_regions(&self, physical_region_id: RegionId) -> Vec { + self.inner + .state + .read() + .unwrap() + .physical_region_states() + .get(&utils::to_data_region_id(physical_region_id)) + .map(|state| state.logical_regions().iter().copied().collect()) + .unwrap_or_default() + } + /// Returns all logical regions associated with the physical region. pub async fn logical_regions(&self, physical_region_id: RegionId) -> Result> { self.inner @@ -608,7 +636,7 @@ impl MetricEngine { } /// Shared definitions must agree before CREATE/ALTER mutates either region. -fn validate_shared_column( +pub(crate) fn validate_shared_column( region_id: store_api::storage::RegionId, existing: &store_api::metadata::ColumnMetadata, proposed: &store_api::metadata::ColumnMetadata, @@ -647,6 +675,42 @@ struct MetricEngineInner { flush_task: RepeatedTask, } +impl MetricEngineInner { + /// Adds columns to the physical region while holding its column-add lock. + /// + /// Column ids are derived from the metadata of the physical region, so requests + /// on logical regions that share one physical region must not allocate ids at + /// the same time, otherwise two different columns could get one id. + async fn add_physical_columns( + &self, + physical_region_id: RegionId, + columns: Vec, + index_options: IndexOptions, + ) -> Result<()> { + if columns.is_empty() { + return Ok(()); + } + + let data_region_id = utils::to_data_region_id(physical_region_id); + let column_add_lock = { + let state = self.state.read().unwrap(); + state + .physical_region_states() + .get(&data_region_id) + .with_context(|| PhysicalRegionNotFoundSnafu { + region_id: data_region_id, + })? + .column_add_lock() + .clone() + }; + + let _guard = column_add_lock.lock().await; + self.data_region + .add_columns(physical_region_id, columns, index_options) + .await + } +} + #[cfg(test)] mod test { use std::assert_matches; @@ -658,6 +722,7 @@ mod test { use common_wal::options::{KafkaWalOptions, WalOptions}; use mito2::sst::location::region_dir_from_table_dir; use mito2::test_util::{kafka_log_store_factory, prepare_test_for_kafka_log_store}; + use rstest::rstest; use store_api::metric_engine_consts::PHYSICAL_TABLE_METADATA_KEY; use store_api::mito_engine_options::WAL_OPTIONS_KEY; use store_api::region_request::{ @@ -920,22 +985,54 @@ mod test { assert!(!object_store.exists(&data_region_dir).await.unwrap()); } + const LEASE: std::time::Duration = std::time::Duration::from_secs(10); + + /// Asserts that both the metadata and the data region of `physical_region_id` recorded + /// `expected` as the candidate lease id of their latest lease. + fn assert_candidate_lease_id_recorded( + mito: &MitoEngine, + physical_region_id: RegionId, + expected: Option, + ) { + for region_id in [ + utils::to_metadata_region_id(physical_region_id), + utils::to_data_region_id(physical_region_id), + ] { + assert_eq!(mito.granted_candidate_lease_id(region_id), expected); + } + } + + /// The region that ignores a Leader grant is the one downgraded after the heartbeat is sent; + /// the other one is downgraded before it. + #[rstest] + #[case::metadata_region_ignores( + utils::to_data_region_id, + utils::to_metadata_region_id, + RegionRole::Follower + )] + #[case::data_region_ignores( + utils::to_metadata_region_id, + utils::to_data_region_id, + RegionRole::Leader + )] #[tokio::test] - async fn test_leader_grant_ignored_by_metadata_region_leaves_data_region() { + async fn test_leader_grant_ignored_by_one_region( + #[case] downgraded_before_sent: fn(RegionId) -> RegionId, + #[case] downgraded_after_sent: fn(RegionId) -> RegionId, + #[case] metadata_role: RegionRole, + ) { let env = TestEnv::new().await; env.init_metric_region().await; let (engine, mito) = (env.metric(), env.mito()); let physical_region_id = env.default_physical_region_id(); - let lease = std::time::Duration::from_secs(10); - // The data region is downgraded before the heartbeat is sent, the metadata region after it. let downgrade = |region_id| { mito.set_region_role_state_gracefully(region_id, SettableRegionRoleState::Follower) }; - downgrade(utils::to_data_region_id(physical_region_id)) + downgrade(downgraded_before_sent(physical_region_id)) .await .unwrap(); let sent_at = Instant::now(); - downgrade(utils::to_metadata_region_id(physical_region_id)) + downgrade(downgraded_after_sent(physical_region_id)) .await .unwrap(); @@ -945,17 +1042,45 @@ mod test { physical_region_id, RegionRole::Leader, sent_at, - sent_at + lease, + sent_at + LEASE, + Some(7), ) .unwrap() }; assert!(!grant(sent_at)); assert_eq!(engine.role(physical_region_id), Some(RegionRole::Follower)); + assert_eq!( + mito.role(utils::to_metadata_region_id(physical_region_id)), + Some(metadata_role) + ); + // Both regions record the grant, including the one that is not handed it. + assert_candidate_lease_id_recorded(&mito, physical_region_id, Some(7)); assert!(grant(Instant::now())); assert_eq!(engine.role(physical_region_id), Some(RegionRole::Leader)); } + #[tokio::test] + async fn test_candidate_lease_id_reaches_metadata_and_data_regions() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let (engine, mito) = (env.metric(), env.mito()); + let physical_region_id = env.default_physical_region_id(); + let sent_at = Instant::now(); + + engine + .set_region_role_with_lease( + physical_region_id, + RegionRole::Follower, + sent_at, + sent_at + LEASE, + Some(7), + ) + .unwrap(); + + assert_candidate_lease_id_recorded(&mito, physical_region_id, Some(7)); + } + #[tokio::test] async fn test_role() { let env = TestEnv::new().await; diff --git a/src/metric-engine/src/engine/alter.rs b/src/metric-engine/src/engine/alter.rs index e940fa4cbbb..a5f3ec4a4d4 100644 --- a/src/metric-engine/src/engine/alter.rs +++ b/src/metric-engine/src/engine/alter.rs @@ -29,8 +29,8 @@ use validate::validate_alter_region_requests; use crate::engine::MetricEngineInner; use crate::error::{ - AddingFieldColumnSnafu, LogicalRegionNotFoundSnafu, PhysicalRegionNotFoundSnafu, Result, - SerializeColumnMetadataSnafu, UnexpectedRequestSnafu, + AddingFieldColumnSnafu, LogicalRegionNotFoundSnafu, MissingPhysicalColumnSnafu, + PhysicalRegionNotFoundSnafu, Result, SerializeColumnMetadataSnafu, UnexpectedRequestSnafu, }; use crate::utils::{append_manifest_info, encode_manifest_info_to_extensions, to_data_region_id}; @@ -163,8 +163,7 @@ impl MetricEngineInner { ); } - self.data_region - .add_columns(data_region_id, new_columns_to_add, index_options) + self.add_physical_columns(physical_region_id, new_columns_to_add, index_options) .await?; let physical_columns = self.data_region.physical_columns(data_region_id).await?; @@ -173,28 +172,39 @@ impl MetricEngineInner { .map(|metadata| (metadata.column_schema.name.as_str(), metadata)) .collect::>(); - let logical_region_columns = requests.iter().map(|(region_id, request)| { + // The physical region now has every requested column, either because this call + // added it or because it was already there with the same definition. + let mut logical_region_columns = Vec::with_capacity(requests.len()); + for (region_id, request) in &requests { let AlterKind::AddColumns { columns } = &request.kind else { unreachable!() }; - ( - *region_id, - columns - .iter() - .map(|col| { - let column_name = col.column_metadata.column_schema.name.as_str(); - let column_metadata = *physical_schema_map.get(column_name).unwrap(); - (column_name, column_metadata) - }) - .collect::>(), - ) - }); + let mut column_metadata = HashMap::with_capacity(columns.len()); + for col in columns { + let column_name = col.column_metadata.column_schema.name.as_str(); + let metadata = + physical_schema_map + .get(column_name) + .copied() + .with_context(|| MissingPhysicalColumnSnafu { + name: column_name, + region_id: physical_region_id, + })?; + column_metadata.insert(column_name, metadata); + } + logical_region_columns.push((*region_id, column_metadata)); + } - let new_add_columns = new_column_indices.keys().map(|name| { - // Safety: previous steps ensure the physical region exist - let column_metadata = *physical_schema_map.get(name).unwrap(); - (name.to_string(), column_metadata.clone()) - }); + let mut new_add_columns = Vec::with_capacity(new_column_indices.len()); + for name in new_column_indices.keys() { + let column_metadata = physical_schema_map.get(name).copied().with_context(|| { + MissingPhysicalColumnSnafu { + name: name.to_string(), + region_id: physical_region_id, + } + })?; + new_add_columns.push((name.to_string(), column_metadata.clone())); + } // The data region has the new columns, whatever happens to the metadata write. self.state @@ -205,7 +215,11 @@ impl MetricEngineInner { // Writes logical regions metadata to metadata region let result = self .metadata_region - .add_logical_regions(physical_region_id, false, logical_region_columns) + .add_logical_regions( + physical_region_id, + false, + logical_region_columns.into_iter(), + ) .await; // A failed write may still be applied, e.g. when the region lease expires after the WAL // append, so the cached logical columns are dropped either way. @@ -284,6 +298,9 @@ mod test { use common_meta::ddl::test_util::assert_column_name_and_id; use common_meta::ddl::utils::{parse_column_metadatas, parse_manifest_infos_from_extensions}; use common_query::prelude::{greptime_timestamp, greptime_value}; + use datatypes::prelude::ConcreteDataType; + use datatypes::schema::ColumnSchema; + use store_api::metadata::ColumnMetadata; use store_api::metric_engine_consts::ALTER_PHYSICAL_EXTENSION_KEY; use store_api::region_engine::RegionEngine; use store_api::region_request::{ @@ -292,7 +309,9 @@ mod test { use store_api::storage::RegionId; use store_api::storage::consts::ReservedColumnId; + use crate::engine::IndexOptions; use crate::test_util::{TestEnv, alter_logical_region_request, create_logical_region_request}; + use crate::utils::to_data_region_id; async fn check_alter_physical_skip_wal( engine: &super::MetricEngineInner, @@ -442,4 +461,109 @@ mod test { ], ); } + + // Column ids are derived from the current metadata, so concurrent requests on + // logical regions sharing one physical region must not hand the same id to + // different columns. + #[tokio::test] + async fn test_add_physical_columns_concurrently_assigns_distinct_ids() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let physical_region_id = env.default_physical_region_id(); + let data_region_id = to_data_region_id(physical_region_id); + let tag = |name: &str| ColumnMetadata { + column_id: 0, + semantic_type: SemanticType::Tag, + column_schema: ColumnSchema::new(name, ConcreteDataType::string_datatype(), false), + }; + + let inner = env.metric().inner.clone(); + let (first, second) = tokio::join!( + inner.add_physical_columns( + physical_region_id, + vec![tag("tag4")], + IndexOptions::Inverted, + ), + inner.add_physical_columns( + physical_region_id, + vec![tag("tag5")], + IndexOptions::Inverted, + ), + ); + first.unwrap(); + second.unwrap(); + + let metadata = env.mito().get_metadata(data_region_id).await.unwrap(); + assert_ne!( + metadata.column_by_name("tag4").unwrap().column_id, + metadata.column_by_name("tag5").unwrap().column_id + ); + } + + // A request that Mito accepted but that never wrote the logical metadata leaves the + // physical column installed while the cache still misses it. The retry must reuse the + // installed column id and repair both the cache and the logical metadata. + #[tokio::test] + async fn test_alter_logical_region_after_cancelled_physical_alter() { + let env = TestEnv::new().await; + env.init_metric_region().await; + let physical_region_id = env.default_physical_region_id(); + let logical_region_id = env.default_logical_region_id(); + let data_region_id = to_data_region_id(physical_region_id); + + env.data_region() + .add_columns( + physical_region_id, + vec![ColumnMetadata { + column_id: 0, + semantic_type: SemanticType::Tag, + column_schema: ColumnSchema::new( + "tag1", + ConcreteDataType::string_datatype(), + false, + ), + }], + IndexOptions::Inverted, + ) + .await + .unwrap(); + let installed_id = env + .mito() + .get_metadata(data_region_id) + .await + .unwrap() + .column_by_name("tag1") + .unwrap() + .column_id; + + env.metric() + .handle_batch_ddl_requests(BatchRegionDdlRequest::Alter(vec![( + logical_region_id, + alter_logical_region_request(&["tag1"]), + )])) + .await + .unwrap(); + + let metadata = env.mito().get_metadata(data_region_id).await.unwrap(); + assert_eq!( + metadata.column_by_name("tag1").unwrap().column_id, + installed_id + ); + assert_eq!( + env.metadata_region() + .column_semantic_type(physical_region_id, logical_region_id, "tag1") + .await + .unwrap(), + Some(SemanticType::Tag) + ); + + let engine = env.metric(); + let state = engine.inner.state.read().unwrap(); + let physical_columns = state + .physical_region_states() + .get(&data_region_id) + .unwrap() + .physical_columns(); + assert!(physical_columns.contains_key("tag1")); + } } diff --git a/src/metric-engine/src/engine/create.rs b/src/metric-engine/src/engine/create.rs index 1061565a80e..91121cb457f 100644 --- a/src/metric-engine/src/engine/create.rs +++ b/src/metric-engine/src/engine/create.rs @@ -45,9 +45,9 @@ use crate::engine::create::extract_new_columns::extract_new_columns; use crate::engine::options::{PhysicalRegionOptions, set_data_region_options}; use crate::error::{ ColumnTypeMismatchSnafu, ConflictRegionOptionSnafu, CreateMitoRegionSnafu, - InternalColumnOccupiedSnafu, InvalidMetadataSnafu, MissingRegionOptionSnafu, - MultipleFieldColumnSnafu, NoFieldColumnSnafu, ParseRegionIdSnafu, PhysicalRegionNotFoundSnafu, - Result, SerializeColumnMetadataSnafu, UnexpectedRequestSnafu, + InternalColumnOccupiedSnafu, InvalidMetadataSnafu, MissingPhysicalColumnSnafu, + MissingRegionOptionSnafu, MultipleFieldColumnSnafu, NoFieldColumnSnafu, ParseRegionIdSnafu, + PhysicalRegionNotFoundSnafu, Result, SerializeColumnMetadataSnafu, UnexpectedRequestSnafu, }; use crate::metrics::PHYSICAL_REGION_COUNT; use crate::utils::{ @@ -289,8 +289,7 @@ impl MetricEngineInner { }; // TODO(weny): we dont need to pass a mutable new_columns here. - self.data_region - .add_columns(data_region_id, new_columns, index_option) + self.add_physical_columns(physical_region_id, new_columns, index_option) .await?; let physical_columns = self.data_region.physical_columns(data_region_id).await?; @@ -302,28 +301,35 @@ impl MetricEngineInner { .iter() .map(|(region_id, _)| *region_id) .collect::>(); - let logical_region_columns = requests.iter().map(|(region_id, request)| { - ( - *region_id, - request - .column_metadatas - .iter() - .map(|metadata| { - // Safety: previous steps ensure the physical region exist - let column_metadata = *physical_schema_map - .get(metadata.column_schema.name.as_str()) - .unwrap(); - (metadata.column_schema.name.as_str(), column_metadata) - }) - .collect::>(), - ) - }); + // The physical region now has every requested column, either because this call + // added it or because it was already there with the same definition. + let mut logical_region_columns = Vec::with_capacity(requests.len()); + for (region_id, request) in &requests { + let mut column_metadata = HashMap::with_capacity(request.column_metadatas.len()); + for metadata in &request.column_metadatas { + let name = metadata.column_schema.name.as_str(); + let physical_metadata = + physical_schema_map.get(name).copied().with_context(|| { + MissingPhysicalColumnSnafu { + name, + region_id: physical_region_id, + } + })?; + column_metadata.insert(name, physical_metadata); + } + logical_region_columns.push((*region_id, column_metadata)); + } - let new_add_columns = new_column_indices.keys().map(|name| { - // Safety: previous steps ensure the physical region exist - let column_metadata = *physical_schema_map.get(name).unwrap(); - (name.to_string(), column_metadata.clone()) - }); + let mut new_add_columns = Vec::with_capacity(new_column_indices.len()); + for name in new_column_indices.keys() { + let column_metadata = physical_schema_map.get(name).copied().with_context(|| { + MissingPhysicalColumnSnafu { + name: name.to_string(), + region_id: physical_region_id, + } + })?; + new_add_columns.push((name.to_string(), column_metadata.clone())); + } // The data region has the new columns, whatever happens to the metadata write. self.state .write() @@ -337,7 +343,7 @@ impl MetricEngineInner { // Writes logical regions metadata to metadata region self.metadata_region - .add_logical_regions(physical_region_id, true, logical_region_columns) + .add_logical_regions(physical_region_id, true, logical_region_columns.into_iter()) .await?; self.state diff --git a/src/metric-engine/src/engine/put.rs b/src/metric-engine/src/engine/put.rs index eac38d0ab39..9b9295f02cd 100644 --- a/src/metric-engine/src/engine/put.rs +++ b/src/metric-engine/src/engine/put.rs @@ -2379,7 +2379,13 @@ mod tests { let at = |n| start + n * Duration::from_secs(10); let grant = |n| { engine - .set_region_role_with_lease(physical_region_id, RegionRole::Leader, start, at(n)) + .set_region_role_with_lease( + physical_region_id, + RegionRole::Leader, + start, + at(n), + None, + ) .unwrap() }; let expire = |n| mito.freeze_lease_clock(at(n)); diff --git a/src/metric-engine/src/engine/state.rs b/src/metric-engine/src/engine/state.rs index e0f407b919b..0333bd75410 100644 --- a/src/metric-engine/src/engine/state.rs +++ b/src/metric-engine/src/engine/state.rs @@ -43,6 +43,15 @@ pub struct PhysicalRegionState { primary_key_encoding: PrimaryKeyEncoding, options: PhysicalRegionOptions, time_index_unit: TimeUnit, + /// Serializes adding columns to this physical region: the column existence + /// check, the column id allocation and the physical ALTER that installs the + /// columns. Column ids are derived from the current metadata, so two + /// concurrent add-column requests on logical regions sharing this physical + /// region could otherwise assign the same id to different columns. + /// + /// Callers clone the [`Arc`] out of the state and hold it across the + /// physical alter; see [`crate::data_region::DataRegion::add_columns`]. + column_add_lock: Arc>, } impl PhysicalRegionState { @@ -67,9 +76,15 @@ impl PhysicalRegionState { primary_key_encoding, options, time_index_unit, + column_add_lock: Arc::new(tokio::sync::Mutex::new(())), } } + /// Returns the lock that serializes adding columns to this physical region. + pub fn column_add_lock(&self) -> &Arc> { + &self.column_add_lock + } + /// Returns a reference to the logical region ids. pub fn logical_regions(&self) -> &HashSet { &self.logical_regions @@ -263,6 +278,7 @@ impl MetricEngineState { for logical_region in logical_regions { self.logical_regions.remove(logical_region); + self.logical_columns.remove(logical_region); } self.physical_regions.remove(&physical_region_id); Ok(()) diff --git a/src/metric-engine/src/error.rs b/src/metric-engine/src/error.rs index f9d61d6af31..f326429e2e0 100644 --- a/src/metric-engine/src/error.rs +++ b/src/metric-engine/src/error.rs @@ -255,6 +255,18 @@ pub enum Error { location: Location, }, + #[snafu(display( + "Column {} is missing from physical region {} after adding columns", + name, + region_id + ))] + MissingPhysicalColumn { + name: String, + region_id: RegionId, + #[snafu(implicit)] + location: Location, + }, + #[snafu(display("Table id count mismatch. Expect {}, got {}", expected, actual))] TableIdCountMismatch { expected: usize, @@ -451,6 +463,10 @@ impl ErrorExt for Error { ColumnNotFound { .. } => StatusCode::TableColumnNotFound, + // The physical region was read twice around the alter and lost a column in + // between, so the failure is an inconsistent server state, not a bad request. + MissingPhysicalColumn { .. } => StatusCode::Unexpected, + CreateMitoRegion { source, .. } | OpenMitoRegion { source, .. } | CleanUpMitoRegion { source, .. } diff --git a/src/mito2/AGENTS.md b/src/mito2/AGENTS.md index 680dd2e4821..8699365f305 100644 --- a/src/mito2/AGENTS.md +++ b/src/mito2/AGENTS.md @@ -36,6 +36,7 @@ snapshot isolation). It implements the `RegionEngine` trait from `store-api`. | `schedule` | `src/mito2/src/schedule/` | Local/remote background job scheduling | | `remap_manifest` | `src/mito2/src/remap_manifest.rs` | Manifest path remapping for region copy/migration | | `config` | `src/mito2/src/config.rs` | `MitoConfig` tuning knobs | +| `tail_replay` | `src/mito2/src/tail_replay.rs`, `src/mito2/src/worker/handle_tail_replay.rs` | Background replay of pending WAL tails into SSTs, published with one manifest edit | | `temp_file_sweeper` | `src/mito2/src/temp_file_sweeper.rs` | Reclaims temporary files a crashed writer left in a region, triggered when the region becomes a writable leader or a create opens an existing region (under the `hdfs-object-store` feature) | | `test_util` | `src/mito2/src/test_util.rs` | `TestEnv` and builders (under the `test` feature) | @@ -118,7 +119,9 @@ Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_tes arbitrary partial groups: readers and index row IDs rely on this layout. - Sequence numbers are strictly increasing per region; dedup and snapshot reads depend on this. Do not change assignment lightly. -- Manifest version is monotonic — never reset or skip it. +- Manifest version is monotonic — never reset or skip it. Staging takes no version: + its actions stay in memory (`manifest/manager.rs`) and are committed once, as the + next version, when staging ends (`MitoRegion::end_staging_locked`). - In regions whose writes require a lease and whose WAL is the object store WAL, each manifest version has one fixed object name, created conditionally. A late create can succeed again once checkpoint cleanup frees the name, so a commit @@ -126,19 +129,36 @@ Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_tes - In those regions a commit after the region's Leader deadline sends nothing, and a version conflict fences the region (`ManifestContext::fence_for_reload`): no lease or catchup promotes it until the worker swaps in an instance reloaded from the new manifest - (`worker/handle_reload.rs`). Never await a worker sender while holding a manifest guard. + (`worker/handle_reload.rs`). A tail replay round that fails while the storage holds a newer + manifest fences a writable leader the same way. Never await a worker sender while holding a + manifest guard. - Pending tails (`RegionManifest::pending_tails`) are WAL ranges a region still has to recover; the flushed entry id never prunes them. Opening a region replays its - WAL after `RegionManifest::wal_replay_start`. The WAL releases at open and after - a flush stay below this node's tails (`cap_wal_release`); truncate, drop, offline - cleanup and catchup release without that cap. Handoff floors are dropped once the - flushed entry id reaches them, in `RegionManifestBuilder::try_build`. While a + WAL after `RegionManifest::wal_replay_start`. The WAL releases at open, after + a flush, and on close stay below this node's tails (`cap_wal_release`); truncate, + drop, offline cleanup and catchup release without that cap. Handoff floors are + dropped once the flushed entry id reaches them, in + `RegionManifestBuilder::try_build`. While a region has pending tails, scans and memtable/SST stats fail with `RegionPendingTails`, and so do truncate, entering staging, a repartition flush, a close with `flush_on_close`, and every ALTER other than adding nullable columns without a default or changing options; compaction pauses. These checks run before any region state changes. Full-listing GC skips a region that has pending tails or whose manifest `has_update()` reports newer, or fails to check. +- A leader that is writable, altering or editing replays its pending tails in one + background task (`tail_replay.rs`), started by the worker when a lease, catchup or + role change makes the region writable, or when it registers a region created or opened + as a writable leader. Each tail goes into its own SSTs, since an + SST can give all its rows its largest sequence. The task commits outside the edit queue + (`ManifestContext::update_manifest_for_tails`) and always reports its end, which + lets the worker start the next replay or reload a fenced region. The edit that records + tails must raise the committed sequence to theirs, or writes accepted during the replay + lose to replayed rows. This node's later entry ids must be above every tail's `upto`, and + its unflushed entries at or below one must be in its own tails, or the tail commit's + flushed entry id skips them. A node bounds its replays' concurrency and SST writes apart + from flushes, and their chunk memory within the global write buffer (`TailReplayLimits`). + Replayed SSTs build indexes as flushed ones (`IndexBuildType::Flush`), under + `create_on_flush`. - Lock ordering: take the manifest lock before updating `version_control`; the reverse deadlocks against concurrent flush/compaction. - All region I/O runs on tokio workers; never `block_on` inside a worker. @@ -149,6 +169,16 @@ Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_tes `All` removes all data; `ByTimeRanges` removes fully contained files. Other regions/readers still protect the shared objects. Stale compaction results must not reinstate truncated inputs. +- Manifest and runtime file views share immutable `FileMetaRef` values. The engine-local + metadata interner compares complete metadata before sharing; file identity alone is insufficient. + Partition expressions are shared by table and complete expression content, so historical + SST partition rules remain distinct after repartition. + Interned references reclaim entries on final release, including delayed readers, + without a maintenance task or new writes. Hash table capacity remains allocated + until the interner is dropped. + Manifest consumers receive the interner directly. Scrape-time entry counts use weak + references and do not add maintenance tasks or retain engine storage. + Index replacement preserves old readers and updates the lifetime guard to the newest metadata. - Local compaction units use the snapshot-free `CompactionContext`. Retaining a full `CompactionRegion` in a unit pins other units' inputs and delays SST purge. - Candidate waiting history is owned by the region scheduler (`scheduler/history.rs`). @@ -163,9 +193,20 @@ Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_tes - In flat batches, equal `__primary_key` dictionary keys imply the same series, but not the reverse: `wrap_pk_binary_to_dict` gives every row its own key. Compare key bytes when grouping rows by series. -- Staging regions never flush: a flush would move rows into the staging manifest and - release their WAL, losing them if staging does not end successfully. Every flush - trigger in `worker/handle_flush.rs` must skip them, and `handle_close` leaves staging first. +- Staging regions, ending staging included, never flush: a flush would stage its SST and + release the rows' WAL, losing them if staging does not end successfully. Every flush + trigger in `worker/handle_flush.rs` must skip them: background triggers test `is_writable()` + and `is_staging()`, a flush request `is_flushable()` and `is_staging()` (`is_writable()` and + `is_flushable()` are false while ending staging). `handle_close` leaves staging first. + Tail replays never run for them: entering staging fails while tails are pending. +- Direct SST imports without a staging binding cannot publish into staging. + `worker/handle_manifest.rs` seals edit admission while EnterStaging drains accepted + edits. Scheduler-owned DDLs resume through execution, bypassing initial admission. +- The latest lease each region received (`GrantedLeases` in `lease.rs`) lives outside the + region instance, so a reopen during catchup keeps it. Every path that removes a region from + a worker's region map must also remove its record: `RegionWorkerLoop::remove_region` + (`worker/handle_close.rs`, also used after a close-time flush), `drop_region` + (`worker/handle_drop.rs`) and `RegionWorkerLoop::clean` (`worker.rs`). ## Maintenance contract diff --git a/src/mito2/Cargo.toml b/src/mito2/Cargo.toml index ba37766d309..f5c6c4d53d4 100644 --- a/src/mito2/Cargo.toml +++ b/src/mito2/Cargo.toml @@ -59,6 +59,7 @@ greptime-proto.workspace = true humantime.workspace = true humantime-serde.workspace = true index.workspace = true +intern-arc.workspace = true itertools.workspace = true lazy_static = "1.4" log-store = { workspace = true } diff --git a/src/mito2/src/access_layer.rs b/src/mito2/src/access_layer.rs index 3a572b395bd..3b73271e302 100644 --- a/src/mito2/src/access_layer.rs +++ b/src/mito2/src/access_layer.rs @@ -26,12 +26,14 @@ use object_store::services::Fs; use object_store::util::{join_dir, with_instrument_layers}; use object_store::{ATOMIC_WRITE_DIR, ErrorKind, OLD_ATOMIC_WRITE_DIR, ObjectStore}; use parquet::file::metadata::PageIndexPolicy; +use prometheus::HistogramVec; use smallvec::SmallVec; use snafu::ResultExt; use store_api::metadata::RegionMetadataRef; use store_api::region_request::PathType; use store_api::sst_entry::StorageSstEntry; use store_api::storage::{FileId, RegionId, SequenceNumber}; +use strum::IntoStaticStr; use crate::cache::file_cache::{FileCacheRef, FileType, IndexKey}; use crate::cache::write_cache::SstUploadRequest; @@ -40,7 +42,7 @@ use crate::config::{BloomFilterConfig, FulltextIndexConfig, IndexConfig, Inverte use crate::error::{ CleanDirSnafu, DeleteIndexSnafu, DeleteIndexesSnafu, DeleteSstsSnafu, OpenDalSnafu, Result, }; -use crate::metrics::{COMPACTION_STAGE_ELAPSED, FLUSH_ELAPSED}; +use crate::metrics::{COMPACTION_STAGE_ELAPSED, FLUSH_ELAPSED, TAIL_REPLAY_ELAPSED}; use crate::read::FlatSource; use crate::region::options::IndexOptions; use crate::sst::file::{FileHandle, RegionFileId, RegionIndexId}; @@ -51,7 +53,7 @@ use crate::sst::index::registry::IndexRegistry; use crate::sst::index::{IndexerBuilderImpl, build_index_from_parquet_bytes}; use crate::sst::location::{self, region_dir_from_table_dir}; use crate::sst::parquet::reader::ParquetReaderBuilder; -use crate::sst::parquet::writer::ParquetWriter; +use crate::sst::parquet::writer::{ParquetWriter, SstWriteContext}; use crate::sst::parquet::{SstInfo, WriteOptions}; use crate::sst::{DEFAULT_WRITE_CONCURRENCY, FormatType}; @@ -60,12 +62,14 @@ pub type AccessLayerRef = Arc; pub type SstInfoArray = SmallVec<[SstInfo; 2]>; /// Write operation type. -#[derive(Eq, PartialEq, Debug)] +#[derive(Eq, PartialEq, Debug, Clone, Copy, IntoStaticStr)] pub enum WriteType { /// Writes from flush Flush, /// Writes from compaction. Compaction, + /// Writes from the replay of pending WAL tails. + TailReplay, } #[derive(Debug)] @@ -104,47 +108,28 @@ impl Metrics { } pub(crate) fn observe(self) { - match self.write_type { - WriteType::Flush => { - FLUSH_ELAPSED - .with_label_values(&["iter_source"]) - .observe(self.iter_source.as_secs_f64()); - FLUSH_ELAPSED - .with_label_values(&["write_batch"]) - .observe(self.write_batch.as_secs_f64()); - FLUSH_ELAPSED - .with_label_values(&["update_index"]) - .observe(self.update_index.as_secs_f64()); - FLUSH_ELAPSED - .with_label_values(&["upload_parquet"]) - .observe(self.upload_parquet.as_secs_f64()); - FLUSH_ELAPSED - .with_label_values(&["upload_puffin"]) - .observe(self.upload_puffin.as_secs_f64()); - if !self.compact_memtable.is_zero() { - FLUSH_ELAPSED - .with_label_values(&["compact_memtable"]) - .observe(self.upload_puffin.as_secs_f64()); - } - } - WriteType::Compaction => { - COMPACTION_STAGE_ELAPSED - .with_label_values(&["iter_source"]) - .observe(self.iter_source.as_secs_f64()); - COMPACTION_STAGE_ELAPSED - .with_label_values(&["write_batch"]) - .observe(self.write_batch.as_secs_f64()); - COMPACTION_STAGE_ELAPSED - .with_label_values(&["update_index"]) - .observe(self.update_index.as_secs_f64()); - COMPACTION_STAGE_ELAPSED - .with_label_values(&["upload_parquet"]) - .observe(self.upload_parquet.as_secs_f64()); - COMPACTION_STAGE_ELAPSED - .with_label_values(&["upload_puffin"]) - .observe(self.upload_puffin.as_secs_f64()); - } + let elapsed: &HistogramVec = match self.write_type { + WriteType::Flush => &FLUSH_ELAPSED, + WriteType::Compaction => &COMPACTION_STAGE_ELAPSED, + WriteType::TailReplay => &TAIL_REPLAY_ELAPSED, }; + let stages = [ + ("iter_source", self.iter_source), + ("write_batch", self.write_batch), + ("update_index", self.update_index), + ("upload_parquet", self.upload_parquet), + ("upload_puffin", self.upload_puffin), + ]; + for (stage, duration) in stages { + elapsed + .with_label_values(&[stage]) + .observe(duration.as_secs_f64()); + } + if !self.compact_memtable.is_zero() { + elapsed + .with_label_values(&["compact_memtable"]) + .observe(self.compact_memtable.as_secs_f64()); + } } } @@ -376,18 +361,13 @@ impl AccessLayer { .await } - /// Writes an SST whose output is registered with `context.guard`. - /// - /// The write cache is bypassed, upload concurrency is fixed to one, and - /// the guard runs before either the SST or its index becomes visible. - /// `context.limits` bounds the encoder and footer workspace. - #[cfg(feature = "enterprise")] - pub(crate) async fn write_sst_with_guard( + /// Writes SSTs with explicit ownership and cooperative cancellation. + pub(crate) async fn write_sst_with_context( &self, request: SstWriteRequest, write_opts: &WriteOptions, metrics: &mut Metrics, - context: crate::sst::parquet::writer::SstWriteContext, + context: SstWriteContext, ) -> Result { self.write_sst_inner(request, write_opts, metrics, Some(context)) .await @@ -398,10 +378,12 @@ impl AccessLayer { request: SstWriteRequest, write_opts: &WriteOptions, metrics: &mut Metrics, - context: Option, + context: Option, ) -> Result { - let limits = context.as_ref().map(|context| context.limits); - let guard = context.map(|context| context.guard); + let context = context.unwrap_or_default(); + let limits = context.limits; + let guard = context.guard.clone(); + let cancellation = context.cancellation.clone(); let op_type = request.op_type; let region_id = request.metadata.region_id; let region_metadata = request.metadata.clone(); @@ -413,11 +395,11 @@ impl AccessLayer { }; let sst_info = if let Some(write_cache) = cache_manager.write_cache() - && guard.is_none() + && limits.is_none() { // Write to the write cache. write_cache - .write_and_upload_sst( + .write_and_upload_sst_with_context( request, SstUploadRequest { dest_path_provider: RegionFilePathFactory::new( @@ -428,10 +410,12 @@ impl AccessLayer { }, write_opts, metrics, + context, ) .await? } else { - // Write cache is disabled, or bypassed by an SST write guard. + // The write cache does not enforce SstWriteLimits, so bounded writes + // bypass it. let store = self.object_store.clone(); let path_provider = RegionFilePathFactory::new(self.table_dir.clone(), self.path_type); let puffin_factory = if limits.is_some() { @@ -469,12 +453,13 @@ impl AccessLayer { ) .await .with_file_cleaner(cleaner) - .with_write_concurrency(if guard.is_some() { + .with_write_concurrency(if limits.is_some() { 1 } else { crate::sst::DEFAULT_WRITE_CONCURRENCY }) .with_write_guard(guard) + .with_cancellation(cancellation) .with_memory_limits(limits); match request.sst_write_format { FormatType::PrimaryKey => { @@ -553,6 +538,7 @@ impl AccessLayer { /// /// Builds the SST's indexes from `data` into `sst_info.index_metadata` when /// `index_build_params` is set. + #[allow(clippy::too_many_arguments)] pub(crate) async fn put_sst( &self, data: &bytes::Bytes, @@ -561,7 +547,8 @@ impl AccessLayer { cache_manager: &CacheManagerRef, write_buffer_size: ReadableSize, index_build_params: Option, - ) -> Result { + metrics: &mut Metrics, + ) -> Result<()> { let region_file_id = RegionFileId::new(region_id, sst_info.file_id); if let Some(write_cache) = cache_manager.write_cache() { // Write to cache and upload to remote store @@ -580,6 +567,7 @@ impl AccessLayer { upload_request, write_buffer_size, index_build_params, + metrics, ) .await } else { @@ -603,7 +591,6 @@ impl AccessLayer { cleaner.clean_by_file_id(region_file_id.file_id()).await; return Err(err); } - let mut metrics = Metrics::new(WriteType::Flush); metrics.write_batch = start.elapsed(); if let Some(params) = index_build_params { @@ -618,7 +605,7 @@ impl AccessLayer { build_index_from_parquet_bytes(&builder, region_file_id, data).await; metrics.update_index = start.elapsed(); } - Ok(metrics) + Ok(()) } } @@ -972,6 +959,7 @@ mod tests { &cache_manager, ReadableSize(chunk_size as u64), None, + &mut Metrics::new(WriteType::Flush), ) .await .unwrap(); @@ -1104,6 +1092,104 @@ mod tests { assert_eq!(1, warm.mem_cache_hit); } + #[rstest::rstest] + #[tokio::test] + async fn test_cancelled_write_cleans_rolled_ssts( + #[values(false, true)] enable_write_cache: bool, + #[values(FormatType::Flat, FormatType::PrimaryKey)] format: FormatType, + ) { + use crate::read::FlatSource; + use crate::sst::file::UncommittedSsts; + use crate::test_util::sst_util::{new_record_batch_by_range, sst_region_metadata}; + use tokio_util::sync::CancellationToken; + + let mut env = TestEnv::new().await; + let store = env.init_object_store_manager(); + let local_dir = create_temp_dir("cancelled-sst-cache"); + let local_store = new_fs_store(local_dir.path().to_str().unwrap()); + let write_cache = if enable_write_cache { + Some( + env.create_write_cache(local_store.clone(), ReadableSize::mb(10)) + .await, + ) + } else { + None + }; + let cache = Arc::new(CacheManager::builder().write_cache(write_cache).build()); + let layer = Arc::new(AccessLayer::new( + "cancelled", + PathType::Bare, + store.clone(), + env.get_puffin_manager(), + env.get_intermediate_manager(), + )); + let metadata = Arc::new(sst_region_metadata()); + let uncommitted = + UncommittedSsts::new(metadata.region_id, layer.clone(), Some(cache.clone())); + let cancellation = CancellationToken::new(); + let signal = cancellation.clone(); + let tracked = uncommitted.clone(); + let schema = new_record_batch_by_range(&["a", "b"], 0, 8).schema(); + let source = FlatSource::new_stream( + schema, + Box::pin(async_stream::try_stream! { + for i in 0..64 { + if tracked.num_tracked_files() >= 3 { + signal.cancel(); + } + let tag = format!("{i:04}"); + yield new_record_batch_by_range(&[&tag, "b"], 0, 4096); + } + }), + ); + let result = layer + .write_sst_with_context( + SstWriteRequest { + index_build_inputs: None, + op_type: OperationType::Compact, + metadata, + source, + cache_manager: cache, + max_sequence: None, + sst_write_format: format, + preserve_row_sequence: false, + index_options: Default::default(), + index_config: Default::default(), + inverted_index_config: Default::default(), + fulltext_index_config: Default::default(), + bloom_filter_index_config: Default::default(), + }, + &WriteOptions { + max_file_size: Some(1), + row_group_size: 8, + ..Default::default() + }, + &mut Metrics::new(WriteType::Compaction), + SstWriteContext { + guard: Some(Arc::new(uncommitted.clone())), + cancellation: Some(cancellation), + ..Default::default() + }, + ) + .await; + assert!(matches!( + result, + Err(crate::error::Error::SstWriteCancelled { .. }) + )); + assert!( + uncommitted.num_tracked_files() >= 2, + "must cover files finalized before cancellation" + ); + uncommitted.cleanup_for_test().await.unwrap(); + for object_store in [&store, &local_store] { + let entries = object_store.list_with("/").recursive(true).await.unwrap(); + assert!( + entries.iter().all(|entry| entry.metadata().is_dir()), + "uncommitted files remain: {entries:?}" + ); + } + } + #[tokio::test] async fn test_new_fs_cache_store() { let root = common_test_util::temp_dir::create_temp_dir("fs-cache-store"); diff --git a/src/mito2/src/cache/write_cache.rs b/src/mito2/src/cache/write_cache.rs index 37a4964a20e..ac4d4f6b3b2 100644 --- a/src/mito2/src/cache/write_cache.rs +++ b/src/mito2/src/cache/write_cache.rs @@ -19,16 +19,15 @@ use std::time::{Duration, Instant}; use common_base::readable_size::ReadableSize; use common_telemetry::{debug, info}; -use futures::AsyncWriteExt; use object_store::ObjectStore; -use snafu::ResultExt; +use snafu::{IntoError, ResultExt}; use store_api::storage::RegionId; use tokio::sync::mpsc::{UnboundedSender, unbounded_channel}; +use tokio_util::sync::CancellationToken; use crate::access_layer::{ EncodedSstIndexBuildParams, FilePathProvider, Metrics, OperationType, RegionFilePathFactory, - SstInfoArray, SstWriteRequest, TempFileCleaner, WriteCachePathProvider, WriteType, - new_fs_cache_store, + SstInfoArray, SstWriteRequest, TempFileCleaner, WriteCachePathProvider, new_fs_cache_store, }; use crate::cache::file_cache::{FileCache, FileCacheRef, FileType, IndexKey, IndexValue}; use crate::cache::manifest_cache::ManifestCache; @@ -40,8 +39,9 @@ use crate::sst::index::intermediate::IntermediateManager; use crate::sst::index::puffin_manager::{PuffinManagerFactory, SstPuffinManager}; use crate::sst::index::registry::IndexRegistry; use crate::sst::index::{IndexerBuilderImpl, build_index_from_parquet_bytes}; -use crate::sst::parquet::writer::ParquetWriter; +use crate::sst::parquet::writer::{ParquetWriter, SstWriteContext}; use crate::sst::parquet::{SstInfo, WriteOptions}; +use crate::sst::upload::{self, UploadError}; use crate::sst::{DEFAULT_WRITE_BUFFER_SIZE, DEFAULT_WRITE_CONCURRENCY}; /// Wraps the remote object store used by write cache uploads. @@ -178,6 +178,7 @@ impl WriteCache { /// /// Builds the SST's indexes from `data` when `index_build_params` is set and uploads them /// with the SST. + #[allow(clippy::too_many_arguments)] pub(crate) async fn put_and_upload_sst( &self, data: &bytes::Bytes, @@ -186,10 +187,10 @@ impl WriteCache { upload_request: SstUploadRequest, write_buffer_size: ReadableSize, index_build_params: Option, - ) -> Result { + metrics: &mut Metrics, + ) -> Result<()> { let file_id = sst_info.file_id; let region_file_id = RegionFileId::new(region_id, file_id); - let mut metrics = Metrics::new(WriteType::Flush); // Create index key for the SST file let parquet_key = IndexKey::new(region_id, file_id, FileType::Parquet); @@ -275,7 +276,7 @@ impl WriteCache { .await; return Err(e); } - Ok(metrics) + Ok(()) } /// Returns the intermediate manager of the write cache. @@ -283,13 +284,14 @@ impl WriteCache { &self.intermediate_manager } - /// Writes SST to the cache and then uploads it to the remote object store. - pub(crate) async fn write_and_upload_sst( + /// Preserves write ownership and cancellation through local encoding and upload. + pub(crate) async fn write_and_upload_sst_with_context( &self, write_request: SstWriteRequest, upload_request: SstUploadRequest, write_opts: &WriteOptions, metrics: &mut Metrics, + context: SstWriteContext, ) -> Result { let region_id = write_request.metadata.region_id; let override_sequence = if write_request.preserve_row_sequence { @@ -328,7 +330,9 @@ impl WriteCache { metrics, ) .await - .with_file_cleaner(cleaner); + .with_file_cleaner(cleaner) + .with_write_guard(context.guard) + .with_cancellation(context.cancellation.clone()); let sst_info = match write_request.sst_write_format { crate::sst::FormatType::PrimaryKey => { @@ -357,13 +361,21 @@ impl WriteCache { let op_type = write_request.op_type; let remote_store = &upload_request.remote_store; for sst in &sst_info { + if context + .cancellation + .as_ref() + .is_some_and(|handle| handle.is_cancelled()) + { + err = Some(error::SstWriteCancelledSnafu.build()); + break; + } let parquet_key = IndexKey::new(region_id, sst.file_id, FileType::Parquet); let parquet_path = upload_request .dest_path_provider .build_sst_file_path(RegionFileId::new(region_id, sst.file_id)); let start = Instant::now(); if let Err(e) = self - .upload( + .upload_with_cancellation( parquet_key, &parquet_path, remote_store, @@ -371,6 +383,7 @@ impl WriteCache { op_type, write_buffer_size: write_opts.write_buffer_size, }, + context.cancellation.as_ref(), ) .await { @@ -387,7 +400,7 @@ impl WriteCache { .build_index_file_path(RegionFileId::new(region_id, sst.file_id)); let start = Instant::now(); if let Err(e) = self - .upload( + .upload_with_cancellation( puffin_key, &puffin_path, remote_store, @@ -395,6 +408,7 @@ impl WriteCache { op_type, write_buffer_size: DEFAULT_WRITE_BUFFER_SIZE, }, + context.cancellation.as_ref(), ) .await { @@ -467,10 +481,21 @@ impl WriteCache { upload_path: &str, remote_store: &ObjectStore, options: UploadOptions, + ) -> Result<()> { + self.upload_with_cancellation(index_key, upload_path, remote_store, options, None) + .await + } + + async fn upload_with_cancellation( + &self, + index_key: IndexKey, + upload_path: &str, + remote_store: &ObjectStore, + options: UploadOptions, + cancellation: Option<&CancellationToken>, ) -> Result<()> { let region_id = index_key.region_id; let file_id = index_key.file_id; - let file_type = index_key.file_type; let cache_path = self.file_cache.cache_file_path(index_key); let start = Instant::now(); @@ -480,13 +505,14 @@ impl WriteCache { .stat(&cache_path) .await .context(error::OpenDalSnafu)?; - let reader = self + let mut reader = self .file_cache .local_store() - .reader(&cache_path) + .reader_with(&cache_path) + .chunk(options.write_buffer_size.as_bytes().max(1) as usize) .await .context(error::OpenDalSnafu)? - .into_futures_async_read(0..cached_value.content_length()) + .into_stream(0..cached_value.content_length()) .await .context(error::OpenDalSnafu)?; @@ -499,24 +525,50 @@ impl WriteCache { .chunk(options.write_buffer_size.as_bytes() as usize) .concurrent(DEFAULT_WRITE_CONCURRENCY) .await - .context(error::OpenDalSnafu)? - .into_futures_async_write(); + .context(error::OpenDalSnafu)?; - let bytes_written = - futures::io::copy(reader, &mut writer) + let result = async { + let bytes_written = upload::copy( + &mut reader, + &mut writer, + options.write_buffer_size.as_bytes() as usize, + cancellation, + ) + .await + .map_err(|error| match error { + UploadError::Cancelled => error::SstWriteCancelledSnafu.build(), + UploadError::Read { source } => error::OpenDalSnafu.into_error(source), + UploadError::Write { source } => error::UploadSnafu { + region_id, + file_id, + file_type: index_key.file_type, + } + .into_error(std::io::Error::from(source)), + })?; + if cancellation.is_some_and(CancellationToken::is_cancelled) { + return error::SstWriteCancelledSnafu.fail(); + } + writer + .close() .await + .map_err(std::io::Error::from) .context(error::UploadSnafu { region_id, file_id, - file_type, + file_type: index_key.file_type, })?; - - // Must close to upload all data. - writer.close().await.context(error::UploadSnafu { - region_id, - file_id, - file_type, - })?; + Ok(bytes_written) + } + .await; + let bytes_written = match result { + Ok(bytes_written) => bytes_written, + Err(error) => { + if let Err(abort_error) = writer.abort().await { + common_telemetry::warn!(abort_error; "Failed to abort SST upload, region: {}, file: {}", region_id, file_id); + } + return Err(error); + } + }; UPLOAD_BYTES_TOTAL.inc_by(bytes_written); @@ -633,7 +685,7 @@ mod tests { use store_api::storage::FileId; use super::*; - use crate::access_layer::OperationType; + use crate::access_layer::{OperationType, WriteType}; use crate::cache::file_cache::IndexValue; use crate::cache::test_util::{assert_parquet_metadata_equal, new_fs_store}; use crate::cache::{CacheManager, CacheStrategy}; @@ -647,6 +699,253 @@ mod tests { sst_file_handle_with_file_id, sst_region_metadata, }; + struct CancellingUploadWriter { + inner: object_store::layers::mock::oio::Writer, + cancellation: CancellationToken, + bytes_written: Arc, + aborted: Arc, + } + + impl object_store::layers::mock::oio::Write for CancellingUploadWriter { + async fn write(&mut self, buffer: object_store::Buffer) -> object_store::Result<()> { + let len = buffer.len(); + self.inner.write(buffer).await?; + self.bytes_written + .fetch_add(len, std::sync::atomic::Ordering::Relaxed); + self.cancellation.cancel(); + Ok(()) + } + async fn close( + &mut self, + ) -> object_store::layers::mock::Result { + self.inner.close().await + } + async fn abort(&mut self) -> object_store::Result<()> { + self.aborted + .store(true, std::sync::atomic::Ordering::Relaxed); + self.inner.abort().await + } + } + + #[tokio::test] + async fn test_cancel_upload_aborts_incomplete_object() { + use object_store::layers::mock::MockLayerBuilder; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + + let env = TestEnv::new().await; + let local_store = ObjectStore::new(Memory::default()).unwrap(); + let cancellation = CancellationToken::new(); + let bytes_written = Arc::new(AtomicUsize::new(0)); + let aborted = Arc::new(AtomicBool::new(false)); + let signal = cancellation.clone(); + let bytes = bytes_written.clone(); + let abort = aborted.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + Box::new(CancellingUploadWriter { + inner, + cancellation: signal.clone(), + bytes_written: bytes.clone(), + aborted: abort.clone(), + }) + })) + .build() + .unwrap(); + let remote_store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let cache = env + .create_write_cache(local_store.clone(), ReadableSize::mb(10)) + .await; + let key = IndexKey::new(RegionId::new(1024, 1), FileId::random(), FileType::Parquet); + let data = vec![42; 1024 * 1024]; + local_store + .write(&cache.file_cache.cache_file_path(key), data.clone()) + .await + .unwrap(); + let result = cache + .upload_with_cancellation( + key, + "cancelled.parquet", + &remote_store, + UploadOptions { + op_type: OperationType::Compact, + write_buffer_size: ReadableSize::kb(1), + }, + Some(&cancellation), + ) + .await; + assert!(matches!( + result, + Err(error::Error::SstWriteCancelled { .. }) + )); + assert!(aborted.load(Ordering::Relaxed)); + let bytes = bytes_written.load(Ordering::Relaxed); + assert!( + bytes > 0 && bytes < data.len(), + "wrote {bytes} bytes before cancelling" + ); + assert!(!remote_store.exists("cancelled.parquet").await.unwrap()); + } + + struct FailingUploadWriter { + inner: object_store::layers::mock::oio::Writer, + fail_close: bool, + fail_abort: bool, + successful_writes: usize, + } + + impl object_store::layers::mock::oio::Write for FailingUploadWriter { + async fn write(&mut self, bytes: object_store::Buffer) -> object_store::Result<()> { + if !self.fail_close && self.successful_writes == 1 { + return Err(object_store::Error::new( + object_store::ErrorKind::NotFound, + "injected write failure", + )); + } + self.inner.write(bytes).await?; + self.successful_writes += 1; + Ok(()) + } + + async fn close(&mut self) -> object_store::Result { + Err(object_store::Error::new( + object_store::ErrorKind::NotFound, + "injected close failure", + )) + } + + async fn abort(&mut self) -> object_store::Result<()> { + if self.fail_abort { + return Err(object_store::Error::new( + object_store::ErrorKind::Unexpected, + "injected abort failure", + )); + } + self.inner.abort().await + } + } + + #[rstest::rstest] + #[case::write(false, false)] + #[case::close(true, false)] + #[case::write_and_abort(false, true)] + #[tokio::test] + async fn test_upload_error_preserves_file_context( + #[case] fail_close: bool, + #[case] fail_abort: bool, + ) { + use object_store::layers::mock::{MockLayerBuilder, ObservedWriter, WriterObservations}; + + let env = TestEnv::new().await; + let local_store = ObjectStore::new(Memory::default()).unwrap(); + let observations = Arc::new(std::sync::Mutex::new(WriterObservations::default())); + let observed = observations.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + Box::new(ObservedWriter::new( + Box::new(FailingUploadWriter { + inner, + fail_close, + fail_abort, + successful_writes: 0, + }), + observed.clone(), + )) + })) + .build() + .unwrap(); + let remote_store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let cache = env + .create_write_cache(local_store.clone(), ReadableSize::mb(10)) + .await; + let key = IndexKey::new(RegionId::new(1024, 1), FileId::random(), FileType::Parquet); + local_store + .write(&cache.file_cache.cache_file_path(key), vec![42; 4096]) + .await + .unwrap(); + assert!(!cache.file_cache.contains_key(&key)); + let err = cache + .upload( + key, + "failed.parquet", + &remote_store, + UploadOptions { + op_type: OperationType::Compact, + write_buffer_size: ReadableSize::kb(1), + }, + ) + .await + .unwrap_err(); + assert!(!err.is_object_not_found()); + assert!(err.may_have_persisted_manifest_update()); + match err { + error::Error::Upload { + region_id, + file_id, + file_type, + error, + .. + } => { + assert_eq!(key.region_id, region_id); + assert_eq!(key.file_id, file_id); + assert_eq!(key.file_type, file_type); + assert_eq!(std::io::ErrorKind::NotFound, error.kind()); + let message = if fail_close { + "injected close failure" + } else { + "injected write failure" + }; + assert!(error.to_string().contains(message)); + } + err => panic!("expected Upload error, got {err:?}"), + } + assert!(!cache.file_cache.contains_key(&key)); + assert!(!remote_store.exists("failed.parquet").await.unwrap()); + let observations = observations.lock().unwrap(); + // The write failure follows one completed chunk, so partial progress is exercised. + assert_eq!( + vec![1024; if fail_close { 4 } else { 2 }], + observations.write_sizes + ); + assert_eq!(1, observations.abort_calls); + assert_eq!(usize::from(fail_close), observations.close_calls); + } + + #[tokio::test] + async fn test_upload_local_read_error() { + use object_store::layers::mock::MockLayerBuilder; + + let env = TestEnv::new().await; + // The unit reader rejects reads after stat and reader creation succeed. + let layer = MockLayerBuilder::default() + .reader_factory(Arc::new(|_, _, _| Box::new(()))) + .build() + .unwrap(); + let local_store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let remote_store = ObjectStore::new(Memory::default()).unwrap(); + let cache = env + .create_write_cache(local_store.clone(), ReadableSize::mb(10)) + .await; + let key = IndexKey::new(RegionId::new(1024, 1), FileId::random(), FileType::Parquet); + local_store + .write(&cache.file_cache.cache_file_path(key), vec![42; 4096]) + .await + .unwrap(); + let err = cache + .upload( + key, + "failed.parquet", + &remote_store, + UploadOptions { + op_type: OperationType::Compact, + write_buffer_size: ReadableSize::kb(1), + }, + ) + .await + .unwrap_err(); + assert!(matches!(err, error::Error::OpenDal { .. })); + assert!(!remote_store.exists("failed.parquet").await.unwrap()); + } + struct RedirectUploadStoreWrapper { target_store: ObjectStore, last_op_type: Mutex>, @@ -776,7 +1075,13 @@ mod tests { // Write to cache and upload sst to mock remote store let mut metrics = Metrics::new(WriteType::Flush); let mut sst_infos = write_cache - .write_and_upload_sst(write_request, upload_request, &write_opts, &mut metrics) + .write_and_upload_sst_with_context( + write_request, + upload_request, + &write_opts, + &mut metrics, + SstWriteContext::default(), + ) .await .unwrap(); let sst_info = sst_infos.remove(0); @@ -890,7 +1195,13 @@ mod tests { let mut metrics = Metrics::new(WriteType::Flush); let mut sst_infos = write_cache - .write_and_upload_sst(write_request, upload_request, &write_opts, &mut metrics) + .write_and_upload_sst_with_context( + write_request, + upload_request, + &write_opts, + &mut metrics, + SstWriteContext::default(), + ) .await .unwrap(); let sst_info = sst_infos.remove(0); @@ -983,7 +1294,13 @@ mod tests { let mut metrics = Metrics::new(WriteType::Flush); write_cache - .write_and_upload_sst(write_request, upload_request, &write_opts, &mut metrics) + .write_and_upload_sst_with_context( + write_request, + upload_request, + &write_opts, + &mut metrics, + SstWriteContext::default(), + ) .await .unwrap_err(); let atomic_write_dir = write_cache_dir.path().join(ATOMIC_WRITE_DIR); diff --git a/src/mito2/src/compaction/compactor.rs b/src/mito2/src/compaction/compactor.rs index 5074dcedadb..7656896c396 100644 --- a/src/mito2/src/compaction/compactor.rs +++ b/src/mito2/src/compaction/compactor.rs @@ -17,7 +17,6 @@ use std::sync::Arc; use std::time::Duration; use common_base::Plugins; -use common_base::cancellation::CancellationHandle; use common_meta::key::SchemaMetadataManagerRef; use common_telemetry::{info, warn}; use common_time::TimeToLive; @@ -31,6 +30,7 @@ use store_api::ManifestVersion; use store_api::metadata::RegionMetadataRef; use store_api::region_request::PathType; use store_api::storage::{RegionId, SequenceNumber}; +use tokio_util::sync::CancellationToken; use crate::access_layer::{ AccessLayer, AccessLayerRef, Metrics, OperationType, SstWriteRequest, WriteType, @@ -54,12 +54,14 @@ use crate::region::{ManifestContext, RegionLeaderState, RegionRoleState}; use crate::schedule::scheduler::LocalScheduler; use crate::sst::FormatType; use crate::sst::file::{self, FileHandle, FileMeta, UncommittedSsts}; +use crate::sst::file_metadata::{FileMetadataInterner, FileMetadataInternerRef}; use crate::sst::file_purger::LocalFilePurger; use crate::sst::index::intermediate::IntermediateManager; use crate::sst::index::puffin_manager::PuffinManagerFactory; use crate::sst::location::region_dir_from_table_dir; use crate::sst::parquet::metadata::extract_primary_key_range; use crate::sst::parquet::parallel_writer::EncodeRuntime; +use crate::sst::parquet::writer::SstWriteContext; use crate::sst::parquet::{SstInfo, WriteOptions}; use crate::sst::version::{SstVersion, SstVersionRef}; @@ -208,6 +210,7 @@ pub struct OpenCompactionRegionRequest { /// stale intermediate files, so repeating it can delete another job's live files. #[derive(Clone)] pub struct CompactionRegionOpener { + file_metadata_interner: FileMetadataInternerRef, mito_config: Arc, object_store_manager: ObjectStoreManagerRef, puffin_manager_factory: PuffinManagerFactory, @@ -232,6 +235,7 @@ impl CompactionRegionOpener { IntermediateManager::init_fs(&mito_config.index.aux_path).await?; Ok(Self { + file_metadata_interner: FileMetadataInterner::new(), mito_config: Arc::new(mito_config), object_store_manager, puffin_manager_factory, @@ -270,8 +274,12 @@ impl CompactionRegionOpener { let manifest_manager = { let region_dir = region_dir_from_table_dir(&req.table_dir, req.region_id, req.path_type); - let region_manifest_options = - RegionManifestOptions::new(mito_config, ®ion_dir, object_store); + let region_manifest_options = RegionManifestOptions::new( + mito_config, + ®ion_dir, + object_store, + self.file_metadata_interner.clone(), + ); RegionManifestManager::open(region_manifest_options, &Default::default()) .await? @@ -548,11 +556,14 @@ pub trait Compactor: Send + Sync + 'static { /// implementations in tests. #[async_trait::async_trait] pub trait SstMerger: Send + Sync + 'static { + /// Merges an output, observing cancellation at safe boundaries and draining + /// in-flight writers before returning. async fn merge_single_output( &self, context: CompactionContext, output: CompactionOutput, write_opts: WriteOptions, + cancellation: CancellationToken, ) -> Result<(Vec, Vec)>; } @@ -611,6 +622,37 @@ impl SstMerger for DefaultSstMerger { context: CompactionContext, output: CompactionOutput, write_opts: WriteOptions, + cancellation: CancellationToken, + ) -> Result<(Vec, Vec)> { + let uncommitted = UncommittedSsts::new( + context.region_id, + context.access_layer.clone(), + Some(context.cache_manager.clone()), + ); + let result = self + .merge_output(context, output, write_opts, &uncommitted, cancellation) + .await; + if result.is_err() { + uncommitted.cleanup().await; + } else { + // Successful outputs are transferred to the caller for publication. + uncommitted.disarm_cleanup(); + } + match result { + Err(error::Error::SstWriteCancelled { .. }) => error::CompactionCancelledSnafu.fail(), + result => result, + } + } +} + +impl DefaultSstMerger { + async fn merge_output( + &self, + context: CompactionContext, + output: CompactionOutput, + write_opts: WriteOptions, + uncommitted: &UncommittedSsts, + cancellation: CancellationToken, ) -> Result<(Vec, Vec)> { let region_id = context.region_id; validate_region_compaction_output(region_id, &output)?; @@ -645,13 +687,18 @@ impl SstMerger for DefaultSstMerger { merge_mode, read_parallelism: write_opts.encode_parallelism, }; - let source = spawn_merge(builder.build_flat_sst_reader().await?); + let source = tokio::select! { + biased; + _ = cancellation.cancelled() => return error::CompactionCancelledSnafu.fail(), + source = builder.build_flat_sst_reader() => source?, + }; + let source = spawn_merge(source); let mut metrics = Metrics::new(WriteType::Compaction); let region_metadata = context.region_metadata.clone(); let sst_infos = context .access_layer - .write_sst( + .write_sst_with_context( SstWriteRequest { index_build_inputs: None, op_type: OperationType::Compact, @@ -676,6 +723,11 @@ impl SstMerger for DefaultSstMerger { }, &write_opts, &mut metrics, + SstWriteContext { + guard: Some(Arc::new(uncommitted.clone())), + cancellation: Some(cancellation), + ..Default::default() + }, ) .await?; // Convert partition expression once outside the map @@ -715,7 +767,7 @@ impl SstMerger for DefaultSstMerger { num_rows: sst_info.num_rows as u64, num_row_groups: sst_info.num_row_groups, sequence: sequence_metadata.file_sequence, - partition_expr: partition_expr.clone(), + partition_expr: partition_expr.clone().map(Into::into), num_series: sst_info.num_series, primary_key_min, primary_key_max, @@ -739,7 +791,7 @@ impl SstMerger for DefaultSstMerger { /// implementations in tests. pub struct DefaultCompactor { merger: M, - cancel_handle: Arc, + cancel_token: CancellationToken, uncommitted: Option, } @@ -748,37 +800,37 @@ impl DefaultCompactor { pub fn with_merger(merger: M) -> Self { Self { merger, - cancel_handle: Arc::new(CancellationHandle::default()), + cancel_token: CancellationToken::new(), uncommitted: None, } } } impl DefaultCompactor { - /// Creates a new `DefaultCompactor` with the given cancel handle and no + /// Creates a new `DefaultCompactor` with the given cancellation token and no /// uncommitted SSTs tracking. /// /// This is the public entry point for external crates that want a /// cancellable compactor without opting into uncommitted-SST tracking. /// - /// Cancellation stops new output batches. Already spawned outputs finish, - /// and their successful SSTs remain in the returned [`MergeOutput`]. This - /// can delay cancellation until the current batch completes. - pub fn new_with_cancel_handle(cancel_handle: Arc) -> Self { + /// Cancellation stops new output batches and interrupts active outputs at + /// batch and upload-chunk boundaries. Successful complete outputs remain in + /// the returned [`MergeOutput`]; incomplete outputs are cleaned up. + pub fn new_with_cancel_token(cancel_token: CancellationToken) -> Self { Self { merger: DefaultSstMerger, - cancel_handle, + cancel_token, uncommitted: None, } } - pub(crate) fn with_cancel_handle( - cancel_handle: Arc, + pub(crate) fn with_cancel_token( + cancel_token: CancellationToken, uncommitted: UncommittedSsts, ) -> Self { Self { merger: DefaultSstMerger, - cancel_handle, + cancel_token, uncommitted: Some(uncommitted), } } @@ -797,11 +849,11 @@ impl DefaultCompactor { ..Default::default() }; for output in &unit.outputs { - if self.cancel_handle.is_cancelled() { + if self.cancel_token.is_cancelled() { return error::CompactionCancelledSnafu.fail(); } - // Let the writer finish and register its files before observing cancellation. - // Dropping it mid-write can lose ownership of already finalized SSTs. + // The merger observes cancellation and drains its writers before returning. + // Keep awaiting it so cleanup cannot race with a late finalized SST. let (files, infos) = self .merger .merge_single_output( @@ -821,6 +873,7 @@ impl DefaultCompactor { ), encode_runtime: EncodeRuntime::Compact, }, + self.cancel_token.clone(), ) .await?; if let Some(uncommitted) = &self.uncommitted { @@ -829,7 +882,7 @@ impl DefaultCompactor { result.files_to_add.extend(files); result.sst_infos.extend(infos); } - if self.cancel_handle.is_cancelled() { + if self.cancel_token.is_cancelled() { return error::CompactionCancelledSnafu.fail(); } Ok(result) @@ -888,9 +941,10 @@ where let merger = self.merger.clone(); let context = context.clone(); let uncommitted = self.uncommitted.clone(); + let cancellation = self.cancel_token.clone(); let fut = async move { let result = merger - .merge_single_output(context, output, write_opts) + .merge_single_output(context, output, write_opts, cancellation) .await; if let (Some(uncommitted), Ok((_, infos))) = (&uncommitted, &result) { uncommitted.track(infos); @@ -908,7 +962,7 @@ where + picker_output.expired_ssts.len(), ); - while !tasks.is_empty() && !self.cancel_handle.is_cancelled() { + while !tasks.is_empty() && !self.cancel_token.is_cancelled() { let mut chunk: Vec<(Vec, _)> = Vec::with_capacity(internal_parallelism); for _ in 0..internal_parallelism { if let Some(task) = tasks.pop() { @@ -923,9 +977,8 @@ where }) .collect(); - // Like merge_unit, cancellation must not drop a writer mid-output: - // its blocking encoders would detach and finalized SSTs could lose - // their owner. Join the whole batch and retain every successful output. + // Each merger cancels cooperatively and owns its incomplete outputs. + // Join every sibling before returning so writers and cleanup cannot race. let mut join_error = None; while let Some((inputs, handle)) = spawned.pop() { match handle.await { @@ -936,6 +989,7 @@ where } compacted_inputs.extend(inputs); } + Ok(Err(error::Error::CompactionCancelled { .. })) => {} Ok(Err(e)) => { warn!( e; "Failed to merge compaction output for region: {}, inputs: [{}]", @@ -959,7 +1013,7 @@ where return Err(e).context(error::JoinSnafu); } - if self.cancel_handle.is_cancelled() { + if self.cancel_token.is_cancelled() { info!("Compaction merge cancelled for region: {}", region_id); break; } @@ -999,9 +1053,12 @@ mod tests { use std::time::Duration; use datatypes::arrow::record_batch::RecordBatch; + use object_store::ObjectStore; + use object_store::layers::mock::{self, MockLayerBuilder}; + use object_store::services::Memory; use parquet::arrow::ArrowWriter; use store_api::storage::{FileId, RegionId}; - use tokio::sync::{mpsc, oneshot}; + use tokio::sync::{Notify, mpsc, oneshot}; use tokio::time::timeout; use super::{DefaultCompactor, *}; @@ -1025,6 +1082,105 @@ mod tests { FileHandle::new(meta, Arc::new(NoopFilePurger)) } + struct PendingCompactionReader { + started: Arc, + } + + impl mock::oio::Read for PendingCompactionReader { + async fn open( + &self, + _: mock::BytesRange, + ) -> object_store::Result<(mock::RpRead, Box)> { + self.started.notify_one(); + std::future::pending().await + } + + async fn read( + &self, + _: mock::BytesRange, + ) -> object_store::Result<(mock::RpRead, object_store::Buffer)> { + self.started.notify_one(); + std::future::pending().await + } + } + + #[rstest::rstest] + #[case::unit(true)] + #[case::outputs(false)] + #[tokio::test] + async fn test_default_merger_observes_compactor_cancellation(#[case] merge_unit: bool) { + let mut region = new_test_compaction_region().await; + let started = Arc::new(Notify::new()); + let signal = started.clone(); + let layer = MockLayerBuilder::default() + .reader_factory(Arc::new(move |_, _, _| { + Box::new(PendingCompactionReader { + started: signal.clone(), + }) + })) + .build() + .unwrap(); + let store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let access = ®ion.access_layer; + region.access_layer = Arc::new(AccessLayer::new( + access.table_dir(), + access.path_type(), + store.clone(), + access.puffin_manager_factory().clone(), + access.intermediate_manager().clone(), + )); + let input = new_file_handle(dummy_file_meta()); + store + .write( + &input.file_path( + region.access_layer.table_dir(), + region.access_layer.path_type(), + ), + vec![0; input.meta_ref().file_size as usize], + ) + .await + .unwrap(); + let picker_output = PickerOutput { + outputs: vec![CompactionOutput { + output_level: 1, + inputs: vec![input], + filter_deleted: false, + output_time_range: None, + }], + ..Default::default() + }; + let compactor = DefaultCompactor::with_merger(DefaultSstMerger); + let context = CompactionContext::from(®ion); + let mut merge = Box::pin(async { + if merge_unit { + let unit = CompactionUnit::from_picker(picker_output) + .unwrap() + .pop() + .unwrap(); + compactor.merge_unit(&context, &unit).await + } else { + compactor.merge_ssts(®ion, picker_output).await + } + }); + tokio::select! { + _ = started.notified() => {}, + result = &mut merge => panic!("merge returned before cancellation: {result:?}"), + _ = tokio::time::sleep(Duration::from_secs(10)) => panic!("merge did not reach the reader"), + } + compactor.cancel_token.cancel(); + let result = timeout(Duration::from_secs(10), merge).await.unwrap(); + if merge_unit { + assert!(matches!( + result, + Err(error::Error::CompactionCancelled { .. }) + )); + } else { + let result = result.unwrap(); + assert!(result.files_to_add.is_empty()); + assert!(result.files_to_remove.is_empty()); + } + } + #[test] fn test_effective_target_trust_and_max_sequence() { let target = RegionId::new(1, 1); @@ -1413,7 +1569,12 @@ mod tests { .await .unwrap(); let (files, _) = DefaultSstMerger - .merge_single_output(context, output, WriteOptions::default()) + .merge_single_output( + context, + output, + WriteOptions::default(), + CancellationToken::new(), + ) .await .unwrap(); assert_eq!(1, files.len()); @@ -1474,6 +1635,7 @@ mod tests { _context: CompactionContext, _output: CompactionOutput, _write_opts: WriteOptions, + _cancellation: CancellationToken, ) -> Result<(Vec, Vec)> { let idx = self.call_idx.fetch_add(1, Ordering::SeqCst); match self.results.lock().unwrap().get(idx) { @@ -1706,6 +1868,7 @@ mod tests { _context: CompactionContext, output: CompactionOutput, _write_opts: WriteOptions, + _cancellation: CancellationToken, ) -> Result<(Vec, Vec)> { let input_id = output.inputs[0].meta_ref().file_id; let mut file = dummy_file_meta(); @@ -1758,7 +1921,7 @@ mod tests { .collect(), ..Default::default() }; - let cancel_handle = Arc::new(CancellationHandle::default()); + let cancel_token = CancellationToken::new(); let (started_tx, mut started_rx) = mpsc::unbounded_channel(); let compactor = DefaultCompactor { merger: GatedEncodingMerger { @@ -1766,7 +1929,7 @@ mod tests { // Outputs are popped from the back, then joined in reverse order. panic_input: panic_merge.then_some(inputs[1].file_id), }, - cancel_handle: cancel_handle.clone(), + cancel_token: cancel_token.clone(), uncommitted: None, }; let memory = new_compaction_memory_manager(1024 * 1024); @@ -1785,7 +1948,7 @@ mod tests { .unwrap(), ); } - cancel_handle.cancel(); + cancel_token.cancel(); assert!(futures::poll!(merge.as_mut()).is_pending()); assert!(memory.try_acquire(memory.limit_bytes()).is_none()); assert!( @@ -1848,7 +2011,7 @@ mod tests { let merger = MockMerger::new(vec![Ok(vec![dummy_file_meta()])]); let calls = merger.call_idx.clone(); let compactor = DefaultCompactor::with_merger(merger); - compactor.cancel_handle.cancel(); + compactor.cancel_token.cancel(); let picker_output = PickerOutput { outputs: vec![CompactionOutput { output_level: 1, diff --git a/src/mito2/src/compaction/multi_region.rs b/src/mito2/src/compaction/multi_region.rs index de4549a4c5c..15741e663bd 100644 --- a/src/mito2/src/compaction/multi_region.rs +++ b/src/mito2/src/compaction/multi_region.rs @@ -28,12 +28,12 @@ use std::collections::{BTreeMap, HashMap, HashSet}; use std::sync::Arc; use std::sync::atomic::AtomicU64; -use common_base::cancellation::{CancellableFuture, CancellationHandle}; use common_time::Timestamp; use partition::PartitionRuleRef; use snafu::{OptionExt, ensure}; use store_api::metadata::RegionMetadataRef; use store_api::storage::RegionId; +use tokio_util::sync::CancellationToken; use crate::access_layer::AccessLayerRef; use crate::cache::CacheManagerRef; @@ -82,7 +82,7 @@ pub struct MultiRegionCompactionContext { pub cache: CacheManagerRef, pub config: Arc, pub protection: Arc, - pub cancellation: Arc, + pub cancellation: CancellationToken, } /// Source I/O and lossless output accounting, excluding footer and query reads. @@ -137,9 +137,11 @@ pub async fn compact_regions( source::SharedSource::build(&plan, &context, metadata, counters.clone()).await?; worker::execute(plan, context, source, counters).await }; - CancellableFuture::new(execution, cancellation) - .await - .map_err(|_| crate::error::CompactionCancelledSnafu.build())? + tokio::select! { + biased; + _ = cancellation.cancelled() => crate::error::CompactionCancelledSnafu.fail(), + result = execution => result, + } } fn validate_plan(plan: &MultiRegionCompactionPlan) -> Result { diff --git a/src/mito2/src/compaction/multi_region/source.rs b/src/mito2/src/compaction/multi_region/source.rs index 313f57778f5..66f38a27f02 100644 --- a/src/mito2/src/compaction/multi_region/source.rs +++ b/src/mito2/src/compaction/multi_region/source.rs @@ -109,7 +109,7 @@ pub(crate) struct SharedSource { window: (Timestamp, Timestamp), pub(crate) limits: MemoryLimits, peak_group_bytes: AtomicU64, - cancellation: Arc, + cancellation: CancellationToken, stop: CancellationToken, counters: Arc, } diff --git a/src/mito2/src/compaction/multi_region/tests.rs b/src/mito2/src/compaction/multi_region/tests.rs index 67709443086..a8a2cc6efdf 100644 --- a/src/mito2/src/compaction/multi_region/tests.rs +++ b/src/mito2/src/compaction/multi_region/tests.rs @@ -277,7 +277,7 @@ impl Fixture { cache: Arc::new(CacheManager::default()), config: self.config.clone(), protection: protection.clone(), - cancellation: Arc::new(CancellationHandle::default()), + cancellation: CancellationToken::new(), }, protection, ) @@ -2002,7 +2002,7 @@ async fn test_interrupted_indexed_worker_cleans_tantivy_files(#[case] fail_sibli } } -struct CancelBeforeCreate(Arc); +struct CancelBeforeCreate(CancellationToken); #[async_trait::async_trait] impl SstWriteGuard for CancelBeforeCreate { diff --git a/src/mito2/src/compaction/multi_region/writer.rs b/src/mito2/src/compaction/multi_region/writer.rs index 0d2ce6ea838..bb449e5f5b2 100644 --- a/src/mito2/src/compaction/multi_region/writer.rs +++ b/src/mito2/src/compaction/multi_region/writer.rs @@ -16,6 +16,7 @@ use common_base::readable_size::ReadableSize; use partition::expr::PartitionExpr; use snafu::{ResultExt, ensure}; use store_api::codec::PrimaryKeyEncoding; +use tokio_util::sync::CancellationToken; use crate::access_layer::{Metrics, OperationType, SstWriteRequest, WriteType}; use crate::compaction::multi_region::memory::{self, MemoryLimits}; @@ -27,6 +28,7 @@ use crate::sst::FormatType; use crate::sst::file::FileMeta; use crate::sst::parquet::metadata::extract_primary_key_range; use crate::sst::parquet::parallel_writer::EncodeRuntime; +use crate::sst::parquet::writer::SstWriteContext; use crate::sst::parquet::{SstInfo, WriteOptions}; pub(crate) struct TargetWriter<'a> { @@ -71,13 +73,14 @@ impl<'a> TargetWriter<'a> { let infos = self .context .access_layer - .write_sst_with_guard( + .write_sst_with_context( request, &self.options(), &mut metrics, - crate::sst::parquet::writer::SstWriteContext { - guard: self.context.protection.clone(), - limits: self.limits.writer, + SstWriteContext { + guard: Some(self.context.protection.clone()), + limits: Some(self.limits.writer), + ..Default::default() }, ) .await?; @@ -195,7 +198,7 @@ impl<'a> TargetWriter<'a> { num_rows: info.num_rows as u64, num_row_groups: info.num_row_groups, sequence: self.sequence, - partition_expr, + partition_expr: partition_expr.map(Into::into), num_series: info.num_series, primary_key_min: range.as_ref().map(|(min, _)| min.clone()), primary_key_max: range.map(|(_, max)| max), @@ -207,7 +210,7 @@ impl<'a> TargetWriter<'a> { fn output_projection( target: &MultiRegionTarget, mut source: FlatSource, - cancellation: Arc, + cancellation: CancellationToken, ) -> Result { let projection: Vec<_> = source .schema() diff --git a/src/mito2/src/compaction/scheduler.rs b/src/mito2/src/compaction/scheduler.rs index c7a370df623..028c7615d07 100644 --- a/src/mito2/src/compaction/scheduler.rs +++ b/src/mito2/src/compaction/scheduler.rs @@ -51,8 +51,8 @@ use crate::error::{ CompactionCancelledSnafu, Error, ManualCompactionAlreadyRunningSnafu, RegionClosedSnafu, RegionDroppedSnafu, RegionTruncatedSnafu, Result, UnsupportedOperationSnafu, }; +use crate::region::ManifestContextRef; use crate::region::version::VersionControlRef; -use crate::region::{ManifestContextRef, RegionLeaderState, RegionRoleState}; use crate::request::{DdlRequest, OptionOutputTx, SenderDdlRequest, WorkerRequestWithTime}; use crate::schedule::RequestCancelResult; use crate::worker::WorkerListener; @@ -516,6 +516,23 @@ impl CompactionScheduler { status.pending_ddl_requests.push(request); Ok(()) } + + /// Pending and handed-off DDLs retain their turn until the final reply. + pub(crate) fn blocks_region_edits(&self, region_id: RegionId) -> bool { + self.region_status + .get(®ion_id) + .is_some_and(|status| status.has_ddl()) + } + + /// Whether a staging request is still waiting for compaction cancellation. + pub(crate) fn has_pending_enter_staging(&self, region_id: RegionId) -> bool { + self.region_status.get(®ion_id).is_some_and(|status| { + status + .pending_ddl_requests + .iter() + .any(|ddl| matches!(ddl.request, DdlRequest::EnterStaging(_))) + }) + } } // Internal state-machine helpers. @@ -569,7 +586,7 @@ impl CompactionScheduler { return Ok(false); } let current_state = manifest_ctx.current_state(); - if current_state == RegionRoleState::Leader(RegionLeaderState::Staging) { + if current_state.in_staging_session() { info!( "Skipping compaction for region {} in staging mode, options: {:?}", region_id, compact_options diff --git a/src/mito2/src/compaction/scheduler_test.rs b/src/mito2/src/compaction/scheduler_test.rs index a4d310a688f..bb9276d2b80 100644 --- a/src/mito2/src/compaction/scheduler_test.rs +++ b/src/mito2/src/compaction/scheduler_test.rs @@ -45,7 +45,7 @@ use crate::engine::pending_tail_test::{pending_tails_edit, tail}; use crate::error::InvalidSchedulerStateSnafu; use crate::manifest::manager::{RegionManifestManager, RegionManifestOptions}; use crate::metrics::COMPACTION_MEMORY_REJECTED; -use crate::region::ManifestContext; +use crate::region::{ManifestContext, RegionLeaderState, RegionRoleState}; use crate::request::{BackgroundNotify, CompactionUnitNotification, OutputTx, WorkerRequest}; use crate::schedule::CancellableTaskState; use crate::schedule::remote_job_scheduler::{RemoteJob, RemoteJobSchedulerRef}; @@ -2314,8 +2314,11 @@ async fn test_scheduler_env_submits_compactions_to_injected_executor() { assert_eq!(1, injected.num_jobs()); } +#[rstest::rstest] +#[case::staging(RegionLeaderState::Staging)] +#[case::ending_staging(RegionLeaderState::EndingStaging)] #[tokio::test] -async fn test_compaction_bypass_in_staging_mode() { +async fn test_compaction_bypass_in_staging_mode(#[case] state: RegionLeaderState) { let env = SchedulerEnv::new().await; let (tx, _rx) = mpsc::channel(4); let mut scheduler = env.mock_compaction_scheduler(tx); @@ -2330,6 +2333,7 @@ async fn test_compaction_bypass_in_staging_mode() { version_control.current().version.metadata.clone(), 0, RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "".to_string(), object_store: env.access_layer.object_store().clone(), compress_type: CompressionType::Uncompressed, @@ -2346,7 +2350,7 @@ async fn test_compaction_bypass_in_staging_mode() { .unwrap(); Arc::new(ManifestContext::new( manager, - RegionRoleState::Leader(RegionLeaderState::Staging), + RegionRoleState::Leader(state), None, Default::default(), )) @@ -2515,7 +2519,7 @@ async fn test_request_cancel_state_transitions() { let state = status.start_local_task(); assert_eq!(status.request_cancel(), RequestCancelResult::CancelIssued); - assert!(state.cancel_handle().is_cancelled()); + assert!(state.cancel_token().is_cancelled()); assert_eq!( status.request_cancel(), RequestCancelResult::AlreadyCancelling diff --git a/src/mito2/src/compaction/task.rs b/src/mito2/src/compaction/task.rs index 666601fcf85..98c5d2552de 100644 --- a/src/mito2/src/compaction/task.rs +++ b/src/mito2/src/compaction/task.rs @@ -15,7 +15,6 @@ use std::fmt::{Debug, Formatter}; use std::sync::Arc; -use common_base::cancellation::CancellableFuture; use common_memory_manager::OnExhaustedPolicy; use common_telemetry::error; use futures::FutureExt; @@ -91,25 +90,24 @@ impl CompactionTask { None } else { let _wait_timer = crate::metrics::COMPACTION_MEMORY_WAIT.start_timer(); + let cancellation = self.state.cancel_token(); Some( - CancellableFuture::new( - self.memory_manager - .acquire_with_policy(bytes, self.memory_policy), - self.state.cancel_handle(), - ) - .await - .map_err(|_| CompactionCancelledSnafu.build())? - .context(CompactionMemoryExhaustedSnafu { - region_id, - policy: format!("{:?}", self.memory_policy), - })?, + tokio::select! { + biased; + _ = cancellation.cancelled() => return CompactionCancelledSnafu.fail(), + memory = self.memory_manager.acquire_with_policy(bytes, self.memory_policy) => memory, + } + .context(CompactionMemoryExhaustedSnafu { + region_id, + policy: format!("{:?}", self.memory_policy), + })?, ) }; let _timer = COMPACTION_STAGE_ELAPSED .with_label_values(&["merge"]) .start_timer(); - let compactor = DefaultCompactor::with_cancel_handle( - self.state.cancel_handle(), + let compactor = DefaultCompactor::with_cancel_token( + self.state.cancel_token(), self.uncommitted.clone(), ); let context = self.compaction_context.clone(); diff --git a/src/mito2/src/engine.rs b/src/mito2/src/engine.rs index 48067a554f7..dd9144cbc0c 100644 --- a/src/mito2/src/engine.rs +++ b/src/mito2/src/engine.rs @@ -57,7 +57,7 @@ mod merge_mode_test; #[cfg(test)] mod object_store_wal_recovery_test; #[cfg(test)] -mod object_store_wal_test; +pub(crate) mod object_store_wal_test; #[cfg(test)] mod open_test; #[cfg(test)] @@ -90,6 +90,8 @@ mod sync_test; #[cfg(feature = "enterprise")] pub mod table_compaction; #[cfg(test)] +mod tail_replay_test; +#[cfg(test)] mod truncate_test; #[cfg(test)] @@ -171,9 +173,9 @@ use crate::metrics::{ }; use crate::read::scan_region::{ScanRegion, Scanner, exact_sequence_range}; use crate::read::stream::ScanBatchStream; -use crate::region::MitoRegionRef; use crate::region::opener::PartitionExprFetcherRef; use crate::region::options::parse_wal_options; +use crate::region::{MitoRegionRef, RegionRoleState}; use crate::request::{RegionEditRequest, WorkerRequest}; use crate::sst::file::{FileMeta, RegionFileId, RegionIndexId}; use crate::sst::file_ref::FileReferenceManagerRef; @@ -350,12 +352,40 @@ impl MitoEngine { self.inner.workers.node_lease().renew(deadline); } + /// Records a lease of a heartbeat sent at `sent_at` as the latest one the region received, if + /// it is the latest. The region may also be a region of a group that received the lease as a + /// whole, such as the data region of a metric engine region. Only records it: the region's role + /// does not change. Does nothing if the region is not open. + pub fn record_granted_lease( + &self, + region_id: RegionId, + sent_at: Instant, + candidate_lease_id: Option, + ) { + self.inner + .workers + .granted_leases() + .record(region_id, sent_at, candidate_lease_id, || { + self.inner.find_region(region_id).is_ok() + }); + } + /// Freezes the clock of the lease checks at `at`. #[cfg(any(test, feature = "test"))] pub fn freeze_lease_clock(&self, at: Instant) { self.inner.workers.node_lease().freeze_clock(at); } + /// Returns the candidate lease id of the latest lease granted to the region. + #[cfg(any(test, feature = "test"))] + pub fn granted_candidate_lease_id(&self, region_id: RegionId) -> Option { + self.inner + .workers + .granted_leases() + .get(region_id) + .and_then(|lease| lease.candidate_lease_id) + } + pub fn object_store_manager(&self) -> &ObjectStoreManagerRef { self.inner.workers.object_store_manager() } @@ -1344,10 +1374,40 @@ impl EngineInner { /// does not check whether the region waits for a reload. fn set_region_role(&self, region_id: RegionId, role: RegionRole) -> Result<()> { let region = self.find_region(region_id)?; - region.set_role(role); + let was_leader = region.region_role() == RegionRole::Leader; + let was_writable = region.is_writable(); + let previous = region.set_role(role); + if role == RegionRole::Leader && !was_leader { + // No lease comes for a direct promotion, so it counts as one. A leader is not promoted. + region.manifest_ctx.record_leader_lease(Instant::now()); + } + self.notify_if_can_no_longer_replay(®ion, previous); + self.notify_if_became_writable(®ion, was_writable); Ok(()) } + /// Asks the worker to cancel the tail replay of `region` if a role change left it unable to + /// replay. `previous` is the state the change replaced; a later state could come from a + /// concurrent promotion. A change to Leader only replaces states that cannot replay. A replay + /// may wait on I/O that never notices the role change. + fn notify_if_can_no_longer_replay( + &self, + region: &MitoRegionRef, + previous: Option, + ) { + if previous.is_some_and(RegionRoleState::can_replay_tails) { + self.workers.notify_cannot_replay(region); + } + } + + /// Asks the worker to replay the pending tails of `region` if a role change made it + /// writable. The worker may already have started an ALTER or an edit of the region. + fn notify_if_became_writable(&self, region: &MitoRegionRef, was_writable: bool) { + if !was_writable && region.can_replay_tails() { + self.workers.notify_writable(region); + } + } + /// Sets read-only for a region and ensures no more writes in the region after it returns. async fn set_region_role_state_gracefully( &self, @@ -1686,18 +1746,25 @@ impl RegionEngine for MitoEngine { role: RegionRole, sent_at: Instant, deadline: Instant, + candidate_lease_id: Option, ) -> Result { + // Before the role, so a lease the region ignores still counts. + self.record_granted_lease(region_id, sent_at, candidate_lease_id); let region = self.inner.find_region(region_id).map_err(BoxedError::new)?; // Before the role, and even if the role leaves the region's state unchanged. if role.writable() { region.manifest_ctx.leader_deadline.extend(deadline); } if role == RegionRole::Leader { - return Ok(region + let was_writable = region.is_writable(); + let promoted = region .manifest_ctx - .promote_unless_downgrade_started_after(sent_at, region_id)); + .promote_unless_downgrade_started_after(sent_at, region_id); + self.inner.notify_if_became_writable(®ion, was_writable); + return Ok(promoted); } - region.set_role(role); + let previous = region.set_role(role); + self.inner.notify_if_can_no_longer_replay(®ion, previous); Ok(true) } diff --git a/src/mito2/src/engine/apply_staging_manifest_test.rs b/src/mito2/src/engine/apply_staging_manifest_test.rs index be6d651b48e..35df5115f5a 100644 --- a/src/mito2/src/engine/apply_staging_manifest_test.rs +++ b/src/mito2/src/engine/apply_staging_manifest_test.rs @@ -18,6 +18,7 @@ use std::{assert_matches, fs}; use api::v1::Rows; use api::v1::region::{StrictWindow, compact_request}; +use common_error::ext::{ErrorExt, RetryHint}; use common_function::utils::partition_expr_version; use common_recordbatch::RecordBatches; use datatypes::arrow::array::{AsArray, UInt64Array}; @@ -41,8 +42,10 @@ use crate::manifest::action::{ }; use crate::sst::FormatType; use crate::sst::file::FileMeta; +use crate::sst::file_metadata::MetadataRef; use crate::test_util::{ - CreateRequestBuilder, TestEnv, build_rows, build_rows_for_key, put_rows, rows_schema, + CreateRequestBuilder, DeltaFaults, TestEnv, build_rows, build_rows_for_key, put_rows, + reopen_region, rows_schema, }; fn range_expr(col_name: &str, start: i64, end: i64) -> PartitionExpr { @@ -609,6 +612,16 @@ async fn test_apply_staging_manifest_success_with_format(flat_format: bool) { assert_eq!(manifest.files.len(), 0); let staging_manifest = region.manifest_ctx.staging_manifest().await.unwrap(); assert_eq!(staging_manifest.files.len(), 0); + // A delta that an older binary left in the staging directory goes when staging ends. + let data_home = env.data_home(); + let region_dir = format!("{}/data/test/1_0000000002", data_home.display()); + let staging_manifest_dir = format!("{}/staging/manifest", region_dir); + fs::create_dir_all(&staging_manifest_dir).unwrap(); + fs::write( + format!("{staging_manifest_dir}/00000000000000000001.json"), + "{}", + ) + .unwrap(); engine .handle_request( @@ -638,9 +651,6 @@ async fn test_apply_staging_manifest_success_with_format(flat_format: bool) { // The staging partition expr should be cleared. assert!(region.manifest_ctx.staging_partition_info().is_none()); // The staging manifest directory should be empty. - let data_home = env.data_home(); - let region_dir = format!("{}/data/test/1_0000000001", data_home.display()); - let staging_manifest_dir = format!("{}/staging/manifest", region_dir); let staging_files = fs::read_dir(&staging_manifest_dir) .map(|entries| { entries @@ -666,11 +676,11 @@ async fn test_apply_staging_manifest_success_with_format(flat_format: bool) { let file_id = FileId::random(); new_manifest_1.files.insert( file_id, - FileMeta { + crate::sst::file::FileMetaRef::from(FileMeta { region_id, file_id, ..Default::default() - }, + }), ); blob_store .put( @@ -1134,6 +1144,32 @@ async fn test_split_repartition_causes_duplicate_data() { assert_eq!(manifest_1.files.len(), source_file_count); assert_eq!(manifest_2.files.len(), source_file_count); + let file_id = *source_manifest.files.keys().next().unwrap(); + let historical_expr = source_manifest.files[&file_id] + .partition_expr + .as_ref() + .unwrap(); + for (region, manifest) in [ + (&target_region_1, &manifest_1), + (&target_region_2, &manifest_2), + ] { + let meta = &manifest.files[&file_id]; + assert_eq!(Some(&source_partition_expr), meta.partition_expr.as_deref()); + assert!(MetadataRef::ptr_eq( + historical_expr, + meta.partition_expr.as_ref().unwrap() + )); + let version = region.version(); + let handle = version + .ssts + .levels() + .iter() + .flat_map(|level| level.files()) + .find(|file| file.file_id() == meta.file_id()) + .unwrap(); + assert!(std::ptr::eq(meta.as_ref(), handle.meta_ref())); + } + // Verify duplication via Scan. let scan_request = ScanRequest::default(); let stream = engine @@ -1179,6 +1215,43 @@ async fn test_split_repartition_causes_duplicate_data() { "actual: {}", batches.pretty_print().unwrap() ); + // Independent manifest recovery must preserve the SST's historical expression. + for region_id in [target_region_id_1, target_region_id_2] { + reopen_region( + &engine, + region_id, + "test".to_string(), + true, + Default::default(), + ) + .await; + } + let recovered_1 = engine + .get_region(target_region_id_1) + .unwrap() + .manifest_ctx + .manifest() + .await; + let recovered_2 = engine + .get_region(target_region_id_2) + .unwrap() + .manifest_ctx + .manifest() + .await; + assert!(MetadataRef::ptr_eq( + recovered_1.files[&file_id].partition_expr.as_ref().unwrap(), + recovered_2.files[&file_id].partition_expr.as_ref().unwrap(), + )); + assert_eq!( + Some(&source_partition_expr), + recovered_2.files[&file_id].partition_expr.as_deref() + ); + let stream = engine + .scan_to_stream(target_region_id_2, ScanRequest::default()) + .await + .unwrap(); + let recovered_batches = RecordBatches::try_collect(stream).await.unwrap(); + assert_eq!(expected, recovered_batches.pretty_print().unwrap()); } #[tokio::test] @@ -1331,6 +1404,18 @@ async fn test_merge_repartition_data_integrity() { "After applying staging manifest, target region should have all files" ); + // A table-scoped pool must preserve both source partition expressions after merge. + for source in [&source_manifest_1, &source_manifest_2] { + for (file_id, source_meta) in &source.files { + let imported = &manifest.files[file_id]; + assert_eq!(source_meta.partition_expr, imported.partition_expr); + assert!(MetadataRef::ptr_eq( + source_meta.partition_expr.as_ref().unwrap(), + imported.partition_expr.as_ref().unwrap(), + )); + } + } + let scan_request = ScanRequest::default(); let stream = engine .scan_to_stream(target_region_id, scan_request) @@ -1359,3 +1444,153 @@ async fn test_merge_repartition_data_integrity() { batches.pretty_print().unwrap() ); } + +/// The end commit fails after its delta landed (a lost response) or before. The retry writes the +/// same delta at the next version, and a reopen sees one commit. +#[rstest::rstest] +#[case::landed(true)] +#[case::not_landed(false)] +#[tokio::test] +async fn test_apply_staging_manifest_retries_failed_commit(#[case] landed: bool) { + let faults = DeltaFaults::default(); + let mut env = TestEnv::with_prefix("apply-failed-commit") + .await + .with_mock_layer(faults.layer()); + let engine = env.create_engine(MitoConfig::default()).await; + let region_id = RegionId::new(2, 2); + let request = CreateRequestBuilder::new().build(); + let table_dir = request.table_dir.clone(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_rows( + &engine, + region_id, + Rows { + schema, + rows: build_rows(0, 3), + }, + ) + .await; + engine + .handle_request(region_id, RegionRequest::Flush(Default::default())) + .await + .unwrap(); + let partition_expr = range_expr("tag_0", 0, 50).as_json_str().unwrap(); + engine + .handle_request( + region_id, + RegionRequest::EnterStaging(EnterStagingRequest { + partition_directive: StagingPartitionDirective::UpdatePartitionExpr( + partition_expr.clone(), + ), + }), + ) + .await + .unwrap(); + let remap_result = engine + .remap_manifests(RemapManifestsRequest { + region_id, + input_regions: vec![region_id], + region_mapping: [(region_id, vec![region_id])].into_iter().collect(), + new_partition_exprs: [(region_id, partition_expr.clone())].into_iter().collect(), + }) + .await + .unwrap(); + let apply = || { + RegionRequest::ApplyStagingManifest(ApplyStagingManifestRequest { + partition_expr: partition_expr.clone(), + central_region_id: region_id, + manifest_path: remap_result.manifest_paths[®ion_id].clone(), + }) + }; + let region = engine.get_region(region_id).unwrap(); + let version = region.manifest_ctx.manifest_version().await; + + if landed { + faults.lose_next_response(); + } else { + faults.fail_next_write(); + } + // The region stays staging and keeps its edits for a retry. + let err = engine.handle_request(region_id, apply()).await.unwrap_err(); + assert_eq!(RetryHint::Retryable, err.retry_hint()); + assert!(region.is_staging()); + // Until the retry, another edit is refused without closing the region. + let edit = RegionEdit { + files_to_add: vec![FileMeta { + region_id, + file_id: FileId::random(), + ..Default::default() + }], + ..Default::default() + }; + let (sender, receiver) = tokio::sync::oneshot::channel(); + let edit = crate::request::RegionEditRequest::new(region_id, edit, true, sender) + .for_staging(partition_expr.clone()); + engine + .inner + .workers + .submit_to_worker(region_id, crate::request::WorkerRequest::EditRegion(edit)) + .await + .unwrap(); + let err = receiver.await.unwrap().unwrap_err(); + assert_eq!(RetryHint::Retryable, err.retry_hint()); + assert!(region.is_staging()); + assert!( + !region + .manifest_ctx + .manifest_manager + .read() + .await + .is_stopped() + ); + + // The retry writes the same delta at the next version. + engine.handle_request(region_id, apply()).await.unwrap(); + assert!(!region.is_staging()); + let committed = region.manifest_ctx.manifest().await; + assert_eq!(version + 2, committed.manifest_version); + assert_eq!( + Some(partition_expr.as_str()), + committed.metadata.partition_expr.as_deref() + ); + let deltas = region + .manifest_ctx + .manifest_manager + .read() + .await + .store() + .fetch_manifests(version + 1, version + 3) + .await + .unwrap(); + if landed { + assert_eq!( + vec![version + 1, version + 2], + deltas.iter().map(|(v, _)| *v).collect::>() + ); + assert_eq!(deltas[0].1, deltas[1].1); + } else { + assert_eq!( + vec![version + 2], + deltas.iter().map(|(v, _)| *v).collect::>() + ); + } + + // A reopen replays the deltas as one commit. + reopen_region(&engine, region_id, table_dir, true, Default::default()).await; + let reopened = engine + .get_region(region_id) + .unwrap() + .manifest_ctx + .manifest() + .await; + assert_eq!(version + 2, reopened.manifest_version); + assert_eq!(committed.files, reopened.files); + assert_eq!( + committed.metadata.partition_expr, + reopened.metadata.partition_expr + ); +} diff --git a/src/mito2/src/engine/close_test.rs b/src/mito2/src/engine/close_test.rs index c3ace2f5c42..f113bb1e611 100644 --- a/src/mito2/src/engine/close_test.rs +++ b/src/mito2/src/engine/close_test.rs @@ -14,12 +14,13 @@ use std::sync::Arc; use std::sync::atomic::Ordering; +use std::time::Duration; use api::v1::Rows; use common_base::Plugins; use common_recordbatch::RecordBatches; use store_api::region_engine::RegionEngine; -use store_api::region_request::{RegionCloseRequest, RegionRequest}; +use store_api::region_request::{PathType, RegionCloseRequest, RegionOpenRequest, RegionRequest}; use store_api::storage::{RegionId, ScanRequest}; use crate::config::MitoConfig; @@ -27,7 +28,8 @@ use crate::engine::flush_test::MockRegionHook; use crate::engine::listener::AlterFlushListener; use crate::engine::region_hook::RegionHookRef; use crate::test_util::{ - CreateRequestBuilder, TestEnv, build_rows, flush_region, put_rows, rows_schema, + CheckpointTaskBlocker, CreateRequestBuilder, TestEnv, build_rows, flush_region, put_rows, + rows_schema, }; #[tokio::test] @@ -277,3 +279,210 @@ async fn test_engine_close_region_flush_on_close_while_flushing() { let batches = RecordBatches::try_collect(stream).await.unwrap(); assert_eq!(6, batches.iter().map(|b| b.num_rows()).sum::()); } + +#[rstest::rstest] +#[case::skip_wal(true, [false, false])] +#[case::flush_on_close(false, [true, false])] +#[tokio::test] +async fn test_repeated_close_during_edit_preserves_flush_requirement( + #[case] skip_wal: bool, + #[case] flush_on_close: [bool; 2], +) { + use crate::manifest::action::RegionEdit; + use crate::region::{RegionLeaderState, RegionRoleState}; + use crate::request::{DdlRequest, SenderDdlRequest, WorkerRequest}; + use store_api::region_request::RegionPutRequest; + + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + let create = CreateRequestBuilder::new().build(); + let schema = rows_schema(&create); + for id in [region_id, other_id] { + engine + .handle_request(id, RegionRequest::Create(create.clone())) + .await + .unwrap(); + } + engine + .handle_request( + region_id, + RegionRequest::Put(RegionPutRequest { + rows: Rows { + schema, + rows: build_rows(0, 3), + }, + hint: None, + partition_expr_version: None, + skip_wal, + }), + ) + .await + .unwrap(); + let region = engine.get_region(region_id).unwrap(); + let guard = region.manifest_ctx.manifest_manager.write().await; + // A removal of a nonexistent file exercises manifest serialization without fake SSTs. + let e = engine.clone(); + let edit = tokio::spawn(async move { + e.edit_region( + region_id, + RegionEdit { + files_to_remove: vec![crate::sst::file::FileMeta { + region_id, + file_id: store_api::storage::FileId::random(), + ..Default::default() + }], + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(5), async { + while region.state() != RegionRoleState::Leader(RegionLeaderState::Editing) { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + let mut closes = Vec::new(); + for flush_on_close in flush_on_close { + let (sender, receiver) = tokio::sync::oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::Ddl(SenderDdlRequest { + region_id, + sender: sender.into(), + request: DdlRequest::Close(RegionCloseRequest { flush_on_close }), + }), + ) + .await + .unwrap(); + closes.push(receiver); + } + tokio::time::timeout( + std::time::Duration::from_secs(5), + flush_region(&engine, other_id, None), + ) + .await + .unwrap(); + for close in &mut closes { + assert!(matches!( + close.try_recv(), + Err(tokio::sync::oneshot::error::TryRecvError::Empty) + )); + } + drop(guard); + tokio::time::timeout(std::time::Duration::from_secs(5), edit) + .await + .unwrap() + .unwrap() + .unwrap(); + for close in closes { + assert_eq!( + 0, + tokio::time::timeout(std::time::Duration::from_secs(5), close) + .await + .unwrap() + .unwrap() + .unwrap() + ); + } + assert!(!engine.is_region_exists(region_id)); + // WAL replay could hide a skipped flush after reopening. Check the committed SST first. + assert!(!region.manifest_ctx.manifest().await.files.is_empty()); + crate::test_util::reopen_region(&engine, region_id, create.table_dir, true, create.options) + .await; + let stream = engine + .scan_to_stream(region_id, ScanRequest::default()) + .await + .unwrap(); + let batches = RecordBatches::try_collect(stream).await.unwrap(); + assert_eq!( + 3, + batches.iter().map(|batch| batch.num_rows()).sum::() + ); + engine.stop().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_engine_stop_waits_for_checkpoint() { + let (blocker, mock_layer) = CheckpointTaskBlocker::block_last_checkpoint_write(); + let mut env = TestEnv::with_prefix("stop-waits-for-checkpoint") + .await + .with_mock_layer(mock_layer); + let config = MitoConfig { + manifest_checkpoint_distance: 1, + ..Default::default() + }; + let engine = env.create_engine(config.clone()).await; + let region_id = RegionId::new(1, 1); + let request = CreateRequestBuilder::new().build(); + let table_dir = request.table_dir.clone(); + let column_schemas = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_rows( + &engine, + region_id, + Rows { + schema: column_schemas, + rows: build_rows(0, 3), + }, + ) + .await; + // The flush starts a checkpoint, which blocks after it has created `_last_checkpoint` + // and before it has written the pointer. + flush_region(&engine, region_id, None).await; + blocker.wait_until_blocked().await; + let wait_started = engine + .get_region(region_id) + .unwrap() + .manifest_ctx + .manifest_manager + .read() + .await + .checkpointer() + .pending_checkpoint_wait_started(); + + let mut reopen = tokio::spawn(async move { + let engine = env.reopen_engine(engine, config).await; + let open = RegionRequest::Open(RegionOpenRequest { + engine: String::new(), + table_dir, + path_type: PathType::Bare, + options: Default::default(), + skip_wal_replay: false, + checkpoint: None, + requirements: Default::default(), + }); + let result = engine.handle_request(region_id, open).await; + (env, result) + }); + tokio::time::timeout(Duration::from_secs(10), wait_started.notified()) + .await + .expect("the engine stop does not wait for the checkpoint"); + // The new engine would read the empty pointer while the old checkpoint is blocked. + assert!( + tokio::time::timeout(Duration::from_millis(500), &mut reopen) + .await + .is_err() + ); + blocker.release(); + let (_env, result) = tokio::time::timeout(Duration::from_secs(10), reopen) + .await + .unwrap() + .unwrap(); + result.unwrap(); +} diff --git a/src/mito2/src/engine/compaction_test.rs b/src/mito2/src/engine/compaction_test.rs index 62a95c3a774..bd626752925 100644 --- a/src/mito2/src/engine/compaction_test.rs +++ b/src/mito2/src/engine/compaction_test.rs @@ -2687,6 +2687,26 @@ async fn test_enter_staging_waits_for_picking_logical_cancellation_ack() { assert!(!compact_task.is_finished()); assert!(!staging_task.is_finished()); + // Moving EnterStaging into the compaction scheduler must keep import admission sealed. + let error = tokio::time::timeout( + Duration::from_secs(5), + engine.edit_region( + region_id, + crate::manifest::action::RegionEdit { + files_to_add: vec![crate::sst::file::FileMeta { + region_id, + file_id: store_api::storage::FileId::random(), + ..Default::default() + }], + ..Default::default() + }, + ), + ) + .await + .unwrap() + .unwrap_err(); + assert_eq!(StatusCode::RegionBusy, error.status_code()); + gate_guard.release(); let compact_err = tokio::time::timeout(Duration::from_secs(5), compact_task) .await diff --git a/src/mito2/src/engine/create_test.rs b/src/mito2/src/engine/create_test.rs index 895d15b4195..219757a4f5e 100644 --- a/src/mito2/src/engine/create_test.rs +++ b/src/mito2/src/engine/create_test.rs @@ -49,6 +49,163 @@ use crate::test_util::{ CreateRequestBuilder, TestEnv, build_rows, flush_region, put_rows, reopen_region, rows_schema, }; +#[tokio::test] +async fn test_engine_create_validates_extension_options_before_manifest() { + use crate::sst::index::IndexBuildType; + use crate::sst::index::handler::{ + IndexKindHandler, IndexKindProvider, IndexKindProviderRef, PlanContext, PlannedIndex, + }; + use crate::sst::index::kind::{ExtensionIndexKind, IndexKind}; + use common_base::Plugins; + use common_error::ext::BoxedError; + + #[derive(Debug)] + struct ValidatingHandler; + impl IndexKindHandler for ValidatingHandler { + fn kinds(&self) -> &[IndexKind] { + &[IndexKind::Extension(ExtensionIndexKind::Hll)] + } + fn validate(&self, context: &PlanContext<'_>) -> Result<(), BoxedError> { + for column in &context.metadata.column_metadatas { + if column + .column_schema + .metadata() + .get(datatypes::schema::HLL_KEY) + .map(String::as_str) + == Some(r#"{"precision":0}"#) + { + return Err(BoxedError::new(common_error::ext::PlainError::new( + "injected invalid extension option".into(), + StatusCode::InvalidArguments, + ))); + } + } + Ok(()) + } + fn plan( + &self, + context: &PlanContext<'_>, + ) -> Result>, BoxedError> { + // Creation must validate options even though this provider skips flush work. + if context.build_type == IndexBuildType::Flush { + return Ok(None); + } + self.validate(context)?; + Ok(None) + } + } + #[derive(Debug)] + struct Provider; + impl IndexKindProvider for Provider { + fn handlers(&self) -> Vec> { + vec![Arc::new(ValidatingHandler)] + } + } + let mut env = TestEnv::new().await; + let plugins = Plugins::new(); + plugins.insert::(Arc::new(Provider)); + let engine = env + .create_engine_with_plugins(MitoConfig::default(), plugins) + .await; + let region_id = RegionId::new(1, 1); + let mut request = CreateRequestBuilder::new().build(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":0}"#.into(), + ); + let dir = manifest_dir(®ion_dir_from_table_dir( + &request.table_dir, + region_id, + request.path_type, + )); + let error = engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap_err(); + assert_eq!(error.status_code(), StatusCode::InvalidArguments); + assert!(engine.get_region(region_id).is_none()); + assert!( + env.get_object_store() + .unwrap() + .list(&dir) + .await + .unwrap() + .is_empty() + ); + // A failed plugin validation must leave a clean path for a subsequent create. + let mut request = CreateRequestBuilder::new().build(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + engine + .handle_request(region_id, RegionRequest::Create(request.clone())) + .await + .unwrap(); + assert!(engine.get_region(region_id).is_some()); + + // Recovery and an idempotent Create do not require an optional provider. + let engine = env.reopen_engine(engine, MitoConfig::default()).await; + engine + .handle_request(region_id, RegionRequest::Create(request.clone())) + .await + .unwrap(); + reopen_region( + &engine, + region_id, + request.table_dir, + true, + Default::default(), + ) + .await; + assert!(engine.get_region(region_id).is_some()); +} + +#[tokio::test] +async fn test_engine_create_rejects_missing_sketch_provider_before_manifest() { + let mut env = TestEnv::new().await; + let engine = env.create_engine(MitoConfig::default()).await; + let region_id = RegionId::new(1, 1); + for (key, value) in [ + (datatypes::schema::HLL_KEY, r#"{"precision":10}"#), + (datatypes::schema::KLL_KEY, r#"{"capacity":64}"#), + (datatypes::schema::MG_KEY, r#"{"max-counters":64}"#), + (datatypes::schema::HLL_KEY, "invalid"), + ] { + let mut request = CreateRequestBuilder::new().build(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert(key.into(), value.into()); + let dir = manifest_dir(®ion_dir_from_table_dir( + &request.table_dir, + region_id, + request.path_type, + )); + let error = engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap_err(); + assert_eq!(error.status_code(), StatusCode::InvalidArguments); + assert!(error.to_string().contains("no registered handler")); + assert!(engine.get_region(region_id).is_none()); + assert!( + env.get_object_store() + .unwrap() + .list(&dir) + .await + .unwrap() + .is_empty() + ); + } +} + #[tokio::test] async fn test_engine_column_compression_validation() { use datatypes::schema::COMPRESSION_KEY; diff --git a/src/mito2/src/engine/edit_region_test.rs b/src/mito2/src/engine/edit_region_test.rs index f58daa55d34..10bf061bda4 100644 --- a/src/mito2/src/engine/edit_region_test.rs +++ b/src/mito2/src/engine/edit_region_test.rs @@ -45,6 +45,64 @@ use crate::sst::location; use crate::test_util::{CreateRequestBuilder, TestEnv, build_rows, rows_schema}; use crate::time_provider::mock::MockTimeProvider; +#[tokio::test] +async fn test_runtime_edits_share_manifest_file_metadata() { + for flush in [false, true] { + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let request = CreateRequestBuilder::new().build(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + if flush { + crate::test_util::put_rows( + &engine, + region_id, + api::v1::Rows { + schema, + rows: build_rows(0, 10), + }, + ) + .await; + crate::test_util::flush_region(&engine, region_id, None).await; + } else { + engine + .edit_region( + region_id, + RegionEdit { + files_to_add: vec![FileMeta { + region_id, + file_id: FileId::random(), + ..Default::default() + }], + ..Default::default() + }, + ) + .await + .unwrap(); + } + let region = engine.get_region(region_id).unwrap(); + let manifest = region.manifest_ctx.manifest().await; + let version = region.version(); + assert!(!manifest.files.is_empty()); + for handle in version.ssts.levels().iter().flat_map(|level| level.files()) { + let meta = &manifest.files[&handle.file_id().file_id()]; + assert!( + std::ptr::eq(meta.as_ref(), handle.meta_ref()), + "flush={flush}" + ); + } + } +} + #[tokio::test] async fn test_edit_region_schedule_compaction() { for flat_format in [true, false] { @@ -1213,7 +1271,7 @@ async fn test_drop_rejects_alter_and_edit_queues_after_flush_commit() { }); wait_worker_recv_event(&mut rx).await; - // The first ALTER waits in the flush scheduler; the second uses pending_alters. + // The first ALTER waits in the flush scheduler; the second uses pending_ddls. let mut alters = Vec::new(); for _ in 0..2 { let e = engine.clone(); @@ -1263,3 +1321,514 @@ async fn test_drop_rejects_alter_and_edit_queues_after_flush_commit() { .await .expect("DROP left queued ALTER or edit requests unresolved"); } + +/// Submits without awaiting the reply so a test can observe a deferred control request. +async fn submit_control( + engine: &MitoEngine, + region_id: RegionId, + request: crate::request::DdlRequest, +) -> oneshot::Receiver> { + let (sender, receiver) = oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::Ddl(crate::request::SenderDdlRequest { + region_id, + request, + sender: sender.into(), + }), + ) + .await + .unwrap(); + receiver +} + +#[tokio::test] +async fn test_edit_retirement_rejects_pending_requests() { + use crate::request::DdlRequest; + use store_api::region_request::RegionFlushRequest; + + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + for id in [region_id, other_id] { + engine + .handle_request( + id, + RegionRequest::Create(CreateRequestBuilder::new().build()), + ) + .await + .unwrap(); + } + let region = engine.get_region(region_id).unwrap(); + let mut guard = region.manifest_ctx.manifest_manager.write().await; + let e = engine.clone(); + let edit = tokio::spawn(async move { + e.edit_region(region_id, test_region_edit(region_id, FileId::random())) + .await + }); + wait_until_region_is_in_editing(®ion).await; + let mut flush = submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest::default()), + ) + .await; + let (sender, queued_edit) = oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::EditRegion(RegionEditRequest::new( + region_id, + test_region_edit(region_id, FileId::random()), + true, + sender, + )), + ) + .await + .unwrap(); + tokio::time::timeout( + Duration::from_secs(5), + crate::test_util::flush_region(&engine, other_id, None), + ) + .await + .unwrap(); + assert!(matches!( + flush.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + + // A stopped manager without reload pending forces edit completion to retire the region + // and return early. Its queued requests must finish without another dispatch notification. + guard.stop().await; + drop(guard); + tokio::time::timeout(Duration::from_secs(5), edit) + .await + .unwrap() + .unwrap() + .unwrap_err(); + let error = tokio::time::timeout(Duration::from_secs(5), flush) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert_eq!(StatusCode::RegionNotFound, error.status_code()); + let error = tokio::time::timeout(Duration::from_secs(5), queued_edit) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert_eq!(StatusCode::RegionNotFound, error.status_code()); + assert!(!engine.is_region_exists(region_id)); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_repartition_flush_waits_for_region_edit() { + use crate::request::DdlRequest; + use store_api::region_request::{RegionFlushReason, RegionFlushRequest}; + + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + for id in [region_id, other_id] { + engine + .handle_request( + id, + RegionRequest::Create(CreateRequestBuilder::new().build()), + ) + .await + .unwrap(); + } + let region = engine.get_region(region_id).unwrap(); + let guard = region.manifest_ctx.manifest_manager.write().await; + let e = engine.clone(); + let edit = tokio::spawn(async move { + e.edit_region(region_id, test_region_edit(region_id, FileId::random())) + .await + }); + wait_until_region_is_in_editing(®ion).await; + let mut flush = submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest { + reason: Some(RegionFlushReason::Repartition), + ..Default::default() + }), + ) + .await; + // The same worker processes this barrier after admitting the flush. It must remain free + // to serve another region while the first region's manifest lock is held. + tokio::time::timeout( + Duration::from_secs(5), + crate::test_util::flush_region(&engine, other_id, None), + ) + .await + .unwrap(); + let early_reply = flush.try_recv(); + assert!( + matches!(&early_reply, Err(oneshot::error::TryRecvError::Empty)), + "flush replied before the edit completed: {early_reply:?}" + ); + drop(guard); + tokio::time::timeout(Duration::from_secs(5), edit) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!( + 0, + tokio::time::timeout(Duration::from_secs(5), flush) + .await + .unwrap() + .unwrap() + .unwrap() + ); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_staging_drains_accepted_edits_before_later_controls() { + use crate::request::DdlRequest; + use store_api::region_request::{ + AlterKind, EnterStagingRequest, RegionAlterRequest, RegionFlushRequest, SetRegionOption, + StagingPartitionDirective, + }; + + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + for id in [region_id, other_id] { + engine + .handle_request( + id, + RegionRequest::Create(CreateRequestBuilder::new().build()), + ) + .await + .unwrap(); + } + let region = engine.get_region(region_id).unwrap(); + let guard = region.manifest_ctx.manifest_manager.write().await; + let e = engine.clone(); + let first = tokio::spawn(async move { + e.edit_region(region_id, test_region_edit(region_id, FileId::random())) + .await + }); + wait_until_region_is_in_editing(®ion).await; + let file_id = FileId::random(); + let (tx, queued) = oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::EditRegion(RegionEditRequest::new( + region_id, + test_region_edit(region_id, file_id), + false, + tx, + )), + ) + .await + .unwrap(); + crate::test_util::flush_region(&engine, other_id, None).await; + let staging = submit_control( + &engine, + region_id, + DdlRequest::EnterStaging(EnterStagingRequest { + partition_directive: StagingPartitionDirective::RejectAllWrites, + }), + ) + .await; + let mut alter = submit_control( + &engine, + region_id, + DdlRequest::Alter(RegionAlterRequest { + kind: AlterKind::SetRegionOptions { + options: vec![SetRegionOption::SkipWal(false)], + }, + }), + ) + .await; + let mut later = submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest::default()), + ) + .await; + crate::test_util::flush_region(&engine, other_id, None).await; + let err = tokio::time::timeout( + Duration::from_secs(5), + engine.edit_region(region_id, test_region_edit(region_id, FileId::random())), + ) + .await + .unwrap() + .unwrap_err(); + assert_eq!(StatusCode::RegionBusy, err.status_code()); + assert!(matches!( + later.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + assert!(matches!( + alter.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + drop(guard); + tokio::time::timeout(Duration::from_secs(5), first) + .await + .unwrap() + .unwrap() + .unwrap(); + tokio::time::timeout(Duration::from_secs(5), queued) + .await + .unwrap() + .unwrap() + .unwrap(); + tokio::time::timeout(Duration::from_secs(5), staging) + .await + .unwrap() + .unwrap() + .unwrap(); + tokio::time::timeout(Duration::from_secs(5), later) + .await + .unwrap() + .unwrap() + .unwrap(); + // ALTER is validated after staging has begun, so it fails rather than crossing the boundary. + assert!( + tokio::time::timeout(Duration::from_secs(5), alter) + .await + .unwrap() + .unwrap() + .is_err() + ); + assert!(region.is_staging()); + // The accepted import belongs to the normal manifest, not the disposable staging one. + assert!( + region + .manifest_ctx + .manifest() + .await + .files + .contains_key(&file_id) + ); + let err = engine + .edit_region(region_id, test_region_edit(region_id, FileId::random())) + .await + .unwrap_err(); + assert!(matches!( + err, + crate::error::Error::RejectWriteInStaging { .. } + )); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_merged_flush_allows_region_edits() { + use crate::request::DdlRequest; + use store_api::region_request::{RegionFlushReason, RegionFlushRequest}; + use tokio::sync::{Notify, RwLock}; + + #[derive(Default)] + struct Listener { + flushing: Notify, + gate: RwLock<()>, + } + #[async_trait::async_trait] + impl EventListener for Listener { + async fn on_flush_begin(&self, _: RegionId) { + self.flushing.notify_one(); + let _guard = self.gate.read().await; + } + } + for repartition in [false, true] { + let mut env = TestEnv::new().await; + let listener = Arc::new(Listener::default()); + let engine = env + .create_engine_with( + MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }, + None, + Some(listener.clone()), + None, + ) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + let create = CreateRequestBuilder::new().build(); + let schema = rows_schema(&create); + for id in [region_id, other_id] { + engine + .handle_request(id, RegionRequest::Create(create.clone())) + .await + .unwrap(); + } + crate::test_util::put_rows( + &engine, + region_id, + Rows { + schema, + rows: build_rows(0, 3), + }, + ) + .await; + let gate = listener.gate.write().await; + let first = submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest::default()), + ) + .await; + tokio::time::timeout(Duration::from_secs(5), listener.flushing.notified()) + .await + .unwrap(); + let merged = submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest { + reason: repartition.then_some(RegionFlushReason::Repartition), + ..Default::default() + }), + ) + .await; + crate::test_util::flush_region(&engine, other_id, None).await; + let (tx, mut edit) = oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::EditRegion(RegionEditRequest::new( + region_id, + test_region_edit(region_id, FileId::random()), + false, + tx, + )), + ) + .await + .unwrap(); + crate::test_util::flush_region(&engine, other_id, None).await; + // Both ordinary and repartition flushes retain concurrency with direct edits. + tokio::time::timeout(Duration::from_secs(5), &mut edit) + .await + .unwrap() + .unwrap() + .unwrap(); + drop(gate); + for flush in [first, merged] { + tokio::time::timeout(Duration::from_secs(5), flush) + .await + .unwrap() + .unwrap() + .unwrap(); + } + engine.stop().await.unwrap(); + } +} + +#[tokio::test] +async fn test_deferred_flush_queue_is_bounded_and_drains_after_role_change() { + use crate::request::DdlRequest; + use store_api::region_request::{RegionFlushReason, RegionFlushRequest}; + + let mut env = TestEnv::new().await; + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1, 1); + let other_id = RegionId::new(1, 2); + for id in [region_id, other_id] { + engine + .handle_request( + id, + RegionRequest::Create(CreateRequestBuilder::new().build()), + ) + .await + .unwrap(); + } + let region = engine.get_region(region_id).unwrap(); + let guard = region.manifest_ctx.manifest_manager.write().await; + let e = engine.clone(); + let edit = tokio::spawn(async move { + e.edit_region(region_id, test_region_edit(region_id, FileId::random())) + .await + }); + wait_until_region_is_in_editing(®ion).await; + let mut replies = Vec::new(); + for _ in 0..crate::worker::MAX_PENDING_DDLS_PER_REGION + 1 { + replies.push( + submit_control( + &engine, + region_id, + DdlRequest::Flush(RegionFlushRequest { + reason: Some(RegionFlushReason::Repartition), + ..Default::default() + }), + ) + .await, + ); + } + crate::test_util::flush_region(&engine, other_id, None).await; + let overflow = replies.pop().unwrap().await.unwrap().unwrap_err(); + assert_eq!(StatusCode::RegionBusy, overflow.status_code()); + for reply in &mut replies { + assert!(matches!( + reply.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + } + engine + .set_region_role(region_id, RegionRole::Follower) + .unwrap(); + drop(guard); + assert!( + tokio::time::timeout(Duration::from_secs(5), edit) + .await + .unwrap() + .unwrap() + .is_err() + ); + for reply in replies { + assert!( + tokio::time::timeout(Duration::from_secs(5), reply) + .await + .unwrap() + .unwrap() + .is_err() + ); + } + assert_eq!(RegionRoleState::Follower, region.state()); + engine.stop().await.unwrap(); +} diff --git a/src/mito2/src/engine/flush_test.rs b/src/mito2/src/engine/flush_test.rs index d01b77ddbd9..19a95383382 100644 --- a/src/mito2/src/engine/flush_test.rs +++ b/src/mito2/src/engine/flush_test.rs @@ -420,7 +420,7 @@ async fn test_ingested_parquet_bytes() { .edit_region( region_id, crate::manifest::action::RegionEdit { - files_to_add: vec![file], + files_to_add: vec![file.as_ref().clone()], files_to_remove: vec![], timestamp_ms: None, compaction_time_window: None, diff --git a/src/mito2/src/engine/index_build_test.rs b/src/mito2/src/engine/index_build_test.rs index 36ae38330ae..1e25a3a035a 100644 --- a/src/mito2/src/engine/index_build_test.rs +++ b/src/mito2/src/engine/index_build_test.rs @@ -15,23 +15,26 @@ //! Index build tests for mito engine. //! use std::collections::{HashMap, HashSet}; -use std::sync::Arc; -use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; use std::time::Duration; use api::v1::region::{StrictWindow, compact_request}; use api::v1::{Rows, SemanticType}; use async_trait::async_trait; +use common_base::Plugins; use common_base::readable_size::ReadableSize; -use common_error::ext::ErrorExt; +use common_error::ext::{BoxedError, ErrorExt, PlainError}; use common_error::status_code::StatusCode; use common_recordbatch::RecordBatches; use datafusion_expr::expr::InList; use datafusion_expr::{Expr, col, lit}; use datatypes::arrow::array::AsArray; use datatypes::arrow::datatypes::TimestampMillisecondType; +use datatypes::arrow::record_batch::RecordBatch; use datatypes::prelude::ConcreteDataType; -use datatypes::schema::{ColumnSchema, SkippingIndexOptions, SkippingIndexType}; +use datatypes::schema::{ColumnSchema, FulltextOptions, SkippingIndexOptions, SkippingIndexType}; +use puffin::puffin_manager::{PuffinManager, PuffinReader, PuffinWriter}; use store_api::metadata::ColumnMetadata; use store_api::region_engine::RegionEngine; use store_api::region_request::{ @@ -48,15 +51,23 @@ use crate::engine::MitoEngine; use crate::engine::compaction_test::put_and_flush; use crate::engine::listener::{EventListener, GateIndexBuildListener, IndexBuildListener}; use crate::manifest::action::RegionEdit; +use crate::read::Batch; use crate::read::scan_region::{ScanRegion, Scanner}; -use crate::sst::file::{FileMeta, RegionFileId, RegionIndexId}; +use crate::sst::file::{FileMeta, IndexType, RegionFileId, RegionIndexId}; +use crate::sst::index::handler::{ + BuildContext, IndexCreator, IndexKindHandler, IndexKindProvider, IndexKindProviderRef, + PlanContext, PlanEntry, PlannedIndex, +}; +use crate::sst::index::kind::{ExtensionIndexKind, IndexKind}; +use crate::sst::index::puffin_manager::SstPuffinWriter; +use crate::sst::index::{CompletedEntry, IndexBuildType}; use crate::sst::location; use crate::test_util::batch_util::sort_batches_and_print; use crate::test_util::{ CreateRequestBuilder, TestEnv, build_rows, flush_region, put_rows, reopen_region, rows_schema, }; -fn async_build_mode_config(is_create_on_flush: bool) -> MitoConfig { +pub(crate) fn async_build_mode_config(is_create_on_flush: bool) -> MitoConfig { let mut config = MitoConfig::default(); config.index.build_mode = IndexBuildMode::Async; if !is_create_on_flush { @@ -736,7 +747,8 @@ async fn test_index_build_uses_physical_file_region_and_logical_manifest_region( .manifest() .await .files - .get(&target_files[0].file_id), + .get(&target_files[0].file_id) + .map(AsRef::as_ref), Some(&target_files[0]) ); assert!(!current_file_metas(&engine, source_region_id).await[0].exists_index()); @@ -1178,22 +1190,23 @@ async fn test_consecutive_schema_changes_publish_latest_index_generation() { )); } +#[rstest::rstest] +#[case(IndexBuildType::Flush)] +#[case(IndexBuildType::Compact)] #[tokio::test] -async fn test_unrelated_schema_change_retries_stale_flush_index_build() { +async fn test_unrelated_schema_change_retries_background_index_without_provider( + #[case] build_type: IndexBuildType, +) { let mut env = TestEnv::with_prefix("test_unrelated_schema_change_retries_stale_index_build_").await; let gate = Arc::new(IndexPublicationGate::new( IndexPublicationPhase::BeforeManifestCommit, )); - let engine = Arc::new( - env.create_engine_with( - async_build_mode_config(true), - None, - Some(gate.clone()), - None, - ) - .await, - ); + let plugins = Plugins::new(); + plugins.insert::(Arc::new(RebuildSketchProvider::default())); + let mut setup_config = async_build_mode_config(false); + setup_config.index.build_mode = IndexBuildMode::Sync; + let setup_engine = env.create_engine_with_plugins(setup_config, plugins).await; let region_id = RegionId::new(1, 1); env.get_schema_metadata_manager() @@ -1207,16 +1220,60 @@ async fn test_unrelated_schema_change_retries_stale_flush_index_build() { ) .await; - let request = CreateRequestBuilder::new().build_with_index(); + let mut request = CreateRequestBuilder::new() + .insert_option("compaction.type", "twcs") + .insert_option("compaction.twcs.trigger_file_num", "4") + .build_with_index(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + let table_dir = request.table_dir.clone(); + let options = request.options.clone(); let column_schemas = rows_schema(&request); - engine + setup_engine .handle_request(region_id, RegionRequest::Create(request)) .await .unwrap(); - put_and_flush(&engine, region_id, &column_schemas, 0..20).await; + if build_type == IndexBuildType::Compact { + put_and_flush(&setup_engine, region_id, &column_schemas, 0..10).await; + put_and_flush(&setup_engine, region_id, &column_schemas, 10..20).await; + } + setup_engine.stop().await.unwrap(); + drop(setup_engine); + env.log_store = None; + let engine = Arc::new( + env.create_engine_with( + async_build_mode_config(build_type == IndexBuildType::Flush), + None, + Some(gate.clone()), + None, + ) + .await, + ); + reopen_region(&engine, region_id, table_dir, true, options).await; + if build_type == IndexBuildType::Flush { + put_and_flush(&engine, region_id, &column_schemas, 0..20).await; + } else { + engine + .handle_request( + region_id, + RegionRequest::Compact(RegionCompactRequest { + options: compact_request::Options::StrictWindow(StrictWindow { + window_seconds: 3600, + }), + ..Default::default() + }), + ) + .await + .unwrap(); + } tokio::time::timeout(Duration::from_secs(10), gate.wait_entered(1)) .await - .expect("flush index build did not reach manifest publication"); + .expect("background index build did not reach manifest publication"); engine .handle_request( @@ -1244,7 +1301,7 @@ async fn test_unrelated_schema_change_retries_stale_flush_index_build() { gate.release(1); tokio::time::timeout(Duration::from_secs(10), gate.wait_entered(2)) .await - .expect("stale flush index build was not retried"); + .expect("stale background index build was not retried"); gate.release(1); tokio::time::timeout(Duration::from_secs(10), gate.wait_stopped(2)) .await @@ -1266,6 +1323,11 @@ async fn test_unrelated_schema_change_retries_stale_flush_index_build() { .await .unwrap(); assert_eq!(num_of_index_files(&engine, &scanner, region_id).await, 1); + assert_eq!( + files[0].available_indexes.as_slice(), + &[IndexType::InvertedIndex] + ); + engine.stop().await.unwrap(); } #[tokio::test] @@ -2577,3 +2639,684 @@ async fn test_truncate_while_indexing_encoded_bulk_part_removes_uploaded_sst() { .collect::>(); assert!(leftovers.is_empty(), "{leftovers:?}"); } + +#[derive(Debug, Default, Clone)] +struct RebuildSketchProvider { + fail_manual_plan: bool, + memory_limit: Option, + manual_build: bool, + manual_finished: Arc, + manual_plans: Arc, + compact_plans: Arc, + creates: Arc, + readable_inputs: Arc>>, +} + +impl IndexKindProvider for RebuildSketchProvider { + fn handlers(&self) -> Vec> { + vec![Arc::new(self.clone())] + } +} + +impl IndexKindHandler for RebuildSketchProvider { + fn kinds(&self) -> &[IndexKind] { + &[IndexKind::Extension(ExtensionIndexKind::Hll)] + } + + fn validate(&self, _: &PlanContext<'_>) -> Result<(), BoxedError> { + Ok(()) + } + + fn plan(&self, context: &PlanContext<'_>) -> Result>, BoxedError> { + if context.build_type == IndexBuildType::Compact { + self.compact_plans.fetch_add(1, Ordering::Relaxed); + } + if context.build_type == IndexBuildType::Manual { + self.manual_plans.fetch_add(1, Ordering::Relaxed); + if self.fail_manual_plan { + return Err(BoxedError::new(PlainError::new( + "injected manual plan failure".into(), + StatusCode::Internal, + ))); + } + } + if context.build_type == IndexBuildType::Flush { + return Ok(None); + } + if matches!( + context.build_type, + IndexBuildType::Manual | IndexBuildType::SchemaChange + ) && self.memory_limit.is_none() + && !context + .existing_indexes + .iter() + .any(|column| column.created_indexes.contains(&IndexType::HllIndex)) + { + return Ok(None); + } + assert!( + context + .metadata + .column_by_id(0) + .unwrap() + .column_schema + .metadata() + .contains_key(datatypes::schema::HLL_KEY) + ); + Ok(Some(Arc::new(Self { + manual_build: context.build_type == IndexBuildType::Manual, + ..self.clone() + }))) + } +} + +#[async_trait] +impl PlannedIndex for RebuildSketchProvider { + fn kinds(&self) -> &[IndexKind] { + IndexKindHandler::kinds(self) + } + fn entries(&self) -> &[PlanEntry] { + &[PlanEntry { + column_id: 0, + kind: IndexKind::Extension(ExtensionIndexKind::Hll), + format_version: 1, + }] + } + fn is_current(&self, file: &FileMeta, _: &PlanContext<'_>) -> bool { + self.manual_finished.load(Ordering::Relaxed) + && file + .indexes + .iter() + .any(|column| column.created_indexes.contains(&IndexType::HllIndex)) + } + + fn memory_limit(&self) -> Option { + self.memory_limit + } + + async fn build( + &self, + context: &BuildContext<'_>, + ) -> Result>, BoxedError> { + for input in context.inputs.iter() { + // Initial compaction may read flush files that have no sketches yet. + if !input.meta_ref().indexes.iter().any(|column| { + column.created_indexes.iter().any(|kind| { + matches!( + kind, + IndexType::HllIndex | IndexType::KllIndex | IndexType::MgIndex + ) + }) + }) { + continue; + } + let reader = context + .input_puffin_manager + .reader(&input.index_id()) + .await + .unwrap(); + let metadata = reader.metadata().await.unwrap(); + assert!( + metadata + .blobs + .iter() + .any(|blob| blob.blob_type == "test.sketch") + ); + self.readable_inputs + .lock() + .unwrap() + .push((input.index_id(), context.can_merge_inputs)); + } + self.creates.fetch_add(1, Ordering::Relaxed); + Ok(Some(Box::new(RebuildSketchCreator { + rows: 0, + manual_finished: self.manual_build.then(|| self.manual_finished.clone()), + }))) + } +} + +#[derive(Debug)] +struct RebuildSketchCreator { + rows: usize, + manual_finished: Option>, +} + +#[async_trait] +impl IndexCreator for RebuildSketchCreator { + async fn update(&mut self, batch: &mut Batch) -> Result<(), BoxedError> { + self.rows += batch.num_rows(); + Ok(()) + } + + async fn update_flat(&mut self, batch: &RecordBatch) -> Result<(), BoxedError> { + self.rows += batch.num_rows(); + Ok(()) + } + + async fn finish( + &mut self, + writer: &mut SstPuffinWriter, + ) -> Result, BoxedError> { + writer + .put_blob( + "test.sketch", + futures::io::Cursor::new(self.rows.to_le_bytes().to_vec()), + Default::default(), + HashMap::new(), + ) + .await + .map_err(BoxedError::new)?; + if let Some(finished) = &self.manual_finished { + finished.store(true, Ordering::Relaxed); + } + Ok(vec![CompletedEntry { + column_id: Some(0), + kind: IndexKind::Extension(ExtensionIndexKind::Hll), + format_version: 1, + }]) + } + + async fn abort(&mut self) -> Result<(), BoxedError> { + Ok(()) + } + + fn memory_usage(&self) -> usize { + self.rows * std::mem::size_of::() + } +} + +#[rstest::rstest] +#[case(false)] +#[case(true)] +#[tokio::test] +async fn test_async_compaction_builds_only_builtin_indexes(#[case] registered: bool) { + let mut env = TestEnv::new().await; + let provider = Arc::new(RebuildSketchProvider::default()); + let plugins = Plugins::new(); + plugins.insert::(provider.clone()); + let mut setup_config = async_build_mode_config(false); + setup_config.index.build_mode = IndexBuildMode::Sync; + let engine = env.create_engine_with_plugins(setup_config, plugins).await; + let region_id = RegionId::new(1, 1); + env.get_schema_metadata_manager() + .register_region_table_info( + region_id.table_id(), + "test_table", + "test_catalog", + "test_schema", + None, + env.get_kv_backend(), + ) + .await; + let mut request = CreateRequestBuilder::new() + .insert_option("compaction.type", "twcs") + .insert_option("compaction.twcs.trigger_file_num", "4") + .build_with_index(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + let table_dir = request.table_dir.clone(); + let options = request.options.clone(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_and_flush(&engine, region_id, &schema, 0..2).await; + put_and_flush(&engine, region_id, &schema, 2..4).await; + let inputs = engine + .get_region(region_id) + .unwrap() + .all_manifest_files() + .await + .0; + assert_eq!(inputs.len(), 2); + assert!(inputs.iter().all(|file| file.available_indexes.is_empty())); + engine.stop().await.unwrap(); + + let plugins = Plugins::new(); + if registered { + plugins.insert::(provider.clone()); + } + let engine = env + .new_mito_engine_with_plugins(async_build_mode_config(false), plugins) + .await; + reopen_region(&engine, region_id, table_dir, true, options).await; + engine + .handle_request( + region_id, + RegionRequest::Compact(RegionCompactRequest { + options: compact_request::Options::StrictWindow(StrictWindow { + window_seconds: 3600, + }), + ..Default::default() + }), + ) + .await + .unwrap(); + + let region = engine.get_region(region_id).unwrap(); + tokio::time::timeout(Duration::from_secs(10), async { + loop { + let files = region.all_manifest_files().await.0; + if let [output] = files.as_slice() + && !inputs.iter().any(|input| input.file_id == output.file_id) + && output.available_indexes.contains(&IndexType::InvertedIndex) + { + assert_eq!( + output.available_indexes.as_slice(), + &[IndexType::InvertedIndex] + ); + assert_eq!(output.num_rows, 4); + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("async compaction did not publish built-in indexes"); + assert_eq!(provider.compact_plans.load(Ordering::Relaxed), 0); + assert_eq!(provider.creates.load(Ordering::Relaxed), 0); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_async_flush_without_sketch_provider_builds_builtin_indexes() { + let mut env = TestEnv::new().await; + let plugins = Plugins::new(); + plugins.insert::(Arc::new(RebuildSketchProvider::default())); + let mut setup_config = async_build_mode_config(false); + setup_config.index.build_mode = IndexBuildMode::Sync; + let engine = env.create_engine_with_plugins(setup_config, plugins).await; + let region_id = RegionId::new(1, 1); + let mut request = CreateRequestBuilder::new().build_with_index(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + request.column_metadatas[0] + .column_schema + .set_skipping_options(&SkippingIndexOptions::new_unchecked( + 1, + 0.0001, + SkippingIndexType::BloomFilter, + )) + .unwrap(); + request.column_metadatas[1].column_schema = + ColumnSchema::new("field_0", ConcreteDataType::string_datatype(), true); + request.column_metadatas[1] + .column_schema + .set_fulltext_options(&FulltextOptions { + enable: true, + ..Default::default() + }) + .unwrap(); + let rows = |start, end| { + let mut rows = build_rows(start, end); + for row in &mut rows { + row.values[1] = api::v1::Value { + value_data: Some(api::v1::value::ValueData::StringValue("sample text".into())), + }; + } + rows + }; + let table_dir = request.table_dir.clone(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_rows( + &engine, + region_id, + Rows { + schema: schema.clone(), + rows: rows(0, 2), + }, + ) + .await; + flush_region(&engine, region_id, None).await; + let region = engine.get_region(region_id).unwrap(); + let mut historical = region.all_manifest_files().await.0.pop().unwrap(); + historical.available_indexes.push(IndexType::HllIndex); + engine + .edit_region( + region_id, + RegionEdit { + files_to_add: vec![historical.clone()], + ..Default::default() + }, + ) + .await + .unwrap(); + let historical = region.all_manifest_files().await.0.pop().unwrap(); + + // Reopen without the provider. Neither schema options nor an unrelated + // historical sketch may prevent built-in indexes on the next flushed file. + let engine = env + .reopen_engine(engine, async_build_mode_config(true)) + .await; + reopen_region(&engine, region_id, table_dir, true, HashMap::new()).await; + put_rows( + &engine, + region_id, + Rows { + schema, + rows: rows(2, 4), + }, + ) + .await; + flush_region(&engine, region_id, None).await; + let region = engine.get_region(region_id).unwrap(); + tokio::time::timeout(Duration::from_secs(10), async { + loop { + let files = region.all_manifest_files().await.0; + if let Some(file) = files.iter().find(|file| file.file_id != historical.file_id) + && [ + IndexType::InvertedIndex, + IndexType::FulltextIndex, + IndexType::BloomFilterIndex, + ] + .iter() + .all(|kind| file.available_indexes.contains(kind)) + { + assert_eq!( + files.iter().find(|file| file.file_id == historical.file_id), + Some(&historical) + ); + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("async flush did not publish built-in indexes"); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_manual_rebuild_without_required_provider_preserves_sketch_metadata() { + let mut env = TestEnv::new().await; + let plugins = Plugins::new(); + plugins.insert::(Arc::new(RebuildSketchProvider { + memory_limit: Some(1024 * 1024), + ..Default::default() + })); + let mut config = async_build_mode_config(false); + config.index.build_mode = IndexBuildMode::Sync; + let engine = env + .create_engine_with_plugins(config.clone(), plugins) + .await; + let region_id = RegionId::new(1, 1); + let mut request = CreateRequestBuilder::new().build_with_index(); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + let table_dir = request.table_dir.clone(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_and_flush(&engine, region_id, &schema, 0..2).await; + engine + .handle_request( + region_id, + RegionRequest::BuildIndex(RegionBuildIndexRequest::default()), + ) + .await + .unwrap(); + let file = engine + .get_region(region_id) + .unwrap() + .all_manifest_files() + .await + .0 + .pop() + .unwrap(); + assert!(file.available_indexes.contains(&IndexType::HllIndex)); + + let engine = env.reopen_engine(engine, config).await; + reopen_region(&engine, region_id, table_dir, true, HashMap::new()).await; + for _ in 0..2 { + let error = engine + .handle_request( + region_id, + RegionRequest::BuildIndex(RegionBuildIndexRequest::default()), + ) + .await + .unwrap_err(); + assert!(format!("{error:?}").contains("no registered handler")); + assert_eq!( + engine + .get_region(region_id) + .unwrap() + .all_manifest_files() + .await + .0, + vec![file.clone()] + ); + } + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_targeted_rebuild_drops_unconfigured_sketch_metadata() { + let mut env = TestEnv::new().await; + let mut config = async_build_mode_config(false); + config.index.build_mode = IndexBuildMode::Sync; + let engine = env.create_engine(config).await; + let region_id = RegionId::new(1, 1); + let request = CreateRequestBuilder::new().build_with_index(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + for start in [0, 2] { + put_rows( + &engine, + region_id, + Rows { + schema: schema.clone(), + rows: build_rows(start, start + 2), + }, + ) + .await; + flush_region(&engine, region_id, None).await; + } + let region = engine.get_region(region_id).unwrap(); + let mut files = region.all_manifest_files().await.0; + assert_eq!(files.len(), 2); + let target = files.pop().unwrap(); + let mut historical = files.pop().unwrap(); + historical.available_indexes.push(IndexType::HllIndex); + engine + .edit_region( + region_id, + RegionEdit { + files_to_add: vec![historical.clone()], + ..Default::default() + }, + ) + .await + .unwrap(); + + for file in [&target, &historical] { + let (result_tx, mut result_rx) = tokio::sync::mpsc::channel(1); + engine + .inner + .workers + .submit_to_worker( + region_id, + crate::request::WorkerRequest::Background { + region_id, + notify: crate::request::BackgroundNotify::IndexBuildRetry( + crate::request::BuildIndexRequest { + region_id, + build_type: IndexBuildType::SchemaChange, + file_metas: vec![file.clone()], + result_senders: vec![result_tx], + }, + ), + }, + ) + .await + .unwrap(); + let outcome = tokio::time::timeout(Duration::from_secs(10), result_rx.recv()) + .await + .unwrap() + .unwrap(); + outcome.unwrap(); + if file.file_id == target.file_id { + let current = region.all_manifest_files().await.0; + let unrelated = current + .iter() + .find(|file| file.file_id == historical.file_id) + .unwrap(); + assert_eq!(unrelated.available_indexes, historical.available_indexes); + assert_eq!(unrelated.index_version, historical.index_version); + } + } + let current = region.all_manifest_files().await.0; + assert!( + current + .iter() + .find(|file| file.file_id == target.file_id) + .unwrap() + .available_indexes + .contains(&IndexType::InvertedIndex) + ); + let historical_current = current + .iter() + .find(|file| file.file_id == historical.file_id) + .unwrap(); + assert_eq!( + historical_current.available_indexes.as_slice(), + &[IndexType::InvertedIndex] + ); + assert!(historical_current.index_version > historical.index_version); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_manual_rebuild_plan_failure_returns_before_scheduling() { + let mut env = TestEnv::new().await; + let plugins = Plugins::new(); + let provider = Arc::new(RebuildSketchProvider { + fail_manual_plan: true, + ..Default::default() + }); + plugins.insert::(provider.clone()); + let engine = env + .create_engine_with_plugins(async_build_mode_config(false), plugins) + .await; + let region_id = RegionId::new(1, 1); + let request = CreateRequestBuilder::new().build(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_rows( + &engine, + region_id, + Rows { + schema, + rows: build_rows(0, 2), + }, + ) + .await; + flush_region(&engine, region_id, None).await; + let region = engine.get_region(region_id).unwrap(); + let before = region.all_manifest_files().await.0; + assert_eq!(before.len(), 1); + let error = engine + .handle_request( + region_id, + RegionRequest::BuildIndex(RegionBuildIndexRequest::default()), + ) + .await + .unwrap_err(); + assert!(format!("{error:?}").contains("injected manual plan failure")); + // Selection and task preparation would each plan again; only entry preflight ran. + assert_eq!(provider.manual_plans.load(Ordering::Relaxed), 1); + assert_eq!(provider.creates.load(Ordering::Relaxed), 0); + assert_eq!(region.all_manifest_files().await.0, before); + engine.stop().await.unwrap(); +} + +#[tokio::test] +async fn test_manual_rebuild_reports_budget_discard_of_new_plugin() { + let mut env = TestEnv::new().await; + let plugins = Plugins::new(); + let provider = Arc::new(RebuildSketchProvider { + memory_limit: Some(1), + ..Default::default() + }); + plugins.insert::(provider.clone()); + let engine = env + .create_engine_with_plugins(async_build_mode_config(false), plugins) + .await; + let region_id = RegionId::new(1, 1); + let mut request = CreateRequestBuilder::new().build(); + request.column_metadatas[0] + .column_schema + .set_inverted_index(true); + request.column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_rows( + &engine, + region_id, + Rows { + schema, + rows: build_rows(0, 2), + }, + ) + .await; + flush_region(&engine, region_id, None).await; + let error = engine + .handle_request( + region_id, + RegionRequest::BuildIndex(RegionBuildIndexRequest::default()), + ) + .await + .unwrap_err(); + assert!( + error.to_string().contains("Index build did not complete"), + "{error:?}" + ); + assert_eq!(error.status_code(), StatusCode::IllegalState); + assert_eq!(provider.creates.load(Ordering::Relaxed), 1); + let region = engine.get_region(region_id).unwrap(); + let files = region.all_manifest_files().await.0; + assert_eq!(files.len(), 1); + assert!(!files[0].available_indexes.contains(&IndexType::HllIndex)); + assert!( + files[0] + .available_indexes + .contains(&IndexType::InvertedIndex) + ); + engine.stop().await.unwrap(); +} diff --git a/src/mito2/src/engine/lease_test.rs b/src/mito2/src/engine/lease_test.rs index 5fa0a897d63..dcce6b57e1a 100644 --- a/src/mito2/src/engine/lease_test.rs +++ b/src/mito2/src/engine/lease_test.rs @@ -28,8 +28,8 @@ use rstest::rstest; use rstest_reuse::apply; use store_api::region_engine::{RegionEngine, RegionRole, SettableRegionRoleState}; use store_api::region_request::{ - EnterStagingRequest, RegionCatchupRequest, RegionPutRequest, RegionRequest, - StagingPartitionDirective, + EnterStagingRequest, RegionCatchupRequest, RegionCloseRequest, RegionDropRequest, + RegionPutRequest, RegionRequest, StagingPartitionDirective, }; use store_api::storage::{RegionId, ScanRequest}; use tokio::task::JoinHandle; @@ -39,6 +39,7 @@ use crate::engine::MitoEngine; use crate::engine::basic_test::build_bulk_insert_request; use crate::engine::listener::WriteWalListener; use crate::error::Error; +use crate::lease::GrantedLease; use crate::region::{RegionLeaderState, RegionRoleState}; use crate::test_util::{ CreateRequestBuilder, LogStoreFactory, TestEnv, build_rows, kafka_log_store_factory, @@ -72,18 +73,34 @@ async fn create_region( fn grant(engine: &MitoEngine, role: RegionRole, deadline: Instant) { engine - .set_region_role_with_lease(REGION_ID, role, Instant::now(), deadline) + .set_region_role_with_lease(REGION_ID, role, Instant::now(), deadline, None) .unwrap(); } -/// Grants Leader for a heartbeat sent at `sent_at`, and returns whether the role was applied. -fn grant_leader_at(engine: &MitoEngine, sent_at: Instant) -> bool { - let deadline = Instant::now() + LEASE; +/// Grants `role` with `candidate_lease_id` for a heartbeat sent at `sent_at`, and returns whether +/// the region applied the role. +fn grant_at( + engine: &MitoEngine, + role: RegionRole, + sent_at: Instant, + candidate_lease_id: Option, +) -> bool { engine - .set_region_role_with_lease(REGION_ID, RegionRole::Leader, sent_at, deadline) + .set_region_role_with_lease( + REGION_ID, + role, + sent_at, + Instant::now() + LEASE, + candidate_lease_id, + ) .unwrap() } +/// Returns the latest lease the region recorded. +fn granted_lease(engine: &MitoEngine) -> Option { + engine.inner.workers.granted_leases().get(REGION_ID) +} + /// Returns a request that writes rows `start..start + 3`. fn put_request(schema: &[ColumnSchema], start: usize) -> RegionRequest { let rows = Rows { @@ -319,11 +336,11 @@ async fn test_stale_leader_grant_after_downgrade(#[case] downgraded_to: Settable .unwrap(); // Grants for heartbeats sent before either downgrade arrive late. - assert!(!grant_leader_at(&engine, sent_before)); - assert!(!grant_leader_at(&engine, sent_between)); + assert!(!grant_at(&engine, RegionRole::Leader, sent_before, None)); + assert!(!grant_at(&engine, RegionRole::Leader, sent_between, None)); assert_eq!(downgraded, engine.get_region(REGION_ID).unwrap().state()); - assert!(grant_leader_at(&engine, Instant::now())); + assert!(grant_at(&engine, RegionRole::Leader, Instant::now(), None)); assert!(engine.get_region(REGION_ID).unwrap().is_writable()); } @@ -356,7 +373,7 @@ async fn test_reopened_region_ignores_earlier_grants(#[case] close: bool) { &engine.get_region(REGION_ID).unwrap() )); - assert!(!grant_leader_at(&engine, sent_before)); + assert!(!grant_at(&engine, RegionRole::Leader, sent_before, None)); assert!(!engine.get_region(REGION_ID).unwrap().is_writable()); grant(&engine, RegionRole::Leader, Instant::now() + LEASE); @@ -384,3 +401,98 @@ async fn test_leader_grant_exits_staging() { state() ); } + +#[tokio::test] +async fn test_granted_lease_keeps_latest_heartbeat_across_catchup_reopen() { + common_telemetry::init_default_ut_logging(); + let mut env = TestEnv::with_prefix("granted-lease-reopen").await; + let (_listener, engine, schema) = create_region(&mut env, None).await; + grant(&engine, RegionRole::Leader, Instant::now() + LEASE); + // The memtable is not empty, so the catchup below reopens the region. + put(&engine, REGION_ID, &schema, 0).await.unwrap().unwrap(); + // After the first grant and before the downgrade. + let sent_before = Instant::now(); + engine + .set_region_role_state_gracefully(REGION_ID, SettableRegionRoleState::Follower) + .await + .unwrap(); + + // A Leader lease the region ignores still counts as the latest one received. + assert!(!grant_at(&engine, RegionRole::Leader, sent_before, Some(9))); + let latest = GrantedLease { + candidate_lease_id: Some(9), + sent_at: sent_before, + }; + assert_eq!(granted_lease(&engine), Some(latest)); + // A lease of an earlier or the same heartbeat that arrives late does not replace it. + let earlier = sent_before - Duration::from_millis(1); + grant_at(&engine, RegionRole::Follower, earlier, Some(1)); + assert_eq!(granted_lease(&engine), Some(latest)); + grant_at(&engine, RegionRole::Follower, sent_before, None); + assert_eq!(granted_lease(&engine), Some(latest)); + + let region = engine.get_region(REGION_ID).unwrap(); + let request = RegionRequest::Catchup(RegionCatchupRequest::default()); + engine.handle_request(REGION_ID, request).await.unwrap(); + assert!(!Arc::ptr_eq( + ®ion, + &engine.get_region(REGION_ID).unwrap() + )); + assert_eq!(granted_lease(&engine), Some(latest)); + + let later = Instant::now(); + grant_at(&engine, RegionRole::Follower, later, None); + assert_eq!( + granted_lease(&engine), + Some(GrantedLease { + candidate_lease_id: None, + sent_at: later, + }) + ); +} + +#[rstest] +#[case::close(RegionRequest::Close(RegionCloseRequest::default()))] +#[case::close_after_flush(RegionRequest::Close(RegionCloseRequest { flush_on_close: true }))] +#[case::drop(RegionRequest::Drop(RegionDropRequest { + fast_path: false, + force: false, + partial_drop: false, +}))] +#[tokio::test] +async fn test_granted_lease_removed_with_region(#[case] request: RegionRequest) { + common_telemetry::init_default_ut_logging(); + let mut env = TestEnv::with_prefix("granted-lease-removed").await; + let (_listener, engine, schema) = create_region(&mut env, None).await; + grant(&engine, RegionRole::Leader, Instant::now() + LEASE); + // The memtable is not empty, so a close with `flush_on_close` closes after the flush. + put(&engine, REGION_ID, &schema, 0).await.unwrap().unwrap(); + assert!(granted_lease(&engine).is_some()); + + engine.handle_request(REGION_ID, request).await.unwrap(); + + assert!(!engine.is_region_exists(REGION_ID)); + assert_eq!(granted_lease(&engine), None); + // A lease that arrives after the removal is not recorded. + let _ = engine.set_region_role_with_lease( + REGION_ID, + RegionRole::Follower, + Instant::now(), + Instant::now() + LEASE, + Some(3), + ); + assert_eq!(granted_lease(&engine), None); +} + +#[tokio::test] +async fn test_granted_lease_removed_on_engine_stop() { + common_telemetry::init_default_ut_logging(); + let mut env = TestEnv::with_prefix("granted-lease-stop").await; + let (_listener, engine, _schema) = create_region(&mut env, None).await; + grant_at(&engine, RegionRole::Follower, Instant::now(), Some(7)); + assert!(granted_lease(&engine).is_some()); + + engine.stop().await.unwrap(); + + assert_eq!(granted_lease(&engine), None); +} diff --git a/src/mito2/src/engine/listener.rs b/src/mito2/src/engine/listener.rs index 3ca23151339..e34fdcd6975 100644 --- a/src/mito2/src/engine/listener.rs +++ b/src/mito2/src/engine/listener.rs @@ -23,6 +23,7 @@ use common_telemetry::info; use store_api::storage::{FileId, RegionId}; use tokio::sync::{Notify, Semaphore}; +use crate::error::Error; use crate::sst::file::RegionFileId; /// Mito engine background event listener. @@ -37,6 +38,22 @@ pub trait EventListener: Send + Sync { /// Notifies the listener before a catchup reopens the region. async fn on_catchup_reopen_begin(&self, _region_id: RegionId) {} + /// Notifies the listener before a tail replay commits its result. + async fn on_tail_commit_begin(&self, _region_id: RegionId) {} + + /// Notifies the listener each time a tail replay checks for a Leader lease while it waits. + fn on_tail_lease_check(&self, _region_id: RegionId) {} + + /// Notifies the listener before a tail replay that found a newer manifest waits to fence + /// the region. + fn on_tail_fence_begin(&self, _region_id: RegionId) {} + + /// Notifies the listener that a round of tail replay failed. + fn on_tail_replay_failed(&self, _region_id: RegionId, _error: &Error) {} + + /// Notifies the listener after the worker learns that a tail replay ended. + fn on_tail_replay_ended(&self, _region_id: RegionId) {} + /// Notifies the listener that a region is flushed successfully. fn on_flush_success(&self, region_id: RegionId) { let _ = region_id; @@ -128,6 +145,10 @@ pub trait EventListener: Send + Sync { /// Notifies the listener that the index build task is aborted. async fn on_index_build_abort(&self, _region_file_id: RegionFileId) {} + /// Notifies the listener that the worker learned an index build task stopped, after it + /// handled any completion the task reported before. + async fn on_index_build_stopped(&self, _region_file_id: RegionFileId) {} + /// Notifies the listener after the final source check and before manifest publication. async fn on_index_build_before_manifest_commit(&self, _region_file_id: RegionFileId) {} @@ -190,6 +211,34 @@ impl EventListener for StallListener { } } +/// Counts write stalls, for tests that wait for more than one. +#[derive(Default)] +pub struct CountingStallListener { + count: AtomicUsize, + notify: Notify, +} + +impl CountingStallListener { + /// Waits until the engine has stalled writes at least `expected` times. + pub async fn wait_for_count(&self, expected: usize) { + loop { + let notified = self.notify.notified(); + if self.count.load(Ordering::Relaxed) >= expected { + return; + } + notified.await; + } + } +} + +#[async_trait] +impl EventListener for CountingStallListener { + fn on_write_stall(&self) { + self.count.fetch_add(1, Ordering::Relaxed); + self.notify.notify_waiters(); + } +} + /// Blocks a flush at its begin event until the worker requests flush cancellation. /// /// Tests use this gate to ensure a flush is active before submitting a lifecycle DDL. diff --git a/src/mito2/src/engine/object_store_wal_recovery_test.rs b/src/mito2/src/engine/object_store_wal_recovery_test.rs index 279014e1e9b..bd2cad053ee 100644 --- a/src/mito2/src/engine/object_store_wal_recovery_test.rs +++ b/src/mito2/src/engine/object_store_wal_recovery_test.rs @@ -34,6 +34,7 @@ use common_wal::config::object_store::{ AckMode, CorruptedSegmentAction, ObjectStoreWalConfig, STANDALONE_GENERATION, }; use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions}; +use futures::TryStreamExt; use log_store::ObjectStoreLogStore; use log_store::metrics::{ METRIC_OBJECT_STORE_WAL_FAILING, METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL, @@ -56,6 +57,8 @@ use store_api::storage::{RegionId, ScanRequest}; use crate::config::MitoConfig; use crate::engine::MitoEngine; +use crate::engine::pending_tail_test::{set_pending_tails, tail}; +use crate::manifest::action::{PendingTail, UptoKind}; use crate::region::{MitoRegionRef, RegionLeaderState, RegionRoleState}; use crate::test_util::{ CreateRequestBuilder, TestEnv, build_rows, build_rows_for_key, flush_region, put_rows, @@ -1975,6 +1978,299 @@ async fn test_enqueued_writes_fail_fast_while_the_object_store_refuses_the_backl ); } +#[rstest] +#[case(AckMode::Durable)] +#[case(AckMode::Enqueued)] +#[tokio::test] +async fn test_close_persists_bounded_release_without_losing_unflushed_rows( + #[case] ack_mode: AckMode, +) { + let mut env = TestEnv::with_prefix("close-bounded-release").await; + let object_store = memory_store(); + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + let (table_dir, schema) = create_region(&engine, REGION_A, &[]).await; + let (other_dir, other_schema) = create_region(&engine, REGION_B, &[]).await; + let mut writer = SealedWriter::new(&store); + writer + .put_and_seal( + &engine, + vec![ + (REGION_A, rows(&schema, 0, 3)), + (REGION_B, rows(&other_schema, 0, 2)), + ], + ) + .await; + flush_region(&engine, REGION_A, None).await; + writer + .put_and_seal(&engine, vec![(REGION_A, rows(&schema, 3, 6))]) + .await; + let expected = scan_rows(&engine, REGION_A).await; + let expected_other = scan_rows(&engine, REGION_B).await; + // Region B pins the shared object: reclaim-boundary recovery alone cannot + // discard region A's flushed entries without the close release record. + let before = wal_object_seqs(&object_store).await; + tokio::time::timeout(WAIT, close_region(&engine, REGION_A)) + .await + .unwrap(); + assert_eq!(before, wal_object_seqs(&object_store).await); + // The close completes without an upload; sealing persists its bounded release. + store.seal_open_batch().await.unwrap(); + drop(writer); + crash(engine, store).await; + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + // Recovery uses the release even before this region is reopened. + let batches = store.read(&provider(REGION_A), 0).await.unwrap(); + let entries: Vec<_> = batches.try_collect().await.unwrap(); + assert_eq!(1, entries.iter().map(|batch| batch.len()).sum::()); + open_region(&engine, REGION_A, &table_dir, &[]) + .await + .unwrap(); + assert_eq!(expected, scan_rows(&engine, REGION_A).await); + open_region(&engine, REGION_B, &other_dir, &[]) + .await + .unwrap(); + assert_eq!(expected_other, scan_rows(&engine, REGION_B).await); + engine.stop().await.unwrap(); +} + +#[rstest] +#[case::local_tail(true)] +#[case::foreign_tail(false)] +#[tokio::test] +async fn test_close_bounded_release_preserves_pending_tails( + #[case] local_tail: bool, + #[values(AckMode::Durable, AckMode::Enqueued)] ack_mode: AckMode, +) { + let mut env = TestEnv::with_prefix("close-pending-tails") + .await + .with_node_id(0); + let object_store = memory_store(); + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + let (table_dir, schema) = create_region(&engine, REGION_A, &[]).await; + let (_, other_schema) = create_region(&engine, REGION_B, &[]).await; + let mut writer = SealedWriter::new(&store); + writer + .put_and_seal( + &engine, + vec![ + (REGION_A, rows(&schema, 0, 1)), + (REGION_B, rows(&other_schema, 0, 1)), + ], + ) + .await; + for i in 1..3 { + writer + .put_and_seal(&engine, vec![(REGION_A, rows(&schema, i, i + 1))]) + .await; + } + // A foreign tail with a lower bound must not cap this node's release. + let mut tails = vec![tail(1, 0, entry_id(3, 1))]; + if local_tail { + tails.push(tail(0, entry_id(2, 1), entry_id(3, 1))); + tails.push(tail(0, entry_id(1, 1), entry_id(2, 1))); + } + set_pending_tails(&engine, REGION_A, tails.clone()).await; + flush_region(&engine, REGION_A, None).await; + assert_eq!( + entry_id(3, 1), + engine + .get_region(REGION_A) + .unwrap() + .version() + .flushed_entry_id + ); + close_region(&engine, REGION_A).await; + store.seal_open_batch().await.unwrap(); + drop(writer); + crash(engine, store).await; + + let store = open_store(&object_store, ack_mode).await; + let batches: Vec<_> = store + .read(&provider(REGION_A), 0) + .await + .unwrap() + .try_collect() + .await + .unwrap(); + let ids: Vec<_> = batches + .into_iter() + .flatten() + .map(|entry| entry.entry_id()) + .collect(); + let expected = if local_tail { + vec![entry_id(2, 1), entry_id(3, 1)] + } else { + vec![] + }; + assert_eq!(expected, ids); + let engine = new_engine(&mut env, store).await; + open_region(&engine, REGION_A, &table_dir, &[]) + .await + .unwrap(); + assert_eq!( + tails, + engine.get_region(REGION_A).unwrap().version().pending_tails + ); + engine.stop().await.unwrap(); +} + +#[rstest] +#[case(AckMode::Durable)] +#[case(AckMode::Enqueued)] +#[tokio::test] +async fn test_close_reopens_after_its_release_record_is_collected(#[case] ack_mode: AckMode) { + let mut env = TestEnv::with_prefix("close-collected-release").await; + let object_store = memory_store(); + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + let (dir_a, schema_a) = create_region(&engine, REGION_A, &[]).await; + let (dir_b, schema_b) = create_region(&engine, REGION_B, &[]).await; + let mut writer = SealedWriter::new(&store); + for i in 0..9 { + let (region_id, schema) = if i < 5 { + (REGION_A, &schema_a) + } else { + (REGION_B, &schema_b) + }; + writer + .put_and_seal(&engine, vec![(region_id, rows(schema, i, i + 1))]) + .await; + } + let expected_a = scan_rows(&engine, REGION_A).await; + let expected_b = scan_rows(&engine, REGION_B).await; + // A's unflushed data pins the prefix while the higher-watermark B closes. + flush_region(&engine, REGION_B, None).await; + assert_eq!( + entry_id(9, 1), + region(&engine, REGION_B).version().flushed_entry_id + ); + close_region(&engine, REGION_B).await; + store.seal_open_batch().await.unwrap(); + wait_for_collection(&store).await; + assert!(wal_object_seqs(&object_store).await.contains(&10)); + flush_region(&engine, REGION_A, None).await; + assert_eq!( + entry_id(5, 1), + region(&engine, REGION_A).version().flushed_entry_id + ); + close_region(&engine, REGION_A).await; + store.seal_open_batch().await.unwrap(); + wait_for_collection(&store).await; + // Only A's release remains: neither B's entries nor its release can + // short-circuit the history check after restart. + assert_eq!(vec![11], wal_object_seqs(&object_store).await); + drop(writer); + crash(engine, store).await; + for _ in 0..2 { + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + open_region(&engine, REGION_B, &dir_b, &[]).await.unwrap(); + open_region(&engine, REGION_A, &dir_a, &[]).await.unwrap(); + assert_eq!(expected_b, scan_rows(&engine, REGION_B).await); + assert_eq!(expected_a, scan_rows(&engine, REGION_A).await); + crash(engine, store).await; + } +} + +#[rstest] +#[case::stopped(true)] +#[case::poisoned(false)] +#[tokio::test] +async fn test_close_handles_unavailable_bounded_release(#[case] stopped: bool) { + let mut env = TestEnv::with_prefix("close-release-admission-failure").await; + let store = open_store(&memory_store(), AckMode::Durable).await; + let engine = new_engine(&mut env, store.clone()).await; + create_region(&engine, REGION_A, &[]).await; + if stopped { + store.stop().await.unwrap(); + } else { + // Exhausting the sequence poisons the live actor: this is a real + // admission failure, unlike the expected stop-before-close order. + assert!( + store + .obsolete(&provider(REGION_A), REGION_A, u64::MAX) + .await + .is_err() + ); + } + let failures = crate::metrics::WAL_RELEASE_ADMISSION_FAILURE_TOTAL.get(); + close_region(&engine, REGION_A).await; + assert!(engine.get_region(REGION_A).is_none()); + assert_eq!( + failures + u64::from(!stopped), + crate::metrics::WAL_RELEASE_ADMISSION_FAILURE_TOTAL.get() + ); + engine.stop().await.unwrap(); +} + +#[rstest] +#[case(AckMode::Durable)] +#[case(AckMode::Enqueued)] +#[tokio::test] +async fn test_flush_on_close_persists_bounded_release(#[case] ack_mode: AckMode) { + let mut env = TestEnv::with_prefix("flush-on-close-bounded-release").await; + let object_store = memory_store(); + let store = open_store(&object_store, ack_mode).await; + let engine = new_engine(&mut env, store.clone()).await; + let (table_dir, schema) = create_region(&engine, REGION_A, &[]).await; + let (_, other_schema) = create_region(&engine, REGION_B, &[]).await; + let mut writer = SealedWriter::new(&store); + writer + .put_and_seal( + &engine, + vec![ + (REGION_A, rows(&schema, 0, 3)), + (REGION_B, rows(&other_schema, 0, 2)), + ], + ) + .await; + let expected = scan_rows(&engine, REGION_A).await; + tokio::time::timeout( + WAIT, + engine.handle_request( + REGION_A, + RegionRequest::Close(RegionCloseRequest { + flush_on_close: true, + }), + ), + ) + .await + .unwrap() + .unwrap(); + assert!(engine.get_region(REGION_A).is_none()); + store.seal_open_batch().await.unwrap(); + drop(writer); + crash(engine, store).await; + let store = open_store(&object_store, ack_mode).await; + // B pins the shared object. Before A is reopened, only the persisted + // release can hide its entries; the manifest has not supplied its floor. + let entries: Vec<_> = store + .read(&provider(REGION_A), 0) + .await + .unwrap() + .try_collect() + .await + .unwrap(); + assert!(entries.iter().all(Vec::is_empty)); + let entries: Vec<_> = store + .read(&provider(REGION_B), 0) + .await + .unwrap() + .try_collect() + .await + .unwrap(); + assert_eq!(1, entries.iter().map(Vec::len).sum::()); + let engine = new_engine(&mut env, store.clone()).await; + open_region(&engine, REGION_A, &table_dir, &[]) + .await + .unwrap(); + assert_eq!(expected, scan_rows(&engine, REGION_A).await); + engine.stop().await.unwrap(); +} + /// Reports the actual scheduler state before the worker can process flush /// completion. This distinguishes skipped admission from a later commit failure. struct PressureAdmissionListener { @@ -2190,6 +2486,61 @@ async fn test_wal_pressure_flushes_blockers_across_workers(#[case] state: Region store.stop().await.unwrap(); } +/// Pressure leaves alone a region whose own pending tail starts at or below its flushed entry id, +/// since a flush cannot move its WAL release past that tail, and flushes any other region. +#[rstest] +#[case::own_tail(0, 0, false)] +#[case::own_tail_ahead(0, 1, true)] +#[case::other_node_tail(1, 0, true)] +#[tokio::test] +async fn test_wal_pressure_skips_regions_capped_by_own_tails( + #[case] node_id: u64, + #[case] after: u64, + #[case] flushed: bool, +) { + use crate::time_provider::mock::MockTimeProvider; + + let mut env = TestEnv::new().await.with_node_id(0); + let store = open_store(&memory_store(), AckMode::Durable).await; + let now = common_time::util::current_time_millis(); + let clock = Arc::new(MockTimeProvider::new(now)); + let (checked, mut observations) = tokio::sync::mpsc::unbounded_channel(); + let engine = env + .create_engine_with_log_store_and_time( + MitoConfig { + auto_flush_interval: Duration::from_secs(3600), + ..Default::default() + }, + store.clone(), + clock.clone(), + Some(Arc::new(PressureAdmissionListener { checked })), + ) + .await; + let (_, schema) = create_region(&engine, REGION_A, &[]).await; + let mut writer = SealedWriter::new(&store); + for i in 0..=1_200 { + writer + .put_and_seal(&engine, vec![(REGION_A, rows(&schema, i, i + 1))]) + .await; + } + assert_eq!(vec![REGION_A], store.flush_candidates()); + let tail = PendingTail { + node_id, + after, + upto: latest(&store, REGION_A), + upto_kind: UptoKind::Reported, + }; + set_pending_tails(&engine, REGION_A, vec![tail]).await; + + clock.set_now(now + 120_000); + clock.set_elapsed(120_000); + + assert_eq!( + flushed, + wait_for_pressure_admission(&mut observations, REGION_A).await + ); +} + /// Different regions pin successive old objects. One refresh must flush all /// blockers, including while new objects arrive, rather than advance one pin /// per minute. Repeat after reclamation to exercise ongoing pressure. diff --git a/src/mito2/src/engine/object_store_wal_test.rs b/src/mito2/src/engine/object_store_wal_test.rs index 26b219f277c..32340a9649b 100644 --- a/src/mito2/src/engine/object_store_wal_test.rs +++ b/src/mito2/src/engine/object_store_wal_test.rs @@ -72,7 +72,7 @@ use crate::time_provider::mock::MockTimeProvider; use crate::wal::EntryId; use crate::worker::CHECK_REGION_INTERVAL; -fn wal_config() -> ObjectStoreWalConfig { +pub(crate) fn wal_config() -> ObjectStoreWalConfig { ObjectStoreWalConfig { prefix: "cluster-a/wal".to_string(), ..Default::default() @@ -85,7 +85,7 @@ fn node_prefix() -> String { wal_config().node_prefix(0, STANDALONE_GENERATION) } -fn wal_options() -> HashMap { +pub(crate) fn wal_options() -> HashMap { let options = WalOptions::ObjectStore(ObjectStoreWalOptions::new(node_prefix())); HashMap::from([( WAL_OPTIONS_KEY.to_string(), @@ -161,7 +161,7 @@ async fn close_region(engine: &MitoEngine, region_id: RegionId) { .unwrap(); } -async fn count_rows(engine: &MitoEngine, region_id: RegionId) -> usize { +pub(crate) async fn count_rows(engine: &MitoEngine, region_id: RegionId) -> usize { let stream = engine .scan_to_stream(region_id, ScanRequest::default()) .await @@ -175,7 +175,10 @@ async fn count_rows(engine: &MitoEngine, region_id: RegionId) -> usize { } /// Returns the ids of the entries the store still hands out for `region_id`. -async fn stored_entry_ids(log_store: &ObjectStoreLogStore, region_id: RegionId) -> Vec { +pub(crate) async fn stored_entry_ids( + log_store: &ObjectStoreLogStore, + region_id: RegionId, +) -> Vec { log_store .read(&object_store_provider(region_id), 0) .await @@ -437,7 +440,7 @@ async fn create_leased_region( /// Creates a region whose writes require a lease on `engine`. Returns its table dir, row schema /// and lease deadline, which the frozen clock of the lease checks has not reached. -async fn create_leased_region_on( +pub(crate) async fn create_leased_region_on( engine: &MitoEngine, region_id: RegionId, ) -> (String, Vec, Instant) { @@ -446,12 +449,18 @@ async fn create_leased_region_on( let (table_dir, schema) = create_region(engine, region_id).await; let deadline = Instant::now() + Duration::from_secs(10); engine - .set_region_role_with_lease(region_id, RegionRole::Leader, Instant::now(), deadline) + .set_region_role_with_lease( + region_id, + RegionRole::Leader, + Instant::now(), + deadline, + None, + ) .unwrap(); (table_dir, schema, deadline) } -async fn count_ssts(store: &ObjectStore, region_dir: &str) -> usize { +pub(crate) async fn count_ssts(store: &ObjectStore, region_dir: &str) -> usize { let entries = store.list(region_dir).await.unwrap(); entries .iter() @@ -656,7 +665,7 @@ impl EventListener for CatchupReopenBlocker { } } -async fn put_range( +pub(crate) async fn put_range( engine: &MitoEngine, region_id: RegionId, schema: &[ColumnSchema], @@ -670,7 +679,7 @@ async fn put_range( put_rows(engine, region_id, rows).await; } -async fn wait_until(condition: impl Fn() -> bool) { +pub(crate) async fn wait_until(condition: impl Fn() -> bool) { tokio::time::timeout(Duration::from_secs(10), async { while !condition() { tokio::time::sleep(Duration::from_millis(10)).await; @@ -681,13 +690,13 @@ async fn wait_until(condition: impl Fn() -> bool) { } /// Returns whether the engine registers another instance of the region than `old`. -fn is_replaced(engine: &MitoEngine, old: &MitoRegionRef) -> bool { +pub(crate) fn is_replaced(engine: &MitoEngine, old: &MitoRegionRef) -> bool { engine .get_region(old.region_id) .is_some_and(|region| !Arc::ptr_eq(®ion, old)) } -async fn next_manifest_version(region: &MitoRegionRef) -> u64 { +pub(crate) async fn next_manifest_version(region: &MitoRegionRef) -> u64 { region .manifest_ctx .manifest_manager @@ -698,7 +707,12 @@ async fn next_manifest_version(region: &MitoRegionRef) -> u64 { } /// Stores `bytes` as manifest version `version` of the region, as another writer would. -async fn publish_delta(store: &ObjectStore, region_dir: &str, version: u64, bytes: Vec) { +pub(crate) async fn publish_delta( + store: &ObjectStore, + region_dir: &str, + version: u64, + bytes: Vec, +) { let path = gen_path( &manifest_dir(region_dir), &delta_file(version), @@ -707,7 +721,7 @@ async fn publish_delta(store: &ObjectStore, region_dir: &str, version: u64, byte store.write(&path, bytes).await.unwrap(); } -async fn encode_delta(action_list: RegionMetaActionList) -> Vec { +pub(crate) async fn encode_delta(action_list: RegionMetaActionList) -> Vec { CompressionType::Gzip .encode(&action_list.encode().unwrap()[..]) .await @@ -828,7 +842,7 @@ async fn test_staging_save_after_deadline_sends_nothing() { let region = engine.get_region(region_id).unwrap(); let version = next_manifest_version(®ion).await; - // The memtable is empty, so entering staging saves its manifest without a flush. + // The memtable is empty, so entering staging stages its change without a flush. engine.freeze_lease_clock(deadline); let error = engine .handle_request(region_id, enter_staging_request()) @@ -840,12 +854,85 @@ async fn test_staging_save_after_deadline_sends_nothing() { "{error:?}" ); assert_eq!(version, next_manifest_version(®ion).await); - let store = region.manifest_ctx.manifest_manager.read().await.store(); - assert!(store.fetch_staging_manifests().await.unwrap().is_empty()); + let manager = region.manifest_ctx.manifest_manager.read().await; + assert!(manager.staged_actions().is_none()); + drop(manager); assert!(!region.manifest_ctx.is_reload_pending()); assert!(region.is_writable()); } +#[tokio::test] +async fn test_end_of_staging_after_deadline_sends_nothing() { + let mut env = TestEnv::with_prefix("end-staging-after-deadline").await; + let region_id = RegionId::new(1, 1); + let (engine, _log_store, table_dir, schema, deadline) = + create_leased_region(&mut env, region_id).await; + put_range(&engine, region_id, &schema, 0, 3).await; + flush_region(&engine, region_id, None).await; + let region = engine.get_region(region_id).unwrap(); + let file = region + .manifest_ctx + .manifest() + .await + .files + .values() + .next() + .unwrap() + .clone(); + engine + .handle_request(region_id, enter_staging_request()) + .await + .unwrap(); + let partition_expr = region + .manifest_ctx + .staging_partition_info() + .unwrap() + .partition_expr() + .unwrap() + .to_string(); + let (sender, receiver) = oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::EditRegion( + RegionEditRequest::new( + region_id, + edit_files(vec![file.as_ref().clone()], vec![]), + false, + sender, + ) + .for_staging(partition_expr), + ), + ) + .await + .unwrap(); + receiver.await.unwrap().unwrap(); + let version = next_manifest_version(®ion).await; + + engine.freeze_lease_clock(deadline); + let response = engine + .set_region_role_state_gracefully(region_id, SettableRegionRoleState::Leader) + .await + .unwrap(); + + assert!(response.is_invalid_transition(), "{response:?}"); + assert_eq!(version, next_manifest_version(®ion).await); + let store = env.get_object_store().unwrap(); + let region_dir = region_dir_from_table_dir(&table_dir, region_id, PathType::Bare); + let delta = gen_path( + &manifest_dir(®ion_dir), + &delta_file(version), + CompressionType::Gzip, + ); + assert!(!store.exists(&delta).await.unwrap()); + assert!(region.is_follower()); + assert!(!region.manifest_ctx.is_reload_pending()); + // The staged edit leaves with staging. + assert!(region.manifest_ctx.staging_manifest().await.is_none()); +} + #[tokio::test] async fn test_default_mode_commit_ignores_leader_deadline() { let mut env = TestEnv::with_prefix("default-mode-deadline").await; @@ -872,7 +959,7 @@ async fn test_lease_on_fenced_region_only_extends_deadline() { let later = deadline + Duration::from_secs(10); let promoted = engine - .set_region_role_with_lease(region_id, RegionRole::Leader, Instant::now(), later) + .set_region_role_with_lease(region_id, RegionRole::Leader, Instant::now(), later, None) .unwrap(); assert!(!promoted); @@ -1011,6 +1098,49 @@ async fn test_reload_waits_for_held_upload() { assert_eq!(3, count_rows(&engine, region_id).await); } +#[tokio::test] +async fn test_reload_waits_for_running_checkpoint() { + let (blocker, layer) = CheckpointTaskBlocker::block_last_checkpoint_write(); + let mut env = TestEnv::with_prefix("reload-waits-for-running-checkpoint") + .await + .with_mock_layer(layer); + let region_id = RegionId::new(1, 1); + let (engine, _log_store, _, schema, _) = create_leased_region(&mut env, region_id).await; + put_range(&engine, region_id, &schema, 0, 3).await; + // The flush starts a checkpoint, which blocks after it has created `_last_checkpoint` + // and before it has written the pointer. + flush_region(&engine, region_id, None).await; + blocker.wait_until_blocked().await; + let old = engine.get_region(region_id).unwrap(); + let wait_started = old + .manifest_ctx + .manifest_manager + .read() + .await + .checkpointer() + .pending_checkpoint_wait_started(); + fence(&old).await; + + notify_fenced(&engine, region_id).await; + tokio::time::timeout(Duration::from_secs(10), wait_started.notified()) + .await + .expect("the reload does not wait for the checkpoint"); + // The worker takes the manifest lock of the fenced instance, for example to close it, + // while the reload waits. + let manager = tokio::time::timeout( + Duration::from_secs(10), + old.manifest_ctx.manifest_manager.write(), + ) + .await + .expect("the reload holds the manifest lock while it waits"); + drop(manager); + assert!(!is_replaced(&engine, &old)); + blocker.release(); + + wait_until(|| is_replaced(&engine, &old)).await; + assert_eq!(3, count_rows(&engine, region_id).await); +} + #[tokio::test] async fn test_reload_waits_for_running_flush() { let mut env = TestEnv::with_prefix("reload-waits-for-flush").await; @@ -1100,7 +1230,7 @@ async fn test_edit_conflict_reloads_region() { .clone(); let added = FileMeta { file_id: FileId::random(), - ..flushed + ..flushed.as_ref().clone() }; let region_dir = region_dir_from_table_dir(&table_dir, region_id, PathType::Bare); let version = next_manifest_version(&old).await; @@ -1128,7 +1258,10 @@ async fn test_reload_ignores_edit_of_replaced_instance() { plugins.insert::(hook.clone()); let engine = env .create_engine_with_log_store_and_plugins( - MitoConfig::default(), + MitoConfig { + num_workers: 1, + ..Default::default() + }, new_log_store().await, plugins, ) @@ -1149,10 +1282,12 @@ async fn test_reload_ignores_edit_of_replaced_instance() { .clone(); let file = |file_id| FileMeta { file_id, - ..flushed.clone() + ..flushed.as_ref().clone() }; let added = file(FileId::random()); + let barrier_region_id = RegionId::new(1, 2); + create_leased_region_on(&engine, barrier_region_id).await; hook.armed.store(true, Ordering::Relaxed); let committed = { let engine = engine.clone(); @@ -1169,9 +1304,8 @@ async fn test_reload_ignores_edit_of_replaced_instance() { .submit_to_worker(region_id, request) .await .unwrap(); - // The worker queues the edit before it handles the flush request sent after it. - let flush = RegionRequest::Flush(RegionFlushRequest::default()); - engine.handle_request(region_id, flush).await.unwrap_err(); + // An empty flush on the same worker acknowledges receipt without waiting for this edit. + flush_region(&engine, barrier_region_id, None).await; let region_dir = region_dir_from_table_dir(&table_dir, region_id, PathType::Bare); let version = next_manifest_version(&old).await; let removal = RegionMetaActionList::with_action(RegionMetaAction::Edit(edit_files( @@ -1225,7 +1359,7 @@ async fn test_reload_ignores_leases_sent_before_swap() { wait_until(|| is_replaced(&engine, &old)).await; let lease = |sent_at| { - engine.set_region_role_with_lease(region_id, RegionRole::Leader, sent_at, deadline) + engine.set_region_role_with_lease(region_id, RegionRole::Leader, sent_at, deadline, None) }; assert!(!lease(sent_before_swap).unwrap()); assert!(engine.get_region(region_id).unwrap().is_follower()); @@ -1347,10 +1481,32 @@ async fn test_end_of_staging_conflict_reloads_region() { .handle_request(region_id, enter_staging_request()) .await .unwrap(); + let partition_expr = old + .manifest_ctx + .staging_partition_info() + .unwrap() + .partition_expr() + .unwrap() + .to_string(); + let (sender, receiver) = oneshot::channel(); engine - .edit_region(region_id, edit_files(vec![file], vec![])) + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::EditRegion( + RegionEditRequest::new( + region_id, + edit_files(vec![file.as_ref().clone()], vec![]), + false, + sender, + ) + .for_staging(partition_expr), + ), + ) .await .unwrap(); + receiver.await.unwrap().unwrap(); let region_dir = region_dir_from_table_dir(&table_dir, region_id, PathType::Bare); let version = next_manifest_version(&old).await; let store = env.get_object_store().unwrap(); diff --git a/src/mito2/src/engine/open_test.rs b/src/mito2/src/engine/open_test.rs index d566e56ccd8..32bd9dea335 100644 --- a/src/mito2/src/engine/open_test.rs +++ b/src/mito2/src/engine/open_test.rs @@ -1198,8 +1198,11 @@ fn new_dummy_fetcher(region_id: RegionId, expr_json: String) -> PartitionExprFet }) } +#[rstest::rstest] +#[case::from_writable(false)] +#[case::after_empty_staging(true)] #[tokio::test] -async fn test_open_backfills_partition_expr_with_fetcher() { +async fn test_open_backfills_partition_expr_with_fetcher(#[case] staging: bool) { let mut env = TestEnv::with_prefix("open-backfill-fetcher").await; // prepare plugins with dummy fetcher @@ -1255,10 +1258,27 @@ async fn test_open_backfills_partition_expr_with_fetcher() { engine .set_region_role(region_id, RegionRole::Leader) .unwrap(); + if staging { + // Ending a staging with nothing to commit also backfills. + engine + .set_region_role_state_gracefully(region_id, SettableRegionRoleState::StagingLeader) + .await + .unwrap(); + } engine .set_region_role_state_gracefully(region_id, SettableRegionRoleState::Leader) .await .unwrap(); + let manifest = engine + .get_region(region_id) + .unwrap() + .manifest_ctx + .manifest() + .await; + assert_eq!( + manifest.metadata.partition_expr.as_deref(), + Some(expr_json.as_str()) + ); // reopen again to ensure no further changes and still Some engine diff --git a/src/mito2/src/engine/pending_tail_test.rs b/src/mito2/src/engine/pending_tail_test.rs index 5f2627f0633..507ea607f37 100644 --- a/src/mito2/src/engine/pending_tail_test.rs +++ b/src/mito2/src/engine/pending_tail_test.rs @@ -23,7 +23,7 @@ use datatypes::schema::{ColumnDefaultConstraint, ColumnSchema}; use rstest::rstest; use store_api::logstore::provider::RaftEngineProvider; use store_api::metadata::ColumnMetadata; -use store_api::region_engine::RegionEngine; +use store_api::region_engine::{RegionEngine, SetRegionRoleStateResponse, SettableRegionRoleState}; use store_api::region_request::{ AddColumn, AlterKind, EnterStagingRequest, RegionAlterRequest, RegionCloseRequest, RegionCompactRequest, RegionCreateRequest, RegionDropRequest, RegionFlushReason, @@ -95,7 +95,7 @@ pub(crate) fn pending_tails_edit(pending_tails: Vec) -> RegionEdit } } -async fn commit_edit(engine: &MitoEngine, region_id: RegionId, edit: RegionEdit) { +pub(crate) async fn commit_edit(engine: &MitoEngine, region_id: RegionId, edit: RegionEdit) { let region = engine.get_region(region_id).unwrap(); region .manifest_ctx @@ -120,7 +120,7 @@ pub(crate) async fn set_pending_tails( } async fn reopen(engine: &MitoEngine, region_id: RegionId, request: RegionCreateRequest) { - reopen_region(engine, region_id, request.table_dir, true, request.options).await; + reopen_region(engine, region_id, request.table_dir, false, request.options).await; } fn wal_span(env: &TestEnv, region_id: RegionId) -> (Option, Option) { @@ -252,7 +252,7 @@ fn assert_pending_tails_error(err: &BoxedError) { ); } -fn add_field(default_constraint: Option) -> AlterKind { +pub(crate) fn add_field(default_constraint: Option) -> AlterKind { let column_schema = ColumnSchema::new("field_1", ConcreteDataType::float64_datatype(), true) .with_default_constraint(default_constraint) .unwrap(); @@ -395,6 +395,14 @@ async fn test_enter_staging_rejects_region_with_pending_tails() { .await .unwrap_err(); + assert_pending_tails_error(&err); + let response = engine + .set_region_role_state_gracefully(region_id, SettableRegionRoleState::StagingLeader) + .await + .unwrap(); + let SetRegionRoleStateResponse::InvalidTransition(err) = response else { + panic!("{response:?}"); + }; assert_pending_tails_error(&err); let region = engine.get_region(region_id).unwrap(); assert!(!region.is_staging()); diff --git a/src/mito2/src/engine/region_hook.rs b/src/mito2/src/engine/region_hook.rs index 9f138405842..ff52553b18f 100644 --- a/src/mito2/src/engine/region_hook.rs +++ b/src/mito2/src/engine/region_hook.rs @@ -24,7 +24,7 @@ //! (see [`SstFileInfo`] and the coverage footnote). //! //! - [`on_manifest_updated`]: Fires after a manifest write is committed to the **live** -//! (normal) manifest directory. Writes to the staging directory (enter staging, +//! (normal) manifest directory. Actions staged in memory (enter staging, //! operations during staging, the intermediate apply-staging edit) are suppressed — //! their effects are accumulated and delivered in a single notification when the //! staged actions are promoted to the live manifest. Receives the full @@ -43,32 +43,33 @@ //! ## Coverage //! //! Only manifest writes to the **normal** (live) manifest directory trigger -//! `on_manifest_updated`. Writes to the staging manifest directory (operations +//! `on_manifest_updated`. Actions staged in memory (operations //! that happen while the region is in staging mode) are intentionally suppressed: //! their effects are accumulated and delivered in a single notification when -//! the staged actions are promoted to the live manifest via `exit_staging_on_success`. +//! the staged actions are committed to the live manifest via `end_staging_locked`. //! //! | Scenario | `on_sst_files_written` | `on_manifest_updated` | //! |------------------------------|:----------------------:|:---------------------:| //! | Flush (memtable → SST) | ✅ Yes | ✅ Yes | //! | Local compaction | ✅ Yes | ✅ Yes | +//! | Pending WAL tail replay | ✅ Yes | ✅ Yes | //! | Remote compaction | ✅ (compactor node) ¹ | ✅ (compactor node) ¹ | //! | RegionEdit / bulk ingestion | ❌ (files pre-written) | ✅ Yes | //! | Copy region | ❌ (object-store copy) | ✅ Yes | //! | Apply staging (promote) | ❌ (delegates to edit) | ✅ Yes ² | //! | Alter (schema change) | ❌ (no SST files) | ✅ Yes | //! | Truncate | ❌ (removes files) | ✅ Yes | -//! | Enter staging | ❌ (no SST files) | ❌ (staging dir) | -//! | Operations during staging | N/A | ❌ (staging dir) | +//! | Enter staging | ❌ (no SST files) | ❌ (in memory) | +//! | Operations during staging | N/A | ❌ (in memory) | //! | Async index build | ❌ (index files only) | ✅ Yes | //! //! ¹ Remote compaction runs on a dedicated compactor node via `CompactionRegionOpener::open()`; //! pass plugins via `OpenCompactionRegionRequest` to enable hooks there. `sst_infos` is //! `#[serde(skip)]` over the wire, so each [`SstInfo`] is rebuilt from [`FileMeta`] with //! empty footer/index — see [`SstFileInfo`] for field-level detail. -//! ² Apply staging fires `on_manifest_updated` once when `exit_staging_on_success` promotes +//! ² Apply staging fires `on_manifest_updated` once when `end_staging_locked` commits //! all staged manifest actions (including the SST file additions) into the live manifest. -//! The intermediate staging `RegionEdit` is written to the staging directory and does not +//! The intermediate staging `RegionEdit` is staged in memory and does not //! fire the hook — its file list is included in the promote notification. //! //! The following paths do **not** trigger any hook: @@ -116,8 +117,9 @@ //! //! ## Invocation points //! -//! `on_sst_files_written` is invoked at the SST write site (flush task or compaction task), -//! immediately after SST files are written but **before** the manifest is committed. +//! `on_sst_files_written` is invoked at the SST write site (flush task, compaction task or +//! tail replay task), immediately after SST files are written but **before** the manifest +//! is committed. //! //! `on_manifest_updated` is funneled through [`ManifestContext::update_locked`], //! which calls the low-level [`RegionManifestManager::update_with_verification`] @@ -176,10 +178,10 @@ use crate::sst::parquet::SstInfo; /// /// ## Staging suppression /// -/// When `is_staging` is `true`, [`fire`](Self::fire) is a no-op — the write went to -/// the staging manifest directory. The hook only observes writes to the live (normal) +/// When `is_staging` is `true`, [`fire`](Self::fire) is a no-op — the actions were +/// staged in memory. The hook only observes writes to the live (normal) /// manifest directory. Staging actions are accumulated and delivered in a single -/// notification when `exit_staging_on_success` promotes them (`is_staging = false`). +/// notification when `end_staging_locked` commits them (`is_staging = false`). #[must_use = "the region hook must be fired after releasing the manifest write lock"] pub(crate) struct PendingManifestHook { region_id: RegionId, @@ -187,7 +189,7 @@ pub(crate) struct PendingManifestHook { action_list: Option, version: ManifestVersion, hook: Option, - /// Whether the manifest write went to the staging directory. + /// Whether the actions were staged in memory. /// When `true`, `fire()` is suppressed — the hook only observes live manifest writes. is_staging: bool, } @@ -214,9 +216,9 @@ impl PendingManifestHook { self.version } - /// Fires the hook if one is registered, **unless** the write went to the staging - /// manifest directory (`is_staging = true`). Safe to call unconditionally: - /// it is a no-op when no hook is registered or when the write is staging-only. + /// Fires the hook if one is registered, **unless** the actions were staged in + /// memory (`is_staging = true`). Safe to call unconditionally: + /// it is a no-op when no hook is registered or when the actions were staged in memory. pub(crate) async fn fire(self) { if self.is_staging { return; @@ -291,7 +293,8 @@ pub struct VerifiedSstFileInfo<'a> { pub trait RegionHook: Send + Sync + Debug { /// Called after SST **data files** are physically written, before manifest commit. /// - /// This fires only when mito2 itself writes SST files (flush and compaction). + /// This fires only when mito2 itself writes SST files (flush, compaction and the replay + /// of pending WAL tails). /// It does **not** fire when SST files are pre-written externally (bulk ingestion, /// copy region) or when only index files are written (async index build). /// @@ -342,8 +345,8 @@ pub trait RegionHook: Send + Sync + Debug { /// (normal) manifest directory. /// /// Fires for: flush, compaction, region edit, copy region, alter, truncate, - /// async index build, and apply-staging promote. Does **not** fire for writes - /// to the staging manifest directory (enter staging, operations during staging, + /// async index build, and apply-staging promote. Does **not** fire for actions + /// staged in memory (enter staging, operations during staging, /// the intermediate apply-staging edit) — those are suppressed because their /// effects are accumulated and delivered in a single promote notification. /// diff --git a/src/mito2/src/engine/scan_test.rs b/src/mito2/src/engine/scan_test.rs index ea8d9f54535..538fb670244 100644 --- a/src/mito2/src/engine/scan_test.rs +++ b/src/mito2/src/engine/scan_test.rs @@ -5463,14 +5463,16 @@ async fn test_reset_state_discards_dynamic_field_pruning() { ))) .unwrap(); - // Dynamic pruning skips the first row group but does not filter individual - // rows in the second. Drop before consuming the third row group. + // Dynamic pruning skips the first row group, and the dynamic prefilter prunes the + // row the producer rejects in the second. Only the row above the producer's + // threshold is left; the live filter above the scan still owns the final result. + // Drop before consuming the third row group. let mut stream = scanner .scan_partition(&Default::default(), &ExecutionPlanMetricsSet::default(), 0) .unwrap(); let first_batch = stream.try_next().await.unwrap().unwrap(); assert_eq!( - vec![2.0, 3.0], + vec![3.0], scan_field_values(std::slice::from_ref(&first_batch)) ); drop(stream); diff --git a/src/mito2/src/engine/set_role_state_test.rs b/src/mito2/src/engine/set_role_state_test.rs index 94a52c4f3c6..0900a117a1f 100644 --- a/src/mito2/src/engine/set_role_state_test.rs +++ b/src/mito2/src/engine/set_role_state_test.rs @@ -499,7 +499,7 @@ async fn test_retried_downgrade_waits_after_first_request_is_cancelled() { let first_region = region.clone(); let first_downgrade = tokio::spawn(async move { first_region - .set_role_state_gracefully(SettableRegionRoleState::DowngradingLeader) + .set_role_state_gracefully(SettableRegionRoleState::DowngradingLeader, async {}) .await }); tokio::time::timeout(Duration::from_secs(5), first_wait_started) @@ -536,7 +536,7 @@ async fn test_retried_downgrade_waits_after_first_request_is_cancelled() { let retry_region = region.clone(); let retry_downgrade = tokio::spawn(async move { retry_region - .set_role_state_gracefully(SettableRegionRoleState::DowngradingLeader) + .set_role_state_gracefully(SettableRegionRoleState::DowngradingLeader, async {}) .await }); tokio::time::timeout(Duration::from_secs(5), second_wait_started) diff --git a/src/mito2/src/engine/shared_file_test.rs b/src/mito2/src/engine/shared_file_test.rs index 36e32697f38..14fe2afac3a 100644 --- a/src/mito2/src/engine/shared_file_test.rs +++ b/src/mito2/src/engine/shared_file_test.rs @@ -12,6 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. +#[cfg(feature = "enterprise")] +use std::collections::HashMap; + use store_api::region_engine::RegionEngine; use store_api::region_request::RegionRequest; use store_api::storage::{FileId, RegionId}; @@ -230,7 +233,7 @@ async fn test_shared_reclassification_requires_add_only() { assert_eq!(LEVEL_SHARED, installed.files[&source.file_id].level); assert!(region.version().ssts.levels()[0].files.is_empty()); assert_eq!( - &installed.files[&source.file_id], + installed.files[&source.file_id].as_ref(), region.version().ssts.levels()[SHARED_LEVEL_SLOT].files[&source.file_id].meta_ref() ); reopen_region( @@ -248,7 +251,7 @@ async fn test_shared_reclassification_requires_add_only() { ); assert!(reopened.version().ssts.levels()[0].files.is_empty()); assert_eq!( - &installed.files[&source.file_id], + installed.files[&source.file_id].as_ref(), reopened.version().ssts.levels()[SHARED_LEVEL_SLOT].files[&source.file_id].meta_ref() ); assert_eq!(expected_rows, scan().await); @@ -446,7 +449,13 @@ async fn test_truncate_shared_references_preserves_other_readers(#[case] truncat .files() .map(|file| (file.file_id().file_id(), file.meta_ref().clone())) .collect::>(); - assert_eq!(expected, visible); + assert_eq!( + expected + .iter() + .map(|(id, meta)| (*id, meta.as_ref().clone())) + .collect::>(), + visible + ); let old_rows = RecordBatches::try_collect(old_reader).await.unwrap(); assert_eq!( 6, diff --git a/src/mito2/src/engine/staging_test.rs b/src/mito2/src/engine/staging_test.rs index a0501362ea8..d8821a9a458 100644 --- a/src/mito2/src/engine/staging_test.rs +++ b/src/mito2/src/engine/staging_test.rs @@ -51,19 +51,23 @@ use store_api::storage::{FileId, RegionId, ScanRequest}; use crate::config::MitoConfig; use crate::engine::MitoEngine; -use crate::engine::listener::{EventListener, FlushListener, NotifyEnterStagingResultListener}; +use crate::engine::basic_test::build_bulk_insert_request; +use crate::engine::listener::{ + CountingStallListener, EventListener, FlushListener, NotifyEnterStagingResultListener, +}; use crate::error::Error; use crate::flush::WriteBufferManager; use crate::manifest::action::{ RegionChange, RegionEdit, RegionMetaAction, RegionMetaActionList, RegionPartitionExprChange, }; use crate::region::{MitoRegionRef, RegionLeaderState, RegionRoleState}; -use crate::request::{RegionEditRequest, WorkerRequest}; +use crate::request::{BackgroundNotify, RegionEditRequest, WorkerRequest}; use crate::sst::FormatType; use crate::sst::file::FileMeta; use crate::test_util::{ - CheckpointTaskBlocker, CreateRequestBuilder, MockWriteBufferManager, TestEnv, build_rows, - flush_region, put_rows, reopen_region, rows_schema, + CheckpointTaskBlocker, CreateRequestBuilder, MockWriteBufferManager, TestEnv, + build_delete_rows, build_rows, delete_rows, delete_rows_schema, flush_region, put_rows, + reopen_region, rows_schema, }; use crate::time_provider::mock::MockTimeProvider; use crate::worker::MAX_INITIAL_CHECK_DELAY_SECS; @@ -572,6 +576,7 @@ async fn test_staging_manifest_directory_with_format(flat_format: bool) { fs::metadata(&normal_manifest_dir).is_ok(), "Normal manifest directory should exist" ); + let normal_files_before = fs::read_dir(&normal_manifest_dir).unwrap().count(); // Now test staging mode manifest creation // Set region to staging mode using the engine API @@ -639,31 +644,17 @@ async fn test_staging_manifest_directory_with_format(flat_format: bool) { Error::StagingPartitionExprMismatch { .. } ); - // Check that manifest files are in staging directory + // Staging keeps its actions in memory: it writes no manifest file. let staging_manifest_dir = format!("{}/staging/manifest", region_dir); - assert!( - fs::metadata(&staging_manifest_dir).is_ok(), - "Staging manifest directory should exist" + assert_eq!( + fs::read_dir(&staging_manifest_dir) + .map(|entries| entries.count()) + .unwrap_or_default(), + 0 ); - - // Check what exists in normal manifest directory - let files: Vec<_> = fs::read_dir(&normal_manifest_dir) - .unwrap() - .collect::, _>>() - .unwrap(); - assert!( - !files.is_empty(), - "Normal manifest directory should contain files" - ); - - // Check what exists in staging manifest directory - let staging_files = fs::read_dir(&staging_manifest_dir) - .unwrap() - .collect::, _>>() - .unwrap(); - assert!( - !staging_files.is_empty(), - "Staging manifest directory should contain files" + assert_eq!( + fs::read_dir(&normal_manifest_dir).unwrap().count(), + normal_files_before ); } @@ -758,45 +749,27 @@ async fn test_staging_exit_success_with_manifests_with_format(flat_format: bool) let region = engine.get_region(region_id).unwrap(); for (file, expected) in source_files.into_iter().zip([11, 12]) { - engine - .edit_region( - region_id, - RegionEdit { - files_to_add: vec![file], - files_to_remove: vec![], - timestamp_ms: None, - compaction_time_window: None, - flushed_entry_id: None, - flushed_sequence: None, - committed_sequence: None, - wal_recovery: Default::default(), - }, - ) - .await - .unwrap(); + submit_staging_edit( + &engine, + region_id, + file.as_ref().clone(), + partition_expr.clone(), + ) + .await + .unwrap(); let current = region.version_control.current(); assert_eq!(current.last_sequence, expected); assert_eq!(current.committed_sequence, 10); } - // Verify we're in staging mode and staging manifests exist + // Staging keeps its actions in memory and takes no version. let data_home = env.data_home(); let region_dir = format!("{}/data/test/1024_0000000000", data_home.display()); - let staging_manifest_dir = format!("{}/staging/manifest", region_dir); - - let staging_files_before = fs::read_dir(&staging_manifest_dir) - .unwrap() - .collect::, _>>() - .unwrap(); - assert_eq!( - staging_files_before.len(), - // Two edits and entering staging mode. - 3, - "Staging manifest directory should contain 3 files before exit, got: {:?}", - staging_files_before - ); + let version_before = region.manifest_ctx.manifest_version().await; { let manager = region.manifest_ctx.manifest_manager.read().await; + // Entering staging and two edits. + assert_eq!(manager.staged_actions().unwrap().actions.len(), 3); let staging_manifest = manager.staging_manifest().unwrap(); assert_eq!(staging_manifest.files.len(), 3); assert_eq!( @@ -891,9 +864,9 @@ async fn test_staging_exit_success_with_manifests_with_format(flat_format: bool) .next() .unwrap() .clone(); - let mut overridden = original.clone(); + let mut overridden = original.as_ref().clone(); overridden.file_size = u64::MAX - 1; - for metadata in [overridden, original.clone()] { + for metadata in [overridden.into(), original.clone()] { region .manifest_ctx .manifest_manager @@ -901,7 +874,7 @@ async fn test_staging_exit_success_with_manifests_with_format(flat_format: bool) .await .update( RegionMetaActionList::new(vec![RegionMetaAction::Edit(RegionEdit { - files_to_add: vec![metadata.clone()], + files_to_add: vec![metadata.as_ref().clone()], files_to_remove: vec![], timestamp_ms: None, compaction_time_window: None, @@ -978,24 +951,16 @@ async fn test_staging_exit_success_with_manifests_with_format(flat_format: bool) ingested_parquet_bytes ); - // Verify staging manifests have been cleared - let staging_files_after = fs::read_dir(&staging_manifest_dir) - .map(|entries| entries.collect::, _>>().unwrap_or_default()) - .unwrap_or_default(); - assert!( - staging_files_after.is_empty(), - "Staging manifest directory should be empty after successful exit" + // The end of staging commits the staged actions as the next version. + assert_eq!( + region.manifest_ctx.manifest_version().await, + version_before + 1 ); - - // Verify normal manifests contain the merged changes let normal_files_after = fs::read_dir(&normal_manifest_dir) .unwrap() .collect::, _>>() .unwrap(); - assert!( - normal_files_after.len() > normal_count_before, - "Normal manifest directory should contain more files after merge" - ); + assert_eq!(normal_files_after.len(), normal_count_before + 1); // Validate in-memory version reflects merged manifests (files visible in levels) let version = region.version(); @@ -1063,24 +1028,20 @@ async fn test_enter_staging_writes_partition_expr_change_action_with_format(flat let region = engine.get_region(region_id).unwrap(); let manager = region.manifest_ctx.manifest_manager.read().await; - let staging_manifests = manager.store().fetch_staging_manifests().await.unwrap(); - assert!(!staging_manifests.is_empty()); + let staged_actions = manager.staged_actions().unwrap(); let mut found_partition_expr_change = false; let mut found_change = false; - for (_, raw_action_list) in staging_manifests { - let action_list = RegionMetaActionList::decode(&raw_action_list).unwrap(); - for action in action_list.actions { - match action { - RegionMetaAction::PartitionExprChange(change) => { - found_partition_expr_change = true; - assert_eq!(change.partition_expr, Some(partition_expr.clone())); - } - RegionMetaAction::Change(_) => { - found_change = true; - } - _ => {} + for action in &staged_actions.actions { + match action { + RegionMetaAction::PartitionExprChange(change) => { + found_partition_expr_change = true; + assert_eq!(change.partition_expr, Some(partition_expr.clone())); } + RegionMetaAction::Change(_) => { + found_change = true; + } + _ => {} } } @@ -1379,28 +1340,6 @@ impl List for MockLister { } } -struct MockWriter { - path: String, - inner: Writer, -} - -impl Write for MockWriter { - async fn write(&mut self, bs: Buffer) -> MockResult<()> { - self.inner.write(bs).await - } - - async fn close(&mut self) -> MockResult { - if self.path.contains("staging") { - return Err(MockError::new(ErrorKind::Unexpected, "mock error")); - } - self.inner.close().await - } - - async fn abort(&mut self) -> MockResult<()> { - self.inner.abort().await - } -} - /// Fails closing Parquet files while `fail_parquet` is set. struct ParquetFailWriter { path: String, @@ -1498,27 +1437,6 @@ async fn test_enter_staging_clean_staging_manifest_error_with_format(flat_format test_enter_staging_error(&mut env, flat_format).await; } -#[tokio::test] -async fn test_enter_staging_save_staging_manifest_error() { - common_telemetry::init_default_ut_logging(); - test_enter_staging_save_staging_manifest_error_with_format(false).await; - test_enter_staging_save_staging_manifest_error_with_format(true).await; -} - -async fn test_enter_staging_save_staging_manifest_error_with_format(flat_format: bool) { - let mock_layer = MockLayerBuilder::default() - .writer_factory(Arc::new(|path, _args, lister| { - Box::new(MockWriter { - path: path.to_string(), - inner: lister, - }) - })) - .build() - .unwrap(); - let mut env = TestEnv::new().await.with_mock_layer(mock_layer); - test_enter_staging_error(&mut env, flat_format).await; -} - async fn enter_staging(engine: &MitoEngine, region_id: RegionId) { engine .handle_request( @@ -1618,7 +1536,7 @@ async fn put_range_skip_wal( /// Returns one of the files in the region's manifest. async fn first_file(region: &MitoRegionRef) -> FileMeta { let manifest = region.manifest_ctx.manifest().await; - manifest.files.values().next().unwrap().clone() + manifest.files.values().next().unwrap().as_ref().clone() } fn edit_adding(file: FileMeta) -> RegionEdit { @@ -1634,14 +1552,26 @@ fn edit_adding(file: FileMeta) -> RegionEdit { } } -/// Spawns an edit importing `file`. +/// Spawns an edit importing `file`, retaining the current staging binding when present. fn spawn_edit( engine: &MitoEngine, region_id: RegionId, file: FileMeta, ) -> tokio::task::JoinHandle> { let edit_engine = engine.clone(); - tokio::spawn(async move { edit_engine.edit_region(region_id, edit_adding(file)).await }) + let partition_expr = engine + .get_region(region_id) + .unwrap() + .manifest_ctx + .staging_partition_info() + .and_then(|info| info.partition_expr().map(ToOwned::to_owned)); + tokio::spawn(async move { + if let Some(partition_expr) = partition_expr { + submit_staging_edit(&edit_engine, region_id, file, partition_expr).await + } else { + edit_engine.edit_region(region_id, edit_adding(file)).await + } + }) } /// Starts an edit importing `file` and waits until the region is `Editing`. @@ -1979,22 +1909,14 @@ async fn test_successful_staging_end_keeps_rows_of_rejected_flush() { .unwrap_err(); let imported_file = first_file(&engine.get_region(region_id).unwrap()).await; - engine - .edit_region( - region_id, - RegionEdit { - files_to_add: vec![imported_file], - files_to_remove: vec![], - timestamp_ms: None, - compaction_time_window: None, - flushed_entry_id: None, - flushed_sequence: None, - committed_sequence: None, - wal_recovery: Default::default(), - }, - ) - .await - .unwrap(); + submit_staging_edit( + &engine, + region_id, + imported_file, + float_range_expr("field_0", 0., 100.).as_json_str().unwrap(), + ) + .await + .unwrap(); engine .set_region_role_state_gracefully(region_id, SettableRegionRoleState::Leader) .await @@ -2084,7 +2006,7 @@ impl EventListener for RequestCountListener { } #[tokio::test] -async fn test_edit_returns_to_staging_after_staging_leader_role() { +async fn test_unbound_edit_rejected_after_staging_leader_role() { let mut env = TestEnv::new().await; let engine = env.create_engine(MitoConfig::default()).await; let region_id = RegionId::new(1024, 0); @@ -2103,11 +2025,44 @@ async fn test_edit_returns_to_staging_after_staging_leader_role() { let region = engine.get_region(region_id).unwrap(); let file = first_file(®ion).await; - spawn_edit(&engine, region_id, file).await.unwrap().unwrap(); - + let err = spawn_edit(&engine, region_id, file) + .await + .unwrap() + .unwrap_err(); + assert_matches!(err, Error::RejectWriteInStaging { .. }); assert!(region.is_staging()); } +#[tokio::test] +async fn test_staged_files_not_listed_after_leaving_staging() { + let mut env = TestEnv::new().await; + let engine = env.create_engine(MitoConfig::default()).await; + let region_id = RegionId::new(1024, 0); + let request = CreateRequestBuilder::new().build(); + let schema = rows_schema(&request); + engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + put_range(&engine, region_id, &schema, (0, 3)).await; + flush_region(&engine, region_id, None).await; + enter_staging(&engine, region_id).await; + let region = engine.get_region(region_id).unwrap(); + let file = FileMeta { + file_id: FileId::random(), + ..first_file(®ion).await + }; + spawn_edit(&engine, region_id, file).await.unwrap().unwrap(); + assert_eq!(2, region.manifest_sst_entries().await.len()); + assert_eq!(2, region.all_manifest_files().await.0.len()); + + // Leaving staging without a commit keeps the staged file out of a later Leader's listing. + region.set_role(RegionRole::Follower); + region.set_role(RegionRole::Leader); + assert_eq!(1, region.manifest_sst_entries().await.len()); + assert_eq!(1, region.all_manifest_files().await.0.len()); +} + #[tokio::test] async fn test_close_time_flush_rejects_queued_and_late_edits() { let mut env = TestEnv::new().await; @@ -2237,7 +2192,21 @@ async fn test_stale_staging_edit_rejected_after_close_time_flush_fails() { }; let file_id = file.file_id; let stale_expr = float_range_expr("field_0", 0., 100.).as_json_str().unwrap(); - let result = submit_staging_edit(&engine, region_id, file.clone(), stale_expr.clone()).await; + // The flush task replies before the worker clears its close-time flush state. + // Wait for that notification before checking the stale staging identity. + // The busy check precedes the identity check, so retrying cannot hide a missed rejection. + let result = tokio::time::timeout(Duration::from_secs(5), async { + loop { + let result = + submit_staging_edit(&engine, region_id, file.clone(), stale_expr.clone()).await; + if !matches!(&result, Err(Error::RegionBusy { .. })) { + break result; + } + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await + .unwrap(); assert_matches!(result.unwrap_err(), Error::RegionState { .. }); // A later staging for another partition expression does not take the stale edit either. @@ -2404,22 +2373,14 @@ async fn test_close_after_end_of_staging_is_queued_keeps_committed_result() { put_range(&engine, region_id, &schema, (3, 5)).await; let region = engine.get_region(region_id).unwrap(); let file = first_file(®ion).await; - engine - .edit_region( - region_id, - RegionEdit { - files_to_add: vec![file], - files_to_remove: vec![], - timestamp_ms: None, - compaction_time_window: None, - flushed_entry_id: None, - flushed_sequence: None, - committed_sequence: None, - wal_recovery: Default::default(), - }, - ) - .await - .unwrap(); + submit_staging_edit( + &engine, + region_id, + file, + float_range_expr("field_0", 0., 100.).as_json_str().unwrap(), + ) + .await + .unwrap(); let manager = region.manifest_ctx.manifest_manager.read().await; let end_engine = engine.clone(); @@ -2452,3 +2413,329 @@ async fn test_close_after_end_of_staging_is_queued_keeps_committed_result() { ); assert_eq!(5, count_rows(&engine, region_id).await); } + +#[rstest::rstest] +#[case::committed(false)] +#[case::downgraded(true)] +#[tokio::test] +async fn test_writes_wait_while_staging_ends(#[case] downgrade: bool) { + let mut env = TestEnv::new().await; + let listener = Arc::new(CountingStallListener::default()); + let engine = env + .create_engine_with(MitoConfig::default(), None, Some(listener.clone()), None) + .await; + let region_id = RegionId::new(1024, 0); + let (schema, _) = create_staging_region(&engine, region_id).await; + let region = engine.get_region(region_id).unwrap(); + let started = region.manifest_ctx.begin_ending_staging(region_id).unwrap(); + + let put_engine = engine.clone(); + let put = tokio::spawn(async move { + put_engine + .handle_request( + region_id, + RegionRequest::Put(RegionPutRequest { + rows: Rows { + schema, + rows: build_rows(0, 3), + }, + hint: None, + partition_expr_version: None, + skip_wal: false, + }), + ) + .await + }); + wait_for_stalls(&listener, 1).await; + assert!(region.version().memtables.is_empty()); + + if downgrade { + region.manifest_ctx.begin_downgrade(region_id); + } + region + .manifest_ctx + .leave_ending_staging(region_id, started, &Ok::<(), Error>(())); + send_staging_ended(&engine, region_id).await; + + // The drained write sees the state that the end of staging left: a downgrade refuses it. + let result = put.await.unwrap(); + assert_eq!(downgrade, result.is_err()); + assert_eq!(downgrade, region.version().memtables.is_empty()); +} + +#[tokio::test] +async fn test_writes_after_staging_ends_wait_for_stalled_writes() { + let mut env = TestEnv::new().await; + let listener = Arc::new(CountingStallListener::default()); + let engine = env + .create_engine_with(MitoConfig::default(), None, Some(listener.clone()), None) + .await; + let region_id = RegionId::new(1024, 0); + let (schema, _) = create_staging_region(&engine, region_id).await; + let region = engine.get_region(region_id).unwrap(); + let started = region.manifest_ctx.begin_ending_staging(region_id).unwrap(); + let put_engine = engine.clone(); + let rows = Rows { + schema, + rows: build_rows(0, 3), + }; + let put = tokio::spawn(async move { put_rows(&put_engine, region_id, rows).await }); + wait_for_stalls(&listener, 1).await; + + // A delete of the same rows that arrives after staging ended, before the worker drains the + // stalled put, waits behind it. + region + .manifest_ctx + .leave_ending_staging(region_id, started, &Ok::<(), Error>(())); + let delete_engine = engine.clone(); + let rows = Rows { + schema: delete_rows_schema(&CreateRequestBuilder::new().build()), + rows: build_delete_rows(0, 3), + }; + let delete = tokio::spawn(async move { delete_rows(&delete_engine, region_id, rows).await }); + wait_for_stalls(&listener, 2).await; + send_staging_ended(&engine, region_id).await; + put.await.unwrap(); + delete.await.unwrap(); + + assert_eq!(0, count_rows(&engine, region_id).await); +} + +#[tokio::test] +async fn test_bulk_writes_after_staging_ends_wait_for_stalled_writes() { + let mut env = TestEnv::new().await; + let listener = Arc::new(CountingStallListener::default()); + let engine = env + .create_engine_with(MitoConfig::default(), None, Some(listener.clone()), None) + .await; + let region_id = RegionId::new(1024, 0); + create_staging_region(&engine, region_id).await; + let region = engine.get_region(region_id).unwrap(); + let started = region.manifest_ctx.begin_ending_staging(region_id).unwrap(); + let bulk_insert = |num_rows| { + let engine = engine.clone(); + let request = RegionRequest::BulkInserts(build_bulk_insert_request(region_id, num_rows)); + tokio::spawn(async move { engine.handle_request(region_id, request).await }) + }; + let first = bulk_insert(3); + wait_for_stalls(&listener, 1).await; + + // A bulk write that arrives after staging ended, before the worker drains the stalled one, + // waits behind it. + region + .manifest_ctx + .leave_ending_staging(region_id, started, &Ok::<(), Error>(())); + let second = bulk_insert(5); + wait_for_stalls(&listener, 2).await; + assert!(region.version().memtables.is_empty()); + send_staging_ended(&engine, region_id).await; + first.await.unwrap().unwrap(); + second.await.unwrap().unwrap(); + + assert_eq!(5, count_rows(&engine, region_id).await); +} + +/// Waits until the engine of `listener` has stalled writes `n` times. +async fn wait_for_stalls(listener: &CountingStallListener, n: usize) { + tokio::time::timeout(Duration::from_secs(5), listener.wait_for_count(n)) + .await + .expect("the writes were not stalled"); +} + +/// Has the worker handle the writes it stalled while the region was ending staging. +async fn send_staging_ended(engine: &MitoEngine, region_id: RegionId) { + let (drained, wait) = tokio::sync::oneshot::channel(); + engine + .inner + .workers + .submit_to_worker( + region_id, + WorkerRequest::Background { + region_id, + notify: BackgroundNotify::StagingEnded(drained), + }, + ) + .await + .unwrap(); + wait.await.unwrap(); +} + +#[tokio::test] +async fn test_migration_flush_retries_while_staging_ends() { + let mut env = TestEnv::new().await; + let engine = env.create_engine(MitoConfig::default()).await; + let region_id = RegionId::new(1024, 0); + let (schema, _) = create_staging_region(&engine, region_id).await; + put_range(&engine, region_id, &schema, (0, 3)).await; + let region = engine.get_region(region_id).unwrap(); + let migration_flush = || { + engine.handle_request( + region_id, + RegionRequest::Flush(RegionFlushRequest { + reason: Some(RegionFlushReason::RegionMigration), + ..Default::default() + }), + ) + }; + + let started = region.manifest_ctx.begin_ending_staging(region_id).unwrap(); + assert_eq!(Some(RegionRole::StagingLeader), engine.role(region_id)); + let error = migration_flush().await.unwrap_err(); + assert_eq!(RetryHint::Retryable, error.retry_hint()); + + region + .manifest_ctx + .leave_ending_staging(region_id, started, &Ok::<(), Error>(())); + assert_eq!(0, migration_flush().await.unwrap().affected_rows); +} + +#[tokio::test] +async fn test_apply_staging_manifest_revalidates_after_close_during_read() { + use object_store::layers::mock; + use tokio::sync::Notify; + + #[derive(Default)] + struct ReadGate { + armed: AtomicBool, + entered: Notify, + release: Notify, + } + + struct ManifestReader { + inner: mock::Reader, + is_staging_blob: bool, + gate: Arc, + } + + impl ManifestReader { + async fn pause(&self) { + if self.is_staging_blob && self.gate.armed.swap(false, Ordering::Relaxed) { + self.gate.entered.notify_one(); + self.gate.release.notified().await; + } + } + } + + impl mock::Read for ManifestReader { + async fn open( + &self, + range: mock::BytesRange, + ) -> mock::Result<(mock::RpRead, Box)> { + let result = self.inner.open_dyn(range).await?; + self.pause().await; + Ok(result) + } + + async fn read(&self, range: mock::BytesRange) -> mock::Result<(mock::RpRead, Buffer)> { + // Retain a successful read across Close. A missing blob must not mask the worker's + // late state/binding check by making the object-store read fail first. + let result = self.inner.read_dyn(range).await?; + self.pause().await; + Ok(result) + } + } + + for reopen in [false, true] { + let gate = Arc::new(ReadGate::default()); + let reader_gate = gate.clone(); + let layer = MockLayerBuilder::default() + .reader_factory(Arc::new(move |path, _, inner| { + Box::new(ManifestReader { + inner, + is_staging_blob: path.contains("/staging/blob/"), + gate: reader_gate.clone(), + }) as mock::Reader + })) + .build() + .unwrap(); + let mut env = TestEnv::new().await.with_mock_layer(layer); + let engine = env + .create_engine(MitoConfig { + num_workers: 1, + schedule_compaction_after_edit: false, + ..Default::default() + }) + .await; + let region_id = RegionId::new(1024, 0); + let create = CreateRequestBuilder::new().build(); + let schema = rows_schema(&create); + let table_dir = create.table_dir.clone(); + engine + .handle_request(region_id, RegionRequest::Create(create)) + .await + .unwrap(); + put_range(&engine, region_id, &schema, (0, 3)).await; + enter_staging(&engine, region_id).await; + let old = engine.get_region(region_id).unwrap(); + let partition_expr = float_range_expr("field_0", 0., 100.).as_json_str().unwrap(); + let manifests = engine + .remap_manifests(RemapManifestsRequest { + region_id, + input_regions: vec![region_id], + region_mapping: [(region_id, vec![region_id])].into_iter().collect(), + new_partition_exprs: [(region_id, partition_expr.clone())].into_iter().collect(), + }) + .await + .unwrap(); + gate.armed.store(true, Ordering::Relaxed); + let apply_engine = engine.clone(); + let apply = tokio::spawn(async move { + apply_engine + .handle_request( + region_id, + RegionRequest::ApplyStagingManifest(ApplyStagingManifestRequest { + partition_expr, + central_region_id: region_id, + manifest_path: manifests.manifest_paths[®ion_id].clone(), + }), + ) + .await + }); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()) + .await + .unwrap(); + tokio::time::timeout( + Duration::from_secs(5), + engine.handle_request( + region_id, + RegionRequest::Close(RegionCloseRequest::default()), + ), + ) + .await + .unwrap() + .unwrap(); + assert!(!apply.is_finished()); + let observed = if reopen { + reopen_region(&engine, region_id, table_dir.clone(), true, HashMap::new()).await; + let replacement = engine.get_region(region_id).unwrap(); + assert!(!Arc::ptr_eq(&old, &replacement)); + replacement + } else { + old.clone() + }; + let before = observed.manifest_ctx.manifest().await; + gate.release.notify_one(); + let error = tokio::time::timeout(Duration::from_secs(5), apply) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert_eq!( + if reopen { + StatusCode::RegionNotReady + } else { + StatusCode::RegionNotFound + }, + error.status_code() + ); + let after = observed.manifest_ctx.manifest().await; + assert_eq!(before.manifest_version, after.manifest_version); + assert_eq!(before.files, after.files); + if !reopen { + assert!(!engine.is_region_exists(region_id)); + reopen_region(&engine, region_id, table_dir, true, HashMap::new()).await; + } + assert_eq!(3, count_rows(&engine, region_id).await); + engine.stop().await.unwrap(); + } +} diff --git a/src/mito2/src/engine/table_compaction/tests.rs b/src/mito2/src/engine/table_compaction/tests.rs index 144c586a33e..cb98f54ef40 100644 --- a/src/mito2/src/engine/table_compaction/tests.rs +++ b/src/mito2/src/engine/table_compaction/tests.rs @@ -254,7 +254,7 @@ async fn test_partial_success_can_retry_both_regions_without_duplicate_files() { let region = engine.get_region(id).unwrap(); let manifest = region.manifest_ctx.manifest().await; assert_eq!(1, manifest.files.len()); - assert_eq!(&output, &manifest.files[&output.file_id]); + assert_eq!(&output, manifest.files[&output.file_id].as_ref()); assert_eq!(3, row_count(&engine, id).await); } } @@ -563,7 +563,10 @@ async fn test_indexed_output_with_real_index_object_publishes() { .unwrap() ); let manifest = region.manifest_ctx.manifest().await; - assert_eq!(Some(&output), manifest.files.get(&output.file_id)); + assert_eq!( + Some(&output), + manifest.files.get(&output.file_id).map(AsRef::as_ref) + ); assert_eq!(3, row_count(&engine, region_id).await); } @@ -653,7 +656,10 @@ async fn test_preserve_row_sequence_option_drives_output_admission() { .unwrap() ); let manifest = region.manifest_ctx.manifest().await; - assert_eq!(Some(&output), manifest.files.get(&output.file_id)); + assert_eq!( + Some(&output), + manifest.files.get(&output.file_id).map(AsRef::as_ref) + ); assert_eq!(3, row_count(&engine, region_id).await); } @@ -1041,7 +1047,10 @@ async fn test_multiple_outputs_verify_notify_once_and_release_footers(#[case] wi let manifest = region.manifest_ctx.manifest().await; assert_eq!(outputs.len(), manifest.files.len()); for output in &outputs { - assert_eq!(Some(output), manifest.files.get(&output.file_id)); + assert_eq!( + Some(output), + manifest.files.get(&output.file_id).map(AsRef::as_ref) + ); } assert_eq!(6, row_count(&engine, region_id).await); } diff --git a/src/mito2/src/engine/tail_replay_test.rs b/src/mito2/src/engine/tail_replay_test.rs new file mode 100644 index 00000000000..66a77462208 --- /dev/null +++ b/src/mito2/src/engine/tail_replay_test.rs @@ -0,0 +1,1946 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Tests for the background replay of pending WAL tails. + +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use api::v1::{Mutation, OpType, Rows, WalEntry}; +use async_trait::async_trait; +use common_base::Plugins; +use common_base::readable_size::ReadableSize; +use common_error::ext::ErrorExt; +use common_error::status_code::StatusCode; +use common_recordbatch::RecordBatches; +use common_wal::config::object_store::STANDALONE_GENERATION; +use common_wal::options::WAL_OPTIONS_KEY; +use datatypes::arrow::array::Array; +use log_store::ObjectStoreLogStore; +use log_store::object_store_wal::{NodeWalPrefix, ObjectStoreWalReaders}; +use object_store::ObjectStore; +use object_store::layers::mock::{ + Buffer, Entry as ListEntry, Error as MockError, ErrorKind, List, Metadata, MockLayer, + MockLayerBuilder, Result as MockResult, Write, Writer, +}; +use object_store::services::Memory; +use rstest::rstest; +use store_api::logstore::LogStore; +use store_api::logstore::provider::Provider; +use store_api::region_engine::{ + RegionEngine, RegionManifestInfo, RegionRole, SettableRegionRoleState, +}; +use store_api::region_request::{ + PathType, RegionAlterRequest, RegionCatchupRequest, RegionCloseRequest, RegionDropRequest, + RegionFlushRequest, RegionRequest, RegionTruncateRequest, +}; +use store_api::storage::{RegionId, ScanRequest}; + +use crate::config::MitoConfig; +use crate::engine::MitoEngine; +use crate::engine::flush_test::MockRegionHook; +use crate::engine::index_build_test::async_build_mode_config; +use crate::engine::listener::EventListener; +use crate::engine::object_store_wal_test::{ + count_rows, count_ssts, create_leased_region_on, create_region, encode_delta, is_replaced, + next_manifest_version, publish_delta, put_range, stored_entry_ids, wait_until, wal_config, + wal_options, +}; +use crate::engine::pending_tail_test::{ + add_field, commit_edit, pending_tails_edit, set_pending_tails, +}; +use crate::engine::region_hook::RegionHookRef; +use crate::error::Error; +use crate::manifest::action::{ + PendingTail, RegionEdit, RegionMetaAction, RegionMetaActionList, UptoKind, +}; +use crate::manifest::storage::delta_file; +use crate::region::version::VersionRef; +use crate::region::{MitoRegion, RegionLeaderState}; +use crate::request::{BackgroundNotify, WorkerRequest}; +use crate::sst::file::RegionFileId; +use crate::sst::location::region_dir_from_table_dir; +use crate::test_util::{ + CreateRequestBuilder, DeltaFaults, TestEnv, build_rows, build_rows_for_key, flush_region, + nop_action, put_rows, reopen_region, rows_schema, +}; +use crate::time_provider::StdTimeProvider; +use crate::wal::{EntryId, Wal}; + +/// The node the engine's object store WAL runs as. +const NODE_ID: u64 = 0; +/// Another node sharing the WAL storage. +const REMOTE_NODE_ID: u64 = 1; + +/// Records the tail replays of an engine, and holds their commits while asked to. +#[derive(Default)] +struct TailReplayListener { + commits: AtomicUsize, + /// Commit attempts numbered from it on wait until it changes; 0 holds none. + held_from: AtomicUsize, + failures: Mutex>, + ended: AtomicUsize, + lease_checks: AtomicUsize, + fence_begins: AtomicUsize, + /// Index builds the worker learned stopped. + index_builds_stopped: AtomicUsize, +} + +#[async_trait] +impl EventListener for TailReplayListener { + async fn on_tail_commit_begin(&self, _region_id: RegionId) { + let attempt = self.commits.fetch_add(1, Ordering::Relaxed) + 1; + while (1..=attempt).contains(&self.held_from.load(Ordering::Relaxed)) { + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + + fn on_tail_lease_check(&self, _region_id: RegionId) { + self.lease_checks.fetch_add(1, Ordering::Relaxed); + } + + fn on_tail_fence_begin(&self, _region_id: RegionId) { + self.fence_begins.fetch_add(1, Ordering::Relaxed); + } + + fn on_tail_replay_failed(&self, _region_id: RegionId, error: &Error) { + let failure = (error.status_code(), format!("{error:?}")); + self.failures.lock().unwrap().push(failure); + } + + fn on_tail_replay_ended(&self, _region_id: RegionId) { + self.ended.fetch_add(1, Ordering::Relaxed); + } + + async fn on_index_build_stopped(&self, _region_file_id: RegionFileId) { + self.index_builds_stopped.fetch_add(1, Ordering::Relaxed); + } +} + +impl TailReplayListener { + fn hold_from(&self, attempt: usize) { + self.held_from.store(attempt, Ordering::Relaxed); + } + + fn commits(&self) -> usize { + self.commits.load(Ordering::Relaxed) + } + + fn ended(&self) -> usize { + self.ended.load(Ordering::Relaxed) + } + + fn failed(&self) -> bool { + !self.failures.lock().unwrap().is_empty() + } + + fn failed_with(&self, code: StatusCode) -> bool { + let failures = self.failures.lock().unwrap(); + failures.iter().any(|(failed, _)| *failed == code) + } + + /// Returns whether a failure's debug output names `cause`. + fn failed_naming(&self, cause: &str) -> bool { + let failures = self.failures.lock().unwrap(); + failures.iter().any(|(_, error)| error.contains(cause)) + } + + fn lease_checks(&self) -> usize { + self.lease_checks.load(Ordering::Relaxed) + } + + fn index_builds_stopped(&self) -> usize { + self.index_builds_stopped.load(Ordering::Relaxed) + } + + fn fence_begins(&self) -> usize { + self.fence_begins.load(Ordering::Relaxed) + } +} + +struct TestRegion { + env: TestEnv, + /// The storage of the WAL of every node. + wal_store: ObjectStore, + log_store: Arc, + engine: MitoEngine, + listener: Arc, + region_id: RegionId, + table_dir: String, + /// The WAL entries holding rows `[0, 3)` and `[3, 6)`. + entries: [EntryId; 2], +} + +impl TestRegion { + /// Creates a region on the object store WAL of node [NODE_ID], with leases when + /// `leased`, and writes rows `[0, 3)` and `[3, 6)` in two WAL entries. + async fn new(prefix: &str, leased: bool, mock_layer: Option) -> Self { + Self::build(prefix, leased, mock_layer, None, MitoConfig::default()).await + } + + /// As [Self::new], with `wal_layer` over the WAL storage of every node. + async fn with_wal_layer( + prefix: &str, + leased: bool, + mock_layer: Option, + wal_layer: Option, + ) -> Self { + Self::build(prefix, leased, mock_layer, wal_layer, MitoConfig::default()).await + } + + /// As [Self::new] without leases, with an engine of `config`. + async fn with_config(prefix: &str, mock_layer: Option, config: MitoConfig) -> Self { + Self::build(prefix, false, mock_layer, None, config).await + } + + async fn build( + prefix: &str, + leased: bool, + mock_layer: Option, + wal_layer: Option, + config: MitoConfig, + ) -> Self { + let mut env = TestEnv::with_prefix(prefix).await.with_node_id(NODE_ID); + if let Some(mock_layer) = mock_layer { + env = env.with_mock_layer(mock_layer); + } + let mut wal_store = ObjectStore::new(Memory::default()).unwrap(); + if let Some(wal_layer) = wal_layer { + wal_store = wal_store.layer(wal_layer); + } + let log_store = ObjectStoreLogStore::try_new( + wal_store.clone(), + &wal_config(), + NODE_ID, + STANDALONE_GENERATION, + ) + .await + .unwrap(); + let listener = Arc::new(TailReplayListener::default()); + let engine = env + .create_engine_with_log_store_and_time( + config, + log_store.clone(), + Arc::new(StdTimeProvider), + Some(listener.clone()), + ) + .await; + let region_id = RegionId::new(1, 1); + let (table_dir, schema) = if leased { + let (table_dir, schema, _) = create_leased_region_on(&engine, region_id).await; + (table_dir, schema) + } else { + create_region(&engine, region_id).await + }; + let mut entries = [0; 2]; + for (i, entry) in entries.iter_mut().enumerate() { + put_range(&engine, region_id, &schema, i * 3, i * 3 + 3).await; + *entry = engine + .get_region(region_id) + .unwrap() + .version_control + .current() + .last_entry_id; + } + Self { + env, + wal_store, + log_store, + engine, + listener, + region_id, + table_dir, + entries, + } + } + + /// Records `tails` and reopens the region as a Follower, which replays none of them. + async fn reopen_with_tails(&self, tails: Vec) { + self.reopen_with_edit(pending_tails_edit(tails)).await; + } + + /// Commits `edit`, which records tails, and reopens the region as a Follower. + async fn reopen_with_edit(&self, edit: RegionEdit) { + commit_edit(&self.engine, self.region_id, edit).await; + reopen_region( + &self.engine, + self.region_id, + self.table_dir.clone(), + false, + wal_options(), + ) + .await; + } + + fn promote(&self) { + self.engine + .set_region_role(self.region_id, RegionRole::Leader) + .unwrap(); + } + + fn promote_with_lease(&self) { + let now = Instant::now(); + self.engine + .set_region_role_with_lease( + self.region_id, + RegionRole::Leader, + now, + now + Duration::from_secs(10), + None, + ) + .unwrap(); + } + + /// Asks the worker to replay the tails, as a role change that makes the region writable does. + async fn start_replay(&self) { + let request = WorkerRequest::Background { + region_id: self.region_id, + notify: BackgroundNotify::RegionBecameWritable, + }; + self.engine + .inner + .workers + .submit_to_worker(self.region_id, request) + .await + .unwrap(); + } + + /// Reopens the follower region as a new instance, through a catchup that finds a manifest + /// version another writer published. + async fn replace_follower(&self) { + let mito_region = self.engine.get_region(self.region_id).unwrap(); + let store = self.env.get_object_store().unwrap(); + let version = next_manifest_version(&mito_region).await; + publish_delta( + &store, + &self.region_dir(), + version, + encode_delta(nop_action()).await, + ) + .await; + let catchup = RegionRequest::Catchup(RegionCatchupRequest::default()); + self.engine + .handle_request(self.region_id, catchup) + .await + .unwrap(); + assert!(is_replaced(&self.engine, &mito_region)); + } + + async fn catchup(&self) { + let catchup = RegionRequest::Catchup(RegionCatchupRequest { + set_writable: true, + ..Default::default() + }); + self.engine + .handle_request(self.region_id, catchup) + .await + .unwrap(); + } + + /// Writes rows `[start, end)` of `region_id` to the WAL of [REMOTE_NODE_ID] in one entry + /// with the sequence 100, and returns the entry id. + async fn write_remote_rows(&self, region_id: RegionId, start: usize, end: usize) -> EntryId { + let store = ObjectStoreLogStore::try_new( + self.wal_store.clone(), + &wal_config(), + REMOTE_NODE_ID, + STANDALONE_GENERATION, + ) + .await + .unwrap(); + let provider = Provider::object_store_provider(region_id, remote_prefix()); + let request = CreateRequestBuilder::new().build(); + let entry = WalEntry { + mutations: vec![Mutation { + op_type: OpType::Put as i32, + sequence: 100, + rows: Some(Rows { + schema: rows_schema(&request), + rows: build_rows(start, end), + }), + write_hint: None, + }], + bulk_entries: vec![], + }; + let mut writer = Wal::new(store.clone()).writer(); + writer.add_entry(region_id, 0, &entry, &provider).unwrap(); + let response = writer.write_to_wal().await.unwrap(); + store.stop().await.unwrap(); + response.last_entry_ids[®ion_id] + } + + /// Reopens the region with a tail in the WAL of [REMOTE_NODE_ID], deletes that WAL and + /// promotes the region, so its replay keeps failing. + async fn start_failing_remote_replay(&self) { + let last = self.write_remote_rows(self.region_id, 6, 9).await; + self.reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + self.delete_remote_objects().await; + self.catchup().await; + } + + /// Deletes every object of the WAL of [REMOTE_NODE_ID], so reading it fails. + async fn delete_remote_objects(&self) { + let objects = self + .wal_store + .list(&format!("{}/objects/", remote_prefix())) + .await + .unwrap(); + for object in objects { + self.wal_store.delete(object.path()).await.unwrap(); + } + } + + /// Returns the bound a takeover would record for the WAL of [REMOTE_NODE_ID]. + async fn remote_listed_bound(&self) -> EntryId { + let readers = ObjectStoreWalReaders::new(self.wal_store.clone()); + let prefix = NodeWalPrefix::try_new(&remote_prefix()).unwrap(); + readers + .reader(&prefix) + .unwrap() + .list_upper_bound(self.region_id) + .await + .unwrap() + } + + /// Writes 99 to the row of key "6", which [Self::write_remote_rows] also writes. + async fn overwrite_remote_row(&self) { + let rows = Rows { + schema: rows_schema(&CreateRequestBuilder::new().build()), + rows: build_rows_for_key("6", 6, 7, 99), + }; + put_rows(&self.engine, self.region_id, rows).await; + } + + /// Returns whether a scan sees the value [Self::overwrite_remote_row] wrote. + async fn sees_overwrite(&self) -> bool { + let request = ScanRequest::default(); + let stream = self.engine.scan_to_stream(self.region_id, request).await; + let batches = RecordBatches::try_collect(stream.unwrap()).await.unwrap(); + batches.pretty_print().unwrap().contains("99.0") + } + + fn version(&self) -> VersionRef { + self.engine.get_region(self.region_id).unwrap().version() + } + + fn has_pending_tails(&self) -> bool { + self.version().has_pending_tails() + } + + async fn wait_until_replayed(&self) { + wait_until(|| !self.has_pending_tails()).await; + } + + fn region_dir(&self) -> String { + region_dir_from_table_dir(&self.table_dir, self.region_id, PathType::Bare) + } + + async fn stored_ssts(&self) -> usize { + count_ssts(&self.env.get_object_store().unwrap(), &self.region_dir()).await + } + + async fn stored_sst_names(&self) -> Vec { + let store = self.env.get_object_store().unwrap(); + let entries = store.list(&self.region_dir()).await.unwrap(); + entries + .iter() + .map(|entry| entry.name().to_string()) + .filter(|name| name.ends_with(".parquet")) + .collect() + } +} + +fn own_tail(after: EntryId, upto: EntryId, upto_kind: UptoKind) -> PendingTail { + PendingTail { + node_id: NODE_ID, + after, + upto, + upto_kind, + } +} + +fn remote_tail(after: EntryId, upto: EntryId, upto_kind: UptoKind) -> PendingTail { + PendingTail { + node_id: REMOTE_NODE_ID, + ..own_tail(after, upto, upto_kind) + } +} + +fn remote_prefix() -> String { + wal_config().node_prefix(REMOTE_NODE_ID, STANDALONE_GENERATION) +} + +#[tokio::test] +async fn test_replays_own_tail_after_reopen() { + let region = TestRegion::new("tail-replay-own", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let version = region.version(); + assert_eq!(0, version.memtables.num_rows()); + + region.promote(); + region.wait_until_replayed().await; + + let version = region.version(); + assert_eq!(1, version.ssts.levels()[0].files.len()); + assert_eq!(region.entries[1], version.flushed_entry_id); + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); + // The commit released the replayed WAL. + assert!( + stored_entry_ids(®ion.log_store, region.region_id) + .await + .is_empty() + ); +} + +#[tokio::test] +async fn test_tail_commit_advances_flushed_sequence() { + let region = TestRegion::new("tail-replay-sequence", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + + region.promote(); + region.wait_until_replayed().await; + + let version = region.version(); + assert_eq!(6, version.flushed_sequence); +} + +#[tokio::test] +async fn test_each_tail_has_its_own_ssts() { + let region = TestRegion::new("tail-replay-two-tails", false, None).await; + let [first, second] = region.entries; + region + .reopen_with_tails(vec![ + own_tail(0, first, UptoKind::Reported), + own_tail(first, second, UptoKind::Reported), + ]) + .await; + + region.promote(); + region.wait_until_replayed().await; + + let version = region.version(); + let files = &version.ssts.levels()[0].files; + assert_eq!(2, files.len()); + let mut num_rows = files + .values() + .map(|file| file.meta_ref().num_rows) + .collect::>(); + num_rows.sort(); + assert_eq!(vec![3, 3], num_rows); +} + +#[tokio::test] +async fn test_own_listed_tail_is_cleared_without_lease() { + let region = TestRegion::new("tail-replay-own-listed", false, None).await; + let upto = region.entries[1] + 1000; + region + .reopen_with_tails(vec![own_tail(0, upto, UptoKind::Listed)]) + .await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + assert!( + mito_region + .manifest_ctx + .promote_after_catchup(region.region_id) + ); + + region.start_replay().await; + region.wait_until_replayed().await; + + let version = region.version(); + assert_eq!(region.entries[1], version.flushed_entry_id); + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_reported_tail_must_reach_its_upto() { + let region = TestRegion::new("tail-replay-missing-upto", false, None).await; + let upto = region.entries[1] + 1; + region + .reopen_with_tails(vec![own_tail(0, upto, UptoKind::Reported)]) + .await; + + region.promote(); + wait_until(|| region.listener.failed_with(StatusCode::StorageUnavailable)).await; + + assert!(region.has_pending_tails()); +} + +#[tokio::test] +async fn test_remote_tail_fails_on_log_store_reading_no_other_node() { + let mut env = TestEnv::with_prefix("tail-replay-raft-engine") + .await + .with_node_id(NODE_ID); + let listener = Arc::new(TailReplayListener::default()); + let engine = env + .create_engine_with(MitoConfig::default(), None, Some(listener.clone()), None) + .await; + let region_id = RegionId::new(1, 1); + let request = RegionRequest::Create(CreateRequestBuilder::new().build()); + engine.handle_request(region_id, request).await.unwrap(); + let tails = vec![remote_tail(0, 1, UptoKind::Listed)]; + set_pending_tails(&engine, region_id, tails.clone()).await; + + // The Raft engine log store cannot read another node's WAL. + let notify = BackgroundNotify::RegionBecameWritable; + let request = WorkerRequest::Background { region_id, notify }; + let workers = &engine.inner.workers; + workers.submit_to_worker(region_id, request).await.unwrap(); + wait_until(|| listener.failed_with(StatusCode::Unsupported)).await; + + let region = engine.get_region(region_id).unwrap(); + assert_eq!(tails, region.version().pending_tails); +} + +#[rstest] +#[case::reported(UptoKind::Reported)] +#[case::listed(UptoKind::Listed)] +#[tokio::test] +async fn test_tail_read_stops_at_its_upto(#[case] upto_kind: UptoKind) { + let region = TestRegion::new("tail-replay-stop-at-upto", false, None).await; + // The object store WAL keeps an entry's position in the low 20 bits, so the listed bound + // is the last id of the first entry's object and names no entry. + let upto = match upto_kind { + UptoKind::Reported => region.entries[0], + UptoKind::Listed => region.entries[0] | ((1 << 20) - 1), + }; + region + .reopen_with_tails(vec![own_tail(0, upto, upto_kind)]) + .await; + // Makes the entry after the tail unreadable. + let prefix = wal_config().node_prefix(NODE_ID, STANDALONE_GENERATION); + let objects = region + .wal_store + .list(&format!("{prefix}/objects/")) + .await + .unwrap(); + let last = objects.iter().map(|entry| entry.path()).max().unwrap(); + region.wal_store.delete(last).await.unwrap(); + + region.promote(); + region.wait_until_replayed().await; + + assert_eq!(region.entries[0], region.version().flushed_entry_id); +} + +#[tokio::test] +async fn test_catchup_replays_remote_reported_tail() { + let region = TestRegion::new("tail-replay-remote-reported", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + + region.catchup().await; + region.wait_until_replayed().await; + + assert_eq!(last, region.version().flushed_entry_id); + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_missing_reported_object_fails_until_restored() { + let region = TestRegion::new("tail-replay-missing-object", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + let objects = region + .wal_store + .list(&format!("{}/objects/", remote_prefix())) + .await + .unwrap(); + let path = objects + .iter() + .map(|entry| entry.path()) + .max() + .unwrap() + .to_string(); + let bytes = region.wal_store.read(&path).await.unwrap(); + region.wal_store.delete(&path).await.unwrap(); + + region.catchup().await; + wait_until(|| { + region + .listener + .failed_naming("does not hold the history of region") + }) + .await; + assert!(region.has_pending_tails()); + region.wal_store.write(&path, bytes).await.unwrap(); + region.wait_until_replayed().await; + + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_failed_round_deletes_chunks_of_earlier_tails() { + let region = TestRegion::new("tail-replay-failed-round", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![ + own_tail(0, region.entries[1], UptoKind::Reported), + remote_tail(0, last, UptoKind::Reported), + ]) + .await; + let objects = region + .wal_store + .list(&format!("{}/objects/", remote_prefix())) + .await + .unwrap(); + let path = objects[objects.len() - 1].path().to_string(); + let bytes = region.wal_store.read(&path).await.unwrap(); + region.wal_store.delete(&path).await.unwrap(); + + region.catchup().await; + // The round wrote the own tail's chunk before the remote read failed. + wait_until(|| region.listener.failed()).await; + assert_eq!(0, region.stored_ssts().await); + region.wal_store.write(&path, bytes).await.unwrap(); + region.wait_until_replayed().await; + + assert_eq!(2, region.stored_ssts().await); + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_remote_listed_tail_shrinks_until_leader_lease() { + let region = TestRegion::new("tail-replay-remote-listed", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + let upto = region.remote_listed_bound().await; + region + .reopen_with_tails(vec![remote_tail(0, upto, UptoKind::Listed)]) + .await; + + region.catchup().await; + let shrunk = vec![remote_tail(last, upto, UptoKind::Listed)]; + wait_until(|| region.version().pending_tails == shrunk).await; + assert_eq!(last, region.version().flushed_entry_id); + region.promote_with_lease(); + region.wait_until_replayed().await; + + assert_eq!(last, region.version().flushed_entry_id); + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_unread_remote_listed_tail_waits_for_leader_lease() { + let region = TestRegion::new("tail-replay-remote-idle", false, None).await; + region.write_remote_rows(RegionId::new(1, 2), 0, 1).await; + let upto = region.remote_listed_bound().await; + region + .reopen_with_tails(vec![remote_tail(0, upto, UptoKind::Listed)]) + .await; + let version = next_manifest_version(®ion.engine.get_region(region.region_id).unwrap()).await; + + region.catchup().await; + // The catchup reopens the region. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + // A second check means the empty read is done and the replay keeps waiting. + wait_until(|| region.listener.lease_checks() >= 2).await; + assert!(region.has_pending_tails()); + assert_eq!(version, next_manifest_version(&mito_region).await); + // The replay waits for the lease instead of ending and starting again. + assert_eq!(0, region.listener.ended()); + region.promote_with_lease(); + region.wait_until_replayed().await; + + assert_eq!(version + 1, next_manifest_version(&mito_region).await); + assert_eq!(0, region.version().flushed_entry_id); +} + +#[tokio::test] +async fn test_writes_after_replay_win_over_replayed_rows() { + let region = TestRegion::new("tail-replay-later-writes", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + region.catchup().await; + region.wait_until_replayed().await; + + region.overwrite_remote_row().await; + + assert!(region.sees_overwrite().await); +} + +#[tokio::test] +async fn test_writes_during_replay_win_over_replayed_rows() { + let region = TestRegion::new("tail-replay-earlier-writes", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + // Whatever records tails raises the committed sequence to theirs (100 to 102) and leaves + // this node no unflushed entry at or below theirs: later writes win and stay in the WAL. + flush_region(®ion.engine, region.region_id, None).await; + let edit = RegionEdit { + committed_sequence: Some(102), + ..pending_tails_edit(vec![remote_tail(0, last, UptoKind::Reported)]) + }; + region.reopen_with_edit(edit).await; + region.listener.hold_from(1); + region.catchup().await; + wait_until(|| region.listener.commits() == 1).await; + + region.overwrite_remote_row().await; + region.listener.hold_from(0); + region.wait_until_replayed().await; + + assert!(region.sees_overwrite().await); + region.reopen_with_tails(vec![]).await; + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); + assert!(region.sees_overwrite().await); +} + +#[tokio::test] +async fn test_create_reopening_region_replays_tails() { + let region = TestRegion::new("tail-replay-create", false, None).await; + let tail = own_tail(0, region.entries[1], UptoKind::Reported); + set_pending_tails(®ion.engine, region.region_id, vec![tail]).await; + let close = RegionRequest::Close(RegionCloseRequest::default()); + region + .engine + .handle_request(region.region_id, close) + .await + .unwrap(); + + let request = CreateRequestBuilder::new() + .insert_option(WAL_OPTIONS_KEY, &wal_options()[WAL_OPTIONS_KEY]) + .build(); + region + .engine + .handle_request(region.region_id, RegionRequest::Create(request)) + .await + .unwrap(); + region.wait_until_replayed().await; + + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_direct_promotion_counts_as_leader_lease() { + let region = TestRegion::new("tail-replay-direct-promotion", false, None).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + let upto = region.remote_listed_bound().await; + region + .reopen_with_tails(vec![remote_tail(0, upto, UptoKind::Listed)]) + .await; + region.catchup().await; + let shrunk = vec![remote_tail(last, upto, UptoKind::Listed)]; + wait_until(|| region.version().pending_tails == shrunk).await; + // Setting a leader to Leader promotes nothing, so it is no lease. + region.promote(); + let mito_region = region.engine.get_region(region.region_id).unwrap(); + assert!(!mito_region.manifest_ctx.leader_lease_received()); + + region + .engine + .set_region_role(region.region_id, RegionRole::Follower) + .unwrap(); + region.promote(); + region.wait_until_replayed().await; +} + +#[tokio::test] +async fn test_leased_first_round_clears_remote_listed_tail() { + let region = TestRegion::new("tail-replay-leased-listed", false, None).await; + region.write_remote_rows(region.region_id, 6, 9).await; + let upto = region.remote_listed_bound().await; + region + .reopen_with_tails(vec![remote_tail(0, upto, UptoKind::Listed)]) + .await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let version = next_manifest_version(&mito_region).await; + + // No catchup promoted this instance, so it counts as leased and no commit keeps part of + // the tail. + region.promote(); + region.wait_until_replayed().await; + + assert_eq!(version + 1, next_manifest_version(&mito_region).await); + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_replay_fills_columns_added_after_the_tail() { + let region = TestRegion::new("tail-replay-alter", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + // Writable without asking the worker to replay, so the ALTER runs first. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + mito_region.set_role(RegionRole::Leader); + let add_column = RegionRequest::Alter(RegionAlterRequest { + kind: add_field(None), + }); + region + .engine + .handle_request(region.region_id, add_column) + .await + .unwrap(); + + region.start_replay().await; + region.wait_until_replayed().await; + + let stream = region + .engine + .scan_to_stream(region.region_id, ScanRequest::default()) + .await + .unwrap(); + let batches = RecordBatches::try_collect(stream).await.unwrap(); + let nulls = batches + .iter() + .map(|batch| batch.column_by_name("field_1").unwrap().null_count()) + .sum::(); + assert_eq!(6, nulls); +} + +#[tokio::test] +async fn test_truncate_after_tail_commit_keeps_flushed_entry_id() { + let region = TestRegion::new("tail-replay-truncate", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.promote(); + region.wait_until_replayed().await; + + region + .engine + .handle_request( + region.region_id, + RegionRequest::Truncate(RegionTruncateRequest::All), + ) + .await + .unwrap(); + + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let manifest = mito_region.manifest_ctx.manifest().await; + assert_eq!(region.entries[1], manifest.flushed_entry_id); +} + +#[tokio::test] +async fn test_tail_commit_conflict_reloads_region() { + let region = TestRegion::new("tail-replay-conflict", true, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let old = region.engine.get_region(region.region_id).unwrap(); + publish_delta( + ®ion.env.get_object_store().unwrap(), + ®ion.region_dir(), + next_manifest_version(&old).await, + encode_delta(nop_action()).await, + ) + .await; + + region.promote_with_lease(); + wait_until(|| is_replaced(®ion.engine, &old)).await; + + assert_eq!(1, region.listener.ended()); + assert!(region.has_pending_tails()); + assert_eq!(1, region.stored_ssts().await); +} + +/// Lets the first write of a manifest delta land, then fails it as if its response was lost. +fn lose_delta_response(version: Arc) -> MockLayer { + let armed = Arc::new(AtomicBool::new(true)); + MockLayerBuilder::default() + .writer_factory(Arc::new(move |path, _args, inner| { + let target = delta_file(version.load(Ordering::Relaxed)); + if path.contains(&target) && armed.swap(false, Ordering::Relaxed) { + Box::new(LostResponseWriter { inner }) + } else { + inner + } + })) + .build() + .unwrap() +} + +struct LostResponseWriter { + inner: Writer, +} + +impl Write for LostResponseWriter { + async fn write(&mut self, bs: Buffer) -> MockResult<()> { + self.inner.write(bs).await + } + + async fn close(&mut self) -> MockResult { + self.inner.close().await?; + Err(MockError::new(ErrorKind::Unexpected, "response lost")) + } + + async fn abort(&mut self) -> MockResult<()> { + self.inner.abort().await + } +} + +#[tokio::test] +async fn test_tail_commit_with_unknown_result_is_retried() { + let version = Arc::new(AtomicU64::new(u64::MAX)); + let region = TestRegion::new( + "tail-replay-unknown-commit", + true, + Some(lose_delta_response(version.clone())), + ) + .await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + version.store(next_manifest_version(&mito_region).await, Ordering::Relaxed); + + region.promote_with_lease(); + region.wait_until_replayed().await; + + assert_eq!(2, region.listener.commits()); + assert_eq!(1, region.stored_ssts().await); + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_uncertain_commit_is_retried_after_expired_lease() { + let version = Arc::new(AtomicU64::new(u64::MAX)); + let region = TestRegion::new( + "tail-replay-uncertain-expired", + true, + Some(lose_delta_response(version.clone())), + ) + .await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + version.store(next_manifest_version(&mito_region).await, Ordering::Relaxed); + region.listener.hold_from(2); + region.promote_with_lease(); + // The first attempt lands but reports an error. + wait_until(|| region.listener.commits() == 2).await; + let expired = Instant::now() + Duration::from_secs(60); + region.engine.freeze_lease_clock(expired); + region.listener.hold_from(3); + + wait_until(|| region.listener.commits() == 3).await; + region.engine.freeze_lease_clock(Instant::now()); + region.listener.hold_from(0); + region.wait_until_replayed().await; + + assert!(!region.listener.failed()); + assert_eq!(1, region.stored_ssts().await); + assert!(Arc::ptr_eq( + &mito_region, + ®ion.engine.get_region(region.region_id).unwrap() + )); +} + +#[tokio::test] +async fn test_closed_replay_keeps_chunks_of_uncertain_commit() { + let version = Arc::new(AtomicU64::new(u64::MAX)); + let region = TestRegion::new( + "tail-replay-uncertain-closed", + true, + Some(lose_delta_response(version.clone())), + ) + .await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + version.store(next_manifest_version(&mito_region).await, Ordering::Relaxed); + region.listener.hold_from(2); + region.promote_with_lease(); + // The first attempt lands but reports an error, and the region closes before a retry. + wait_until(|| region.listener.commits() == 2).await; + let close = RegionRequest::Close(RegionCloseRequest::default()); + region + .engine + .handle_request(region.region_id, close) + .await + .unwrap(); + region.listener.hold_from(0); + wait_until(|| region.listener.ended() == 1).await; + + assert_eq!(1, region.stored_ssts().await); + reopen_region( + ®ion.engine, + region.region_id, + region.table_dir.clone(), + false, + wal_options(), + ) + .await; + assert!(!region.has_pending_tails()); + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_tail_commit_after_deadline_deletes_chunks() { + let region = TestRegion::new("tail-replay-deadline", true, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote_with_lease(); + wait_until(|| region.listener.commits() == 1).await; + assert_eq!(1, region.stored_ssts().await); + + region + .engine + .freeze_lease_clock(Instant::now() + Duration::from_secs(60)); + region.listener.hold_from(0); + wait_until(|| region.listener.failed_with(StatusCode::RegionNotReady)).await; + region + .engine + .set_region_role(region.region_id, RegionRole::Follower) + .unwrap(); + wait_until(|| region.listener.ended() == 1).await; + + assert!(region.has_pending_tails()); + assert_eq!(0, region.stored_ssts().await); +} + +#[derive(Debug)] +enum Stop { + Downgrade, + Close, + Drop, +} + +#[rstest] +#[case::downgrade(Stop::Downgrade)] +#[case::close(Stop::Close)] +#[case::drop(Stop::Drop)] +#[tokio::test] +async fn test_stopping_region_cancels_replay(#[case] stop: Stop) { + let region = TestRegion::new("tail-replay-cancel", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + + match stop { + Stop::Downgrade => { + region + .engine + .set_region_role_state_gracefully( + region.region_id, + SettableRegionRoleState::Follower, + ) + .await + .unwrap(); + } + Stop::Close => { + region + .engine + .handle_request( + region.region_id, + RegionRequest::Close(RegionCloseRequest::default()), + ) + .await + .unwrap(); + } + Stop::Drop => { + region + .engine + .handle_request( + region.region_id, + RegionRequest::Drop(RegionDropRequest { + fast_path: false, + force: false, + partial_drop: false, + }), + ) + .await + .unwrap(); + } + } + region.listener.hold_from(0); + wait_until(|| region.listener.ended() == 1).await; + + assert_eq!(1, region.listener.commits()); + assert_eq!(0, region.stored_ssts().await); +} + +#[tokio::test] +async fn test_cancelled_replay_does_not_publish_after_promotion() { + let region = TestRegion::new("tail-replay-cancel-promote", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + + region + .engine + .set_region_role_state_gracefully(region.region_id, SettableRegionRoleState::Follower) + .await + .unwrap(); + region.promote(); + region.listener.hold_from(0); + region.wait_until_replayed().await; + + assert_eq!(2, region.listener.commits()); +} + +#[rstest] +#[case::demoted(true)] +#[case::tails_changed(false)] +#[tokio::test] +async fn test_tail_commit_checks_role_and_tails_under_lock(#[case] demote: bool) { + let region = TestRegion::new("tail-replay-locked-check", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + + if demote { + // The replay waits for the manifest lock while the region is demoted: a read held + // here queues its write, which then fails a further read. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let manager = &mito_region.manifest_ctx.manifest_manager; + let reading = manager.read().await; + region.listener.hold_from(0); + wait_until(|| manager.try_read().is_err()).await; + region + .engine + .set_region_role(region.region_id, RegionRole::Follower) + .unwrap(); + drop(reading); + } else { + set_pending_tails(®ion.engine, region.region_id, vec![]).await; + region.listener.hold_from(0); + } + wait_until(|| region.listener.ended() == 1).await; + + assert_eq!(1, region.listener.commits()); + assert_eq!(demote, region.has_pending_tails()); + assert_eq!(0, region.stored_ssts().await); +} + +/// A commit notice that arrives after a sync rebuilt the version from a later manifest does +/// not bring back the tails that manifest cleared. +#[tokio::test] +async fn test_late_tail_commit_notice_keeps_synced_tails() { + let region = TestRegion::new("tail-replay-late-notice", false, None).await; + let replayed = vec![remote_tail(0, 10, UptoKind::Listed)]; + let shrunk = vec![remote_tail(5, 10, UptoKind::Listed)]; + // A sync expects a follower with empty memtables. + flush_region(®ion.engine, region.region_id, None).await; + region.reopen_with_tails(shrunk.clone()).await; + // A new leader clears the tails, and this node syncs its manifest. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let version = next_manifest_version(&mito_region).await; + let clear = + RegionMetaActionList::with_action(RegionMetaAction::Edit(pending_tails_edit(vec![]))); + let store = region.env.get_object_store().unwrap(); + publish_delta( + &store, + ®ion.region_dir(), + version, + encode_delta(clear).await, + ) + .await; + let manifest_info = RegionManifestInfo::mito(version, 0, 0); + region + .engine + .sync_region(region.region_id, manifest_info.into()) + .await + .unwrap(); + assert!(!region.has_pending_tails()); + + // The notice of the replay that left `shrunk` arrives after the sync. + let notify = BackgroundNotify::TailReplayCommitted { + region: mito_region, + edit: pending_tails_edit(shrunk), + tails: replayed, + }; + let request = WorkerRequest::Background { + region_id: region.region_id, + notify, + }; + let workers = ®ion.engine.inner.workers; + workers + .submit_to_worker(region.region_id, request) + .await + .unwrap(); + // The worker handles the notice before this request, which keeps the version. + region + .engine + .set_region_role_state_gracefully(region.region_id, SettableRegionRoleState::Follower) + .await + .unwrap(); + + assert!(!region.has_pending_tails()); +} + +/// Parks every listing of a path `is_target` accepts once `stall` is set, and counts them. +fn stall_listings( + is_target: fn(&str) -> bool, + stall: Arc, + stalled: Arc, +) -> MockLayer { + MockLayerBuilder::default() + .lister_factory(Arc::new(move |path, _args, inner| { + if is_target(path) && stall.load(Ordering::Relaxed) { + stalled.fetch_add(1, Ordering::Relaxed); + Box::new(StalledLister) + } else { + inner + } + })) + .build() + .unwrap() +} + +/// Whether `path` is in the WAL of [REMOTE_NODE_ID]. +fn is_remote_wal(path: &str) -> bool { + path.starts_with(&remote_prefix()) +} + +/// Whether `path` is in a manifest directory. +fn is_manifest(path: &str) -> bool { + path.contains("/manifest/") +} + +struct StalledLister; + +impl List for StalledLister { + async fn next(&mut self) -> MockResult> { + std::future::pending().await + } +} + +#[rstest] +#[case::close(false)] +#[case::engine_stop(true)] +#[tokio::test] +async fn test_stalled_remote_read_ends_when_region_stops(#[case] stop_engine: bool) { + let stall = Arc::new(AtomicBool::new(false)); + let stalled = Arc::new(AtomicUsize::new(0)); + let layer = stall_listings(is_remote_wal, stall.clone(), stalled.clone()); + let region = + TestRegion::with_wal_layer("tail-replay-stalled-read", false, None, Some(layer)).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + stall.store(true, Ordering::Relaxed); + region.catchup().await; + wait_until(|| stalled.load(Ordering::Relaxed) > 0).await; + let replaying = Arc::downgrade(®ion.engine.get_region(region.region_id).unwrap()); + + if stop_engine { + region.engine.stop().await.unwrap(); + } else { + let close = RegionRequest::Close(RegionCloseRequest::default()); + region + .engine + .handle_request(region.region_id, close) + .await + .unwrap(); + } + + wait_until(|| replaying.strong_count() == 0).await; +} + +#[tokio::test] +async fn test_replacing_region_cancels_its_replay() { + let stall = Arc::new(AtomicBool::new(false)); + let stalled = Arc::new(AtomicUsize::new(0)); + let layer = stall_listings(is_remote_wal, stall.clone(), stalled.clone()); + let region = TestRegion::with_wal_layer("tail-replay-replaced", false, None, Some(layer)).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + stall.store(true, Ordering::Relaxed); + region.promote(); + wait_until(|| stalled.load(Ordering::Relaxed) > 0).await; + let replaying = Arc::downgrade(®ion.engine.get_region(region.region_id).unwrap()); + + // Demotes without telling the worker, so only the replacement cancels the replay. + region + .engine + .get_region(region.region_id) + .unwrap() + .set_role(RegionRole::Follower); + region.replace_follower().await; + + wait_until(|| replaying.strong_count() == 0).await; +} + +#[tokio::test] +async fn test_commit_notice_of_replaced_instance_changes_nothing() { + let region = TestRegion::new("tail-replay-replaced-notice", false, None).await; + let tails = vec![own_tail(0, region.entries[1], UptoKind::Reported)]; + region.reopen_with_tails(tails.clone()).await; + let old = region.engine.get_region(region.region_id).unwrap(); + region.replace_follower().await; + + let notify = BackgroundNotify::TailReplayCommitted { + region: old.clone(), + edit: pending_tails_edit(vec![]), + tails: tails.clone(), + }; + let request = WorkerRequest::Background { + region_id: region.region_id, + notify, + }; + let workers = ®ion.engine.inner.workers; + workers + .submit_to_worker(region.region_id, request) + .await + .unwrap(); + // The worker handles the notice before this request, which keeps the version. + region + .engine + .set_region_role_state_gracefully(region.region_id, SettableRegionRoleState::Follower) + .await + .unwrap(); + + assert_eq!(tails, old.version().pending_tails); + assert_eq!(tails, region.version().pending_tails); +} + +/// Moves a writable leader to `busy`, Altering or Editing. +fn set_busy(region: &MitoRegion, busy: RegionLeaderState) { + match busy { + RegionLeaderState::Altering => region.set_altering(), + _ => region.set_editing(RegionLeaderState::Writable), + } + .unwrap(); +} + +#[rstest] +#[case::altering(RegionLeaderState::Altering)] +#[case::editing(RegionLeaderState::Editing)] +#[tokio::test] +async fn test_replay_backs_off_while_busy(#[case] busy: RegionLeaderState) { + let region = TestRegion::new("tail-replay-busy", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + let chunks = region.stored_sst_names().await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + set_busy(&mito_region, busy); + + region.listener.hold_from(0); + wait_until(|| region.listener.commits() == 2).await; + assert!(region.has_pending_tails()); + mito_region.switch_state_to_writable(busy); + region.wait_until_replayed().await; + + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); + assert!(!region.listener.failed()); + // The retries commit the chunks written before the region became busy. + assert_eq!(chunks, region.stored_sst_names().await); +} + +#[tokio::test] +async fn test_replay_round_keeps_no_flushed_memtables() { + let region = TestRegion::new("tail-replay-memtables", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + let memtables = Arc::downgrade(®ion.version().memtables); + + region.overwrite_remote_row().await; + flush_region(®ion.engine, region.region_id, None).await; + wait_until(|| memtables.strong_count() == 0).await; + region.listener.hold_from(0); + region.wait_until_replayed().await; +} + +#[rstest] +#[case::altering(RegionLeaderState::Altering)] +#[case::editing(RegionLeaderState::Editing)] +#[tokio::test] +async fn test_promotion_replays_tails_of_region_busy_meanwhile(#[case] busy: RegionLeaderState) { + let region = TestRegion::new("tail-replay-promote-busy", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + // The worker starts an ALTER or an edit between the promotion and its check of the role. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + mito_region.set_role(RegionRole::Leader); + set_busy(&mito_region, busy); + region + .engine + .inner + .notify_if_became_writable(&mito_region, false); + + wait_until(|| region.listener.commits() > 0).await; + mito_region.switch_state_to_writable(busy); + region.wait_until_replayed().await; +} + +#[tokio::test] +async fn test_triggers_during_replay_start_no_second_one() { + let region = TestRegion::new("tail-replay-repeated-trigger", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + // Every replay task holds the region. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let holders = Arc::strong_count(&mito_region); + + region.start_replay().await; + region.start_replay().await; + // The worker handles both triggers before this request, which leaves the region writable. + region + .engine + .set_region_role_state_gracefully(region.region_id, SettableRegionRoleState::Leader) + .await + .unwrap(); + // The graceful role change holds the region until its task ends. + wait_until(|| Arc::strong_count(&mito_region) == holders).await; + + region.listener.hold_from(0); + region.wait_until_replayed().await; + assert_eq!(1, region.listener.commits()); +} + +#[tokio::test] +async fn test_replay_chunks_fire_sst_written_hook() { + let mut env = TestEnv::with_prefix("tail-replay-hook") + .await + .with_node_id(NODE_ID); + let wal_store = ObjectStore::new(Memory::default()).unwrap(); + let log_store = + ObjectStoreLogStore::try_new(wal_store, &wal_config(), NODE_ID, STANDALONE_GENERATION) + .await + .unwrap(); + let hook = Arc::new(MockRegionHook::new()); + let plugins = Plugins::new(); + plugins.insert(hook.clone() as RegionHookRef); + let engine = env + .create_engine_with_log_store_and_plugins(MitoConfig::default(), log_store, plugins) + .await; + let region_id = RegionId::new(1, 1); + let (table_dir, schema) = create_region(&engine, region_id).await; + put_range(&engine, region_id, &schema, 0, 3).await; + let region = engine.get_region(region_id).unwrap(); + let last = region.version_control.current().last_entry_id; + let tails = vec![own_tail(0, last, UptoKind::Reported)]; + commit_edit(&engine, region_id, pending_tails_edit(tails)).await; + reopen_region(&engine, region_id, table_dir, false, wal_options()).await; + + engine + .set_region_role(region_id, RegionRole::Leader) + .unwrap(); + let region = engine.get_region(region_id).unwrap(); + wait_until(|| !region.version().has_pending_tails()).await; + + // Nothing else wrote an SST: the region never flushed. + assert_eq!(1, hook.sst_written_count.load(Ordering::Relaxed)); +} + +#[tokio::test] +async fn test_replay_writes_chunk_when_memory_budget_is_exhausted() { + // The replay budget is 8 KiB: a 256 B floor and a 4 KiB pool, which a few hundred rows + // exceed. The engine flushes at 32 KiB, above what the region itself writes. + let config = MitoConfig { + global_write_buffer_size: ReadableSize::kb(64), + ..Default::default() + }; + let region = TestRegion::with_config("tail-replay-chunk-budget", None, config).await; + let first = region.write_remote_rows(region.region_id, 6, 300).await; + let last = region.write_remote_rows(region.region_id, 300, 600).await; + assert!(first < last); + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + assert_eq!(0, region.stored_ssts().await); + + region.catchup().await; + region.wait_until_replayed().await; + + // Each entry of the tail went to its own chunk. + assert_eq!(2, region.version().ssts.levels()[0].files.len()); + assert_eq!(600, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_failed_replay_reloads_region_from_newer_manifest() { + let region = TestRegion::new("tail-replay-failed-reload", false, None).await; + region.start_failing_remote_replay().await; + wait_until(|| region.listener.failed()).await; + + // Another writer clears the tails, which nothing can replay any more. + let old = region.engine.get_region(region.region_id).unwrap(); + let action_list = + RegionMetaActionList::with_action(RegionMetaAction::Edit(pending_tails_edit(vec![]))); + publish_delta( + ®ion.env.get_object_store().unwrap(), + ®ion.region_dir(), + next_manifest_version(&old).await, + encode_delta(action_list).await, + ) + .await; + wait_until(|| is_replaced(®ion.engine, &old)).await; + + assert!(!region.has_pending_tails()); + assert_eq!(1, region.listener.ended()); + // The region's own rows survive the fence and the reload. + assert_eq!(6, count_rows(®ion.engine, region.region_id).await); +} + +#[rstest] +#[case::create_on_flush(true)] +#[case::no_create_on_flush(false)] +#[tokio::test] +async fn test_replayed_ssts_build_index_as_flushed(#[case] create_on_flush: bool) { + let region = TestRegion::with_config( + "tail-replay-index", + None, + async_build_mode_config(create_on_flush), + ) + .await; + let region_id = RegionId::new(1, 2); + let request = CreateRequestBuilder::new() + .insert_option(WAL_OPTIONS_KEY, &wal_options()[WAL_OPTIONS_KEY]) + .build_with_index(); + let table_dir = request.table_dir.clone(); + region + .engine + .handle_request(region_id, RegionRequest::Create(request)) + .await + .unwrap(); + let last = region.write_remote_rows(region_id, 0, 3).await; + let tails = vec![remote_tail(0, last, UptoKind::Reported)]; + commit_edit(®ion.engine, region_id, pending_tails_edit(tails)).await; + reopen_region(®ion.engine, region_id, table_dir, false, wal_options()).await; + + let catchup = RegionRequest::Catchup(RegionCatchupRequest { + set_writable: true, + ..Default::default() + }); + region + .engine + .handle_request(region_id, catchup) + .await + .unwrap(); + let replayed = region.engine.get_region(region_id).unwrap(); + wait_until(|| !replayed.version().has_pending_tails()).await; + + wait_until(|| region.listener.index_builds_stopped() == 1).await; + let version = replayed.version(); + let files = &version.ssts.levels()[0].files; + let has_index = files.values().any(|file| file.meta_ref().exists_index()); + assert_eq!(create_on_flush, has_index); +} + +#[tokio::test] +async fn test_stalled_manifest_check_ends_when_region_closes() { + let stall = Arc::new(AtomicBool::new(false)); + let stalled = Arc::new(AtomicUsize::new(0)); + let layer = stall_listings(is_manifest, stall.clone(), stalled.clone()); + let region = TestRegion::new("tail-replay-stalled-check", false, Some(layer)).await; + region.start_failing_remote_replay().await; + // A failed round checks the manifest for an update, which stalls. + stall.store(true, Ordering::Relaxed); + wait_until(|| stalled.load(Ordering::Relaxed) > 0).await; + let mito_region = region.engine.get_region(region.region_id).unwrap(); + // The stalled listing holds no manifest lock. + let write = tokio::time::timeout( + Duration::from_secs(5), + mito_region.manifest_ctx.manifest_manager.write(), + ) + .await + .unwrap(); + drop(write); + let replaying = Arc::downgrade(&mito_region); + drop(mito_region); + + let close = RegionRequest::Close(RegionCloseRequest::default()); + tokio::time::timeout( + Duration::from_secs(10), + region.engine.handle_request(region.region_id, close), + ) + .await + .unwrap() + .unwrap(); + + wait_until(|| replaying.strong_count() == 0).await; +} + +/// Parks the first SST write once `stall` is set, and counts the parked writes. +fn stall_first_sst_write(stall: Arc, stalled: Arc) -> MockLayer { + MockLayerBuilder::default() + .writer_factory(Arc::new(move |path, _args, inner| { + if path.ends_with(".parquet") && stall.swap(false, Ordering::Relaxed) { + stalled.fetch_add(1, Ordering::Relaxed); + Box::new(StalledWriter) + } else { + inner + } + })) + .build() + .unwrap() +} + +struct StalledWriter; + +impl Write for StalledWriter { + async fn write(&mut self, _bs: Buffer) -> MockResult<()> { + std::future::pending().await + } + + async fn close(&mut self) -> MockResult { + std::future::pending().await + } + + async fn abort(&mut self) -> MockResult<()> { + Ok(()) + } +} + +#[tokio::test] +async fn test_flush_runs_while_replay_writes_are_saturated() { + let stall = Arc::new(AtomicBool::new(false)); + let stalled = Arc::new(AtomicUsize::new(0)); + let layer = stall_first_sst_write(stall.clone(), stalled.clone()); + // One replay SST write takes every replay write permit. + let config = MitoConfig { + max_background_flushes: 1, + ..Default::default() + }; + let region = TestRegion::with_config("tail-replay-saturated", Some(layer), config).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + stall.store(true, Ordering::Relaxed); + region.catchup().await; + wait_until(|| stalled.load(Ordering::Relaxed) == 1).await; + + // The memtable holds rows `[0, 6)`, which the reopen replayed from this node's WAL. + tokio::time::timeout( + Duration::from_secs(10), + flush_region(®ion.engine, region.region_id, None), + ) + .await + .unwrap(); + + assert_eq!(1, region.stored_ssts().await); + assert!(region.has_pending_tails()); +} + +#[tokio::test] +async fn test_replay_waiting_to_fence_ends_when_demoted() { + let region = TestRegion::new("tail-replay-fence-wait", false, None).await; + region.start_failing_remote_replay().await; + wait_until(|| region.listener.failed()).await; + + // The next failed round finds the newer manifest and waits for the write lock behind + // this read guard, unless the replay is cancelled. + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let version = next_manifest_version(&mito_region).await; + let guard = mito_region.manifest_ctx.manifest_manager.read().await; + publish_delta( + ®ion.env.get_object_store().unwrap(), + ®ion.region_dir(), + version, + encode_delta(nop_action()).await, + ) + .await; + wait_until(|| region.listener.fence_begins() == 1).await; + // The worker cancels the replay before the downgrade waits for the manifest lock. + let engine = region.engine.clone(); + let region_id = region.region_id; + let downgrade = tokio::spawn(async move { + engine + .set_region_role_state_gracefully(region_id, SettableRegionRoleState::Follower) + .await + }); + + wait_until(|| region.listener.ended() == 1).await; + drop(guard); + downgrade.await.unwrap().unwrap(); + + assert!(mito_region.is_follower()); +} + +#[rstest] +#[case::fences(false, false)] +#[case::version_changed(true, false)] +#[case::not_writable(false, true)] +#[tokio::test] +async fn test_fence_for_reload_rechecks_version_and_role( + #[case] version_changed: bool, + #[case] demoted: bool, +) { + let region = TestRegion::new("tail-replay-fence-recheck", false, None).await; + if demoted { + region + .engine + .set_region_role(region.region_id, RegionRole::Follower) + .unwrap(); + } + let mito_region = region.engine.get_region(region.region_id).unwrap(); + let manifest_ctx = &mito_region.manifest_ctx; + let mut manager = manifest_ctx.manifest_manager.write().await; + let installed = manager.manifest().manifest_version + u64::from(version_changed); + + let fenced = manifest_ctx + .fence_for_reload_if_unchanged(&mut manager, region.region_id, installed) + .await; + + let expected = !version_changed && !demoted; + assert_eq!(expected, fenced); + assert_eq!(expected, manifest_ctx.is_reload_pending()); +} + +#[rstest] +#[case::set_role(false)] +#[case::set_role_with_lease(true)] +#[tokio::test] +async fn test_direct_demotion_ends_stalled_replay(#[case] with_lease: bool) { + let stall = Arc::new(AtomicBool::new(false)); + let stalled = Arc::new(AtomicUsize::new(0)); + let layer = stall_listings(is_remote_wal, stall.clone(), stalled.clone()); + let region = + TestRegion::with_wal_layer("tail-replay-direct-demotion", false, None, Some(layer)).await; + let last = region.write_remote_rows(region.region_id, 6, 9).await; + region + .reopen_with_tails(vec![remote_tail(0, last, UptoKind::Reported)]) + .await; + stall.store(true, Ordering::Relaxed); + region.catchup().await; + wait_until(|| stalled.load(Ordering::Relaxed) > 0).await; + + if with_lease { + let now = Instant::now(); + region + .engine + .set_region_role_with_lease( + region.region_id, + RegionRole::Follower, + now, + now + Duration::from_secs(10), + None, + ) + .unwrap(); + } else { + region + .engine + .set_region_role(region.region_id, RegionRole::Follower) + .unwrap(); + } + wait_until(|| region.listener.ended() == 1).await; + + // A later promotion replays the tails again. + stall.store(false, Ordering::Relaxed); + region.promote(); + region.wait_until_replayed().await; + assert_eq!(9, count_rows(®ion.engine, region.region_id).await); +} + +#[tokio::test] +async fn test_late_demotion_notice_of_replaced_instance_keeps_replay() { + let region = TestRegion::new("tail-replay-stale-notice", false, None).await; + region + .reopen_with_tails(vec![own_tail(0, region.entries[1], UptoKind::Reported)]) + .await; + let old = region.engine.get_region(region.region_id).unwrap(); + region.replace_follower().await; + region.listener.hold_from(1); + region.promote(); + wait_until(|| region.listener.commits() == 1).await; + + // A demotion notice of the replaced instance arrives late. + let request = WorkerRequest::Background { + region_id: region.region_id, + notify: BackgroundNotify::RegionCannotReplay { region: old }, + }; + region + .engine + .inner + .workers + .submit_to_worker(region.region_id, request) + .await + .unwrap(); + // The worker handles the notice before this write. + region.overwrite_remote_row().await; + region.listener.hold_from(0); + region.wait_until_replayed().await; + + assert_eq!(1, region.listener.commits()); +} + +#[tokio::test] +async fn test_failed_replay_reloads_region_after_failed_commit() { + let faults = DeltaFaults::default(); + let region = TestRegion::new( + "tail-replay-failed-commit-reload", + false, + Some(faults.layer()), + ) + .await; + region.start_failing_remote_replay().await; + wait_until(|| region.listener.failed()).await; + + // A failed flush commit takes the next version number without installing it. + let old = region.engine.get_region(region.region_id).unwrap(); + let next = next_manifest_version(&old).await; + faults.fail_next_write(); + let flush = RegionRequest::Flush(RegionFlushRequest::default()); + assert!( + region + .engine + .handle_request(region.region_id, flush) + .await + .is_err() + ); + assert_eq!( + next, + old.manifest_ctx + .manifest_manager + .read() + .await + .last_version() + ); + + // Another writer then publishes that version, clearing the tails. + let action_list = + RegionMetaActionList::with_action(RegionMetaAction::Edit(pending_tails_edit(vec![]))); + publish_delta( + ®ion.env.get_object_store().unwrap(), + ®ion.region_dir(), + next, + encode_delta(action_list).await, + ) + .await; + wait_until(|| is_replaced(®ion.engine, &old)).await; + + assert!(!region.has_pending_tails()); +} diff --git a/src/mito2/src/engine/truncate_test.rs b/src/mito2/src/engine/truncate_test.rs index 856644ca12d..8f138ddee67 100644 --- a/src/mito2/src/engine/truncate_test.rs +++ b/src/mito2/src/engine/truncate_test.rs @@ -32,7 +32,7 @@ use tokio::sync::Notify; use super::ScanRequest; use crate::config::MitoConfig; -use crate::engine::listener::{EventListener, FlushCancellationListener}; +use crate::engine::listener::{CountingStallListener, EventListener, FlushCancellationListener}; use crate::test_util::{ CreateRequestBuilder, MockWriteBufferManager, TestEnv, build_rows, put_rows, rows_schema, }; @@ -75,32 +75,6 @@ impl EventListener for FlushCommitListener { } } -#[derive(Default)] -struct CountingStallListener { - count: AtomicUsize, - notify: Notify, -} - -impl CountingStallListener { - async fn wait_for_count(&self, expected: usize) { - loop { - let notified = self.notify.notified(); - if self.count.load(Ordering::Relaxed) >= expected { - return; - } - notified.await; - } - } -} - -#[async_trait::async_trait] -impl EventListener for CountingStallListener { - fn on_write_stall(&self) { - self.count.fetch_add(1, Ordering::Relaxed); - self.notify.notify_waiters(); - } -} - #[tokio::test] async fn test_engine_truncate_region_basic() { test_engine_truncate_region_basic_with_format(false).await; diff --git a/src/mito2/src/error.rs b/src/mito2/src/error.rs index 7aeda169df0..43853bb565c 100644 --- a/src/mito2/src/error.rs +++ b/src/mito2/src/error.rs @@ -516,6 +516,15 @@ pub enum Error { source: BoxedError, }, + #[snafu(display("Failed to read the WAL of node {} for region {}", node_id, region_id))] + ReadNodeWal { + region_id: RegionId, + node_id: u64, + #[snafu(implicit)] + location: Location, + source: BoxedError, + }, + #[snafu(display("Failed to decode WAL entry, region_id: {}", region_id))] DecodeWal { region_id: RegionId, @@ -751,6 +760,32 @@ pub enum Error { location: Location, }, + #[snafu(display( + "Region {} cannot read pending WAL tails of node {}", + region_id, + node_id + ))] + RemoteWalUnsupported { + region_id: RegionId, + node_id: u64, + #[snafu(implicit)] + location: Location, + }, + + #[snafu(display( + "Region {} cannot read entry {} of the WAL of node {}, where its pending tail ends", + region_id, + entry_id, + node_id + ))] + TailEntryMissing { + region_id: RegionId, + node_id: u64, + entry_id: u64, + #[snafu(implicit)] + location: Location, + }, + #[snafu(display("Region {} does not hold a valid leader lease", region_id))] RegionLeaseExpired { region_id: RegionId, @@ -784,6 +819,24 @@ pub enum Error { location: Location, }, + #[snafu(display( + "Failed to end staging of region {}, which is still staging", + region_id + ))] + EndStaging { + region_id: RegionId, + source: Arc, + #[snafu(implicit)] + location: Location, + }, + + #[snafu(display("Region {} waits for a retry of its end-of-staging commit", region_id))] + EndStagingPending { + region_id: RegionId, + #[snafu(implicit)] + location: Location, + }, + #[snafu(display( "Failed to compat readers for region {}, reason: {}", region_id, @@ -1353,6 +1406,9 @@ pub enum Error { #[snafu(display("Compaction is cancelled."))] CompactionCancelled {}, + #[snafu(display("SST write is cancelled."))] + SstWriteCancelled {}, + #[snafu(display("Flush is cancelled."))] FlushCancelled {}, @@ -1572,6 +1628,7 @@ impl Error { | Error::RegionTruncated { .. } | Error::RegionStopped { .. } | Error::RegionLeaseExpired { .. } + | Error::EndStagingPending { .. } | Error::SerdeJson { .. } | Error::CompressObject { .. } => false, Error::OpenDal { error, .. } => !matches!( @@ -1611,6 +1668,7 @@ impl ErrorExt for Error { | ReadParquet { .. } => StatusCode::StorageUnavailable, WriteWal { source, .. } | ReadWal { source, .. } + | ReadNodeWal { source, .. } | DeleteWal { source, .. } | WaitWalDurable { source, .. } | CheckWalHistory { source, .. } => source.status_code(), @@ -1711,12 +1769,14 @@ impl ErrorExt for Error { RegionTruncated { .. } => StatusCode::Cancelled, RejectWrite { .. } | RejectWriteInStaging { .. } => StatusCode::StorageUnavailable, CompactRegion { source, .. } => source.status_code(), - EditRegion { source, .. } => source.status_code(), + EditRegion { source, .. } | EndStaging { source, .. } => source.status_code(), CompatReader { .. } => StatusCode::Unexpected, InvalidRegionRequest { source, .. } => source.status_code(), RegionState { .. } | UpdateManifest { .. } => StatusCode::RegionNotReady, RegionLeaseExpired { .. } | UnacknowledgedWrite { .. } => StatusCode::RegionNotReady, RegionPendingTails { .. } => StatusCode::RegionNotReady, + RemoteWalUnsupported { .. } => StatusCode::Unsupported, + TailEntryMissing { .. } => StatusCode::StorageUnavailable, JsonOptions { .. } => StatusCode::InvalidArguments, EmptyRegionDir { .. } | EmptyManifestDir { .. } => StatusCode::RegionNotFound, ConvertValue { source, .. } => source.status_code(), @@ -1759,7 +1819,7 @@ impl ErrorExt for Error { | StatsNotPresent { .. } | InvalidPrimaryKeyRange { .. } | DecodePrimaryKeyRange { .. } => StatusCode::Internal, - RegionBusy { .. } => StatusCode::RegionBusy, + RegionBusy { .. } | EndStagingPending { .. } => StatusCode::RegionBusy, GetSchemaMetadata { source, .. } => source.status_code(), Timeout { .. } => StatusCode::Cancelled, @@ -1769,9 +1829,10 @@ impl ErrorExt for Error { source.status_code() } - ManualCompactionOverride {} | CompactionCancelled {} | FlushCancelled {} => { - StatusCode::Cancelled - } + ManualCompactionOverride {} + | CompactionCancelled {} + | FlushCancelled {} + | SstWriteCancelled {} => StatusCode::Cancelled, // A concurrent manual compaction fails fast instead of being queued; // the conflict is reported to the caller, which decides whether to @@ -1832,6 +1893,8 @@ impl ErrorExt for Error { | RegionStopped { .. } | RegionBusy { .. } | RegionPendingTails { .. } + | EndStaging { .. } + | EndStagingPending { .. } | ManualCompactionAlreadyRunning { .. } | FlushableRegionState { .. } | ManifestDeltaNotFound { .. } @@ -1845,6 +1908,7 @@ impl ErrorExt for Error { WriteWal { source, .. } | ReadWal { source, .. } + | ReadNodeWal { source, .. } | DeleteWal { source, .. } | WaitWalDurable { source, .. } | CheckWalHistory { source, .. } @@ -2056,6 +2120,14 @@ mod tests { } .build(); assert!(!error.may_have_persisted_manifest_update()); + // Staging that waits for a retried end commit refuses before storage I/O. + let error = EndStagingPendingSnafu { region_id }.build(); + assert!(!error.may_have_persisted_manifest_update()); + assert!(error.retry_hint().is_retryable()); + assert_eq!(error.status_code(), StatusCode::RegionBusy); + // Other busy errors keep the conservative classification. + let error = RegionBusySnafu { region_id }.build(); + assert!(error.may_have_persisted_manifest_update()); // A commit refused before its PUT because the Leader deadline has passed. let error = RegionLeaseExpiredSnafu { region_id }.build(); diff --git a/src/mito2/src/flush.rs b/src/mito2/src/flush.rs index 2951d109d54..1a2dd13ad4f 100644 --- a/src/mito2/src/flush.rs +++ b/src/mito2/src/flush.rs @@ -21,10 +21,10 @@ use std::sync::{Arc, Mutex}; use std::time::Instant; use bytes::Bytes; -use common_base::cancellation::CancellableFuture; use common_telemetry::{debug, error, info}; use datatypes::arrow::datatypes::SchemaRef; use partition::expr::PartitionExpr; +use prometheus::{Histogram, IntCounter}; use smallvec::{SmallVec, smallvec}; use snafu::{ResultExt, ensure}; use store_api::metadata::RegionMetadataRef; @@ -40,7 +40,7 @@ use crate::access_layer::{ use crate::cache::CacheManagerRef; use crate::compaction::{collect_json2_rewrite_plans, rewrite_json2_batch, rewrite_json2_schema}; use crate::config::{IndexBuildMode, MitoConfig}; -use crate::engine::region_hook::SstFileInfo; +use crate::engine::region_hook::{RegionHookRef, SstFileInfo}; use crate::error::{ Error, FlushCancelledSnafu, FlushRegionSnafu, JoinSnafu, RegionBusySnafu, RegionClosedSnafu, RegionDroppedSnafu, RegionTruncatedSnafu, Result, UnexpectedSnafu, @@ -53,14 +53,14 @@ use crate::memtable::{ }; use crate::metrics::{ FLUSH_BYTES_TOTAL, FLUSH_ELAPSED, FLUSH_FAILURE_TOTAL, FLUSH_FILE_TOTAL, FLUSH_REQUESTS_TOTAL, - INFLIGHT_FLUSH_COUNT, + INFLIGHT_FLUSH_COUNT, TAIL_REPLAY_BYTES_TOTAL, TAIL_REPLAY_ELAPSED, TAIL_REPLAY_FILE_TOTAL, }; use crate::read::FlatSource; use crate::read::flat_dedup::{FlatDedupIterator, FlatLastNonNull, FlatLastRow}; use crate::read::flat_merge::FlatMergeIterator; use crate::region::options::{IndexOptions, MergeMode, RegionOptions}; use crate::region::version::{VersionControlData, VersionControlRef, VersionRef}; -use crate::region::{ManifestContextRef, RegionLeaderState, parse_partition_expr}; +use crate::region::{ManifestContextRef, MitoRegion, RegionLeaderState, parse_partition_expr}; use crate::request::{ BackgroundNotify, DdlRequest, FlushFailed, FlushFinished, OnFailure, OptionOutputTx, OutputTx, SenderBulkRequest, SenderDdlRequest, SenderWriteRequest, WorkerRequest, WorkerRequestWithTime, @@ -271,24 +271,53 @@ pub(crate) struct RegionFlushTask { pub(crate) senders: Vec, /// Request sender to notify the worker. pub(crate) request_sender: mpsc::Sender, - - pub(crate) access_layer: AccessLayerRef, pub(crate) listener: WorkerListener, - pub(crate) engine_config: Arc, - pub(crate) row_group_size: Option, - pub(crate) cache_manager: CacheManagerRef, pub(crate) manifest_ctx: ManifestContextRef, + /// Writes the immutable memtables to SSTs. + pub(crate) sst_writer: MemtableSstWriter, + /// Waits until the WAL is durable through the entry id the flush records. + pub(crate) durability_barrier: DurabilityBarrier, +} + +/// Writes the immutable memtables of a version to level 0 SSTs, for a flush or a tail replay. +pub(crate) struct MemtableSstWriter { + region_id: RegionId, + /// Kind of the writes, which selects their metrics. + kind: MemtableWriteKind, + access_layer: AccessLayerRef, + engine_config: Arc, + row_group_size: Option, + cache_manager: CacheManagerRef, + /// Hook told of the written SSTs. + hook: Option, /// Index options for the region. - pub(crate) index_options: IndexOptions, - /// Semaphore to control flush concurrency. - pub(crate) flush_semaphore: Arc, + index_options: IndexOptions, + /// Semaphore to control concurrent SST writes. + write_semaphore: Arc, /// Partition expression of the region. /// /// This is used to generate the file meta. - pub(crate) partition_expr: Option, - /// Waits until the WAL is durable through the entry id the flush records. - pub(crate) durability_barrier: DurabilityBarrier, + partition_expr: Option, +} + +/// Kind of the writes of a [MemtableSstWriter]. +#[derive(Clone, Copy)] +enum MemtableWriteKind { + /// Writes of a flush, for the reason its log names. + Flush(FlushReason), + /// Writes of the replay of pending WAL tails. + TailReplay, +} + +impl MemtableWriteKind { + /// Returns the [WriteType] of the writes' metrics. + fn write_type(self) -> WriteType { + match self { + MemtableWriteKind::Flush(_) => WriteType::Flush, + MemtableWriteKind::TailReplay => WriteType::TailReplay, + } + } } struct FlushTaskWaiters { @@ -383,8 +412,8 @@ impl RegionFlushTask { let timer = FLUSH_ELAPSED.with_label_values(&["total"]).start_timer(); let uncommitted = UncommittedSsts::new( self.region_id, - self.access_layer.clone(), - Some(self.cache_manager.clone()), + self.sst_writer.access_layer.clone(), + Some(self.sst_writer.cache_manager.clone()), ); self.listener.on_flush_begin(self.region_id).await; let flush_result = if state.is_cancelled() { @@ -420,7 +449,10 @@ impl RegionFlushTask { } Err(e) => { let err = Arc::new(e); - let failed = FlushFailed { err: err.clone() }; + let failed = FlushFailed { + err: err.clone(), + senders: std::mem::take(&mut self.senders), + }; if failed.is_cancelled() { info!("Flush cancelled for region {}", self.region_id); } else { @@ -430,7 +462,6 @@ impl RegionFlushTask { timer.stop_and_discard(); uncommitted.cleanup().await; - self.on_failure(err.clone()); WorkerRequest::Background { region_id: self.region_id, notify: BackgroundNotify::FlushFailed(failed), @@ -450,80 +481,20 @@ impl RegionFlushTask { ) -> Result { // We must use the immutable memtables list and entry ids from the `version_data` // for consistency as others might already modify the version in the `version_control`. - let version = &version_data.version; - let timer = FLUSH_ELAPSED - .with_label_values(&["flush_memtables"]) - .start_timer(); - - let mut write_opts = WriteOptions { - write_buffer_size: self.engine_config.sst_write_buffer_size, - float_field_encoding: version.options.float_field_encoding, - ..Default::default() - }; - if let Some(row_group_size) = self.row_group_size { - write_opts.row_group_size = row_group_size; - } - - let DoFlushMemtablesResult { - file_metas, - flushed_bytes, - series_count, - encoded_part_count, - flush_metrics, - sst_infos, - } = self - .do_flush_memtables(version, write_opts, state, uncommitted) + let file_metas = self + .sst_writer + .write_ssts(&version_data.version, state, uncommitted) .await?; - if !file_metas.is_empty() { - FLUSH_BYTES_TOTAL.inc_by(flushed_bytes); - } - - let mut file_ids = Vec::with_capacity(file_metas.len()); - let mut total_rows = 0; - let mut total_bytes = 0; - for meta in &file_metas { - file_ids.push(meta.file_id); - total_rows += meta.num_rows; - total_bytes += meta.file_size; - } - info!( - "Successfully flush memtables, region: {}, reason: {}, files: {:?}, series count: {}, total_rows: {}, total_bytes: {}, cost: {:?}, encoded_part_count: {}, metrics: {:?}", - self.region_id, - self.reason.as_str(), - file_ids, - series_count, - total_rows, - total_bytes, - timer.stop_and_record(), - encoded_part_count, - flush_metrics, - ); - flush_metrics.observe(); - - let hook = self.manifest_ctx.hook(); - if let Some(hook) = &hook { - let files: Vec> = sst_infos - .iter() - .zip(file_metas.iter()) - .map(|(sst_info, file_meta)| SstFileInfo { - sst_info_ref: sst_info, - file_meta, - }) - .collect(); - hook.on_sst_files_written(self.region_id, &version.metadata, &files) - .await; - } - // The manifest may name only a durable entry as flushed. A log store // that acknowledges appends before their entries are durable can have // handed out `last_entry_id` for an entry that is still in its backlog. // A DDL that cancels the flush must not wait behind that upload. - let durable = self.durability_barrier.wait(version_data.last_entry_id); - tokio::pin!(durable); - match CancellableFuture::new(durable.as_mut(), state.cancel_handle()).await { - Ok(result) => result?, - Err(_) => return FlushCancelledSnafu.fail(), + let cancellation = state.cancel_token(); + tokio::select! { + biased; + _ = cancellation.cancelled() => return FlushCancelledSnafu.fail(), + result = self.durability_barrier.wait(version_data.last_entry_id) => result?, } let edit = RegionEdit { @@ -583,6 +554,193 @@ impl RegionFlushTask { Ok(edit) } + /// Notify flush job status. + pub(crate) async fn send_worker_request(&self, request: WorkerRequest) { + if let Err(e) = self + .request_sender + .send(WorkerRequestWithTime::new(request)) + .await + { + let request = e.0.request; + error!( + "Failed to notify flush job status for region {}, request: {:?}", + self.region_id, request + ); + if let WorkerRequest::Background { notify, .. } = request { + let err = RegionClosedSnafu { + region_id: self.region_id, + } + .build(); + match notify { + BackgroundNotify::FlushFinished(mut finished) => { + finished.on_failure(err); + } + BackgroundNotify::FlushFailed(failed) => { + let err = Arc::new(err); + for sender in failed.senders { + sender.send(Err(err.clone()).context(FlushRegionSnafu { + region_id: self.region_id, + })); + } + } + _ => {} + } + } + } + } + + /// Merge two flush tasks. + fn merge(&mut self, mut other: RegionFlushTask) { + assert_eq!(self.region_id, other.region_id); + // Now we only merge senders. They share the same flush reason. + self.senders.append(&mut other.senders); + } +} + +impl MemtableSstWriter { + /// Creates a writer of the `region`'s immutable memtables for a flush with `reason`. + pub(crate) fn new_flush( + region: &MitoRegion, + reason: FlushReason, + row_group_size: Option, + engine_config: Arc, + cache_manager: CacheManagerRef, + write_semaphore: Arc, + ) -> Self { + Self::new( + MemtableWriteKind::Flush(reason), + region, + row_group_size, + engine_config, + cache_manager, + write_semaphore, + ) + } + + /// Creates a writer of the `region`'s immutable memtables for a tail replay. Only + /// [TailReplayLimits::new_sst_writer](crate::tail_replay::TailReplayLimits::new_sst_writer) + /// calls it, with the replays' write semaphore. + pub(crate) fn new_tail_replay( + region: &MitoRegion, + engine_config: Arc, + cache_manager: CacheManagerRef, + write_semaphore: Arc, + ) -> Self { + Self::new( + MemtableWriteKind::TailReplay, + region, + None, + engine_config, + cache_manager, + write_semaphore, + ) + } + + fn new( + kind: MemtableWriteKind, + region: &MitoRegion, + row_group_size: Option, + engine_config: Arc, + cache_manager: CacheManagerRef, + write_semaphore: Arc, + ) -> Self { + Self { + region_id: region.region_id, + kind, + access_layer: region.access_layer.clone(), + engine_config, + // The request-provided size (e.g. manual flush) takes precedence over the + // region option. + row_group_size: row_group_size.or(region.version().options.max_row_group_row_count), + cache_manager, + hook: region.manifest_ctx.hook(), + index_options: region.version().options.index_options.clone(), + write_semaphore, + partition_expr: region.metadata().partition_expr.clone(), + } + } + + /// Returns the cache manager the written SSTs go through. + pub(crate) fn cache_manager(&self) -> &CacheManagerRef { + &self.cache_manager + } + + /// Writes the immutable memtables of `version` to level 0 SSTs, which `uncommitted` + /// tracks until a manifest edit references them, and returns their file metas. Its log + /// names the flush reason, or the write type when the write is not a flush. + pub(crate) async fn write_ssts( + &self, + version: &VersionRef, + state: &CancellableTaskState, + uncommitted: &UncommittedSsts, + ) -> Result> { + let timer = self.write_elapsed().start_timer(); + + let mut write_opts = WriteOptions { + write_buffer_size: self.engine_config.sst_write_buffer_size, + float_field_encoding: version.options.float_field_encoding, + ..Default::default() + }; + if let Some(row_group_size) = self.row_group_size { + write_opts.row_group_size = row_group_size; + } + + let DoFlushMemtablesResult { + file_metas, + flushed_bytes, + series_count, + encoded_part_count, + flush_metrics, + sst_infos, + } = self + .do_flush_memtables(version, write_opts, state, uncommitted) + .await?; + + if !file_metas.is_empty() { + self.bytes_total().inc_by(flushed_bytes); + } + + let mut file_ids = Vec::with_capacity(file_metas.len()); + let mut total_rows = 0; + let mut total_bytes = 0; + for meta in &file_metas { + file_ids.push(meta.file_id); + total_rows += meta.num_rows; + total_bytes += meta.file_size; + } + info!( + "Successfully flush memtables, region: {}, reason: {}, files: {:?}, series count: {}, total_rows: {}, total_bytes: {}, cost: {:?}, encoded_part_count: {}, metrics: {:?}", + self.region_id, + match self.kind { + MemtableWriteKind::Flush(reason) => reason.as_str(), + MemtableWriteKind::TailReplay => WriteType::TailReplay.into(), + }, + file_ids, + series_count, + total_rows, + total_bytes, + timer.stop_and_record(), + encoded_part_count, + flush_metrics, + ); + flush_metrics.observe(); + + if let Some(hook) = &self.hook { + let files: Vec> = sst_infos + .iter() + .zip(file_metas.iter()) + .map(|(sst_info, file_meta)| SstFileInfo { + sst_info_ref: sst_info, + file_meta, + }) + .collect(); + hook.on_sst_files_written(self.region_id, &version.metadata, &files) + .await; + } + + Ok(file_metas) + } + async fn do_flush_memtables( &self, version: &VersionRef, @@ -595,9 +753,8 @@ impl RegionFlushTask { let mut flushed_bytes = 0; let mut series_count = 0; let mut encoded_part_count = 0; - let mut flush_metrics = Metrics::new(WriteType::Flush); + let mut flush_metrics = Metrics::new(self.kind.write_type()); let partition_expr = parse_partition_expr(self.partition_expr.as_deref())?; - let hook = self.manifest_ctx.hook(); let mut all_sst_infos = Vec::new(); for mem in memtables { if mem.is_empty() { @@ -673,7 +830,7 @@ impl RegionFlushTask { version.options.preserve_row_sequence, )?); } - if hook.is_some() { + if self.hook.is_some() { all_sst_infos.extend(ssts_written); } } @@ -722,22 +879,24 @@ impl RegionFlushTask { &version.options, field_column_start, )?; + let file_total = self.file_total(); let mut tasks = Vec::with_capacity(flat_sources.encoded.len() + flat_sources.sources.len()); let num_encoded = flat_sources.encoded.len(); for (source, max_sequence) in flat_sources.sources { let write_request = self.new_write_request(version, max_sequence, source); let access_layer = self.access_layer.clone(); let write_opts = write_opts.clone(); - let semaphore = self.flush_semaphore.clone(); + let semaphore = self.write_semaphore.clone(); let uncommitted = uncommitted.clone(); + let write_type = self.kind.write_type(); let task = common_runtime::spawn_global(async move { let _permit = semaphore.acquire().await.unwrap(); - let mut metrics = Metrics::new(WriteType::Flush); + let mut metrics = Metrics::new(write_type); let ssts = access_layer .write_sst(write_request, &write_opts, &mut metrics) .await?; uncommitted.track(&ssts); - FLUSH_FILE_TOTAL.inc_by(ssts.len() as u64); + file_total.inc_by(ssts.len() as u64); Ok((max_sequence, ssts, metrics)) }); tasks.push(task); @@ -746,10 +905,11 @@ impl RegionFlushTask { for (mut encoded, max_sequence) in flat_sources.encoded { let access_layer = self.access_layer.clone(); let cache_manager = self.cache_manager.clone(); - let region_id = version.metadata.region_id; + let region_id = self.region_id; let write_buffer_size = write_opts.write_buffer_size; - let semaphore = self.flush_semaphore.clone(); + let semaphore = self.write_semaphore.clone(); let uncommitted = uncommitted.clone(); + let write_type = self.kind.write_type(); let index_build_params = build_index.then(|| EncodedSstIndexBuildParams { metadata: version.metadata.clone(), index_options: self.index_options.clone(), @@ -762,7 +922,8 @@ impl RegionFlushTask { uncommitted.track_before_upload(encoded.sst_info.file_id, index_build_params.is_some()); let task = common_runtime::spawn_global(async move { let _permit = semaphore.acquire().await.unwrap(); - let metrics = access_layer + let mut metrics = Metrics::new(write_type); + access_layer .put_sst( &encoded.data, region_id, @@ -770,10 +931,11 @@ impl RegionFlushTask { &cache_manager, write_buffer_size, index_build_params, + &mut metrics, ) .await?; uncommitted.track(std::slice::from_ref(&encoded.sst_info)); - FLUSH_FILE_TOTAL.inc(); + file_total.inc(); Ok((max_sequence, smallvec![encoded.sst_info], metrics)) }); tasks.push(task); @@ -785,12 +947,17 @@ impl RegionFlushTask { .collect::>(); let join_all = futures::future::join_all(tasks); tokio::pin!(join_all); - let results = match CancellableFuture::new(join_all.as_mut(), state.cancel_handle()).await { - Ok(results) => results + let cancellation = state.cancel_token(); + let results = match tokio::select! { + biased; + _ = cancellation.cancelled() => None, + results = join_all.as_mut() => Some(results), + } { + Some(results) => results .into_iter() .map(|result| result.context(JoinSnafu)) .collect::>>()?, - Err(_) => { + None => { for handle in abort_handles { handle.abort(); } @@ -833,7 +1000,7 @@ impl RegionFlushTask { num_rows: sst_info.num_rows as u64, num_row_groups: sst_info.num_row_groups, sequence: NonZeroU64::new(max_sequence), - partition_expr, + partition_expr: partition_expr.map(Into::into), num_series: sst_info.num_series, primary_key_min, primary_key_max, @@ -876,38 +1043,30 @@ impl RegionFlushTask { } } - /// Notify flush job status. - pub(crate) async fn send_worker_request(&self, request: WorkerRequest) { - if let Err(e) = self - .request_sender - .send(WorkerRequestWithTime::new(request)) - .await - { - let request = e.0.request; - error!( - "Failed to notify flush job status for region {}, request: {:?}", - self.region_id, request - ); - if let WorkerRequest::Background { - notify: BackgroundNotify::FlushFinished(mut finished), - .. - } = request - { - finished.on_failure( - RegionClosedSnafu { - region_id: self.region_id, - } - .build(), - ); + /// Returns the histogram that times [Self::write_ssts] for this writer's kind. + fn write_elapsed(&self) -> Histogram { + match self.kind { + MemtableWriteKind::TailReplay => { + TAIL_REPLAY_ELAPSED.with_label_values(&["write_chunk"]) } + MemtableWriteKind::Flush(_) => FLUSH_ELAPSED.with_label_values(&["flush_memtables"]), } } - /// Merge two flush tasks. - fn merge(&mut self, mut other: RegionFlushTask) { - assert_eq!(self.region_id, other.region_id); - // Now we only merge senders. They share the same flush reason. - self.senders.append(&mut other.senders); + /// Returns the counter of SST bytes this writer writes. + fn bytes_total(&self) -> &'static IntCounter { + match self.kind { + MemtableWriteKind::TailReplay => &TAIL_REPLAY_BYTES_TOTAL, + MemtableWriteKind::Flush(_) => &FLUSH_BYTES_TOTAL, + } + } + + /// Returns the counter of SST files this writer writes. + fn file_total(&self) -> &'static IntCounter { + match self.kind { + MemtableWriteKind::TailReplay => &TAIL_REPLAY_FILE_TOTAL, + MemtableWriteKind::Flush(_) => &FLUSH_FILE_TOTAL, + } } } @@ -1290,13 +1449,13 @@ impl FlushScheduler { return Ok(()); } - let closing = task.reason == FlushReason::Closing; + let reason = task.reason; let state = self.schedule_flush_task(version_control, task)?; // Add this region to status map. let _ = self.region_status.insert( region_id, - FlushStatus::new(region_id, version_control.clone(), state, closing), + FlushStatus::new(region_id, version_control.clone(), state, reason), ); Ok(()) @@ -1488,13 +1647,13 @@ impl FlushScheduler { status.pending_bulk_writes.push(request); } - /// Whether an ALTER owns a turn while waiting for its memtables to flush. - pub(crate) fn has_pending_alter(&self, region_id: RegionId) -> bool { + /// Whether staging is waiting for its memtables to flush. + pub(crate) fn has_pending_enter_staging(&self, region_id: RegionId) -> bool { self.region_status.get(®ion_id).is_some_and(|status| { status .pending_ddls .iter() - .any(|ddl| matches!(ddl.request, DdlRequest::Alter(_))) + .any(|ddl| matches!(ddl.request, DdlRequest::EnterStaging(_))) }) } @@ -1550,14 +1709,14 @@ impl FlushStatus { region_id: RegionId, version_control: VersionControlRef, state: CancellableTaskState, - closing: bool, + reason: FlushReason, ) -> FlushStatus { FlushStatus { region_id, version_control, state, pending_task: None, - closing, + closing: reason == FlushReason::Closing, pending_ddls: Vec::new(), pending_writes: Vec::new(), pending_bulk_writes: Vec::new(), @@ -1703,15 +1862,20 @@ mod tests { reason, senders: Vec::new(), request_sender, - access_layer: env.access_layer.clone(), listener: WorkerListener::default(), - engine_config: Arc::new(MitoConfig::default()), - row_group_size: None, - cache_manager: Arc::new(CacheManager::default()), manifest_ctx, - index_options: IndexOptions::default(), - flush_semaphore: Arc::new(Semaphore::new(2)), - partition_expr: None, + sst_writer: MemtableSstWriter { + region_id, + kind: MemtableWriteKind::Flush(reason), + access_layer: env.access_layer.clone(), + engine_config: Arc::new(MitoConfig::default()), + row_group_size: None, + cache_manager: Arc::new(CacheManager::default()), + hook: None, + index_options: IndexOptions::default(), + write_semaphore: Arc::new(Semaphore::new(2)), + partition_expr: None, + }, durability_barrier: DurabilityBarrier::noop(), } } @@ -1846,24 +2010,16 @@ mod tests { let version_control = Arc::new(builder.build()); let (output_tx, output_rx) = oneshot::channel(); - let mut task = RegionFlushTask { - region_id: builder.region_id(), - reason: FlushReason::Manual, - senders: Vec::new(), - request_sender: tx, - access_layer: env.access_layer.clone(), - listener: WorkerListener::default(), - engine_config: Arc::new(MitoConfig::default()), - row_group_size: None, - cache_manager: Arc::new(CacheManager::default()), - manifest_ctx: env - .mock_manifest_context(version_control.current().version.metadata.clone()) - .await, - index_options: IndexOptions::default(), - flush_semaphore: Arc::new(Semaphore::new(2)), - partition_expr: None, - durability_barrier: DurabilityBarrier::noop(), - }; + let manifest_ctx = env + .mock_manifest_context(version_control.current().version.metadata.clone()) + .await; + let mut task = new_test_flush_task( + &env, + builder.region_id(), + FlushReason::Manual, + tx, + manifest_ctx, + ); task.push_sender(OptionOutputTx::from(output_tx)); scheduler .schedule_flush(builder.region_id(), &version_control, task) @@ -1956,6 +2112,82 @@ mod tests { assert!(output.is_err()); } + #[tokio::test] + async fn test_flush_failure_defers_waiter_notification_to_worker() { + let env = SchedulerEnv::new().await; + let (tx, mut rx) = mpsc::channel(1); + let builder = VersionControlBuilder::new(); + let region_id = builder.region_id(); + let version_control = builder.build(); + let version_data = version_control.current(); + let manifest_ctx = env + .mock_manifest_context(version_data.version.metadata.clone()) + .await; + let mut task = new_test_flush_task(&env, region_id, FlushReason::Closing, tx, manifest_ctx); + let (output_tx, mut output_rx) = oneshot::channel(); + task.push_sender(OptionOutputTx::from(output_tx)); + let state = CancellableTaskState::new(); + state.request_cancel(); + + // Cancellation exercises the job's failure path without depending on I/O or scheduling. + task.do_flush(version_data, state).await; + let request = rx.try_recv().unwrap().request; + assert!(matches!( + output_rx.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + let WorkerRequest::Background { + region_id: failed_region_id, + notify: BackgroundNotify::FlushFailed(failed), + } = request + else { + panic!("expected a flush failure notification"); + }; + assert_eq!(failed_region_id, region_id); + assert_eq!(failed.senders.len(), 1); + assert!(failed.is_cancelled()); + } + + #[tokio::test] + async fn test_send_worker_request_failure_notifies_flush_failed_waiter() { + let env = SchedulerEnv::new().await; + let (tx, rx) = mpsc::channel(1); + drop(rx); + let builder = VersionControlBuilder::new(); + let version_control = Arc::new(builder.build()); + let manifest_ctx = env + .mock_manifest_context(version_control.current().version.metadata.clone()) + .await; + let task = new_test_flush_task( + &env, + builder.region_id(), + FlushReason::Manual, + tx, + manifest_ctx, + ); + let (output_tx, output_rx) = oneshot::channel(); + let request = WorkerRequest::Background { + region_id: builder.region_id(), + notify: BackgroundNotify::FlushFailed(FlushFailed { + err: Arc::new( + RegionBusySnafu { + region_id: builder.region_id(), + } + .build(), + ), + senders: vec![OutputTx::new(output_tx)], + }), + }; + + task.send_worker_request(request).await; + + let err = output_rx + .await + .expect("waiter must receive explicit error") + .unwrap_err(); + assert_eq!(err.status_code(), StatusCode::Cancelled); + } + #[tokio::test] async fn test_flush_waiters_drop_notifies_waiter() { let region_id = RegionId::new(1, 1); @@ -2149,21 +2381,14 @@ mod tests { .await; // Creates 3 tasks. let mut tasks: Vec<_> = (0..3) - .map(|_| RegionFlushTask { - region_id: builder.region_id(), - reason: FlushReason::Manual, - senders: Vec::new(), - request_sender: tx.clone(), - access_layer: env.access_layer.clone(), - listener: WorkerListener::default(), - engine_config: Arc::new(MitoConfig::default()), - row_group_size: None, - cache_manager: Arc::new(CacheManager::default()), - manifest_ctx: manifest_ctx.clone(), - index_options: IndexOptions::default(), - flush_semaphore: Arc::new(Semaphore::new(2)), - partition_expr: None, - durability_barrier: DurabilityBarrier::noop(), + .map(|_| { + new_test_flush_task( + &env, + builder.region_id(), + FlushReason::Manual, + tx.clone(), + manifest_ctx.clone(), + ) }) .collect(); // Schedule first task. @@ -2456,21 +2681,14 @@ mod tests { .await; // Creates 2 tasks. let mut tasks: Vec<_> = (0..2) - .map(|_| RegionFlushTask { - region_id: builder.region_id(), - reason: FlushReason::Manual, - senders: Vec::new(), - request_sender: tx.clone(), - access_layer: env.access_layer.clone(), - listener: WorkerListener::default(), - engine_config: Arc::new(MitoConfig::default()), - row_group_size: None, - cache_manager: Arc::new(CacheManager::default()), - manifest_ctx: manifest_ctx.clone(), - index_options: IndexOptions::default(), - flush_semaphore: Arc::new(Semaphore::new(2)), - partition_expr: None, - durability_barrier: DurabilityBarrier::noop(), + .map(|_| { + new_test_flush_task( + &env, + builder.region_id(), + FlushReason::Manual, + tx.clone(), + manifest_ctx.clone(), + ) }) .collect(); // Schedule first task. diff --git a/src/mito2/src/lease.rs b/src/mito2/src/lease.rs index 95e076825eb..1544bd650ab 100644 --- a/src/mito2/src/lease.rs +++ b/src/mito2/src/lease.rs @@ -12,14 +12,17 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::sync::LazyLock; +use std::collections::HashMap; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::{LazyLock, Mutex}; use std::time::Instant; +use store_api::storage::RegionId; + /// The instant lease deadlines count from, so that a deadline fits in an [AtomicU64]. static LEASE_EPOCH: LazyLock = LazyLock::new(Instant::now); -fn lease_nanos(instant: Instant) -> u64 { +pub(crate) fn lease_nanos(instant: Instant) -> u64 { instant.saturating_duration_since(*LEASE_EPOCH).as_nanos() as u64 } @@ -89,3 +92,76 @@ impl NodeLease { .fetch_max(self.deadline.get(), Ordering::Relaxed); } } + +/// A region lease the metasrv granted for a heartbeat sent at `sent_at`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct GrantedLease { + pub(crate) candidate_lease_id: Option, + /// When the heartbeat request for which this lease was granted was sent. + pub(crate) sent_at: Instant, +} + +/// The lease of the latest heartbeat each region received, whether or not the region applied it. +/// It is kept apart from the region instance, so a reopen during catchup keeps it. +#[derive(Debug, Default)] +pub(crate) struct GrantedLeases(Mutex>); + +impl GrantedLeases { + /// Records the lease if `region_is_open` holds. The check runs under the same lock as + /// [Self::remove], which a worker calls after it removes the region, so a lease recorded + /// concurrently with the removal does not outlive it. + pub(crate) fn record( + &self, + region_id: RegionId, + sent_at: Instant, + candidate_lease_id: Option, + region_is_open: impl FnOnce() -> bool, + ) { + let lease = GrantedLease { + candidate_lease_id, + sent_at, + }; + let mut leases = self.0.lock().unwrap(); + if !region_is_open() { + return; + } + leases + .entry(region_id) + .and_modify(|latest| { + if sent_at > latest.sent_at { + *latest = lease; + } + }) + .or_insert(lease); + } + + pub(crate) fn remove(&self, region_id: RegionId) { + self.0.lock().unwrap().remove(®ion_id); + } + + #[cfg(any(test, feature = "test"))] + pub(crate) fn get(&self, region_id: RegionId) -> Option { + self.0.lock().unwrap().get(®ion_id).copied() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_record_checks_region_under_removal_lock() { + let leases = GrantedLeases::default(); + let region_id = RegionId::new(1, 1); + leases.record(region_id, Instant::now(), Some(1), || { + // A concurrent `remove` waits until the record is done. + assert!(leases.0.try_lock().is_err()); + true + }); + assert_eq!(leases.get(region_id).unwrap().candidate_lease_id, Some(1)); + + leases.remove(region_id); + leases.record(region_id, Instant::now(), Some(2), || false); + assert_eq!(leases.get(region_id), None); + } +} diff --git a/src/mito2/src/lib.rs b/src/mito2/src/lib.rs index 116e8ed5833..0ee0de24f3e 100644 --- a/src/mito2/src/lib.rs +++ b/src/mito2/src/lib.rs @@ -43,6 +43,7 @@ pub mod request; pub mod schedule; pub mod series_index; pub mod sst; +mod tail_replay; #[cfg(feature = "hdfs-object-store")] mod temp_file_sweeper; mod time_provider; diff --git a/src/mito2/src/manifest/action.rs b/src/mito2/src/manifest/action.rs index cd00237087a..03ed1cc5444 100644 --- a/src/mito2/src/manifest/action.rs +++ b/src/mito2/src/manifest/action.rs @@ -30,7 +30,8 @@ use crate::error::{RegionMetadataNotFoundSnafu, Result, SerdeJsonSnafu, Utf8Snaf use crate::manifest::manager::RemoveFileOptions; use crate::region::ManifestStats; use crate::sst::FormatType; -use crate::sst::file::{self, FileMeta}; +use crate::sst::file::{self, FileMeta, FileMetaRef}; +use crate::sst::file_metadata::FileMetadataInternerRef; use crate::wal::EntryId; /// Actions that can be applied to region manifest. @@ -217,7 +218,7 @@ pub struct RegionManifest { /// Metadata of the region. pub metadata: RegionMetadataRef, /// SST files. - pub files: HashMap, + pub files: HashMap, /// Removed files, which are not in the current manifest but may still be kept for a while. /// This is a list of (set of files, timestamp) pairs, where the timestamp is the time when /// the files are removed from manifest. The timestamp is in milliseconds since unix epoch. @@ -308,8 +309,9 @@ impl RegionManifest { #[derive(Debug, Default)] pub struct RegionManifestBuilder { + file_metadata_interner: FileMetadataInternerRef, metadata: Option, - files: HashMap, + files: HashMap, pub removed_files: RemovedFilesRecord, flushed_entry_id: EntryId, flushed_sequence: SequenceNumber, @@ -324,19 +326,43 @@ pub struct RegionManifestBuilder { } impl RegionManifestBuilder { + /// Creates an empty builder with the owner's metadata interner. + pub(crate) fn new(file_metadata_interner: FileMetadataInternerRef) -> Self { + Self { + file_metadata_interner, + ..Default::default() + } + } + + /// Canonicalizes independently decoded checkpoint entries before publication. + pub(crate) fn with_checkpoint( + checkpoint: Option, + file_metadata_interner: FileMetadataInternerRef, + ) -> Self { + let mut builder = Self::from_shared_manifest(checkpoint, file_metadata_interner); + for meta in builder.files.values_mut() { + builder.file_metadata_interner.intern_in_place(meta); + } + builder + } + fn removed_file(&self, file: &FileMeta) -> RemovedFile { let index_version = self .files .get(&file.file_id) - .and_then(FileMeta::index_version) + .and_then(|meta| meta.index_version()) .or_else(|| file.index_version()); RemovedFile::File(file.file_id, index_version) } - /// Start with a checkpoint. - pub fn with_checkpoint(checkpoint: Option) -> Self { + /// Reuses entries from a snapshot already canonicalized by the same owner. + pub(crate) fn from_shared_manifest( + checkpoint: Option, + file_metadata_interner: FileMetadataInternerRef, + ) -> Self { if let Some(s) = checkpoint { Self { + file_metadata_interner, metadata: Some(s.metadata), files: s.files, removed_files: s.removed_files, @@ -352,7 +378,7 @@ impl RegionManifestBuilder { handoff_floors: s.handoff_floors, } } else { - Default::default() + Self::new(file_metadata_interner) } } @@ -392,6 +418,8 @@ impl RegionManifestBuilder { let mut removed_files = vec![]; for file in edit.files_to_add { + let file = FileMetaRef::from(file); + let file = self.file_metadata_interner.intern(file); if let Some(old_file) = self.files.insert(file.file_id, file.clone()) && let Some(old_index) = old_file.index_version() && !old_file.is_index_up_to_date(&file) @@ -480,7 +508,7 @@ impl RegionManifestBuilder { } } - pub fn files(&self) -> &HashMap { + pub fn files(&self) -> &HashMap { &self.files } @@ -670,7 +698,7 @@ impl RegionCheckpoint { } } -#[derive(Serialize, Deserialize, Clone, Debug, PartialEq, Eq)] +#[derive(Serialize, Deserialize, Clone, Debug, Default, PartialEq, Eq)] pub struct RegionMetaActionList { pub actions: Vec, } @@ -765,6 +793,8 @@ mod tests { use common_time::Timestamp; + use crate::sst::file_metadata::MetadataRef; + use super::*; // These tests are used to ensure backward compatibility of manifest files. @@ -1560,6 +1590,56 @@ mod tests { builder } + #[test] + fn test_checkpoint_sharing_preserves_owned_allocations() { + let mut checkpoint = manifest_builder().try_build().unwrap(); + let expr = partition::expr::col("tag").lt(datatypes::value::Value::UInt32(10)); + let mut allocations = HashMap::new(); + for _ in 0..4 { + let meta = FileMetaRef::from(FileMeta { + region_id: RegionId::new(1, 1), + file_id: FileId::random(), + indexes: vec![crate::sst::file::ColumnIndexMetadata::default()], + partition_expr: Some(expr.clone().into()), + ..Default::default() + }); + allocations.insert( + meta.file_id, + (std::ptr::from_ref(meta.as_ref()), meta.indexes.as_ptr()), + ); + checkpoint.files.insert(meta.file_id, meta); + } + let interner = Arc::new(crate::sst::file_metadata::FileMetadataInterner::default()); + let manifest = RegionManifestBuilder::with_checkpoint(Some(checkpoint), interner.clone()) + .try_build() + .unwrap(); + let expression = manifest + .files + .values() + .next() + .unwrap() + .partition_expr + .as_ref() + .unwrap(); + for meta in manifest.files.values() { + assert_eq!( + allocations[&meta.file_id], + (std::ptr::from_ref(meta.as_ref()), meta.indexes.as_ptr()) + ); + assert!(MetadataRef::ptr_eq( + expression, + meta.partition_expr.as_ref().unwrap() + )); + } + let decoded = serde_json::from_slice(&serde_json::to_vec(&manifest).unwrap()).unwrap(); + let recovered = RegionManifestBuilder::with_checkpoint(Some(decoded), interner) + .try_build() + .unwrap(); + for (id, meta) in &manifest.files { + assert!(MetadataRef::ptr_eq(meta, &recovered.files[id])); + } + } + #[test] fn test_pending_tails_change_only_when_set() { let mut builder = manifest_builder(); diff --git a/src/mito2/src/manifest/checkpointer.rs b/src/mito2/src/manifest/checkpointer.rs index 3d33942a2f9..8f5a33c8c7c 100644 --- a/src/mito2/src/manifest/checkpointer.rs +++ b/src/mito2/src/manifest/checkpointer.rs @@ -22,6 +22,7 @@ use store_api::storage::RegionId; use store_api::{MIN_VERSION, ManifestVersion}; #[cfg(test)] use tokio::sync::Notify; +use tokio::sync::watch; use crate::error::Result; use crate::manifest::action::{RegionCheckpoint, RegionManifest}; @@ -34,11 +35,19 @@ use crate::metrics::MANIFEST_OP_ELAPSED; pub(crate) struct Checkpointer { manifest_options: RegionManifestOptions, inner: Arc, - checkpoint_task: Option>, + checkpoint_task: Option, #[cfg(test)] pending_checkpoint_wait_started: Arc, } +/// A spawned checkpoint task and the signal that it has ended. +#[derive(Debug)] +struct CheckpointTask { + handle: JoinHandle<()>, + /// Closes when the task ends, so that it can be awaited without borrowing the checkpointer. + done: watch::Receiver<()>, +} + #[derive(Debug)] struct Inner { region_id: RegionId, @@ -142,7 +151,7 @@ impl Checkpointer { if self .checkpoint_task .as_ref() - .is_some_and(|handle| !handle.is_finished()) + .is_some_and(|task| !task.handle.is_finished()) { return; } @@ -187,9 +196,36 @@ impl Checkpointer { fn do_checkpoint(&mut self, checkpoint: RegionCheckpoint) { let inner = self.inner.clone(); - self.checkpoint_task = Some(common_runtime::spawn_global(async move { + let (done_sender, done) = watch::channel(()); + let handle = common_runtime::spawn_global(async move { + // The sender is dropped however the task ends, which closes the channel. + let _done_sender = done_sender; inner.do_checkpoint(checkpoint).await; - })); + }); + self.checkpoint_task = Some(CheckpointTask { handle, done }); + } + + /// Returns a future that resolves once the checkpoint task running now has ended. + /// + /// Unlike [Self::wait_for_pending_checkpoint], which borrows the checkpointer mutably and reaps + /// the task handle, this does neither, so the caller can await the future without holding the + /// manifest lock. + pub(crate) fn pending_checkpoint_completion(&self) -> impl Future + Send + use<> { + let done = self.checkpoint_task.as_ref().map(|task| task.done.clone()); + let region_id = self.inner.region_id(); + #[cfg(test)] + let wait_started = self.pending_checkpoint_wait_started.clone(); + async move { + // The channel is closed once the task has ended. + let Some(mut done) = done.filter(|receiver| receiver.has_changed().is_ok()) else { + return; + }; + #[cfg(test)] + wait_started.notify_one(); + info!("Waiting for the pending checkpoint of region {}", region_id); + // Nothing is ever sent, so this returns once the task drops the sender. + let _ = done.changed().await; + } } /// Waits for the current checkpoint task without removing its handle first. @@ -197,14 +233,14 @@ impl Checkpointer { /// Keeping the handle in `self` while awaiting is important. If the caller is /// cancelled, another lifecycle transition can still wait for the same task. pub(crate) async fn wait_for_pending_checkpoint(&mut self) { - let Some(handle) = self.checkpoint_task.as_mut() else { + let Some(task) = self.checkpoint_task.as_mut() else { return; }; #[cfg(test)] self.pending_checkpoint_wait_started.notify_one(); - let result = (&mut *handle).await; + let result = (&mut task.handle).await; // There is no cancellation point between observing completion and // clearing the handle. self.checkpoint_task = None; @@ -218,7 +254,16 @@ impl Checkpointer { pub(crate) fn is_doing_checkpoint(&self) -> bool { self.checkpoint_task .as_ref() - .is_some_and(|handle| !handle.is_finished()) + .is_some_and(|task| !task.handle.is_finished()) + } + + #[cfg(test)] + pub(crate) fn abort_checkpoint_task(&self) { + self.checkpoint_task + .as_ref() + .expect("checkpoint task must exist") + .handle + .abort(); } #[cfg(test)] diff --git a/src/mito2/src/manifest/manager.rs b/src/mito2/src/manifest/manager.rs index da8cac5125e..35817d50078 100644 --- a/src/mito2/src/manifest/manager.rs +++ b/src/mito2/src/manifest/manager.rs @@ -12,39 +12,40 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::collections::HashSet; use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::Instant; use common_datasource::compression::CompressionType; use common_telemetry::{debug, info, warn}; -use futures::TryStreamExt; use object_store::ObjectStore; -use snafu::{OptionExt, ResultExt, ensure}; +use snafu::ensure; use store_api::metadata::RegionMetadataRef; use store_api::{MAX_VERSION, MIN_VERSION, ManifestVersion}; use crate::cache::manifest_cache::ManifestCache; use crate::config::MitoConfig; use crate::error::{ - self, InstallManifestToSnafu, NoCheckpointSnafu, NoManifestsSnafu, RegionStoppedSnafu, Result, + EndStagingPendingSnafu, InstallManifestToSnafu, NoCheckpointSnafu, NoManifestsSnafu, + RegionStoppedSnafu, Result, }; use crate::manifest::action::{ RegionChange, RegionCheckpoint, RegionEdit, RegionManifest, RegionManifestBuilder, RegionMetaAction, RegionMetaActionList, RemovedFile, }; use crate::manifest::checkpointer::Checkpointer; -use crate::manifest::storage::{ - ManifestObjectStore, file_version, is_checkpoint_file, is_delta_file, list_start_after, - manifest_compress_type, manifest_dir, -}; +use crate::manifest::storage::{ManifestObjectStore, manifest_compress_type, manifest_dir}; use crate::metrics::MANIFEST_OP_ELAPSED; -use crate::region::{ManifestStats, RegionLeaderState, RegionRoleState}; +use crate::region::ManifestStats; +use crate::sst::file_metadata::FileMetadataInternerRef; use crate::sst::{FormatType, file}; /// Options for [RegionManifestManager]. #[derive(Debug, Clone)] pub struct RegionManifestOptions { + /// Shares immutable file metadata with runtime readers in this engine. + pub(crate) file_metadata_interner: FileMetadataInternerRef, /// Directory to store manifest. pub manifest_dir: String, pub object_store: ObjectStore, @@ -61,9 +62,15 @@ pub struct RegionManifestOptions { } impl RegionManifestOptions { - /// Creates a new [RegionManifestOptions] with the given region directory, object store, and configuration. - pub fn new(config: &MitoConfig, region_dir: &str, object_store: &ObjectStore) -> Self { + /// Creates options using the same metadata interner as the owner's runtime file views. + pub fn new( + config: &MitoConfig, + region_dir: &str, + object_store: &ObjectStore, + file_metadata_interner: FileMetadataInternerRef, + ) -> Self { RegionManifestOptions { + file_metadata_interner, manifest_dir: manifest_dir(region_dir), object_store: object_store.clone(), // We don't allow users to set the compression algorithm as we use it as a file suffix. @@ -162,18 +169,52 @@ pub struct RemoveFileOptions { /// ``` #[derive(Debug)] pub struct RegionManifestManager { + file_metadata_interner: FileMetadataInternerRef, store: ManifestObjectStore, last_version: Arc, checkpointer: Checkpointer, manifest: Arc, - // Staging manifest is used to store the manifest of the staging region before it becomes available. - // It is initially inherited from the previous manifest(i.e., `self.manifest`). - // When the staging manifest becomes available, it will be used to construct the new manifest. - staging_manifest: Option>, + /// Actions of a staging region, committed as one version when staging ends. + staged: StagedActions, stats: ManifestStats, stopped: bool, } +/// Actions appended while a region is staging. They take no version and write nothing until +/// staging ends. +#[derive(Debug, Default)] +struct StagedActions { + actions: RegionMetaActionList, + /// The manifest with `actions` applied, while any are staged. + view: Option>, + /// Encoded bytes of an end commit that may have been written. A retry commits the same bytes, + /// and no other action is appended while they are kept. + attempted: Option>, +} + +impl StagedActions { + /// Returns whether `action_list` only adds files that are already staged, as a retried + /// import does. + fn only_adds_staged_files(&self, action_list: &RegionMetaActionList) -> bool { + let staged: HashSet<_> = self + .actions + .actions + .iter() + .flat_map(|action| match action { + RegionMetaAction::Edit(edit) => edit.files_to_add.as_slice(), + _ => &[], + }) + .map(|meta| meta.file_id) + .collect(); + !action_list.actions.is_empty() + && action_list.actions.iter().all(|action| { + matches!(action, RegionMetaAction::Edit(edit) + if edit.files_to_remove.is_empty() + && edit.files_to_add.iter().all(|meta| staged.contains(&meta.file_id))) + }) + } +} + impl RegionManifestManager { /// Constructs a region's manifest and persist it. pub async fn new( @@ -195,6 +236,7 @@ impl RegionManifestManager { ) .with_conditional_publication(options.conditional_publication)?; let manifest_version = stats.manifest_version.clone(); + let file_metadata_interner = options.file_metadata_interner.clone(); info!( "Creating region manifest in {} with metadata {:?}, flushed_entry_id: {}", @@ -202,7 +244,7 @@ impl RegionManifestManager { ); let version = MIN_VERSION; - let mut manifest_builder = RegionManifestBuilder::default(); + let mut manifest_builder = RegionManifestBuilder::new(file_metadata_interner.clone()); // set the initial metadata. manifest_builder.apply_change( version, @@ -241,9 +283,7 @@ impl RegionManifestManager { // Persist region change. let action_list = RegionMetaActionList::new(actions); - // New region is not in staging mode. - // TODO(ruihang): add staging mode support if needed. - store.save(version, &action_list.encode()?, false).await?; + store.save(version, &action_list.encode()?).await?; let checkpointer = Checkpointer::new(region_id, options, store.clone(), MIN_VERSION); manifest_version.store(version, Ordering::Relaxed); @@ -251,11 +291,12 @@ impl RegionManifestManager { .removed_files .update_file_removed_cnt_to_stats(stats); let manager = Self { + file_metadata_interner, store, last_version: manifest_version, checkpointer, manifest: Arc::new(manifest), - staging_manifest: None, + staged: StagedActions::default(), stats: stats.clone(), stopped: false, }; @@ -285,6 +326,7 @@ impl RegionManifestManager { ) .with_conditional_publication(options.conditional_publication)?; let manifest_version = stats.manifest_version.clone(); + let file_metadata_interner = options.file_metadata_interner.clone(); // recover from storage // construct manifest builder @@ -301,13 +343,16 @@ impl RegionManifestManager { options.manifest_dir, checkpoint.last_version ); version = version.max(checkpoint.last_version + 1); - RegionManifestBuilder::with_checkpoint(checkpoint.checkpoint) + RegionManifestBuilder::with_checkpoint( + checkpoint.checkpoint, + file_metadata_interner.clone(), + ) } else { info!( "Checkpoint not found in {}, build manifest from scratch", options.manifest_dir ); - RegionManifestBuilder::default() + RegionManifestBuilder::new(file_metadata_interner.clone()) }; let replay_start_version = version; @@ -381,19 +426,24 @@ impl RegionManifestManager { open_start.elapsed(), ); let manager = Self { + file_metadata_interner, store, last_version: manifest_version, checkpointer, manifest: Arc::new(manifest), - // TODO(weny): open the staging manifest if exists. - staging_manifest: None, + staged: StagedActions::default(), stats: stats.clone(), stopped: false, }; Ok(Some(manager)) } - /// Stops the manager. + /// Stops the manager. A stopped manager takes no update and starts no checkpoint, but a + /// checkpoint already running keeps writing the manifest directory until it ends. + /// + /// A caller that hands the storage to another instance waits for that checkpoint with + /// `pending_checkpoint_completion` after the stop, as engine stop and reload do. Close and + /// drop, which run in the region worker loop, deliberately do not. pub async fn stop(&mut self) { self.stopped = true; } @@ -495,8 +545,10 @@ impl RegionManifestManager { } debug_assert_eq!(manifests.first().unwrap().0, last_version + 1); - let mut manifest_builder = - RegionManifestBuilder::with_checkpoint(Some(self.manifest.as_ref().clone())); + let mut manifest_builder = RegionManifestBuilder::from_shared_manifest( + Some(self.manifest.as_ref().clone()), + self.file_metadata_interner.clone(), + ); for (manifest_version, raw_action_list) in manifests { let action_list = RegionMetaActionList::decode(&raw_action_list)?; @@ -559,7 +611,10 @@ impl RegionManifestManager { } .fail(); }; - let builder = RegionManifestBuilder::with_checkpoint(checkpoint.checkpoint); + let builder = RegionManifestBuilder::with_checkpoint( + checkpoint.checkpoint, + self.file_metadata_interner.clone(), + ); let manifest = builder.try_build()?; let last_version = self.set_version(manifest.manifest_version); manifest @@ -576,6 +631,9 @@ impl RegionManifestManager { /// Updates the manifest. Returns the current manifest version number. /// + /// With `is_staging`, the actions are staged in memory: they take no version and write + /// nothing until [Self::commit_staged] commits them when staging ends. + /// /// The caller has no Leader deadline to show that its commit returned in time, so a normal /// commit under conditional publication is always verified and starts no checkpoint. pub async fn update( @@ -597,7 +655,7 @@ impl RegionManifestManager { &mut self, action_list: RegionMetaActionList, is_staging: bool, - mut allow_checkpoint: bool, + allow_checkpoint: bool, must_verify: impl FnOnce() -> bool, ) -> Result { let _t = MANIFEST_OP_ELAPSED @@ -624,33 +682,141 @@ impl RegionManifestManager { } } } + if is_staging { + self.stage(action_list)?; + return Ok(self.last_version()); + } + + let bytes = action_list.encode()?; + self.commit(action_list.actions, &bytes, allow_checkpoint, must_verify) + .await + } + + /// Saves `bytes`, the encoded `actions`, as the next version and applies the actions. + async fn commit( + &mut self, + actions: Vec, + bytes: &[u8], + mut allow_checkpoint: bool, + must_verify: impl FnOnce() -> bool, + ) -> Result { let conditional = self.store.is_conditional_publication(); let version = if conditional { self.last_version() + 1 } else { self.increase_version() }; - self.store - .save(version, &action_list.encode()?, is_staging) - .await?; + self.store.save(version, bytes).await?; if conditional { - if !is_staging && must_verify() { + if must_verify() { self.store.ensure_version_not_checkpointed(version).await?; allow_checkpoint = false; } self.set_version(version); } - // For a staging region, the manifest is initially inherited from the previous manifest(i.e., `self.manifest`). - // When the staging manifest becomes available, it will be used to construct the new manifest. - let mut manifest_builder = - if is_staging && let Some(staging_manifest) = self.staging_manifest.as_ref() { - RegionManifestBuilder::with_checkpoint(Some(staging_manifest.as_ref().clone())) - } else { - RegionManifestBuilder::with_checkpoint(Some(self.manifest.as_ref().clone())) - }; + let mut manifest_builder = RegionManifestBuilder::from_shared_manifest( + Some(self.manifest.as_ref().clone()), + self.file_metadata_interner.clone(), + ); + self.apply_actions(&mut manifest_builder, version, actions); + let new_manifest = manifest_builder.try_build()?; + new_manifest + .removed_files + .update_file_removed_cnt_to_stats(&self.stats); + let updated_manifest = self + .checkpointer + .update_manifest_removed_files(new_manifest)?; + self.manifest = Arc::new(updated_manifest); + if allow_checkpoint { + self.checkpointer + .maybe_do_checkpoint(self.manifest.as_ref()) + .await; + } - for action in action_list.actions { + Ok(version) + } + + /// Commits the staged actions as one version and clears them. Returns the version and the + /// committed actions. + /// + /// If the commit fails with an error that may have written it, its encoded bytes are kept: + /// the next call commits the same bytes, and no other action is staged until the staged + /// actions are cleared. + pub(crate) async fn commit_staged( + &mut self, + must_verify: impl FnOnce() -> bool, + ) -> Result<(ManifestVersion, RegionMetaActionList)> { + let _t = MANIFEST_OP_ELAPSED + .with_label_values(&["update"]) + .start_timer(); + + ensure!( + !self.stopped, + RegionStoppedSnafu { + region_id: self.manifest.metadata.region_id, + } + ); + + let bytes = match &self.staged.attempted { + Some(bytes) => bytes.clone(), + None => self.staged.actions.encode()?, + }; + // A copy builds the new manifest; the staged list itself goes back for the region version. + let actions = self.staged.actions.actions.clone(); + let result = self.commit(actions, &bytes, true, must_verify).await; + if let Err(e) = &result + && e.may_have_persisted_manifest_update() + { + self.staged.attempted = Some(bytes); + } + let version = result?; + let committed = std::mem::take(&mut self.staged).actions; + Ok((version, committed)) + } + + /// Returns the staged actions, or `None` if nothing is staged. + pub(crate) fn staged_actions(&self) -> Option<&RegionMetaActionList> { + (!self.staged.actions.actions.is_empty()).then_some(&self.staged.actions) + } + + /// Stages `action_list` in memory. + fn stage(&mut self, action_list: RegionMetaActionList) -> Result<()> { + if self.staged.attempted.is_some() { + // A kept end commit must be retried with the same content: a retried import of + // files that are already staged changes nothing, and nothing else is staged. + ensure!( + self.staged.only_adds_staged_files(&action_list), + EndStagingPendingSnafu { + region_id: self.manifest.metadata.region_id, + } + ); + return Ok(()); + } + + let base = self.staged.view.as_ref().unwrap_or(&self.manifest); + let mut manifest_builder = RegionManifestBuilder::from_shared_manifest( + Some(base.as_ref().clone()), + self.file_metadata_interner.clone(), + ); + self.apply_actions( + &mut manifest_builder, + self.last_version(), + action_list.actions.iter().cloned(), + ); + self.staged.view = Some(Arc::new(manifest_builder.try_build()?)); + self.staged.actions.actions.extend(action_list.actions); + Ok(()) + } + + /// Applies `actions` of `version` to `manifest_builder`. + fn apply_actions( + &self, + manifest_builder: &mut RegionManifestBuilder, + version: ManifestVersion, + actions: impl IntoIterator, + ) { + for action in actions { match action { RegionMetaAction::Change(action) => { manifest_builder.apply_change(version, action); @@ -672,32 +838,11 @@ impl RegionManifestManager { } } } + } - if is_staging { - let new_manifest = manifest_builder.try_build()?; - self.staging_manifest = Some(Arc::new(new_manifest)); - - info!( - "Skipping checkpoint for region {} in staging mode, manifest version: {}", - self.manifest.metadata.region_id, self.manifest.manifest_version - ); - } else { - let new_manifest = manifest_builder.try_build()?; - new_manifest - .removed_files - .update_file_removed_cnt_to_stats(&self.stats); - let updated_manifest = self - .checkpointer - .update_manifest_removed_files(new_manifest)?; - self.manifest = Arc::new(updated_manifest); - if allow_checkpoint { - self.checkpointer - .maybe_do_checkpoint(self.manifest.as_ref()) - .await; - } - } - - Ok(version) + /// Clears the staged actions. + fn clear_staged(&mut self) { + self.staged = StagedActions::default(); } /// Waits for an in-flight checkpoint to finish, including its cleanup. @@ -705,6 +850,13 @@ impl RegionManifestManager { self.checkpointer.wait_for_pending_checkpoint().await; } + /// Returns a future that resolves once the in-flight checkpoint has finished, including its + /// cleanup. Unlike [Self::wait_for_pending_checkpoint], which borrows the manager mutably and + /// reaps the task handle, this does neither. + pub(crate) fn pending_checkpoint_completion(&self) -> impl Future + Send + use<> { + self.checkpointer.pending_checkpoint_completion() + } + /// Clear deleted files from manifest's `removed_files` field without update version. Notice if datanode exit before checkpoint then new manifest by open region may still contain these deleted files, which is acceptable for gc process. pub fn clear_deleted_files(&mut self, deleted_files: Vec) { let mut manifest = (*self.manifest()).clone(); @@ -721,9 +873,9 @@ impl RegionManifestManager { self.manifest.clone() } - /// Retrieves the current [RegionManifest]. + /// Returns the manifest with the staged actions applied, or `None` if nothing is staged. pub fn staging_manifest(&self) -> Option> { - self.staging_manifest.clone() + self.staged.view.clone() } /// Returns total manifest size. @@ -735,37 +887,11 @@ impl RegionManifestManager { /// /// It is typically used in read-only regions to catch up with manifest. /// It doesn't lock the manifest directory in the object store so the result - /// may be inaccurate if there are concurrent writes. + /// may be inaccurate if there are concurrent writes. A caller holds the region's manifest + /// `RwLock` read guard across the listing (e.g. `ManifestContext::has_update`), which + /// `ManifestContext::installed_version_if_updated` does not hold while it lists. pub async fn has_update(&self) -> Result { - let last_version = self.last_version(); - - // Skip older files at the object-store layer. Files for `v == last_version` - // may still appear (`{path}{v:020}` sorts before `{path}{v:020}.json`) but - // they are filtered out below by the `version > last_version` check. - let start_after = list_start_after(self.store.manifest_dir(), last_version); - let streamer = self - .store - .manifest_lister(false, Some(&start_after)) - .await? - .context(error::EmptyManifestDirSnafu { - manifest_dir: self.store.manifest_dir(), - })?; - - let need_update = streamer - .try_any(|entry| async move { - let file_name = entry.name(); - if is_delta_file(file_name) || is_checkpoint_file(file_name) { - let version = file_version(file_name); - if version > last_version { - return true; - } - } - false - }) - .await - .context(error::OpenDalSnafu)?; - - Ok(need_update) + self.store.has_version_after(self.last_version()).await } /// Increases last version and returns the increased version. @@ -810,71 +936,14 @@ impl RegionManifestManager { &self.checkpointer } - /// Merge all staged manifest actions into a single action list ready for submission. - /// This collects all staging manifests, applies them sequentially, and returns the merged actions. - pub(crate) async fn merge_staged_actions( - &mut self, - region_state: RegionRoleState, - ) -> Result> { - // Only merge if we're in staging mode - if region_state != RegionRoleState::Leader(RegionLeaderState::Staging) { - return Ok(None); - } - - // Fetch all staging manifests - let staging_manifests = self.store.fetch_staging_manifests().await?; - - if staging_manifests.is_empty() { - info!( - "No staging manifests to merge for region {}", - self.manifest.metadata.region_id - ); - return Ok(None); - } - - info!( - "Merging {} staging manifests for region {}", - staging_manifests.len(), - self.manifest.metadata.region_id - ); - - // Start with current manifest state as the base - let mut merged_actions = Vec::new(); - let mut latest_version = self.last_version(); - - // Apply all staging actions in order - for (manifest_version, raw_action_list) in staging_manifests { - let action_list = RegionMetaActionList::decode(&raw_action_list)?; - - for action in action_list.actions { - merged_actions.push(action); - } - - latest_version = latest_version.max(manifest_version); - } - - if merged_actions.is_empty() { - return Ok(None); - } - - info!( - "Successfully merged {} actions from staging manifests for region {}, latest version: {}", - merged_actions.len(), - self.manifest.metadata.region_id, - latest_version - ); - - Ok(Some(RegionMetaActionList::new(merged_actions))) - } - - /// Unsets the staging manifest. + /// Clears the staged actions. pub(crate) fn unset_staging_manifest(&mut self) { - self.staging_manifest = None; + self.clear_staged(); } - /// Clear all staging manifests. + /// Clears the staged actions and the staging directory. pub(crate) async fn clear_staging_manifest_and_dir(&mut self) -> Result<()> { - self.staging_manifest = None; + self.clear_staged(); self.store.clear_staging_manifests().await?; info!( "Cleared all staging manifests for region {}", @@ -984,6 +1053,67 @@ mod test { assert_eq!(manifest.metadata.partition_expr.as_deref(), Some(expr_json)); } + #[tokio::test] + async fn recover_historical_file_expression_types() { + use datatypes::value::Value; + use partition::expr::col; + use store_api::storage::FileId; + + use crate::sst::file::FileMeta; + + for checkpoint_distance in [0, 1] { + let env = TestEnv::new().await; + let metadata = Arc::new(basic_region_metadata()); + let mut manager = env + .create_manifest_manager( + CompressionType::Uncompressed, + checkpoint_distance, + Some(metadata.clone()), + ) + .await + .unwrap() + .unwrap(); + let mut files = Vec::new(); + for value in [Value::Int32(10), Value::Int64(10)] { + let file = FileMeta { + region_id: metadata.region_id, + file_id: FileId::random(), + partition_expr: Some(col("tag").lt(value).into()), + ..Default::default() + }; + manager + .update( + RegionMetaActionList::with_action(RegionMetaAction::Edit(RegionEdit { + files_to_add: vec![file.clone()], + ..Default::default() + })), + false, + ) + .await + .unwrap(); + manager.wait_for_pending_checkpoint().await; + files.push(file); + } + assert_eq!( + checkpoint_distance != 0, + manager.checkpointer.last_checkpoint_version() > 0 + ); + manager.stop().await; + drop(manager); + // The helper supplies a fresh interner, so recovery cannot reuse old entries. + let recovered = env + .create_manifest_manager(CompressionType::Uncompressed, checkpoint_distance, None) + .await + .unwrap() + .unwrap(); + let manifest = recovered.manifest(); + assert_eq!(files.len(), manifest.files.len()); + for file in files { + assert_eq!(&file, manifest.files[&file.file_id].as_ref()); + } + } + } + #[tokio::test] async fn region_change_add_column() { let metadata = Arc::new(basic_region_metadata()); diff --git a/src/mito2/src/manifest/storage.rs b/src/mito2/src/manifest/storage.rs index 7de29a319df..1b641ccbc32 100644 --- a/src/mito2/src/manifest/storage.rs +++ b/src/mito2/src/manifest/storage.rs @@ -32,14 +32,14 @@ use lazy_static::lazy_static; use object_store::util::join_dir; use object_store::{Lister, ObjectStore, util}; use regex::Regex; -use snafu::{ResultExt, ensure}; +use snafu::{OptionExt, ResultExt, ensure}; use store_api::storage::RegionId; use store_api::{MIN_VERSION, ManifestVersion}; use crate::cache::manifest_cache::ManifestCache; use crate::error::{ - ChecksumMismatchSnafu, ManifestVersionConflictSnafu, OpenDalSnafu, Result, - UnsupportedOperationSnafu, + ChecksumMismatchSnafu, EmptyManifestDirSnafu, ManifestVersionConflictSnafu, OpenDalSnafu, + Result, UnsupportedOperationSnafu, }; use crate::manifest::storage::checkpoint::CheckpointStorage; use crate::manifest::storage::delta::DeltaStorage; @@ -219,8 +219,7 @@ impl ManifestObjectStore { manifest_cache.clone(), delta_tracker, ); - let staging_storage = - StagingStorage::new(path.clone(), object_store.clone(), compress_type); + let staging_storage = StagingStorage::new(path.clone(), object_store.clone()); Self { object_store, @@ -263,21 +262,44 @@ impl ManifestObjectStore { &self.path } - /// Returns an iterator of manifests from normal or staging directory. + /// Returns whether the directory holds a delta or checkpoint of a version after + /// `last_version`. It reads the directory without locking it. + pub(crate) async fn has_version_after(&self, last_version: ManifestVersion) -> Result { + // Skip older files at the object-store layer. Files for `v == last_version` + // may still appear (`{path}{v:020}` sorts before `{path}{v:020}.json`) but + // they are filtered out below by the `version > last_version` check. + let start_after = list_start_after(self.manifest_dir(), last_version); + let streamer = + self.manifest_lister(Some(&start_after)) + .await? + .context(EmptyManifestDirSnafu { + manifest_dir: self.manifest_dir(), + })?; + + streamer + .try_any(|entry| async move { + let file_name = entry.name(); + if is_delta_file(file_name) || is_checkpoint_file(file_name) { + let version = file_version(file_name); + if version > last_version { + return true; + } + } + false + }) + .await + .context(OpenDalSnafu) + } + + /// Returns an iterator of manifests. /// - /// `start_after` is forwarded to the non-staging lister to skip entries - /// whose name is lexicographically less than or equal to it. It is - /// ignored for the staging directory. + /// `start_after` is forwarded to the lister to skip entries whose name is + /// lexicographically less than or equal to it. pub(crate) async fn manifest_lister( &self, - is_staging: bool, start_after: Option<&str>, ) -> Result> { - if is_staging { - self.staging_storage.manifest_lister().await - } else { - self.delta_storage.manifest_lister(start_after).await - } + self.delta_storage.manifest_lister(start_after).await } /// Fetches manifests in range [start_version, end_version). @@ -411,15 +433,8 @@ impl ManifestObjectStore { } /// Save the delta manifest file. - pub async fn save( - &mut self, - version: ManifestVersion, - bytes: &[u8], - is_staging: bool, - ) -> Result<()> { - if is_staging { - self.staging_storage.save(version, bytes).await - } else if self.conditional_publication { + pub async fn save(&mut self, version: ManifestVersion, bytes: &[u8]) -> Result<()> { + if self.conditional_publication { self.delta_storage.save_conditional(version, bytes).await } else { self.delta_storage.save(version, bytes).await @@ -521,11 +536,6 @@ impl ManifestObjectStore { self.size_tracker.retain_after(version); } - /// Fetch all staging manifest files and return them as (version, action_list) pairs. - pub async fn fetch_staging_manifests(&self) -> Result)>> { - self.staging_storage.fetch_manifests().await - } - /// Clear all staging manifest files. pub async fn clear_staging_manifests(&mut self) -> Result<()> { self.staging_storage.clear().await @@ -583,7 +593,6 @@ impl ManifestObjectStore { pub(crate) fn set_compress_type(&mut self, compress_type: CompressionType) { self.checkpoint_storage.set_compress_type(compress_type); self.delta_storage.set_compress_type(compress_type); - self.staging_storage.set_compress_type(compress_type); } } @@ -649,7 +658,7 @@ mod tests { async fn test_manifest_log_store_case(mut log_store: ManifestObjectStore) { for v in 0..5 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -726,7 +735,7 @@ mod tests { log_store.set_compress_type(CompressionType::Uncompressed); for v in 0..5 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -746,7 +755,7 @@ mod tests { // write compressed data to stimulate compress algorithm take effect for v in 5..10 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -803,7 +812,7 @@ mod tests { log_store.set_compress_type(CompressionType::Uncompressed); for v in 0..5 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -844,7 +853,7 @@ mod tests { // write 5 manifest files for v in 0..5 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -903,7 +912,7 @@ mod tests { async fn test_scan_with_start_after_case(mut log_store: ManifestObjectStore) { for v in 0..10 { log_store - .save(v, format!("hello, {v}").as_bytes(), false) + .save(v, format!("hello, {v}").as_bytes()) .await .unwrap(); } @@ -1024,7 +1033,7 @@ mod tests { if matches!(payload, PayloadKind::Checkpoint { .. }) { store.save_checkpoint(3, data).await.unwrap(); } else { - store.save(3, data, false).await.unwrap(); + store.save(3, data).await.unwrap(); } let encoded = remote.read(&key).await.unwrap().to_vec(); if source == CacheSource::LegacyDecoded { diff --git a/src/mito2/src/manifest/storage/delta.rs b/src/mito2/src/manifest/storage/delta.rs index 8b3a0a1d34a..ccbb8c31084 100644 --- a/src/mito2/src/manifest/storage/delta.rs +++ b/src/mito2/src/manifest/storage/delta.rs @@ -64,14 +64,6 @@ impl DeltaStorage { } } - pub(crate) fn path(&self) -> &str { - &self.path - } - - pub(crate) fn object_store(&self) -> &ObjectStore { - &self.object_store - } - fn delta_file_path(&self, version: ManifestVersion) -> String { gen_path(&self.path, &delta_file(version), self.compress_type) } diff --git a/src/mito2/src/manifest/storage/size_tracker.rs b/src/mito2/src/manifest/storage/size_tracker.rs index b28f898094d..36596f1bbc1 100644 --- a/src/mito2/src/manifest/storage/size_tracker.rs +++ b/src/mito2/src/manifest/storage/size_tracker.rs @@ -56,15 +56,6 @@ impl Tracker for DeltaTracker { } } -#[derive(Debug, Clone)] -pub struct NoopTracker; - -impl Tracker for NoopTracker { - fn record(&self, _version: ManifestVersion, _size: u64) { - // noop - } -} - #[derive(Debug, Clone, Default)] pub(crate) struct SizeTracker { inventory: Arc>, diff --git a/src/mito2/src/manifest/storage/staging.rs b/src/mito2/src/manifest/storage/staging.rs index e4e9e7f177f..836fd96edac 100644 --- a/src/mito2/src/manifest/storage/staging.rs +++ b/src/mito2/src/manifest/storage/staging.rs @@ -12,19 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::sync::Arc; - -use common_datasource::compression::CompressionType; use common_telemetry::debug; -use object_store::{Lister, ObjectStore, util}; +use object_store::{ObjectStore, util}; use snafu::ResultExt; -use store_api::ManifestVersion; use crate::error::{OpenDalSnafu, Result}; -use crate::manifest::storage::delta::DeltaStorage; -use crate::manifest::storage::size_tracker::NoopTracker; -use crate::manifest::storage::utils::{sort_manifests, write_and_put_cache}; -use crate::manifest::storage::{file_version, is_delta_file}; +use crate::manifest::storage::utils::write_and_put_cache; /// A simple blob storage for arbitrary binary data in the staging directory. /// @@ -97,18 +90,16 @@ impl StagingBlobStorage { } } -/// Storage for staging manifest files and blobs used during repartition operations. -/// -/// Fields: -/// - `delta_storage`: Manages incremental manifest delta files specific to the staging region. -/// - `blob_storage`: Manages arbitrary blobs, such as generated manifests for regions. +/// Storage for the staging directory used during repartition operations. /// /// Directory structure: -/// - `{region_dir}/staging/manifest/` — for incremental manifest delta files for the staging region. +/// - `{region_dir}/staging/manifest/` — manifest deltas that older versions staged. It is only +/// cleared. /// - `{region_dir}/staging/blob/` — for arbitrary blobs (e.g., generated region manifests). #[derive(Debug, Clone)] pub(crate) struct StagingStorage { - delta_storage: DeltaStorage, + object_store: ObjectStore, + manifest_path: String, blob_storage: StagingBlobStorage, } @@ -125,23 +116,12 @@ pub fn staging_manifest_path(manifest_path: &str) -> String { } impl StagingStorage { - pub fn new(path: String, object_store: ObjectStore, compress_type: CompressionType) -> Self { - let staging_blob_path = staging_blob_path(&path); - let blob_storage = StagingBlobStorage::new(staging_blob_path, object_store.clone()); - let staging_manifest_path = staging_manifest_path(&path); - let delta_storage = DeltaStorage::new( - staging_manifest_path.clone(), - object_store.clone(), - compress_type, - // StagingStorage does not use a manifest cache; set to None. - None, - // StagingStorage does not track file sizes, since all staging files are - // deleted after exiting staging mode. - Arc::new(NoopTracker), - ); + pub fn new(path: String, object_store: ObjectStore) -> Self { + let blob_storage = StagingBlobStorage::new(staging_blob_path(&path), object_store.clone()); Self { - delta_storage, + object_store, + manifest_path: staging_manifest_path(&path), blob_storage, } } @@ -151,44 +131,10 @@ impl StagingStorage { &self.blob_storage } - /// Returns an iterator of manifests from staging directory. - pub(crate) async fn manifest_lister(&self) -> Result> { - self.delta_storage.manifest_lister(None).await - } - - /// Fetch all staging manifest files and return them as (version, action_list) pairs. - pub(crate) async fn fetch_manifests(&self) -> Result)>> { - let manifest_entries = self - .delta_storage - .get_paths(None, |entry| { - let file_name = entry.name(); - if is_delta_file(file_name) { - let version = file_version(file_name); - Some((version, entry)) - } else { - None - } - }) - .await?; - - let mut sorted_entries = manifest_entries; - sort_manifests(&mut sorted_entries); - - self.delta_storage - .fetch_manifests_from_entries(sorted_entries) - .await - } - - /// Save the delta manifest file. - pub(crate) async fn save(&mut self, version: ManifestVersion, bytes: &[u8]) -> Result<()> { - self.delta_storage.save(version, bytes).await - } - /// Clean all staging manifest files. pub(crate) async fn clear(&self) -> Result<()> { - self.delta_storage - .object_store() - .delete_with(self.delta_storage.path()) + self.object_store + .delete_with(&self.manifest_path) .recursive(true) .await .context(OpenDalSnafu)?; @@ -202,7 +148,7 @@ impl StagingStorage { debug!( "Cleared all staging manifest files from {}, blob directory: {}", - self.delta_storage.path(), + self.manifest_path, self.blob_storage.path() ); @@ -210,18 +156,10 @@ impl StagingStorage { } } -#[cfg(test)] -impl StagingStorage { - pub fn set_compress_type(&mut self, compress_type: CompressionType) { - self.delta_storage.set_compress_type(compress_type); - } -} - #[cfg(test)] mod tests { use std::sync::{Arc, Mutex}; - use common_datasource::compression::CompressionType; use common_test_util::temp_dir::create_temp_dir; use futures::TryStreamExt; use object_store::layers::mock::{MockLayerBuilder, ObservedWriter, WriterObservations, oio}; @@ -319,15 +257,18 @@ mod tests { let builder = Fs::default().root(&tmp_dir.path().to_string_lossy()); let object_store = ObjectStore::new(builder).unwrap(); let manifest_path = "/data/table/region_0001/manifest/"; - let mut storage = StagingStorage::new( - manifest_path.to_string(), - object_store.clone(), - CompressionType::Uncompressed, - ); + let storage = StagingStorage::new(manifest_path.to_string(), object_store.clone()); let manifest_dir = staging_manifest_path(manifest_path); let blob_dir = staging_blob_path(manifest_path); - storage.save(0, b"manifest").await.unwrap(); + // A delta staged by an older version. + object_store + .write( + &format!("{manifest_dir}00000000000000000000.json"), + b"manifest".to_vec(), + ) + .await + .unwrap(); storage .blob_storage() .put("region_0002_manifest", b"blob".to_vec()) diff --git a/src/mito2/src/manifest/tests/checkpoint.rs b/src/mito2/src/manifest/tests/checkpoint.rs index 708ca51ceb4..0a8a7bdab59 100644 --- a/src/mito2/src/manifest/tests/checkpoint.rs +++ b/src/mito2/src/manifest/tests/checkpoint.rs @@ -14,7 +14,7 @@ use std::assert_matches; use std::sync::Arc; -use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::time::Duration; use common_datasource::compression::CompressionType; @@ -25,6 +25,7 @@ use object_store::layers::mock::{ }; use store_api::storage::{FileId, RegionId}; use strum::IntoEnumIterator; +use tokio::sync::Notify; use crate::error::Error::{ChecksumMismatch, ManifestDeltaNotFound}; use crate::manifest::action::{ @@ -507,11 +508,13 @@ async fn manifest_install_manifest_to_with_checkpoint() { async fn test_checkpoint_bypass_in_staging_mode() { common_telemetry::init_default_ut_logging(); let (_env, mut manager) = build_manager(1, CompressionType::Uncompressed).await; + let version = manager.last_version(); // Apply actions in staging mode - checkpoint should be bypassed for _ in 0..15 { manager.update(nop_action(), true).await.unwrap(); } + assert_eq!(manager.last_version(), version); assert!(!manager.checkpointer().is_doing_checkpoint()); // Verify no checkpoint was created in staging mode @@ -524,8 +527,9 @@ async fn test_checkpoint_bypass_in_staging_mode() { .is_none() ); - // Now switch to normal mode and apply one more action - manager.update(nop_action(), false).await.unwrap(); + // The end of staging commits all of them as the next version. + let (committed_version, _) = manager.commit_staged(|| false).await.unwrap(); + assert_eq!(committed_version, version + 1); // Wait for potential checkpoint while manager.checkpointer().is_doing_checkpoint() { @@ -541,8 +545,8 @@ async fn test_checkpoint_bypass_in_staging_mode() { ); let (last_version, _checkpoint_data) = checkpoint_result.unwrap(); - // Checkpoint should include all 16 actions (15 from staging + 1 from writable) - assert_eq!(last_version, 16); + // Checkpoint should include the staged actions, which took one version + assert_eq!(last_version, version + 1); } /// Simulates both a rejected delete and a close failure after eager deletion. @@ -705,7 +709,7 @@ async fn install_preserves_listed_delta_not_found_retry_hint() { .unwrap(); let mut store = manager.store(); store - .save(1, &nop_action().encode().unwrap(), false) + .save(1, &nop_action().encode().unwrap()) .await .unwrap(); @@ -825,6 +829,132 @@ async fn completed_checkpoint_is_reaped_before_scheduling_the_next_one() { assert_eq!(2, manager.checkpointer().last_checkpoint_version()); } +#[rstest::rstest] +#[case::cleanup_finished(false)] +#[case::task_aborted(true)] +#[tokio::test] +async fn pending_checkpoint_completion_waits_until_the_task_ends(#[case] abort: bool) { + let (blocker, mock_layer) = CheckpointTaskBlocker::block_cleanup(); + let env = TestEnv::new().await.with_mock_layer(mock_layer); + let metadata = Arc::new(basic_region_metadata()); + let mut manager = env + .create_manifest_manager(CompressionType::Uncompressed, 1, Some(metadata)) + .await + .unwrap() + .unwrap(); + + manager.update(nop_action(), false).await.unwrap(); + tokio::time::timeout(Duration::from_secs(5), blocker.wait_until_blocked()) + .await + .expect("checkpoint cleanup did not start"); + + // Two waiters stay on their own tasks until the checkpoint ends; a third is cancelled. + let waiters = [ + manager.pending_checkpoint_completion(), + manager.pending_checkpoint_completion(), + ] + .map(tokio::spawn); + let mut cancelled = Box::pin(manager.pending_checkpoint_completion()); + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut cancelled) + .await + .is_err(), + "the cancelled wait ended before the checkpoint cleanup" + ); + drop(cancelled); + assert!( + waiters.iter().all(|waiter| !waiter.is_finished()), + "a spawned wait ended before the checkpoint cleanup" + ); + // The cleanup stays blocked in the abort case, so only the abort can end the task. + if abort { + manager.checkpointer().abort_checkpoint_task(); + } else { + blocker.release(); + } + for waiter in waiters { + tokio::time::timeout(Duration::from_secs(5), waiter) + .await + .expect("the wait did not end with the checkpoint task") + .unwrap(); + } + let wait_started = manager.checkpointer().pending_checkpoint_wait_started(); + // Consumes the permit the waiters above left. + let _ = tokio::time::timeout(Duration::ZERO, wait_started.notified()).await; + tokio::time::timeout(Duration::ZERO, manager.pending_checkpoint_completion()) + .await + .expect("a wait after the checkpoint task ended did not return at once"); + assert!( + tokio::time::timeout(Duration::ZERO, wait_started.notified()) + .await + .is_err(), + "a wait after the checkpoint task ended started waiting" + ); +} + +/// Panics on the first delete, which a checkpoint issues in its cleanup, once released. +struct PanickingDeleter { + entered: Arc, + release: Arc, + panicked: Arc, +} + +impl oio::Delete for PanickingDeleter { + async fn delete(&mut self, _path: &str, _args: OpDelete) -> MockResult<()> { + self.entered.notify_one(); + self.release.notified().await; + self.panicked.store(true, Ordering::Relaxed); + panic!("mock manifest delete panic"); + } + + async fn close(&mut self) -> MockResult<()> { + Ok(()) + } +} + +#[tokio::test] +async fn pending_checkpoint_completion_ends_when_the_task_panics() { + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + let panicked = Arc::new(AtomicBool::new(false)); + let (factory_entered, factory_release, factory_panicked) = + (entered.clone(), release.clone(), panicked.clone()); + let mock_layer = MockLayerBuilder::default() + .deleter_factory(Arc::new(move |_inner| { + Box::new(PanickingDeleter { + entered: factory_entered.clone(), + release: factory_release.clone(), + panicked: factory_panicked.clone(), + }) + })) + .build() + .unwrap(); + let env = TestEnv::new().await.with_mock_layer(mock_layer); + let metadata = Arc::new(basic_region_metadata()); + let mut manager = env + .create_manifest_manager(CompressionType::Uncompressed, 1, Some(metadata)) + .await + .unwrap() + .unwrap(); + + manager.update(nop_action(), false).await.unwrap(); + tokio::time::timeout(Duration::from_secs(5), entered.notified()) + .await + .expect("checkpoint cleanup did not start"); + let mut done = Box::pin(manager.pending_checkpoint_completion()); + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut done) + .await + .is_err(), + "the wait ended before the checkpoint task panicked" + ); + release.notify_one(); + tokio::time::timeout(Duration::from_secs(5), done) + .await + .expect("the wait did not end after the checkpoint task panicked"); + assert!(panicked.load(Ordering::Relaxed)); +} + /// Retention runs inside the checkpoint task, so waiting for task completion also /// makes logical size accounting observable to the assertions below. async fn wait_for_manifest_checkpoint(manager: &RegionManifestManager) { @@ -1008,6 +1138,7 @@ async fn manifest_size_warm_cache_matches_recovery_bytes( .unwrap(); let cache = ManifestCache::new(local_store, ReadableSize::mb(10), None, true).await; let options = RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "/".to_string(), object_store, compress_type, @@ -1110,6 +1241,7 @@ async fn manifest_cache_write_through_survives_reopen(#[case] compression: Compr let cache = ManifestCache::new(local.clone(), ReadableSize::mb(32), None, true).await; // Use a relative prefix, matching production keys and the recovered index. let mut options = RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "region/".to_string(), object_store: remote, compress_type: compression, diff --git a/src/mito2/src/manifest/tests/conditional.rs b/src/mito2/src/manifest/tests/conditional.rs index e270d205475..b98b0c381f8 100644 --- a/src/mito2/src/manifest/tests/conditional.rs +++ b/src/mito2/src/manifest/tests/conditional.rs @@ -29,8 +29,10 @@ use object_store::layers::mock::{ use object_store::services::Fs; use rstest::rstest; use store_api::ManifestVersion; +use store_api::storage::FileId; use crate::error::{Error, Result}; +use crate::manifest::action::{RegionEdit, RegionMetaAction, RegionMetaActionList}; use crate::manifest::manager::{RegionManifestManager, RegionManifestOptions}; use crate::manifest::storage::{ LAST_CHECKPOINT_FILE, ManifestObjectStore, delta_file, gen_path, is_checkpoint_file, @@ -38,7 +40,8 @@ use crate::manifest::storage::{ }; use crate::manifest::tests::utils::{basic_region_metadata, fail_manifest_reads_layer}; use crate::sst::FormatType; -use crate::test_util::nop_action; +use crate::sst::file::FileMeta; +use crate::test_util::{DeltaFaults, nop_action}; fn temp_store() -> (TempDir, ObjectStore) { let dir = create_temp_dir(""); @@ -53,6 +56,7 @@ fn options( conditional_publication: bool, ) -> RegionManifestOptions { RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "/".to_string(), object_store: store.clone(), compress_type, @@ -141,6 +145,16 @@ fn assert_conflict(error: &Error) { ); } +fn import(file_id: FileId) -> RegionMetaActionList { + RegionMetaActionList::with_action(RegionMetaAction::Edit(RegionEdit { + files_to_add: vec![FileMeta { + file_id, + ..Default::default() + }], + ..Default::default() + })) +} + #[rstest] #[case::conditional(true)] #[case::legacy(false)] @@ -212,17 +226,17 @@ async fn test_compression_settings_share_one_name() { let mut compressed = conditional_store(&store, CompressionType::Gzip); let data = vec![b'a'; 1024]; - stored.save(1, &data, false).await.unwrap(); + stored.save(1, &data).await.unwrap(); let stored_object = store.read(&delta_key(1)).await.unwrap().to_vec(); assert!(stored_object.len() > data.len()); // A retry by a writer with the other setting succeeds and leaves the object alone. - compressed.save(1, &data, false).await.unwrap(); + compressed.save(1, &data).await.unwrap(); let retried_object = store.read(&delta_key(1)).await.unwrap().to_vec(); assert_eq!(stored_object, retried_object); assert_eq!(stored_object.len() as u64, compressed.total_manifest_size()); - compressed.save(2, &data, false).await.unwrap(); + compressed.save(2, &data).await.unwrap(); assert!(store.read(&delta_key(2)).await.unwrap().len() < data.len()); assert_eq!( vec![(1, data.clone()), (2, data)], @@ -253,7 +267,7 @@ async fn test_existing_delta_conflicts(#[case] undecodable: bool, #[case] unread }; let error = conditional_store(&writer_store, CompressionType::Gzip) - .save(1, b"mine", false) + .save(1, b"mine") .await .unwrap_err(); @@ -370,3 +384,34 @@ async fn test_deltas_are_readable_across_modes( assert_eq!(1, reader.last_version()); assert_eq!(2, commit(&mut reader).await.unwrap()); } + +#[tokio::test] +async fn test_staged_commit_retry_writes_same_content() { + let (_dir, store) = temp_store(); + let faults = DeltaFaults::default(); + let store = store.layer(faults.layer()); + let mut manager = create_manager(options(&store, CompressionType::Gzip, 0, true)).await; + let imported = FileId::random(); + manager.update(import(imported), true).await.unwrap(); + + faults.lose_next_response(); + let error = manager.commit_staged(|| false).await.unwrap_err(); + assert!(error.may_have_persisted_manifest_update()); + assert_eq!(0, manager.last_version()); + + // Until the kept commit is retried, only a retried import (empty for a target with no + // files) is staged. + manager.update(import(imported), true).await.unwrap(); + let empty = RegionMetaActionList::with_action(RegionMetaAction::Edit(Default::default())); + manager.update(empty, true).await.unwrap(); + let refused = manager.update(import(FileId::random()), true).await; + assert!( + matches!(refused, Err(Error::EndStagingPending { .. })), + "{refused:?}" + ); + + // The retry writes the same content, so it recognizes its own landed write. + assert_eq!(1, manager.commit_staged(|| false).await.unwrap().0); + assert!(manager.staged_actions().is_none()); + assert!(manager.manifest().files.contains_key(&imported)); +} diff --git a/src/mito2/src/memtable/bulk/context.rs b/src/mito2/src/memtable/bulk/context.rs index 994d3a027d4..14b8d98634f 100644 --- a/src/mito2/src/memtable/bulk/context.rs +++ b/src/mito2/src/memtable/bulk/context.rs @@ -106,7 +106,6 @@ impl BulkIterContext { codec, // we don't need to compat batch since all batch in memtable have the same schema. compat_batch: None, - compaction_projection_mapper: None, pre_filter_mode, partition_filter: None, }, diff --git a/src/mito2/src/metrics.rs b/src/mito2/src/metrics.rs index 812a827f8f7..a24cdd0bf6d 100644 --- a/src/mito2/src/metrics.rs +++ b/src/mito2/src/metrics.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +pub(crate) mod file_metadata; pub(crate) mod manifest_size; use std::time::{Duration, Instant}; @@ -68,6 +69,12 @@ lazy_static! { ) .unwrap(); + /// Close-time WAL releases that the log store failed to admit. + pub static ref WAL_RELEASE_ADMISSION_FAILURE_TOTAL: IntCounter = register_int_counter!( + "greptime_mito_wal_release_admission_failure_total", + "Close-time WAL release admission failures" + ).unwrap(); + // ------ Flush related metrics /// Counter of scheduled flush requests. /// Note that the flush scheduler may merge some flush requests. @@ -728,6 +735,35 @@ lazy_static! { /// Counter of flush files. pub static ref FLUSH_FILE_TOTAL: IntCounter = register_int_counter!("greptime_mito_flush_file_total", "mito flushed file count").unwrap(); + + /// Elapsed time of the SST writes of tail replays, by stage. + pub static ref TAIL_REPLAY_ELAPSED: HistogramVec = register_histogram_vec!( + "greptime_mito_tail_replay_elapsed", + "mito tail replay elapsed", + &[TYPE_LABEL], + // 1 ~ 3125 + exponential_buckets(1.0, 5.0, 6).unwrap(), + ) + .unwrap(); + /// Bytes of SSTs from completed chunk writes of tail replays, including chunks of rounds + /// that fail later. + pub static ref TAIL_REPLAY_BYTES_TOTAL: IntCounter = register_int_counter!( + "greptime_mito_tail_replay_bytes_total", + "mito tail replay bytes total" + ) + .unwrap(); + /// Count of SSTs written by tail replays, including SSTs of failed rounds that were deleted. + pub static ref TAIL_REPLAY_FILE_TOTAL: IntCounter = register_int_counter!( + "greptime_mito_tail_replay_file_total", + "mito tail replay file count" + ) + .unwrap(); + /// Count of failed tail replay rounds. + pub static ref TAIL_REPLAY_FAILURE_TOTAL: IntCounter = register_int_counter!( + "greptime_mito_tail_replay_failure_total", + "mito tail replay failure total" + ) + .unwrap(); } #[cfg(feature = "hdfs-object-store")] diff --git a/src/mito2/src/metrics/file_metadata.rs b/src/mito2/src/metrics/file_metadata.rs new file mode 100644 index 00000000000..595c300651d --- /dev/null +++ b/src/mito2/src/metrics/file_metadata.rs @@ -0,0 +1,93 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Scrape-time counts of registered SST metadata values. + +use std::sync::{Arc, Mutex, Weak}; + +use lazy_static::lazy_static; +use prometheus::core::{Collector, Desc}; +use prometheus::proto::MetricFamily; +use prometheus::{IntGaugeVec, Opts}; + +use crate::metrics::TYPE_LABEL; +use crate::sst::file_metadata::FileMetadataInternerState; + +lazy_static! { + pub(crate) static ref FILE_METADATA_COLLECTOR: FileMetadataCollector = { + let collector = FileMetadataCollector::default(); + prometheus::register(Box::new(collector.clone())).unwrap(); + collector + }; +} + +/// Aggregates live interners without retaining them or adding work to intern/drop. +/// Labels have fixed cardinality; entries are distinct values, not SST view counts. +#[derive(Clone)] +pub(crate) struct FileMetadataCollector { + sources: Arc>>>, + entries: IntGaugeVec, +} + +impl Default for FileMetadataCollector { + fn default() -> Self { + Self { + sources: Arc::new(Mutex::new(Vec::new())), + entries: IntGaugeVec::new( + Opts::new( + "greptime_mito_sst_metadata_interner_entries", + "Registered values in live SST metadata interners", + ), + &[TYPE_LABEL], + ) + .unwrap(), + } + } +} + +impl FileMetadataCollector { + pub(crate) fn register(&self, source: &Arc) { + let mut sources = self.sources.lock().unwrap(); + sources.retain(|source| source.strong_count() > 0); + sources.push(Arc::downgrade(source)); + } +} + +impl Collector for FileMetadataCollector { + fn desc(&self) -> Vec<&Desc> { + self.entries.desc() + } + + fn collect(&self) -> Vec { + // Serialize gathers so concurrent scrapes cannot overwrite each other's gauges. + // Registration never holds a shard lock; intern/drop never take this mutex. + let mut sources = self.sources.lock().unwrap(); + let mut files = 0; + let mut expressions = 0; + sources.retain(|source| { + let Some(source) = source.upgrade() else { + return false; + }; + let counts = source.entry_counts(); + files += counts.0; + expressions += counts.1; + true + }); + self.entries.with_label_values(&["file"]).set(files as i64); + self.entries + .with_label_values(&["partition_expr"]) + .set(expressions as i64); + self.entries.collect() + } +} diff --git a/src/mito2/src/read/compat.rs b/src/mito2/src/read/compat.rs index 377b096db26..321e274b96b 100644 --- a/src/mito2/src/read/compat.rs +++ b/src/mito2/src/read/compat.rs @@ -1552,7 +1552,7 @@ mod tests { } #[test] - fn test_flat_compat_batch_compact_sparse() { + fn test_flat_compat_batch_compact_sparse_added_and_dropped_fields() { let mut actual_metadata = new_metadata( &[ ( @@ -1561,6 +1561,7 @@ mod tests { ConcreteDataType::timestamp_millisecond_datatype(), ), (2, SemanticType::Field, ConcreteDataType::int64_datatype()), + (4, SemanticType::Field, ConcreteDataType::int64_datatype()), ], &[], ); @@ -1599,6 +1600,7 @@ mod tests { let sparse_k1 = encode_sparse_key(&[]); let input_columns: Vec = vec![ Arc::new(Int64Array::from(vec![100, 200])), + Arc::new(Int64Array::from(vec![300, 400])), // dropped field Arc::new(TimestampMillisecondArray::from_iter_values([1000, 2000])), build_flat_test_pk_array(&[&sparse_k1, &sparse_k1]), Arc::new(UInt64Array::from_iter_values([1, 2])), @@ -1613,6 +1615,11 @@ mod tests { ); let input_batch = RecordBatch::try_new(input_schema, input_columns).unwrap(); + // The physical read projection omits the dropped source field. Sparse + // conversion preserves this encoded layout, which goes directly to compat. + assert!(read_format.projected_index_by_id(4).is_none()); + let input_batch = input_batch.project(&[0, 2, 3, 4, 5]).unwrap(); + let input_batch = read_format.convert_batch(input_batch, None).unwrap(); let result = compat_batch.compat(input_batch).unwrap(); let expected_columns: Vec = vec![ diff --git a/src/mito2/src/read/flat_projection.rs b/src/mito2/src/read/flat_projection.rs index 69df6256762..28fa18f6b05 100644 --- a/src/mito2/src/read/flat_projection.rs +++ b/src/mito2/src/read/flat_projection.rs @@ -31,7 +31,7 @@ use store_api::metadata::{RegionMetadata, RegionMetadataRef}; use store_api::storage::ColumnId; use crate::cache::CacheStrategy; -use crate::error::{InvalidRequestSnafu, NewRecordBatchSnafu, Result}; +use crate::error::{InvalidRequestSnafu, Result}; use crate::read::projection::{read_column_ids_from_projection, repeated_vector_with_cache}; use crate::read::read_columns::{JsonTargetTypes, ReadColumns}; use crate::sst::parquet::Json2RewriteTargets; @@ -46,6 +46,11 @@ use crate::sst::{ /// /// This mapper support duplicate and unsorted projection indices. /// The output schema is determined by the projection indices. +/// Input indices address the compatible merge schema, not physical Parquet +/// columns or temporary predicate projections. Sparse compaction uses +/// `input_arrow_schema(true)` without expanded tags and bypasses query output +/// conversion. Sparse compaction retains encoded keys through filtering and +/// passes batches directly to compatibility; predicate tags stay temporary. pub struct FlatProjectionMapper { /// Metadata of the region. metadata: RegionMetadataRef, @@ -477,47 +482,6 @@ pub(crate) fn compute_input_arrow_schema( Arc::new(datatypes::arrow::datatypes::Schema::new(new_fields)) } -/// Helper to project compaction batches into flat format columns -/// (fields + time index + __primary_key + __sequence + __op_type). -pub(crate) struct CompactionProjectionMapper { - projection: Vec, -} - -impl CompactionProjectionMapper { - pub(crate) fn try_new(metadata: &RegionMetadataRef) -> Result { - let projection = metadata - .column_metadatas - .iter() - .enumerate() - .filter_map(|(idx, col)| { - if matches!(col.semantic_type, SemanticType::Field) { - Some(idx) - } else { - None - } - }) - .chain([metadata.time_index_column_pos()]) - .collect::>(); - - let read_col_ids = metadata.column_metadatas.iter().map(|col| col.column_id); - let read_cols = ReadColumns::new(read_col_ids); - let mapper = FlatProjectionMapper::new_with_read_columns(metadata, projection, read_cols)?; - Ok(Self { - projection: mapper.batch_indices, - }) - } - - /// Projects columns and appends internal columns for compaction output. - /// - /// The input batch is expected to be in flat format with internal columns appended. - pub(crate) fn project(&self, batch: DfRecordBatch) -> Result { - let mut projection = self.projection.clone(); - projection.extend(batch.num_columns() - 3..batch.num_columns()); - // Projection must preserve the Arrow layouts already produced by compaction Rewrite. - batch.project(&projection).context(NewRecordBatchSnafu) - } -} - #[cfg(test)] mod tests { use datatypes::schema::ColumnSchema; diff --git a/src/mito2/src/read/scan_region.rs b/src/mito2/src/read/scan_region.rs index 0147f7883c8..e908560ca09 100644 --- a/src/mito2/src/read/scan_region.rs +++ b/src/mito2/src/read/scan_region.rs @@ -1518,7 +1518,7 @@ impl ScanInput { self.region_partition_expr.as_ref(), file.meta_ref().partition_expr.as_ref(), ) { - (Some(region_expr), Some(file_expr)) => region_expr == file_expr, + (Some(region_expr), Some(file_expr)) => region_expr == file_expr.as_ref(), _ => false, } } @@ -2680,7 +2680,7 @@ mod tests { FileHandle::new( FileMeta { num_rows: 1, - partition_expr, + partition_expr: partition_expr.map(Into::into), ..Default::default() }, Arc::new(crate::sst::file_purger::NoopFilePurger), diff --git a/src/mito2/src/read/scan_util.rs b/src/mito2/src/read/scan_util.rs index 69366015155..6c6c4d93839 100644 --- a/src/mito2/src/read/scan_util.rs +++ b/src/mito2/src/read/scan_util.rs @@ -1616,13 +1616,7 @@ pub fn build_flat_file_range_scan_stream( .file_handle() .is_effective_target_sequence_trusted(stream_ctx.input.region_metadata().region_id); - let mapper = range.compaction_projection_mapper(); while let Some(record_batch) = reader.next_batch().await? { - let record_batch = if let Some(mapper) = mapper { - mapper.project(record_batch)? - } else { - record_batch - }; let Some(record_batch) = filter_flat_batch_by_sequence( record_batch, stream_ctx.input.sequence_range, diff --git a/src/mito2/src/read/series_reader.rs b/src/mito2/src/read/series_reader.rs index 4399533c4e5..3d5e3137c55 100644 --- a/src/mito2/src/read/series_reader.rs +++ b/src/mito2/src/read/series_reader.rs @@ -575,11 +575,6 @@ fn scan_series_file_ranges( reader_metrics.filter_metrics.rows_precise_filtered += num_rows_before_filter - record_batch.num_rows(); - let record_batch = if let Some(mapper) = range.compaction_projection_mapper() { - mapper.project(record_batch)? - } else { - record_batch - }; if let Some(compat) = range.compat_batch() { yield compat.compat(record_batch)?; } else { diff --git a/src/mito2/src/region.rs b/src/mito2/src/region.rs index ad53b7a9424..5fe44773b67 100644 --- a/src/mito2/src/region.rs +++ b/src/mito2/src/region.rs @@ -57,13 +57,13 @@ use crate::access_layer::AccessLayerRef; use crate::compaction::CompactionStateRef; use crate::engine::region_hook::{PendingManifestHook, RegionHookRef}; use crate::error::{ - Error, FlushableRegionStateSnafu, InvalidPartitionExprSnafu, InvalidRequestSnafu, - RegionBusySnafu, RegionLeaseExpiredSnafu, RegionNotFoundSnafu, RegionStateSnafu, - RegionTruncatedSnafu, Result, UnexpectedSnafu, UpdateManifestSnafu, + EndStagingSnafu, Error, FlushableRegionStateSnafu, InvalidPartitionExprSnafu, + InvalidRequestSnafu, RegionBusySnafu, RegionLeaseExpiredSnafu, RegionNotFoundSnafu, + RegionStateSnafu, RegionTruncatedSnafu, Result, UnexpectedSnafu, UpdateManifestSnafu, }; -use crate::lease::{Deadline, NodeLease}; +use crate::lease::{Deadline, NodeLease, lease_nanos}; use crate::manifest::action::{ - RegionChange, RegionManifest, RegionMetaAction, RegionMetaActionList, + PendingTail, RegionChange, RegionManifest, RegionMetaAction, RegionMetaActionList, }; use crate::manifest::manager::RegionManifestManager; use crate::metrics::manifest_size::RegionManifestSizeMetric; @@ -101,6 +101,12 @@ pub enum RegionLeaderState { Staging, /// The region is entering staging mode. - write requests will be stalled. EnteringStaging, + /// The region is committing its staged edits in `end_staging`. Writes stall until + /// `StagingEnded` has the worker handle them, after the region leaves this state. `set_role` + /// ignores role changes, since `leave_ending_staging` works out the next state from the + /// commit result, a downgrade begun meanwhile and the Leader lease. Graceful role changes + /// wait for the manifest lock, which `end_staging` holds until the region leaves. + EndingStaging, /// The region is altering. Altering, /// The region is dropping. @@ -120,6 +126,20 @@ pub enum RegionRoleState { } impl RegionRoleState { + /// Returns whether a region in this state can replay pending tails: a leader that is + /// Writable, Altering or Editing. A region with pending tails cannot enter staging; a + /// downgrading one is leaving. + pub(crate) fn can_replay_tails(self) -> bool { + matches!( + self, + RegionRoleState::Leader( + RegionLeaderState::Writable + | RegionLeaderState::Altering + | RegionLeaderState::Editing + ) + ) + } + /// Converts the region role state to leader state if it is a leader state. pub fn into_leader_state(self) -> Option { match self { @@ -128,6 +148,13 @@ impl RegionRoleState { } } + /// Returns whether the region is staging or ending staging. A staged edit belongs to the + /// session too, which only [MitoRegion::in_staging_session] can tell. + pub(crate) fn in_staging_session(self) -> bool { + self == Self::Leader(RegionLeaderState::Staging) + || self == Self::Leader(RegionLeaderState::EndingStaging) + } + pub(crate) fn as_str(&self) -> &'static str { match self { RegionRoleState::Follower => "Follower", @@ -136,6 +163,7 @@ impl RegionRoleState { RegionRoleState::Leader(RegionLeaderState::EnteringStaging) => { "Leader(EnteringStaging)" } + RegionRoleState::Leader(RegionLeaderState::EndingStaging) => "Leader(EndingStaging)", RegionRoleState::Leader(RegionLeaderState::Altering) => "Leader(Altering)", RegionRoleState::Leader(RegionLeaderState::Dropping) => "Leader(Dropping)", RegionRoleState::Leader(RegionLeaderState::Truncating) => "Leader(Truncating)", @@ -385,14 +413,15 @@ impl MitoRegion { /// Returns whether the region should abort index building. pub(crate) fn should_abort_index(&self) -> bool { - matches!( - self.manifest_ctx.state.load(), - RegionRoleState::Follower - | RegionRoleState::Leader(RegionLeaderState::Dropping) - | RegionRoleState::Leader(RegionLeaderState::Truncating) - | RegionRoleState::Leader(RegionLeaderState::Downgrading) - | RegionRoleState::Leader(RegionLeaderState::Staging) - ) + let state = self.manifest_ctx.state.load(); + state.in_staging_session() + || matches!( + state, + RegionRoleState::Follower + | RegionRoleState::Leader(RegionLeaderState::Dropping) + | RegionRoleState::Leader(RegionLeaderState::Truncating) + | RegionRoleState::Leader(RegionLeaderState::Downgrading) + ) } /// Returns whether the region is downgrading. @@ -408,10 +437,11 @@ impl MitoRegion { self.manifest_ctx.state.load() == RegionRoleState::Leader(RegionLeaderState::Staging) } - /// Returns whether the region is in staging mode, counting a staged edit in progress. - pub(crate) fn is_staging_or_staged_edit(&self) -> bool { + /// Returns whether the region is in a staging session: staging, ending staging, or in a + /// staged edit. + pub(crate) fn in_staging_session(&self) -> bool { match self.manifest_ctx.state.load() { - RegionRoleState::Leader(RegionLeaderState::Staging) => true, + state if state.in_staging_session() => true, RegionRoleState::Leader(RegionLeaderState::Editing) => { self.manifest_ctx.staging_partition_info().is_some() } @@ -430,6 +460,12 @@ impl MitoRegion { self.manifest_ctx.state.load() == RegionRoleState::Leader(RegionLeaderState::Truncating) } + /// Returns whether the region can replay pending tails (see + /// [RegionRoleState::can_replay_tails]). + pub(crate) fn can_replay_tails(&self) -> bool { + self.manifest_ctx.state.load().can_replay_tails() + } + pub fn region_id(&self) -> RegionId { self.region_id } @@ -457,15 +493,16 @@ impl MitoRegion { self.manifest_ctx.state.load() } - /// Sets the region role state. - pub(crate) fn set_role(&self, next_role: RegionRole) { - self.manifest_ctx.set_role(next_role, self.region_id); + /// Sets the region role state. Returns the state this call replaced, None if it left the + /// state unchanged. + pub(crate) fn set_role(&self, next_role: RegionRole) -> Option { + self.manifest_ctx.set_role(next_role, self.region_id) } pub(crate) fn region_role(&self) -> RegionRole { match self.state() { RegionRoleState::Follower => RegionRole::Follower, - RegionRoleState::Leader(RegionLeaderState::Staging) => RegionRole::StagingLeader, + state if state.in_staging_session() => RegionRole::StagingLeader, RegionRoleState::Leader(RegionLeaderState::Downgrading) => { RegionRole::DowngradingLeader } @@ -512,7 +549,7 @@ impl MitoRegion { &self, manager: &mut RwLockWriteGuard<'_, RegionManifestManager>, ) -> Result<()> { - manager.store().clear_staging_manifests().await?; + manager.clear_staging_manifest_and_dir().await?; self.compare_exchange_state( RegionLeaderState::Writable, @@ -542,7 +579,7 @@ impl MitoRegion { /// Leaves staging without committing it, abandoning a staged edit in progress. /// /// You should call this method in the worker loop, which is what restores the state of an - /// edit, and only when [Self::is_staging_or_staged_edit] holds. + /// edit, and only when [Self::in_staging_session] holds. pub(crate) fn abort_staging(&self) { let aborted = self .manifest_ctx @@ -563,9 +600,13 @@ impl MitoRegion { } /// Sets the region role state gracefully. This acquires the manifest write lock. + /// + /// Ending staging awaits `drain_stalled_writes` (see [Self::end_staging]); other changes + /// drop it. pub(crate) async fn set_role_state_gracefully( &self, state: SettableRegionRoleState, + drain_stalled_writes: impl Future, ) -> Result<()> { let mut manager: RwLockWriteGuard<'_, RegionManifestManager> = self.manifest_ctx.manifest_manager.write().await; @@ -591,8 +632,9 @@ impl MitoRegion { match current_state { RegionRoleState::Leader(RegionLeaderState::Staging) => { info!("Exiting staging mode for region {}", self.region_id); - // Use the success exit path that merges all staged manifests - self.exit_staging_on_success(&mut manager).await? + let pending = self.end_staging(manager, drain_stalled_writes).await?; + manager = self.manifest_ctx.manifest_manager.write().await; + pending } RegionRoleState::Leader(RegionLeaderState::Writable) => { // Already in desired state - no-op @@ -616,6 +658,7 @@ impl MitoRegion { // Only allowed from writable leader state match current_state { RegionRoleState::Leader(RegionLeaderState::Writable) => { + self.version().ensure_no_pending_tails()?; info!("Entering staging mode for region {}", self.region_id); self.set_staging(&mut manager).await?; } @@ -936,6 +979,8 @@ impl MitoRegion { let manager = self.manifest_ctx.manifest_manager.read().await; (manager.manifest(), manager.staging_manifest()) }; + // A region that left staging without a commit may still hold its staged files. + let staging = staging.filter(|_| self.in_staging_session()); let normal_files = manifest.files.values().map(|meta| (false, meta)); let staging_files = staging.iter().flat_map(|s| s.files.values()).map(|meta| (true, meta)); for (index, (is_staging, meta)) in normal_files.chain(staging_files).enumerate() { @@ -1037,58 +1082,100 @@ impl MitoRegion { file_ids .iter() - .map(|file_id| manifest_files.get(file_id).cloned()) + .map(|file_id| { + manifest_files + .get(file_id) + .map(|meta| meta.as_ref().clone()) + }) .collect::>() } - /// Returns all live SST file metas and the current manifest version from - /// the region manifest, merging both the normal and staging manifests. + /// Returns all live SST file metas and the normal manifest version, merging in the staging + /// manifest while the region is in a staging session. /// - /// While the region is in staging mode (e.g. during region copy/migration), - /// the authoritative live file set lives in the staging manifest, so this - /// method merges both — matching the semantics of [`manifest_sst_entries`]. - /// - /// The returned manifest version is the staging version when a staging - /// manifest is present, otherwise the normal manifest version. + /// In a staging session (e.g. during region copy/migration), the authoritative live file set + /// lives in the staging manifest, so this method merges both — matching the semantics of + /// [`manifest_sst_entries`]. Outside a session it ignores staged files, since a region that + /// left staging may still hold them. pub async fn all_manifest_files(&self) -> (Vec, ManifestVersion) { let manifest = self.manifest_ctx.manifest().await; let staging = self .manifest_ctx .staging_manifest() .await - .map(|m| (m.files.clone(), m.manifest_version)); - - let version = staging - .as_ref() - .map(|(_, v)| *v) - .unwrap_or(manifest.manifest_version); + .filter(|_| self.in_staging_session()) + .map(|m| m.files.clone()); let files = match staging { - Some((staging_files, _)) => { + Some(staging_files) => { let merged = manifest .files .clone() .into_iter() .chain(staging_files) .collect::>(); - merged.into_values().collect() + merged + .into_values() + .map(|meta| meta.as_ref().clone()) + .collect() } - None => manifest.files.values().cloned().collect(), + None => manifest + .files + .values() + .map(|meta| meta.as_ref().clone()) + .collect(), }; - (files, version) + (files, manifest.manifest_version) } - /// Exit staging mode successfully by merging all staged manifests and making them visible. - /// Merges staged manifest actions into the live manifest and exits staging mode. + /// Commits the staged edits as one manifest version and ends staging (see + /// [Self::end_staging_locked]) under the manifest write lock `manager`, then drops the lock, + /// awaits `drain_stalled_writes` and clears the staging directory. The drain waits for the + /// worker to handle the writes it stalled, and the worker may be waiting for the lock, so the + /// drain is never awaited under the lock. + /// Returns the receipt of the commit, as [Self::end_staging_locked] does. + pub(crate) async fn end_staging( + &self, + mut manager: RwLockWriteGuard<'_, RegionManifestManager>, + drain_stalled_writes: impl Future, + ) -> Result> { + let result = self.end_staging_locked(&mut manager).await; + drop(manager); + + // The worker cannot be awaited under the lock, and its writes need not wait for the + // slower clear of the staging directory. + drain_stalled_writes.await; + let pending = result?; + self.clear_ended_staging().await; + Ok(pending) + } + + /// Clears the staging directory after staging ended, unless the region was closed (it may be + /// open again by now) or a new staging has begun. + async fn clear_ended_staging(&self) { + let mut manager = self.manifest_ctx.manifest_manager.write().await; + if manager.is_stopped() || self.is_enter_staging() || self.in_staging_session() { + return; + } + if let Err(e) = manager.clear_staging_manifest_and_dir().await { + error!(e; "Failed to clear staging manifest dir for region {}", self.region_id); + } + } + + /// Commits the staged edits as one manifest version and ends staging. /// - /// The caller must hold the manifest write lock and pass it via `manager`. - /// Returns `Ok(Some(pending))` when staging manifests were merged, or - /// `Ok(None)` when there were no staged manifests to merge. + /// The caller must hold the manifest write lock and pass it via `manager`. The commit runs in + /// [RegionLeaderState::EndingStaging], which stalls writes and holds role changes back (see + /// [ManifestContext::leave_ending_staging] for the state it leaves for). If the region stays + /// staging, its staged edits are kept for a retry and the error is retryable; otherwise they + /// are dropped. + /// + /// Returns `Ok(None)` when there was nothing to commit. /// /// **Important:** [`fire`](PendingManifestHook::fire) the receipt only after /// dropping the lock — the hook may read the manifest and deadlock otherwise. - pub(crate) async fn exit_staging_on_success( + pub(crate) async fn end_staging_locked( &self, manager: &mut RwLockWriteGuard<'_, RegionManifestManager>, ) -> Result> { @@ -1102,47 +1189,42 @@ impl MitoRegion { } ); - // Merge all staged manifest actions - let merged_actions = match manager.merge_staged_actions(current_state).await? { - Some(actions) => actions, - None => { - info!( - "No staged manifests to merge for region {}, exiting staging mode without changes", - self.region_id - ); - // Even if no manifests to merge, we still need to exit staging mode - self.exit_staging()?; - return Ok(None); - } + let Some(staged_actions) = manager.staged_actions() else { + info!( + "No staged actions for region {}, exiting staging mode without changes", + self.region_id + ); + self.exit_staging()?; + return Ok(None); }; - let expect_change = merged_actions.actions.iter().any(|a| a.is_change()); - let expect_partition_expr_change = merged_actions + let expect_change = staged_actions.actions.iter().any(|a| a.is_change()); + let expect_partition_expr_change = staged_actions .actions .iter() .any(|a| a.is_partition_expr_change()); - let expect_edit = merged_actions.actions.iter().any(|a| a.is_edit()); + let expect_edit = staged_actions.actions.iter().any(|a| a.is_edit()); ensure!( !(expect_change && expect_partition_expr_change), UnexpectedSnafu { - reason: "unexpected both change and partition expr change actions in merged actions" + reason: "unexpected both change and partition expr change actions in staged actions" } ); ensure!( expect_change || expect_partition_expr_change, UnexpectedSnafu { - reason: "expect a change or partition expr change action in merged actions" + reason: "expect a change or partition expr change action in staged actions" } ); ensure!( expect_edit, UnexpectedSnafu { - reason: "expect an edit action in merged actions" + reason: "expect an edit action in staged actions" } ); - let (merged_partition_expr_change, merged_change, merged_edit) = - merged_actions.clone().split_region_change_and_edit(); - if let Some(change) = &merged_change { + if let Some(RegionMetaAction::Change(change)) = + staged_actions.actions.iter().find(|a| a.is_change()) + { // In staging exit we only allow metadata-only updates. A `Change` // action is accepted only when column definitions are unchanged; // otherwise it is treated as a schema change and rejected. @@ -1155,38 +1237,47 @@ impl MitoRegion { ); } - // Submit merged actions using the manifest manager's update method. - // Pass `false` so it saves to normal directory, not staging. - let pending = self - .manifest_ctx - .update_locked(manager, merged_actions, false) - .await?; - let new_version = pending.version(); - info!( - "Successfully submitted merged staged manifests for region {}, new version: {}", - self.region_id, new_version - ); - - // Apply the merged changes to in-memory version control - if let Some(change) = merged_partition_expr_change { - let mut new_metadata = self.version().metadata.as_ref().clone(); - new_metadata.set_partition_expr(change.partition_expr); - self.version_control.alter_metadata(new_metadata.into()); + let ending_started_at = self.manifest_ctx.begin_ending_staging(self.region_id)?; + let mut result = self.manifest_ctx.commit_staged_locked(manager).await; + if let Ok((pending, committed)) = &mut result { + info!( + "Successfully committed staged manifests for region {}, new version: {}", + self.region_id, + pending.version() + ); + // Apply the committed changes to in-memory version control + let (staged_partition_expr_change, staged_change, staged_edit) = + std::mem::take(committed).split_region_change_and_edit(); + if let Some(change) = staged_partition_expr_change { + let mut new_metadata = self.version().metadata.as_ref().clone(); + new_metadata.set_partition_expr(change.partition_expr); + self.version_control.alter_metadata(new_metadata.into()); + } + if let Some(change) = staged_change { + self.version_control.alter_metadata(change.metadata); + } + self.version_control + .apply_edit(Some(staged_edit), &[], self.file_purger.clone()); } - if let Some(change) = merged_change { - self.version_control.alter_metadata(change.metadata); + let result = result.map(|(pending, _)| pending); + let next_state = + self.manifest_ctx + .leave_ending_staging(self.region_id, ending_started_at, &result); + if next_state != RegionRoleState::Leader(RegionLeaderState::Staging) { + // Only a retry commits the staged actions. The listings gate on the state and do not + // rely on this. + manager.unset_staging_manifest(); } - self.version_control - .apply_edit(Some(merged_edit), &[], self.file_purger.clone()); - // Clear all staging manifests and transit state - if let Err(e) = manager.clear_staging_manifest_and_dir().await { - error!(e; "Failed to clear staging manifest dir for region {}", self.region_id); + match result { + // A retry of the request commits the kept edits. + Err(e) if next_state == RegionRoleState::Leader(RegionLeaderState::Staging) => { + Err(Arc::new(e)).context(EndStagingSnafu { + region_id: self.region_id, + }) + } + result => result.map(Some), } - self.exit_staging()?; - - // Return the hook payload; the caller invokes the hook after dropping the lock. - Ok(Some(pending)) } pub fn expected_partition_expr_version(&self) -> u64 { @@ -1237,6 +1328,17 @@ pub(crate) enum IndexPublication { Stale(IndexPublicationStale), } +/// Result of publishing the result of a pending tail replay to the manifest. +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum TailPublication { + /// The result was committed. + Committed(ManifestVersion), + /// The region is altering or applying an edit, so the result waits. + Busy, + /// The region is no longer a writable leader with the replayed tails. + Stale, +} + /// Why an index publication became stale. #[derive(Debug, PartialEq, Eq)] pub(crate) enum IndexPublicationStale { @@ -1279,6 +1381,10 @@ pub(crate) struct ManifestContext { /// Decides whether a promotion to Leader applies. The lock covers checking or changing it and /// the role change. promotion_fence: Mutex, + /// When a catchup last made the region writable, in the unit of [Deadline], until a Leader + /// lease of a heartbeat sent after it applies; 0 otherwise. While it is set, the region keeps + /// the unread part of another node's listed tails. + catchup_promoted_at: AtomicU64, /// Deadline of the region's latest Leader or StagingLeader lease from the metasrv, which is how /// long the Leader role stays valid. The engine extends it from those leases, and writes check /// it. @@ -1308,6 +1414,7 @@ impl ManifestContext { downgrade_started_at: Instant::now(), reload_pending: false, }), + catchup_promoted_at: AtomicU64::new(0), leader_deadline: Deadline::default(), node_lease, staging_partition_info: Mutex::new(None), @@ -1356,6 +1463,67 @@ impl ManifestContext { Ok(()) } + /// Switches a staging region to [RegionLeaderState::EndingStaging] and returns the instant + /// it switched. + /// + /// It holds the lock of [Self::begin_downgrade], so a downgrade either has left staging + /// already, or begins after the returned instant. + pub(crate) fn begin_ending_staging(&self, region_id: RegionId) -> Result { + let _fence = self.promotion_fence.lock().unwrap(); + self.state + .compare_exchange( + RegionRoleState::Leader(RegionLeaderState::Staging), + RegionRoleState::Leader(RegionLeaderState::EndingStaging), + ) + .map_err(|actual| { + RegionStateSnafu { + region_id, + state: actual, + expect: RegionRoleState::Leader(RegionLeaderState::Staging), + } + .build() + })?; + Ok(Instant::now()) + } + + /// Leaves [RegionLeaderState::EndingStaging], entered at `ending_started_at`, after the end + /// commit returned `result`. Returns the state the region leaves it for, which is in order: + /// - Follower if the commit conflicted with another writer or failed on an expired lease; + /// - Downgrading if a downgrade began meanwhile; + /// - Follower if the Leader lease has expired; + /// - Writable if the commit succeeded; + /// - Staging otherwise, keeping the staging partition info for a retry. + pub(crate) fn leave_ending_staging( + &self, + region_id: RegionId, + ending_started_at: Instant, + result: &Result, + ) -> RegionRoleState { + let fence = self.promotion_fence.lock().unwrap(); + let lease_expired = self.node_lease.is_enabled() && self.is_leader_lease_expired(); + let next_state = match result { + Err(Error::ManifestVersionConflict { .. } | Error::RegionLeaseExpired { .. }) => { + RegionRoleState::Follower + } + _ if fence.downgrade_started_at > ending_started_at => { + RegionRoleState::Leader(RegionLeaderState::Downgrading) + } + _ if lease_expired => RegionRoleState::Follower, + Ok(_) => RegionRoleState::Leader(RegionLeaderState::Writable), + Err(_) => RegionRoleState::Leader(RegionLeaderState::Staging), + }; + // Role changes leave a region that is ending staging alone. + self.state.store(next_state); + if next_state != RegionRoleState::Leader(RegionLeaderState::Staging) { + self.clear_staging_partition_info(); + } + info!( + "Region {} ended staging, next state: {:?}", + region_id, next_state + ); + next_state + } + pub(crate) async fn manifest_version(&self) -> ManifestVersion { self.manifest_manager .read() @@ -1368,6 +1536,60 @@ impl ManifestContext { self.manifest_manager.read().await.has_update().await } + /// Waits for the in-flight checkpoint of a stopped manager. A stopped manager starts no + /// checkpoint, so this instance no longer writes the manifest directory afterwards. + /// + /// Callers must not hold the manifest lock: this takes its read lock, and the lock is not + /// reentrant. Under the lock, use [RegionManifestManager::wait_for_pending_checkpoint] + /// instead. The wait itself holds no lock. + pub(crate) async fn wait_for_checkpoint_after_stop(&self) { + let done = { + let manager = self.manifest_manager.read().await; + debug_assert!(manager.is_stopped()); + manager.pending_checkpoint_completion() + }; + done.await; + } + + /// Returns the installed manifest version if the storage holds a newer one. The version is + /// read under the manifest read lock, which is not held across the listing, so a caller + /// rechecks it under the write lock before acting on it. It is the installed version, not + /// `last_version`, which a failed commit leaves ahead of it. + pub(crate) async fn installed_version_if_updated(&self) -> Result> { + let (store, installed) = { + let manager = self.manifest_manager.read().await; + (manager.store(), manager.manifest().manifest_version) + }; + // Lists without the read lock: held across the listing, it would let a queued writer + // block every later reader. + Ok(store + .has_version_after(installed) + .await? + .then_some(installed)) + } + + /// Fences the region for a reload, see [Self::fence_for_reload], if it is still a Writable + /// leader at `expected_version`. Returns whether it fenced the region. + pub(crate) async fn fence_for_reload_if_unchanged( + &self, + manager: &mut RegionManifestManager, + region_id: RegionId, + expected_version: ManifestVersion, + ) -> bool { + let state = self.state.load(); + let version = manager.manifest().manifest_version; + if state != RegionRoleState::Leader(RegionLeaderState::Writable) + || version != expected_version + { + info!( + "Region {region_id} is {state:?} at manifest version {version}, not a writable leader at version {expected_version}, not fencing it" + ); + return false; + } + self.fence_for_reload(manager, region_id).await; + true + } + /// Returns the current region role state. pub(crate) fn current_state(&self) -> RegionRoleState { self.state.load() @@ -1494,7 +1716,12 @@ impl ManifestContext { )); } - if manifest.files.get(&source.file_meta.file_id) != Some(&source.file_meta) { + if manifest + .files + .get(&source.file_meta.file_id) + .map(AsRef::as_ref) + != Some(&source.file_meta) + { return Ok(IndexPublication::Stale( IndexPublicationStale::SourceChanged, )); @@ -1532,6 +1759,28 @@ impl ManifestContext { }) } + /// Publishes the result of a replay of the pending tails `replayed`, if the region is + /// still a writable leader whose manifest records exactly those tails. + pub(crate) async fn update_manifest_for_tails( + &self, + replayed: &[PendingTail], + action_list: RegionMetaActionList, + ) -> Result { + let manager = self.manifest_manager.write().await; + match self.state.load() { + RegionRoleState::Leader(RegionLeaderState::Writable) => {} + RegionRoleState::Leader(RegionLeaderState::Altering | RegionLeaderState::Editing) => { + return Ok(TailPublication::Busy); + } + _ => return Ok(TailPublication::Stale), + } + if manager.manifest().pending_tails != replayed { + return Ok(TailPublication::Stale); + } + let manifest_version = self.update_and_fire(manager, action_list, false).await?; + Ok(TailPublication::Committed(manifest_version)) + } + /// Performs a manifest write under a caller-held write lock and returns a /// [`PendingManifestHook`] to [`fire`](PendingManifestHook::fire) after /// dropping the lock. It calls @@ -1540,7 +1789,7 @@ impl ManifestContext { /// /// Does not validate state or edit applicability — the caller must do so /// while still holding the lock (see `update_manifest_with_state_check` - /// and `MitoRegion::exit_staging_on_success`). + /// and `MitoRegion::end_staging_locked`). pub(crate) async fn update_locked( &self, manager: &mut RegionManifestManager, @@ -1555,26 +1804,15 @@ impl ManifestContext { // checkpoint barrier. let allow_checkpoint = !is_staging && self.state.load() != RegionRoleState::Leader(RegionLeaderState::Downgrading); - // A writer past its Leader deadline may no longer hold the region, so it sends nothing. - let result = if manager.is_conditional_publication() && self.is_leader_lease_expired() { - RegionLeaseExpiredSnafu { region_id }.fail() - } else { - manager - .update_with_verification(action_list, is_staging, allow_checkpoint, || { - self.is_leader_lease_expired() - }) - .await - }; - let version = match result { - Ok(version) => version, - Err(e) => { - error!(e; "Failed to update manifest, region_id: {}", region_id); - if matches!(e, Error::ManifestVersionConflict { .. }) { - self.fence_for_reload(manager, region_id).await; - } - return Err(e); - } - }; + let version = self + .commit_locked(manager, async |manager| { + manager + .update_with_verification(action_list, is_staging, allow_checkpoint, || { + self.is_leader_lease_expired() + }) + .await + }) + .await?; Ok(PendingManifestHook::new( region_id, @@ -1585,6 +1823,57 @@ impl ManifestContext { )) } + /// Commits the staged actions of `manager` as one version under a caller-held write lock. + /// Returns a [`PendingManifestHook`] to [`fire`](PendingManifestHook::fire) after dropping + /// the lock, and the committed actions. + pub(crate) async fn commit_staged_locked( + &self, + manager: &mut RegionManifestManager, + ) -> Result<(PendingManifestHook, RegionMetaActionList)> { + let region_id = manager.manifest().metadata.region_id; + let (version, committed) = self + .commit_locked(manager, async |manager| { + manager + .commit_staged(|| self.is_leader_lease_expired()) + .await + }) + .await?; + let actions_for_hook = self.hook.as_ref().map(|_| committed.clone()); + + let pending = PendingManifestHook::new( + region_id, + actions_for_hook, + version, + self.hook.clone(), + false, + ); + Ok((pending, committed)) + } + + /// Runs the manifest commit `commit` under a caller-held write lock. A writer past its + /// Leader deadline may no longer hold the region, so in conditional publication it sends + /// nothing; a commit that finds a newer manifest fences the region (see + /// [Self::fence_for_reload]). + async fn commit_locked( + &self, + manager: &mut RegionManifestManager, + commit: impl AsyncFnOnce(&mut RegionManifestManager) -> Result, + ) -> Result { + let region_id = manager.manifest().metadata.region_id; + let result = if manager.is_conditional_publication() && self.is_leader_lease_expired() { + RegionLeaseExpiredSnafu { region_id }.fail() + } else { + commit(manager).await + }; + if let Err(e) = &result { + error!(e; "Failed to update manifest, region_id: {}", region_id); + if matches!(e, Error::ManifestVersionConflict { .. }) { + self.fence_for_reload(manager, region_id).await; + } + } + result + } + /// Returns whether the region's Leader deadline has passed. No deadline counts as passed. fn is_leader_lease_expired(&self) -> bool { self.node_lease.now() >= self.leader_deadline.get() @@ -1681,7 +1970,7 @@ impl ManifestContext { .is_none_or(|current| current.level != file::LEVEL_SHARED) }) && edit.files_to_add.iter().all(|file| { manifest.files.get(&file.file_id).is_none_or(|current| { - current.level != file::LEVEL_SHARED || current == file + current.level != file::LEVEL_SHARED || current.as_ref() == file }) }), InvalidRequestSnafu { @@ -1754,29 +2043,67 @@ impl ManifestContext { ) -> bool { // A writable leader needs no change; a staging leader is promoted below. if self.state.load() == RegionRoleState::Leader(RegionLeaderState::Writable) { + self.record_leader_lease(issued_at); return true; } let fence = self.promotion_fence.lock().unwrap(); - issued_at >= fence.downgrade_started_at - && self.promote_unless_reload_pending(&fence, region_id) + if issued_at < fence.downgrade_started_at { + return false; + } + self.record_leader_lease(issued_at); + self.promote_unless_reload_pending(&fence, region_id, false) } /// Promotes a region that caught up to a writable leader. Returns false if the region waits /// for a reload. pub(crate) fn promote_after_catchup(&self, region_id: RegionId) -> bool { let fence = self.promotion_fence.lock().unwrap(); - self.promote_unless_reload_pending(&fence, region_id) + self.promote_unless_reload_pending(&fence, region_id, true) } /// Promotes the region unless it waits for a reload. The caller holds the fence lock. - fn promote_unless_reload_pending(&self, fence: &PromotionFence, region_id: RegionId) -> bool { + fn promote_unless_reload_pending( + &self, + fence: &PromotionFence, + region_id: RegionId, + by_catchup: bool, + ) -> bool { if fence.reload_pending { return false; } + if by_catchup { + // Before the role, so a lease that finds the region writable compares with it. + self.catchup_promoted_at + .store(lease_nanos(Instant::now()).max(1), Ordering::Relaxed); + } self.set_role(RegionRole::Leader, region_id); true } + /// Counts an applied Leader lease of a heartbeat sent at `issued_at` as received since the + /// last catchup promotion. A heartbeat sent before the promotion does not count: its lease + /// may have been granted before an earlier downgrade of this node, which the fast path of + /// [Self::promote_unless_downgrade_started_after] does not check, and the lease of the next + /// heartbeat counts instead. It reads one atomic, since every lease of a writable leader + /// comes here. + pub(crate) fn record_leader_lease(&self, issued_at: Instant) { + let promoted_at = self.catchup_promoted_at.load(Ordering::Relaxed); + if promoted_at != 0 && lease_nanos(issued_at) >= promoted_at { + let _ = self.catchup_promoted_at.compare_exchange( + promoted_at, + 0, + Ordering::Relaxed, + Ordering::Relaxed, + ); + } + } + + /// Returns whether a Leader lease of a heartbeat sent after the last catchup promotion applied, + /// or no catchup promoted the region. + pub(crate) fn leader_lease_received(&self) -> bool { + self.catchup_promoted_at.load(Ordering::Relaxed) == 0 + } + /// Sets the [`RegionRole`]. /// /// ```text @@ -1810,7 +2137,17 @@ impl ManifestContext { /// From `Downgrading Leader`: /// - `Downgrading Leader -> Leader` /// - `Downgrading Leader -> Follower` - pub(crate) fn set_role(&self, next_role: RegionRole, region_id: RegionId) { + /// + /// A region that is ending staging keeps its state: the end of staging reads the Leader + /// deadline and the downgrade start when it leaves that state. + /// + /// Returns the state this call replaced, None if it left the state unchanged. It comes from + /// the atomic operation that replaced it. + pub(crate) fn set_role( + &self, + next_role: RegionRole, + region_id: RegionId, + ) -> Option { match next_role { RegionRole::Follower => { if self @@ -1822,19 +2159,26 @@ impl ManifestContext { region_id, RegionRoleState::Leader(RegionLeaderState::Staging) ); - return; + return Some(RegionRoleState::Leader(RegionLeaderState::Staging)); } match self.state.fetch_update(|state| { - if !matches!(state, RegionRoleState::Follower) { + if !matches!( + state, + RegionRoleState::Follower + | RegionRoleState::Leader(RegionLeaderState::EndingStaging) + ) { Some(RegionRoleState::Follower) } else { None } }) { - Ok(state) => info!( - "Convert region {} to follower, previous role state: {:?}", - region_id, state - ), + Ok(state) => { + info!( + "Convert region {} to follower, previous role state: {:?}", + region_id, state + ); + Some(state) + } Err(state) => { if state != RegionRoleState::Follower { warn!( @@ -1842,6 +2186,7 @@ impl ManifestContext { region_id, state ) } + None } } } @@ -1858,7 +2203,7 @@ impl ManifestContext { region_id, RegionRoleState::Leader(RegionLeaderState::Staging) ); - return; + return Some(RegionRoleState::Leader(RegionLeaderState::Staging)); } match self.state.fetch_update(|state| { if matches!( @@ -1871,10 +2216,13 @@ impl ManifestContext { None } }) { - Ok(state) => info!( - "Convert region {} to leader, previous role state: {:?}", - region_id, state - ), + Ok(state) => { + info!( + "Convert region {} to leader, previous role state: {:?}", + region_id, state + ); + Some(state) + } Err(state) => { // These states already have the leader role; keep their ongoing operation. if !matches!( @@ -1886,6 +2234,7 @@ impl ManifestContext { | RegionLeaderState::Truncating | RegionLeaderState::Editing | RegionLeaderState::EnteringStaging + | RegionLeaderState::EndingStaging ) ) { warn!( @@ -1893,6 +2242,7 @@ impl ManifestContext { region_id, state ) } + None } } } @@ -1901,6 +2251,7 @@ impl ManifestContext { "Ignore direct conversion of region {} to staging leader; staging requires the dedicated workflow", region_id ); + None } RegionRole::DowngradingLeader => { if self @@ -1915,16 +2266,19 @@ impl ManifestContext { region_id, RegionRoleState::Leader(RegionLeaderState::Staging) ); - return; + return Some(RegionRoleState::Leader(RegionLeaderState::Staging)); } match self.state.compare_exchange( RegionRoleState::Leader(RegionLeaderState::Writable), RegionRoleState::Leader(RegionLeaderState::Downgrading), ) { - Ok(state) => info!( - "Convert region {} to downgrading region, previous role state: {:?}", - region_id, state - ), + Ok(state) => { + info!( + "Convert region {} to downgrading region, previous role state: {:?}", + region_id, state + ); + Some(state) + } Err(state) => { if state != RegionRoleState::Leader(RegionLeaderState::Downgrading) { warn!( @@ -1932,6 +2286,7 @@ impl ManifestContext { region_id, state ) } + None } } } @@ -2348,18 +2703,20 @@ mod tests { use object_store::services::Fs; use store_api::logstore::provider::Provider; use store_api::region_engine::RegionRole; - use store_api::region_request::PathType; + use store_api::region_request::{PathType, StagingPartitionDirective}; use store_api::storage::{FileId, RegionId}; use crate::access_layer::AccessLayer; - use crate::error::Error; + use crate::error::{Error, ManifestVersionConflictSnafu, Result, UnexpectedSnafu}; use crate::manifest::action::{ RegionChange, RegionEdit, RegionMetaAction, RegionMetaActionList, RegionPartitionExprChange, }; use crate::manifest::manager::{RegionManifestManager, RegionManifestOptions}; + use crate::manifest::storage::delta_file; + use crate::manifest::storage::staging::staging_manifest_path; use crate::region::{ IndexBuildSource, IndexPublication, IndexPublicationStale, ManifestContext, ManifestStats, - MitoRegion, RegionLeaderState, RegionRoleState, RegionStats, + MitoRegion, RegionLeaderState, RegionRoleState, RegionStats, StagingPartitionInfo, }; use crate::sst::FormatType; use crate::sst::index::intermediate::IntermediateManager; @@ -2459,6 +2816,7 @@ mod tests { metadata.clone(), 0, RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "".to_string(), object_store: env.access_layer.object_store().clone(), compress_type: CompressionType::Uncompressed, @@ -2629,6 +2987,23 @@ mod tests { assert!(rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap()); } + #[tokio::test] + async fn test_lease_sent_before_catchup_promotion_does_not_count() { + let env = SchedulerEnv::new().await; + let region = build_test_region(&env).await; + let manifest_ctx = ®ion.manifest_ctx; + let sent_before = std::time::Instant::now(); + region.set_role(RegionRole::Follower); + std::thread::sleep(std::time::Duration::from_millis(1)); + assert!(manifest_ctx.promote_after_catchup(region.region_id)); + + assert!(manifest_ctx.promote_unless_downgrade_started_after(sent_before, region.region_id)); + assert!(!manifest_ctx.leader_lease_received()); + let sent_after = std::time::Instant::now(); + assert!(manifest_ctx.promote_unless_downgrade_started_after(sent_after, region.region_id)); + assert!(manifest_ctx.leader_lease_received()); + } + #[tokio::test] async fn test_compaction_update_manifest_allows_editing_state() { let env = SchedulerEnv::new().await; @@ -2748,7 +3123,8 @@ mod tests { .manifest() .await .files - .get(&source.file_id), + .get(&source.file_id) + .map(AsRef::as_ref), Some(&newer_committed) ); } @@ -2809,30 +3185,38 @@ mod tests { let manifest = region.manifest_ctx.manifest().await; assert_eq!(manifest.metadata.schema_version, old_schema_version + 1); - assert_eq!(manifest.files.get(&source_meta.file_id), Some(&source_meta)); + assert_eq!( + manifest.files.get(&source_meta.file_id).map(AsRef::as_ref), + Some(&source_meta) + ); } #[tokio::test] - async fn test_exit_staging_partition_expr_change_and_edit_success() { + async fn test_end_staging_partition_expr_change_and_edit_success() { let env = SchedulerEnv::new().await; let region = build_test_region(&env).await; let mut manager = region.manifest_ctx.manifest_manager.write().await; region.set_staging(&mut manager).await.unwrap(); - manager - .update( - RegionMetaActionList::new(vec![ - RegionMetaAction::PartitionExprChange(RegionPartitionExprChange { - partition_expr: Some("expr_a".to_string()), - }), - RegionMetaAction::Edit(empty_edit()), - ]), - true, - ) - .await - .unwrap(); + let version = manager.last_version(); + for action in [ + RegionMetaAction::PartitionExprChange(RegionPartitionExprChange { + partition_expr: Some("expr_a".to_string()), + }), + RegionMetaAction::Edit(empty_edit()), + ] { + manager + .update(RegionMetaActionList::with_action(action), true) + .await + .unwrap(); + } + // Staged actions take no version. + assert_eq!(manager.last_version(), version); - let _hook_payload = region.exit_staging_on_success(&mut manager).await.unwrap(); + let pending = region.end_staging_locked(&mut manager).await.unwrap(); + assert_eq!(pending.unwrap().version(), version + 1); + assert_eq!(manager.last_version(), version + 1); + assert!(manager.staged_actions().is_none()); drop(manager); assert_eq!( @@ -2846,7 +3230,7 @@ mod tests { } #[tokio::test] - async fn test_exit_staging_change_with_same_columns_success() { + async fn test_end_staging_change_with_same_columns_success() { let env = SchedulerEnv::new().await; let region = build_test_region(&env).await; @@ -2871,7 +3255,7 @@ mod tests { .await .unwrap(); - let _hook_payload = region.exit_staging_on_success(&mut manager).await.unwrap(); + let _hook_payload = region.end_staging_locked(&mut manager).await.unwrap(); drop(manager); assert_eq!( @@ -2885,7 +3269,7 @@ mod tests { } #[tokio::test] - async fn test_exit_staging_change_with_different_columns_fails() { + async fn test_end_staging_change_with_different_columns_fails() { let env = SchedulerEnv::new().await; let region = build_test_region(&env).await; @@ -2910,12 +3294,12 @@ mod tests { .await .unwrap(); - let result = region.exit_staging_on_success(&mut manager).await; + let result = region.end_staging_locked(&mut manager).await; assert!(matches!(result, Err(Error::Unexpected { .. }))); } #[tokio::test] - async fn test_exit_staging_partition_expr_change_and_change_conflict_fails() { + async fn test_end_staging_partition_expr_change_and_change_conflict_fails() { let env = SchedulerEnv::new().await; let region = build_test_region(&env).await; @@ -2943,10 +3327,122 @@ mod tests { .await .unwrap(); - let result = region.exit_staging_on_success(&mut manager).await; + let result = region.end_staging_locked(&mut manager).await; assert!(matches!(result, Err(Error::Unexpected { .. }))); } + #[tokio::test] + async fn test_end_staging_ignores_files_in_staging_dir() { + let env = SchedulerEnv::new().await; + let region = build_test_region(&env).await; + + let mut manager = region.manifest_ctx.manifest_manager.write().await; + region.set_staging(&mut manager).await.unwrap(); + // A late writer leaves a delta in the staging directory. + let late_file = crate::sst::file::FileMeta { + file_id: FileId::random(), + ..Default::default() + }; + let late_edit = RegionEdit { + files_to_add: vec![late_file.clone()], + ..Default::default() + }; + region + .access_layer + .object_store() + .write( + &format!( + "{}{}", + staging_manifest_path(manager.store().manifest_dir()), + delta_file(1) + ), + RegionMetaActionList::with_action(RegionMetaAction::Edit(late_edit)) + .encode() + .unwrap(), + ) + .await + .unwrap(); + manager + .update( + RegionMetaActionList::new(vec![ + RegionMetaAction::PartitionExprChange(RegionPartitionExprChange { + partition_expr: Some("expr_a".to_string()), + }), + RegionMetaAction::Edit(empty_edit()), + ]), + true, + ) + .await + .unwrap(); + + let _hook_payload = region.end_staging_locked(&mut manager).await.unwrap(); + + assert!(!manager.manifest().files.contains_key(&late_file.file_id)); + } + + #[tokio::test] + async fn test_ending_staging_holds_role_changes() { + let env = SchedulerEnv::new().await; + let version_control = VersionControlBuilder::new().build(); + let ctx = env + .mock_manifest_context(version_control.current().version.metadata.clone()) + .await; + let region_id = RegionId::new(1024, 0); + let staging = RegionRoleState::Leader(RegionLeaderState::Staging); + let committed: Result<()> = Ok(()); + let unknown: Result<()> = UnexpectedSnafu { reason: "mock" }.fail(); + let conflict: Result<()> = ManifestVersionConflictSnafu { + version: 1u64, + manifest_dir: "manifest/", + reason: "mock", + } + .fail(); + // Ends staging with `result`, running `during` while the commit runs. + let end = |result: &Result<()>, during: &dyn Fn()| { + ctx.state.store(staging); + if ctx.staging_partition_info().is_none() { + ctx.set_staging_partition_info(StagingPartitionInfo::from_partition_directive( + StagingPartitionDirective::RejectAllWrites, + )); + } + let started = ctx.begin_ending_staging(region_id).unwrap(); + during(); + assert_eq!( + ctx.state.load(), + RegionRoleState::Leader(RegionLeaderState::EndingStaging) + ); + let state = ctx.leave_ending_staging(region_id, started, result); + // Only a return to staging keeps the partition info for the retry. + assert_eq!(ctx.staging_partition_info().is_some(), state == staging); + state + }; + let set_roles = || { + for role in [ + RegionRole::Follower, + RegionRole::Leader, + RegionRole::StagingLeader, + RegionRole::DowngradingLeader, + ] { + ctx.set_role(role, region_id); + } + }; + + assert_eq!(end(&unknown, &set_roles), staging); + assert_eq!( + end(&committed, &|| ctx.begin_downgrade(region_id)), + RegionRoleState::Leader(RegionLeaderState::Downgrading) + ); + assert_eq!( + end(&committed, &|| {}), + RegionRoleState::Leader(RegionLeaderState::Writable) + ); + assert_eq!(end(&conflict, &|| {}), RegionRoleState::Follower); + // The Leader lease expired while the commit ran; no deadline counts as expired. + ctx.node_lease.enable(); + assert_eq!(end(&committed, &|| {}), RegionRoleState::Follower); + assert_eq!(end(&unknown, &|| {}), RegionRoleState::Follower); + } + #[tokio::test] async fn test_set_region_state() { let env = SchedulerEnv::new().await; @@ -2957,49 +3453,115 @@ mod tests { .await; let region_id = RegionId::new(1024, 0); + // Each call returns the state it replaced, None if it left the state unchanged. // Leader -> Follower - manifest_ctx.set_role(RegionRole::Follower, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::Follower, region_id), + Some(RegionRoleState::Leader(RegionLeaderState::Writable)) + ); assert_eq!(manifest_ctx.state.load(), RegionRoleState::Follower); + // Follower -> Follower + assert_eq!(manifest_ctx.set_role(RegionRole::Follower, region_id), None); + // Follower -> Leader - manifest_ctx.set_role(RegionRole::Leader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::Leader, region_id), + Some(RegionRoleState::Follower) + ); assert_eq!( manifest_ctx.state.load(), RegionRoleState::Leader(RegionLeaderState::Writable) ); // Direct Leader -> StagingLeader should be ignored. - manifest_ctx.set_role(RegionRole::StagingLeader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::StagingLeader, region_id), + None + ); assert_eq!( manifest_ctx.state.load(), RegionRoleState::Leader(RegionLeaderState::Writable) ); // Leader -> Downgrading Leader - manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id), + Some(RegionRoleState::Leader(RegionLeaderState::Writable)) + ); assert_eq!( manifest_ctx.state.load(), RegionRoleState::Leader(RegionLeaderState::Downgrading) ); // Downgrading Leader -> Follower - manifest_ctx.set_role(RegionRole::Follower, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::Follower, region_id), + Some(RegionRoleState::Leader(RegionLeaderState::Downgrading)) + ); assert_eq!(manifest_ctx.state.load(), RegionRoleState::Follower); // Can't downgrade from follower (Follower -> Downgrading Leader) - manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id), + None + ); assert_eq!(manifest_ctx.state.load(), RegionRoleState::Follower); // Set region role too Downgrading Leader - manifest_ctx.set_role(RegionRole::Leader, region_id); - manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::Leader, region_id), + Some(RegionRoleState::Follower) + ); + assert_eq!( + manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id), + Some(RegionRoleState::Leader(RegionLeaderState::Writable)) + ); assert_eq!( manifest_ctx.state.load(), RegionRoleState::Leader(RegionLeaderState::Downgrading) ); + assert_eq!( + manifest_ctx.set_role(RegionRole::DowngradingLeader, region_id), + None + ); + // Downgrading Leader -> Leader - manifest_ctx.set_role(RegionRole::Leader, region_id); + assert_eq!( + manifest_ctx.set_role(RegionRole::Leader, region_id), + Some(RegionRoleState::Leader(RegionLeaderState::Downgrading)) + ); + assert_eq!( + manifest_ctx.state.load(), + RegionRoleState::Leader(RegionLeaderState::Writable) + ); + + assert_eq!(manifest_ctx.set_role(RegionRole::Leader, region_id), None); + + // Altering Leader -> Follower + let altering = RegionRoleState::Leader(RegionLeaderState::Altering); + manifest_ctx.state.store(altering); + assert_eq!( + manifest_ctx.set_role(RegionRole::Follower, region_id), + Some(altering) + ); + assert_eq!(manifest_ctx.state.load(), RegionRoleState::Follower); + + // Staging Leader -> Follower + let staging = RegionRoleState::Leader(RegionLeaderState::Staging); + manifest_ctx.state.store(staging); + assert_eq!( + manifest_ctx.set_role(RegionRole::Follower, region_id), + Some(staging) + ); + assert_eq!(manifest_ctx.state.load(), RegionRoleState::Follower); + + manifest_ctx.state.store(staging); + assert_eq!( + manifest_ctx.set_role(RegionRole::Leader, region_id), + Some(staging) + ); assert_eq!( manifest_ctx.state.load(), RegionRoleState::Leader(RegionLeaderState::Writable) @@ -3018,6 +3580,7 @@ mod tests { version_control.current().version.metadata.clone(), 0, RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "".to_string(), object_store: env.access_layer.object_store().clone(), compress_type: CompressionType::Uncompressed, @@ -3089,6 +3652,7 @@ mod tests { metadata.clone(), 0, RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "".to_string(), object_store: access_layer.object_store().clone(), compress_type: CompressionType::Uncompressed, @@ -3154,42 +3718,38 @@ mod tests { ); assert!(!region.is_staging()); - // Test staging directory cleanup: Create dirty staging files before entering staging mode + // Entering staging clears the staging directory and the staged actions. { - // Create some dummy staging manifest files to simulate interrupted session - let manager = manifest_ctx.manifest_manager.write().await; - let dummy_actions = RegionMetaActionList::new(vec![]); - let dummy_bytes = dummy_actions.encode().unwrap(); - - // Create dirty staging files with versions 100 and 101 - manager.store().save(100, &dummy_bytes, true).await.unwrap(); - manager.store().save(101, &dummy_bytes, true).await.unwrap(); - drop(manager); - - // Verify dirty files exist before entering staging - let manager = manifest_ctx.manifest_manager.read().await; - let dirty_manifests = manager.store().fetch_staging_manifests().await.unwrap(); - assert_eq!( - dirty_manifests.len(), - 2, - "Should have 2 dirty staging files" - ); - drop(manager); - - // Enter staging mode - this should clean up the dirty files let mut manager = manifest_ctx.manifest_manager.write().await; - region.set_staging(&mut manager).await.unwrap(); - drop(manager); - - // Verify dirty files are cleaned up after entering staging - let manager = manifest_ctx.manifest_manager.read().await; - let cleaned_manifests = manager.store().fetch_staging_manifests().await.unwrap(); - assert_eq!( - cleaned_manifests.len(), - 0, - "Dirty staging files should be cleaned up" + let object_store = region.access_layer.object_store().clone(); + // A delta that an older version staged. + let stale_delta = format!( + "{}{}", + staging_manifest_path(manager.store().manifest_dir()), + delta_file(100) ); - drop(manager); + object_store + .write( + &stale_delta, + RegionMetaActionList::new(vec![]).encode().unwrap(), + ) + .await + .unwrap(); + region.set_staging(&mut manager).await.unwrap(); + assert!(!object_store.exists(&stale_delta).await.unwrap()); + + manager + .update( + RegionMetaActionList::with_action(RegionMetaAction::Edit(empty_edit())), + true, + ) + .await + .unwrap(); + assert!(manager.staged_actions().is_some()); + region.exit_staging().unwrap(); + region.set_staging(&mut manager).await.unwrap(); + assert!(manager.staged_actions().is_none()); + assert!(manager.staging_manifest().is_none()); // Exit staging to restore normal state for remaining tests region.exit_staging().unwrap(); diff --git a/src/mito2/src/region/opener.rs b/src/mito2/src/region/opener.rs index 25167a7d418..f7bfedabedb 100644 --- a/src/mito2/src/region/opener.rs +++ b/src/mito2/src/region/opener.rs @@ -20,6 +20,7 @@ use std::sync::atomic::{AtomicI64, AtomicU64}; use std::sync::{Arc, LazyLock}; use std::time::Instant; +use api::v1::WalEntry; use arrow_schema::extension::ExtensionType; use common_telemetry::{debug, error, info, warn}; use common_wal::options::WalOptions; @@ -77,8 +78,11 @@ use crate::sst::FormatType; use crate::sst::file::{FileHandle, RegionFileId, RegionIndexId}; use crate::sst::file_purger::{FilePurgerRef, create_file_purger}; use crate::sst::file_ref::FileReferenceManagerRef; +use crate::sst::index::IndexBuildType; +use crate::sst::index::handler::PlanContext; use crate::sst::index::intermediate::IntermediateManager; use crate::sst::index::puffin_manager::PuffinManagerFactory; +use crate::sst::index::registry::IndexRegistryRef; use crate::sst::location::{self, region_dir_from_table_dir}; use crate::sst::parquet::metadata::{MetadataLoader, extract_primary_key_range}; use crate::sst::parquet::reader::MetadataCacheMetrics; @@ -148,6 +152,7 @@ pub type PartitionExprFetcherRef = Arc; /// Builder to create a new [MitoRegion] or open an existing one. pub(crate) struct RegionOpener { + index_registry: Option, region_id: RegionId, metadata_builder: Option, memtable_builder_provider: MemtableBuilderProvider, @@ -191,6 +196,7 @@ impl RegionOpener { partition_expr_fetcher: PartitionExprFetcherRef, ) -> RegionOpener { RegionOpener { + index_registry: None, region_id, metadata_builder: None, memtable_builder_provider, @@ -216,6 +222,12 @@ impl RegionOpener { } } + /// Sets the registry used to validate new-region index options. + pub(crate) fn with_index_registry(mut self, registry: IndexRegistryRef) -> Self { + self.index_registry = Some(registry); + self + } + /// Sets the store for companion range indexes. pub(crate) fn series_index_store(mut self, store: Option) -> Self { self.series_index_store = store; @@ -383,17 +395,39 @@ impl RegionOpener { }; ensure_json2_uses_flat_format(&metadata, &options)?; options.validate_column_compression(&metadata)?; + // Validate explicit new-region configuration before creating its manifest. + // Opening an existing manifest above does not require its optional providers. + if let Some(registry) = &self.index_registry { + registry + .validate(&PlanContext { + metadata: &metadata, + index_options: &options.index_options, + // Validate initial options without building indexes; extensions skip flush work. + build_type: IndexBuildType::Flush, + existing_indexes: &[], + }) + .map_err(|err| { + InvalidRegionOptionsSnafu { + reason: err.to_string(), + } + .build() + })?; + } // The region's WAL history starts in this log store, so reopening // the region on it passes the history check without any entry. It is // recorded before the manifest exists: a log store that gave up its // history meanwhile fails the create before it leaves a manifest. wal.start_region_history(region_id, &provider)?; // Create a manifest manager for this region and writes regions to the manifest file. - let mut region_manifest_options = - RegionManifestOptions::new(config, ®ion_dir, &object_store) - .with_conditional_publication( - options.requires_conditional_manifest_publication(self.node_lease.is_enabled()), - ); + let mut region_manifest_options = RegionManifestOptions::new( + config, + ®ion_dir, + &object_store, + self.file_ref_manager.file_metadata_interner.clone(), + ) + .with_conditional_publication( + options.requires_conditional_manifest_publication(self.node_lease.is_enabled()), + ); // Set manifest cache if available region_manifest_options.manifest_cache = self .cache_manager @@ -525,12 +559,15 @@ impl RegionOpener { let now = Instant::now(); let mut region_options = self.options.as_ref().unwrap().clone(); let object_storage = get_object_store(®ion_options.storage, &self.object_store_manager)?; - let mut region_manifest_options = - RegionManifestOptions::new(config, &self.region_dir(), &object_storage) - .with_conditional_publication( - region_options - .requires_conditional_manifest_publication(self.node_lease.is_enabled()), - ); + let mut region_manifest_options = RegionManifestOptions::new( + config, + &self.region_dir(), + &object_storage, + self.file_ref_manager.file_metadata_interner.clone(), + ) + .with_conditional_publication( + region_options.requires_conditional_manifest_publication(self.node_lease.is_enabled()), + ); // Set manifest cache if available region_manifest_options.manifest_cache = self .cache_manager @@ -1001,7 +1038,6 @@ where // data in the WAL. let mut last_entry_id = flushed_entry_id; let replay_from_entry_id = flushed_entry_id + 1; - let region_metadata = version_control.current().version.metadata.clone(); let mut wal_stream = wal_entry_reader.read(provider, replay_from_entry_id)?; while let Some(res) = wal_stream.next().await { @@ -1021,63 +1057,8 @@ where ); } last_entry_id = last_entry_id.max(entry_id); - - let mut region_write_ctx = RegionWriteCtx::new( - region_id, - version_control, - provider.clone(), - // For WAL replay, we don't need to track the write bytes rate. - None, - WriteSource::WalReplay, - ); - for mutation in entry.mutations { - rows_replayed += mutation - .rows - .as_ref() - .map(|rows| rows.rows.len()) - .unwrap_or(0); - region_write_ctx.push_mutation( - mutation.op_type, - mutation.rows, - mutation.write_hint, - OptionOutputTx::none(), - // We should respect the sequence in WAL during replay. - Some(mutation.sequence), - false, - ); - } - - for bulk_entry in entry.bulk_entries { - let mut part = BulkPart::try_from(bulk_entry)?; - // The entry may miss columns added by a concurrent alter if it was - // written by a writer with a stale schema (older versions kept the - // stale raw data when filling missing columns). Fills missing - // columns like the write path, otherwise the memtable rejects the - // batch. Skips sparse batches as they don't carry tag columns by - // design. - part.fill_missing_columns(®ion_metadata)?; - rows_replayed += part.num_rows(); - // During replay, we should adopt the sequence from WAL. - let bulk_sequence_from_wal = part.sequence; - ensure!( - region_write_ctx.push_bulk( - OptionOutputTx::none(), - part, - Some(bulk_sequence_from_wal), - false - ), - RegionCorruptedSnafu { - region_id, - reason: "unable to replay memtable with bulk entries", - } - ); - } - - // set next_entry_id and write to memtable. - region_write_ctx.set_next_entry_id(last_entry_id + 1); - // Publish the replayed sequences only after all rows (including bulk - // parts) are installed, matching the write path ordering. - region_write_ctx.write_memtables().await; + rows_replayed += + replay_entry(provider, region_id, version_control, entry, last_entry_id).await?; } // TODO(weny): We need to update `flushed_entry_id` in the region manifest @@ -1098,6 +1079,77 @@ where Ok(last_entry_id) } +/// Writes one WAL entry into the mutable memtable of `version_control`, keeping the +/// sequences the WAL recorded, and returns the number of rows. `last_entry_id` is the +/// largest entry id replayed so far, this one included. +pub(crate) async fn replay_entry( + provider: &Provider, + region_id: RegionId, + version_control: &VersionControlRef, + entry: WalEntry, + last_entry_id: EntryId, +) -> Result { + let mut rows_replayed = 0; + let mut region_write_ctx = RegionWriteCtx::new( + region_id, + version_control, + provider.clone(), + // For WAL replay, we don't need to track the write bytes rate. + None, + WriteSource::WalReplay, + ); + for mutation in entry.mutations { + rows_replayed += mutation + .rows + .as_ref() + .map(|rows| rows.rows.len()) + .unwrap_or(0); + region_write_ctx.push_mutation( + mutation.op_type, + mutation.rows, + mutation.write_hint, + OptionOutputTx::none(), + // We should respect the sequence in WAL during replay. + Some(mutation.sequence), + false, + ); + } + + let region_metadata = region_write_ctx.version().metadata.clone(); + for bulk_entry in entry.bulk_entries { + let mut part = BulkPart::try_from(bulk_entry)?; + // The entry may miss columns added by a concurrent alter if it was + // written by a writer with a stale schema (older versions kept the + // stale raw data when filling missing columns). Fills missing + // columns like the write path, otherwise the memtable rejects the + // batch. Skips sparse batches as they don't carry tag columns by + // design. + part.fill_missing_columns(®ion_metadata)?; + rows_replayed += part.num_rows(); + // During replay, we should adopt the sequence from WAL. + let bulk_sequence_from_wal = part.sequence; + ensure!( + region_write_ctx.push_bulk( + OptionOutputTx::none(), + part, + Some(bulk_sequence_from_wal), + false + ), + RegionCorruptedSnafu { + region_id, + reason: "unable to replay memtable with bulk entries", + } + ); + } + + // set next_entry_id and write to memtable. + region_write_ctx.set_next_entry_id(last_entry_id + 1); + // Publish the replayed sequences only after all rows (including bulk + // parts) are installed, matching the write path ordering. + region_write_ctx.write_memtables().await; + Ok(rows_replayed) +} + /// A task to load and fill the region file cache. pub(crate) struct RegionLoadCacheTask { region: MitoRegionRef, @@ -1476,6 +1528,7 @@ fn can_load_cache(state: RegionRoleState) -> bool { match state { RegionRoleState::Leader(RegionLeaderState::Writable) | RegionRoleState::Leader(RegionLeaderState::Staging) + | RegionRoleState::Leader(RegionLeaderState::EndingStaging) | RegionRoleState::Leader(RegionLeaderState::Altering) | RegionRoleState::Leader(RegionLeaderState::EnteringStaging) | RegionRoleState::Leader(RegionLeaderState::Editing) diff --git a/src/mito2/src/region/utils.rs b/src/mito2/src/region/utils.rs index 496668cfce0..3faf66c9dd3 100644 --- a/src/mito2/src/region/utils.rs +++ b/src/mito2/src/region/utils.rs @@ -32,6 +32,7 @@ use crate::manifest::manager::{RegionManifestManager, RegionManifestOptions}; use crate::region::opener::get_object_store; use crate::region::options::RegionOptions; use crate::sst::file::{RegionFileId, RegionIndexId}; +use crate::sst::file_metadata::FileMetadataInternerRef; use crate::sst::location; /// A loader for loading metadata from a region dir. @@ -40,6 +41,7 @@ pub struct RegionMetadataLoader { config: Arc, object_store_manager: ObjectStoreManagerRef, lease_enabled: bool, + file_metadata_interner: FileMetadataInternerRef, } impl RegionMetadataLoader { @@ -48,11 +50,13 @@ impl RegionMetadataLoader { config: Arc, object_store_manager: ObjectStoreManagerRef, lease_enabled: bool, + file_metadata_interner: FileMetadataInternerRef, ) -> Self { Self { config, object_store_manager, lease_enabled, + file_metadata_interner, } } @@ -73,11 +77,15 @@ impl RegionMetadataLoader { region_options: &RegionOptions, ) -> Result>> { let object_store = get_object_store(®ion_options.storage, &self.object_store_manager)?; - let region_manifest_options = - RegionManifestOptions::new(&self.config, region_dir, &object_store) - .with_conditional_publication( - region_options.requires_conditional_manifest_publication(self.lease_enabled), - ); + let region_manifest_options = RegionManifestOptions::new( + &self.config, + region_dir, + &object_store, + self.file_metadata_interner.clone(), + ) + .with_conditional_publication( + region_options.requires_conditional_manifest_publication(self.lease_enabled), + ); let Some(manifest_manager) = RegionManifestManager::open(region_manifest_options, &Default::default()).await? else { diff --git a/src/mito2/src/region/version.rs b/src/mito2/src/region/version.rs index a9866ecd748..64710774866 100644 --- a/src/mito2/src/region/version.rs +++ b/src/mito2/src/region/version.rs @@ -37,7 +37,7 @@ use crate::memtable::time_partition::TimePartitions; use crate::memtable::version::{MemtableVersion, MemtableVersionRef}; use crate::memtable::{MemtableBuilderRef, MemtableId}; use crate::region::options::RegionOptions; -use crate::sst::file::FileMeta; +use crate::sst::file::{FileMeta, FileMetaRef}; use crate::sst::file_purger::FilePurgerRef; use crate::sst::version::{SstVersion, SstVersionRef}; use crate::wal::EntryId; @@ -554,7 +554,7 @@ impl VersionBuilder { pub(crate) fn add_files( mut self, file_purger: FilePurgerRef, - files: impl Iterator, + files: impl Iterator>, ) -> Self { let mut ssts = (*self.ssts).clone(); ssts.add_files(file_purger, files); diff --git a/src/mito2/src/remap_manifest.rs b/src/mito2/src/remap_manifest.rs index fec0cad0547..6994cf5185c 100644 --- a/src/mito2/src/remap_manifest.rs +++ b/src/mito2/src/remap_manifest.rs @@ -436,7 +436,7 @@ mod tests { num_rows: 100, num_row_groups: 1, sequence: NonZeroU64::new(1), - partition_expr, + partition_expr: partition_expr.map(Into::into), num_series: 1, ..Default::default() } @@ -454,7 +454,7 @@ mod tests { for _ in 0..num_files { let file_id = FileId::random(); let file_meta = create_file_meta(region_id, file_id, partition_expr.clone()); - files.insert(file_id, file_meta); + files.insert(file_id, file_meta.into()); } RegionManifest { @@ -646,10 +646,12 @@ mod tests { let file_meta = create_file_meta(old_region_id_1, shared_file_id, None); let mut manifest_1 = create_manifest(old_region_id_1, 0, None, 100, 200); - manifest_1.files.insert(shared_file_id, file_meta.clone()); + manifest_1 + .files + .insert(shared_file_id, file_meta.clone().into()); let mut manifest_2 = create_manifest(old_region_id_2, 0, None, 100, 200); - manifest_2.files.insert(shared_file_id, file_meta); + manifest_2.files.insert(shared_file_id, file_meta.into()); let mut old_manifests = HashMap::new(); old_manifests.insert(old_region_id_1, manifest_1); diff --git a/src/mito2/src/request.rs b/src/mito2/src/request.rs index de9594229b3..be3e76f53c9 100644 --- a/src/mito2/src/request.rs +++ b/src/mito2/src/request.rs @@ -58,7 +58,7 @@ use crate::error::{ Result, UnexpectedSnafu, }; use crate::flush::FlushReason; -use crate::manifest::action::{RegionEdit, TruncateKind}; +use crate::manifest::action::{PendingTail, RegionEdit, TruncateKind}; use crate::memtable::MemtableId; use crate::memtable::bulk::part::BulkPart; use crate::metrics::COMPACTION_ELAPSED_TOTAL; @@ -1143,6 +1143,9 @@ pub(crate) enum BackgroundNotify { RegionEdit(RegionEditResult), /// Enter staging result. EnterStaging(EnterStagingResult), + /// The region left [RegionLeaderState::EndingStaging](crate::region::RegionLeaderState::EndingStaging), + /// so writes stalled meanwhile can be handled. The sender is notified once they are. + StagingEnded(oneshot::Sender<()>), /// Copy region result. CopyRegionFromFinished(CopyRegionFromFinished), /// A commit outside the worker fenced the region until it is reloaded. @@ -1154,6 +1157,65 @@ pub(crate) enum BackgroundNotify { /// The reloaded instance. result: Result, }, + /// The region became writable and may have pending tails to replay. + RegionBecameWritable, + /// A role change left the region unable to replay its tails (see + /// [MitoRegion::can_replay_tails](crate::region::MitoRegion::can_replay_tails)); a running + /// replay is cancelled. + RegionCannotReplay { + /// The instance the role change applied to. + region: MitoRegionRef, + }, + /// A tail replay published its result in the manifest. + TailReplayCommitted { + /// The instance the replay ran for. + region: MitoRegionRef, + /// The published edit. + edit: RegionEdit, + /// The tails the replay read, which the manifest recorded when the edit was published. + tails: Vec, + }, + /// A tail replay ended. + TailReplayEnded, +} + +impl BackgroundNotify { + /// Whether the worker dispatches pending requests after handling this notification. + /// + /// Dispatch consumes queued requests and can determine their terminal errors. A fenced + /// region must let reload reject its old requests as not found, rather than dispatching + /// them against the fenced instance and returning busy. + /// Keep this exhaustive so new notifications must declare their scheduling effect. + pub(crate) fn should_dispatch_pending(&self) -> bool { + match self { + Self::CompactionUnit(_) + | Self::CompactionDdlComplete { .. } + | Self::CompactionPickFinished(_) + | Self::FlushFinished(_) + | Self::FlushFailed(_) + | Self::CompactionFinished(_) + | Self::CompactionFailed(_) + | Self::RegionChange(_) + | Self::StagingEnded(_) + | Self::EnterStaging(_) => true, + // Edit completion dispatches before scheduling compaction, after stalled writes. + // A completion from an old instance must not dispatch the new instance's queue. + Self::RegionEdit(_) => false, + Self::RegionOpened { .. } + | Self::IndexBuildFinished(_) + | Self::IndexBuildStopped(_) + | Self::IndexBuildRetry(_) + | Self::Truncate(_) + | Self::DiscardUnflushed(_) + | Self::CopyRegionFromFinished(_) + | Self::RegionFenced + | Self::RegionReloaded { .. } + | Self::RegionBecameWritable + | Self::RegionCannotReplay { .. } + | Self::TailReplayCommitted { .. } + | Self::TailReplayEnded => false, + } + } } /// Notifies a flush job is finished. @@ -1204,6 +1266,8 @@ impl OnFailure for FlushFinished { pub(crate) struct FlushFailed { /// The error source of the failure. pub(crate) err: Arc, + /// Flush result senders. + pub(crate) senders: Vec, } impl FlushFailed { diff --git a/src/mito2/src/schedule.rs b/src/mito2/src/schedule.rs index e0b91e1ad59..9b09a83927d 100644 --- a/src/mito2/src/schedule.rs +++ b/src/mito2/src/schedule.rs @@ -15,7 +15,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; -use common_base::cancellation::CancellationHandle; +use tokio_util::sync::CancellationToken; pub mod remote_job_scheduler; pub mod scheduler; @@ -23,7 +23,7 @@ pub mod scheduler; /// Shared state for cooperatively cancelling a task until it starts committing. #[derive(Debug, Clone)] pub(crate) struct CancellableTaskState { - cancel_handle: Arc, + cancel_token: CancellationToken, commit_started: Arc>, started: Arc, } @@ -31,7 +31,7 @@ pub(crate) struct CancellableTaskState { impl CancellableTaskState { pub(crate) fn new() -> Self { Self { - cancel_handle: Arc::new(CancellationHandle::default()), + cancel_token: CancellationToken::new(), commit_started: Arc::new(Mutex::new(false)), started: Arc::new(AtomicBool::new(false)), } @@ -47,12 +47,12 @@ impl CancellableTaskState { self.started.load(Ordering::Relaxed) } - pub(crate) fn cancel_handle(&self) -> Arc { - self.cancel_handle.clone() + pub(crate) fn cancel_token(&self) -> CancellationToken { + self.cancel_token.clone() } pub(crate) fn is_cancelled(&self) -> bool { - self.cancel_handle.is_cancelled() + self.cancel_token.is_cancelled() } /// Returns whether the task has crossed the non-cancellable publication boundary. @@ -65,7 +65,7 @@ impl CancellableTaskState { /// Returns false if cancellation was requested first. pub(crate) fn mark_commit_started(&self) -> bool { let mut commit_started = self.commit_started.lock().unwrap(); - if self.cancel_handle.is_cancelled() { + if self.cancel_token.is_cancelled() { return false; } *commit_started = true; @@ -78,11 +78,11 @@ impl CancellableTaskState { if *commit_started { return RequestCancelResult::TooLateToCancel; } - if self.cancel_handle.is_cancelled() { + if self.cancel_token.is_cancelled() { return RequestCancelResult::AlreadyCancelling; } - self.cancel_handle.cancel(); + self.cancel_token.cancel(); RequestCancelResult::CancelIssued } } diff --git a/src/mito2/src/series_index/task.rs b/src/mito2/src/series_index/task.rs index 909444c3f97..6c75d05e7dc 100644 --- a/src/mito2/src/series_index/task.rs +++ b/src/mito2/src/series_index/task.rs @@ -609,7 +609,9 @@ mod tests { LifecycleChange::Remove => { regions.remove_region(region.region_id).unwrap(); } - LifecycleChange::Demote => region.set_role(RegionRole::Follower), + LifecycleChange::Demote => { + region.set_role(RegionRole::Follower); + } LifecycleChange::Stop => state.stop(), } release.notify_one(); diff --git a/src/mito2/src/sst.rs b/src/mito2/src/sst.rs index 7a0e3603836..cc579a11843 100644 --- a/src/mito2/src/sst.rs +++ b/src/mito2/src/sst.rs @@ -42,6 +42,7 @@ use crate::error::{InvalidNativeHistogramFieldIdSnafu, InvalidNativeHistogramSub use crate::sst::parquet::flat_format::time_index_column_index; pub mod file; +pub mod file_metadata; pub mod file_purger; pub mod file_ref; pub mod index; @@ -49,6 +50,7 @@ pub mod location; pub mod parquet; pub(crate) mod primary_key; pub mod range_index; +pub(crate) mod upload; pub(crate) mod version; /// Default write buffer size, it should be greater than the default minimum upload part of S3 (5mb). diff --git a/src/mito2/src/sst/file.rs b/src/mito2/src/sst/file.rs index a29cc33af0c..fc7f09bed1f 100644 --- a/src/mito2/src/sst/file.rs +++ b/src/mito2/src/sst/file.rs @@ -17,6 +17,7 @@ use std::collections::HashMap; use std::fmt; use std::fmt::{Debug, Formatter}; +use std::hash::{Hash, Hasher}; use std::num::NonZeroU64; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex, RwLock}; @@ -38,11 +39,13 @@ use crate::access_layer::AccessLayerRef; use crate::cache::CacheManagerRef; use crate::cache::file_cache::{FileType, IndexKey}; use crate::region::options::IndexOptions; +use crate::sst::file_metadata::MetadataRef; use crate::sst::file_purger::FilePurgerRef; -use crate::sst::file_ref::FileReferenceManagerRef; +use crate::sst::file_ref::{FileReferenceManager, FileReferenceManagerRef}; use crate::sst::index::fulltext_index::FULLTEXT_BLOOM_VERSION; use crate::sst::location; use crate::sst::parquet::SstInfo; +use crate::sst::parquet::writer::SstWriteGuard; use crate::sst::primary_key::PrimaryKeyRangeMapper; /// Custom serde functions for Bytes fields serialized as base64 strings. @@ -74,7 +77,7 @@ where /// Custom serde functions for partition_expr field in FileMeta fn serialize_partition_expr( - partition_expr: &Option, + partition_expr: &Option, serializer: S, ) -> Result where @@ -91,7 +94,7 @@ where } } -fn deserialize_partition_expr<'de, D>(deserializer: D) -> Result, D::Error> +fn deserialize_partition_expr<'de, D>(deserializer: D) -> Result, D::Error> where D: serde::Deserializer<'de>, { @@ -106,7 +109,9 @@ where Ok(None) } else { // Parse the JSON string to PartitionExpr - PartitionExpr::from_json_str(&json_str).map_err(D::Error::custom) + PartitionExpr::from_json_str(&json_str) + .map(|expr| expr.map(PartitionExprRef::from)) + .map_err(D::Error::custom) } } } @@ -231,6 +236,12 @@ pub(crate) fn overlaps(l: &FileTimeRange, r: &FileTimeRange) -> bool { r_start <= l_end } +/// Shared immutable metadata of an SST file. +pub type FileMetaRef = MetadataRef; + +/// Shared immutable partition expression. +pub type PartitionExprRef = MetadataRef; + /// Metadata of a SST file. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize, Default)] #[serde(default)] @@ -289,7 +300,7 @@ pub struct FileMeta { pub sequence: Option, /// Partition expression from the region metadata when the file is created. /// - /// This is stored as a PartitionExpr object in memory for convenience, + /// Immutable expressions are shared by table and content where canonicalized, /// but serialized as JSON string for manifest compatibility. /// Compatibility behavior: /// - None: no partition expr was set when the file was created (legacy files). @@ -298,7 +309,7 @@ pub struct FileMeta { serialize_with = "serialize_partition_expr", deserialize_with = "deserialize_partition_expr" )] - pub partition_expr: Option, + pub partition_expr: Option, /// Number of series in the file. /// /// The number is 0 if the series number is not available. @@ -343,6 +354,14 @@ impl) -> fmt::Result> std::fmt::Debug for DebugFmt { } } +// Hashing file identity avoids traversing indexes and expressions on every lookup. +// Full equality still distinguishes metadata versions of the same physical file. +impl Hash for FileMeta { + fn hash(&self, state: &mut H) { + RegionFileId::new(self.region_id, self.file_id).hash(state); + } +} + impl Debug for FileMeta { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { let mut debug_struct = f.debug_struct("FileMeta"); @@ -404,7 +423,7 @@ impl Debug for FileMeta { } /// Type of index. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] pub enum IndexType { /// Inverted index. InvertedIndex, @@ -431,7 +450,7 @@ pub enum IndexType { /// This is primarily used for: /// - Manual index building in asynchronous index construction mode /// - Verifying index consistency between files and region metadata -#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, Default)] +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize, Default)] #[serde(default)] pub struct ColumnIndexMetadata { /// The column ID this index metadata applies to. @@ -634,7 +653,8 @@ pub(crate) fn known_max_input_sequence<'a>( impl FileHandle { /// Creates a new logical SST reference. Index replacements must reuse its lifetime. - pub fn new(meta: FileMeta, file_purger: FilePurgerRef) -> FileHandle { + pub fn new(meta: impl Into, file_purger: FilePurgerRef) -> FileHandle { + let meta = meta.into(); let pk_range = meta.primary_key_range(); FileHandle { inner: Arc::new(FileHandleInner::new(meta, file_purger, pk_range)), @@ -648,18 +668,22 @@ impl FileHandle { primary_key_range: Option<(Bytes, Bytes)>, ) -> FileHandle { FileHandle { - inner: Arc::new(FileHandleInner::new(meta, file_purger, primary_key_range)), + inner: Arc::new(FileHandleInner::new( + meta.into(), + file_purger, + primary_key_range, + )), } } /// Replaces an index metadata view without splitting the underlying SST lifetime. - pub(crate) fn with_updated_index(&self, meta: FileMeta) -> Self { + pub(crate) fn with_updated_index(&self, meta: impl Into) -> Self { + let meta = meta.into(); debug_assert_eq!(self.file_id(), meta.file_id()); let primary_key_range = meta .primary_key_range() .or_else(|| self.raw_primary_key_range()); let lifetime = self.inner.lifetime.clone(); - *lifetime.meta.lock().unwrap() = meta.clone(); Self { inner: Arc::new(FileHandleInner::with_lifetime( meta, @@ -770,12 +794,18 @@ impl FileHandle { .primary_key_range .read() .unwrap() - .aligned(mapper.schema_version()) + .as_deref() + .and_then(|range| range.aligned(mapper.schema_version())) { return range.clone(); } // Recheck under the write lock: another snapshot may have replaced the cached schema. - let aligned = self.inner.primary_key_range.write().unwrap().align(mapper); + let mut cached_range = self.inner.primary_key_range.write().unwrap(); + if cached_range.is_none() { + let raw = self.inner.meta.primary_key_range()?; + *cached_range = Some(Box::new(PrimaryKeyRange::Raw(raw))); + } + let aligned = cached_range.as_deref_mut()?.align(mapper); match aligned { Ok(range) => range, Err(err) => { @@ -788,15 +818,22 @@ impl FileHandle { /// Returns original statistics for metadata hydration, never schema-aligned bounds. pub fn raw_primary_key_range(&self) -> Option<(Bytes, Bytes)> { - self.inner.primary_key_range.read().unwrap().raw().cloned() + self.inner + .primary_key_range + .read() + .unwrap() + .as_deref() + .map(PrimaryKeyRange::raw) + .cloned() + .or_else(|| self.inner.meta.primary_key_range()) } pub(crate) fn set_primary_key_range(&self, primary_key_range: (Bytes, Bytes)) { // SST contents are immutable. Hydrate missing raw statistics without // replacing the source of already cached schema views. let mut range = self.inner.primary_key_range.write().unwrap(); - if matches!(*range, PrimaryKeyRange::Missing) { - *range = PrimaryKeyRange::Raw(primary_key_range); + if range.is_none() && self.inner.meta.primary_key_range().is_none() { + *range = Some(Box::new(PrimaryKeyRange::Raw(primary_key_range))); } } } @@ -806,7 +843,6 @@ type PrimaryKeyBounds = (Bytes, Bytes); /// A single-slot cache shared by file handles. Always retain the source bounds so /// older snapshots and changed defaults can realign without interpreting padded values as stored. enum PrimaryKeyRange { - Missing, Raw(PrimaryKeyBounds), Aligned { raw: PrimaryKeyBounds, @@ -816,10 +852,9 @@ enum PrimaryKeyRange { } impl PrimaryKeyRange { - fn raw(&self) -> Option<&PrimaryKeyBounds> { + fn raw(&self) -> &PrimaryKeyBounds { match self { - Self::Missing => None, - Self::Raw(raw) | Self::Aligned { raw, .. } => Some(raw), + Self::Raw(raw) | Self::Aligned { raw, .. } => raw, } } @@ -841,9 +876,7 @@ impl PrimaryKeyRange { if let Some(bounds) = self.aligned(mapper.schema_version()) { return Ok(bounds.clone()); } - let Some(raw) = self.raw().cloned() else { - return Ok(None); - }; + let raw = self.raw().clone(); let aligned = mapper.map(raw.clone()); // Failed mappings also occupy the cache, so repeated hits don't repeat the warning. *self = Self::Aligned { @@ -859,9 +892,10 @@ impl PrimaryKeyRange { /// /// Contains meta of the file, and other mutable info like whether the file is compacting. struct FileHandleInner { - meta: FileMeta, + meta: FileMetaRef, compacting: AtomicBool, - primary_key_range: RwLock, + /// Allocated only for derived bounds; boxing keeps handles without a cache small. + primary_key_range: RwLock>>, lifetime: Arc, } @@ -875,34 +909,56 @@ impl Drop for FileHandleInner { impl FileHandleInner { fn new( - meta: FileMeta, + meta: FileMetaRef, file_purger: FilePurgerRef, primary_key_range: Option<(Bytes, Bytes)>, ) -> Self { let file_ref_manager = file_purger.file_ref_manager(); + // Normalize expressions while metadata is still exclusively owned. + let meta = Self::register_metadata(meta, file_ref_manager.as_deref()); let lifetime = Arc::new(SstLifetime { meta: Mutex::new(meta.clone()), deleted: AtomicBool::new(false), file_purger, file_ref_manager, }); - Self::with_lifetime(meta, lifetime, primary_key_range) + Self::from_registered_metadata(meta, lifetime, primary_key_range) } fn with_lifetime( - meta: FileMeta, + meta: FileMetaRef, lifetime: Arc, primary_key_range: Option<(Bytes, Bytes)>, ) -> Self { - if let Some(manager) = &lifetime.file_ref_manager { + let meta = Self::register_metadata(meta, lifetime.file_ref_manager.as_deref()); + *lifetime.meta.lock().unwrap() = meta.clone(); + Self::from_registered_metadata(meta, lifetime, primary_key_range) + } + + /// Every runtime view is interned before it enters the reference registry. + fn register_metadata(meta: FileMetaRef, manager: Option<&FileReferenceManager>) -> FileMetaRef { + if let Some(manager) = manager { + let meta = manager.file_metadata_interner.intern(meta); manager.register_file(&meta); + meta + } else { + meta } + } + + fn from_registered_metadata( + meta: FileMetaRef, + lifetime: Arc, + primary_key_range: Option<(Bytes, Bytes)>, + ) -> Self { + // Raw bounds already in metadata need no per-handle cache until alignment. + let primary_key_range = primary_key_range + .filter(|range| meta.primary_key_range().as_ref() != Some(range)) + .map(|range| Box::new(PrimaryKeyRange::Raw(range))); Self { meta, compacting: AtomicBool::new(false), - primary_key_range: RwLock::new( - primary_key_range.map_or(PrimaryKeyRange::Missing, PrimaryKeyRange::Raw), - ), + primary_key_range: RwLock::new(primary_key_range), lifetime, } } @@ -912,7 +968,7 @@ impl FileHandleInner { /// Independent logical references (for example, copied regions) do not share this guard. struct SstLifetime { /// Latest metadata ensures final purge includes indexes newer than the last reader's view. - meta: Mutex, + meta: Mutex, deleted: AtomicBool, file_purger: FilePurgerRef, /// Tracks each metadata view's readers independently for object-store index GC. @@ -923,7 +979,7 @@ impl Drop for SstLifetime { fn drop(&mut self) { if self.deleted.load(Ordering::Acquire) { self.file_purger - .purge_file(self.meta.get_mut().unwrap().clone()); + .purge_file(self.meta.get_mut().unwrap().as_ref().clone()); } } } @@ -987,7 +1043,7 @@ pub async fn delete_files( Ok(()) } -/// Tracks finalized SSTs until their manifest edit is committed. +/// Tracks SSTs from creation until their manifest edit is committed. /// /// Flush and local compaction jobs use this to remove files that were written successfully but /// abandoned by cancellation or a later failure. @@ -999,6 +1055,15 @@ pub(crate) struct UncommittedSsts { cache_manager: Option, } +#[async_trait::async_trait] +impl SstWriteGuard for UncommittedSsts { + async fn before_create(&self, file_id: RegionFileId) -> crate::error::Result<()> { + // An index can be created before its final metadata is available. + self.track_before_upload(file_id.file_id(), true); + Ok(()) + } +} + impl UncommittedSsts { pub(crate) fn new( region_id: RegionId, @@ -1199,6 +1264,87 @@ mod tests { use super::*; + #[test] + fn test_shared_metadata_keeps_independent_lifetimes() { + let meta = FileMetaRef::from(FileMeta::default()); + let purger = Arc::new(crate::sst::file_purger::NoopFilePurger); + let first = FileHandle::new(meta.clone(), purger.clone()); + let second = FileHandle::new(meta.clone(), purger); + assert!(FileMetaRef::ptr_eq(&meta, &first.inner.meta)); + assert!(FileMetaRef::ptr_eq( + &meta, + &first.inner.lifetime.meta.lock().unwrap() + )); + assert!(!Arc::ptr_eq(&first.inner.lifetime, &second.inner.lifetime)); + assert!(first.inner.primary_key_range.read().unwrap().is_none()); + first.mark_deleted(); + assert!(!second.is_deleted()); + + let mut updated = meta.as_ref().clone(); + updated.index_version = 1; + let updated = FileMetaRef::from(updated); + let replacement = second.with_updated_index(updated.clone()); + assert_eq!(second.meta_ref().index_version, 0); + assert_eq!(replacement.meta_ref().index_version, 1); + assert!(Arc::ptr_eq( + &replacement.inner.lifetime, + &second.inner.lifetime + )); + assert!(FileMetaRef::ptr_eq( + &updated, + &second.inner.lifetime.meta.lock().unwrap() + )); + } + + #[test] + fn test_runtime_handles_share_partition_expressions() { + #[derive(Debug)] + struct TrackedPurger(FileReferenceManagerRef); + + impl crate::sst::file_purger::FilePurger for TrackedPurger { + fn purge_file(&self, _: FileMeta) {} + + fn file_ref_manager(&self) -> Option { + Some(self.0.clone()) + } + } + + let purger = Arc::new(TrackedPurger(Arc::new(FileReferenceManager::new(None)))); + let handles: Vec<_> = (0..2) + .map(|_| { + FileHandle::new( + FileMeta { + region_id: RegionId::new(1, 1), + file_id: FileId::random(), + partition_expr: Some(col("tag").lt(Value::UInt32(10)).into()), + ..Default::default() + }, + purger.clone(), + ) + }) + .collect(); + assert!(PartitionExprRef::ptr_eq( + handles[0].meta_ref().partition_expr.as_ref().unwrap(), + handles[1].meta_ref().partition_expr.as_ref().unwrap(), + )); + } + + #[test] + fn test_raw_metadata_bounds_do_not_allocate_a_cache() { + let bounds = (Bytes::from_static(b"a"), Bytes::from_static(b"z")); + let meta = FileMeta { + primary_key_min: Some(bounds.0.clone()), + primary_key_max: Some(bounds.1.clone()), + ..Default::default() + }; + let handle = FileHandle::new(meta, Arc::new(crate::sst::file_purger::NoopFilePurger)); + assert_eq!(Some(bounds.clone()), handle.raw_primary_key_range()); + assert!(handle.inner.primary_key_range.read().unwrap().is_none()); + handle.set_primary_key_range((Bytes::from_static(b"b"), Bytes::from_static(b"y"))); + assert_eq!(Some(bounds), handle.raw_primary_key_range()); + assert!(handle.inner.primary_key_range.read().unwrap().is_none()); + } + fn create_file_meta(file_id: FileId, level: Level) -> FileMeta { FileMeta { region_id: 0.into(), @@ -1341,7 +1487,7 @@ mod tests { num_rows: 0, num_row_groups: 0, sequence: None, - partition_expr: Some(partition_expr.clone()), + partition_expr: Some(partition_expr.clone().into()), num_series: 0, ..Default::default() }; diff --git a/src/mito2/src/sst/file_metadata.rs b/src/mito2/src/sst/file_metadata.rs new file mode 100644 index 00000000000..a8673c3bc7f --- /dev/null +++ b/src/mito2/src/sst/file_metadata.rs @@ -0,0 +1,567 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Sharing of immutable SST metadata across manifests and runtime readers. + +use std::collections::hash_map::RandomState; +use std::fmt::{self, Debug, Display}; +use std::hash::{BuildHasher, Hash, Hasher}; +use std::ops::Deref; +use std::sync::Arc; + +use intern_arc::{HashInterner, InternedHash}; +use partition::expr::PartitionExpr; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use store_api::storage::TableId; + +use crate::metrics::file_metadata::FILE_METADATA_COLLECTOR; +use crate::sst::file::{FileMeta, FileMetaRef, RegionFileId}; + +const SHARD_COUNT: usize = 32; + +/// Immutable metadata, either freshly decoded or registered with an engine interner. +/// Keeping the decoded allocation avoids copying index vectors during recovery. +#[derive(Clone)] +pub struct MetadataRef(MetadataRefInner); + +#[derive(Clone)] +enum MetadataRefInner { + Owned(Arc), + Interned(InternedHash>), +} + +#[derive(PartialEq, Eq, Hash)] +struct ScopedMetadata { + table_id: TableId, + value: Arc, +} + +impl MetadataRef { + fn as_arc(&self) -> &Arc { + match &self.0 { + MetadataRefInner::Owned(value) => value, + MetadataRefInner::Interned(value) => &value.value, + } + } + + fn get_mut(&mut self) -> Option<&mut T> { + match &mut self.0 { + MetadataRefInner::Owned(value) => Arc::get_mut(value), + // Mutating a registered value would invalidate the hash table. + MetadataRefInner::Interned(_) => None, + } + } + + #[cfg(test)] + pub(crate) fn ptr_eq(left: &Self, right: &Self) -> bool { + Arc::ptr_eq(left.as_arc(), right.as_arc()) + } +} + +impl From for MetadataRef { + fn from(value: T) -> Self { + Self::from(Arc::new(value)) + } +} + +impl From> for MetadataRef { + fn from(value: Arc) -> Self { + Self(MetadataRefInner::Owned(value)) + } +} + +impl Default for MetadataRef { + fn default() -> Self { + T::default().into() + } +} + +impl Deref for MetadataRef { + type Target = T; + + fn deref(&self) -> &T { + self.as_arc() + } +} + +impl AsRef for MetadataRef { + fn as_ref(&self) -> &T { + self + } +} + +impl Debug for MetadataRef { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + Debug::fmt(self.as_ref(), f) + } +} + +impl Display for MetadataRef { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + Display::fmt(self.as_ref(), f) + } +} + +impl PartialEq for MetadataRef { + fn eq(&self, other: &Self) -> bool { + self.as_arc() == other.as_arc() + } +} + +impl Eq for MetadataRef {} + +impl Hash for MetadataRef { + fn hash(&self, state: &mut H) { + self.as_ref().hash(state); + } +} + +impl Serialize for MetadataRef { + fn serialize(&self, serializer: S) -> Result { + self.as_ref().serialize(serializer) + } +} + +impl<'de, T: Eq + Hash + Deserialize<'de>> Deserialize<'de> for MetadataRef { + fn deserialize>(deserializer: D) -> Result { + // Decoding must not register values in the crate's process-global pool. + T::deserialize(deserializer).map(Self::from) + } +} + +/// Engine-local sharing. File and expression operations never hold nested locks. +/// The final registered reference removes its entry. Hash table capacity remains +/// allocated until the interner is dropped. +pub struct FileMetadataInterner { + inner: Arc, +} + +/// Shared storage observed through weak references by the metrics collector. +#[derive(Default)] +pub(crate) struct FileMetadataInternerState { + files: [HashInterner>; SHARD_COUNT], + expressions: [HashInterner>; SHARD_COUNT], + hasher: RandomState, +} + +/// Shared engine-local metadata interner. +pub type FileMetadataInternerRef = Arc; + +impl Default for FileMetadataInterner { + fn default() -> Self { + let inner = Arc::new(FileMetadataInternerState::default()); + FILE_METADATA_COLLECTOR.register(&inner); + Self { inner } + } +} + +impl FileMetadataInternerState { + /// Counts registered values, not runtime views or allocations outside this interner. + /// Each shard exposes its entry count without traversing entries. + pub(crate) fn entry_counts(&self) -> (usize, usize) { + ( + self.files.iter().map(HashInterner::len).sum(), + self.expressions.iter().map(HashInterner::len).sum(), + ) + } +} + +impl Debug for FileMetadataInterner { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("FileMetadataInterner") + .finish_non_exhaustive() + } +} + +impl FileMetadataInterner { + /// Creates an interner with automatic reclamation and scrape-time metrics. + pub fn new() -> FileMetadataInternerRef { + Arc::new(Self::default()) + } + + /// Normalizes exclusively owned metadata before publishing it to readers. + /// Shared inputs remain immutable: expression deduplication must not deep-clone + /// indexes merely to replace an equivalent expression. + pub(crate) fn intern_in_place(&self, meta: &mut FileMetaRef) { + if let Some(expr) = &meta.partition_expr { + let table_id = meta.region_id.table_id(); + let shard = self.inner.hasher.hash_one(table_id) as usize % SHARD_COUNT; + let shared = MetadataRef(MetadataRefInner::Interned( + self.inner.expressions[shard].intern_sized(ScopedMetadata { + table_id, + value: expr.as_arc().clone(), + }), + )); + if let Some(owned) = meta.get_mut() { + owned.partition_expr = Some(shared); + } + } + let key = RegionFileId::new(meta.region_id, meta.file_id); + let shard = self.inner.hasher.hash_one(key) as usize % SHARD_COUNT; + *meta = MetadataRef(MetadataRefInner::Interned( + self.inner.files[shard].intern_sized(ScopedMetadata { + table_id: meta.region_id.table_id(), + value: meta.as_arc().clone(), + }), + )); + } + + /// Shares only fully equal metadata; file identity alone is insufficient. + pub(crate) fn intern(&self, mut meta: FileMetaRef) -> FileMetaRef { + self.intern_in_place(&mut meta); + meta + } +} + +#[cfg(test)] +mod tests { + use store_api::storage::{FileId, RegionId}; + + use super::*; + use crate::sst::file::FileMeta; + + #[test] + fn test_retirement_after_delayed_reader_release() { + let interner = FileMetadataInterner::new(); + let values: Vec<_> = (0..4096) + .map(|_| { + interner.intern( + FileMeta { + region_id: RegionId::new(1, 1), + file_id: FileId::random(), + partition_expr: Some( + partition::expr::col("tag") + .lt(datatypes::value::Value::UInt32(10)) + .into(), + ), + ..Default::default() + } + .into(), + ) + }) + .collect(); + let reader = values[0].clone(); + drop(values); + assert_eq!( + interner + .inner + .files + .iter() + .map(HashInterner::len) + .sum::(), + 1 + ); + assert_eq!( + interner + .inner + .expressions + .iter() + .map(HashInterner::len) + .sum::(), + 1 + ); + let recovered = interner.intern(reader.as_ref().clone().into()); + assert!(MetadataRef::ptr_eq(&reader, &recovered)); + drop(recovered); + drop(reader); + assert!(interner.inner.files.iter().all(HashInterner::is_empty)); + assert!( + interner + .inner + .expressions + .iter() + .all(HashInterner::is_empty) + ); + } + + #[test] + fn test_historical_expression_types_share_only_equal_metadata() { + use datatypes::value::Value; + use partition::expr::col; + + let interner = FileMetadataInterner::default(); + let mut retained = Vec::new(); + let values = [ + Value::Int32(10), + Value::Int64(10), + Value::UInt32(10), + serde_json::from_value(serde_json::json!({ + "Decimal128": {"value": 10, "precision": 10, "scale": 0} + })) + .unwrap(), + serde_json::from_value(serde_json::json!({ + "Decimal128": {"value": 10, "precision": 10, "scale": 1} + })) + .unwrap(), + ]; + let expected_count = values.len(); + for value in values { + let meta = FileMeta { + region_id: RegionId::new(1, 1), + partition_expr: Some(col("tag").lt(value).into()), + ..Default::default() + }; + // Recovery creates independent allocations for historical metadata. + let encoded = serde_json::to_vec(&meta).unwrap(); + let first = interner.intern(serde_json::from_slice(&encoded).unwrap()); + let recovered = interner.intern(serde_json::from_slice(&encoded).unwrap()); + assert!(MetadataRef::ptr_eq(&first, &recovered)); + for old in &retained { + assert!(!MetadataRef::ptr_eq(old, &first)); + assert_eq!( + interner.inner.hasher.hash_one(old), + interner.inner.hasher.hash_one(&first) + ); + assert!(!MetadataRef::ptr_eq( + old.partition_expr.as_ref().unwrap(), + first.partition_expr.as_ref().unwrap(), + )); + } + retained.push(first); + } + assert_eq!( + (expected_count, expected_count), + interner.inner.entry_counts() + ); + drop(retained); + assert_eq!((0, 0), interner.inner.entry_counts()); + } + + #[test] + fn test_metadata_views_and_recovered_copies() { + let pool = FileMetadataInterner::default(); + let original = MetadataRef::from(FileMeta { + region_id: RegionId::new(1, 1), + file_id: FileId::random(), + partition_expr: PartitionExpr::from_json_str( + r#"{"Expr":{"lhs":{"Column":"tag"},"op":"GtEq","rhs":{"Value":{"UInt32":10}}}}"#, + ) + .unwrap() + .map(MetadataRef::from), + ..Default::default() + }); + let encoded = serde_json::to_vec(original.as_ref()).unwrap(); + let first = pool.intern(original); + let recovered = pool.intern(serde_json::from_slice(&encoded).unwrap()); + assert!(MetadataRef::ptr_eq(&first, &recovered)); + assert_eq!(encoded, serde_json::to_vec(&recovered).unwrap()); + + let mut changed = first.as_ref().clone(); + changed.index_version += 1; + let changed = pool.intern(MetadataRef::from(changed)); + assert!(!MetadataRef::ptr_eq(&first, &changed)); + assert_eq!(first.index_version, 0); + assert_eq!(changed.index_version, 1); + + let mut other_file = first.as_ref().clone(); + other_file.file_id = FileId::random(); + other_file.region_id = RegionId::new(1, 2); + let other_file = + pool.intern(serde_json::from_slice(&serde_json::to_vec(&other_file).unwrap()).unwrap()); + assert!(!MetadataRef::ptr_eq(&first, &other_file)); + assert!(MetadataRef::ptr_eq( + first.partition_expr.as_ref().unwrap(), + other_file.partition_expr.as_ref().unwrap() + )); + } + + #[test] + fn test_table_pool_keeps_historical_expressions_distinct() { + let pool = FileMetadataInterner::default(); + let old_expr = partition::expr::col("tag").lt(datatypes::value::Value::UInt32(10)); + let new_expr = partition::expr::col("tag").lt(datatypes::value::Value::UInt32(5)); + let old = pool.intern(MetadataRef::from(FileMeta { + region_id: RegionId::new(1, 1), + partition_expr: Some(MetadataRef::from(old_expr.clone())), + ..Default::default() + })); + for (region_id, expr) in [ + (RegionId::new(1, 2), new_expr.clone()), + (RegionId::new(2, 1), old_expr.clone()), + ] { + let other = pool.intern(MetadataRef::from(FileMeta { + region_id, + partition_expr: Some(MetadataRef::from(expr.clone())), + ..Default::default() + })); + assert!(!MetadataRef::ptr_eq( + old.partition_expr.as_ref().unwrap(), + other.partition_expr.as_ref().unwrap(), + )); + assert_eq!(Some(&expr), other.partition_expr.as_deref()); + } + assert_eq!(Some(&old_expr), old.partition_expr.as_deref()); + } + + #[test] + fn test_same_index_different_contents_remain_shared() { + let pool = FileMetadataInterner::default(); + let first = pool.intern(MetadataRef::from(FileMeta::default())); + let second = pool.intern(MetadataRef::from(FileMeta { + file_size: 123, + ..first.as_ref().clone() + })); + assert!(!MetadataRef::ptr_eq(&first, &second)); + assert!(MetadataRef::ptr_eq( + &first, + &pool.intern(MetadataRef::from(first.as_ref().clone())) + )); + assert!(MetadataRef::ptr_eq( + &second, + &pool.intern(MetadataRef::from(second.as_ref().clone())) + )); + } + #[test] + fn test_concurrent_publication_and_retirement() { + let interner = FileMetadataInterner::new(); + let collector = crate::metrics::file_metadata::FileMetadataCollector::default(); + collector.register(&interner.inner); + let barrier = std::sync::Barrier::new(8); + std::thread::scope(|scope| { + let workers: Vec<_> = (0..8) + .map(|_| { + let interner = &interner; + let barrier = &barrier; + let collector = &collector; + scope.spawn(move || { + barrier.wait(); + for i in 0..1000 { + if i % 64 == 0 { + use prometheus::core::Collector; + assert_eq!(collector.collect().len(), 1); + } + let meta = FileMeta { + region_id: RegionId::new(1, 1), + file_size: i % 16, + partition_expr: Some( + partition::expr::col("tag") + .lt(datatypes::value::Value::UInt32(10)) + .into(), + ), + ..Default::default() + }; + let first = interner.intern(meta.clone().into()); + let second = interner.intern(meta.into()); + assert!(MetadataRef::ptr_eq(&first, &second)); + assert!(MetadataRef::ptr_eq( + first.partition_expr.as_ref().unwrap(), + second.partition_expr.as_ref().unwrap() + )); + } + }) + }) + .collect(); + for worker in workers { + worker.join().unwrap(); + } + }); + assert!(interner.inner.files.iter().all(HashInterner::is_empty)); + assert!( + interner + .inner + .expressions + .iter() + .all(HashInterner::is_empty) + ); + } + + #[test] + fn test_engine_isolation_and_reader_outlives_interner() { + let first = FileMetadataInterner::new(); + let second = FileMetadataInterner::new(); + let left = first.intern(FileMeta::default().into()); + let right = second.intern(FileMeta::default().into()); + assert!(!MetadataRef::ptr_eq(&left, &right)); + drop(first); + assert_eq!(*left, *right); + } + #[test] + fn test_scrape_counts_distinct_values_and_releases_sources() { + use std::collections::BTreeMap; + + use crate::metrics::file_metadata::FileMetadataCollector; + + let collector = FileMetadataCollector::default(); + let registry = prometheus::Registry::new(); + registry.register(Box::new(collector.clone())).unwrap(); + let counts = || { + let families = registry.gather(); + assert_eq!(families.len(), 1); + assert_eq!( + families[0].name(), + "greptime_mito_sst_metadata_interner_entries" + ); + families[0] + .get_metric() + .iter() + .map(|metric| { + assert_eq!(metric.get_label().len(), 1); + let label = &metric.get_label()[0]; + assert_eq!(label.name(), "type"); + ( + label.value().to_string(), + metric.get_gauge().value() as usize, + ) + }) + .collect::>() + }; + let expected = |files, expressions| { + BTreeMap::from([ + ("file".to_string(), files), + ("partition_expr".to_string(), expressions), + ]) + }; + assert_eq!(counts(), expected(0, 0)); + let first = FileMetadataInterner::default(); + let second = FileMetadataInterner::new(); + collector.register(&first.inner); + collector.register(&second.inner); + let make_meta = || FileMeta { + region_id: RegionId::new(1, 1), + partition_expr: Some( + partition::expr::col("tag") + .lt(datatypes::value::Value::UInt32(10)) + .into(), + ), + ..Default::default() + }; + let value = first.intern(make_meta().into()); + let view = first.intern(make_meta().into()); + let other_engine = second.intern(make_meta().into()); + let changed = first.intern( + FileMeta { + file_size: 1, + ..make_meta() + } + .into(), + ); + // Views deduplicate, content versions and engine scopes do not. + assert_eq!(counts(), expected(3, 2)); + drop(value); + drop(changed); + drop(other_engine); + assert_eq!(counts(), expected(1, 1)); + drop(view); + assert_eq!(counts(), expected(0, 0)); + + let reader = first.intern(make_meta().into()); + let weak = Arc::downgrade(&first.inner); + drop(first); + drop(second); + assert!(weak.upgrade().is_none()); + // Counts describe registered entries in live interners, not detached readers. + assert_eq!(counts(), expected(0, 0)); + assert_eq!(reader.region_id, RegionId::new(1, 1)); + } +} diff --git a/src/mito2/src/sst/file_ref.rs b/src/mito2/src/sst/file_ref.rs index b82a7b9c217..eb9c8373493 100644 --- a/src/mito2/src/sst/file_ref.rs +++ b/src/mito2/src/sst/file_ref.rs @@ -23,6 +23,7 @@ use crate::error::Result; use crate::metrics::GC_REF_FILE_CNT; use crate::region::MitoRegionRef; use crate::sst::file::FileMeta; +use crate::sst::file_metadata::{FileMetadataInterner, FileMetadataInternerRef}; /// File references grouped by their physical source region. #[derive(Debug, Clone, Default)] @@ -38,6 +39,7 @@ pub struct FileReferenceManager { /// this node in region manifests. node_id: Option, files_per_region: DashMap, + pub(crate) file_metadata_interner: FileMetadataInternerRef, /// Whether global GC is enabled. /// This is only meaningful when using object store (not local filesystem). gc_enabled: bool, @@ -50,6 +52,7 @@ impl FileReferenceManager { Self { node_id, files_per_region: Default::default(), + file_metadata_interner: FileMetadataInterner::new(), gc_enabled: false, } } @@ -59,6 +62,7 @@ impl FileReferenceManager { Self { node_id, files_per_region: Default::default(), + file_metadata_interner: FileMetadataInterner::new(), gc_enabled, } } diff --git a/src/mito2/src/sst/index.rs b/src/mito2/src/sst/index.rs index 47f7834235f..2ef1cef59e0 100644 --- a/src/mito2/src/sst/index.rs +++ b/src/mito2/src/sst/index.rs @@ -63,8 +63,8 @@ use crate::cache::{CacheManagerRef, CacheStrategy}; #[cfg(test)] use crate::config::{BloomFilterConfig, InvertedIndexConfig}; use crate::error::{ - BuildIndexAsyncSnafu, Error, ExternalSnafu, InvalidConfigSnafu, JoinSnafu, RegionClosedSnafu, - RegionDroppedSnafu, RegionTruncatedSnafu, Result, + BuildIndexAsyncSnafu, Error, ExternalSnafu, InvalidConfigSnafu, JoinSnafu, + OperateAbortedIndexSnafu, RegionClosedSnafu, RegionDroppedSnafu, RegionTruncatedSnafu, Result, }; use crate::metrics::{ INDEX_ARTIFACT_CLEANUP_FAILURE_TOTAL, INDEX_CREATE_DISCARDED_TOTAL, INDEX_CREATE_MEMORY_USAGE, @@ -512,6 +512,20 @@ pub(crate) struct IndexerBuilderImpl { pub(crate) index_options: IndexOptions, } +impl IndexerBuilderImpl { + fn plan(&self) -> Arc { + if let Some(inputs) = &self.index_build_inputs { + return inputs.plan.clone(); + } + Arc::new(self.registry.plan(&handler::PlanContext { + metadata: &self.metadata, + index_options: &self.index_options, + build_type: self.build_type.clone(), + existing_indexes: &[], + })) + } +} + #[async_trait::async_trait] impl IndexerBuilder for IndexerBuilderImpl { /// Sanity check for arguments and create a new [Indexer] if arguments are valid. @@ -530,17 +544,7 @@ impl IndexerBuilder for IndexerBuilderImpl { ..Default::default() }; - let plan = if let Some(inputs) = &self.index_build_inputs { - inputs.plan.clone() - } else { - let context = handler::PlanContext { - metadata: &self.metadata, - index_options: &self.index_options, - build_type: self.build_type.clone(), - existing_indexes: &[], - }; - Arc::new(self.registry.plan(&context)) - }; + let plan = self.plan(); indexer.failure = plan.check_success().err(); let mut context = handler::BuildContext { metadata: &self.metadata, @@ -796,7 +800,8 @@ pub struct IndexBuildTask { pub file_purger: FilePurgerRef, /// When write cache is enabled, the indexer builder should be built from the write cache. /// Otherwise, it should be built from the access layer. - pub indexer_builder: Arc, + /// Rebuild owns plan assembly; external implementations enter through the registry. + pub(crate) indexer_builder: Arc, /// Request sender to notify the region worker. pub(crate) request_sender: Sender, /// Senders of callers waiting for this build. Coalesced tasks and @@ -957,16 +962,20 @@ impl IndexBuildTask { // Taken before `prepare_source` replaces the source, and used for the // index files and listener events of this build. let region_file_id = self.region_file_id(); - if let ControlFlow::Break(report) = - self.prepare_source(region_file_id, &version_control).await + if let ControlFlow::Break(report) = self + .prepare_source(region_file_id, &version_control) + .await? { return Ok(report); } - let new_index_version = self - .source - .file_meta - .index_version() - .map_or(0, |version| version + 1); + // Metadata-only removal leaves no current index, but its version still reserves + // every older path until GC finishes. Never reuse those paths on a later build. + let file_meta = &self.source.file_meta; + let new_index_version = if file_meta.index_version > 0 || file_meta.exists_index() { + file_meta.index_version + 1 + } else { + 0 + }; let (mut indexer, index_output) = match self.create_index(region_file_id, new_index_version).await? { ControlFlow::Continue(created) => created, @@ -992,13 +1001,13 @@ impl IndexBuildTask { &mut self, region_file_id: RegionFileId, version_control: &VersionControlRef, - ) -> ControlFlow { + ) -> Result> { // Check SST file existence before building index to avoid failure of parquet reader. if !self.check_sst_file_exists(version_control).await { - return ControlFlow::Break( + return Ok(ControlFlow::Break( self.abort_and_notify(region_file_id, "SST file not found during index build") .await, - ); + )); } // A task may have waited behind another build of the same SST. Build on @@ -1011,41 +1020,96 @@ impl IndexBuildTask { .get(&self.source.file_meta.file_id) .cloned(); let Some(current_file_meta) = current_file_meta else { - return ControlFlow::Break( + return Ok(ControlFlow::Break( self.abort_and_notify( region_file_id, "SST file not found in manifest during index build", ) .await, - ); + )); }; + if matches!( + self.reason, + IndexBuildType::Manual | IndexBuildType::SchemaChange + ) { + self.indexer_builder + .registry + .validate_schema_kinds(&self.target_region_metadata) + .with_context(|_| ExternalSnafu { + context: "validating index handlers for rebuild".to_string(), + })?; + } + let context = handler::PlanContext { + metadata: &self.target_region_metadata, + index_options: &self.index_options, + build_type: self.reason.clone(), + existing_indexes: ¤t_file_meta.indexes, + }; + let plan = Arc::new(self.indexer_builder.registry.plan(&context)); + plan.check_success().context(BuildIndexAsyncSnafu { + region_id: self.region_id, + })?; + // Manual requests skip current indexes; schema changes must run their maintenance plans. if self.reason == IndexBuildType::Manual && current_file_meta.is_index_consistent_with_region( &self.target_region_metadata.column_metadatas, &self.index_options, ) + && plan.is_current(¤t_file_meta, &context) { - return ControlFlow::Break( + return Ok(ControlFlow::Break( self.abort_and_notify(region_file_id, "Index is already up to date") .await, - ); + )); } - self.source.file_meta = current_file_meta; - ControlFlow::Continue(()) + let current_file = self.file.with_updated_index(current_file_meta.clone()); + let builder = Arc::make_mut(&mut self.indexer_builder); + let output_rows = (current_file_meta.num_rows != 0).then_some(current_file_meta.num_rows); + if let Some(inputs) = &mut builder.index_build_inputs { + inputs.plan = plan; + inputs.inputs = vec![current_file.clone()].into(); + inputs.output_rows = output_rows; + } else { + builder.index_build_inputs = Some(build::IndexBuildInputs { + plan, + inputs: vec![current_file.clone()].into(), + input_puffin_manager: self.access_layer.build_puffin_manager(), + can_merge_inputs: self + .target_region_metadata + .partition_expr + .as_deref() + .is_none_or(str::is_empty) + && current_file_meta.region_id == self.region_id, + output_rows, + memory_limit: None, + }); + } + self.file = current_file; + self.source.file_meta = current_file_meta.as_ref().clone(); + Ok(ControlFlow::Continue(())) } - /// Reads the SST and creates the index files. + /// Builds a replacement index file, scanning data only when a creator needs rows. async fn create_index( &self, region_file_id: RegionFileId, new_index_version: u64, ) -> Result> { - let mut parquet_reader = self + let reader_builder = self .access_layer - .read_sst(self.file.clone()) // use the latest file handle instead of creating a new one - .expected_metadata(Some(self.target_region_metadata.clone())) - .build() - .await?; + .read_sst(self.file.clone()) + .expected_metadata(Some(self.target_region_metadata.clone())); + let plan = self.indexer_builder.plan(); + // Inverted segments must align with the actual SST row groups. Other + // creators can decide input reuse without opening the Parquet file. + let needs_row_group_size = plan + .entries() + .any(|entry| entry.kind.index_type() == IndexType::InvertedIndex); + let mut parquet_reader = if needs_row_group_size { + reader_builder.build().await? + } else { + None + }; let row_group_size = parquet_reader.as_ref().and_then(|reader| { reader @@ -1055,39 +1119,139 @@ impl IndexBuildTask { .map(|row_group| row_group.num_rows() as usize) .filter(|size| *size > 0) }); - let mut indexer = self .indexer_builder .build(region_file_id, new_index_version, row_group_size) .await; - // The build type or index options may leave nothing to build. Return before reading - // any row; an empty indexer finishes with no output, which publishes nothing. + self.check_rebuild_indexer_or_abort(&mut indexer).await?; if indexer.is_empty() { + if self.can_remove_all_indexes() { + return Ok(ControlFlow::Continue((indexer, IndexOutput::default()))); + } return Ok(ControlFlow::Break(self.aborted("No index to build"))); } - if let Some(mut parquet_reader) = parquet_reader.take() { - // TODO(SNC123): optimize index batch - loop { - match parquet_reader.next_record_batch().await { + if indexer.requires_row_scan() && !needs_row_group_size { + parquet_reader = match reader_builder.build().await { + Ok(reader) => reader, + Err(err) => { + indexer.abort().await; + return Err(err); + } + }; + } + if indexer.requires_row_scan() + && let Some(mut reader) = parquet_reader.take() + { + while indexer.requires_row_scan() { + match reader.next_record_batch().await { Ok(Some(batch)) => { indexer.update_flat(&batch).await; + self.check_rebuild_indexer_or_abort(&mut indexer).await?; } Ok(None) => break, - Err(e) => { + Err(err) => { indexer.abort().await; - return Err(e); + return Err(err); } } } } - let index_output = indexer.finish().await; - - if index_output.file_size == 0 { - // The indexer logs and drops index creation failures. + let output = indexer.finish().await; + self.check_rebuild_indexer_or_abort(&mut indexer).await?; + if let Err(err) = self.check_rebuild_output(&output) { + indexer.abort().await; + return Err(err).context(BuildIndexAsyncSnafu { + region_id: self.region_id, + }); + } + if output.file_size == 0 { return Ok(ControlFlow::Break(self.aborted("No index was created"))); } - Ok(ControlFlow::Continue((indexer, index_output))) + Ok(ControlFlow::Continue((indexer, output))) + } + + /// Aborts a failed replacement before propagating the rebuild error. + async fn check_rebuild_indexer_or_abort(&self, indexer: &mut Indexer) -> Result<()> { + if let Err(err) = self.check_rebuild_indexer(indexer) { + indexer.abort().await; + return Err(err).context(BuildIndexAsyncSnafu { + region_id: self.region_id, + }); + } + Ok(()) + } + + /// A budget skip may omit a new kind, but must not erase an existing one. + fn check_rebuild_indexer(&self, indexer: &Indexer) -> Result<(), Arc> { + indexer.check_success()?; + let source = &self.source.file_meta; + if indexer.discarded().iter().any(|kind| { + let index_type = kind.index_type(); + source.available_indexes.contains(&index_type) + || source + .indexes + .iter() + .any(|column| column.created_indexes.contains(&index_type)) + }) { + warn!( + "Preserving previous index generation after budget discard, region_id: {}, file_id: {}, discarded: {:?}", + self.region_id, + source.file_id, + indexer.discarded() + ); + return Err(Arc::new(OperateAbortedIndexSnafu.build())); + } + Ok(()) + } + + /// A skipped creator or partial completion must not erase a requested existing extension. + fn check_rebuild_output(&self, output: &IndexOutput) -> Result<(), Arc> { + let source = &self.source.file_meta; + for entry in self.indexer_builder.plan().entries() { + if entry.kind.is_builtin() { + continue; + } + let index_type = entry.kind.index_type(); + let existing_column = source.indexes.iter().any(|column| { + column.column_id == entry.column_id && column.created_indexes.contains(&index_type) + }); + // Legacy file-level descriptors do not identify individual indexed columns. + let legacy_kind = source.available_indexes.contains(&index_type) + && !source + .indexes + .iter() + .any(|column| column.created_indexes.contains(&index_type)); + if (existing_column || legacy_kind) + && !output.entries.iter().any(|completed| { + completed.kind == entry.kind + && (legacy_kind || completed.column_id == Some(entry.column_id)) + }) + { + warn!( + "Preserving previous index generation after incomplete rebuild, region_id: {}, file_id: {}, column_id: {}, kind: {}", + self.region_id, + source.file_id, + entry.column_id, + entry.kind.as_str() + ); + return Err(Arc::new(OperateAbortedIndexSnafu.build())); + } + } + Ok(()) + } + + /// Disabled options may remove old indexes after handler validation. + /// A failed plan is not an empty removal plan and must retain the old file. + fn can_remove_all_indexes(&self) -> bool { + self.source.file_meta.exists_index() + && self + .indexer_builder + .index_build_inputs + .as_ref() + .is_some_and(|inputs| { + inputs.plan.check_success().is_ok() && inputs.plan.entries().next().is_none() + }) } /// Uploads the index files and commits them to the manifest. @@ -1171,7 +1335,7 @@ impl IndexBuildTask { // cannot storm. self.notify_worker(BackgroundNotify::IndexBuildRetry(BuildIndexRequest { region_id: self.region_id, - build_type: IndexBuildType::SchemaChange, + build_type: self.reason.clone(), file_metas: vec![self.source.file_meta.clone()], result_senders: self.result_senders.clone(), })) @@ -1185,6 +1349,9 @@ impl IndexBuildTask { index_file_id: FileId, index_version: u64, ) -> Result<()> { + if output.file_size == 0 { + return Ok(()); + } if let Some(write_cache) = &self.write_cache { let file_id = self.source.file_meta.file_id; let region_id = self.source.file_meta.region_id; @@ -1448,6 +1615,25 @@ impl IndexBuildScheduler { &mut self, version_control: &VersionControlRef, task: IndexBuildTask, + ) { + self.schedule_build_inner(version_control, task, false) + .await; + } + + /// A retry is queued before its previous task releases the active slot. + pub(crate) async fn schedule_retry( + &mut self, + version_control: &VersionControlRef, + task: IndexBuildTask, + ) { + self.schedule_build_inner(version_control, task, true).await; + } + + async fn schedule_build_inner( + &mut self, + version_control: &VersionControlRef, + task: IndexBuildTask, + is_retry: bool, ) { let status = self .region_status @@ -1461,7 +1647,8 @@ impl IndexBuildScheduler { ); // A manual build must not report success while an active build that may // not create the requested indexes is still running, so it waits as well. - let can_wait_for_active = status.retiring + let can_wait_for_active = is_retry + || status.retiring || matches!( task.reason, IndexBuildType::SchemaChange | IndexBuildType::Manual @@ -1633,7 +1820,7 @@ impl IndexBuildScheduler { #[cfg(test)] mod tests { - use std::sync::Arc; + use std::sync::{Arc, Mutex}; use api::v1::SemanticType; use common_base::readable_size::ReadableSize; @@ -1663,7 +1850,7 @@ mod tests { use crate::region::version::{VersionBuilder, VersionControl}; use crate::schedule::scheduler::{LocalScheduler, Scheduler}; use crate::sst::file::{FileMeta, RegionFileId}; - use crate::sst::file_purger::NoopFilePurger; + use crate::sst::file_purger::{FilePurger, NoopFilePurger}; use crate::sst::location; use crate::sst::parquet::WriteOptions; use crate::test_util::memtable_util::EmptyMemtableBuilder; @@ -2080,6 +2267,13 @@ mod tests { } impl handler::IndexKindHandler for TestProvider { + fn validate( + &self, + _context: &handler::PlanContext<'_>, + ) -> std::result::Result<(), common_error::ext::BoxedError> { + Ok(()) + } + fn kinds(&self) -> &[kind::IndexKind] { match self.kind { Some(kind::ExtensionIndexKind::Kll) => { @@ -2118,6 +2312,9 @@ mod tests { }], } } + fn is_current(&self, _: &crate::sst::file::FileMeta, _: &handler::PlanContext<'_>) -> bool { + false + } fn memory_limit(&self) -> Option { self.hard_limit @@ -2183,6 +2380,10 @@ mod tests { Reused, Budget, Create, + SkipCreate, + EmptyFinish, + CompletePair, + PartialFinish, Update, UpdateFlat, Finish, @@ -2206,6 +2407,13 @@ mod tests { } impl handler::IndexKindHandler for FailingProvider { + fn validate( + &self, + _context: &handler::PlanContext<'_>, + ) -> std::result::Result<(), common_error::ext::BoxedError> { + Ok(()) + } + fn kinds(&self) -> &[kind::IndexKind] { &[kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll)] } @@ -2235,12 +2443,32 @@ mod tests { handler::IndexKindHandler::kinds(self) } fn entries(&self) -> &[handler::PlanEntry] { + if matches!( + self.0, + ProviderFailure::CompletePair | ProviderFailure::PartialFinish + ) { + return &[ + handler::PlanEntry { + column_id: 0, + kind: kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll), + format_version: 2, + }, + handler::PlanEntry { + column_id: 1, + kind: kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll), + format_version: 2, + }, + ]; + } &[handler::PlanEntry { column_id: 0, kind: kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll), format_version: 2, }] } + fn is_current(&self, _: &crate::sst::file::FileMeta, _: &handler::PlanContext<'_>) -> bool { + false + } fn memory_limit(&self) -> Option { (self.0 == ProviderFailure::Budget).then_some(16) @@ -2256,6 +2484,9 @@ mod tests { if self.0 == ProviderFailure::Create { return Err(provider_test_error()); } + if self.0 == ProviderFailure::SkipCreate { + return Ok(None); + } Ok(Some(Box::new(FailingCreator(self.0)))) } } @@ -2300,6 +2531,9 @@ mod tests { ) -> std::result::Result, common_error::ext::BoxedError> { use puffin::puffin_manager::PuffinWriter; + if self.0 == ProviderFailure::EmptyFinish { + return Ok(Vec::new()); + } writer .put_blob( "test-sketch", @@ -2313,11 +2547,18 @@ mod tests { // Fail after writing to exercise partially written artifact handling. return Err(provider_test_error()); } - Ok(vec![CompletedEntry { + let mut entries = vec![CompletedEntry { column_id: Some(0), kind: kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll), format_version: 2, - }]) + }]; + if self.0 == ProviderFailure::CompletePair { + entries.push(CompletedEntry { + column_id: Some(1), + ..entries[0].clone() + }); + } + Ok(entries) } async fn abort(&mut self) -> std::result::Result<(), common_error::ext::BoxedError> { @@ -2475,6 +2716,41 @@ mod tests { } } + #[tokio::test] + async fn test_extension_finish_failure_discards_completed_builtin_output() { + for failure in [ProviderFailure::None, ProviderFailure::Finish] { + let env = SchedulerEnv::new().await; + let mut builder = mock_indexer_builder(Arc::new(sst_region_metadata()), &env).await; + let builder = Arc::make_mut(&mut builder); + builder.build_type = IndexBuildType::Compact; + Arc::make_mut(&mut builder.registry) + .register(Arc::new(FailingProvider(failure))) + .unwrap(); + let mut indexer = builder.build(random_region_file_id(), 0, Some(1024)).await; + // Registry order puts real built-in creators before the failing extension. + assert!(indexer.creators.first().unwrap().is_builtin()); + assert!(!indexer.creators.last().unwrap().is_builtin()); + indexer + .update_flat(&new_record_batch_by_range(&["a", "b"], 0, 10)) + .await; + assert!(indexer.check_success().is_ok()); + let output = indexer.finish().await; + if failure == ProviderFailure::None { + assert!(output.entries.iter().any(|entry| entry.kind.is_builtin())); + assert!(output.entries.iter().any(|entry| !entry.kind.is_builtin())); + assert!(output.file_size > 0); + assert!(indexer.check_success().is_ok()); + } else { + // A shared writer cannot publish the preceding successful index alone. + assert!(output.entries.is_empty()); + assert!(output.build_indexes().unwrap().is_empty()); + assert_eq!(output.file_size, 0); + assert!(indexer.check_success().is_err()); + } + assert!(indexer.creators.is_empty()); + } + } + #[tokio::test] async fn test_provider_compaction_only_and_per_output() { let (dir, factory) = PuffinManagerFactory::new_for_test_async("provider_scope").await; @@ -3154,6 +3430,11 @@ mod tests { file_size: sst_info.file_size, max_row_group_uncompressed_size: sst_info.max_row_group_uncompressed_size, available_indexes: smallvec![IndexType::InvertedIndex], + indexes: vec![ColumnIndexMetadata { + column_id: 2, + created_indexes: smallvec::smallvec![IndexType::InvertedIndex], + ..Default::default() + }], // Old manifests may publish an index without recording its size. index_file_size: 0, index_version: 0, @@ -3210,6 +3491,18 @@ mod tests { assert!(updated_meta.index_file_size > 0); assert_eq!(updated_meta.file_id, file_meta.file_id); assert_eq!(updated_meta.index_version, 1); + // This built-in-only rebuild must not advertise extension indexes. + assert!( + updated_meta + .indexes + .iter() + .all(|index| index.created_indexes.iter().all(|kind| !matches!( + kind, + crate::sst::file::IndexType::HllIndex + | crate::sst::file::IndexType::KllIndex + | crate::sst::file::IndexType::MgIndex + ))) + ); let field_0_index = updated_meta .indexes .iter() @@ -3343,8 +3636,489 @@ mod tests { schedule_index_build_task_with_mode(IndexBuildMode::Sync).await; } + #[derive(Debug, Default)] + struct RecordingRebuildPurger(Mutex>); + + impl FilePurger for RecordingRebuildPurger { + fn purge_file(&self, meta: FileMeta) { + self.0.lock().unwrap().push(meta); + } + } + + #[rstest::rstest] + #[case(false)] + #[case(true)] + #[tokio::test] + async fn test_rebuild_metadata_refresh_retains_sst_lifetime(#[case] unknown_rows: bool) { + let env = SchedulerEnv::new().await; + let metadata = Arc::new(sst_region_metadata()); + let manifest_ctx = env.mock_manifest_context(metadata.clone()).await; + let purger = Arc::new(RecordingRebuildPurger::default()); + let info = mock_sst_file(metadata.clone(), &env, IndexBuildMode::Async).await; + let meta = FileMeta { + region_id: metadata.region_id, + file_id: info.file_id, + file_size: info.file_size, + num_rows: if unknown_rows { + 0 + } else { + info.num_rows as u64 + }, + num_row_groups: info.num_row_groups, + ..Default::default() + }; + let mut refreshed = meta.clone(); + refreshed.index_version = 1; + seed_manifest_file(&manifest_ctx, &refreshed).await; + let version = mock_version_control( + metadata.clone(), + Arc::new(NoopFilePurger), + HashMap::from([(meta.file_id, refreshed.clone())]), + ) + .await; + let original = FileHandle::new(meta.clone(), purger.clone()); + let builder = mock_indexer_builder(metadata.clone(), &env).await; + let (tx, _rx) = mpsc::channel(4); + let mut task = IndexBuildTask { + region_id: metadata.region_id, + file: original.clone(), + target_region_metadata: metadata.clone(), + source: IndexBuildSource::new(meta.clone(), metadata.schema_version), + reason: IndexBuildType::SchemaChange, + access_layer: env.access_layer.clone(), + listener: WorkerListener::default(), + manifest_ctx, + write_cache: None, + cache_manager: None, + file_purger: purger.clone(), + indexer_builder: builder, + request_sender: tx, + index_options: IndexOptions::default(), + result_senders: Vec::new(), + }; + assert!(matches!( + task.prepare_source(meta.file_id(), &version).await.unwrap(), + ControlFlow::Continue(()) + )); + assert_eq!(task.file.meta_ref().index_version, 1); + assert_eq!( + task.indexer_builder + .index_build_inputs + .as_ref() + .unwrap() + .output_rows, + (!unknown_rows).then_some(info.num_rows as u64), + ); + let reader = task + .indexer_builder + .index_build_inputs + .as_ref() + .unwrap() + .inputs[0] + .clone(); + // Removal after preparation must wait for the task and its plugin readers. + original.mark_deleted(); + drop(original); + assert!(purger.0.lock().unwrap().is_empty()); + drop(task); + assert!(purger.0.lock().unwrap().is_empty()); + drop(reader); + assert_eq!(*purger.0.lock().unwrap(), vec![refreshed]); + } + + #[tokio::test] + async fn test_rebuild_scans_only_when_a_creator_requires_rows() { + for with_builtin in [false, true] { + let env = SchedulerEnv::new().await; + let mut metadata = sst_region_metadata(); + if !with_builtin { + for column in &mut metadata.column_metadatas { + column.column_schema.set_inverted_index(false); + column.column_schema.unset_skipping_options().unwrap(); + } + } + let metadata = Arc::new(metadata); + let manifest_ctx = env.mock_manifest_context(metadata.clone()).await; + let file_purger = Arc::new(NoopFilePurger {}); + let info = mock_sst_file(metadata.clone(), &env, IndexBuildMode::Async).await; + let file_meta = FileMeta { + region_id: metadata.region_id, + file_id: info.file_id, + file_size: info.file_size, + max_row_group_uncompressed_size: info.max_row_group_uncompressed_size, + num_rows: info.num_rows as u64, + num_row_groups: info.num_row_groups, + ..Default::default() + }; + seed_manifest_file(&manifest_ctx, &file_meta).await; + let version = mock_version_control( + metadata.clone(), + file_purger.clone(), + HashMap::from([(file_meta.file_id, file_meta.clone())]), + ) + .await; + let mut builder = mock_indexer_builder(metadata.clone(), &env).await; + let mutable = Arc::make_mut(&mut builder); + mutable.build_type = IndexBuildType::Manual; + Arc::make_mut(&mut mutable.registry) + .register(Arc::new(FailingProvider(ProviderFailure::Reused))) + .unwrap(); + let (tx, _rx) = mpsc::channel(4); + let mut task = IndexBuildTask { + region_id: metadata.region_id, + file: FileHandle::new(file_meta.clone(), file_purger.clone()), + target_region_metadata: metadata.clone(), + source: IndexBuildSource::new(file_meta.clone(), metadata.schema_version), + reason: IndexBuildType::Manual, + access_layer: env.access_layer.clone(), + listener: WorkerListener::default(), + manifest_ctx: manifest_ctx.clone(), + write_cache: None, + cache_manager: None, + file_purger, + indexer_builder: builder, + request_sender: tx, + index_options: IndexOptions::default(), + result_senders: Vec::new(), + }; + if !with_builtin { + // A reuse-only build must not open even the Parquet footer. + // Keep the file identity published, but make footer reads fail. + let path = task + .file + .file_path(env.access_layer.table_dir(), env.access_layer.path_type()); + env.access_layer + .object_store() + .write(&path, Bytes::from_static(b"not a parquet file")) + .await + .unwrap(); + } + // The reused creator rejects row updates, even when built-in creators need a scan. + task.index_build(version).await.unwrap(); + let published = manifest_ctx.manifest().await.files[&file_meta.file_id].clone(); + assert!(published.available_indexes.contains(&IndexType::HllIndex)); + assert!(published.index_file_size > 0); + let reader = env + .access_layer + .build_puffin_manager() + .reader(&RegionIndexId::new( + RegionFileId::new(metadata.region_id, file_meta.file_id), + published.index_version, + )) + .await + .unwrap(); + reader.blob("test-sketch").await.unwrap(); + if with_builtin { + assert!( + published + .available_indexes + .contains(&IndexType::InvertedIndex) + ); + let blob = reader.blob(inverted_index::INDEX_BLOB_TYPE).await.unwrap(); + let index_metadata = + index::inverted_index::format::reader::InvertedIndexBlobReader::new( + blob.reader().await.unwrap(), + ) + .metadata(None) + .await + .unwrap(); + assert_eq!(index_metadata.total_row_count, file_meta.num_rows); + } else { + assert_eq!( + published.available_indexes.as_slice(), + &[IndexType::HllIndex] + ); + } + } + } + + #[tokio::test] + async fn test_rebuild_failure_preserves_published_indexes() { + for (failure, old_hll, legacy, with_builtin) in [ + (Some(ProviderFailure::Plan), true, false, true), + (Some(ProviderFailure::Create), true, false, true), + (Some(ProviderFailure::UpdateFlat), true, false, true), + (Some(ProviderFailure::Finish), true, false, true), + (Some(ProviderFailure::Budget), true, false, true), + (Some(ProviderFailure::Budget), false, false, true), + (Some(ProviderFailure::SkipCreate), true, false, true), + (Some(ProviderFailure::SkipCreate), true, false, false), + (Some(ProviderFailure::SkipCreate), true, true, true), + (Some(ProviderFailure::SkipCreate), false, false, true), + (Some(ProviderFailure::EmptyFinish), true, false, true), + (Some(ProviderFailure::PartialFinish), true, false, true), + (Some(ProviderFailure::None), true, false, true), + (None, true, false, true), + (None, false, false, true), + ] { + let env = SchedulerEnv::new().await; + let mut metadata = sst_region_metadata(); + if !with_builtin { + for column in &mut metadata.column_metadatas { + column.column_schema.set_inverted_index(false); + column.column_schema.unset_skipping_options().unwrap(); + } + } + let metadata = Arc::new(metadata); + let region_id = metadata.region_id; + let manifest_ctx = env.mock_manifest_context(metadata.clone()).await; + let file_purger = Arc::new(NoopFilePurger {}); + let info = mock_sst_file(metadata.clone(), &env, IndexBuildMode::Async).await; + let initial = FileMeta { + region_id, + file_id: info.file_id, + file_size: info.file_size, + max_row_group_uncompressed_size: info.max_row_group_uncompressed_size, + num_rows: info.num_rows as u64, + num_row_groups: info.num_row_groups, + ..Default::default() + }; + seed_manifest_file(&manifest_ctx, &initial).await; + let version = mock_version_control( + metadata.clone(), + file_purger.clone(), + HashMap::from([(initial.file_id, initial.clone())]), + ) + .await; + let mut builder = mock_indexer_builder(metadata.clone(), &env).await; + Arc::make_mut(&mut builder).build_type = IndexBuildType::Manual; + if old_hll { + Arc::make_mut(&mut Arc::make_mut(&mut builder).registry) + .register(Arc::new(FailingProvider( + if matches!( + failure, + Some(ProviderFailure::PartialFinish | ProviderFailure::None) + ) { + ProviderFailure::CompletePair + } else { + ProviderFailure::None + }, + ))) + .unwrap(); + } + let (tx, _rx) = mpsc::channel(4); + let mut task = IndexBuildTask { + region_id, + file: FileHandle::new(initial.clone(), file_purger.clone()), + target_region_metadata: metadata.clone(), + source: IndexBuildSource::new(initial.clone(), metadata.schema_version), + reason: IndexBuildType::Manual, + access_layer: env.access_layer.clone(), + listener: WorkerListener::default(), + manifest_ctx: manifest_ctx.clone(), + write_cache: None, + cache_manager: None, + file_purger: file_purger.clone(), + indexer_builder: builder, + request_sender: tx, + index_options: IndexOptions::default(), + result_senders: Vec::new(), + }; + // First publish real built-in blobs and the test provider's HLL blob. + task.index_build(version).await.unwrap(); + let mut old = manifest_ctx.manifest().await.files[&initial.file_id].clone(); + if legacy { + let mut legacy_meta = old.as_ref().clone(); + for column in &mut legacy_meta.indexes { + column + .created_indexes + .retain(|kind| *kind != IndexType::HllIndex); + } + old = legacy_meta.into(); + seed_manifest_file(&manifest_ctx, old.as_ref()).await; + } + assert_eq!( + old.available_indexes.contains(&IndexType::HllIndex), + old_hll + ); + assert_eq!( + old.available_indexes + .iter() + .any(|kind| *kind != IndexType::HllIndex), + with_builtin, + ); + let old_path = location::index_file_path( + env.access_layer.table_dir(), + RegionIndexId::new(RegionFileId::new(region_id, old.file_id), old.index_version), + env.access_layer.path_type(), + ); + let old_bytes = env + .access_layer + .object_store() + .read(&old_path) + .await + .unwrap() + .to_bytes(); + assert!(!old_bytes.is_empty()); + let version = mock_version_control( + metadata.clone(), + file_purger.clone(), + HashMap::from([(old.file_id, old.as_ref().clone())]), + ) + .await; + let mut builder = mock_indexer_builder(metadata.clone(), &env).await; + Arc::make_mut(&mut builder).build_type = IndexBuildType::Manual; + if let Some(failure) = failure { + Arc::make_mut(&mut Arc::make_mut(&mut builder).registry) + .register(Arc::new(FailingProvider(failure))) + .unwrap(); + } + if failure.is_none() && !old_hll { + // Newly configured sketches must also fail before replacing the + // built-in indexes when their provider is unavailable. + Arc::make_mut(&mut task.target_region_metadata).column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + } + if failure == Some(ProviderFailure::None) { + // Keep only the first column configured; the second old HLL is obsolete. + Arc::make_mut(&mut task.target_region_metadata).column_metadatas[0] + .column_schema + .mut_metadata() + .insert( + datatypes::schema::HLL_KEY.into(), + r#"{"precision":10}"#.into(), + ); + Arc::make_mut(&mut builder).metadata = task.target_region_metadata.clone(); + } + task.file = FileHandle::new(old.clone(), file_purger); + task.source = IndexBuildSource::new(old.as_ref().clone(), metadata.schema_version); + task.indexer_builder = builder; + let result = task.index_build(version).await; + let published = manifest_ctx.manifest().await.files[&old.file_id].clone(); + if !with_builtin { + // No creator means no publication; the requested plan is not a removal plan. + assert!(matches!( + result.unwrap(), + IndexBuildReport::Done(IndexBuildOutcome::Aborted(_)) + )); + assert_eq!(old, published); + } else if failure == Some(ProviderFailure::None) { + result.unwrap(); + assert_eq!(published.index_version, old.index_version + 1); + let hll_columns = published + .indexes + .iter() + .filter(|column| column.created_indexes.contains(&IndexType::HllIndex)) + .map(|column| column.column_id) + .collect::>(); + assert_eq!(hll_columns, vec![0]); + } else if (old_hll && failure.is_some()) || (failure.is_none() && !old_hll) { + assert!(result.is_err(), "maintenance must fail for {failure:?}"); + assert_eq!( + old, published, + "failed {failure:?} must not change manifest" + ); + } else { + result.unwrap(); + assert_eq!(published.index_version, old.index_version + 1); + assert!(!published.available_indexes.contains(&IndexType::HllIndex)); + assert!( + published + .indexes + .iter() + .all(|column| !column.created_indexes.contains(&IndexType::HllIndex)) + ); + let expected: Vec<_> = old + .available_indexes + .iter() + .copied() + .filter(|kind| *kind != IndexType::HllIndex) + .collect(); + assert_eq!(published.available_indexes.as_slice(), expected.as_slice()); + assert!(published.index_file_size > 0); + } + // NoopFilePurger keeps the historical generation available for byte comparison. + assert_eq!( + old_bytes, + env.access_layer + .object_store() + .read(&old_path) + .await + .unwrap() + .to_bytes() + ); + } + } + #[tokio::test] async fn test_index_build_task_no_index() { + for remove_sketches in [false, true] { + check_empty_index_build(remove_sketches, true).await; + if remove_sketches { + check_empty_index_build(true, false).await; + } + } + } + + #[derive(Debug)] + struct RemoveTestIndex; + + impl handler::IndexKindHandler for RemoveTestIndex { + fn validate( + &self, + _context: &handler::PlanContext<'_>, + ) -> std::result::Result<(), common_error::ext::BoxedError> { + Ok(()) + } + + fn kinds(&self) -> &[kind::IndexKind] { + &[kind::IndexKind::Extension(kind::ExtensionIndexKind::Hll)] + } + + fn plan( + &self, + context: &handler::PlanContext<'_>, + ) -> std::result::Result< + Option>, + common_error::ext::BoxedError, + > { + Ok(context + .existing_indexes + .iter() + .any(|column| { + column + .created_indexes + .contains(&crate::sst::file::IndexType::HllIndex) + }) + .then(|| Arc::new(Self) as Arc)) + } + } + + #[async_trait::async_trait] + impl handler::PlannedIndex for RemoveTestIndex { + fn kinds(&self) -> &[kind::IndexKind] { + handler::IndexKindHandler::kinds(self) + } + + fn entries(&self) -> &[handler::PlanEntry] { + &[] + } + + async fn build( + &self, + _: &handler::BuildContext<'_>, + ) -> std::result::Result< + Option>, + common_error::ext::BoxedError, + > { + Ok(None) + } + + fn is_current(&self, file: &FileMeta, _: &handler::PlanContext<'_>) -> bool { + !file.indexes.iter().any(|column| { + column + .created_indexes + .contains(&crate::sst::file::IndexType::HllIndex) + }) + } + } + + async fn check_empty_index_build(remove_sketches: bool, registered: bool) { let env = SchedulerEnv::new().await; let mut scheduler = env.mock_index_build_scheduler(4); let mut metadata = sst_region_metadata(); @@ -3358,7 +4132,7 @@ mod tests { let manifest_ctx = env.mock_manifest_context(metadata.clone()).await; let file_purger = Arc::new(NoopFilePurger {}); let sst_info = mock_sst_file(metadata.clone(), &env, IndexBuildMode::Async).await; - let file_meta = FileMeta { + let mut file_meta = FileMeta { region_id, file_id: sst_info.file_id, file_size: sst_info.file_size, @@ -3368,13 +4142,47 @@ mod tests { num_row_groups: sst_info.num_row_groups, ..Default::default() }; + if remove_sketches { + file_meta + .indexes + .push(crate::sst::file::ColumnIndexMetadata { + column_id: metadata.column_metadatas[0].column_id, + created_indexes: smallvec::smallvec![crate::sst::file::IndexType::HllIndex], + fulltext_bloom_version: 0, + }); + file_meta.index_file_size = 123; + } seed_manifest_file(&manifest_ctx, &file_meta).await; let files = HashMap::from([(file_meta.file_id, file_meta.clone())]); let version_control = mock_version_control(metadata.clone(), file_purger.clone(), files).await; - let indexer_builder = mock_indexer_builder(metadata.clone(), &env).await; - + let mut indexer_builder = mock_indexer_builder(metadata.clone(), &env).await; let file = FileHandle::new(file_meta.clone(), file_purger.clone()); + if remove_sketches { + // A registered owner can remove its index after options are disabled. + let builder = Arc::make_mut(&mut indexer_builder); + if registered { + Arc::make_mut(&mut builder.registry) + .register(Arc::new(RemoveTestIndex)) + .unwrap(); + } + let plan = builder.registry.plan(&handler::PlanContext { + metadata: &metadata, + index_options: &builder.index_options, + build_type: IndexBuildType::SchemaChange, + existing_indexes: &file_meta.indexes, + }); + assert_eq!(!plan.is_empty(), registered); + assert!(plan.entries().next().is_none()); + builder.index_build_inputs = Some(build::IndexBuildInputs { + output_rows: None, + plan: Arc::new(plan), + inputs: vec![file.clone()].into(), + input_puffin_manager: builder.puffin_manager.clone(), + can_merge_inputs: false, + memory_limit: None, + }); + } // Create mock task. let (tx, mut rx) = mpsc::channel(4); @@ -3387,10 +4195,14 @@ mod tests { file_meta.clone(), version_control.current().version.metadata.schema_version, ), - reason: IndexBuildType::Flush, + reason: if registered { + IndexBuildType::SchemaChange + } else { + IndexBuildType::Manual + }, access_layer: env.access_layer.clone(), listener: WorkerListener::default(), - manifest_ctx, + manifest_ctx: manifest_ctx.clone(), write_cache: None, cache_manager: None, file_purger, @@ -3400,7 +4212,74 @@ mod tests { result_senders: vec![result_tx], }; - scheduler.schedule_build(&version_control, task).await; + if remove_sketches { + assert!(task.can_remove_all_indexes()); + for column_level in [false, true] { + let mut mixed = task.clone(); + if column_level { + mixed.source.file_meta.indexes[0] + .created_indexes + .push(IndexType::InvertedIndex); + } else { + mixed + .source + .file_meta + .available_indexes + .push(IndexType::InvertedIndex); + } + assert!(mixed.can_remove_all_indexes()); + } + } + + scheduler + .schedule_build(&version_control, task.clone()) + .await; + + if remove_sketches { + let finished = recv_index_build_finished(&mut rx).await; + assert!(finished.file_meta.indexes.is_empty()); + assert!(finished.file_meta.available_indexes.is_empty()); + assert_eq!(finished.file_meta.index_file_size, 0); + assert_eq!(finished.file_meta.index_version, 1); + assert!( + manifest_ctx.manifest().await.files[&file_meta.file_id] + .indexes + .is_empty() + ); + // Reconstruct the file from persisted metadata as reopening does, then enable + // an ordinary index. Its new Puffin must not reuse the historical version 0. + let persisted = serde_json::to_vec(&finished.file_meta).unwrap(); + let reopened: FileMeta = serde_json::from_slice(&persisted).unwrap(); + assert!(!reopened.exists_index()); + let target_metadata = Arc::new(sst_region_metadata()); + let reopened_version = mock_version_control( + target_metadata.clone(), + task.file_purger.clone(), + HashMap::from([(reopened.file_id, reopened.clone())]), + ) + .await; + let (tx, mut rx) = mpsc::channel(4); + let mut rebuild = task; + rebuild.file = FileHandle::new(reopened.clone(), rebuild.file_purger.clone()); + rebuild.source.file_meta = reopened; + rebuild.target_region_metadata = target_metadata.clone(); + rebuild.indexer_builder = mock_indexer_builder(target_metadata, &env).await; + rebuild.request_sender = tx; + let mut reopened_scheduler = env.mock_index_build_scheduler(4); + reopened_scheduler + .schedule_build(&reopened_version, rebuild) + .await; + let rebuilt = recv_index_build_finished(&mut rx).await; + assert!(rebuilt.file_meta.exists_index()); + assert_eq!(rebuilt.file_meta.index_version, 2); + let path = location::index_file_path( + env.access_layer.table_dir(), + RegionIndexId::new(RegionFileId::new(region_id, file_meta.file_id), 2), + env.access_layer.path_type(), + ); + assert!(env.access_layer.object_store().exists(&path).await.unwrap()); + return; + } // No index needs to be built, so the SST is not read. assert!(matches!( diff --git a/src/mito2/src/sst/index/build.rs b/src/mito2/src/sst/index/build.rs index 863aa84b7a4..180b1e26272 100644 --- a/src/mito2/src/sst/index/build.rs +++ b/src/mito2/src/sst/index/build.rs @@ -21,13 +21,13 @@ use crate::sst::index::puffin_manager::SstPuffinManager; use crate::sst::index::registry::IndexBuildPlan; /// An immutable plan with operation-scoped input files and output constraints. -/// Currently compaction and explicit rebuild supply these inputs; flush does not -/// inject extension handlers and builds only the existing built-in indexes. +/// Compaction and SST index rebuilds supply these inputs, including asynchronous +/// post-flush indexing. Synchronous memtable flushes have no persisted input files. #[derive(Clone)] pub struct IndexBuildInputs { /// Typed plans shared by output files of this operation. pub plan: Arc, - /// Pinned physical input files; flushes have no inputs. + /// Pinned physical input files. Post-flush indexing reads the newly written SST. pub inputs: Arc<[FileHandle]>, /// Reads persisted inputs even when output uses a local write cache. /// Input availability is determined by `inputs.is_empty()`, not this manager. diff --git a/src/mito2/src/sst/index/handler.rs b/src/mito2/src/sst/index/handler.rs index dba75c143ce..de9fba783a9 100644 --- a/src/mito2/src/sst/index/handler.rs +++ b/src/mito2/src/sst/index/handler.rs @@ -28,7 +28,7 @@ use store_api::storage::ColumnId; use crate::read::Batch; use crate::region::options::IndexOptions; -use crate::sst::file::{ColumnIndexMetadata, FileHandle, RegionIndexId}; +use crate::sst::file::{ColumnIndexMetadata, FileHandle, FileMeta, RegionIndexId}; use crate::sst::index::intermediate::IntermediateManager; use crate::sst::index::kind::IndexKind; use crate::sst::index::puffin_manager::{SstPuffinManager, SstPuffinWriter}; @@ -104,6 +104,11 @@ pub trait IndexKindHandler: Send + Sync + Debug { /// entirely built-in or entirely extension kinds. fn kinds(&self) -> &[IndexKind]; + /// Validates options at region creation independently of whether this lifecycle + /// event builds indexes. Implementations must not skip validation based on + /// `context.build_type`. + fn validate(&self, context: &PlanContext<'_>) -> Result<(), BoxedError>; + /// Prepares typed parameters. `None` means no work; an empty entry list may /// still request removal of obsolete descriptors during maintenance. fn plan(&self, context: &PlanContext<'_>) -> Result>, BoxedError>; @@ -124,6 +129,10 @@ pub trait PlannedIndex: Send + Sync + Debug { context: &BuildContext<'_>, ) -> Result>, BoxedError>; + /// Whether persisted entries satisfy this plan. + /// The context uses the same file's indexes as the planning input. + fn is_current(&self, file: &FileMeta, context: &PlanContext<'_>) -> bool; + /// Whether building this plan can reuse persisted input summaries. /// Row-only plans do not require checking input object existence. fn needs_input_reuse(&self) -> bool { diff --git a/src/mito2/src/sst/index/handler/builtin.rs b/src/mito2/src/sst/index/handler/builtin.rs index c814cb4b43b..9c561a5dcb1 100644 --- a/src/mito2/src/sst/index/handler/builtin.rs +++ b/src/mito2/src/sst/index/handler/builtin.rs @@ -24,6 +24,7 @@ use common_error::ext::BoxedError; use index::target::IndexTarget; use crate::config::{BloomFilterConfig, FulltextIndexConfig, InvertedIndexConfig}; +use crate::sst::file::FileMeta; use crate::sst::index::handler::builtin::bloom::BloomPlan; use crate::sst::index::handler::builtin::fulltext::FulltextPlan; use crate::sst::index::handler::builtin::inverted::InvertedPlan; @@ -47,6 +48,12 @@ impl IndexKindHandler for BuiltinIndexHandler { } } + fn validate(&self, _context: &PlanContext<'_>) -> Result<(), BoxedError> { + // Built-in options retain their existing schema/build-time validation. + // Planning here would make region creation reject inputs it previously accepted. + Ok(()) + } + fn plan(&self, context: &PlanContext<'_>) -> Result>, BoxedError> { match self { Self::Inverted(config) => InvertedPlan::plan(config, context) @@ -83,6 +90,11 @@ fn normalize_entries(entries: &mut Vec) { }); } +fn builtin_is_current(file: &FileMeta, context: &PlanContext<'_>) -> bool { + // Preserve the established schema/version policy, including its JSON rebuild scope. + file.is_index_consistent_with_region(&context.metadata.column_metadatas, context.index_options) +} + #[cfg(test)] mod tests { use super::*; @@ -90,6 +102,45 @@ mod tests { use crate::sst::index::IndexBuildType; use crate::test_util::sst_util; + #[test] + fn test_builtin_validation_preserves_build_time_option_errors() { + for (handler, key) in [ + ( + BuiltinIndexHandler::Fulltext(FulltextIndexConfig::default()), + datatypes::schema::FULLTEXT_KEY, + ), + ( + BuiltinIndexHandler::Bloom(BloomFilterConfig::default()), + datatypes::schema::SKIPPING_INDEX_KEY, + ), + ] { + let mut metadata = sst_util::sst_region_metadata(); + let column = metadata + .column_metadatas + .iter_mut() + .find(|column| column.semantic_type == api::v1::SemanticType::Field) + .unwrap(); + column.column_schema = + column + .column_schema + .clone() + .with_metadata(std::collections::HashMap::from([( + key.to_string(), + "invalid json".to_string(), + )])); + let metadata = Arc::new(metadata); + let options = IndexOptions::default(); + let context = PlanContext { + metadata: &metadata, + index_options: &options, + build_type: IndexBuildType::Flush, + existing_indexes: &[], + }; + assert!(handler.validate(&context).is_ok()); + assert!(handler.plan(&context).is_err()); + } + } + #[test] fn test_builtin_planning_reports_invalid_options() { for (handler, key) in [ diff --git a/src/mito2/src/sst/index/handler/builtin/bloom.rs b/src/mito2/src/sst/index/handler/builtin/bloom.rs index aff50955bb8..18f9867337b 100644 --- a/src/mito2/src/sst/index/handler/builtin/bloom.rs +++ b/src/mito2/src/sst/index/handler/builtin/bloom.rs @@ -24,8 +24,11 @@ use index::target::IndexTarget; use crate::config::BloomFilterConfig; use crate::read::Batch; +use crate::sst::file::FileMeta; use crate::sst::index::bloom_filter::creator::BloomFilterIndexer; -use crate::sst::index::handler::builtin::{column_target_id, normalize_entries, target_column_id}; +use crate::sst::index::handler::builtin::{ + builtin_is_current, column_target_id, normalize_entries, target_column_id, +}; use crate::sst::index::handler::{ BuildContext, IndexCreator, PlanContext, PlanEntry, PlannedIndex, }; @@ -105,6 +108,10 @@ impl PlannedIndex for BloomPlan { .map(|creator| creator.map(|c| Box::new(c) as Box)) .map_err(BoxedError::new) } + + fn is_current(&self, file: &FileMeta, context: &PlanContext<'_>) -> bool { + builtin_is_current(file, context) + } } #[async_trait] diff --git a/src/mito2/src/sst/index/handler/builtin/fulltext.rs b/src/mito2/src/sst/index/handler/builtin/fulltext.rs index 4a287ef4017..5b82a73e662 100644 --- a/src/mito2/src/sst/index/handler/builtin/fulltext.rs +++ b/src/mito2/src/sst/index/handler/builtin/fulltext.rs @@ -24,9 +24,12 @@ use index::target::IndexTarget; use crate::config::FulltextIndexConfig; use crate::read::Batch; +use crate::sst::file::FileMeta; use crate::sst::index::fulltext_index; use crate::sst::index::fulltext_index::creator::{FulltextIndexer, FulltextTarget}; -use crate::sst::index::handler::builtin::{column_target_id, normalize_entries, target_column_id}; +use crate::sst::index::handler::builtin::{ + builtin_is_current, column_target_id, normalize_entries, target_column_id, +}; use crate::sst::index::handler::{ BuildContext, IndexCreator, PlanContext, PlanEntry, PlannedIndex, }; @@ -118,6 +121,10 @@ impl PlannedIndex for FulltextPlan { .map(|creator| creator.map(|c| Box::new(c) as Box)) .map_err(BoxedError::new) } + + fn is_current(&self, file: &FileMeta, context: &PlanContext<'_>) -> bool { + builtin_is_current(file, context) + } } #[async_trait] diff --git a/src/mito2/src/sst/index/handler/builtin/inverted.rs b/src/mito2/src/sst/index/handler/builtin/inverted.rs index 353afd5cff9..44010b629ba 100644 --- a/src/mito2/src/sst/index/handler/builtin/inverted.rs +++ b/src/mito2/src/sst/index/handler/builtin/inverted.rs @@ -27,7 +27,10 @@ use index::target::IndexTarget; use crate::config::InvertedIndexConfig; use crate::read::Batch; -use crate::sst::index::handler::builtin::{column_target_id, normalize_entries, target_column_id}; +use crate::sst::file::FileMeta; +use crate::sst::index::handler::builtin::{ + builtin_is_current, column_target_id, normalize_entries, target_column_id, +}; use crate::sst::index::handler::{ BuildContext, IndexCreator, PlanContext, PlanEntry, PlannedIndex, }; @@ -131,6 +134,10 @@ impl PlannedIndex for InvertedPlan { self.targets.clone(), )))) } + + fn is_current(&self, file: &FileMeta, context: &PlanContext<'_>) -> bool { + builtin_is_current(file, context) + } } fn json_inverted_targets(context: &PlanContext<'_>) -> Vec { diff --git a/src/mito2/src/sst/index/registry.rs b/src/mito2/src/sst/index/registry.rs index 7b9da2e51dd..d4f53f30632 100644 --- a/src/mito2/src/sst/index/registry.rs +++ b/src/mito2/src/sst/index/registry.rs @@ -72,6 +72,16 @@ impl IndexRegistry { Ok(self) } + /// Validates required providers and their options before creating a new region. + /// Built-in handlers preserve their existing schema/build-time validation. + pub fn validate(&self, context: &PlanContext<'_>) -> Result<(), BoxedError> { + self.validate_schema_kinds(context.metadata)?; + for handler in &self.handlers { + handler.validate(context)?; + } + Ok(()) + } + /// Installs a handler, rejecting empty or overlapping kind ownership. pub fn register(&mut self, handler: Arc) -> Result<(), BoxedError> { self.validate_registration(handler.kinds())?; @@ -89,6 +99,14 @@ impl IndexRegistry { files: impl IntoIterator, ) -> Result<(), BoxedError> { self.validate_schema_kinds(metadata)?; + self.validate_file_kinds(files) + } + + /// Rejects unavailable implementations for indexes being replaced. + fn validate_file_kinds<'a>( + &self, + files: impl IntoIterator, + ) -> Result<(), BoxedError> { for file in files { for kind in file.available_indexes.iter().chain( file.indexes @@ -223,7 +241,10 @@ impl IndexRegistry { }); } - fn validate_schema_kinds(&self, metadata: &RegionMetadataRef) -> Result<(), BoxedError> { + pub(crate) fn validate_schema_kinds( + &self, + metadata: &RegionMetadataRef, + ) -> Result<(), BoxedError> { for column in &metadata.column_metadatas { let schema = &column.column_schema; for kind in ExtensionIndexKind::ALL { @@ -270,6 +291,11 @@ impl IndexBuildPlan { &self.planned } + /// Whether no handler requested work, including metadata-only maintenance. + pub fn is_empty(&self) -> bool { + self.planned.is_empty() + } + /// All requested column formats. pub fn entries(&self) -> impl Iterator { self.planned.iter().flat_map(|plan| plan.entries()) @@ -301,6 +327,51 @@ impl IndexBuildPlan { self.planned .retain(|plan| plan.kinds().iter().all(|kind| kind.is_builtin())); } + + /// Whether every handler considers the published file current. + pub fn is_current(&self, file: &FileMeta, context: &PlanContext<'_>) -> bool { + let has_removed_extensions = matches!( + context.build_type, + IndexBuildType::Manual | IndexBuildType::SchemaChange + ) && ExtensionIndexKind::ALL.iter().any(|kind| { + let index_type = IndexKind::Extension(*kind).index_type(); + let configured = |column_id| { + context + .metadata + .column_by_id(column_id) + .is_some_and(|column| { + column + .column_schema + .metadata() + .contains_key(kind.required_metadata_key()) + }) + }; + let mut has_column_descriptor = false; + for column in &file.indexes { + if column.created_indexes.contains(&index_type) { + has_column_descriptor = true; + if !configured(column.column_id) { + return true; + } + } + } + // Legacy file-level descriptors cannot identify individual columns. + !has_column_descriptor + && file.available_indexes.contains(&index_type) + && !context.metadata.column_metadatas.iter().any(|column| { + column + .column_schema + .metadata() + .contains_key(kind.required_metadata_key()) + }) + }); + self.failure.is_none() + && !has_removed_extensions + && self + .planned + .iter() + .all(|plan| plan.is_current(file, context)) + } } #[cfg(test)] @@ -315,6 +386,11 @@ mod tests { fn kinds(&self) -> &[IndexKind] { &self.0 } + + fn validate(&self, _: &PlanContext<'_>) -> Result<(), BoxedError> { + Ok(()) + } + fn plan(&self, _: &PlanContext<'_>) -> Result>, BoxedError> { Ok(None) } @@ -340,6 +416,10 @@ mod tests { ) -> Result>, BoxedError> { unreachable!("admission must not construct creators") } + + fn is_current(&self, _: &FileMeta, _: &PlanContext<'_>) -> bool { + false + } } #[test] @@ -427,6 +507,10 @@ mod tests { &self.owned } + fn validate(&self, _: &PlanContext<'_>) -> Result<(), BoxedError> { + Ok(()) + } + fn plan(&self, _: &PlanContext<'_>) -> Result>, BoxedError> { Ok(Some(self.plan.clone())) } diff --git a/src/mito2/src/sst/parquet.rs b/src/mito2/src/sst/parquet.rs index 911e02cbf76..63bf566b2b8 100644 --- a/src/mito2/src/sst/parquet.rs +++ b/src/mito2/src/sst/parquet.rs @@ -1632,7 +1632,8 @@ mod tests { sequence: None, partition_expr: match &metadata.partition_expr { Some(json_str) => partition::expr::PartitionExpr::from_json_str(json_str) - .expect("partition expression should be valid JSON"), + .expect("partition expression should be valid JSON") + .map(Into::into), None => None, }, num_series: 0, @@ -2176,7 +2177,8 @@ mod tests { sequence: None, partition_expr: match &metadata.partition_expr { Some(json_str) => partition::expr::PartitionExpr::from_json_str(json_str) - .expect("partition expression should be valid JSON"), + .expect("partition expression should be valid JSON") + .map(Into::into), None => None, }, num_series: 0, diff --git a/src/mito2/src/sst/parquet/file_range.rs b/src/mito2/src/sst/parquet/file_range.rs index 37bae5f1d34..cd529cc6c7d 100644 --- a/src/mito2/src/sst/parquet/file_range.rs +++ b/src/mito2/src/sst/parquet/file_range.rs @@ -46,7 +46,6 @@ use crate::error::{ NewRecordBatchSnafu, RecordBatchSnafu, Result, StatsNotPresentSnafu, UnexpectedSnafu, }; use crate::read::compat::FlatCompatBatch; -use crate::read::flat_projection::CompactionProjectionMapper; use crate::read::last_row::FlatRowGroupLastRowCachedReader; use crate::read::prune::FlatPruneReader; use crate::sst::file::FileHandle; @@ -388,11 +387,6 @@ impl FileRange { self.context.compat_batch() } - /// Returns the helper to project batches. - pub(crate) fn compaction_projection_mapper(&self) -> Option<&CompactionProjectionMapper> { - self.context.compaction_projection_mapper() - } - /// Filters a full-projection batch using this range's precise filters. pub(crate) fn precise_filter_flat( &self, @@ -535,19 +529,12 @@ impl FileRangeContext { self.base.compat_batch.as_ref() } - /// Returns the helper to project batches. - pub(crate) fn compaction_projection_mapper(&self) -> Option<&CompactionProjectionMapper> { - self.base.compaction_projection_mapper.as_ref() - } - /// Sets the compat helper to the context. pub(crate) fn set_compat_batch(&mut self, compat: Option) { self.base.compat_batch = compat; } - /// Filters the input RecordBatch by the pushed down predicate and returns RecordBatch. - /// If a partition expr filter is configured, it is also applied. - /// Physical filter exprs are not evaluated here; they are only applied during prefiltering. + /// Applies source-batch filtering; see [RangeBase::precise_filter_flat] for layout invariants. pub(crate) fn precise_filter_flat( &self, input: RecordBatch, @@ -626,8 +613,6 @@ pub(crate) struct RangeBase { pub(crate) codec: Arc, /// Optional helper to compat batches. pub(crate) compat_batch: Option, - /// Optional helper to project batches. - pub(crate) compaction_projection_mapper: Option, /// Mode to pre-filter columns. pub(crate) pre_filter_mode: PreFilterMode, /// Partition filter. @@ -649,9 +634,11 @@ impl TagDecodeState { } impl RangeBase { - /// Filters the input RecordBatch by the pushed down predicate and returns RecordBatch. + /// Filters converted source batches before compatibility and merge. + /// Predicate projections are temporary: only their row mask is applied to the + /// original batch, preserving its schema and complete encoded primary keys. /// - /// It assumes all necessary tags are already decoded from the primary key. + /// Missing predicate tags are decoded from the primary key on demand. /// /// # Arguments /// * `input` - The RecordBatch to filter @@ -1041,7 +1028,6 @@ mod tests { prune_schema: metadata.schema.clone(), codec: mito_codec::row_converter::build_primary_key_codec(metadata.as_ref()), compat_batch: None, - compaction_projection_mapper: None, pre_filter_mode: PreFilterMode::All, partition_filter: None, } @@ -1087,11 +1073,28 @@ mod tests { assert_eq!(batch, filtered); } - #[test] - fn test_sparse_filter_and_partition_tags_preserve_skip_modes() { - let metadata = Arc::new(sst_region_metadata_with_encoding( - PrimaryKeyEncoding::Sparse, - )); + #[rstest::rstest] + #[case(2)] + #[case(40)] + #[case(3600)] + fn test_sparse_filter_and_partition_tags_preserve_skip_modes(#[case] tag_count: u32) { + let source = sst_region_metadata_with_encoding(PrimaryKeyEncoding::Sparse); + let mut keys = source.primary_key.clone(); + let mut builder = RegionMetadataBuilder::from_existing(source); + for id in 4..tag_count + 2 { + builder.push_column_metadata(store_api::metadata::ColumnMetadata { + column_schema: ColumnSchema::new( + format!("tag_{id}"), + ConcreteDataType::string_datatype(), + true, + ), + semantic_type: SemanticType::Tag, + column_id: id, + }); + keys.push(id); + } + builder.primary_key(keys); + let metadata = Arc::new(builder.build().unwrap()); let codec = SparsePrimaryKeyCodec::schemaless(); let long_tag = "x".repeat(80); let mut pk_builder = BinaryDictionaryBuilder::::new(); @@ -1159,7 +1162,10 @@ mod tests { ) .unwrap(); let batch = read_format.convert_batch(raw.clone(), None).unwrap(); - let base = RangeBase { + if !materialized { + assert_eq!(batch, raw); + } + let mut base = RangeBase { filters: filters.clone(), dyn_filters: vec![], read_format, @@ -1167,7 +1173,6 @@ mod tests { prune_schema: metadata.schema.clone(), codec: Arc::new(codec.clone()), compat_batch: None, - compaction_projection_mapper: None, pre_filter_mode: PreFilterMode::All, partition_filter: Some(PartitionFilterContext { partition_schema: partition_schema.clone(), @@ -1180,6 +1185,19 @@ mod tests { (false, true, vec![3, 4, 5]), (true, true, vec![0, 1, 3, 4, 5]), ] { + let mut state = TagDecodeState::new(); + base.compute_filter_mask_flat(&batch, skip_fields, skip_tags, &mut state) + .unwrap(); + let mut decoded = state.decoded_tag_cache.keys().copied().collect::>(); + decoded.sort_unstable(); + assert_eq!( + decoded, + if !materialized && !skip_tags { + vec![0, 1] + } else { + vec![1] + } + ); let output = base .precise_filter_flat(batch.clone(), skip_fields, skip_tags) .unwrap() @@ -1194,7 +1212,76 @@ mod tests { expected.as_slice(), "materialized={materialized}, skip_fields={skip_fields}, skip_tags={skip_tags}" ); + let mask = BooleanArray::from( + (0..6) + .map(|row| expected.contains(&row)) + .collect::>(), + ); + assert_eq!( + output, + datatypes::arrow::compute::filter_record_batch(&batch, &mask).unwrap() + ); + // Query materialization stays separate from compaction. Both + // compaction consumers pass encoded batches directly to compat. + if materialized { + continue; + } + let compacted = output; + assert_eq!( + compacted, + datatypes::arrow::compute::filter_record_batch(&raw, &mask).unwrap() + ); + // Exercise that boundary with an added nullable field. + let mut target = RegionMetadataBuilder::from_existing(metadata.as_ref().clone()); + target.push_column_metadata(store_api::metadata::ColumnMetadata { + column_schema: ColumnSchema::new( + "added", + ConcreteDataType::uint64_datatype(), + true, + ), + semantic_type: SemanticType::Field, + column_id: tag_count + 2, + }); + let target = Arc::new(target.build().unwrap()); + let projection = + crate::read::flat_projection::FlatProjectionMapper::all(&target).unwrap(); + let compat = FlatCompatBatch::try_new(&projection, &base.read_format, true) + .unwrap() + .unwrap(); + let compatible = compat.compat(compacted.clone()).unwrap(); + assert_eq!(compatible.schema(), projection.input_arrow_schema(true)); + assert_eq!(compatible.column(1).null_count(), compatible.num_rows()); + assert_eq!(compatible.column(0), compacted.column(0)); + assert_eq!(&compatible.columns()[2..], &compacted.columns()[1..]); } + // Missing partition tags produce NULL, which must reject the row + // even when ordinary tag and field predicates are skipped. + base.filters.clear(); + let schema = Arc::new(Schema::new(vec![ColumnSchema::new( + "tag_0", + ConcreteDataType::dictionary_datatype( + ConcreteDataType::uint32_datatype(), + ConcreteDataType::string_datatype(), + ), + true, + )])); + base.partition_filter = Some(PartitionFilterContext { + region_partition_physical_expr: partition_col("tag_0") + .eq(Value::String("".into())) + .try_as_physical_expr(schema.arrow_schema()) + .unwrap(), + partition_schema: schema.clone(), + }); + let temporary = base + .project_record_batch_for_pruning_flat(&batch, &schema, &mut TagDecodeState::new()) + .unwrap(); + // Dictionary keys can be valid while the referenced value is NULL. + assert_eq!(temporary.column(0).logical_null_count(), 1); + let filtered = base + .precise_filter_flat(batch.clone(), true, true) + .unwrap() + .unwrap(); + assert_eq!(filtered, batch.slice(1, 1)); } } diff --git a/src/mito2/src/sst/parquet/flat_format.rs b/src/mito2/src/sst/parquet/flat_format.rs index 3f40f7ab3a1..1734d7e1926 100644 --- a/src/mito2/src/sst/parquet/flat_format.rs +++ b/src/mito2/src/sst/parquet/flat_format.rs @@ -183,7 +183,11 @@ pub(crate) fn field_column_start(metadata: &RegionMetadata, num_columns: usize) // sparse encoding). /// Helper for reading the flat SST format with projection. /// -/// It only supports flat format that stores primary keys additionally. +/// Physical Parquet projections describe stored columns and drive pruning and +/// prefiltering. Converted projections describe batches returned by `convert_batch`. +/// Sparse compaction can retain encoded-only tags; query reads may expand them. +/// Neither layout is the final user projection. Compatibility aligns converted +/// batches before merge; temporary predicate columns never change their layout. pub struct FlatReadFormat { /// Sequence number to override the sequence read from the SST. override_sequence: Option, @@ -252,7 +256,7 @@ impl FlatReadFormat { self.pk_dict_wrap_schema = Some(output_schema); } - /// Index of a column in the projected batch by its column id. + /// Index in the converted batch, never in a separately projected prefilter batch. pub fn projected_index_by_id(&self, column_id: ColumnId) -> Option { self.format_projection() .column_id_to_projected_index diff --git a/src/mito2/src/sst/parquet/format.rs b/src/mito2/src/sst/parquet/format.rs index 0f1f0ab38e2..8c0332fea1b 100644 --- a/src/mito2/src/sst/parquet/format.rs +++ b/src/mito2/src/sst/parquet/format.rs @@ -1651,8 +1651,10 @@ mod tests { tag1: String, } - #[test] - fn test_flat_read_format_convert_format_with_dense_encoding() { + #[rstest::rstest] + #[case(false)] + #[case(true)] + fn test_flat_read_format_convert_format_with_dense_encoding(#[case] skip_auto_convert: bool) { let metadata = build_test_region_metadata(); let column_ids: Vec<_> = metadata @@ -1665,7 +1667,7 @@ mod tests { ReadColumns::new(column_ids), Some(build_test_arrow_schema()), "test", - false, + skip_auto_convert, ) .unwrap(); diff --git a/src/mito2/src/sst/parquet/index_writer.rs b/src/mito2/src/sst/parquet/index_writer.rs index 6d6414b9cd2..24da3f1a9b4 100644 --- a/src/mito2/src/sst/parquet/index_writer.rs +++ b/src/mito2/src/sst/parquet/index_writer.rs @@ -12,15 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -use bytes::Bytes; use datatypes::arrow::datatypes::SchemaRef; use datatypes::arrow::record_batch::RecordBatch; -use futures::future::BoxFuture; -use object_store::{ObjectStore, Writer}; +use object_store::ObjectStore; use parquet::arrow::AsyncArrowWriter; -use parquet::arrow::async_writer::AsyncFileWriter; use parquet::basic::{Compression, Encoding, ZstdLevel}; -use parquet::errors::ParquetError; use parquet::file::metadata::KeyValue; use parquet::file::properties::WriterProperties; use snafu::{OptionExt, ResultExt}; @@ -29,55 +25,9 @@ use crate::access_layer::TempFileCleaner; use crate::error::{OpenDalSnafu, Result, UnexpectedSnafu, WriteParquetSnafu}; use crate::sst::{DEFAULT_WRITE_BUFFER_SIZE, DEFAULT_WRITE_CONCURRENCY}; -type ArrowWriter = AsyncArrowWriter; +use crate::sst::parquet::writer::ObjectStoreWriter; -/// Bridges an OpenDAL [`Writer`] with Parquet's [`AsyncFileWriter`] and tracks -/// the number of bytes successfully submitted to the object store. -struct AsyncWriter { - inner: Writer, - output_bytes: u64, -} - -impl AsyncWriter { - fn new(inner: Writer) -> Self { - Self { - inner, - output_bytes: 0, - } - } - - fn output_bytes(&self) -> u64 { - self.output_bytes - } - - fn into_inner(self) -> Writer { - self.inner - } -} - -impl AsyncFileWriter for AsyncWriter { - fn write(&mut self, bytes: Bytes) -> BoxFuture<'_, parquet::errors::Result<()>> { - Box::pin(async move { - let len = bytes.len() as u64; - self.inner - .write(bytes) - .await - .map_err(|error| ParquetError::External(Box::new(error)))?; - self.output_bytes += len; - Ok(()) - }) - } - - fn complete(&mut self) -> BoxFuture<'_, parquet::errors::Result<()>> { - Box::pin(async move { - self.inner - .close() - .await - .map(|_| ()) - .map_err(|error| ParquetError::External(Box::new(error))) - }) - } -} +type ArrowWriter = AsyncArrowWriter; /// Shared Parquet output and cleanup lifecycle for index writers. pub(crate) struct ParquetIndexWriter { @@ -112,9 +62,12 @@ impl ParquetIndexWriter { .set_statistics_truncate_length(None) .set_key_value_metadata(key_value_metadata) .build(); - let writer = - AsyncArrowWriter::try_new(AsyncWriter::new(output), schema.clone(), Some(properties)) - .context(WriteParquetSnafu)?; + let writer = AsyncArrowWriter::try_new( + ObjectStoreWriter::new(output, DEFAULT_WRITE_BUFFER_SIZE.as_bytes() as usize, None), + schema.clone(), + Some(properties), + ) + .context(WriteParquetSnafu)?; Ok(Self { name, diff --git a/src/mito2/src/sst/parquet/parallel_writer.rs b/src/mito2/src/sst/parquet/parallel_writer.rs index d257574cb79..d0caf95285c 100644 --- a/src/mito2/src/sst/parquet/parallel_writer.rs +++ b/src/mito2/src/sst/parquet/parallel_writer.rs @@ -19,6 +19,7 @@ use std::sync::Arc; use std::time::{Duration, Instant}; use bytes::Bytes; +use common_base::cancellation::Cancelled; use common_runtime::JoinHandle; use datatypes::arrow::datatypes::SchemaRef; use datatypes::arrow::record_batch::RecordBatch; @@ -31,6 +32,7 @@ use parquet::errors::{ParquetError, Result}; use parquet::file::metadata::ParquetMetaData; use parquet::file::properties::WriterProperties; use parquet::file::writer::SerializedFileWriter; +use tokio_util::sync::CancellationToken; /// Runtime whose blocking pool encodes the columns of a writer. #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] @@ -90,6 +92,7 @@ pub struct ParallelArrowWriter { /// Tasks encoding the previous row group. encoding: Vec>>, num_row_groups: usize, + cancellation: Option, } impl ParallelArrowWriter { @@ -137,13 +140,20 @@ impl ParallelArrowWriter { buffered_rows: 0, encoding: Vec::new(), num_row_groups: 0, + cancellation: None, }) } + pub(crate) fn with_cancellation(mut self, cancellation: Option) -> Self { + self.cancellation = cancellation; + self + } + /// Buffers `batch`, starting to encode each row group once it is full. pub async fn write(&mut self, batch: &RecordBatch) -> Result<()> { let mut offset = 0; while offset < batch.num_rows() { + check_cancelled(self.cancellation.as_ref())?; let len = (self.max_row_group_rows - self.buffered_rows).min(batch.num_rows() - offset); self.buffered.push(batch.slice(offset, len)); self.buffered_rows += len; @@ -165,6 +175,11 @@ impl ParallelArrowWriter { /// Writes the remaining rows and the footer, then completes the async writer. pub async fn close(mut self) -> Result { + self.finish().await + } + + /// Finalizes the file while retaining the upload for abort if it fails. + pub(crate) async fn finish(&mut self) -> Result { self.flush().await?; let metadata = self.file_writer.finish()?; self.write_buffered_bytes().await?; @@ -173,9 +188,10 @@ impl ParallelArrowWriter { } /// Discards buffered input and waits for encoders without writing a failed SST. - pub(crate) async fn abort(mut self) { + pub(crate) async fn abort(mut self) -> W { self.buffered.clear(); abort_encoding_tasks(&mut self.encoding).await; + self.async_writer } /// Installs controlled tasks to test SST write-error cleanup independently of codec timing. @@ -191,6 +207,7 @@ impl ParallelArrowWriter { /// Starts encoding the buffered rows once the previous row group is written. async fn encode_buffered(&mut self) -> Result<()> { self.finish_encoding().await?; + check_cancelled(self.cancellation.as_ref())?; let batches: Arc<[RecordBatch]> = mem::take(&mut self.buffered).into(); self.buffered_rows = 0; @@ -209,8 +226,9 @@ impl ParallelArrowWriter { .map(|fields| { let batches = batches.clone(); let schema = self.arrow_schema.clone(); + let cancellation = self.cancellation.clone(); self.runtime - .spawn(move || encode_fields(&schema, &batches, fields)) + .spawn(move || encode_fields(&schema, &batches, fields, cancellation.as_ref())) }) .collect(); Ok(()) @@ -301,17 +319,26 @@ fn assign_fields( .collect() } +fn check_cancelled(cancellation: Option<&CancellationToken>) -> Result<()> { + if cancellation.is_some_and(CancellationToken::is_cancelled) { + return Err(ParquetError::External(Box::new(Cancelled))); + } + Ok(()) +} + /// Encodes `fields` of all `batches` and closes their column writers. fn encode_fields( schema: &SchemaRef, batches: &[RecordBatch], fields: Vec, + cancellation: Option<&CancellationToken>, ) -> Result { let mut chunks = Vec::new(); let mut costs = Vec::with_capacity(fields.len()); for (field, mut writers) in fields { let start = Instant::now(); for batch in batches { + check_cancelled(cancellation)?; let leaves = compute_leaves(schema.field(field), batch.column(field))?; if leaves.len() != writers.len() { return Err(ParquetError::General(format!( @@ -322,10 +349,12 @@ fn encode_fields( ))); } for (leaf, (_, writer)) in leaves.iter().zip(writers.iter_mut()) { + check_cancelled(cancellation)?; writer.write(leaf)?; } } for (leaf, writer) in writers { + check_cancelled(cancellation)?; chunks.push((leaf, writer.close()?)); } costs.push((field, start.elapsed())); @@ -534,8 +563,12 @@ mod tests { let running = runtime.spawn_blocking(move || { started_tx.send(()).unwrap(); release_rx.blocking_recv().unwrap(); - let _encoded = - encode_fields(&schema(), &running_batches, vec![(0, vec![(0, column)])])?; + let _encoded = encode_fields( + &schema(), + &running_batches, + vec![(0, vec![(0, column)])], + None, + )?; Err(ParquetError::General("later encoding failure".to_string())) }); // With one blocking thread, this also proves the first task has ended. diff --git a/src/mito2/src/sst/parquet/prefilter.rs b/src/mito2/src/sst/parquet/prefilter.rs index d5401f4a6de..fbb1e0b5101 100644 --- a/src/mito2/src/sst/parquet/prefilter.rs +++ b/src/mito2/src/sst/parquet/prefilter.rs @@ -24,6 +24,9 @@ use std::sync::Arc; use api::v1::SemanticType; use common_recordbatch::filter::SimpleFilterEvaluator; +use common_telemetry::debug; +use datafusion::physical_expr::utils::collect_columns; +use datafusion::physical_plan::PhysicalExpr; use datafusion_common::ScalarValue; use datafusion_expr::Expr; use datatypes::arrow::array::{Array, BinaryArray, BooleanArray, BooleanBufferBuilder}; @@ -53,8 +56,8 @@ use crate::sst::parquet::file_range::PreFilterMode; use crate::sst::parquet::flat_format::FlatReadFormat; use crate::sst::parquet::format::{PrimaryKeyArray, StatValues}; use crate::sst::parquet::reader::{ - MaybeFilter, PhysicalFilterContext, RowGroupBuildContext, RowGroupReaderBuilder, - SimpleFilterContext, + DynamicFilterSnapshot, MaybeFilter, PhysicalFilterContext, RowGroupBuildContext, + RowGroupReaderBuilder, SimpleFilterContext, }; pub(crate) fn matching_row_ranges_by_primary_key( @@ -401,6 +404,12 @@ pub(crate) fn build_bulk_filter_plan( /// the prefilter pass. A caller can postpone simple timestamp filters to the normal /// precise-filter path when the scan time range covers the SST. When predicate /// prefiltering is disabled, all simple filters remain on the normal path instead. +/// +/// Producer-owned dynamic filters, such as the membership filter a single-key `HashJoin` +/// publishes, are admitted the same best-effort way, from their published physical +/// expression instead of a logical [`Expr`]: see [`PhysicalFilterContext::new_dynamic`]. +/// Their masks are never cached, because the producer may republish the expression +/// between row groups. #[allow(clippy::too_many_arguments)] pub(crate) fn build_reader_filter_plan( predicate: Option<&Predicate>, @@ -517,6 +526,20 @@ pub(crate) fn build_reader_filter_plan( }; } + // Producer-owned dynamic filters are physical expressions the producer republishes + // while the scan runs. Evaluate their current snapshot as a prefilter pass so the row + // selection shrinks before the full projection is read. A dynamic filter never + // replaces the live filter above the scan, and its mask is never cached. + let dyn_filters = predicate.dyn_filters(); + for handle in dyn_filters.iter() { + if let Some(filter) = + PhysicalFilterContext::new_dynamic(metadata, expected_metadata, read_format, handle) + && can_direct_prefilter(filter.semantic_type()) + { + prefilter_physical_filters.push(filter); + } + } + let pk_filter_expr_strs = (!pk_filter_contexts.is_empty()).then(|| { let mut expr_strs = pk_filter_contexts .iter() @@ -567,6 +590,12 @@ pub(crate) struct PrefilterContext { /// Physical filters that can be evaluated directly from the prefilter batch. /// Physical expressions are only applied in the prefilter phase. physical_filters: Vec, + /// Producer-owned dynamic filter snapshots, aligned with `physical_filters`. + /// + /// Captured when this row group's prefilter starts and evaluated for the whole pass. + /// `None` for static filters and for dynamic filters this pass cannot use, which + /// keeps every row instead. + dynamic_snapshots: Vec>, /// Region schema version used in per-filter cache keys. schema_version: u64, /// Sorted expression strings for the encoded-PK filter group. @@ -577,6 +606,41 @@ pub(crate) struct PrefilterContext { proven_simple_filters: Vec, } +impl PrefilterContext { + /// Returns true when the dynamic filter at `physical_filters[idx]` contributes no mask + /// for this row group: no snapshot was captured, or the snapshot references no column. + /// + /// A producer that has not published yet still holds its `true` placeholder, and + /// reading the column for this row group would only cost I/O. A constant that rejects + /// every row is left to the row-group statistics pruning, which skips the whole row + /// group before any read. + fn dynamic_filter_is_noop(&self, idx: usize) -> bool { + let Some(filter) = self.physical_filters.get(idx) else { + return false; + }; + if !filter.is_dynamic() { + return false; + } + match self.dynamic_snapshots.get(idx) { + Some(Some(snapshot)) => collect_columns(&snapshot.expr).is_empty(), + _ => true, + } + } + + /// Returns true when the producer has not republished the dynamic filter at + /// `physical_filters[idx]` since this pass captured its snapshot. + fn dynamic_snapshot_is_current(&self, idx: usize) -> bool { + let (Some(filter), Some(Some(snapshot))) = ( + self.physical_filters.get(idx), + self.dynamic_snapshots.get(idx), + ) else { + return false; + }; + + filter.dynamic_snapshot_is_current(snapshot) + } +} + /// Pre-built state for constructing [PrefilterContext] per row group. /// /// Fields invariant across row groups (projection mask, codec, metadata, filters) @@ -675,6 +739,10 @@ impl PrefilterContextBuilder { } /// Builds a [PrefilterContext] for a specific row group. + /// + /// This is where a row group's prefilter starts, so producer-owned dynamic filters + /// are snapshotted here: one stable expression per filter, evaluated for the whole + /// pass and rechecked against the producer before its mask is committed. pub(crate) fn build(&self, row_group_idx: usize) -> PrefilterContext { let pk_filter = self .build_primary_key_filter() @@ -683,6 +751,11 @@ impl PrefilterContextBuilder { pk_filter, filters: self.filters.clone(), physical_filters: self.physical_filters.clone(), + dynamic_snapshots: self + .physical_filters + .iter() + .map(|filter| filter.capture_dynamic_snapshot()) + .collect(), schema_version: self.schema_version, pk_filter_expr_strs: self.pk_filter_expr_strs.clone(), arrow_schema: self.arrow_schema.clone(), @@ -935,8 +1008,10 @@ async fn execute_prefilter_with_result_cache( .copied() .map(|idx| PrefilterEntry::without_cache(PrefilterEntryKind::Physical(idx))), ); - let (uncached_mask, read_rows) = + let masks = build_prefilter_masks(prefilter_ctx, reader_builder, build_ctx, &uncached_entries).await?; + let read_rows = masks.rows; + let uncached_mask = masks.commit(prefilter_ctx); let final_mask = match (hit_mask, uncached_mask) { (Some(hit_mask), Some(uncached_mask)) => hit_mask.bitand(&uncached_mask), @@ -960,16 +1035,90 @@ fn non_cacheable_physical_filters(prefilter_ctx: &PrefilterContext) -> Vec, + /// Masks of producer-owned dynamic filters, keyed by their `physical_filters` index. + /// Only commit them while the producer stays at the generation they were snapshotted + /// at. + dynamic_masks: Vec<(usize, BooleanBuffer)>, + /// Number of rows this pass read. + rows: usize, +} + +impl PrefilterMasks { + /// ANDs the masks that may be committed for this row group. + /// + /// A producer may publish a new expression while this pass runs. The committed mask + /// must never be tighter than the live filter, and nothing here orders one generation + /// against another, so a dynamic mask whose generation moved on is discarded and the + /// incoming selection of the row group is kept as it was. Only the live filter above + /// the scan decides which rows a query returns. + /// + /// # Producer contract + /// + /// Committing a dynamic mask takes two properties from the producer: + /// + /// * **Stable snapshot.** The mask was computed from the single expression captured + /// at `generation` and is only committed while the handle is still at that + /// generation (see [`PhysicalFilterContext::capture_dynamic_snapshot`]). + /// * **A publication is a necessary condition for the same execution.** A committed + /// mask removes every row that generation rejects, so a publication is only allowed + /// to reject rows the enclosing join or limit rejects too. The supported producers + /// publish completed state: a `HashJoin` publishes its immutable, completed + /// build-side map, and `TopK` publishes a monotone (only ever tightening) threshold. + /// Arbitrary independently mutable nested expressions are not promised, and nothing + /// here compares two generations: a newer publication is simply re-evaluated by the + /// next pass, while the live filter above the scan stays authoritative. + /// + /// # Why a stale mask is dropped instead of retained + /// + /// For a monotone tightening producer such as `TopK`, the newer expression accepts a + /// subset of the rows the snapshotted one accepts, so keeping the older mask would + /// stay correct and prune more. Nothing observable here tells that producer apart from + /// one whose next publication widens, reverses, or rewrites its expression, and + /// keeping a mask across such a publication would return rows the live filter rejects. + /// A mask whose generation moved on is therefore treated as untrusted rather than + /// assumed safe: the mask is dropped, the incoming selection of the row group is kept, + /// and the next prefilter pass evaluates the new expression from scratch. Retaining a + /// stale mask needs an explicit monotone/tightening contract published by the producer + /// next to its generation; no producer exposes one today, so it can be added together + /// with the producer that relies on it. + fn commit(self, prefilter_ctx: &PrefilterContext) -> Option { + let mut mask = self.static_mask; + for (idx, dynamic_mask) in self.dynamic_masks { + // The producer moved on. Keeping this mask would assume the new expression only + // tightens the snapshotted one, which nothing here proves. + if !prefilter_ctx.dynamic_snapshot_is_current(idx) { + continue; + } + mask = Some(match mask { + Some(mask) => mask.bitand(&dynamic_mask), + None => dynamic_mask, + }); + } + + mask + } +} + async fn build_prefilter_masks( prefilter_ctx: &mut PrefilterContext, reader_builder: &RowGroupReaderBuilder, build_ctx: &RowGroupBuildContext<'_>, entries: &[PrefilterEntry], -) -> Result<(Option, usize)> { +) -> Result { let prefilter_column_names = prefilter_column_names_for_entries(prefilter_ctx, entries); let parquet_schema = reader_builder .parquet_metadata() @@ -994,7 +1143,14 @@ async fn build_prefilter_masks( .iter() .map(|entry| entry.key.is_some().then(|| BooleanBufferBuilder::new(0))) .collect::>(); - let mut combined_builder = (!entries.is_empty()).then(|| BooleanBufferBuilder::new(0)); + // Dynamic filter masks are kept per filter, so the commit step can still drop one + // whose generation moved on without discarding the others. + let mut dynamic_builders = prefilter_ctx + .dynamic_snapshots + .iter() + .map(|snapshot| snapshot.is_some().then(|| BooleanBufferBuilder::new(0))) + .collect::>(); + let mut combined_builder = BooleanBufferBuilder::new(0); let mut rows_before_filter = 0usize; while let Some(batch_result) = stream.next().await { @@ -1013,14 +1169,21 @@ async fn build_prefilter_masks( entry.kind, reader_builder.file_path(), )?; - batch_mask = batch_mask.bitand(&mask); + let dynamic_builder = match entry.kind { + PrefilterEntryKind::Physical(physical_idx) => dynamic_builders + .get_mut(physical_idx) + .and_then(Option::as_mut), + PrefilterEntryKind::Simple(_) | PrefilterEntryKind::PkGroup => None, + }; + match dynamic_builder { + Some(builder) => builder.append_buffer(&mask), + None => batch_mask = batch_mask.bitand(&mask), + } if let Some(Some(builder)) = cache_builders.get_mut(idx) { builder.append_buffer(&mask); } } - if let Some(builder) = &mut combined_builder { - builder.append_buffer(&batch_mask); - } + combined_builder.append_buffer(&batch_mask); } for (entry, builder) in entries.iter().zip(cache_builders) { @@ -1031,10 +1194,18 @@ async fn build_prefilter_masks( } } - Ok(( - combined_builder.map(|mut builder| builder.finish()), - rows_before_filter, - )) + Ok(PrefilterMasks { + static_mask: Some(combined_builder.finish()) + .filter(|mask: &BooleanBuffer| !mask.is_empty()), + dynamic_masks: dynamic_builders + .into_iter() + .enumerate() + .filter_map(|(idx, builder)| builder.map(|mut builder| (idx, builder.finish()))) + // A dynamic filter without a usable snapshot reads no row for this pass. + .filter(|(_, mask)| !mask.is_empty()) + .collect(), + rows: rows_before_filter, + }) } fn prefilter_column_names_for_entries( @@ -1073,8 +1244,9 @@ async fn execute_prefilter_by_reading_columns( if entries.is_empty() { return Ok(identity_prefilter_result(reader_builder, build_ctx)); } - let (mask, rows_before_filter) = - build_prefilter_masks(prefilter_ctx, reader_builder, build_ctx, &entries).await?; + let masks = build_prefilter_masks(prefilter_ctx, reader_builder, build_ctx, &entries).await?; + let rows_before_filter = masks.rows; + let mask = masks.commit(prefilter_ctx); let final_mask = mask.unwrap_or_else(|| BooleanBuffer::new_set(rows_before_filter)); let rows_selected = final_mask.count_set_bits(); @@ -1111,6 +1283,7 @@ fn all_prefilter_entries(prefilter_ctx: &PrefilterContext) -> Vec { - eval_physical_filter_mask(batch, &prefilter_ctx.physical_filters[idx], file_path) + let filter_ctx = &prefilter_ctx.physical_filters[idx]; + let Some(snapshot) = prefilter_ctx + .dynamic_snapshots + .get(idx) + .and_then(Option::as_ref) + else { + // A static filter; a dynamic filter without a usable snapshot keeps all + // rows instead of being evaluated at its live, ever-changing generation. + let Some(filter) = filter_ctx.filter() else { + return Ok(BooleanBuffer::new_set(batch.num_rows())); + }; + return evaluate_physical_mask(batch, filter_ctx, filter, file_path); + }; + // A producer-owned expression is only checked against the prefilter schema + // when it is snapshotted. Keep every row if evaluating it fails, exactly like + // any other unsupported prefilter candidate. + match evaluate_physical_mask(batch, filter_ctx, &snapshot.expr, file_path) { + Ok(mask) => Ok(mask), + Err(error) => { + // The producer owns the expression, so a snapshot that cannot be + // evaluated is a miss for this pass, not a scan error. + debug!( + "Ignored dynamic prefilter snapshot of {} that failed to evaluate: {}, file: {}", + filter_ctx.column_name(), + error, + file_path + ); + Ok(BooleanBuffer::new_set(batch.num_rows())) + } + } } PrefilterEntryKind::PkGroup => { let pk_filter = prefilter_ctx.pk_filter.as_mut().context(UnexpectedSnafu { @@ -1322,13 +1524,18 @@ fn eval_simple_filter_mask( filter.evaluate_array(&column).context(RecordBatchSnafu) } -fn eval_physical_filter_mask( +/// Evaluates a physical prefilter expression against the single filter column of `batch`. +/// +/// `NULL` results count as filtered out. This is safe because the published predicate is +/// already a necessary condition for the enclosing operation; legacy consumer-side NULL +/// guards may admit additional rows that the operation does not need. No `IS NULL` guard +/// is added here because the producer owns the null semantics of its expression. +pub(crate) fn evaluate_physical_mask( batch: &RecordBatch, filter_ctx: &PhysicalFilterContext, + filter: &Arc, file_path: &str, ) -> Result { - let filter = filter_ctx.filter(); - let (idx, _) = batch .schema() .column_with_name(filter_ctx.column_name()) @@ -1371,12 +1578,15 @@ mod tests { use std::sync::atomic::{AtomicUsize, Ordering}; use bytes::Bytes; - use common_recordbatch::filter::SimpleFilterEvaluator; + use common_recordbatch::filter::{SimpleFilterEvaluator, batch_filter}; + use datafusion::physical_plan::expressions::{ + Column, DynamicFilterPhysicalExpr, lit as physical_lit, + }; use datafusion_common::ScalarValue; use datafusion_expr::{col, lit}; use datatypes::arrow::array::{ - ArrayRef, DictionaryArray, Int32Array, TimestampMillisecondArray, UInt8Array, UInt32Array, - UInt64Array, + ArrayRef, DictionaryArray, Float64Array, Int32Array, TimestampMillisecondArray, UInt8Array, + UInt32Array, UInt64Array, }; use datatypes::arrow::datatypes::{DataType, Field, Schema, UInt32Type}; use datatypes::arrow::record_batch::RecordBatch; @@ -2011,6 +2221,7 @@ mod tests { pk_filter: None, filters, physical_filters: Vec::new(), + dynamic_snapshots: Vec::new(), schema_version: metadata.schema_version, pk_filter_expr_strs: None, arrow_schema: metadata.schema.arrow_schema().clone(), @@ -2027,6 +2238,102 @@ mod tests { )); } + /// A generation change discards the dynamic mask even when the new expression is a + /// monotone tightening of the snapshotted one. + /// + /// A `TopK` only ever tightens its threshold, so the mask of the generation a pass + /// snapshotted would still be implied by the newer expression and keeping it would + /// prune more. Nothing here can tell such a producer apart from one whose next + /// publication widens, so a mask from a superseded generation is treated as untrusted + /// and dropped: the row group keeps its incoming selection and the next pass evaluates + /// the new expression from scratch. + #[test] + fn test_commit_discards_dynamic_mask_after_monotone_tightening_publication() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = FlatReadFormat::new( + metadata.clone(), + ReadColumns::new(metadata.column_metadatas.iter().map(|c| c.column_id)), + None, + "memtable", + false, + ) + .unwrap(); + let producer_schema = Arc::new(Schema::new(vec![Field::new( + "field_0", + DataType::UInt64, + true, + )])); + let publish = |members: Vec| { + let expr = col("field_0").in_list(members.into_iter().map(lit).collect(), false); + Predicate::to_physical_expr(&expr, &producer_schema).unwrap() + }; + let key = Arc::new(Column::new("field_0", 0)) as Arc; + // The producer starts from `{1, 2}` and only ever tightens the membership below. + let handle = Arc::new(DynamicFilterPhysicalExpr::new( + vec![key], + physical_lit(true), + )); + handle.update(publish(vec![1, 2])).unwrap(); + + // A prefilter pass of three rows that snapshots the current generation. + let new_context = || { + let filter = + PhysicalFilterContext::new_dynamic(&metadata, None, &read_format, &handle).unwrap(); + let snapshot = filter.capture_dynamic_snapshot(); + PrefilterContext { + pk_filter: None, + filters: Vec::new(), + physical_filters: vec![filter], + dynamic_snapshots: vec![snapshot], + schema_version: metadata.schema_version, + pk_filter_expr_strs: None, + arrow_schema: read_format.arrow_schema().clone(), + proven_simple_filters: Vec::new(), + } + }; + let mask_bits = |mask: BooleanBuffer| mask.iter().collect::>(); + let keep_all = BooleanBuffer::new_set(3); + // `{1, 2}` keeps rows 1 and 2 of the three rows the pass read. + let membership_12 = BooleanBuffer::from(vec![false, true, true]); + + let context = new_context(); + let committed = PrefilterMasks { + static_mask: Some(keep_all.clone()), + dynamic_masks: vec![(0, membership_12.clone())], + rows: 3, + } + .commit(&context); + assert_eq!( + vec![false, true, true], + mask_bits(committed.unwrap()), + "a mask of the snapshotted generation is committed" + ); + + // Tightening `{1, 2}` to `{2}` supersedes the generation the pass snapshotted. + handle.update(publish(vec![2])).unwrap(); + let committed = PrefilterMasks { + static_mask: Some(keep_all.clone()), + dynamic_masks: vec![(0, membership_12.clone())], + rows: 3, + } + .commit(&context); + assert_eq!( + vec![true, true, true], + mask_bits(committed.unwrap()), + "the superseded mask must be dropped, not retained as a tightening" + ); + + // The next pass snapshots the tightened generation and prunes with it. + let context = new_context(); + let committed = PrefilterMasks { + static_mask: Some(keep_all), + dynamic_masks: vec![(0, BooleanBuffer::from(vec![false, false, true]))], + rows: 3, + } + .commit(&context); + assert_eq!(vec![false, false, true], mask_bits(committed.unwrap())); + } + #[test] fn test_identity_row_selection_preserves_input() { let sparse = RowSelection::from(vec![ @@ -2242,6 +2549,116 @@ mod tests { ); } + /// Creates the dynamic filter a single-key `HashJoin` publishes for `members`. + fn new_dynamic_field_filter( + read_format: &FlatReadFormat, + members: Vec, + ) -> Arc { + let arrow_schema = read_format.arrow_schema(); + let key = Arc::new(Column::new("field_0", 0)) as Arc; + let handle = Arc::new(DynamicFilterPhysicalExpr::new( + vec![key], + Predicate::to_physical_expr(&lit(true), arrow_schema).unwrap(), + )); + let expr = col("field_0").in_list(members.into_iter().map(lit).collect(), false); + handle + .update(Predicate::to_physical_expr(&expr, arrow_schema).unwrap()) + .unwrap(); + handle + } + + fn dynamic_filter_count(builder: &PrefilterContextBuilder) -> usize { + builder + .physical_filters + .iter() + .filter(|filter| filter.is_dynamic()) + .count() + } + + /// A dynamic filter on a field column is admitted only where field pruning is safe. + /// `SkipFields` preserves the dedup semantics of a multi-source `LastRow` / + /// `LastNonNull` merge, so the dynamic filter must not prune fields there either. + #[test] + fn test_build_reader_filter_plan_gates_dynamic_filters_by_prefilter_mode() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = FlatReadFormat::new( + metadata.clone(), + ReadColumns::new(metadata.column_metadatas.iter().map(|c| c.column_id)), + None, + "test", + true, + ) + .unwrap(); + let codec = build_primary_key_codec(metadata.as_ref()); + let parquet_metadata = stats_metadata(&[vec![1]], None); + let predicate = || { + Predicate::with_dyn_filters( + vec![], + vec![new_dynamic_field_filter(&read_format, vec![1])], + ) + }; + + let all_plan = build_reader_filter_plan( + Some(&predicate()), + None, + PreFilterMode::All, + true, + false, + &read_format, + &codec, + &parquet_metadata, + ); + assert_eq!( + 1, + dynamic_filter_count(all_plan.prefilter_builder.as_ref().unwrap()) + ); + + // The dynamic filter is the only prefilter candidate, and it is gated out. + let skip_fields_plan = build_reader_filter_plan( + Some(&predicate()), + None, + PreFilterMode::SkipFields, + true, + false, + &read_format, + &codec, + &parquet_metadata, + ); + assert!(skip_fields_plan.prefilter_builder.is_none()); + + // With a tag filter the prefilter still runs, without the dynamic field filter. + let skip_fields_plan = build_reader_filter_plan( + Some(&Predicate::with_dyn_filters( + vec![col("tag_0").eq(lit("a"))], + vec![new_dynamic_field_filter(&read_format, vec![1])], + )), + None, + PreFilterMode::SkipFields, + true, + false, + &read_format, + &codec, + &parquet_metadata, + ); + assert_eq!( + 0, + dynamic_filter_count(skip_fields_plan.prefilter_builder.as_ref().unwrap()) + ); + + // Disabled predicate prefiltering keeps dynamic filters on the normal path. + let disabled_plan = build_reader_filter_plan( + Some(&predicate()), + None, + PreFilterMode::All, + false, + false, + &read_format, + &codec, + &parquet_metadata, + ); + assert!(disabled_plan.prefilter_builder.is_none()); + } + #[test] fn test_pk_filter_expr_strings_are_stable_under_expr_order() { let metadata: RegionMetadataRef = Arc::new(sst_region_metadata_with_encoding( @@ -2348,10 +2765,177 @@ mod tests { let pk = new_primary_key(&["a", "x"]); let batch = new_raw_batch(&[pk.as_slice(), pk.as_slice(), pk.as_slice()], &[9, 10, 11]); - let mask = eval_physical_filter_mask(&batch, &physical_filters[0], "test").unwrap(); + let filter = physical_filters[0].filter().unwrap(); + let mask = evaluate_physical_mask(&batch, &physical_filters[0], filter, "test").unwrap(); assert_eq!(mask.count_set_bits(), 1); } + /// The producer's input schema, where `field_0` is its fourth column rather than the + /// only column of a prefilter batch. + fn producer_schema(field_type: DataType) -> SchemaRef { + Arc::new(Schema::new(vec![ + Field::new("tag_0", DataType::Utf8, true), + Field::new("tag_1", DataType::Utf8, true), + Field::new("ts", DataType::UInt64, true), + Field::new("field_0", field_type, true), + ])) + } + + fn full_read_format(metadata: &RegionMetadataRef) -> FlatReadFormat { + FlatReadFormat::new( + metadata.clone(), + ReadColumns::new(metadata.column_metadatas.iter().map(|c| c.column_id)), + None, + "test", + false, + ) + .unwrap() + } + + /// Evaluates a producer-owned dynamic filter over `batch` both ways: as a prefilter + /// mask, and as the live `batch_filter` that runs above the scan. + /// + /// `batch` carries the filter column plus a `row` column holding row indexes, so the + /// rows the upper filter keeps come back as values. Both evaluations see the same + /// rows, so the returned row lists are directly comparable. + fn prefilter_and_upper_rows( + metadata: &RegionMetadataRef, + read_format: &FlatReadFormat, + producer_schema: &SchemaRef, + published: &Expr, + batch: &RecordBatch, + ) -> (Vec, Vec) { + let key = Arc::new(Column::new("field_0", 3)) as Arc; + let handle = Arc::new(DynamicFilterPhysicalExpr::new( + vec![key], + Predicate::to_physical_expr(&lit(true), producer_schema).unwrap(), + )); + handle + .update(Predicate::to_physical_expr(published, producer_schema).unwrap()) + .unwrap(); + let filter_ctx = + PhysicalFilterContext::new_dynamic(metadata, None, read_format, &handle).unwrap(); + let snapshot = filter_ctx.capture_dynamic_snapshot().unwrap(); + + let mask = evaluate_physical_mask(batch, &filter_ctx, &snapshot.expr, "test").unwrap(); + let prefilter_rows = mask + .iter() + .enumerate() + .filter_map(|(row, keep)| keep.then_some(row)) + .collect::>(); + + let filtered = batch_filter(batch, &snapshot.expr).unwrap(); + let upper_rows = filtered + .column(1) + .as_any() + .downcast_ref::() + .unwrap() + .values() + .iter() + .map(|row| *row as usize) + .collect::>(); + + (prefilter_rows, upper_rows) + } + + /// A batch of `field_0` values plus a `row` column holding their indexes. + fn indexed_batch(field: ArrayRef, data_type: DataType) -> RecordBatch { + let schema = Arc::new(Schema::new(vec![ + Field::new("field_0", data_type, true), + Field::new("row", DataType::UInt64, false), + ])); + let rows = Arc::new(UInt64Array::from_iter_values(0..field.len() as u64)) as ArrayRef; + RecordBatch::try_new(schema, vec![field, rows]).unwrap() + } + + /// A bare `col IN (...)` leaves a NULL comparison UNKNOWN, and the prefilter must + /// reject that row exactly like the live filter above the scan: it adds no NULL guard + /// of its own, because the producer owns the NULL semantics of its expression. + #[test] + fn test_dynamic_filter_mask_matches_upper_filter_for_nullable_bare_in() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = full_read_format(&metadata); + let published = col("field_0").in_list(vec![lit(1_u64), lit(3_u64)], false); + // Row 1 is NULL, and `NULL IN (1, 3)` is neither a member nor a non-member. + let batch = indexed_batch( + Arc::new(UInt64Array::from(vec![Some(1), None, Some(2), Some(3)])) as ArrayRef, + DataType::UInt64, + ); + + let (prefilter_rows, upper_rows) = prefilter_and_upper_rows( + &metadata, + &read_format, + &producer_schema(DataType::UInt64), + &published, + &batch, + ); + + assert_eq!(vec![0, 3], prefilter_rows); + assert_eq!( + prefilter_rows, upper_rows, + "the prefilter must keep exactly the rows the upper filter keeps" + ); + } + + /// Float membership compares with a total order, so `NaN` is a member of a list that + /// holds `NaN` while `-0.0` is not a member of a list that holds `0.0`. The prefilter + /// must reproduce those semantics exactly, which it can only do by evaluating the + /// producer's whole expression instead of substituting an IEEE `==` or a re-derived + /// comparison. + #[test] + fn test_dynamic_filter_mask_matches_upper_filter_for_float_nan_membership() { + let metadata = metadata_with_field_type(ConcreteDataType::float64_datatype()); + let read_format = full_read_format(&metadata); + let published = col("field_0").in_list(vec![lit(0.0_f64), lit(f64::NAN)], false); + // Rows: `0.0` and `NaN` are members, `-0.0` is not (total order, not IEEE `==`), + // and the NULL row is UNKNOWN. + let batch = indexed_batch( + Arc::new(Float64Array::from(vec![ + Some(0.0), + Some(-0.0), + Some(f64::NAN), + None, + ])) as ArrayRef, + DataType::Float64, + ); + let unlisted = indexed_batch( + Arc::new(Float64Array::from(vec![ + Some(1.5), + Some(f64::NAN), + Some(0.0), + None, + ])) as ArrayRef, + DataType::Float64, + ); + + let (prefilter_rows, upper_rows) = prefilter_and_upper_rows( + &metadata, + &read_format, + &producer_schema(DataType::Float64), + &published, + &batch, + ); + + assert_eq!( + prefilter_rows, upper_rows, + "the prefilter must keep exactly the rows the upper filter keeps" + ); + assert_eq!(vec![0, 2], prefilter_rows); + + let (unlisted_prefilter_rows, unlisted_upper_rows) = prefilter_and_upper_rows( + &metadata, + &read_format, + &producer_schema(DataType::Float64), + &published, + &unlisted, + ); + assert_eq!( + unlisted_prefilter_rows, unlisted_upper_rows, + "the prefilter must keep exactly the rows the upper filter keeps" + ); + assert_eq!(vec![1, 2], unlisted_prefilter_rows); + } + #[test] fn test_eval_pk_group_mask_finds_pk_column_by_name() { let metadata = Arc::new(sst_region_metadata()); diff --git a/src/mito2/src/sst/parquet/reader.rs b/src/mito2/src/sst/parquet/reader.rs index e92ac60f93b..a5feb33ca85 100644 --- a/src/mito2/src/sst/parquet/reader.rs +++ b/src/mito2/src/sst/parquet/reader.rs @@ -25,7 +25,9 @@ use arrow_schema::extension::{ use common_recordbatch::QueryMemoryTracker; use common_recordbatch::filter::{SimpleFilterEvaluator, TimestampUnitCast}; use common_telemetry::{debug, error, tracing, warn}; +use datafusion::physical_expr::utils::{collect_columns, reassign_expr_columns}; use datafusion::physical_plan::PhysicalExpr; +use datafusion::physical_plan::expressions::DynamicFilterPhysicalExpr; use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion}; use datafusion_expr::utils::expr_to_columns; use datafusion_expr::{Expr, Volatility}; @@ -45,7 +47,6 @@ use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData}; use parquet::file::properties::DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT; use partition::expr::PartitionExpr; use snafu::{OptionExt, ResultExt}; -use store_api::codec::PrimaryKeyEncoding; use store_api::metadata::{ColumnMetadata, RegionMetadata, RegionMetadataRef}; use store_api::region_request::PathType; use store_api::storage::consts::PRIMARY_KEY_COLUMN_NAME; @@ -62,7 +63,6 @@ use crate::error::{ UnexpectedSnafu, }; use crate::metrics::{READ_ROWS_TOTAL, READ_STAGE_ELAPSED}; -use crate::read::flat_projection::CompactionProjectionMapper; use crate::read::prune::FlatPruneReader; use crate::read::read_columns::ReadColumns; use crate::series_index::SeriesIndexReadContext; @@ -467,36 +467,9 @@ impl ParquetReaderBuilder { let mut parquet_meta = sst_meta.parquet_metadata(); let mut parquet_metadata_size = sst_meta.parquet_metadata_size(); let region_meta = sst_meta.region_metadata(); - let region_partition_expr_str = self - .expected_metadata - .as_ref() - .and_then(|meta| meta.partition_expr.as_ref()) - .map(|expr| expr.as_str()); - let (_, is_same_region_partition) = Self::is_same_region_partition( - region_partition_expr_str, - self.file_handle.meta_ref().partition_expr.as_ref(), - )?; - // Skip auto convert when: - // - compaction is enabled - // - region partition expr is same with file partition expr (no need to auto convert) - let skip_auto_convert = self.compaction && is_same_region_partition; - - // Build a compaction projection helper when: - // - compaction is enabled - // - region partition expr differs from file partition expr - // - flat format is enabled - // - primary key encoding is sparse - // - // This is applied after row-group filtering to align batches with flat output schema - // before compat handling. - let compaction_projection_mapper = if self.compaction - && !is_same_region_partition - && region_meta.primary_key_encoding == PrimaryKeyEncoding::Sparse - { - Some(CompactionProjectionMapper::try_new(®ion_meta)?) - } else { - None - }; + // Sparse compaction preserves encoded keys. Partition predicates decode only + // their dependencies into temporary arrays during precise filtering. + let skip_auto_convert = self.compaction; let read_cols = if let Some(read_cols) = &self.read_cols { read_cols.clone() @@ -715,7 +688,6 @@ impl ParquetReaderBuilder { prune_schema, codec, compat_batch: None, - compaction_projection_mapper, pre_filter_mode: self.pre_filter_mode, partition_filter, }, @@ -740,6 +712,9 @@ impl ParquetReaderBuilder { Ok((region_partition_expr, is_same)) } + /// Builds the dependency-only schema for row-level partition evaluation. + /// Its indices are independent of both Parquet and converted batch projections; + /// precise filtering gathers these columns temporarily before compatibility. /// Compare partition expressions from expected metadata and file metadata, /// and build a partition filter if they differ. fn build_partition_filter( @@ -750,13 +725,11 @@ impl ParquetReaderBuilder { let region_partition_expr_str = self .expected_metadata .as_ref() - .and_then(|meta| meta.partition_expr.as_ref()); - let file_partition_expr_ref = self.file_handle.meta_ref().partition_expr.as_ref(); + .and_then(|meta| meta.partition_expr.as_deref()); + let file_partition_expr_ref = self.file_handle.meta_ref().partition_expr.as_deref(); - let (region_partition_expr, is_same_region_partition) = Self::is_same_region_partition( - region_partition_expr_str.map(|s| s.as_str()), - file_partition_expr_ref, - )?; + let (region_partition_expr, is_same_region_partition) = + Self::is_same_region_partition(region_partition_expr_str, file_partition_expr_ref)?; if is_same_region_partition { return Ok(None); @@ -928,6 +901,8 @@ impl ParquetReaderBuilder { } /// Computes row groups to read, along with their respective row selections. + /// Statistics and index lookups use source-file column mappings, independently + /// of whether conversion will later expand encoded tags. #[tracing::instrument( skip_all, fields( @@ -2159,7 +2134,12 @@ impl SimpleFilterContext { #[derive(Clone)] pub(crate) struct PhysicalFilterContext { /// Filter to evaluate. - filter: Arc, + /// + /// `None` for a producer-owned dynamic filter, whose expression changes while the + /// scan runs and is therefore captured per prefilter pass. + filter: Option>, + /// Producer-owned dynamic filter to snapshot for each prefilter pass. + dynamic: Option>, /// Debug string of the original logical expression. expr_str: String, /// Id of the column to evaluate. @@ -2170,10 +2150,24 @@ pub(crate) struct PhysicalFilterContext { semantic_type: SemanticType, /// Schema containing only the referenced column. schema: SchemaRef, - /// Whether the original logical expression is immutable across queries. + /// Whether the filter is immutable across queries. immutable: bool, } +/// A stable snapshot of the expression a producer published for a dynamic filter. +/// +/// The expression is rebound to the single-column prefilter schema and is only +/// trustworthy while the producer stays at `generation`. See +/// [`PhysicalFilterContext::capture_dynamic_snapshot`] for the producer contract this +/// snapshot depends on. +#[derive(Clone)] +pub(crate) struct DynamicFilterSnapshot { + /// Captured expression, rebound to the prefilter schema. + pub(crate) expr: Arc, + /// Producer generation the expression was captured at. + generation: u64, +} + impl PhysicalFilterContext { /// Creates a context for the `expr`. /// @@ -2190,27 +2184,7 @@ impl PhysicalFilterContext { } let expr_str = format!("{expr:?}"); let column_name = Self::single_column_name(expr)?; - let column_metadata = match expected_meta { - Some(meta) => { - let column = meta.column_by_name(&column_name)?; - let sst_column = sst_meta.column_by_id(column.column_id)?; - // Physical expr requires the column name to match the SST column name. - if sst_column.column_schema.name != column_name { - return None; - } - // Schema evolution (an altered field column, or a widened time - // index unit) can make the column's file type differ from the - // expected type; evaluating the original expr against the - // file's column would raise a cross-type comparison error - // (e.g. Timestamp(ms) >= Timestamp(µs)). Drop the prefilter; - // the query layer re-applies the predicate above the scan. - if sst_column.column_schema.data_type != column.column_schema.data_type { - return None; - } - column - } - None => sst_meta.column_by_name(&column_name)?, - }; + let column_metadata = Self::validated_column(sst_meta, expected_meta, &column_name)?; // The column must be present in the projected arrow schema for the // prefilter to be able to read it. @@ -2225,7 +2199,8 @@ impl PhysicalFilterContext { let immutable = expr_is_immutable(expr); Some(Self { - filter: physical_expr, + filter: Some(physical_expr), + dynamic: None, expr_str, column_id: column_metadata.column_id, column_name, @@ -2235,6 +2210,167 @@ impl PhysicalFilterContext { }) } + /// Creates a context for a producer-owned dynamic filter. + /// + /// Unlike [`Self::new_opt`], the expression is built by the producer (for example a + /// single-key `HashJoin`) against its own input schema, so this constructor admits only + /// filters whose declared dependencies are the filter column itself, and rebinds that + /// column to the single-column prefilter schema. It does not restrict the shape of the + /// published expression: the whole expression is evaluated as-is. + /// + /// Returns `None` when the dependencies are not exactly the filter column, or when + /// the SST and the expected metadata disagree on the column's identity or type. The + /// prefilter then keeps every row. + pub(crate) fn new_dynamic( + sst_meta: &RegionMetadataRef, + expected_meta: Option<&RegionMetadata>, + read_format: &FlatReadFormat, + handle: &Arc, + ) -> Option { + // The columns a dynamic filter may reference are fixed when it is created, so + // they decide admission before any expression is published. + let column_name = Self::single_dynamic_column(handle)?; + let column_metadata = Self::validated_column(sst_meta, expected_meta, &column_name)?; + let (_, field) = read_format.arrow_schema().column_with_name(&column_name)?; + // The prefilter batch must carry the type the producer built its expression + // against. A changed type (an altered column, or a tag column that the flat + // format dictionary-encodes) would fail the comparison or silently compare + // across types. + if *field.data_type() != column_metadata.column_schema.data_type.as_arrow_type() { + return None; + } + let schema = Arc::new(ArrowSchema::new(vec![field.clone()])); + + Some(Self { + filter: None, + dynamic: Some(Arc::clone(handle)), + // A stable debug string: the published expression changes between generations. + expr_str: format!( + "dynamic filter {:?} on {}", + handle.expression_id(), + column_name + ), + column_id: column_metadata.column_id, + column_name, + semantic_type: column_metadata.semantic_type, + schema, + // The producer republishes the expression while the scan runs. + immutable: false, + }) + } + + /// Captures the current expression of a producer-owned dynamic filter. + /// + /// Returns `None` for a static filter, and whenever no trustworthy snapshot exists: + /// the producer updated the filter while the snapshot was taken (the generation read + /// before and after [`DynamicFilterPhysicalExpr::current`] differs), or the published + /// expression cannot be rebound to the single-column prefilter schema, which includes + /// every reference outside this filter's column. Callers must keep every row then. + /// + /// # Producer contract + /// + /// The snapshot is evaluated as-is for a whole row-group prefilter pass, so the + /// published expression has to be usable on its own: + /// + /// * **A publication is a stable snapshot.** It is a value the producer hands over, + /// not a view of state the producer keeps mutating: one expression is captured per + /// pass and evaluated against every batch of that pass. + /// * **A publication is a necessary condition for the same execution.** The prefilter + /// only ever drops rows its snapshot rejects, so a publication that accepts every + /// row the enclosing join or limit keeps can never change the query result, however + /// loose it is. A publication must therefore never reject a row the execution's own + /// semantics keeps. + /// * **Only this handle's generation is observed.** A + /// [`DynamicFilterPhysicalExpr`] nested inside the published expression that + /// republishes on its own is invisible to + /// [`Self::dynamic_snapshot_is_current`]; such an expression is not promised to be + /// snapshotted at all. + /// + /// The supported producers publish completed, self-contained state: a `HashJoin` + /// publishes its immutable, completed build-side map (the build side does not change + /// once the filter is published), and `TopK` publishes its monotone (only ever + /// tightening) threshold as a fresh comparison expression. Arbitrary independently + /// mutable nested expressions are not promised: the snapshot is then either rejected + /// here and the pass keeps every row, or the producer's own generation keeps it + /// honest. The live filter above the scan stays authoritative either way. + pub(crate) fn capture_dynamic_snapshot(&self) -> Option { + let handle = self.dynamic.as_ref()?; + let generation = handle.snapshot_generation(); + let expr = handle.current().ok()?; + if handle.snapshot_generation() != generation { + return None; + } + let expr = reassign_expr_columns(expr, &self.schema).ok()?; + + Some(DynamicFilterSnapshot { expr, generation }) + } + + /// Returns true when the producer has not published a newer expression since + /// `snapshot` was captured. + pub(crate) fn dynamic_snapshot_is_current(&self, snapshot: &DynamicFilterSnapshot) -> bool { + self.dynamic + .as_ref() + .is_some_and(|handle| handle.snapshot_generation() == snapshot.generation) + } + + /// Returns true for a producer-owned dynamic filter. + pub(crate) fn is_dynamic(&self) -> bool { + self.dynamic.is_some() + } + + /// Resolves the column a filter reads after checking that the SST and the expected + /// metadata agree on its identity (id and name) and type. + fn validated_column<'a>( + sst_meta: &'a RegionMetadataRef, + expected_meta: Option<&'a RegionMetadata>, + column_name: &str, + ) -> Option<&'a ColumnMetadata> { + match expected_meta { + Some(meta) => { + // Gets the column metadata from the expected metadata. + let column = meta.column_by_name(column_name)?; + // Checks if the column is present in the SST metadata. We still uses the + // column from the expected metadata. + let sst_column = sst_meta.column_by_id(column.column_id)?; + debug_assert_eq!(column.semantic_type, sst_column.semantic_type); + // Physical exprs reference the column by the expected name; a renamed + // column keeps its old name in the SST. + if sst_column.column_schema.name != column_name { + return None; + } + // Schema evolution (an altered field column, or a widened time index + // unit) can make columns with the same id have different types across + // SSTs; evaluating the expression against the file's column would raise + // a cross-type comparison error. Drop the prefilter; the query layer + // re-applies the predicate above the scan. + if sst_column.column_schema.data_type != column.column_schema.data_type { + return None; + } + Some(column) + } + None => sst_meta.column_by_name(column_name), + } + } + + /// Returns the single column all declared dependencies of a dynamic filter read. + /// + /// A multi-key producer publishes one filter over several key columns; such a filter + /// cannot be evaluated on a single-column prefilter batch, so it is rejected. + fn single_dynamic_column(handle: &DynamicFilterPhysicalExpr) -> Option { + let mut column_name = None; + for child in handle.children() { + for column in collect_columns(child) { + match &column_name { + Some(name) if *name != column.name() => return None, + Some(_) => {} + None => column_name = Some(column.name().to_string()), + } + } + } + + column_name + } + /// Returns true if the expression is a variant we want to evaluate as a /// physical prefilter. Binary exprs are intentionally excluded because /// [`SimpleFilterEvaluator`] already handles them. @@ -2266,8 +2402,12 @@ impl PhysicalFilterContext { } /// Returns the filter to evaluate. - pub(crate) fn filter(&self) -> &Arc { - &self.filter + /// + /// `None` for a producer-owned dynamic filter: its expression changes while the scan + /// runs, so callers capture a snapshot with [`Self::capture_dynamic_snapshot`] + /// instead of evaluating this field. + pub(crate) fn filter(&self) -> Option<&Arc> { + self.filter.as_ref() } /// Returns the original logical expression string. @@ -2498,12 +2638,20 @@ mod tests { use common_function::scalars::json::json_get::JsonGetWithType; use common_function::scalars::udf::create_udf; use common_recordbatch::ext::RecordBatchExt; + use common_recordbatch::filter::batch_filter; use datafusion::arrow::datatypes::DataType; + use datafusion::physical_plan::expressions::{ + BinaryExpr, CaseExpr, Column, lit as physical_lit, + }; + use datafusion::physical_plan::joins::join_hash_map::{JoinHashMapType, JoinHashMapU32}; + use datafusion::physical_plan::joins::{HashExpr, HashTableLookupExpr, Map, SeededRandomState}; + use datafusion::physical_plan::repartition::REPARTITION_RANDOM_STATE; use datafusion_common::ScalarValue; + use datafusion_common::hash_utils::create_hashes; use datafusion_expr::expr::ScalarFunction; use datafusion_expr::{ - ColumnarValue, Expr, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility, - col, lit, + ColumnarValue, Expr, Operator, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, + Volatility, col, lit, }; use datatypes::arrow::array::{ ArrayRef, BinaryDictionaryBuilder, Int64Array, StringArray, StructArray, @@ -2530,10 +2678,14 @@ mod tests { use crate::cache::CacheManager; use crate::cache::test_util::{GatedStorageReader, parquet_file_data_with_region_metadata}; use crate::sst::parquet::metadata::MetadataLoader; - use crate::sst::parquet::prefilter::{build_reader_filter_plan, execute_prefilter}; + use crate::sst::parquet::prefilter::{ + build_reader_filter_plan, evaluate_physical_mask, execute_prefilter, + }; use crate::sst::parquet::read_columns::{ParquetReadColumn, ParquetReadColumns}; use crate::sst::parquet::row_group::ParquetFetchMetrics; - use crate::test_util::sst_util::{sst_file_handle, sst_region_metadata}; + use crate::test_util::sst_util::{ + new_record_batch_with_custom_sequence, sst_file_handle, sst_region_metadata, + }; #[tokio::test] async fn test_shared_file_metadata_resolution_counts_without_duplicate_io() { @@ -2798,10 +2950,145 @@ mod tests { } } + #[rstest::rstest] + #[case(None)] + #[case(Some(false))] + #[case(Some(true))] + #[tokio::test] + async fn test_sparse_compaction_partition_mismatch_keeps_encoded_layout( + #[case] prefilter: Option, + ) { + let source = crate::test_util::sst_util::sst_region_metadata_with_encoding( + PrimaryKeyEncoding::Sparse, + ); + let props = WriterProperties::builder() + .set_key_value_metadata(Some(vec![parquet::file::metadata::KeyValue::new( + crate::sst::parquet::PARQUET_METADATA_KEY.to_string(), + source.to_json().unwrap(), + )])) + .build(); + let store = ObjectStore::new(Memory::default()).unwrap(); + let (fixture, metadata) = prefilter_test_builder_with( + store.clone(), + 3, + 2, + PageIndexPolicy::Skip, + Some(props), + Predicate::new(vec![col("tag_0").eq(lit("a"))]), + CacheStrategy::Disabled, + ) + .await; + for partition in [None, Some("a"), Some("b"), Some("missing")] { + let mut expected = metadata.as_ref().clone(); + expected.set_partition_expr(partition.map(|value| { + partition::expr::col("tag_0") + .eq(datatypes::value::Value::String(value.into())) + .as_json_str() + .unwrap() + })); + for compaction in [false, true] { + let builder = ParquetReaderBuilder::new( + "prefilter_test".into(), + PathType::Bare, + fixture.file_handle.clone(), + store.clone(), + ) + .expected_metadata(Some(Arc::new(expected.clone()))) + .compaction(compaction); + let builder = if let Some(enabled) = prefilter { + builder + .predicate(Some(Predicate::new(vec![ + col("tag_1").eq(lit("x")), + col("field_0").gt(lit(1_u64)), + ]))) + .enable_predicate_prefilter(enabled) + } else { + builder + }; + let mut reader = builder.build().await.unwrap().unwrap(); + assert_eq!( + reader.context.reader_builder().prefilter_builder.is_some(), + prefilter == Some(true), + ); + let mut count = 0; + while let Some(batch) = reader.next_record_batch().await.unwrap() { + assert_eq!( + batch.num_columns(), + if compaction { + 5 + } else { + metadata.column_metadatas.len() + 3 + } + ); + if compaction { + assert!( + reader + .context + .read_format() + .projected_index_by_id(0) + .is_none() + ); + assert!( + reader + .context + .read_format() + .projected_index_by_id(1) + .is_none() + ); + } + count += batch.num_rows(); + } + assert_eq!( + count, + match partition { + None => + if prefilter.is_some() { + 4 + } else { + 6 + }, + Some("a") => + if prefilter.is_some() { + 3 + } else { + 5 + }, + Some("b") => 1, + _ => 0, + } + ); + } + } + } + async fn prefilter_test_builder( object_store: ObjectStore, predicate: Predicate, cache_strategy: CacheStrategy, + ) -> (RowGroupReaderBuilder, Arc) { + prefilter_test_builder_with( + object_store, + 3, + 2, + PageIndexPolicy::Skip, + None, + predicate, + cache_strategy, + ) + .await + } + + /// Builds a reader over one SST with `num_row_groups` row groups of + /// `rows_per_group` rows. `writer_props` lets a test write several pages per row + /// group. + async fn prefilter_test_builder_with( + object_store: ObjectStore, + rows_per_group: i64, + num_row_groups: i64, + page_index_policy: PageIndexPolicy, + writer_props: Option, + predicate: Predicate, + cache_strategy: CacheStrategy, ) -> (RowGroupReaderBuilder, Arc) { let metadata = Arc::new( crate::test_util::sst_util::sst_region_metadata_with_encoding( @@ -2843,27 +3130,26 @@ mod tests { ) .unwrap() }; - let first_batch = batch(0, 3); - let second_batch = batch(3, 6); + let first_batch = batch(0, rows_per_group); let mut bytes = Vec::new(); - let mut writer = ArrowWriter::try_new(&mut bytes, first_batch.schema(), None).unwrap(); - writer.write(&first_batch).unwrap(); - writer.flush().unwrap(); - writer.write(&second_batch).unwrap(); + let mut writer = + ArrowWriter::try_new(&mut bytes, first_batch.schema(), writer_props).unwrap(); + for row_group in 0..num_row_groups { + let start = row_group * rows_per_group; + writer.write(&batch(start, start + rows_per_group)).unwrap(); + writer.flush().unwrap(); + } writer.close().unwrap(); - let file_handle = sst_file_handle(0, 6); + let file_handle = sst_file_handle(0, num_row_groups * rows_per_group); let file_path = file_handle.file_path("prefilter_test", PathType::Bare); let file_size = bytes.len() as u64; object_store.write(&file_path, bytes).await.unwrap(); let mut cache_metrics = MetadataCacheMetrics::default(); - let parquet_meta = Arc::new( - MetadataLoader::new(object_store.clone(), &file_path, file_size) - .load(&mut cache_metrics) - .await - .unwrap(), - ); + let mut loader = MetadataLoader::new(object_store.clone(), &file_path, file_size); + loader.with_page_index_policy(page_index_policy); + let parquet_meta = Arc::new(loader.load(&mut cache_metrics).await.unwrap()); let read_format = FlatReadFormat::new( metadata.clone(), ReadColumns::new( @@ -2926,6 +3212,9 @@ mod tests { ) } + /// A row group whose filters are already proven by its statistics keeps the incoming + /// selection, and the prefilter pass requests no pages or ranges for it: every fetch + /// counter stays at zero. #[tokio::test(flavor = "current_thread")] async fn test_execute_prefilter_proven_filters_preserve_selection_without_fetching() { let object_store = ObjectStore::new(Memory::default()).unwrap(); @@ -2965,6 +3254,8 @@ mod tests { assert_eq!(result.refined_selection, expected); assert_eq!(result.filtered_rows, 0); let metrics = fetch_metrics.data.lock().unwrap(); + // `pages_to_fetch_*` count the page ranges requested per source, so these + // are store, mem-cache and write-cache range counts rather than pages. assert_eq!(metrics.pages_to_fetch_store, 0); assert_eq!(metrics.pages_to_fetch_mem, 0); assert_eq!(metrics.pages_to_fetch_write_cache, 0); @@ -3013,6 +3304,8 @@ mod tests { RowSelection::from(vec![RowSelector::select(1), RowSelector::skip(2),]) ); if pass == 1 { + // The cached pass adds no store range count: the second pass reuses the + // cached masks instead of reading the row group again. assert_eq!(fetch_metrics.data.lock().unwrap().pages_to_fetch_store, 0); } } @@ -3047,6 +3340,495 @@ mod tests { ); } + /// Builds a dynamic filter over `field_0` that a `HashJoin` starts as `true` and + /// republishes with the membership of its build side. + fn new_test_dynamic_filter() -> Arc { + let key = Arc::new(Column::new("field_0", 0)) as Arc; + Arc::new(DynamicFilterPhysicalExpr::new( + vec![key], + physical_lit(true), + )) + } + + fn publish_membership( + handle: &Arc, + producer_schema: &SchemaRef, + members: Vec, + ) { + let expr = col("field_0").in_list(members.into_iter().map(lit).collect(), false); + let published = Predicate::to_physical_expr(&expr, producer_schema).unwrap(); + handle.update(published).unwrap(); + } + + async fn execute_test_prefilter( + reader_builder: &RowGroupReaderBuilder, + row_group_idx: usize, + ) -> crate::sst::parquet::prefilter::PrefilterResult { + execute_test_prefilter_with_metrics(reader_builder, row_group_idx) + .await + .0 + } + + /// Runs one prefilter pass over a whole row group and keeps the fetch metrics of that + /// pass, so a test can tell a pass that read the row group from one served by the + /// prefilter-result cache. + async fn execute_test_prefilter_with_metrics( + reader_builder: &RowGroupReaderBuilder, + row_group_idx: usize, + ) -> ( + crate::sst::parquet::prefilter::PrefilterResult, + ParquetFetchMetrics, + ) { + let fetch_metrics = ParquetFetchMetrics::default(); + let prefilter_builder = reader_builder.prefilter_builder.as_ref().unwrap(); + let mut prefilter_ctx = prefilter_builder.build(row_group_idx); + let result = execute_prefilter( + &mut prefilter_ctx, + reader_builder, + &RowGroupBuildContext { + row_group_idx, + row_selection: None, + fetch_metrics: Some(&fetch_metrics), + }, + ) + .await + .unwrap(); + (result, fetch_metrics) + } + + /// A dynamic filter starts as `true` and is published after the plan and the reader + /// are constructed. Each row group snapshots the current generation when its prefilter + /// starts, so a later publication still prunes. + #[tokio::test(flavor = "current_thread")] + async fn test_execute_prefilter_dynamic_filter_published_after_reader_construction() { + let producer_schema = dynamic_producer_schema(); + let handle = new_test_dynamic_filter(); + let (reader_builder, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + 3, + 3, + PageIndexPolicy::Skip, + None, + Predicate::with_dyn_filters(vec![], vec![handle.clone()]), + CacheStrategy::Disabled, + ) + .await; + + // Nothing is published yet: the placeholder keeps every row. + let result = execute_test_prefilter(&reader_builder, 0).await; + assert_eq!(3, result.refined_selection.row_count()); + assert_eq!(0, result.filtered_rows); + + // The build side finishes and the producer publishes its membership. + publish_membership(&handle, &producer_schema, vec![1]); + let result = execute_test_prefilter(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![ + RowSelector::skip(1), + RowSelector::select(1), + RowSelector::skip(1), + ]), + result.refined_selection + ); + assert_eq!(2, result.filtered_rows); + + // The mask is never cached across generations: the next publication is evaluated + // instead of replaying the previous result. + publish_membership(&handle, &producer_schema, vec![2]); + let result = execute_test_prefilter(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![RowSelector::skip(2), RowSelector::select(1)]), + result.refined_selection + ); + assert_eq!(2, result.filtered_rows); + } + + /// A producer may republish while a row group's prefilter runs. Nothing proves the + /// new expression keeps the rows the stale snapshot kept, so the mask is discarded and + /// the incoming selection is kept: the upper filter stays authoritative. + #[tokio::test(flavor = "current_thread")] + async fn test_execute_prefilter_discards_dynamic_mask_after_republish() { + let producer_schema = dynamic_producer_schema(); + let handle = new_test_dynamic_filter(); + publish_membership(&handle, &producer_schema, vec![1]); + let (reader_builder, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + 3, + 3, + PageIndexPolicy::Skip, + None, + Predicate::with_dyn_filters(vec![], vec![handle.clone()]), + CacheStrategy::Disabled, + ) + .await; + + // The row group's prefilter starts and snapshots the current generation. + let prefilter_builder = reader_builder.prefilter_builder.as_ref().unwrap(); + let mut prefilter_ctx = prefilter_builder.build(0); + // The producer publishes a new expression while the pass is in flight. + publish_membership(&handle, &producer_schema, vec![2]); + let incoming = RowSelection::from(vec![RowSelector::select(2), RowSelector::skip(1)]); + let result = execute_prefilter( + &mut prefilter_ctx, + &reader_builder, + &RowGroupBuildContext { + row_group_idx: 0, + row_selection: Some(incoming.clone()), + fetch_metrics: None, + }, + ) + .await + .unwrap(); + + assert_eq!(incoming, result.refined_selection); + assert_eq!(0, result.filtered_rows); + } + + /// With the prefilter-result cache enabled, a static predicate keeps its cached mask + /// while a producer republishes a dynamic filter. + /// + /// The static `ts` predicate is cacheable, so the row selection it produces is written + /// to the cache and reused. The dynamic membership is not: each pass evaluates the + /// generation it snapshotted, a newer publication replaces the previous membership + /// instead of replaying a cached mask, and disabling the filter leaves no stale mask + /// behind. Only the static mask is reused, and the rows the pass keeps stay the rows + /// the static predicate and the current membership keep. + #[tokio::test(flavor = "current_thread")] + async fn test_prefilter_result_cache_keeps_static_mask_but_not_dynamic_mask() { + let producer_schema = dynamic_producer_schema(); + let handle = new_test_dynamic_filter(); + let cache = CacheStrategy::EnableAll(Arc::new( + CacheManager::builder() + .prefilter_result_cache_size(1024) + .build(), + )); + // `ts >= 1` keeps the last two of the three rows, and the row group's time index + // statistics cannot prove it, so it stays a cacheable prefilter entry. The dynamic + // filter reads another column, so the cached static mask is the only way a later + // pass can skip reading `ts`. + let static_predicate = + col("ts").gt_eq(lit(ScalarValue::TimestampMillisecond(Some(1), None))); + let predicate = Predicate::with_dyn_filters(vec![static_predicate], vec![handle.clone()]); + let (reader_builder, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + 3, + 1, + PageIndexPolicy::Skip, + None, + predicate.clone(), + cache, + ) + .await; + + // Pass 1: membership A keeps row 1. This pass writes the static mask to the cache. + publish_membership(&handle, &producer_schema, vec![1]); + let (result, metrics) = execute_test_prefilter_with_metrics(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![ + RowSelector::skip(1), + RowSelector::select(1), + RowSelector::skip(1), + ]), + result.refined_selection, + "the pass keeps the row membership A keeps" + ); + assert_eq!(2, result.filtered_rows); + assert!( + metrics.data.lock().unwrap().pages_to_fetch_store > 0, + "the first pass has to read the row group" + ); + + // Pass 2: publishing membership B keeps row 2. A pass that replayed a cached mask + // would still keep row 1. + publish_membership(&handle, &producer_schema, vec![2]); + let (result, _) = execute_test_prefilter_with_metrics(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![RowSelector::skip(2), RowSelector::select(1)]), + result.refined_selection, + "the new membership must apply instead of the cached static-only selection" + ); + assert_eq!(2, result.filtered_rows); + + // Pass 3: a wider membership keeps every row, so only the cached static mask + // prunes. A stale mask of A or B would still drop row 1 or row 2. + publish_membership(&handle, &producer_schema, vec![0, 1, 2]); + let (result, metrics) = execute_test_prefilter_with_metrics(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![RowSelector::skip(1), RowSelector::select(2)]), + result.refined_selection, + "no mask of a superseded membership may survive" + ); + assert_eq!(1, result.filtered_rows); + assert!( + metrics.data.lock().unwrap().pages_to_fetch_store > 0, + "the dynamic filter still reads its column" + ); + + // Pass 4: the producer republishes its `true` placeholder, which disables the + // dynamic filter for this row group. The pass then prunes nothing dynamically and + // reads no page at all: the static mask comes from the cache. + handle.update(physical_lit(true)).unwrap(); + let (result, metrics) = execute_test_prefilter_with_metrics(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![RowSelector::skip(1), RowSelector::select(2)]), + result.refined_selection, + "a disabled dynamic filter keeps the rows the static predicate keeps" + ); + assert_eq!(1, result.filtered_rows); + assert_eq!( + 0, + metrics.data.lock().unwrap().pages_to_fetch_store, + "the static mask must come from the cache" + ); + + // The rows those selections promise are the rows the projected reader returns. + let (batches, _) = + read_test_row_group(&reader_builder, 0, Some(result.refined_selection)).await; + assert_eq!(vec![1, 2], test_field_values(&batches)); + + // Control: the same predicate (the shared handle still holds the `true` placeholder) + // without a cache reads the row group and keeps the same rows, so the pass above + // was a cache hit rather than a skipped read. + let (control, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + 3, + 1, + PageIndexPolicy::Skip, + None, + predicate, + CacheStrategy::Disabled, + ) + .await; + let (result, metrics) = execute_test_prefilter_with_metrics(&control, 0).await; + assert_eq!( + RowSelection::from(vec![RowSelector::skip(1), RowSelector::select(2)]), + result.refined_selection + ); + assert!( + metrics.data.lock().unwrap().pages_to_fetch_store > 0, + "the uncached control reads the row group" + ); + } + + /// The prefilter of the expression a real `HashJoin` publishes keeps exactly the rows + /// the live `batch_filter` above the same scan keeps. + /// + /// The published shape is what a partitioned join builds for a large build side: a + /// hash-table lookup over the probe key per build partition, routed by + /// `hash_repartition(key) % 2` inside a `CASE` — not a logical `IN` list. The + /// prefilter evaluates it on the reduced projection; the filter above the scan + /// evaluates the same expression on the projected read. Both must select the same + /// rows of the same row group. + #[tokio::test(flavor = "current_thread")] + async fn test_execute_prefilter_hash_lookup_matches_upper_filter() { + // The build side holds 1, 3 and 5; rows 0, 2 and 4 are not members. + let build_keys = [1_u64, 3, 5]; + let handle = Arc::new(DynamicFilterPhysicalExpr::new( + vec![Arc::new(Column::new("field_0", 3)) as Arc], + physical_lit(true), + )); + handle + .update(partitioned_hash_join_lookup( + &dynamic_producer_schema(), + &build_keys, + )) + .unwrap(); + let published = handle.current().unwrap(); + assert!( + published.is::(), + "a partitioned join routes its lookups: {published}" + ); + assert!( + published.to_string().contains("hash_lookup") + && published.to_string().contains("hash_repartition"), + "the published shape must be the hash lookup a producer builds: {published}" + ); + + let (reader_builder, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + 6, + 1, + PageIndexPolicy::Skip, + None, + Predicate::with_dyn_filters(vec![], vec![handle.clone()]), + CacheStrategy::Disabled, + ) + .await; + + let result = execute_test_prefilter(&reader_builder, 0).await; + assert_eq!( + 3, result.filtered_rows, + "the lookup must prune the rows outside the build side" + ); + + let (all_batches, _) = read_test_row_group(&reader_builder, 0, None).await; + let (selected_batches, _) = + read_test_row_group(&reader_builder, 0, Some(result.refined_selection)).await; + + // The upper filter evaluates the same published expression, but on the wide + // projected batch it guards instead of the single prefilter column. + let schema = all_batches[0].schema(); + let upper_expr = reassign_expr_columns(Arc::clone(&published), schema.as_ref()).unwrap(); + let upper_batches = all_batches + .iter() + .map(|batch| batch_filter(batch, &upper_expr).unwrap()) + .collect::>(); + let upper_values = test_field_values(&upper_batches); + assert_eq!( + build_keys.to_vec(), + upper_values, + "the upper filter keeps exactly the build-side members" + ); + assert_eq!( + upper_values, + test_field_values(&selected_batches), + "the prefilter must keep exactly the rows the upper filter keeps" + ); + } + + /// A multi-page row group returns the same rows for the rows a dynamic prefilter + /// selects, and the projected reader fetches fewer bytes for them. + /// + /// The metrics below belong to the projected read only: `page_size_to_fetch_store` is + /// the number of bytes of the projected columns that read covers, and + /// `pages_to_fetch_store` the number of store ranges it requests. The saving is a + /// projection-byte saving, not a net scan saving: this row group's filter column was + /// already read by the prefilter pass, and store range count is not part of the claim — + /// a partial selection can split a column into more ranges than a full read. The row + /// counts asserted here are the rows the projected reader returns, not the rows of the + /// filter column the prefilter decoded. + /// + /// The row group is large enough that the pages a partial selection skips span more + /// than the store's fetch merge gap; below that the fetched byte ranges merge back + /// into the whole row group whatever the selection is. + #[tokio::test(flavor = "current_thread")] + async fn test_execute_prefilter_dynamic_selection_saves_projected_store_bytes() { + const ROWS: i64 = 100_000; + const PAGE_ROWS: usize = 20_000; + + let props = WriterProperties::builder() + .set_data_page_row_count_limit(PAGE_ROWS) + .set_write_batch_size(8192) + .build(); + let producer_schema = dynamic_producer_schema(); + let handle = new_test_dynamic_filter(); + // Only the first row of the row group survives the producer's membership. + publish_membership(&handle, &producer_schema, vec![0]); + let (reader_builder, _) = prefilter_test_builder_with( + ObjectStore::new(Memory::default()).unwrap(), + ROWS, + 1, + // Translating a row selection into byte ranges needs the offset index. + PageIndexPolicy::Optional, + Some(props), + Predicate::with_dyn_filters(vec![], vec![handle.clone()]), + CacheStrategy::Disabled, + ) + .await; + assert_eq!( + ROWS as usize, + reader_builder.parquet_meta.row_group(0).num_rows() as usize + ); + assert!( + reader_builder.parquet_meta.offset_index().is_some(), + "the row selection can only skip pages with the offset index" + ); + assert!( + reader_builder + .parquet_meta + .row_group(0) + .columns() + .iter() + .any(|column| column.num_values() as usize > PAGE_ROWS), + "the row group must span several pages" + ); + + let result = execute_test_prefilter(&reader_builder, 0).await; + assert_eq!( + RowSelection::from(vec![ + RowSelector::select(1), + RowSelector::skip(ROWS as usize - 1), + ]), + result.refined_selection + ); + + let (all_batches, all_metrics) = read_test_row_group(&reader_builder, 0, None).await; + let (selected_batches, selected_metrics) = + read_test_row_group(&reader_builder, 0, Some(result.refined_selection)).await; + + // The selected read returns exactly the rows of the full read that the producer + // keeps. + let all_values = test_field_values(&all_batches); + assert_eq!( + (0..ROWS).map(|value| value as u64).collect::>(), + all_values + ); + assert_eq!( + vec![0], + test_field_values(&selected_batches), + "the row selection must not change the rows the projected reader returns" + ); + + let all = all_metrics.data.lock().unwrap().clone(); + let selected = selected_metrics.data.lock().unwrap().clone(); + assert!( + all.page_size_to_fetch_store > 0, + "the control read must fetch projected bytes from the store" + ); + // Store range count (`pages_to_fetch_store`) is deliberately not asserted: a + // partial selection can request more ranges than a full read, because one range + // per column can become one range per page. + // + // Projection-byte savings: the projected read covers fewer bytes, because the + // selection drops the pages of the rows the producer rejects. + assert!( + selected.page_size_to_fetch_store < all.page_size_to_fetch_store, + "selected {} bytes vs all {} bytes", + selected.page_size_to_fetch_store, + all.page_size_to_fetch_store + ); + } + + /// Reads one row group of the projected columns through the projected reader and + /// returns its batches with the fetch metrics of exactly that read. + async fn read_test_row_group( + reader_builder: &RowGroupReaderBuilder, + row_group_idx: usize, + row_selection: Option, + ) -> (Vec, ParquetFetchMetrics) { + let fetch_metrics = ParquetFetchMetrics::default(); + let mut stream = reader_builder + .build_with_projection( + row_group_idx, + row_selection, + reader_builder.projection.mask.clone(), + Some(&fetch_metrics), + ) + .await + .unwrap(); + let mut batches = Vec::new(); + while let Some(batch) = stream.next().await { + batches.push(batch.unwrap()); + } + + (batches, fetch_metrics) + } + + fn test_field_values(batches: &[RecordBatch]) -> Vec { + batches + .iter() + .flat_map(|batch| { + batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values() + .to_vec() + }) + .collect() + } + #[test] fn test_skip_prefilter_for_json_get() -> WhateverResult<()> { fn json_get_expr(base: Expr, path: &str) -> Expr { @@ -3623,6 +4405,306 @@ mod tests { assert!(PhysicalFilterContext::new_opt(&metadata, None, &read_format, &binary).is_none()); } + /// The producer's input schema, where `field_0` is its fourth column. + fn dynamic_producer_schema() -> SchemaRef { + Arc::new(Schema::new(vec![ + Field::new("tag_0", DataType::Utf8, true), + Field::new("tag_1", DataType::Utf8, true), + Field::new("ts", DataType::UInt64, true), + Field::new("field_0", DataType::UInt64, true), + ])) + } + + fn full_read_format(metadata: &RegionMetadataRef) -> FlatReadFormat { + FlatReadFormat::new( + metadata.clone(), + ReadColumns::new(metadata.column_metadatas.iter().map(|c| c.column_id)), + None, + "test", + true, + ) + .unwrap() + } + + /// Creates the dynamic filter a single-key `HashJoin` produces: the probe key is + /// declared as the filter's only child at the producer's column index, the + /// placeholder is `true`, and `published` is what the build side sends once known. + fn new_hash_join_dynamic_filter( + producer_schema: &SchemaRef, + published: &Expr, + ) -> Arc { + let key = Arc::new(Column::new("field_0", 3)) as Arc; + let handle = Arc::new(DynamicFilterPhysicalExpr::new( + vec![key], + physical_lit(true), + )); + let published = Predicate::to_physical_expr(published, producer_schema).unwrap(); + handle.update(published).unwrap(); + handle + } + + /// Returns a hash-table lookup over the probe key that contains `keys`, i.e. one + /// build partition of a `HashJoin` that picked the hash-map pushdown strategy. + fn hash_lookup_expr(keys: &[u64], seed: &SeededRandomState) -> Arc { + // The lookup compares hashes computed with the seed the producer built its map + // with, so the map is filled with the same `create_hashes` output. + let keys: ArrayRef = Arc::new(UInt64Array::from(keys.to_vec())); + let mut hashes = vec![0_u64; keys.len()]; + create_hashes([keys], seed.random_state(), &mut hashes).unwrap(); + let mut map = JoinHashMapU32::with_capacity(hashes.len().max(1)); + map.update_from_iter(Box::new(hashes.iter().enumerate()), 0); + + Arc::new(HashTableLookupExpr::new( + vec![Arc::new(Column::new("field_0", 3)) as Arc], + seed.clone(), + Arc::new(Map::HashMap(Box::new(map))), + "hash_lookup".to_string(), + None, + )) + } + + /// The dynamic filter a partitioned `HashJoin` publishes once every build partition + /// reported: one hash-table lookup per build partition, selected by + /// `hash_repartition(key) % partitions` in a `CASE`. + /// + /// `build_keys` is the whole build side. Each key is routed like the join routes a + /// probe row, so the branch that owns a key is the branch whose map contains it. + fn partitioned_hash_join_lookup( + producer_schema: &SchemaRef, + build_keys: &[u64], + ) -> Arc { + const PARTITIONS: usize = 2; + let seed = SeededRandomState::with_seed(0); + let probe_key = || Arc::new(Column::new("field_0", 3)) as Arc; + let route = Arc::new(BinaryExpr::new( + Arc::new(HashExpr::new( + vec![probe_key()], + REPARTITION_RANDOM_STATE, + "hash_repartition".to_string(), + )), + Operator::Modulo, + physical_lit(ScalarValue::UInt64(Some(PARTITIONS as u64))), + )) as Arc; + + // Route the build keys exactly like probe rows, with the producer schema so the + // routing expression reads `field_0` at the producer's column index. + let build_batch = RecordBatch::try_new( + Arc::clone(producer_schema), + vec![ + Arc::new(StringArray::from(vec![""; build_keys.len()])) as ArrayRef, + Arc::new(StringArray::from(vec![""; build_keys.len()])) as ArrayRef, + Arc::new(UInt64Array::from(vec![0_u64; build_keys.len()])) as ArrayRef, + Arc::new(UInt64Array::from(build_keys.to_vec())) as ArrayRef, + ], + ) + .unwrap(); + let routes = route + .evaluate(&build_batch) + .unwrap() + .into_array(build_keys.len()) + .unwrap(); + let routes = routes.as_any().downcast_ref::().unwrap(); + let mut partition_keys = vec![Vec::new(); PARTITIONS]; + for (key, partition) in build_keys.iter().zip(routes.values()) { + partition_keys[*partition as usize].push(*key); + } + + // The last partition is the `ELSE` branch, as the producer emits it. + let when_then_expr = partition_keys[..PARTITIONS - 1] + .iter() + .enumerate() + .map(|(partition, keys)| { + ( + physical_lit(ScalarValue::UInt64(Some(partition as u64))), + hash_lookup_expr(keys, &seed), + ) + }) + .collect::>(); + let else_expr = hash_lookup_expr(&partition_keys[PARTITIONS - 1], &seed); + + Arc::new(CaseExpr::try_new(Some(route), when_then_expr, Some(else_expr)).unwrap()) + } + + #[test] + fn test_dynamic_filter_context_rebinds_published_column_to_prefilter_schema() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = full_read_format(&metadata); + let handle = new_hash_join_dynamic_filter( + &dynamic_producer_schema(), + &col("field_0").in_list(vec![lit(1_u64), lit(2_u64)], false), + ); + + let ctx = + PhysicalFilterContext::new_dynamic(&metadata, None, &read_format, &handle).unwrap(); + assert!(ctx.is_dynamic()); + assert!( + !ctx.is_immutable(), + "a producer-owned mask must never enter the result cache" + ); + assert!(ctx.filter().is_none()); + assert_eq!("field_0", ctx.column_name()); + + let snapshot = ctx.capture_dynamic_snapshot().unwrap(); + assert!(ctx.dynamic_snapshot_is_current(&snapshot)); + // The producer published `field_0` as its fourth column; the prefilter batch + // carries that column alone, so the snapshot must use index 0. + let columns = collect_columns(&snapshot.expr); + assert_eq!(1, columns.len()); + assert_eq!(0, columns.iter().next().unwrap().index()); + + // `new_record_batch_with_custom_sequence` sets field_0 to 0, 1, 2, 3. + let batch = new_record_batch_with_custom_sequence(&["a", "x"], 0, 4, 1); + let mask = evaluate_physical_mask(&batch, &ctx, &snapshot.expr, "test").unwrap(); + assert_eq!( + vec![false, true, true, false], + mask.iter().collect::>() + ); + } + + /// The published expression is evaluated whole: a producer that routes NULL keys + /// itself keeps them, so the prefilter must not narrow the expression to its `IN` + /// leaf. + #[test] + fn test_dynamic_filter_prefilter_keeps_producer_null_semantics() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = full_read_format(&metadata); + let producer_schema = dynamic_producer_schema(); + let key = || col("field_0"); + // Duplicate members are the producer's list verbatim. + let members = || vec![lit(1_u64), lit(1_u64), lit(3_u64)]; + + // Both shapes a single-key `HashJoin` publishes for a nullable probe key: the + // membership filter ORed with the producer-owned null guard, and the same choice + // routed through `CASE`. + let guarded = key().is_null().or(key().in_list(members(), false)); + let case_routed = Expr::Case(datafusion_expr::expr::Case::new( + None, + vec![(Box::new(key().is_null()), Box::new(lit(true)))], + Some(Box::new(key().in_list(members(), false))), + )); + + let batch = RecordBatch::try_new( + Arc::new(Schema::new(vec![Field::new( + "field_0", + DataType::UInt64, + true, + )])), + vec![Arc::new(UInt64Array::from(vec![Some(1), None, Some(2), Some(3)])) as ArrayRef], + ) + .unwrap(); + + for published in [guarded, case_routed] { + let handle = new_hash_join_dynamic_filter(&producer_schema, &published); + let ctx = + PhysicalFilterContext::new_dynamic(&metadata, None, &read_format, &handle).unwrap(); + let snapshot = ctx.capture_dynamic_snapshot().unwrap(); + let mask = evaluate_physical_mask(&batch, &ctx, &snapshot.expr, "test").unwrap(); + // The NULL key and both members survive, only `2` is dropped. Evaluating the + // bare `IN` leaf would have dropped the NULL row. + assert_eq!( + vec![true, true, false, true], + mask.iter().collect::>(), + "{published}" + ); + } + } + + #[test] + fn test_dynamic_filter_context_keeps_all_without_a_single_filter_column() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = full_read_format(&metadata); + let producer_schema = dynamic_producer_schema(); + + // A two-key producer declares both keys, so the filter cannot be evaluated on a + // single-column prefilter batch. + let multi_key = Arc::new(DynamicFilterPhysicalExpr::new( + vec![ + Arc::new(Column::new("field_0", 3)) as Arc, + Arc::new(Column::new("ts", 2)) as Arc, + ], + physical_lit(true), + )); + assert!( + PhysicalFilterContext::new_dynamic(&metadata, None, &read_format, &multi_key).is_none() + ); + + let published = col("field_0").in_list(vec![lit(1_u64)], false); + let handle = new_hash_join_dynamic_filter(&producer_schema, &published); + + // The expected metadata points the filter column at an id the SST does not have. + let mut missing = (*metadata).clone(); + for column in missing.column_metadatas.iter_mut() { + if column.column_schema.name == "field_0" { + column.column_id = 20; + } + } + let missing = Arc::new(missing); + assert!( + PhysicalFilterContext::new_dynamic( + &metadata, + Some(missing.as_ref()), + &read_format, + &handle + ) + .is_none() + ); + + // A producer that publishes a wider expression after admission keeps every row: + // the snapshot is rejected instead of evaluated against the wrong columns. + let ctx = + PhysicalFilterContext::new_dynamic(&metadata, None, &read_format, &handle).unwrap(); + let widened = + Predicate::to_physical_expr(&published.and(col("ts").gt(lit(0_u64))), &producer_schema) + .unwrap(); + handle.update(widened).unwrap(); + assert!(ctx.capture_dynamic_snapshot().is_none()); + } + + #[test] + fn test_dynamic_filter_context_skips_renamed_and_type_changed_columns() { + let metadata: RegionMetadataRef = Arc::new(sst_region_metadata()); + let read_format = full_read_format(&metadata); + let producer_schema = dynamic_producer_schema(); + let handle = new_hash_join_dynamic_filter( + &producer_schema, + &col("field_0").in_list(vec![lit(1_u64)], false), + ); + + // The SST keeps the pre-rename name for the filter column's id. + let mut sst_metadata = (*metadata).clone(); + for column in sst_metadata.column_metadatas.iter_mut() { + if column.column_schema.name == "field_0" { + column.column_schema.name = "field_old".to_string(); + } + } + assert!( + PhysicalFilterContext::validated_column( + &Arc::new(sst_metadata), + Some(metadata.as_ref()), + "field_0" + ) + .is_none() + ); + + // An altered column type between the file and the expected metadata. + let mut type_changed = (*metadata).clone(); + for column in type_changed.column_metadatas.iter_mut() { + if column.column_schema.name == "field_0" { + column.column_schema.data_type = ConcreteDataType::int64_datatype(); + } + } + let type_changed = Arc::new(type_changed); + assert!( + PhysicalFilterContext::new_dynamic( + &metadata, + Some(type_changed.as_ref()), + &read_format, + &handle + ) + .is_none() + ); + } + fn write_test_parquet_with_pk_column(values: &[&[u8]]) -> bytes::Bytes { use datatypes::arrow::array::{ BinaryArray, TimestampMillisecondArray, UInt8Array, UInt64Array, diff --git a/src/mito2/src/sst/parquet/writer.rs b/src/mito2/src/sst/parquet/writer.rs index 3b4c02b3080..cec4af689b5 100644 --- a/src/mito2/src/sst/parquet/writer.rs +++ b/src/mito2/src/sst/parquet/writer.rs @@ -15,15 +15,15 @@ //! Parquet writer. use std::collections::HashMap; +use std::error::Error as StdError; use std::future::Future; use std::mem; -use std::pin::Pin; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; -use std::task::{Context, Poll}; use std::time::Instant; use bytes::Bytes; +use common_base::cancellation::Cancelled; use common_telemetry::debug; use common_time::Timestamp; use datatypes::arrow::array::{ @@ -34,23 +34,24 @@ use datatypes::arrow::compute::{max, min}; use datatypes::arrow::datatypes::{DataType, SchemaRef, TimeUnit}; use datatypes::arrow::record_batch::RecordBatch; use datatypes::extension::json::is_json2_extension_type; -use object_store::{FuturesAsyncWriter, ObjectStore}; +use futures::future::BoxFuture; +use object_store::{ObjectStore, Writer}; use parquet::arrow::AsyncArrowWriter; use parquet::arrow::async_writer::AsyncFileWriter; use parquet::basic::{Compression, Encoding, ZstdLevel}; +use parquet::errors::ParquetError; use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData}; use parquet::file::properties::WriterProperties; use smallvec::smallvec; -use snafu::{OptionExt, ResultExt, ensure}; +use snafu::{IntoError, OptionExt, ResultExt, ensure}; use store_api::metadata::RegionMetadataRef; use store_api::storage::{FileId, SequenceNumber}; -use tokio::io::AsyncWrite; -use tokio_util::compat::{Compat, FuturesAsyncWriteCompatExt}; +use tokio_util::sync::CancellationToken; use crate::access_layer::{FilePathProvider, Metrics, SstInfoArray, TempFileCleaner}; use crate::config::{IndexBuildMode, IndexConfig}; use crate::error::{ - InvalidMetadataSnafu, InvalidRecordBatchSnafu, OpenDalSnafu, Result, UnexpectedSnafu, + Error, InvalidMetadataSnafu, InvalidRecordBatchSnafu, OpenDalSnafu, Result, UnexpectedSnafu, WriteParquetSnafu, }; use crate::read::FlatSource; @@ -65,6 +66,7 @@ use crate::sst::parquet::{ COLUMN_INDEX_TRUNCATE_LENGTH, PARQUET_METADATA_KEY, SstInfo, WriteOptions, apply_column_compression, apply_float_field_encoding, apply_internal_column_encoding, }; +use crate::sst::upload::{self, UploadError}; use crate::sst::{ DEFAULT_WRITE_CONCURRENCY, FlatSchemaOptions, SeriesEstimator, maybe_wrap_schema, }; @@ -179,16 +181,17 @@ impl SstArrowWriter { } } - async fn close(self) -> parquet::errors::Result { + async fn finish(&mut self) -> parquet::errors::Result { match self { - SstArrowWriter::Serial(writer) => writer.close().await, - SstArrowWriter::Parallel(writer) => writer.close().await, + SstArrowWriter::Serial(writer) => writer.finish().await, + SstArrowWriter::Parallel(writer) => writer.finish().await, } } - async fn abort(self) { - if let SstArrowWriter::Parallel(writer) = self { - writer.abort().await; + async fn abort(self) -> W { + match self { + SstArrowWriter::Serial(writer) => writer.into_inner(), + SstArrowWriter::Parallel(writer) => writer.abort().await, } } } @@ -248,28 +251,105 @@ pub struct ParquetWriter<'a, F: WriterFactory, I: IndexerBuilder, P: FilePathPro /// Write metrics metrics: &'a mut Metrics, write_guard: Option>, + cancellation: Option, memory: WriteMemory, } -/// Registers durable ownership before either an SST or its index is published to storage. +/// Registers output ownership before either an SST or its index is created in storage. #[async_trait::async_trait] pub trait SstWriteGuard: Send + Sync { async fn before_create(&self, file_id: RegionFileId) -> Result<()>; } -/// Protection and resource limits travel together for table-compaction output. +/// Optional ownership, cancellation, and resource limits for an SST write. +#[derive(Default)] pub(crate) struct SstWriteContext { - pub(crate) guard: Arc, - pub(crate) limits: SstWriteLimits, + /// Registers output ownership before creating files, including cached files. + pub(crate) guard: Option>, + /// Resource bounds enforced by the direct writer, with one upload at a time. + /// The write cache does not enforce these limits, so bounded writes bypass it. + pub(crate) limits: Option, + /// Signals cancellation; the writer still drains encoders and aborts unfinished output. + pub(crate) cancellation: Option, } pub trait WriterFactory { - type Writer: AsyncWrite + Send + Unpin; + type Writer: AsyncFileWriter; + + /// Opens output with the same cancellation signal as its encoders. fn create( &mut self, file_path: &str, write_buffer_size: usize, + cancellation: Option<&CancellationToken>, ) -> impl Future>; + + /// Aborts unfinished output after all encoders have stopped using it. + fn abort(writer: Self::Writer) -> impl Future>; +} + +/// Bridges an OpenDAL [`Writer`] with Parquet's [`AsyncFileWriter`] and tracks +/// the number of bytes successfully submitted to the object store. +pub struct ObjectStoreWriter { + inner: Writer, + output_bytes: u64, + write_buffer_size: usize, + cancellation: Option, +} + +impl ObjectStoreWriter { + pub(crate) fn new( + inner: Writer, + write_buffer_size: usize, + cancellation: Option, + ) -> Self { + Self { + inner, + output_bytes: 0, + write_buffer_size, + cancellation, + } + } + + pub(crate) fn output_bytes(&self) -> u64 { + self.output_bytes + } + + pub(crate) fn into_inner(self) -> Writer { + self.inner + } +} + +impl AsyncFileWriter for ObjectStoreWriter { + fn write(&mut self, bytes: Bytes) -> BoxFuture<'_, parquet::errors::Result<()>> { + Box::pin(async move { + let written = upload::write( + &mut self.inner, + bytes.into(), + self.write_buffer_size, + self.cancellation.as_ref(), + ) + .await + .map_err(|error| match error { + UploadError::Cancelled => ParquetError::External(Box::new(Cancelled)), + UploadError::Read { source } | UploadError::Write { source } => { + ParquetError::External(Box::new(source)) + } + })?; + self.output_bytes += written; + Ok(()) + }) + } + + fn complete(&mut self) -> BoxFuture<'_, parquet::errors::Result<()>> { + Box::pin(async move { + self.inner + .close() + .await + .map(|_| ()) + .map_err(|error| ParquetError::External(Box::new(error))) + }) + } } pub struct ObjectStoreWriterFactory { @@ -278,17 +358,26 @@ pub struct ObjectStoreWriterFactory { } impl WriterFactory for ObjectStoreWriterFactory { - type Writer = Compat; + type Writer = ObjectStoreWriter; - async fn create(&mut self, file_path: &str, write_buffer_size: usize) -> Result { + async fn create( + &mut self, + file_path: &str, + write_buffer_size: usize, + cancellation: Option<&CancellationToken>, + ) -> Result { self.object_store .writer_with(file_path) .chunk(write_buffer_size) .concurrent(self.concurrency) .await - .map(|v| v.into_futures_async_write().compat_write()) + .map(|writer| ObjectStoreWriter::new(writer, write_buffer_size, cancellation.cloned())) .context(OpenDalSnafu) } + + async fn abort(writer: Self::Writer) -> Result<()> { + writer.into_inner().abort().await.context(OpenDalSnafu) + } } impl<'a, I, P> ParquetWriter<'a, ObjectStoreWriterFactory, I, P> @@ -361,9 +450,26 @@ where file_cleaner: None, metrics, write_guard: None, + cancellation: None, } } + pub(crate) fn with_cancellation(mut self, cancellation: Option) -> Self { + self.cancellation = cancellation; + self + } + + fn check_cancelled(&self) -> Result<()> { + ensure!( + !self + .cancellation + .as_ref() + .is_some_and(|handle| handle.is_cancelled()), + crate::error::SstWriteCancelledSnafu + ); + Ok(()) + } + pub(crate) fn with_write_guard(mut self, guard: Option>) -> Self { self.write_guard = guard; self @@ -409,8 +515,9 @@ where ssts: &mut SstInfoArray, stats: &mut SourceStats, ) -> Result<()> { + self.check_cancelled()?; // maybe_init_writer will re-create a new file. - if let Some(mut current_writer) = mem::take(&mut self.writer) { + if self.writer.is_some() { let mut stats = mem::take(stats); // At least one row has been written. assert!(stats.num_rows > 0); @@ -436,9 +543,28 @@ where ); } } - current_writer.flush().await.context(WriteParquetSnafu)?; - - let parquet_metadata = current_writer.close().await.context(WriteParquetSnafu)?; + self.check_cancelled()?; + self.writer + .as_mut() + .context(UnexpectedSnafu { + reason: "Missing SST writer during finalization", + })? + .flush() + .await + .map_err(map_write_error)?; + self.check_cancelled()?; + // Keep the upload owned until finalization succeeds, so an error + // still reaches the common abort path. + let parquet_metadata = self + .writer + .as_mut() + .context(UnexpectedSnafu { + reason: "Missing SST writer during finalization", + })? + .finish() + .await + .map_err(map_write_error)?; + self.writer.take(); let file_size = self.bytes_written.load(Ordering::Relaxed) as u64; if self.memory.limits.is_some() { self.memory.completed = self.memory.completed.saturating_add( @@ -517,6 +643,7 @@ where ); let res = self.write_all_flat_inner(source, &converter, opts).await; if res.is_err() { + self.abort_current_indexer().await; self.abort_current_file().await; } res @@ -542,6 +669,7 @@ where }; let res = self.write_all_flat_inner(source, &converter, opts).await; if res.is_err() { + self.abort_current_indexer().await; self.abort_current_file().await; } res @@ -558,7 +686,17 @@ where loop { let start = Instant::now(); - let batch = match source.next_batch().await { + self.check_cancelled()?; + let next = if let Some(cancellation) = &self.cancellation { + tokio::select! { + biased; + _ = cancellation.cancelled() => return crate::error::SstWriteCancelledSnafu.fail(), + next = source.next_batch() => next, + } + } else { + source.next_batch().await + }; + let batch = match next { Ok(Some(batch)) => batch, Ok(None) => break, Err(e) => { @@ -622,15 +760,18 @@ where stats: &mut SourceStats, ) -> Result<()> { let result = async { + self.check_cancelled()?; let arrow_batch = converter.convert_batch(batch)?; + self.check_cancelled()?; let start = Instant::now(); self.maybe_init_writer(arrow_batch.schema_ref(), opts) .await? .write(&arrow_batch) .await - .context(WriteParquetSnafu)?; + .map_err(map_write_error)?; self.metrics.write_batch += start.elapsed(); + self.check_cancelled()?; stats.update_flat(batch)?; if matches!(self.index_config.build_mode, IndexBuildMode::Sync) { let start = Instant::now(); @@ -664,7 +805,10 @@ where /// Joins encoders before dropping the writer and deleting its failed output. async fn abort_current_file(&mut self) { if let Some(writer) = self.writer.take() { - writer.abort().await; + let output = writer.abort().await; + if let Err(error) = F::abort(output.inner).await { + common_telemetry::warn!(error; "Failed to abort SST writer, file: {}", self.current_file); + } } if let Some(cleaner) = &self.file_cleaner { cleaner.clean_by_file_id(self.current_file).await; @@ -672,7 +816,7 @@ where } async fn abort_current_indexer(&mut self) { - if let Some(indexer) = &mut self.current_indexer { + if let Some(mut indexer) = self.current_indexer.take() { indexer.abort().await; } } @@ -682,6 +826,7 @@ where schema: &SchemaRef, opts: &WriteOptions, ) -> Result<&mut SstArrowWriter>> { + self.check_cancelled()?; if let Some(ref mut w) = self.writer { Ok(w) } else { @@ -719,7 +864,11 @@ where } let writer = SizeAwareWriter::new( self.writer_factory - .create(&sst_file_path, opts.write_buffer_size.as_bytes() as usize) + .create( + &sst_file_path, + opts.write_buffer_size.as_bytes() as usize, + self.cancellation.as_ref(), + ) .await?, self.bytes_written.clone(), ); @@ -736,12 +885,13 @@ where opts.encode_parallelism, opts.encode_runtime, ) - .context(WriteParquetSnafu)?, + .map_err(map_write_error)? + .with_cancellation(self.cancellation.clone()), ) } else { SstArrowWriter::Serial( AsyncArrowWriter::try_new(writer, schema, Some(writer_props)) - .context(WriteParquetSnafu)?, + .map_err(map_write_error)?, ) }; self.writer = Some(arrow_writer); @@ -762,6 +912,19 @@ where } } +/// Keeps cancellation inside the SST write error contract, including nested encoder errors. +fn map_write_error(error: ParquetError) -> Error { + let cancelled = std::iter::successors(Some(&error as &(dyn StdError + 'static)), |source| { + (*source).source() + }) + .any(|source| source.is::()); + if cancelled { + crate::error::SstWriteCancelledSnafu.build() + } else { + WriteParquetSnafu.into_error(error) + } +} + #[derive(Default)] struct SourceStats { /// Number of rows fetched. @@ -875,47 +1038,34 @@ impl SizeAwareWriter { } } -impl AsyncWrite for SizeAwareWriter -where - W: AsyncWrite + Unpin, -{ - fn poll_write( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &[u8], - ) -> Poll> { - let this = self.as_mut().get_mut(); - - match Pin::new(&mut this.inner).poll_write(cx, buf) { - Poll::Ready(Ok(bytes_written)) => { - this.size.fetch_add(bytes_written, Ordering::Relaxed); - Poll::Ready(Ok(bytes_written)) - } - other => other, - } +impl AsyncFileWriter for SizeAwareWriter { + fn write(&mut self, bytes: Bytes) -> BoxFuture<'_, parquet::errors::Result<()>> { + Box::pin(async move { + let len = bytes.len(); + self.inner.write(bytes).await?; + self.size.fetch_add(len, Ordering::Relaxed); + Ok(()) + }) } - fn poll_flush( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - ) -> Poll> { - Pin::new(&mut self.inner).poll_flush(cx) - } - - fn poll_shutdown( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - ) -> Poll> { - Pin::new(&mut self.inner).poll_shutdown(cx) + fn complete(&mut self) -> BoxFuture<'_, parquet::errors::Result<()>> { + self.inner.complete() } } #[cfg(test)] mod tests { + use std::sync::Mutex; use std::sync::atomic::AtomicBool; use std::time::Duration; + use common_base::readable_size::ReadableSize; use common_runtime::runtime::{BuilderBuild, RuntimeTrait}; + use object_store::layers::mock::{ + Metadata, MockLayerBuilder, ObservedWriter, WriterObservations, oio, + }; + use object_store::services::Memory; + use object_store::{Buffer, Error as ObjectStoreError, ErrorKind, Result as ObjectStoreResult}; use parquet::arrow::ArrowWriter; use parquet::errors::ParquetError; use store_api::region_request::PathType; @@ -925,20 +1075,445 @@ mod tests { use super::*; use crate::access_layer::{RegionFilePathFactory, WriteType}; use crate::compaction::memory_manager::new_compaction_memory_manager; - use crate::error::Error; use crate::sst::FormatType; use crate::sst::parquet::tests::NoopIndexBuilder; use crate::test_util::TestEnv; use crate::test_util::sst_util::{new_record_batch_by_range, sst_region_metadata}; + struct GatedObjectStoreWriter { + inner: oio::Writer, + started: Option>, + release: Option>, + completed: Arc, + write_error: Option, + } + + impl oio::Write for GatedObjectStoreWriter { + async fn write(&mut self, bytes: Buffer) -> ObjectStoreResult<()> { + if let Some(started) = self.started.take() { + started.send(()).unwrap(); + self.release.take().unwrap().await.unwrap(); + } + if let Some(error) = self.write_error.take() { + return Err(error); + } + self.inner.write(bytes).await?; + self.completed.fetch_add(1, Ordering::Relaxed); + Ok(()) + } + + async fn close(&mut self) -> ObjectStoreResult { + self.inner.close().await + } + + async fn abort(&mut self) -> ObjectStoreResult<()> { + self.inner.abort().await + } + } + + #[rstest::rstest] + #[tokio::test] + async fn test_object_store_writer_cancellation_between_chunks( + #[values(false, true)] observe_cancellation: bool, + ) { + let cancellation = CancellationToken::new(); + let observations = Arc::new(Mutex::new(WriterObservations::default())); + let completed = Arc::new(AtomicUsize::new(0)); + let (started_tx, started_rx) = oneshot::channel(); + let (release_tx, release_rx) = oneshot::channel(); + let gate = Mutex::new(Some((started_tx, release_rx))); + let observed = observations.clone(); + let completed_writes = completed.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + let (started, release) = gate.lock().unwrap().take().unwrap(); + Box::new(ObservedWriter::new( + Box::new(GatedObjectStoreWriter { + inner, + started: Some(started), + release: Some(release), + completed: completed_writes.clone(), + write_error: None, + }), + observed.clone(), + )) + })) + .build() + .unwrap(); + let store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let inner = store + .writer_with("output.parquet") + .chunk(4) + .concurrent(1) + .await + .unwrap(); + let mut writer = + ObjectStoreWriter::new(inner, 4, observe_cancellation.then(|| cancellation.clone())); + let data = Bytes::from(vec![42; 32]); + let mut write = Box::pin(writer.write(data.clone())); + assert!(futures::poll!(write.as_mut()).is_pending()); + timeout(Duration::from_secs(5), started_rx) + .await + .unwrap() + .unwrap(); + cancellation.cancel(); + // A started storage write retains its owner until it completes. + assert!(futures::poll!(write.as_mut()).is_pending()); + assert_eq!(0, completed.load(Ordering::Relaxed)); + release_tx.send(()).unwrap(); + let result = timeout(Duration::from_secs(5), write).await.unwrap(); + { + let observed = observations.lock().unwrap(); + assert_eq!(0, observed.abort_calls); + assert_eq!(0, observed.close_calls); + } + if observe_cancellation { + assert!( + matches!(result, Err(ParquetError::External(ref source)) if source.is::()) + ); + assert_eq!(1, completed.load(Ordering::Relaxed)); + assert_eq!(vec![4], observations.lock().unwrap().write_sizes); + // The caller owns abort; the adapter only reports cancellation. + writer.into_inner().abort().await.unwrap(); + assert_eq!(1, observations.lock().unwrap().abort_calls); + assert!(!store.exists("output.parquet").await.unwrap()); + } else { + result.unwrap(); + writer.complete().await.unwrap(); + assert_eq!(data, store.read("output.parquet").await.unwrap().to_bytes()); + let observed = observations.lock().unwrap(); + assert_eq!(1, observed.close_calls); + assert_eq!(0, observed.abort_calls); + assert_eq!(vec![4; 8], observed.write_sizes); + } + } + + #[rstest::rstest] + #[tokio::test] + async fn test_parquet_write_error_aborts_upload( + #[values(1, 2)] parallelism: usize, + #[values(false, true)] cancel_during_write: bool, + ) { + let cancellation = CancellationToken::new(); + let observations = Arc::new(Mutex::new(WriterObservations::default())); + let completed = Arc::new(AtomicUsize::new(0)); + let (started_tx, started_rx) = oneshot::channel(); + let (release_tx, release_rx) = oneshot::channel(); + let gate = Mutex::new(Some((started_tx, release_rx))); + let observed = observations.clone(); + let completed_writes = completed.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + let (started, release) = gate.lock().unwrap().take().unwrap(); + Box::new(ObservedWriter::new( + Box::new(GatedObjectStoreWriter { + inner, + started: Some(started), + release: Some(release), + completed: completed_writes.clone(), + write_error: (!cancel_during_write).then(|| { + ObjectStoreError::new( + ErrorKind::PermissionDenied, + "injected upload failure", + ) + }), + }), + observed.clone(), + )) + })) + .build() + .unwrap(); + let store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let metadata = Arc::new(sst_region_metadata()); + let mut metrics = Metrics::new(WriteType::Compaction); + let mut writer = ParquetWriter::new_with_object_store( + store, + metadata, + IndexConfig::default(), + NoopIndexBuilder, + RegionFilePathFactory::new(String::new(), PathType::Bare), + &mut metrics, + ) + .await + .with_write_concurrency(1) + .with_cancellation(Some(cancellation.clone())); + let batch = new_record_batch_by_range(&["a", "b"], 0, 8); + let opts = WriteOptions { + // A partial group reaches the upload through finish_current_file. + row_group_size: batch.num_rows() + 1, + write_buffer_size: ReadableSize(1), + encode_parallelism: parallelism, + ..Default::default() + }; + let source_finished = Arc::new(AtomicBool::new(false)); + let finished = source_finished.clone(); + let source = FlatSource::new_stream( + batch.schema(), + Box::pin(async_stream::try_stream! { + yield batch; + finished.store(true, Ordering::Relaxed); + }), + ); + let mut write = Box::pin(writer.write_all_flat(source, None, &opts)); + tokio::select! { + started = started_rx => started.unwrap(), + result = &mut write => panic!("writer returned before the upload gate: {result:?}"), + _ = tokio::time::sleep(Duration::from_secs(5)) => panic!("writer did not reach the upload gate"), + } + assert!(source_finished.load(Ordering::Relaxed)); + if cancel_during_write { + cancellation.cancel(); + } + assert!(futures::poll!(write.as_mut()).is_pending()); + assert_eq!(0, observations.lock().unwrap().abort_calls); + release_tx.send(()).unwrap(); + let error = timeout(Duration::from_secs(5), write) + .await + .unwrap() + .unwrap_err(); + if cancel_during_write { + assert!( + matches!(error, Error::SstWriteCancelled { .. }), + "{error:?}" + ); + } else { + assert!(matches!(error, Error::WriteParquet { .. }), "{error:?}"); + let source = + std::iter::successors(Some(&error as &(dyn StdError + 'static)), |error| { + (*error).source() + }) + .find_map(|error| error.downcast_ref::()) + .unwrap(); + assert_eq!(ErrorKind::PermissionDenied, source.kind()); + assert!(source.to_string().contains("injected upload failure")); + } + assert_eq!( + usize::from(cancel_during_write), + completed.load(Ordering::Relaxed) + ); + let observed = observations.lock().unwrap(); + assert_eq!(vec![1], observed.write_sizes); + assert_eq!(1, observed.abort_calls); + assert_eq!(0, observed.close_calls); + assert!(writer.writer.is_none()); + assert!(writer.current_indexer.is_none()); + } + #[rstest::rstest] #[case::flat(FormatType::Flat)] #[case::primary_key(FormatType::PrimaryKey)] #[tokio::test] - async fn test_source_error_waits_for_encoding_cleanup(#[case] format: FormatType) { + async fn test_encoding_error_classification( + #[case] format: FormatType, + #[values(false, true)] finish_file: bool, + #[values(false, true)] cancelled: bool, + ) { let mut env = TestEnv::new().await; let metadata = Arc::new(sst_region_metadata()); let mut metrics = Metrics::new(WriteType::Compaction); + let mut writer = ParquetWriter::new_with_object_store( + env.init_object_store_manager(), + metadata.clone(), + IndexConfig::default(), + NoopIndexBuilder, + RegionFilePathFactory::new(String::new(), PathType::Bare), + &mut metrics, + ) + .await; + let batch = new_record_batch_by_range(&["a", "b"], 0, 4); + let converter = match format { + FormatType::Flat => FlatBatchConverter::Flat(FlatWriteFormat::new( + metadata.clone(), + &FlatSchemaOptions::default(), + )), + FormatType::PrimaryKey => FlatBatchConverter::PrimaryKey { + format: PrimaryKeyWriteFormat::new(metadata.clone()), + num_fields: metadata.field_columns().count(), + }, + }; + let schema = converter.convert_batch(&batch).unwrap().schema(); + let opts = WriteOptions { + // A full row group joins the previous task in append_flat_batch; + // a partial group joins it in finish_current_file instead. + row_group_size: batch.num_rows() + usize::from(finish_file), + encode_parallelism: 2, + ..Default::default() + }; + let SstArrowWriter::Parallel(parallel) = + writer.maybe_init_writer(&schema, &opts).await.unwrap() + else { + panic!("expected parallel encoding"); + }; + let task = common_runtime::spawn_blocking_compact(move || { + let source: Box = if cancelled { + Box::new(Cancelled) + } else { + Box::new(std::io::Error::other("encoding failed")) + }; + Err(ParquetError::External(source)) + }); + // No SST-level cancellation handle: only the encoder's error can report cancellation. + parallel.set_encoding_tasks_for_test(vec![task]); + let source = + FlatSource::new_stream(batch.schema(), Box::pin(futures::stream::iter([Ok(batch)]))); + let error = match format { + FormatType::Flat => writer.write_all_flat(source, None, &opts).await, + FormatType::PrimaryKey => { + writer + .write_all_flat_as_primary_key(source, None, &opts) + .await + } + } + .unwrap_err(); + if cancelled { + assert!(matches!(error, Error::SstWriteCancelled { .. })); + } else { + assert!( + matches!(error, Error::WriteParquet { error: ParquetError::External(ref source), .. } + if source.downcast_ref::().is_some_and(|error| error.to_string() == "encoding failed")) + ); + } + assert!(writer.writer.is_none()); + assert!(writer.current_indexer.is_none()); + assert_eq!(0, writer.bytes_written.load(Ordering::Relaxed)); + } + + #[rstest::rstest] + #[case::flat(FormatType::Flat)] + #[case::primary_key(FormatType::PrimaryKey)] + #[tokio::test] + async fn test_cancel_pending_source( + #[case] format: FormatType, + #[values(1, 2)] parallelism: usize, + ) { + let mut env = TestEnv::new().await; + let metadata = Arc::new(sst_region_metadata()); + let cancellation = CancellationToken::new(); + let mut metrics = Metrics::new(WriteType::Compaction); + let mut writer = ParquetWriter::new_with_object_store( + env.init_object_store_manager(), + metadata, + IndexConfig::default(), + NoopIndexBuilder, + RegionFilePathFactory::new(String::new(), PathType::Bare), + &mut metrics, + ) + .await + .with_cancellation(Some(cancellation.clone())); + let batch = new_record_batch_by_range(&["a", "b"], 0, 4); + let schema = batch.schema(); + let source = FlatSource::new_stream( + schema, + Box::pin(async_stream::try_stream! { + yield batch; + futures::future::pending::<()>().await; + }), + ); + let opts = WriteOptions { + row_group_size: 8, + encode_parallelism: parallelism, + ..Default::default() + }; + let mut write = Box::pin(async { + match format { + FormatType::Flat => writer.write_all_flat(source, None, &opts).await, + FormatType::PrimaryKey => { + writer + .write_all_flat_as_primary_key(source, None, &opts) + .await + } + } + }); + assert!(futures::poll!(write.as_mut()).is_pending()); + cancellation.cancel(); + let error = timeout(Duration::from_secs(5), write) + .await + .unwrap() + .unwrap_err(); + assert!(matches!(error, Error::SstWriteCancelled { .. })); + assert!(writer.writer.is_none()); + } + + /// Controlled comparison: identical batches and cancellation point, changing + /// only whether the writer observes cancellation. + #[rstest::rstest] + #[case::flat(FormatType::Flat)] + #[case::primary_key(FormatType::PrimaryKey)] + #[tokio::test] + async fn test_batch_cancellation_ablation( + #[case] format: FormatType, + #[values(1, 2)] parallelism: usize, + ) { + for enabled in [false, true] { + let mut env = TestEnv::new().await; + let cancellation = CancellationToken::new(); + let mut metrics = Metrics::new(WriteType::Compaction); + let mut writer = ParquetWriter::new_with_object_store( + env.init_object_store_manager(), + Arc::new(sst_region_metadata()), + IndexConfig::default(), + NoopIndexBuilder, + RegionFilePathFactory::new(String::new(), PathType::Bare), + &mut metrics, + ) + .await + .with_cancellation(enabled.then(|| cancellation.clone())); + let batch = new_record_batch_by_range(&["a", "b"], 0, 4); + let consumed = Arc::new(AtomicUsize::new(0)); + let observed = consumed.clone(); + let source = FlatSource::new_stream( + batch.schema(), + Box::pin(async_stream::try_stream! { + for i in 0..32 { + if i == 1 { cancellation.cancel(); } + observed.fetch_add(1, Ordering::Relaxed); + yield new_record_batch_by_range(&["a", "b"], i * 8, i * 8 + 8); + } + }), + ); + let opts = WriteOptions { + row_group_size: 8, + encode_parallelism: parallelism, + ..Default::default() + }; + let result = match format { + FormatType::Flat => writer.write_all_flat(source, None, &opts).await, + FormatType::PrimaryKey => { + writer + .write_all_flat_as_primary_key(source, None, &opts) + .await + } + }; + if enabled { + assert!(result.is_err()); + assert_eq!(2, consumed.load(Ordering::Relaxed)); + assert!(writer.writer.is_none()); + } else { + assert_eq!( + 256, + result + .unwrap() + .iter() + .map(|info| info.num_rows) + .sum::() + ); + assert_eq!(32, consumed.load(Ordering::Relaxed)); + } + } + } + + #[rstest::rstest] + #[case::flat(FormatType::Flat)] + #[case::primary_key(FormatType::PrimaryKey)] + #[tokio::test] + async fn test_source_error_waits_for_encoding_cleanup( + #[case] format: FormatType, + #[values(false, true)] cancel_source: bool, + ) { + let mut env = TestEnv::new().await; + let metadata = Arc::new(sst_region_metadata()); + let cancellation = CancellationToken::new(); + let mut metrics = Metrics::new(WriteType::Compaction); let mut writer = ParquetWriter::new_with_object_store( env.init_object_store_manager(), metadata.clone(), @@ -947,7 +1522,8 @@ mod tests { RegionFilePathFactory::new("".to_string(), PathType::Bare), &mut metrics, ) - .await; + .await + .with_cancellation(Some(cancellation.clone())); let opts = WriteOptions { row_group_size: 8, encode_parallelism: 2, @@ -1010,13 +1586,11 @@ mod tests { parallel.set_encoding_tasks_for_test(vec![running, queued]); let source = FlatSource::new_stream( batch.schema(), - Box::pin(futures::stream::iter(vec![ - Ok(batch), - InvalidRecordBatchSnafu { - reason: "source read failed", - } - .fail(), - ])), + Box::pin(async_stream::try_stream! { + yield batch; + if cancel_source { futures::future::pending::<()>().await; } + Err(InvalidRecordBatchSnafu { reason: "source read failed" }.build())?; + }), ); let memory = new_compaction_memory_manager(1024 * 1024); let budget = memory.try_acquire(memory.limit_bytes()).unwrap(); @@ -1034,6 +1608,10 @@ mod tests { // The next row group is buffered, then the source fails. No codec timing // or I/O wait can make the old indexer-only cleanup pass this assertion. assert!(futures::poll!(write.as_mut()).is_pending()); + if cancel_source { + cancellation.cancel(); + assert!(futures::poll!(write.as_mut()).is_pending()); + } assert!(memory.try_acquire(memory.limit_bytes()).is_none()); assert!(retained_field.upgrade().is_some()); release_tx.send(()).unwrap(); @@ -1041,9 +1619,13 @@ mod tests { .await .unwrap() .unwrap_err(); - assert!( - matches!(error, Error::InvalidRecordBatch { reason, .. } if reason == "source read failed") - ); + if cancel_source { + assert!(matches!(error, Error::SstWriteCancelled { .. })); + } else { + assert!( + matches!(error, Error::InvalidRecordBatch { reason, .. } if reason == "source read failed") + ); + } assert!(!queued_started.load(Ordering::SeqCst)); assert!(retained_field.upgrade().is_none()); assert!(memory.try_acquire(memory.limit_bytes()).is_some()); diff --git a/src/mito2/src/sst/upload.rs b/src/mito2/src/sst/upload.rs new file mode 100644 index 00000000000..a5dfeaf2384 --- /dev/null +++ b/src/mito2/src/sst/upload.rs @@ -0,0 +1,243 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Cooperative byte transfer for SST uploads. Callers own close, abort, and file cleanup. + +use futures::{Stream, StreamExt}; +use object_store::{Buffer, Error as ObjectStoreError, Result as ObjectStoreResult, Writer}; +use snafu::Snafu; +use tokio_util::sync::CancellationToken; + +/// Preserves read/write attribution until the caller adds its file context. +#[derive(Debug, Snafu)] +pub(crate) enum UploadError { + #[snafu(display("Upload cancelled"))] + Cancelled, + #[snafu(display("Failed to read upload source: {source}"))] + Read { source: ObjectStoreError }, + #[snafu(display("Failed to write upload: {source}"))] + Write { source: ObjectStoreError }, +} + +/// Copies chunks without dropping an in-flight write when cancellation arrives. +pub(crate) async fn copy( + reader: &mut (impl Stream> + Unpin), + writer: &mut Writer, + write_buffer_size: usize, + cancellation: Option<&CancellationToken>, +) -> Result { + let mut written = 0; + loop { + let next = if let Some(cancellation) = cancellation { + tokio::select! { + biased; + _ = cancellation.cancelled() => return Err(UploadError::Cancelled), + next = reader.next() => next, + } + } else { + reader.next().await + }; + let Some(bytes) = next else { + return Ok(written); + }; + let bytes = bytes.map_err(|source| UploadError::Read { source })?; + written += write(writer, bytes, write_buffer_size, cancellation).await?; + } +} + +/// Writes bounded, zero-copy slices, awaiting each started write to completion. +pub(crate) async fn write( + writer: &mut Writer, + mut bytes: Buffer, + write_buffer_size: usize, + cancellation: Option<&CancellationToken>, +) -> Result { + let len = bytes.len() as u64; + // Keep each slice nonempty even when the configured size is zero. + let write_buffer_size = write_buffer_size.max(1); + while !bytes.is_empty() { + if cancellation.is_some_and(CancellationToken::is_cancelled) { + return Err(UploadError::Cancelled); + } + let chunk = bytes.split_to(bytes.len().min(write_buffer_size)); + writer + .write(chunk) + .await + .map_err(|source| UploadError::Write { source })?; + } + if cancellation.is_some_and(CancellationToken::is_cancelled) { + return Err(UploadError::Cancelled); + } + Ok(len) +} + +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + use std::time::Duration; + + use bytes::Bytes; + use futures::stream; + use object_store::ObjectStore; + use object_store::layers::mock::{ + Metadata, MockLayerBuilder, ObservedWriter, WriterObservations, oio, + }; + use object_store::services::Memory; + use tokio::sync::Notify; + use tokio::time::timeout; + + use super::*; + + #[tokio::test] + async fn test_copy_cancels_pending_read() { + let store = ObjectStore::new(Memory::default()).unwrap(); + let mut writer = store.writer("pending").await.unwrap(); + let mut reader = stream::pending(); + let cancellation = CancellationToken::new(); + let mut copy = Box::pin(super::copy( + &mut reader, + &mut writer, + 4, + Some(&cancellation), + )); + assert!(futures::poll!(copy.as_mut()).is_pending()); + cancellation.cancel(); + let error = timeout(Duration::from_secs(5), copy) + .await + .unwrap() + .unwrap_err(); + assert!(matches!(error, UploadError::Cancelled)); + writer.abort().await.unwrap(); + assert!(!store.exists("pending").await.unwrap()); + } + + struct GatedWriter { + inner: oio::Writer, + started: Arc, + release: Arc, + } + + impl oio::Write for GatedWriter { + async fn write(&mut self, bytes: Buffer) -> ObjectStoreResult<()> { + self.started.notify_one(); + self.release.notified().await; + self.inner.write(bytes).await + } + + async fn close(&mut self) -> ObjectStoreResult { + self.inner.close().await + } + + async fn abort(&mut self) -> ObjectStoreResult<()> { + self.inner.abort().await + } + } + + #[tokio::test] + async fn test_copy_cancellation_waits_for_started_write() { + let started = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + let observations = Arc::new(Mutex::new(WriterObservations::default())); + let observer = observations.clone(); + let started_write = started.clone(); + let release_write = release.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + Box::new(ObservedWriter::new( + Box::new(GatedWriter { + inner, + started: started_write.clone(), + release: release_write.clone(), + }), + observer.clone(), + )) + })) + .build() + .unwrap(); + let store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + // Leave the OpenDAL writer unbuffered so the observer sees helper boundaries. + let mut writer = store.writer("gated").await.unwrap(); + let mut reader = stream::iter([Ok(Buffer::from(Bytes::from_static(b"abcdefghijkl")))]); + let cancellation = CancellationToken::new(); + let mut copy = Box::pin(super::copy( + &mut reader, + &mut writer, + 4, + Some(&cancellation), + )); + assert!(futures::poll!(copy.as_mut()).is_pending()); + timeout(Duration::from_secs(5), started.notified()) + .await + .unwrap(); + + cancellation.cancel(); + assert!(futures::poll!(copy.as_mut()).is_pending()); + release.notify_one(); + let error = timeout(Duration::from_secs(5), copy) + .await + .unwrap() + .unwrap_err(); + assert!(matches!(error, UploadError::Cancelled)); + { + let observations = observations.lock().unwrap(); + assert_eq!(vec![4], observations.write_sizes); + assert_eq!(0, observations.close_calls); + assert_eq!(0, observations.abort_calls); + } + writer.abort().await.unwrap(); + assert_eq!(1, observations.lock().unwrap().abort_calls); + assert!(!store.exists("gated").await.unwrap()); + } + + #[rstest::rstest] + #[tokio::test] + async fn test_copy_preserves_contents(#[values(false, true)] cancellable: bool) { + let observations = Arc::new(Mutex::new(WriterObservations::default())); + let observer = observations.clone(); + let layer = MockLayerBuilder::default() + .writer_factory(Arc::new(move |_, _, inner| { + Box::new(ObservedWriter::new(inner, observer.clone())) + })) + .build() + .unwrap(); + let store = ObjectStore::new(Memory::default()).unwrap().layer(layer); + let mut writer = store.writer("complete").await.unwrap(); + let mut reader = stream::iter([ + Ok(Buffer::from(Bytes::from_static(b"abcde"))), + Ok(Buffer::from(Bytes::from_static(b"fghijkl"))), + ]); + let cancellation = CancellationToken::new(); + let copied = super::copy( + &mut reader, + &mut writer, + 4, + cancellable.then_some(&cancellation), + ) + .await + .unwrap(); + assert_eq!(12, copied); + { + let observations = observations.lock().unwrap(); + assert_eq!(vec![4, 1, 4, 3], observations.write_sizes); + assert_eq!(0, observations.close_calls); + assert_eq!(0, observations.abort_calls); + } + writer.close().await.unwrap(); + assert_eq!(1, observations.lock().unwrap().close_calls); + assert_eq!( + Bytes::from_static(b"abcdefghijkl"), + store.read("complete").await.unwrap().to_bytes() + ); + } +} diff --git a/src/mito2/src/sst/version.rs b/src/mito2/src/sst/version.rs index 0f040d3bb3f..0d339073310 100644 --- a/src/mito2/src/sst/version.rs +++ b/src/mito2/src/sst/version.rs @@ -21,7 +21,7 @@ use common_time::{TimeToLive, Timestamp}; use store_api::metadata::RegionMetadataRef; use store_api::storage::{FileId, RegionId}; -use crate::sst::file::{self, FileHandle, FileMeta, FileTimeRange, Level}; +use crate::sst::file::{self, FileHandle, FileMeta, FileMetaRef, FileTimeRange, Level}; use crate::sst::file_purger::FilePurgerRef; use crate::sst::primary_key::PrimaryKeyRangeMapper; @@ -77,10 +77,11 @@ impl SstVersion { pub(crate) fn add_files( &mut self, file_purger: FilePurgerRef, - files_to_add: impl Iterator, + files_to_add: impl Iterator>, ) { let levels = Arc::make_mut(&mut self.levels); for file in files_to_add { + let file = file.into(); let level = file.level; let new_index_version = file.index_version; // A trusted L0 adoption changes classification, not the physical object. @@ -98,7 +99,7 @@ impl SstVersion { .files .entry(file.file_id) .and_modify(|f| { - if *f.meta_ref() == file || f.meta_ref().is_index_up_to_date(&file) { + if f.meta_ref() == file.as_ref() || f.meta_ref().is_index_up_to_date(&file) { // same file meta or current file handle's index is up-to-date, skip adding if f.index_id().version > new_index_version { // what does it mean for us to see older index version? diff --git a/src/mito2/src/tail_replay.rs b/src/mito2/src/tail_replay.rs new file mode 100644 index 00000000000..5a8e712cd79 --- /dev/null +++ b/src/mito2/src/tail_replay.rs @@ -0,0 +1,708 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Background replay of the pending WAL tails of a region into SSTs. + +use std::num::NonZeroU64; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Duration; + +use common_telemetry::{error, info, warn}; +use futures::TryStreamExt; +use snafu::ensure; +use store_api::logstore::{LogStore, WalReadUpperBound}; +use store_api::storage::RegionId; +use tokio::sync::Semaphore; +use tokio::sync::mpsc::Sender; + +use crate::cache::CacheManagerRef; +use crate::config::MitoConfig; +use crate::error::{Error, FlushCancelledSnafu, Result, TailEntryMissingSnafu, UnexpectedSnafu}; +use crate::flush::MemtableSstWriter; +use crate::manifest::action::{ + PendingTail, RegionEdit, RegionMetaAction, RegionMetaActionList, UptoKind, WalRecoveryEdit, + cap_wal_release, +}; +use crate::memtable::time_partition::TimePartitions; +use crate::metrics::TAIL_REPLAY_FAILURE_TOTAL; +use crate::region::opener::replay_entry; +use crate::region::version::{VersionBuilder, VersionControl, VersionControlRef}; +use crate::region::{MitoRegion, MitoRegionRef, TailPublication}; +use crate::request::{BackgroundNotify, WorkerRequest, WorkerRequestWithTime}; +use crate::schedule::CancellableTaskState; +use crate::sst::file::{FileMeta, UncommittedSsts}; +use crate::wal::{EntryId, Wal, WalEntryStream}; +use crate::worker::WorkerListener; + +/// Memtable usage at which replayed rows are written to SSTs at the latest. +const CHUNK_MEMTABLE_USAGE: usize = 64 * 1024 * 1024; +/// Replays that read and write at once on a node. +const MAX_CONCURRENT_TAIL_REPLAYS: usize = 16; +/// The replay chunks of a node may use `1 / TAIL_REPLAY_MEMORY_FACTOR` of the global write +/// buffer size. +const TAIL_REPLAY_MEMORY_FACTOR: usize = 8; +/// Delay before the first retry of a failed round or an unfinished commit. +const MIN_BACKOFF: Duration = Duration::from_secs(1); +/// Longest delay between retries. +const MAX_BACKOFF: Duration = Duration::from_secs(30); +/// How often a replay checks whether the region received a Leader lease. +const LEASE_CHECK_INTERVAL: Duration = Duration::from_millis(500); + +/// Returns whether the pending tails of `region` are to be replayed: it has some and +/// [can replay them](MitoRegion::can_replay_tails). +pub(crate) fn should_replay_tails(region: &MitoRegion) -> bool { + region.can_replay_tails() && region.version().has_pending_tails() +} + +/// Replays the pending tails of a region into SSTs and publishes them in one manifest edit. +/// +/// Each round reads the tails in order and writes each into its own SSTs: an SST can give all its +/// rows its largest sequence, so in an SST shared with a later tail, an earlier tail's rows would +/// take the later tail's sequence and outrank rows that a leader between the two tails flushed. A +/// tail's chunks only bound memory: writes accepted during the replay already take sequences +/// above every tail's. A chunk is written once it reaches 64 MiB, or once it grows above its floor +/// and the node's replay memory pool cannot cover the growth. The edit adds the SSTs, clears +/// the tails and advances the flushed entry id and sequence. Another node's listed tail is only a +/// bound, so until the region receives a Leader lease after a catchup made it writable, the edit +/// keeps its unread part, which a second edit clears once a lease arrives. A failed round is +/// retried after a back-off while [should_replay_tails] holds, unless the storage holds a newer +/// manifest and the region is still a Writable leader at its installed version: then the task +/// fences the region for a reload and ends. The task always reports its end to the worker. +pub(crate) struct TailReplayTask { + pub(crate) region: MitoRegionRef, + /// Id of this node, whose tails are read from `wal`. + pub(crate) node_id: Option, + pub(crate) wal: Wal, + /// Writes the chunks, built by [TailReplayLimits::new_sst_writer]. + pub(crate) sst_writer: MemtableSstWriter, + /// Test listener told of failed rounds, lease and fence checks and commits. + pub(crate) listener: WorkerListener, + /// Sends the task's results and its end to the worker. + pub(crate) request_sender: Sender, + /// Limits of the node's tail replays. + pub(crate) limits: TailReplayLimits, + pub(crate) state: CancellableTaskState, +} + +impl TailReplayTask { + pub(crate) async fn run(self) { + let region_id = self.region.region_id; + let _ended = TailReplayEndGuard { + sender: self.request_sender.clone(), + region_id, + }; + let mut backoff = MIN_BACKOFF; + while !self.state.is_cancelled() && should_replay_tails(&self.region) { + match self.replay_round().await { + Ok(()) => return, + Err(_) if self.state.is_cancelled() => break, + Err(e) => { + error!(e; "Failed to replay the pending WAL tails of region {region_id}"); + TAIL_REPLAY_FAILURE_TOTAL.inc(); + self.listener.on_tail_replay_failed(region_id, &e); + // The tails may be gone from a newer manifest, e.g. after another writer + // committed them and their source released its WAL. The worker reloads the + // fenced region once this task ends. + match self.fence_if_manifest_updated().await { + Ok(true) => break, + Ok(false) => {} + Err(_) if self.state.is_cancelled() => break, + Err(e) => { + warn!(e; "Failed to check region {region_id} for a newer manifest"); + } + } + } + } + if !self.sleep(backoff).await { + break; + } + backoff = (backoff * 2).min(MAX_BACKOFF); + } + info!( + "Region {region_id} stopped replaying its pending WAL tails before they were published" + ); + } + + /// Fences the region for a reload if the storage holds a newer manifest and the region is + /// still a Writable leader at its installed version. Returns whether it fenced the region. + /// Cancellation ends it until it holds the manifest write lock. + async fn fence_if_manifest_updated(&self) -> Result { + let manifest_ctx = &self.region.manifest_ctx; + let Some(installed) = self + .unless_cancelled(manifest_ctx.installed_version_if_updated()) + .await? + else { + return Ok(false); + }; + let region_id = self.region.region_id; + info!( + "Region {region_id} found a manifest newer than its version {installed} after a failed replay round" + ); + self.listener.on_tail_fence_begin(region_id); + let mut manager = self + .unless_cancelled(async { Ok(manifest_ctx.manifest_manager.write().await) }) + .await?; + Ok(manifest_ctx + .fence_for_reload_if_unchanged(&mut manager, region_id, installed) + .await) + } + + /// Replays the current tails and publishes the result. An error asks for another round. + async fn replay_round(&self) -> Result<()> { + let region_id = self.region.region_id; + let replay_semaphore = &self.limits.replay_semaphore; + let permit = match replay_semaphore.try_acquire() { + Ok(permit) => permit, + Err(_) => { + info!("Region {region_id} waits for a tail replay permit"); + self.unless_cancelled(async { + replay_semaphore.acquire().await.map_err(|error| { + UnexpectedSnafu { + reason: format!("failed to acquire a tail replay permit: {error}"), + } + .build() + }) + }) + .await? + } + }; + // Keeps no version past this point: it would hold the region's memtables, and their + // write buffer memory, after a flush released them. + let (tails, flushed_entry_id) = { + let version = self.region.version(); + (version.pending_tails.clone(), version.flushed_entry_id) + }; + let uncommitted = UncommittedSsts::new( + region_id, + self.region.access_layer.clone(), + Some(self.sst_writer.cache_manager().clone()), + ); + let replayed = self.replay_tails(&tails, &uncommitted).await; + drop(permit); + let (files, last_entry_ids) = match replayed { + Ok(replayed) => replayed, + Err(e) => { + uncommitted.cleanup().await; + return Err(e); + } + }; + let lease_received = self.region.manifest_ctx.leader_lease_received(); + let remaining = tails + .iter() + .zip(&last_entry_ids) + .filter_map(|(tail, last_entry_id)| { + self.remaining_tail(tail, *last_entry_id, lease_received) + }) + .collect::>(); + if !files.is_empty() || remaining != tails { + let last_entry_id = last_entry_ids.into_iter().flatten().max().unwrap_or(0); + let flushed_sequence = files + .iter() + .filter_map(|file| file.sequence) + .map(NonZeroU64::get) + .max(); + let edit = RegionEdit { + flushed_sequence, + files_to_add: files, + files_to_remove: Vec::new(), + timestamp_ms: Some(chrono::Utc::now().timestamp_millis()), + compaction_time_window: None, + flushed_entry_id: (last_entry_id > flushed_entry_id).then_some(last_entry_id), + // Later writes of the region take sequences above the replayed rows. + committed_sequence: flushed_sequence, + wal_recovery: WalRecoveryEdit { + pending_tails: Some(remaining.clone()), + handoff_floor: None, + }, + }; + if !self.publish(&tails, edit, &uncommitted).await? { + return Ok(()); + } + } + if remaining.is_empty() || !self.wait_for_leader_lease().await { + return Ok(()); + } + let edit = RegionEdit { + timestamp_ms: Some(chrono::Utc::now().timestamp_millis()), + wal_recovery: WalRecoveryEdit { + pending_tails: Some(Vec::new()), + handoff_floor: None, + }, + ..Default::default() + }; + self.publish(&remaining, edit, &uncommitted).await?; + Ok(()) + } + + /// Returns what stays recorded of `tail` after a replay read it up to `last_entry_id`. + /// Only another node's listed tail stays, from its last entry read, and only until the + /// region receives a Leader lease. + fn remaining_tail( + &self, + tail: &PendingTail, + last_entry_id: Option, + lease_received: bool, + ) -> Option { + if lease_received + || tail.upto_kind == UptoKind::Reported + || Some(tail.node_id) == self.node_id + { + return None; + } + let after = last_entry_id.unwrap_or(tail.after); + (after < tail.upto).then(|| PendingTail { + after, + ..tail.clone() + }) + } + + /// Replays `tails` into chunks of SSTs. Returns the SSTs and the last entry id read of + /// each tail. + async fn replay_tails( + &self, + tails: &[PendingTail], + uncommitted: &UncommittedSsts, + ) -> Result<(Vec, Vec>)> { + let region_id = self.region.region_id; + let provider = &self.region.provider; + let mut files = Vec::new(); + let mut last_entry_ids = Vec::with_capacity(tails.len()); + for tail in tails { + let mut entries = self.unless_cancelled(self.read_tail_entries(tail)).await?; + let mut chunk = self.new_chunk_version(); + let mut reservation = self.limits.memory.chunk_reservation(); + let mut last_entry_id = None; + while let Some((entry_id, entry)) = self.unless_cancelled(entries.try_next()).await? { + if entry_id > tail.upto { + break; + } + replay_entry(provider, region_id, &chunk, entry, entry_id).await?; + last_entry_id = Some(entry_id); + let usage = chunk.current().version.memtables.mutable_usage(); + if usage >= CHUNK_MEMTABLE_USAGE || !reservation.grow_to(usage) { + files.extend(self.write_chunk_ssts(&chunk, uncommitted).await?); + chunk = self.new_chunk_version(); + reservation.clear(); + } + if entry_id == tail.upto { + break; + } + } + ensure!( + last_entry_id == Some(tail.upto) || tail.upto_kind == UptoKind::Listed, + TailEntryMissingSnafu { + region_id, + node_id: tail.node_id, + entry_id: tail.upto, + } + ); + files.extend(self.write_chunk_ssts(&chunk, uncommitted).await?); + last_entry_ids.push(last_entry_id); + } + Ok((files, last_entry_ids)) + } + + /// Streams the region's entries of `tail`, starting after `tail.after`. This node's reported + /// tail comes from its own WAL, whose stream may go past `tail.upto`. Every other tail is read + /// up to its bound through [Wal::read_node], since a listed bound may name no entry. + async fn read_tail_entries(&self, tail: &PendingTail) -> Result> { + let (provider, region_id) = (&self.region.provider, self.region.region_id); + let upper = match tail.upto_kind { + UptoKind::Reported if Some(tail.node_id) == self.node_id => { + let mut reader = self.wal.wal_entry_reader(provider, region_id); + return reader.read(provider, tail.after + 1); + } + UptoKind::Reported => WalReadUpperBound::LastEntry(tail.upto), + UptoKind::Listed => WalReadUpperBound::Listed(tail.upto), + }; + self.wal + .read_node(region_id, tail.node_id, tail.after, upper) + .await + } + + /// Waits until the region receives a Leader lease. Returns false if the region stops being + /// a writable, altering or editing leader, or the task is cancelled, first. + async fn wait_for_leader_lease(&self) -> bool { + let region_id = self.region.region_id; + info!("Region {region_id} waits for a Leader lease to clear its unread listed tails"); + while self.region.can_replay_tails() { + self.listener.on_tail_lease_check(region_id); + if self.region.manifest_ctx.leader_lease_received() { + return true; + } + if !self.sleep(LEASE_CHECK_INTERVAL).await { + info!("Region {region_id} cancelled its tail replay while waiting for a lease"); + return false; + } + } + info!("Region {region_id} can no longer replay, keeping its unread listed tails"); + false + } + + /// Returns a scratch version with an empty memtable for one chunk of replayed rows. + fn new_chunk_version(&self) -> VersionControlRef { + let version = self.region.version(); + let mutable = TimePartitions::new( + version.metadata.clone(), + version.memtables.mutable.memtable_builder().clone(), + 0, + Some(version.memtables.mutable.part_duration()), + ); + Arc::new(VersionControl::new( + VersionBuilder::new(version.metadata.clone(), mutable) + .options(version.options.clone()) + .build(), + )) + } + + /// Writes the rows of `chunk` to SSTs. + async fn write_chunk_ssts( + &self, + chunk: &VersionControl, + uncommitted: &UncommittedSsts, + ) -> Result> { + chunk.freeze_mutable()?; + let version = chunk.current().version; + if version.memtables.immutables().is_empty() { + return Ok(Vec::new()); + } + self.sst_writer + .write_ssts(&version, &self.state, uncommitted) + .await + } + + /// Commits `edit` for the replayed `tails`. It commits the same content again while the + /// result is unknown or the region is busy. Returns whether it committed; if not, the task + /// ends. An error asks for another round. + async fn publish( + &self, + tails: &[PendingTail], + edit: RegionEdit, + uncommitted: &UncommittedSsts, + ) -> Result { + let region_id = self.region.region_id; + let action_list = RegionMetaActionList::with_action(RegionMetaAction::Edit(edit.clone())); + let mut backoff = MIN_BACKOFF; + // Whether an attempt may have persisted the edit, which only the same content confirms. + let mut uncertain = false; + loop { + self.listener.on_tail_commit_begin(region_id).await; + if self.state.is_cancelled() { + info!("Region {region_id} cancelled its tail replay before publishing"); + uncommitted.cleanup().await; + return Ok(false); + } + match self + .region + .manifest_ctx + .update_manifest_for_tails(tails, action_list.clone()) + .await + { + Ok(TailPublication::Committed(manifest_version)) => { + uncommitted.disarm_cleanup(); + info!( + "Published the replayed WAL tails of region {region_id} in manifest version {manifest_version}" + ); + self.release_wal().await; + let region = self.region.clone(); + let tails = tails.to_vec(); + let notify = BackgroundNotify::TailReplayCommitted { + region, + edit, + tails, + }; + notify_worker(&self.request_sender, region_id, notify).await; + return Ok(true); + } + Ok(TailPublication::Busy) => {} + Ok(TailPublication::Stale) => { + info!("Region {region_id} changed its role or tails, dropping the replay"); + uncommitted.cleanup().await; + return Ok(false); + } + // The commit fenced the region, which the worker reloads once this task ends. + Err(e @ Error::ManifestVersionConflict { .. }) => { + uncommitted.disarm_cleanup(); + warn!(e; "Publishing the replayed WAL tails of region {region_id} conflicted"); + return Ok(false); + } + Err(e) if e.may_have_persisted_manifest_update() => { + uncommitted.disarm_cleanup(); + uncertain = true; + error!(e; "Publishing the replayed WAL tails of region {region_id} may have failed, retrying"); + } + Err(Error::RegionStopped { .. }) => { + info!("Region {region_id} stopped, dropping the replay"); + uncommitted.cleanup().await; + return Ok(false); + } + Err(e) if uncertain => { + error!(e; "Failed to publish the replayed WAL tails of region {region_id} again, retrying"); + } + Err(e) => { + uncommitted.cleanup().await; + return Err(e); + } + } + if !self.sleep(backoff).await { + info!( + "Region {region_id} cancelled its tail replay while retrying the publication" + ); + uncommitted.cleanup().await; + return Ok(false); + } + backoff = (backoff * 2).min(MAX_BACKOFF); + } + } + + /// Releases the WAL up to where the region now replays it on open, keeping this node's + /// remaining tails. A failure is only logged. + async fn release_wal(&self) { + let region = &self.region; + let manifest = region.manifest_ctx.manifest().await; + let entry_id = cap_wal_release( + &manifest.pending_tails, + self.node_id, + manifest.wal_replay_start(self.node_id), + ); + if let Err(e) = self + .wal + .obsolete(region.region_id, entry_id, ®ion.provider) + .await + { + error!(e; "Failed to release the WAL of region {} after publishing its tails", region.region_id); + } + } + + /// Awaits `future` unless the task is cancelled first. + async fn unless_cancelled(&self, future: impl Future>) -> Result { + let cancellation = self.state.cancel_token(); + tokio::select! { + biased; + _ = cancellation.cancelled() => FlushCancelledSnafu.fail(), + result = future => result, + } + } + + /// Sleeps for `delay`. Returns false if the task is cancelled first. + async fn sleep(&self, delay: Duration) -> bool { + self.unless_cancelled(async { + tokio::time::sleep(delay).await; + Ok(()) + }) + .await + .is_ok() + } +} + +/// Sends `notify` about the region `region_id` to its worker. +pub(crate) async fn notify_worker( + sender: &Sender, + region_id: RegionId, + notify: BackgroundNotify, +) { + let request = WorkerRequest::Background { region_id, notify }; + if sender + .send(WorkerRequestWithTime::new(request)) + .await + .is_err() + { + warn!("The worker of region {region_id} has stopped"); + } +} + +/// Reports the end of a tail replay to the worker, however the task ends. +struct TailReplayEndGuard { + sender: Sender, + region_id: RegionId, +} + +impl Drop for TailReplayEndGuard { + fn drop(&mut self) { + let sender = self.sender.clone(); + let region_id = self.region_id; + common_runtime::spawn_global(async move { + notify_worker(&sender, region_id, BackgroundNotify::TailReplayEnded).await; + }); + } +} + +/// Bounds the tail replays of a node. They take none of the flush permits, while their chunk +/// memtables still count in the global write buffer. +#[derive(Clone)] +pub(crate) struct TailReplayLimits { + /// Replays that read and write at once. + replay_semaphore: Arc, + /// SST writes of the replays, as many as the flushes may run, so a node may write up to + /// `2 * max_background_flushes` level 0 SSTs at once. Writer buffers are not part of the + /// replay memory budget. + write_semaphore: Arc, + /// Memtable memory of the replay chunks. + memory: Arc, +} + +impl TailReplayLimits { + /// Creates the limits of a node from `config`: `MAX_CONCURRENT_TAIL_REPLAYS` replays, + /// `max_background_flushes` SST writes and + /// `global_write_buffer_size / TAIL_REPLAY_MEMORY_FACTOR` bytes of chunk memory. + pub(crate) fn new(config: &MitoConfig) -> Self { + Self { + replay_semaphore: Arc::new(Semaphore::new(MAX_CONCURRENT_TAIL_REPLAYS)), + write_semaphore: Arc::new(Semaphore::new(config.max_background_flushes)), + memory: Arc::new(TailReplayMemory::new( + config.global_write_buffer_size.as_bytes() as usize / TAIL_REPLAY_MEMORY_FACTOR, + )), + } + } + + /// Creates a writer of the `region`'s replay chunks, which takes these limits' write + /// permits. + pub(crate) fn new_sst_writer( + &self, + region: &MitoRegion, + engine_config: Arc, + cache_manager: CacheManagerRef, + ) -> MemtableSstWriter { + MemtableSstWriter::new_tail_replay( + region, + engine_config, + cache_manager, + self.write_semaphore.clone(), + ) + } +} + +/// Memtable memory of the replay chunks on a node, out of a `budget`. The chunks share a pool of +/// half the budget above their floors, and each of the [MAX_CONCURRENT_TAIL_REPLAYS] replays may +/// always use a floor of an equal share of the other half. +struct TailReplayMemory { + /// Bytes each replay's chunk may always use without a reservation. + floor: usize, + /// Bytes the chunks share above their floors. + pool: usize, + /// Bytes of the pool the chunks hold. + reserved: AtomicUsize, +} + +impl TailReplayMemory { + /// Splits `budget` into the floors and the pool. + fn new(budget: usize) -> Self { + Self { + floor: budget / (2 * MAX_CONCURRENT_TAIL_REPLAYS), + pool: budget / 2, + reserved: AtomicUsize::new(0), + } + } + + /// Returns an empty reservation for one chunk. + fn chunk_reservation(&self) -> ChunkReservation<'_> { + ChunkReservation { + memory: self, + bytes: 0, + } + } + + /// Reserves `bytes` of the pool. Returns false if the pool has not enough left. + fn try_reserve(&self, bytes: usize) -> bool { + self.reserved + .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |reserved| { + (reserved + bytes <= self.pool).then_some(reserved + bytes) + }) + .is_ok() + } + + /// Returns `bytes` to the pool. + fn release(&self, bytes: usize) { + self.reserved.fetch_sub(bytes, Ordering::Relaxed); + } +} + +/// The pool memory a chunk uses above its floor. It is released on drop. +struct ChunkReservation<'a> { + /// Memory the reservation is taken from. + memory: &'a TailReplayMemory, + /// Bytes of the pool the chunk holds. + bytes: usize, +} + +impl ChunkReservation<'_> { + /// Grows the reservation to cover a chunk using `usage` bytes. Returns false if the pool + /// has not enough left, in which case the reservation is unchanged. + fn grow_to(&mut self, usage: usize) -> bool { + let needed = usage.saturating_sub(self.memory.floor); + if needed <= self.bytes { + return true; + } + if !self.memory.try_reserve(needed - self.bytes) { + return false; + } + self.bytes = needed; + true + } + + /// Releases the reservation for a new chunk. + fn clear(&mut self) { + self.memory.release(self.bytes); + self.bytes = 0; + } +} + +impl Drop for ChunkReservation<'_> { + fn drop(&mut self) { + self.clear(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_chunk_reservation_keeps_floor_when_pool_is_exhausted() { + // Floor 1 KiB, pool 16 KiB. + let memory = TailReplayMemory::new(32 * 1024); + let mut first = memory.chunk_reservation(); + let mut second = memory.chunk_reservation(); + assert!(first.grow_to(17 * 1024)); + assert!(!first.grow_to(17 * 1024 + 1)); + // The pool is used up, but another chunk still grows to its floor. + assert!(second.grow_to(1024)); + assert!(!second.grow_to(1025)); + + first.clear(); + assert!(second.grow_to(17 * 1024)); + drop(second); + assert_eq!(0, memory.reserved.load(Ordering::Relaxed)); + } + + #[test] + fn test_chunk_reservation_grows_by_increments_and_reuses_after_clear() { + // Floor 1 KiB, pool 16 KiB. + let memory = TailReplayMemory::new(32 * 1024); + let reserved = || memory.reserved.load(Ordering::Relaxed); + let mut chunk = memory.chunk_reservation(); + assert!(chunk.grow_to(5 * 1024)); + assert_eq!(4 * 1024, reserved()); + assert!(chunk.grow_to(9 * 1024)); + assert_eq!(8 * 1024, reserved()); + + chunk.clear(); + assert_eq!(0, reserved()); + assert!(chunk.grow_to(3 * 1024)); + assert_eq!(2 * 1024, reserved()); + drop(chunk); + assert_eq!(0, reserved()); + } +} diff --git a/src/mito2/src/test_util.rs b/src/mito2/src/test_util.rs index 2c06c2b4978..7af095b4ca6 100644 --- a/src/mito2/src/test_util.rs +++ b/src/mito2/src/test_util.rs @@ -55,7 +55,8 @@ use log_store::test_util::log_store_util; use moka::future::CacheBuilder; use object_store::ObjectStore; use object_store::layers::mock::{ - Buffer, Deleter, Metadata, MockLayer, MockLayerBuilder, OpDelete, Result as MockResult, Writer, + Buffer, Deleter, Error as MockError, ErrorKind, Metadata, MockLayer, MockLayerBuilder, + OpDelete, Result as MockResult, Writer, }; use object_store::manager::{ObjectStoreManager, ObjectStoreManagerRef}; use object_store::services::{Fs, Memory}; @@ -252,6 +253,84 @@ impl object_store::layers::mock::Write for BlockingManifestWriter { } } +/// Injects faults into the next manifest delta write of a mock object store. +#[derive(Clone, Default)] +pub(crate) struct DeltaFaults { + /// Whether the next delta write lands but reports an error, like a lost response. + lose_response: Arc, + /// Whether the next delta write fails without landing. + fail_write: Arc, +} + +impl DeltaFaults { + pub(crate) fn lose_next_response(&self) { + self.lose_response.store(true, Ordering::Relaxed); + } + + pub(crate) fn fail_next_write(&self) { + self.fail_write.store(true, Ordering::Relaxed); + } + + pub(crate) fn layer(&self) -> MockLayer { + let faults = self.clone(); + MockLayerBuilder::default() + .writer_factory(Arc::new(move |path, _args, inner| { + if !is_delta_file(path.rsplit('/').next().unwrap_or_default()) { + inner + } else if faults.fail_write.swap(false, Ordering::Relaxed) { + Box::new(FailedWriter) + } else { + Box::new(LostResponseWriter { + inner, + faults: faults.clone(), + }) + } + })) + .build() + .unwrap() + } +} + +struct LostResponseWriter { + inner: Writer, + faults: DeltaFaults, +} + +impl object_store::layers::mock::Write for LostResponseWriter { + async fn write(&mut self, bs: Buffer) -> MockResult<()> { + self.inner.write(bs).await + } + + async fn close(&mut self) -> MockResult { + let metadata = self.inner.close().await?; + if self.faults.lose_response.swap(false, Ordering::Relaxed) { + return Err(MockError::new(ErrorKind::Unexpected, "mock lost response")); + } + Ok(metadata) + } + + async fn abort(&mut self) -> MockResult<()> { + self.inner.abort().await + } +} + +/// A write that fails without touching the store. +struct FailedWriter; + +impl object_store::layers::mock::Write for FailedWriter { + async fn write(&mut self, _: Buffer) -> MockResult<()> { + Err(MockError::new(ErrorKind::Unexpected, "mock failed write")) + } + + async fn close(&mut self) -> MockResult { + Err(MockError::new(ErrorKind::Unexpected, "mock failed write")) + } + + async fn abort(&mut self) -> MockResult<()> { + Ok(()) + } +} + pub(crate) fn new_noop_file_purger() -> FilePurgerRef { Arc::new(NoopFilePurger) } @@ -904,6 +983,7 @@ impl TestEnv { let manifest_dir = "/".to_string(); let manifest_opts = RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir, object_store, compress_type, diff --git a/src/mito2/src/test_util/bench_util.rs b/src/mito2/src/test_util/bench_util.rs index e0334676e5b..7a8a941cea6 100644 --- a/src/mito2/src/test_util/bench_util.rs +++ b/src/mito2/src/test_util/bench_util.rs @@ -84,7 +84,6 @@ pub fn tag_filter_for_bench( prune_schema: metadata.schema.clone(), codec: mito_codec::row_converter::build_primary_key_codec(&metadata), compat_batch: None, - compaction_projection_mapper: None, pre_filter_mode: PreFilterMode::All, partition_filter: None, }; diff --git a/src/mito2/src/test_util/scheduler_util.rs b/src/mito2/src/test_util/scheduler_util.rs index 3249104978a..585c4339638 100644 --- a/src/mito2/src/test_util/scheduler_util.rs +++ b/src/mito2/src/test_util/scheduler_util.rs @@ -149,6 +149,7 @@ impl SchedulerEnv { metadata, 0, RegionManifestOptions { + file_metadata_interner: Default::default(), manifest_dir: "".to_string(), object_store: self.access_layer.object_store().clone(), compress_type: CompressionType::Uncompressed, diff --git a/src/mito2/src/test_util/version_util.rs b/src/mito2/src/test_util/version_util.rs index 136bf11db28..21786def89c 100644 --- a/src/mito2/src/test_util/version_util.rs +++ b/src/mito2/src/test_util/version_util.rs @@ -121,7 +121,8 @@ impl VersionControlBuilder { sequence: NonZeroU64::new(start_ms as u64), partition_expr: match &self.metadata.partition_expr { Some(json_str) => partition::expr::PartitionExpr::from_json_str(json_str) - .expect("partition expression should be valid JSON"), + .expect("partition expression should be valid JSON") + .map(Into::into), None => None, }, ..Default::default() @@ -209,7 +210,8 @@ pub(crate) fn apply_edit( sequence: NonZeroU64::new(*start_ms as u64), partition_expr: match &version_control.current().version.metadata.partition_expr { Some(json_str) => partition::expr::PartitionExpr::from_json_str(json_str) - .expect("partition expression should be valid JSON"), + .expect("partition expression should be valid JSON") + .map(Into::into), None => None, }, ..Default::default() diff --git a/src/mito2/src/wal.rs b/src/mito2/src/wal.rs index e8123988853..c87cb7724e8 100644 --- a/src/mito2/src/wal.rs +++ b/src/mito2/src/wal.rs @@ -31,18 +31,19 @@ use encoder::WalEntryEncoder; use entry_reader::NoopEntryReader; use futures::future::BoxFuture; use futures::stream::BoxStream; -use snafu::ResultExt; +use futures::{StreamExt, TryStreamExt}; +use snafu::{OptionExt, ResultExt}; use store_api::logstore::entry::Entry; use store_api::logstore::provider::Provider; -use store_api::logstore::{AppendBatchResponse, LogStore}; +use store_api::logstore::{AppendBatchResponse, LogStore, WalReadUpperBound}; use store_api::storage::RegionId; use tokio::sync::watch; use crate::error::{ - BuildEntrySnafu, CheckWalHistorySnafu, DeleteWalSnafu, Result, WaitWalDurableSnafu, - WriteWalSnafu, + BuildEntrySnafu, CheckWalHistorySnafu, DeleteWalSnafu, ReadNodeWalSnafu, + RemoteWalUnsupportedSnafu, Result, WaitWalDurableSnafu, WriteWalSnafu, }; -use crate::wal::entry_reader::{LogStoreEntryReader, WalEntryReader}; +use crate::wal::entry_reader::{LogStoreEntryReader, WalEntryReader, decode_raw_entry}; use crate::wal::raw_entry_reader::{LogStoreRawEntryReader, RegionRawEntryReader}; /// WAL entry id. @@ -243,6 +244,33 @@ impl Wal { } } + /// Streams the region's entries in `(after, upper]` from the WAL of node `node_id`, which may + /// be this node, without writing to it. + pub(crate) async fn read_node( + &self, + region_id: RegionId, + node_id: u64, + after: EntryId, + upper: WalReadUpperBound, + ) -> Result> { + let entries = self + .store + .read_node(node_id, region_id, after, upper) + .await + .map_err(BoxedError::new) + .context(ReadNodeWalSnafu { region_id, node_id })? + .context(RemoteWalUnsupportedSnafu { region_id, node_id })?; + Ok(entries + .map(move |entries| { + entries + .map_err(BoxedError::new) + .context(ReadNodeWalSnafu { region_id, node_id }) + }) + .map_ok(|entries| futures::stream::iter(entries.into_iter().map(decode_raw_entry))) + .try_flatten() + .boxed()) + } + /// Scan entries of specific region starting from `start_id` (inclusive). /// Currently only used in tests. pub fn scan<'a>( @@ -296,6 +324,24 @@ impl Wal { .context(DeleteWalSnafu { region_id }) } + /// Queues a persistent release through the region's safe recovery watermark. + /// Success acknowledges admission, not persistence; see [LogStore::release_entries]. + pub async fn release_entries( + &self, + region_id: RegionId, + entry_id: EntryId, + provider: &Provider, + ) -> Result<()> { + if let Provider::Noop = provider { + return Ok(()); + } + self.store + .release_entries(provider, region_id, entry_id) + .await + .map_err(BoxedError::new) + .context(DeleteWalSnafu { region_id }) + } + /// Marks all WAL entries of a region as obsolete and removes its dedicated namespace when /// supported by the backend. pub async fn obsolete_all(&self, region_id: RegionId, provider: &Provider) -> Result<()> { diff --git a/src/mito2/src/worker.rs b/src/mito2/src/worker.rs index c02fbfc1c85..23e193a65a6 100644 --- a/src/mito2/src/worker.rs +++ b/src/mito2/src/worker.rs @@ -32,10 +32,11 @@ mod handle_reload; mod handle_remap; #[cfg(feature = "enterprise")] mod handle_table_compaction; +mod handle_tail_replay; mod handle_truncate; mod handle_write; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet, VecDeque}; use std::path::Path; use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; @@ -46,7 +47,7 @@ use common_error::ext::BoxedError; use common_meta::key::SchemaMetadataManagerRef; use common_runtime::JoinHandle; use common_stat::get_total_memory_bytes; -use common_telemetry::{error, info, warn}; +use common_telemetry::{debug, error, info, warn}; use futures::future::{join_all, try_join_all}; use object_store::ObjectStore; use object_store::manager::ObjectStoreManagerRef; @@ -67,10 +68,10 @@ use crate::cache::{CacheManager, CacheManagerRef, WriteCacheUploadStoreWrapperRe use crate::compaction::memory_manager::{CompactionMemoryManager, new_compaction_memory_manager}; use crate::compaction::{CancelCompactionResult, CompactionCoordinator, CompactionScheduler}; use crate::config::MitoConfig; -use crate::error::{self, CreateDirSnafu, JoinSnafu, Result, WorkerStoppedSnafu}; +use crate::error::{self, CreateDirSnafu, ExternalSnafu, JoinSnafu, Result, WorkerStoppedSnafu}; use crate::flush::{FlushScheduler, WriteBufferManagerImpl, WriteBufferManagerRef}; use crate::gc::{GcLimiter, GcLimiterRef}; -use crate::lease::NodeLease; +use crate::lease::{GrantedLeases, NodeLease}; use crate::memtable::MemtableBuilderProvider; use crate::memtable::bulk::CompactDispatcher; use crate::metrics::{REGION_COUNT, REQUEST_WAIT_TIME, WRITE_STALLING}; @@ -83,6 +84,7 @@ use crate::request::{ BackgroundNotify, BulkInsertRequest, DdlRequest, OptionOutputTx, SenderBulkRequest, SenderDdlRequest, SenderWriteRequest, WorkerRequest, WorkerRequestWithTime, }; +use crate::schedule::CancellableTaskState; use crate::schedule::scheduler::{LocalScheduler, SchedulerRef}; use crate::series_index::{ IndexFilePurger, SeriesIndexTaskState, series_index_channel, spawn_series_index_tasks, @@ -90,8 +92,11 @@ use crate::series_index::{ use crate::sst::file::RegionFileId; use crate::sst::file_ref::FileReferenceManagerRef; use crate::sst::index::IndexBuildScheduler; +use crate::sst::index::handler::IndexKindProviderRef; use crate::sst::index::intermediate::IntermediateManager; use crate::sst::index::puffin_manager::PuffinManagerFactory; +use crate::sst::index::registry::{IndexRegistry, IndexRegistryRef}; +use crate::tail_replay::TailReplayLimits; #[cfg(feature = "hdfs-object-store")] use crate::temp_file_sweeper; use crate::time_provider::{StdTimeProvider, TimeProviderRef}; @@ -164,6 +169,7 @@ pub(crate) struct WorkerGroup { /// Gc limiter to limit concurrent gc jobs. gc_limiter: GcLimiterRef, node_lease: Arc, + granted_leases: Arc, /// Object store manager. object_store_manager: ObjectStoreManagerRef, /// Puffin manager factory. @@ -215,6 +221,7 @@ impl WorkerGroup { let memtable_compact_dispatcher = Arc::new(CompactDispatcher::new(config.max_background_compactions)); let flush_semaphore = Arc::new(Semaphore::new(config.max_background_flushes)); + let tail_replay_limits = TailReplayLimits::new(&config); // We use another scheduler to avoid purge jobs blocking other jobs. let purge_scheduler = Arc::new(LocalScheduler::new(config.max_background_purges)); let upload_store_wrapper = plugins.get::(); @@ -266,6 +273,7 @@ impl WorkerGroup { let sweep_limiter = (config.gc.max_concurrent_gc_job > 0) .then(|| Arc::new(Semaphore::new(config.gc.max_concurrent_gc_job))); let node_lease = Arc::new(NodeLease::default()); + let granted_leases = Arc::new(GrantedLeases::default()); let staging_regions = StagingRegionsRef::default(); let wal = Wal::new(log_store); @@ -297,9 +305,11 @@ impl WorkerGroup { file_ref_manager: file_ref_manager.clone(), partition_expr_fetcher: partition_expr_fetcher.clone(), flush_semaphore: flush_semaphore.clone(), + tail_replay_limits: tail_replay_limits.clone(), #[cfg(feature = "hdfs-object-store")] sweep_limiter: sweep_limiter.clone(), node_lease: node_lease.clone(), + granted_leases: granted_leases.clone(), staging_regions: staging_regions.clone(), } .start() @@ -316,6 +326,7 @@ impl WorkerGroup { file_ref_manager, gc_limiter, node_lease, + granted_leases, object_store_manager, puffin_manager_factory, intermediate_manager, @@ -424,6 +435,21 @@ impl WorkerGroup { &self.node_lease } + /// Returns the latest lease each region of this group received. + pub(crate) fn granted_leases(&self) -> &GrantedLeases { + &self.granted_leases + } + + /// Asks the worker of `region` to replay its pending tails, if it has any, without waiting. + pub(crate) fn notify_writable(&self, region: &MitoRegionRef) { + handle_tail_replay::notify_writable(&self.worker(region.region_id).sender, region); + } + + /// Asks the worker of `region` to cancel its tail replay, if one runs, without waiting. + pub(crate) fn notify_cannot_replay(&self, region: &MitoRegionRef) { + handle_tail_replay::notify_cannot_replay(&self.worker(region.region_id).sender, region); + } + /// Get worker for specific `region_id`. pub(crate) fn worker(&self, region_id: RegionId) -> &RegionWorker { let index = region_id_to_index(region_id, self.workers.len()); @@ -491,6 +517,7 @@ impl WorkerGroup { config.max_background_compactions, )); let flush_semaphore = Arc::new(Semaphore::new(config.max_background_flushes)); + let tail_replay_limits = TailReplayLimits::new(&config); let purge_scheduler = Arc::new(LocalScheduler::new(config.max_background_flushes)); let puffin_manager_factory = PuffinManagerFactory::new( &config.index.aux_path, @@ -544,6 +571,7 @@ impl WorkerGroup { let sweep_limiter = (config.gc.max_concurrent_gc_job > 0) .then(|| Arc::new(Semaphore::new(config.gc.max_concurrent_gc_job))); let node_lease = Arc::new(NodeLease::default()); + let granted_leases = Arc::new(GrantedLeases::default()); let staging_regions = StagingRegionsRef::default(); let wal = Wal::new(log_store); let workers = (0..config.num_workers) @@ -574,9 +602,11 @@ impl WorkerGroup { file_ref_manager: file_ref_manager.clone(), partition_expr_fetcher: partition_expr_fetcher.clone(), flush_semaphore: flush_semaphore.clone(), + tail_replay_limits: tail_replay_limits.clone(), #[cfg(feature = "hdfs-object-store")] sweep_limiter: sweep_limiter.clone(), node_lease: node_lease.clone(), + granted_leases: granted_leases.clone(), staging_regions: staging_regions.clone(), } .start() @@ -593,6 +623,7 @@ impl WorkerGroup { file_ref_manager, gc_limiter, node_lease, + granted_leases, object_store_manager, puffin_manager_factory, intermediate_manager, @@ -704,6 +735,23 @@ pub(crate) fn worker_init_check_delay() -> Duration { Duration::from_secs(init_check_delay) } +/// Constructs the shared registry for region creation and index maintenance. +fn index_registry( + config: &MitoConfig, + provider: Option, +) -> Result { + IndexRegistry::new( + config.inverted_index.clone(), + config.fulltext_index.clone(), + config.bloom_filter_index.clone(), + ) + .with_provider(provider) + .map(Arc::new) + .with_context(|_| ExternalSnafu { + context: "registering index handlers".to_string(), + }) +} + /// Worker start config. struct WorkerStarter { id: WorkerId, @@ -733,11 +781,14 @@ struct WorkerStarter { file_ref_manager: FileReferenceManagerRef, partition_expr_fetcher: PartitionExprFetcherRef, flush_semaphore: Arc, + /// Limits of the node's tail replays. + tail_replay_limits: TailReplayLimits, #[cfg(feature = "hdfs-object-store")] /// Limiter for the temporary file sweeps, or `None` if the datanode allows /// no concurrent GC job. sweep_limiter: Option>, node_lease: Arc, + granted_leases: Arc, staging_regions: StagingRegionsRef, } @@ -836,16 +887,19 @@ impl WorkerStarter { region_count: REGION_COUNT.with_label_values(&[&id_string]), request_wait_time: REQUEST_WAIT_TIME.with_label_values(&[&id_string]), region_edit_queues: RegionEditQueues::default(), - pending_alters: Default::default(), + pending_ddls: PendingDdls::default(), schema_metadata_manager: self.schema_metadata_manager, file_ref_manager: self.file_ref_manager.clone(), partition_expr_fetcher: self.partition_expr_fetcher, flush_semaphore: self.flush_semaphore, + tail_replay_limits: self.tail_replay_limits, plugins: self.plugins, #[cfg(feature = "hdfs-object-store")] sweep_limiter: self.sweep_limiter, node_lease: self.node_lease, + granted_leases: self.granted_leases, staging_regions: self.staging_regions, + tail_replays: HashMap::new(), }; let handle = common_runtime::spawn_global(async move { worker_thread.run().await; @@ -1024,7 +1078,8 @@ type RequestBuffer = Vec; /// Buffer for stalled write requests. /// -/// Maintains stalled write requests and their estimated size. +/// Maintains stalled write requests, their estimated size, and the regions whose writes wait for +/// the end of staging to drain them. #[derive(Default)] pub(crate) struct StalledRequests { /// Stalled requests. @@ -1037,6 +1092,10 @@ pub(crate) struct StalledRequests { HashMap, Vec)>, /// Estimated size of all stalled requests. pub(crate) estimated_size: usize, + /// Regions with writes stalled while they were ending staging. Later writes of these regions + /// stall behind them until [BackgroundNotify::StagingEnded] handles them, so they keep their + /// order. + staging_drains: HashSet, } impl StalledRequests { @@ -1100,6 +1159,118 @@ impl StalledRequests { .map(|(_, reqs, bulk_reqs)| reqs.len() + bulk_reqs.len()) .sum() } + + /// Records that `region_id` stalled a write while ending staging, so its later writes stall + /// behind it until [Self::remove_staging_drain]. + pub(crate) fn add_staging_drain(&mut self, region_id: RegionId) { + self.staging_drains.insert(region_id); + } + + /// Returns whether writes of `region_id` stall behind the ones it stalled while ending + /// staging. + pub(crate) fn has_staging_drain(&self, region_id: &RegionId) -> bool { + self.staging_drains.contains(region_id) + } + + /// Lets later writes of `region_id` through once the end of staging drains its stalled ones. + pub(crate) fn remove_staging_drain(&mut self, region_id: &RegionId) { + self.staging_drains.remove(region_id); + } +} + +/// Has the worker handle the writes it stalled while the region was ending staging, and waits +/// until it has. Await it only as `MitoRegion::end_staging` does. +async fn drain_ended_staging(sender: &Sender, region_id: RegionId) { + let (drained, wait) = oneshot::channel(); + let notify = WorkerRequest::Background { + region_id, + notify: BackgroundNotify::StagingEnded(drained), + }; + let _ = sender.send(WorkerRequestWithTime::new(notify)).await; + let _ = wait.await; +} + +/// Maximum number of pending DDL requests per region. +pub(crate) const MAX_PENDING_DDLS_PER_REGION: usize = 128; + +/// Lifecycle requests waiting for a region manifest operation to finish. +#[derive(Default)] +struct PendingDdls(HashMap>); + +impl PendingDdls { + fn enqueue(&mut self, ddl: SenderDdlRequest) -> std::result::Result<(), SenderDdlRequest> { + let queue = self.0.entry(ddl.region_id).or_default(); + if queue.len() >= MAX_PENDING_DDLS_PER_REGION { + return Err(ddl); + } + debug!( + region_id = %ddl.region_id, + request = ?ddl.request, + queue_len = queue.len() + 1, + "Deferring DDL behind region manifest work" + ); + queue.push_back(ddl); + Ok(()) + } + + fn pop_front(&mut self, region_id: RegionId) -> Option { + let queue = self.0.get_mut(®ion_id)?; + let ddl = queue.pop_front(); + if queue.is_empty() { + self.0.remove(®ion_id); + } + ddl + } + + fn remove(&mut self, region_id: RegionId) -> Option> { + self.0.remove(®ion_id) + } + + fn contains_region(&self, region_id: RegionId) -> bool { + self.0.contains_key(®ion_id) + } + + fn has_close(&self, region_id: RegionId) -> bool { + self.0.get(®ion_id).is_some_and(|queue| { + queue + .iter() + .any(|ddl| matches!(ddl.request, DdlRequest::Close(_))) + }) + } + + fn has_enter_staging(&self, region_id: RegionId) -> bool { + self.0.get(®ion_id).is_some_and(|queue| { + queue + .iter() + .any(|ddl| matches!(ddl.request, DdlRequest::EnterStaging(_))) + }) + } + + fn front_is_enter_staging(&self, region_id: RegionId) -> bool { + self.0 + .get(®ion_id) + .and_then(VecDeque::front) + .is_some_and(|ddl| matches!(ddl.request, DdlRequest::EnterStaging(_))) + } + + fn take_closes(&mut self, region_id: RegionId) -> Vec { + let Some(mut queue) = self.0.remove(®ion_id) else { + return Vec::new(); + }; + let mut matching = Vec::new(); + let mut remaining = VecDeque::new(); + while let Some(ddl) = queue.pop_front() { + if matches!(ddl.request, DdlRequest::Close(_)) { + matching.push(ddl); + } else { + remaining.push_back(ddl); + } + } + if !remaining.is_empty() { + self.0.insert(region_id, remaining); + } + matching + } } /// Background worker loop to handle requests. @@ -1171,8 +1342,8 @@ struct RegionWorkerLoop { request_wait_time: Histogram, /// Queues for region edit requests. region_edit_queues: RegionEditQueues, - /// ALTER requests waiting for the current edit or ALTER to finish. - pending_alters: handle_alter::PendingAlters, + /// DDL requests waiting for the current manifest operation to finish. + pending_ddls: PendingDdls, /// Database level metadata manager. schema_metadata_manager: SchemaMetadataManagerRef, /// Datanode level file references manager. @@ -1181,6 +1352,8 @@ struct RegionWorkerLoop { partition_expr_fetcher: PartitionExprFetcherRef, /// Semaphore to control flush concurrency. flush_semaphore: Arc, + /// Limits of the node's tail replays. + tail_replay_limits: TailReplayLimits, /// Plugins for flush hooks. plugins: Plugins, #[cfg(feature = "hdfs-object-store")] @@ -1188,8 +1361,11 @@ struct RegionWorkerLoop { /// no concurrent GC job. sweep_limiter: Option>, node_lease: Arc, + granted_leases: Arc, /// Staging regions of this node that have received writes. staging_regions: StagingRegionsRef, + /// Running tail replays, which stay here until they report their end. + tail_replays: HashMap, } impl RegionWorkerLoop { @@ -1197,6 +1373,7 @@ impl RegionWorkerLoop { fn register_region(&mut self, region: MitoRegionRef) { if let Some(old) = self.regions.get_region(region.region_id) { old.deactivate_region_manifest_size_metric(); + self.cancel_tail_replay(region.region_id); } self.compaction_scheduler.register_region( @@ -1206,6 +1383,8 @@ impl RegionWorkerLoop { ); self.regions.insert_region(region.clone()); region.activate_region_manifest_size_metric(); + // A create can open an existing region as a writable leader. + self.maybe_replay_tails(region.region_id); } /// Starts the worker loop. @@ -1441,7 +1620,7 @@ impl RegionWorkerLoop { .await; } WorkerRequest::EditRegion(request) => { - self.handle_region_edit(request); + self.handle_incoming_region_edit(request); } WorkerRequest::CancelCompaction { region_id, @@ -1488,86 +1667,95 @@ impl RegionWorkerLoop { self.handle_write_requests(write_requests, bulk_requests, true) .await; - self.handle_ddl_requests(ddl_requests).await; + for ddl in ddl_requests.drain(..) { + let region_id = ddl.region_id; + if self.should_defer_region_ddl(&ddl) { + self.enqueue_region_ddl(ddl); + } else { + self.execute_ddl_request(ddl).await; + } + self.dispatch_pending_region_requests(region_id).await; + } } /// Takes and handles all ddl requests. async fn handle_ddl_requests(&mut self, ddl_requests: &mut Vec) { - if ddl_requests.is_empty() { - return; - } - for ddl in ddl_requests.drain(..) { - let res = match ddl.request { - DdlRequest::Create(req) => self.handle_create_request(ddl.region_id, req).await, - DdlRequest::Drop(req) => { - self.handle_drop_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::Open((req, wal_entry_receiver)) => { - self.handle_open_request(ddl.region_id, req, wal_entry_receiver, ddl.sender) - .await; - continue; - } - DdlRequest::OfflineCleanup(req) => { - self.handle_offline_cleanup_request(ddl.region_id, req) - .await - } - DdlRequest::Close(req) => { - self.handle_close_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::Alter(req) => { - self.handle_alter_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::Flush(req) => { - self.handle_flush_request(ddl.region_id, req, ddl.sender); - continue; - } - DdlRequest::Compact(req) => { - self.handle_compaction_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::BuildIndex(req) => { - self.handle_build_index_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::Truncate(req) => { - self.handle_truncate_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - DdlRequest::Catchup((req, wal_entry_receiver)) => { - self.handle_catchup_request(ddl.region_id, req, wal_entry_receiver, ddl.sender) - .await; - continue; - } - DdlRequest::EnterStaging(req) => { - self.handle_enter_staging_request( - ddl.region_id, - req.partition_directive, - ddl.sender, - ) - .await; - continue; - } - DdlRequest::ApplyStagingManifest(req) => { - self.handle_apply_staging_manifest_request(ddl.region_id, req, ddl.sender) - .await; - continue; - } - }; - - ddl.sender.send(res); + self.execute_ddl_request(ddl).await; } } + /// Executes an admitted request, including requests handed back by a scheduler. + async fn execute_ddl_request(&mut self, ddl: SenderDdlRequest) { + let res = match ddl.request { + DdlRequest::Create(req) => self.handle_create_request(ddl.region_id, req).await, + DdlRequest::Drop(req) => { + self.handle_drop_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::Open((req, wal_entry_receiver)) => { + self.handle_open_request(ddl.region_id, req, wal_entry_receiver, ddl.sender) + .await; + return; + } + DdlRequest::OfflineCleanup(req) => { + self.handle_offline_cleanup_request(ddl.region_id, req) + .await + } + DdlRequest::Close(req) => { + self.handle_close_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::Alter(req) => { + self.execute_alter_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::Flush(req) => { + self.handle_flush_request(ddl.region_id, req, ddl.sender); + return; + } + DdlRequest::Compact(req) => { + self.handle_compaction_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::BuildIndex(req) => { + self.handle_build_index_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::Truncate(req) => { + self.handle_truncate_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + DdlRequest::Catchup((req, wal_entry_receiver)) => { + self.handle_catchup_request(ddl.region_id, req, wal_entry_receiver, ddl.sender) + .await; + return; + } + DdlRequest::EnterStaging(req) => { + self.handle_enter_staging_request( + ddl.region_id, + req.partition_directive, + ddl.sender, + ) + .await; + return; + } + DdlRequest::ApplyStagingManifest(req) => { + self.handle_apply_staging_manifest_request(ddl.region_id, req, ddl.sender) + .await; + return; + } + }; + + ddl.sender.send(res); + } + /// Handle periodical tasks such as region auto flush. fn handle_periodical_tasks(&mut self) { let interval = CHECK_REGION_INTERVAL.as_millis() as i64; @@ -1596,6 +1784,7 @@ impl RegionWorkerLoop { async fn handle_background_notify(&mut self, region_id: RegionId, notify: BackgroundNotify) { // A reload that fails for good would otherwise restart itself forever. let is_reload_result = matches!(notify, BackgroundNotify::RegionReloaded { .. }); + let should_dispatch_pending = notify.should_dispatch_pending(); match notify { BackgroundNotify::RegionOpened { region, registered } => { self.register_region(region); @@ -1636,7 +1825,7 @@ impl RegionWorkerLoop { self.handle_index_build_stopped(region_id, req).await } BackgroundNotify::IndexBuildRetry(req) => { - self.handle_rebuild_index(req, OptionOutputTx::new(None)) + self.handle_rebuild_index_inner(req, OptionOutputTx::new(None), true) .await } BackgroundNotify::CompactionFinished(req) => { @@ -1651,6 +1840,11 @@ impl RegionWorkerLoop { self.handle_manifest_region_change_result(req).await } BackgroundNotify::EnterStaging(req) => self.handle_enter_staging_result(req).await, + BackgroundNotify::StagingEnded(drained) => { + self.stalled_requests.remove_staging_drain(®ion_id); + self.handle_region_stalled_requests(®ion_id, true).await; + let _ = drained.send(()); + } BackgroundNotify::RegionEdit(req) => self.handle_region_edit_result(req).await, BackgroundNotify::CopyRegionFromFinished(req) => { self.handle_copy_region_from_finished(req) @@ -1659,8 +1853,27 @@ impl RegionWorkerLoop { BackgroundNotify::RegionReloaded { old, result } => { self.handle_region_reloaded(region_id, old, result).await } + BackgroundNotify::RegionBecameWritable => self.maybe_replay_tails(region_id), + BackgroundNotify::RegionCannotReplay { region } => { + self.handle_region_cannot_replay(region_id, region) + } + BackgroundNotify::TailReplayCommitted { + region, + edit, + tails, + } => { + self.handle_tail_replay_committed(region_id, region, edit, tails) + .await + } + BackgroundNotify::TailReplayEnded => self.handle_tail_replay_ended(region_id), } - // A commit that found a newer manifest fenced the region before its job reported here. + // Completion handlers may return early after releasing their scheduler. Dispatch + // here covers those paths as well as their normal completion. + if should_dispatch_pending { + self.dispatch_pending_region_requests(region_id).await; + } + // A commit or a tail replay that found a newer manifest fenced the region before its job + // reported here. if !is_reload_result { self.maybe_reload_region(region_id); } @@ -1683,6 +1896,7 @@ impl RegionWorkerLoop { // changes a writable or staging leader, but always fences // promotions issued before. region.manifest_ctx.begin_downgrade(region_id); + self.cancel_tail_replay(region_id); } #[cfg(feature = "hdfs-object-store")] let sweep_older_than = self.config.gc.unknown_file_lingering_time; @@ -1691,7 +1905,11 @@ impl RegionWorkerLoop { let request_sender = self.sender.clone(); // We need to do this in background as we need the manifest lock. common_runtime::spawn_global(async move { - match region.set_role_state_gracefully(region_role_state).await { + let drain = drain_ended_staging(&request_sender, region_id); + match region + .set_role_state_gracefully(region_role_state, drain) + .await + { Ok(()) => { let last_entry_id = region.version_control.current().last_entry_id; let _ = sender.send(SetRegionRoleStateResponse::success( @@ -1728,14 +1946,29 @@ impl RegionWorkerLoop { impl RegionWorkerLoop { /// Cleans up the worker. async fn clean(&self) { + for state in self.tail_replays.values() { + let _ = state.request_cancel(); + } // Closes remaining regions. let regions = self.regions.list_regions(); - for region in regions { + for region in ®ions { region.deactivate_region_manifest_size_metric(); region.stop().await; } + // Another engine may open these regions from the same storage after the stop. Regions + // closed earlier are not waited for; a datanode or standalone shutdown closes every region + // first, so there this only covers regions whose close failed. + join_all( + regions + .iter() + .map(|region| region.manifest_ctx.wait_for_checkpoint_after_stop()), + ) + .await; self.regions.clear(); + for region in regions { + self.granted_leases.remove(region.region_id); + } } /// Notifies the whole group that a flush job is finished so other @@ -1787,6 +2020,46 @@ impl WorkerListener { } } + /// A tail replay is about to commit its result. + pub(crate) async fn on_tail_commit_begin(&self, _region_id: RegionId) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_tail_commit_begin(_region_id).await; + } + } + + /// A tail replay checks for a Leader lease while it waits. + pub(crate) fn on_tail_lease_check(&self, _region_id: RegionId) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_tail_lease_check(_region_id); + } + } + + /// A tail replay that found a newer manifest is about to wait to fence the region. + pub(crate) fn on_tail_fence_begin(&self, _region_id: RegionId) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_tail_fence_begin(_region_id); + } + } + + /// A round of tail replay failed. + pub(crate) fn on_tail_replay_failed(&self, _region_id: RegionId, _error: &crate::error::Error) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_tail_replay_failed(_region_id, _error); + } + } + + /// The worker learned that a tail replay ended. + pub(crate) fn on_tail_replay_ended(&self, _region_id: RegionId) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_tail_replay_ended(_region_id); + } + } + /// Flush is finished successfully. pub(crate) fn on_flush_success(&self, region_id: RegionId) { #[cfg(any(test, feature = "test"))] @@ -1982,6 +2255,13 @@ impl WorkerListener { } } + pub(crate) async fn on_index_build_stopped(&self, _region_file_id: RegionFileId) { + #[cfg(any(test, feature = "test"))] + if let Some(listener) = &self.listener { + listener.on_index_build_stopped(_region_file_id).await; + } + } + pub(crate) async fn on_index_build_before_manifest_commit( &self, _region_file_id: RegionFileId, diff --git a/src/mito2/src/worker/handle_alter.rs b/src/mito2/src/worker/handle_alter.rs index 592b508249e..eb5c33aca09 100644 --- a/src/mito2/src/worker/handle_alter.rs +++ b/src/mito2/src/worker/handle_alter.rs @@ -45,42 +45,7 @@ use crate::request::{DdlRequest, OptionOutputTx, SenderDdlRequest}; use crate::sst::FormatType; use crate::worker::RegionWorkerLoop; -pub(crate) type PendingAlters = std::collections::HashMap< - RegionId, - std::collections::VecDeque<(RegionAlterRequest, OptionOutputTx)>, ->; - impl RegionWorkerLoop { - pub(crate) async fn handle_alter_request( - &mut self, - region_id: RegionId, - request: RegionAlterRequest, - sender: OptionOutputTx, - ) { - if self.flush_scheduler.has_pending_alter(region_id) - || self.pending_alters.contains_key(®ion_id) - || self.regions.get_region(region_id).is_some_and(|region| { - matches!( - region.state(), - crate::region::RegionRoleState::Leader( - crate::region::RegionLeaderState::Editing - | crate::region::RegionLeaderState::Altering - ) - ) - }) - { - let queue = self.pending_alters.entry(region_id).or_default(); - if queue.len() >= 128 { - sender.send(crate::error::RegionBusySnafu { region_id }.fail()); - } else { - queue.push_back((request, sender)); - } - self.dispatch_pending_region_requests(region_id).await; - return; - } - self.execute_alter_request(region_id, request, sender).await; - } - pub(crate) async fn execute_alter_request( &mut self, region_id: RegionId, diff --git a/src/mito2/src/worker/handle_apply_staging.rs b/src/mito2/src/worker/handle_apply_staging.rs index 31a4679c64d..d5924ce16ad 100644 --- a/src/mito2/src/worker/handle_apply_staging.rs +++ b/src/mito2/src/worker/handle_apply_staging.rs @@ -27,6 +27,7 @@ use crate::region::{MitoRegionRef, RegionLeaderState, RegionRoleState}; use crate::request::{OptionOutputTx, RegionEditRequest, WorkerRequest, WorkerRequestWithTime}; use crate::sst::location::region_dir_from_table_dir; use crate::worker::RegionWorkerLoop; +use crate::worker::drain_ended_staging; use crate::worker::handle_reload::notify_if_fenced; impl RegionWorkerLoop { @@ -116,7 +117,11 @@ impl RegionWorkerLoop { return; } - let files_to_add = staging_manifest.files.values().cloned().collect::>(); + let files_to_add = staging_manifest + .files + .values() + .map(|meta| meta.as_ref().clone()) + .collect::>(); let edit = RegionEdit { files_to_add, files_to_remove: vec![], @@ -151,10 +156,9 @@ impl RegionWorkerLoop { sender.send(result.map(|_| 0)); return; }; - let mut manager = region.manifest_ctx.manifest_manager.write().await; - let result = region.exit_staging_on_success(&mut manager).await; - drop(manager); - let hook_payload = match result { + let manager = region.manifest_ctx.manifest_manager.write().await; + let drain = drain_ended_staging(&worker_sender, region_id); + let hook_payload = match region.end_staging(manager, drain).await { Ok(payload) => payload, Err(e) => { notify_if_fenced(&worker_sender, ®ion).await; diff --git a/src/mito2/src/worker/handle_catchup.rs b/src/mito2/src/worker/handle_catchup.rs index 1db62929941..47ad46447bb 100644 --- a/src/mito2/src/worker/handle_catchup.rs +++ b/src/mito2/src/worker/handle_catchup.rs @@ -32,7 +32,7 @@ use crate::request::OptionOutputTx; #[cfg(feature = "hdfs-object-store")] use crate::temp_file_sweeper; use crate::wal::entry_distributor::WalEntryReceiver; -use crate::worker::RegionWorkerLoop; +use crate::worker::{RegionWorkerLoop, drain_ended_staging, handle_tail_replay}; impl RegionWorkerLoop { pub(crate) async fn handle_catchup_request( @@ -79,6 +79,7 @@ impl RegionWorkerLoop { #[cfg(feature = "hdfs-object-store")] let sweep_limiter = self.sweep_limiter.clone(); let node_lease = self.node_lease.clone(); + let worker_sender = self.sender.clone(); common_runtime::spawn_global(async move { let mut task = RegionCatchupTask::new(region.clone(), wal, allow_stale_entries) .with_entry_receiver(entry_receiver) @@ -93,8 +94,9 @@ impl RegionWorkerLoop { node_lease.seed(®ion.manifest_ctx.leader_deadline); if region.manifest_ctx.promote_after_catchup(region_id) { // Finalize leadership: persist backfilled metadata. + let drain = drain_ended_staging(&worker_sender, region_id); match region - .set_role_state_gracefully(SettableRegionRoleState::Leader) + .set_role_state_gracefully(SettableRegionRoleState::Leader, drain) .await { Ok(()) => { @@ -110,6 +112,7 @@ impl RegionWorkerLoop { error!(err; "Failed to set region {region_id} to leader"); } } + handle_tail_replay::notify_writable(&worker_sender, ®ion); } } catchup_regions.remove_region(region_id); diff --git a/src/mito2/src/worker/handle_close.rs b/src/mito2/src/worker/handle_close.rs index 1e447e3ea54..03da3578a58 100644 --- a/src/mito2/src/worker/handle_close.rs +++ b/src/mito2/src/worker/handle_close.rs @@ -14,11 +14,14 @@ //! Handling close request. -use common_telemetry::info; +use common_telemetry::{info, warn}; use store_api::logstore::LogStore; use store_api::region_request::{RegionCloseRequest, RegionFlushReason, RegionFlushRequest}; use store_api::storage::RegionId; +use crate::manifest::action::cap_wal_release; +use crate::metrics::WAL_RELEASE_ADMISSION_FAILURE_TOTAL; +use crate::region::MitoRegionRef; use crate::request::{DdlRequest, OptionOutputTx, SenderDdlRequest}; use crate::worker::RegionWorkerLoop; @@ -43,24 +46,20 @@ impl RegionWorkerLoop { return; } - if region.is_staging_or_staged_edit() { + if region.in_staging_session() { // A staging region cannot flush, so it leaves staging before the close-time flush // below. The manifest lock lets a successful end of staging in progress finish first, // after which the region is already out of staging. let _manager = region.manifest_ctx.manifest_manager.write().await; region.abort_staging(); // Edits queued behind a staged edit belong to the staging that has just ended. - self.reject_region_edit_queue_as_not_found(region_id); + self.reject_region_edits_as_not_found(region_id); } // Memtables containing skip-WAL writes must be flushed before closing. // Followers cannot flush; explicit flush-on-close and region WAL options // retain their existing behavior. - let version = region.version(); - if (request.flush_on_close || region.skip_wal() || version.memtables.has_skip_wal_writes()) - && !version.memtables.is_empty() - && region.is_flushable() - { + if close_needs_flush(®ion, &request) && region.is_flushable() { info!("Region {} has pending data, waiting for flush", region_id); if self.flush_scheduler.is_flush_requested(region_id) { self.flush_scheduler @@ -69,6 +68,7 @@ impl RegionWorkerLoop { sender, request: DdlRequest::Close(request), }); + self.attach_pending_closes_to_flush(region_id); return; } self.handle_flush_request( @@ -79,24 +79,59 @@ impl RegionWorkerLoop { }, sender, ); + if self.flush_scheduler.is_flush_requested(region_id) { + self.attach_pending_closes_to_flush(region_id); + } return; } // No close-time flush is required, or the region cannot flush. - self.remove_region(region_id).await; + let pending_closes = self.take_pending_closes(region_id); + self.close_region(region_id).await; + for ddl in pending_closes { + ddl.sender.send(Ok(0)); + } info!("Region {} closed, worker: {}", region_id, self.id); sender.send(Ok(0)) } + /// Closes a region and queues a release of its persisted WAL history. + /// Pending tails cap the release, as they do after a flush. + /// The stopped manifest makes the watermark stable, and the worker awaits + /// admission before it can process a reopen. Upload failures are retried by + /// the store; failure to admit a release retains extra WAL history. + pub(crate) async fn close_region(&mut self, region_id: RegionId) { + let Some(region) = self.regions.get_region(region_id) else { + return; + }; + self.remove_region(region_id).await; + let version = region.version(); + let release_entry_id = cap_wal_release( + &version.pending_tails, + self.file_ref_manager.node_id(), + version.flushed_entry_id, + ); + if let Err(error) = self + .wal + .release_entries(region_id, release_entry_id, ®ion.provider) + .await + { + WAL_RELEASE_ADMISSION_FAILURE_TOTAL.inc(); + warn!(error; "Failed to queue WAL release for closed region {}", region_id); + } + } + /// Remove a region and stop all related tasks. pub(crate) async fn remove_region(&mut self, region_id: RegionId) { let Some(region) = self.regions.remove_region(region_id) else { return; }; + self.granted_leases.remove(region_id); region.deactivate_region_manifest_size_metric(); + self.cancel_tail_replay(region_id); region.stop().await; self.fail_region_stalled_requests_as_not_found(®ion_id); - self.reject_region_edit_queue_as_not_found(region_id); + self.reject_region_requests_as_not_found(region_id); // Clean flush status. self.flush_scheduler.on_region_closed(region_id); // Clean compaction status. @@ -113,4 +148,22 @@ impl RegionWorkerLoop { hook.on_region_closed(region_id, &metadata).await; } } + + /// Gives closes accepted during an edit the same terminal flush result. + fn attach_pending_closes_to_flush(&mut self, region_id: RegionId) { + for ddl in self.take_pending_closes(region_id) { + self.flush_scheduler.add_ddl_request_to_pending(ddl); + } + } + + fn take_pending_closes(&mut self, region_id: RegionId) -> Vec { + self.pending_ddls.take_closes(region_id) + } +} + +/// Whether closing requires flushing data, independent of temporary region state. +pub(crate) fn close_needs_flush(region: &MitoRegionRef, request: &RegionCloseRequest) -> bool { + let version = region.version(); + !version.memtables.is_empty() + && (request.flush_on_close || region.skip_wal() || version.memtables.has_skip_wal_writes()) } diff --git a/src/mito2/src/worker/handle_compaction.rs b/src/mito2/src/worker/handle_compaction.rs index 269cf258340..b22ed7f27ff 100644 --- a/src/mito2/src/worker/handle_compaction.rs +++ b/src/mito2/src/worker/handle_compaction.rs @@ -298,7 +298,7 @@ impl RegionWorkerLoop { /// Schedule compaction for the region if necessary. pub(crate) async fn schedule_compaction(&mut self, region: &MitoRegionRef) { - if region.is_staging() || region.is_enter_staging() || region.is_truncating() { + if region.in_staging_session() || region.is_enter_staging() || region.is_truncating() { info!( "Region {} is staging, entering staging, or truncating, skip compaction", region.region_id diff --git a/src/mito2/src/worker/handle_copy_region.rs b/src/mito2/src/worker/handle_copy_region.rs index fbd02e8210b..a62ef28000d 100644 --- a/src/mito2/src/worker/handle_copy_region.rs +++ b/src/mito2/src/worker/handle_copy_region.rs @@ -69,6 +69,7 @@ impl RegionWorkerLoop { self.config.clone(), self.object_store_manager.clone(), self.node_lease.is_enabled(), + self.file_ref_manager.file_metadata_interner.clone(), ); let worker_sender = self.sender.clone(); @@ -230,7 +231,7 @@ impl RegionWorkerLoop { for (file_id, file_meta) in &source_region_manifest.files { if !target_region_manifest.files.contains_key(file_id) { new_file_metas.push(if file_meta.level == file::LEVEL_SHARED { - file_meta.clone() + file_meta.as_ref().clone() } else { remap_copied_file_meta(file_meta, target_region_id) }); diff --git a/src/mito2/src/worker/handle_create.rs b/src/mito2/src/worker/handle_create.rs index 5c7da5779f3..3dd9c8e5aca 100644 --- a/src/mito2/src/worker/handle_create.rs +++ b/src/mito2/src/worker/handle_create.rs @@ -22,9 +22,10 @@ use store_api::storage::RegionId; use crate::error::Result; use crate::region::opener::{RegionOpener, check_recovered_region}; +use crate::sst::index::handler::IndexKindProviderRef; #[cfg(feature = "hdfs-object-store")] use crate::temp_file_sweeper; -use crate::worker::RegionWorkerLoop; +use crate::worker::{RegionWorkerLoop, index_registry}; impl RegionWorkerLoop { pub(crate) async fn handle_create_request( @@ -57,6 +58,7 @@ impl RegionWorkerLoop { // Create a MitoRegion from the RegionMetadata. let requirements = request.requirements; + let registry = index_registry(&self.config, self.plugins.get::())?; let opener = RegionOpener::new( region_id, &request.table_dir, @@ -76,7 +78,8 @@ impl RegionWorkerLoop { .cache(Some(self.cache_manager.clone())) .series_index_store(self.series_index_store.clone()) .series_index_purger(self.series_index_purger.clone()) - .hook(self.plugins.get()); + .hook(self.plugins.get()) + .with_index_registry(registry); opener.ensure_region_requirements(requirements)?; diff --git a/src/mito2/src/worker/handle_drop.rs b/src/mito2/src/worker/handle_drop.rs index e3950e0b264..709618428b3 100644 --- a/src/mito2/src/worker/handle_drop.rs +++ b/src/mito2/src/worker/handle_drop.rs @@ -114,7 +114,9 @@ where region.stop().await; // Removes this region from region map to prevent other requests from accessing this region self.regions.remove_region(region_id); + self.granted_leases.remove(region_id); region.deactivate_region_manifest_size_metric(); + self.cancel_tail_replay(region_id); self.dropping_regions.insert_region(region.clone()); // The region is no longer served once it leaves the map, so unregister its // runtime state before the fallible WAL cleanup below. Otherwise a WAL @@ -225,7 +227,7 @@ where pub(crate) async fn cleanup_dropped_region_runtime_state(&mut self, region_id: RegionId) { // A successful flush may hand off DROP before queued ALTERs and edits resume. // Reject their waiters here because the removed region has no next turn. - self.reject_region_edit_queue_as_not_found(region_id); + self.reject_region_requests_as_not_found(region_id); // Notifies flush scheduler. self.flush_scheduler.on_region_dropped(region_id); // Notifies compaction scheduler. diff --git a/src/mito2/src/worker/handle_flush.rs b/src/mito2/src/worker/handle_flush.rs index bcc71520c2b..e51845c71ce 100644 --- a/src/mito2/src/worker/handle_flush.rs +++ b/src/mito2/src/worker/handle_flush.rs @@ -20,13 +20,14 @@ use std::sync::{Arc, Mutex, Weak}; use common_base::readable_size::ReadableSize; use common_telemetry::{debug, error, info}; +use snafu::ResultExt; use store_api::logstore::LogStore; use store_api::region_request::{RegionFlushReason, RegionFlushRequest}; use store_api::storage::RegionId; use crate::config::{IndexBuildMode, MitoConfig}; -use crate::error::{RegionBusySnafu, RegionNotFoundSnafu, Result}; -use crate::flush::{FlushReason, RegionFlushTask}; +use crate::error::{FlushRegionSnafu, RegionBusySnafu, RegionNotFoundSnafu, Result}; +use crate::flush::{FlushReason, MemtableSstWriter, RegionFlushTask}; use crate::manifest::action::cap_wal_release; use crate::region::version::VersionRef; use crate::region::{MitoRegion, MitoRegionRef}; @@ -57,7 +58,7 @@ impl StagingRegions { regions.insert(region.region_id, Arc::downgrade(region)); let mut usage = 0; regions.retain(|_, region| match region.upgrade() { - Some(region) if region.is_staging_or_staged_edit() => { + Some(region) if region.in_staging_session() => { usage += region_memtable_usage(®ion.version()); true } @@ -88,7 +89,12 @@ fn resolve_flush_reason( impl RegionWorkerLoop { /// On region flush job failed. pub(crate) async fn handle_flush_failed(&mut self, region_id: RegionId, request: FlushFailed) { - let mut pending_ddls = self.flush_scheduler.on_flush_failed(region_id, request.err); + let mut pending_ddls = self + .flush_scheduler + .on_flush_failed(region_id, request.err.clone()); + for sender in request.senders { + sender.send(Err(request.err.clone()).context(FlushRegionSnafu { region_id })); + } if !pending_ddls.is_empty() { info!( "Flush terminated for region {}, handling {} pending lifecycle DDL requests", @@ -102,7 +108,6 @@ impl RegionWorkerLoop { // Handle stalled requests. self.handle_stalled_requests().await; - self.dispatch_pending_region_requests(region_id).await; } /// Checks whether the engine reaches flush threshold. If so, finds regions in this @@ -276,17 +281,16 @@ impl RegionWorkerLoop { reason, senders: Vec::new(), request_sender: self.sender.clone(), - access_layer: region.access_layer.clone(), listener: self.listener.clone(), - engine_config, - // The request-provided size (e.g. manual flush) takes precedence over the - // region option. - row_group_size: row_group_size.or(region.version().options.max_row_group_row_count), - cache_manager: self.cache_manager.clone(), manifest_ctx: region.manifest_ctx.clone(), - index_options: region.version().options.index_options.clone(), - flush_semaphore: self.flush_semaphore.clone(), - partition_expr: region.metadata().partition_expr.clone(), + sst_writer: MemtableSstWriter::new_flush( + region, + reason, + row_group_size, + engine_config, + self.cache_manager.clone(), + self.flush_semaphore.clone(), + ), durability_barrier: self .wal .durability_barrier(region.region_id, ®ion.provider), @@ -403,14 +407,21 @@ impl RegionWorkerLoop { /// Flushes locally writable candidates selected by the WAL. pub(crate) fn flush_for_wal_pressure(&mut self, candidates: &[RegionId]) -> Result<()> { + let node_id = self.file_ref_manager.node_id(); for ®ion_id in candidates.iter() { let Some(region) = self.regions.get_region(region_id) else { continue; }; + let version = region.version(); + // A flush cannot move the WAL release of a region past one of this node's tails. + let release_capped = version.pending_tails.iter().any(|tail| { + Some(tail.node_id) == node_id && tail.after <= version.flushed_entry_id + }); if !region.is_writable() || region.is_staging() || self.flush_scheduler.is_flush_requested(region_id) - || region_memtable_usage(®ion.version()) == 0 + || region_memtable_usage(&version) == 0 + || release_capped { continue; } @@ -531,8 +542,14 @@ impl RegionWorkerLoop { error!(err; "Failed to write wal, region: {}", region_id); request.on_shared_failure(err.clone()); // The flush scheduler still tracks the flush, so release it as a failed flush does. - self.handle_flush_failed(region_id, FlushFailed { err }) - .await; + self.handle_flush_failed( + region_id, + FlushFailed { + err, + senders: Vec::new(), + }, + ) + .await; return; } @@ -555,7 +572,7 @@ impl RegionWorkerLoop { } if flush_on_close { - self.remove_region(region_id).await; + self.close_region(region_id).await; info!("Region {} closed after flush", region_id); request.on_success(); self.listener.on_flush_success(region_id); @@ -569,15 +586,8 @@ impl RegionWorkerLoop { self.flush_scheduler.on_flush_success(region_id) { // Perform DDLs first because they require empty memtables. - // These ALTERs already own the flush turn; bypass initial admission. - for ddl in ddl_requests.drain(..) { - if let crate::request::DdlRequest::Alter(request) = ddl.request { - self.execute_alter_request(ddl.region_id, request, ddl.sender) - .await; - } else { - self.handle_ddl_requests(&mut vec![ddl]).await; - } - } + // Scheduler-owned requests bypass initial admission but retain handler validation. + self.handle_ddl_requests(&mut ddl_requests).await; if self.flush_scheduler.is_flush_requested(region_id) { // The DDL may schedule another flush, e.g. a close-time flush after writes // arrived in the mutable memtable during the previous flush. Keep pending @@ -612,7 +622,6 @@ impl RegionWorkerLoop { // Schedules compaction. self.schedule_compaction(®ion).await; - self.dispatch_pending_region_requests(region_id).await; self.listener.on_flush_success(region_id); } diff --git a/src/mito2/src/worker/handle_manifest.rs b/src/mito2/src/worker/handle_manifest.rs index 8cd0d77c5d3..60e64304d72 100644 --- a/src/mito2/src/worker/handle_manifest.rs +++ b/src/mito2/src/worker/handle_manifest.rs @@ -31,7 +31,8 @@ use crate::cache::CacheManagerRef; use crate::cache::file_cache::{FileType, IndexKey}; use crate::config::IndexBuildMode; use crate::error::{ - EditRegionSnafu, RegionBusySnafu, RegionNotFoundSnafu, RegionStateSnafu, Result, + EditRegionSnafu, RegionBusySnafu, RegionNotFoundSnafu, RegionStateSnafu, + RejectWriteInStagingSnafu, Result, }; use crate::manifest::action::{ RegionChange, RegionEdit, RegionMetaAction, RegionMetaActionList, RegionTruncate, TruncateKind, @@ -44,13 +45,14 @@ use crate::region::options::RegionOptions; use crate::region::version::VersionControlRef; use crate::region::{MitoRegionRef, RegionLeaderState, RegionRoleState}; use crate::request::{ - BackgroundNotify, BuildIndexRequest, DiscardUnflushedResult, OptionOutputTx, + BackgroundNotify, BuildIndexRequest, DdlRequest, DiscardUnflushedResult, OptionOutputTx, RegionChangeResult, RegionEditKind, RegionEditRequest, RegionEditResult, RegionSyncRequest, - TruncateResult, WorkerRequest, WorkerRequestWithTime, + SenderDdlRequest, TruncateResult, WorkerRequest, WorkerRequestWithTime, }; use crate::sst::index::IndexBuildType; use crate::sst::{file, location}; use crate::wal::EntryId; +use crate::worker::handle_close; use crate::worker::{RegionWorkerLoop, WorkerListener}; pub(crate) type RegionEditQueues = HashMap; @@ -85,7 +87,7 @@ impl RegionEditQueue { } fn enqueue(&mut self, request: RegionEditRequest) -> RegionEditDispatch { - if self.requests.len() > Self::QUEUE_MAX_LEN { + if self.requests.len() >= Self::QUEUE_MAX_LEN { request.waiters.reply_with(|| { RegionBusySnafu { region_id: self.region_id, @@ -128,7 +130,10 @@ impl RegionEditQueue { return Some(merged); } - while let Some(request) = self.requests.pop_front_if(|request| can_merge(request)) { + // Merging must preserve the staging binding of every waiter. + while let Some(request) = self.requests.pop_front_if(|request| { + can_merge(request) && request.staging_partition_expr == merged.staging_partition_expr + }) { merged.edit.files_to_add.extend(request.edit.files_to_add); merged.waiters.merge(request.waiters); } @@ -162,13 +167,18 @@ impl RegionEditQueue { } impl RegionWorkerLoop { - /// Rejects queued ALTER and region edit requests as region not found. - pub(crate) fn reject_region_edit_queue_as_not_found(&mut self, region_id: RegionId) { - if let Some(queue) = self.pending_alters.remove(®ion_id) { - for (_, sender) in queue { - sender.send(RegionNotFoundSnafu { region_id }.fail()); + /// Rejects queued DDL and region edit requests as region not found. + pub(crate) fn reject_region_requests_as_not_found(&mut self, region_id: RegionId) { + if let Some(queue) = self.pending_ddls.remove(region_id) { + for ddl in queue { + ddl.sender.send(RegionNotFoundSnafu { region_id }.fail()); } } + self.reject_region_edits_as_not_found(region_id); + } + + /// Rejects edits belonging to an abandoned staging without dropping queued controls. + pub(crate) fn reject_region_edits_as_not_found(&mut self, region_id: RegionId) { if let Some(edit_queue) = self.region_edit_queues.remove(®ion_id) { edit_queue.reject_all_as_not_found(); } @@ -224,8 +234,6 @@ impl RegionWorkerLoop { // Handles the stalled requests. self.handle_region_stalled_requests(&change_result.region_id, true) .await; - self.dispatch_pending_region_requests(change_result.region_id) - .await; } /// Handles region sync request. @@ -305,8 +313,39 @@ impl RegionWorkerLoop { } impl RegionWorkerLoop { - /// Admits an edit and reports whether it started, queued, or failed synchronously. - pub(crate) fn handle_region_edit(&mut self, request: RegionEditRequest) -> RegionEditDispatch { + fn is_import_without_staging_binding(request: &RegionEditRequest) -> bool { + matches!(request.kind, RegionEditKind::Import) && request.staging_partition_expr.is_none() + } + + /// Whether EnterStaging is pending in the worker, flush, or compaction queue. + fn has_pending_enter_staging(&self, region_id: RegionId) -> bool { + self.flush_scheduler.has_pending_enter_staging(region_id) + || self + .compaction_scheduler + .has_pending_enter_staging(region_id) + || self.pending_ddls.has_enter_staging(region_id) + } + + /// Handles a newly received edit, rejecting unbound imports while EnterStaging drains + /// previously accepted edits. Queued edits bypass this admission check. + pub(crate) fn handle_incoming_region_edit( + &mut self, + request: RegionEditRequest, + ) -> RegionEditDispatch { + let region_id = request.region_id; + if Self::is_import_without_staging_binding(&request) + && self.has_pending_enter_staging(region_id) + { + request + .waiters + .reply_with(|| RegionBusySnafu { region_id }.fail()); + return RegionEditDispatch::Rejected; + } + self.handle_region_edit(request) + } + + /// Executes an admitted edit and reports whether it started, queued, or failed synchronously. + fn handle_region_edit(&mut self, request: RegionEditRequest) -> RegionEditDispatch { let region_id = request.region_id; let Some(region) = self.regions.get_region(region_id) else { request @@ -315,6 +354,14 @@ impl RegionWorkerLoop { return RegionEditDispatch::Rejected; }; + // Unbound imports must not be acknowledged into a manifest that staging can discard. + if Self::is_import_without_staging_binding(&request) && region.in_staging_session() { + request + .waiters + .reply_with(|| RejectWriteInStagingSnafu { region_id }.fail()); + return RegionEditDispatch::Rejected; + } + // A close-time flush ends with the region removed. A close that left staging keeps the // region writable until then, so an edit meant for staging would run as a normal edit. if self.flush_scheduler.has_close_time_flush(region_id) { @@ -345,11 +392,11 @@ impl RegionWorkerLoop { return RegionEditDispatch::Rejected; } - if !region.is_writable() || self.flush_scheduler.has_pending_alter(region_id) { + if !region.is_writable() || self.scheduler_blocks_region_edits(region_id) { if matches!( region.state(), RegionRoleState::Leader(RegionLeaderState::Editing | RegionLeaderState::Altering) - ) || (region.is_writable() && self.flush_scheduler.has_pending_alter(region_id)) + ) || (region.is_writable() && self.scheduler_blocks_region_edits(region_id)) { return self .region_edit_queues @@ -463,7 +510,7 @@ impl RegionWorkerLoop { // Fail writes stalled behind this edit if the region was removed before the // edit-completion notification reached the worker. self.fail_region_stalled_requests_as_not_found(&edit_result.region_id); - self.reject_region_edit_queue_as_not_found(edit_result.region_id); + self.reject_region_requests_as_not_found(edit_result.region_id); edit_result.waiters.reply_with(|| { RegionNotFoundSnafu { @@ -557,6 +604,8 @@ impl RegionWorkerLoop { .await; } + // Edit completion owns dispatch: stalled writes precede queued edits, and queued + // controls precede compaction. Early returns either retire waiters or ignore an old instance. self.dispatch_pending_region_requests(edit_result.region_id) .await; @@ -565,53 +614,150 @@ impl RegionWorkerLoop { } } - /// ALTER has priority over the edit backlog. Flush-pending DDLs and the - /// Altering state retain that priority until their terminal notification. + /// Retains a control request without blocking the worker that receives its completion. + pub(crate) fn enqueue_region_ddl(&mut self, ddl: SenderDdlRequest) { + let region_id = ddl.region_id; + if let Err(ddl) = self.pending_ddls.enqueue(ddl) { + ddl.sender.send(RegionBusySnafu { region_id }.fail()); + } + } + + /// Whether a scheduler holds lifecycle work that must finish before edits run. + /// + /// Running ordinary and repartition flushes retain concurrency with edits through the + /// existing manifest synchronization. Ordinary and repartition flushes alone do not block + /// edits. + fn scheduler_blocks_region_edits(&self, region_id: RegionId) -> bool { + self.flush_scheduler.has_pending_ddls(region_id) + || self.compaction_scheduler.blocks_region_edits(region_id) + } + + /// Defers only recoverable local contention; the original handler validates other states. + pub(crate) fn should_defer_region_ddl(&self, ddl: &SenderDdlRequest) -> bool { + let Some(region) = self.regions.get_region(ddl.region_id) else { + return false; + }; + self.can_defer_ddl_request(ddl, ®ion) + && self.has_work_ahead_of_ddl(ddl.region_id, ®ion, &ddl.request) + } + + /// Whether this request may wait for ongoing work in the current region state. + /// Ineligible requests go directly to their handlers for normal validation. + fn can_defer_ddl_request(&self, ddl: &SenderDdlRequest, region: &MitoRegionRef) -> bool { + if !matches!( + region.state(), + RegionRoleState::Leader( + RegionLeaderState::Writable + | RegionLeaderState::Staging + | RegionLeaderState::Editing + | RegionLeaderState::Altering + | RegionLeaderState::EnteringStaging + ) + ) { + return false; + } + let version = region.version(); + match &ddl.request { + DdlRequest::Alter(_) | DdlRequest::Flush(_) | DdlRequest::ApplyStagingManifest(_) => { + true + } + DdlRequest::EnterStaging(request) => { + if version.has_pending_tails() { + return false; + } + // A different directive is not temporary contention with the active staging. + if let Some(info) = region.manifest_ctx.staging_partition_info() + && info.partition_directive != request.partition_directive + { + return false; + } + true + } + DdlRequest::Close(request) if !region.in_staging_session() => { + // Another Close can take queued closes and reply without executing their + // handlers, so reject invalid flush-on-close requests before that handoff. + !self.flush_scheduler.has_close_time_flush(ddl.region_id) + && !(request.flush_on_close && version.has_pending_tails()) + // A later close cannot bypass an earlier close's flush requirement. + && (handle_close::close_needs_flush(region, request) + || self.pending_ddls.has_close(ddl.region_id)) + } + _ => false, + } + } + + /// Whether active work or queue ordering requires this DDL to wait. + /// EnterStaging also waits for previously accepted edits to drain. + fn has_work_ahead_of_ddl( + &self, + region_id: RegionId, + region: &MitoRegionRef, + request: &DdlRequest, + ) -> bool { + self.pending_ddls.contains_region(region_id) + || self.scheduler_blocks_region_edits(region_id) + || matches!( + region.state(), + RegionRoleState::Leader( + RegionLeaderState::Editing + | RegionLeaderState::Altering + | RegionLeaderState::EnteringStaging + ) + ) + || (matches!(request, DdlRequest::EnterStaging(_)) + && self + .region_edit_queues + .get(®ion_id) + .is_some_and(|queue| !queue.is_empty())) + } + + /// Controls retain priority, except that staging first drains its accepted edit prefix. pub(crate) async fn dispatch_pending_region_requests(&mut self, region_id: RegionId) { loop { - if self.flush_scheduler.has_pending_alter(region_id) { + if self.scheduler_blocks_region_edits(region_id) { return; } if self.regions.get_region(region_id).is_some_and(|region| { matches!( region.state(), RegionRoleState::Leader( - RegionLeaderState::Editing | RegionLeaderState::Altering + RegionLeaderState::Editing + | RegionLeaderState::Altering + | RegionLeaderState::EnteringStaging ) ) }) { return; } + let drain_edits = self.pending_ddls.front_is_enter_staging(region_id); + if !drain_edits + || self + .region_edit_queues + .get(®ion_id) + .is_none_or(|queue| queue.is_empty()) + { + let next = self.pending_ddls.pop_front(region_id); + if let Some(ddl) = next { + self.execute_ddl_request(ddl).await; + continue; + } + } let next = self - .pending_alters - .get_mut(®ion_id) - .and_then(|q| q.pop_front()); - let Some((request, sender)) = next else { - break; - }; - self.execute_alter_request(region_id, request, sender).await; - } - self.pending_alters.remove(®ion_id); - loop { - let next_request = self .region_edit_queues .get_mut(®ion_id) - .and_then(|queue| queue.dequeue()); + .and_then(RegionEditQueue::dequeue); if self .region_edit_queues .get(®ion_id) - .is_some_and(|queue| queue.is_empty()) + .is_some_and(RegionEditQueue::is_empty) { self.region_edit_queues.remove(®ion_id); } - let Some(request) = next_request else { + let Some(request) = next else { return; }; - // Start at most one edit batch per turn. Started and queued requests - // rely on a completion notification to resume dispatch. Only synchronous - // rejections continue: they produce no notification, so stopping would - // strand the remaining waiters. Use the admission outcome rather than - // rereading region state, which can race with a role change. + // Rejected edits produce no completion notification. Keep draining them so neither + // the next edit nor the staging boundary is stranded. Execution bypasses admission. match self.handle_region_edit(request) { RegionEditDispatch::Started | RegionEditDispatch::Queued => return, RegionEditDispatch::Rejected => continue, @@ -1021,6 +1167,46 @@ mod tests { use crate::sst::file::{FileMeta, LEVEL_SHARED}; use crate::test_util::{CreateRequestBuilder, TestEnv}; + #[test] + fn test_edit_queue_preserves_staging_bindings() { + for (left, right, merge) in [ + (None, None, true), + (Some("a"), Some("a"), true), + (None, Some("a"), false), + (Some("a"), None, false), + (Some("a"), Some("b"), false), + ] { + let region_id = RegionId::new(1, 1); + let mut queue = RegionEditQueue::new(region_id); + let mut receivers = Vec::new(); + for binding in [left, right] { + let (sender, receiver) = tokio::sync::oneshot::channel(); + let mut request = + RegionEditRequest::new(region_id, RegionEdit::default(), false, sender); + request.staging_partition_expr = binding.map(str::to_string); + assert!(matches!(queue.enqueue(request), RegionEditDispatch::Queued)); + receivers.push(receiver); + } + let first = queue.dequeue().unwrap(); + assert_eq!(left, first.staging_partition_expr.as_deref()); + first.waiters.reply_with(|| Ok(())); + assert!(receivers[0].try_recv().unwrap().is_ok()); + if merge { + assert!(receivers[1].try_recv().unwrap().is_ok()); + } else { + assert!(matches!( + receivers[1].try_recv(), + Err(tokio::sync::oneshot::error::TryRecvError::Empty) + )); + let second = queue.dequeue().unwrap(); + assert_eq!(right, second.staging_partition_expr.as_deref()); + second.waiters.reply_with(|| Ok(())); + assert!(receivers[1].try_recv().unwrap().is_ok()); + } + assert!(queue.is_empty()); + } + } + #[rstest::rstest] #[case::shared_removal(false)] #[case::unsupported_level(true)] diff --git a/src/mito2/src/worker/handle_rebuild_index.rs b/src/mito2/src/worker/handle_rebuild_index.rs index c17008bab27..e897a113be5 100644 --- a/src/mito2/src/worker/handle_rebuild_index.rs +++ b/src/mito2/src/worker/handle_rebuild_index.rs @@ -19,6 +19,7 @@ use std::sync::Arc; use api::v1::region::build_index_request; use common_telemetry::{debug, warn}; +use snafu::{IntoError, ResultExt}; use store_api::metadata::RegionMetadataRef; use store_api::region_request::RegionBuildIndexRequest; use store_api::storage::{FileId, RegionId}; @@ -26,7 +27,8 @@ use tokio::sync::mpsc; use crate::cache::CacheStrategy; use crate::error::{ - IndexBuildIncompleteSnafu, InvalidRequestSnafu, RegionStateSnafu, Result, WorkerStoppedSnafu, + BuildIndexAsyncSnafu, ExternalSnafu, IndexBuildIncompleteSnafu, InvalidRequestSnafu, + RegionStateSnafu, Result, WorkerStoppedSnafu, }; use crate::manifest::action::RegionEdit; use crate::metrics::INDEX_PUBLICATION_STALE_TOTAL; @@ -34,11 +36,13 @@ use crate::region::options::IndexOptions; use crate::region::{IndexBuildSource, MitoRegionRef, RegionLeaderState, RegionRoleState}; use crate::request::{BuildIndexRequest, IndexBuildFinished, IndexBuildStopped, OptionOutputTx}; use crate::sst::file::{self, FileHandle, FileMeta, RegionFileId, RegionIndexId}; +use crate::sst::index::handler::{IndexKindProviderRef, PlanContext}; +use crate::sst::index::registry::{IndexRegistry, IndexRegistryRef}; use crate::sst::index::{ IndexBuildOutcome, IndexBuildTask, IndexBuildType, IndexerBuilderImpl, ResultMpscSender, cleanup_stale_index_caches, }; -use crate::worker::RegionWorkerLoop; +use crate::worker::{RegionWorkerLoop, index_registry}; /// Parts of a region version that index build tasks need. /// @@ -54,11 +58,12 @@ impl RegionWorkerLoop { &self, region: &MitoRegionRef, snapshot: &IndexBuildSnapshot, - file: FileHandle, - file_meta: FileMeta, + registry: IndexRegistryRef, + source: (FileHandle, FileMeta), build_type: IndexBuildType, result_senders: Vec, ) -> IndexBuildTask { + let (file, file_meta) = source; let access_layer = region.access_layer.clone(); let puffin_manager = if let Some(write_cache) = self.cache_manager.write_cache() { @@ -74,14 +79,10 @@ impl RegionWorkerLoop { }; let indexer_builder_ref = Arc::new(IndexerBuilderImpl { + index_build_inputs: None, build_type: build_type.clone(), metadata: snapshot.metadata.clone(), - index_build_inputs: None, - registry: Arc::new(crate::sst::index::registry::IndexRegistry::new( - self.config.inverted_index.clone(), - self.config.fulltext_index.clone(), - self.config.bloom_filter_index.clone(), - )), + registry: registry.clone(), index_options: snapshot.index_options.clone(), intermediate_manager, puffin_manager, @@ -107,6 +108,25 @@ impl RegionWorkerLoop { } } + /// Validates providers and the current plan before selecting or scheduling files. + fn prepare_rebuild_registry( + &self, + snapshot: &IndexBuildSnapshot, + build_type: IndexBuildType, + ) -> Result { + // Post-flush/compaction tasks build the built-in indexes of a new SST. + // Compaction extension construction belongs to the compaction writer. + let provider = match build_type { + IndexBuildType::Flush | IndexBuildType::Compact => None, + IndexBuildType::Manual | IndexBuildType::SchemaChange => { + self.plugins.get::() + } + }; + let registry = index_registry(&self.config, provider)?; + validate_rebuild_plan(®istry, snapshot, build_type)?; + Ok(registry) + } + /// Handles manual build index requests. pub(crate) async fn handle_build_index_request( &mut self, @@ -165,9 +185,19 @@ impl RegionWorkerLoop { } pub(crate) async fn handle_rebuild_index( + &mut self, + request: BuildIndexRequest, + sender: OptionOutputTx, + ) { + self.handle_rebuild_index_inner(request, sender, false) + .await; + } + + pub(crate) async fn handle_rebuild_index_inner( &mut self, request: BuildIndexRequest, mut sender: OptionOutputTx, + is_retry: bool, ) { let region_id = request.region_id; let Some(region) = self.regions.writable_region_or(region_id, &mut sender) else { @@ -223,19 +253,10 @@ impl RegionWorkerLoop { file_meta.cloned() }; - let build_tasks = if request.file_metas.is_empty() { - // If no specific files are provided, find files whose index is inconsistent with the region metadata. - // Check the version rather than the manifest: a publication committed - // to the manifest but not yet applied is still invisible to queries. - // Its build still holds the file, so the manual task waits for it. + // Collect eligible inputs before currency selection. + let candidates = if request.file_metas.is_empty() { all_files .values() - .filter(|file| { - !file.meta_ref().is_index_consistent_with_region( - &snapshot.metadata.column_metadatas, - &snapshot.index_options, - ) - }) .filter_map(|file| { let file_meta = current_file_meta(file.meta_ref().file_id)?; Some((file.clone(), file_meta)) @@ -253,6 +274,44 @@ impl RegionWorkerLoop { .collect::>() }; + let registry = match self.prepare_rebuild_registry(&snapshot, request.build_type.clone()) { + Ok(registry) => registry, + Err(err) => { + warn!(err; + "Failed to prepare index build, region_id: {}, build_type: {:?}, candidate_files: {}", + region_id, request.build_type, candidates.len() + ); + let err = Arc::new(err); + for result_sender in &request.result_senders { + let _ = result_sender + .send(Err( + BuildIndexAsyncSnafu { region_id }.into_error(err.clone()) + )) + .await; + } + sender.send(Err(BuildIndexAsyncSnafu { region_id }.into_error(err))); + return; + } + }; + let build_tasks = if request.file_metas.is_empty() { + // Check the version rather than the manifest: a publication committed + // to the manifest but not yet applied is still invisible to queries. + // Its build still holds the file, so the manual task waits for it. + candidates + .into_iter() + .filter(|(file, _)| { + index_needs_rebuild( + ®istry, + &snapshot, + file.meta_ref(), + request.build_type.clone(), + ) + }) + .collect::>() + } else { + candidates + }; + if build_tasks.is_empty() { debug!( "No files need to build index for region {}, request: {:?}", @@ -292,14 +351,20 @@ impl RegionWorkerLoop { let task = self.new_index_build_task( ®ion, &snapshot, - file_handle, - file_meta, + registry.clone(), + (file_handle, file_meta.as_ref().clone()), request.build_type.clone(), result_senders, ); - self.index_build_scheduler - .schedule_build(®ion.version_control, task) - .await; + if is_retry { + self.index_build_scheduler + .schedule_retry(®ion.version_control, task) + .await; + } else { + self.index_build_scheduler + .schedule_build(®ion.version_control, task) + .await; + } } drop(tx); @@ -336,10 +401,11 @@ impl RegionWorkerLoop { let incomplete = present .into_iter() .filter(|file_id| { - !manifest.files[file_id].is_index_consistent_with_region( + let file = &manifest.files[file_id]; + !file.is_index_consistent_with_region( &snapshot.metadata.column_metadatas, &snapshot.index_options, - ) + ) || index_needs_rebuild(®istry, &snapshot, file, build_type.clone()) }) .collect::>(); let result = if incomplete.is_empty() { @@ -389,7 +455,8 @@ impl RegionWorkerLoop { let manager = region.manifest_ctx.manifest_manager.read().await; let manifest = manager.manifest(); - let is_current = manifest.files.get(&file_meta.file_id) == Some(&file_meta); + let is_current = + manifest.files.get(&file_meta.file_id).map(AsRef::as_ref) == Some(&file_meta); if is_current { region.version_control.apply_edit( Some(RegionEdit { @@ -448,5 +515,349 @@ impl RegionWorkerLoop { ) { self.index_build_scheduler .on_task_stopped(region_id, request.file_id); + self.listener + .on_index_build_stopped(RegionFileId::new(region_id, request.file_id)) + .await; + } +} + +/// Rejects invalid current options before selecting files or scheduling any task. +fn validate_rebuild_plan( + registry: &IndexRegistry, + snapshot: &IndexBuildSnapshot, + build_type: IndexBuildType, +) -> Result<()> { + // New SSTs need built-in indexes even when optional providers are unavailable. + // Explicit maintenance follows current options, allowing removed sketch options + // to discard their historical descriptors without requiring a provider. + if matches!( + build_type, + IndexBuildType::Manual | IndexBuildType::SchemaChange + ) { + registry + .validate_schema_kinds(&snapshot.metadata) + .with_context(|_| ExternalSnafu { + context: "validating index handlers for rebuild".to_string(), + })?; + } + let plan = registry.plan(&PlanContext { + metadata: &snapshot.metadata, + index_options: &snapshot.index_options, + build_type, + existing_indexes: &[], + }); + plan.check_success().context(BuildIndexAsyncSnafu { + region_id: snapshot.metadata.region_id, + }) +} + +fn index_needs_rebuild( + registry: &IndexRegistry, + snapshot: &IndexBuildSnapshot, + file: &FileMeta, + build_type: IndexBuildType, +) -> bool { + let context = PlanContext { + metadata: &snapshot.metadata, + index_options: &snapshot.index_options, + build_type, + existing_indexes: &file.indexes, + }; + let plan = registry.plan(&context); + !plan.is_current(file, &context) +} + +#[cfg(test)] +mod tests { + use common_error::ext::ErrorExt; + use common_error::status_code::StatusCode; + use store_api::metadata::RegionMetadataBuilder; + + use super::*; + use crate::sst::file::ColumnIndexMetadata; + use crate::sst::index::handler::{ + BuildContext, IndexCreator, IndexKindHandler, PlanEntry, PlannedIndex, + }; + use crate::sst::index::kind::{ExtensionIndexKind, IndexKind}; + use crate::test_util::CreateRequestBuilder; + + #[derive(Debug)] + struct MaintenanceHandler; + + impl IndexKindHandler for MaintenanceHandler { + fn kinds(&self) -> &[IndexKind] { + &[IndexKind::Extension(ExtensionIndexKind::Hll)] + } + + fn validate( + &self, + _context: &PlanContext<'_>, + ) -> std::result::Result<(), common_error::ext::BoxedError> { + Ok(()) + } + + fn plan( + &self, + context: &PlanContext<'_>, + ) -> std::result::Result>, common_error::ext::BoxedError> + { + let existing = context.existing_indexes.iter().any(|column| { + column + .created_indexes + .contains(&crate::sst::file::IndexType::HllIndex) + }); + Ok((existing + && matches!( + context.build_type, + IndexBuildType::Manual | IndexBuildType::SchemaChange + )) + .then(|| Arc::new(Self) as Arc)) + } + } + + #[async_trait::async_trait] + impl PlannedIndex for MaintenanceHandler { + fn kinds(&self) -> &[IndexKind] { + IndexKindHandler::kinds(self) + } + + fn entries(&self) -> &[PlanEntry] { + &[] + } + + async fn build( + &self, + _context: &BuildContext<'_>, + ) -> std::result::Result>, common_error::ext::BoxedError> + { + Ok(None) + } + + fn is_current(&self, _file: &FileMeta, _context: &PlanContext<'_>) -> bool { + false + } + } + + fn snapshot_without_builtin_indexes() -> IndexBuildSnapshot { + let request = CreateRequestBuilder::new().build(); + let mut metadata = RegionMetadataBuilder::new(RegionId::new(1, 1)); + metadata.primary_key(request.primary_key); + for mut column in request.column_metadatas { + column.column_schema.set_inverted_index(false); + column.column_schema.unset_skipping_options().unwrap(); + metadata.push_column_metadata(column); + } + IndexBuildSnapshot { + metadata: Arc::new(metadata.build().unwrap()), + index_options: IndexOptions::default(), + } + } + + #[test] + fn test_rebuild_selection_uses_registered_currency_and_empty_maintenance_plans() { + let snapshot = snapshot_without_builtin_indexes(); + let builtin = + IndexRegistry::new(Default::default(), Default::default(), Default::default()); + let mut registered = builtin.clone(); + registered.register(Arc::new(MaintenanceHandler)).unwrap(); + let empty = FileMeta::default(); + let mut existing = empty.clone(); + existing.indexes.push(ColumnIndexMetadata { + column_id: 0, + created_indexes: smallvec::smallvec![crate::sst::file::IndexType::HllIndex], + ..Default::default() + }); + // Missing owners are rejected before selection; currency only compares + // the plans of registered handlers with the persisted indexes. + assert!( + builtin + .validate_required_kinds(&snapshot.metadata, [&existing]) + .is_err() + ); + for reason in [IndexBuildType::Manual, IndexBuildType::SchemaChange] { + assert!(!index_needs_rebuild( + ®istered, + &snapshot, + &empty, + reason.clone() + )); + assert!( + index_needs_rebuild(®istered, &snapshot, &existing, reason), + "an explicit empty maintenance plan still requests publication" + ); + } + for reason in [IndexBuildType::Flush, IndexBuildType::Compact] { + assert!(!index_needs_rebuild( + ®istered, + &snapshot, + &existing, + reason + )); + } + } + + #[derive(Debug)] + struct FailingPlanHandler; + + impl IndexKindHandler for FailingPlanHandler { + fn kinds(&self) -> &[IndexKind] { + &[IndexKind::Extension(ExtensionIndexKind::Hll)] + } + + fn validate( + &self, + _context: &PlanContext<'_>, + ) -> std::result::Result<(), common_error::ext::BoxedError> { + Ok(()) + } + + fn plan( + &self, + _context: &PlanContext<'_>, + ) -> std::result::Result>, common_error::ext::BoxedError> + { + Err(common_error::ext::BoxedError::new( + common_error::ext::PlainError::new( + "injected plan failure".into(), + common_error::status_code::StatusCode::Internal, + ), + )) + } + } + + #[test] + fn test_failed_plan_requires_rebuild_even_without_planned_entries() { + let snapshot = snapshot_without_builtin_indexes(); + let mut registry = + IndexRegistry::new(Default::default(), Default::default(), Default::default()); + registry.register(Arc::new(FailingPlanHandler)).unwrap(); + let file = FileMeta { + indexes: vec![ColumnIndexMetadata { + column_id: 0, + created_indexes: smallvec::smallvec![crate::sst::file::IndexType::HllIndex], + ..Default::default() + }], + ..Default::default() + }; + assert!( + registry + .validate_required_kinds(&snapshot.metadata, [&file]) + .is_ok() + ); + for reason in [IndexBuildType::Manual, IndexBuildType::SchemaChange] { + let plan = registry.plan(&PlanContext { + metadata: &snapshot.metadata, + index_options: &snapshot.index_options, + build_type: reason.clone(), + existing_indexes: &file.indexes, + }); + assert!(plan.entries().next().is_none()); + assert!(plan.check_success().is_err()); + assert!(index_needs_rebuild(®istry, &snapshot, &file, reason)); + } + } + + #[derive(Debug)] + struct DuplicateKindProvider; + + impl crate::sst::index::handler::IndexKindProvider for DuplicateKindProvider { + fn handlers(&self) -> Vec> { + vec![Arc::new(MaintenanceHandler), Arc::new(MaintenanceHandler)] + } + } + + #[test] + fn test_rebuild_registry_registration_failure_is_not_provider_absence() { + let config = crate::config::MitoConfig::default(); + assert!(index_registry(&config, None).is_ok()); + let error = index_registry(&config, Some(Arc::new(DuplicateKindProvider))).unwrap_err(); + assert_eq!(error.status_code(), StatusCode::InvalidArguments); + assert!(error.to_string().contains("registering index handlers")); + } + + #[test] + fn test_rebuild_preflight_rejects_plan_failure_before_file_selection() { + let snapshot = snapshot_without_builtin_indexes(); + let mut registry = + IndexRegistry::new(Default::default(), Default::default(), Default::default()); + assert!(validate_rebuild_plan(®istry, &snapshot, IndexBuildType::Manual).is_ok()); + registry.register(Arc::new(FailingPlanHandler)).unwrap(); + assert!(validate_rebuild_plan(®istry, &snapshot, IndexBuildType::Manual).is_err()); + } + + #[test] + fn test_required_handlers_cover_schema_and_both_persisted_locations() { + use crate::sst::file::IndexType; + let registry = + IndexRegistry::new(Default::default(), Default::default(), Default::default()); + for kind in [IndexType::HllIndex, IndexType::KllIndex, IndexType::MgIndex] { + let mut snapshot = snapshot_without_builtin_indexes(); + let schema = + &mut Arc::make_mut(&mut snapshot.metadata).column_metadatas[0].column_schema; + let (key, value) = match kind { + IndexType::HllIndex => (datatypes::schema::HLL_KEY, r#"{"precision":10}"#), + IndexType::KllIndex => (datatypes::schema::KLL_KEY, r#"{"capacity":64}"#), + IndexType::MgIndex => (datatypes::schema::MG_KEY, r#"{"max-counters":64}"#), + _ => unreachable!(), + }; + schema.mut_metadata().insert(key.into(), value.into()); + assert!( + registry + .validate_required_kinds(&snapshot.metadata, []) + .is_err() + ); + assert!(validate_rebuild_plan(®istry, &snapshot, IndexBuildType::Manual).is_err()); + assert!( + registry + .plan(&PlanContext { + metadata: &snapshot.metadata, + index_options: &snapshot.index_options, + build_type: IndexBuildType::Flush, + existing_indexes: &[], + }) + .check_success() + .is_ok() + ); + let snapshot = snapshot_without_builtin_indexes(); + for column_level in [false, true] { + let mut file = FileMeta::default(); + if column_level { + file.indexes.push(ColumnIndexMetadata { + column_id: 0, + created_indexes: smallvec::smallvec![kind], + ..Default::default() + }); + } else { + file.available_indexes.push(kind); + } + // Historical descriptors alone do not require the provider. + assert!( + validate_rebuild_plan(®istry, &snapshot, IndexBuildType::Manual).is_ok() + ); + assert!(index_needs_rebuild( + ®istry, + &snapshot, + &file, + IndexBuildType::Manual + )); + // Compaction's strict input validation remains available to its writer. + assert!( + registry + .validate_required_kinds(&snapshot.metadata, [&file]) + .is_err() + ); + } + } + let snapshot = snapshot_without_builtin_indexes(); + let legacy = FileMeta { + available_indexes: smallvec::smallvec![IndexType::VectorIndex], + ..Default::default() + }; + assert!(validate_rebuild_plan(®istry, &snapshot, IndexBuildType::Manual).is_ok()); + assert!( + registry + .validate_required_kinds(&snapshot.metadata, [&legacy]) + .is_ok() + ); } } diff --git a/src/mito2/src/worker/handle_reload.rs b/src/mito2/src/worker/handle_reload.rs index 67f6514fc77..b489c84aef9 100644 --- a/src/mito2/src/worker/handle_reload.rs +++ b/src/mito2/src/worker/handle_reload.rs @@ -12,7 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! Reloading a region that a manifest version conflict fenced. +//! Reloading a region fenced by a manifest version conflict, or by a failed tail replay round +//! that found a newer manifest. use std::sync::Arc; @@ -28,7 +29,8 @@ use crate::worker::RegionWorkerLoop; impl RegionWorkerLoop { /// Starts reloading the region from its manifest and this node's WAL if a manifest version - /// conflict fenced it and no catchup or reload of it runs. + /// conflict or a failed tail replay round that found a newer manifest fenced it, and no + /// catchup or reload of it runs. pub(crate) fn maybe_reload_region(&mut self, region_id: RegionId) { let Some(region) = self.regions.get_region(region_id) else { return; @@ -66,6 +68,9 @@ impl RegionWorkerLoop { let sender = self.sender.clone(); common_runtime::spawn_global(async move { let result = async { + // The fenced instance is stopped, but its last checkpoint may still be writing + // the manifest that the reload reads. + region.manifest_ctx.wait_for_checkpoint_after_stop().await; // The replay reads only what the log store has made durable. durability_barrier.wait(last_entry_id).await?; opener.open(&config, &wal).await @@ -115,7 +120,7 @@ impl RegionWorkerLoop { region.manifest_ctx.ignore_earlier_promotions(); // Requests waiting for the old instance are rejected as a close rejects them. self.fail_region_stalled_requests_as_not_found(®ion_id); - self.reject_region_edit_queue_as_not_found(region_id); + self.reject_region_requests_as_not_found(region_id); self.register_region(region); if let Some(state) = &self.series_index_task_state { state.wake(); diff --git a/src/mito2/src/worker/handle_remap.rs b/src/mito2/src/worker/handle_remap.rs index 24480f0f490..97b83026297 100644 --- a/src/mito2/src/worker/handle_remap.rs +++ b/src/mito2/src/worker/handle_remap.rs @@ -62,6 +62,7 @@ impl RegionWorkerLoop { self.config.clone(), self.object_store_manager.clone(), self.node_lease.is_enabled(), + self.file_ref_manager.file_metadata_interner.clone(), ); common_runtime::spawn_global(async move { let result = Self::fetch_and_remap_manifests( diff --git a/src/mito2/src/worker/handle_table_compaction.rs b/src/mito2/src/worker/handle_table_compaction.rs index eddc81636e4..b80f6f5c66c 100644 --- a/src/mito2/src/worker/handle_table_compaction.rs +++ b/src/mito2/src/worker/handle_table_compaction.rs @@ -93,7 +93,7 @@ fn is_current( && edit .files_to_remove .iter() - .all(|file| manifest.files.get(&file.file_id) == Some(file)) + .all(|file| manifest.files.get(&file.file_id).map(AsRef::as_ref) == Some(file)) } fn validate_files( @@ -139,7 +139,7 @@ fn validate_files( && file.level == 0 && outputs.insert(file.file_id) && !manifest.files.contains_key(&file.file_id) - && file.partition_expr == partition + && file.partition_expr.as_deref() == partition.as_ref() && file.preserve_row_sequence == trusted && file.sequence == sequence), InvalidRequestSnafu { diff --git a/src/mito2/src/worker/handle_tail_replay.rs b/src/mito2/src/worker/handle_tail_replay.rs new file mode 100644 index 00000000000..46d6b2f6911 --- /dev/null +++ b/src/mito2/src/worker/handle_tail_replay.rs @@ -0,0 +1,158 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Starting, applying and ending background replays of pending WAL tails. + +use std::sync::Arc; + +use common_telemetry::info; +use store_api::logstore::LogStore; +use store_api::storage::RegionId; +use tokio::sync::mpsc::Sender; + +use crate::config::IndexBuildMode; +use crate::manifest::action::{PendingTail, RegionEdit}; +use crate::region::MitoRegionRef; +use crate::request::{BackgroundNotify, BuildIndexRequest, OptionOutputTx, WorkerRequestWithTime}; +use crate::schedule::CancellableTaskState; +use crate::sst::index::IndexBuildType; +use crate::tail_replay::{TailReplayTask, notify_worker, should_replay_tails}; +use crate::worker::RegionWorkerLoop; + +impl RegionWorkerLoop { + /// Starts replaying the pending tails of the region, unless a replay of it still runs. + pub(crate) fn maybe_replay_tails(&mut self, region_id: RegionId) { + let Some(region) = self.regions.get_region(region_id) else { + return; + }; + if self.tail_replays.contains_key(®ion_id) || !should_replay_tails(®ion) { + return; + } + info!("Replaying the pending WAL tails of region {region_id}"); + let state = CancellableTaskState::new(); + let task = TailReplayTask { + sst_writer: self.tail_replay_limits.new_sst_writer( + ®ion, + self.config.clone(), + self.cache_manager.clone(), + ), + region, + node_id: self.file_ref_manager.node_id(), + wal: self.wal.clone(), + listener: self.listener.clone(), + request_sender: self.sender.clone(), + limits: self.tail_replay_limits.clone(), + state: state.clone(), + }; + self.tail_replays.insert(region_id, state); + common_runtime::spawn_global(task.run()); + } + + /// Cancels the tail replay of the region. The replay still reports its end. + pub(crate) fn cancel_tail_replay(&self, region_id: RegionId) { + if let Some(state) = self.tail_replays.get(®ion_id) { + let _ = state.request_cancel(); + } + } + + /// Applies a published tail replay unless its instance was replaced, or its version no + /// longer records `tails`, the tails it replayed, because a sync rebuilt it from a later + /// manifest. + pub(crate) async fn handle_tail_replay_committed( + &mut self, + region_id: RegionId, + region: MitoRegionRef, + edit: RegionEdit, + tails: Vec, + ) { + let registered = self.regions.get_region(region_id); + if !registered.is_some_and(|registered| Arc::ptr_eq(®istered, ®ion)) { + info!("Region {region_id} changed after its tails were published"); + return; + } + if region.version().pending_tails != tails { + info!("Region {region_id} no longer records the replayed tails, skipping their edit"); + return; + } + // A truncate uses the last entry id, which must not be below the flushed entry id. + if let Some(flushed_entry_id) = edit.flushed_entry_id + && flushed_entry_id > region.version_control.current().last_entry_id + { + region.version_control.set_entry_id(flushed_entry_id); + } + let file_metas = edit.files_to_add.clone(); + region + .version_control + .apply_edit(Some(edit), &[], region.file_purger.clone()); + if self.config.index.build_mode == IndexBuildMode::Async && !file_metas.is_empty() { + self.handle_rebuild_index( + BuildIndexRequest { + region_id, + build_type: IndexBuildType::Flush, + file_metas, + result_senders: Vec::new(), + }, + OptionOutputTx::new(None), + ) + .await; + } + self.schedule_compaction(®ion).await; + } + + /// Cancels the tail replay of the region unless `region`, the instance a role change made + /// unable to replay, was replaced since. + pub(crate) fn handle_region_cannot_replay(&self, region_id: RegionId, region: MitoRegionRef) { + let registered = self.regions.get_region(region_id); + if registered.is_some_and(|registered| Arc::ptr_eq(®istered, ®ion)) { + self.cancel_tail_replay(region_id); + } + } + + /// Forgets the ended tail replay of the region and starts another if it still has tails. + pub(crate) fn handle_tail_replay_ended(&mut self, region_id: RegionId) { + self.tail_replays.remove(®ion_id); + self.listener.on_tail_replay_ended(region_id); + self.maybe_replay_tails(region_id); + } +} + +/// Asks the worker to replay the pending tails of a region that became writable, if it has +/// any, without waiting. +pub(crate) fn notify_writable(sender: &Sender, region: &MitoRegionRef) { + if !region.version().has_pending_tails() { + return; + } + let sender = sender.clone(); + let region_id = region.region_id; + common_runtime::spawn_global(async move { + notify_worker(&sender, region_id, BackgroundNotify::RegionBecameWritable).await; + }); +} + +/// Asks the worker to cancel the tail replay of a region that a role change outside the worker +/// made unable to replay, if it has pending tails, without waiting. A replay the cancel reaches +/// after the region became writable again ends and the worker starts another. +pub(crate) fn notify_cannot_replay(sender: &Sender, region: &MitoRegionRef) { + if !region.version().has_pending_tails() { + return; + } + let sender = sender.clone(); + let region_id = region.region_id; + let notify = BackgroundNotify::RegionCannotReplay { + region: region.clone(), + }; + common_runtime::spawn_global(async move { + notify_worker(&sender, region_id, notify).await; + }); +} diff --git a/src/mito2/src/worker/handle_write.rs b/src/mito2/src/worker/handle_write.rs index 269859cb7e8..44beecaaad6 100644 --- a/src/mito2/src/worker/handle_write.rs +++ b/src/mito2/src/worker/handle_write.rs @@ -321,6 +321,13 @@ impl RegionWorkerLoop { .add_write_request_to_pending(sender_req); continue; } + if self.stalled_requests.has_staging_drain(®ion_id) { + self.stalling_count.add(1); + WRITE_STALL_TOTAL.inc(); + self.stalled_requests.push(sender_req); + self.listener.on_write_stall(); + continue; + } // Checks whether the region exists and is it stalling. if let hash_map::Entry::Vacant(e) = region_ctxs.entry(region_id) { @@ -365,8 +372,9 @@ impl RegionWorkerLoop { e.insert(region_ctx); } - RegionRoleState::Leader(RegionLeaderState::Altering) - | RegionRoleState::Leader(RegionLeaderState::Editing) => { + state @ (RegionRoleState::Leader(RegionLeaderState::Altering) + | RegionRoleState::Leader(RegionLeaderState::Editing) + | RegionRoleState::Leader(RegionLeaderState::EndingStaging)) => { // Editing is transient: queue the write so edit completion can drain it // before starting the next queued edit. debug!( @@ -377,6 +385,10 @@ impl RegionWorkerLoop { self.stalling_count.add(1); WRITE_STALL_TOTAL.inc(); self.stalled_requests.push(sender_req); + if state == RegionRoleState::Leader(RegionLeaderState::EndingStaging) { + self.stalled_requests.add_staging_drain(region_id); + self.listener.on_write_stall(); + } continue; } RegionRoleState::Leader(RegionLeaderState::EnteringStaging) => { @@ -487,6 +499,13 @@ impl RegionWorkerLoop { self.flush_scheduler.add_bulk_request_to_pending(bulk_req); continue; } + if self.stalled_requests.has_staging_drain(®ion_id) { + self.stalling_count.add(1); + WRITE_STALL_TOTAL.inc(); + self.stalled_requests.push_bulk(bulk_req); + self.listener.on_write_stall(); + continue; + } // Checks whether the region exists and is it stalling. if let hash_map::Entry::Vacant(e) = region_ctxs.entry(region_id) { @@ -521,8 +540,9 @@ impl RegionWorkerLoop { e.insert(region_ctx); } - RegionRoleState::Leader(RegionLeaderState::Altering) - | RegionRoleState::Leader(RegionLeaderState::Editing) => { + state @ (RegionRoleState::Leader(RegionLeaderState::Altering) + | RegionRoleState::Leader(RegionLeaderState::Editing) + | RegionRoleState::Leader(RegionLeaderState::EndingStaging)) => { // Editing is transient: queue the bulk write so edit completion can drain // it before starting the next queued edit. debug!( @@ -533,6 +553,10 @@ impl RegionWorkerLoop { self.stalling_count.add(1); WRITE_STALL_TOTAL.inc(); self.stalled_requests.push_bulk(bulk_req); + if state == RegionRoleState::Leader(RegionLeaderState::EndingStaging) { + self.stalled_requests.add_staging_drain(region_id); + self.listener.on_write_stall(); + } continue; } state => { diff --git a/src/object-store/src/config.rs b/src/object-store/src/config.rs index 2917bdbe817..90df5aeb0db 100644 --- a/src/object-store/src/config.rs +++ b/src/object-store/src/config.rs @@ -234,7 +234,8 @@ pub struct AzblobConnection { #[serde(skip_serializing)] pub account_key: SecretString, pub endpoint: String, - pub sas_token: Option, + #[serde(skip_serializing)] + pub sas_token: Option, } impl From<&AzblobConnection> for Azblob { @@ -248,7 +249,7 @@ impl From<&AzblobConnection> for Azblob { .account_key(connection.account_key.expose_secret()); if let Some(token) = &connection.sas_token { - builder = builder.sas_token(token); + builder = builder.sas_token(token.expose_secret()); }; builder @@ -445,6 +446,20 @@ mod tests { use super::*; use crate::config::ObjectStoreConfig; + #[test] + fn test_azblob_sas_token_is_not_disclosed() { + let config: AzblobConnection = + toml::from_str("container = 'test'\nsas_token = 'sas-secret'").unwrap(); + assert_eq!( + config.sas_token.as_ref().unwrap().expose_secret(), + "sas-secret" + ); + let serialized = toml::to_string(&config).unwrap(); + assert!(!serialized.contains("sas_token")); + assert!(!format!("{config:#?}").contains("sas-secret")); + assert!(serialized.contains("container = \"test\"")); + } + #[test] fn test_config_name() { let object_store_config = ObjectStoreConfig::default(); diff --git a/src/operator/src/batcher.rs b/src/operator/src/batcher.rs index bc9ef86b34e..5d05f6f6d8b 100644 --- a/src/operator/src/batcher.rs +++ b/src/operator/src/batcher.rs @@ -19,16 +19,26 @@ use async_trait::async_trait; use common_query::Output; use session::context::QueryContextRef; use table::metadata::TableInfoRef; -use tokio::sync::OwnedSemaphorePermit; use crate::error::Result; +/// Keeps a request's admission slot and conversion memory alive across its submissions. +pub type PendingRowsPermit = Arc; + /// Accepts prepared table writes without coupling the inserter to a batcher implementation. /// Schema creation, alteration and default evaluation remain the caller's responsibility. #[async_trait] pub trait PendingRowsBatcher: Send + Sync { - /// Acquires one slot per original request, shared by all of its table submissions. - async fn acquire(&self) -> Result>; + /// Returns the conversion memory limit, or zero when byte admission is disabled. + fn memory_limit_bytes(&self) -> u64 { + 0 + } + + /// Admits an original request before conversion, retaining its slot and estimated + /// Arrow memory across all table submissions. Memory acquisition must not wait + /// because the caller can already hold transport and decoded-row reservations. + /// `held_bytes` counts those reservations, which cannot be released during conversion. + async fn acquire(&self, estimated_bytes: usize, held_bytes: u64) -> Result; /// Submits rows according to the acknowledgement policy, retaining the slot until /// writing completes even when the response acknowledges queue admission only. @@ -39,6 +49,6 @@ pub trait PendingRowsBatcher: Send + Sync { table_info: TableInfoRef, batch: RecordBatch, ctx: QueryContextRef, - permit: Arc, + permit: PendingRowsPermit, ) -> Result; } diff --git a/src/operator/src/expr_helper.rs b/src/operator/src/expr_helper.rs index af6c960556c..359ab9722b5 100644 --- a/src/operator/src/expr_helper.rs +++ b/src/operator/src/expr_helper.rs @@ -836,6 +836,14 @@ pub(crate) fn to_alter_table_expr( .map_err(BoxedError::new) .context(ExternalSnafu)?; + // SQL must not reach the gRPC entry point's default-schema fallback. + ensure!( + !schema_name.is_empty(), + crate::error::SchemaNotFoundSnafu { + schema_info: &schema_name + } + ); + let kind = match alter_table.alter_operation { AlterTableOperation::AddConstraint(_) => { return NotSupportedSnafu { @@ -1130,6 +1138,13 @@ pub fn to_create_flow_task_expr( .map(|s| s.to_owned()) .unwrap_or(query_ctx.current_schema()); + ensure!( + !schema.is_empty(), + crate::error::SchemaNotFoundSnafu { + schema_info: &schema + } + ); + let sink_table_name = TableName { catalog_name: catalog, schema_name: schema, @@ -1153,6 +1168,13 @@ pub fn to_create_flow_task_expr( .map(|s| s.to_string()) .unwrap_or(query_ctx.current_schema()); + ensure!( + !schema.is_empty(), + crate::error::SchemaNotFoundSnafu { + schema_info: &schema + } + ); + let table_name = TableName { catalog_name: catalog, schema_name: schema, @@ -1453,6 +1475,39 @@ SELECT * FROM tql; ); } + #[test] + fn test_create_flow_requires_schema_for_unqualified_tables() { + let ctx = QueryContext::arc(); + ctx.set_current_schema(""); + for (sink, source, valid) in [ + ("sink", "public.source", false), + ("public.sink", "source", false), + ("public.sink", "public.source", true), + ] { + let sql = format!("CREATE FLOW test_flow SINK TO {sink} AS SELECT * FROM {source}"); + let stmt = ParserContext::create_with_dialect( + &sql, + &GreptimeDbDialect {}, + ParseOptions::default(), + ) + .unwrap() + .pop() + .unwrap(); + let Statement::CreateFlow(flow) = stmt else { + unreachable!() + }; + let result = to_create_flow_task_expr(flow, &ctx); + if valid { + assert!(result.is_ok(), "{result:?}"); + } else { + assert!( + matches!(result, Err(crate::error::Error::SchemaNotFound { ref schema_info, .. }) if schema_info.is_empty()), + "{result:?}" + ); + } + } + } + #[test] fn test_create_flow_expr() { let sql = r" diff --git a/src/operator/src/insert.rs b/src/operator/src/insert.rs index c0792ee56e3..e69063734db 100644 --- a/src/operator/src/insert.rs +++ b/src/operator/src/insert.rs @@ -16,7 +16,7 @@ mod otlp; pub use otlp::MetricInsertPreflight; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, LazyLock}; use std::time::{Duration, Instant}; @@ -55,7 +55,7 @@ use common_meta::rpc::ddl::TriggerReason; use common_query::Output; use common_query::native_histogram::{is_native_histogram_value_type, native_histogram_value_type}; use common_query::prelude::{ - EXEMPLAR_LABELS_COLUMN, EXEMPLAR_METRIC_COLUMN, greptime_timestamp, greptime_value, + EXEMPLAR_METRIC_COLUMN, greptime_exemplar_labels, greptime_timestamp, greptime_value, }; use common_telemetry::tracing_context::TracingContext; use common_telemetry::{debug, error, warn}; @@ -105,7 +105,7 @@ use crate::region_req_factory::RegionRequestFactory; use crate::req_convert::common::preprocess_row_insert_requests; use crate::req_convert::insert::{ ColumnToRow, ImpureDefaultFiller, RowToRegion, StatementToRegion, TableToRegion, - fill_reqs_with_impure_default, rows_to_record_batch, + fill_reqs_with_impure_default, rows_to_record_batch, value_matches_type, }; use crate::statement::StatementExecutor; use crate::write_dispatcher::{WriteDispatcher, WriteDispatcherRef}; @@ -120,6 +120,11 @@ pub struct Inserter { /// When `false`, missing tables are never auto-created regardless of the /// per-request `auto_create_table` hint. When `true`, the hint still applies. auto_create_table: bool, + /// Time index unit for physical metric tables created by auto table creation. + /// New logical tables created under such a physical table inherit its unit, + /// so all metric-engine ingestion paths write timestamps in this unit. + /// Defaults to milliseconds for compatibility. + auto_create_physical_table_timestamp_unit: TimeUnit, pending_rows_batcher: Option>, /// Rows handed to detached flow mirror tasks that have not finished yet. /// Bounds how much row data in-flight mirror writes may keep alive, see @@ -342,11 +347,18 @@ impl Inserter { node_manager, table_flownode_set_cache, auto_create_table, + auto_create_physical_table_timestamp_unit: TimeUnit::Millisecond, pending_rows_batcher: None, mirror_pending_rows: Arc::new(AtomicU64::new(0)), } } + /// Sets the time index unit for auto-created physical metric tables. + pub fn with_auto_create_physical_table_timestamp_unit(mut self, unit: TimeUnit) -> Self { + self.auto_create_physical_table_timestamp_unit = unit; + self + } + /// Installs the shared dispatcher used by direct writes and batch flushes. pub fn with_write_dispatcher(mut self, dispatcher: WriteDispatcherRef) -> Self { self.write_dispatcher = dispatcher; @@ -411,9 +423,10 @@ impl Inserter { reason: "invalid exemplar destination" } ); - self.bind_exemplar_table_ids(&mut requests, &ctx, sources) - .await?; + validate_column_count_match(&requests)?; preprocess_row_insert_requests(&mut requests.inserts)?; + self.bind_exemplar_table_ids(&mut requests, &ctx, sources, statement_executor) + .await?; self.handle_row_inserts_with_create_type( requests, ctx, @@ -430,9 +443,10 @@ impl Inserter { requests: &mut RowInsertRequests, ctx: &QueryContextRef, sources: &[TableInfoRef], + statement_executor: &StatementExecutor, ) -> Result<()> { - validate_column_count_match(requests)?; let mut tables = HashMap::new(); + let mut required_tags = BTreeMap::>::new(); for info in sources.iter().filter(|info| { info.catalog_name == ctx.current_catalog() && info.schema_name == ctx.current_schema() }) { @@ -452,6 +466,7 @@ impl Inserter { let Some(rows) = request.rows.as_mut() else { continue; }; + validate_exemplar_rows(rows)?; ensure!( !rows .schema @@ -497,6 +512,16 @@ impl Inserter { ), } ); + let tags = required_tags.entry(name.clone()).or_default(); + for (column, value) in rows.schema.iter().zip(&row.values) { + if column.semantic_type == SemanticType::Tag as i32 + && column.column_name != EXEMPLAR_METRIC_COLUMN + && column.column_name != DATA_SCHEMA_TSID_COLUMN_NAME + && value.value_data.is_some() + { + tags.insert(column.column_name.clone()); + } + } row.values.push(ValueData::U32Value(info.table_id()).into()); } rows.schema.push(api::v1::helper::tag_column_schema( @@ -504,6 +529,83 @@ impl Inserter { ColumnDataType::Uint32, )); } + + let mut alters = Vec::new(); + let mut refresh = Vec::new(); + for (name, tags) in required_tags { + let info = &tables[&name]; + let tags = tags + .into_iter() + .map(|name| api::v1::helper::tag_column_schema(&name, ColumnDataType::String)) + .collect::>(); + if let Some(add_columns) = exemplar_source_add_columns(info, &tags)? { + ensure!( + self.auto_create_disabled_reason(ctx)?.is_none(), + InvalidInsertRequestSnafu { + reason: format!( + "exemplar source {name} requires new tags, but automatic DDL is disabled" + ), + } + ); + alters.push(( + info.table_id(), + AlterTableExpr { + catalog_name: info.catalog_name.clone(), + schema_name: info.schema_name.clone(), + table_name: name, + kind: Some(Kind::AddColumns(add_columns)), + }, + )); + refresh.push((info, tags)); + } + } + if !alters.is_empty() { + if let Some(table) = self + .get_table( + ctx.current_catalog(), + &ctx.current_schema(), + METRICS_EXEMPLARS_TABLE, + ) + .await? + { + // Reject an incompatible destination before changing any source schema. + validate_exemplar_table(&table.table_info())?; + } + statement_executor + .alter_logical_tables_with_ids(alters, ctx.clone(), TriggerReason::AutoAlter) + .await?; + for (pinned, tags) in refresh { + let table = self + .get_table(&pinned.catalog_name, &pinned.schema_name, &pinned.name) + .await? + .with_context(|| InvalidInsertRequestSnafu { + reason: format!( + "exemplar source {} disappeared during reconciliation", + pinned.name + ), + })?; + let info = table.table_info(); + ensure!( + info.table_id() == pinned.table_id(), + InvalidInsertRequestSnafu { + reason: format!( + "exemplar source {} changed during reconciliation", + pinned.name + ), + } + ); + // Concurrent ADD IF NOT EXISTS can succeed on names alone; verify their definitions. + ensure!( + exemplar_source_add_columns(&info, &tags)?.is_none(), + InvalidInsertRequestSnafu { + reason: format!( + "exemplar source {} is still missing inherited tags", + pinned.name + ), + } + ); + } + } Ok(()) } @@ -654,7 +756,7 @@ impl Inserter { .values() .map(|info| (info.name.as_str(), info)) .collect::>(); - let mut prepared = Vec::with_capacity(requests.inserts.len()); + let mut estimated_bytes = Some(0usize); for request in &mut requests.inserts { let table_info = by_name @@ -666,8 +768,34 @@ impl Inserter { continue; }; ImpureDefaultFiller::new((*table_info).clone())?.fill_rows(rows); - let batch = rows_to_record_batch(rows, table_info)?; - prepared.push(((*table_info).clone(), batch)); + if batcher.memory_limit_bytes() != 0 { + estimated_bytes = estimated_bytes.and_then(|bytes| { + bytes.checked_add(crate::req_convert::insert::estimated_record_batch_bytes( + rows, table_info, + )?) + }); + } + } + // Reserve once for the whole request before constructing any Arrow arrays. + // PB rows and the converted batches coexist, so this is additional memory. + let permit = if requests.inserts.is_empty() { + None + } else { + Some( + batcher + .acquire( + estimated_bytes.unwrap_or(usize::MAX), + ctx.held_memory_bytes(), + ) + .await?, + ) + }; + let mut prepared = Vec::with_capacity(requests.inserts.len()); + for request in &requests.inserts { + let table_info = by_name[request.table_name.as_str()]; + if let Some(rows) = &request.rows { + prepared.push((table_info.clone(), rows_to_record_batch(rows, table_info)?)); + } } // Preserve the existing meter input and original request boundary. These // envelopes are only for accounting; routing happens after batching. @@ -698,11 +826,8 @@ impl Inserter { .await .context(WriteRejectedSnafu)?; prepared.retain(|(_, batch)| batch.num_rows() != 0); - let results = if prepared.is_empty() { - Vec::new() - } else { + let results = if let Some(permit) = permit { // One original request shares admission across all table submissions. - let permit = batcher.acquire().await?; let submissions = prepared.into_iter().map(|(info, batch)| { // Route to the same target database used for admission above. let mut target_ctx = ctx.fork(); @@ -712,6 +837,8 @@ impl Inserter { }); // Observe every table completion even when another table fails. future::join_all(submissions).await + } else { + Vec::new() }; let mut affected_rows = 0; let mut meta = OutputMeta::new_with_cost(write_cost as _); @@ -1753,7 +1880,9 @@ impl Inserter { let default_schema = vec![ ColumnSchema { column_name: greptime_timestamp().to_string(), - datatype: ColumnDataType::TimestampMillisecond as _, + datatype: api::helper::timestamp_datatype( + self.auto_create_physical_table_timestamp_unit, + ) as _, semantic_type: SemanticType::Timestamp as _, datatype_extension: None, options: None, @@ -2195,6 +2324,95 @@ fn validate_column_count_match(requests: &RowInsertRequests) -> Result<()> { Ok(()) } +fn validate_exemplar_rows(rows: &Rows) -> Result<()> { + let mut names = HashSet::new(); + for (index, column) in rows.schema.iter().enumerate() { + let data_type: datatypes::prelude::ConcreteDataType = + api::helper::ColumnDataTypeWrapper::try_new( + column.datatype, + column.datatype_extension.clone(), + ) + .context(crate::error::ColumnDataTypeSnafu)? + .into(); + let valid = match column.column_name.as_str() { + DATA_SCHEMA_TSID_COLUMN_NAME => { + data_type == datatypes::prelude::ConcreteDataType::uint64_datatype() + && column.semantic_type == SemanticType::Tag as i32 + } + EXEMPLAR_METRIC_COLUMN => { + data_type.is_string() && column.semantic_type == SemanticType::Tag as i32 + } + name if name == greptime_timestamp() => { + data_type.is_timestamp() && column.semantic_type == SemanticType::Timestamp as i32 + } + name if name == greptime_value() => { + data_type == datatypes::prelude::ConcreteDataType::float64_datatype() + && column.semantic_type == SemanticType::Field as i32 + } + name if name == greptime_exemplar_labels() => { + data_type.is_json2() && column.semantic_type == SemanticType::Field as i32 + } + _ => data_type.is_string() && column.semantic_type == SemanticType::Tag as i32, + }; + ensure!( + valid && names.insert(column.column_name.as_str()), + InvalidInsertRequestSnafu { + reason: format!( + "invalid or duplicate exemplar column {}", + column.column_name + ), + } + ); + ensure!( + rows.rows.iter().all(|row| value_matches_type( + &row.values[index], + column.datatype, + column.datatype_extension.as_ref(), + )), + InvalidInsertRequestSnafu { + reason: format!("invalid value for exemplar column {}", column.column_name), + } + ); + } + ensure!( + [ + DATA_SCHEMA_TSID_COLUMN_NAME, + EXEMPLAR_METRIC_COLUMN, + greptime_timestamp(), + greptime_value(), + greptime_exemplar_labels(), + ] + .iter() + .all(|name| names.contains(name)), + InvalidInsertRequestSnafu { + reason: "missing required exemplar columns", + } + ); + Ok(()) +} + +fn exemplar_source_add_columns( + info: &TableInfo, + tags: &[ColumnSchema], +) -> Result> { + for tag in tags { + if let Some(index) = info.meta.schema.column_index_by_name(&tag.column_name) { + let column = &info.meta.schema.column_schemas()[index]; + // Existing source defaults may already be part of the normalized series identity. + ensure!( + info.meta.primary_key_indices.contains(&index) && column.data_type.is_string(), + InvalidInsertRequestSnafu { + reason: format!( + "exemplar source {} column {} must be a string tag", + info.name, tag.column_name + ), + } + ); + } + } + expr_helper::extract_add_columns_expr(&info.meta.schema, ColumnExpr::from_column_schemas(tags)) +} + /// Validates the fixed exemplar contract before writing or scanning a shared table. pub fn validate_exemplar_table(table: &TableInfo) -> Result<()> { let meta = &table.meta; @@ -2271,11 +2489,11 @@ pub fn validate_exemplar_table(table: &TableInfo) -> Result<()> { reason: "exemplar value must be a DOUBLE field" } ); - let (labels_index, labels) = lookup(EXEMPLAR_LABELS_COLUMN)?; + let (labels_index, labels) = lookup(greptime_exemplar_labels())?; ensure!( labels.data_type.is_json2() && !meta.primary_key_indices.contains(&labels_index), InvalidInsertRequestSnafu { - reason: "exemplar_labels must be a JSON2 field" + reason: format!("{} must be a JSON2 field", greptime_exemplar_labels()) } ); if let Some(extension) = labels @@ -2304,7 +2522,7 @@ pub fn validate_exemplar_table(table: &TableInfo) -> Result<()> { greptime_timestamp(), greptime_value(), EXEMPLAR_METRIC_COLUMN, - EXEMPLAR_LABELS_COLUMN, + greptime_exemplar_labels(), DATA_SCHEMA_TABLE_ID_COLUMN_NAME, DATA_SCHEMA_TSID_COLUMN_NAME, ] @@ -4103,7 +4321,7 @@ mod tests { #[async_trait::async_trait] impl PendingRowsBatcher for UnexpectedBatcher { - async fn acquire(&self) -> Result> { + async fn acquire(&self, _: usize, _: u64) -> Result { panic!("empty writes must not acquire batch admission") } @@ -4112,7 +4330,7 @@ mod tests { _table_info: TableInfoRef, _batch: arrow::record_batch::RecordBatch, _ctx: QueryContextRef, - _permit: Arc, + _permit: crate::batcher::PendingRowsPermit, ) -> Result { panic!("empty writes must not submit a batch") } diff --git a/src/operator/src/insert/otlp.rs b/src/operator/src/insert/otlp.rs index 9405480d3a2..aa6bf665eb6 100644 --- a/src/operator/src/insert/otlp.rs +++ b/src/operator/src/insert/otlp.rs @@ -118,12 +118,9 @@ impl Inserter { tables.insert(backing.clone(), table); } physical_columns.entry(backing.as_str()).or_insert_with(|| { - PhysicalColumns::new( - tables[backing] - .as_ref() - .map(table_columns) - .unwrap_or_else(default_physical_columns), - ) + PhysicalColumns::new(tables[backing].as_ref().map(table_columns).unwrap_or_else( + || default_physical_columns(self.auto_create_physical_table_timestamp_unit), + )) }); } // Match ordinary insertion: impure defaults are evaluated once per table @@ -176,7 +173,9 @@ impl Inserter { let unit = target .and_then(table_time_index_unit) .or_else(|| physical.and_then(table_time_index_unit)) - .unwrap_or(TimeUnit::Millisecond); + // The physical table will be auto-created in the configured + // unit, so validate its rows in that unit as well. + .unwrap_or(self.auto_create_physical_table_timestamp_unit); if let Some(rows) = &mut request.rows { convert_rows_time_unit(rows, unit)?; } @@ -378,13 +377,13 @@ fn table_columns(table: &TableRef) -> Vec<(DataColumn, SemanticType)> { .collect() } -fn default_physical_columns() -> Vec<(DataColumn, SemanticType)> { +fn default_physical_columns(unit: TimeUnit) -> Vec<(DataColumn, SemanticType)> { use datatypes::prelude::ConcreteDataType; vec![ ( DataColumn::new( greptime_timestamp(), - ConcreteDataType::timestamp_millisecond_datatype(), + ConcreteDataType::timestamp_datatype(unit), false, ) .with_time_index(true), @@ -698,7 +697,7 @@ mod tests { #[test] fn test_preflight_omissions_defaults_and_explicit_nulls() { - let mut columns = default_physical_columns(); + let mut columns = default_physical_columns(TimeUnit::Millisecond); columns.push(( DataColumn::new("host", ConcreteDataType::string_datatype(), false) .with_default_constraint(Some(ColumnDefaultConstraint::Value(DataValue::String( @@ -739,7 +738,7 @@ mod tests { #[test] fn test_preflight_existing_columns_and_physical_definitions() { - let columns = default_physical_columns(); + let columns = default_physical_columns(TimeUnit::Millisecond); let mut rows = Rows { schema: vec![ColumnSchema { column_name: greptime_timestamp().into(), @@ -805,7 +804,7 @@ mod tests { (defaulted, None), (impure, Some("impure default for physical field `impure`")), ] { - let mut columns = default_physical_columns(); + let mut columns = default_physical_columns(TimeUnit::Millisecond); columns.extend((0..1024).map(|index| { ( DataColumn::new( diff --git a/src/operator/src/req_convert/insert.rs b/src/operator/src/req_convert/insert.rs index 50000e8dc93..5e1757f2c03 100644 --- a/src/operator/src/req_convert/insert.rs +++ b/src/operator/src/req_convert/insert.rs @@ -25,7 +25,7 @@ pub use column_to_row::ColumnToRow; pub(crate) use fill_impure_default::ImpureDefaultFiller; pub use fill_impure_default::fill_reqs_with_impure_default; pub use row_to_batch::rows_to_record_batch; -pub(crate) use row_to_batch::value_matches_type; +pub(crate) use row_to_batch::{estimated_record_batch_bytes, value_matches_type}; pub use row_to_region::RowToRegion; use snafu::{OptionExt, ResultExt}; pub use stmt_to_region::StatementToRegion; diff --git a/src/operator/src/req_convert/insert/row_to_batch.rs b/src/operator/src/req_convert/insert/row_to_batch.rs index 440c66c93a0..0e92e74e859 100644 --- a/src/operator/src/req_convert/insert/row_to_batch.rs +++ b/src/operator/src/req_convert/insert/row_to_batch.rs @@ -12,6 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. +mod memory; + use std::collections::HashMap; use api::helper::{ @@ -29,6 +31,8 @@ use table::metadata::TableInfo; use crate::error::{self, Result}; +pub(crate) use memory::estimated_record_batch_bytes; + /// Converts prepared rows in target-schema order without Prom-specific renaming. /// /// Missing columns evaluate their defaults once per conversion. Explicit nulls @@ -198,10 +202,14 @@ mod tests { use std::sync::Arc; use std::thread::yield_now; + use api::helper::to_grpc_value; use api::v1::value::ValueData; use api::v1::{ColumnDataType, ColumnSchema as ProtoColumnSchema, Row, Value}; use arrow::array::{Array, Int32Array, StringArray}; + use datatypes::extension::json::{Json2ExtensionType, JsonMetadata}; + use datatypes::json::{JsonSettings, JsonTypeHint}; use datatypes::schema::{ColumnDefaultConstraint, ColumnSchema, Schema}; + use datatypes::types::json_type::JsonNativeType; use datatypes::value::Value as DtValue; use table::metadata::{TableInfoBuilder, TableMetaBuilder}; @@ -260,6 +268,230 @@ mod tests { (rows, schema) } + fn json_column(metadata: JsonMetadata) -> ColumnSchema { + let mut column = ColumnSchema::new( + "json", + ConcreteDataType::json2(JsonNativeType::object()), + true, + ); + column.with_extension_type(&Json2ExtensionType::new(Arc::new(metadata))); + column + } + + fn json_rows(column: &ColumnSchema, values: impl IntoIterator) -> Rows { + let (kind, extension) = ColumnDataTypeWrapper::try_from(column.data_type.clone()) + .unwrap() + .to_parts(); + let mut input = source(&column.name, kind); + input.datatype_extension = extension; + Rows { + schema: vec![input], + rows: values + .into_iter() + .map(|value| Row { + values: vec![to_grpc_value(value)], + }) + .collect(), + } + } + + #[test] + fn test_conversion_estimate_covers_destination_padding_and_defaults() { + let (mut rows, schema) = fixture(); + rows.rows[0].values[0] = ValueData::StringValue("x".repeat(10_000)).into(); + rows.rows.resize(1000, rows.rows[0].clone()); + let mut columns = schema.column_schemas().to_vec(); + columns.extend((0..2048).map(|i| { + ColumnSchema::new( + format!("tag_{i}"), + ConcreteDataType::string_datatype(), + true, + ) + })); + columns.push( + ColumnSchema::new("default", ConcreteDataType::string_datatype(), true) + .with_default_constraint(Some(ColumnDefaultConstraint::Value(DtValue::String( + "default".repeat(1000).into(), + )))) + .unwrap(), + ); + let table = table_info(Schema::new(columns)); + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!(estimate >= batch.get_array_memory_size()); + assert!(estimate > 4 * 1024 * 1024); + } + + #[test] + fn test_conversion_estimate_covers_nested_values_and_defaults() { + use datatypes::value::ListValue; + + let value = DtValue::List(ListValue::new( + vec![ + DtValue::String("a".into()), + DtValue::String("b".repeat(30_000).into()), + ], + Arc::new(ConcreteDataType::string_datatype()), + )); + let datatype = value.data_type(); + let table = table_info(Schema::new(vec![ + ColumnSchema::new("input", datatype.clone(), true), + ColumnSchema::new("default", datatype.clone(), true) + .with_default_constraint(Some(ColumnDefaultConstraint::Value(value.clone()))) + .unwrap(), + ])); + let (kind, extension) = ColumnDataTypeWrapper::try_from(datatype) + .unwrap() + .to_parts(); + let mut input = source("input", kind); + input.datatype_extension = extension; + let rows = Rows { + schema: vec![input], + rows: vec![ + Row { + values: vec![to_grpc_value(value)] + }; + 10 + ], + }; + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!(estimate >= batch.get_array_memory_size()); + } + + #[test] + fn test_conversion_estimate_covers_json2_expansion() { + for (legacy, max_paths) in [ + (false, None), + (false, Some(0)), + (false, Some(1)), + (true, None), + ] { + let settings = JsonSettings::try_new( + (0..200) + .map(|i| JsonTypeHint { + path: vec![format!("hint_{i}")], + data_type: ConcreteDataType::string_datatype(), + inverted_index: false, + skipping_index: None, + fulltext_index: None, + }) + .collect(), + max_paths, + ) + .unwrap(); + let column = json_column(if legacy { + JsonMetadata::new_v1(settings.clone()) + } else { + JsonMetadata::new(settings.clone()) + }); + let rows = json_rows( + &column, + (0..200).map(|i| { + settings + .encode(serde_json::json!({ + format!("field_{i}"): "x".repeat(i * 100), + "nested": [{"name": "y".repeat(i)}, {"other": i}], + })) + .unwrap() + }), + ); + let table = table_info(Schema::new(vec![column])); + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!( + estimate >= batch.get_array_memory_size(), + "legacy={legacy}, max_paths={max_paths:?}" + ); + } + } + + #[test] + fn test_conversion_estimate_respects_json2_expansion_limit() { + for (legacy, max_paths) in [ + (false, Some(0)), + (false, Some(100)), + (false, None), + (true, Some(0)), + ] { + let settings = JsonSettings::try_new(vec![], max_paths).unwrap(); + let column = json_column(if legacy { + JsonMetadata::new_v1(settings.clone()) + } else { + JsonMetadata::new(settings.clone()) + }); + let rows = json_rows( + &column, + (0..1000).map(|i| { + settings + .encode(serde_json::json!({format!("field_{i}"): "a"})) + .unwrap() + }), + ); + let table = table_info(Schema::new(vec![column])); + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!( + estimate >= batch.get_array_memory_size(), + "legacy={legacy}, max_paths={max_paths:?}" + ); + if !legacy { + // Small batches must fit this quota even when every row has a different key. + assert!(estimate < 64 * 1024 * 1024, "max_paths={max_paths:?}"); + } + } + } + + #[test] + fn test_conversion_estimate_covers_nested_json2_paths_and_defaults() { + // One selected array path can contain many fields, with padding for every element. + let items = (0..256) + .map(|i| serde_json::json!({format!("field_{i}"): "a"})) + .collect::>(); + for use_default in [false, true] { + let settings = + JsonSettings::try_new(vec![], Some(if use_default { 0 } else { 1 })).unwrap(); + let value = settings + .encode(serde_json::json!({"outer": {"inner": {"items": items}}})) + .unwrap(); + let mut column = json_column(JsonMetadata::new(settings)); + let rows = if use_default { + column = column + .with_default_constraint(Some(ColumnDefaultConstraint::Value(value))) + .unwrap(); + Rows { + schema: vec![], + rows: vec![Row { values: vec![] }; 2], + } + } else { + json_rows(&column, [value.clone(), value]) + }; + let table = table_info(Schema::new(vec![column])); + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!( + estimate >= batch.get_array_memory_size(), + "use_default={use_default}" + ); + } + } + + #[test] + fn test_conversion_estimate_covers_small_json2_view_buffers() { + let settings = JsonSettings::default(); + let column = json_column(JsonMetadata::new_v1(settings.clone())); + let rows = json_rows( + &column, + [settings + .encode(serde_json::json!({"value": "x".repeat(13)})) + .unwrap()], + ); + let table = table_info(Schema::new(vec![column])); + let estimate = estimated_record_batch_bytes(&rows, &table).unwrap(); + let batch = rows_to_record_batch(&rows, &table).unwrap(); + assert!(estimate >= batch.get_array_memory_size()); + } + #[test] fn test_reorder_multifield_and_defaults() { let (rows, schema) = fixture(); @@ -349,11 +581,7 @@ mod tests { #[test] fn test_json2_expanded_schema() { - use api::helper::to_grpc_value; - use datatypes::extension::json::{Json2ExtensionType, JsonMetadata}; - use datatypes::json::JsonSettings; use datatypes::schema::SchemaBuilder; - use datatypes::types::json_type::JsonNativeType; let settings = JsonSettings::default(); let mut column_schema = ColumnSchema::new( diff --git a/src/operator/src/req_convert/insert/row_to_batch/memory.rs b/src/operator/src/req_convert/insert/row_to_batch/memory.rs new file mode 100644 index 00000000000..7fdff3feba7 --- /dev/null +++ b/src/operator/src/req_convert/insert/row_to_batch/memory.rs @@ -0,0 +1,345 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::{HashMap, HashSet}; + +use api::v1::value::ValueData; +use api::v1::{JsonValue, Rows, json_value}; +use arrow::datatypes::DataType; +use arrow_schema::extension::ExtensionType; +use datatypes::extension::json::Json2ExtensionType; +use datatypes::json::JSON2_DEFAULT_MAX_AUTO_EXPANDED_PATHS; +use datatypes::json::value::JsonVariant; +use datatypes::schema::ColumnDefaultConstraint; +use datatypes::value::Value; +use datatypes::vectors::json; +use table::metadata::TableInfo; + +const ARRAY_OVERHEAD: usize = 512; +const VIEW_BLOCK_BYTES: usize = 8192; +const VALUE_SCRATCH_BYTES: usize = 256; + +/// Estimates additional memory for conversion, including destination columns absent +/// from the request. Overflow fails closed. Source rows remain separately reserved. +/// +/// This budgets buffers, capacity growth and JSON2 conversion scratch, not allocator +/// RSS. shortcut: account for flush concatenation, partitioning and IPC separately +/// when extending admission to the entire flush pipeline. +pub(crate) fn estimated_record_batch_bytes(rows: &Rows, table: &TableInfo) -> Option { + let source = rows + .schema + .iter() + .enumerate() + .map(|(i, column)| (column.column_name.as_str(), i)) + .collect::>(); + let mut total = 0usize; + for (column, field) in table + .meta + .schema + .column_schemas() + .iter() + .zip(table.meta.schema.arrow_schema().fields()) + { + let mut values = ValueSize::default(); + let source_index = source.get(column.name.as_str()); + if let Some(&index) = source_index { + for row in &rows.rows { + values.proto(row.values.get(index)?.value_data.as_ref())?; + } + } else if let Some(ColumnDefaultConstraint::Value(value)) = column.default_constraint() { + values.default_value(value)?; + values.payload = values.payload.checked_mul(rows.rows.len())?; + values.items = values.items.checked_mul(rows.rows.len())?; + values.scratch = values.scratch.checked_mul(rows.rows.len())?; + } + + let count = rows.rows.len().checked_add(values.items)?; + let extension = column.extension_type::().ok()?; + // Only supplied v2 JSON2 values use the settings-aware builder; defaults use the datatype. + let settings = extension + .as_ref() + .filter(|extension| { + source_index.is_some() + && column.data_type.is_json2() + && extension.metadata().is_version_2() + }) + .map(|extension| extension.metadata().json_settings()); + let (mut bytes, paths) = if let Some(settings) = settings { + let limit = settings + .max_auto_expanded_paths() + .unwrap_or(JSON2_DEFAULT_MAX_AUTO_EXPANDED_PATHS); + ( + array_bytes(&json::json2_physical_data_type(settings), count)?, + values.expanded_paths(usize::try_from(limit).ok()?)?, + ) + } else { + (array_bytes(field.data_type(), count)?, values.paths.len()) + }; + // Each selected path can need buffers for every row or nested element. + let path_bytes = count + .checked_add(1)? + .checked_mul(48)? + .checked_add(3 * ARRAY_OVERHEAD + VIEW_BLOCK_BYTES)?; + bytes = bytes.checked_add(paths.checked_mul(path_bytes)?)?; + // Builders can double capacity; payload and JSON scratch can coexist with + // the final arrays. Fixed per-array slack covers alignment and empty buffers. + total = total + .checked_add(bytes.checked_mul(2)?)? + .checked_add(field.size().checked_mul(2)?)? + .checked_add(values.payload.checked_mul(4)?)? + .checked_add(values.scratch)?; + } + Some(total) +} + +fn array_bytes(data_type: &DataType, count: usize) -> Option { + let width = match data_type { + DataType::Null => 0, + DataType::Boolean => 1, + DataType::Utf8 | DataType::Binary | DataType::List(_) => 4, + DataType::LargeUtf8 | DataType::LargeBinary | DataType::LargeList(_) => 8, + DataType::Utf8View | DataType::BinaryView => 16, + DataType::Struct(_) | DataType::FixedSizeList(_, _) | DataType::Dictionary(_, _) => 0, + DataType::FixedSizeBinary(width) => usize::try_from(*width).ok()?, + other => other.primitive_width()?, + }; + // A byte per validity bit is deliberately conservative, including null padding. + let mut bytes = count + .checked_add(1)? + .checked_mul(width.checked_add(1)?)? + .checked_add(ARRAY_OVERHEAD)?; + if matches!(data_type, DataType::Utf8View | DataType::BinaryView) { + // Arrow starts out-of-line view data with an 8 KiB block. + bytes = bytes.checked_add(VIEW_BLOCK_BYTES)?; + } + match data_type { + DataType::Struct(fields) => { + for field in fields { + bytes = bytes.checked_add(array_bytes(field.data_type(), count)?)?; + } + } + DataType::List(field) | DataType::LargeList(field) => { + bytes = bytes.checked_add(array_bytes(field.data_type(), count)?)?; + } + DataType::FixedSizeList(field, len) => { + bytes = bytes.checked_add(array_bytes( + field.data_type(), + count.checked_mul(usize::try_from(*len).ok()?)?, + )?)?; + } + DataType::Dictionary(key, value) => { + bytes = bytes + .checked_add(array_bytes(key, count)?)? + .checked_add(array_bytes(value, count)?)?; + } + _ => {} + } + Some(bytes) +} + +#[derive(Default)] +struct ValueSize<'a> { + payload: usize, + items: usize, + scratch: usize, + paths: HashSet>>, + path: Vec>, +} + +#[derive(Clone, PartialEq, Eq, Hash)] +enum PathComponent<'a> { + Key(&'a str), + Field(usize), + Element, +} + +impl<'a> ValueSize<'a> { + fn expanded_paths(&self, limit: usize) -> Option { + if limit == 0 { + return Some(0); + } + if self.paths.len() <= limit { + return Some(self.paths.len()); + } + + // Arrays count as one selectable leaf, but all their element fields can expand. + let mut path_sizes = HashMap::new(); + for path in &self.paths { + let element = path + .iter() + .position(|part| matches!(part, PathComponent::Element)); + let leaf = &path[..element.unwrap_or(path.len())]; + let size = path_sizes.entry(leaf).or_insert(leaf.len()); + if element.is_some() { + *size = size.checked_add(1)?; + } + } + // The largest candidates bound any frequency-based selection, including parent structs. + let mut sizes = path_sizes.into_values().collect::>(); + sizes.sort_unstable_by(|a, b| b.cmp(a)); + let count = sizes + .into_iter() + .take(limit) + .try_fold(0usize, usize::checked_add)?; + Some(count.min(self.paths.len())) + } + + fn payload(&mut self, bytes: usize) -> Option<()> { + self.payload = self.payload.checked_add(bytes)?; + Some(()) + } + + fn children(&mut self, len: usize, array: bool) -> Option<()> { + if array { + self.items = self.items.checked_add(len)?; + } + self.scratch = self + .scratch + .checked_add(len.checked_mul(VALUE_SCRATCH_BYTES)?)?; + Some(()) + } + + fn variant(&mut self, bytes: usize) -> Option<()> { + self.payload(bytes)?; + // Opaque JSONB is decoded into a tree when written to a Variant remainder. + self.scratch = self.scratch.checked_add(bytes.checked_mul(64)?)?; + Some(()) + } + + fn proto(&mut self, value: Option<&'a ValueData>) -> Option<()> { + match value { + Some(ValueData::StringValue(value)) => self.payload(value.len())?, + Some(ValueData::BinaryValue(value)) => self.payload(value.len())?, + Some(ValueData::ListValue(value)) => { + self.children(value.items.len(), true)?; + self.path.push(PathComponent::Element); + for value in &value.items { + self.proto(value.value_data.as_ref())?; + } + self.path.pop(); + } + Some(ValueData::StructValue(value)) => { + self.children(value.items.len(), false)?; + for (index, value) in value.items.iter().enumerate() { + self.path.push(PathComponent::Field(index)); + self.proto(value.value_data.as_ref())?; + self.path.pop(); + } + } + Some(ValueData::JsonValue(value)) => self.json(value)?, + _ => {} + } + Some(()) + } + + fn json(&mut self, value: &'a JsonValue) -> Option<()> { + if !self.paths.contains(&self.path) { + self.paths.insert(self.path.clone()); + } + self.scratch = self.scratch.checked_add(VALUE_SCRATCH_BYTES)?; + match value.value.as_ref() { + Some(json_value::Value::Str(value)) => self.payload(value.len())?, + Some(json_value::Value::Variant(value)) => self.variant(value.len())?, + Some(json_value::Value::Array(value)) => { + self.items = self.items.checked_add(value.items.len())?; + self.path.push(PathComponent::Element); + for value in &value.items { + self.json(value)?; + } + self.path.pop(); + } + Some(json_value::Value::Object(value)) => { + for entry in &value.entries { + self.payload(entry.key.len())?; + self.path.push(PathComponent::Key(&entry.key)); + if let Some(value) = &entry.value { + self.json(value)?; + } + self.path.pop(); + } + } + _ => {} + } + Some(()) + } + + fn default_value(&mut self, value: &'a Value) -> Option<()> { + match value { + Value::String(value) => self.payload(value.len())?, + Value::Binary(value) => self.payload(value.len())?, + Value::List(value) => { + self.children(value.items().len(), true)?; + self.path.push(PathComponent::Element); + for value in value.items() { + self.default_value(value)?; + } + self.path.pop(); + } + Value::Struct(value) => { + self.children(value.items().len(), false)?; + for (index, value) in value.items().iter().enumerate() { + self.path.push(PathComponent::Field(index)); + self.default_value(value)?; + self.path.pop(); + } + } + Value::Json(value) => self.default_json(value.variant())?, + _ => {} + } + Some(()) + } + + fn default_json(&mut self, value: &'a JsonVariant) -> Option<()> { + if !self.paths.contains(&self.path) { + self.paths.insert(self.path.clone()); + } + self.scratch = self.scratch.checked_add(VALUE_SCRATCH_BYTES)?; + match value { + JsonVariant::String(value) => self.payload(value.len())?, + JsonVariant::Variant(value) => self.variant(value.len())?, + JsonVariant::Array(values) => { + self.items = self.items.checked_add(values.len())?; + self.path.push(PathComponent::Element); + for value in values { + self.default_json(value)?; + } + self.path.pop(); + } + JsonVariant::Object(values) => { + for (key, value) in values { + self.payload(key.len())?; + self.path.push(PathComponent::Key(key)); + self.default_json(value)?; + self.path.pop(); + } + } + _ => {} + } + Some(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_estimate_overflow_fails_closed() { + assert!(array_bytes(&DataType::Utf8, usize::MAX).is_none()); + let mut values = ValueSize::default(); + assert!(values.payload(usize::MAX).is_some()); + assert!(values.payload(1).is_none()); + assert!(values.children(usize::MAX, true).is_none()); + } +} diff --git a/src/operator/src/statement.rs b/src/operator/src/statement.rs index b52cb814cbb..feab3b15cba 100644 --- a/src/operator/src/statement.rs +++ b/src/operator/src/statement.rs @@ -522,11 +522,48 @@ impl StatementExecutor { SchemaNotFoundSnafu { schema_info: &db } ); + let previous_schema = query_ctx.current_schema(); query_ctx.set_current_schema(&db); + debug!(user = query_ctx.current_user().username(), protocol = %query_ctx.channel(), + catalog, schema = %db, previous_schema, process_id = query_ctx.process_id(), + "Database selected"); Ok(Output::new_with_record_batches(RecordBatches::empty())) } + /// Selects the first existing schema from candidates authorized by the caller. + pub async fn set_search_path( + &self, + schemas: Vec, + query_ctx: QueryContextRef, + ) -> Result { + let mut selected = String::new(); + for schema in schemas { + if !schema.is_empty() + && self + .catalog_manager + .schema_exists(query_ctx.current_catalog(), &schema, Some(&query_ctx)) + .await + .context(CatalogSnafu)? + { + selected = schema; + break; + } + debug!(user = query_ctx.current_user().username(), protocol = %query_ctx.channel(), + catalog = query_ctx.current_catalog(), schema = %schema, + process_id = query_ctx.process_id(), reason = "schema_not_found", + "Skipping unavailable search_path candidate"); + } + // An empty effective path must not retain access to the previous schema. + let previous_schema = query_ctx.current_schema(); + query_ctx.set_current_schema(&selected); + debug!(user = query_ctx.current_user().username(), protocol = %query_ctx.channel(), + catalog = query_ctx.current_catalog(), schema = %selected, previous_schema, + process_id = query_ctx.process_id(), empty_path = selected.is_empty(), + "Search path schema selected"); + Ok(Output::new_with_affected_rows(0)) + } + fn set_variables(&self, set_var: SetVariables, query_ctx: QueryContextRef) -> Result { let variable = sql::ast::object_name_to_string_unquoted(&set_var.variable); let var_name = variable.to_uppercase(); @@ -594,17 +631,10 @@ impl StatementExecutor { } }, "SEARCH_PATH" => { - if query_ctx.channel() == Channel::Postgres { - let search_path = set_var.search_path().context(NotSupportedSnafu { - feat: "Unsupported search path in set variable statement", - })?; - query_ctx.set_current_schema(search_path); - } else { - return NotSupportedSnafu { - feat: format!("Unsupported set variable {}", var_name), - } - .fail(); + return NotSupportedSnafu { + feat: "Unsupported search_path assignment", } + .fail(); } _ => { if query_ctx.channel() == Channel::Postgres || query_ctx.channel() == Channel::Mysql diff --git a/src/operator/src/statement/ddl.rs b/src/operator/src/statement/ddl.rs index ffbde46d420..9ee9ecffaa5 100644 --- a/src/operator/src/statement/ddl.rs +++ b/src/operator/src/statement/ddl.rs @@ -55,8 +55,8 @@ use common_meta::rpc::ddl::trigger::CreateTriggerTask; #[cfg(feature = "enterprise")] use common_meta::rpc::ddl::trigger::DropTriggerTask; use common_meta::rpc::ddl::{ - CreateFlowTask, CreatorGrantIntent, DdlTask, DropFlowTask, DropViewTask, SubmitDdlTaskRequest, - SubmitDdlTaskResponse, TriggerReason, + ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY, CreateFlowTask, CreatorGrantIntent, DdlTask, + DropFlowTask, DropViewTask, SubmitDdlTaskRequest, SubmitDdlTaskResponse, TriggerReason, }; use common_query::Output; use common_recordbatch::{RecordBatch, RecordBatches}; @@ -1564,16 +1564,7 @@ impl StatementExecutor { query_context: QueryContextRef, trigger_reason: TriggerReason, ) -> Result { - let _timer = crate::metrics::DIST_ALTER_TABLES.start_timer(); - ensure!( - !alter_table_exprs.is_empty(), - EmptyDdlExprSnafu { - name: "alter logical tables" - } - ); - - // group by physical table id - let mut groups: HashMap> = HashMap::new(); + let mut tables = Vec::with_capacity(alter_table_exprs.len()); for expr in alter_table_exprs { // Get table_id from catalog_manager let catalog = if expr.catalog_name.is_empty() { @@ -1596,20 +1587,49 @@ impl StatementExecutor { table_name: format_full_table_name(catalog, &schema, table_name), })?; let table_id = table.table_info().ident.table_id; + tables.push((table_id, expr)); + } + self.alter_logical_tables_with_ids(tables, query_context, trigger_reason) + .await + } + + /// Alters the captured table generations, rejecting concurrent drop/recreate under DDL locks. + pub(crate) async fn alter_logical_tables_with_ids( + &self, + tables: Vec<(TableId, AlterTableExpr)>, + query_context: QueryContextRef, + trigger_reason: TriggerReason, + ) -> Result { + let _timer = crate::metrics::DIST_ALTER_TABLES.start_timer(); + ensure!( + !tables.is_empty(), + EmptyDdlExprSnafu { + name: "alter logical tables" + } + ); + let mut groups: HashMap> = HashMap::new(); + for (table_id, expr) in tables { let physical_table_id = self .table_metadata_manager .table_route_manager() .get_physical_table_id(table_id) .await .context(TableMetadataManagerSnafu)?; - groups.entry(physical_table_id).or_default().push(expr); + groups + .entry(physical_table_id) + .or_default() + .push((table_id, expr)); } // Submit procedure for each physical table let mut handles = Vec::with_capacity(groups.len()); - for (_physical_table_id, exprs) in groups { - let fut = - self.alter_logical_tables_procedure(exprs, query_context.clone(), trigger_reason); + for (physical_table_id, exprs) in groups { + let fut = self.alter_logical_tables_procedure( + physical_table_id, + exprs, + query_context.clone(), + trigger_reason, + ); handles.push(fut); } let _results = futures::future::try_join_all(handles).await?; @@ -2336,17 +2356,54 @@ impl StatementExecutor { async fn alter_logical_tables_procedure( &self, - tables_data: Vec, + physical_table_id: TableId, + tables_data: Vec<(TableId, AlterTableExpr)>, query_context: QueryContextRef, trigger_reason: TriggerReason, ) -> Result { - let executor_context = to_executor_context(query_context, trigger_reason); - let request = SubmitDdlTaskRequest::new(DdlTask::new_alter_logical_tables(tables_data)); + let mut invalidate_keys = vec![CacheIdent::TableId(physical_table_id)]; + if let Some(physical) = self + .table_metadata_manager + .table_info_manager() + .get(physical_table_id) + .await + .context(TableMetadataManagerSnafu)? + { + invalidate_keys.push(CacheIdent::TableName(physical.table_name())); + } + for (id, expr) in &tables_data { + invalidate_keys.push(CacheIdent::TableId(*id)); + invalidate_keys.push(CacheIdent::TableName(TableName::new( + &expr.catalog_name, + &expr.schema_name, + &expr.table_name, + ))); + } + let mut executor_context = to_executor_context(query_context, trigger_reason); + let (table_ids, exprs): (Vec<_>, Vec<_>) = tables_data.into_iter().unzip(); + if let Some(context) = &mut executor_context.query_context { + context.extensions.insert( + ALTER_LOGICAL_TABLE_IDS_EXTENSION_KEY.to_string(), + table_ids + .iter() + .map(ToString::to_string) + .collect::>() + .join(","), + ); + } + let request = SubmitDdlTaskRequest::new(DdlTask::new_alter_logical_tables(exprs)); - self.procedure_executor + let response = self + .procedure_executor .submit_ddl_task(executor_context, request) .await - .context(error::ExecuteDdlSnafu) + .context(error::ExecuteDdlSnafu)?; + // Metasrv notifications are asynchronous; callers need the updated local schema now. + self.cache_invalidator + .invalidate(&Context::default(), &invalidate_keys) + .await + .context(error::InvalidateTableCacheSnafu)?; + Ok(response) } async fn drop_table_procedure( diff --git a/src/partition/src/expr.rs b/src/partition/src/expr.rs index 55139cd9dca..b2f2941c390 100644 --- a/src/partition/src/expr.rs +++ b/src/partition/src/expr.rs @@ -44,14 +44,14 @@ use crate::partition::PartitionBound; /// by [`Self::to_parser_expr`]. /// /// [Expr]: sqlparser::ast::Expr -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] pub struct PartitionExpr { pub lhs: Box, pub op: RestrictedOp, pub rhs: Box, } -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] pub enum Operand { Column(String), Value(Value), @@ -186,7 +186,7 @@ impl Display for Operand { /// A restricted set of [Operator](datafusion_expr::Operator) that can be used in /// partition expressions. -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] pub enum RestrictedOp { // Evaluate to binary Eq, @@ -488,6 +488,27 @@ mod tests { use super::*; + #[test] + fn test_expression_hash_respects_value_equality() { + use std::collections::hash_map::RandomState; + use std::hash::BuildHasher; + + use common_time::Timestamp; + + let seconds = col("ts").lt(Value::Timestamp(Timestamp::new_second(1))); + let millis = col("ts").lt(Value::Timestamp(Timestamp::new_millisecond(1000))); + assert_eq!(seconds, millis); + let hasher = RandomState::new(); + assert_eq!(hasher.hash_one(&seconds), hasher.hash_one(&millis)); + for value in [Value::Int32(10), Value::Int64(10)] { + let expr = col("tag").lt(value); + let decoded: PartitionExpr = + serde_json::from_slice(&serde_json::to_vec(&expr).unwrap()).unwrap(); + assert_eq!(expr, decoded); + assert_eq!(hasher.hash_one(&expr), hasher.hash_one(&decoded)); + } + } + #[test] fn test_partition_expr() { let cases = [ diff --git a/src/partition/src/function.rs b/src/partition/src/function.rs index cd121deb7f4..4fc45f13c0a 100644 --- a/src/partition/src/function.rs +++ b/src/partition/src/function.rs @@ -98,7 +98,7 @@ impl IcebergHash { pub fn to_logical_expr(&self) -> Expr { let column = Expr::Column(datafusion_common::Column::from_name(self.column())); - ScalarUDF::new_from_impl(IcebergHashUdf::default()).call(vec![column]) + iceberg_hash_udf().call(vec![column]) } } @@ -108,6 +108,15 @@ impl Display for IcebergHash { } } +/// Returns the DataFusion form of [`IcebergHash`]. +/// +/// Partition rules plan this UDF directly, and the SQL function `iceberg_hash` +/// is registered from the same constructor, so a persisted rule and a user query +/// always hash through one implementation. +pub fn iceberg_hash_udf() -> ScalarUDF { + ScalarUDF::new_from_impl(IcebergHashUdf::default()) +} + fn is_signed_integer(data_type: &ConcreteDataType) -> bool { matches!( data_type, @@ -120,6 +129,8 @@ fn is_signed_integer(data_type: &ConcreteDataType) -> bool { fn iceberg_hash_array(array: &dyn Array) -> datafusion_common::Result { let hashes: Int32Array = match array.data_type() { + // `iceberg_hash(NULL)` reaches here as an untyped null. + DataType::Null => Int32Array::from(vec![None; array.len()]), DataType::Utf8 => array .as_string::() .iter() diff --git a/src/plugins/src/frontend.rs b/src/plugins/src/frontend.rs index b8e56cf4668..608a9215216 100644 --- a/src/plugins/src/frontend.rs +++ b/src/plugins/src/frontend.rs @@ -18,6 +18,7 @@ use auth::{DefaultPermissionChecker, PermissionCheckerRef, UserProviderRef}; use common_base::Plugins; use common_base::secrets::ExposeSecret; use common_meta::cache::CacheRegistryBuilder; +use common_meta::cache_invalidator::KvCacheInvalidatorRef; use frontend::error::{IllegalAuthConfigSnafu, Result}; use frontend::frontend::FrontendOptions; use frontend::heartbeat::FrontendHeartbeatExtensions; @@ -87,7 +88,10 @@ pub async fn start_frontend_plugins(_instance: &Instance) -> Result<()> { } /// Allows frontend plugins to add cache invalidators to the layered registry. -pub fn configure_cache_registry(_plugins: &Plugins) -> Option { +pub fn configure_cache_registry( + _plugins: &Plugins, + _kv_cache: KvCacheInvalidatorRef, +) -> Option { None } diff --git a/src/query/src/dist_plan/dyn_filter_bridge.rs b/src/query/src/dist_plan/dyn_filter_bridge.rs index fb49839b304..0ebd0dcbf3e 100644 --- a/src/query/src/dist_plan/dyn_filter_bridge.rs +++ b/src/query/src/dist_plan/dyn_filter_bridge.rs @@ -18,15 +18,14 @@ use std::sync::Arc; use common_query::request::{ DynFilterPayload, INITIAL_REMOTE_DYN_FILTER_REGISTRATIONS_EXTENSION_KEY, InitialDynFilterReg, InitialDynFilterRegs, InitialDynFilterSnapshot, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + is_dyn_filter_payload_too_large, is_dyn_filter_unsupported, }; use datafusion::physical_plan::filter_pushdown::PushedDown; use datafusion_common::Result; use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion}; use datafusion_expr::ColumnarValue; use datafusion_physical_expr::PhysicalExpr; -use datafusion_physical_expr::expressions::{ - Column, DynamicFilterPhysicalExpr, InListExpr, Literal, lit, -}; +use datafusion_physical_expr::expressions::{Column, DynamicFilterPhysicalExpr, Literal, lit}; use datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr; use datafusion_physical_expr::utils::collect_columns; use session::context::{QueryContext, QueryContextRef}; @@ -36,6 +35,7 @@ use crate::dist_plan::filter_id::build_remote_dyn_filter_id; use crate::dist_plan::{ FilterId, QueryDynFilterRegistry, RemoteDynFilterProducerId, Subscriber, SubscriberRegistration, }; +use crate::metrics::REMOTE_DYN_FILTER_ENCODE_TOTAL; use crate::region_query::RegionQueryTarget; #[derive(Debug, Clone)] @@ -82,7 +82,7 @@ pub(crate) fn capture_remote_dyn_filters_for_pushdown( producer_local_ordinal, alive_dyn_filter, ) - .map(|captured| (captured, PushedDown::Yes)) + .map(pushdown_for_captured_filter) } else if is_scalar_subquery_filter(&filter) { // The filter is only forwarded to regions for pruning; the parent keeps // evaluating it, so report it as not pushed down. @@ -128,24 +128,43 @@ pub(crate) fn capture_remote_dyn_filters_for_pushdown( } } +/// A captured filter is only reported as pushed down when its initial snapshot could be +/// shipped. +/// +/// An unsupported (or oversized) snapshot is never rewritten to `true`, so a region +/// receives **no initial predicate** from the initial registration. The registration +/// itself is still sent - filter id plus children - and the receiver installs a `true` +/// placeholder for it, which is why the region is not filtered yet but the later updates +/// already have somewhere to land. The parent must therefore keep evaluating the filter +/// itself, which is what [`PushedDown::No`] tells DataFusion. +fn pushdown_for_captured_filter(captured: CapturedDynFilter) -> (CapturedDynFilter, PushedDown) { + let pushed_down = if captured.initial_registration.initial_snapshot.is_some() { + PushedDown::Yes + } else { + PushedDown::No + }; + + (captured, pushed_down) +} + /// Returns true if `filter` references at least one scalar subquery and is otherwise /// built only from columns and literals, so it becomes a plain predicate once the /// subqueries are resolved. /// -/// Filters containing `IN` lists are rejected: an oversized payload replaces `IN` with -/// `true`, which only widens a conjunction but can narrow an arbitrary predicate such as -/// `(host IN (...)) = false`. -// TODO(dennis): make the bounds-only fallback in `DynFilterPayload` replace only `IN` lists -// in conjunctive positions, then drop this restriction. +/// `IN` lists are accepted: the retention contract in +/// [`DynFilterPayload::from_datafusion_expr`] only ever drops whole top-level `AND` +/// conjuncts and never substitutes `true` for a node, so an `IN` list inside an arbitrary +/// predicate such as `(host IN (...)) = false` cannot be narrowed by transport. A payload +/// that cannot be shipped as it stands is an explicit error, which the capture answers by +/// only attaching no initial predicate (the region keeps its `true` placeholder) while the +/// parent keeps evaluating the filter. fn is_scalar_subquery_filter(filter: &Arc) -> bool { let mut has_scalar_subquery = false; let mut supported = true; let _ = filter.apply(|expr| { if expr.is::() { has_scalar_subquery = true; - } else if expr.is::() - || (expr.children().is_empty() && !expr.is::() && !expr.is::()) - { + } else if expr.children().is_empty() && !expr.is::() && !expr.is::() { supported = false; return Ok(TreeNodeRecursion::Stop); } @@ -323,6 +342,15 @@ fn attach_initial_snapshot( initial_registration.with_initial_snapshot(initial_snapshot) } +/// Initial snapshot of a captured filter, or `None` when regions must not be filtered yet. +/// +/// A predicate transport cannot represent (or that does not fit the payload budget) is +/// *not* replaced with `true`: only the initial snapshot is dropped (warn + metric), so the +/// region starts with no initial predicate but still receives the registration itself +/// (filter id plus children) and installs its `true` placeholder. Later updates land in +/// that placeholder - a representable payload, or the explicit `true` reset the registry +/// publishes when a generation has no wire form. The capture reports `PushedDown::No` so +/// the parent keeps evaluating the filter itself. fn initial_snapshot( alive_dyn_filter: &DynamicFilterPhysicalExpr, ) -> Option { @@ -340,6 +368,24 @@ fn initial_snapshot( REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, ) { Ok(payload) => payload, + Err(error) if is_dyn_filter_unsupported(&error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["unsupported"]) + .inc(); + common_telemetry::warn!( + error; "Remote dyn filter initial snapshot is dropped because the predicate has no wire form" + ); + return None; + } + Err(error) if is_dyn_filter_payload_too_large(&error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["too_large"]) + .inc(); + common_telemetry::warn!( + error; "Remote dyn filter initial snapshot is dropped because the predicate does not fit the payload budget" + ); + return None; + } Err(error) => { common_telemetry::warn!(error; "Failed to encode remote dyn filter initial snapshot"); return None; @@ -453,11 +499,13 @@ mod tests { use common_meta::peer::Peer; use datafusion::execution::TaskContext; + use datafusion::physical_plan::joins::join_hash_map::JoinHashMapU32; + use datafusion::physical_plan::joins::{HashTableLookupExpr, Map, SeededRandomState}; use datafusion_common::ScalarValue; use datafusion_expr::ColumnarValue; use datafusion_expr::Operator; use datafusion_expr::physical_planning_context::{ScalarSubqueryResults, SubqueryIndex}; - use datafusion_physical_expr::expressions::BinaryExpr; + use datafusion_physical_expr::expressions::{BinaryExpr, NotExpr}; use session::query_id::QueryId; use uuid::Uuid; @@ -557,6 +605,33 @@ mod tests { .unwrap() } + /// A probe filter whose current predicate has no wire form. + /// + /// A hand-built lookup has no build lease, hence no key list, and `NOT lookup` is a + /// single top-level conjunct that carries the membership: neither a key list nor a + /// membership-free conjunct exists. + fn test_unsupported_snapshot_filter( + column_name: &str, + column_index: usize, + ) -> Arc { + let column = Arc::new(Column::new(column_name, column_index)) as Arc; + let lookup = Arc::new(HashTableLookupExpr::new( + vec![Arc::clone(&column)], + SeededRandomState::with_seed(0), + Arc::new(Map::HashMap(Box::new(JoinHashMapU32::with_capacity(0)))), + "hash_lookup".to_string(), + None, + )) as Arc; + let dyn_filter = Arc::new(DynamicFilterPhysicalExpr::new( + vec![Arc::clone(&column) as _], + lit(true) as _, + )); + dyn_filter + .update(Arc::new(NotExpr::new(lookup)) as _) + .unwrap(); + dyn_filter + } + fn test_dyn_filter_with_snapshot_payload( column_name: &str, column_index: usize, @@ -654,7 +729,12 @@ mod tests { } #[test] - fn scalar_subquery_filter_with_in_list_is_left_to_parent() { + fn scalar_subquery_filter_with_in_list_is_captured_and_never_narrowed() { + // `(host IN ('a', 'b')) = (SELECT flag ...)` used to be rejected because an + // oversized payload was thought to replace `IN` with `true`. That is not the + // contract any more: transport never substitutes `true` for a node and only drops + // whole top-level conjuncts, so a membership inside an arbitrary predicate can only + // be kept or refused as a whole, never narrowed. let results = ScalarSubqueryResults::new(1); let schema = arrow_schema::Schema::new(vec![arrow_schema::Field::new( "host", @@ -668,7 +748,6 @@ mod tests { &schema, ) .unwrap(); - // `(host IN ('a', 'b')) = (SELECT flag ...)` let filter = Arc::new(BinaryExpr::new( host_in_list, Operator::Eq, @@ -676,7 +755,7 @@ mod tests { arrow_schema::DataType::Boolean, true, SubqueryIndex::new(0), - results, + results.clone(), )), )) as Arc; @@ -687,8 +766,40 @@ mod tests { vec![filter], ); + // Captured like any other scalar subquery filter, and still left to the parent. assert_eq!(pushed_down(&pushdown), vec![false]); - assert!(pushdown.captured_dyn_filters.is_empty()); + assert_eq!(pushdown.captured_dyn_filters.len(), 1); + let captured = &pushdown.captured_dyn_filters[0]; + + // Once resolved, the published predicate keeps the `IN` list structurally instead + // of widening it to `true`: `(host IN (...)) = true`. + results + .set(SubqueryIndex::new(0), ScalarValue::Boolean(Some(true))) + .unwrap(); + resolve_scalar_subquery_dyn_filters(std::slice::from_ref(captured)); + let snapshot = refreshed_regs(std::slice::from_ref(captured)).regs[0] + .initial_snapshot + .clone() + .expect("the resolved predicate is representable"); + let display = decode_snapshot(&snapshot, "host"); + assert!( + display.contains("IN"), + "the IN list must survive the wire: {display}" + ); + assert_ne!(display, "true", "transport must not widen to `true`"); + assert!( + display.ends_with("= true"), + "the equality over the IN list must survive: {display}" + ); + + // A payload that does not fit is refused outright rather than narrowed: the whole + // conjunct is dropped and nothing is left to ship. + let current = captured.alive_dyn_filter.current().unwrap(); + let refused = DynFilterPayload::from_datafusion_expr(¤t, 8).unwrap_err(); + assert!( + is_dyn_filter_payload_too_large(&refused), + "an oversized payload must be refused as too large, got: {refused:?}" + ); } #[test] @@ -851,6 +962,88 @@ mod tests { ); } + #[test] + fn capture_remote_dyn_filters_for_pushdown_skips_unsupported_initial_snapshot() { + // The parent retention and the registration are checked end to end: the capture + // still sends its registration (filter id plus children) but no initial predicate, + // the region installs its `true` placeholder, and a later representable generation + // still reaches it. + let dyn_filter = test_unsupported_snapshot_filter("host", 0); + let parent_filters = + vec![Arc::clone(&dyn_filter) as Arc]; + + let pushdown = capture_remote_dyn_filters_for_pushdown( + test_remote_dyn_filter_producer_id(42), + 0, + &[], + parent_filters, + ); + + // No snapshot is published (and it is never replaced with `true`), so the parent + // must keep evaluating the filter itself. + assert_eq!(pushed_down(&pushdown), vec![false]); + assert_eq!(pushdown.captured_dyn_filters.len(), 1); + let captured = &pushdown.captured_dyn_filters[0]; + assert!(captured.initial_registration.initial_snapshot.is_none()); + assert!( + !captured + .initial_registration + .child_exprs_datafusion_proto + .is_empty(), + "the registration must still describe the filter children" + ); + + // The registration survives the wire: filter id and children are intact, and the + // region installs the filter with a `true` placeholder and no initial predicate. + let regs = build_initial_dyn_filter_regs_for_region(&pushdown.captured_dyn_filters); + let serialized = regs.to_extension_value().unwrap(); + let received = InitialDynFilterRegs::from_extension_value(&serialized).unwrap(); + assert_eq!(received.regs.len(), 1); + assert_eq!( + received.regs[0].filter_id, + captured.filter_id.to_string(), + "the filter id must survive serialization" + ); + assert!(received.regs[0].initial_snapshot.is_none()); + let schema = Arc::new(arrow_schema::Schema::new(vec![arrow_schema::Field::new( + "host", + arrow_schema::DataType::Utf8, + true, + )])); + let children = received.regs[0] + .decode_children( + &TaskContext::default(), + &schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap(); + assert_eq!(children.len(), 1); + assert!(children[0].is::()); + + // A later generation transport can represent is delivered to the installed filter. + dyn_filter.update(lit(false) as _).unwrap(); + let current = captured.alive_dyn_filter.current().unwrap(); + let payload = + DynFilterPayload::from_datafusion_expr(¤t, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES) + .unwrap(); + let decoded = payload + .decode_datafusion_expr( + &TaskContext::default(), + &schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap(); + assert_eq!(decoded.to_string(), "false"); + let installed = Arc::new(DynamicFilterPhysicalExpr::new(children, lit(true) as _)); + let generation = installed.snapshot_generation(); + installed.update(decoded).unwrap(); + assert!( + installed.snapshot_generation() > generation, + "the receiver generation must advance" + ); + assert_eq!(installed.current().unwrap().to_string(), "false"); + } + #[test] fn capture_remote_dyn_filters_for_pushdown_attaches_initial_snapshot_after_update() { let dyn_filter = Arc::new(DynamicFilterPhysicalExpr::new( diff --git a/src/query/src/dist_plan/remote_dyn_filter_receiver.rs b/src/query/src/dist_plan/remote_dyn_filter_receiver.rs index 562cfd3c77c..71eeca16198 100644 --- a/src/query/src/dist_plan/remote_dyn_filter_receiver.rs +++ b/src/query/src/dist_plan/remote_dyn_filter_receiver.rs @@ -238,11 +238,21 @@ impl RemoteDynFilterReceiverExtensionPlanner { mod tests { use std::sync::Arc; - use datafusion::arrow::datatypes::{DataType, Field, Schema}; + use datafusion::arrow::array::{ + BooleanArray, Float64Array, StringArray, TimestampMillisecondArray, + }; + use datafusion::arrow::datatypes::{DataType, Field, Schema, TimeUnit}; + use datafusion::arrow::record_batch::RecordBatch; + use datafusion::execution::TaskContext; use datafusion::physical_plan::expressions::{Column, DynamicFilterPhysicalExpr, lit}; use datafusion_common::DFSchema; use datafusion_expr::{EmptyRelation, UserDefinedLogicalNodeCore}; + use common_query::request::{ + DynFilterPayload, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + remote_membership_test_support::{key, one_column_schema, run_real_hash_join, utf8}, + }; + use super::*; fn empty_input() -> LogicalPlan { @@ -318,4 +328,118 @@ mod tests { assert_eq!(column.name(), "instance"); assert_eq!(column.index(), 1); } + + /// The receiver keeps the recovered payload's columns as the filter's children, so + /// a name-based remap must also remap the recovered `InList` operand when the input + /// is pruned to a different column order. + fn pruned_input_with_trace_id() -> LogicalPlan { + let schema = Arc::new(Schema::new(vec![ + Field::new( + "ts", + DataType::Timestamp(TimeUnit::Millisecond, None), + false, + ), + Field::new("value", DataType::Float64, true), + Field::new("trace_id", DataType::Utf8, true), + ])); + LogicalPlan::EmptyRelation(EmptyRelation { + produce_one_row: false, + schema: Arc::new(DFSchema::try_from(schema).unwrap()), + }) + } + + #[tokio::test] + async fn recovered_membership_payload_survives_receiver_remap() { + // Coordinator side: a real hash join over the trace id publishes a runtime + // lookup that the GT encoder recovers as exact build membership. + let producer_schema = one_column_schema("trace_id", DataType::Utf8); + let trace_id = key("trace_id", 0); + let members = ["trace-a", "trace-b"]; + let member_batch = + RecordBatch::try_new(Arc::clone(&producer_schema), vec![utf8(&members)]).unwrap(); + let filter = run_real_hash_join( + 1, + Arc::clone(&producer_schema), + Arc::clone(&producer_schema), + vec![vec![member_batch.clone()]], + vec![vec![member_batch]], + vec![(Arc::clone(&trace_id), Arc::clone(&trace_id))], + ) + .await; + let payload = DynFilterPayload::from_datafusion_expr( + &filter.expr, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap(); + let decoded = payload + .decode_datafusion_expr( + &TaskContext::default(), + &producer_schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap(); + + // Receiver side: the payload's columns become the filter's children. + let payload_columns = collect_columns(&decoded) + .into_iter() + .map(|column| Arc::new(column) as Arc) + .collect::>(); + assert_eq!(payload_columns.len(), 1); + let dyn_filter = Arc::new(DynamicFilterPhysicalExpr::new( + payload_columns, + Arc::clone(&decoded), + )); + + let plan = + RemoteDynFilterReceiverLogicalPlan::new(pruned_input_with_trace_id(), vec![dyn_filter]); + let remapped = UserDefinedLogicalNodeCore::with_exprs_and_inputs( + &plan, + vec![], + vec![pruned_input_with_trace_id()], + ) + .unwrap(); + + // The recovered membership is remapped by name to the pruned input ordinal. + let receiver_filter = remapped.dyn_filters()[0] + .downcast_ref::() + .expect("receiver keeps the dynamic filter"); + let current = receiver_filter.current().unwrap(); + let remapped_columns = collect_columns(¤t); + assert_eq!(remapped_columns.len(), 1); + let remapped_column = remapped_columns.iter().next().unwrap(); + assert_eq!(remapped_column.name(), "trace_id"); + assert_eq!(remapped_column.index(), 2); + + // ...and still selects exactly the build keys on the pruned schema. + let pruned_schema = Arc::new(Schema::new(vec![ + Field::new( + "ts", + DataType::Timestamp(TimeUnit::Millisecond, None), + false, + ), + Field::new("value", DataType::Float64, true), + Field::new("trace_id", DataType::Utf8, true), + ])); + let batch = RecordBatch::try_new( + pruned_schema, + vec![ + Arc::new(TimestampMillisecondArray::from(vec![0i64, 1, 2])), + Arc::new(Float64Array::from(vec![1.0, 2.0, 3.0])), + Arc::new(StringArray::from(vec!["trace-a", "trace-b", "miss"])), + ], + ) + .unwrap(); + let selected = current + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let selected = selected.as_any().downcast_ref::().unwrap(); + assert_eq!( + (0..batch.num_rows()) + .map(|row| selected.value(row)) + .collect::>(), + vec![true, true, false] + ); + } } diff --git a/src/query/src/dist_plan/remote_dyn_filter_registry.rs b/src/query/src/dist_plan/remote_dyn_filter_registry.rs index 6a44db70513..42a60d6d653 100644 --- a/src/query/src/dist_plan/remote_dyn_filter_registry.rs +++ b/src/query/src/dist_plan/remote_dyn_filter_registry.rs @@ -18,11 +18,14 @@ use std::sync::{Arc, Mutex, RwLock, Weak}; use std::time::Duration; use api::v1::region::{RemoteDynFilterUnregister, RemoteDynFilterUpdate}; -use common_query::request::{DynFilterPayload, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES}; +use common_query::request::{ + DynFilterPayload, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, is_dyn_filter_payload_too_large, + is_dyn_filter_unsupported, +}; use common_runtime::spawn_global; use common_telemetry::{debug, warn}; use datafusion_physical_expr::PhysicalExpr; -use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr; +use datafusion_physical_expr::expressions::{DynamicFilterPhysicalExpr, lit}; use session::query_id::QueryId; use store_api::storage::RegionId; use tokio::sync::{Notify, watch}; @@ -381,6 +384,20 @@ async fn run_entry_fanout( unregister_entry_once_for_query(®ion_query_handler, query_id, &entry).await; } +/// Publishes one producer snapshot to every subscriber of `entry`. +/// +/// # Explicit `true` reset instead of a stale predicate +/// +/// A generation the wire cannot carry (a predicate with no wire form, or one that stays +/// over budget after top-level conjunct retention) is **not** skipped any more: the +/// subscriber would keep the previous, usually narrower predicate, and a producer that +/// then widens its filter would have rows pruned away by that stale predicate. Such a +/// generation is instead published as a literal-`true` payload at the same generation, so +/// the region drops the stale predicate. +/// +/// This reset is an explicit invalidation, not a serialization fallback: the producer +/// expression is never rewritten and no predicate is ever *narrowed* on the way out. The +/// query keeps running in every case; nothing in here returns an error. async fn fanout_snapshot_for_query( query_id: QueryId, region_query_handler: &RegionQueryHandlerRef, @@ -405,31 +422,9 @@ async fn fanout_snapshot_for_query( let _ = entry.mark_generation_sent(generation); } - let payload = - match DynFilterPayload::from_datafusion_expr(¤t, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES) - { - Ok(DynFilterPayload::Datafusion(payload)) => { - REMOTE_DYN_FILTER_ENCODE_TOTAL - .with_label_values(&["success"]) - .inc(); - REMOTE_DYN_FILTER_PAYLOAD_BYTES.observe(payload.len() as f64); - payload - } - Ok(_) => { - REMOTE_DYN_FILTER_ENCODE_TOTAL - .with_label_values(&["unsupported"]) - .inc(); - warn!("Ignored unsupported remote dynamic filter producer payload"); - return true; - } - Err(error) => { - REMOTE_DYN_FILTER_ENCODE_TOTAL - .with_label_values(&["error"]) - .inc(); - warn!(error; "Failed to encode remote dynamic filter producer snapshot"); - return true; - } - }; + let Some(payload) = snapshot_payload(¤t) else { + return true; + }; fanout_update_for_query( query_id, @@ -444,6 +439,100 @@ async fn fanout_snapshot_for_query( .await } +/// Payload to publish for one producer snapshot, or `None` when nothing can be published. +/// +/// A representable snapshot is published exactly as the producer wrote it. A snapshot the +/// wire cannot carry is **not** published as the producer wrote it (that is impossible) and +/// is **not** skipped either: the stale region predicate it would leave behind could prune +/// away rows the producer keeps after widening, so it is replaced by the explicit `true` +/// reset described on [`fanout_snapshot_for_query`]. The producer expression itself is +/// never rewritten. +fn snapshot_payload(current: &Arc) -> Option> { + let max_payload_bytes = REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES; + match DynFilterPayload::from_datafusion_expr(current, max_payload_bytes) { + Ok(DynFilterPayload::Datafusion(payload)) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["success"]) + .inc(); + REMOTE_DYN_FILTER_PAYLOAD_BYTES.observe(payload.len() as f64); + Some(payload) + } + Ok(_) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["unsupported"]) + .inc(); + warn!( + "Remote dynamic filter producer payload has an unknown encoding; publishing an explicit `true` reset" + ); + true_reset_payload() + } + Err(error) if is_dyn_filter_unsupported(&error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["unsupported"]) + .inc(); + warn!( + error; + "Remote dynamic filter snapshot has no wire form; publishing an explicit `true` reset to invalidate the stale region predicate" + ); + true_reset_payload() + } + Err(error) if is_dyn_filter_payload_too_large(&error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["too_large"]) + .inc(); + warn!( + error; + "Remote dynamic filter snapshot stays over budget; publishing an explicit `true` reset to invalidate the stale region predicate" + ); + true_reset_payload() + } + Err(error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["error"]) + .inc(); + warn!( + error; + "Failed to encode remote dynamic filter producer snapshot; publishing an explicit `true` reset to invalidate the stale region predicate" + ); + true_reset_payload() + } + } +} + +/// Encodes the literal-`true` payload that invalidates a stale remote predicate. +/// +/// This is the wire form of the explicit reset described on [`fanout_snapshot_for_query`]: +/// it is a *payload the transport builds itself*, never a reuse of the producer's +/// unrepresentable expression, so no predicate is rewritten to `true` on the way out. It +/// has no budget problem by construction and can only fail on a transport defect, which +/// is reported as `None` so best-effort fanout continues with the query running. +fn true_reset_payload() -> Option> { + let reset = lit(true) as Arc; + match DynFilterPayload::from_datafusion_expr(&reset, REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES) { + Ok(DynFilterPayload::Datafusion(payload)) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["reset"]) + .inc(); + REMOTE_DYN_FILTER_PAYLOAD_BYTES.observe(payload.len() as f64); + Some(payload) + } + Ok(_) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["reset_error"]) + .inc(); + warn!("Ignored remote dynamic filter `true` reset with an unknown encoding"); + None + } + Err(error) => { + REMOTE_DYN_FILTER_ENCODE_TOTAL + .with_label_values(&["reset_error"]) + .inc(); + warn!(error; "Failed to encode the remote dynamic filter `true` reset"); + None + } + } +} + #[allow(clippy::too_many_arguments)] async fn fanout_update_for_query( query_id: QueryId, @@ -828,7 +917,10 @@ mod tests { use async_trait::async_trait; use common_meta::peer::Peer; use common_query::request::QueryRequest; - use datafusion_physical_expr::expressions::{Column, lit}; + use datafusion::execution::TaskContext; + use datafusion::physical_plan::joins::join_hash_map::JoinHashMapU32; + use datafusion::physical_plan::joins::{HashTableLookupExpr, Map, SeededRandomState}; + use datafusion_physical_expr::expressions::{Column, NotExpr, lit}; use session::ReadPreference; use uuid::Uuid; @@ -1009,6 +1101,42 @@ mod tests { Arc::new(DynamicFilterPhysicalExpr::new(children, lit(true) as _)) } + /// `NOT lookup` over a hand-built lookup: no key list, and no membership-free conjunct. + fn unsupported_lookup_predicate() -> Arc { + let column = Arc::new(Column::new("host", 0)) as Arc; + let lookup = Arc::new(HashTableLookupExpr::new( + vec![Arc::clone(&column)], + SeededRandomState::with_seed(0), + Arc::new(Map::HashMap(Box::new(JoinHashMapU32::with_capacity(0)))), + "hash_lookup".to_string(), + None, + )) as Arc; + + Arc::new(NotExpr::new(lookup)) as Arc + } + + fn host_schema() -> Arc { + Arc::new(arrow_schema::Schema::new(vec![arrow_schema::Field::new( + "host", + arrow_schema::DataType::Utf8, + true, + )])) + } + + /// Decodes a recorded update the way a region receiver does. + fn decode_recorded_payload( + update: &RecordedUpdate, + schema: &Arc, + ) -> Arc { + DynFilterPayload::Datafusion(update.payload.clone()) + .decode_datafusion_expr( + &TaskContext::default(), + schema, + REMOTE_DYN_FILTER_PAYLOAD_MAX_BYTES, + ) + .unwrap() + } + #[test] fn registry_manager_returns_same_registry_for_same_query() { let manager = Arc::new(DynFilterRegistryManager::default()); @@ -1337,6 +1465,145 @@ mod tests { assert_eq!(entry.subscribers().len(), 2); } + #[tokio::test] + async fn fanout_resets_stale_predicate_when_a_wider_generation_has_no_wire_form() { + let query_id = test_query_id(11); + let registry = Arc::new(QueryDynFilterRegistry::new(query_id)); + let filter = test_dyn_filter(&["host"]); + let filter_id = test_filter_id(1); + let entry = match registry.register_remote_dyn_filter(filter_id.clone(), filter.clone()) { + EntryRegistration::Inserted(entry) => entry, + other => panic!("unexpected registration result: {other:?}"), + }; + let subscriber = Subscriber::new(RegionId::new(1024, 7), test_target(1)); + assert_eq!( + registry.register_subscriber(&filter_id, subscriber), + SubscriberRegistration::Added + ); + + let handler = Arc::new(RecordingRegionQueryHandler::default()); + let handler_ref = handler.clone() as RegionQueryHandlerRef; + let schema = host_schema(); + + // A representable narrow generation reaches the region and is applied there. + filter.update(lit(false) as _).unwrap(); + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + let updates = handler.updates(); + assert_eq!(updates.len(), 1); + let narrow = decode_recorded_payload(&updates[0], &schema); + assert_eq!(narrow.to_string(), "false"); + let receiver = Arc::new(DynamicFilterPhysicalExpr::new( + vec![Arc::new(Column::new("host", 0)) as _], + lit(true) as _, + )); + receiver.update(narrow).unwrap(); + let narrow_generation = receiver.snapshot_generation(); + assert_eq!(receiver.current().unwrap().to_string(), "false"); + + // `NOT lookup` has no wire form and no membership-free conjunct. The producer + // expression is never rewritten, but leaving the stale `false` installed would let + // it prune away the rows of a producer that widened, so the region gets an explicit + // `true` reset at the new generation instead. + filter.update(unsupported_lookup_predicate() as _).unwrap(); + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + let updates = handler.updates(); + assert_eq!( + updates.len(), + 2, + "the unsupported generation must be published as a reset" + ); + assert_eq!(updates[1].generation, filter.snapshot_generation()); + assert!(updates[1].generation > updates[0].generation); + let reset = decode_recorded_payload(&updates[1], &schema); + assert_eq!(reset.to_string(), "true"); + receiver.update(reset).unwrap(); + assert!( + receiver.snapshot_generation() > narrow_generation, + "the receiver generation must advance for the reset" + ); + assert_eq!(receiver.current().unwrap().to_string(), "true"); + + // A representable generation after the reset is still published as the producer + // wrote it, and the reset is not repeated for the same generation. + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + assert_eq!(handler.updates().len(), 2); + filter.update(lit(false) as _).unwrap(); + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + let updates = handler.updates(); + assert_eq!(updates.len(), 3); + assert_eq!(updates[2].generation, filter.snapshot_generation()); + assert_eq!( + decode_recorded_payload(&updates[2], &schema).to_string(), + "false" + ); + } + + #[tokio::test] + async fn fanout_publishes_producer_supplied_true_after_restrictive_generation() { + let query_id = test_query_id(12); + let registry = Arc::new(QueryDynFilterRegistry::new(query_id)); + let filter = test_dyn_filter(&["host"]); + let filter_id = test_filter_id(1); + let entry = match registry.register_remote_dyn_filter(filter_id.clone(), filter.clone()) { + EntryRegistration::Inserted(entry) => entry, + other => panic!("unexpected registration result: {other:?}"), + }; + let subscriber = Subscriber::new(RegionId::new(1024, 7), test_target(1)); + assert_eq!( + registry.register_subscriber(&filter_id, subscriber), + SubscriberRegistration::Added + ); + + let handler = Arc::new(RecordingRegionQueryHandler::default()); + let handler_ref = handler.clone() as RegionQueryHandlerRef; + let schema = host_schema(); + let receiver = Arc::new(DynamicFilterPhysicalExpr::new( + vec![Arc::new(Column::new("host", 0)) as _], + lit(true) as _, + )); + + // A restrictive generation reaches the region and is applied there. + filter.update(lit(false) as _).unwrap(); + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + let updates = handler.updates(); + assert_eq!(updates.len(), 1); + let restrictive = decode_recorded_payload(&updates[0], &schema); + assert_eq!(restrictive.to_string(), "false"); + receiver.update(restrictive).unwrap(); + let restrictive_generation = receiver.snapshot_generation(); + assert_eq!(receiver.current().unwrap().to_string(), "false"); + + // A `true` the *producer* itself publishes is a real predicate and is published + // normally - it is not the transport reset, which only exists for generations the + // wire cannot carry, and it is not a synthesized fallback either. + filter.update(lit(true) as _).unwrap(); + registry + .fanout_snapshot(&handler_ref, &entry, filter.as_ref(), false) + .await; + let updates = handler.updates(); + assert_eq!(updates.len(), 2); + assert_eq!(updates[1].generation, filter.snapshot_generation()); + assert!(updates[1].generation > updates[0].generation); + let widened = decode_recorded_payload(&updates[1], &schema); + assert_eq!(widened.to_string(), "true"); + receiver.update(widened).unwrap(); + assert!( + receiver.snapshot_generation() > restrictive_generation, + "the receiver generation must advance for a producer-supplied `true`" + ); + assert_eq!(receiver.current().unwrap().to_string(), "true"); + } + #[tokio::test] async fn fanout_task_waits_for_dynamic_filter_notifications() { let query_id = test_query_id(3); @@ -1590,6 +1857,75 @@ mod tests { wait_for_registry_drop(registry_weak).await; } + #[tokio::test] + async fn fanout_task_resets_unsupported_final_generation_and_cleans_up_once() { + let query_id = test_query_id(13); + let manager = Arc::new(DynFilterRegistryManager::default()); + let lease = manager.acquire_lease(query_id); + let registry_weak = Arc::downgrade(lease.registry.as_ref().unwrap()); + let filter = test_dyn_filter(&["host"]); + // A narrow, representable generation is the predicate the region starts with. + filter.update(lit(false) as _).unwrap(); + let filter_id = test_filter_id(1); + let _ = lease + .registry() + .register_remote_dyn_filter(filter_id.clone(), filter.clone()); + let subscriber = Subscriber::new(RegionId::new(1024, 7), test_target(1)); + assert_eq!( + lease.registry().register_subscriber(&filter_id, subscriber), + SubscriberRegistration::Added + ); + + let handler = Arc::new(RecordingRegionQueryHandler::default()); + lease.ensure_fanout_task(handler.clone() as RegionQueryHandlerRef); + + // The supported initial generation installs the narrow predicate on the region. + handler.wait_for_update_count(1).await; + assert_eq!( + decode_recorded_payload(&handler.updates()[0], &host_schema()).to_string(), + "false" + ); + + // The final generation has no wire form, so the stale `false` must be invalidated + // instead of being kept: an explicit `true` reset is published at that generation + // (complete, if the completion was already visible), and the fanout still cleans up + // exactly once. Whether the update and the completion are observed in one snapshot + // or two is racy, so both shapes are accepted. + filter.update(unsupported_lookup_predicate() as _).unwrap(); + filter.mark_complete(); + handler.wait_for_complete_update().await; + tokio::time::sleep(Duration::from_millis(100)).await; + + drop(lease); + handler.wait_for_unregister_count(1).await; + + let schema = host_schema(); + let updates = handler.updates(); + assert!(updates.len() >= 2, "the final generation must be reset"); + for reset in &updates[1..] { + assert_eq!( + decode_recorded_payload(reset, &schema).to_string(), + "true", + "every snapshot after the narrow one must be the true reset" + ); + assert_eq!( + reset.generation, + filter.snapshot_generation(), + "the reset must carry the current generation" + ); + } + assert!( + updates.last().unwrap().is_complete, + "the complete final generation must be marked complete" + ); + assert_eq!( + handler.unregisters().len(), + 1, + "the unsupported final generation must still clean up exactly once" + ); + wait_for_registry_drop(registry_weak).await; + } + #[tokio::test] async fn fanout_task_unregisters_after_lifecycle_close_during_blocked_update() { let query_id = test_query_id(5); diff --git a/src/query/src/optimizer/test_util.rs b/src/query/src/optimizer/test_util.rs index 09f72bb08b7..befaa62daad 100644 --- a/src/query/src/optimizer/test_util.rs +++ b/src/query/src/optimizer/test_util.rs @@ -109,6 +109,7 @@ impl RegionEngine for MetaRegionEngine { _role: RegionRole, _sent_at: std::time::Instant, _deadline: std::time::Instant, + _candidate_lease_id: Option, ) -> Result { unimplemented!() } diff --git a/src/relational-engine/src/engine.rs b/src/relational-engine/src/engine.rs index 3d84c10971d..9af0b520c92 100644 --- a/src/relational-engine/src/engine.rs +++ b/src/relational-engine/src/engine.rs @@ -116,6 +116,7 @@ impl RegionEngine for RelationalEngine { role: RegionRole, _sent_at: Instant, _deadline: Instant, + _candidate_lease_id: Option, ) -> Result { self.set_region_role(region_id, role).map(|_| true) } diff --git a/src/servers/AGENTS.md b/src/servers/AGENTS.md index 7bbdcf21861..d276becb685 100644 --- a/src/servers/AGENTS.md +++ b/src/servers/AGENTS.md @@ -26,6 +26,11 @@ provided through handler traits, mostly implemented by `frontend`. after bumping it. - Auth or `QueryContext` changes must cover HTTP, gRPC, MySQL, and PostgreSQL entry points. +- A protocol handler that changes the `QueryContext` catalog or schema after + authentication (HTTP `db` query/form parameters, MySQL `COM_INIT_DB`) calls + `UserProvider::authorize` for the new scope before mutating the context. The + HTTP middleware authorizes only the scope it extracts itself + (`extract_catalog_and_schema` in `http/authorize.rs`). - Internal frontend gRPC listeners set `Channel::Internal` in server-owned request extensions. Handlers preserve it through query execution and metering; client flow metadata does not grant an internal channel. diff --git a/src/servers/benches/to_http_output.rs b/src/servers/benches/to_http_output.rs index f3a9149083a..674c0c912bd 100644 --- a/src/servers/benches/to_http_output.rs +++ b/src/servers/benches/to_http_output.rs @@ -190,6 +190,7 @@ async fn limited_response( ..Default::default() }), Extension(QueryContext::with_db_name(None)), + None, Form(SqlQuery::default()), ) .await; diff --git a/src/servers/dashboard/VERSION b/src/servers/dashboard/VERSION index 574be600071..1d6cdeb31f8 100644 --- a/src/servers/dashboard/VERSION +++ b/src/servers/dashboard/VERSION @@ -1 +1 @@ -v0.14.5 +v0.14.6 diff --git a/src/servers/src/batcher/logical_table.rs b/src/servers/src/batcher/logical_table.rs index 707afe2ea0f..b7c0817e03a 100644 --- a/src/servers/src/batcher/logical_table.rs +++ b/src/servers/src/batcher/logical_table.rs @@ -108,6 +108,10 @@ pub struct LogicalTablePendingRowsBatcher { worker_channel_capacity: usize, prom_store_with_metric_engine: bool, schema_alterer: PendingRowsSchemaAltererRef, + /// Time index unit the schema alterer uses when auto-creating a missing + /// physical metric table; keeps the batcher's logical-create alignment + /// consistent with the created physical table. + auto_create_physical_table_timestamp_unit: TimeUnit, pending_rows_batch_sync: bool, shutdown: broadcast::Sender<()>, } @@ -123,6 +127,7 @@ impl LogicalTablePendingRowsBatcher { table_flownode_set_cache: TableFlownodeSetCacheRef, prom_store_with_metric_engine: bool, schema_alterer: PendingRowsSchemaAltererRef, + auto_create_physical_table_timestamp_unit: TimeUnit, flush_interval: Duration, max_batch_rows: usize, max_concurrent_flushes: usize, @@ -163,6 +168,7 @@ impl LogicalTablePendingRowsBatcher { flow_notification_tx, prom_store_with_metric_engine, schema_alterer, + auto_create_physical_table_timestamp_unit, flush_limiter, request_limiter, worker_channel_capacity, @@ -184,8 +190,8 @@ impl LogicalTablePendingRowsBatcher { } /// Returns the physical metric table's time index unit resolved from - /// `ctx`, defaulting to millisecond when the table does not exist yet - /// (the schema alterer auto-creates it as millisecond). + /// `ctx`, falling back to the configured auto-create unit when the table + /// does not exist yet (the schema alterer auto-creates it with that unit). async fn physical_time_index_unit_or_default(&self, ctx: &QueryContextRef) -> TimeUnit { let key = batch_key_from_ctx(ctx); let Ok(Some(table)) = self @@ -193,7 +199,7 @@ impl LogicalTablePendingRowsBatcher { .table(&key.catalog, &key.schema, &key.physical_table, None) .await else { - return TimeUnit::Millisecond; + return self.auto_create_physical_table_timestamp_unit; }; table .table_info() diff --git a/src/servers/src/batcher/logical_table/tables.rs b/src/servers/src/batcher/logical_table/tables.rs index 140dd2d4f23..8aef0df0b39 100644 --- a/src/servers/src/batcher/logical_table/tables.rs +++ b/src/servers/src/batcher/logical_table/tables.rs @@ -94,8 +94,8 @@ impl LogicalTablePendingRowsBatcher { // New tables are created on the request's selected physical table; // their time index must use the physical table's unit (a missing - // physical table is auto-created as millisecond by the schema - // alterer, matching the default here). + // physical table is auto-created in the configured auto-create unit + // by the schema alterer, matching the fallback here). if !plan.tables_to_create.is_empty() { let physical_unit = self.physical_time_index_unit_or_default(ctx).await; for (_, request_schema) in &mut plan.tables_to_create { diff --git a/src/servers/src/batcher/logical_table/test_util.rs b/src/servers/src/batcher/logical_table/test_util.rs index 364cf0c8f5a..6cf95af1f1c 100644 --- a/src/servers/src/batcher/logical_table/test_util.rs +++ b/src/servers/src/batcher/logical_table/test_util.rs @@ -32,6 +32,7 @@ use common_meta::node_manager::{ }; use common_meta::test_util::MockDatanodeManager; use common_query::prelude::GREPTIME_PHYSICAL_TABLE; +use common_time::timestamp::TimeUnit; use datatypes::prelude::ConcreteDataType; use datatypes::schema::{ColumnSchema as DtColumnSchema, Schema as DtSchema}; use operator::test_util::{create_partition_rule_manager, prepare_mocked_backend}; @@ -110,6 +111,7 @@ pub(crate) async fn submission_batcher( mock_table_flownode_cache(1, vec![]).await, true, Arc::new(UnexpectedSchemaAlterer), + TimeUnit::Millisecond, Duration::from_secs(1), 2, 1, diff --git a/src/servers/src/batcher/table.rs b/src/servers/src/batcher/table.rs index 7840fd355c4..db5ab6b77aa 100644 --- a/src/servers/src/batcher/table.rs +++ b/src/servers/src/batcher/table.rs @@ -31,20 +31,22 @@ use common_batcher::flush_policy::timing::TimingFlushPolicy; use common_batcher::request_limiter::RequestLimiter; use common_batcher::worker_registry::WorkerRegistry; use common_error::ext::BoxedError; +use common_memory_manager::OnExhaustedPolicy; use common_query::{Output, WriteCompletion, WriteProgress}; -use operator::batcher::PendingRowsBatcher; +use operator::batcher::{PendingRowsBatcher, PendingRowsPermit}; use operator::error::{ExternalSnafu, Result, UnexpectedSnafu}; use operator::insert::Inserter; use session::context::QueryContextRef; -use snafu::ResultExt; +use snafu::{IntoError, ResultExt}; use table::metadata::TableInfoRef; -use tokio::sync::{OwnedSemaphorePermit, Semaphore, broadcast, oneshot}; +use tokio::sync::{Semaphore, broadcast, oneshot}; use crate::batcher::pending_rows_batch_sync_enabled; use crate::batcher::table::flow_notifier::FlowNotifier; use crate::batcher::table::metrics::PENDING_WORKERS; use crate::batcher::table::pending_batch::PendingBatch; use crate::batcher::table::pending_worker::{PendingWorker, WorkerCommand, start_worker}; +use crate::request_memory_limiter::ServerMemoryLimiter; #[derive(Clone, Debug, Hash, PartialEq, Eq)] struct BatchKey { @@ -72,6 +74,7 @@ pub struct TablePendingRowsBatcher { flush_policy: TimingFlushPolicy, flush_limiter: FlushLimiter, request_limiter: RequestLimiter, + memory_limiter: ServerMemoryLimiter, worker_channel_capacity: usize, pending_rows_batch_sync: bool, worker_idle_timeout: Duration, @@ -82,6 +85,7 @@ pub struct TablePendingRowsBatcher { impl TablePendingRowsBatcher { /// Creates a shared timing batcher, rejecting disabled or unsupported limits. + #[allow(clippy::too_many_arguments)] pub fn try_new( flush_interval: Duration, max_batch_rows: usize, @@ -89,6 +93,7 @@ impl TablePendingRowsBatcher { worker_channel_capacity: usize, max_inflight_requests: usize, flow_notification_queue_capacity: NonZeroUsize, + memory_limiter: ServerMemoryLimiter, inserter: Arc, ) -> Option> { if worker_channel_capacity == 0 @@ -113,6 +118,7 @@ impl TablePendingRowsBatcher { flush_policy, flush_limiter, request_limiter, + memory_limiter, worker_channel_capacity, pending_rows_batch_sync: pending_rows_batch_sync_enabled(), worker_idle_timeout: flush_interval.checked_mul(3).unwrap_or(flush_interval), @@ -154,15 +160,43 @@ impl Drop for TablePendingRowsBatcher { #[async_trait] impl PendingRowsBatcher for TablePendingRowsBatcher { + fn memory_limit_bytes(&self) -> u64 { + self.memory_limiter.limit_bytes() + } + /// Acquires once per original request, before submitting any of its tables. /// Retain a clone in every submission until its flush has completed. - async fn acquire(&self) -> Result> { - self.request_limiter.acquire().await.map_err(|_| { + async fn acquire(&self, estimated_bytes: usize, held_bytes: u64) -> Result { + let permit = self.request_limiter.acquire().await.map_err(|_| { UnexpectedSnafu { violated: "batcher admission closed".to_string(), } .build() - }) + })?; + if self.memory_limit_bytes() == 0 { + return Ok(permit); + } + let bytes = u64::try_from(estimated_bytes).unwrap_or(u64::MAX); + let reserve = || { + self.memory_limiter.check_request_fits(bytes, held_bytes)?; + self.memory_limiter.try_acquire(bytes).ok_or_else(|| { + let source = common_memory_manager::Error::MemoryLimitExceeded { + requested_bytes: bytes, + limit_bytes: self.memory_limit_bytes(), + }; + // Remote Write senders must retry contention under the wait policy. + match self.memory_limiter.policy() { + OnExhaustedPolicy::Wait { .. } => { + crate::error::OtherSnafu.into_error(BoxedError::new(source)) + } + OnExhaustedPolicy::Fail => { + crate::error::MemoryLimitExceededSnafu.into_error(source) + } + } + }) + }; + let memory = reserve().map_err(BoxedError::new).context(ExternalSnafu)?; + Ok(Arc::new((permit, memory))) } /// Acknowledges according to the global batching policy. Cancellation does @@ -172,7 +206,7 @@ impl PendingRowsBatcher for TablePendingRowsBatcher { table_info: TableInfoRef, batch: RecordBatch, ctx: QueryContextRef, - permit: Arc, + permit: PendingRowsPermit, ) -> Result { let total_rows = batch.num_rows(); let confirmed = ctx.confirmed_writes(); @@ -262,6 +296,7 @@ mod tests { use common_batcher::worker_registry::WorkerRegistry; use common_catalog::consts::default_engine; use common_grpc::flight::FlightDecoder; + use common_memory_manager::OnExhaustedPolicy; use common_meta::error::Result as MetaResult; use common_meta::peer::Peer; use common_meta::test_util::{MockDatanodeHandler, MockDatanodeManager}; @@ -292,6 +327,7 @@ mod tests { use crate::batcher::table::pending_worker::{WorkerCommand, start_worker}; use crate::batcher::table::{PendingWorkers, TablePendingRowsBatcher, batch_key_from_ctx}; use crate::batcher::test_util::mock_table_flownode_cache; + use crate::request_memory_limiter::ServerMemoryLimiter; #[derive(Clone)] struct BulkHandler { @@ -337,6 +373,8 @@ mod tests { #[tokio::test] async fn test_acknowledgement_preserves_admission() { + use common_error::ext::ErrorExt; + use common_error::status_code::StatusCode; use operator::error::UnexpectedSnafu; use crate::batcher::pending_rows_batch_sync_enabled; @@ -344,7 +382,8 @@ mod tests { use crate::batcher::table::pending_batch::notify_batches; for sync in [false, true] { - for fail in [false, true] { + for outcome in ["success", "failure", "cancel"] { + let fail = outcome == "failure"; let backend = prepare_mocked_backend().await; let nodes = Arc::new(MockDatanodeManager::new(BulkHandler { requests: Arc::new(Mutex::new(Vec::new())), @@ -359,6 +398,12 @@ mod tests { mock_table_flownode_cache(1, vec![]).await, true, )); + let policy = if sync { + OnExhaustedPolicy::default() + } else { + OnExhaustedPolicy::Fail + }; + let memory = ServerMemoryLimiter::new(4096, policy); let mut batcher = TablePendingRowsBatcher::try_new( Duration::from_secs(3600), 1, @@ -366,6 +411,7 @@ mod tests { 1, 1, NonZeroUsize::new(1).unwrap(), + memory.clone(), inserter, ) .unwrap(); @@ -374,6 +420,31 @@ mod tests { pending_rows_batch_sync_enabled() ); Arc::get_mut(&mut batcher).unwrap().pending_rows_batch_sync = sync; + assert!(batcher.acquire(4097, 0).await.is_err()); + let held = memory.try_acquire(3072).unwrap(); + let Err(error) = batcher.acquire(2048, held.granted_bytes()).await else { + panic!("body and conversion together exceed the entire quota"); + }; + assert_eq!(error.status_code(), StatusCode::RateLimited); + let additional = batcher.acquire(1024, held.granted_bytes()).await.unwrap(); + assert_eq!(memory.used_bytes(), 4096); + drop(additional); + // The same occupied quota belongs to another request and can be released. + let acquire = batcher.acquire(2048, 0); + tokio::pin!(acquire); + let std::task::Poll::Ready(Err(error)) = futures::poll!(acquire.as_mut()) else { + panic!("conversion memory must not wait while a request holds other permits"); + }; + assert_eq!( + error.status_code(), + if sync { + StatusCode::RuntimeResourcesExhausted + } else { + StatusCode::RateLimited + } + ); + drop(held); + assert_eq!(memory.used_bytes(), 0); let table = Arc::new(new_test_table_info(1, "ack", [0].into_iter())); let progress = Arc::new(session::write_progress::WriteProgress::default()); let mut ctx = QueryContext::with("greptime", "public"); @@ -388,7 +459,7 @@ mod tests { Arc::new(Int32Array::from(vec![1])) as ArrayRef, )]) .unwrap(); - let permit = batcher.acquire().await.unwrap(); + let permit = batcher.acquire(2048, 0).await.unwrap(); let submitter = batcher.clone(); let submitted = tokio::spawn(async move { submitter.submit(table, batch, ctx, permit).await }); @@ -415,8 +486,16 @@ mod tests { } assert_eq!(progress.accepted(), !sync); assert_eq!(progress.written(), 0); + if outcome == "cancel" { + if let Some(submitted) = submitted.take() { + submitted.abort(); + assert!(submitted.await.unwrap_err().is_cancelled()); + } + drop(completion.take()); + } + assert_eq!(memory.used_bytes(), 2048); // Early acknowledgement must not release capacity before completion. - let acquire = batcher.acquire(); + let acquire = batcher.acquire(0, 0); tokio::pin!(acquire); assert!(futures::poll!(acquire.as_mut()).is_pending()); let result = if fail { @@ -430,6 +509,7 @@ mod tests { Ok(()) }; notify_batches(vec![pending], result); + assert_eq!(memory.used_bytes(), 0); if let Some(submitted) = submitted { let result = timeout(Duration::from_secs(5), submitted) .await @@ -538,6 +618,7 @@ mod tests { 4, if shared_request { 1 } else { 4 }, NonZeroUsize::new(16).unwrap(), + ServerMemoryLimiter::default(), inserter, ) .unwrap(); @@ -555,7 +636,7 @@ mod tests { opentsdb_ctx.set_skip_wal(skip_wal); let (first_result, second_result) = timeout(Duration::from_secs(5), async { if shared_request { - let permit = batcher.acquire().await.unwrap(); + let permit = batcher.acquire(0, 0).await.unwrap(); let (first, second) = tokio::join!( batcher.submit(table.clone(), first, influx_ctx, permit.clone()), batcher.submit(table.clone(), second, opentsdb_ctx, permit), @@ -740,6 +821,7 @@ mod tests { 4, 4, NonZeroUsize::new(16).unwrap(), + ServerMemoryLimiter::default(), inserter, ) .unwrap(); @@ -756,8 +838,8 @@ mod tests { ctx.set_write_progress(counter.clone()); Arc::new(ctx) }); - let first_permit = batcher.acquire().await.unwrap(); - let second_permit = batcher.acquire().await.unwrap(); + let first_permit = batcher.acquire(0, 0).await.unwrap(); + let second_permit = batcher.acquire(0, 0).await.unwrap(); let (first, second) = tokio::join!( batcher.submit(table.clone(), first, contexts[0].clone(), first_permit), batcher.submit(table, second, contexts[1].clone(), second_permit), diff --git a/src/servers/src/batcher/table/pending_batch.rs b/src/servers/src/batcher/table/pending_batch.rs index c4545f5992a..4fd235499ed 100644 --- a/src/servers/src/batcher/table/pending_batch.rs +++ b/src/servers/src/batcher/table/pending_batch.rs @@ -17,11 +17,12 @@ use std::sync::Arc; use arrow::record_batch::RecordBatch; use common_error::ext::BoxedError; use common_query::WriteProgress; +use operator::batcher::PendingRowsPermit; use operator::error::Error; use session::context::QueryContextRef; use snafu::IntoError; use table::metadata::TableInfoRef; -use tokio::sync::{OwnedSemaphorePermit, oneshot}; +use tokio::sync::oneshot; /// One complete input and its completion/admission ownership. pub(in crate::batcher::table) struct PendingBatch { @@ -29,7 +30,7 @@ pub(in crate::batcher::table) struct PendingBatch { pub batch: RecordBatch, pub ctx: QueryContextRef, pub response_tx: oneshot::Sender, - pub _permit: Arc, + pub _permit: PendingRowsPermit, pub _completion: Option, } diff --git a/src/servers/src/exemplar.rs b/src/servers/src/exemplar.rs index 986c7212181..ed7d381502c 100644 --- a/src/servers/src/exemplar.rs +++ b/src/servers/src/exemplar.rs @@ -24,7 +24,7 @@ use api::v1::{ColumnDataType, Row, RowInsertRequests, Value}; use common_catalog::consts::METRICS_EXEMPLARS_TABLE; use common_grpc::precision::Precision; use common_query::prelude::{ - EXEMPLAR_LABELS_COLUMN, EXEMPLAR_METRIC_COLUMN, greptime_timestamp, greptime_value, + EXEMPLAR_METRIC_COLUMN, greptime_exemplar_labels, greptime_timestamp, greptime_value, }; use datatypes::json; use metric_engine::row_modifier::TsidGenerator; @@ -120,7 +120,7 @@ where greptime_timestamp(), greptime_value(), EXEMPLAR_METRIC_COLUMN, - EXEMPLAR_LABELS_COLUMN, + greptime_exemplar_labels(), DATA_SCHEMA_TABLE_ID_COLUMN_NAME, DATA_SCHEMA_TSID_COLUMN_NAME, ] @@ -225,7 +225,7 @@ pub fn into_rows(exemplars: Vec) -> Result<(RowInsertRequests, BTreeSe row_writer::write_by_schema( table, std::iter::once(( - row_writer::build_json2_column_schema(EXEMPLAR_LABELS_COLUMN)?, + row_writer::build_json2_column_schema(greptime_exemplar_labels())?, Some(row_writer::encode_json2(exemplar.labels)?), )), &mut row, @@ -276,9 +276,18 @@ mod tests { for name in [ DATA_SCHEMA_TABLE_ID_COLUMN_NAME, DATA_SCHEMA_TSID_COLUMN_NAME, + greptime_exemplar_labels(), ] { assert!(labels([(name.into(), "x".into())], false, validate_label_name).is_err()); } + assert!( + labels( + [("exemplar_labels".into(), "x".into())], + false, + validate_label_name, + ) + .is_ok() + ); assert_eq!(timestamp_from_millis(-1).unwrap(), -1_000_000); assert!(timestamp_from_millis(i64::MAX).is_err()); } diff --git a/src/servers/src/exemplar/query.rs b/src/servers/src/exemplar/query.rs index 64cf609bd9d..0087c964b21 100644 --- a/src/servers/src/exemplar/query.rs +++ b/src/servers/src/exemplar/query.rs @@ -137,6 +137,7 @@ fn resolve(selector: &VectorSelector, default_schema: &str) -> Result, pub exemplars: Vec, diff --git a/src/servers/src/http.rs b/src/servers/src/http.rs index 13be8b03b2a..a8d70e8fb37 100644 --- a/src/servers/src/http.rs +++ b/src/servers/src/http.rs @@ -240,9 +240,16 @@ pub async fn set_http_write_batching( .extensions() .get::() .is_some_and(|protocols| protocols.0.contains(&protocol)); + let body_memory_bytes = req + .extensions() + .get::() + .map(memory_limit::BodyMemoryAccounting::reserved_bytes_counter); if let Some(ctx) = req.extensions_mut().get_mut::() { ctx.set_batching_enabled(enabled); ctx.set_logical_batching_enabled(logical_enabled); + if let Some(bytes) = body_memory_bytes { + ctx.set_body_memory_bytes(bytes); + } } next.run(req).await } diff --git a/src/servers/src/http/authorize.rs b/src/servers/src/http/authorize.rs index 5c5211ce79f..325efd129db 100644 --- a/src/servers/src/http/authorize.rs +++ b/src/servers/src/http/authorize.rs @@ -30,7 +30,7 @@ use common_time::Timezone; use common_time::timezone::parse_timezone; use headers::Header; use serde::Serialize; -use session::context::QueryContextBuilder; +use session::context::{QueryContext, QueryContextBuilder}; use snafu::{OptionExt, ResultExt, ensure}; use crate::error::{ @@ -113,6 +113,7 @@ pub async fn inner_auth( { Ok(userinfo) => { query_ctx.set_current_user(userinfo); + req.extensions_mut().insert(user_provider); let _ = req.extensions_mut().insert(query_ctx); return Ok(req); } @@ -163,6 +164,7 @@ pub async fn inner_auth( { Ok(userinfo) => { query_ctx.set_current_user(userinfo); + req.extensions_mut().insert(user_provider); let _ = req.extensions_mut().insert(query_ctx); Ok(req) } @@ -180,6 +182,34 @@ pub async fn inner_auth( } } +pub(crate) async fn select_database( + query_ctx: &mut QueryContext, + db: Option<&str>, + user_provider: Option<&UserProviderRef>, +) -> auth::error::Result<()> { + let Some(db) = db else { + return Ok(()); + }; + let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); + if catalog == query_ctx.current_catalog() && schema == query_ctx.current_schema() { + return Ok(()); + } + // Middleware authorized the initial scope; form and query parameters can change it. + if let Some(provider) = user_provider + && let Err(error) = provider + .authorize(&catalog, &schema, &query_ctx.current_user()) + .await + { + crate::metrics::METRIC_AUTH_FAILURE + .with_label_values(&[error.status_code().as_ref()]) + .inc(); + return Err(error); + } + query_ctx.set_current_catalog(&catalog); + query_ctx.set_current_schema(&schema); + Ok(()) +} + pub async fn check_http_auth( State(auth_state): State, req: Request, @@ -454,13 +484,54 @@ fn extract_influxdb_user_from_query(query: &str) -> (Option<&str>, Option<&str>) } #[cfg(test)] -mod tests { +pub(crate) mod tests { use std::assert_matches; use common_base::secrets::ExposeSecret; use super::*; + pub(crate) struct ScopedUserProvider; + + #[async_trait::async_trait] + impl auth::UserProvider for ScopedUserProvider { + fn name(&self) -> &str { + "scoped-test" + } + + async fn authenticate( + &self, + _: auth::Identity<'_>, + _: auth::Password<'_>, + ) -> auth::error::Result { + Ok(auth::userinfo_by_name(Some("reader".to_string()))) + } + + async fn authorize( + &self, + catalog: &str, + schema: &str, + _: &auth::UserInfoRef, + ) -> auth::error::Result<()> { + if catalog == "greptime" && matches!(schema, "public" | "%64enied") { + Ok(()) + } else { + auth::error::PermissionDeniedSnafu.fail() + } + } + + async fn auth_bearer_token( + &self, + _: &str, + catalog: &str, + schema: &str, + ) -> auth::error::Result { + let user = auth::userinfo_by_name(Some("reader".to_string())); + self.authorize(catalog, schema, &user).await?; + Ok(user) + } + } + #[test] fn test_need_auth() { let req = Request::builder() diff --git a/src/servers/src/http/handler.rs b/src/servers/src/http/handler.rs index 8cbfc25ff07..fd85d22402b 100644 --- a/src/servers/src/http/handler.rs +++ b/src/servers/src/http/handler.rs @@ -19,12 +19,12 @@ use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, Instant}; +use auth::UserProviderRef; use axum::extract::rejection::FormRejection; use axum::extract::{Json, Query, State}; use axum::response::sse::{Event, KeepAlive, Sse}; use axum::response::{IntoResponse, Response}; use axum::{Extension, Form}; -use common_catalog::parse_catalog_and_schema_from_db_string; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; use common_plugins::GREPTIME_EXEC_WRITE_COST; @@ -48,6 +48,7 @@ use crate::error::{ CollectRecordbatchSnafu, FailedToParseQuerySnafu, InvalidQuerySnafu, Result, log_error_if_necessary, }; +use crate::http::authorize::select_database; use crate::http::header::collect_plan_metrics; use crate::http::result::arrow_result::ArrowResponse; use crate::http::result::csv_result::CsvResponse; @@ -143,14 +144,19 @@ pub async fn sql( State(state): State, Query(query_params): Query, Extension(mut query_ctx): Extension, + user_provider: Option>, Form(form_params): Form, ) -> HttpResponse { let start = Instant::now(); let sql_handler = &state.sql_handler; - if let Some(db) = &query_params.db.or(form_params.db) { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - query_ctx.set_current_catalog(&catalog); - query_ctx.set_current_schema(&schema); + if let Err(error) = select_database( + &mut query_ctx, + query_params.db.or(form_params.db).as_deref(), + user_provider.as_ref().map(|provider| &provider.0), + ) + .await + { + return HttpResponse::Error(ErrorResponse::from_error(error)); } let db = query_ctx.get_db_string(); @@ -243,6 +249,7 @@ pub struct LogicalTableDdl { pub async fn create_logical_tables( State(state): State, Extension(mut query_ctx): Extension, + user_provider: Option>, form: std::result::Result, FormRejection>, ) -> HttpResponse { let start = Instant::now(); @@ -267,10 +274,14 @@ pub async fn create_logical_tables( "logical-table DDL exceeds 1 MiB".to_string(), )); } - if let Some(db) = form.db { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(&db); - query_ctx.set_current_catalog(&catalog); - query_ctx.set_current_schema(&schema); + if let Err(error) = select_database( + &mut query_ctx, + form.db.as_deref(), + user_provider.as_ref().map(|provider| &provider.0), + ) + .await + { + return HttpResponse::Error(ErrorResponse::from_error(error)); } query_ctx.set_channel(Channel::HttpSql); let query_ctx = Arc::new(query_ctx); @@ -374,6 +385,7 @@ pub async fn sql_analyze_stream( State(state): State, Query(query_params): Query, Extension(mut query_ctx): Extension, + user_provider: Option>, form_params: std::result::Result, FormRejection>, ) -> Response { let start = Instant::now(); @@ -392,10 +404,14 @@ pub async fn sql_analyze_stream( } }; let sql_handler = &state.sql_handler; - if let Some(db) = &query_params.db.or(form_params.db) { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - query_ctx.set_current_catalog(&catalog); - query_ctx.set_current_schema(&schema); + if let Err(error) = select_database( + &mut query_ctx, + query_params.db.or(form_params.db).as_deref(), + user_provider.as_ref().map(|provider| &provider.0), + ) + .await + { + return ErrorResponse::from_error(error).into_response(); } query_ctx.set_channel(Channel::HttpSql); query_ctx.enable_live_analyze_metrics(); @@ -1086,6 +1102,7 @@ mod tests { use snafu::IntoError; use super::*; + use crate::http::authorize::tests::ScopedUserProvider; use crate::query_handler::sql::SqlQueryHandler; struct TestSqlQueryHandler { @@ -1134,6 +1151,84 @@ mod tests { } } + #[tokio::test] + async fn test_sql_endpoints_authorize_effective_database() { + use axum::http::{Request, StatusCode as HttpStatusCode}; + use tower::ServiceExt; + + for authorization in ["Basic cmVhZGVyOnB3ZA==", "Bearer test-token"] { + for (path, query, body, method) in [ + ("/v1/sql", "", "sql=select+1&db=denied", "POST"), + ("/v1/sql", "?db=denied&sql=select+1", "", "GET"), + ("/v1/sql", "?db=%64enied&sql=select+1", "", "GET"), + ("/v1/sql", "?db=denied", "sql=select+1&db=public", "POST"), + ("/v1/ddl", "", "sql=create+table+t&db=denied", "POST"), + ("/v1/analyze", "", "sql=select+1&db=denied", "POST"), + ("/v1/analyze", "?db=denied&sql=select+1", "", "GET"), + ] { + let handler = Arc::new(TestSqlQueryHandler { + outputs: Mutex::new(vec![Ok(Output::new_with_affected_rows(0))]), + }); + let app = axum::Router::new() + .route("/v1/sql", axum::routing::get(sql).post(sql)) + .route("/v1/ddl", axum::routing::post(create_logical_tables)) + .route( + "/v1/analyze", + axum::routing::get(sql_analyze_stream).post(sql_analyze_stream), + ) + .with_state(ApiState { + sql_handler: handler.clone(), + }) + .layer(axum::middleware::from_fn_with_state( + crate::http::authorize::AuthState::new(Some(Arc::new(ScopedUserProvider))), + crate::http::authorize::check_http_auth, + )); + let mut request = Request::builder() + .method(method) + .uri(format!("{path}{query}")) + .header("authorization", authorization) + .header("content-type", "application/x-www-form-urlencoded"); + if !query.contains("%64") { + request = request.header("x-greptime-db-name", "public"); + } + let response = app + .oneshot(request.body(axum::body::Body::from(body)).unwrap()) + .await + .unwrap(); + assert_eq!( + HttpStatusCode::FORBIDDEN, + response.status(), + "{path}{query}" + ); + assert_eq!(1, handler.outputs.lock().unwrap().len()); + } + } + } + + #[tokio::test] + async fn test_select_database_authorizes_before_mutating_context() { + let provider: UserProviderRef = Arc::new(ScopedUserProvider); + let mut ctx = QueryContext::with_db_name(Some("initial")); + select_database(&mut ctx, Some("public"), Some(&provider)) + .await + .unwrap(); + assert_eq!("public", ctx.current_schema()); + let failures = crate::metrics::METRIC_AUTH_FAILURE + .with_label_values(&[StatusCode::PermissionDenied.as_ref()]); + let before = failures.get(); + assert!( + select_database(&mut ctx, Some("denied"), Some(&provider)) + .await + .is_err() + ); + assert!(failures.get() > before); + assert_eq!("public", ctx.current_schema()); + select_database(&mut ctx, Some("other"), None) + .await + .unwrap(); + assert_eq!("other", ctx.current_schema()); + } + async fn sql_response( outputs: Vec>, format: &str, @@ -1152,6 +1247,7 @@ mod tests { ..Default::default() }), Extension(QueryContext::with_db_name(None)), + None, Form(SqlQuery::default()), ) .await @@ -1181,6 +1277,7 @@ mod tests { }), }), Extension(QueryContext::with_db_name(None)), + None, Ok(Form(LogicalTableDdl { sql: Some(sql.to_string()), db: None, diff --git a/src/servers/src/http/memory_limit.rs b/src/servers/src/http/memory_limit.rs index 255920974fd..16c40ba0006 100644 --- a/src/servers/src/http/memory_limit.rs +++ b/src/servers/src/http/memory_limit.rs @@ -31,8 +31,8 @@ //! buffers that outlive the body stream itself). use std::pin::Pin; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, Weak}; use std::task::{Context, Poll}; use axum::body::Body; @@ -182,6 +182,7 @@ pub async fn decoded_body_accounting_middleware( #[derive(Clone, Default)] pub(crate) struct BodyMemoryAccounting { guards: Arc>>>, + reserved_bytes: Arc, /// Set when an incremental charge hit the quota, so the rejection can be /// rewritten into a 429 after the middleware regains control. quota_exceeded: Arc, @@ -190,15 +191,17 @@ pub(crate) struct BodyMemoryAccounting { impl BodyMemoryAccounting { /// Includes permit rounding and all frames collected so far. pub(crate) fn reserved_bytes(&self) -> u64 { - self.guards - .lock() - .unwrap() - .iter() - .map(MemoryGuard::granted_bytes) - .sum() + self.reserved_bytes.load(Ordering::Acquire) + } + + /// Observes subsequent body charges without extending the guards' lifetime. + pub(crate) fn reserved_bytes_counter(&self) -> Weak { + Arc::downgrade(&self.reserved_bytes) } fn hold(&self, guard: MemoryGuard) { + self.reserved_bytes + .fetch_add(guard.granted_bytes(), Ordering::Release); self.guards.lock().unwrap().push(guard); } @@ -534,6 +537,79 @@ mod tests { Body::from(compressed) } + #[tokio::test] + async fn test_body_reservations_follow_context_without_extending_lifetime() { + use std::sync::{Arc, Mutex}; + + use axum::Extension; + use session::context::QueryContext; + use tower_http::decompression::RequestDecompressionLayer; + + use crate::batcher::BatchingProtocol; + use crate::http::set_http_write_batching; + + for limit in [0, 8 * 1024] { + for compressed in [false, true] { + let limiter = ServerMemoryLimiter::new(limit, OnExhaustedPolicy::default()); + let captured = Arc::new(Mutex::new(None)); + let output = captured.clone(); + let memory = limiter.clone(); + let app = Router::new() + .route( + "/echo", + post( + move |Extension(ctx): Extension, body: Bytes| async move { + assert_eq!(body.len(), 1536); + assert!(ctx.batching_enabled()); + assert!(ctx.held_memory_bytes() >= body.len() as u64); + assert_eq!(ctx.held_memory_bytes(), memory.used_bytes()); + *output.lock().unwrap() = Some(ctx.with_write_admission()); + StatusCode::OK + }, + ), + ) + .layer(middleware::from_fn_with_state( + limiter.clone(), + super::decoded_body_accounting_middleware, + )) + .layer(RequestDecompressionLayer::new()) + .layer(middleware::from_fn_with_state( + BatchingProtocol::Influxdb, + set_http_write_batching, + )) + .layer(Extension(QueryContext::with("greptime", "public"))) + .layer(Extension(Arc::new(vec![BatchingProtocol::Influxdb]))) + .layer(middleware::from_fn_with_state( + limiter.clone(), + memory_limit_middleware, + )); + let body = vec![b'x'; 1536]; + let body = if compressed { + zstd::stream::encode_all(body.as_slice(), 3).unwrap() + } else { + body + }; + let mut request = chunked_request("/echo", body, 512); + if compressed { + request + .headers_mut() + .insert(header::CONTENT_ENCODING, "zstd".parse().unwrap()); + } + assert_eq!(app.oneshot(request).await.unwrap().status(), StatusCode::OK); + assert_eq!(limiter.used_bytes(), 0); + assert_eq!( + captured + .lock() + .unwrap() + .as_ref() + .unwrap() + .held_memory_bytes(), + 0 + ); + } + } + } + #[tokio::test] async fn test_decompressed_body_is_charged_for_content_encoded_requests() { // The decoded size (64 KiB) exceeds the 16 KiB quota: the accounting diff --git a/src/servers/src/http/prom_store.rs b/src/servers/src/http/prom_store.rs index b4f182a5a41..86d8bba795f 100644 --- a/src/servers/src/http/prom_store.rs +++ b/src/servers/src/http/prom_store.rs @@ -173,6 +173,71 @@ async fn remote_write_v1( body: Bytes, body_reserved_bytes: u64, ) -> Result { + if let Some(response) = vm_proto_version_response(¶ms) { + return Ok(response); + } + + let (db, query_ctx, _timer) = + prepare_remote_write_context(¶ms, query_ctx, REMOTE_WRITE_V1_VERSION); + let query_ctx = Arc::new(query_ctx); + + match ingest_remote_write_v1( + &state, + &db, + pipeline_info, + is_zstd, + query_ctx, + body, + body_reserved_bytes, + ) + .await + { + Ok(outcome) => { + Ok((StatusCode::NO_CONTENT, write_cost_header_map(outcome.cost)).into_response()) + } + Err(error) => Err(error.error), + } +} + +/// The axum-free core of a remote-write v1 ingestion, shared with +/// `COPY FROM STDIN WITH (FORMAT promwrite)`: decode, convert and write +/// exactly as the HTTP endpoint does. +pub(crate) async fn ingest_remote_write_v1( + state: &PromStoreState, + db: &str, + pipeline_info: PipelineInfo, + is_zstd: bool, + query_ctx: QueryContextRef, + body: Bytes, + body_reserved_bytes: u64, +) -> std::result::Result { + let result = ingest_remote_write_v1_inner( + state, + pipeline_info, + is_zstd, + query_ctx, + body, + body_reserved_bytes, + ) + .await; + // Account for partial writes as well as successful requests, regardless of transport. + let (samples, exemplars) = match &result { + Ok(outcome) => (outcome.written[0], outcome.written[2]), + Err(error) => (error.rows_written, error.exemplars_written), + }; + record_remote_write_samples(db, REMOTE_WRITE_V1_VERSION, samples); + record_remote_write_exemplars(db, REMOTE_WRITE_V1_VERSION, exemplars); + result +} + +async fn ingest_remote_write_v1_inner( + state: &PromStoreState, + pipeline_info: PipelineInfo, + is_zstd: bool, + query_ctx: QueryContextRef, + body: Bytes, + body_reserved_bytes: u64, +) -> std::result::Result { let PromStoreState { prom_store_handler, pipeline_handler, @@ -182,29 +247,30 @@ async fn remote_write_v1( memory_limiter, } = state; - if let Some(response) = vm_proto_version_response(¶ms) { - return Ok(response); - } - - let (db, query_ctx, _timer) = - prepare_remote_write_context(¶ms, query_ctx, REMOTE_WRITE_V1_VERSION); - let query_ctx = Arc::new(query_ctx); - + let no_progress = |error: error::Error| PromWriteError { + error, + rows_written: 0, + exemplars_written: 0, + }; let mut processor = PromSeriesProcessor::default_processor(); if let Some(pipeline_name) = pipeline_info.pipeline_name { let pipeline_def = PipelineDefinition::from_name( &pipeline_name, to_pipeline_version(pipeline_info.pipeline_version.as_deref()) - .context(PipelineSnafu)?, + .context(PipelineSnafu) + .map_err(no_progress)?, None, ) - .context(PipelineSnafu)?; - let pipeline_handler = pipeline_handler.context(InternalSnafu { - err_msg: "pipeline handler is not set".to_string(), - })?; + .context(PipelineSnafu) + .map_err(no_progress)?; + let Some(pipeline_handler) = pipeline_handler else { + return Err(no_progress(error::Error::Internal { + err_msg: "pipeline handler is not set".to_string(), + })); + }; - processor.set_pipeline(pipeline_handler, query_ctx.clone(), pipeline_def); + processor.set_pipeline(pipeline_handler.clone(), query_ctx.clone(), pipeline_def); } // Keep the decode guards alive until the batches are built: the @@ -212,11 +278,13 @@ async fn remote_write_v1( let (mut decoded, decode_guards) = decode_remote_write_request( is_zstd, body, - prom_validation_mode, + *prom_validation_mode, &mut processor, - &memory_limiter, + memory_limiter, + body_reserved_bytes, ) - .await?; + .await + .map_err(no_progress)?; // Parsing borrows the decode buffer, but row building copies out of it: tag // values through `decode_string`, column names through `to_owned`, and the @@ -226,7 +294,7 @@ async fn remote_write_v1( let exemplars = std::mem::take(&mut decoded.exemplars); let req = if processor.use_pipeline { drop(decoded); - processor.exec_pipeline().await? + processor.exec_pipeline().await.map_err(no_progress)? } else { let req = decoded.as_insert_requests(); drop(decoded); @@ -240,82 +308,90 @@ async fn remote_write_v1( let bytes = crate::exemplar::estimated_row_bytes(exemplars.values().map(Vec::as_slice)) .and_then(|bytes| u64::try_from(bytes).ok()) .ok_or_else(|| { - error::MemoryLimitExceededSnafu.into_error( + no_progress(error::MemoryLimitExceededSnafu.into_error( common_memory_manager::Error::MemoryLimitExceeded { requested_bytes: u64::MAX, limit_bytes: memory_limiter.limit_bytes(), }, - ) + )) })?; - memory_limiter.check_request_fits(bytes, body_reserved_bytes)?; + memory_limiter + .check_request_fits(bytes, body_reserved_bytes) + .map_err(no_progress)?; // Holding body permits while waiting for expansion can stall concurrent requests. let reservation = Arc::new(if body_reserved_bytes > 0 { - memory_limiter.try_acquire(bytes).ok_or_else(|| { - let source = common_memory_manager::Error::MemoryLimitExceeded { - requested_bytes: bytes, - limit_bytes: memory_limiter.limit_bytes(), - }; - // Wait-policy contention must remain retryable by Remote Write senders. - match memory_limiter.policy() { - OnExhaustedPolicy::Wait { .. } => { - error::OtherSnafu.into_error(BoxedError::new(source)) + memory_limiter + .try_acquire(bytes) + .ok_or_else(|| { + let source = common_memory_manager::Error::MemoryLimitExceeded { + requested_bytes: bytes, + limit_bytes: memory_limiter.limit_bytes(), + }; + // Wait-policy contention must remain retryable by Remote Write senders. + match memory_limiter.policy() { + OnExhaustedPolicy::Wait { .. } => { + error::OtherSnafu.into_error(BoxedError::new(source)) + } + OnExhaustedPolicy::Fail => { + error::MemoryLimitExceededSnafu.into_error(source) + } } - OnExhaustedPolicy::Fail => error::MemoryLimitExceededSnafu.into_error(source), - } - })? + }) + .map_err(no_progress)? } else { memory_limiter .acquire(bytes) .await - .map_err(prom_remote_write::map_memory_acquire_timeout)? + .map_err(prom_remote_write::map_memory_acquire_timeout) + .map_err(no_progress)? }); let mut ctx = query_ctx.fork(); ctx.set_write_progress(Arc::new( - session::write_progress::WriteProgress::with_reservation(reservation), + session::write_progress::WriteProgress::with_reservation( + reservation.granted_bytes(), + reservation, + ), )); Arc::new(ctx) }; let batches = into_prom_write_batches(req, query_ctx.clone()); if !exemplars.is_empty() { let outcome = exemplar::write( - prom_store_handler, - pending_rows_batcher, - prom_store_with_metric_engine, + prom_store_handler.clone(), + pending_rows_batcher.clone(), + *prom_store_with_metric_engine, batches, Vec::new(), exemplars, query_ctx, ) .await; - record_remote_write_samples(&db, REMOTE_WRITE_V1_VERSION, outcome.written[0]); - record_remote_write_exemplars(&db, REMOTE_WRITE_V1_VERSION, outcome.written[2]); if let Some(error) = outcome.error { - return Err(error); + return Err(PromWriteError { + error, + rows_written: outcome.written[0], + exemplars_written: outcome.written[2], + }); } - return Ok((StatusCode::NO_CONTENT, write_cost_header_map(outcome.cost)).into_response()); + return Ok(PromWriteResult { + written: outcome.written, + cost: outcome.cost, + accepted: outcome.accepted, + }); } - let outcome = match write_prometheus_rows_with_progress( - prom_store_handler, - pending_rows_batcher, - prom_store_with_metric_engine, + write_prometheus_rows_with_progress( + prom_store_handler.clone(), + pending_rows_batcher.clone(), + *prom_store_with_metric_engine, batches, ) .await - { - Ok(outcome) => outcome, - Err(error) => { - record_remote_write_samples(&db, REMOTE_WRITE_V1_VERSION, error.rows_written); - return Err(error.error); - } - }; - record_remote_write_samples(&db, REMOTE_WRITE_V1_VERSION, outcome.rows_written); - - Ok(( - StatusCode::NO_CONTENT, - write_cost_header_map(outcome.write_cost), - ) - .into_response()) + .map(|outcome| PromWriteResult { + written: [outcome.rows_written, 0, 0], + cost: outcome.write_cost, + accepted: false, + }) } async fn remote_write_v2( @@ -327,41 +403,99 @@ async fn remote_write_v2( body: Bytes, body_reserved_bytes: u64, ) -> Result { - let PromStoreState { - prom_store_handler, - pipeline_handler: _, - prom_store_with_metric_engine, - prom_validation_mode: _, - pending_rows_batcher, - memory_limiter, - } = state; - if let Some(response) = vm_proto_version_response(¶ms) { return Ok(response); } - // Pipeline processing is not supported for remote write v2 yet. Ignore the - // optional pipeline parameter and ingest samples directly. + // Pipeline processing is not supported for remote write v2 yet. Only the + // pipeline presence matters, for the exemplar restriction below. let has_pipeline = pipeline_info.pipeline_name.is_some(); - let (db, mut query_ctx, _timer) = + let (db, query_ctx, _timer) = prepare_remote_write_context(¶ms, query_ctx, REMOTE_WRITE_V2_VERSION); - let req = - match decode_remote_write_v2(is_zstd, body, &memory_limiter, body_reserved_bytes).await { - Ok(req) => req, - Err(error) => return Ok(remote_write_v2_error_response(error, 0, 0, 0)), - }; + match ingest_remote_write_v2( + &state, + &db, + has_pipeline, + is_zstd, + query_ctx, + body, + body_reserved_bytes, + ) + .await + { + Ok(PromWriteResult { + written, + cost, + accepted, + }) => { + let mut headers = write_cost_header_map(cost); + append_remote_write_v2_written_headers( + &mut headers, + written[0], + written[1], + written[2], + ); + let status = if accepted { + StatusCode::ACCEPTED + } else { + StatusCode::NO_CONTENT + }; + Ok((status, headers).into_response()) + } + Err(error) => Ok(remote_write_v2_error_response( + error.error, + error.samples_written, + error.histograms_written, + error.exemplars_written, + )), + } +} + +/// The axum-free core of a remote-write v2 ingestion, shared with +/// `COPY FROM STDIN WITH (FORMAT promwrite, version 2)`. `db` labels the +/// per-batch write-progress counters exactly as the HTTP route does. +pub(crate) async fn ingest_remote_write_v2( + state: &PromStoreState, + db: &str, + has_pipeline: bool, + is_zstd: bool, + query_ctx: QueryContext, + body: Bytes, + body_reserved_bytes: u64, +) -> std::result::Result { + let PromStoreState { + prom_store_handler, + pending_rows_batcher, + prom_store_with_metric_engine, + memory_limiter, + .. + } = state; + + let mut query_ctx = query_ctx; + let req = match decode_remote_write_v2(is_zstd, body, memory_limiter, body_reserved_bytes).await + { + Ok(req) => req, + Err(error) => { + return Err(PromWriteV2Error { + error, + samples_written: 0, + histograms_written: 0, + exemplars_written: 0, + }); + } + }; if has_pipeline && req.exemplar_count > 0 { - return Ok(remote_write_v2_error_response( - error::NotSupportedSnafu { + return Err(PromWriteV2Error { + error: error::NotSupportedSnafu { feat: "pipelines with exemplars", } .build(), - 0, - 0, - 0, - )); + samples_written: 0, + histograms_written: 0, + exemplars_written: 0, + }); } // The v2 per-series metadata upgrades the written tables' semantic options // (metric type/unit, declared quality) at auto-create time. @@ -370,10 +504,13 @@ async fn remote_write_v2( } if req.exemplar_count > 0 { let counter = crate::metrics::PROM_STORE_REMOTE_WRITE_EXEMPLARS - .with_label_values(&[&db, REMOTE_WRITE_V2_VERSION]); + .with_label_values(&[db, REMOTE_WRITE_V2_VERSION]); query_ctx.set_write_progress(Arc::new( - session::write_progress::WriteProgress::with_reservation(req.memory_guard.clone()) - .with_observer(move |rows| counter.inc_by(rows)), + session::write_progress::WriteProgress::with_reservation( + req.memory_guard.granted_bytes(), + req.memory_guard.clone(), + ) + .with_observer(move |rows| counter.inc_by(rows)), )); } let query_ctx = Arc::new(query_ctx); @@ -383,16 +520,18 @@ async fn remote_write_v2( for (batches, histogram) in [(&mut sample_batches, false), (&mut histogram_batches, true)] { let counter = if histogram { crate::metrics::PROM_STORE_REMOTE_WRITE_HISTOGRAMS - .with_label_values(&[&db, REMOTE_WRITE_V2_VERSION]) + .with_label_values(&[db, REMOTE_WRITE_V2_VERSION]) } else { crate::metrics::PROM_STORE_REMOTE_WRITE_SAMPLES - .with_label_values(&[&db, REMOTE_WRITE_V2_VERSION]) + .with_label_values(&[db, REMOTE_WRITE_V2_VERSION]) }; for (ctx, _) in batches { let counter = counter.clone(); - let progress = - session::write_progress::WriteProgress::with_reservation(req.memory_guard.clone()) - .with_observer(move |rows| counter.inc_by(rows)); + let progress = session::write_progress::WriteProgress::with_reservation( + req.memory_guard.granted_bytes(), + req.memory_guard.clone(), + ) + .with_observer(move |rows| counter.inc_by(rows)); let mut fork = ctx.fork(); fork.set_write_progress(Arc::new(progress)); *ctx = Arc::new(fork); @@ -400,9 +539,9 @@ async fn remote_write_v2( } if req.exemplar_count > 0 { let outcome = exemplar::write( - prom_store_handler, - pending_rows_batcher, - prom_store_with_metric_engine, + prom_store_handler.clone(), + pending_rows_batcher.clone(), + *prom_store_with_metric_engine, sample_batches, histogram_batches, req.exemplars, @@ -410,62 +549,34 @@ async fn remote_write_v2( ) .await; if let Some(error) = outcome.error { - return Ok(remote_write_v2_error_response( + return Err(PromWriteV2Error { error, - outcome.written[0], - outcome.written[1], - outcome.written[2], - )); + samples_written: outcome.written[0], + histograms_written: outcome.written[1], + exemplars_written: outcome.written[2], + }); } - let mut headers = write_cost_header_map(outcome.cost); - append_remote_write_v2_written_headers( - &mut headers, - outcome.written[0], - outcome.written[1], - outcome.written[2], - ); - let status = if outcome.accepted { - StatusCode::ACCEPTED - } else { - StatusCode::NO_CONTENT - }; - return Ok((status, headers).into_response()); + return Ok(PromWriteResult { + written: outcome.written, + cost: outcome.cost, + accepted: outcome.accepted, + }); } - let outcome = match write_prometheus_v2_rows_with_progress( - prom_store_handler, - pending_rows_batcher, - prom_store_with_metric_engine, + let outcome = write_prometheus_v2_rows_with_progress( + prom_store_handler.clone(), + pending_rows_batcher.clone(), + *prom_store_with_metric_engine, sample_batches, histogram_batches, ) - .await - { - Ok(outcome) => outcome, - Err(error) => { - return Ok(remote_write_v2_error_response( - error.error, - error.samples_written, - error.histograms_written, - 0, - )); - } - }; + .await?; debug_assert!(outcome.samples_written <= sample_count); debug_assert!(outcome.histograms_written <= histogram_count); - let mut headers = write_cost_header_map(outcome.write_cost); - append_remote_write_v2_written_headers( - &mut headers, - outcome.samples_written, - outcome.histograms_written, - 0, - ); - - let status = if outcome.accepted { - StatusCode::ACCEPTED - } else { - StatusCode::NO_CONTENT - }; - Ok((status, headers).into_response()) + Ok(PromWriteResult { + written: [outcome.samples_written, outcome.histograms_written, 0], + cost: outcome.write_cost, + accepted: outcome.accepted, + }) } fn vm_proto_version_response(params: &RemoteWriteQuery) -> Option { @@ -485,9 +596,28 @@ fn prepare_remote_write_context( ) -> (String, QueryContext, HistogramTimer) { let db = params.db.clone().unwrap_or_default(); query_ctx.set_channel(Channel::Prometheus); - let physical_table = params - .physical_table - .clone() + stamp_remote_write_extensions( + &mut query_ctx, + params.physical_table.as_deref(), + remote_write_version, + ); + let timer = crate::metrics::METRIC_HTTP_PROM_STORE_WRITE_ELAPSED + .with_label_values(&[db.as_str(), remote_write_version]) + .start_timer(); + + (db, query_ctx, timer) +} + +/// Stamps the remote-write context extensions shared by the HTTP +/// endpoint and COPY FROM STDIN: physical table and the Prometheus +/// semantic identity the batched and direct write paths both rely on. +pub(crate) fn stamp_remote_write_extensions( + query_ctx: &mut QueryContext, + physical_table: Option<&str>, + remote_write_version: &str, +) { + let physical_table = physical_table + .map(String::from) .unwrap_or_else(|| GREPTIME_PHYSICAL_TABLE.to_string()); query_ctx.set_extension(PHYSICAL_TABLE_PARAM, physical_table); // Stamp the Prometheus metric identity here, before `as_req_iter` splits into the @@ -499,11 +629,6 @@ fn prepare_remote_write_context( query_ctx.set_extension(SEMANTIC_SOURCE, SOURCE_PROMETHEUS); query_ctx.set_extension(SEMANTIC_SOURCE_VERSION, remote_write_version); query_ctx.set_extension(SEMANTIC_METRIC_METADATA_QUALITY, METADATA_QUALITY_INFERRED); - let timer = crate::metrics::METRIC_HTTP_PROM_STORE_WRITE_ELAPSED - .with_label_values(&[db.as_str(), remote_write_version]) - .start_timer(); - - (db, query_ctx, timer) } struct PromWriteOutcome { @@ -511,9 +636,18 @@ struct PromWriteOutcome { rows_written: u64, } -struct PromWriteError { - error: error::Error, - rows_written: u64, +pub(crate) struct PromWriteError { + pub(crate) error: error::Error, + pub(crate) rows_written: u64, + pub(crate) exemplars_written: u64, +} + +/// Transport-independent remote-write outcome for samples, histograms and exemplars. +pub(crate) struct PromWriteResult { + /// Written counts, or accepted counts when v2 asynchronous batching is enabled. + pub(crate) written: [u64; 3], + pub(crate) cost: usize, + pub(crate) accepted: bool, } struct PromWriteV2Outcome { @@ -523,10 +657,11 @@ struct PromWriteV2Outcome { histograms_written: u64, } -struct PromWriteV2Error { - error: error::Error, - samples_written: u64, - histograms_written: u64, +pub(crate) struct PromWriteV2Error { + pub(crate) error: error::Error, + pub(crate) samples_written: u64, + pub(crate) histograms_written: u64, + pub(crate) exemplars_written: u64, } type PromWriteBatch = (QueryContextRef, RowInsertRequests); @@ -585,6 +720,7 @@ async fn write_prometheus_rows_with_progress( .map_err(|error| PromWriteError { error, rows_written: 0, + exemplars_written: 0, })?; // Destinations bound to another physical table must stay on the // ordinary insert path, which routes per destination; time index @@ -603,6 +739,7 @@ async fn write_prometheus_rows_with_progress( .map_err(|error| PromWriteError { error, rows_written, + exemplars_written: 0, })?; rows_written += rows; } @@ -630,6 +767,7 @@ async fn write_prometheus_rows_with_progress( .map_err(|error| PromWriteError { error, rows_written: 0, + exemplars_written: 0, })?; let output_count = outputs.len(); let mut write_cost = 0; @@ -638,6 +776,7 @@ async fn write_prometheus_rows_with_progress( let output = output.map_err(|error| PromWriteError { error, rows_written, + exemplars_written: 0, })?; write_cost += output.meta.cost; rows_written += rows; @@ -646,6 +785,7 @@ async fn write_prometheus_rows_with_progress( return Err(PromWriteError { error: incomplete_prom_write_error(), rows_written, + exemplars_written: 0, }); } @@ -672,6 +812,7 @@ async fn write_prepared_prometheus_rows_with_progress( .map_err(|error| PromWriteError { error, rows_written, + exemplars_written: 0, })?; write_cost += output.meta.cost; rows_written += rows; @@ -698,6 +839,7 @@ async fn write_prometheus_v2_rows_with_progress( error, samples_written: 0, histograms_written: 0, + exemplars_written: 0, })?; let batcher = if prom_store_with_metric_engine && let Some(batcher) = pending_rows_batcher.as_ref() @@ -795,6 +937,7 @@ async fn write_batched_prometheus_v2_rows_with_progress( error, samples_written, histograms_written, + exemplars_written: 0, }); } } @@ -977,6 +1120,214 @@ async fn decode_remote_read_request( ReadRequest::decode(&buf[..]).context(error::DecodePromRemoteRequestSnafu) } +#[cfg(test)] +mod exemplar_accounting_tests { + use super::*; + use crate::http::{HttpOptions, HttpServerBuilder}; + use crate::query_handler::PromStoreProtocolHandler; + use common_memory_manager::OnExhaustedPolicy; + use common_query::Output; + use tower::ServiceExt; + + use axum::body::Body; + use axum::http::Request; + + /// Reports written rows without touching counters, with an optional exemplar + /// failure, so ingestion accounting can be observed in isolation. + struct RecordingHandler { + fail_exemplars: bool, + } + + #[async_trait] + impl PromStoreProtocolHandler for RecordingHandler { + async fn pre_write(&self, _: &RowInsertRequests, _: QueryContextRef) -> Result<()> { + Ok(()) + } + async fn check_exemplar_sources( + &self, + _: &std::collections::BTreeSet, + _: QueryContextRef, + ) -> Result<()> { + Ok(()) + } + async fn write_exemplars_prepared( + &self, + request: RowInsertRequests, + _: QueryContextRef, + ) -> Result { + if self.fail_exemplars { + return Err(crate::error::NotSupportedSnafu { + feat: "injected failure", + } + .build()); + } + Ok(Output::new_with_affected_rows( + prom_write_row_count(&request) as usize, + )) + } + async fn write_prepared( + &self, + request: RowInsertRequests, + _: QueryContextRef, + _: bool, + ) -> Result { + Ok(Output::new_with_affected_rows( + prom_write_row_count(&request) as usize, + )) + } + async fn write(&self, _: RowInsertRequests, _: QueryContextRef, _: bool) -> Result { + unreachable!() + } + async fn write_all(&self, _: Vec, _: bool) -> Result>> { + unreachable!() + } + async fn read(&self, _: ReadRequest, _: QueryContextRef) -> Result { + unreachable!() + } + } + + async fn post_v1_exemplars( + db: &str, + handler: Arc, + sample_count: usize, + exemplar_count: usize, + through_http: bool, + ) -> StatusCode { + use api::prom_store::remote; + + let request = remote::WriteRequest { + // Separate sources keep sample writes on write_prepared rather than + // requiring the stub to resolve real metric-engine table identities. + timeseries: vec![ + remote::TimeSeries { + labels: vec![remote::Label { + name: "__name__".into(), + value: "counter_test_samples".into(), + }], + samples: (0..sample_count) + .map(|i| remote::Sample { + value: i as f64, + timestamp: i as i64, + }) + .collect(), + ..Default::default() + }, + remote::TimeSeries { + labels: vec![remote::Label { + name: "__name__".into(), + value: "counter_test_metric".into(), + }], + exemplars: vec![remote::Exemplar::default(); exemplar_count], + ..Default::default() + }, + ], + ..Default::default() + }; + let body = crate::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + let memory_limiter = ServerMemoryLimiter::new(1024 * 1024, OnExhaustedPolicy::Fail); + if !through_http { + let state = PromStoreState { + prom_store_handler: handler, + pipeline_handler: None, + prom_store_with_metric_engine: true, + prom_validation_mode: PromValidationMode::Strict, + pending_rows_batcher: None, + memory_limiter, + }; + let result = ingest_remote_write_v1( + &state, + db, + PipelineInfo { + pipeline_name: None, + pipeline_version: None, + pipeline_params: Default::default(), + }, + false, + QueryContext::arc(), + Bytes::from(body), + 0, + ) + .await; + return if result.is_ok() { + StatusCode::NO_CONTENT + } else { + StatusCode::BAD_REQUEST + }; + } + let server = HttpServerBuilder::new(HttpOptions::default()) + .with_memory_limiter(memory_limiter) + .with_prom_handler(handler, None, true, PromValidationMode::Strict, None) + .build(); + server + .build(server.make_app()) + .unwrap() + .oneshot( + Request::post(format!("/v1/prometheus/write?db={db}")) + .header(header::CONTENT_TYPE, "application/x-protobuf") + .header(header::CONTENT_ENCODING, DEFAULT_ENCODING) + .body(Body::from(body)) + .unwrap(), + ) + .await + .unwrap() + .status() + } + + fn counter(db: &str, version: &str, exemplars: bool) -> u64 { + if exemplars { + crate::metrics::PROM_STORE_REMOTE_WRITE_EXEMPLARS + .with_label_values(&[db, version]) + .get() + } else { + crate::metrics::PROM_STORE_REMOTE_WRITE_SAMPLES + .with_label_values(&[db, version]) + .get() + } + } + + /// HTTP and direct core callers (COPY) count successful and partial writes + /// exactly once; transport wrappers must not duplicate core accounting. + #[tokio::test] + async fn v1_exemplar_counters_are_recorded_once() { + // Counters are process-global, so use an isolated db label and deltas. + let db = format!("once-{}", std::process::id()); + for (through_http, fail_exemplars, samples, exemplars) in [ + (true, false, 10, 2), + (true, true, 3, 1), + (false, false, 10, 2), + (false, true, 3, 1), + ] { + let samples_before = counter(&db, REMOTE_WRITE_V1_VERSION, false); + let exemplars_before = counter(&db, REMOTE_WRITE_V1_VERSION, true); + let response = post_v1_exemplars( + &db, + Arc::new(RecordingHandler { fail_exemplars }), + samples, + exemplars, + through_http, + ) + .await; + assert_eq!( + response, + if fail_exemplars { + StatusCode::BAD_REQUEST + } else { + StatusCode::NO_CONTENT + } + ); + // A failed exemplar write still leaves the successful samples counted. + assert_eq!( + counter(&db, REMOTE_WRITE_V1_VERSION, false) - samples_before, + samples as u64 + ); + assert_eq!( + counter(&db, REMOTE_WRITE_V1_VERSION, true) - exemplars_before, + if fail_exemplars { 0 } else { exemplars as u64 } + ); + } + } +} + #[cfg(test)] mod tests { use std::sync::Mutex; diff --git a/src/servers/src/http/prom_store/exemplar.rs b/src/servers/src/http/prom_store/exemplar.rs index d6524e9ce71..30ad9246bc2 100644 --- a/src/servers/src/http/prom_store/exemplar.rs +++ b/src/servers/src/http/prom_store/exemplar.rs @@ -17,6 +17,7 @@ use std::sync::Arc; use api::v1::RowInsertRequests; use common_error::ext::BoxedError; +use common_query::Output; use futures::future::ready; use operator::insert::PreparedMetricInserts; use session::context::QueryContextRef; @@ -37,6 +38,24 @@ pub(super) struct Outcome { pub error: Option, } +impl Outcome { + async fn record(&mut self, kind: usize, output: Result) { + match output { + Ok(output) => { + self.cost += output.meta.cost; + let progress = output.write_progress().await; + self.written[kind] += progress.affected_rows as u64; + if let Err(error) = progress.into_result().context(error::ExecuteGrpcQuerySnafu) { + self.error.get_or_insert(error); + } + } + Err(error) => { + self.error.get_or_insert(error); + } + } + } +} + enum PreparedWrite { Metric(PreparedMetricInserts), Rows(RowInsertRequests), @@ -56,7 +75,7 @@ pub(super) async fn write( Ok((writes, targets)) => { let mut sources = Vec::new(); let mut prepared = Vec::new(); - let mut source_preparation_failed = false; + let mut source_failed = false; for (kind, ctx, request) in writes { if kind != 2 && metric_engine { let (pinned, other): (Vec<_>, Vec<_>) = @@ -84,7 +103,7 @@ pub(super) async fn write( if let Ok(request) = &result { sources.extend(request.source_tables().cloned()); } else { - source_preparation_failed = true; + source_failed = true; } prepared.push((kind, ctx, result.map(PreparedWrite::Metric))); } @@ -93,7 +112,7 @@ pub(super) async fn write( } } // A failed source preparation must not fall back to a newer table with the same name. - if source_preparation_failed { + if source_failed { prepared.retain(|(kind, _, _)| *kind != 2); } let sample_batches = prepared.iter().filter_map(|(kind, ctx, request)| { @@ -113,9 +132,11 @@ pub(super) async fn write( } else { None }; - let mut outputs = Vec::with_capacity(prepared.len()); - // Serialize schema preparation across batches that resolve to the same table. - for (kind, ctx, request) in prepared { + let (related, other): (Vec<_>, Vec<_>) = + prepared.into_iter().partition(|(kind, _, request)| { + *kind == 2 || !matches!(request, Ok(PreparedWrite::Rows(_))) + }); + let submit = async |(kind, ctx, request)| { let mut deferred = false; let output = match request { Err(error) => Err(error), @@ -145,29 +166,41 @@ pub(super) async fn write( } } }; - outputs.push((kind, output, deferred)); - } + (output, deferred) + }; let mut outcome = Outcome::default(); + let ((), outputs) = tokio::join!( + async { + for write in related { + let kind = write.0; + if kind == 2 && source_failed { + continue; + } + let (output, _) = submit(write).await; + // Successful submissions can still carry source storage failures. + outcome.record(kind, output).await; + source_failed |= kind != 2 && outcome.error.is_some(); + } + }, + async { + let mut outputs = Vec::with_capacity(other.len()); + // Serialize schema preparation for batches resolving to the same table. + for write in other { + let kind = write.0; + let (output, deferred) = submit(write).await; + outputs.push((kind, output, deferred)); + } + outputs + }, + ); let mut pending_samples = Vec::new(); for (kind, output, deferred) in outputs { match output { - Ok(output) => { + Ok(output) if deferred && !output.meta.write_completions.is_empty() => { outcome.cost += output.meta.cost; - if deferred && !output.meta.write_completions.is_empty() { - pending_samples.push(output); - continue; - } - let progress = output.write_progress().await; - outcome.written[kind] += progress.affected_rows as u64; - if let Err(error) = - progress.into_result().context(error::ExecuteGrpcQuerySnafu) - { - outcome.error.get_or_insert(error); - } - } - Err(error) => { - outcome.error.get_or_insert(error); + pending_samples.push(output); } + output => outcome.record(kind, output).await, } } if outcome.error.is_none() { @@ -265,17 +298,85 @@ mod tests { use crate::exemplar::Exemplar; use crate::query_handler::{PromStoreProtocolHandler, PromStoreResponse}; + #[derive(Clone, Copy, Default)] + enum WriteResult { + #[default] + Success, + ImmediateFailure, + CompletionFailure, + } + #[derive(Default)] struct Handler { events: Mutex>, - exemplar_done: Arc, + source_result: WriteResult, + other_result: WriteResult, + source_completion: Option>, + other_submission: Option>, deny_schema: bool, fail_preparation: bool, - successful_writes: bool, check_serial: bool, writing: AtomicBool, } + impl Handler { + async fn write_metric(&self, ctx: QueryContextRef, source: bool) -> Result { + if self.check_serial && !source { + assert!(!self.writing.swap(true, Ordering::SeqCst)); + tokio::task::yield_now().await; + self.writing.store(false, Ordering::SeqCst); + } + assert!(ctx.confirmed_writes()); + assert_eq!( + ctx.write_rows_to_admit(ctx.current_catalog(), &ctx.current_schema(), 3), + 0 + ); + self.events.lock().unwrap().push("metric".into()); + let result = if source { + self.source_result + } else { + self.other_result + }; + if matches!(result, WriteResult::ImmediateFailure) { + return error::InternalSnafu { + err_msg: "immediate metric failure", + } + .fail(); + } + let gate = source.then(|| self.source_completion.clone()).flatten(); + let counter = ctx.write_progress().cloned(); + let completion = WriteCompletion::with_progress(async move { + if let Some(gate) = gate { + gate.notified().await; + } + let failed = matches!(result, WriteResult::CompletionFailure); + let affected_rows = if failed { 1 } else { 2 }; + if let Some(counter) = counter { + counter.add_written(affected_rows); + } + WriteProgress { + affected_rows, + error: failed.then(|| { + Arc::new(BoxedError::new( + error::InternalSnafu { + err_msg: "late metric failure", + } + .build(), + )) + }), + } + }); + Ok(Output::new( + OutputData::AffectedRows(2), + OutputMeta { + cost: 2, + write_completions: vec![completion], + ..Default::default() + }, + )) + } + } + #[async_trait] impl PromStoreProtocolHandler for Handler { async fn prepare_metric_write( @@ -297,8 +398,7 @@ mod tests { _: PreparedMetricInserts, ctx: QueryContextRef, ) -> Result { - self.write_prepared(RowInsertRequests::default(), ctx, true) - .await + self.write_metric(ctx, true).await } async fn pre_write(&self, request: &RowInsertRequests, ctx: QueryContextRef) -> Result<()> { @@ -338,48 +438,10 @@ mod tests { ctx: QueryContextRef, _: bool, ) -> Result { - if self.check_serial { - assert!(!self.writing.swap(true, Ordering::SeqCst)); - tokio::task::yield_now().await; - self.writing.store(false, Ordering::SeqCst); + if let Some(gate) = &self.other_submission { + gate.notified().await; } - assert!(ctx.confirmed_writes()); - assert_eq!( - ctx.write_rows_to_admit(ctx.current_catalog(), &ctx.current_schema(), 3), - 0 - ); - self.events.lock().unwrap().push("metric".into()); - if self.successful_writes { - if let Some(counter) = ctx.write_progress() { - counter.add_written(2); - } - return Ok(Output::new_with_affected_rows(2)); - } - let done = self.exemplar_done.clone(); - let counter = ctx.write_progress().cloned(); - let completion = WriteCompletion::with_progress(async move { - done.notified().await; - if let Some(counter) = counter { - counter.add_written(1); - } - WriteProgress { - affected_rows: 1, - error: Some(Arc::new(BoxedError::new( - error::InternalSnafu { - err_msg: "late metric failure", - } - .build(), - ))), - } - }); - Ok(Output::new( - OutputData::AffectedRows(2), - OutputMeta { - cost: 2, - write_completions: vec![completion], - ..Default::default() - }, - )) + self.write_metric(ctx, false).await } async fn write_exemplars_prepared( @@ -396,7 +458,6 @@ mod tests { if let Some(counter) = ctx.write_progress() { counter.add_written(1); } - self.exemplar_done.notify_one(); Ok(Output::new( OutputData::AffectedRows(1), OutputMeta::new_with_cost(1), @@ -418,6 +479,34 @@ mod tests { } } + fn metric_request(tables: &[&str], num_rows: usize) -> RowInsertRequests { + RowInsertRequests { + inserts: tables + .iter() + .map(|name| RowInsertRequest { + table_name: (*name).into(), + rows: Some(Rows { + schema: vec![], + rows: vec![Row { values: vec![] }; num_rows], + }), + }) + .collect(), + } + } + + fn exemplar_requests(schema: Option) -> ExemplarRequests { + ExemplarRequests::from([( + schema, + vec![Exemplar { + metric: "metric".into(), + series_labels: BTreeMap::new().into(), + labels: BTreeMap::new(), + timestamp: 1, + value: 1.0, + }], + )]) + } + #[tokio::test] async fn exemplar_requires_successful_metric_engine_source_preparation() { for metric_engine in [false, true] { @@ -427,38 +516,17 @@ mod tests { }); let ctx = QueryContext::arc(); let samples = if metric_engine { - vec![( - ctx.clone(), - RowInsertRequests { - inserts: vec![RowInsertRequest { - table_name: "metric".into(), - rows: Some(Rows { - schema: vec![], - rows: vec![Row { values: vec![] }], - }), - }], - }, - )] + vec![(ctx.clone(), metric_request(&["metric"], 1))] } else { vec![] }; - let exemplars = ExemplarRequests::from([( - None, - vec![Exemplar { - metric: "metric".into(), - series_labels: BTreeMap::new().into(), - labels: BTreeMap::new(), - timestamp: 1, - value: 1.0, - }], - )]); let outcome = write( handler.clone(), None, metric_engine, samples, vec![], - exemplars, + exemplar_requests(None), ctx, ) .await; @@ -488,29 +556,8 @@ mod tests { if let Some(physical) = physical { target.set_extension(PHYSICAL_TABLE_PARAM, physical); } - ( - Arc::new(target), - RowInsertRequests { - inserts: vec![RowInsertRequest { - table_name: "metric".into(), - rows: Some(Rows { - schema: vec![], - rows: vec![Row { values: vec![] }], - }), - }], - }, - ) + (Arc::new(target), metric_request(&["metric"], 1)) }; - let exemplars = ExemplarRequests::from([( - None, - vec![Exemplar { - metric: "metric".into(), - series_labels: BTreeMap::new().into(), - labels: BTreeMap::new(), - timestamp: 1, - value: 1.0, - }], - )]); let (writes, targets) = prepare( &handler, vec![ @@ -519,7 +566,7 @@ mod tests { batch(None), ], vec![batch(Some("histogram_physical"))], - exemplars, + exemplar_requests(None), ctx, ) .await @@ -547,45 +594,154 @@ mod tests { } #[tokio::test] - async fn exemplar_submits_metric_batches_sequentially() { + async fn exemplar_submits_unrelated_metric_batches_sequentially() { let handler = Arc::new(Handler { - successful_writes: true, check_serial: true, ..Default::default() }); let ctx = QueryContext::arc(); let samples = ["metric", "other", "other"] .into_iter() - .map(|name| { - ( - Arc::new(ctx.fork()), - RowInsertRequests { - inserts: vec![RowInsertRequest { - table_name: name.into(), - rows: Some(Rows { - schema: vec![], - rows: vec![Row { values: vec![] }; 2], - }), - }], - }, - ) - }) + .map(|name| (Arc::new(ctx.fork()), metric_request(&[name], 2))) .collect(); - let exemplars = ExemplarRequests::from([( + let outcome = write( + handler, None, - vec![Exemplar { - metric: "metric".into(), - series_labels: BTreeMap::new().into(), - labels: BTreeMap::new(), - timestamp: 1, - value: 1.0, - }], - )]); - let outcome = write(handler, None, true, samples, vec![], exemplars, ctx).await; + true, + samples, + vec![], + exemplar_requests(None), + ctx, + ) + .await; assert!(outcome.error.is_none()); assert_eq!(outcome.written, [6, 0, 1]); } + #[tokio::test] + async fn exemplar_waits_for_source_completion() { + for histogram in [false, true] { + for failed in [false, true] { + for block_source in [false, true] { + let gate = Arc::new(Notify::new()); + let handler = Arc::new(Handler { + source_result: if failed { + WriteResult::CompletionFailure + } else { + WriteResult::Success + }, + source_completion: block_source.then(|| gate.clone()), + other_submission: (!block_source).then(|| gate.clone()), + ..Default::default() + }); + let ctx = QueryContext::arc(); + let batch = |name: &str| (ctx.clone(), metric_request(&[name], 2)); + let (samples, histograms) = if histogram { + (vec![batch("other")], vec![batch("metric")]) + } else { + (vec![batch("metric"), batch("other")], vec![]) + }; + let response = write( + handler.clone(), + None, + true, + samples, + histograms, + exemplar_requests(None), + ctx, + ); + tokio::pin!(response); + assert!(futures::poll!(&mut response).is_pending()); + { + let events = handler.events.lock().unwrap(); + assert_eq!( + events.iter().filter(|event| *event == "metric").count(), + 1 + usize::from(block_source) + ); + assert_eq!( + events.iter().any(|event| event == "exemplar"), + !block_source && !failed + ); + } + gate.notify_one(); + let outcome = response.await; + let source_rows = if failed { 1 } else { 2 }; + let exemplars = u64::from(!failed); + assert_eq!(outcome.error.is_some(), failed); + assert_eq!( + outcome.written, + if histogram { + [2, source_rows, exemplars] + } else { + [2 + source_rows, 0, exemplars] + } + ); + assert_eq!(outcome.cost, 4 + exemplars as usize); + } + } + } + } + + #[tokio::test] + async fn exemplar_only_depends_on_its_source_writes() { + for (source_result, other_result, source_rows, other_rows, exemplars) in [ + ( + WriteResult::ImmediateFailure, + WriteResult::Success, + Some(0), + 2, + 0, + ), + ( + WriteResult::Success, + WriteResult::ImmediateFailure, + Some(2), + 0, + 1, + ), + ( + WriteResult::Success, + WriteResult::CompletionFailure, + Some(2), + 1, + 1, + ), + ( + WriteResult::Success, + WriteResult::CompletionFailure, + None, + 1, + 1, + ), + ] { + let handler = Arc::new(Handler { + source_result, + other_result, + ..Default::default() + }); + let ctx = QueryContext::arc(); + let tables = [Some("other"), source_rows.map(|_| "metric")] + .into_iter() + .flatten() + .collect::>(); + let outcome = write( + handler, + None, + true, + vec![(ctx.clone(), metric_request(&tables, 2))], + vec![], + exemplar_requests(None), + ctx, + ) + .await; + assert!(outcome.error.is_some()); + assert_eq!( + outcome.written, + [source_rows.unwrap_or(0) + other_rows, 0, exemplars] + ); + } + } + #[tokio::test] async fn exemplar_v2_batching_acknowledgement_and_partial_counts() { use api::greptime_proto::io::prometheus::write::v2; @@ -650,12 +806,17 @@ mod tests { }, ], }; + let handler = Arc::new(Handler { + source_result: if successful_writes { + WriteResult::Success + } else { + WriteResult::CompletionFailure + }, + ..Default::default() + }); let response = remote_write_v2( PromStoreState { - prom_store_handler: Arc::new(Handler { - successful_writes, - ..Default::default() - }), + prom_store_handler: handler.clone(), pipeline_handler: None, prom_store_with_metric_engine: true, prom_validation_mode: PromValidationMode::Strict, @@ -689,6 +850,15 @@ mod tests { command = receiver.recv() => command.unwrap(), }; assert_eq!(early_response.is_some(), !sync && successful_writes); + assert_eq!( + handler + .events + .lock() + .unwrap() + .iter() + .any(|event| event == "exemplar"), + successful_writes + ); assert!(limiter.used_bytes() > 0); let WorkerCommand::Submit { ctx, @@ -730,18 +900,17 @@ mod tests { for (kind, count) in [ ("samples", samples), ("histograms", "0"), - ("exemplars", "1"), + ("exemplars", if successful_writes { "1" } else { "0" }), ] { assert_eq!( response.headers()[format!("x-prometheus-remote-write-{kind}-written")], count ); } - for ((counter, before), count) in counters - .into_iter() - .zip(before) - .zip([if successful_writes { 4 } else { 2 }, 1]) - { + for ((counter, before), count) in counters.into_iter().zip(before).zip([ + if successful_writes { 4 } else { 2 }, + u64::from(successful_writes), + ]) { assert_eq!(counter.get() - before, count); } drop(ctx); @@ -754,38 +923,15 @@ mod tests { for denied in [false, true] { let handler = Arc::new(Handler { deny_schema: denied, + source_result: WriteResult::CompletionFailure, ..Default::default() }); - let mut exemplars = ExemplarRequests::new(); - for schema in if denied { - vec![None, Some("other".into())] - } else { - vec![None] - } { - exemplars.insert( - schema, - vec![Exemplar { - metric: "metric".into(), - series_labels: BTreeMap::new().into(), - labels: BTreeMap::new(), - timestamp: 1, - value: 1.0, - }], - ); + let mut exemplars = exemplar_requests(None); + if denied { + exemplars.extend(exemplar_requests(Some("other".into()))); } let ctx = QueryContext::arc(); - let samples = vec![( - ctx.clone(), - RowInsertRequests { - inserts: vec![RowInsertRequest { - table_name: "metric".into(), - rows: Some(Rows { - schema: vec![], - rows: vec![Row { values: vec![] }; 2], - }), - }], - }, - )]; + let samples = vec![(ctx.clone(), metric_request(&["metric"], 2))]; let outcome = write(handler.clone(), None, true, samples, vec![], exemplars, ctx).await; assert!(outcome.error.is_some()); let events = handler.events.lock().unwrap(); @@ -793,17 +939,11 @@ mod tests { assert_eq!(outcome.written, [0, 0, 0]); assert!(!events.iter().any(|e| e == "metric" || e == "exemplar")); } else { - assert_eq!(outcome.written, [1, 0, 1]); - assert_eq!(outcome.cost, 3); + assert_eq!(outcome.written, [1, 0, 0]); + assert_eq!(outcome.cost, 2); assert_eq!( *events, - [ - "pre:public", - "pre:public", - "source:public", - "metric", - "exemplar" - ] + ["pre:public", "pre:public", "source:public", "metric"] ); } } diff --git a/src/servers/src/http/prometheus.rs b/src/servers/src/http/prometheus.rs index 28fed296ed0..ed2238bcde2 100644 --- a/src/servers/src/http/prometheus.rs +++ b/src/servers/src/http/prometheus.rs @@ -26,11 +26,11 @@ use arrow::datatypes::{ UInt8Type, UInt16Type, UInt32Type, UInt64Type, }; use arrow_schema::{DataType, IntervalUnit}; -use auth::{PermissionTableTarget, PermissionTableTargets}; +use auth::{PermissionTableTarget, PermissionTableTargets, UserProviderRef}; use axum::extract::{Path, Query, State}; use axum::{Extension, Form}; use catalog::CatalogManagerRef; -use common_catalog::parse_catalog_and_schema_from_db_string; +use common_catalog::consts; use common_decimal::Decimal128; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; @@ -77,6 +77,7 @@ use crate::error::{ CollectRecordbatchSnafu, ConvertScalarValueSnafu, DataFusionSnafu, Error, InvalidQuerySnafu, NotSupportedSnafu, ParseTimestampSnafu, Result, TableNotFoundSnafu, UnexpectedResultSnafu, }; +use crate::http::authorize::select_database; use crate::http::header::collect_plan_metrics; use crate::otlp::metrics::ucum_to_openmetrics_unit; use crate::prom_store::{FIELD_NAME_LABEL, METRIC_NAME_LABEL, is_database_selection_label}; @@ -238,14 +239,20 @@ pub struct ExemplarQuery { fields(protocol = "prometheus", request_type = "query_exemplars") )] pub async fn query_exemplars( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut ctx): Extension, Form(form): Form, ) -> PrometheusJsonResponse { - if let Some(db) = params.db.or(form.db) { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(&db); - try_update_catalog_schema(&mut ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut ctx, + params.db.or(form.db).as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); } ctx.set_channel(Channel::Prometheus); let ctx = Arc::new(ctx); @@ -357,12 +364,20 @@ pub async fn build_info_query() -> PrometheusJsonResponse { fields(protocol = "prometheus", request_type = "metadata_query") )] pub async fn metadata_query( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, ) -> PrometheusJsonResponse { - let (catalog, schema) = get_catalog_schema(¶ms.db, &query_ctx); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); + } let query_ctx = Arc::new(query_ctx); if let Err(err) = handler.check_query_permission(&[], &query_ctx).await { @@ -460,6 +475,7 @@ macro_rules! try_call_return_response { fields(protocol = "prometheus", request_type = "instant_query") )] pub async fn instant_query( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, @@ -483,9 +499,14 @@ pub async fn instant_query( }; // update catalog and schema in query context if necessary - if let Some(db) = ¶ms.db { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); } let query_ctx = Arc::new(query_ctx); let _timer = crate::metrics::METRIC_HTTP_PROMETHEUS_PROMQL_ELAPSED @@ -597,6 +618,7 @@ pub struct RangeQuery { fields(protocol = "prometheus", request_type = "range_query") )] pub async fn range_query( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, @@ -615,9 +637,14 @@ pub async fn range_query( }; // update catalog and schema in query context if necessary - if let Some(db) = ¶ms.db { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); } let query_ctx = Arc::new(query_ctx); let _timer = crate::metrics::METRIC_HTTP_PROMETHEUS_PROMQL_ELAPSED @@ -797,13 +824,21 @@ macro_rules! handle_schema_err { fields(protocol = "prometheus", request_type = "labels_query") )] pub async fn labels_query( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, Form(form_params): Form, ) -> PrometheusJsonResponse { - let (catalog, schema) = get_catalog_schema(¶ms.db, &query_ctx); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); + } let query_ctx = Arc::new(query_ctx); let mut queries = params.matches.0; @@ -856,10 +891,13 @@ pub async fn labels_query( prom_queries }; + let catalog = query_ctx.current_catalog(); + let schema = query_ctx.current_schema(); + // Fetch all columns if no query matcher is provided if prom_queries.is_empty() { let (mut labels, table_names) = - match get_all_column_names(&catalog, &schema, &handler.catalog_manager()).await { + match get_all_column_names(catalog, &schema, &handler.catalog_manager()).await { Ok(result) => result, Err(e) => return PrometheusJsonResponse::error(e.status_code(), e.output_msg()), }; @@ -885,7 +923,7 @@ pub async fn labels_query( Err(e) => return PrometheusJsonResponse::error(e.status_code(), e.output_msg()), } } - None => match get_all_column_names(&catalog, &schema, &handler.catalog_manager()).await { + None => match get_all_column_names(catalog, &schema, &handler.catalog_manager()).await { Ok((labels, _)) => labels, Err(e) => return PrometheusJsonResponse::error(e.status_code(), e.output_msg()), }, @@ -927,7 +965,7 @@ async fn get_all_column_names( let mut table_names = Vec::new(); let mut tables = manager.tables(catalog, schema, None); while let Some(table) = tables.try_next().await? { - if is_internal_physical_metric_table(&table) { + if is_internal_metric_table(&table) { continue; } table_names.push(table.table_info().name.clone()); @@ -952,7 +990,7 @@ async fn get_target_column_names( Some(query_ctx), ) .await? - && !is_internal_physical_metric_table(&table) + && !is_internal_metric_table(&table) { extend_tag_column_names(&mut labels, &table); } @@ -1402,27 +1440,6 @@ pub(crate) fn retrieve_metric_name_and_result_type( ) } -/// Tries to get catalog and schema from an optional db param. And retrieves -/// them from [QueryContext] if they don't present. -pub(crate) fn get_catalog_schema(db: &Option, ctx: &QueryContext) -> (String, String) { - if let Some(db) = db { - parse_catalog_and_schema_from_db_string(db) - } else { - ( - ctx.current_catalog().to_string(), - ctx.current_schema().clone(), - ) - } -} - -/// Update catalog and schema in [QueryContext] if necessary. -pub(crate) fn try_update_catalog_schema(ctx: &mut QueryContext, catalog: &str, schema: &str) { - if ctx.current_catalog() != catalog || ctx.current_schema() != schema { - ctx.set_current_catalog(catalog); - ctx.set_current_schema(schema); - } -} - fn current_schema_metric_targets( query_ctx: &QueryContext, metric_names: &[String], @@ -1441,8 +1458,9 @@ fn current_schema_metric_targets( ) } -fn is_internal_physical_metric_table(table: &TableRef) -> bool { - table.table_info().is_physical_table() +fn is_internal_metric_table(table: &TableRef) -> bool { + let info = table.table_info(); + info.is_physical_table() || info.name == consts::METRICS_EXEMPLARS_TABLE } fn promql_expr_to_metric_name(expr: &PromqlExpr) -> Option { @@ -1831,13 +1849,21 @@ pub struct LabelValueQuery { fields(protocol = "prometheus", request_type = "label_values_query") )] pub async fn label_values_query( + user_provider: Option>, State(handler): State, Path(label_name): Path, Extension(mut query_ctx): Extension, Query(params): Query, ) -> PrometheusJsonResponse { - let (catalog, schema) = get_catalog_schema(¶ms.db, &query_ctx); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); + } let query_ctx = Arc::new(query_ctx); let _timer = crate::metrics::METRIC_HTTP_PROMETHEUS_PROMQL_ELAPSED @@ -2198,7 +2224,7 @@ async fn retrieve_table_names( .options .extra_options .contains_key(LOGICAL_TABLE_METADATA_KEY) - || is_internal_physical_metric_table(&table) + || is_internal_metric_table(&table) { // skip non-prometheus (non-metricengine) tables for __name__ query continue; @@ -2389,7 +2415,7 @@ async fn readable_metadata_tables( } } candidates.retain(|table| { - !is_internal_physical_metric_table(table) + !is_internal_metric_table(table) && (selectors.is_empty() || selectors .iter() @@ -2492,6 +2518,7 @@ pub struct SeriesQuery { fields(protocol = "prometheus", request_type = "series_query") )] pub async fn series_query( + user_provider: Option>, State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, @@ -2521,9 +2548,14 @@ pub async fn series_query( .unwrap_or_else(|| DEFAULT_LOOKBACK_STRING.to_string()); // update catalog and schema in query context if necessary - if let Some(db) = ¶ms.db { - let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - try_update_catalog_schema(&mut query_ctx, &catalog, &schema); + if let Err(error) = select_database( + &mut query_ctx, + params.db.as_deref(), + user_provider.as_ref().map(|Extension(provider)| provider), + ) + .await + { + return PrometheusJsonResponse::error(error.status_code(), error.output_msg()); } let query_ctx = Arc::new(query_ctx); @@ -2872,6 +2904,105 @@ mod tests { ); } + #[tokio::test] + async fn test_prometheus_endpoints_authorize_effective_database() { + use axum::http::{Request, StatusCode as HttpStatusCode}; + use axum::response::IntoResponse; + use tower::ServiceExt; + + for authorization in ["Basic cmVhZGVyOnB3ZA==", "Bearer test-token"] { + for (path, form_db, encoded_db) in [ + ("/v1/prometheus/api/v1/query", false, false), + ("/v1/prometheus/api/v1/query_range", false, false), + ("/v1/prometheus/api/v1/metadata", false, false), + ("/v1/prometheus/api/v1/labels", false, false), + ("/v1/prometheus/api/v1/label/job/values", false, false), + ("/v1/prometheus/api/v1/series", false, false), + ("/v1/prometheus/api/v1/query_exemplars", false, false), + ("/v1/prometheus/api/v1/query_exemplars", true, false), + ("/v1/prometheus/api/v1/query", false, true), + ] { + let handler = Arc::new(TestPrometheusHandler { + catalog_manager: MemoryCatalogManager::new(), + deny_operation: false, + denied_table: None, + metric_names: Vec::new(), + label_metric_names: Vec::new(), + label_lookups: Mutex::new(Vec::new()), + queries: Mutex::new(Vec::new()), + ordered_outputs: Mutex::new(Vec::new()), + }); + let app = axum::Router::new() + .route( + "/v1/prometheus/api/v1/query", + axum::routing::get(instant_query), + ) + .route( + "/v1/prometheus/api/v1/query_range", + axum::routing::get(range_query), + ) + .route( + "/v1/prometheus/api/v1/metadata", + axum::routing::get(metadata_query), + ) + .route( + "/v1/prometheus/api/v1/labels", + axum::routing::get(labels_query), + ) + .route( + "/v1/prometheus/api/v1/label/{name}/values", + axum::routing::get(label_values_query), + ) + .route( + "/v1/prometheus/api/v1/series", + axum::routing::get(series_query), + ) + .route( + "/v1/prometheus/api/v1/query_exemplars", + axum::routing::get(query_exemplars).post(query_exemplars), + ) + .with_state(handler.clone() as PrometheusHandlerRef) + .layer(axum::middleware::from_fn_with_state( + crate::http::authorize::AuthState::new(Some(Arc::new( + crate::http::authorize::tests::ScopedUserProvider, + ))), + crate::http::authorize::check_http_auth, + )); + let db = if encoded_db { "%64enied" } else { "denied" }; + let query = if form_db { + String::new() + } else { + format!("&db={db}") + }; + let mut request = Request::builder() + .method(if form_db { "POST" } else { "GET" }) + .uri(format!( + "{path}?query=requests&start=0&end=1&step=1&match[]=requests{query}" + )) + .header("authorization", authorization) + .header("content-type", "application/x-www-form-urlencoded"); + if !encoded_db { + request = request.header("x-greptime-db-name", "public"); + } + let body = if form_db { "db=denied" } else { "" }; + let response = tokio::time::timeout( + std::time::Duration::from_secs(5), + app.oneshot(request.body(axum::body::Body::from(body)).unwrap()), + ) + .await + .unwrap() + .unwrap(); + assert_eq!( + HttpStatusCode::FORBIDDEN, + response.into_response().status(), + "{path}" + ); + assert!(handler.queries.lock().unwrap().is_empty()); + assert!(handler.label_lookups.lock().unwrap().is_empty()); + } + } + } + #[tokio::test] async fn test_exemplar_timeout_cancels_scan() { for explicit in [false, true] { @@ -2890,6 +3021,7 @@ mod tests { ctx.set_query_timeout(std::time::Duration::from_millis(1)); } let response = query_exemplars( + None, State(handler.clone()), Query(ExemplarQuery { query: Some("requests".into()), @@ -2921,6 +3053,7 @@ mod tests { }); let state: PrometheusHandlerRef = handler.clone(); instant_query( + None, State(state.clone()), Query(InstantQuery { query: Some("sort(vector(1))".to_string()), @@ -2938,6 +3071,7 @@ mod tests { // Both a single-point and a multi-step range query take the same path. for end in ["0", "1"] { range_query( + None, State(state.clone()), Query(RangeQuery { query: Some("sort(vector(1))".to_string()), @@ -2975,6 +3109,7 @@ mod tests { ordered_outputs: Mutex::new(Vec::new()), }); let response = range_query( + None, State(handler.clone() as PrometheusHandlerRef), Query(RangeQuery { query: Some("vector(1)[5m:]".to_string()), @@ -3020,6 +3155,7 @@ mod tests { .with_label_values(&[db.as_str(), "instant_query"]); let instant_count = instant_histogram.get_sample_count(); let response = instant_query( + None, State(handler.clone()), Query(InstantQuery { query: Some("up{".to_string()), @@ -3036,6 +3172,7 @@ mod tests { .with_label_values(&[db.as_str(), "range_query"]); let range_count = range_histogram.get_sample_count(); let response = range_query( + None, State(handler), Query(RangeQuery { query: Some("up{".to_string()), @@ -3163,6 +3300,7 @@ mod tests { assert!(schemas.contains(&"empty".to_string())); assert!(!schemas.contains(&"denied_schema".to_string())); let response = label_values_query( + None, State(handler), Path(FIELD_NAME_LABEL.to_string()), Extension(QueryContext::with( @@ -3222,6 +3360,7 @@ mod tests { ) -> Vec { let state: PrometheusHandlerRef = handler; let response = label_values_query( + None, State(state), Path(METRIC_NAME_LABEL.to_string()), Extension(QueryContext::with( @@ -3359,6 +3498,7 @@ mod tests { let query_ctx = QueryContext::with(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME); let target_ctx = Arc::new(query_ctx.clone()); let response = series_query( + None, State(state), Query(SeriesQuery { matches: Matches(vec![r#"{__name__=~"cpu_.*"}"#.to_string()]), @@ -4019,6 +4159,126 @@ mod tests { assert_eq!(fields, HashSet::from(["value".to_string()])); } + #[tokio::test] + async fn test_exemplar_table_is_hidden_from_metadata() { + use common_catalog::consts::METRICS_EXEMPLARS_TABLE; + use common_query::prelude::{EXEMPLAR_METRIC_COLUMN, greptime_exemplar_labels}; + use datatypes::types::json_type::JsonNativeType; + + let manager = MemoryCatalogManager::with_default_setup(); + for (id, name, tag, field, data_type) in [ + ( + 1024, + "cpu_usage", + "host", + "value", + ConcreteDataType::float64_datatype(), + ), + ( + 1025, + METRICS_EXEMPLARS_TABLE, + EXEMPLAR_METRIC_COLUMN, + greptime_exemplar_labels(), + ConcreteDataType::json2(JsonNativeType::Null), + ), + ] { + let mut info = test_table_info( + id, + name, + DEFAULT_SCHEMA_NAME, + DEFAULT_CATALOG_NAME, + Arc::new(Schema::new(vec![ + ColumnSchema::new(tag, ConcreteDataType::string_datatype(), true), + ColumnSchema::new(field, data_type, true), + ])), + ); + info.meta.primary_key_indices = vec![0]; + manager + .register_table_sync(RegisterTableRequest { + catalog: DEFAULT_CATALOG_NAME.to_string(), + schema: DEFAULT_SCHEMA_NAME.to_string(), + table_name: name.to_string(), + table_id: id, + table: EmptyTable::from_table_info(&info), + }) + .unwrap(); + } + let manager: CatalogManagerRef = manager; + let handler: PrometheusHandlerRef = Arc::new(TestPrometheusHandler { + catalog_manager: manager.clone(), + deny_operation: false, + denied_table: Some(METRICS_EXEMPLARS_TABLE), + metric_names: Vec::new(), + label_metric_names: Vec::new(), + label_lookups: Mutex::new(Vec::new()), + queries: Mutex::new(Vec::new()), + ordered_outputs: Mutex::new(Vec::new()), + }); + let ctx = QueryContext::with(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME); + let response = labels_query( + None, + State(handler.clone()), + Query(LabelsQuery::default()), + Extension(ctx.fork()), + Form(LabelsQuery::default()), + ) + .await; + assert_eq!(None, response.status_code, "{:?}", response.error); + assert_eq!( + response.data, + PrometheusResponse::Labels(vec![METRIC_NAME_LABEL.into(), "host".into()]) + ); + + let targets = ["cpu_usage", METRICS_EXEMPLARS_TABLE].map(|name| { + PermissionTableTarget::new(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME, name) + }); + assert_eq!( + get_target_column_names(&targets, &manager, &ctx) + .await + .unwrap(), + HashSet::from(["host".to_string()]) + ); + + let exemplar_selector = format!(r#"{{__name__=~"{METRICS_EXEMPLARS_TABLE}"}}"#); + let mixed_selector = format!(r#"{{__name__=~"cpu_usage|{METRICS_EXEMPLARS_TABLE}"}}"#); + for (label, selectors, expected) in [ + (FIELD_NAME_LABEL, vec![], vec!["value"]), + (FIELD_NAME_LABEL, vec!["cpu_usage"], vec!["value"]), + ( + FIELD_NAME_LABEL, + vec![mixed_selector.as_str()], + vec!["value"], + ), + (FIELD_NAME_LABEL, vec![METRICS_EXEMPLARS_TABLE], vec![]), + (FIELD_NAME_LABEL, vec![exemplar_selector.as_str()], vec![]), + ("__schema__", vec![METRICS_EXEMPLARS_TABLE], vec![]), + ("__schema__", vec![exemplar_selector.as_str()], vec![]), + ( + "__schema__", + vec![mixed_selector.as_str()], + vec![DEFAULT_SCHEMA_NAME], + ), + ] { + let response = label_values_query( + None, + State(handler.clone()), + Path(label.to_string()), + Extension(ctx.fork()), + Query(LabelValueQuery { + matches: Matches(selectors.iter().map(|value| value.to_string()).collect()), + ..Default::default() + }), + ) + .await; + assert_eq!(None, response.status_code, "{:?}", response.error); + assert_eq!( + response.data, + PrometheusResponse::LabelValues(expected.into_iter().map(String::from).collect()), + "label={label}, selectors={selectors:?}", + ); + } + } + #[tokio::test] async fn test_get_target_column_names_uses_selected_schema() { let manager = MemoryCatalogManager::with_default_setup(); @@ -4462,6 +4722,7 @@ mod tests { assert!(metadata.contains_key("allowed")); let response = metadata_query( + None, State(Arc::new(TestPrometheusHandler { catalog_manager: manager.clone(), deny_operation: true, @@ -4489,6 +4750,7 @@ mod tests { ordered_outputs: Mutex::new(Vec::new()), }); let response = metadata_query( + None, State(handler.clone()), Query(MetadataQuery::default()), Extension(query_ctx.clone()), @@ -4504,6 +4766,7 @@ mod tests { ); let response = metadata_query( + None, State(handler), Query(MetadataQuery { metric: Some("denied".to_string()), diff --git a/src/servers/src/mysql/handler.rs b/src/servers/src/mysql/handler.rs index bdfc7a72057..5d202c77af1 100644 --- a/src/servers/src/mysql/handler.rs +++ b/src/servers/src/mysql/handler.rs @@ -151,6 +151,124 @@ impl MysqlInstanceShim { } } + async fn authenticate_handshake_user( + &self, + auth_plugin: &str, + username: &str, + salt: &[u8], + auth_data: &[u8], + catalog: &str, + ) -> Option { + let Some(user_provider) = &self.user_provider else { + return Some(auth::userinfo_by_name(Some(username.to_string()))); + }; + let addr = self + .session + .conn_info() + .client_addr + .map(|addr| addr.to_string()); + let result = if username == BEARER_TOKEN_USER { + if auth_plugin != MysqlAuthMethod::CLEAR_PASSWORD_PLUGIN { + warn!("Bearer-token MySQL authentication requires mysql_clear_password"); + return None; + } + let token = auth_data.strip_suffix(&[0]).unwrap_or(auth_data); + let Ok(token) = std::str::from_utf8(token) else { + warn!("Bearer token is not valid UTF-8"); + return None; + }; + user_provider + .authenticate_bearer_token(token, catalog) + .await + } else { + let user_id = Identity::UserId(username, addr.as_deref()); + let password = match auth_plugin { + MysqlAuthMethod::NATIVE_PASSWORD_PLUGIN => { + Password::MysqlNativePassword(auth_data, salt) + } + MysqlAuthMethod::CLEAR_PASSWORD_PLUGIN => { + // The raw bytes received could be represented in C-like string, ended in '\0'. + // We must "trim" it to get the real password string. + let password = auth_data.strip_suffix(&[0]).unwrap_or(auth_data); + Password::PlainText(String::from_utf8_lossy(password).to_string().into()) + } + other => { + error!("Unsupported mysql auth plugin: {}", other); + return None; + } + }; + user_provider.authenticate(user_id, password).await + }; + match result { + Ok(user_info) => Some(user_info), + Err(error) => { + METRIC_AUTH_FAILURE + .with_label_values(&[error.status_code().as_ref()]) + .inc(); + warn!(error; "Failed to auth"); + None + } + } + } + + // on_init can write an ERR without ending the handshake. Validate here before + // publishing the authenticated session or entering opensrv's command loop. + async fn select_handshake_database( + &self, + catalog: &str, + schema: String, + user_info: &auth::UserInfoRef, + explicit_database: bool, + ) -> Option { + if let Some(provider) = &self.user_provider + && let Err(error) = provider.authorize(catalog, &schema, user_info).await + { + let denied = matches!( + error.status_code(), + common_error::status_code::StatusCode::AccessDenied + | common_error::status_code::StatusCode::PermissionDenied + ); + if !explicit_database && denied { + debug!(user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + reason = %error.status_code(), + "Default database not authorized; starting without a selected database"); + return Some(String::new()); + } + METRIC_AUTH_FAILURE + .with_label_values(&[error.status_code().as_ref()]) + .inc(); + if denied { + warn!(user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + reason = %error.status_code(), "Handshake database authorization denied"); + } else { + error!(error; user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + "Handshake database authorization failed"); + } + return None; + } + if !explicit_database && schema.is_empty() { + return Some(schema); + } + match self.query_handler.is_valid_schema(catalog, &schema).await { + Ok(true) => Some(schema), + Ok(false) if !explicit_database => { + debug!(user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + "Default database does not exist; starting without a selected database"); + Some(String::new()) + } + Ok(false) => { + warn!(user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + "Unknown MySQL handshake database"); + None + } + Err(error) => { + error!(error; user = user_info.username(), protocol = "mysql", catalog, schema = %schema, + "Failed to validate MySQL handshake database"); + None + } + } + } + /// Describe the statement async fn do_describe( &self, @@ -432,76 +550,40 @@ impl AsyncMysqlShim for MysqlInstanceShi auth_data: &[u8], database: Option<&[u8]>, ) -> bool { + let database = match database.map(std::str::from_utf8).transpose() { + Ok(database) => database, + Err(_) => { + warn!("MySQL database is not valid UTF-8"); + return false; + } + }; + let (catalog, schema) = if let Some(database) = database { + let (catalog, schema) = parse_optional_catalog_and_schema_from_db_string(database); + (catalog.unwrap_or_else(|| self.session.catalog()), schema) + } else { + (self.session.catalog(), self.session.schema()) + }; // if not specified then **greptime** will be used let username = String::from_utf8_lossy(username); - let mut user_info = None; - let addr = self - .session - .conn_info() - .client_addr - .map(|addr| addr.to_string()); - if let Some(user_provider) = &self.user_provider { - let result = if username.as_ref() == BEARER_TOKEN_USER { - if auth_plugin != MysqlAuthMethod::CLEAR_PASSWORD_PLUGIN { - warn!("Bearer-token MySQL authentication requires mysql_clear_password"); - return false; - } - let token = auth_data.strip_suffix(&[0]).unwrap_or(auth_data); - let Ok(token) = std::str::from_utf8(token) else { - warn!("Bearer token is not valid UTF-8"); - return false; - }; - let catalog = if let Some(database) = database { - let Ok(database) = std::str::from_utf8(database) else { - warn!("MySQL database is not valid UTF-8"); - return false; - }; - parse_optional_catalog_and_schema_from_db_string(database) - .0 - .unwrap_or_else(|| self.session.catalog()) - } else { - self.session.catalog() - }; - user_provider - .authenticate_bearer_token(token, &catalog) - .await - } else { - let user_id = Identity::UserId(&username, addr.as_deref()); - let password = match auth_plugin { - MysqlAuthMethod::NATIVE_PASSWORD_PLUGIN => { - Password::MysqlNativePassword(auth_data, salt) - } - MysqlAuthMethod::CLEAR_PASSWORD_PLUGIN => { - // The raw bytes received could be represented in C-like string, ended in '\0'. - // We must "trim" it to get the real password string. - let password = auth_data.strip_suffix(&[0]).unwrap_or(auth_data); - Password::PlainText(String::from_utf8_lossy(password).to_string().into()) - } - other => { - error!("Unsupported mysql auth plugin: {}", other); - return false; - } - }; - user_provider.authenticate(user_id, password).await - }; - match result { - Ok(userinfo) => { - user_info = Some(userinfo); - } - Err(e) => { - METRIC_AUTH_FAILURE - .with_label_values(&[e.status_code().as_ref()]) - .inc(); - warn!(e; "Failed to auth"); - return false; - } - }; - } - let user_info = - user_info.unwrap_or_else(|| auth::userinfo_by_name(Some(username.to_string()))); - + let Some(user_info) = self + .authenticate_handshake_user(auth_plugin, &username, salt, auth_data, &catalog) + .await + else { + return false; + }; + let Some(schema) = self + .select_handshake_database(&catalog, schema, &user_info, database.is_some()) + .await + else { + return false; + }; + self.session.set_catalog(catalog); + self.session.set_schema(schema); self.session.set_user_info(user_info); + debug!(user = self.session.user_info().username(), protocol = "mysql", + catalog = %self.session.catalog(), schema = %self.session.schema(), + client_addr = ?self.session.conn_info().client_addr, "Handshake database selected"); true } @@ -610,9 +692,12 @@ impl AsyncMysqlShim for MysqlInstanceShi return Ok(()); } Err(e) => { - writer - .error(ErrorKind::ER_SP_BADSTATEMENT, e.output_msg().as_bytes()) - .await?; + let kind = if writer::is_no_database_selected(&e) { + ErrorKind::ER_NO_DB_ERROR + } else { + ErrorKind::ER_SP_BADSTATEMENT + }; + writer.error(kind, e.output_msg().as_bytes()).await?; return Ok(()); } } @@ -684,20 +769,6 @@ impl AsyncMysqlShim for MysqlInstanceShi self.session.catalog() }; - if !self - .query_handler - .is_valid_schema(&catalog, &schema) - .await? - { - return w - .error( - ErrorKind::ER_WRONG_DB_NAME, - format!("Unknown database '{}'", database).as_bytes(), - ) - .await - .map_err(|e| e.into()); - } - let user_info = &self.session.user_info(); if let Some(schema_validator) = &self.user_provider @@ -708,6 +779,15 @@ impl AsyncMysqlShim for MysqlInstanceShi METRIC_AUTH_FAILURE .with_label_values(&[e.status_code().as_ref()]) .inc(); + if e.status_code().should_log_error() { + error!(e; user = user_info.username(), protocol = "mysql", catalog = %catalog, schema = %schema, + client_addr = ?self.session.conn_info().client_addr, reason = %e.status_code(), + "Database selection authorization failed"); + } else { + warn!(user = user_info.username(), protocol = "mysql", catalog = %catalog, schema = %schema, + previous_schema = %self.session.schema(), client_addr = ?self.session.conn_info().client_addr, + reason = %e.status_code(), "Database selection denied"); + } return w .error( ErrorKind::ER_DBACCESS_DENIED_ERROR, @@ -717,10 +797,32 @@ impl AsyncMysqlShim for MysqlInstanceShi .map_err(|e| e.into()); } + if !self + .query_handler + .is_valid_schema(&catalog, &schema) + .await? + { + debug!(user = user_info.username(), protocol = "mysql", catalog = %catalog, schema = %schema, + client_addr = ?self.session.conn_info().client_addr, reason = "schema_not_found", + "Database selection failed"); + return w + .error( + ErrorKind::ER_WRONG_DB_NAME, + format!("Unknown database '{}'", database).as_bytes(), + ) + .await + .map_err(|e| e.into()); + } + + let previous_catalog = self.session.catalog(); + let previous_schema = self.session.schema(); if catalog_from_db.is_some() { self.session.set_catalog(catalog) } self.session.set_schema(schema); + debug!(user = user_info.username(), protocol = "mysql", catalog = %self.session.catalog(), + schema = %self.session.schema(), previous_catalog, previous_schema, + client_addr = ?self.session.conn_info().client_addr, "Database selected"); w.ok().await.map_err(|e| e.into()) } @@ -996,6 +1098,7 @@ mod tests { use std::sync::Arc; use async_trait::async_trait; + use auth::UserProvider; use common_query::Output; use datafusion_expr::LogicalPlan; use query::parser::PromQuery; @@ -1040,8 +1143,14 @@ mod tests { unimplemented!() } - async fn is_valid_schema(&self, _: &str, _: &str) -> Result { - Ok(true) + async fn is_valid_schema(&self, _: &str, schema: &str) -> Result { + if schema == "backend_error" { + return crate::error::InternalSnafu { + err_msg: "Schema backend unavailable", + } + .fail(); + } + Ok(schema != "missing") } } @@ -1055,6 +1164,97 @@ mod tests { ) } + struct HandshakeUserProvider { + inner: auth::tests::MockUserProvider, + fail_authorization: bool, + } + + #[async_trait] + impl UserProvider for HandshakeUserProvider { + fn name(&self) -> &str { + "handshake_test" + } + + async fn authenticate( + &self, + identity: Identity<'_>, + password: Password<'_>, + ) -> auth::error::Result { + self.inner.authenticate(identity, password).await + } + + async fn authorize( + &self, + catalog: &str, + schema: &str, + user_info: &auth::UserInfoRef, + ) -> auth::error::Result<()> { + if self.fail_authorization { + auth::error::InternalStateSnafu { + msg: "Authorization backend unavailable", + } + .fail() + } else { + self.inner.authorize(catalog, schema, user_info).await + } + } + } + + #[tokio::test] + async fn test_handshake_database_selection() { + for (allowed, default_schema, database, fail_authorization, expected_schema) in [ + ("allowed", "public", None, false, Some("")), + ("allowed", "public", Some("denied"), false, None), + ("allowed", "public", Some("allowed"), false, Some("allowed")), + ("missing", "public", Some("missing"), false, None), + ("public", "public", None, false, Some("public")), + ("missing", "missing", None, false, Some("")), + ("backend_error", "backend_error", None, false, None), + ("public", "public", None, true, None), + ] { + let mut shim = create_shim(); + shim.session.set_schema(default_schema.to_string()); + shim.user_provider = Some(Arc::new(HandshakeUserProvider { + inner: auth::tests::MockUserProvider { + schema: allowed.to_string(), + ..Default::default() + }, + fail_authorization, + })); + let result = + >::authenticate_with_database( + &shim, + MysqlAuthMethod::CLEAR_PASSWORD_PLUGIN, + b"greptime", + &[], + b"greptime\0", + database.map(str::as_bytes), + ) + .await; + assert_eq!(expected_schema.is_some(), result, "{allowed}: {database:?}"); + assert_eq!( + expected_schema.unwrap_or(default_schema), + shim.session.schema() + ); + } + } + + #[tokio::test] + async fn test_reject_invalid_handshake_database() { + let shim = create_shim(); + assert!( + !>::authenticate_with_database( + &shim, + MysqlAuthMethod::NATIVE_PASSWORD_PLUGIN, + b"greptime", + &[], + &[], + Some(&[0xff]), + ) + .await + ); + } + #[test] fn test_prepared_statement_cache_metrics() { use common_frontend::metrics::{CACHE_HIT, CACHE_MISS}; diff --git a/src/servers/src/mysql/writer.rs b/src/servers/src/mysql/writer.rs index c485a5189fc..cde1e082d07 100644 --- a/src/servers/src/mysql/writer.rs +++ b/src/servers/src/mysql/writer.rs @@ -151,10 +151,31 @@ pub async fn write_output( Ok(()) } +pub(super) fn is_no_database_selected(error: &impl ErrorExt) -> bool { + matches!( + error.as_any().downcast_ref::(), + Some(session::SessionError::NoDatabaseSelected {}) + ) || std::iter::successors(error.source(), |error| error.source()).any(|error| { + let session_error = error.downcast_ref::().or_else(|| { + error + .downcast_ref::() + .and_then(|error| error.as_any().downcast_ref::()) + }); + matches!( + session_error, + Some(session::SessionError::NoDatabaseSelected {}) + ) + }) +} + /// Handle GreptimeDB error, convert it to MySQL error pub fn handle_err(e: impl ErrorExt, query_ctx: QueryContextRef) -> (ErrorKind, String) { let status_code = e.status_code(); - let kind = mysql_error_kind(&status_code); + let kind = if is_no_database_selected(&e) { + ErrorKind::ER_NO_DB_ERROR + } else { + mysql_error_kind(&status_code) + }; if status_code.should_log_error() { let root_error = e.root_cause().unwrap_or(&e); diff --git a/src/servers/src/postgres.rs b/src/servers/src/postgres.rs index e9032bc79bd..d3bccb88d4e 100644 --- a/src/servers/src/postgres.rs +++ b/src/servers/src/postgres.rs @@ -96,6 +96,10 @@ pub struct PostgresServerHandlerInner { otlp_handler: Option, otlp_with_metric_engine: bool, memory_limiter: ServerMemoryLimiter, + /// Completes `COPY FROM STDIN WITH (FORMAT promwrite)` payloads + /// through the shared remote-write ingestion core — the same state + /// the `/v1/prometheus/write` endpoint runs. + prom_write_state: Option, force_tls: bool, auth_driver: AuthDriver, UserInfoRef>, /// SCRAM selected; failures get the uniform metric label. @@ -122,6 +126,8 @@ pub(crate) struct MakePostgresServerHandler { otlp_with_metric_engine: bool, #[builder(default)] memory_limiter: ServerMemoryLimiter, + #[builder(default)] + prom_write_state: Option, user_provider: Option, #[builder(default = "Arc::new(GreptimeDBStartupParameters::new())")] param_provider: Arc, @@ -176,6 +182,7 @@ impl MakePostgresServerHandler { otlp_handler: self.otlp_handler.clone(), otlp_with_metric_engine: self.otlp_with_metric_engine, memory_limiter: self.memory_limiter.clone(), + prom_write_state: self.prom_write_state.clone(), force_tls: self.force_tls, auth_driver: AuthDriver::new(self.param_provider.clone()) .with_pid_secret_key_generator(Arc::new(ZeroPidSecretKeyGenerator)), diff --git a/src/servers/src/postgres/auth_handler.rs b/src/servers/src/postgres/auth_handler.rs index 169ce594f5e..1c1211490b8 100644 --- a/src/servers/src/postgres/auth_handler.rs +++ b/src/servers/src/postgres/auth_handler.rs @@ -24,6 +24,7 @@ use async_trait::async_trait; use common_catalog::parse_catalog_and_schema_from_db_string; use common_error::ext::ErrorExt; use common_error::status_code::StatusCode; +use common_telemetry::{debug, error, warn}; use common_time::Timezone; use futures::{Sink, SinkExt}; use pgwire::api::auth::StartupHandler; @@ -88,8 +89,7 @@ impl PgUserProviderSource { } /// (user, catalog, schema) of this login. Catalog/schema are parsed from - /// the `database` startup parameter, which was validated at connection - /// start. + /// the `database` startup parameter. Existence is checked after authorization. fn resolve(login: &WireLoginInfo<'_>) -> Option<(String, String, String)> { let user = login.user()?.to_string(); let (catalog, schema) = parse_catalog_and_schema_from_db_string(login.database()?); @@ -136,6 +136,13 @@ impl PgUserProviderSource { { Err(e) => { record_auth_failure(e.status_code()); + if e.status_code().should_log_error() { + error!(e; user = user_info.username(), protocol = "postgres", catalog = %catalog, + schema = %schema, client_addr = login.host(), reason = %e.status_code(), "Startup database authorization failed"); + } else { + warn!(user = user_info.username(), protocol = "postgres", catalog = %catalog, + schema = %schema, client_addr = login.host(), reason = %e.status_code(), "Startup database authorization denied"); + } Err(PgWireError::ApiError(Box::new(AuthSnafu.into_error(e)))) } Ok(()) => Ok(()), @@ -233,6 +240,9 @@ where if let Some(current_schema) = client.metadata().get(super::METADATA_SCHEMA) { session.set_schema(current_schema.clone()); } + debug!(user = session.user_info().username(), protocol = "postgres", + catalog = %session.catalog(), schema = %session.schema(), + client_addr = ?session.conn_info().client_addr, "Handshake database selected"); } enum MechanismSelection { @@ -291,18 +301,18 @@ impl StartupHandler for PostgresServerHandlerInner { auth::protocol_negotiation(client, startup).await?; auth::save_startup_parameters_to_metadata(client, startup); - // check if db is valid - match resolve_db_info(client, self.query_handler.clone()).await? { - DbResolution::Resolved(catalog, schema) => { - let metadata = client.metadata_mut(); - let _ = metadata.insert(super::METADATA_CATALOG.to_owned(), catalog); - let _ = metadata.insert(super::METADATA_SCHEMA.to_owned(), schema); - } - DbResolution::NotFound(msg) => { - send_error(client, PgErrorCode::Ec3D000.to_err_info(msg)).await?; - return Ok(()); - } - } + let Some(db) = client.metadata().get(super::METADATA_DATABASE) else { + send_error( + client, + PgErrorCode::Ec3D000.to_err_info("Database not specified".to_owned()), + ) + .await?; + return Ok(()); + }; + let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); + let metadata = client.metadata_mut(); + let _ = metadata.insert(super::METADATA_CATALOG.to_owned(), catalog); + let _ = metadata.insert(super::METADATA_SCHEMA.to_owned(), schema); // try to set TimeZone if let Some(tz) = client.metadata().get("TimeZone") { @@ -323,6 +333,9 @@ impl StartupHandler for PostgresServerHandlerInner { match self.select_mechanism(client).await { MechanismSelection::NoAuth => { + if !check_database_exists(client, &self.query_handler).await? { + return Ok(()); + } self.session.set_user_info(userinfo_by_name( client.metadata().get(super::METADATA_USER).cloned(), )); @@ -387,6 +400,9 @@ impl StartupHandler for PostgresServerHandlerInner { return send_password_authentication_failed(client).await; } + if !check_database_exists(client, &self.query_handler).await? { + return Ok(()); + } self.session.set_user_info(user_info); set_client_info(client, &self.session); self.auth_driver @@ -437,32 +453,35 @@ where .await } -enum DbResolution { - Resolved(String, String), - NotFound(String), -} - -/// A function extracted to resolve lifetime and readability issues: -async fn resolve_db_info( +async fn check_database_exists( client: &mut C, - query_handler: ServerSqlQueryHandlerRef, -) -> PgWireResult + query_handler: &ServerSqlQueryHandlerRef, +) -> PgWireResult where - C: ClientInfo + Unpin + Send, + C: ClientInfo + Sink + Unpin + Send, + C::Error: Debug, + PgWireError: From<>::Error>, { - let db_ref = client.metadata().get(super::METADATA_DATABASE); - if let Some(db) = db_ref { + let db = client.metadata().get(super::METADATA_DATABASE); + let error = if let Some(db) = db { let (catalog, schema) = parse_catalog_and_schema_from_db_string(db); - if query_handler - .is_valid_schema(&catalog, &schema) - .await - .map_err(convert_err)? - { - Ok(DbResolution::Resolved(catalog, schema)) - } else { - Ok(DbResolution::NotFound(format!("Database not found: {db}"))) + match query_handler.is_valid_schema(&catalog, &schema).await { + Ok(true) => return Ok(true), + Ok(false) => {} + Err(error) => { + let mut error_info: ErrorInfo = convert_err(error).into(); + error_info.severity = "FATAL".to_string(); + send_error(client, error_info).await?; + return Ok(false); + } } + debug!(protocol = "postgres", user = ?client.metadata().get(super::METADATA_USER), + catalog = %catalog, schema = %schema, reason = "schema_not_found", + "Startup database validation failed"); + format!("Database not found: {db}") } else { - Ok(DbResolution::NotFound("Database not specified".to_owned())) - } + "Database not specified".to_owned() + }; + send_error(client, PgErrorCode::Ec3D000.to_err_info(error)).await?; + Ok(false) } diff --git a/src/servers/src/postgres/copy_in/codec.rs b/src/servers/src/postgres/copy_in/codec.rs index 9918061c273..0cebcf42ed9 100644 --- a/src/servers/src/postgres/copy_in/codec.rs +++ b/src/servers/src/postgres/copy_in/codec.rs @@ -56,6 +56,7 @@ use pgwire::error::PgWireResult; use crate::postgres::copy_in::csv::CsvCodec; use crate::postgres::copy_in::influxdb::InfluxdbOptions; use crate::postgres::copy_in::otlp::OtlpOptions; +use crate::postgres::copy_in::prom_write::PromWriteOptions; /// A resolved COPY FROM STDIN format with its validated options. #[derive(Debug, Clone)] @@ -74,6 +75,11 @@ pub(super) enum CopyFormat { /// matching OTLP pipeline the OTLP/HTTP endpoints use. Advertises /// wire format code 1. Otlp(OtlpOptions), + /// Table-less, **binary** Prometheus Remote Write: one request body + /// (v1 or v2, compressed protobuf), buffered whole and completed + /// through the same ingestion core the `/v1/prometheus/write` + /// endpoint runs. Advertises wire format code 1. + PromWrite(PromWriteOptions), } /// A parsed, insertable row: one gRPC value per targeted column, in COPY diff --git a/src/servers/src/postgres/copy_in/mod.rs b/src/servers/src/postgres/copy_in/mod.rs index d63b50367ce..0676557ca3f 100644 --- a/src/servers/src/postgres/copy_in/mod.rs +++ b/src/servers/src/postgres/copy_in/mod.rs @@ -37,6 +37,7 @@ mod influxdb; mod options; mod otlp; mod parse; +mod prom_write; use std::fmt::Debug; use std::sync::Arc; @@ -68,6 +69,7 @@ use pgwire::api::{ClientInfo, PgWireConnectionState}; use pgwire::error::{ErrorInfo, PgWireError, PgWireResult}; use pgwire::messages::PgWireBackendMessage; use pgwire::messages::copy::{CopyData, CopyDone, CopyFail}; +use prom_write::{PromWriteOptions, PromWriteVersion}; use prost::Message; use session::context::QueryContextRef; use table::metadata::TableInfo; @@ -109,6 +111,7 @@ const REGISTERED_FORMATS: &[(&str, FormatBuilder)] = &[ ("influxdb_line", influxdb::build_influxdb_codec), ("otlp", otlp::build_otlp_codec), ("otlp_metrics", otlp::build_otlp_codec), + ("promwrite", prom_write::build_prom_write_codec), ("text", build_unimplemented_text_codec), ("txt", build_unimplemented_text_codec), ("binary", build_unimplemented_binary_codec), @@ -247,6 +250,20 @@ fn resolve_table_name( pub(crate) struct CopyInState { query_ctx: QueryContextRef, payload: Payload, + /// Reservations against the shared request-memory limiter for the + /// bytes buffered by a table-less payload, acquired per CopyData + /// chunk as it arrives and held until the buffer is flushed or the + /// copy ends — the receive-stage counterpart of HTTP's + /// body-accounting middleware, so concurrent unfinished COPY + /// requests cannot retain unaccounted memory. + memory_guards: Vec< + common_memory_manager::MemoryGuard, + >, + /// Payload bytes already covered by limiter grants, in the + /// limiter's KiB grant granularity. Chunks acquire only the + /// uncovered delta (`buffered_total - granted`), so rounding adds at + /// most one KiB for the whole stream instead of one per chunk. + granted_bytes: u64, /// The resolved codec's name, for logging. codec_name: &'static str, /// The wire format code of the resolved codec. @@ -254,6 +271,28 @@ pub(crate) struct CopyInState { rows_written: u64, } +/// The receive-stage reservation for one buffered payload, handed to +/// the flush that consumes the payload: the guards must outlive the +/// payload's bytes (they may coexist with the decode buffer or wait on +/// decode quota), and `charged_bytes` is the payload's pre-charged +/// reservation to hand to decode-side fits checks. +pub(crate) struct BufferedReservation { + /// Held only for its Drop (the reservation release); never read. + #[allow(dead_code)] + guards: Vec< + common_memory_manager::MemoryGuard, + >, + granted_bytes: u64, +} + +impl BufferedReservation { + /// The payload bytes already charged at the receive stage; decode + /// paths take this as their pre-held reservation. + pub(crate) fn charged_bytes(&self) -> u64 { + self.granted_bytes + } +} + enum Payload { /// Table-bound rows: parsed against `grpc_schema` and flushed to /// `table_name` via `copy_in_insert`. The schema is cloned into each @@ -279,6 +318,13 @@ enum Payload { /// decoded and completed as one request at `CopyDone` through the /// matching OTLP pipeline. Otlp { data: Vec, options: OtlpOptions }, + /// The whole buffered Prometheus Remote Write body with its + /// validated options, completed as one request at `CopyDone` + /// through the shared remote-write ingestion core. + PromWrite { + data: Vec, + options: PromWriteOptions, + }, } impl CopyInState { @@ -317,6 +363,8 @@ impl CopyInState { parser, rows: Vec::with_capacity(COPY_IN_FLUSH_ROWS), }, + memory_guards: Vec::new(), + granted_bytes: 0, codec_name: "csv", codec_format_code: 0, rows_written: 0, @@ -332,6 +380,8 @@ impl CopyInState { data: Vec::new(), precision: options.precision, }, + memory_guards: Vec::new(), + granted_bytes: 0, codec_name: "influxdb", codec_format_code: 0, rows_written: 0, @@ -348,23 +398,96 @@ impl CopyInState { data: Vec::new(), options: options.clone(), }, + memory_guards: Vec::new(), + granted_bytes: 0, codec_name: "otlp", codec_format_code: 1, rows_written: 0, } } + /// State for the promwrite format: the whole request body is + /// buffered and completed once at `CopyDone` through the shared + /// remote-write ingestion core. Binary: the advertised wire format + /// code is 1. + fn new_prom_write(options: &PromWriteOptions, query_ctx: QueryContextRef) -> Self { + Self { + query_ctx, + payload: Payload::PromWrite { + data: Vec::new(), + options: options.clone(), + }, + memory_guards: Vec::new(), + granted_bytes: 0, + codec_name: "promwrite", + codec_format_code: 1, + rows_written: 0, + } + } + /// The wire format code advertised to the client. pub(crate) fn format_code(&self) -> i8 { self.codec_format_code } + /// Whether the payload buffers raw CopyData bytes until `CopyDone` + /// (the table-less formats); such bytes are charged against the + /// shared request-memory limiter as they arrive. + fn buffers_payload(&self) -> bool { + matches!( + self.payload, + Payload::Influxdb { .. } | Payload::Otlp { .. } | Payload::PromWrite { .. } + ) + } + + /// The payload bytes currently buffered (charged or not). + fn buffered_bytes(&self) -> u64 { + match &self.payload { + Payload::Influxdb { data, .. } + | Payload::Otlp { data, .. } + | Payload::PromWrite { data, .. } => data.len() as u64, + Payload::Table { .. } => 0, + } + } + + /// The payload bytes already covered by grants (KiB-aligned). + fn granted_bytes(&self) -> u64 { + self.granted_bytes + } + + /// Retains a new reservation and records the granted coverage: the + /// limiter grants in KiB granularity, so coverage rounds the + /// buffered total up to the next KiB — subsequent chunks inside + /// that headroom need no acquisition. + fn charge_chunk( + &mut self, + guard: common_memory_manager::MemoryGuard< + crate::request_memory_metrics::RequestMemoryMetrics, + >, + buffered_total: u64, + ) { + self.memory_guards.push(guard); + self.granted_bytes = buffered_total.div_ceil(1024) * 1024; + } + + /// Detaches the buffered-payload reservation so a flush can hold it + /// alongside the payload until the payload's memory is actually + /// consumed; see [`Self::memory_guards`]. + fn take_reservation(&mut self) -> BufferedReservation { + BufferedReservation { + guards: std::mem::take(&mut self.memory_guards), + granted_bytes: std::mem::take(&mut self.granted_bytes), + } + } + /// Feeds a `CopyData` chunk into the parser and buffers parsed rows /// (csv) or payload bytes (influxdb). fn feed(&mut self, data: &[u8]) -> PgWireResult<()> { match &mut self.payload { Payload::Table { parser, rows, .. } => rows.extend(parser.feed(data)?), - Payload::Influxdb { data: buffered, .. } | Payload::Otlp { data: buffered, .. } => { + Payload::Influxdb { data: buffered, .. } + | Payload::Otlp { data: buffered, .. } + | Payload::PromWrite { data: buffered, .. } => { if buffered.len() + data.len() > COPY_IN_MAX_PAYLOAD_BYTES { return Err(bad_copy_data(format!( "COPY FROM STDIN payload exceeds the {} limit; split the import into multiple COPY statements", @@ -383,7 +506,7 @@ impl CopyInState { match &mut self.payload { Payload::Table { parser, rows, .. } => rows.extend(parser.finish()?), // The whole payload is one request body; nothing to finalize. - Payload::Influxdb { .. } | Payload::Otlp { .. } => {} + Payload::Influxdb { .. } | Payload::Otlp { .. } | Payload::PromWrite { .. } => {} } Ok(()) } @@ -392,7 +515,7 @@ impl CopyInState { match &self.payload { Payload::Table { rows, .. } => rows.len() >= COPY_IN_FLUSH_ROWS, // Completed once at CopyDone, not mid-stream. - Payload::Influxdb { .. } | Payload::Otlp { .. } => false, + Payload::Influxdb { .. } | Payload::Otlp { .. } | Payload::PromWrite { .. } => false, } } @@ -423,32 +546,52 @@ impl CopyInState { } /// Takes the buffered OTLP payload and its options, if any. - fn take_otlp(&mut self) -> Option<(OtlpOptions, Vec)> { + fn take_otlp(&mut self) -> Option<(OtlpOptions, Vec, BufferedReservation)> { let Payload::Otlp { options, data } = &mut self.payload else { return None; }; if data.is_empty() { return None; } - Some((options.clone(), std::mem::take(data))) + Some(( + options.clone(), + std::mem::take(data), + self.take_reservation(), + )) + } + + /// Takes the buffered remote-write body and its options, if any. + fn take_prom_write(&mut self) -> Option<(PromWriteOptions, Vec, BufferedReservation)> { + let Payload::PromWrite { options, data } = &mut self.payload else { + return None; + }; + if data.is_empty() { + return None; + } + Some(( + options.clone(), + std::mem::take(data), + self.take_reservation(), + )) } /// Takes the buffered payload as one line-protocol request body, if /// any — the same bytes an HTTP write of the same records would /// carry. - fn take_lines(&mut self) -> PgWireResult> { + fn take_lines(&mut self) -> PgWireResult> { let Payload::Influxdb { data, precision } = &mut self.payload else { return Ok(None); }; if data.is_empty() { return Ok(None); } - let lines = String::from_utf8(std::mem::take(data)) - .map_err(|e| bad_copy_data(format!("COPY FROM STDIN data is not valid UTF-8: {e}")))?; - Ok(Some(InfluxdbRequest { + let request = InfluxdbRequest { precision: *precision, - lines, - })) + lines: String::from_utf8(std::mem::take(data)).map_err(|e| { + bad_copy_data(format!("COPY FROM STDIN data is not valid UTF-8: {e}")) + })?, + }; + Ok(Some((request, self.take_reservation()))) } } @@ -611,11 +754,29 @@ impl PostgresServerHandlerInner { } CopyInState::new_otlp(options, query_ctx) } + // promwrite forbids a target table too — the body's series + // carry their own metric names — and completes through the + // shared remote-write ingestion core; the server must be + // wired. + CopyFormat::PromWrite(options) => { + if !stmt.table.is_empty() { + return Err(unsupported(format!( + "format promwrite routes each series to its own metric table; \"COPY {} FROM STDIN\" must not name one", + stmt.table.join(".") + ))); + } + if self.prom_write_state.is_none() { + return Err(unsupported( + "this server is not wired for the promwrite format", + )); + } + CopyInState::new_prom_write(options, query_ctx) + } }; let num_columns = match &state.payload { Payload::Table { grpc_schema, .. } => grpc_schema.len(), // No fixed column set: the format routes per record. - Payload::Influxdb { .. } | Payload::Otlp { .. } => 0, + Payload::Influxdb { .. } | Payload::Otlp { .. } | Payload::PromWrite { .. } => 0, }; let format_code = state.format_code(); @@ -638,15 +799,35 @@ impl PostgresServerHandlerInner { // timestamp alignment and inserts are identical to an HTTP // write of the same bytes. The context stays this connection's // postgres context. - if let Some((options, bytes)) = state.take_otlp() { + if let Some((options, bytes, reservation)) = state.take_otlp() { let Some(otlp_handler) = &self.otlp_handler else { return Err(unsupported("this server is not wired for the otlp format")); }; + // The reservation outlives the payload: it is dropped when + // the payload's bytes are consumed below. + let rows = self + .flush_otlp( + otlp_handler, + options, + bytes, + reservation.charged_bytes(), + query_ctx, + ) + .await?; + drop(reservation); + return Ok(rows); + } + if let Some((options, bytes, reservation)) = state.take_prom_write() { + let Some(prom_write_state) = &self.prom_write_state else { + return Err(unsupported( + "this server is not wired for the promwrite format", + )); + }; return self - .flush_otlp(otlp_handler, options, bytes, query_ctx) + .flush_prom_write(prom_write_state, options, bytes, reservation, query_ctx) .await; } - if let Some(request) = state.take_lines()? { + if let Some((request, reservation)) = state.take_lines()? { let Some(influxdb_handler) = &self.influxdb_handler else { return Err(unsupported( "this server is not wired for the influxdb line protocol format", @@ -656,6 +837,7 @@ impl PostgresServerHandlerInner { .exec(request, query_ctx) .await .map_err(convert_err)?; + drop(reservation); return match output.data { OutputData::AffectedRows(rows) => Ok(rows as u64), OutputData::RecordBatches(_) | OutputData::Stream(_) => Ok(0), @@ -689,6 +871,7 @@ impl PostgresServerHandlerInner { otlp_handler: &crate::query_handler::OpenTelemetryProtocolHandlerRef, options: OtlpOptions, bytes: Vec, + body_reserved_bytes: u64, query_ctx: QueryContextRef, ) -> PgWireResult { use opentelemetry_proto::tonic::collector::logs::v1::ExportLogsServiceRequest; @@ -724,7 +907,7 @@ impl PostgresServerHandlerInner { .metrics( request, Arc::new(ctx), - Some((self.memory_limiter.clone(), 0)), + Some((self.memory_limiter.clone(), body_reserved_bytes)), ) .await .map_err(convert_err)?; @@ -824,6 +1007,83 @@ impl PostgresServerHandlerInner { } } } + + /// Completes a promwrite copy through the shared remote-write + /// ingestion core: the same decode, validation, conversion and + /// write paths the `/v1/prometheus/write` endpoint runs, with the + /// same context extensions, under the connection's regular postgres + /// query context. No parsing or conversion lives at the copy layer. + async fn flush_prom_write( + &self, + prom_write_state: &crate::http::prom_store::PromStoreState, + options: PromWriteOptions, + bytes: Vec, + reservation: BufferedReservation, + query_ctx: QueryContextRef, + ) -> PgWireResult { + use crate::http::prom_store::{ + ingest_remote_write_v1, ingest_remote_write_v2, stamp_remote_write_extensions, + }; + use crate::prom_remote_write::{REMOTE_WRITE_V1_VERSION, REMOTE_WRITE_V2_VERSION}; + + let version_str = match options.version { + PromWriteVersion::V1 => REMOTE_WRITE_V1_VERSION, + PromWriteVersion::V2 => REMOTE_WRITE_V2_VERSION, + }; + // Same context extensions the HTTP endpoint stamps (physical + // table, semantic identity); the channel stays this + // connection's postgres channel. + // Capture pipeline presence BEFORE the options are moved: the + // v2 core rejects pipelines combined with exemplars, exactly + // like the HTTP route, so the same payload+options cannot be + // silently ingested without the pipeline over COPY. + let has_pipeline = options.pipeline_name.is_some(); + let mut ctx = (*query_ctx).clone(); + stamp_remote_write_extensions(&mut ctx, options.physical_table.as_deref(), version_str); + let pipeline_info = crate::http::extractor::PipelineInfo { + pipeline_name: options.pipeline_name, + pipeline_version: options.pipeline_version, + pipeline_params: Default::default(), + }; + let body = axum::body::Bytes::from(bytes); + // The db labels the write-progress counters. The body's receive + // reservation stays held for the whole ingestion (the guards in + // `reservation` drop when it does), and its charged bytes ride + // along as the decode-side pre-held reservation — the same + // contract HTTP's body accounting passes downstream. + let body_reserved_bytes = reservation.charged_bytes(); + let db = query_ctx.get_db_string(); + match options.version { + PromWriteVersion::V1 => { + let result = ingest_remote_write_v1( + prom_write_state, + &db, + pipeline_info, + options.zstd, + Arc::new(ctx), + body, + body_reserved_bytes, + ) + .await + .map_err(|error| convert_err(error.error))?; + Ok(result.written.iter().sum()) + } + PromWriteVersion::V2 => { + let result = ingest_remote_write_v2( + prom_write_state, + &db, + has_pipeline, + options.zstd, + ctx, + body, + body_reserved_bytes, + ) + .await + .map_err(|error| convert_err(error.error))?; + Ok(result.written.iter().sum()) + } + } + } } /// The error for an OTLP payload whose points/spans were all rejected, @@ -868,7 +1128,45 @@ impl CopyHandler for PostgresServerHandlerInner { // below aborts the whole copy and drops the buffered data — // otherwise a pooled connection would retain failed-import // buffers until its next COPY. On success it is put back. - state.feed(©_data.data)?; + // + // Buffered table-less payloads charge their bytes against the + // shared request-memory limiter as they arrive (the + // receive-stage counterpart of HTTP's body accounting); the + // reservation is held until the buffer is flushed or the state + // is dropped. + if state.buffers_payload() { + let chunk_bytes = u64::try_from(copy_data.data.len()) + .map_err(|_| bad_copy_data("chunk too large"))?; + // Charge only this chunk's uncovered delta against the + // shared budget: the limiter rounds each acquisition up to a + // KiB, so per-chunk full charges would accumulate rounding + // overhead (64 x 16-byte chunks would retain 64 KiB of + // permits for 1 KiB of payload). The fits check covers only + // THIS request's own holdings — cross-request contention is + // left to `acquire`, which applies the configured + // Wait/Reject policy instead of failing fast. + let buffered_after = state.buffered_bytes() + chunk_bytes; + // Granted coverage is KiB-aligned, so it can exceed the + // buffered bytes: chunks inside the rounding headroom are + // already covered. + let uncovered = buffered_after.saturating_sub(state.granted_bytes()); + if uncovered > 0 { + self.memory_limiter + .check_request_fits(uncovered, state.granted_bytes()) + .map_err(convert_err)?; + let memory_guard = self.memory_limiter.acquire(uncovered).await.map_err(|e| { + convert_err(crate::prom_remote_write::map_memory_acquire_timeout(e)) + })?; + state.feed(©_data.data)?; + state.charge_chunk(memory_guard, buffered_after); + } else { + // Fully covered by the previous acquisitions' rounding + // headroom. + state.feed(©_data.data)?; + } + } else { + state.feed(©_data.data)?; + } if state.should_flush() { let rows = self.flush_copy_in(&mut state).await?; state.rows_written += rows; diff --git a/src/servers/src/postgres/copy_in/prom_write.rs b/src/servers/src/postgres/copy_in/prom_write.rs new file mode 100644 index 00000000000..059dfb8185a --- /dev/null +++ b/src/servers/src/postgres/copy_in/prom_write.rs @@ -0,0 +1,199 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! The `promwrite` format for COPY FROM STDIN: one Prometheus Remote +//! Write request body (v1 or v2), snappy- or zstd-compressed protobuf. +//! +//! `COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, ])` +//! is a GreptimeDB-specific, **binary** format (wire format code 1). +//! The payload is byte-for-byte the body a Prometheus remote-write +//! client POSTs to `/v1/prometheus/write` — compression and protobuf +//! framing included — buffered until `CopyDone` and then completed +//! through the exact ingestion core that endpoint runs: the same +//! decode, validation, conversion, batched/prepared write paths and +//! context extensions, under the connection's regular postgres query +//! context. +//! +//! The `WITH` options carry what the HTTP endpoint takes as its +//! content-type/-encoding and query parameters; values already provided +//! by the postgres session (database, user) are not repeated: +//! +//! - `version 1|2` — the remote-write protocol version, matching the +//! HTTP content-type's proto (default `1`). +//! - `zstd` — the body uses zstd instead of the default snappy +//! compression, matching `Content-Encoding: zstd`. +//! - `physical_table 'name'` — the `?physical_table=` query parameter. +//! - `pipeline_name`/`pipeline_version` — the pipeline headers (v1 +//! only; for v2 they are accepted and ignored by the core, exactly +//! like the HTTP route — except that their presence still feeds the +//! pipelines-with-exemplars restriction). +//! +//! One COPY statement is one bounded ingestion request (see +//! [`crate::postgres::copy_in::COPY_IN_MAX_PAYLOAD_BYTES`]); the row +//! count includes samples, histograms and exemplars. With asynchronous v2 +//! batching it reports accepted counts; COPY cannot distinguish HTTP 202 +//! acceptance from completed writes or return partial counts on an error. + +use pgwire::error::PgWireResult; + +use crate::postgres::copy_in::bad_copy_data; +use crate::postgres::copy_in::codec::CopyFormat; +use crate::postgres::copy_in::options::CopyOptionSet; + +/// The format's validated options, carried to the flush that completes +/// the copy through the shared remote-write ingestion core. +#[derive(Debug, Clone)] +pub(crate) struct PromWriteOptions { + pub(crate) version: PromWriteVersion, + /// The `?physical_table=` query-parameter equivalent. + pub(crate) physical_table: Option, + /// Pipeline headers (v1 only). + pub(crate) pipeline_name: Option, + pub(crate) pipeline_version: Option, + /// `Content-Encoding: zstd` equivalent; snappy by default. + pub(crate) zstd: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub(crate) enum PromWriteVersion { + #[default] + V1, + V2, +} + +pub(crate) fn build_prom_write_codec(mut options: CopyOptionSet) -> PgWireResult { + let version = match options.take_value("version") { + Some(value) => match &value { + crate::postgres::copy_in::options::CopyOptionValue::Number(n) if n == "2" => { + PromWriteVersion::V2 + } + crate::postgres::copy_in::options::CopyOptionValue::Number(n) if n == "1" => { + PromWriteVersion::V1 + } + crate::postgres::copy_in::options::CopyOptionValue::String(s) + if s == "2" || s.eq_ignore_ascii_case("v2") => + { + PromWriteVersion::V2 + } + crate::postgres::copy_in::options::CopyOptionValue::String(s) + if s == "1" || s.eq_ignore_ascii_case("v1") => + { + PromWriteVersion::V1 + } + other => { + return Err(bad_copy_data(format!( + "invalid promwrite version {:?}, expected 1 or 2", + other.kind() + ))); + } + }, + None => PromWriteVersion::V1, + }; + let physical_table = options.take_string("physical_table")?; + let pipeline_name = options.take_string("pipeline_name")?; + let pipeline_version = options.take_string("pipeline_version")?; + let zstd = options.take_flag("zstd")?.unwrap_or(false); + options.reject_leftovers("promwrite")?; + // The pipeline options are accepted for version 2 but ignored by its + // ingestion core, exactly like the HTTP route — except that their + // PRESENCE still feeds the pipelines-with-exemplars restriction the + // core enforces, so the option must be retained here (see + // `flush_prom_write`). + Ok(CopyFormat::PromWrite(PromWriteOptions { + version, + physical_table, + pipeline_name, + pipeline_version, + zstd, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::postgres::copy_in::options::CopyOptionValue; + + fn opts(entries: &[(&str, CopyOptionValue)]) -> CopyOptionSet { + let mut set = CopyOptionSet::default(); + for (k, v) in entries { + set.insert((*k).into(), v.clone()).unwrap(); + } + set + } + + #[test] + fn test_defaults_and_versions() { + let CopyFormat::PromWrite(options) = + build_prom_write_codec(CopyOptionSet::default()).unwrap() + else { + panic!("expected promwrite"); + }; + assert_eq!(options.version, PromWriteVersion::V1); + assert!(!options.zstd); + assert!(options.physical_table.is_none()); + + for (entry, expected) in [ + (CopyOptionValue::Number("2".into()), PromWriteVersion::V2), + (CopyOptionValue::String("v2".into()), PromWriteVersion::V2), + (CopyOptionValue::Number("1".into()), PromWriteVersion::V1), + ] { + let CopyFormat::PromWrite(options) = + build_prom_write_codec(opts(&[("version", entry)])).unwrap() + else { + panic!("expected promwrite"); + }; + assert_eq!(options.version, expected); + } + + let set = opts(&[ + ("version", CopyOptionValue::Number("2".into())), + ("zstd", CopyOptionValue::Flag), + ("physical_table", CopyOptionValue::String("phys".into())), + ]); + let CopyFormat::PromWrite(options) = build_prom_write_codec(set).unwrap() else { + panic!("expected promwrite"); + }; + assert_eq!(options.version, PromWriteVersion::V2); + assert!(options.zstd); + assert_eq!(options.physical_table.as_deref(), Some("phys")); + } + + #[test] + fn test_rejects_bad_options() { + let set = opts(&[("version", CopyOptionValue::Number("3".into()))]); + assert!(build_prom_write_codec(set).is_err()); + + let set = opts(&[("unknown", CopyOptionValue::Flag)]); + assert!(build_prom_write_codec(set).is_err()); + } + + #[test] + fn test_v2_accepts_and_ignores_pipeline_options() { + // The HTTP v2 route deliberately ignores pipeline information + // except for the pipelines-with-exemplars restriction; COPY + // accepts the options and retains them so the same restriction + // can fire, while the core otherwise ignores them. + let CopyFormat::PromWrite(options) = build_prom_write_codec(opts(&[ + ("version", CopyOptionValue::Number("2".into())), + ("pipeline_name", CopyOptionValue::String("p".into())), + ("pipeline_version", CopyOptionValue::String("v1".into())), + ])) + .unwrap() else { + panic!("expected promwrite"); + }; + assert_eq!(options.version, PromWriteVersion::V2); + assert_eq!(options.pipeline_name.as_deref(), Some("p")); + assert_eq!(options.pipeline_version.as_deref(), Some("v1")); + } +} diff --git a/src/servers/src/postgres/server.rs b/src/servers/src/postgres/server.rs index d12382a0506..375891b37ce 100644 --- a/src/servers/src/postgres/server.rs +++ b/src/servers/src/postgres/server.rs @@ -27,11 +27,15 @@ use futures::StreamExt; use pgwire::tokio::process_socket; use tokio_rustls::TlsAcceptor; +use crate::batcher::logical_table::LogicalTablePendingRowsBatcher; use crate::error::Result; use crate::postgres::{MakePostgresServerHandler, MakePostgresServerHandlerBuilder}; +use crate::prom_remote_write::validation::PromValidationMode; use crate::query_handler::CopyInHandlerRef; use crate::query_handler::InfluxdbLineProtocolHandlerRef; use crate::query_handler::OpenTelemetryProtocolHandlerRef; +use crate::query_handler::PipelineHandlerRef; +use crate::query_handler::PromStoreProtocolHandlerRef; use crate::query_handler::sql::ServerSqlQueryHandlerRef; use crate::request_memory_limiter::ServerMemoryLimiter; use crate::server::{AbortableStream, BaseTcpServer, Server}; @@ -47,6 +51,7 @@ pub struct PostgresServer { otlp_handler: Option, otlp_with_metric_engine: bool, memory_limiter: ServerMemoryLimiter, + prom_write_state: Option, bind_addr: Option, process_manager: Option, } @@ -88,6 +93,7 @@ impl PostgresServer { otlp_handler: None, otlp_with_metric_engine: false, memory_limiter: ServerMemoryLimiter::default(), + prom_write_state: None, bind_addr: None, process_manager, } @@ -129,6 +135,32 @@ impl PostgresServer { self } + /// Wires the Prometheus Remote Write ingestion used by table-less + /// `COPY FROM STDIN WITH (FORMAT promwrite)` — the same handler, + /// metric-engine flag, validation mode and optional shared logical-table + /// batcher the HTTP endpoint's state carries. + pub fn with_prom_remote_write( + mut self, + handler: PromStoreProtocolHandlerRef, + pipeline_handler: Option, + with_metric_engine: bool, + validation_mode: PromValidationMode, + pending_rows_batcher: Option>, + ) -> Self { + self.prom_write_state = Some(crate::http::prom_store::PromStoreState { + prom_store_handler: handler, + pipeline_handler, + prom_store_with_metric_engine: with_metric_engine, + prom_validation_mode: validation_mode, + pending_rows_batcher, + // Placeholder; `accept` injects the server's shared limiter + // (see below) so the receive-stage charging and the + // ingestion core enforce ONE budget. + memory_limiter: ServerMemoryLimiter::default(), + }); + self + } + fn accept( &self, io_runtime: Runtime, @@ -140,6 +172,14 @@ impl PostgresServer { // going through the builder — keeps this infallible: no // `.build().unwrap()` on the hot accept path. let make_handler = &self.make_handler; + // The ingestion state enforces the same shared request-memory + // budget (`max_in_flight_write_bytes`) as the HTTP endpoints: + // receive-stage COPY buffering and decode/conversion memory are + // charged against one limiter. + let prom_write_state = self.prom_write_state.clone().map(|mut state| { + state.memory_limiter = self.memory_limiter.clone(); + state + }); let handler_maker = Arc::new(MakePostgresServerHandler { query_handler: make_handler.query_handler.clone(), copy_in_handler: make_handler.copy_in_handler.clone(), @@ -147,6 +187,7 @@ impl PostgresServer { otlp_handler: self.otlp_handler.clone(), otlp_with_metric_engine: self.otlp_with_metric_engine, memory_limiter: self.memory_limiter.clone(), + prom_write_state, user_provider: make_handler.user_provider.clone(), param_provider: make_handler.param_provider.clone(), force_tls: make_handler.force_tls, diff --git a/src/servers/src/prom_remote_write/README.md b/src/servers/src/prom_remote_write/README.md index 9e86a8753dd..48fcfbacbf3 100644 --- a/src/servers/src/prom_remote_write/README.md +++ b/src/servers/src/prom_remote_write/README.md @@ -15,32 +15,42 @@ flowchart TD C --> D["borrow symbols and time-series bytes"] D --> E["decode leaf messages and build rows"] - E --> F["samples ContextReq"] - E --> G["histograms ContextReq"] - - F --> H["write_prometheus_rows_with_progress"] - H --> I["metric engine / pending batcher when enabled"] - - G --> J["histogram ContextOpt"] - J --> K["write_prometheus_rows_with_progress"] - K --> L["same metric-engine flag as samples, no batcher"] - L --> M["table: "] - - M --> N["field: configured native-histogram Struct"] - N --> O["struct children: counts, spans, buckets, sum, schema"] - I --> P["written headers and counters"] - L --> P + E --> F{"Retained exemplars?"} + F -->|No| G["write_prometheus_v2_rows_with_progress"] + F -->|Yes| H["exemplar::write"] + G --> I["sample / native histogram writes"] + H --> I + H --> J["exemplar writes: greptime_metrics_exemplars"] + I --> K["written headers and counters"] + J --> K ``` -The conversion step splits one v2 request into two `ContextReq`s: - The `decode` codec metric covers decompression and the borrowed request envelope. The `convert` metric covers per-series scans, leaf prost decoding, validation, and row construction. +The conversion step produces sample and histogram `ContextReq`s plus validated +`ExemplarRequests`: + - samples keep the existing sample table name and can use the metric-engine physical table path and pending rows batcher; -- native histograms keep the existing metric table name. +- native histograms keep the existing metric table name; +- exemplars are grouped by database and written through `exemplar::write`, which + prepares their source metric tables and tracks progress for each kind of write. + +Accompanying source sample and histogram writes must succeed before exemplars +are written. A source write failure skips the request's exemplars while unrelated +metric writes proceed independently. Exemplar-only inputs remain supported for +existing logical metric tables. + +Exemplar persistence requires metric-engine ingestion and logical metric-engine +source tables. Exemplars are stored in the target database's +`greptime_metrics_exemplars` table using ordinary Mito storage with sparse primary +keys and append mode. Remote Write v1 uses the same exemplar write path. + +Exemplar-owned labels are stored in a JSON2 field named +`_exemplar_labels`. The default name is +`greptime_exemplar_labels`; an empty prefix produces `exemplar_labels`. Native histogram rows follow the same metric-engine switch as samples. They do not use the pending rows batcher yet because the batcher assumes the classic @@ -56,6 +66,7 @@ Each histogram row stores one Struct field named - list children for custom values, spans, and positive/negative buckets; - original Prometheus labels as Greptime tags. -The v2 response always reports written sample, histogram, and exemplar counts in -Prometheus remote-write headers. Exemplars are currently ignored and reported as -zero. +The v2 response reports sample, histogram, and exemplar counts in Prometheus +remote-write headers. Exemplar counts reflect confirmed storage writes, including +partial writes on error. Invalid exemplar observations are dropped during +conversion and excluded from the exemplar count. diff --git a/src/servers/src/prom_remote_write/mod.rs b/src/servers/src/prom_remote_write/mod.rs index 2b67304b620..7b5b08240d4 100644 --- a/src/servers/src/prom_remote_write/mod.rs +++ b/src/servers/src/prom_remote_write/mod.rs @@ -115,6 +115,7 @@ pub async fn decode_remote_write_request( prom_validation_mode: PromValidationMode, processor: &mut PromSeriesProcessor, limiter: &ServerMemoryLimiter, + body_reserved_bytes: u64, ) -> crate::error::Result<( TablesBuilder<'static>, Vec>, @@ -126,7 +127,8 @@ pub async fn decode_remote_write_request( // Holds the memory permits for the decompressed bytes; the caller must // keep them alive as long as the returned `TablesBuilder`, which retains // the decompressed buffer as its raw data. - let buf = decompress_remote_write_body(is_zstd, &body[..], limiter, 0).await?; + let buf = + decompress_remote_write_body(is_zstd, &body[..], limiter, body_reserved_bytes).await?; // Decompression copied the payload out, so the compressed body is no longer needed. drop(body); let (data, guards) = buf.into_parts(); diff --git a/src/servers/src/prom_remote_write/v2.rs b/src/servers/src/prom_remote_write/v2.rs index 6f82247b727..7ef8c7848fd 100644 --- a/src/servers/src/prom_remote_write/v2.rs +++ b/src/servers/src/prom_remote_write/v2.rs @@ -1782,8 +1782,10 @@ mod tests { .await .unwrap(); assert!(limiter.used_bytes() >= label.len() as u64); - let progress = - session::write_progress::WriteProgress::with_reservation(decoded.memory_guard.clone()); + let progress = session::write_progress::WriteProgress::with_reservation( + decoded.memory_guard.granted_bytes(), + decoded.memory_guard.clone(), + ); drop(decoded); assert!(limiter.used_bytes() >= label.len() as u64); drop(progress); @@ -1816,8 +1818,10 @@ mod tests { .unwrap(); assert_eq!(decoded.exemplar_count, 1); assert!(limiter.used_bytes() >= label.len() as u64); - let progress = - session::write_progress::WriteProgress::with_reservation(decoded.memory_guard.clone()); + let progress = session::write_progress::WriteProgress::with_reservation( + decoded.memory_guard.granted_bytes(), + decoded.memory_guard.clone(), + ); drop(decoded); assert!(limiter.used_bytes() >= label.len() as u64); drop(progress); diff --git a/src/servers/tests/http/http_handler_test.rs b/src/servers/tests/http/http_handler_test.rs index 2c8064727a6..7492ec50ab8 100644 --- a/src/servers/tests/http/http_handler_test.rs +++ b/src/servers/tests/http/http_handler_test.rs @@ -266,6 +266,7 @@ async fn test_sql_not_provided() { State(api_state.clone()), Query(query), axum::Extension(ctx.clone()), + None, Form(http_handler::SqlQuery::default()), ) .await @@ -294,6 +295,7 @@ async fn test_sql_output_rows() { State(api_state.clone()), query, axum::Extension(ctx.clone()), + None, Form(http_handler::SqlQuery::default()), ) .await; @@ -397,6 +399,7 @@ async fn test_dashboard_sql_limit() { State(api_state.clone()), query, axum::Extension(ctx.clone()), + None, Form(http_handler::SqlQuery::default()), ) .await; @@ -441,6 +444,7 @@ async fn test_sql_form() { State(api_state.clone()), Query(http_handler::SqlQuery::default()), axum::Extension(ctx.clone()), + None, form, ) .await; @@ -750,6 +754,7 @@ async fn test_analyze_stream_worker_panic_emits_one_error_terminal() { ..Default::default() }), axum::Extension(ctx), + None, Ok(Form(http_handler::SqlQuery::default())), ) .await; @@ -791,6 +796,7 @@ async fn test_analyze_stream_worker_panic_after_metrics_uses_current_sequence() ..Default::default() }), axum::Extension(ctx), + None, Ok(Form(http_handler::SqlQuery::default())), ) .await; diff --git a/src/servers/tests/mod.rs b/src/servers/tests/mod.rs index d78f57b7b83..84d1d27255a 100644 --- a/src/servers/tests/mod.rs +++ b/src/servers/tests/mod.rs @@ -233,13 +233,13 @@ impl SqlQueryHandler for DummyInstance { } async fn is_valid_schema(&self, catalog: &str, schema: &str) -> Result { - Ok(self - .query_engine + self.query_engine .engine_state() .catalog_manager() .schema_exists(catalog, schema, None) .await - .unwrap()) + .map_err(common_error::ext::BoxedError::new) + .context(servers::error::CheckDatabaseValiditySnafu) } } diff --git a/src/servers/tests/mysql/mysql_server_test.rs b/src/servers/tests/mysql/mysql_server_test.rs index fdbdd4fd8f9..c1c6eb5e739 100644 --- a/src/servers/tests/mysql/mysql_server_test.rs +++ b/src/servers/tests/mysql/mysql_server_test.rs @@ -278,6 +278,13 @@ async fn test_schema_validation() -> Result<()> { let _ = create_connection_default_db_name(server_port, false) .await .unwrap(); + let mut connection = create_connection(server_port, None, false).await.unwrap(); + for schema in ["information_schema", "missing"] { + match connection.query_drop(format!("USE {schema}")).await { + Err(mysql_async::Error::Server(error)) => assert_eq!(1044, error.code), + result => panic!("expected database access denied for {schema}, got {result:?}"), + } + } mysql_server.shutdown().await.unwrap(); // change to another username @@ -290,6 +297,15 @@ async fn test_schema_validation() -> Result<()> { let fail = create_connection_default_db_name(server_port, false).await; assert!(fail.is_err()); + let mut connection = create_connection(server_port, None, false).await.unwrap(); + match connection.query_drop("USE public").await { + Err(mysql_async::Error::Server(error)) => assert_eq!(1044, error.code), + result => panic!("expected database access denied, got {result:?}"), + } + let database: Option> = + connection.query_first("SELECT DATABASE()").await.unwrap(); + assert_eq!(Some(None), database); + connection.disconnect().await.unwrap(); mysql_server.shutdown().await.unwrap(); Ok(()) @@ -461,7 +477,7 @@ async fn test_bearer_token_auth_over_clear_password() -> Result<()> { assert_eq!(1, value); assert_eq!(1, provider.authentications.load(Ordering::Relaxed)); assert_eq!( - usize::from(database.is_some()), + 1 + usize::from(database.is_some()), provider.authorizations.load(Ordering::Relaxed) ); connection.disconnect().await.unwrap(); @@ -579,9 +595,23 @@ async fn test_db_name() -> Result<()> { mysql_server.start(listening).await.unwrap(); let server_addr = mysql_server.bind_addr().unwrap(); - // None actually uses default database name - let r = create_connection_default_db_name(server_addr.port(), client_tls).await; - let _ = r.unwrap(); + let mut connection = create_connection(server_addr.port(), None, client_tls) + .await + .unwrap(); + let database: Option = connection + .query_first("SELECT DATABASE()") + .await + .unwrap() + .flatten(); + assert_eq!(Some("public".to_string()), database); + connection.query_drop("USE public").await.unwrap(); + let database: String = connection + .query_first("SELECT DATABASE()") + .await + .unwrap() + .unwrap(); + assert_eq!("public", database); + connection.disconnect().await.unwrap(); let r = create_connection(server_addr.port(), Some("tomcat"), client_tls).await; assert!(r.is_err()); diff --git a/src/servers/tests/postgres/mod.rs b/src/servers/tests/postgres/mod.rs index 1bc5e79c236..a47da20c177 100644 --- a/src/servers/tests/postgres/mod.rs +++ b/src/servers/tests/postgres/mod.rs @@ -346,6 +346,132 @@ async fn test_schema_validating() -> Result<()> { Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_database_existence_is_checked_after_authentication_and_authorization() -> Result<()> { + let provider = Arc::new(MixedAuthProvider { + scram_verifier: PgScramSha256Verifier::from_password( + MIXED_AUTH_SCRAM_PASSWORD, + b"database-auth-salt", + 4096, + ) + .unwrap(), + selections: Mutex::new(HashMap::new()), + authorize_calls: Mutex::new(HashMap::new()), + }); + let (server, port) = start_test_server_with_user_provider(provider, Default::default()).await?; + for database in [DEFAULT_SCHEMA_NAME, "missing", "missing_catalog-public"] { + for (user, password, authenticated) in [ + (MIXED_AUTH_CLEARTEXT_USER, "wrong", false), + (MIXED_AUTH_SCRAM_USER, "wrong", false), + ("unknown", "wrong", false), + (MIXED_AUTH_DENIED_USER, "scram-password", false), + ( + MIXED_AUTH_CLEARTEXT_USER, + MIXED_AUTH_CLEARTEXT_PASSWORD, + true, + ), + (MIXED_AUTH_SCRAM_USER, "scram-password", true), + ] { + let result = tokio_postgres::Config::new() + .host("127.0.0.1") + .port(port) + .user(user) + .password(password) + .dbname(database) + .connect_timeout(Duration::from_secs(2)) + .connect(NoTls) + .await; + if authenticated && database == DEFAULT_SCHEMA_NAME { + let (client, connection) = result.unwrap(); + let task = tokio::spawn(connection); + client.simple_query("SELECT 1").await.unwrap(); + drop(client); + task.await.unwrap().unwrap(); + } else { + let error = match result { + Err(error) => error, + Ok(_) => panic!("unexpected successful login for {user} in {database}"), + }; + assert_eq!( + Some(if !authenticated { + &SqlState::INVALID_PASSWORD + } else if database == "missing_catalog-public" { + &SqlState::INTERNAL_ERROR + } else { + &SqlState::INVALID_CATALOG_NAME + }), + error.code(), + "{user} in {database}: {error}" + ); + } + } + } + server.shutdown().await.unwrap(); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_startup_database_backend_error_closes_connection() -> Result<()> { + let provider = Arc::new(MixedAuthProvider { + scram_verifier: PgScramSha256Verifier::from_password( + MIXED_AUTH_SCRAM_PASSWORD, + b"database-auth-salt", + 4096, + ) + .unwrap(), + selections: Mutex::new(HashMap::new()), + authorize_calls: Mutex::new(HashMap::new()), + }); + let (server, port) = start_test_server_with_user_provider(provider, Default::default()).await?; + let mut stream = TcpStream::connect(("127.0.0.1", port)).await.unwrap(); + // The in-memory catalog reports an error for an unknown catalog, exercising + // the backend-error path after successful authentication and authorization. + let mut startup = 196608i32.to_be_bytes().to_vec(); + startup.extend_from_slice( + format!("user\0{MIXED_AUTH_CLEARTEXT_USER}\0database\0missing_catalog-public\0\0") + .as_bytes(), + ); + stream + .write_all(&((startup.len() + 4) as i32).to_be_bytes()) + .await + .unwrap(); + stream.write_all(&startup).await.unwrap(); + let mut header = [0; 5]; + stream.read_exact(&mut header).await.unwrap(); + assert_eq!(b'R', header[0]); + let mut body = vec![0; i32::from_be_bytes(header[1..].try_into().unwrap()) as usize - 4]; + stream.read_exact(&mut body).await.unwrap(); + assert_eq!(3i32.to_be_bytes(), body.as_slice()); + let password = format!("{MIXED_AUTH_CLEARTEXT_PASSWORD}\0"); + let mut message = vec![b'p']; + message.extend_from_slice(&((password.len() + 4) as i32).to_be_bytes()); + message.extend_from_slice(password.as_bytes()); + stream.write_all(&message).await.unwrap(); + stream.read_exact(&mut header).await.unwrap(); + assert_eq!(b'E', header[0]); + let mut body = vec![0; i32::from_be_bytes(header[1..].try_into().unwrap()) as usize - 4]; + stream.read_exact(&mut body).await.unwrap(); + assert!(body.windows(7).any(|part| part == b"SFATAL\0")); + assert!(body.windows(7).any(|part| part == b"CXX000\0")); + // A client that ignores the startup error must not enter the query loop. + let query = b"SELECT 1\0"; + let mut message = vec![b'Q']; + message.extend_from_slice(&((query.len() + 4) as i32).to_be_bytes()); + message.extend_from_slice(query); + let _ = stream.write_all(&message).await; + let mut response = [0]; + let read = tokio::time::timeout(Duration::from_secs(2), stream.read(&mut response)) + .await + .unwrap(); + assert!( + matches!(read, Ok(0)) + || matches!(read, Err(ref error) if error.kind() == std::io::ErrorKind::ConnectionReset), + "{read:?}" + ); + server.shutdown().await.unwrap(); + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn test_pg_scram_sha256_auth() -> Result<()> { let verifier = diff --git a/src/session/src/context.rs b/src/session/src/context.rs index ce1b4261e29..65a62a249d3 100644 --- a/src/session/src/context.rs +++ b/src/session/src/context.rs @@ -16,7 +16,8 @@ use std::collections::HashMap; use std::fmt::{Display, Formatter}; use std::net::SocketAddr; use std::str::FromStr; -use std::sync::{Arc, RwLock}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, RwLock, Weak}; use std::time::Duration; use api::v1::ExplainOptions; @@ -126,6 +127,9 @@ pub struct QueryContext { /// Process-local storage acknowledgements; never accepted from a wire context. #[builder(setter(skip))] write_progress: Option>, + /// Reads transport-owned reservations without retaining them in queued writes. + #[builder(setter(skip))] + body_memory_bytes: Weak, /// Preserve storage acknowledgements for protocol-level partial-write accounting. #[builder(setter(skip))] confirmed_writes: bool, @@ -347,6 +351,23 @@ impl QueryContext { self.write_progress = Some(progress); } + /// Attaches the live, process-local count of transport-owned memory permits. + pub fn set_body_memory_bytes(&mut self, bytes: Weak) { + self.body_memory_bytes = bytes; + } + + /// Returns permits held through conversion, excluding independently flushing Arrow batches. + pub fn held_memory_bytes(&self) -> u64 { + self.body_memory_bytes + .upgrade() + .map_or(0, |bytes| bytes.load(Ordering::Acquire)) + .saturating_add( + self.write_progress + .as_ref() + .map_or(0, |progress| progress.reserved_bytes()), + ) + } + /// Forks this context to retain storage acknowledgements for partial write accounting. /// /// Insertion APIs returning `Output` may return `Ok` while carrying storage errors @@ -841,6 +862,7 @@ impl QueryContextBuilder { admitted_write: None, metric_discovery_scope: MetricDiscoveryScope::default(), write_progress: None, + body_memory_bytes: Weak::new(), confirmed_writes: false, logical_batching_enabled: self.logical_batching_enabled.unwrap_or_default(), process_id: self.process_id.unwrap_or_default(), diff --git a/src/session/src/lib.rs b/src/session/src/lib.rs index fe4e7bdcc01..ae6bae9906c 100644 --- a/src/session/src/lib.rs +++ b/src/session/src/lib.rs @@ -38,6 +38,26 @@ use derive_more::Debug; use crate::context::{Channel, ConnInfo, QueryContextRef, dialect_for_channel}; +/// Errors caused by the current connection state. +#[derive(snafu::Snafu)] +#[common_macro::stack_trace_debug] +pub enum SessionError { + #[snafu(display("No database selected"))] + NoDatabaseSelected {}, +} + +impl common_error::ext::ErrorExt for SessionError { + fn status_code(&self) -> common_error::status_code::StatusCode { + // Keep the shared database-error category; the MySQL writer inspects this + // typed cause through error wrappers to emit ER_NO_DB_ERROR (1046). + common_error::status_code::StatusCode::DatabaseNotFound + } + + fn as_any(&self) -> &dyn std::any::Any { + self + } +} + /// Maximum number of warnings to store per session (similar to MySQL's max_error_count) const MAX_WARNINGS: usize = 64; diff --git a/src/session/src/write_progress.rs b/src/session/src/write_progress.rs index e6f7fc511d4..df473cbe72e 100644 --- a/src/session/src/write_progress.rs +++ b/src/session/src/write_progress.rs @@ -25,7 +25,7 @@ pub struct WriteProgress { written: AtomicU64, accepted: AtomicBool, // A queued write must keep its decoder's memory reservation alive. - _reservation: Option>, + reservation: Option<(u64, Arc)>, on_write: Option>, pending: AtomicUsize, completed: Notify, @@ -70,13 +70,19 @@ impl WriteProgress { } } - pub fn with_reservation(reservation: Arc) -> Self { + /// Retains a fixed-size reservation and its granted bytes for later memory admission. + pub fn with_reservation(reserved_bytes: u64, reservation: Arc) -> Self { Self { - _reservation: Some(reservation), + reservation: Some((reserved_bytes, reservation)), ..Self::default() } } + /// Returns the quota retained by this request's decoder or exemplar rows. + pub fn reserved_bytes(&self) -> u64 { + self.reservation.as_ref().map_or(0, |(bytes, _)| *bytes) + } + pub fn add_written(&self, rows: usize) { self.written.fetch_add(rows as u64, Ordering::Relaxed); self.observe(rows as u64); @@ -125,16 +131,27 @@ mod tests { fn confirmations_do_not_double_count_and_contexts_keep_local_ownership() { let observed = Arc::new(AtomicUsize::new(0)); let counter = observed.clone(); - let progress = Arc::new(WriteProgress::default().with_observer(move |rows| { - counter.fetch_add(rows as usize, Ordering::Relaxed); - })); + let progress = Arc::new( + WriteProgress::with_reservation(2048, Arc::new(())).with_observer(move |rows| { + counter.fetch_add(rows as usize, Ordering::Relaxed); + }), + ); let mut context = crate::context::QueryContext::with("greptime", "public"); + let body_bytes = Arc::new(AtomicU64::new(0)); + context.set_body_memory_bytes(Arc::downgrade(&body_bytes)); context.set_write_progress(progress.clone()); let fork = context.with_write_admission(); assert!(Arc::ptr_eq(fork.write_progress().unwrap(), &progress)); + assert_eq!(fork.held_memory_bytes(), 2048); + body_bytes.store(1024, Ordering::Release); + assert_eq!(context.held_memory_bytes(), 3072); + assert_eq!(fork.held_memory_bytes(), 3072); let wire = api::v1::QueryContext::from(context); let received = crate::context::QueryContext::try_from(wire).unwrap(); assert!(received.write_progress().is_none()); + assert_eq!(received.held_memory_bytes(), 0); + drop(body_bytes); + assert_eq!(fork.held_memory_bytes(), 2048); progress.add_written(2); progress.confirm(3); progress.confirm(3); diff --git a/src/sql/src/statements/set_variables.rs b/src/sql/src/statements/set_variables.rs index 346f1762e62..8b70e3e7fa8 100644 --- a/src/sql/src/statements/set_variables.rs +++ b/src/sql/src/statements/set_variables.rs @@ -26,23 +26,36 @@ pub struct SetVariables { } impl SetVariables { - /// Returns the first supported `search_path` value. - pub fn search_path(&self) -> Option<&str> { + /// Returns the schema candidates for a supported `search_path` assignment. + pub fn search_path(&self) -> Option> { let [ObjectNamePart::Identifier(variable)] = self.variable.0.as_slice() else { return None; }; - if variable.quote_style.is_some() || !variable.value.eq_ignore_ascii_case("search_path") { + if !variable.value.eq_ignore_ascii_case("search_path") { return None; } - match self.value.first()? { - Expr::Value(ValueWithSpan { - value: Value::SingleQuotedString(value) | Value::DoubleQuotedString(value), - .. - }) => Some(value), - Expr::Identifier(value) => Some(&value.value), - _ => None, + if let [Expr::Identifier(value)] = self.value.as_slice() + && value.quote_style.is_none() + && value.value.eq_ignore_ascii_case("default") + { + return Some(vec!["$user".to_string(), "public".to_string()]); } + + self.value + .iter() + .map(|value| match value { + Expr::Value(ValueWithSpan { + value: Value::SingleQuotedString(value) | Value::DoubleQuotedString(value), + .. + }) => Some(value.clone()), + Expr::Identifier(value) if value.quote_style.is_none() => { + Some(value.value.to_lowercase()) + } + Expr::Identifier(value) => Some(value.value.clone()), + _ => None, + }) + .collect() } } @@ -95,11 +108,20 @@ SET delayed_insert_timeout = 300"#, #[test] fn test_search_path() { for (sql, expected) in [ - ("SET search_path TO public", Some("public")), - ("SET SEARCH_PATH TO 'private'", Some("private")), - ("SET search_path TO \"quoted\"", Some("quoted")), - ("SET \"search_path\" TO private", None), + ("SET search_path TO public", Some(vec!["public"])), + ("SET search_path TO DEFAULT", Some(vec!["$user", "public"])), + (r#"SET search_path TO "default""#, Some(vec!["default"])), + ("SET search_path TO 'default'", Some(vec!["default"])), + ("SET SEARCH_PATH TO 'private'", Some(vec!["private"])), + ("SET search_path TO \"quoted\"", Some(vec!["quoted"])), + ("SET \"search_path\" TO private", Some(vec!["private"])), ("SET timezone TO 'UTC'", None), + ( + r#"SET search_path TO "$user", MISSING, "Mixed", 'a,b'"#, + Some(vec!["$user", "missing", "Mixed", "a,b"]), + ), + ("SET search_path TO ''", Some(vec![""])), + ("SET search_path TO public, 42", None), ] { let statements = ParserContext::create_with_dialect( sql, @@ -110,7 +132,10 @@ SET delayed_insert_timeout = 300"#, let [Statement::SetVariables(set)] = statements.as_slice() else { unreachable!() }; - assert_eq!(expected, set.search_path()); + assert_eq!( + expected.map(|v| v.into_iter().map(str::to_owned).collect::>()), + set.search_path() + ); } } } diff --git a/src/standalone/src/options.rs b/src/standalone/src/options.rs index fe7d91371d0..3c7a9e6ce13 100644 --- a/src/standalone/src/options.rs +++ b/src/standalone/src/options.rs @@ -21,6 +21,7 @@ use common_event_recorder::EventRecorderOptions; use common_memory_manager::OnExhaustedPolicy; use common_options::memory::MemoryOptions; use common_telemetry::logging::{LoggingOptions, SlowQueryOptions, TracingOptions}; +use common_time::timestamp::TimeUnit; use common_wal::config::DatanodeWalConfig; use datanode::config::{DatanodeOptions, ProcedureConfig, RegionEngineConfig, StorageConfig}; use file_engine::config::EngineConfig as FileEngineConfig; @@ -47,11 +48,20 @@ pub struct StandaloneOptions { /// Upper bound: when `false`, missing tables are never auto-created even if a /// request sets the `auto_create_table` hint to `true`. Default: `true`. pub auto_create_table: bool, - /// Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight). + /// Time index unit for physical metric tables created by auto table creation + /// (Prometheus remote write, OTLP metrics, and other metric-engine ingestion). + /// New logical tables created under such a physical table inherit its unit. + /// Options: "Second", "Millisecond" (default), "Microsecond", "Nanosecond". + pub auto_create_physical_table_timestamp_unit: TimeUnit, + /// Shared quota for write bodies/messages, retained decoder memory, and estimated + /// ordinary-table batch conversion memory. A batch must fit alongside its request's + /// existing reservations; only additional memory is acquired. /// Set to 0 to disable the limit. Default: "0" (unlimited) pub max_in_flight_write_bytes: ReadableSize, /// Policy when write bytes quota is exhausted. /// Options: "wait" (default, 10s), "wait()", "fail" + /// Table-batch conversion never waits for additional memory. Under a wait policy, + /// temporary contention returns a retryable error; requests that cannot fit are rejected. pub write_bytes_exhausted_policy: OnExhaustedPolicy, pub http: HttpOptions, pub grpc: GrpcOptions, @@ -107,6 +117,7 @@ impl Default for StandaloneOptions { default_timezone: None, default_column_prefix: None, auto_create_table: true, + auto_create_physical_table_timestamp_unit: TimeUnit::Millisecond, max_in_flight_write_bytes: ReadableSize(0), write_bytes_exhausted_policy: OnExhaustedPolicy::default(), http: HttpOptions::default(), @@ -177,6 +188,8 @@ impl StandaloneOptions { FrontendOptions { default_timezone: cloned_opts.default_timezone, auto_create_table: cloned_opts.auto_create_table, + auto_create_physical_table_timestamp_unit: cloned_opts + .auto_create_physical_table_timestamp_unit, max_in_flight_write_bytes: cloned_opts.max_in_flight_write_bytes, write_bytes_exhausted_policy: cloned_opts.write_bytes_exhausted_policy, http: cloned_opts.http, diff --git a/src/store-api/src/logstore.rs b/src/store-api/src/logstore.rs index 934ae3d3d17..3716b914ef0 100644 --- a/src/store-api/src/logstore.rs +++ b/src/store-api/src/logstore.rs @@ -134,6 +134,23 @@ pub trait LogStore: Send + Sync + 'static + std::fmt::Debug { Ok(()) } + /// Queues a persistent release of a region's entries through `entry_id`. + /// + /// The caller must ensure these entries are no longer needed for recovery. + /// Success means the store accepted the release, not that it is durable. + /// A stopped store may ignore the hint successfully, retaining extra history. + /// Implementations may batch records and retry failed writes; a crash before + /// persistence may retain extra history. Later entries remain readable. + /// The default does nothing, preserving other stores' reclamation policies. + async fn release_entries( + &self, + _provider: &Provider, + _region_id: RegionId, + _entry_id: EntryId, + ) -> Result<(), Self::Error> { + Ok(()) + } + /// Checks, before an existing region is replayed from its persisted /// state, that the log store holds the region's WAL history through /// `entry_id`, the largest replay watermark the region persisted, so that @@ -159,6 +176,39 @@ pub trait LogStore: Send + Sync + 'static + std::fmt::Debug { fn start_region_history(&self, _provider: &Provider) -> Result<(), Self::Error> { Ok(()) } + + /// Streams the entries of `region_id` in `(after, upper]` from the WAL of node `node_id`, + /// which shares this log store's storage, without writing to that WAL. + /// + /// The default returns `None`: the log store cannot read the WAL of another node. + async fn read_node( + &self, + _node_id: u64, + _region_id: RegionId, + _after: EntryId, + _upper: WalReadUpperBound, + ) -> Result>, Self::Error> { + Ok(None) + } +} + +/// Upper bound of a WAL range read through [`LogStore::read_node`]. A reported final entry +/// must be present, whereas a LIST boundary need not contain entries of the region. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WalReadUpperBound { + /// The durable last entry reported by the source on downgrade. + LastEntry(EntryId), + /// A conservative bound obtained by listing the source prefix. + Listed(EntryId), +} + +impl WalReadUpperBound { + /// Returns the bound. + pub fn entry_id(self) -> EntryId { + match self { + Self::LastEntry(id) | Self::Listed(id) => id, + } + } } /// The response of an `append` operation. diff --git a/src/store-api/src/region_engine.rs b/src/store-api/src/region_engine.rs index c7bc2e03a6f..296b88998c3 100644 --- a/src/store-api/src/region_engine.rs +++ b/src/store-api/src/region_engine.rs @@ -185,6 +185,10 @@ impl SetRegionRoleStateResponse { pub struct GrantedRegion { pub region_id: RegionId, pub region_role: RegionRole, + /// An opaque non-zero id the metadata plane attaches to some leases, `None` for a lease + /// without one. It reaches the engine through [`RegionEngine::set_region_role_with_lease`]; + /// engines may keep it with the lease and must not interpret it. + pub candidate_lease_id: Option, pub extensions: HashMap>, } @@ -193,6 +197,7 @@ impl GrantedRegion { Self { region_id, region_role, + candidate_lease_id: None, extensions: HashMap::new(), } } @@ -203,6 +208,7 @@ impl From for PbGrantedRegion { PbGrantedRegion { region_id: value.region_id.as_u64(), role: PbRegionRole::from(value.region_role).into(), + candidate_lease_id: value.candidate_lease_id.unwrap_or_default(), extensions: value.extensions, } } @@ -213,6 +219,8 @@ impl From for GrantedRegion { GrantedRegion { region_id: RegionId::from_u64(value.region_id), region_role: value.role().into(), + // 0 means the lease carries no candidate lease id. + candidate_lease_id: (value.candidate_lease_id != 0).then_some(value.candidate_lease_id), extensions: value.extensions, } } @@ -1096,6 +1104,49 @@ mod tests { ); } + #[test] + fn test_legacy_granted_region_wire() { + use prost::Message; + + let legacy_bytes: &[u8] = include_bytes!( + "../../../tests/compatibility/wire/candidate_lease_id/legacy_granted_region.pb" + ); + let granted = GrantedRegion::from(PbGrantedRegion::decode(legacy_bytes).unwrap()); + assert_eq!( + granted, + GrantedRegion { + candidate_lease_id: None, + extensions: HashMap::from([("plugin".to_string(), b"payload".to_vec())]), + ..GrantedRegion::new(RegionId::new(1024, 1), RegionRole::Follower) + } + ); + + assert_eq!(PbGrantedRegion::from(granted).encode_to_vec(), legacy_bytes); + } + + #[test] + fn test_granted_region_candidate_lease_id_wire() { + use prost::Message; + + let granted = GrantedRegion { + candidate_lease_id: Some(7), + ..GrantedRegion::new(RegionId::new(1024, 1), RegionRole::Follower) + }; + // Region id (field 1), role Follower (field 2), then candidate_lease_id as a uint64 + // varint in field 3. + let bytes: &[u8] = &[ + 0x08, 0x81, 0x80, 0x80, 0x80, 0x80, 0x80, 0x01, 0x10, 0x01, 0x18, 0x07, + ]; + assert_eq!( + PbGrantedRegion::from(granted.clone()).encode_to_vec(), + bytes + ); + assert_eq!( + GrantedRegion::from(PbGrantedRegion::decode(bytes).unwrap()), + granted + ); + } + #[test] fn region_statistic_flush_bytes_compatibility() { let legacy_bytes = br#"{ @@ -1479,12 +1530,15 @@ pub trait RegionEngine: Send + Sync { /// The region must not acknowledge a write whose WAL append completes at or after the /// deadline of its latest Leader or StagingLeader lease. Leases of other roles do not extend /// that deadline. + /// + /// `candidate_lease_id` is the lease's [GrantedRegion::candidate_lease_id]. fn set_region_role_with_lease( &self, region_id: RegionId, role: RegionRole, sent_at: Instant, deadline: Instant, + candidate_lease_id: Option, ) -> Result; /// Syncs the region manifest to the given manifest version. diff --git a/src/store-api/src/storage/file.rs b/src/store-api/src/storage/file.rs index b0b76146825..6e5867e1f60 100644 --- a/src/store-api/src/storage/file.rs +++ b/src/store-api/src/storage/file.rs @@ -33,7 +33,9 @@ pub struct ParseIdError { } /// Unique id for [SST File]. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize, Default)] +#[derive( + Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize, Default, +)] pub struct FileId(Uuid); impl FileId { diff --git a/src/table/src/predicate.rs b/src/table/src/predicate.rs index 8dc85b610f4..d3c66430045 100644 --- a/src/table/src/predicate.rs +++ b/src/table/src/predicate.rs @@ -121,12 +121,27 @@ impl Predicate { /// Returns the dynamic filter physical exprs. Notice this return a live dynamic filters which /// can change during query execution. + /// + /// # Precondition: pruning only, never row filtering + /// + /// A consumer may only use these expressions to *prune*: to skip a region, a file or a + /// row group whose statistics prove that no row can match, as + /// [`Self::prune_with_stats`] does. A remote dynamic filter is a bounded pruning hint, + /// not a copy of the producer's predicate — the coordinator ships the join's build-key + /// list as `IN`/`NOT IN` and drops the top-level conjuncts it cannot ship, so the + /// predicate can be *wider* than the one the producer evaluates (see + /// `DynFilterPayload::decode_datafusion_expr` in `common-query`). Over-approximating is + /// free for pruning but loses rows for filtering, so row filtering must not consume + /// these expressions unconditionally. pub fn dyn_filters(&self) -> Arc>> { self.dyn_filters.load_full() } /// Returns the dynamic filter as physical exprs. Notice this return a "snapshot" of /// dynamic filters at the time of calling this method. + /// + /// Pruning only: see [`Self::dyn_filters`] for why these expressions must not filter + /// rows. pub fn dyn_filter_phy_exprs(&self) -> error::Result>> { self.dyn_filters .load() diff --git a/tests-integration/Cargo.toml b/tests-integration/Cargo.toml index 481b402aad7..9b4f3d68415 100644 --- a/tests-integration/Cargo.toml +++ b/tests-integration/Cargo.toml @@ -33,6 +33,7 @@ common-datasource.workspace = true common-error.workspace = true common-event-recorder.workspace = true common-grpc.workspace = true +common-memory-manager.workspace = true common-meta.workspace = true common-procedure.workspace = true common-query.workspace = true diff --git a/tests-integration/src/instance.rs b/tests-integration/src/instance.rs index dd9509b3958..c2ab3a678d1 100644 --- a/tests-integration/src/instance.rs +++ b/tests-integration/src/instance.rs @@ -754,7 +754,7 @@ mod tests { let instance = standalone.fe_instance(); query(instance, "CREATE DATABASE other").await; for schema in ["public", "other"] { - create_table(instance, &format!("CREATE TABLE {schema}.{METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(9) TIME INDEX, greptime_metric STRING, greptime_value DOUBLE, exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')")).await; + create_table(instance, &format!("CREATE TABLE {schema}.{METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(9) TIME INDEX, greptime_metric STRING, greptime_value DOUBLE, greptime_exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')")).await; } for expression in [ "rate(metric{__schema__=\"public\"}[5m]) + metric{__database__=\"secret\"}", diff --git a/tests-integration/src/otlp.rs b/tests-integration/src/otlp.rs index 47e169e54ec..8e193843627 100644 --- a/tests-integration/src/otlp.rs +++ b/tests-integration/src/otlp.rs @@ -1923,8 +1923,9 @@ WITH( } } + #[rstest::rstest] #[tokio::test(flavor = "multi_thread")] - async fn test_exemplar_otlp_memory_wait_and_lifetime() { + async fn test_exemplar_otlp_memory_wait_and_lifetime(#[values(false, true)] gate_source: bool) { use common_memory_manager::OnExhaustedPolicy; use servers::request_memory_limiter::ServerMemoryLimiter; @@ -2052,12 +2053,32 @@ WITH( timeout: Duration::from_secs(10), }, ); + let mut request = exemplar_metric_request(2); + for (index, metric) in request.resource_metrics[0].scope_metrics[0] + .metrics + .iter_mut() + .enumerate() + { + let Some(metric::Data::Gauge(gauge)) = &mut metric.data else { + unreachable!() + }; + let point = &mut gauge.data_points[0]; + point.time_unix_nano = 120_000_000_000; + point.exemplars[0].time_unix_nano = point.time_unix_nano; + if index == 1 { + point.exemplars.clear(); + } + } let table = instance .catalog_manager() .table( DEFAULT_CATALOG_NAME, "public", - METRICS_EXEMPLARS_TABLE, + if gate_source { + "exemplar_memory_0" + } else { + METRICS_EXEMPLARS_TABLE + }, None, ) .await @@ -2071,6 +2092,7 @@ WITH( .region_routes[0] .region .id; + let region_id = RegionId::new(table.table_info().table_id(), region_id.region_number()); for cancel in [false, true] { let (started_tx, started_rx) = oneshot::channel(); let (release_tx, release_rx) = oneshot::channel(); @@ -2109,6 +2131,18 @@ WITH( let released = limiter.acquire(limit).await.unwrap(); drop(released); assert_eq!(limiter.used_bytes(), 0); + if gate_source { + for sql in [ + format!( + "SELECT COUNT(*)=0 FROM {METRICS_EXEMPLARS_TABLE} WHERE CAST(greptime_timestamp AS BIGINT)=120000000000" + ), + "SELECT COUNT(*)=1 FROM exemplar_memory_1".into(), + ] { + assert_trace_v2_query(&instance, &sql, ctx.clone()) + .await + .unwrap(); + } + } } } @@ -2136,9 +2170,9 @@ WITH( let mut metrics = Vec::new(); for (i, key) in [ "greptime_metric", - "exemplar_labels", + "greptime_exemplar_labels", "greptime.metric", - "exemplar.labels", + "greptime.exemplar.labels", ] .into_iter() .enumerate() @@ -2177,9 +2211,9 @@ WITH( assert_eq!(dropped.get(), before + 9); for (table, column) in [ ("collision_0", "greptime_metric"), - ("collision_1", "exemplar_labels"), + ("collision_1", "greptime_exemplar_labels"), ("collision_2", "greptime_metric"), - ("collision_3", "exemplar_labels"), + ("collision_3", "greptime_exemplar_labels"), ("collision_default", "greptime_metric"), ] { assert_trace_v2_query( @@ -3965,6 +3999,94 @@ WITH( } } + #[tokio::test(flavor = "multi_thread")] + async fn test_otlp_initial_write_auto_create_physical_timestamp_unit() { + use common_time::timestamp::TimeUnit as ConfigTimeUnit; + + for (unit, arrow_unit, expected) in [ + (ConfigTimeUnit::Second, TimeUnit::Second, 60), + (ConfigTimeUnit::Millisecond, TimeUnit::Millisecond, 60_123), + ( + ConfigTimeUnit::Microsecond, + TimeUnit::Microsecond, + 60_123_456, + ), + ( + ConfigTimeUnit::Nanosecond, + TimeUnit::Nanosecond, + 60_123_456_789, + ), + ] { + let standalone = GreptimeDbStandaloneBuilder::new(&format!("otlp_initial_{unit:?}")) + .with_auto_create_physical_table_timestamp_unit(unit) + .build() + .await; + let instance = standalone.fe_instance(); + let mut ctx = QueryContext::with(DEFAULT_CATALOG_NAME, "public"); + ctx.set_protocol_ctx(ProtocolCtx::OtlpMetric(OtlpMetricCtx { + with_metric_engine: true, + ..Default::default() + })); + let ctx = Arc::new(ctx); + + // No pre-created tables: preflight must accept the first point so + // ingestion can create both tables with the configured time index. + let outcome = instance + .metrics( + metric_request(vec![gauge_metric( + "initial_metric", + &[(60_123_456_789, 1.0, None)], + )]), + ctx.clone(), + None, + ) + .await + .unwrap(); + assert_eq!( + (outcome.accepted_data_points, outcome.rejected_data_points), + (1, 0), + "unit={unit:?}" + ); + + for table in ["greptime_physical_table", "initial_metric"] { + let output = instance + .do_query( + &format!("SELECT greptime_timestamp FROM {table}"), + ctx.clone(), + ) + .await + .remove(0) + .unwrap(); + let OutputData::Stream(stream) = output.data else { + panic!("expected stream") + }; + let batches = RecordBatches::try_collect(stream).await.unwrap().take(); + assert_eq!( + batches.iter().map(|batch| batch.num_rows()).sum::(), + 1 + ); + let timestamp = batches[0].column(0); + assert_eq!( + timestamp.data_type(), + &DataType::Timestamp(arrow_unit, None) + ); + let actual = match arrow_unit { + TimeUnit::Second => timestamp.as_primitive::().value(0), + TimeUnit::Millisecond => timestamp + .as_primitive::() + .value(0), + TimeUnit::Microsecond => timestamp + .as_primitive::() + .value(0), + TimeUnit::Nanosecond => { + timestamp.as_primitive::().value(0) + } + }; + assert_eq!(actual, expected, "unit={unit:?}, table={table}"); + } + } + } + #[tokio::test(flavor = "multi_thread")] async fn test_otlp_preflight_shared_physical_and_fanout_conflicts() { let standalone = GreptimeDbStandaloneBuilder::new("otlp_physical_conflicts") diff --git a/tests-integration/src/prom_store.rs b/tests-integration/src/prom_store.rs index 8e78b32cdee..c248f5124f3 100644 --- a/tests-integration/src/prom_store.rs +++ b/tests-integration/src/prom_store.rs @@ -45,6 +45,315 @@ mod tests { use crate::standalone::GreptimeDbStandaloneBuilder; use crate::tests; + #[tokio::test(flavor = "multi_thread")] + async fn test_table_batch_combined_memory_limit() { + use std::time::Duration; + + use api::greptime_proto::io::prometheus::write::v2; + use frontend::service_config::BatcherOptions; + use servers::batcher::BatchingProtocol; + use servers::prom_remote_write::validation::PromValidationMode; + + let label = "x".repeat(64 * 1024); + let v1 = WriteRequest { + timeseries: vec![TimeSeries { + labels: vec![ + Label { + name: "__name__".into(), + value: "quota_probe".into(), + }, + Label { + name: "job".into(), + value: label.clone(), + }, + ], + samples: vec![Sample { + value: 1.0, + timestamp: 1, + }], + ..Default::default() + }], + ..Default::default() + }; + let v2 = v2::Request { + symbols: vec![ + "".into(), + "__name__".into(), + "quota_probe".into(), + "job".into(), + "x".repeat(32 * 1024), + ], + timeseries: vec![v2::TimeSeries { + labels_refs: vec![1, 2, 3, 4], + samples: vec![v2::Sample { + value: 1.0, + timestamp: 1, + ..Default::default() + }], + ..Default::default() + }], + }; + for limit in [320 * 1024, 768 * 1024, 0] { + let standalone = GreptimeDbStandaloneBuilder::new(&format!("combined_memory_{limit}")) + .with_write_memory_limit(limit) + .with_table_batcher(BatcherOptions { + protocols: vec![BatchingProtocol::Prom, BatchingProtocol::Influxdb], + pending_rows_flush_interval: Duration::from_millis(10), + ..Default::default() + }) + .build() + .await; + let instance = standalone.fe_instance(); + let memory = instance.server_memory_limiter(); + let build_server = |protocols| { + HttpServerBuilder::new(HttpOptions::default()) + .with_memory_limiter(memory.clone()) + .with_batching_protocols(protocols) + .with_influxdb_handler(instance.clone()) + .with_prom_handler( + instance.clone(), + None, + false, + PromValidationMode::Strict, + None, + ) + .build() + }; + let server = build_server(vec![BatchingProtocol::Prom, BatchingProtocol::Influxdb]); + let client = TestClient::new(server.build(server.make_app()).unwrap()).await; + let v2_body = prom_store::snappy_compress(&v2.encode_to_vec()).unwrap(); + // The v2 decoder itself fits; the rejection below must happen during Arrow admission. + let unbatched = build_server(vec![]); + let unbatched = TestClient::new(unbatched.build(unbatched.make_app()).unwrap()).await; + let response = unbatched + .post("/v1/prometheus/write") + .header( + "content-type", + "application/x-protobuf;proto=io.prometheus.write.v2.Request", + ) + .header("content-encoding", "snappy") + .body(v2_body.clone()) + .send() + .await; + assert!(response.status().is_success(), "{}", response.text().await); + assert_eq!(memory.used_bytes(), 0); + + for (path, content_type, encoding, body, fits_alone) in [ + // The longer label still fits in Arrow when only the compressed body is retained. + ( + "/v1/prometheus/write", + "application/x-protobuf", + "snappy", + prom_store::snappy_compress(&v1.encode_to_vec()).unwrap(), + true, + ), + ( + "/v1/influxdb/write", + "text/plain", + "identity", + format!("quota_probe,job={label} greptime_value=1 1000000").into_bytes(), + false, + ), + ( + "/v1/prometheus/write", + "application/x-protobuf;proto=io.prometheus.write.v2.Request", + "snappy", + v2_body, + false, + ), + ] { + let response = client + .post(path) + .header("content-type", content_type) + .header("content-encoding", encoding) + .body(body) + .send() + .await; + if limit == 320 * 1024 && !fits_alone { + assert_eq!(response.status(), 429, "{}", response.text().await); + } else { + assert!(response.status().is_success(), "{}", response.text().await); + } + tokio::time::timeout(Duration::from_secs(5), async { + while memory.used_bytes() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + } + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn test_exemplar_wide_schema_conversion_memory_limit() { + use std::time::Duration; + + use api::prom_store::remote::Exemplar; + use frontend::service_config::BatcherOptions; + use otel_arrow_rust::proto::opentelemetry::collector::metrics::v1::ExportMetricsServiceRequest; + use otel_arrow_rust::proto::opentelemetry::metrics::v1 as otlp; + use servers::batcher::BatchingProtocol; + use servers::prom_remote_write::validation::PromValidationMode; + + let v1 = WriteRequest { + timeseries: vec![TimeSeries { + labels: vec![ + Label { + name: "__name__".into(), + value: "probe_metric".into(), + }, + Label { + name: "job".into(), + value: "api".into(), + }, + ], + samples: vec![Sample { + value: 1.0, + timestamp: 60_000, + }], + exemplars: vec![ + Exemplar { + labels: vec![Label { + name: "trace_id".into(), + value: "abc".into() + }], + value: 0.5, + timestamp: 60_000, + }; + 1000 + ], + ..Default::default() + }], + ..Default::default() + }; + let otlp = ExportMetricsServiceRequest { + resource_metrics: vec![otlp::ResourceMetrics { + scope_metrics: vec![otlp::ScopeMetrics { + metrics: vec![otlp::Metric { + name: "probe_metric".into(), + data: Some(otlp::metric::Data::Gauge(otlp::Gauge { + data_points: vec![otlp::NumberDataPoint { + time_unix_nano: 62_000_000_000, + value: Some(otlp::number_data_point::Value::AsDouble(1.0)), + exemplars: vec![ + otlp::Exemplar { + time_unix_nano: 62_000_000_000, + value: Some(otlp::exemplar::Value::AsDouble(0.5)), + trace_id: vec![0xab; 16], + ..Default::default() + }; + 1000 + ], + ..Default::default() + }], + })), + ..Default::default() + }], + ..Default::default() + }], + ..Default::default() + }], + }; + for (limit, remote_write) in [ + (4 * 1024 * 1024, false), + (64 * 1024 * 1024, false), + // Metric-engine Remote Write bypasses ordinary-table Arrow batching. + (4 * 1024 * 1024, true), + ] { + let standalone = + GreptimeDbStandaloneBuilder::new(&format!("exemplar_wide_{limit}_{remote_write}")) + .with_write_memory_limit(limit) + .with_table_batcher(BatcherOptions { + protocols: vec![BatchingProtocol::Prom, BatchingProtocol::Otlp], + pending_rows_flush_interval: Duration::from_millis(10), + ..Default::default() + }) + .build() + .await; + let instance = standalone.fe_instance(); + let names = (0..2048).map(|i| format!("tag_{i}")).collect::>(); + for sql in [ + "CREATE TABLE probe_physical (greptime_timestamp TIMESTAMP(3) TIME INDEX, greptime_value DOUBLE) ENGINE=metric WITH ('physical_metric_table'='')".to_string(), + "CREATE TABLE probe_metric (greptime_timestamp TIMESTAMP(3) TIME INDEX, greptime_value DOUBLE, job STRING, PRIMARY KEY(job)) ENGINE=metric WITH ('on_physical_table'='probe_physical')".to_string(), + format!("CREATE TABLE greptime_metrics_exemplars (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_metric STRING, greptime_timestamp TIMESTAMP(9) TIME INDEX, greptime_value DOUBLE, greptime_exemplar_labels JSON2, job STRING, {}, PRIMARY KEY (__table_id, __tsid, greptime_metric, job, {})) ENGINE=mito WITH ('append_mode'='true', 'primary_key_encoding'='sparse')", + names.iter().map(|n| format!("{n} STRING")).collect::>().join(","), names.join(",")), + ] { + instance.do_query(&sql, QueryContext::arc()).await.remove(0).unwrap(); + } + let memory = instance.server_memory_limiter(); + assert_eq!(memory.limit_bytes(), limit); + let server = HttpServerBuilder::new(HttpOptions::default()) + .with_memory_limiter(memory.clone()) + .with_batching_protocols(vec![BatchingProtocol::Prom, BatchingProtocol::Otlp]) + .with_otlp_handler(instance.clone(), true) + .with_prom_handler( + instance.clone(), + None, + true, + PromValidationMode::Strict, + None, + ) + .build(); + let client = TestClient::new(server.build(server.make_app()).unwrap()).await; + let (path, encoding, body) = if remote_write { + ( + "/v1/prometheus/write", + "snappy", + prom_store::snappy_compress(&v1.encode_to_vec()).unwrap(), + ) + } else { + ("/v1/otlp/v1/metrics", "identity", otlp.encode_to_vec()) + }; + let response = client + .post(path) + .header("content-type", "application/x-protobuf") + .header("content-encoding", encoding) + .body(body) + .send() + .await; + let success = remote_write || limit > 4 * 1024 * 1024; + assert_eq!( + response.status().is_success(), + success, + "{path}, quota={limit}: {}", + response.text().await + ); + if !success { + assert_eq!(response.status(), 429); + } + tokio::time::timeout(Duration::from_secs(5), async { + while memory.used_bytes() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + let output = instance + .do_query( + "SELECT COUNT(*) FROM greptime_metrics_exemplars", + QueryContext::arc(), + ) + .await + .remove(0) + .unwrap(); + let OutputData::Stream(stream) = output.data else { + unreachable!() + }; + let batches = common_recordbatch::RecordBatches::try_collect(stream) + .await + .unwrap() + .take(); + assert_eq!( + batches[0] + .column(0) + .as_primitive::() + .value(0), + if success { 1000 } else { 0 } + ); + } + } + #[tokio::test(flavor = "multi_thread")] async fn test_standalone_prom_store_remote_rw_default_physical_table() { common_telemetry::init_default_ut_logging(); @@ -169,6 +478,202 @@ mod tests { .await; } + #[tokio::test(flavor = "multi_thread")] + async fn test_standalone_prom_remote_write_auto_create_microsecond_physical_table() { + common_telemetry::init_default_ut_logging(); + let standalone = GreptimeDbStandaloneBuilder::new("test_prom_rw_auto_create_us_physical") + .with_auto_create_physical_table_timestamp_unit( + common_time::timestamp::TimeUnit::Microsecond, + ) + .build() + .await; + let instance = standalone.fe_instance(); + + let db = "prometheus_auto_create_us"; + let ctx = Arc::new(QueryContext::with(DEFAULT_CATALOG_NAME, db)); + assert!( + SqlQueryHandler::do_query( + instance.as_ref(), + &format!("CREATE DATABASE IF NOT EXISTS {db}"), + ctx.clone(), + ) + .await + .first() + .unwrap() + .is_ok() + ); + + // No manual table creation: both the physical table and the logical + // table `metric1` must be auto-created by the write, with a + // microsecond time index inherited from the physical table. + let write_request = WriteRequest { + timeseries: vec![prom_store::mock_timeseries()[0].clone()], + ..Default::default() + }; + let (row_inserts, _) = to_grpc_row_insert_requests(&write_request).unwrap(); + instance + .write(row_inserts, ctx.clone(), true) + .await + .unwrap(); + + for table in ["greptime_physical_table", "metric1"] { + let mut output = instance + .do_query( + &format!("SELECT greptime_timestamp FROM {table}"), + ctx.clone(), + ) + .await; + let OutputData::Stream(stream) = output.remove(0).unwrap().data else { + unreachable!() + }; + let batches = common_recordbatch::RecordBatches::try_collect(stream) + .await + .unwrap() + .take(); + assert_eq!(batches[0].num_rows(), 2, "unexpected rows in {table}"); + let ts_column = batches[0].column(0); + assert_eq!( + ts_column.data_type(), + &DataType::Timestamp(TimeUnit::Microsecond, None), + "unexpected time index type in {table}" + ); + let timestamps = ts_column + .as_primitive::() + .values(); + assert_eq!(timestamps.to_vec(), vec![1_000_000, 2_000_000]); + } + + // Remote read narrows microsecond timestamps back to milliseconds. + let read_request = ReadRequest { + queries: vec![Query { + start_timestamp_ms: 1000, + end_timestamp_ms: 2000, + matchers: vec![LabelMatcher { + name: prom_store::METRIC_NAME_LABEL.to_string(), + value: "metric1".to_string(), + r#type: 0, + }], + ..Default::default() + }], + ..Default::default() + }; + let resp = instance.read(read_request, ctx.clone()).await.unwrap(); + let body = prom_store::snappy_decompress(&resp.body).unwrap(); + let read_response = ReadResponse::decode(&body[..]).unwrap(); + let samples = &read_response.results[0].timeseries[0].samples; + assert_eq!( + samples.clone(), + vec![ + Sample { + value: 1.0, + timestamp: 1000 + }, + Sample { + value: 2.0, + timestamp: 2000 + } + ] + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn test_standalone_prom_batcher_auto_create_microsecond_physical_table() { + use servers::batcher::logical_table::PendingRowsSchemaAlterer; + + common_telemetry::init_default_ut_logging(); + let standalone = GreptimeDbStandaloneBuilder::new("test_prom_batcher_auto_create_us") + .with_auto_create_physical_table_timestamp_unit( + common_time::timestamp::TimeUnit::Microsecond, + ) + .build() + .await; + let instance = standalone.fe_instance(); + + // Drives the batcher's schema-alterer path: a missing physical table is + // auto-created with the configured microsecond unit, and new logical + // tables aligned to it. + let db = "prometheus_batcher_auto_create_us"; + let ctx = Arc::new(QueryContext::with(DEFAULT_CATALOG_NAME, db)); + assert!( + SqlQueryHandler::do_query( + instance.as_ref(), + &format!("CREATE DATABASE IF NOT EXISTS {db}"), + ctx.clone(), + ) + .await + .first() + .unwrap() + .is_ok() + ); + // The batcher aligns request schemas to the physical table's unit + // before invoking the schema alterer, so the schema below arrives in + // the configured microsecond unit. A missing physical table must be + // auto-created with the same unit. + let request_schema = vec![ + api::v1::ColumnSchema { + column_name: "greptime_timestamp".to_string(), + datatype: ColumnDataType::TimestampMicrosecond as i32, + semantic_type: SemanticType::Timestamp as i32, + datatype_extension: None, + options: None, + }, + api::v1::ColumnSchema { + column_name: "greptime_value".to_string(), + datatype: ColumnDataType::Float64 as i32, + semantic_type: SemanticType::Field as i32, + datatype_extension: None, + options: None, + }, + api::v1::ColumnSchema { + column_name: "job".to_string(), + datatype: ColumnDataType::String as i32, + semantic_type: SemanticType::Tag as i32, + datatype_extension: None, + options: None, + }, + ]; + instance + .create_tables_if_missing_batch( + DEFAULT_CATALOG_NAME, + db, + &[("metric1", request_schema.as_slice())], + true, + ctx.clone(), + ) + .await + .unwrap(); + + for table in ["greptime_physical_table", "metric1"] { + let mut output = instance + .do_query( + &format!( + "SELECT greptime_data_type FROM information_schema.columns \ + WHERE table_schema='{db}' AND table_name='{table}' \ + AND column_name='greptime_timestamp'" + ), + ctx.clone(), + ) + .await; + let OutputData::Stream(stream) = output.remove(0).unwrap().data else { + unreachable!() + }; + let batches = common_recordbatch::RecordBatches::try_collect(stream) + .await + .unwrap() + .take(); + assert_eq!( + batches[0].num_rows(), + 1, + "missing time index column in {table}" + ); + let data_type = batches[0].column(0).as_string::().value(0); + assert_eq!( + data_type, "TimestampMicrosecond", + "unexpected time index type in {table}" + ); + } + } + /// Regression test for : /// prometheus remote write/read against a physical metric table /// pre-created with a non-millisecond time index. Millisecond samples are diff --git a/tests-integration/src/standalone.rs b/tests-integration/src/standalone.rs index 7bda0debb26..c18dc910b64 100644 --- a/tests-integration/src/standalone.rs +++ b/tests-integration/src/standalone.rs @@ -22,6 +22,7 @@ use cache::{ use catalog::kvbackend::KvBackendCatalogManagerBuilder; use catalog::process_manager::ProcessManager; use common_base::Plugins; +use common_base::readable_size::ReadableSize; use common_catalog::consts::{MIN_USER_FLOW_ID, MIN_USER_TABLE_ID}; use common_config::KvBackendConfig; use common_datasource::object_store::LocalFileAccess; @@ -90,8 +91,10 @@ pub struct GreptimeDbStandaloneBuilder { slow_query_options: SlowQueryOptions, event_recorder_options: EventRecorderOptions, auto_create_table: bool, + auto_create_physical_table_timestamp_unit: Option, logical_batcher: Option, table_batcher: BatcherOptions, + max_in_flight_write_bytes: ReadableSize, mito_config: Option, query_options: query::options::QueryOptions, pipeline_options: pipeline::PipelineOptions, @@ -113,8 +116,10 @@ impl GreptimeDbStandaloneBuilder { }, event_recorder_options: EventRecorderOptions::default(), auto_create_table: true, + auto_create_physical_table_timestamp_unit: None, logical_batcher: None, table_batcher: BatcherOptions::default(), + max_in_flight_write_bytes: ReadableSize(0), mito_config: None, query_options: query::options::QueryOptions::default(), pipeline_options: pipeline::PipelineOptions::default(), @@ -149,6 +154,13 @@ impl GreptimeDbStandaloneBuilder { self } + /// Configures the shared protocol and conversion memory quota. + #[must_use] + pub fn with_write_memory_limit(mut self, bytes: u64) -> Self { + self.max_in_flight_write_bytes = ReadableSize(bytes); + self + } + /// Configures logical-table batching for integration tests. #[must_use] pub fn with_logical_batcher(mut self, options: BatcherOptions) -> Self { @@ -162,6 +174,16 @@ impl GreptimeDbStandaloneBuilder { self } + /// Sets the time index unit for auto-created physical metric tables. + #[must_use] + pub fn with_auto_create_physical_table_timestamp_unit( + mut self, + unit: common_time::timestamp::TimeUnit, + ) -> Self { + self.auto_create_physical_table_timestamp_unit = Some(unit); + self + } + /// Sets the event recorder options for the standalone instance. #[must_use] pub fn with_event_recorder_options( @@ -420,7 +442,11 @@ impl GreptimeDbStandaloneBuilder { event_recorder: self.event_recorder_options.clone(), auto_create_table: self.auto_create_table, query: self.query_options.clone(), + auto_create_physical_table_timestamp_unit: self + .auto_create_physical_table_timestamp_unit + .unwrap_or(common_time::timestamp::TimeUnit::Millisecond), pipeline: self.pipeline_options.clone(), + max_in_flight_write_bytes: self.max_in_flight_write_bytes, pending_rows_batcher: PendingRowsBatcherOptions { logical_table: self.logical_batcher.clone(), table: self.table_batcher.clone(), diff --git a/tests-integration/src/test_util.rs b/tests-integration/src/test_util.rs index caccd4d4eca..9742fb5bb61 100644 --- a/tests-integration/src/test_util.rs +++ b/tests-integration/src/test_util.rs @@ -913,6 +913,7 @@ pub fn build_test_prom_server( frontend_ref.table_flownode_set_cache().clone(), true, frontend_ref.clone(), + common_time::timestamp::TimeUnit::Millisecond, Duration::from_millis(50), 1000, 4, @@ -1242,7 +1243,14 @@ pub async fn setup_pg_server_with_slow_query_threshold( None, ) .with_influxdb_handler(fe_instance_ref.clone()) - .with_otlp_handler(fe_instance_ref.clone(), true), + .with_otlp_handler(fe_instance_ref.clone(), true) + .with_prom_remote_write( + fe_instance_ref.clone(), + Some(fe_instance_ref.clone()), + true, + Default::default(), + None, + ), ); pg_server @@ -1299,7 +1307,14 @@ pub async fn setup_pg_server_for_frontend_instance( None, ) .with_influxdb_handler(fe_instance_ref.clone()) - .with_otlp_handler(fe_instance_ref.clone(), true), + .with_otlp_handler(fe_instance_ref.clone(), true) + .with_prom_remote_write( + fe_instance_ref.clone(), + Some(fe_instance_ref.clone()), + true, + Default::default(), + None, + ), ); pg_server @@ -1359,7 +1374,14 @@ pub async fn setup_pg_server_with_prom_native_histogram( None, ) .with_influxdb_handler(instance.fe_instance().clone()) - .with_otlp_handler(instance.fe_instance().clone(), true), + .with_otlp_handler(instance.fe_instance().clone(), true) + .with_prom_remote_write( + instance.fe_instance().clone(), + Some(instance.fe_instance().clone()), + true, + Default::default(), + None, + ), ); pg_server diff --git a/tests-integration/src/tests/instance_test.rs b/tests-integration/src/tests/instance_test.rs index 2d8034b1510..8c54a5c485a 100644 --- a/tests-integration/src/tests/instance_test.rs +++ b/tests-integration/src/tests/instance_test.rs @@ -3935,6 +3935,15 @@ async fn execute_sql(instance: &Arc, sql: &str) -> Output { execute_sql_with(instance, sql, QueryContext::arc()).await } +async fn get_table(instance: &Arc, name: &str) -> table::TableRef { + instance + .catalog_manager() + .table(DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME, name, None) + .await + .unwrap() + .unwrap() +} + async fn execute_sql_with_query_parallelism( instance: &Arc, sql: &str, @@ -4558,17 +4567,7 @@ async fn test_exemplar_preserves_physical_table_routing(#[case] batching: bool) ); } for suffix in ["a", "b"] { - let table = frontend - .catalog_manager() - .table( - DEFAULT_CATALOG_NAME, - "public", - &format!("routed_{version}_{suffix}"), - None, - ) - .await - .unwrap() - .unwrap(); + let table = get_table(frontend, &format!("routed_{version}_{suffix}")).await; assert_eq!( table.table_info().meta.options.extra_options["on_physical_table"], format!("physical_{suffix}") @@ -4582,17 +4581,7 @@ async fn test_exemplar_preserves_physical_table_routing(#[case] batching: bool) "+----------+\n| count(*) |\n+----------+\n| 1 |\n+----------+", ) .await; - let table = frontend - .catalog_manager() - .table( - DEFAULT_CATALOG_NAME, - "public", - METRICS_EXEMPLARS_TABLE, - None, - ) - .await - .unwrap() - .unwrap(); + let table = get_table(frontend, METRICS_EXEMPLARS_TABLE).await; let info = table.table_info(); let ids = info.name_to_ids().unwrap(); assert_eq!( @@ -4782,7 +4771,7 @@ async fn test_exemplar_precreated_table_options( execute_sql(frontend, "CREATE TABLE source_physical (ts TIMESTAMP TIME INDEX, val DOUBLE) ENGINE=metric WITH ('physical_metric_table'='')").await; execute_sql(frontend, "CREATE TABLE metric_source (job STRING PRIMARY KEY, ts TIMESTAMP TIME INDEX, val DOUBLE) ENGINE=metric WITH ('on_physical_table'='source_physical')").await; execute_sql(frontend, &format!( - "CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_metric STRING, greptime_timestamp TIMESTAMP({precision}) TIME INDEX, greptime_value DOUBLE, exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric)) ENGINE=mito WITH ('append_mode'='{append_mode}', 'primary_key_encoding'='{encoding}', 'compaction.type'='twcs')" + "CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_metric STRING, greptime_timestamp TIMESTAMP({precision}) TIME INDEX, greptime_value DOUBLE, greptime_exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric)) ENGINE=mito WITH ('append_mode'='{append_mode}', 'primary_key_encoding'='{encoding}', 'compaction.type'='twcs')" )).await; let (request, _) = into_rows(vec![Exemplar { metric: "metric_source".into(), @@ -4807,17 +4796,7 @@ async fn test_exemplar_precreated_table_options( "{error:?}" ); } - let table = frontend - .catalog_manager() - .table( - DEFAULT_CATALOG_NAME, - "public", - METRICS_EXEMPLARS_TABLE, - None, - ) - .await - .unwrap() - .unwrap(); + let table = get_table(frontend, METRICS_EXEMPLARS_TABLE).await; // Invalid destinations must fail before auto-ALTER adds the inherited label. assert_eq!( table.schema().column_schema_by_name("job").is_some(), @@ -4899,24 +4878,36 @@ async fn test_exemplar_source_identity_policy() { execute_sql(frontend, "CREATE TABLE source_physical (ts TIMESTAMP TIME INDEX, val DOUBLE) ENGINE=metric WITH ('physical_metric_table'='')").await; let create = "CREATE TABLE metric_source (job STRING PRIMARY KEY, ts TIMESTAMP TIME INDEX, val DOUBLE) ENGINE=metric WITH ('on_physical_table'='source_physical')"; execute_sql(frontend, create).await; - let source = frontend - .catalog_manager() - .table(DEFAULT_CATALOG_NAME, "public", "metric_source", None) - .await - .unwrap() - .unwrap() - .table_info(); + let source = get_table(frontend, "metric_source").await.table_info(); // A metric write's captured table ID must not be replaced by a later name lookup. execute_sql(frontend, "DROP TABLE metric_source").await; execute_sql(frontend, create).await; - let replacement = frontend - .catalog_manager() - .table(DEFAULT_CATALOG_NAME, "public", "metric_source", None) - .await - .unwrap() - .unwrap() - .table_info(); + let replacement = get_table(frontend, "metric_source").await.table_info(); assert_ne!(source.table_id(), replacement.table_id()); + let mut missing_tag_request = request("metric_source"); + let rows = missing_tag_request.inserts[0].rows.as_mut().unwrap(); + rows.schema.push(api::v1::helper::tag_column_schema( + "new_tag", + api::v1::ColumnDataType::String, + )); + rows.rows[0] + .values + .push(api::v1::value::ValueData::StringValue("present".into()).into()); + assert!( + frontend + .write_exemplars_with_sources( + missing_tag_request, + QueryContext::arc(), + std::slice::from_ref(&source), + ) + .await + .is_err() + ); + let after = get_table(frontend, "metric_source").await.table_info(); + assert_eq!( + replacement, after, + "reconciliation must not alter a replacement table" + ); frontend .write_exemplars_with_sources( request("metric_source"), @@ -4929,17 +4920,7 @@ async fn test_exemplar_source_identity_policy() { .write_exemplars_prepared(request("metric_source"), QueryContext::arc()) .await .unwrap(); - let table = frontend - .catalog_manager() - .table( - DEFAULT_CATALOG_NAME, - "public", - METRICS_EXEMPLARS_TABLE, - None, - ) - .await - .unwrap() - .unwrap(); + let table = get_table(frontend, METRICS_EXEMPLARS_TABLE).await; let info = table.table_info(); assert_eq!( info.meta @@ -4955,7 +4936,7 @@ async fn test_exemplar_source_identity_policy() { "job", "greptime_value", "greptime_timestamp", - "exemplar_labels", + "greptime_exemplar_labels", ] ); assert_eq!(info.meta.primary_key_indices, [0, 1, 2, 3]); @@ -5363,6 +5344,198 @@ async fn create_exemplar_source(frontend: &Arc, schema: &str, name: &s )).await; } +#[tokio::test(flavor = "multi_thread")] +async fn test_exemplar_source_tag_reconciliation() { + use std::collections::BTreeMap; + + use servers::exemplar::query::Query; + use servers::exemplar::{Exemplar, into_rows}; + use servers::prometheus_handler::PrometheusHandler; + use servers::query_handler::PromStoreProtocolHandler; + use table::requests::AUTO_CREATE_TABLE_KEY; + + use crate::standalone::GreptimeDbStandaloneBuilder; + + let instance = GreptimeDbStandaloneBuilder::new("exemplar_source_tags") + .build() + .await; + let frontend = instance.fe_instance(); + for name in ["first", "second"] { + create_exemplar_source(frontend, "public", name).await; + } + let exemplar = |metric: &str, tag: &str, value: &str| Exemplar { + metric: metric.into(), + series_labels: BTreeMap::from([(tag.into(), value.into())]).into(), + timestamp: 1, + value: 1.0, + labels: BTreeMap::from([("trace_id".into(), "trace".into())]), + }; + let exemplars = vec![ + exemplar("first", "first_only", ""), + exemplar("second", "second_only", "present"), + exemplar("first", "later", "present"), + ]; + let requests = into_rows(exemplars.clone()).unwrap().0; + let rows = requests.inserts[0].rows.as_ref().unwrap(); + let tsid_index = rows + .schema + .iter() + .position(|c| c.column_name == "__tsid") + .unwrap(); + let tsids = rows + .rows + .iter() + .map(|row| { + let Some(api::v1::value::ValueData::U64Value(tsid)) = + &row.values[tsid_index].value_data + else { + panic!("expected a source TSID") + }; + tsid.to_string() + }) + .collect::>(); + frontend + .write_exemplars_prepared(requests, QueryContext::arc()) + .await + .unwrap(); + let mut source_ids = BTreeMap::new(); + for (name, expected) in [ + ("first", vec!["first_only", "later"]), + ("second", vec!["second_only"]), + ] { + let info = get_table(frontend, name).await.table_info(); + source_ids.insert(name, info.table_id()); + let tags = info + .meta + .primary_key_indices + .iter() + .map(|&index| info.meta.schema.column_schemas()[index].name.as_str()) + .collect::>(); + assert_eq!(tags, expected); + assert!(info.meta.schema.column_schema_by_name("trace_id").is_none()); + let output = execute_sql(frontend, &format!("SELECT COUNT(*) FROM {name}")).await; + check_output_stream( + output.data, + "+----------+\n| count(*) |\n+----------+\n| 0 |\n+----------+", + ) + .await; + } + let query = || Query::parse("first + second".into(), "0", "1", "public").unwrap(); + let before = frontend + .query_exemplars(query(), QueryContext::arc()) + .await + .unwrap(); + assert_eq!(before.len(), 3); + for (exemplar, tsid) in exemplars.iter().zip(tsids) { + let mut expected = (*exemplar.series_labels).clone(); + expected.insert("__name__".into(), exemplar.metric.to_string()); + expected.insert( + "__table_id".into(), + source_ids[exemplar.metric.as_ref()].to_string(), + ); + expected.insert("__tsid".into(), tsid); + assert!(before.iter().any(|series| series.series_labels == expected)); + } + + // Disabled automatic DDL and malformed values must not add source tags. + let source = get_table(frontend, "first").await.table_info(); + let mut ctx = QueryContext::arc().fork(); + ctx.set_extension(AUTO_CREATE_TABLE_KEY, "false"); + let blocked = into_rows(vec![exemplar("first", "blocked", "present")]) + .unwrap() + .0; + assert!( + frontend + .write_exemplars_prepared(blocked.clone(), Arc::new(ctx)) + .await + .is_err() + ); + let mut malformed = blocked; + let rows = malformed.inserts[0].rows.as_mut().unwrap(); + let index = rows + .schema + .iter() + .position(|c| c.column_name == "blocked") + .unwrap(); + rows.rows[0].values[index] = api::v1::value::ValueData::F64Value(1.0).into(); + assert!( + frontend + .write_exemplars_prepared(malformed, QueryContext::arc()) + .await + .is_err() + ); + let after = get_table(frontend, "first").await.table_info(); + assert_eq!(source, after); + + execute_sql(frontend, "CREATE TABLE conflict_physical (ts TIMESTAMP TIME INDEX, occupied_field DOUBLE) ENGINE=metric WITH ('physical_metric_table'='')").await; + execute_sql(frontend, "CREATE TABLE conflicting_source (ts TIMESTAMP TIME INDEX, occupied_field DOUBLE) ENGINE=metric WITH ('on_physical_table'='conflict_physical')").await; + let request = into_rows(vec![exemplar( + "conflicting_source", + "occupied_field", + "present", + )]) + .unwrap() + .0; + let error = frontend + .write_exemplars_prepared(request, QueryContext::arc()) + .await + .unwrap_err(); + assert!( + error.output_msg().contains("must be a string tag"), + "{error}" + ); + // Missing sources retain all tags; recreated sources define the exact query's tag projection. + execute_sql(frontend, "DROP TABLE first").await; + let missing = frontend + .query_exemplars(query(), QueryContext::arc()) + .await + .unwrap(); + assert_eq!(before, missing); + create_exemplar_source(frontend, "public", "first").await; + let after = frontend + .query_exemplars(query(), QueryContext::arc()) + .await + .unwrap(); + let mut pruned = before.clone(); + for series in &mut pruned { + series.series_labels.remove("first_only"); + series.series_labels.remove("later"); + } + assert_eq!(pruned, after); + assert_eq!( + after.len(), + 3, + "stored IDs must keep pruned series distinct" + ); + let filtered = frontend + .query_exemplars( + Query::parse("first{later=\"present\"}".into(), "0", "1", "public").unwrap(), + QueryContext::arc(), + ) + .await + .unwrap(); + assert_eq!(filtered.len(), 1); + assert!(!filtered[0].series_labels.contains_key("later")); + assert_eq!( + filtered[0].series_labels["__table_id"], + source_ids["first"].to_string() + ); + for expression in [ + "{__name__=~\"first|second\"}", + "first + {__name__=~\"second\"}", + "first + second + missing", + ] { + let broad = frontend + .query_exemplars( + Query::parse(expression.into(), "0", "1", "public").unwrap(), + QueryContext::arc(), + ) + .await + .unwrap(); + assert_eq!(before, broad, "{expression}"); + } +} + #[rstest] #[case::standalone(false, false, false, false)] #[case::standalone_batched(false, true, false, false)] @@ -5446,7 +5619,7 @@ async fn exemplar_protocol_storage_lifecycle(distributed: bool, batching: bool, greptime_timestamp TIMESTAMP(6) TIME INDEX, greptime_metric STRING, greptime_value DOUBLE, - exemplar_labels JSON2(max_auto_expanded_paths = 0, trace_id STRING), + greptime_exemplar_labels JSON2(max_auto_expanded_paths = 0, trace_id STRING), PRIMARY KEY (__table_id, __tsid, greptime_metric) ) WITH ('append_mode'='true', 'primary_key_encoding'='sparse', 'sst_format'='primary_key')" ), @@ -5566,26 +5739,25 @@ async fn exemplar_protocol_storage_lifecycle(distributed: bool, batching: bool, .await .unwrap(); let status = response.status(); - assert_eq!( - response - .headers() - .get("x-prometheus-remote-write-exemplars-written") - .unwrap(), - "1" - ); - assert_eq!( - response - .headers() - .get("x-prometheus-remote-write-samples-written") - .unwrap(), - "0" - ); + let headers = response.headers().clone(); assert_eq!( status, 204, "{:?}", to_bytes(response.into_body(), usize::MAX).await.unwrap() ); + assert_eq!( + headers + .get("x-prometheus-remote-write-exemplars-written") + .unwrap(), + "1" + ); + assert_eq!( + headers + .get("x-prometheus-remote-write-samples-written") + .unwrap(), + "0" + ); let output = execute_sql( &context.frontend(), "SELECT COUNT(*) FROM retained_exemplar", @@ -5597,6 +5769,62 @@ async fn exemplar_protocol_storage_lifecycle(distributed: bool, batching: bool, ) .await; + let source = get_table(&context.frontend(), "retained_exemplar") + .await + .table_info(); + assert!(source.meta.schema.column_schema_by_name("zone").is_some()); + create_exemplar_source(&context.frontend(), "public", "v1_exemplar_only").await; + let request = WriteRequest { + timeseries: vec![TimeSeries { + labels: vec![ + Label { + name: "__name__".into(), + value: "v1_exemplar_only".into(), + }, + Label { + name: "v1_tag".into(), + value: "present".into(), + }, + ], + exemplars: vec![Exemplar { + timestamp: 1720000000124, + value: 2.0, + ..Default::default() + }], + ..Default::default() + }], + ..Default::default() + }; + let response = app + .clone() + .oneshot( + Request::post("/v1/prometheus/write") + .header("Content-Encoding", "snappy") + .header("Content-Type", "application/x-protobuf") + .body(Body::from( + snappy_compress(&request.encode_to_vec()).unwrap(), + )) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!( + response.status(), + 204, + "{:?}", + to_bytes(response.into_body(), usize::MAX).await.unwrap() + ); + let source = get_table(&context.frontend(), "v1_exemplar_only") + .await + .table_info(); + assert!(source.meta.schema.column_schema_by_name("v1_tag").is_some()); + let output = execute_sql(&context.frontend(), "SELECT COUNT(*) FROM v1_exemplar_only").await; + check_output_stream( + output.data, + "+----------+\n| count(*) |\n+----------+\n| 0 |\n+----------+", + ) + .await; + use otel_arrow_rust::proto::opentelemetry::collector::metrics::v1::ExportMetricsServiceRequest; use otel_arrow_rust::proto::opentelemetry::metrics::v1 as otlp; let request = ExportMetricsServiceRequest { @@ -5685,8 +5913,12 @@ async fn exemplar_protocol_storage_lifecycle(distributed: bool, batching: bool, let value: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); assert_eq!(value["status"], "success", "{phase}: {value}"); for series in value["data"].as_array().unwrap() { - assert!(series["seriesLabels"].get("__table_id").is_none()); - assert!(series["seriesLabels"].get("__tsid").is_none()); + let labels = &series["seriesLabels"]; + let source = get_table(&context.frontend(), labels["__name__"].as_str().unwrap()) + .await + .table_info(); + assert_eq!(labels["__table_id"], source.table_id().to_string()); + assert!(labels["__tsid"].as_str().unwrap().parse::().is_ok()); } assert_eq!( value["data"].as_array().unwrap().len(), @@ -5813,7 +6045,7 @@ async fn test_exemplar_query_bounds_and_malformed_rows( .build() .await; let frontend = instance.fe_instance(); - execute_sql(frontend, &format!("CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(9) TIME INDEX, greptime_metric STRING, job STRING, greptime_value DOUBLE, exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric, job)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')")).await; + execute_sql(frontend, &format!("CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(9) TIME INDEX, greptime_metric STRING, job STRING, greptime_value DOUBLE, greptime_exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric, job)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')")).await; execute_sql( frontend, &format!( @@ -5883,6 +6115,8 @@ async fn test_exemplar_query_bounds_and_malformed_rows( assert!(response.get("warnings").is_none(), "{response}"); } if series > 0 { + assert_eq!(response["data"][0]["seriesLabels"]["__table_id"], "42"); + assert_eq!(response["data"][0]["seriesLabels"]["__tsid"], "7"); assert_eq!( response["data"][0]["exemplars"].as_array().unwrap().len(), 2, @@ -5957,15 +6191,13 @@ async fn test_exemplar_concurrent_ddl_and_precreated_contract() { }); assert_eq!(futures::future::join_all(writes).await, vec![1; 8]); } - let info = frontend - .catalog_manager() - .table("greptime", "public", METRICS_EXEMPLARS_TABLE, None) + let info = get_table(frontend, METRICS_EXEMPLARS_TABLE) .await - .unwrap() - .unwrap() .table_info(); assert_eq!(info.meta.primary_key_indices.len(), 12); operator::insert::validate_exemplar_table(&info).unwrap(); + let source = get_table(frontend, "concurrent").await.table_info(); + assert_eq!(source.meta.primary_key_indices.len(), 9); let response = frontend .query_exemplars( Query::parse("concurrent".into(), "0", "1", "public").unwrap(), @@ -5980,7 +6212,7 @@ async fn test_exemplar_concurrent_ddl_and_precreated_contract() { // Incompatible pre-created tables must be rejected before adding inherited tags. let definition = format!( - "CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(6) TIME INDEX, greptime_metric STRING, greptime_value DOUBLE, exemplar_labels JSON2(max_auto_expanded_paths=0, trace_id STRING), PRIMARY KEY (__table_id, __tsid, greptime_metric)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')" + "CREATE TABLE {METRICS_EXEMPLARS_TABLE} (__table_id INT UNSIGNED NOT NULL, __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(6) TIME INDEX, greptime_metric STRING, greptime_value DOUBLE, greptime_exemplar_labels JSON2(max_auto_expanded_paths=0, trace_id STRING), PRIMARY KEY (__table_id, __tsid, greptime_metric)) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')" ); for definition in [ definition.replace("greptime_value DOUBLE", "greptime_value BIGINT"), @@ -5990,12 +6222,8 @@ async fn test_exemplar_concurrent_ddl_and_precreated_contract() { ] { execute_sql(frontend, &format!("DROP TABLE {METRICS_EXEMPLARS_TABLE}")).await; execute_sql(frontend, &definition).await; - let before = frontend - .catalog_manager() - .table("greptime", "public", METRICS_EXEMPLARS_TABLE, None) + let before = get_table(frontend, METRICS_EXEMPLARS_TABLE) .await - .unwrap() - .unwrap() .table_info(); assert!( frontend @@ -6004,14 +6232,15 @@ async fn test_exemplar_concurrent_ddl_and_precreated_contract() { .is_err(), "{definition}" ); - let after = frontend - .catalog_manager() - .table("greptime", "public", METRICS_EXEMPLARS_TABLE, None) + let after = get_table(frontend, METRICS_EXEMPLARS_TABLE) .await - .unwrap() - .unwrap() .table_info(); assert_eq!(before, after, "incompatible table must not be altered"); + let source_after = get_table(frontend, "concurrent").await.table_info(); + assert_eq!( + source, source_after, + "incompatible destination must not alter sources" + ); } let disabled = GreptimeDbStandaloneBuilder::new("exemplar_disabled_ddl") @@ -6034,6 +6263,8 @@ async fn test_exemplar_concurrent_ddl_and_precreated_contract() { .unwrap() .is_none() ); + let source = get_table(frontend, "concurrent").await.table_info(); + assert!(source.meta.primary_key_indices.is_empty()); } #[rstest] @@ -6088,7 +6319,7 @@ async fn test_exemplar_precreated_precision_and_label_defaults( __tsid BIGINT UNSIGNED NOT NULL, greptime_timestamp TIMESTAMP(6) TIME INDEX, greptime_metric STRING, site STRING {default}, - greptime_value DOUBLE, exemplar_labels JSON2, + greptime_value DOUBLE, greptime_exemplar_labels JSON2, PRIMARY KEY (__table_id, __tsid, greptime_metric, site) ) WITH ('append_mode'='true', 'primary_key_encoding'='sparse')" ), @@ -6194,14 +6425,24 @@ async fn test_exemplar_cross_schema_exact_identity() { .query_exemplars(query, QueryContext::arc()) .await .unwrap(); - assert_eq!(response.len(), 1); - assert_eq!(response[0].exemplars.len(), values.len()); + assert_eq!(response.len(), 2); + assert_ne!( + response[0].series_labels["__table_id"], + response[1].series_labels["__table_id"] + ); + assert_eq!( + response[0].series_labels["__tsid"], + response[1].series_labels["__tsid"] + ); let encoded = serde_json::to_string(&response).unwrap(); - for value in ["0", "-0", "NaN", "+Inf", "-Inf"] { - assert!( - response[0].exemplars.iter().any(|e| e.value == value), - "{encoded}" - ); + for series in &response { + assert_eq!(series.exemplars.len(), values.len()); + for value in ["0", "-0", "NaN", "+Inf", "-Inf"] { + assert!( + series.exemplars.iter().any(|e| e.value == value), + "{encoded}" + ); + } } assert!(encoded.contains("\"timestamp\":-0.001"), "{encoded}"); } diff --git a/tests-integration/src/tests/postgres_copy_in_test.rs b/tests-integration/src/tests/postgres_copy_in_test.rs index 5d382330808..d4b3ed0d4ad 100644 --- a/tests-integration/src/tests/postgres_copy_in_test.rs +++ b/tests-integration/src/tests/postgres_copy_in_test.rs @@ -1173,3 +1173,902 @@ async fn test_postgres_copy_in_otlp_all_rejected() { server.shutdown().await.unwrap(); } + +/// `COPY FROM STDIN WITH (FORMAT promwrite)` ingests snappy- and zstd- +/// compressed Prometheus Remote Write v1 bodies through the same +/// ingestion core as `/v1/prometheus/write`. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_v1() { + use api::prom_store::remote::WriteRequest; + use prost::Message; + + init_default_ut_logging(); + let (_guard, server) = setup_pg_server(StorageType::File, "pg_copy_in_promwrite_v1").await; + let port = server.bind_addr().unwrap().port(); + let client = connect(port).await; + + let request = WriteRequest { + timeseries: servers::prom_store::mock_timeseries(), + ..Default::default() + }; + let payload = request.encode_to_vec(); + for (sql, body) in [ + ( + "COPY FROM STDIN WITH (FORMAT promwrite)", + servers::prom_store::snappy_compress(&payload).unwrap(), + ), + ( + "COPY FROM STDIN WITH (FORMAT promwrite, zstd)", + zstd::stream::encode_all(payload.as_slice(), 0).unwrap(), + ), + ] { + // Split either compression format across CopyData boundaries. + let rows = feed_copy_in( + &client, + sql, + body.chunks(16).map(Bytes::copy_from_slice).collect(), + ) + .await + .unwrap(); + assert_eq!(rows, 7); + } + + // mock_timeseries writes metric1..metric3; verify one of them. + let messages = client + .simple_query("SELECT greptime_value FROM metric1 ORDER BY greptime_timestamp") + .await + .unwrap(); + let values: Vec = messages + .iter() + .filter_map(|m| match m { + tokio_postgres::SimpleQueryMessage::Row(row) => row.get(0).map(str::to_string), + _ => None, + }) + .collect(); + assert!(!values.is_empty(), "metric1 must have samples"); + + server.shutdown().await.unwrap(); +} + +/// Production wiring shares Prom logical-table batching between HTTP and COPY, +/// including when the Prom HTTP endpoint is disabled. Ordinary tables keep the +/// direct-write fallback even when the logical batcher is enabled. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_batching_parity() { + use std::sync::Arc; + use std::sync::atomic::{AtomicU8, Ordering}; + use std::time::Duration; + + use api::greptime_proto::io::prometheus::write::v2::Sample; + use api::prom_store::remote; + use frontend::server::Services; + use frontend::service_config::BatcherOptions; + use prost::Message; + use servers::batcher::BatchingProtocol; + use servers::prom_remote_write::v2::test_util as remote_write_v2; + + // Exercise both permission gates through the real COPY/session and batching + // wiring: 0 allows writes, 1 denies the action, 2 denies table access. + #[derive(Default)] + struct WritePermission(AtomicU8); + impl auth::PermissionChecker for WritePermission { + fn check_permission( + &self, + _: auth::UserInfoRef, + req: auth::PermissionReq, + ) -> auth::error::Result { + Ok( + if matches!(req, auth::PermissionReq::Action(auth::PROM_STORE_WRITE)) + && self.0.load(Ordering::Relaxed) == 1 + { + auth::PermissionResp::Reject + } else { + auth::PermissionResp::Allow + }, + ) + } + + fn check_permission_with_table_targets( + &self, + _: auth::UserInfoRef, + req: auth::PermissionReq, + targets: auth::PermissionTableTargets, + ) -> auth::error::Result { + if matches!(req, auth::PermissionReq::Action(auth::PROM_STORE_WRITE)) { + assert!( + matches!(targets, auth::PermissionTableTargets::Resolved(ref t) if !t.is_empty()) + ); + if self.0.load(Ordering::Relaxed) == 2 { + return Ok(auth::PermissionResp::Reject); + } + } + Ok(auth::PermissionResp::Allow) + } + } + + init_default_ut_logging(); + for (prom_batching, http_prom_enabled) in [(false, true), (true, true), (true, false)] { + let checker = Arc::new(WritePermission::default()); + let plugins = common_base::Plugins::new(); + plugins.insert::(checker.clone()); + let mut instance = + crate::standalone::GreptimeDbStandaloneBuilder::new("pg_copy_prom_batch") + .with_plugin(plugins.clone()) + .build() + .await; + let mut opts = instance.opts.clone(); + opts.http.addr = "127.0.0.1:0".into(); + opts.grpc.bind_addr = "127.0.0.1:0".into(); + opts.postgres.addr = "127.0.0.1:0".into(); + opts.prom_store.enable = http_prom_enabled; + // An OTLP-only batcher must not opt Prom COPY into batching. Conversely, + // the PostgreSQL-only Prom case must initialize without another endpoint. + opts.otlp.enable = !prom_batching; + let batcher_options = BatcherOptions { + protocols: if prom_batching { + vec![BatchingProtocol::Prom] + } else { + vec![BatchingProtocol::Otlp] + }, + pending_rows_flush_interval: Duration::from_millis(10), + ..Default::default() + }; + // Cover both the legacy shared configuration and the logical-table + // override. Neither opts into PostgreSQL SQL batching. + opts.pending_rows_batcher.table.protocols.clear(); + if prom_batching && http_prom_enabled { + opts.pending_rows_batcher.table = batcher_options; + } else { + opts.pending_rows_batcher.logical_table = Some(batcher_options); + } + let mut servers = Services::new(opts, instance.fe_instance().clone(), plugins) + .build() + .unwrap(); + servers.start_all().await.unwrap(); + let client = connect(servers.addr("POSTGRES_SERVER").unwrap().port()).await; + + client + .simple_query( + "CREATE TABLE copy_prom_direct (greptime_timestamp TIMESTAMP TIME INDEX, \ + greptime_value DOUBLE) ENGINE=mito", + ) + .await + .unwrap(); + for (metric, eligible) in [("copy_prom_batched", true), ("copy_prom_direct", false)] { + for version in [1, 2] { + let timestamp = 1_700_000_000_000 + version; + let payload = if version == 1 { + remote::WriteRequest { + timeseries: vec![remote::TimeSeries { + labels: vec![remote::Label { + name: "__name__".into(), + value: metric.into(), + }], + samples: vec![remote::Sample { + value: 42.0, + timestamp, + }], + ..Default::default() + }], + ..Default::default() + } + .encode_to_vec() + } else { + remote_write_v2::request_with_labels_and_samples( + vec![("__name__", metric)], + vec![Sample { + value: 42.0, + timestamp, + ..Default::default() + }], + ) + .encode_to_vec() + }; + let body = servers::prom_store::snappy_compress(&payload).unwrap(); + let sql = format!("COPY FROM STDIN WITH (FORMAT promwrite, version {version})"); + for denied_gate in [1, 2] { + checker.0.store(denied_gate, Ordering::Relaxed); + let before = servers::metrics::FLUSH_TOTAL.get(); + let error = feed_copy_in(&client, &sql, vec![Bytes::from(body.clone())]) + .await + .unwrap_err(); + assert_eq!(error.as_db_error().unwrap().code().code(), "42501"); + assert_eq!(servers::metrics::FLUSH_TOTAL.get(), before); + } + checker.0.store(0, Ordering::Relaxed); + let before = servers::metrics::FLUSH_TOTAL.get(); + let rows = feed_copy_in(&client, &sql, vec![Bytes::from(body)]) + .await + .unwrap(); + assert_eq!(rows, 1); + assert_eq!( + servers::metrics::FLUSH_TOTAL.get() - before, + u64::from(prom_batching && eligible), + "batcher selection: prom={prom_batching}, http={http_prom_enabled}, \ + eligible={eligible}, version={version}" + ); + // Default synchronous acknowledgement makes both paths visible + // immediately, with COPY's exact affected-row count preserved. + let rows = client + .simple_query(&format!( + "SELECT greptime_value FROM {metric} ORDER BY greptime_timestamp" + )) + .await + .unwrap(); + let values: Vec<_> = rows + .iter() + .filter_map(|message| match message { + tokio_postgres::SimpleQueryMessage::Row(row) => { + Some(row.get(0).unwrap().parse::().unwrap()) + } + _ => None, + }) + .collect(); + assert_eq!(values, vec![42.0; version as usize]); + } + } + drop(client); + servers.shutdown_all().await.unwrap(); + instance.guard.remove_all().await; + } +} + +/// `COPY FROM STDIN WITH (FORMAT promwrite, version 2)` ingests samples +/// through the shared v2 core and reports their written count. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_v2() { + use api::greptime_proto::io::prometheus::write::v2::Sample; + use prost::Message; + use servers::prom_remote_write::v2::test_util as remote_write_v2; + + init_default_ut_logging(); + let (_guard, server) = setup_pg_server(StorageType::File, "pg_copy_in_promwrite_v2").await; + let port = server.bind_addr().unwrap().port(); + let client = connect(port).await; + + let request = remote_write_v2::request_with_labels_and_samples( + vec![("__name__", "copy_promwrite_v2_gauge"), ("job", "copy")], + vec![Sample { + value: 42.0, + timestamp: 1_700_000_000_000, + ..Default::default() + }], + ); + let body = servers::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + let rows = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite, version 2)", + vec![Bytes::from(body)], + ) + .await + .unwrap(); + assert_eq!(rows, 1); + + let messages = client + .simple_query("SELECT greptime_value FROM copy_promwrite_v2_gauge") + .await + .unwrap(); + let values: Vec = messages + .iter() + .filter_map(|m| match m { + tokio_postgres::SimpleQueryMessage::Row(row) => row.get(0).map(str::to_string), + _ => None, + }) + .collect(); + assert_eq!(values, vec!["42.0".to_string()]); + + server.shutdown().await.unwrap(); +} + +/// The header-equivalent options flow through: a named physical table +/// routes the series, and a garbage body fails the copy with the +/// decoder's error while the connection stays usable. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_options_and_errors() { + use api::prom_store::remote::WriteRequest; + use prost::Message; + + init_default_ut_logging(); + let (_guard, server) = setup_pg_server(StorageType::File, "pg_copy_in_promwrite_opt").await; + let port = server.bind_addr().unwrap().port(); + let client = connect(port).await; + + let request = WriteRequest { + timeseries: servers::prom_store::mock_timeseries_new_label(), + ..Default::default() + }; + let body = servers::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + let rows = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite, physical_table 'copy_phys')", + vec![Bytes::from(body)], + ) + .await + .unwrap(); + assert!(rows > 0); + + // Garbage body: decode error, connection reusable. + let error = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite)", + vec![Bytes::from_static(b"not a snappy body")], + ) + .await + .unwrap_err(); + assert!(error.as_db_error().is_some(), "unexpected error: {error}"); + let messages = client.simple_query("SELECT 1").await.unwrap(); + assert!(!messages.is_empty()); + + server.shutdown().await.unwrap(); +} + +/// The COPY remote-write flush enforces the shared request-memory +/// quota: a payload whose decompressed size exceeds the configured +/// budget fails instead of consuming unbounded memory, matching the +/// HTTP endpoint's behavior under the same limiter. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_memory_quota() { + use api::prom_store::remote::WriteRequest; + use prost::Message; + + init_default_ut_logging(); + // A standalone instance plus a pg server wired with a tiny shared + // quota, as the frontend does for HTTP ingestion. + let instance = + crate::standalone::GreptimeDbStandaloneBuilder::new("pg_copy_in_promwrite_quota") + .build() + .await; + let fe = instance.fe_instance().clone(); + let fe_pg_addr = format!("127.0.0.1:{}", common_test_util::ports::get_port()); + let opts = frontend::service_config::PostgresOptions { + addr: fe_pg_addr.clone(), + ..Default::default() + }; + let tls = std::sync::Arc::new( + servers::tls::ReloadableTlsServerConfig::try_new(opts.tls.clone()).unwrap(), + ); + let runtime = { + use common_runtime::runtime::BuilderBuild as _; + let mut builder = common_runtime::Builder::default(); + builder.worker_threads(2); + builder.thread_name("pg-runtime"); + builder.build().unwrap() + }; + let pg_server = servers::postgres::PostgresServer::new( + fe.clone(), + fe.clone(), + opts.tls.should_force_tls(), + tls, + 0, + runtime, + None, + None, + ) + .with_influxdb_handler(fe.clone()) + .with_otlp_handler(fe.clone(), true) + .with_memory_limiter(servers::request_memory_limiter::ServerMemoryLimiter::new( + // A budget far below the decompressed payload below. + 64 * 1024, + common_memory_manager::OnExhaustedPolicy::default(), + )) + .with_prom_remote_write(fe.clone(), Some(fe.clone()), true, Default::default(), None); + use servers::server::Server; + let mut pg_server = Box::new(pg_server) as Box; + pg_server + .start(fe_pg_addr.parse::().unwrap()) + .await + .unwrap(); + let port = pg_server.bind_addr().unwrap().port(); + let client = connect(port).await; + + // A highly compressible v1 body: tiny compressed, large decompressed. + let big_label = "x".repeat(512 * 1024); + let request = WriteRequest { + timeseries: vec![api::prom_store::remote::TimeSeries { + labels: vec![api::prom_store::remote::Label { + name: "__name__".into(), + value: big_label, + }], + samples: vec![api::prom_store::remote::Sample { + value: 1.0, + timestamp: 1_700_000_000_000, + }], + ..Default::default() + }], + ..Default::default() + }; + let body = servers::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + assert!( + body.len() < 64 * 1024, + "the compressed body must fit the COPY limit" + ); + + let error = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite)", + vec![Bytes::from(body)], + ) + .await + .unwrap_err(); + assert!(error.as_db_error().is_some(), "unexpected error: {error}"); + + // The connection stays usable. + let messages = client.simple_query("SELECT 1").await.unwrap(); + assert!(!messages.is_empty()); + + pg_server.shutdown().await.unwrap(); +} + +/// Concurrent unfinished COPY requests share the receive-stage request +/// memory quota. The fits check covers only a request's OWN holdings; +/// cross-request contention is left to `acquire` and its policy: under +/// `Wait`, a second connection parks until the first releases (via a +/// client CopyFail here, so no decode is involved); under `Fail`, it is +/// rejected immediately. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_shared_receive_quota() { + init_default_ut_logging(); + + // --- Wait policy (the default): B parks, then proceeds once A's + // client abort (CopyFail) drops A's reservation. + let server = quota_test_server( + "pg_copy_in_pw_quota_wait", + common_memory_manager::OnExhaustedPolicy::default(), + ) + .await; + let port = server.bind_addr().unwrap().port(); + + let chunk = vec![b'x'; 48 * 1024]; + let client_a = connect(port).await; + let client_b = connect(port).await; + + let mut sink_a = Box::pin( + client_a + .copy_in("COPY FROM STDIN WITH (FORMAT promwrite)") + .await + .unwrap(), + ); + sink_a + .as_mut() + .send(Bytes::from(chunk.clone())) + .await + .unwrap(); + + // B's chunk does not fit alongside A's holding: its acquisition + // parks. The sink buffers sends locally, so drive it to completion. + let mut sink_b = Box::pin( + client_b + .copy_in("COPY FROM STDIN WITH (FORMAT promwrite)") + .await + .unwrap(), + ); + sink_b + .as_mut() + .send(Bytes::from(chunk.clone())) + .await + .unwrap(); + let b_finish = tokio::spawn(async move { + let mut sink_b = sink_b; + sink_b.as_mut().finish().await + }); + // Let B's acquisition park, then release A's reservation by aborting + // A's copy client-side (CopyFail drops the buffered state). + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + drop(sink_a); + + let b_result = b_finish.await.unwrap(); + match &b_result { + // B got past the parked acquisition and reached the decoder, + // which rejects the garbage body — the point is that it is NOT + // the quota timeout. + Err(e) => assert!( + !e.to_string().contains("Timeout waiting for memory quota"), + "B must be unblocked by A's release, got: {e}" + ), + Ok(rows) => assert!(*rows < 48 * 1024, "unexpected full success: {rows}"), + } + assert!(!client_b.simple_query("SELECT 1").await.unwrap().is_empty()); + server.shutdown().await.unwrap(); + + // --- Fail policy: the same contention is rejected immediately. + let server = quota_test_server( + "pg_copy_in_pw_quota_fail", + common_memory_manager::OnExhaustedPolicy::Fail, + ) + .await; + let port = server.bind_addr().unwrap().port(); + + let client_a = connect(port).await; + let client_b = connect(port).await; + let mut sink_a = Box::pin( + client_a + .copy_in("COPY FROM STDIN WITH (FORMAT promwrite)") + .await + .unwrap(), + ); + sink_a + .as_mut() + .send(Bytes::from(chunk.clone())) + .await + .unwrap(); + + let mut sink_b = Box::pin( + client_b + .copy_in("COPY FROM STDIN WITH (FORMAT promwrite)") + .await + .unwrap(), + ); + sink_b.as_mut().send(Bytes::from(chunk)).await.unwrap(); + let error = sink_b.as_mut().finish().await.unwrap_err(); + let message = error + .as_db_error() + .map(|d| d.message().to_string()) + .unwrap_or_default(); + assert!( + message.to_lowercase().contains("memory"), + "expected immediate memory rejection under Fail, got: {error}" + ); + // A still holds its reservation; both connections stay usable. + drop(sink_a); + assert!(!client_a.simple_query("SELECT 1").await.unwrap().is_empty()); + assert!(!client_b.simple_query("SELECT 1").await.unwrap().is_empty()); + server.shutdown().await.unwrap(); +} + +/// A pg server with a tiny shared request-memory limiter carrying the +/// given policy, wired like the frontend does. +async fn quota_test_server( + name: &str, + policy: common_memory_manager::OnExhaustedPolicy, +) -> std::sync::Arc> { + let instance = crate::standalone::GreptimeDbStandaloneBuilder::new(name) + .build() + .await; + let fe = instance.fe_instance().clone(); + let fe_pg_addr = format!("127.0.0.1:{}", common_test_util::ports::get_port()); + let opts = frontend::service_config::PostgresOptions { + addr: fe_pg_addr.clone(), + ..Default::default() + }; + let tls = std::sync::Arc::new( + servers::tls::ReloadableTlsServerConfig::try_new(opts.tls.clone()).unwrap(), + ); + let runtime = { + use common_runtime::runtime::BuilderBuild as _; + let mut builder = common_runtime::Builder::default(); + builder.worker_threads(2); + builder.thread_name("pg-runtime"); + builder.build().unwrap() + }; + let pg_server = servers::postgres::PostgresServer::new( + fe.clone(), + fe.clone(), + opts.tls.should_force_tls(), + tls, + 0, + runtime, + None, + None, + ) + .with_influxdb_handler(fe.clone()) + .with_otlp_handler(fe.clone(), true) + .with_memory_limiter(servers::request_memory_limiter::ServerMemoryLimiter::new( + 64 * 1024, + policy, + )) + .with_prom_remote_write(fe.clone(), Some(fe.clone()), true, Default::default(), None); + use servers::server::Server; + let mut pg_server = Box::new(pg_server) as Box; + pg_server + .start(fe_pg_addr.parse::().unwrap()) + .await + .unwrap(); + std::sync::Arc::new(pg_server) +} + +/// COPY preserves the v2 pipeline+exemplars restriction: the same +/// payload and options rejected by HTTP ("pipelines with exemplars") +/// are rejected over COPY too, not silently ingested without the +/// pipeline. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_v2_pipeline_exemplar_rejection() { + use api::greptime_proto::io::prometheus::write::v2::{Exemplar, Request, Sample, TimeSeries}; + use prost::Message; + + init_default_ut_logging(); + let (_guard, server) = setup_pg_server(StorageType::File, "pg_copy_in_pw_v2_pipex").await; + let port = server.bind_addr().unwrap().port(); + let client = connect(port).await; + + // One series with a sample and an exemplar; the exemplar makes the + // pipeline combination invalid. + let mut symbols = vec![String::new()]; + let push = |symbols: &mut Vec, s: &str| -> u32 { + match symbols.iter().position(|x| x == s) { + Some(i) => i as u32, + None => { + symbols.push(s.to_string()); + (symbols.len() - 1) as u32 + } + } + }; + let name_ref = push(&mut symbols, "__name__"); + let value_ref = push(&mut symbols, "copy_pw_v2_exemplar_metric"); + let trace_ref = push(&mut symbols, "trace_id"); + let trace_value_ref = push(&mut symbols, "trace-1"); + let request = Request { + symbols, + timeseries: vec![TimeSeries { + labels_refs: vec![name_ref, value_ref], + samples: vec![Sample { + value: 1.0, + timestamp: 1_700_000_000_000, + ..Default::default() + }], + histograms: Vec::new(), + exemplars: vec![Exemplar { + labels_refs: vec![trace_ref, trace_value_ref], + value: 1.0, + timestamp: 1_700_000_000_000, + }], + metadata: None, + }], + }; + let body = servers::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + + let error = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite, version 2, pipeline_name 'some_pipeline')", + vec![Bytes::from(body)], + ) + .await + .unwrap_err(); + let message = error + .as_db_error() + .map(|d| d.message().to_string()) + .unwrap_or_default(); + assert!( + message.contains("pipelines with exemplars"), + "unexpected error: {error}" + ); + + // Without the pipeline option the same payload ingests. + let request = Request { + symbols: request.symbols, + timeseries: vec![TimeSeries { + labels_refs: vec![name_ref, value_ref], + samples: vec![Sample { + value: 1.0, + timestamp: 1_700_000_000_001, + ..Default::default() + }], + histograms: Vec::new(), + exemplars: vec![Exemplar { + labels_refs: vec![trace_ref, trace_value_ref], + value: 1.0, + timestamp: 1_700_000_000_001, + }], + metadata: None, + }], + }; + let body = servers::prom_store::snappy_compress(&request.encode_to_vec()).unwrap(); + let rows = feed_copy_in( + &client, + "COPY FROM STDIN WITH (FORMAT promwrite, version 2)", + vec![Bytes::from(body)], + ) + .await + .unwrap(); + assert!(rows > 0); + + server.shutdown().await.unwrap(); +} + +/// Chunked delivery has the same receive-quota behavior as a single +/// chunk: each chunk charges only its uncovered delta, so the limiter's +/// per-acquisition KiB rounding adds at most one rounding for the whole +/// stream. 48 KiB in 16-byte chunks (3072 acquisitions) fits a 64 KiB +/// budget where per-chunk full charges would retain ~3 MiB of permits +/// and reject. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_chunked_quota_parity() { + init_default_ut_logging(); + let server = quota_test_server( + "pg_copy_in_pw_quota_chunks", + common_memory_manager::OnExhaustedPolicy::Fail, + ) + .await; + let port = server.bind_addr().unwrap().port(); + let client = connect(port).await; + + let payload: Vec = vec![b'x'; 48 * 1024]; + let mut sink = Box::pin( + client + .copy_in("COPY FROM STDIN WITH (FORMAT promwrite)") + .await + .unwrap(), + ); + // 3072 chunks of 16 bytes: under per-chunk charging each acquisition + // rounds up to 1 KiB, totalling ~3 MiB of permits — far past the + // 64 KiB budget; delta charging holds only the payload's bytes. + for chunk in payload.chunks(16) { + sink.as_mut() + .send(Bytes::copy_from_slice(chunk)) + .await + .unwrap(); + } + // The payload must have been accepted at the receive stage; the + // garbage body then fails at decode (not with a quota error). + let error = sink.as_mut().finish().await.unwrap_err(); + let message = error + .as_db_error() + .map(|d| d.message().to_string()) + .unwrap_or_default(); + assert!( + !message.to_lowercase().contains("memory") && !message.contains("quota"), + "chunked delivery must not exhaust the quota, got: {error} ({message})" + ); + + // The connection stays usable. + let messages = client.simple_query("SELECT 1").await.unwrap(); + assert!(!messages.is_empty()); + + server.shutdown().await.unwrap(); +} + +/// A COPY must reject allocations that cannot fit alongside its own body, +/// rather than waiting for the body permits it cannot release until flush ends. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_prom_write_held_body_quota() { + use std::time::Duration; + + init_default_ut_logging(); + let server = quota_test_server( + "pg_copy_in_pw_held_body", + common_memory_manager::OnExhaustedPolicy::Wait { + timeout: Duration::from_secs(30), + }, + ) + .await; + let client = connect(server.bind_addr().unwrap().port()).await; + + // Incompressible bytes make both the compressed body and decompression + // reservation fit individually, but not together. Rejection must precede + // protobuf decoding, so the uncompressed contents need not be valid protobuf. + let mut state = 1u32; + let decoded: Vec = (0..48 * 1024) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state as u8 + }) + .collect(); + let body = servers::prom_store::snappy_compress(&decoded).unwrap(); + assert!(body.len() < 64 * 1024); + assert!(body.len() + decoded.len() > 64 * 1024); + + assert_copy_rejects_held_body(&client, "COPY FROM STDIN WITH (FORMAT promwrite)", body).await; + server.shutdown().await.unwrap(); +} + +/// Existing table-less formats enforce the same receive budget as Prometheus. +/// An oversized body is rejected before either format's decoder runs. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_buffered_receive_quota() { + let server = quota_test_server( + "pg_copy_buffered_receive_quota", + common_memory_manager::OnExhaustedPolicy::Fail, + ) + .await; + let client = connect(server.bind_addr().unwrap().port()).await; + for format in ["influxdb", "otlp"] { + let error = feed_copy_in( + &client, + &format!("COPY FROM STDIN WITH (FORMAT {format})"), + vec![Bytes::from(vec![b'x'; 65 * 1024])], + ) + .await + .unwrap_err(); + assert!( + error + .as_db_error() + .unwrap() + .message() + .to_lowercase() + .contains("memory") + ); + assert!(!client.simple_query("SELECT 1").await.unwrap().is_empty()); + } + server.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_postgres_copy_in_otlp_held_body_quota() { + use std::time::Duration; + + use otel_arrow_rust::proto::opentelemetry::collector::metrics::v1::ExportMetricsServiceRequest; + use otel_arrow_rust::proto::opentelemetry::common::v1::{AnyValue, KeyValue, any_value}; + use otel_arrow_rust::proto::opentelemetry::metrics::v1::{ + Exemplar, Gauge, Metric, NumberDataPoint, ResourceMetrics, ScopeMetrics, exemplar, metric, + number_data_point, + }; + use prost::Message; + + init_default_ut_logging(); + let server = quota_test_server( + "pg_copy_in_otlp_held_body", + common_memory_manager::OnExhaustedPolicy::Wait { + timeout: Duration::from_secs(30), + }, + ) + .await; + let client = connect(server.bind_addr().unwrap().port()).await; + let request = ExportMetricsServiceRequest { + resource_metrics: vec![ResourceMetrics { + scope_metrics: vec![ScopeMetrics { + metrics: vec![Metric { + name: "copy_otlp_held_body".into(), + data: Some(metric::Data::Gauge(Gauge { + data_points: vec![NumberDataPoint { + time_unix_nano: 1_700_000_000_000_000_000, + value: Some(number_data_point::Value::AsDouble(1.0)), + attributes: vec![KeyValue { + key: "host".into(), + value: Some(AnyValue { + value: Some(any_value::Value::StringValue( + "x".repeat(13 * 1024), + )), + }), + }], + exemplars: vec![Exemplar { + time_unix_nano: 1_700_000_000_000_000_000, + value: Some(exemplar::Value::AsDouble(1.0)), + ..Default::default() + }], + ..Default::default() + }], + })), + ..Default::default() + }], + ..Default::default() + }], + ..Default::default() + }], + }; + let body = request.encode_to_vec(); + let conversion = servers::otlp::metrics::to_grpc_insert_requests( + request, + &mut session::protocol_ctx::OtlpMetricCtx { + with_metric_engine: true, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(conversion.exemplars.len(), 1); + let expansion = + servers::exemplar::estimated_row_bytes([conversion.exemplars.as_slice()]).unwrap(); + assert!(body.len() < 64 * 1024); + assert!(expansion < 64 * 1024); + assert!(body.len() + expansion > 64 * 1024); + + assert_copy_rejects_held_body(&client, "COPY FROM STDIN WITH (FORMAT otlp)", body).await; + server.shutdown().await.unwrap(); +} + +async fn assert_copy_rejects_held_body(client: &tokio_postgres::Client, sql: &str, body: Vec) { + let error = tokio::time::timeout( + std::time::Duration::from_secs(5), + feed_copy_in(client, sql, vec![Bytes::from(body)]), + ) + .await + .expect("COPY must reject promptly, not wait on its own body reservation") + .unwrap_err(); + let message = error.as_db_error().unwrap().message(); + assert!( + message.to_lowercase().contains("memory") + && !message.contains("Timeout waiting for memory quota"), + "expected a self-fit rejection, got: {message}" + ); + assert!(!client.simple_query("SELECT 1").await.unwrap().is_empty()); +} diff --git a/tests-integration/tests/http.rs b/tests-integration/tests/http.rs index ef06aa2556a..9556774d2a4 100644 --- a/tests-integration/tests/http.rs +++ b/tests-integration/tests/http.rs @@ -4686,6 +4686,7 @@ providers = []"#, r#" enable_telemetry = true auto_create_table = true +auto_create_physical_table_timestamp_unit = "Millisecond" max_in_flight_write_bytes = "0KiB" write_bytes_exhausted_policy = "wait" init_regions_in_background = false diff --git a/tests/cases/standalone/common/drop/drop_database_logical_tables.result b/tests/cases/standalone/common/drop/drop_database_logical_tables.result new file mode 100644 index 00000000000..a6920679e4d --- /dev/null +++ b/tests/cases/standalone/common/drop/drop_database_logical_tables.result @@ -0,0 +1,126 @@ +CREATE DATABASE drop_database_logical_tables; + +Affected Rows: 1 + +USE drop_database_logical_tables; + +Affected Rows: 0 + +CREATE TABLE physical (ts TIMESTAMP TIME INDEX, val DOUBLE) +ENGINE=metric WITH ("physical_metric_table" = ""); + +Affected Rows: 0 + +CREATE TABLE logical_a (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); + +Affected Rows: 0 + +CREATE TABLE logical_b (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); + +Affected Rows: 0 + +CREATE TABLE ordinary (ts TIMESTAMP TIME INDEX, val DOUBLE); + +Affected Rows: 0 + +CREATE VIEW logical_view AS SELECT * FROM logical_a; + +Affected Rows: 0 + +INSERT INTO logical_a (ts, val, host) VALUES (0, 1, 'a'); + +Affected Rows: 1 + +INSERT INTO logical_b (ts, val, host) VALUES (0, 2, 'b'); + +Affected Rows: 1 + +SELECT * FROM logical_view; + ++------+---------------------+-----+ +| host | ts | val | ++------+---------------------+-----+ +| a | 1970-01-01T00:00:00 | 1.0 | ++------+---------------------+-----+ + +SHOW TABLES; + ++----------------------------------------+ +| Tables_in_drop_database_logical_tables | ++----------------------------------------+ +| logical_a | +| logical_b | +| logical_view | +| ordinary | +| physical | ++----------------------------------------+ + +USE public; + +Affected Rows: 0 + +DROP DATABASE drop_database_logical_tables; + +Affected Rows: 0 + +SHOW TABLES FROM drop_database_logical_tables; + ++----------------------------------------+ +| Tables_in_drop_database_logical_tables | ++----------------------------------------+ ++----------------------------------------+ + +-- Reusing names must not retain the dropped tables or their rows. +CREATE DATABASE drop_database_logical_tables; + +Affected Rows: 1 + +USE drop_database_logical_tables; + +Affected Rows: 0 + +CREATE TABLE physical (ts TIMESTAMP TIME INDEX, val DOUBLE) +ENGINE=metric WITH ("physical_metric_table" = ""); + +Affected Rows: 0 + +CREATE TABLE logical_a (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); + +Affected Rows: 0 + +SELECT COUNT(*) FROM logical_a; + ++----------+ +| count(*) | ++----------+ +| 0 | ++----------+ + +INSERT INTO logical_a (ts, val, host) VALUES (0, 7, 'new'); + +Affected Rows: 1 + +-- SQLNESS ARG restart=true +SELECT * FROM logical_a; + ++------+---------------------+-----+ +| host | ts | val | ++------+---------------------+-----+ +| new | 1970-01-01T00:00:00 | 7.0 | ++------+---------------------+-----+ + +USE public; + +Affected Rows: 0 + +DROP DATABASE drop_database_logical_tables; + +Affected Rows: 0 + +DROP DATABASE IF EXISTS drop_database_logical_tables; + +Affected Rows: 0 + diff --git a/tests/cases/standalone/common/drop/drop_database_logical_tables.sql b/tests/cases/standalone/common/drop/drop_database_logical_tables.sql new file mode 100644 index 00000000000..753a34a251d --- /dev/null +++ b/tests/cases/standalone/common/drop/drop_database_logical_tables.sql @@ -0,0 +1,37 @@ +CREATE DATABASE drop_database_logical_tables; +USE drop_database_logical_tables; + +CREATE TABLE physical (ts TIMESTAMP TIME INDEX, val DOUBLE) +ENGINE=metric WITH ("physical_metric_table" = ""); +CREATE TABLE logical_a (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); +CREATE TABLE logical_b (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); +CREATE TABLE ordinary (ts TIMESTAMP TIME INDEX, val DOUBLE); +CREATE VIEW logical_view AS SELECT * FROM logical_a; + +INSERT INTO logical_a (ts, val, host) VALUES (0, 1, 'a'); +INSERT INTO logical_b (ts, val, host) VALUES (0, 2, 'b'); +SELECT * FROM logical_view; +SHOW TABLES; + +USE public; +DROP DATABASE drop_database_logical_tables; +SHOW TABLES FROM drop_database_logical_tables; + +-- Reusing names must not retain the dropped tables or their rows. +CREATE DATABASE drop_database_logical_tables; +USE drop_database_logical_tables; +CREATE TABLE physical (ts TIMESTAMP TIME INDEX, val DOUBLE) +ENGINE=metric WITH ("physical_metric_table" = ""); +CREATE TABLE logical_a (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) +ENGINE=metric WITH ("on_physical_table" = "physical"); +SELECT COUNT(*) FROM logical_a; +INSERT INTO logical_a (ts, val, host) VALUES (0, 7, 'new'); + +-- SQLNESS ARG restart=true +SELECT * FROM logical_a; + +USE public; +DROP DATABASE drop_database_logical_tables; +DROP DATABASE IF EXISTS drop_database_logical_tables; diff --git a/tests/cases/standalone/common/partition_iceberg_hash_functions.result b/tests/cases/standalone/common/partition_iceberg_hash_functions.result new file mode 100644 index 00000000000..6355cea0b0b --- /dev/null +++ b/tests/cases/standalone/common/partition_iceberg_hash_functions.result @@ -0,0 +1,168 @@ +-- The SQL forms of the partition hash: measure a value, derive breakpoints, and +-- render a rule list that can be pasted into `PARTITION ON COLUMNS`. +SELECT iceberg_hash('iceberg'), iceberg_hash(34), iceberg_hash(NULL); + ++-------------------------------+-------------------------+--------------------+ +| iceberg_hash(Utf8("iceberg")) | iceberg_hash(Int64(34)) | iceberg_hash(NULL) | ++-------------------------------+-------------------------+--------------------+ +| 1210000089 | 2017239379 | | ++-------------------------------+-------------------------+--------------------+ + +SELECT iceberg_hash_bounds(2), iceberg_hash_bounds(4); + ++-------------------------------+-------------------------------+ +| iceberg_hash_bounds(Int64(2)) | iceberg_hash_bounds(Int64(4)) | ++-------------------------------+-------------------------------+ +| 0 | -1073741824, 0, 1073741824 | ++-------------------------------+-------------------------------+ + +SELECT iceberg_hash_breakpoints('iceberg', 'a', 'iceberg'); + ++---------------------------------------------------------------------+ +| iceberg_hash_breakpoints(Utf8("iceberg"),Utf8("a"),Utf8("iceberg")) | ++---------------------------------------------------------------------+ +| 1009084850, 1210000089 | ++---------------------------------------------------------------------+ + +-- The column name is quoted, so a name that is not a plain lowercase identifier +-- can be pasted as-is. +SELECT iceberg_hash_ranges_sql('host-name', 4); + ++-----------------------------------------------------------------------------+ +| iceberg_hash_ranges_sql(Utf8("host-name"),Int64(4)) | ++-----------------------------------------------------------------------------+ +| iceberg_hash("host-name") < -1073741824, | +| iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0, | +| iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824, | +| iceberg_hash("host-name") >= 1073741824 | ++-----------------------------------------------------------------------------+ + +-- A column name in another Arrow string representation renders the same rules. +SELECT iceberg_hash_ranges_sql(arrow_cast('host-name', 'Utf8View'), 4); + ++----------------------------------------------------------------------------------+ +| iceberg_hash_ranges_sql(arrow_cast(Utf8("host-name"),Utf8("Utf8View")),Int64(4)) | ++----------------------------------------------------------------------------------+ +| iceberg_hash("host-name") < -1073741824, | +| iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0, | +| iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824, | +| iceberg_hash("host-name") >= 1073741824 | ++----------------------------------------------------------------------------------+ + +-- The rules below are the exact output of +-- `iceberg_hash_ranges_sql('host-name', 4)` pasted into a table definition. +CREATE TABLE hash_function_rules ( + "host-name" STRING, + ts TIMESTAMP TIME INDEX, + v INT, + PRIMARY KEY ("host-name") +) +PARTITION ON COLUMNS ("host-name") ( + iceberg_hash("host-name") < -1073741824, + iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0, + iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824, + iceberg_hash("host-name") >= 1073741824 +); + +Affected Rows: 0 + +SHOW CREATE TABLE hash_function_rules; + ++---------------------+-------------------------------------------------------------------------------+ +| Table | Create Table | ++---------------------+-------------------------------------------------------------------------------+ +| hash_function_rules | CREATE TABLE IF NOT EXISTS "hash_function_rules" ( | +| | "host-name" STRING NULL, | +| | "ts" TIMESTAMP(3) NOT NULL, | +| | "v" INT NULL, | +| | TIME INDEX ("ts"), | +| | PRIMARY KEY ("host-name") | +| | ) | +| | PARTITION ON COLUMNS ("host-name") ( | +| | iceberg_hash("host-name") < -1073741824, | +| | iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0, | +| | iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824, | +| | iceberg_hash("host-name") >= 1073741824 | +| | ) | +| | ENGINE=mito | +| | | ++---------------------+-------------------------------------------------------------------------------+ + +SELECT partition_name, partition_description +FROM information_schema.partitions +WHERE table_name = 'hash_function_rules' +ORDER BY partition_name; + ++----------------+------------------------------------------------------------------------+ +| partition_name | partition_description | ++----------------+------------------------------------------------------------------------+ +| p0 | iceberg_hash(host-name) < -1073741824 | +| p1 | iceberg_hash(host-name) >= -1073741824 AND iceberg_hash(host-name) < 0 | +| p2 | iceberg_hash(host-name) >= 0 AND iceberg_hash(host-name) < 1073741824 | +| p3 | iceberg_hash(host-name) >= 1073741824 | ++----------------+------------------------------------------------------------------------+ + +-- One value per bucket, plus NULL, which the first rule catches. +INSERT INTO hash_function_rules VALUES + ('b', 1, 1), + ('delta', 2, 2), + ('a', 3, 3), + ('iceberg', 4, 4), + (NULL, 5, 5); + +Affected Rows: 5 + +-- The hashes the rendered predicates compare against, computed by the function +-- the rules evaluate: b < -1073741824, -1073741824 <= delta < 0, +-- 0 <= a < 1073741824, iceberg >= 1073741824, NULL hashes to NULL. +SELECT "host-name", iceberg_hash("host-name") FROM hash_function_rules ORDER BY ts; + ++-----------+---------------------------------------------+ +| host-name | iceberg_hash(hash_function_rules.host-name) | ++-----------+---------------------------------------------+ +| b | -1780580861 | +| delta | -418823380 | +| a | 1009084850 | +| iceberg | 1210000089 | +| | | ++-----------+---------------------------------------------+ + +-- Row counts per rendered predicate, plus the NULL hash no comparison matches ... +SELECT + sum(CASE WHEN iceberg_hash("host-name") < -1073741824 THEN 1 ELSE 0 END) AS p0, + sum(CASE WHEN iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0 THEN 1 ELSE 0 END) AS p1, + sum(CASE WHEN iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824 THEN 1 ELSE 0 END) AS p2, + sum(CASE WHEN iceberg_hash("host-name") >= 1073741824 THEN 1 ELSE 0 END) AS p3, + sum(CASE WHEN iceberg_hash("host-name") IS NULL THEN 1 ELSE 0 END) AS nulls +FROM hash_function_rules; + ++----+----+----+----+-------+ +| p0 | p1 | p2 | p3 | nulls | ++----+----+----+----+-------+ +| 1 | 1 | 1 | 1 | 1 | ++----+----+----+----+-------+ + +-- ... must match the rows each region actually stored: region 0 holds p0 plus +-- the NULL rows, and every other region holds exactly its bucket. +-- SQLNESS SLEEP 3s +SELECT region_number, region_rows +FROM information_schema.region_statistics +WHERE table_id IN ( + SELECT table_id FROM information_schema.tables + WHERE table_name = 'hash_function_rules' AND table_schema = 'public' +) +ORDER BY region_number; + ++---------------+-------------+ +| region_number | region_rows | ++---------------+-------------+ +| 0 | 2 | +| 1 | 1 | +| 2 | 1 | +| 3 | 1 | ++---------------+-------------+ + +DROP TABLE hash_function_rules; + +Affected Rows: 0 + diff --git a/tests/cases/standalone/common/partition_iceberg_hash_functions.sql b/tests/cases/standalone/common/partition_iceberg_hash_functions.sql new file mode 100644 index 00000000000..44429a51709 --- /dev/null +++ b/tests/cases/standalone/common/partition_iceberg_hash_functions.sql @@ -0,0 +1,72 @@ +-- The SQL forms of the partition hash: measure a value, derive breakpoints, and +-- render a rule list that can be pasted into `PARTITION ON COLUMNS`. + +SELECT iceberg_hash('iceberg'), iceberg_hash(34), iceberg_hash(NULL); + +SELECT iceberg_hash_bounds(2), iceberg_hash_bounds(4); + +SELECT iceberg_hash_breakpoints('iceberg', 'a', 'iceberg'); + +-- The column name is quoted, so a name that is not a plain lowercase identifier +-- can be pasted as-is. +SELECT iceberg_hash_ranges_sql('host-name', 4); + +-- A column name in another Arrow string representation renders the same rules. +SELECT iceberg_hash_ranges_sql(arrow_cast('host-name', 'Utf8View'), 4); + +-- The rules below are the exact output of +-- `iceberg_hash_ranges_sql('host-name', 4)` pasted into a table definition. +CREATE TABLE hash_function_rules ( + "host-name" STRING, + ts TIMESTAMP TIME INDEX, + v INT, + PRIMARY KEY ("host-name") +) +PARTITION ON COLUMNS ("host-name") ( + iceberg_hash("host-name") < -1073741824, + iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0, + iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824, + iceberg_hash("host-name") >= 1073741824 +); + +SHOW CREATE TABLE hash_function_rules; + +SELECT partition_name, partition_description +FROM information_schema.partitions +WHERE table_name = 'hash_function_rules' +ORDER BY partition_name; + +-- One value per bucket, plus NULL, which the first rule catches. +INSERT INTO hash_function_rules VALUES + ('b', 1, 1), + ('delta', 2, 2), + ('a', 3, 3), + ('iceberg', 4, 4), + (NULL, 5, 5); + +-- The hashes the rendered predicates compare against, computed by the function +-- the rules evaluate: b < -1073741824, -1073741824 <= delta < 0, +-- 0 <= a < 1073741824, iceberg >= 1073741824, NULL hashes to NULL. +SELECT "host-name", iceberg_hash("host-name") FROM hash_function_rules ORDER BY ts; + +-- Row counts per rendered predicate, plus the NULL hash no comparison matches ... +SELECT + sum(CASE WHEN iceberg_hash("host-name") < -1073741824 THEN 1 ELSE 0 END) AS p0, + sum(CASE WHEN iceberg_hash("host-name") >= -1073741824 AND iceberg_hash("host-name") < 0 THEN 1 ELSE 0 END) AS p1, + sum(CASE WHEN iceberg_hash("host-name") >= 0 AND iceberg_hash("host-name") < 1073741824 THEN 1 ELSE 0 END) AS p2, + sum(CASE WHEN iceberg_hash("host-name") >= 1073741824 THEN 1 ELSE 0 END) AS p3, + sum(CASE WHEN iceberg_hash("host-name") IS NULL THEN 1 ELSE 0 END) AS nulls +FROM hash_function_rules; + +-- ... must match the rows each region actually stored: region 0 holds p0 plus +-- the NULL rows, and every other region holds exactly its bucket. +-- SQLNESS SLEEP 3s +SELECT region_number, region_rows +FROM information_schema.region_statistics +WHERE table_id IN ( + SELECT table_id FROM information_schema.tables + WHERE table_name = 'hash_function_rules' AND table_schema = 'public' +) +ORDER BY region_number; + +DROP TABLE hash_function_rules; diff --git a/tests/cases/standalone/common/select/prune_field.result b/tests/cases/standalone/common/select/prune_field.result index 6a19b67c21a..27aee9296ed 100644 --- a/tests/cases/standalone/common/select/prune_field.result +++ b/tests/cases/standalone/common/select/prune_field.result @@ -67,3 +67,61 @@ drop table prune_field; Affected Rows: 0 +-- -0.0 equals 0.0, so flushed rows holding either zero must keep matching. +CREATE TABLE prune_float_zero (ts TIMESTAMP TIME INDEX, v DOUBLE); + +Affected Rows: 0 + +insert into prune_float_zero values (1, -0.0), (2, 0.0), (3, 1.0); + +Affected Rows: 3 + +admin flush_table('prune_float_zero'); + ++---------------------------------------+ +| ADMIN flush_table('prune_float_zero') | ++---------------------------------------+ +| 0 | ++---------------------------------------+ + +select ts from prune_float_zero where v = 0.0 order by ts; + ++-------------------------+ +| ts | ++-------------------------+ +| 1970-01-01T00:00:00.001 | +| 1970-01-01T00:00:00.002 | ++-------------------------+ + +select ts from prune_float_zero where v >= 0 order by ts; + ++-------------------------+ +| ts | ++-------------------------+ +| 1970-01-01T00:00:00.001 | +| 1970-01-01T00:00:00.002 | +| 1970-01-01T00:00:00.003 | ++-------------------------+ + +select ts from prune_float_zero where v = -0.0 order by ts; + ++-------------------------+ +| ts | ++-------------------------+ +| 1970-01-01T00:00:00.001 | +| 1970-01-01T00:00:00.002 | ++-------------------------+ + +select ts from prune_float_zero where v in (0.0) order by ts; + ++-------------------------+ +| ts | ++-------------------------+ +| 1970-01-01T00:00:00.001 | +| 1970-01-01T00:00:00.002 | ++-------------------------+ + +drop table prune_float_zero; + +Affected Rows: 0 + diff --git a/tests/cases/standalone/common/select/prune_field.sql b/tests/cases/standalone/common/select/prune_field.sql index a5cf255a0d6..03b4e5d89b4 100644 --- a/tests/cases/standalone/common/select/prune_field.sql +++ b/tests/cases/standalone/common/select/prune_field.sql @@ -22,3 +22,20 @@ select * from prune_field; select * from prune_field where a = 1 and b = 1; drop table prune_field; + +-- -0.0 equals 0.0, so flushed rows holding either zero must keep matching. +CREATE TABLE prune_float_zero (ts TIMESTAMP TIME INDEX, v DOUBLE); + +insert into prune_float_zero values (1, -0.0), (2, 0.0), (3, 1.0); + +admin flush_table('prune_float_zero'); + +select ts from prune_float_zero where v = 0.0 order by ts; + +select ts from prune_float_zero where v >= 0 order by ts; + +select ts from prune_float_zero where v = -0.0 order by ts; + +select ts from prune_float_zero where v in (0.0) order by ts; + +drop table prune_float_zero; diff --git a/tests/compatibility/README.md b/tests/compatibility/README.md index 0dfcba20215..47f80bafc76 100644 --- a/tests/compatibility/README.md +++ b/tests/compatibility/README.md @@ -15,6 +15,18 @@ read-preference defaulting and unrelated metadata preservation: cargo nextest run -p session -E 'test(test_legacy_query_context_wire_defaults_to_leader)' ``` +The same legacy fixture also checks that logical ALTER requests without pinned +table IDs retain their original behavior, and that new pinned IDs survive the +query-context protobuf conversion: + +```shell +cargo nextest run -p common-meta -E 'test(test_pinned_logical_table_ids_wire_compatibility)' +``` + +The ID guard requires an updated metasrv; older metasrvs ignore the new internal +extension. Update metasrv before enabling exemplar source-schema reconciliation +on frontends. No procedure-state or protobuf fields are added. + The [native histogram Remote Read fixture](wire/native_histogram_remote_read/README.md) checks the sampled encoder against independent Prometheus protobuf values, including exact integer counts, custom bounds, and NaN payloads: @@ -23,6 +35,14 @@ including exact integer counts, custom bounds, and NaN payloads: cargo nextest run -p servers -E 'test(native_histogram_remote_read_wire_fixture)' ``` +The [legacy region lease fixture](wire/candidate_lease_id/README.md) checks that a +lease without a candidate lease id decodes as such and encodes to the same bytes +as before the field existed: + +```shell +cargo nextest run -p store-api -E 'test(test_legacy_granted_region_wire)' +``` + ## Object store WAL fixtures Frozen objects of the object store WAL are checked byte for byte by the format tests of diff --git a/tests/compatibility/wire/candidate_lease_id/README.md b/tests/compatibility/wire/candidate_lease_id/README.md new file mode 100644 index 00000000000..6d8253afffc --- /dev/null +++ b/tests/compatibility/wire/candidate_lease_id/README.md @@ -0,0 +1,36 @@ +# Legacy region lease wire compatibility + +`legacy_granted_region.pb` is a generated legacy-shaped protobuf fixture, not a +capture from a released binary. It uses the `greptime.v1.meta.GrantedRegion` +schema at greptime-proto revision +`5870f5adc4d601173e742888e7db2032a690b583`, which has no `candidate_lease_id`. + +Logical contents: + +- Region `4398046511105` (table `1024`, region number `1`), role `Follower`. +- Unrelated extension `plugin = "payload"`. + +SHA-256: `9b0c4796ce8c5094a0336c1febb00e3bef62c9deb4b8a04eccc26f28ea265ec0`. + +Run the regression: + +```shell +cargo nextest run -p store-api -E 'test(test_legacy_granted_region_wire)' +``` + +The test checks that the legacy bytes decode to a lease without a candidate +lease id, and that such a lease encodes back to exactly the legacy bytes, so a +release without the field reads it unchanged. + +## Reproduce the fixture + +With `PROTO_ROOT` pointing to the `proto` directory of the revision above: + +```shell +printf '%s\n' \ + 'region_id: 4398046511105' \ + 'role: Follower' \ + 'extensions { key: "plugin" value: "payload" }' | + protoc --proto_path="$PROTO_ROOT" --encode=greptime.v1.meta.GrantedRegion \ + greptime/v1/meta/heartbeat.proto > legacy_granted_region.pb +``` diff --git a/tests/compatibility/wire/candidate_lease_id/legacy_granted_region.pb b/tests/compatibility/wire/candidate_lease_id/legacy_granted_region.pb new file mode 100644 index 00000000000..bc4d199252e --- /dev/null +++ b/tests/compatibility/wire/candidate_lease_id/legacy_granted_region.pb @@ -0,0 +1,2 @@ +�€€€€€š +pluginpayload \ No newline at end of file