Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
discord9
2026-04-16 16:57:47 +08:00
parent 2775832022
commit b04e35ff62
2 changed files with 41 additions and 37 deletions

View File

@@ -1,10 +1,10 @@
# How Dynamic Filtering Speeds Up TopK Queries in GreptimeDB
# From Tens of Seconds to Sub-Second: How GreptimeDB Speeds Up TopK Queries
`ORDER BY ... LIMIT 10` looks cheap because it only returns a few rows. On a large table, it often is not. If the sort key is not the time-index column, the database may still need to read a large amount of data before it can determine which 10 rows belong in the final result.
GreptimeDB improves this path by passing the runtime boundary formed by TopK to the scan as early as possible. Once later data can no longer enter the final result, the scan can skip it. On a real traces dataset, this turns `ORDER BY end_time DESC LIMIT 10` from tens of seconds into a sub-second query.
## Why `LIMIT 10` used to be slow
## Why `ORDER BY ... LIMIT` can still be expensive
Consider this query:
@@ -17,9 +17,9 @@ LIMIT 10;
Under the old path, the scan could not see TopK's runtime state and had to keep reading until the query finished. Even though the query returned only 10 rows, it could still scan a large amount of data first. Only after the query finished was it clear that much of that work had not been necessary.
Here `start_time` is the table's time-index column, while `end_time` is not. GreptimeDB previously had a more complex optimization called window sort, which depended more on time-index data distribution. Now both kinds of TopK queries go through the same dyn filter path.
Here `start_time` is the table's time-index column, while `end_time` is not. GreptimeDB previously had a more complex optimization called window sort, which depended more on time-index data distribution. Now both kinds of TopK queries go through the same dynamic-filtering path.
## What dyn filter changes
## How dynamic filtering changes the scan path
The execution path is now straightforward:
@@ -30,7 +30,7 @@ The execution path is now straightforward:
For queries like `ORDER BY end_time DESC LIMIT 10`, the win comes from narrowing the scan range earlier, not from returning fewer rows.
## How the runtime threshold prunes data
## How the runtime threshold becomes a pruning condition
Once TopK has a meaningful boundary, GreptimeDB uses it as part of the dynamic predicate seen by the scan.
@@ -49,7 +49,7 @@ flowchart TD
C --> D[Irrelevant data is skipped]
```
## What to look for in the plan
## How to verify it in the execution plan
`EXPLAIN ANALYZE VERBOSE` shows whether this path is active.
@@ -72,9 +72,9 @@ Together, these two fragments show:
- `SortExec: TopK(..., filter=[...])`: TopK has formed a runtime threshold.
- scan `dyn_filters`: the scan receives that threshold and uses it for pruning.
If both signals are present, dyn filter is usually active on this query.
If both signals are present, dynamic filtering is usually active on this query.
## How much faster can it get?
## What the speedup looks like in practice
On this traces dataset, the clearest example is this query:
@@ -87,19 +87,19 @@ LIMIT 10;
| Query | Before | Now | Notes |
| --- | ---: | ---: | --- |
| `ORDER BY end_time DESC LIMIT 10` | ~`28.9s` | ~`0.21s` | `end_time` is not the time-index column, and the old unoptimized path effectively scanned the whole table; after switching to dyn filter, later scan work can be pruned much earlier |
| `ORDER BY start_time DESC LIMIT 10` | `0.334s` / `0.336s` / `0.342s` | `0.336s` / `0.340s` / `0.336s` | `start_time` is the time-index column; window sort was already fast in this case, so switching to dyn filter changes little in practice |
| `ORDER BY end_time DESC LIMIT 10` | ~`28.9s` | ~`0.21s` | `end_time` is not the time-index column, and the old unoptimized path effectively scanned the whole table; after switching to dynamic filtering, later scan work can be pruned much earlier |
| `ORDER BY start_time DESC LIMIT 10` | `0.334s` / `0.336s` / `0.342s` | `0.336s` / `0.340s` / `0.336s` | `start_time` is the time-index column; window sort was already fast in this case, so switching to dynamic filtering changes little in practice |
In practice:
- `end_time`-style queries on non-time-index sort keys were slow mainly because the old path scanned the whole table; dyn filter lets TopK's boundary prune later scan work.
- Both `start_time` and `end_time` TopK queries now go through dyn filter. The older window sort path was more complex and more dependent on time-index data distribution; dyn filter replaces it with a more unified execution path.
- `end_time`-style queries on non-time-index sort keys were slow mainly because the old path scanned the whole table; dynamic filtering lets TopK's boundary prune later scan work.
- Both `start_time` and `end_time` TopK queries now go through dynamic filtering. The older window sort path was more complex and more dependent on time-index data distribution; dynamic filtering replaces it with a more unified execution path.
This optimization changes scan volume. The exact speedup still depends on data distribution, row-group statistics, and the size of `LIMIT`.
## Which queries benefit most
## Which queries benefit the most
dyn filter is strongest when all of the following line up:
Dynamic filtering is strongest when all of the following line up:
- `k` is small;
- row-group statistics are selective enough to make pruning useful;
@@ -114,15 +114,17 @@ It is much less dramatic when:
If scan cost dominates and the data distribution is suitable, this path can significantly reduce query cost.
## Current scope
The dyn filter path discussed here is, for now, mainly the local TopK path. Remote dynamic-filter propagation in distributed queries—especially the remote propagation of join-related dynamic filters—is still under active development; for the current design status, see [PR #7931](https://github.com/GreptimeTeam/greptimedb/pull/7931).
## What to check on your own queries
For now, the dynamic-filtering path discussed here is mainly the local TopK path. Remote dynamic-filter propagation in distributed queries—especially for join-related dynamic filters—is still under active development; for the current design status, see the [remote dyn filter RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/remote-dyn-filter-rfc.md).
If you have an `ORDER BY ... LIMIT k` query on a non-indexed sort key that is still slow, start with these two signals in the plan:
- a filter on the TopK node;
- propagated `dyn_filters` on the scan node.
If both are present, the query is probably on the dyn filter path.
If both are present, the query is probably on the dynamic-filtering path.
If neither shows up, first check whether the query is even on a path that supports dynamic filtering today.
The key change is not that the result only returns a few rows. It is that the scan can stop much earlier. That is where dynamic filtering changes the cost of TopK queries.

View File

@@ -1,10 +1,10 @@
# GreptimeDB 如何用动态过滤加速 TopK 查询
# 从几十秒到亚秒级:GreptimeDB 如何加速 TopK 查询
`ORDER BY ... LIMIT 10` 看起来只返回几行,很多时候却不便宜。表一大、排序键又不是时间索引时,数据库往往还是得先读很多数据,才能确定最终该返回哪 10 行。
GreptimeDB 在这条路径上的做法,是把 TopK 在执行过程中形成的运行时边界尽早交给扫描侧。后面的数据如果已经不可能进入最终结果,就不必继续读。在一组真实 traces 数据上,`ORDER BY end_time DESC LIMIT 10` 因此从几十秒降到了亚秒级。
## 为什么 `LIMIT 10` 过去可能很慢
## 为什么 `ORDER BY ... LIMIT` 仍然可能很慢
先看一个典型查询:
@@ -17,9 +17,9 @@ LIMIT 10;
在旧路径里,扫描侧看不到 TopK 在执行过程中的状态,只能一直读到查询结束。即使最后只返回 10 行,前面也还是可能要扫大量数据;等查询跑完之后,系统才知道其中很多数据其实根本没必要读。
这里还有一个背景:`start_time` 是这张表的时间索引列,`end_time` 不是。更早有一套名为 window sort 的优化,逻辑更复杂,也更依赖 time index 的数据分布。现在这两类 TopK 查询都统一走 dyn filter 这条路径。
这里还有一个背景:`start_time` 是这张表的时间索引列,`end_time` 不是。更早有一套名为 window sort 的优化,逻辑更复杂,也更依赖 time index 的数据分布。现在这两类 TopK 查询都统一走动态过滤(dyn filter这条路径。
## dyn filter 改变了什么
## 动态过滤怎样改变扫描路径
执行路径现在是这样的:
@@ -30,7 +30,7 @@ LIMIT 10;
`ORDER BY end_time DESC LIMIT 10` 这种查询,收益主要来自扫描范围更早收窄,而不是结果集更小。
## 运行时阈值如何参与剪枝
## 运行时阈值怎样变成剪枝条件
当 TopK 已经形成一个有意义的边界时GreptimeDB 会把这个边界作为动态过滤条件的一部分交给扫描侧。
@@ -49,7 +49,7 @@ flowchart TD
C --> D[已经不可能命中的数据被跳过]
```
## 执行计划里能看到什么
## 怎样从执行计划里确认它生效了
`EXPLAIN ANALYZE VERBOSE` 可以直接看到这条路径是否生效。
@@ -72,9 +72,9 @@ UnorderedScan: ..., "dyn_filters":
- `SortExec: TopK(..., filter=[...])`TopK 已经形成了运行时阈值;
- 扫描节点里的 `dyn_filters`:扫描侧收到了这个阈值,并开始据此剪枝。
如果执行计划里这两个信号都在,通常就说明 dyn filter 已经走通了。
如果执行计划里这两个信号都在,通常就说明动态过滤已经走通了。
## 实际能快多少
## 实际收益有多大
在这组 traces 数据上,最能体现变化的是下面这条查询:
@@ -87,19 +87,19 @@ LIMIT 10;
| Query | 之前 | 现在 | 说明 |
| --- | ---: | ---: | --- |
| `ORDER BY end_time DESC LIMIT 10` | ~`28.9s` | ~`0.21s` | `end_time` 不是时间索引列,旧的未优化路径基本会把表扫完;切到 dyn filter 之后,后续扫描可以更早被剪掉 |
| `ORDER BY start_time DESC LIMIT 10` | `0.334s` / `0.336s` / `0.342s` | `0.336s` / `0.340s` / `0.336s` | `start_time` 是时间索引列window sort 在这类场景里本来就已经很快,所以切到 dyn filter 后变化不大 |
| `ORDER BY end_time DESC LIMIT 10` | ~`28.9s` | ~`0.21s` | `end_time` 不是时间索引列,旧的未优化路径基本会把表扫完;切到动态过滤路径之后,后续扫描可以更早被剪掉 |
| `ORDER BY start_time DESC LIMIT 10` | `0.334s` / `0.336s` / `0.342s` | `0.336s` / `0.340s` / `0.336s` | `start_time` 是时间索引列window sort 在这类场景里本来就已经很快,所以切到动态过滤后变化不大 |
这里主要有两点:
- `end_time` 这类非时间索引排序查询,之前慢,主要是因为旧路径会扫描全表;切到 dyn filter 之后TopK 的边界可以直接拿来提前剪掉后续扫描。
- `start_time``end_time` 这两类 TopK 查询,现在都走 dyn filterwindow sort 那条更复杂、也更依赖 time index 数据分布的路径,已经被这条更统一的执行方式取代。
- `end_time` 这类非时间索引排序查询,之前慢,主要是因为旧路径会扫描全表;切到动态过滤之后TopK 的边界可以直接拿来提前剪掉后续扫描。
- `start_time``end_time` 这两类 TopK 查询,现在都走动态过滤window sort 那条更复杂、也更依赖 time index 数据分布的路径,已经被这条更统一的执行方式取代。
这项优化真正改变的是扫描量。具体能快多少还是取决于数据分布、row group 统计信息和 `LIMIT` 的大小。
## 哪些场景收益更大
## 哪些查询最容易受益
dyn filter 通常在下面这些条件同时满足时收益最明显:
动态过滤通常在下面这些条件同时满足时收益最明显:
- `k` 比较小;
- row group 统计信息有足够选择性;
@@ -114,11 +114,9 @@ dyn filter 通常在下面这些条件同时满足时收益最明显:
如果查询本身足够受扫描成本影响,数据形态也合适,这条路径就有机会明显改变查询成本。
## 当前支持范围
## 自己排查时先看什么
这里讨论的 dyn filter,目前主要还是本地 TopK 这条路径。分布式查询里的远程 dyn filter 传播,尤其是 join 相关 dyn filter 的远程传播,还在开发中;具体可以参考 [PR #7931](https://github.com/GreptimeTeam/greptimedb/pull/7931)。
## 自己排查时可以先看什么
这里讨论的动态过滤,目前主要还是本地 TopK 这条路径。分布式查询里的远程动态过滤传播,尤其是 join 相关场景,还在开发中;具体可以参考 [remote dyn filter RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/remote-dyn-filter-rfc.md)。
如果你手头正好有 `ORDER BY ... LIMIT k`(排序键不是索引列)仍然偏慢的查询,先看执行计划里的两个信号:
@@ -126,3 +124,7 @@ dyn filter 通常在下面这些条件同时满足时收益最明显:
- 扫描节点里有没有传播下来的 `dyn_filters`
这两个信号如果都在,通常就说明这条优化路径已经生效了。
如果这两个信号都不在,那就要先确认这条查询是不是还没走到当前支持的动态过滤路径上。
对这类 TopK 查询来说,真正关键的变化不是结果只返回几行,而是后面的扫描能不能更早停下来。这正是动态过滤带来的价值。