mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs
This commit is contained in:
Dennis Zhuang
Generated
+2
@@ -4559,6 +4559,7 @@ dependencies = [
|
||||
"jsonb",
|
||||
"num",
|
||||
"num-traits",
|
||||
"once_cell",
|
||||
"ordered-float 4.6.0",
|
||||
"parquet-variant",
|
||||
"parquet-variant-compute",
|
||||
@@ -8763,6 +8764,7 @@ dependencies = [
|
||||
"tokio-util",
|
||||
"toml 0.8.23",
|
||||
"tracing",
|
||||
"twox-hash",
|
||||
"uuid",
|
||||
"winner_tree",
|
||||
"zstd",
|
||||
|
||||
@@ -0,0 +1,604 @@
|
||||
# Mito columnar series format
|
||||
|
||||
**Status: implemented as an opt-in internal backend.** The byte assignments
|
||||
below are the version 1 compatibility contract. This document describes series-index artifacts only;
|
||||
sample files and query-consumer integration are separate work.
|
||||
|
||||
The presentation follows the field tables and diagrams in the
|
||||
[Prometheus chunks format documentation](https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/chunks.md).
|
||||
This is an independent format, not a Prometheus-compatible file.
|
||||
|
||||
The format sections follow storage order: artifact manifest, then each part's
|
||||
header, column payloads, metadata pages, and trailer. Writer/reader behavior and
|
||||
validation follow the layout specification.
|
||||
|
||||
## Artifact manifest
|
||||
|
||||
The catalog format identifier is `columnar-series`. Its wire version is separate
|
||||
from that identifier and from all column-encoding and compression identifiers.
|
||||
Legacy catalog entries and the `parquet` identifier keep their existing meaning.
|
||||
|
||||
One immutable artifact owns the following objects under a unique generation:
|
||||
|
||||
```text
|
||||
<region_id>/series/<index_uuid>/manifest.json
|
||||
<region_id>/series/<index_uuid>/part-0.series
|
||||
<region_id>/series/<index_uuid>/part-1.series
|
||||
...
|
||||
```
|
||||
|
||||
`manifest.json` contains these fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `format` | String `columnar-series` |
|
||||
| `version` | Integer 1 |
|
||||
| `region_id` | Physical region ID |
|
||||
| `generation` | Index UUID, using the existing `FileId` serialization |
|
||||
| `tags` | Ordered objects with `column_id` and `name`; all are nullable UTF-8 |
|
||||
| `coverage` | Existing `SeriesIndexEntry` representation, with `artifact` omitted and `file_size` zero |
|
||||
| `files` | Ordered objects with `part_id`, `file_size`, `num_series`, `first`, and `last` |
|
||||
|
||||
`part_id` is an unsigned 32-bit integer, starting at zero and increasing by one
|
||||
in file order. The object name is derived as `part-<part_id>.series`, using the
|
||||
decimal ID without leading zeros, under the artifact's generation prefix. No
|
||||
path is stored in a file entry. `first` and `last` are inclusive identifiers,
|
||||
represented as objects with `table_id` and `tsid`. Ranges are strictly ordered
|
||||
and disjoint. Coverage is captured from
|
||||
the build request; source SST IDs remain diagnostic information, not a reuse
|
||||
criterion. Validate coverage and generation against the catalog handle on open.
|
||||
|
||||
The tag list records the schema at build time, including tags currently absent
|
||||
from some logical tables. The logical Arrow schema is `__table_id`, `__tsid`,
|
||||
`__series_row_count`, followed by stored tags in manifest order. It has no
|
||||
timestamp columns and is separate from the existing Parquet index schema.
|
||||
|
||||
The manifest is written only after every part is closed successfully. The
|
||||
existing catalog publication commits the artifact. `ArtifactMetadata.objects`
|
||||
lists the manifest and every part as `{ path, file_size }` objects, and the
|
||||
artifact-level `file_size` is their total stored size. Object sizes are recorded
|
||||
by the writer; the manifest does not store its own size.
|
||||
These lifecycle object paths are derived from the generation and part IDs;
|
||||
artifact ownership and deferred purge behavior remain unchanged.
|
||||
The reader obtains part order from the manifest, not object-list ordering.
|
||||
An empty build has a manifest with an empty file list and creates no part files.
|
||||
|
||||
## Part file
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Header 16 bytes │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Independently encoded/compressed columns │
|
||||
│ Block 0: TSIDs, counts, tags │
|
||||
│ Block 1: ... │
|
||||
├──────────────────────────────────────────────┤ ← payload_end
|
||||
│ Leaf pages (one is the root at depth 1) │
|
||||
│ Optional internal root (depth 2) │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Trailer 96 bytes │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
All fixed-width integers are little-endian. Offsets are absolute from the start
|
||||
of the part; ranges are half-open. There is
|
||||
no alignment padding. A string is `length <u64>` followed by that many UTF-8
|
||||
bytes. Parse fields explicitly; Rust struct layout, `usize`, and enum declaration
|
||||
order are not wire contracts. Reject trailing bytes in decoded structures.
|
||||
|
||||
### Header
|
||||
|
||||
| Field | Bytes | Value |
|
||||
| --- | ---: | --- |
|
||||
| Magic | 8 | ASCII `GTSERIES` |
|
||||
| File version | 4 | 1 |
|
||||
| Reserved | 4 | Zero |
|
||||
|
||||
### Column payloads
|
||||
|
||||
An artifact indexes the series in one Mito series-index bucket. Authoritative
|
||||
data remains in Parquet SSTs; these indexes can be rebuilt. Each record contains:
|
||||
|
||||
| Logical column | Type | Storage |
|
||||
| --- | --- | --- |
|
||||
| `__table_id` | UInt32 | One constant in each block descriptor |
|
||||
| `__tsid` | UInt64 | Independently encoded column chunk |
|
||||
| `__series_row_count` | UInt64 | Independently encoded column chunk |
|
||||
| Tag columns | Nullable UTF-8 | Independent dictionary/code chunks |
|
||||
|
||||
Group records from one table into a block, using a writer row limit that defaults
|
||||
to 8,192. This is a packing choice, not a wire-format limit: every block records
|
||||
its actual number of series. A writer can use another limit without changing
|
||||
the file version, provided the encoded counts and payload sizes remain supported.
|
||||
Readers must not assume that blocks contain 8,192 rows. Write each stored column
|
||||
as an independent encoded and compressed chunk, then continue with the next
|
||||
block. Chunks have no inline header: their locations, lengths, codecs, and
|
||||
checksums are recorded in the later [metadata pages](#metadata-pages).
|
||||
|
||||
`(__table_id, __tsid)` is the complete identifier **under the current uniqueness
|
||||
assumption**. Equal TSIDs in different tables are distinct. Records are ordered
|
||||
lexicographically by this pair and identifiers do not repeat after aggregation.
|
||||
There is no stored `__primary_key` column. The writer extracts identifiers and
|
||||
labels from input sparse primary keys without retaining their bytes on disk.
|
||||
|
||||
Repeated input rows for a series, including rows split across input batches,
|
||||
contribute to its row count. Tags are constant for that identifier under the
|
||||
uniqueness assumption. Aggregation keeps
|
||||
the current index writer's input visibility semantics; this format does not add
|
||||
sample deduplication or make the row count an exact query-visible count for
|
||||
update-mode data.
|
||||
|
||||
No per-series or per-block timestamps are stored, and the series schema has no
|
||||
timestamp unit. The catalog's bucket coverage allows coarse time pruning. A series selected
|
||||
from an overlapping bucket is only a candidate for a time-constrained query;
|
||||
SST/sample reads must apply the exact time filter. The format reader cannot prune
|
||||
individual series by time or accept filters on timestamp columns.
|
||||
|
||||
TODO: evaluate support for optional per-series minimum/maximum timestamps and
|
||||
benchmark their storage size, build cost, and time-pruning benefit before adding
|
||||
them to the format. Keep this version free of per-series and per-block time bounds.
|
||||
|
||||
TODO: benchmark storing exact input primary-key bytes and reconsider collision
|
||||
handling. Different label sets can hash to the same TSID within one table. This
|
||||
version neither preserves a collision discriminator nor promises collision-safe
|
||||
identity. Add the corresponding TODO at the implementation's identity boundary.
|
||||
|
||||
Dedicated `job_id` and `instance_id` identifier/locality components are deferred.
|
||||
Ordinary tags named `job` or `instance` remain ordinary tag columns.
|
||||
|
||||
#### Integer encoding 1
|
||||
|
||||
Start each chunk with `previous = 0`. For each value, compute wrapping u64
|
||||
subtraction from the previous value, reinterpret the result as i64,
|
||||
ZigZag-encode it, and write an unsigned base-128 varint. The previous value
|
||||
becomes the current value. Decoding
|
||||
uses wrapping addition after reversing ZigZag. This preserves all UInt64 values,
|
||||
including differences beyond the signed range.
|
||||
|
||||
A varint has seven low data bits per byte and the high continuation bit. It is
|
||||
at most ten bytes; the tenth byte can contain only bit zero and cannot continue.
|
||||
The writer uses the shortest representation. Decode exactly the block's number
|
||||
of series and reject truncation, overflow, and trailing bytes. TSID values must
|
||||
be strictly increasing and agree with the descriptor's first/last values.
|
||||
|
||||
#### Dictionary encoding 2
|
||||
|
||||
```text
|
||||
dictionary_byte_length <uvarint>
|
||||
dictionary:
|
||||
entry_count <u64>
|
||||
repeated entries:
|
||||
present <u8> # 0 = null, 1 = UTF-8 string
|
||||
string <length + bytes> # only if present = 1
|
||||
row_codes <one uvarint per series>
|
||||
```
|
||||
|
||||
Entry zero is null; remaining entries are distinct non-null strings in
|
||||
first-occurrence order. Null cannot occur at another entry. The empty string is
|
||||
a non-null entry with zero bytes. Dictionary length includes its count and entry
|
||||
framing. Codes must be in range; the dictionary and code streams must each be
|
||||
consumed exactly. Keep dictionary/code buffers during filtering rather than
|
||||
expanding one owned string per row.
|
||||
|
||||
TODO: implement and benchmark a plain UTF-8 encoding alternative for string
|
||||
chunks, particularly high-cardinality labels. Compare dictionary encoding with
|
||||
null bitmap + offsets + UTF-8 bytes, including Zstd-compressed size, build cost,
|
||||
and filtering/decoding cost. Evaluate per-chunk selection using estimated encoded
|
||||
size rather than cardinality alone. A plain encoding gets its own encoding ID;
|
||||
it can use the existing compression IDs and coexist with dictionary chunks.
|
||||
This port implements dictionary strings only; the alternative and automatic
|
||||
selection remain future work.
|
||||
|
||||
### Metadata pages
|
||||
|
||||
The metadata is a tree that maps identifier ranges to column locations:
|
||||
|
||||
- A **leaf page** describes data blocks and where their column chunks are stored.
|
||||
- An **internal page** points to other metadata pages, with an identifier range
|
||||
for each child so the reader can skip unrelated children.
|
||||
- The **root page** is the tree's entrypoint, referenced by the trailer. It is
|
||||
either a leaf or an internal page; there is no separate root page encoding.
|
||||
|
||||
This implementation supports only depths 1 and 2. It does not build or traverse
|
||||
internal pages below the root. The existing depth field and page references
|
||||
leave room for deeper trees as future work.
|
||||
|
||||
Each page is encoded and compressed independently. The diagrams below show its
|
||||
**decoded bytes**. The stored bytes are the compressed form of that entire page;
|
||||
the kind and entry count are inside the compressed payload. The parent's page
|
||||
reference supplies the location, lengths, checksum, and codecs needed to read it.
|
||||
For the root, that reference is in the trailer.
|
||||
|
||||
#### Leaf page
|
||||
|
||||
A leaf uses kind 0. Its entry count is the number of block descriptors, not the
|
||||
number of series. Block descriptors have variable length because their column
|
||||
names and label statistics have variable length.
|
||||
|
||||
```text
|
||||
┌───────────────────────────────────────────────────────┐
|
||||
│ kind = 0 u32 │
|
||||
│ entry_count = N u64 │
|
||||
├───────────────────────────────────────────────────────┤
|
||||
│ Block descriptor 0 │
|
||||
│ table_id u32 │
|
||||
│ first_tsid, last_tsid 2 × u64│
|
||||
│ num_series, num_columns 2 × u32│
|
||||
│ Column descriptor 0 │
|
||||
│ name string │
|
||||
│ payload reference 32 bytes │──→ column chunk
|
||||
│ null_count, statistics_kind 2 × u32│ in payload area
|
||||
│ optional label min/max strings │
|
||||
│ ... │
|
||||
│ Column descriptor num_columns - 1 │
|
||||
├───────────────────────────────────────────────────────┤
|
||||
│ ... │
|
||||
│ Block descriptor N - 1 │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The leaf contains locations and statistics, not the column values themselves.
|
||||
Its table ID supplies the constant `__table_id` value for each series in a block.
|
||||
The exact descriptor fields are specified under [Block descriptor](#block-descriptor).
|
||||
|
||||
#### Internal page
|
||||
|
||||
An internal page uses kind 1. Its entry count is the number of child metadata
|
||||
pages. Each entry is a fixed-size 56-byte page reference:
|
||||
|
||||
```text
|
||||
┌───────────────────────────────────────────────────────┐
|
||||
│ kind = 1 u32 │
|
||||
│ entry_count = N u64 │
|
||||
├───────────────────────────────────────────────────────┤
|
||||
│ Child page reference 0 56 bytes │
|
||||
│ first = (table_id, tsid) 12 bytes │
|
||||
│ last = (table_id, tsid) 12 bytes │
|
||||
│ payload reference 32 bytes │──→ child page
|
||||
├───────────────────────────────────────────────────────┤
|
||||
│ ... │
|
||||
│ Child page reference N - 1 56 bytes │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
An internal page's decoded size is `12 + N × 56` bytes. In this implementation,
|
||||
it is always the root and all its children are leaves. An internal page has no
|
||||
block descriptors or column data. At the 65,536-byte page limit, the root holds
|
||||
at most `floor((65536 - 12) / 56) = 1170` leaf references.
|
||||
|
||||
#### Root page and traversal example
|
||||
|
||||
If all block descriptors fit in one leaf, that leaf is also the root and the
|
||||
trailer records depth 1. Otherwise the root is an internal page and the trailer
|
||||
records depth 2: root → leaves. Depth 3 would mean root → internal pages → leaves,
|
||||
but is not implemented in this port.
|
||||
|
||||
For example, four data blocks can be indexed by two leaves and an internal root.
|
||||
The ranges below are inclusive `(table_id, tsid)` bounds; they do not imply that
|
||||
every ID between the endpoints exists.
|
||||
|
||||
```text
|
||||
Trailer: depth = 2
|
||||
root reference: [(7,10), (7,49)]
|
||||
│
|
||||
▼
|
||||
Root R: internal page, kind = 1, entry_count = 2
|
||||
├─ reference [(7,10), (7,29)] ──→ Leaf A: kind = 0, entry_count = 2
|
||||
│ ├─ Block 0: table 7, TSIDs [10,19]
|
||||
│ └─ Block 1: table 7, TSIDs [20,29]
|
||||
└─ reference [(7,30), (7,49)] ──→ Leaf B: kind = 0, entry_count = 2
|
||||
├─ Block 2: table 7, TSIDs [30,39]
|
||||
└─ Block 3: table 7, TSIDs [40,49]
|
||||
|
||||
Physical order, from payload_end to end of part:
|
||||
[compressed Leaf A] [compressed Leaf B] [compressed Root R] [Trailer]
|
||||
```
|
||||
|
||||
To find `(7,35)`, read the trailer and Root R, then follow only the reference to
|
||||
Leaf B. Its Block 2 descriptor locates the TSID and requested label/count chunks
|
||||
in the earlier column payload area. Decode the TSID chunk to find the exact row;
|
||||
metadata bounds only identify a candidate block. Leaf A and Block 3's payloads
|
||||
are not needed.
|
||||
|
||||
The root's own bounds and storage reference live in the trailer; its page bytes
|
||||
have exactly the leaf or internal layout above. At depth 2 the writer emits all
|
||||
leaves and then the internal root. At depth 1 it emits only the single leaf.
|
||||
|
||||
TODO: evaluate depth 3+ if larger parts or metadata volumes justify it. The
|
||||
current page-reference representation can point to another internal page, so
|
||||
the extension need not change the byte layout. It would require new writer and
|
||||
reader traversal support, benchmarks, and a rollout policy for older readers.
|
||||
Current readers reject depths other than 1 or 2 explicitly before traversal;
|
||||
they must never interpret unsupported trees as empty indexes. Future writers
|
||||
must not assume that an unchanged file version implies old-reader support for
|
||||
new depths. Older files remain readable by a future deeper-tree implementation.
|
||||
|
||||
#### Page ordering and size limits
|
||||
|
||||
Pages are nonempty. A leaf's bounds are its first/last block bounds; an internal
|
||||
page's bounds are its first/last child bounds. Child key ranges are strictly
|
||||
ordered and disjoint, as are block ranges. Children physically precede their
|
||||
parent and their stored ranges do not overlap. Columns within a leaf are likewise
|
||||
physically ordered and non-overlapping.
|
||||
|
||||
Pages target 65,536 decoded bytes including the 12-byte prefix. A descriptor
|
||||
never straddles pages. A descriptor larger than the target gets its own leaf,
|
||||
capped at 8 MiB including framing; a larger descriptor fails the build. Internal
|
||||
pages never exceed 65,536 decoded bytes. Stored pages are capped at 8 MiB plus
|
||||
64 KiB. Readers validate sizes and counts before allocating or decompressing.
|
||||
|
||||
#### Payload reference
|
||||
|
||||
Every column chunk and metadata page has this 32-byte reference:
|
||||
|
||||
| Field | Bytes | Meaning |
|
||||
| --- | ---: | --- |
|
||||
| Offset | 8 | Absolute location of stored bytes |
|
||||
| Stored length | 4 | Bytes to fetch |
|
||||
| Decoded length | 4 | Bytes after decompression, before decoding |
|
||||
| Checksum | 8 | XXH64 of stored bytes, seed zero |
|
||||
| Encoding ID | 2 | Column or metadata encoding, according to context |
|
||||
| Compression ID | 2 | Compression of the encoded payload |
|
||||
| Reserved | 4 | Zero |
|
||||
|
||||
The offset is u64, so a payload can start beyond the 4 GiB boundary. The two u32
|
||||
lengths limit a single stored or decoded payload to at most `2^32 - 1` bytes;
|
||||
they do not limit the whole part to 4 GiB. File length and `payload_end` are also
|
||||
u64. This version keeps u32 payload lengths because chunks and pages have much
|
||||
smaller bounded sizes. Check `offset + stored_length` in u64 with overflow
|
||||
handling, and validate decoded sizes before converting to allocation sizes.
|
||||
|
||||
The checksum is verified before decompression. Decompression must produce exactly
|
||||
the declared length. Encoding and compression are independent: encode first,
|
||||
compress second. Compression parameters needed for decoding must be in the codec
|
||||
frame; the Zstd level is a writer choice, not a decoding requirement.
|
||||
|
||||
#### Page reference
|
||||
|
||||
A page reference is 56 bytes: `first table ID <u32>`, `first TSID <u64>`,
|
||||
`last table ID <u32>`, `last TSID <u64>`, then a 32-byte payload reference.
|
||||
Bounds are inclusive. Compare identifiers as integer pairs, not by the byte
|
||||
ordering of their little-endian representation.
|
||||
|
||||
#### Block descriptor
|
||||
|
||||
A block contains series from one table; its actual row count is recorded below.
|
||||
The writer defaults to 8,192 series per block. A table boundary ends the current
|
||||
block without necessarily ending the part. The decoded block descriptor has the
|
||||
following fields in order:
|
||||
|
||||
| Field | Encoding |
|
||||
| --- | --- |
|
||||
| Table ID | u32 |
|
||||
| First and last TSID | u64 each, inclusive |
|
||||
| Number of series | u32, nonzero |
|
||||
| Number of stored columns | u32 |
|
||||
| Column descriptors | Repeated in logical-schema order, excluding table ID |
|
||||
|
||||
Each column descriptor contains `name <string>`, `payload <32-byte reference>`,
|
||||
`null_count <u32>`, and `statistics_kind <u32>`. Statistics kind 0 has no extra
|
||||
fields; kind 1 adds minimum and maximum non-null strings, in that order. Only
|
||||
tag columns use kind 1. String bounds use UTF-8 byte lexicographic order.
|
||||
|
||||
TSID and row count are mandatory, non-null columns. An all-null tag column can
|
||||
be omitted; its logical values are all null
|
||||
and its inferred null count equals the number of series. A present tag chunk
|
||||
has string bounds if it contains any non-null values. Null and the empty string
|
||||
are distinct. Names must be unique and belong to the manifest schema.
|
||||
|
||||
TODO: benchmark replacing repeated column names with numeric references to a
|
||||
shared schema, including metadata size, decode cost, and selective-read latency.
|
||||
The manifest already stores the tag schema; leaf descriptors currently repeat
|
||||
names to identify their chunks. Keep the current descriptors for this port.
|
||||
Names can be deduplicated later, but payload locations, codecs, null counts, and
|
||||
statistics remain specific to each block's column chunk.
|
||||
|
||||
### Trailer
|
||||
|
||||
| Field | Bytes | Meaning |
|
||||
| --- | ---: | --- |
|
||||
| Root page reference | 56 | Root bounds and payload location/codecs |
|
||||
| Index depth | 4 | Leaf-only index = 1 |
|
||||
| File version | 4 | 1 |
|
||||
| File length | 8 | Includes header and trailer |
|
||||
| `payload_end` | 8 | End of column area, start of metadata area |
|
||||
| Magic | 8 | ASCII `GTSERIES` |
|
||||
| Trailer checksum | 8 | XXH64 of the preceding 88 trailer bytes, seed zero |
|
||||
|
||||
The root ends immediately before the trailer. Supported depths are 1 and 2. Page ranges
|
||||
lie in the metadata area; column ranges lie between the header and `payload_end`.
|
||||
Checksums detect accidental corruption; they do not authenticate data.
|
||||
|
||||
TODO: evaluate and benchmark a variable-length Protobuf footer located by a
|
||||
small fixed trailer, so optional file metadata can evolve without enlarging the
|
||||
fixed trailer. Measure footer size, open/read cost, and compatibility behavior
|
||||
before adopting it. This port retains the 96-byte trailer above; no Protobuf
|
||||
footer or schema-reference changes are included.
|
||||
|
||||
## Codec selection and evolution
|
||||
|
||||
Encoding IDs have separate column and metadata namespaces. IDs are explicitly
|
||||
assigned and never reused. Version 1 supports:
|
||||
|
||||
| Namespace | ID | Meaning |
|
||||
| --- | ---: | --- |
|
||||
| Column encoding | 1 | Wrapping delta, ZigZag, unsigned varint integer stream |
|
||||
| Column encoding | 2 | Nullable UTF-8 dictionary and unsigned varint codes |
|
||||
| Metadata encoding | 1 | Block descriptors / page references defined above |
|
||||
| Compression | 0 | None; stored length equals decoded length |
|
||||
| Compression | 1 | Zstd frame; writer default is level 3 |
|
||||
|
||||
The default writer uses compression 1 for columns and metadata. Compression 0
|
||||
provides an explicit uncompressed representation and a second dispatch path to
|
||||
test. Codec selection is internal; this change adds no public codec configuration.
|
||||
|
||||
Changing the writer's block row limit (for example, from 8,192 to 16,384) requires
|
||||
no codec ID, trailer extension, or file-version change: the block descriptor
|
||||
already supplies the actual row count. The same applies to part rollover choices.
|
||||
These choices remain subject to field-width and reader resource limits. They
|
||||
are distinct from changing the serialized structure of a descriptor or trailer.
|
||||
|
||||
A new encoding gets a new ID and its own documented payload interpretation;
|
||||
it can coexist with older encodings in the same file. A codec can define its
|
||||
own payload prefix for parameters. A new compression method likewise receives
|
||||
a new ID. Neither change requires rewriting the outer layout or changing the
|
||||
format API. Changing the outer structures or the meaning of an existing field
|
||||
requires a new file version. Reserved fields must remain zero in version 1.
|
||||
|
||||
Unknown versions, reserved bits, or metadata codecs fail open/read explicitly.
|
||||
An unknown column codec may be skipped when its column is neither projected nor
|
||||
needed for filtering; attempting to read it returns an unsupported-codec error.
|
||||
It must never be interpreted as null or silently decoded with a default codec.
|
||||
|
||||
## Writer and resource bounds
|
||||
|
||||
The implementation lives behind `SeriesIndexFormat::writer` and `FormatWriter`.
|
||||
Its input remains sorted flat `RecordBatch` data. Reuse the existing writer's
|
||||
sparse-primary-key and string-tag validation, but do not depend on its timestamp
|
||||
aggregation or timestamp-bearing output schema. This writer consumes primary
|
||||
keys and counts input rows without decoding timestamp values. Preserve
|
||||
aggregation across `write` calls; reject decreasing identifiers and checked-count
|
||||
overflow.
|
||||
|
||||
Write column payloads as blocks complete, retaining only the current series,
|
||||
current block, and bounded per-part descriptors. Finish a part at block boundaries
|
||||
when the stored position reaches 512 MiB or cumulative serialized descriptors
|
||||
reach 8 MiB. These are soft limits: the completing block may overshoot them.
|
||||
The root's leaf-reference capacity is a separate hard packing limit: account for
|
||||
the next block descriptor's leaf-page placement before writing that block. If
|
||||
it would require more than 1,170 leaves, finish the current part and write the
|
||||
block to a new part. Never create a third metadata level. Emit leaf pages, the
|
||||
internal root only when needed, and then the trailer.
|
||||
|
||||
For bounded decoding, an individual column chunk is capped at 64 MiB decoded and
|
||||
64 MiB plus 1 MiB stored. Oversized chunks fail the build; no partial artifact is
|
||||
published. A block contains at most 8,388,608 series, bounding each decoded u64
|
||||
column and row-selection buffer to 64 MiB. Bound the manifest to 64 MiB before parsing. These safety limits are
|
||||
part of version 1, not new server configuration. Check offsets, lengths, counts,
|
||||
and arithmetic before allocation; row counts and lengths alone must not authorize
|
||||
allocations larger than the enclosing decoded data permits.
|
||||
|
||||
## OpenDAL I/O and lifecycle
|
||||
|
||||
Use `object_store::ObjectStore` (the OpenDAL operator) and its `Reader`/`Writer`
|
||||
types, as other Mito index implementations do. Do not add a parallel storage trait
|
||||
or a native filesystem adapter. Memory and filesystem OpenDAL services and the
|
||||
existing mock layers provide test backends.
|
||||
|
||||
- Use the artifact metadata's recorded object sizes to read the manifest and
|
||||
locate part trailers without stat requests. Validate part sizes against the
|
||||
manifest and reject duplicate object paths or inconsistent totals. Fetch headers,
|
||||
trailers, metadata pages, and selected column ranges through asynchronous range reads.
|
||||
Reuse `fetch_byte_ranges` where applicable. Its range coalescing can fetch gaps
|
||||
between selected chunks; those bytes do not cause unselected columns to decode.
|
||||
- Open outputs through `writer_with`, using `DEFAULT_WRITE_BUFFER_SIZE` and
|
||||
`DEFAULT_WRITE_CONCURRENCY`. Stream bytes without buffering an entire part.
|
||||
Call `close` before reporting success; count all committed object bytes.
|
||||
- Use existing `OpenDalSnafu` error conversion. Keep codec work separate from I/O
|
||||
and run codecs inline, matching the format benchmark. TODO: compare inline
|
||||
and offloaded codecs for latency, concurrent throughput, and async-worker
|
||||
responsiveness before choosing an offloading policy. No comparative benchmark
|
||||
currently establishes a performance advantage for either policy.
|
||||
- Until successful finish, the writer owns every output in its generation.
|
||||
Failure or explicit abort aborts live OpenDAL writers, removes completed
|
||||
unpublished objects, and aborts each live OpenDAL writer to clean its own atomic-write temporary
|
||||
file. Cleanup never scans temporary files by generic part basenames. A cancellation/drop guard schedules the same cleanup
|
||||
on the shared runtime. Cleanup errors are reported without hiding the original
|
||||
operation error. Process-crash orphan recovery remains the existing lifecycle's
|
||||
responsibility; an incomplete generation is never published as valid.
|
||||
- After successful finish, ownership transfers to the build/catalog lifecycle.
|
||||
Reuse existing publication-failure cleanup and deferred retirement. A reader
|
||||
and each returned stream retain a `SeriesIndexFileHandle` until dropped.
|
||||
|
||||
## Reader API and execution
|
||||
|
||||
Add a reader factory to `SeriesIndexFormat`, accepting the store and pinned
|
||||
artifact handle, and returning a format reader. The reader exposes its stored
|
||||
Arrow schema and a `read` method returning a stream of projected record batches.
|
||||
Reads have independent traversal state and may outlive the reader factory.
|
||||
|
||||
The initial read request contains an ordered projection of column names, an
|
||||
optional table ID, an optional set of `MetricSeriesId` values, and a conjunction
|
||||
of `SimpleFilterEvaluator` filters. No ID restriction means all IDs; an explicitly
|
||||
empty set means no rows. Reject unknown projection/filter columns instead of
|
||||
guessing their meaning. An empty projection still preserves selected row counts.
|
||||
Returned rows preserve identifier order, even when IDs are not projected.
|
||||
|
||||
The reader accepts filters only on its stored logical columns; it has no time
|
||||
filter. Bucket-level time selection belongs to the caller, and exact time
|
||||
filtering belongs to subsequent SST/sample reads. This API evaluates supplied
|
||||
filters exactly; conversion of arbitrary query expressions or schema-evolution predicates into
|
||||
conservative index candidates remains the future query adapter's responsibility.
|
||||
|
||||
1. Open and validate the manifest and selected parts' framing. Use file bounds
|
||||
to skip irrelevant parts and page bounds to traverse only relevant metadata.
|
||||
2. Apply conservative block pruning using identifier and label bounds.
|
||||
Missing statistics never exclude possible matches. Label filters do not prune
|
||||
internal pages. Full scans need not load the entire metadata tree at once.
|
||||
3. Decode IDs when needed for selection and predicate columns before output
|
||||
columns. Apply existing Mito filter/null semantics, including evaluation on
|
||||
tag dictionaries where possible. Null is not converted to the empty string.
|
||||
4. Fetch and decode remaining projected columns only for blocks with surviving
|
||||
rows; each selected chunk is still decompressed as a whole. Materialize output
|
||||
only for selected rows. Synthesize table IDs from the block constant and
|
||||
omitted tags as null arrays. Reuse already decoded filter columns. Output
|
||||
batches target 8 MiB of materialized values (one larger row is allowed), so
|
||||
repeated dictionary values do not expand an entire block at once. This is a
|
||||
reader packing choice and does not change the wire format.
|
||||
|
||||
Keep page traversal state to the active leaf and pending references. There is no
|
||||
cross-read cache in this implementation. Centralize fetch/validate/decode work in
|
||||
`load_metadata_page` and `load_column_chunk` entrypoints.
|
||||
|
||||
TODO: integrate these entrypoints with Mito's existing cache system. Future keys
|
||||
need region/store identity, immutable artifact generation, part ID, byte range,
|
||||
checksum, codec IDs, and representation. Keep that information available; do not
|
||||
introduce a new cache framework, global cache, configuration, or unused parameter.
|
||||
|
||||
Ordinary series scans continue using the existing Parquet path and fall back to
|
||||
SST discovery for this format. Merely adding a format reader must not mark these
|
||||
artifacts as supported query coverage. Samples remain disabled by default.
|
||||
|
||||
## Compatibility and validation
|
||||
|
||||
Existing Parquet files and catalogs remain readable. The new backend is opt-in
|
||||
through the internal format selection; defaults do not write this format. An
|
||||
older binary need not decode new codec bytes. These rebuildable artifacts must
|
||||
not become necessary for reading authoritative SST data. There is no promise
|
||||
that old binaries understand the new catalog identifier.
|
||||
|
||||
Validation covers the layouts, identifier assumption, codec IDs, schema handling,
|
||||
and safety limits above. Implementation acceptance includes:
|
||||
|
||||
- Codec unit tests and writer/reader round trips. Dedicated compatibility tests
|
||||
and frozen format fixtures are outside this PR's scope.
|
||||
- Empty input, repeated series across batches, mixed tables/schemas, equal TSIDs
|
||||
across tables, nullable/empty tags, forced rollover, depths 1 and 2, and
|
||||
oversized payloads/descriptors.
|
||||
- Root-capacity rollover before a third level is needed, using reduced packing
|
||||
limits in tests. Reject depth 0 and unsupported depths including 3 before
|
||||
traversal, and reject an internal child where a leaf is required at depth 2.
|
||||
- Exact projection/filter results, no irrelevant payload decoding, selective
|
||||
range reads, and independent readers/streams with caching absent.
|
||||
- Different block row limits, including 16,384, read by the same format reader
|
||||
without a version change. Validate references beyond 4 GiB with small mocked
|
||||
payloads, plus overflowing offsets and oversized individual payload lengths.
|
||||
- Part-ID-to-path derivation, invalid/misordered part IDs, absence of timestamp
|
||||
columns, and rejection of timestamp filters. Varying input timestamps must not
|
||||
change encoded series records when identifiers, labels, and row counts match.
|
||||
- Truncation, bad checksums, invalid offsets/counts/order, dictionary corruption,
|
||||
unsupported codec/version IDs, and bounded decompression/decoding failures.
|
||||
- Real-format reconciliation, reopen, publication failures, OpenDAL write/close
|
||||
failures, abort/cancellation cleanup, accounting, and retirement with pinned
|
||||
streams. Failed builds must not replace a valid catalog entry.
|
||||
- `cargo nextest run -p mito2`, crate Clippy, formatting, and license checks for
|
||||
new source files. No SQL query behavior change is claimed by this phase.
|
||||
|
||||
Performance must be measured again after the port: the row-count column, checksums,
|
||||
metadata framing, OpenDAL range coalescing, and the absence of decoded caches
|
||||
change the measured workload. Future experiments also cover exact primary keys,
|
||||
dedicated job/instance locality components, and optional per-series time bounds.
|
||||
Shared-schema column references and a Protobuf footer require separate benchmarks
|
||||
and remain outside this port.
|
||||
Depth 3+ metadata is also deferred until larger parts justify the added traversal.
|
||||
No performance equivalence is asserted by this document.
|
||||
@@ -26,6 +26,7 @@ enum_dispatch = "0.3"
|
||||
jsonb.workspace = true
|
||||
num = "0.4"
|
||||
num-traits = "0.2"
|
||||
once_cell.workspace = true
|
||||
ordered-float.workspace = true
|
||||
parquet-variant.workspace = true
|
||||
parquet-variant-compute.workspace = true
|
||||
|
||||
@@ -16,6 +16,7 @@ use std::borrow::Cow;
|
||||
use std::sync::Arc;
|
||||
|
||||
use arrow::buffer::ScalarBuffer;
|
||||
use once_cell::sync::OnceCell;
|
||||
use snafu::OptionExt;
|
||||
|
||||
use crate::error::{InvalidSeriesSnafu, Result};
|
||||
@@ -77,31 +78,27 @@ impl GridSpec {
|
||||
}
|
||||
}
|
||||
|
||||
/// A materialized, shareable timestamp grid.
|
||||
/// A lazily materialized, shareable timestamp grid.
|
||||
#[derive(Debug)]
|
||||
pub struct EvalGrid {
|
||||
spec: GridSpec,
|
||||
timestamps: ScalarBuffer<i64>,
|
||||
timestamps: OnceCell<ScalarBuffer<i64>>,
|
||||
}
|
||||
|
||||
impl EvalGrid {
|
||||
/// Allocates the grid timestamps once for sharing across result series.
|
||||
pub fn try_new(spec: GridSpec) -> Result<Self> {
|
||||
GridSpec::try_new(spec.start_ms, spec.step_ms, spec.len)?;
|
||||
let len =
|
||||
usize::try_from(spec.len).map_err(|_| invalid_series("grid length overflows usize"))?;
|
||||
let mut values = Vec::new();
|
||||
values
|
||||
.try_reserve_exact(len)
|
||||
.map_err(|_| invalid_series("grid timestamp allocation failed"))?;
|
||||
for index in 0..len {
|
||||
// The validated endpoint bounds every point in this positive-step grid.
|
||||
let timestamp = (spec.start_ms as i128) + (index as i128) * (spec.step_ms as i128);
|
||||
values.push(timestamp as i64);
|
||||
}
|
||||
let grid = Self::try_deferred(spec)?;
|
||||
grid.timestamps()?;
|
||||
Ok(grid)
|
||||
}
|
||||
|
||||
/// Creates a validated grid without allocating its timestamp array.
|
||||
pub fn try_deferred(spec: GridSpec) -> Result<Self> {
|
||||
let spec = GridSpec::try_new(spec.start_ms, spec.step_ms, spec.len)?;
|
||||
Ok(Self {
|
||||
spec,
|
||||
timestamps: values.into(),
|
||||
timestamps: OnceCell::new(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -110,9 +107,32 @@ impl EvalGrid {
|
||||
self.spec
|
||||
}
|
||||
|
||||
/// Returns the shared timestamp buffer.
|
||||
pub fn timestamps(&self) -> &ScalarBuffer<i64> {
|
||||
&self.timestamps
|
||||
/// Returns the shared timestamp buffer, materializing it on first access.
|
||||
pub fn timestamps(&self) -> Result<&ScalarBuffer<i64>> {
|
||||
self.timestamps.get_or_try_init(|| {
|
||||
let len = usize::try_from(self.spec.len)
|
||||
.map_err(|_| invalid_series("grid length overflows usize"))?;
|
||||
let mut values = Vec::new();
|
||||
values
|
||||
.try_reserve_exact(len)
|
||||
.map_err(|_| invalid_series("grid timestamp allocation failed"))?;
|
||||
for index in 0..len {
|
||||
let timestamp =
|
||||
(self.spec.start_ms as i128) + (index as i128) * (self.spec.step_ms as i128);
|
||||
values.push(timestamp as i64);
|
||||
}
|
||||
Ok(values.into())
|
||||
})
|
||||
}
|
||||
|
||||
fn timestamp_at(&self, position: usize) -> Option<i64> {
|
||||
if position as u64 >= self.spec.len {
|
||||
return None;
|
||||
}
|
||||
let timestamp =
|
||||
(self.spec.start_ms as i128) + (position as i128) * (self.spec.step_ms as i128);
|
||||
// GridSpec validation bounds every position, and i128 holds usize * i64.
|
||||
Some(timestamp as i64)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,7 +150,10 @@ impl GridRange {
|
||||
let end = start.checked_add(len).context(InvalidSeriesSnafu {
|
||||
reason: "grid range overflow".to_string(),
|
||||
})?;
|
||||
if end > grid.timestamps.len() {
|
||||
if end
|
||||
> usize::try_from(grid.spec.len)
|
||||
.map_err(|_| invalid_series("grid length overflows usize"))?
|
||||
{
|
||||
return invalid("grid range exceeds its grid");
|
||||
}
|
||||
Ok(Self { grid, start, len })
|
||||
@@ -153,8 +176,8 @@ impl GridRange {
|
||||
self.len == 0
|
||||
}
|
||||
/// Returns this range as a zero-copy Arrow buffer slice.
|
||||
pub fn timestamps(&self) -> ScalarBuffer<i64> {
|
||||
self.grid.timestamps.slice(self.start, self.len)
|
||||
pub fn timestamps(&self) -> Result<ScalarBuffer<i64>> {
|
||||
Ok(self.grid.timestamps()?.slice(self.start, self.len))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -230,7 +253,8 @@ impl GridRanges {
|
||||
/// path expects, so callers outside this crate build axes through
|
||||
/// [`TimeAxis::from_grid_spans`], which keeps [`TimeAxis::Grid`] for a single span.
|
||||
pub(crate) fn try_new(grid: Arc<EvalGrid>, spans: Vec<GridSpan>) -> Result<Self> {
|
||||
let grid_len = grid.timestamps().len();
|
||||
let grid_len = usize::try_from(grid.spec.len)
|
||||
.map_err(|_| invalid_series("grid length overflows usize"))?;
|
||||
let mut rows = Vec::new();
|
||||
rows.try_reserve_exact(spans.len())
|
||||
.map_err(|_| invalid_series("grid span allocation failed"))?;
|
||||
@@ -584,11 +608,11 @@ impl TimeAxis {
|
||||
pub fn timestamp(&self, index: usize) -> Option<i64> {
|
||||
match self {
|
||||
Self::Grid(range) => (index < range.len)
|
||||
.then(|| range.grid.timestamps.get(range.start + index).copied())
|
||||
.then(|| range.grid.timestamp_at(range.start.checked_add(index)?))
|
||||
.flatten(),
|
||||
Self::GridRanges(ranges) => {
|
||||
let (span, offset) = ranges.span_of(index)?;
|
||||
ranges.grid.timestamps.get(span.start + offset).copied()
|
||||
ranges.grid.timestamp_at(span.start.checked_add(offset)?)
|
||||
}
|
||||
Self::Explicit(values) => values.get(index).copied(),
|
||||
Self::ExplicitChunks(chunks) => {
|
||||
@@ -751,28 +775,29 @@ impl TimeAxis {
|
||||
///
|
||||
/// A single-run grid axis lends its buffer without copying; an axis of several runs, and a
|
||||
/// chunked explicit axis, allocate one array of the carried rows.
|
||||
pub fn timestamps(&self) -> Cow<'_, [i64]> {
|
||||
pub fn timestamps(&self) -> Result<Cow<'_, [i64]>> {
|
||||
match self {
|
||||
Self::Grid(range) => {
|
||||
Cow::Borrowed(&range.grid.timestamps()[range.start()..range.start() + range.len()])
|
||||
}
|
||||
Self::Grid(range) => Ok(Cow::Borrowed(
|
||||
&range.grid.timestamps()?[range.start()..range.start() + range.len()],
|
||||
)),
|
||||
Self::GridRanges(ranges) => {
|
||||
let mut timestamps = Vec::with_capacity(ranges.len());
|
||||
for (position, _) in self.positions() {
|
||||
match ranges.grid.timestamps.get(position) {
|
||||
Some(timestamp) => timestamps.push(*timestamp),
|
||||
// Unreachable: the constructor checks every span against the grid.
|
||||
None => break,
|
||||
}
|
||||
timestamps.push(
|
||||
ranges
|
||||
.grid
|
||||
.timestamp_at(position)
|
||||
.ok_or_else(|| invalid_series("grid span timestamp is out of range"))?,
|
||||
);
|
||||
}
|
||||
Cow::Owned(timestamps)
|
||||
Ok(Cow::Owned(timestamps))
|
||||
}
|
||||
Self::Explicit(values) => Cow::Borrowed(values),
|
||||
Self::ExplicitChunks(_) => Cow::Owned(
|
||||
Self::Explicit(values) => Ok(Cow::Borrowed(values)),
|
||||
Self::ExplicitChunks(_) => Ok(Cow::Owned(
|
||||
(0..self.len())
|
||||
.filter_map(|row| self.timestamp(row))
|
||||
.collect(),
|
||||
),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -32,18 +32,99 @@ fn grid_specs_validate_and_ranges_are_inclusive_without_rounding_up() {
|
||||
assert!(GridSpec::try_new(0, -1, 1).is_err());
|
||||
assert!(GridSpec::try_new(0, 1, 0).is_err());
|
||||
assert!(GridSpec::try_new(i64::MAX, 1, 2).is_err());
|
||||
let extreme =
|
||||
Arc::new(EvalGrid::try_new(GridSpec::try_new(i64::MIN, i64::MAX, 3).unwrap()).unwrap());
|
||||
assert_eq!(
|
||||
&EvalGrid::try_new(GridSpec::try_new(i64::MIN, i64::MAX, 3).unwrap())
|
||||
.unwrap()
|
||||
.timestamps()[..],
|
||||
&extreme.timestamps().unwrap()[..],
|
||||
&[i64::MIN, -1, i64::MAX - 1]
|
||||
);
|
||||
let axis = TimeAxis::Grid(GridRange::try_new(extreme, 0, 3).unwrap());
|
||||
assert_eq!(axis.timestamp(0), Some(i64::MIN));
|
||||
assert_eq!(axis.timestamp(1), Some(-1));
|
||||
assert_eq!(axis.timestamp(2), Some(i64::MAX - 1));
|
||||
assert_eq!(axis.timestamp(3), None);
|
||||
let deferred_extreme = Arc::new(
|
||||
EvalGrid::try_deferred(GridSpec::try_new(i64::MIN, i64::MAX, 3).unwrap()).unwrap(),
|
||||
);
|
||||
let axis = TimeAxis::Grid(GridRange::try_new(deferred_extreme.clone(), 0, 3).unwrap());
|
||||
assert_eq!(axis.timestamp(0), Some(i64::MIN));
|
||||
assert_eq!(axis.timestamp(1), Some(-1));
|
||||
assert_eq!(axis.timestamp(2), Some(i64::MAX - 1));
|
||||
assert_eq!(axis.timestamp(3), None);
|
||||
assert!(deferred_extreme.timestamps.get().is_none());
|
||||
assert!(GridSpec::try_new(i64::MIN, i64::MAX, 4).is_err());
|
||||
assert!(GridSpec::try_new(0, i64::MAX, u64::MAX).is_err());
|
||||
assert!(GridSpec::try_new(i64::MIN, 1, 2).is_ok());
|
||||
assert!(GridSpec::try_new(i64::MAX - 1, 1, 2).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deferred_grid_keeps_large_grid_unmaterialized_until_requested() {
|
||||
let len = (isize::MAX as u64 / std::mem::size_of::<i64>() as u64) + 1;
|
||||
let spec = GridSpec::try_new(i64::MIN, 1, len).unwrap();
|
||||
let grid = Arc::new(EvalGrid::try_deferred(spec).unwrap());
|
||||
assert!(grid.timestamps.get().is_none());
|
||||
let range = GridRange::try_new(grid.clone(), 0, len as usize).unwrap();
|
||||
let ranges = GridRanges::try_new(grid.clone(), vec![GridSpan::new(1, 2)]).unwrap();
|
||||
assert_eq!(ranges.len(), 2);
|
||||
let range_axis = TimeAxis::Grid(range);
|
||||
assert_eq!(range_axis.timestamp(1), Some(i64::MIN + 1));
|
||||
assert_eq!(
|
||||
range_axis.timestamp(len as usize - 1),
|
||||
Some(i64::MIN + (len as i64 - 1))
|
||||
);
|
||||
assert_eq!(range_axis.timestamp(len as usize), None);
|
||||
let axis = TimeAxis::GridRanges(ranges);
|
||||
assert_eq!(axis.timestamp(1), Some(i64::MIN + 2));
|
||||
assert_eq!(axis.timestamp(2), None);
|
||||
assert_eq!(axis.timestamp(len as usize - 1), None);
|
||||
assert!(grid.timestamps.get().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deferred_grid_materialization_is_shared_across_concurrent_accesses() {
|
||||
let grid = Arc::new(EvalGrid::try_deferred(GridSpec::try_new(-10, 2, 4).unwrap()).unwrap());
|
||||
let eager = EvalGrid::try_new(GridSpec::try_new(-10, 2, 4).unwrap()).unwrap();
|
||||
assert!(eager.timestamps.get().is_some());
|
||||
let threads: Vec<_> = (0..8)
|
||||
.map(|_| {
|
||||
let grid = grid.clone();
|
||||
std::thread::spawn(move || grid.timestamps().unwrap().as_ptr() as usize)
|
||||
})
|
||||
.collect();
|
||||
let pointers: Vec<usize> = threads
|
||||
.into_iter()
|
||||
.map(|thread| thread.join().unwrap())
|
||||
.collect();
|
||||
assert!(pointers.iter().all(|pointer| *pointer == pointers[0]));
|
||||
assert_eq!(&grid.timestamps().unwrap()[..], &[-10, -8, -6, -4]);
|
||||
assert!(grid.timestamps.get().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deferred_sparse_gather_materializes_only_requested_timestamps() {
|
||||
let grid = Arc::new(EvalGrid::try_deferred(GridSpec::try_new(10, 10, 6).unwrap()).unwrap());
|
||||
let axis = TimeAxis::GridRanges(
|
||||
GridRanges::try_new(grid.clone(), vec![GridSpan::new(1, 2), GridSpan::new(5, 1)]).unwrap(),
|
||||
);
|
||||
assert_eq!(axis.timestamps().unwrap().as_ref(), &[20, 30, 60]);
|
||||
assert!(grid.timestamps.get().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deferred_grid_allocation_failure_is_reported_and_not_cached() {
|
||||
let len = (isize::MAX as u64 / std::mem::size_of::<i64>() as u64) + 1;
|
||||
let grid =
|
||||
Arc::new(EvalGrid::try_deferred(GridSpec::try_new(i64::MIN, 1, len).unwrap()).unwrap());
|
||||
assert!(grid.timestamps().is_err());
|
||||
assert!(grid.timestamps.get().is_none());
|
||||
let range = GridRange::try_new(grid.clone(), 0, 1).unwrap();
|
||||
assert!(range.timestamps().is_err());
|
||||
assert!(TimeAxis::Grid(range).timestamps().is_err());
|
||||
assert!(grid.timestamps().is_err());
|
||||
assert!(grid.timestamps.get().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn grid_materialization_revalidates_public_specs_and_shares_ranges_by_value() {
|
||||
assert!(
|
||||
@@ -70,14 +151,26 @@ fn grid_materialization_revalidates_public_specs_and_shares_ranges_by_value() {
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
EvalGrid::try_deferred(GridSpec {
|
||||
start_ms: 0,
|
||||
step_ms: 0,
|
||||
len: 1
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
|
||||
let grid = Arc::new(EvalGrid::try_new(GridSpec::try_new(-5, 3, 5).unwrap()).unwrap());
|
||||
assert_eq!(&grid.timestamps()[..], &[-5, -2, 1, 4, 7]);
|
||||
assert!(grid.timestamps.get().is_some());
|
||||
assert_eq!(&grid.timestamps().unwrap()[..], &[-5, -2, 1, 4, 7]);
|
||||
let a = GridRange::try_new(grid.clone(), 1, 3).unwrap();
|
||||
let b = GridRange::try_new(Arc::new(EvalGrid::try_new(grid.spec()).unwrap()), 1, 3).unwrap();
|
||||
assert_eq!(a, b);
|
||||
let values = a.timestamps();
|
||||
assert_eq!(values.as_ptr(), grid.timestamps().as_ptr().wrapping_add(1));
|
||||
let values = a.timestamps().unwrap();
|
||||
assert_eq!(
|
||||
values.as_ptr(),
|
||||
grid.timestamps().unwrap().as_ptr().wrapping_add(1)
|
||||
);
|
||||
assert_eq!(&values[..], &[-2, 1, 4]);
|
||||
assert!(GridRange::try_new(grid.clone(), 5, 0).unwrap().is_empty());
|
||||
assert!(GridRange::try_new(grid.clone(), 4, 2).is_err());
|
||||
|
||||
+1
-1
@@ -66,7 +66,7 @@ filtered `RecordBatch` stream.
|
||||
and defer new builds when the shared estimate is full. Cleanup still prunes obsolete
|
||||
range entries and expires series indexes to reclaim published usage. Concurrent builds may
|
||||
overshoot; closed-region files are not counted. Series handles and the SST purger own file deletion.
|
||||
- **Series format PoC**: samples are rebuildable local SST sidecars. Optional writer interfaces (`format.rs`) integrate with reconciliation, catalog publication, and deferred purge. Defaults retain the original Parquet writer and disable samples. Prepared sources, read APIs, and concrete new formats are deferred; ordinary scans only use Parquet index artifacts.
|
||||
- **Series format PoC**: samples are rebuildable local SST sidecars. Optional writer interfaces (`format.rs`) integrate with reconciliation, catalog publication, and deferred purge. Defaults retain the original Parquet writer and disable samples. The opt-in columnar backend (`series_index/columnar.rs`) implements streaming OpenDAL writes and pinned format reads; its wire contract is in `docs/formats/mito-series.md`. Artifact objects record paths and sizes, including the manifest, so reads need no stat. Codecs currently run inline pending scheduling benchmarks. Prepared sources and query-consumer integration are deferred; ordinary scans only use Parquet index artifacts.
|
||||
- **Series-index coverage** (`series_index/catalog.rs`): `SeriesIndexEntry` stores
|
||||
compaction-window width and SST summaries keyed by aligned start in both catalogs and Parquet footers.
|
||||
- **Request types** (`request.rs`): usually tied to proto definitions consumed by `datanode`.
|
||||
|
||||
@@ -88,6 +88,7 @@ tokio.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
tokio-util.workspace = true
|
||||
tracing.workspace = true
|
||||
twox-hash.workspace = true
|
||||
uuid.workspace = true
|
||||
winner_tree.workspace = true
|
||||
zstd.workspace = true
|
||||
|
||||
@@ -20,6 +20,11 @@ pub(crate) mod artifact;
|
||||
mod bucket;
|
||||
mod builder;
|
||||
mod catalog;
|
||||
#[allow(
|
||||
dead_code,
|
||||
reason = "Opt-in format; query consumer integration is deferred."
|
||||
)]
|
||||
pub(crate) mod columnar;
|
||||
pub(crate) mod format;
|
||||
#[cfg(test)]
|
||||
mod format_tests;
|
||||
@@ -70,7 +75,9 @@ pub(crate) const ROW_COUNT_COLUMN: &str = "__series_row_count";
|
||||
pub(crate) const METRIC_SERIES_ID_BATCH_SIZE: usize = 500;
|
||||
|
||||
/// Identifies one series in a physical metric region.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
#[derive(
|
||||
Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize,
|
||||
)]
|
||||
pub struct MetricSeriesId {
|
||||
/// Logical table ID inside the physical metric region.
|
||||
pub table_id: u32,
|
||||
|
||||
@@ -28,6 +28,8 @@ use crate::sst::file::RegionFileId;
|
||||
pub(crate) enum ArtifactFormat {
|
||||
#[serde(rename = "parquet")]
|
||||
Parquet,
|
||||
#[serde(rename = "columnar-series")]
|
||||
ColumnarSeries,
|
||||
#[cfg(test)]
|
||||
#[serde(rename = "fake-samples")]
|
||||
FakeSamples,
|
||||
@@ -36,13 +38,20 @@ pub(crate) enum ArtifactFormat {
|
||||
FakeIndex,
|
||||
}
|
||||
|
||||
/// One immutable object produced by a backend build.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub(crate) struct ArtifactObject {
|
||||
pub(crate) path: String,
|
||||
pub(crate) file_size: u64,
|
||||
}
|
||||
|
||||
/// Identifies the exact bytes produced by one backend build.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub(crate) struct ArtifactMetadata {
|
||||
pub(crate) format: ArtifactFormat,
|
||||
pub(crate) generation: FileId,
|
||||
/// Every object owned by this build, including metadata and auxiliary files.
|
||||
pub(crate) objects: Vec<String>,
|
||||
pub(crate) objects: Vec<ArtifactObject>,
|
||||
pub(crate) file_size: u64,
|
||||
}
|
||||
|
||||
@@ -86,8 +95,15 @@ impl SamplesFileHandle {
|
||||
impl Drop for SamplesFileInner {
|
||||
fn drop(&mut self) {
|
||||
if self.retired.load(Ordering::Acquire) {
|
||||
self.purger
|
||||
.purge_objects(self.file_id, self.entry.artifact.objects.clone());
|
||||
self.purger.purge_objects(
|
||||
self.file_id,
|
||||
self.entry
|
||||
.artifact
|
||||
.objects
|
||||
.iter()
|
||||
.map(|object| object.path.clone())
|
||||
.collect(),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -313,6 +313,7 @@ mod tests {
|
||||
);
|
||||
for (format, name) in [
|
||||
(ArtifactFormat::Parquet, "parquet"),
|
||||
(ArtifactFormat::ColumnarSeries, "columnar-series"),
|
||||
(ArtifactFormat::FakeSamples, "fake-samples"),
|
||||
(ArtifactFormat::FakeIndex, "fake-index"),
|
||||
] {
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Columnar series format. See `docs/formats/mito-series.md` for the wire contract.
|
||||
|
||||
mod codec;
|
||||
mod metadata;
|
||||
mod reader;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
mod writer;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use datatypes::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use store_api::metadata::RegionMetadataRef;
|
||||
use store_api::storage::FileId;
|
||||
|
||||
use crate::error::{InvalidMetaSnafu, Result};
|
||||
use crate::series_index::artifact::ArtifactFormat;
|
||||
use crate::series_index::catalog::SeriesIndexEntry;
|
||||
use crate::series_index::format::{FormatReader, FormatWriter, SeriesIndexFormat};
|
||||
use crate::series_index::{
|
||||
MetricSeriesId, ROW_COUNT_COLUMN, SeriesIndexFileHandle, TABLE_ID_COLUMN, TSID_COLUMN,
|
||||
};
|
||||
|
||||
pub(crate) const VERSION: u32 = 1;
|
||||
pub(crate) const MANIFEST_LIMIT: usize = 64 * 1024 * 1024;
|
||||
|
||||
/// Packing and compression choices are internal and do not change the wire version.
|
||||
#[derive(Debug, Clone)]
|
||||
pub(crate) struct ColumnarSeriesFormat {
|
||||
pub(crate) block_rows: usize,
|
||||
pub(crate) zstd_level: i32,
|
||||
pub(crate) compression: codec::Compression,
|
||||
pub(crate) part_bytes: u64,
|
||||
pub(crate) metadata_bytes: usize,
|
||||
pub(crate) page_bytes: usize,
|
||||
pub(crate) root_capacity: usize,
|
||||
}
|
||||
|
||||
impl Default for ColumnarSeriesFormat {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
block_rows: 8192,
|
||||
zstd_level: 3,
|
||||
compression: codec::Compression::Zstd,
|
||||
part_bytes: 512 * 1024 * 1024,
|
||||
metadata_bytes: 8 * 1024 * 1024,
|
||||
page_bytes: metadata::PAGE_TARGET,
|
||||
root_capacity: metadata::ROOT_CAPACITY,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl SeriesIndexFormat for ColumnarSeriesFormat {
|
||||
fn format(&self) -> ArtifactFormat {
|
||||
ArtifactFormat::ColumnarSeries
|
||||
}
|
||||
|
||||
async fn writer(
|
||||
&self,
|
||||
metadata: RegionMetadataRef,
|
||||
store: ObjectStore,
|
||||
entry: &SeriesIndexEntry,
|
||||
) -> Result<Box<dyn FormatWriter>> {
|
||||
Ok(Box::new(writer::ColumnarWriter::new(
|
||||
self.clone(),
|
||||
metadata,
|
||||
store,
|
||||
entry,
|
||||
)?))
|
||||
}
|
||||
|
||||
async fn reader(
|
||||
&self,
|
||||
store: ObjectStore,
|
||||
handle: SeriesIndexFileHandle,
|
||||
) -> Result<Box<dyn FormatReader>> {
|
||||
Ok(Box::new(reader::ColumnarReader::open(store, handle).await?))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub(crate) struct Tag {
|
||||
pub(crate) column_id: u32,
|
||||
pub(crate) name: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub(crate) struct Part {
|
||||
pub(crate) part_id: u32,
|
||||
pub(crate) file_size: u64,
|
||||
pub(crate) num_series: u64,
|
||||
pub(crate) first: MetricSeriesId,
|
||||
pub(crate) last: MetricSeriesId,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub(crate) struct Manifest {
|
||||
pub(crate) format: ArtifactFormat,
|
||||
pub(crate) version: u32,
|
||||
pub(crate) region_id: u64,
|
||||
pub(crate) generation: FileId,
|
||||
pub(crate) tags: Vec<Tag>,
|
||||
pub(crate) coverage: SeriesIndexEntry,
|
||||
pub(crate) files: Vec<Part>,
|
||||
}
|
||||
|
||||
impl Manifest {
|
||||
pub(crate) fn prefix(&self) -> String {
|
||||
format!("{}/series/{}/", self.region_id, self.generation)
|
||||
}
|
||||
pub(crate) fn part_path(&self, id: u32) -> String {
|
||||
format!("{}part-{id}.series", self.prefix())
|
||||
}
|
||||
pub(crate) fn path(&self) -> String {
|
||||
format!("{}manifest.json", self.prefix())
|
||||
}
|
||||
pub(crate) fn schema(&self) -> SchemaRef {
|
||||
let mut fields = vec![
|
||||
Field::new(TABLE_ID_COLUMN, DataType::UInt32, false),
|
||||
Field::new(TSID_COLUMN, DataType::UInt64, false),
|
||||
Field::new(ROW_COUNT_COLUMN, DataType::UInt64, false),
|
||||
];
|
||||
fields.extend(
|
||||
self.tags
|
||||
.iter()
|
||||
.map(|tag| Field::new(&tag.name, DataType::Utf8, true)),
|
||||
);
|
||||
Arc::new(Schema::new(fields))
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn invalid(reason: impl Into<String>) -> crate::error::Error {
|
||||
InvalidMetaSnafu {
|
||||
reason: reason.into(),
|
||||
}
|
||||
.build()
|
||||
}
|
||||
|
||||
pub(crate) fn require(valid: bool, reason: &str) -> Result<()> {
|
||||
if valid { Ok(()) } else { Err(invalid(reason)) }
|
||||
}
|
||||
|
||||
pub(crate) fn unsupported(reason: impl Into<String>) -> crate::error::Error {
|
||||
crate::error::UnsupportedOperationSnafu {
|
||||
err_msg: reason.into(),
|
||||
}
|
||||
.build()
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Bounded, explicit wire codecs. Encoding and compression have independent IDs.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::io::Read;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::series_index::columnar::{invalid, require, unsupported};
|
||||
|
||||
pub(crate) const COLUMN_LIMIT: usize = 64 * 1024 * 1024;
|
||||
pub(crate) const COLUMN_STORED_LIMIT: usize = COLUMN_LIMIT + 1024 * 1024;
|
||||
pub(crate) const INTEGER: u16 = 1;
|
||||
pub(crate) const DICTIONARY: u16 = 2;
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
#[repr(u16)]
|
||||
pub(crate) enum Compression {
|
||||
None = 0,
|
||||
Zstd = 1,
|
||||
}
|
||||
|
||||
pub(crate) fn checksum(bytes: &[u8]) -> u64 {
|
||||
twox_hash::XxHash64::oneshot(0, bytes)
|
||||
}
|
||||
|
||||
pub(crate) fn compress(bytes: &[u8], compression: Compression, level: i32) -> Result<Vec<u8>> {
|
||||
match compression {
|
||||
Compression::None => Ok(bytes.to_vec()),
|
||||
Compression::Zstd => zstd::stream::encode_all(bytes, level)
|
||||
.map_err(|e| invalid(format!("series compression: {e}"))),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn decompress(bytes: &[u8], compression: u16, decoded: usize) -> Result<Vec<u8>> {
|
||||
let output = match compression {
|
||||
0 => bytes.to_vec(),
|
||||
1 => {
|
||||
let decoder = zstd::stream::read::Decoder::new(bytes)
|
||||
.map_err(|e| invalid(format!("series zstd header: {e}")))?;
|
||||
let mut output = Vec::new();
|
||||
// Never let a frame's claimed size control allocation or output growth.
|
||||
decoder
|
||||
.take(decoded as u64 + 1)
|
||||
.read_to_end(&mut output)
|
||||
.map_err(|e| invalid(format!("series decompression: {e}")))?;
|
||||
output
|
||||
}
|
||||
id => return Err(unsupported(format!("series compression {id}"))),
|
||||
};
|
||||
require(output.len() == decoded, "series decoded length mismatch")?;
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
pub(crate) fn put_u32(out: &mut Vec<u8>, value: u32) {
|
||||
out.extend(value.to_le_bytes());
|
||||
}
|
||||
pub(crate) fn put_u64(out: &mut Vec<u8>, value: u64) {
|
||||
out.extend(value.to_le_bytes());
|
||||
}
|
||||
pub(crate) fn put_string(out: &mut Vec<u8>, value: &str) {
|
||||
put_u64(out, value.len() as u64);
|
||||
out.extend(value.as_bytes());
|
||||
}
|
||||
pub(crate) fn put_varint(out: &mut Vec<u8>, mut value: u64) {
|
||||
while value >= 128 {
|
||||
out.push(value as u8 | 128);
|
||||
value >>= 7;
|
||||
}
|
||||
out.push(value as u8);
|
||||
}
|
||||
|
||||
pub(crate) struct Cursor<'a> {
|
||||
bytes: &'a [u8],
|
||||
offset: usize,
|
||||
}
|
||||
impl<'a> Cursor<'a> {
|
||||
pub(crate) fn new(bytes: &'a [u8]) -> Self {
|
||||
Self { bytes, offset: 0 }
|
||||
}
|
||||
pub(crate) fn remaining(&self) -> usize {
|
||||
self.bytes.len() - self.offset
|
||||
}
|
||||
pub(crate) fn take(&mut self, count: usize) -> Result<&'a [u8]> {
|
||||
let end = self
|
||||
.offset
|
||||
.checked_add(count)
|
||||
.ok_or_else(|| invalid("series length overflow"))?;
|
||||
let result = self
|
||||
.bytes
|
||||
.get(self.offset..end)
|
||||
.ok_or_else(|| invalid("truncated series payload"))?;
|
||||
self.offset = end;
|
||||
Ok(result)
|
||||
}
|
||||
pub(crate) fn u8(&mut self) -> Result<u8> {
|
||||
Ok(self.take(1)?[0])
|
||||
}
|
||||
pub(crate) fn u16(&mut self) -> Result<u16> {
|
||||
let b = self.take(2)?;
|
||||
Ok(u16::from_le_bytes([b[0], b[1]]))
|
||||
}
|
||||
pub(crate) fn u32(&mut self) -> Result<u32> {
|
||||
let b = self.take(4)?;
|
||||
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||
}
|
||||
pub(crate) fn u64(&mut self) -> Result<u64> {
|
||||
let b = self.take(8)?;
|
||||
Ok(u64::from_le_bytes([
|
||||
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
|
||||
]))
|
||||
}
|
||||
pub(crate) fn length(&mut self) -> Result<usize> {
|
||||
usize::try_from(self.u64()?).map_err(|_| invalid("series length exceeds address space"))
|
||||
}
|
||||
pub(crate) fn string(&mut self) -> Result<String> {
|
||||
let len = self.length()?;
|
||||
std::str::from_utf8(self.take(len)?)
|
||||
.map(str::to_owned)
|
||||
.map_err(|_| invalid("invalid series UTF-8"))
|
||||
}
|
||||
pub(crate) fn varint(&mut self) -> Result<u64> {
|
||||
let mut value = 0;
|
||||
for shift in (0..70).step_by(7) {
|
||||
let byte = self.u8()?;
|
||||
require(shift != 63 || byte <= 1, "series varint overflow")?;
|
||||
value |= u64::from(byte & 127) << shift;
|
||||
if byte & 128 == 0 {
|
||||
return Ok(value);
|
||||
}
|
||||
}
|
||||
Err(invalid("series varint overflow"))
|
||||
}
|
||||
pub(crate) fn end(self) -> Result<()> {
|
||||
require(self.remaining() == 0, "trailing series payload bytes")
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn encode_integers(values: &[u64]) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
let mut previous = 0u64;
|
||||
for &value in values {
|
||||
let delta = value.wrapping_sub(previous) as i64;
|
||||
put_varint(&mut out, ((delta as u64) << 1) ^ ((delta >> 63) as u64));
|
||||
previous = value;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub(crate) fn decode_integers(bytes: &[u8], count: usize) -> Result<Vec<u64>> {
|
||||
require(count <= bytes.len(), "integer count exceeds payload")?;
|
||||
let mut cursor = Cursor::new(bytes);
|
||||
let mut out = Vec::with_capacity(count);
|
||||
let mut previous = 0u64;
|
||||
for _ in 0..count {
|
||||
let value = cursor.varint()?;
|
||||
let delta = (value >> 1) ^ (0u64.wrapping_sub(value & 1));
|
||||
previous = previous.wrapping_add(delta);
|
||||
out.push(previous);
|
||||
}
|
||||
cursor.end()?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct Dictionary {
|
||||
pub(crate) values: Vec<Option<String>>,
|
||||
pub(crate) codes: Vec<u32>,
|
||||
}
|
||||
|
||||
pub(crate) fn encode_dictionary(values: &[Option<&str>]) -> Result<Vec<u8>> {
|
||||
let mut lookup: HashMap<&str, u32> = HashMap::new();
|
||||
let mut dictionary = Vec::new();
|
||||
put_u64(&mut dictionary, 0); // patched with the entry count below
|
||||
dictionary.push(0); // slot zero is always null
|
||||
let mut codes = Vec::new();
|
||||
for value in values {
|
||||
let code = match value {
|
||||
None => 0,
|
||||
Some(value) => {
|
||||
if let Some(code) = lookup.get(value) {
|
||||
*code
|
||||
} else {
|
||||
let code = u32::try_from(lookup.len() + 1)
|
||||
.map_err(|_| invalid("too many dictionary entries"))?;
|
||||
lookup.insert(value, code);
|
||||
dictionary.push(1);
|
||||
put_string(&mut dictionary, value);
|
||||
code
|
||||
}
|
||||
}
|
||||
};
|
||||
put_varint(&mut codes, u64::from(code));
|
||||
require(
|
||||
dictionary.len() + codes.len() <= COLUMN_LIMIT,
|
||||
"series dictionary exceeds chunk limit",
|
||||
)?;
|
||||
}
|
||||
dictionary[..8].copy_from_slice(&((lookup.len() + 1) as u64).to_le_bytes());
|
||||
let mut out = Vec::new();
|
||||
put_varint(&mut out, dictionary.len() as u64);
|
||||
out.extend(dictionary);
|
||||
out.extend(codes);
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
pub(crate) fn decode_dictionary(bytes: &[u8], count: usize) -> Result<Dictionary> {
|
||||
let mut cursor = Cursor::new(bytes);
|
||||
let length =
|
||||
usize::try_from(cursor.varint()?).map_err(|_| invalid("dictionary length overflow"))?;
|
||||
let mut dictionary = Cursor::new(cursor.take(length)?);
|
||||
let entries = dictionary.length()?;
|
||||
require(
|
||||
entries > 0 && entries <= dictionary.remaining(),
|
||||
"invalid dictionary count",
|
||||
)?;
|
||||
require(
|
||||
count <= cursor.remaining(),
|
||||
"dictionary code count exceeds payload",
|
||||
)?;
|
||||
let mut values = Vec::new();
|
||||
let mut unique = std::collections::HashSet::new();
|
||||
for index in 0..entries {
|
||||
let present = dictionary.u8()?;
|
||||
if index == 0 {
|
||||
require(present == 0, "dictionary slot zero must be null")?;
|
||||
values.push(None);
|
||||
} else {
|
||||
require(present == 1, "nonzero dictionary slot must be present")?;
|
||||
let value = dictionary.string()?;
|
||||
require(unique.insert(value.clone()), "duplicate dictionary value")?;
|
||||
values.push(Some(value));
|
||||
}
|
||||
}
|
||||
dictionary.end()?;
|
||||
let mut codes = Vec::with_capacity(count);
|
||||
for _ in 0..count {
|
||||
let code = cursor.varint()?;
|
||||
require(
|
||||
code < entries as u64 && code <= u32::MAX as u64,
|
||||
"invalid dictionary code",
|
||||
)?;
|
||||
codes.push(code as u32);
|
||||
}
|
||||
cursor.end()?;
|
||||
Ok(Dictionary { values, codes })
|
||||
}
|
||||
@@ -0,0 +1,456 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Explicit little-endian framing and the two-level metadata index.
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::series_index::columnar::codec::{
|
||||
self, Compression, Cursor, put_string, put_u32, put_u64,
|
||||
};
|
||||
use crate::series_index::columnar::{VERSION, invalid, require, unsupported};
|
||||
use crate::series_index::{MetricSeriesId, ROW_COUNT_COLUMN, TSID_COLUMN};
|
||||
|
||||
pub(crate) const MAGIC: &[u8; 8] = b"GTSERIES";
|
||||
pub(crate) const HEADER_SIZE: u64 = 16;
|
||||
pub(crate) const TRAILER_SIZE: u64 = 96;
|
||||
pub(crate) const PAGE_TARGET: usize = 65536;
|
||||
pub(crate) const PAGE_LIMIT: usize = 8 * 1024 * 1024;
|
||||
pub(crate) const PAGE_STORED_LIMIT: usize = PAGE_LIMIT + PAGE_TARGET;
|
||||
pub(crate) const ROOT_CAPACITY: usize = (PAGE_TARGET - 12) / 56;
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub(crate) struct PayloadReference {
|
||||
pub(crate) offset: u64,
|
||||
pub(crate) stored: u32,
|
||||
pub(crate) decoded: u32,
|
||||
pub(crate) checksum: u64,
|
||||
pub(crate) encoding: u16,
|
||||
pub(crate) compression: u16,
|
||||
}
|
||||
impl PayloadReference {
|
||||
pub(crate) fn end(&self) -> Result<u64> {
|
||||
self.offset
|
||||
.checked_add(u64::from(self.stored))
|
||||
.ok_or_else(|| invalid("series offset overflow"))
|
||||
}
|
||||
pub(crate) fn validate(&self, start: u64, end: u64, page: bool) -> Result<()> {
|
||||
let (decoded_limit, stored_limit) = if page {
|
||||
(PAGE_LIMIT, PAGE_STORED_LIMIT)
|
||||
} else {
|
||||
(codec::COLUMN_LIMIT, codec::COLUMN_STORED_LIMIT)
|
||||
};
|
||||
require(
|
||||
self.stored > 0
|
||||
&& self.decoded > 0
|
||||
&& self.stored as usize <= stored_limit
|
||||
&& self.decoded as usize <= decoded_limit,
|
||||
"series payload exceeds size limits",
|
||||
)?;
|
||||
require(
|
||||
self.offset >= start && self.end()? <= end,
|
||||
"series payload outside its area",
|
||||
)
|
||||
}
|
||||
pub(crate) fn encode(&self, out: &mut Vec<u8>) {
|
||||
put_u64(out, self.offset);
|
||||
put_u32(out, self.stored);
|
||||
put_u32(out, self.decoded);
|
||||
put_u64(out, self.checksum);
|
||||
out.extend(self.encoding.to_le_bytes());
|
||||
out.extend(self.compression.to_le_bytes());
|
||||
put_u32(out, 0);
|
||||
}
|
||||
pub(crate) fn decode(c: &mut Cursor<'_>) -> Result<Self> {
|
||||
let reference = Self {
|
||||
offset: c.u64()?,
|
||||
stored: c.u32()?,
|
||||
decoded: c.u32()?,
|
||||
checksum: c.u64()?,
|
||||
encoding: c.u16()?,
|
||||
compression: c.u16()?,
|
||||
};
|
||||
require(c.u32()? == 0, "nonzero series reference reserved field")?;
|
||||
Ok(reference)
|
||||
}
|
||||
pub(crate) fn unpack(&self, bytes: &[u8]) -> Result<Vec<u8>> {
|
||||
require(
|
||||
bytes.len() == self.stored as usize && codec::checksum(bytes) == self.checksum,
|
||||
"series payload checksum mismatch",
|
||||
)?;
|
||||
codec::decompress(bytes, self.compression, self.decoded as usize)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn pack(
|
||||
bytes: &[u8],
|
||||
offset: u64,
|
||||
encoding: u16,
|
||||
compression: Compression,
|
||||
level: i32,
|
||||
page: bool,
|
||||
) -> Result<(PayloadReference, Vec<u8>)> {
|
||||
let limit = if page {
|
||||
PAGE_LIMIT
|
||||
} else {
|
||||
codec::COLUMN_LIMIT
|
||||
};
|
||||
require(
|
||||
!bytes.is_empty() && bytes.len() <= limit,
|
||||
"series decoded payload exceeds size limit",
|
||||
)?;
|
||||
let stored = codec::compress(bytes, compression, level)?;
|
||||
let reference = PayloadReference {
|
||||
offset,
|
||||
stored: u32::try_from(stored.len())
|
||||
.map_err(|_| invalid("series stored length overflow"))?,
|
||||
decoded: bytes.len() as u32,
|
||||
checksum: codec::checksum(&stored),
|
||||
encoding,
|
||||
compression: compression as u16,
|
||||
};
|
||||
reference.validate(offset, reference.end()?, page)?;
|
||||
Ok((reference, stored))
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(crate) struct PageReference {
|
||||
pub(crate) first: MetricSeriesId,
|
||||
pub(crate) last: MetricSeriesId,
|
||||
pub(crate) payload: PayloadReference,
|
||||
}
|
||||
impl PageReference {
|
||||
pub(crate) fn encode(&self, out: &mut Vec<u8>) {
|
||||
put_u32(out, self.first.table_id);
|
||||
put_u64(out, self.first.tsid);
|
||||
put_u32(out, self.last.table_id);
|
||||
put_u64(out, self.last.tsid);
|
||||
self.payload.encode(out);
|
||||
}
|
||||
pub(crate) fn decode(c: &mut Cursor<'_>) -> Result<Self> {
|
||||
let first = MetricSeriesId {
|
||||
table_id: c.u32()?,
|
||||
tsid: c.u64()?,
|
||||
};
|
||||
let last = MetricSeriesId {
|
||||
table_id: c.u32()?,
|
||||
tsid: c.u64()?,
|
||||
};
|
||||
require(first <= last, "reversed series page bounds")?;
|
||||
Ok(Self {
|
||||
first,
|
||||
last,
|
||||
payload: PayloadReference::decode(c)?,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(crate) struct Column {
|
||||
pub(crate) name: String,
|
||||
pub(crate) payload: PayloadReference,
|
||||
pub(crate) null_count: u32,
|
||||
pub(crate) bounds: Option<(String, String)>,
|
||||
}
|
||||
#[derive(Debug, Clone)]
|
||||
pub(crate) struct Block {
|
||||
pub(crate) table_id: u32,
|
||||
pub(crate) first: u64,
|
||||
pub(crate) last: u64,
|
||||
pub(crate) rows: u32,
|
||||
pub(crate) columns: Vec<Column>,
|
||||
}
|
||||
impl Block {
|
||||
pub(crate) fn first_id(&self) -> MetricSeriesId {
|
||||
MetricSeriesId {
|
||||
table_id: self.table_id,
|
||||
tsid: self.first,
|
||||
}
|
||||
}
|
||||
pub(crate) fn last_id(&self) -> MetricSeriesId {
|
||||
MetricSeriesId {
|
||||
table_id: self.table_id,
|
||||
tsid: self.last,
|
||||
}
|
||||
}
|
||||
pub(crate) fn encode(&self, out: &mut Vec<u8>) {
|
||||
put_u32(out, self.table_id);
|
||||
put_u64(out, self.first);
|
||||
put_u64(out, self.last);
|
||||
put_u32(out, self.rows);
|
||||
put_u32(out, self.columns.len() as u32);
|
||||
for column in &self.columns {
|
||||
put_string(out, &column.name);
|
||||
column.payload.encode(out);
|
||||
put_u32(out, column.null_count);
|
||||
if let Some((min, max)) = &column.bounds {
|
||||
put_u32(out, 1);
|
||||
put_string(out, min);
|
||||
put_string(out, max);
|
||||
} else {
|
||||
put_u32(out, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
fn decode(
|
||||
c: &mut Cursor<'_>,
|
||||
names: &[String],
|
||||
payload_end: u64,
|
||||
previous_end: &mut u64,
|
||||
) -> Result<Self> {
|
||||
let table_id = c.u32()?;
|
||||
let first = c.u64()?;
|
||||
let last = c.u64()?;
|
||||
let rows = c.u32()?;
|
||||
let count = c.u32()? as usize;
|
||||
require(
|
||||
rows > 0
|
||||
&& rows as usize <= codec::COLUMN_LIMIT / 8
|
||||
&& first <= last
|
||||
&& u64::from(rows - 1) <= last - first,
|
||||
"invalid series block bounds/count",
|
||||
)?;
|
||||
require(
|
||||
count >= 2 && count <= names.len() && count <= c.remaining() / 48,
|
||||
"invalid series column count",
|
||||
)?;
|
||||
let mut columns = Vec::with_capacity(count);
|
||||
let mut next_name = 0;
|
||||
for index in 0..count {
|
||||
let name = c.string()?;
|
||||
let position = names[next_name..]
|
||||
.iter()
|
||||
.position(|n| *n == name)
|
||||
.map(|p| p + next_name)
|
||||
.ok_or_else(|| invalid("unknown or unordered series column"))?;
|
||||
require(
|
||||
index >= 2 || position == index,
|
||||
"missing required series column",
|
||||
)?;
|
||||
next_name = position + 1;
|
||||
let payload = PayloadReference::decode(c)?;
|
||||
payload.validate(*previous_end, payload_end, false)?;
|
||||
*previous_end = payload.end()?;
|
||||
let null_count = c.u32()?;
|
||||
require(null_count <= rows, "invalid series null count")?;
|
||||
let bounds = match c.u32()? {
|
||||
0 => None,
|
||||
1 => {
|
||||
let min = c.string()?;
|
||||
let max = c.string()?;
|
||||
require(min <= max, "reversed string bounds")?;
|
||||
Some((min, max))
|
||||
}
|
||||
_ => return Err(unsupported("series statistics kind")),
|
||||
};
|
||||
if name == TSID_COLUMN || name == ROW_COUNT_COLUMN {
|
||||
require(
|
||||
null_count == 0 && bounds.is_none(),
|
||||
"invalid integer column statistics",
|
||||
)?;
|
||||
} else {
|
||||
require(
|
||||
bounds.is_some() == (null_count < rows),
|
||||
"missing or invalid string statistics",
|
||||
)?;
|
||||
}
|
||||
columns.push(Column {
|
||||
name,
|
||||
payload,
|
||||
null_count,
|
||||
bounds,
|
||||
});
|
||||
}
|
||||
require(
|
||||
columns.iter().take(2).all(|column| {
|
||||
column.payload.encoding != codec::INTEGER || rows <= column.payload.decoded
|
||||
}),
|
||||
"series row count exceeds integer payload lengths",
|
||||
)?;
|
||||
Ok(Self {
|
||||
table_id,
|
||||
first,
|
||||
last,
|
||||
rows,
|
||||
columns,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) enum Page {
|
||||
Leaf(Vec<Block>),
|
||||
Internal(Vec<PageReference>),
|
||||
}
|
||||
impl Page {
|
||||
pub(crate) fn encode(&self) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
match self {
|
||||
Self::Leaf(blocks) => {
|
||||
put_u32(&mut out, 0);
|
||||
put_u64(&mut out, blocks.len() as u64);
|
||||
for block in blocks {
|
||||
block.encode(&mut out);
|
||||
}
|
||||
}
|
||||
Self::Internal(children) => {
|
||||
put_u32(&mut out, 1);
|
||||
put_u64(&mut out, children.len() as u64);
|
||||
for child in children {
|
||||
child.encode(&mut out);
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
pub(crate) fn decode(
|
||||
bytes: &[u8],
|
||||
reference: &PageReference,
|
||||
depth: u32,
|
||||
payload_end: u64,
|
||||
names: &[String],
|
||||
) -> Result<Self> {
|
||||
let mut c = Cursor::new(bytes);
|
||||
let kind = c.u32()?;
|
||||
let count = c.length()?;
|
||||
require(count > 0, "empty series metadata page")?;
|
||||
let (page, first, last) = match (kind, depth) {
|
||||
(0, 1) => {
|
||||
require(
|
||||
count <= c.remaining() / 28,
|
||||
"series block count exceeds page",
|
||||
)?;
|
||||
let mut blocks: Vec<Block> = Vec::new();
|
||||
let mut end = HEADER_SIZE;
|
||||
for _ in 0..count {
|
||||
let block = Block::decode(&mut c, names, payload_end, &mut end)?;
|
||||
if let Some(previous) = blocks.last() {
|
||||
require(
|
||||
previous.last_id() < block.first_id(),
|
||||
"unordered series blocks",
|
||||
)?;
|
||||
}
|
||||
blocks.push(block);
|
||||
}
|
||||
let first = blocks[0].first_id();
|
||||
let last = blocks[count - 1].last_id();
|
||||
(Self::Leaf(blocks), first, last)
|
||||
}
|
||||
(1, 2) => {
|
||||
require(
|
||||
bytes.len() <= PAGE_TARGET
|
||||
&& count <= ROOT_CAPACITY
|
||||
&& count <= c.remaining() / 56,
|
||||
"series root exceeds capacity",
|
||||
)?;
|
||||
let mut children: Vec<PageReference> = Vec::with_capacity(count);
|
||||
let mut end = payload_end;
|
||||
for _ in 0..count {
|
||||
let child = PageReference::decode(&mut c)?;
|
||||
child
|
||||
.payload
|
||||
.validate(end, reference.payload.offset, true)?;
|
||||
end = child.payload.end()?;
|
||||
if let Some(previous) = children.last() {
|
||||
require(previous.last < child.first, "unordered series child ranges")?;
|
||||
}
|
||||
children.push(child);
|
||||
}
|
||||
let first = children[0].first;
|
||||
let last = children[count - 1].last;
|
||||
(Self::Internal(children), first, last)
|
||||
}
|
||||
_ => return Err(unsupported("series metadata kind/depth")),
|
||||
};
|
||||
c.end()?;
|
||||
require(
|
||||
first == reference.first && last == reference.last,
|
||||
"series page bounds disagree with parent",
|
||||
)?;
|
||||
Ok(page)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn header() -> Vec<u8> {
|
||||
let mut out = MAGIC.to_vec();
|
||||
put_u32(&mut out, VERSION);
|
||||
put_u32(&mut out, 0);
|
||||
out
|
||||
}
|
||||
pub(crate) fn validate_header(bytes: &[u8]) -> Result<()> {
|
||||
let mut c = Cursor::new(bytes);
|
||||
require(c.take(8)? == MAGIC, "invalid series header magic")?;
|
||||
if c.u32()? != VERSION {
|
||||
return Err(unsupported("series file version"));
|
||||
}
|
||||
require(c.u32()? == 0, "nonzero series header reserved field")?;
|
||||
c.end()
|
||||
}
|
||||
|
||||
pub(crate) struct Trailer {
|
||||
pub(crate) root: PageReference,
|
||||
pub(crate) depth: u32,
|
||||
pub(crate) file_length: u64,
|
||||
pub(crate) payload_end: u64,
|
||||
}
|
||||
impl Trailer {
|
||||
pub(crate) fn encode(&self) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
self.root.encode(&mut out);
|
||||
put_u32(&mut out, self.depth);
|
||||
put_u32(&mut out, VERSION);
|
||||
put_u64(&mut out, self.file_length);
|
||||
put_u64(&mut out, self.payload_end);
|
||||
out.extend(MAGIC);
|
||||
let checksum = codec::checksum(&out);
|
||||
put_u64(&mut out, checksum);
|
||||
out
|
||||
}
|
||||
pub(crate) fn decode(bytes: &[u8], file_length: u64) -> Result<Self> {
|
||||
require(
|
||||
bytes.len() == TRAILER_SIZE as usize,
|
||||
"truncated series trailer",
|
||||
)?;
|
||||
let mut c = Cursor::new(bytes);
|
||||
let root = PageReference::decode(&mut c)?;
|
||||
let depth = c.u32()?;
|
||||
let version = c.u32()?;
|
||||
if !(1..=2).contains(&depth) || version != VERSION {
|
||||
return Err(unsupported("series trailer version/depth"));
|
||||
}
|
||||
let length = c.u64()?;
|
||||
let payload_end = c.u64()?;
|
||||
require(
|
||||
c.take(8)? == MAGIC && c.u64()? == codec::checksum(&bytes[..88]),
|
||||
"series trailer checksum/magic mismatch",
|
||||
)?;
|
||||
require(
|
||||
length == file_length
|
||||
&& payload_end >= HEADER_SIZE
|
||||
&& file_length >= HEADER_SIZE + TRAILER_SIZE,
|
||||
"invalid series file framing",
|
||||
)?;
|
||||
root.payload
|
||||
.validate(payload_end, file_length - TRAILER_SIZE, true)?;
|
||||
require(
|
||||
root.payload.end()? == file_length - TRAILER_SIZE,
|
||||
"series root does not end at trailer",
|
||||
)?;
|
||||
Ok(Self {
|
||||
root,
|
||||
depth,
|
||||
file_length,
|
||||
payload_end,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,557 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Lazy range reads with per-stream traversal and dictionary predicate evaluation.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_recordbatch::filter::SimpleFilterEvaluator;
|
||||
use datatypes::arrow::array::{ArrayRef, StringArray, UInt32Array, UInt64Array};
|
||||
use datatypes::arrow::datatypes::SchemaRef;
|
||||
use datatypes::arrow::record_batch::{RecordBatch, RecordBatchOptions};
|
||||
use datatypes::value::Value;
|
||||
use futures::stream::BoxStream;
|
||||
use object_store::ObjectStore;
|
||||
use snafu::ResultExt;
|
||||
|
||||
use crate::error::{NewRecordBatchSnafu, OpenDalSnafu, RecordBatchSnafu, Result, SerdeJsonSnafu};
|
||||
use crate::series_index::artifact::ArtifactFormat;
|
||||
use crate::series_index::columnar::codec::{self, Dictionary};
|
||||
use crate::series_index::columnar::metadata::{self, Block, Column, Page, PageReference, Trailer};
|
||||
use crate::series_index::columnar::{
|
||||
MANIFEST_LIMIT, Manifest, VERSION, invalid, require, unsupported,
|
||||
};
|
||||
use crate::series_index::format::{FormatReader, SeriesReadRequest};
|
||||
use crate::series_index::{
|
||||
MetricSeriesId, ROW_COUNT_COLUMN, SeriesIndexFileHandle, TABLE_ID_COLUMN, TSID_COLUMN,
|
||||
};
|
||||
use crate::sst::parquet::helper::fetch_byte_ranges;
|
||||
|
||||
pub(crate) struct ColumnarReader {
|
||||
store: ObjectStore,
|
||||
manifest: Arc<Manifest>,
|
||||
handle: SeriesIndexFileHandle,
|
||||
}
|
||||
impl ColumnarReader {
|
||||
pub(crate) async fn open(store: ObjectStore, handle: SeriesIndexFileHandle) -> Result<Self> {
|
||||
let entry = handle.entry();
|
||||
let artifact = entry
|
||||
.artifact
|
||||
.as_ref()
|
||||
.ok_or_else(|| invalid("missing columnar artifact"))?;
|
||||
require(
|
||||
artifact.format == ArtifactFormat::ColumnarSeries
|
||||
&& artifact.generation == entry.index_uuid,
|
||||
"series artifact identity mismatch",
|
||||
)?;
|
||||
let region = handle.file_id().region_id().as_u64();
|
||||
let path = format!("{region}/series/{}/manifest.json", entry.index_uuid);
|
||||
let mut objects = HashMap::with_capacity(artifact.objects.len());
|
||||
for object in &artifact.objects {
|
||||
require(
|
||||
objects
|
||||
.insert(object.path.as_str(), object.file_size)
|
||||
.is_none(),
|
||||
"duplicate series artifact object",
|
||||
)?;
|
||||
}
|
||||
let length = *objects
|
||||
.get(path.as_str())
|
||||
.ok_or_else(|| invalid("missing series manifest object"))?;
|
||||
require(
|
||||
length <= MANIFEST_LIMIT as u64,
|
||||
"series manifest exceeds size limit",
|
||||
)?;
|
||||
let bytes = read_range(&store, &path, 0, length).await?;
|
||||
let manifest: Manifest = serde_json::from_slice(&bytes).context(SerdeJsonSnafu)?;
|
||||
if manifest.version != VERSION {
|
||||
return Err(unsupported("series manifest version"));
|
||||
}
|
||||
require(
|
||||
manifest.format == ArtifactFormat::ColumnarSeries
|
||||
&& manifest.region_id == region
|
||||
&& manifest.generation == entry.index_uuid,
|
||||
"series manifest identity mismatch",
|
||||
)?;
|
||||
let mut coverage = entry.clone();
|
||||
coverage.artifact = None;
|
||||
coverage.file_size = 0;
|
||||
require(
|
||||
manifest.coverage == coverage,
|
||||
"series manifest coverage mismatch",
|
||||
)?;
|
||||
let mut names = HashSet::from([
|
||||
TABLE_ID_COLUMN.to_owned(),
|
||||
TSID_COLUMN.to_owned(),
|
||||
ROW_COUNT_COLUMN.to_owned(),
|
||||
]);
|
||||
let mut ids = HashSet::new();
|
||||
for tag in &manifest.tags {
|
||||
require(
|
||||
names.insert(tag.name.clone()) && ids.insert(tag.column_id),
|
||||
"duplicate series tag name/id",
|
||||
)?;
|
||||
}
|
||||
let mut paths = HashSet::from([path]);
|
||||
let mut file_size = length;
|
||||
let mut previous = None;
|
||||
for (index, part) in manifest.files.iter().enumerate() {
|
||||
require(
|
||||
index as u64 == u64::from(part.part_id)
|
||||
&& part.first <= part.last
|
||||
&& part.num_series > 0
|
||||
&& part.file_size >= metadata::HEADER_SIZE + metadata::TRAILER_SIZE,
|
||||
"invalid series part entry",
|
||||
)?;
|
||||
if let Some(last) = previous {
|
||||
require(last < part.first, "unordered series parts")?;
|
||||
}
|
||||
previous = Some(part.last);
|
||||
let part_path = manifest.part_path(part.part_id);
|
||||
require(
|
||||
objects.get(part_path.as_str()) == Some(&part.file_size),
|
||||
"series part length mismatch",
|
||||
)?;
|
||||
paths.insert(part_path);
|
||||
file_size = file_size
|
||||
.checked_add(part.file_size)
|
||||
.ok_or_else(|| invalid("series artifact length overflow"))?;
|
||||
}
|
||||
require(
|
||||
paths.len() == artifact.objects.len()
|
||||
&& artifact
|
||||
.objects
|
||||
.iter()
|
||||
.all(|object| paths.contains(&object.path)),
|
||||
"series artifact object set mismatch",
|
||||
)?;
|
||||
require(
|
||||
file_size == artifact.file_size && file_size == entry.file_size,
|
||||
"series artifact size mismatch",
|
||||
)?;
|
||||
Ok(Self {
|
||||
store,
|
||||
manifest: Arc::new(manifest),
|
||||
handle,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl FormatReader for ColumnarReader {
|
||||
fn schema(&self) -> SchemaRef {
|
||||
self.manifest.schema()
|
||||
}
|
||||
|
||||
fn read(&self, request: SeriesReadRequest) -> Result<BoxStream<'static, Result<RecordBatch>>> {
|
||||
let schema = self.schema();
|
||||
let mut projection = Vec::new();
|
||||
for name in &request.projection {
|
||||
projection.push(
|
||||
schema
|
||||
.index_of(name)
|
||||
.map_err(|_| invalid(format!("unknown series projection {name}")))?,
|
||||
);
|
||||
}
|
||||
for filter in &request.filters {
|
||||
schema
|
||||
.index_of(filter.column_name())
|
||||
.map_err(|_| invalid(format!("unknown series filter {}", filter.column_name())))?;
|
||||
}
|
||||
let projected = Arc::new(
|
||||
schema
|
||||
.project(&projection)
|
||||
.map_err(|e| invalid(format!("series projection: {e}")))?,
|
||||
);
|
||||
let manifest = self.manifest.clone();
|
||||
let store = self.store.clone();
|
||||
let handle = self.handle.clone();
|
||||
let names: Vec<_> = schema
|
||||
.fields()
|
||||
.iter()
|
||||
.skip(1)
|
||||
.map(|f| f.name().clone())
|
||||
.collect();
|
||||
Ok(Box::pin(async_stream::try_stream! {
|
||||
// Retained across every yield, even after the reader object is dropped.
|
||||
let _handle = handle;
|
||||
for part in &manifest.files {
|
||||
if !matches_range(part.first, part.last, &request) { continue; }
|
||||
let path = manifest.part_path(part.part_id);
|
||||
let size = part.file_size;
|
||||
let framing = fetch_byte_ranges(&path, store.clone(), &[0..metadata::HEADER_SIZE, size-metadata::TRAILER_SIZE..size]).await.context(OpenDalSnafu)?;
|
||||
require(
|
||||
framing.len() == 2
|
||||
&& framing[0].len() as u64 == metadata::HEADER_SIZE
|
||||
&& framing[1].len() as u64 == metadata::TRAILER_SIZE,
|
||||
"series framing length mismatch",
|
||||
)?;
|
||||
metadata::validate_header(&framing[0])?;
|
||||
let trailer = Trailer::decode(&framing[1], size)?;
|
||||
require(trailer.root.first == part.first && trailer.root.last == part.last, "series part/root bounds mismatch")?;
|
||||
let root = load_metadata_page(&store, &path, &trailer.root, trailer.depth, trailer.payload_end, &names).await?;
|
||||
let (mut active, children) = match root {
|
||||
Page::Leaf(blocks) => (Some(blocks), Vec::new()),
|
||||
Page::Internal(children) => (None, children),
|
||||
};
|
||||
let mut child_iter = children.into_iter();
|
||||
let mut observed_rows = 0u64;
|
||||
let mut previous_payload_end = metadata::HEADER_SIZE;
|
||||
loop {
|
||||
let blocks = if let Some(blocks) = active.take() { blocks } else {
|
||||
let Some(child) = child_iter.next() else { break; };
|
||||
if !matches_range(child.first, child.last, &request) { continue; }
|
||||
match load_metadata_page(&store, &path, &child, 1, trailer.payload_end, &names).await? {
|
||||
Page::Leaf(blocks) => blocks,
|
||||
Page::Internal(_) => Err(unsupported("internal child in depth-two series index"))?,
|
||||
}
|
||||
};
|
||||
for block in blocks {
|
||||
observed_rows = observed_rows.checked_add(u64::from(block.rows)).ok_or_else(|| invalid("series part row count overflow"))?;
|
||||
for column in &block.columns {
|
||||
column.payload.validate(previous_payload_end, trailer.payload_end, false)?;
|
||||
previous_payload_end = column.payload.end()?;
|
||||
}
|
||||
if !matches_range(block.first_id(), block.last_id(), &request) || !matches_statistics(&block, &request.filters) { continue; }
|
||||
if let Some(mut output) = read_block(&store, &path, &block, &request, projected.clone()).await? {
|
||||
while let Some(batch) = output.next_batch()? { yield batch; }
|
||||
}
|
||||
}
|
||||
}
|
||||
if request.ids.is_none() && request.table_id.is_none() {
|
||||
require(observed_rows == part.num_series, "series part row count mismatch")?;
|
||||
}
|
||||
}
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
fn matches_range(first: MetricSeriesId, last: MetricSeriesId, request: &SeriesReadRequest) -> bool {
|
||||
let (mut first, mut last) = (first, last);
|
||||
if let Some(table) = request.table_id {
|
||||
first = first.max(MetricSeriesId {
|
||||
table_id: table,
|
||||
tsid: 0,
|
||||
});
|
||||
last = last.min(MetricSeriesId {
|
||||
table_id: table,
|
||||
tsid: u64::MAX,
|
||||
});
|
||||
if first > last {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
request
|
||||
.ids
|
||||
.as_ref()
|
||||
.is_none_or(|ids| ids.range(first..=last).next().is_some())
|
||||
}
|
||||
|
||||
fn matches_statistics(block: &Block, filters: &[SimpleFilterEvaluator]) -> bool {
|
||||
for filter in filters {
|
||||
if !filter.is_eq() {
|
||||
continue;
|
||||
}
|
||||
if let Some(Value::String(value)) = filter.literal_value()
|
||||
&& let Some((min, max)) = block
|
||||
.columns
|
||||
.iter()
|
||||
.find(|c| c.name == filter.column_name())
|
||||
.and_then(|c| c.bounds.as_ref())
|
||||
&& (value.as_utf8() < min.as_str() || value.as_utf8() > max.as_str())
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
async fn read_range(store: &ObjectStore, path: &str, start: u64, end: u64) -> Result<bytes::Bytes> {
|
||||
let mut buffers = fetch_byte_ranges(path, store.clone(), std::slice::from_ref(&(start..end)))
|
||||
.await
|
||||
.context(OpenDalSnafu)?;
|
||||
let bytes = buffers
|
||||
.pop()
|
||||
.ok_or_else(|| invalid("missing series range response"))?;
|
||||
require(
|
||||
bytes.len() as u64 == end - start,
|
||||
"series range length mismatch",
|
||||
)?;
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
/// TODO: integrate with Mito's cache using store/region, generation, part, reference,
|
||||
/// checksum and codec IDs. The immutable path and reference remain available here.
|
||||
async fn load_metadata_page(
|
||||
store: &ObjectStore,
|
||||
path: &str,
|
||||
reference: &PageReference,
|
||||
depth: u32,
|
||||
payload_end: u64,
|
||||
names: &[String],
|
||||
) -> Result<Page> {
|
||||
if reference.payload.encoding != 1 {
|
||||
return Err(unsupported(format!(
|
||||
"series metadata encoding {}",
|
||||
reference.payload.encoding
|
||||
)));
|
||||
}
|
||||
let bytes = read_range(
|
||||
store,
|
||||
path,
|
||||
reference.payload.offset,
|
||||
reference.payload.end()?,
|
||||
)
|
||||
.await?;
|
||||
// TODO: benchmark inline versus offloaded codecs for latency, throughput,
|
||||
// and async-worker responsiveness before adding scheduling overhead.
|
||||
let decoded = reference.payload.unpack(&bytes)?;
|
||||
Page::decode(&decoded, reference, depth, payload_end, names)
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
enum Decoded {
|
||||
Integers(Vec<u64>),
|
||||
Strings(Dictionary),
|
||||
Table(u32, usize),
|
||||
Null(usize),
|
||||
}
|
||||
impl Decoded {
|
||||
fn value_bytes(&self, index: usize) -> usize {
|
||||
match self {
|
||||
Self::Integers(_) => 8,
|
||||
Self::Table(_, _) => 4,
|
||||
Self::Null(_) => 5,
|
||||
Self::Strings(dict) => {
|
||||
dict.values[dict.codes[index] as usize]
|
||||
.as_ref()
|
||||
.map_or(0, String::len)
|
||||
+ 5
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn array(&self, selected: &[usize]) -> ArrayRef {
|
||||
match self {
|
||||
Self::Integers(values) => Arc::new(UInt64Array::from_iter_values(
|
||||
selected.iter().map(|&i| values[i]),
|
||||
)),
|
||||
Self::Strings(dict) => Arc::new(StringArray::from_iter(
|
||||
selected
|
||||
.iter()
|
||||
.map(|&i| dict.values[dict.codes[i] as usize].as_deref()),
|
||||
)),
|
||||
Self::Table(value, _) => Arc::new(UInt32Array::from(vec![*value; selected.len()])),
|
||||
Self::Null(_) => Arc::new(StringArray::from(vec![None::<&str>; selected.len()])),
|
||||
}
|
||||
}
|
||||
fn filter(&self, filter: &SimpleFilterEvaluator, selected: &mut Vec<usize>) -> Result<()> {
|
||||
if let Self::Strings(dict) = self {
|
||||
let values: ArrayRef = Arc::new(StringArray::from_iter(
|
||||
dict.values.iter().map(|v| v.as_deref()),
|
||||
));
|
||||
let mask = filter.evaluate_array(&values).context(RecordBatchSnafu)?;
|
||||
selected.retain(|&i| mask.value(dict.codes[i] as usize));
|
||||
} else {
|
||||
let length = match self {
|
||||
Self::Integers(v) => v.len(),
|
||||
Self::Null(n) | Self::Table(_, n) => *n,
|
||||
Self::Strings(_) => 0,
|
||||
};
|
||||
let values = self.array(&(0..length).collect::<Vec<_>>());
|
||||
let mask = filter.evaluate_array(&values).context(RecordBatchSnafu)?;
|
||||
selected.retain(|&i| mask.value(i));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// TODO: share decoded chunks through the existing Mito cache, not a new cache framework.
|
||||
async fn load_column_chunk(
|
||||
store: &ObjectStore,
|
||||
path: &str,
|
||||
column: &Column,
|
||||
block: &Block,
|
||||
) -> Result<Decoded> {
|
||||
let integer = column.name == TSID_COLUMN || column.name == ROW_COUNT_COLUMN;
|
||||
let expected = if integer {
|
||||
codec::INTEGER
|
||||
} else {
|
||||
codec::DICTIONARY
|
||||
};
|
||||
if column.payload.encoding != expected {
|
||||
return Err(unsupported(format!(
|
||||
"series column encoding {} for {}",
|
||||
column.payload.encoding, column.name
|
||||
)));
|
||||
}
|
||||
let bytes = read_range(store, path, column.payload.offset, column.payload.end()?).await?;
|
||||
let rows = block.rows as usize;
|
||||
let first = block.first;
|
||||
let last = block.last;
|
||||
let bytes = column.payload.unpack(&bytes)?;
|
||||
if integer {
|
||||
let values = codec::decode_integers(&bytes, rows)?;
|
||||
if column.name == TSID_COLUMN {
|
||||
require(
|
||||
values.first() == Some(&first)
|
||||
&& values.last() == Some(&last)
|
||||
&& values.windows(2).all(|v| v[0] < v[1]),
|
||||
"invalid series TSID ordering/bounds",
|
||||
)?;
|
||||
} else {
|
||||
require(values.iter().all(|&v| v > 0), "zero series row count")?;
|
||||
}
|
||||
Ok(Decoded::Integers(values))
|
||||
} else {
|
||||
let dictionary = codec::decode_dictionary(&bytes, rows)?;
|
||||
require(
|
||||
dictionary.codes.iter().filter(|&&code| code == 0).count()
|
||||
== column.null_count as usize,
|
||||
"series dictionary null count mismatch",
|
||||
)?;
|
||||
let min = dictionary
|
||||
.codes
|
||||
.iter()
|
||||
.filter_map(|&code| dictionary.values[code as usize].as_ref())
|
||||
.min();
|
||||
let max = dictionary
|
||||
.codes
|
||||
.iter()
|
||||
.filter_map(|&code| dictionary.values[code as usize].as_ref())
|
||||
.max();
|
||||
require(
|
||||
column.bounds.as_ref().map(|(a, b)| (a, b)) == min.zip(max),
|
||||
"series dictionary bounds mismatch",
|
||||
)?;
|
||||
Ok(Decoded::Strings(dictionary))
|
||||
}
|
||||
}
|
||||
|
||||
async fn get_column(store: &ObjectStore, path: &str, block: &Block, name: &str) -> Result<Decoded> {
|
||||
if name == TABLE_ID_COLUMN {
|
||||
return Ok(Decoded::Table(block.table_id, block.rows as usize));
|
||||
}
|
||||
match block.columns.iter().find(|c| c.name == name) {
|
||||
Some(column) => load_column_chunk(store, path, column, block).await,
|
||||
None => Ok(Decoded::Null(block.rows as usize)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn read_block(
|
||||
store: &ObjectStore,
|
||||
path: &str,
|
||||
block: &Block,
|
||||
request: &SeriesReadRequest,
|
||||
schema: SchemaRef,
|
||||
) -> Result<Option<ProjectedBlock>> {
|
||||
let mut decoded = HashMap::new();
|
||||
let mut selected: Vec<_> = (0..block.rows as usize).collect();
|
||||
if let Some(ids) = &request.ids {
|
||||
let column = get_column(store, path, block, TSID_COLUMN).await?;
|
||||
if let Decoded::Integers(values) = &column {
|
||||
selected.retain(|&i| {
|
||||
ids.contains(&MetricSeriesId {
|
||||
table_id: block.table_id,
|
||||
tsid: values[i],
|
||||
})
|
||||
});
|
||||
}
|
||||
decoded.insert(TSID_COLUMN.to_owned(), Arc::new(column));
|
||||
}
|
||||
for filter in &request.filters {
|
||||
if selected.is_empty() {
|
||||
return Ok(None);
|
||||
}
|
||||
let name = filter.column_name();
|
||||
if !decoded.contains_key(name) {
|
||||
decoded.insert(
|
||||
name.to_owned(),
|
||||
Arc::new(get_column(store, path, block, name).await?),
|
||||
);
|
||||
}
|
||||
decoded
|
||||
.get(name)
|
||||
.ok_or_else(|| invalid("missing decoded filter column"))?
|
||||
.filter(filter, &mut selected)?;
|
||||
}
|
||||
if selected.is_empty() {
|
||||
return Ok(None);
|
||||
}
|
||||
let mut columns = Vec::with_capacity(request.projection.len());
|
||||
for name in &request.projection {
|
||||
if !decoded.contains_key(name) {
|
||||
decoded.insert(
|
||||
name.clone(),
|
||||
Arc::new(get_column(store, path, block, name).await?),
|
||||
);
|
||||
}
|
||||
columns.push(
|
||||
decoded
|
||||
.get(name)
|
||||
.ok_or_else(|| invalid("missing decoded output column"))?
|
||||
.clone(),
|
||||
);
|
||||
}
|
||||
Ok(Some(ProjectedBlock {
|
||||
columns,
|
||||
selected,
|
||||
schema,
|
||||
position: 0,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Dictionary expansion can be much larger than the stored chunk. Bound each
|
||||
/// materialized batch, allowing one oversized row, without expanding the block.
|
||||
struct ProjectedBlock {
|
||||
columns: Vec<Arc<Decoded>>,
|
||||
selected: Vec<usize>,
|
||||
schema: SchemaRef,
|
||||
position: usize,
|
||||
}
|
||||
impl ProjectedBlock {
|
||||
fn next_batch(&mut self) -> Result<Option<RecordBatch>> {
|
||||
const TARGET_BYTES: usize = 8 * 1024 * 1024;
|
||||
if self.position == self.selected.len() {
|
||||
return Ok(None);
|
||||
}
|
||||
let start = self.position;
|
||||
let mut bytes = 0usize;
|
||||
while self.position < self.selected.len() {
|
||||
let index = self.selected[self.position];
|
||||
let row_bytes = self.columns.iter().fold(0usize, |sum, column| {
|
||||
sum.saturating_add(column.value_bytes(index))
|
||||
});
|
||||
if self.position > start && bytes.saturating_add(row_bytes) > TARGET_BYTES {
|
||||
break;
|
||||
}
|
||||
bytes = bytes.saturating_add(row_bytes);
|
||||
self.position += 1;
|
||||
}
|
||||
let selected = &self.selected[start..self.position];
|
||||
let arrays = self
|
||||
.columns
|
||||
.iter()
|
||||
.map(|column| column.array(selected))
|
||||
.collect();
|
||||
Ok(Some(
|
||||
RecordBatch::try_new_with_options(
|
||||
self.schema.clone(),
|
||||
arrays,
|
||||
&RecordBatchOptions::new().with_row_count(Some(selected.len())),
|
||||
)
|
||||
.context(NewRecordBatchSnafu)?,
|
||||
))
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,614 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Streaming aggregation and owned OpenDAL outputs.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use bytes::Bytes;
|
||||
use datatypes::arrow::array::{Array, BinaryArray, DictionaryArray};
|
||||
use datatypes::arrow::datatypes::UInt32Type;
|
||||
use datatypes::arrow::record_batch::RecordBatch;
|
||||
use mito_codec::row_converter::SparseOffsetsCache;
|
||||
use object_store::{ObjectStore, Writer};
|
||||
use snafu::{IntoError, ResultExt};
|
||||
use store_api::metadata::RegionMetadataRef;
|
||||
|
||||
use crate::error::{JoinSnafu, OpenDalSnafu, Result, SerdeJsonSnafu};
|
||||
use crate::series_index::artifact::{ArtifactFormat, ArtifactMetadata, ArtifactObject};
|
||||
use crate::series_index::catalog::SeriesIndexEntry;
|
||||
use crate::series_index::columnar::codec::{self, encode_dictionary, encode_integers};
|
||||
use crate::series_index::columnar::metadata::{
|
||||
self, Block, Column, Page, PageReference, Trailer, pack,
|
||||
};
|
||||
use crate::series_index::columnar::{
|
||||
ColumnarSeriesFormat, MANIFEST_LIMIT, Manifest, Part, Tag, VERSION, invalid, require,
|
||||
};
|
||||
use crate::series_index::format::FormatWriter;
|
||||
use crate::series_index::writer::{decode_series_identity, tag_columns, validate_metadata};
|
||||
use crate::series_index::{MetricSeriesId, ROW_COUNT_COLUMN, TSID_COLUMN};
|
||||
use crate::sst::{DEFAULT_WRITE_BUFFER_SIZE, DEFAULT_WRITE_CONCURRENCY};
|
||||
|
||||
/// Maximum owned row bytes per block; a single larger row is flushed on its own.
|
||||
const BLOCK_BYTE_BUDGET: usize = 8 * 1024 * 1024;
|
||||
|
||||
/// Drop owns the live OpenDAL writer, including its private atomic-write temporary file.
|
||||
struct LiveFile {
|
||||
writer: Option<Writer>,
|
||||
store: ObjectStore,
|
||||
path: String,
|
||||
}
|
||||
impl Drop for LiveFile {
|
||||
fn drop(&mut self) {
|
||||
if let Some(mut writer) = self.writer.take() {
|
||||
let store = self.store.clone();
|
||||
let path = self.path.clone();
|
||||
common_runtime::spawn_compact(async move {
|
||||
if let Err(error) = writer.abort().await {
|
||||
common_telemetry::warn!(error; "Aborting series output {path}");
|
||||
}
|
||||
if let Err(error) = store.delete(&path).await {
|
||||
common_telemetry::warn!(error; "Deleting series output {path}");
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct Outputs {
|
||||
store: ObjectStore,
|
||||
active: Option<LiveFile>,
|
||||
paths: Vec<String>,
|
||||
committed: bool,
|
||||
}
|
||||
impl Outputs {
|
||||
async fn open(&mut self, path: String) -> Result<()> {
|
||||
require(self.active.is_none(), "series output already open")?;
|
||||
self.paths.push(path.clone());
|
||||
let store = self.store.clone();
|
||||
// The task owns opening: cancellation drops the returned LiveFile and aborts it.
|
||||
self.active = Some(
|
||||
common_runtime::spawn_compact(async move {
|
||||
let writer = store
|
||||
.writer_with(&path)
|
||||
.chunk(DEFAULT_WRITE_BUFFER_SIZE.as_bytes() as usize)
|
||||
.concurrent(DEFAULT_WRITE_CONCURRENCY)
|
||||
.await
|
||||
.context(OpenDalSnafu)?;
|
||||
Ok::<_, crate::error::Error>(LiveFile {
|
||||
writer: Some(writer),
|
||||
store,
|
||||
path,
|
||||
})
|
||||
})
|
||||
.await
|
||||
.context(JoinSnafu)??,
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
async fn write(&mut self, bytes: Vec<u8>) -> Result<()> {
|
||||
self.active
|
||||
.as_mut()
|
||||
.and_then(|f| f.writer.as_mut())
|
||||
.ok_or_else(|| invalid("series output is not open"))?
|
||||
.write(Bytes::from(bytes))
|
||||
.await
|
||||
.context(OpenDalSnafu)?;
|
||||
Ok(())
|
||||
}
|
||||
async fn close(&mut self) -> Result<()> {
|
||||
let file = self
|
||||
.active
|
||||
.as_mut()
|
||||
.ok_or_else(|| invalid("series output is not open"))?;
|
||||
file.writer
|
||||
.as_mut()
|
||||
.ok_or_else(|| invalid("series output is closed"))?
|
||||
.close()
|
||||
.await
|
||||
.context(OpenDalSnafu)?;
|
||||
file.writer = None;
|
||||
self.active = None;
|
||||
Ok(())
|
||||
}
|
||||
async fn cleanup(&mut self) -> Result<()> {
|
||||
let mut first_error = None;
|
||||
if let Some(file) = self.active.as_mut()
|
||||
&& let Some(writer) = file.writer.as_mut()
|
||||
{
|
||||
match writer.abort().await {
|
||||
Ok(()) => file.writer = None,
|
||||
Err(error) => first_error = Some(OpenDalSnafu.into_error(error)),
|
||||
}
|
||||
}
|
||||
self.active = None;
|
||||
for path in &self.paths {
|
||||
if let Err(error) = self.store.delete(path).await {
|
||||
first_error.get_or_insert_with(|| OpenDalSnafu.into_error(error));
|
||||
}
|
||||
}
|
||||
if let Some(error) = first_error {
|
||||
Err(error)
|
||||
} else {
|
||||
self.committed = true;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Drop for Outputs {
|
||||
fn drop(&mut self) {
|
||||
if !self.committed {
|
||||
let mut owned = Self {
|
||||
store: self.store.clone(),
|
||||
active: self.active.take(),
|
||||
paths: std::mem::take(&mut self.paths),
|
||||
committed: true,
|
||||
};
|
||||
common_runtime::spawn_compact(async move {
|
||||
if let Err(error) = owned.cleanup().await {
|
||||
common_telemetry::warn!(error; "Cleaning unpublished series artifact");
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Row {
|
||||
id: MetricSeriesId,
|
||||
count: u64,
|
||||
tags: Vec<Option<String>>,
|
||||
}
|
||||
|
||||
pub(crate) struct ColumnarWriter {
|
||||
options: ColumnarSeriesFormat,
|
||||
manifest: Manifest,
|
||||
outputs: Outputs,
|
||||
tags: Vec<(u32, String)>,
|
||||
offsets: SparseOffsetsCache,
|
||||
buffer: Vec<u8>,
|
||||
last_key: Vec<u8>,
|
||||
current: Option<Row>,
|
||||
rows: Vec<Row>,
|
||||
row_bytes: usize,
|
||||
leaves: Vec<Vec<Block>>,
|
||||
leaf_bytes: usize,
|
||||
descriptor_bytes: usize,
|
||||
position: u64,
|
||||
part_rows: u64,
|
||||
failed: bool,
|
||||
}
|
||||
impl ColumnarWriter {
|
||||
pub(crate) fn new(
|
||||
options: ColumnarSeriesFormat,
|
||||
metadata: RegionMetadataRef,
|
||||
store: ObjectStore,
|
||||
entry: &SeriesIndexEntry,
|
||||
) -> Result<Self> {
|
||||
validate_metadata(&metadata)?;
|
||||
require(
|
||||
options.block_rows > 0
|
||||
&& options.block_rows <= codec::COLUMN_LIMIT / 10
|
||||
&& options.root_capacity > 0
|
||||
&& options.root_capacity <= metadata::ROOT_CAPACITY
|
||||
&& (12..=metadata::PAGE_TARGET).contains(&options.page_bytes),
|
||||
"invalid columnar writer options",
|
||||
)?;
|
||||
let tags = tag_columns(&metadata);
|
||||
let mut coverage = entry.clone();
|
||||
coverage.file_size = 0;
|
||||
coverage.artifact = None;
|
||||
let manifest = Manifest {
|
||||
format: ArtifactFormat::ColumnarSeries,
|
||||
version: VERSION,
|
||||
region_id: metadata.region_id.as_u64(),
|
||||
generation: entry.index_uuid,
|
||||
tags: tags
|
||||
.iter()
|
||||
.map(|(id, name)| Tag {
|
||||
column_id: *id,
|
||||
name: name.clone(),
|
||||
})
|
||||
.collect(),
|
||||
coverage,
|
||||
files: Vec::new(),
|
||||
};
|
||||
Ok(Self {
|
||||
options,
|
||||
manifest,
|
||||
outputs: Outputs {
|
||||
store,
|
||||
active: None,
|
||||
paths: Vec::new(),
|
||||
committed: false,
|
||||
},
|
||||
tags,
|
||||
offsets: SparseOffsetsCache::new(),
|
||||
buffer: Vec::new(),
|
||||
last_key: Vec::new(),
|
||||
current: None,
|
||||
rows: Vec::new(),
|
||||
row_bytes: 0,
|
||||
leaves: Vec::new(),
|
||||
leaf_bytes: 12,
|
||||
descriptor_bytes: 0,
|
||||
position: 0,
|
||||
part_rows: 0,
|
||||
failed: false,
|
||||
})
|
||||
}
|
||||
|
||||
async fn push_current(&mut self) -> Result<()> {
|
||||
if let Some(row) = self.current.take() {
|
||||
let row_bytes = std::mem::size_of::<Row>()
|
||||
+ row.tags.capacity() * std::mem::size_of::<Option<String>>()
|
||||
+ row
|
||||
.tags
|
||||
.iter()
|
||||
.flatten()
|
||||
.map(String::capacity)
|
||||
.sum::<usize>();
|
||||
if self.row_bytes.saturating_add(row_bytes) > BLOCK_BYTE_BUDGET
|
||||
|| self
|
||||
.rows
|
||||
.first()
|
||||
.is_some_and(|first| first.id.table_id != row.id.table_id)
|
||||
{
|
||||
self.flush_block().await?;
|
||||
}
|
||||
self.row_bytes += row_bytes;
|
||||
self.rows.push(row);
|
||||
if self.rows.len() >= self.options.block_rows || self.row_bytes >= BLOCK_BYTE_BUDGET {
|
||||
self.flush_block().await?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn append_key(&mut self, key: &[u8]) -> Result<()> {
|
||||
if key == self.last_key && self.current.is_some() {
|
||||
if let Some(row) = self.current.as_mut() {
|
||||
row.count = row
|
||||
.count
|
||||
.checked_add(1)
|
||||
.ok_or_else(|| invalid("series row count overflow"))?;
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
let (id, tags) =
|
||||
decode_series_identity(key, &self.tags, &mut self.offsets, &mut self.buffer)?;
|
||||
if let Some(row) = self.current.as_mut() {
|
||||
require(id >= row.id, "series identifiers are not sorted")?;
|
||||
if id == row.id {
|
||||
row.count = row
|
||||
.count
|
||||
.checked_add(1)
|
||||
.ok_or_else(|| invalid("series row count overflow"))?;
|
||||
self.last_key.clear();
|
||||
self.last_key.extend(key);
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
self.push_current().await?;
|
||||
self.current = Some(Row { id, count: 1, tags });
|
||||
self.last_key.clear();
|
||||
self.last_key.extend(key);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn write_batch(&mut self, batch: &RecordBatch) -> Result<()> {
|
||||
require(batch.num_columns() >= 4, "series input is not a flat batch")?;
|
||||
let pk = batch.column(crate::sst::parquet::flat_format::primary_key_column_index(
|
||||
batch.num_columns(),
|
||||
));
|
||||
if let Some(keys) = pk.as_any().downcast_ref::<BinaryArray>() {
|
||||
require(keys.null_count() == 0, "null series primary key")?;
|
||||
for key in keys.iter().flatten() {
|
||||
self.append_key(key).await?;
|
||||
}
|
||||
} else if let Some(keys) = pk.as_any().downcast_ref::<DictionaryArray<UInt32Type>>() {
|
||||
require(keys.null_count() == 0, "null series primary key")?;
|
||||
let values = keys
|
||||
.values()
|
||||
.as_any()
|
||||
.downcast_ref::<BinaryArray>()
|
||||
.ok_or_else(|| invalid("series dictionary values must be binary"))?;
|
||||
for key in keys.keys().values() {
|
||||
require(
|
||||
(*key as usize) < values.len() && !values.is_null(*key as usize),
|
||||
"invalid series dictionary primary key",
|
||||
)?;
|
||||
self.append_key(values.value(*key as usize)).await?;
|
||||
}
|
||||
} else {
|
||||
return Err(invalid(
|
||||
"series primary key must be binary or UInt32 dictionary",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn flush_block(&mut self) -> Result<()> {
|
||||
if self.rows.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let rows = std::mem::take(&mut self.rows);
|
||||
self.row_bytes = 0;
|
||||
// TODO: benchmark inline versus offloaded codecs for latency, throughput,
|
||||
// and async-worker responsiveness before adding scheduling overhead.
|
||||
let (mut block, chunks) = encode_block(rows, &self.tags, &self.options)?;
|
||||
let mut descriptor = Vec::new();
|
||||
block.encode(&mut descriptor);
|
||||
require(
|
||||
descriptor.len() + 12 <= metadata::PAGE_LIMIT,
|
||||
"series descriptor exceeds leaf limit",
|
||||
)?;
|
||||
let needs_leaf =
|
||||
!self.leaves.is_empty() && self.leaf_bytes + descriptor.len() > self.options.page_bytes;
|
||||
if needs_leaf && self.leaves.len() == self.options.root_capacity {
|
||||
self.finish_part().await?;
|
||||
}
|
||||
if self.outputs.active.is_none() {
|
||||
let id = u32::try_from(self.manifest.files.len())
|
||||
.map_err(|_| invalid("too many series parts"))?;
|
||||
self.outputs.open(self.manifest.part_path(id)).await?;
|
||||
self.outputs.write(metadata::header()).await?;
|
||||
self.position = metadata::HEADER_SIZE;
|
||||
}
|
||||
for (column, chunk) in block.columns.iter_mut().zip(chunks) {
|
||||
column.payload.offset = self.position;
|
||||
self.position = column.payload.end()?;
|
||||
self.outputs.write(chunk).await?;
|
||||
}
|
||||
if self.leaves.is_empty() || self.leaf_bytes + descriptor.len() > self.options.page_bytes {
|
||||
self.leaves.push(Vec::new());
|
||||
self.leaf_bytes = 12;
|
||||
}
|
||||
self.leaf_bytes += descriptor.len();
|
||||
self.descriptor_bytes += descriptor.len();
|
||||
self.part_rows += u64::from(block.rows);
|
||||
self.leaves
|
||||
.last_mut()
|
||||
.ok_or_else(|| invalid("missing series leaf"))?
|
||||
.push(block);
|
||||
if self.position >= self.options.part_bytes
|
||||
|| self.descriptor_bytes >= self.options.metadata_bytes
|
||||
{
|
||||
self.finish_part().await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn write_page(
|
||||
&mut self,
|
||||
page: Page,
|
||||
first: MetricSeriesId,
|
||||
last: MetricSeriesId,
|
||||
) -> Result<PageReference> {
|
||||
let position = self.position;
|
||||
let (payload, bytes) = pack(
|
||||
&page.encode(),
|
||||
position,
|
||||
1,
|
||||
self.options.compression,
|
||||
self.options.zstd_level,
|
||||
true,
|
||||
)?;
|
||||
self.position = payload.end()?;
|
||||
self.outputs.write(bytes).await?;
|
||||
Ok(PageReference {
|
||||
first,
|
||||
last,
|
||||
payload,
|
||||
})
|
||||
}
|
||||
|
||||
async fn finish_part(&mut self) -> Result<()> {
|
||||
if self.leaves.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let payload_end = self.position;
|
||||
let mut children = Vec::new();
|
||||
for leaf in std::mem::take(&mut self.leaves) {
|
||||
let first = leaf
|
||||
.first()
|
||||
.ok_or_else(|| invalid("empty series leaf"))?
|
||||
.first_id();
|
||||
let last = leaf
|
||||
.last()
|
||||
.ok_or_else(|| invalid("empty series leaf"))?
|
||||
.last_id();
|
||||
children.push(self.write_page(Page::Leaf(leaf), first, last).await?);
|
||||
}
|
||||
let first = children[0].first;
|
||||
let last = children[children.len() - 1].last;
|
||||
let depth = if children.len() == 1 { 1 } else { 2 };
|
||||
let root = if depth == 1 {
|
||||
children.remove(0)
|
||||
} else {
|
||||
self.write_page(Page::Internal(children), first, last)
|
||||
.await?
|
||||
};
|
||||
let file_length = self
|
||||
.position
|
||||
.checked_add(metadata::TRAILER_SIZE)
|
||||
.ok_or_else(|| invalid("series file length overflow"))?;
|
||||
self.outputs
|
||||
.write(
|
||||
Trailer {
|
||||
root,
|
||||
depth,
|
||||
file_length,
|
||||
payload_end,
|
||||
}
|
||||
.encode(),
|
||||
)
|
||||
.await?;
|
||||
self.outputs.close().await?;
|
||||
self.manifest.files.push(Part {
|
||||
part_id: u32::try_from(self.manifest.files.len())
|
||||
.map_err(|_| invalid("too many series parts"))?,
|
||||
file_size: file_length,
|
||||
num_series: self.part_rows,
|
||||
first,
|
||||
last,
|
||||
});
|
||||
self.position = 0;
|
||||
self.part_rows = 0;
|
||||
self.descriptor_bytes = 0;
|
||||
self.leaf_bytes = 12;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn complete(&mut self) -> Result<ArtifactMetadata> {
|
||||
require(!self.failed, "series writer failed previously")?;
|
||||
self.push_current().await?;
|
||||
self.flush_block().await?;
|
||||
self.finish_part().await?;
|
||||
let bytes = serde_json::to_vec(&self.manifest).context(SerdeJsonSnafu)?;
|
||||
require(
|
||||
bytes.len() <= MANIFEST_LIMIT,
|
||||
"series manifest exceeds size limit",
|
||||
)?;
|
||||
let mut file_size = bytes.len() as u64;
|
||||
let manifest_size = file_size;
|
||||
let mut objects = Vec::with_capacity(self.manifest.files.len() + 1);
|
||||
for part in &self.manifest.files {
|
||||
objects.push(ArtifactObject {
|
||||
path: self.manifest.part_path(part.part_id),
|
||||
file_size: part.file_size,
|
||||
});
|
||||
file_size = file_size
|
||||
.checked_add(part.file_size)
|
||||
.ok_or_else(|| invalid("series artifact length overflow"))?;
|
||||
}
|
||||
objects.push(ArtifactObject {
|
||||
path: self.manifest.path(),
|
||||
file_size: manifest_size,
|
||||
});
|
||||
self.outputs.open(self.manifest.path()).await?;
|
||||
self.outputs.write(bytes).await?;
|
||||
self.outputs.close().await?;
|
||||
self.outputs.committed = true;
|
||||
Ok(ArtifactMetadata {
|
||||
format: ArtifactFormat::ColumnarSeries,
|
||||
generation: self.manifest.generation,
|
||||
objects,
|
||||
file_size,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl FormatWriter for ColumnarWriter {
|
||||
async fn write(&mut self, batch: &RecordBatch) -> Result<()> {
|
||||
require(!self.failed, "series writer failed previously")?;
|
||||
// A cancelled write must not allow a later finish to publish partial input.
|
||||
self.failed = true;
|
||||
let result = self.write_batch(batch).await;
|
||||
if result.is_ok() {
|
||||
self.failed = false;
|
||||
}
|
||||
result
|
||||
}
|
||||
async fn finish(mut self: Box<Self>) -> Result<ArtifactMetadata> {
|
||||
let result = self.complete().await;
|
||||
if result.is_err()
|
||||
&& let Err(error) = self.outputs.cleanup().await
|
||||
{
|
||||
common_telemetry::warn!(error; "Cleaning failed series build");
|
||||
}
|
||||
result
|
||||
}
|
||||
async fn abort(mut self: Box<Self>) -> Result<()> {
|
||||
self.outputs.cleanup().await
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_block(
|
||||
rows: Vec<Row>,
|
||||
tags: &[(u32, String)],
|
||||
options: &ColumnarSeriesFormat,
|
||||
) -> Result<(Block, Vec<Vec<u8>>)> {
|
||||
let first = rows
|
||||
.first()
|
||||
.ok_or_else(|| invalid("empty series block"))?
|
||||
.id;
|
||||
let last = rows.last().ok_or_else(|| invalid("empty series block"))?.id;
|
||||
let mut block = Block {
|
||||
table_id: first.table_id,
|
||||
first: first.tsid,
|
||||
last: last.tsid,
|
||||
rows: rows.len() as u32,
|
||||
columns: Vec::new(),
|
||||
};
|
||||
let mut chunks = Vec::new();
|
||||
for (name, values) in [
|
||||
(
|
||||
TSID_COLUMN,
|
||||
rows.iter().map(|r| r.id.tsid).collect::<Vec<_>>(),
|
||||
),
|
||||
(ROW_COUNT_COLUMN, rows.iter().map(|r| r.count).collect()),
|
||||
] {
|
||||
let (payload, bytes) = pack(
|
||||
&encode_integers(&values),
|
||||
0,
|
||||
codec::INTEGER,
|
||||
options.compression,
|
||||
options.zstd_level,
|
||||
false,
|
||||
)?;
|
||||
block.columns.push(Column {
|
||||
name: name.into(),
|
||||
payload,
|
||||
null_count: 0,
|
||||
bounds: None,
|
||||
});
|
||||
chunks.push(bytes);
|
||||
}
|
||||
for (index, (_, name)) in tags.iter().enumerate() {
|
||||
let values: Vec<_> = rows.iter().map(|row| row.tags[index].as_deref()).collect();
|
||||
let null_count = values.iter().filter(|v| v.is_none()).count() as u32;
|
||||
if null_count == block.rows {
|
||||
continue;
|
||||
}
|
||||
let min = values
|
||||
.iter()
|
||||
.flatten()
|
||||
.min()
|
||||
.copied()
|
||||
.ok_or_else(|| invalid("missing tag minimum"))?;
|
||||
let max = values
|
||||
.iter()
|
||||
.flatten()
|
||||
.max()
|
||||
.copied()
|
||||
.ok_or_else(|| invalid("missing tag maximum"))?;
|
||||
let (payload, bytes) = pack(
|
||||
&encode_dictionary(&values)?,
|
||||
0,
|
||||
codec::DICTIONARY,
|
||||
options.compression,
|
||||
options.zstd_level,
|
||||
false,
|
||||
)?;
|
||||
block.columns.push(Column {
|
||||
name: name.clone(),
|
||||
payload,
|
||||
null_count,
|
||||
bounds: Some((min.to_owned(), max.to_owned())),
|
||||
});
|
||||
chunks.push(bytes);
|
||||
}
|
||||
Ok((block, chunks))
|
||||
}
|
||||
@@ -14,11 +14,15 @@
|
||||
|
||||
//! Storage-format boundaries. Encodings stay outside reconciliation and source scheduling.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::fmt::Debug;
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use common_recordbatch::filter::SimpleFilterEvaluator;
|
||||
use datatypes::arrow::datatypes::SchemaRef;
|
||||
use datatypes::arrow::record_batch::RecordBatch;
|
||||
use futures::stream::BoxStream;
|
||||
use object_store::ObjectStore;
|
||||
use store_api::metadata::RegionMetadataRef;
|
||||
use store_api::storage::FileId;
|
||||
@@ -26,6 +30,7 @@ use store_api::storage::FileId;
|
||||
use crate::error::Result;
|
||||
use crate::series_index::artifact::{ArtifactFormat, ArtifactMetadata};
|
||||
use crate::series_index::catalog::SeriesIndexEntry;
|
||||
use crate::series_index::{MetricSeriesId, SeriesIndexFileHandle};
|
||||
|
||||
/// Writers own partial outputs until finish succeeds; abort removes every partial object.
|
||||
/// A failed finish must clean up its output before returning; implementations also own
|
||||
@@ -37,10 +42,43 @@ pub(crate) trait FormatWriter: Send {
|
||||
async fn abort(self: Box<Self>) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Builds series-index artifacts; read interfaces are deferred.
|
||||
/// A read has independent traversal state and pins its immutable artifact.
|
||||
#[allow(
|
||||
dead_code,
|
||||
reason = "Query adapter will consume the format reader separately."
|
||||
)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub(crate) struct SeriesReadRequest {
|
||||
pub(crate) projection: Vec<String>,
|
||||
pub(crate) table_id: Option<u32>,
|
||||
pub(crate) ids: Option<BTreeSet<MetricSeriesId>>,
|
||||
pub(crate) filters: Vec<SimpleFilterEvaluator>,
|
||||
}
|
||||
|
||||
#[allow(
|
||||
dead_code,
|
||||
reason = "Query adapter will consume the format reader separately."
|
||||
)]
|
||||
pub(crate) trait FormatReader: Send + Sync {
|
||||
fn schema(&self) -> SchemaRef;
|
||||
fn read(&self, request: SeriesReadRequest) -> Result<BoxStream<'static, Result<RecordBatch>>>;
|
||||
}
|
||||
|
||||
/// Builds and opens immutable series-index artifacts.
|
||||
#[async_trait]
|
||||
pub(crate) trait SeriesIndexFormat: Debug + Send + Sync {
|
||||
fn format(&self) -> ArtifactFormat;
|
||||
#[allow(dead_code, reason = "Query adapter is deferred.")]
|
||||
async fn reader(
|
||||
&self,
|
||||
_store: ObjectStore,
|
||||
_handle: SeriesIndexFileHandle,
|
||||
) -> Result<Box<dyn FormatReader>> {
|
||||
crate::error::UnsupportedOperationSnafu {
|
||||
err_msg: "This series format has no reader",
|
||||
}
|
||||
.fail()
|
||||
}
|
||||
async fn writer(
|
||||
&self,
|
||||
metadata: RegionMetadataRef,
|
||||
|
||||
@@ -30,7 +30,7 @@ use store_api::storage::FileId;
|
||||
|
||||
use crate::config::MitoConfig;
|
||||
use crate::error::{OpenDalSnafu, Result, UnexpectedSnafu};
|
||||
use crate::series_index::artifact::{ArtifactFormat, ArtifactMetadata};
|
||||
use crate::series_index::artifact::{ArtifactFormat, ArtifactMetadata, ArtifactObject};
|
||||
use crate::series_index::catalog::{SeriesIndexEntry, load_version_control};
|
||||
use crate::series_index::format::{FormatWriter, SamplesFormat, SeriesFormats, SeriesIndexFormat};
|
||||
use crate::series_index::maintenance::reconcile_series_indexes;
|
||||
@@ -64,8 +64,11 @@ impl FormatWriter for FakeWriter {
|
||||
Ok(self.artifact)
|
||||
}
|
||||
async fn abort(self: Box<Self>) -> Result<()> {
|
||||
for path in &self.artifact.objects {
|
||||
self.store.delete(path).await.context(OpenDalSnafu)?;
|
||||
for object in &self.artifact.objects {
|
||||
self.store
|
||||
.delete(&object.path)
|
||||
.await
|
||||
.context(OpenDalSnafu)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -92,7 +95,10 @@ impl SamplesFormat for FakeSamples {
|
||||
artifact: ArtifactMetadata {
|
||||
format: self.format(),
|
||||
generation,
|
||||
objects,
|
||||
objects: objects
|
||||
.into_iter()
|
||||
.map(|path| ArtifactObject { path, file_size: 4 })
|
||||
.collect(),
|
||||
file_size: 8,
|
||||
},
|
||||
fail: self.fail.clone(),
|
||||
@@ -156,7 +162,7 @@ async fn format_skeleton_build_reopen_and_retire() {
|
||||
// Rebuilding a missing sidecar replaces its generation but retains old readers' files.
|
||||
let old = snapshot.samples.values().next().unwrap().clone();
|
||||
store
|
||||
.delete(&old.entry().artifact.objects[0])
|
||||
.delete(&old.entry().artifact.objects[0].path)
|
||||
.await
|
||||
.unwrap();
|
||||
let stats = reconcile_series_indexes(
|
||||
@@ -244,7 +250,7 @@ impl SeriesIndexFormat for FakeIndex {
|
||||
artifact: ArtifactMetadata {
|
||||
format: self.format(),
|
||||
generation: entry.index_uuid,
|
||||
objects: vec![path],
|
||||
objects: vec![ArtifactObject { path, file_size: 4 }],
|
||||
file_size: 4,
|
||||
},
|
||||
fail: Arc::new(AtomicBool::new(false)),
|
||||
@@ -321,7 +327,7 @@ async fn format_index_rebuild_is_scoped_and_scans_fall_back_to_ssts() {
|
||||
|
||||
let missing = before.series_indexes.values().next().unwrap();
|
||||
store
|
||||
.delete(&missing.entry().artifact.as_ref().unwrap().objects[0])
|
||||
.delete(&missing.entry().artifact.as_ref().unwrap().objects[0].path)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(1, reconcile().await.unwrap().built_series);
|
||||
@@ -397,7 +403,7 @@ async fn format_catalog_failure_preserves_published_artifacts() {
|
||||
let catalog = store.read(&catalog_path).await.unwrap().to_bytes();
|
||||
let old = before.samples.values().next().unwrap();
|
||||
store
|
||||
.delete(&old.entry().artifact.objects[0])
|
||||
.delete(&old.entry().artifact.objects[0].path)
|
||||
.await
|
||||
.unwrap();
|
||||
let layer = MockLayerBuilder::default()
|
||||
@@ -427,11 +433,13 @@ async fn format_catalog_failure_preserves_published_artifacts() {
|
||||
assert_eq!(catalog, store.read(&catalog_path).await.unwrap().to_bytes());
|
||||
let cleanup = purges.try_recv().unwrap().objects.unwrap();
|
||||
assert_eq!(2, cleanup.len());
|
||||
assert!(
|
||||
cleanup
|
||||
assert!(cleanup.iter().all(|path| {
|
||||
!old.entry()
|
||||
.artifact
|
||||
.objects
|
||||
.iter()
|
||||
.all(|path| !old.entry().artifact.objects.contains(path))
|
||||
);
|
||||
.any(|object| &object.path == path)
|
||||
}));
|
||||
assert!(purges.try_recv().is_err());
|
||||
let reopened = load_version_control(&store, region.region_id, &purger).await;
|
||||
assert_eq!(
|
||||
|
||||
@@ -235,7 +235,7 @@ async fn build_index_version(
|
||||
if reusable && let Some(artifact) = &entry.artifact {
|
||||
for object in &artifact.objects {
|
||||
reusable &= store
|
||||
.exists(object)
|
||||
.exists(&object.path)
|
||||
.await
|
||||
.context(crate::error::OpenDalSnafu)?;
|
||||
}
|
||||
@@ -288,9 +288,9 @@ async fn build_index_version(
|
||||
let reusable = if let Some(handle) = samples.get(&id) {
|
||||
let artifact = &handle.entry().artifact;
|
||||
let mut exists = artifact.format == backend.format();
|
||||
for path in &artifact.objects {
|
||||
for object in &artifact.objects {
|
||||
exists &= store
|
||||
.exists(path)
|
||||
.exists(&object.path)
|
||||
.await
|
||||
.context(crate::error::OpenDalSnafu)?;
|
||||
}
|
||||
|
||||
@@ -83,8 +83,14 @@ impl Drop for SeriesIndexFileHandleInner {
|
||||
fn drop(&mut self) {
|
||||
if self.deleted.load(Ordering::Acquire) {
|
||||
if let Some(artifact) = &self.entry.artifact {
|
||||
self.purger
|
||||
.purge_objects(self.file_id, artifact.objects.clone());
|
||||
self.purger.purge_objects(
|
||||
self.file_id,
|
||||
artifact
|
||||
.objects
|
||||
.iter()
|
||||
.map(|object| object.path.clone())
|
||||
.collect(),
|
||||
);
|
||||
} else {
|
||||
self.purger.purge(PurgeRequest {
|
||||
file_id: self.file_id,
|
||||
|
||||
@@ -477,7 +477,7 @@ fn time_index_unit(metadata: &RegionMetadataRef) -> Result<TimeUnit> {
|
||||
.unit())
|
||||
}
|
||||
|
||||
fn validate_metadata(metadata: &RegionMetadataRef) -> Result<()> {
|
||||
pub(crate) fn validate_metadata(metadata: &RegionMetadataRef) -> Result<()> {
|
||||
for column in &metadata.column_metadatas {
|
||||
ensure!(
|
||||
!matches!(
|
||||
@@ -540,7 +540,7 @@ fn validate_metadata(metadata: &RegionMetadataRef) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn tag_columns(metadata: &RegionMetadataRef) -> Vec<(ColumnId, String)> {
|
||||
pub(crate) fn tag_columns(metadata: &RegionMetadataRef) -> Vec<(ColumnId, String)> {
|
||||
metadata
|
||||
.primary_key_columns()
|
||||
.filter(|column| !is_reserved_column(column.column_id))
|
||||
@@ -595,16 +595,9 @@ fn decode_primary_key(
|
||||
offsets: &mut SparseOffsetsCache,
|
||||
buf: &mut Vec<u8>,
|
||||
) -> Result<SeriesIndexRow> {
|
||||
let mut view = SparsePrimaryKeyView::new(primary_key, offsets).context(DecodeSnafu)?;
|
||||
|
||||
let table_id = view.table_id();
|
||||
let tsid = view.tsid();
|
||||
|
||||
let mut tags = Vec::with_capacity(tag_columns.len());
|
||||
for (column_id, _) in tag_columns {
|
||||
let value = view.label(*column_id, buf).context(DecodeSnafu)?;
|
||||
tags.push(value.map(str::to_owned));
|
||||
}
|
||||
let (id, tags) = decode_series_identity(primary_key, tag_columns, offsets, buf)?;
|
||||
let table_id = id.table_id;
|
||||
let tsid = id.tsid;
|
||||
|
||||
Ok(SeriesIndexRow {
|
||||
min_ts,
|
||||
@@ -616,6 +609,29 @@ fn decode_primary_key(
|
||||
})
|
||||
}
|
||||
|
||||
/// Shared sparse-key extraction, independent of timestamp aggregation.
|
||||
pub(crate) fn decode_series_identity(
|
||||
primary_key: &[u8],
|
||||
tag_columns: &[(ColumnId, String)],
|
||||
offsets: &mut SparseOffsetsCache,
|
||||
buf: &mut Vec<u8>,
|
||||
) -> Result<(crate::series_index::MetricSeriesId, Vec<Option<String>>)> {
|
||||
let mut view = SparsePrimaryKeyView::new(primary_key, offsets).context(DecodeSnafu)?;
|
||||
let id = crate::series_index::MetricSeriesId {
|
||||
table_id: view.table_id(),
|
||||
tsid: view.tsid(),
|
||||
};
|
||||
let mut tags = Vec::with_capacity(tag_columns.len());
|
||||
for (column_id, _) in tag_columns {
|
||||
tags.push(
|
||||
view.label(*column_id, buf)
|
||||
.context(DecodeSnafu)?
|
||||
.map(str::to_owned),
|
||||
);
|
||||
}
|
||||
Ok((id, tags))
|
||||
}
|
||||
|
||||
/// Builds a timestamp array in `unit` from raw i64 values.
|
||||
fn ts_array(timestamps: impl Iterator<Item = i64>, unit: TimeUnit) -> ArrayRef {
|
||||
match unit {
|
||||
|
||||
+85
-91
@@ -16,9 +16,14 @@
|
||||
//! is keyed by absolute grid index, never by local row number; sparse state and trimmed output
|
||||
//! do not expand the full grid.
|
||||
|
||||
mod state;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
pub use state::{
|
||||
AccumulatorState, FloatMerger, PartialAggregate, PartialGroup, StateField, merge_count_values,
|
||||
};
|
||||
|
||||
use std::cmp::Reverse;
|
||||
use std::collections::hash_map::Entry;
|
||||
use std::collections::{BTreeMap, BinaryHeap, HashMap};
|
||||
@@ -28,12 +33,14 @@ use datafusion::error::{DataFusionError, Result};
|
||||
use datatypes::series::MetricName;
|
||||
use datatypes::series::samples::{SampleRef, Series, SeriesSchema};
|
||||
|
||||
use crate::go_format::format_float_f;
|
||||
use crate::grid_series::{GridSeriesBuilder, external};
|
||||
use crate::map::{drop_empty_labels, validate_float_series, validate_scalar};
|
||||
use crate::histogram_eval::histogram_string;
|
||||
use crate::map::{drop_empty_labels, validate_float_series, validate_grid_series, validate_scalar};
|
||||
use crate::range_eval::quantile;
|
||||
|
||||
/// Float aggregations without a parameter.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum AggregateOp {
|
||||
Sum,
|
||||
Count,
|
||||
@@ -139,30 +146,39 @@ impl FloatAggregator {
|
||||
|
||||
/// The result of every group, in the order the groups first appeared.
|
||||
pub fn finish(self) -> Result<Vec<Series>> {
|
||||
let width = self.schema.fields.len();
|
||||
let mut builder = GridSeriesBuilder::try_new(self.schema)?;
|
||||
let mut output = Vec::with_capacity(self.groups.len());
|
||||
let mut row = Vec::with_capacity(width);
|
||||
for group in self.groups {
|
||||
// One builder serves every group, so each result series starts explicitly.
|
||||
builder.begin_series();
|
||||
for (&block, states) in &group.blocks {
|
||||
// The last block can extend past the grid; only points with a sample are pushed.
|
||||
for (offset, point) in states.chunks_exact(width).enumerate() {
|
||||
if !point.iter().any(|state| state.seen) {
|
||||
continue;
|
||||
}
|
||||
row.clear();
|
||||
row.extend(point.iter().map(|state| state.finish(self.op)));
|
||||
builder.push(block * BLOCK_POINTS + offset, &row)?;
|
||||
}
|
||||
}
|
||||
output.extend(builder.finish(group.metric)?);
|
||||
}
|
||||
Ok(output)
|
||||
finish_groups(self.schema, self.op, self.groups)
|
||||
}
|
||||
}
|
||||
|
||||
/// Turns the state of `groups` into result series, in group order.
|
||||
fn finish_groups(
|
||||
schema: Arc<SeriesSchema>,
|
||||
op: AggregateOp,
|
||||
groups: Vec<Group>,
|
||||
) -> Result<Vec<Series>> {
|
||||
let width = schema.fields.len();
|
||||
let mut builder = GridSeriesBuilder::try_new(schema)?;
|
||||
let mut output = Vec::with_capacity(groups.len());
|
||||
let mut row = Vec::with_capacity(width);
|
||||
for group in groups {
|
||||
// One builder serves every group, so each result series starts explicitly.
|
||||
builder.begin_series();
|
||||
for (&block, states) in &group.blocks {
|
||||
// The last block can extend past the grid; only points with a sample are pushed.
|
||||
for (offset, point) in states.chunks_exact(width).enumerate() {
|
||||
if !point.iter().any(|state| state.seen) {
|
||||
continue;
|
||||
}
|
||||
row.clear();
|
||||
row.extend(point.iter().map(|state| state.finish(op)));
|
||||
builder.push(block * BLOCK_POINTS + offset, &row)?;
|
||||
}
|
||||
}
|
||||
output.extend(builder.finish(group.metric)?);
|
||||
}
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// `quantile(q, v)`, with `q` read at each grid point from an unlabelled single-field scalar on
|
||||
/// the same grid, as Prometheus evaluates the parameter at every step. A point where `q` has no
|
||||
/// value has no result; `q` outside `[0, 1]` or NaN gives what Prometheus' `quantile` gives.
|
||||
@@ -262,7 +278,8 @@ pub fn aggregate_quantile(
|
||||
|
||||
/// `count_values(label, v)`: at each grid point, the number of samples in each group of series
|
||||
/// that share a value, with the value written to `label` in Prometheus' `FormatFloat(v, 'f', -1)`
|
||||
/// form. With `by`, `label` is part of the grouping; with `without`, it is dropped if listed.
|
||||
/// form, or for a native histogram as its `FloatHistogram.String`. With `by`, `label` is part of
|
||||
/// the grouping; with `without`, it is dropped if listed.
|
||||
pub fn count_values(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
input: &[Series],
|
||||
@@ -283,7 +300,11 @@ pub fn count_values(
|
||||
}
|
||||
without => without,
|
||||
};
|
||||
validate_input(schema, input, checked)?;
|
||||
// Unlike the other aggregations, this one reads histogram samples too.
|
||||
validate_grouping(schema, checked)?;
|
||||
for series in input {
|
||||
validate_grid_series(schema, series)?;
|
||||
}
|
||||
if label.is_empty() {
|
||||
return Err(DataFusionError::Execution(format!(
|
||||
"invalid label name {label:?}"
|
||||
@@ -301,23 +322,29 @@ pub fn count_values(
|
||||
let mut group_index = HashMap::new();
|
||||
// Counts are kept per point that has one, not in blocks: a gauge with distinct values makes
|
||||
// about one group per sample.
|
||||
let mut groups: Vec<(MetricName, BTreeMap<usize, Vec<f64>>)> = Vec::new();
|
||||
let mut groups: Counts = Vec::new();
|
||||
for series in input {
|
||||
// Equal bits format to the same label value, so each distinct value of a series
|
||||
// resolves its group once.
|
||||
let mut group_of_value: HashMap<u64, usize> = HashMap::new();
|
||||
// resolves its group once. A histogram is keyed by the text it is written as.
|
||||
let mut group_of_value: HashMap<Value, usize> = HashMap::new();
|
||||
for (field_index, field) in series.fields.iter().enumerate() {
|
||||
for ((point, _), sample) in series.time.positions().zip(field.iter()) {
|
||||
let SampleRef::Float(value) = sample else {
|
||||
continue;
|
||||
let key = match sample {
|
||||
SampleRef::Missing => continue,
|
||||
SampleRef::Float(value) => Value::Float(value.to_bits()),
|
||||
SampleRef::Histogram(h) => Value::Histogram(histogram_string(&h)?),
|
||||
};
|
||||
let group = match group_of_value.entry(value.to_bits()) {
|
||||
let group = match group_of_value.entry(key) {
|
||||
Entry::Occupied(entry) => *entry.get(),
|
||||
Entry::Vacant(entry) => {
|
||||
// Prometheus writes a float in Go's `'f'` form and a histogram as
|
||||
// `FloatHistogram.String` does.
|
||||
let text = match entry.key() {
|
||||
Value::Float(bits) => format_float_f(f64::from_bits(*bits)),
|
||||
Value::Histogram(text) => text.clone(),
|
||||
};
|
||||
let mut metric = series.metric.clone();
|
||||
metric
|
||||
.set(label, Arc::from(format_value(value)))
|
||||
.map_err(external)?;
|
||||
metric.set(label, Arc::from(text)).map_err(external)?;
|
||||
let metric = grouping.apply(&metric)?;
|
||||
let index = *group_index.entry(label_key(&metric)).or_insert_with(|| {
|
||||
groups.push((metric, BTreeMap::new()));
|
||||
@@ -333,9 +360,16 @@ pub fn count_values(
|
||||
}
|
||||
}
|
||||
}
|
||||
count_series(schema, groups)
|
||||
}
|
||||
|
||||
/// Counts per point and field: one series per label set, a count of 0 being no sample.
|
||||
type Counts = Vec<(MetricName, BTreeMap<usize, Vec<f64>>)>;
|
||||
|
||||
fn count_series(schema: &Arc<SeriesSchema>, groups: Counts) -> Result<Vec<Series>> {
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone())?;
|
||||
let mut output = Vec::with_capacity(groups.len());
|
||||
let mut row = Vec::with_capacity(width);
|
||||
let mut row = Vec::with_capacity(schema.fields.len());
|
||||
for (metric, counts) in groups {
|
||||
builder.begin_series();
|
||||
for (point, counts) in counts {
|
||||
@@ -348,69 +382,29 @@ pub fn count_values(
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// Go's `strconv.FormatFloat(value, 'f', -1, 64)`: the fewest significant digits that round-trip,
|
||||
/// in positional notation. When two candidates of that length are equally close, Go takes the
|
||||
/// even one while Rust's shortest formatting rounds up, so the digits are recomputed with Rust's
|
||||
/// exact formatting, which rounds half to even. The recomputed digits only count when they still
|
||||
/// parse back to `value`: just above a power of two the lower neighbour is half as far away, so
|
||||
/// the even candidate can name that neighbour, and only the other one round-trips.
|
||||
fn format_value(value: f64) -> String {
|
||||
match value {
|
||||
f64::INFINITY => return "+Inf".to_string(),
|
||||
f64::NEG_INFINITY => return "-Inf".to_string(),
|
||||
_ if value.is_nan() => return "NaN".to_string(),
|
||||
_ => {}
|
||||
}
|
||||
let shortest = format!("{value:e}");
|
||||
let mantissa = shortest.split_once('e').expect("exponent format").0;
|
||||
let significant = mantissa.chars().filter(char::is_ascii_digit).count();
|
||||
let exact = format!("{value:.precision$e}", precision = significant - 1);
|
||||
let digits = if exact.parse::<f64>() == Ok(value) {
|
||||
&exact
|
||||
} else {
|
||||
&shortest
|
||||
};
|
||||
let (mantissa, exponent) = digits.split_once('e').expect("exponent format");
|
||||
let exponent: i64 = exponent.parse().expect("integer exponent");
|
||||
let (sign, mantissa) = match mantissa.strip_prefix('-') {
|
||||
Some(unsigned) => ("-", unsigned),
|
||||
None => ("", mantissa),
|
||||
};
|
||||
let digits: String = mantissa.chars().filter(char::is_ascii_digit).collect();
|
||||
// The decimal point goes after `exponent + 1` digits.
|
||||
let point = exponent + 1;
|
||||
let mut out = String::from(sign);
|
||||
if point <= 0 {
|
||||
out.push_str("0.");
|
||||
out.extend(std::iter::repeat_n('0', (-point) as usize));
|
||||
out.push_str(digits.trim_end_matches('0'));
|
||||
} else if point as usize >= digits.len() {
|
||||
out.push_str(&digits);
|
||||
out.extend(std::iter::repeat_n('0', point as usize - digits.len()));
|
||||
} else {
|
||||
let (whole, fraction) = digits.split_at(point as usize);
|
||||
out.push_str(whole);
|
||||
let fraction = fraction.trim_end_matches('0');
|
||||
if !fraction.is_empty() {
|
||||
out.push('.');
|
||||
out.push_str(fraction);
|
||||
}
|
||||
}
|
||||
out
|
||||
/// The value of a `count_values` sample, as it is grouped.
|
||||
#[derive(PartialEq, Eq, Hash)]
|
||||
enum Value {
|
||||
Float(u64),
|
||||
Histogram(String),
|
||||
}
|
||||
|
||||
/// Schema validation is repeated because Series exposes its fields publicly.
|
||||
fn validate_input(schema: &SeriesSchema, input: &[Series], grouping: Grouping<'_>) -> Result<()> {
|
||||
validate_grouping(schema, grouping)?;
|
||||
for series in input {
|
||||
validate_float_series(schema, series)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_grouping(schema: &SeriesSchema, grouping: Grouping<'_>) -> Result<()> {
|
||||
if schema.eval_grid.is_none() {
|
||||
return Err(DataFusionError::Execution(
|
||||
"Aggregate requires an evaluated grid".into(),
|
||||
));
|
||||
}
|
||||
grouping.validate(schema)?;
|
||||
for series in input {
|
||||
validate_float_series(schema, series)?;
|
||||
}
|
||||
Ok(())
|
||||
grouping.validate(schema)
|
||||
}
|
||||
|
||||
impl Grouping<'_> {
|
||||
@@ -530,7 +524,6 @@ impl FloatState {
|
||||
*self = Self {
|
||||
seen: true,
|
||||
value: match op {
|
||||
AggregateOp::Group => 1.0,
|
||||
// The running sum of squared deviations; Prometheus starts it at NaN for a
|
||||
// non-finite first sample.
|
||||
AggregateOp::Stddev | AggregateOp::Stdvar if !sample.is_finite() => f64::NAN,
|
||||
@@ -597,7 +590,8 @@ impl FloatState {
|
||||
AggregateOp::Avg if self.incremental_mean => self.mean + self.compensation,
|
||||
AggregateOp::Avg => self.value / self.count + self.compensation / self.count,
|
||||
AggregateOp::Count => self.count,
|
||||
AggregateOp::Min | AggregateOp::Max | AggregateOp::Group => self.value,
|
||||
AggregateOp::Min | AggregateOp::Max => self.value,
|
||||
AggregateOp::Group => 1.0,
|
||||
AggregateOp::Stdvar => self.value / self.count,
|
||||
AggregateOp::Stddev => (self.value / self.count).sqrt(),
|
||||
})
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Partial aggregation: the state one execution partition folds its series into, and the merge
|
||||
//! of such states into the final result.
|
||||
//!
|
||||
//! A merge regroups the operations of a serial fold, so floating-point results differ from
|
||||
//! [`aggregate_float`](crate::aggregate::aggregate_float) by rounding. The same reassociation can
|
||||
//! also move an overflow of finite values: a serial `sum` of `[1e308, 1e308, -1e308]` is `+Inf`,
|
||||
//! while the merge of `[1e308]` and `[1e308, -1e308]` is finite. `stddev` and `stdvar` differ
|
||||
//! more where the samples differ by little more than the rounding of their mean: Prometheus'
|
||||
//! uncompensated Welford update rounds the mean on every sample, and no pairwise merge reproduces
|
||||
//! those roundings. For `[1e16]` and `[1e16 + 2]` Prometheus gives a variance of 2 and the merge
|
||||
//! gives the exact 1. Explicit NaN and infinite inputs follow the same rules as the serial fold. Where the serial fold keeps the first of two equal
|
||||
//! values, as `min` and `max` do for `0` and `-0`, a merge keeps the one merged first.
|
||||
|
||||
use std::collections::hash_map::Entry;
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::sync::Arc;
|
||||
|
||||
use datafusion::error::{DataFusionError, Result};
|
||||
use datatypes::series::MetricName;
|
||||
use datatypes::series::samples::{SampleRef, Series, SeriesSchema};
|
||||
|
||||
use crate::aggregate::{
|
||||
AggregateOp, BLOCK_POINTS, Counts, FloatAggregator, FloatState, Group, LabelKey, count_series,
|
||||
finish_groups, kahan_inc, label_key,
|
||||
};
|
||||
use crate::map::validate_float_series;
|
||||
|
||||
/// A component of [`AccumulatorState`].
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum StateField {
|
||||
/// The running sum (`sum`, and `avg` until it switches to a mean), the extreme (`min`,
|
||||
/// `max`), or the sum of squared deviations (`stddev`, `stdvar`).
|
||||
Value,
|
||||
/// The running mean (`avg` once switched, `stddev`, `stdvar`).
|
||||
Mean,
|
||||
/// The Kahan compensation of `Value`, or of `Mean` once `avg` has switched.
|
||||
Compensation,
|
||||
/// The number of samples folded in.
|
||||
Count,
|
||||
/// Whether `avg` has switched from a sum to a mean, which it does when the sum overflows.
|
||||
IncrementalMean,
|
||||
}
|
||||
|
||||
impl AggregateOp {
|
||||
/// The components the state of this aggregation carries; the others are unused and need not
|
||||
/// be stored. `group` has none: a state that exists has a value of 1.
|
||||
pub fn state_fields(self) -> &'static [StateField] {
|
||||
use StateField::*;
|
||||
match self {
|
||||
Self::Sum => &[Value, Compensation],
|
||||
Self::Count => &[Count],
|
||||
Self::Avg => &[Value, Mean, Compensation, Count, IncrementalMean],
|
||||
Self::Min | Self::Max => &[Value],
|
||||
Self::Group => &[],
|
||||
Self::Stddev | Self::Stdvar => &[Value, Mean, Count],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The state of one field of one group at one grid point.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq)]
|
||||
pub struct AccumulatorState {
|
||||
pub value: f64,
|
||||
pub mean: f64,
|
||||
pub compensation: f64,
|
||||
pub count: f64,
|
||||
pub incremental_mean: bool,
|
||||
}
|
||||
|
||||
/// The state of one group: its points in ascending order, each with one state per field, `None`
|
||||
/// where no sample of that field was folded in.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PartialGroup {
|
||||
pub metric: MetricName,
|
||||
pub points: Vec<(usize, Vec<Option<AccumulatorState>>)>,
|
||||
}
|
||||
|
||||
/// The state of one execution partition, with its groups in the order they first appeared.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PartialAggregate {
|
||||
pub op: AggregateOp,
|
||||
pub groups: Vec<PartialGroup>,
|
||||
}
|
||||
|
||||
impl FloatAggregator {
|
||||
/// The state folded so far, for a merge with the state of other partitions.
|
||||
pub fn finish_partial(self) -> PartialAggregate {
|
||||
let width = self.schema.fields.len();
|
||||
let groups = self
|
||||
.groups
|
||||
.into_iter()
|
||||
.map(|group| {
|
||||
let mut points = Vec::new();
|
||||
for (&block, states) in &group.blocks {
|
||||
for (offset, point) in states.chunks_exact(width).enumerate() {
|
||||
if point.iter().any(|state| state.seen) {
|
||||
points.push((
|
||||
block * BLOCK_POINTS + offset,
|
||||
point.iter().map(|state| state.partial()).collect(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
PartialGroup {
|
||||
metric: group.metric,
|
||||
points,
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
PartialAggregate {
|
||||
op: self.op,
|
||||
groups,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Merges the partial states of one aggregation into its result. Groups are identified by their
|
||||
/// full label set, already grouped by the partitions, and keep the order they first arrive in.
|
||||
pub struct FloatMerger {
|
||||
schema: Arc<SeriesSchema>,
|
||||
op: AggregateOp,
|
||||
group_index: HashMap<LabelKey, usize>,
|
||||
groups: Vec<Group>,
|
||||
}
|
||||
|
||||
impl FloatMerger {
|
||||
pub fn try_new(schema: &Arc<SeriesSchema>, op: AggregateOp) -> Result<Self> {
|
||||
if schema.eval_grid.is_none() {
|
||||
return Err(DataFusionError::Execution(
|
||||
"Aggregate requires an evaluated grid".into(),
|
||||
));
|
||||
}
|
||||
Ok(Self {
|
||||
schema: schema.clone(),
|
||||
op,
|
||||
group_index: HashMap::new(),
|
||||
groups: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Merges the state of one partition. A partial that does not fit this merge, by its
|
||||
/// aggregation, field count or grid positions, fails and leaves the merge unchanged.
|
||||
pub fn merge(&mut self, partial: PartialAggregate) -> Result<()> {
|
||||
let width = self.schema.fields.len();
|
||||
let points = self
|
||||
.schema
|
||||
.eval_grid
|
||||
.as_ref()
|
||||
.expect("checked at construction")
|
||||
.spec()
|
||||
.len as usize;
|
||||
if partial.op != self.op {
|
||||
return Err(DataFusionError::Execution(format!(
|
||||
"cannot merge {:?} state into {:?}",
|
||||
partial.op, self.op
|
||||
)));
|
||||
}
|
||||
for group in &partial.groups {
|
||||
for (point, states) in &group.points {
|
||||
if *point >= points || states.len() != width {
|
||||
return Err(DataFusionError::Execution(
|
||||
"aggregate state does not match the grid or fields of the merge".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
for group in partial.groups {
|
||||
let index = match self.group_index.entry(label_key(&group.metric)) {
|
||||
Entry::Occupied(entry) => *entry.get(),
|
||||
Entry::Vacant(entry) => {
|
||||
self.groups.push(Group {
|
||||
metric: group.metric,
|
||||
blocks: BTreeMap::new(),
|
||||
});
|
||||
*entry.insert(self.groups.len() - 1)
|
||||
}
|
||||
};
|
||||
let blocks = &mut self.groups[index].blocks;
|
||||
for (point, states) in group.points {
|
||||
let block = blocks.entry(point / BLOCK_POINTS).or_insert_with(|| {
|
||||
vec![FloatState::default(); BLOCK_POINTS * width].into_boxed_slice()
|
||||
});
|
||||
let offset = (point % BLOCK_POINTS) * width;
|
||||
for (slot, state) in block[offset..offset + width].iter_mut().zip(states) {
|
||||
if let Some(state) = state {
|
||||
slot.merge(self.op, FloatState::from_partial(state));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The result of every group, in the order the groups first arrived.
|
||||
pub fn finish(self) -> Result<Vec<Series>> {
|
||||
finish_groups(self.schema, self.op, self.groups)
|
||||
}
|
||||
}
|
||||
|
||||
impl FloatState {
|
||||
fn partial(&self) -> Option<AccumulatorState> {
|
||||
self.seen.then_some(AccumulatorState {
|
||||
value: self.value,
|
||||
mean: self.mean,
|
||||
compensation: self.compensation,
|
||||
count: self.count,
|
||||
incremental_mean: self.incremental_mean,
|
||||
})
|
||||
}
|
||||
|
||||
fn from_partial(state: AccumulatorState) -> Self {
|
||||
Self {
|
||||
seen: true,
|
||||
value: state.value,
|
||||
mean: state.mean,
|
||||
compensation: state.compensation,
|
||||
count: state.count,
|
||||
incremental_mean: state.incremental_mean,
|
||||
}
|
||||
}
|
||||
|
||||
/// Folds the state of another partition into this one.
|
||||
fn merge(&mut self, op: AggregateOp, other: Self) {
|
||||
if !self.seen {
|
||||
*self = other;
|
||||
return;
|
||||
}
|
||||
match op {
|
||||
AggregateOp::Sum => {
|
||||
(self.value, self.compensation) = add_compensated(*self, other);
|
||||
}
|
||||
AggregateOp::Count => self.count += other.count,
|
||||
AggregateOp::Group => {}
|
||||
AggregateOp::Max => {
|
||||
if self.value < other.value || self.value.is_nan() {
|
||||
self.value = other.value;
|
||||
}
|
||||
}
|
||||
AggregateOp::Min => {
|
||||
if self.value > other.value || self.value.is_nan() {
|
||||
self.value = other.value;
|
||||
}
|
||||
}
|
||||
AggregateOp::Avg => self.merge_avg(other),
|
||||
AggregateOp::Stddev | AggregateOp::Stdvar => {
|
||||
// Chan et al.'s pairwise update. `delta * (delta * weight)` keeps the product
|
||||
// finite where it is: `delta²` alone overflows for `[0]` and `[1.5e154]`, whose
|
||||
// merged sum of squares is finite.
|
||||
let count = self.count + other.count;
|
||||
let delta = other.mean - self.mean;
|
||||
let weight = self.count * other.count / count;
|
||||
self.mean += delta * (other.count / count);
|
||||
self.value = self.value + other.value + delta * (delta * weight);
|
||||
self.count = count;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Adds the sums while they stay finite, and otherwise continues with the count-weighted
|
||||
/// mean, as the serial update switches to an incremental mean once its sum overflows.
|
||||
fn merge_avg(&mut self, other: Self) {
|
||||
let count = self.count + other.count;
|
||||
if !self.incremental_mean && !other.incremental_mean {
|
||||
let (value, compensation) = add_compensated(*self, other);
|
||||
if !value.is_infinite() {
|
||||
self.value = value;
|
||||
self.compensation = compensation;
|
||||
self.count = count;
|
||||
return;
|
||||
}
|
||||
}
|
||||
let (mean, compensation) = self.mean_form();
|
||||
let (other_mean, other_compensation) = other.mean_form();
|
||||
let (weight, other_weight) = (self.count / count, other.count / count);
|
||||
(self.mean, self.compensation) = kahan_inc(
|
||||
other_mean * other_weight,
|
||||
mean * weight,
|
||||
compensation * weight + other_compensation * other_weight,
|
||||
);
|
||||
self.incremental_mean = true;
|
||||
self.count = count;
|
||||
}
|
||||
|
||||
/// The mean and its compensation, converting a sum as the serial update does on switching.
|
||||
fn mean_form(&self) -> (f64, f64) {
|
||||
if self.incremental_mean {
|
||||
(self.mean, self.compensation)
|
||||
} else {
|
||||
(self.value / self.count, self.compensation / self.count)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The compensated sum of two partial sums. Both the other sum and its compensation are added as
|
||||
/// Kahan addends: adding the compensation plainly would drop what the compensation already
|
||||
/// holds once later partials cancel the sums, as merging `[1e100, 1]`, `[-1e100, 1e84]` and
|
||||
/// `[-1e84]` would lose the `1`.
|
||||
fn add_compensated(this: FloatState, other: FloatState) -> (f64, f64) {
|
||||
let (value, compensation) = kahan_inc(other.value, this.value, this.compensation);
|
||||
kahan_inc(other.compensation, value, compensation)
|
||||
}
|
||||
|
||||
/// Merges `count_values` results of several partitions. Each partial already carries its full
|
||||
/// output label set, value label included, so series with one label set add their counts point
|
||||
/// by point and field by field.
|
||||
pub fn merge_count_values(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
partials: impl IntoIterator<Item = Series>,
|
||||
) -> Result<Vec<Series>> {
|
||||
let width = schema.fields.len();
|
||||
let mut group_index: HashMap<LabelKey, usize> = HashMap::new();
|
||||
let mut groups: Counts = Vec::new();
|
||||
for series in partials {
|
||||
validate_float_series(schema, &series)?;
|
||||
let index = match group_index.entry(label_key(&series.metric)) {
|
||||
Entry::Occupied(entry) => *entry.get(),
|
||||
Entry::Vacant(entry) => {
|
||||
groups.push((series.metric.clone(), BTreeMap::new()));
|
||||
*entry.insert(groups.len() - 1)
|
||||
}
|
||||
};
|
||||
for (field_index, field) in series.fields.iter().enumerate() {
|
||||
for ((point, _), sample) in series.time.positions().zip(field.iter()) {
|
||||
if let SampleRef::Float(count) = sample {
|
||||
groups[index]
|
||||
.1
|
||||
.entry(point)
|
||||
.or_insert_with(|| vec![0.0; width])[field_index] += count;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
count_series(schema, groups)
|
||||
}
|
||||
@@ -20,6 +20,10 @@ use datatypes::series::histogram::HistogramBatchBuilder;
|
||||
use datatypes::series::samples::{FieldSpec, Samples, SeriesFields};
|
||||
use datatypes::series::time::{EvalGrid, GridRange, GridSpec, TimeAxis};
|
||||
|
||||
use rand::rngs::StdRng;
|
||||
use rand::seq::SliceRandom;
|
||||
use rand::{Rng, SeedableRng};
|
||||
|
||||
use super::*;
|
||||
use crate::golden::{assert_result, cases};
|
||||
use crate::grid_series::reference::{Labels, piece_ranges};
|
||||
@@ -391,7 +395,7 @@ fn fields_are_aggregated_independently() {
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
let grid_len = multi.eval_grid.as_ref().unwrap().timestamps().len();
|
||||
let grid_len = multi.eval_grid.as_ref().unwrap().spec().len as usize;
|
||||
let q_values: Vec<Option<f64>> = (0..grid_len)
|
||||
.map(|i| Some([0.0, 0.25, 0.5, 0.9, 1.0][i % 5]))
|
||||
.collect();
|
||||
@@ -898,28 +902,6 @@ fn an_aggregation_of_sparse_inputs_is_compared_in_full_against_the_reference() {
|
||||
assert_eq!(piece_ranges(&counts), vec![(2, 1), (6, 1)]);
|
||||
}
|
||||
|
||||
/// `count_values` writes value labels the way Go's `strconv.FormatFloat(v, 'f', -1, 64)` does,
|
||||
/// including the round-half-even choice of the last shortest digit; see the header of
|
||||
/// `format_float_cases.txt` (promql-compat `cmd/narthex-golden-formatfloat`).
|
||||
#[test]
|
||||
fn value_labels_format_like_go() {
|
||||
let mut checked = 0;
|
||||
for line in include_str!("format_float_cases.txt").lines() {
|
||||
let (bits, expected) = line.split_once(' ').unwrap();
|
||||
let value = f64::from_bits(u64::from_str_radix(bits, 16).unwrap());
|
||||
assert_eq!(format_value(value), expected, "{bits}");
|
||||
checked += 1;
|
||||
}
|
||||
assert!(checked > 2_400, "{checked}");
|
||||
for (value, expected) in [
|
||||
(f64::NAN, "NaN"),
|
||||
(f64::INFINITY, "+Inf"),
|
||||
(f64::NEG_INFINITY, "-Inf"),
|
||||
] {
|
||||
assert_eq!(format_value(value), expected);
|
||||
}
|
||||
}
|
||||
|
||||
/// A point where the `q` scalar has no value has no `quantile` result, as a missing scalar point
|
||||
/// yields no output in the other scalar operators.
|
||||
#[test]
|
||||
@@ -1022,6 +1004,351 @@ fn quantile_over_one_point_members_reads_each_member_at_its_point() {
|
||||
assert_eq!(values_of(&output, 0, 128), (0, expected));
|
||||
}
|
||||
|
||||
/// A float sample for the partial tests: ordinary values, repeats, signed zeros, NaN and
|
||||
/// infinities, but no finite values large enough to overflow when regrouped.
|
||||
fn partial_sample(rng: &mut StdRng) -> f64 {
|
||||
match rng.random_range(0..20) {
|
||||
0 => f64::NAN,
|
||||
1 => f64::INFINITY,
|
||||
2 => f64::NEG_INFINITY,
|
||||
3 => 0.0,
|
||||
4 => -0.0,
|
||||
5..=8 => rng.random_range(0..4) as f64,
|
||||
_ => (rng.random::<f64>() - 0.5) * 2e6,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every valued (point, field) of `output` by rendered label set.
|
||||
fn flatten(output: &[Series]) -> BTreeMap<String, BTreeMap<(usize, usize), f64>> {
|
||||
let mut flat: BTreeMap<String, BTreeMap<(usize, usize), f64>> = BTreeMap::new();
|
||||
for series in output {
|
||||
let points = flat.entry(format!("{:?}", series.metric)).or_default();
|
||||
for (field, samples) in series.fields.iter().enumerate() {
|
||||
for ((point, _), sample) in series.time.positions().zip(samples.iter()) {
|
||||
if let SampleRef::Float(value) = sample {
|
||||
assert!(points.insert((point, field), value).is_none());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
flat
|
||||
}
|
||||
|
||||
fn assert_flat_close(
|
||||
actual: &[Series],
|
||||
expected: &[Series],
|
||||
epsilon: f64,
|
||||
context: &dyn std::fmt::Display,
|
||||
) {
|
||||
let (actual, expected) = (flatten(actual), flatten(expected));
|
||||
assert_eq!(
|
||||
actual.keys().collect::<Vec<_>>(),
|
||||
expected.keys().collect::<Vec<_>>(),
|
||||
"{context}"
|
||||
);
|
||||
for (labels, expected) in &expected {
|
||||
let actual = &actual[labels];
|
||||
assert_eq!(
|
||||
actual.keys().collect::<Vec<_>>(),
|
||||
expected.keys().collect::<Vec<_>>(),
|
||||
"{context} {labels}"
|
||||
);
|
||||
for (key, &value) in expected {
|
||||
// An epsilon of 0 asks for the same bits, NaN payloads aside.
|
||||
let same = if epsilon == 0.0 {
|
||||
actual[key].to_bits() == value.to_bits() || (actual[key].is_nan() && value.is_nan())
|
||||
} else {
|
||||
crate::grid_series::reference::close(actual[key], value, epsilon)
|
||||
};
|
||||
assert!(
|
||||
same,
|
||||
"{context} {labels} {key:?}: merged {} serial {value}",
|
||||
actual[key]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `partial` with only the components its aggregation declares in `state_fields`, as an encoder
|
||||
/// that stores the layout and nothing else would hand it over.
|
||||
fn to_layout(mut partial: PartialAggregate) -> PartialAggregate {
|
||||
let kept = partial.op.state_fields();
|
||||
for group in &mut partial.groups {
|
||||
for (_, states) in &mut group.points {
|
||||
for state in states.iter_mut().flatten() {
|
||||
let mut stripped = AccumulatorState::default();
|
||||
for field in kept {
|
||||
match field {
|
||||
StateField::Value => stripped.value = state.value,
|
||||
StateField::Mean => stripped.mean = state.mean,
|
||||
StateField::Compensation => stripped.compensation = state.compensation,
|
||||
StateField::Count => stripped.count = state.count,
|
||||
StateField::IncrementalMean => {
|
||||
stripped.incremental_mean = state.incremental_mean
|
||||
}
|
||||
}
|
||||
}
|
||||
*state = stripped;
|
||||
}
|
||||
}
|
||||
}
|
||||
partial
|
||||
}
|
||||
|
||||
/// One group's value at point 0, folding each list of samples in its own partition and merging
|
||||
/// the partitions in order, next to the serial fold of all samples.
|
||||
fn merged_and_serial(op: AggregateOp, partitions: &[&[f64]]) -> (f64, f64) {
|
||||
let schema = schema(&["value"]);
|
||||
let series = |samples: &[f64]| -> Vec<Series> {
|
||||
samples
|
||||
.iter()
|
||||
.map(|&value| input(&schema, metric("m", &[]), 0, &[vec![Some(value)]]))
|
||||
.collect()
|
||||
};
|
||||
let mut merger = FloatMerger::try_new(&schema, op).unwrap();
|
||||
for samples in partitions {
|
||||
let mut aggregator = FloatAggregator::try_new(&schema, op, Grouping::By(&[])).unwrap();
|
||||
for series in series(samples) {
|
||||
aggregator.update(&series).unwrap();
|
||||
}
|
||||
merger
|
||||
.merge(to_layout(aggregator.finish_partial()))
|
||||
.unwrap();
|
||||
}
|
||||
let all: Vec<Series> = partitions
|
||||
.iter()
|
||||
.flat_map(|samples| series(samples))
|
||||
.collect();
|
||||
let value = |output: Vec<Series>| values_of(&output, 0, 128).1[0].unwrap();
|
||||
(
|
||||
value(merger.finish().unwrap()),
|
||||
value(aggregate_float(&schema, &all, op, Grouping::By(&[])).unwrap()),
|
||||
)
|
||||
}
|
||||
|
||||
/// Partial states merged in any partition and merge order give the serial result, within the
|
||||
/// rounding a regrouped fold may change: the same groups, points and fields, the same NaN and
|
||||
/// infinities. Each partial keeps only the components its aggregation declares, so the layout
|
||||
/// in `state_fields` is all a merge needs.
|
||||
#[test]
|
||||
fn partials_merge_to_the_serial_result() {
|
||||
const OPS: [AggregateOp; 8] = [
|
||||
AggregateOp::Sum,
|
||||
AggregateOp::Count,
|
||||
AggregateOp::Avg,
|
||||
AggregateOp::Min,
|
||||
AggregateOp::Max,
|
||||
AggregateOp::Group,
|
||||
AggregateOp::Stddev,
|
||||
AggregateOp::Stdvar,
|
||||
];
|
||||
for seed in 0..300 {
|
||||
let mut rng = StdRng::seed_from_u64(seed);
|
||||
let fields: Vec<String> = (0..rng.random_range(1..=3))
|
||||
.map(|field| format!("f{field}"))
|
||||
.collect();
|
||||
let names: Vec<&str> = fields.iter().map(String::as_str).collect();
|
||||
let schema = schema_with_grid_len(&names, rng.random_range(1..80));
|
||||
let points = schema.eval_grid.as_ref().unwrap().spec().len as usize;
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone()).unwrap();
|
||||
let mut series = Vec::new();
|
||||
for index in 0..rng.random_range(1..12) {
|
||||
builder.begin_series();
|
||||
for point in 0..points {
|
||||
let row: Vec<Option<f64>> = (0..names.len())
|
||||
.map(|_| rng.random_bool(0.6).then(|| partial_sample(&mut rng)))
|
||||
.collect();
|
||||
builder.push(point, &row).unwrap();
|
||||
}
|
||||
let job = ["x", "y"][rng.random_range(0..2)];
|
||||
let index = index.to_string();
|
||||
series.extend(
|
||||
builder
|
||||
.finish(metric("m", &[("job", job), ("i", &index)]))
|
||||
.unwrap(),
|
||||
);
|
||||
}
|
||||
let parts: Vec<usize> = series.iter().map(|_| rng.random_range(0..4)).collect();
|
||||
let mut order: Vec<usize> = (0..4).collect();
|
||||
order.shuffle(&mut rng);
|
||||
let grouping = if rng.random_bool(0.5) {
|
||||
Grouping::By(&["job"])
|
||||
} else {
|
||||
Grouping::Without(&["i"])
|
||||
};
|
||||
|
||||
for op in OPS {
|
||||
let serial = aggregate_float(&schema, &series, op, grouping).unwrap();
|
||||
let mut merger = FloatMerger::try_new(&schema, op).unwrap();
|
||||
for &part in &order {
|
||||
let mut aggregator = FloatAggregator::try_new(&schema, op, grouping).unwrap();
|
||||
for (series, _) in series.iter().zip(&parts).filter(|(_, p)| **p == part) {
|
||||
aggregator.update(series).unwrap();
|
||||
}
|
||||
merger
|
||||
.merge(to_layout(aggregator.finish_partial()))
|
||||
.unwrap();
|
||||
}
|
||||
let merged = merger.finish().unwrap();
|
||||
assert_flat_close(&merged, &serial, 1e-10, &format!("seed {seed} {op:?}"));
|
||||
|
||||
// One partition holding every series is the serial fold, bit for bit.
|
||||
let mut aggregator = FloatAggregator::try_new(&schema, op, grouping).unwrap();
|
||||
for series in &series {
|
||||
aggregator.update(series).unwrap();
|
||||
}
|
||||
let mut merger = FloatMerger::try_new(&schema, op).unwrap();
|
||||
merger
|
||||
.merge(to_layout(aggregator.finish_partial()))
|
||||
.unwrap();
|
||||
assert_flat_close(
|
||||
&merger.finish().unwrap(),
|
||||
&serial,
|
||||
0.0,
|
||||
&format!("seed {seed} {op:?} one partition"),
|
||||
);
|
||||
}
|
||||
|
||||
let serial = count_values(&schema, &series, "value", grouping).unwrap();
|
||||
let partials = order.iter().flat_map(|&part| {
|
||||
let members: Vec<Series> = series
|
||||
.iter()
|
||||
.zip(&parts)
|
||||
.filter(|(_, p)| **p == part)
|
||||
.map(|(series, _)| series.clone())
|
||||
.collect();
|
||||
count_values(&schema, &members, "value", grouping).unwrap()
|
||||
});
|
||||
let merged = merge_count_values(&schema, partials).unwrap();
|
||||
assert_flat_close(&merged, &serial, 0.0, &format!("seed {seed} count_values"));
|
||||
}
|
||||
}
|
||||
|
||||
/// The compensation travels with a partial and absorbs what adding two partial sums loses: `1`
|
||||
/// survives next to `1e100` and `-1e100` whether it is a partition's own compensation or the
|
||||
/// error of a merge, and `avg` divides it like the sum.
|
||||
#[test]
|
||||
fn compensation_survives_partials_and_merges() {
|
||||
assert_eq!(
|
||||
merged_and_serial(AggregateOp::Sum, &[&[-1e100], &[1e100, 1.0]]),
|
||||
(1.0, 1.0)
|
||||
);
|
||||
assert_eq!(
|
||||
merged_and_serial(AggregateOp::Sum, &[&[1e100], &[1.0], &[-1e100]]),
|
||||
(1.0, 1.0)
|
||||
);
|
||||
let (merged, serial) = merged_and_serial(AggregateOp::Avg, &[&[-1e100], &[1e100, 1.0]]);
|
||||
assert_eq!(merged, serial);
|
||||
assert!(crate::grid_series::reference::close(
|
||||
merged,
|
||||
1.0 / 3.0,
|
||||
1e-10
|
||||
));
|
||||
// A partial's own compensation must survive a later cancellation of the sums: `1` rides in
|
||||
// the first partial's compensation while `1e84` and `-1e84` cancel across partials.
|
||||
let cancelling: &[&[f64]] = &[&[1e100, 1.0], &[-1e100, 1e84], &[-1e84]];
|
||||
assert_eq!(merged_and_serial(AggregateOp::Sum, cancelling), (1.0, 1.0));
|
||||
let (merged, serial) = merged_and_serial(AggregateOp::Avg, cancelling);
|
||||
assert_eq!(serial, 0.2);
|
||||
assert!(
|
||||
crate::grid_series::reference::close(merged, serial, 1e-10),
|
||||
"{merged}"
|
||||
);
|
||||
let (merged, serial) =
|
||||
merged_and_serial(AggregateOp::Avg, &[&[1e32, 1.0], &[-1e32, 1e16], &[-1e16]]);
|
||||
assert_eq!(serial, 0.2);
|
||||
assert!(
|
||||
crate::grid_series::reference::close(merged, serial, 1e-10),
|
||||
"{merged}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Merging finite partials does not overflow where the result is finite: `avg` switches to a
|
||||
/// count-weighted mean when the summed partials overflow, and `stdvar` does not square the mean
|
||||
/// difference on its own.
|
||||
#[test]
|
||||
fn merges_of_large_finite_partials_stay_finite() {
|
||||
for partitions in [
|
||||
&[&[1e308][..], &[1e308][..]][..],
|
||||
// Unequal counts, one partition already a mean: the weights decide the result.
|
||||
&[&[1e308, 1e308][..], &[0.0][..]][..],
|
||||
&[&[0.0][..], &[1e308, 1e308][..]][..],
|
||||
] {
|
||||
let (merged, serial) = merged_and_serial(AggregateOp::Avg, partitions);
|
||||
assert!(
|
||||
merged.is_finite() && crate::grid_series::reference::close(merged, serial, 1e-10),
|
||||
"{partitions:?}: merged {merged} serial {serial}"
|
||||
);
|
||||
}
|
||||
let (stdvar, serial) = merged_and_serial(AggregateOp::Stdvar, &[&[0.0], &[1.5e154]]);
|
||||
assert!(stdvar.is_finite(), "{stdvar}");
|
||||
assert!(crate::grid_series::reference::close(stdvar, serial, 1e-10));
|
||||
}
|
||||
|
||||
/// Samples that differ by about the rounding of their mean are outside the merge's agreement with
|
||||
/// Prometheus: its serial Welford update rounds the mean back and doubles the variance, while the
|
||||
/// pairwise merge keeps the exact one.
|
||||
#[test]
|
||||
fn ill_conditioned_variance_merges_to_the_exact_value() {
|
||||
let partitions: &[&[f64]] = &[&[1e16], &[1e16 + 2.0]];
|
||||
assert_eq!(
|
||||
merged_and_serial(AggregateOp::Stdvar, partitions),
|
||||
(1.0, 2.0)
|
||||
);
|
||||
assert_eq!(
|
||||
merged_and_serial(AggregateOp::Stddev, partitions),
|
||||
(1.0, std::f64::consts::SQRT_2)
|
||||
);
|
||||
}
|
||||
|
||||
/// `min` and `max` keep the first of two equal values, so with `0` and `-0` the sign is that of
|
||||
/// the partial merged first.
|
||||
#[test]
|
||||
fn equal_zero_extremes_follow_the_merge_order() {
|
||||
let schema = schema(&["value"]);
|
||||
for op in [AggregateOp::Min, AggregateOp::Max] {
|
||||
for (first, second) in [(0.0, -0.0), (-0.0, 0.0)] {
|
||||
let mut merger = FloatMerger::try_new(&schema, op).unwrap();
|
||||
for value in [first, second] {
|
||||
let mut aggregator =
|
||||
FloatAggregator::try_new(&schema, op, Grouping::By(&[])).unwrap();
|
||||
aggregator
|
||||
.update(&input(&schema, metric("m", &[]), 0, &[vec![Some(value)]]))
|
||||
.unwrap();
|
||||
merger.merge(aggregator.finish_partial()).unwrap();
|
||||
}
|
||||
let value = values_of(&merger.finish().unwrap(), 0, 128).1[0].unwrap();
|
||||
assert_eq!(
|
||||
value.to_bits(),
|
||||
f64::to_bits(first),
|
||||
"{op:?} {first} {second}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A partial of another aggregation, field count or grid fails the merge.
|
||||
#[test]
|
||||
fn a_merge_rejects_state_that_does_not_fit() {
|
||||
let schema = schema(&["value"]);
|
||||
let partial = |op| {
|
||||
let mut aggregator = FloatAggregator::try_new(&schema, op, Grouping::By(&[])).unwrap();
|
||||
aggregator
|
||||
.update(&input(&schema, metric("m", &[]), 5, &[vec![Some(1.0)]]))
|
||||
.unwrap();
|
||||
aggregator.finish_partial()
|
||||
};
|
||||
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
|
||||
assert!(merger.merge(partial(AggregateOp::Avg)).is_err());
|
||||
let mut wide = partial(AggregateOp::Sum);
|
||||
wide.groups[0].points[0].1.push(None);
|
||||
assert!(merger.merge(wide).is_err());
|
||||
let mut late = partial(AggregateOp::Sum);
|
||||
late.groups[0].points[0].0 = 128;
|
||||
assert!(merger.merge(late).is_err());
|
||||
assert!(merger.finish().unwrap().is_empty());
|
||||
}
|
||||
|
||||
/// `2^-24` and the float below it format to different labels, so they count as two values: the
|
||||
/// even 16-digit candidate for `2^-24` names the lower neighbour.
|
||||
#[test]
|
||||
@@ -1064,3 +1391,48 @@ fn count_values_keeps_neighbours_of_a_power_of_two_apart() {
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
/// Prometheus counts a histogram sample under the text `FloatHistogram.String` writes it as, so
|
||||
/// equal histograms share a group and a float beside them has its own.
|
||||
#[test]
|
||||
fn count_values_counts_histograms_by_their_text() {
|
||||
use crate::grid_series::reference::OwnedHistogram;
|
||||
use crate::grid_series::{GridSample, GridSeriesBuilder};
|
||||
use crate::histogram_eval::float_histogram::FloatHistogram;
|
||||
use crate::histogram_eval::histogram_string;
|
||||
|
||||
let schema = schema_with_grid_len(&["value"], 1);
|
||||
let histogram =
|
||||
OwnedHistogram::parse("0/0/0/4000000000000000/3ff0000000000000/0/1_1/4000000000000000///");
|
||||
let text = histogram_string(&histogram.view()).unwrap();
|
||||
let series = |job: &str, sample: GridSample| {
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone()).unwrap();
|
||||
builder.push_samples(0, &[Some(sample)]).unwrap();
|
||||
builder
|
||||
.finish(metric("m", &[("job", job)]))
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
};
|
||||
let sample = || {
|
||||
GridSample::Histogram(Box::new(
|
||||
FloatHistogram::try_from_view(&histogram.view()).unwrap(),
|
||||
))
|
||||
};
|
||||
let input = [
|
||||
series("a", sample()),
|
||||
series("b", sample()),
|
||||
series("c", GridSample::Float(2.0)),
|
||||
];
|
||||
let mut counts: Vec<(String, f64)> = count_values(&schema, &input, "v", Grouping::By(&[]))
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|series| {
|
||||
let SampleRef::Float(count) = series.fields.as_slice()[0].sample(0) else {
|
||||
panic!("a count is a float");
|
||||
};
|
||||
(series.metric.get("v").unwrap().to_string(), count)
|
||||
})
|
||||
.collect();
|
||||
counts.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
assert_eq!(counts, [("2".to_string(), 1.0), (text, 2.0)]);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! The shortest float formats of Go's `strconv`, which Prometheus writes label values in.
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
/// Go's `strconv.FormatFloat(value, 'f', -1, 64)`, the form of `count_values` labels.
|
||||
pub(crate) fn format_float_f(value: f64) -> String {
|
||||
match special(value) {
|
||||
Some(text) => text.to_string(),
|
||||
None => {
|
||||
let (negative, digits, point) = shortest(value);
|
||||
positional(negative, &digits, point)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Prometheus `labels.FormatOpenMetricsFloat`, the form of `histogram_quantiles` labels: Go's
|
||||
/// shortest `'g'` format, with `.0` appended to an integer and `0`, `1` and `-1` written as
|
||||
/// `0.0`, `1.0` and `-1.0`.
|
||||
pub(crate) fn format_open_metrics_float(value: f64) -> String {
|
||||
if value == 1.0 {
|
||||
return "1.0".to_string();
|
||||
}
|
||||
if value == 0.0 {
|
||||
return "0.0".to_string();
|
||||
}
|
||||
if value == -1.0 {
|
||||
return "-1.0".to_string();
|
||||
}
|
||||
if let Some(text) = special(value) {
|
||||
return text.to_string();
|
||||
}
|
||||
let mut text = format_float_g(value);
|
||||
if !text.contains(['e', '.']) {
|
||||
text.push_str(".0");
|
||||
}
|
||||
text
|
||||
}
|
||||
|
||||
/// Go's `strconv.FormatFloat(value, 'g', -1, 64)`, which `%g` and `%v` print a float in: the
|
||||
/// shortest digits, with an exponent of at least two digits below 1e-4 and from 1e6 on.
|
||||
pub(crate) fn format_float_g(value: f64) -> String {
|
||||
if let Some(text) = special(value) {
|
||||
return text.to_string();
|
||||
}
|
||||
let (negative, digits, point) = shortest(value);
|
||||
let exponent = point - 1;
|
||||
if (-4..6).contains(&exponent) {
|
||||
return positional(negative, &digits, point);
|
||||
}
|
||||
let mut text = String::from(if negative { "-" } else { "" });
|
||||
text.push_str(&digits[..1]);
|
||||
if digits.len() > 1 {
|
||||
text.push('.');
|
||||
text.push_str(&digits[1..]);
|
||||
}
|
||||
text.push_str(&format!(
|
||||
"e{}{:02}",
|
||||
if exponent < 0 { '-' } else { '+' },
|
||||
exponent.abs()
|
||||
));
|
||||
text
|
||||
}
|
||||
|
||||
fn special(value: f64) -> Option<&'static str> {
|
||||
match value {
|
||||
f64::INFINITY => Some("+Inf"),
|
||||
f64::NEG_INFINITY => Some("-Inf"),
|
||||
_ if value.is_nan() => Some("NaN"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The fewest significant digits of a finite `value` that round-trip, without trailing zeros, and
|
||||
/// the position of the decimal point among them: `value = ±0.digits × 10^point`. Zero is `"0"` at
|
||||
/// point 1.
|
||||
///
|
||||
/// When two candidates of that length are equally close, Go takes the even one while Rust's
|
||||
/// shortest formatting rounds up, so the digits are recomputed with Rust's exact formatting,
|
||||
/// which rounds half to even. The recomputed digits only count when they still parse back to
|
||||
/// `value`: just above a power of two the lower neighbour is half as far away, so the even
|
||||
/// candidate can name that neighbour, and only the other one round-trips.
|
||||
fn shortest(value: f64) -> (bool, String, i64) {
|
||||
let shortest = format!("{value:e}");
|
||||
let mantissa = shortest.split_once('e').expect("exponent format").0;
|
||||
let significant = mantissa.chars().filter(char::is_ascii_digit).count();
|
||||
let exact = format!("{value:.precision$e}", precision = significant - 1);
|
||||
let chosen = if exact.parse::<f64>() == Ok(value) {
|
||||
&exact
|
||||
} else {
|
||||
&shortest
|
||||
};
|
||||
let (mantissa, exponent) = chosen.split_once('e').expect("exponent format");
|
||||
let exponent: i64 = exponent.parse().expect("integer exponent");
|
||||
let (negative, mantissa) = match mantissa.strip_prefix('-') {
|
||||
Some(unsigned) => (true, unsigned),
|
||||
None => (false, mantissa),
|
||||
};
|
||||
let digits: String = mantissa.chars().filter(char::is_ascii_digit).collect();
|
||||
let trimmed = digits.trim_end_matches('0');
|
||||
let digits = if trimmed.is_empty() { "0" } else { trimmed };
|
||||
(negative, digits.to_string(), exponent + 1)
|
||||
}
|
||||
|
||||
/// `±0.digits × 10^point` in positional notation without trailing fractional zeros.
|
||||
fn positional(negative: bool, digits: &str, point: i64) -> String {
|
||||
let mut out = String::from(if negative { "-" } else { "" });
|
||||
if point <= 0 {
|
||||
out.push_str("0.");
|
||||
out.extend(std::iter::repeat_n('0', (-point) as usize));
|
||||
out.push_str(digits);
|
||||
} else if point as usize >= digits.len() {
|
||||
out.push_str(digits);
|
||||
out.extend(std::iter::repeat_n('0', point as usize - digits.len()));
|
||||
} else {
|
||||
let (whole, fraction) = digits.split_at(point as usize);
|
||||
out.push_str(whole);
|
||||
out.push('.');
|
||||
out.push_str(fraction);
|
||||
}
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,358 @@
|
||||
0 0.0
|
||||
8000000000000000 0.0
|
||||
3ff0000000000000 1.0
|
||||
bff0000000000000 -1.0
|
||||
3fe0000000000000 0.5
|
||||
3feccccccccccccd 0.9
|
||||
3fefae147ae147ae 0.99
|
||||
3feff7ced916872b 0.999
|
||||
3f1a36e2eb1c432d 0.0001
|
||||
3ee4f8b588e368f1 1e-05
|
||||
3f201f31f46ed246 0.000123
|
||||
40f869f000000000 99999.0
|
||||
40f86a0000000000 100000.0
|
||||
412e847e00000000 999999.0
|
||||
412e848000000000 1e+06
|
||||
4132d68700000000 1.234567e+06
|
||||
444b1ae4d6e2ef50 1e+21
|
||||
3e7ad7f29abcaf48 1e-07
|
||||
4000000000000000 2.0
|
||||
c004000000000000 -2.5
|
||||
1 5e-324
|
||||
7fefffffffffffff 1.7976931348623157e+308
|
||||
7ff8000000000001 NaN
|
||||
7ff0000000000000 +Inf
|
||||
fff0000000000000 -Inf
|
||||
3e0fffffffffffff 9.313225746154784e-10
|
||||
3e10000000000000 9.313225746154785e-10
|
||||
3e10000000000001 9.313225746154787e-10
|
||||
3e1fffffffffffff 1.8626451492309568e-09
|
||||
3e20000000000000 1.862645149230957e-09
|
||||
3e20000000000001 1.8626451492309574e-09
|
||||
3e2fffffffffffff 3.7252902984619136e-09
|
||||
3e30000000000000 3.725290298461914e-09
|
||||
3e30000000000001 3.725290298461915e-09
|
||||
3e3fffffffffffff 7.450580596923827e-09
|
||||
3e40000000000000 7.450580596923828e-09
|
||||
3e40000000000001 7.45058059692383e-09
|
||||
3e4fffffffffffff 1.4901161193847655e-08
|
||||
3e50000000000000 1.4901161193847656e-08
|
||||
3e50000000000001 1.490116119384766e-08
|
||||
3e5fffffffffffff 2.980232238769531e-08
|
||||
3e60000000000000 2.9802322387695312e-08
|
||||
3e60000000000001 2.980232238769532e-08
|
||||
3e6fffffffffffff 5.960464477539062e-08
|
||||
3e70000000000000 5.960464477539063e-08
|
||||
3e70000000000001 5.960464477539064e-08
|
||||
3e7fffffffffffff 1.1920928955078124e-07
|
||||
3e80000000000000 1.1920928955078125e-07
|
||||
3e80000000000001 1.1920928955078128e-07
|
||||
3e8fffffffffffff 2.3841857910156247e-07
|
||||
3e90000000000000 2.384185791015625e-07
|
||||
3e90000000000001 2.3841857910156255e-07
|
||||
3e9fffffffffffff 4.7683715820312495e-07
|
||||
3ea0000000000000 4.76837158203125e-07
|
||||
3ea0000000000001 4.768371582031251e-07
|
||||
3eafffffffffffff 9.536743164062499e-07
|
||||
3eb0000000000000 9.5367431640625e-07
|
||||
3eb0000000000001 9.536743164062502e-07
|
||||
3ebfffffffffffff 1.9073486328124998e-06
|
||||
3ec0000000000000 1.9073486328125e-06
|
||||
3ec0000000000001 1.9073486328125004e-06
|
||||
3ecfffffffffffff 3.8146972656249996e-06
|
||||
3ed0000000000000 3.814697265625e-06
|
||||
3ed0000000000001 3.814697265625001e-06
|
||||
3edfffffffffffff 7.629394531249999e-06
|
||||
3ee0000000000000 7.62939453125e-06
|
||||
3ee0000000000001 7.629394531250002e-06
|
||||
3eefffffffffffff 1.5258789062499998e-05
|
||||
3ef0000000000000 1.52587890625e-05
|
||||
3ef0000000000001 1.5258789062500003e-05
|
||||
3effffffffffffff 3.0517578124999997e-05
|
||||
3f00000000000000 3.0517578125e-05
|
||||
3f00000000000001 3.051757812500001e-05
|
||||
3f0fffffffffffff 6.103515624999999e-05
|
||||
3f10000000000000 6.103515625e-05
|
||||
3f10000000000001 6.103515625000001e-05
|
||||
3f1fffffffffffff 0.00012207031249999999
|
||||
3f20000000000000 0.0001220703125
|
||||
3f20000000000001 0.00012207031250000003
|
||||
3f2fffffffffffff 0.00024414062499999997
|
||||
3f30000000000000 0.000244140625
|
||||
3f30000000000001 0.00024414062500000005
|
||||
3f3fffffffffffff 0.00048828124999999995
|
||||
3f40000000000000 0.00048828125
|
||||
3f40000000000001 0.0004882812500000001
|
||||
3f4fffffffffffff 0.0009765624999999999
|
||||
3f50000000000000 0.0009765625
|
||||
3f50000000000001 0.0009765625000000002
|
||||
3f5fffffffffffff 0.0019531249999999998
|
||||
3f60000000000000 0.001953125
|
||||
3f60000000000001 0.0019531250000000004
|
||||
3f6fffffffffffff 0.0039062499999999996
|
||||
3f70000000000000 0.00390625
|
||||
3f70000000000001 0.003906250000000001
|
||||
3f7fffffffffffff 0.007812499999999999
|
||||
3f80000000000000 0.0078125
|
||||
3f80000000000001 0.007812500000000002
|
||||
3f8fffffffffffff 0.015624999999999998
|
||||
3f90000000000000 0.015625
|
||||
3f90000000000001 0.015625000000000003
|
||||
3f9fffffffffffff 0.031249999999999997
|
||||
3fa0000000000000 0.03125
|
||||
3fa0000000000001 0.03125000000000001
|
||||
3fafffffffffffff 0.06249999999999999
|
||||
3fb0000000000000 0.0625
|
||||
3fb0000000000001 0.06250000000000001
|
||||
3fbfffffffffffff 0.12499999999999999
|
||||
3fc0000000000000 0.125
|
||||
3fc0000000000001 0.12500000000000003
|
||||
3fcfffffffffffff 0.24999999999999997
|
||||
3fd0000000000000 0.25
|
||||
3fd0000000000001 0.25000000000000006
|
||||
3fdfffffffffffff 0.49999999999999994
|
||||
3fe0000000000000 0.5
|
||||
3fe0000000000001 0.5000000000000001
|
||||
3fefffffffffffff 0.9999999999999999
|
||||
3ff0000000000000 1.0
|
||||
3ff0000000000001 1.0000000000000002
|
||||
3fffffffffffffff 1.9999999999999998
|
||||
4000000000000000 2.0
|
||||
4000000000000001 2.0000000000000004
|
||||
400fffffffffffff 3.9999999999999996
|
||||
4010000000000000 4.0
|
||||
4010000000000001 4.000000000000001
|
||||
401fffffffffffff 7.999999999999999
|
||||
4020000000000000 8.0
|
||||
4020000000000001 8.000000000000002
|
||||
402fffffffffffff 15.999999999999998
|
||||
4030000000000000 16.0
|
||||
4030000000000001 16.000000000000004
|
||||
403fffffffffffff 31.999999999999996
|
||||
4040000000000000 32.0
|
||||
4040000000000001 32.00000000000001
|
||||
404fffffffffffff 63.99999999999999
|
||||
4050000000000000 64.0
|
||||
4050000000000001 64.00000000000001
|
||||
405fffffffffffff 127.99999999999999
|
||||
4060000000000000 128.0
|
||||
4060000000000001 128.00000000000003
|
||||
406fffffffffffff 255.99999999999997
|
||||
4070000000000000 256.0
|
||||
4070000000000001 256.00000000000006
|
||||
407fffffffffffff 511.99999999999994
|
||||
4080000000000000 512.0
|
||||
4080000000000001 512.0000000000001
|
||||
408fffffffffffff 1023.9999999999999
|
||||
4090000000000000 1024.0
|
||||
4090000000000001 1024.0000000000002
|
||||
409fffffffffffff 2047.9999999999998
|
||||
40a0000000000000 2048.0
|
||||
40a0000000000001 2048.0000000000005
|
||||
40afffffffffffff 4095.9999999999995
|
||||
40b0000000000000 4096.0
|
||||
40b0000000000001 4096.000000000001
|
||||
40bfffffffffffff 8191.999999999999
|
||||
40c0000000000000 8192.0
|
||||
40c0000000000001 8192.000000000002
|
||||
40cfffffffffffff 16383.999999999998
|
||||
40d0000000000000 16384.0
|
||||
40d0000000000001 16384.000000000004
|
||||
40dfffffffffffff 32767.999999999996
|
||||
40e0000000000000 32768.0
|
||||
40e0000000000001 32768.00000000001
|
||||
40efffffffffffff 65535.99999999999
|
||||
40f0000000000000 65536.0
|
||||
40f0000000000001 65536.00000000001
|
||||
40ffffffffffffff 131071.99999999999
|
||||
4100000000000000 131072.0
|
||||
4100000000000001 131072.00000000003
|
||||
410fffffffffffff 262143.99999999997
|
||||
4110000000000000 262144.0
|
||||
4110000000000001 262144.00000000006
|
||||
411fffffffffffff 524287.99999999994
|
||||
4120000000000000 524288.0
|
||||
4120000000000001 524288.0000000001
|
||||
412fffffffffffff 1.0485759999999999e+06
|
||||
4130000000000000 1.048576e+06
|
||||
4130000000000001 1.0485760000000002e+06
|
||||
413fffffffffffff 2.0971519999999998e+06
|
||||
4140000000000000 2.097152e+06
|
||||
4140000000000001 2.0971520000000005e+06
|
||||
414fffffffffffff 4.1943039999999995e+06
|
||||
4150000000000000 4.194304e+06
|
||||
4150000000000001 4.194304000000001e+06
|
||||
415fffffffffffff 8.388607999999999e+06
|
||||
4160000000000000 8.388608e+06
|
||||
4160000000000001 8.388608000000002e+06
|
||||
416fffffffffffff 1.6777215999999998e+07
|
||||
4170000000000000 1.6777216e+07
|
||||
4170000000000001 1.6777216000000004e+07
|
||||
417fffffffffffff 3.3554431999999996e+07
|
||||
4180000000000000 3.3554432e+07
|
||||
4180000000000001 3.355443200000001e+07
|
||||
418fffffffffffff 6.710886399999999e+07
|
||||
4190000000000000 6.7108864e+07
|
||||
4190000000000001 6.710886400000001e+07
|
||||
419fffffffffffff 1.3421772799999999e+08
|
||||
41a0000000000000 1.34217728e+08
|
||||
41a0000000000001 1.3421772800000003e+08
|
||||
41afffffffffffff 2.6843545599999997e+08
|
||||
41b0000000000000 2.68435456e+08
|
||||
41b0000000000001 2.6843545600000006e+08
|
||||
41bfffffffffffff 5.3687091199999994e+08
|
||||
41c0000000000000 5.36870912e+08
|
||||
41c0000000000001 5.368709120000001e+08
|
||||
41cfffffffffffff 1.0737418239999999e+09
|
||||
41d0000000000000 1.073741824e+09
|
||||
41d0000000000001 1.0737418240000002e+09
|
||||
3e65822107fcfd52 4.0062361426977984e-08
|
||||
bfd04dc76695721d -0.2547472478670743
|
||||
3dda858149c6e2d1 9.648466959782995e-11
|
||||
c276cb397916001e -1.5663811505600073e+12
|
||||
40197f48392907a0 6.374298947457561
|
||||
40b27cc6f3875d04 4732.777153454026
|
||||
be2cffa2ba517936 -3.3758786482808152e-09
|
||||
c27621587cb3ad0b -1.5207796437068152e+12
|
||||
c06a768b7c4e0b68 -211.70452704661443
|
||||
424c3f2525632186 2.426369665982619e+11
|
||||
41f2070f169c1121 4.839240041754182e+09
|
||||
bd8bd268b68e6a3f -3.162985000843313e-12
|
||||
c0b4c47f2cdf5b8a -5316.496778449868
|
||||
3ea4502fc5d6d268 6.053813980584764e-07
|
||||
c1034c367674cb74 -158086.80783995579
|
||||
3f8c8b35bb9457d8 0.013937396800013532
|
||||
c00419db794209ff -2.5126256440428274
|
||||
c1e4b7a55a25e0cb -2.7806379051836905e+09
|
||||
3e72b02936d4ff9b 6.961870520783569e-08
|
||||
40d442fd5ad145f4 20747.958668058694
|
||||
be4d968d3f71f8cb -1.3778029368671855e-08
|
||||
be06e3cafccae224 -6.661813267564451e-10
|
||||
c169d9f17d9e8fbc -1.3553547925605647e+07
|
||||
420403def63a5b6f 1.0745536199294645e+10
|
||||
4221017fbc897d06 3.651979629274419e+10
|
||||
c265a315724c7af1 -7.434432395238419e+11
|
||||
41ca71f52257bb7d 8.873518766854092e+08
|
||||
40bb9ca8be8e9981 7068.659157669522
|
||||
be33fac825416fed -4.651866964628022e-09
|
||||
41b9f7676678e7aa 4.356442624722849e+08
|
||||
3ef7830356aa6b86 2.2422570888562812e-05
|
||||
4228f8bd3e99b0a8 5.362618350084503e+10
|
||||
403a156ae58348b0 26.083662361671202
|
||||
bfa0a1e9406aec9d -0.03248528395985171
|
||||
bd9d8e9fa5ead0bd -6.720511814114512e-12
|
||||
40408e4b738b6829 33.11167759234406
|
||||
41b780ea967cd710 3.943246304876566e+08
|
||||
bfb9c16725fd3d7f -0.10060734441252882
|
||||
411f01c926fbf29b 508018.2880704791
|
||||
bf12787c2bf3394b -7.045991786755916e-05
|
||||
c16ed31f1cb9e6c6 -1.616101689769305e+07
|
||||
c0e92d0169a39144 -51560.04414537785
|
||||
c02b5389421657ff -13.663156571605212
|
||||
3f7e91e98b02c917 0.007463371541393002
|
||||
4052560383d17909 73.34396453338935
|
||||
c1a58fbeaf635d47 -1.8087099969407102e+08
|
||||
bfc03382be14186f -0.1265719822042146
|
||||
bdb7ae24e9174548 -2.1537062557132775e-11
|
||||
41f5855807b73658 5.7769657554507675e+09
|
||||
beb2a18da250bf34 -1.1104981549984584e-06
|
||||
bedb3641fe3dea79 -6.487833136388005e-06
|
||||
c1eb36dbc9b344e9 -3.652640333602162e+09
|
||||
40f5cb471941e52a 89268.44366635816
|
||||
c080d6326b40eabc -538.7746186324725
|
||||
bd82aafe336eacca -2.1222992129454815e-12
|
||||
c0ef278a67149b9c -63804.325083068776
|
||||
bf1c44bf5a5ffefb -0.0001078359406108085
|
||||
bf6d29286f00f02b -0.0035596646994502565
|
||||
bdbfafc3545836c4 -2.8818771389702537e-11
|
||||
bd7f325712f4128d -1.7733237396191494e-12
|
||||
c05e3e3e4a305605 -120.97255186767877
|
||||
bfeca80d72521a90 -0.8955142243219871
|
||||
3f85f26283efc6c6 0.01071621861590034
|
||||
411cfc5dc1540864 474903.4387971221
|
||||
bfbb185b70ab53ba -0.1058404111930277
|
||||
413fb024878d4063 2.076708529499077e+06
|
||||
c274d6353cb62f07 -1.4318905455069392e+12
|
||||
40b900fb4ffbf3a9 6400.981689211833
|
||||
3fd796b83f190476 0.36857420122038975
|
||||
3f89c820f5c353cf 0.012588746547786215
|
||||
bf1c9b582d230176 -0.00010912633595961348
|
||||
3f0b626c3f1d7427 5.223172037347622e-05
|
||||
c1b98e40b01aa47d -4.287530721040724e+08
|
||||
bf40e7ac7417bf38 -0.0005158989007991504
|
||||
c0fa23eb55ece0ea -107070.70847785813
|
||||
c144a3c813d346b5 -2.705296154885138e+06
|
||||
420cfdbc6db30e37 1.556450859838194e+10
|
||||
bffaedf1a6e222f5 -1.6830917853241634
|
||||
bdaa7d712e166bdf -1.2046261072190693e-11
|
||||
bd9166a221898f34 -3.956509186266304e-12
|
||||
41f29c7719af53ba 4.995903898957941e+09
|
||||
c257d70a133c9673 -4.0956803813035077e+11
|
||||
3e03f10e0f212551 5.803775500632443e-10
|
||||
bdd09a3e6f241c57 -6.039959833086004e-11
|
||||
3def70ff3be70cb0 2.2876625416160384e-10
|
||||
3e0b4e8e20f31127 7.947341135001321e-10
|
||||
c101aa87289cbd2c -144720.89483020583
|
||||
c1a30cf840a79c3b -1.5980854432736382e+08
|
||||
c1f2cee10365790c -5.048766518342052e+09
|
||||
4241f57b0bb131e8 1.542658517463899e+11
|
||||
41644786139efcca 1.0632240613157649e+07
|
||||
c10c4694a54af747 -231634.5807093924
|
||||
3fb93fb09f6c47ac 0.09862808124697037
|
||||
410283b654043a66 151670.7910236895
|
||||
3f955397b4926431 0.020826692959051655
|
||||
c2770eb58f0d2558 -1.584496439506334e+12
|
||||
c2574b9d7f803594 -4.0021144320083716e+11
|
||||
c1798c4033fe2656 -2.678886724954828e+07
|
||||
bdda20af6b41292d -9.505193404385634e-11
|
||||
bd9d5ac882093298 -6.6744682973325806e-12
|
||||
c10fbff5b2d5a113 -260094.71232152786
|
||||
3ea8e96f5b8e56a9 7.42431146882456e-07
|
||||
3d98fa6e04f9ce92 5.679394522162092e-12
|
||||
bfb82947a67e52b1 -0.09437988104611851
|
||||
3f819e7792417fc3 0.008603033209681038
|
||||
3ef67230b027b7e0 2.1406226560154065e-05
|
||||
410bdd8f44c9ab40 228273.908587778
|
||||
bdaddb0efd7a84af -1.3576799338601639e-11
|
||||
bfde7f03e3abff4a -0.476502392135434
|
||||
be3c374d736e41cc -6.569555391781771e-09
|
||||
bd8fd8d37b3ccebc -3.6205821004411243e-12
|
||||
bf015dbd315ae6af -3.312333698830599e-05
|
||||
bef0a23119ea0f2f -1.58630006077541e-05
|
||||
4147e80684a6bfdc 3.1334530363387894e+06
|
||||
418ffa8aa44a03a4 6.706414853613976e+07
|
||||
41b39bb8b580c8ba 3.2897246950306284e+08
|
||||
c13b6a22b648e604 -1.7966427120498428e+06
|
||||
c235fcdd893a310f -9.443669023419164e+10
|
||||
421ec18437f5198b 3.3023921661274944e+10
|
||||
bf8ae37ae39d02db -0.013129196234900877
|
||||
bef95e8cca599392 -2.4194083968604218e-05
|
||||
3dcad1bdc176163d 4.8784025588835895e-11
|
||||
3f1df0dcf9f95ede 0.00011421536191258015
|
||||
c1757d0f8a928585 -2.2532344660771865e+07
|
||||
c0a5dcfd887953e8 -2798.4951818385052
|
||||
c1ad00c4e533e9a3 -2.4329483460139188e+08
|
||||
be189b251dba77ae -1.432259747359515e-09
|
||||
c0faea4753b428d7 -110244.45793548539
|
||||
3f1888203391860a 9.358116666782087e-05
|
||||
c07eccfcba9075b0 -492.81170135907723
|
||||
c1a9bf542b517f2b -2.1598466165917334e+08
|
||||
be6f489bc56ba9c3 -5.827029585742761e-08
|
||||
c07aeee5fdde63b1 -430.9311503111922
|
||||
bdbff35e51e90a1f -2.905895413133007e-11
|
||||
c08585e696d511e1 -688.7375923772889
|
||||
3e1584aa5403510d 1.2525254861511635e-09
|
||||
425175c897a14c51 2.9996209728519244e+11
|
||||
bfefb52e3298237e -0.9908667553109891
|
||||
3ece74d4b80f63d4 3.630682329430478e-06
|
||||
be56a526ee438107 -2.108991638348143e-08
|
||||
3e7365a824bf7034 7.225981696286206e-08
|
||||
405b24ce99aa99ff 108.57510987911517
|
||||
bd8f8f5522164965 -3.5879444088569995e-12
|
||||
c11bb2a14ccdd7af -453800.3250039769
|
||||
be24ed37e9a7b972 -2.4361808796334138e-09
|
||||
bec4827163240d09 -2.444928008606154e-06
|
||||
c1c39851da458093 -6.574990605429863e+08
|
||||
3e46091f323f491a 1.0261140888290476e-08
|
||||
bdaa570dd2d99eb9 -1.197807008879946e-11
|
||||
40f265b33b61e4f4 75355.2019976562
|
||||
@@ -0,0 +1,51 @@
|
||||
// Copyright 2023 Greptime Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use crate::go_format::{format_float_f, format_open_metrics_float};
|
||||
|
||||
/// Prometheus `labels.FormatOpenMetricsFloat`; `open_metrics_cases.txt` comes from promql-compat
|
||||
/// `narthex-golden-formatfloat -openmetrics` as `<hex bits> <formatted>`.
|
||||
#[test]
|
||||
fn open_metrics_floats_format_like_prometheus() {
|
||||
let mut checked = 0;
|
||||
for line in include_str!("open_metrics_cases.txt").lines() {
|
||||
let (bits, expected) = line.split_once(' ').unwrap();
|
||||
let value = f64::from_bits(u64::from_str_radix(bits, 16).unwrap());
|
||||
assert_eq!(format_open_metrics_float(value), expected, "{bits}");
|
||||
checked += 1;
|
||||
}
|
||||
assert!(checked > 300, "{checked}");
|
||||
}
|
||||
|
||||
/// Go's `strconv.FormatFloat(v, 'f', -1, 64)`, the form `count_values` writes value labels in,
|
||||
/// including the round-half-even choice of the last shortest digit and the round-trip check at
|
||||
/// powers of two; `float_cases.txt` comes from promql-compat `narthex-golden-formatfloat`.
|
||||
#[test]
|
||||
fn floats_format_like_go() {
|
||||
let mut checked = 0;
|
||||
for line in include_str!("float_cases.txt").lines() {
|
||||
let (bits, expected) = line.split_once(' ').unwrap();
|
||||
let value = f64::from_bits(u64::from_str_radix(bits, 16).unwrap());
|
||||
assert_eq!(format_float_f(value), expected, "{bits}");
|
||||
checked += 1;
|
||||
}
|
||||
assert!(checked > 2_400, "{checked}");
|
||||
for (value, expected) in [
|
||||
(f64::NAN, "NaN"),
|
||||
(f64::INFINITY, "+Inf"),
|
||||
(f64::NEG_INFINITY, "-Inf"),
|
||||
] {
|
||||
assert_eq!(format_float_f(value), expected);
|
||||
}
|
||||
}
|
||||
@@ -254,7 +254,7 @@ fn fold_one(pieces: Vec<Series>) -> Series {
|
||||
None => panic!("expected a grid series, got {:?}", pieces[0].time),
|
||||
};
|
||||
let mut rows: Vec<Option<Vec<Option<f64>>>> =
|
||||
(0..grid.timestamps().len()).map(|_| None).collect();
|
||||
(0..grid.spec().len as usize).map(|_| None).collect();
|
||||
for piece in &pieces {
|
||||
let mut cursors = piece
|
||||
.fields
|
||||
|
||||
@@ -163,10 +163,10 @@ impl GridSeriesBuilder {
|
||||
self.schema.fields.len()
|
||||
)));
|
||||
}
|
||||
if index < self.next || index >= self.grid.timestamps().len() {
|
||||
if index < self.next || index >= self.grid.spec().len as usize {
|
||||
return Err(DataFusionError::Execution(format!(
|
||||
"grid series index {index} is not increasing or exceeds grid length {}",
|
||||
self.grid.timestamps().len()
|
||||
self.grid.spec().len as usize
|
||||
)));
|
||||
}
|
||||
self.next = index + 1;
|
||||
@@ -302,6 +302,15 @@ pub(crate) const METADATA_LABELS: [&str; 3] = ["__name__", "__type__", "__unit__
|
||||
|
||||
pub(crate) type LabelSet = Vec<(Arc<str>, Arc<str>)>;
|
||||
|
||||
/// A label set as Prometheus prints one in errors: `{name="value", ...}`.
|
||||
pub(crate) fn render_label_set(labels: &LabelSet) -> String {
|
||||
let labels: Vec<String> = labels
|
||||
.iter()
|
||||
.map(|(name, value)| format!("{name}=\"{value}\""))
|
||||
.collect();
|
||||
format!("{{{}}}", labels.join(", "))
|
||||
}
|
||||
|
||||
/// The label set Prometheus compares, which has no empty values.
|
||||
pub(crate) fn label_set(metric: &MetricName) -> LabelSet {
|
||||
let name = metric
|
||||
|
||||
@@ -43,7 +43,7 @@ pub(crate) fn expand(series: &Series) -> Dense {
|
||||
panic!("expected a grid series, got {:?}", series.time);
|
||||
};
|
||||
let fields = series.fields.as_slice();
|
||||
let mut dense = vec![vec![None; fields.len()]; grid.timestamps().len()];
|
||||
let mut dense = vec![vec![None; fields.len()]; grid.spec().len as usize];
|
||||
let mut cursors = fields
|
||||
.iter()
|
||||
.map(|field| field.cursor())
|
||||
@@ -124,7 +124,7 @@ fn assert_series_matches(
|
||||
last + 1 - first,
|
||||
"{context}: the result carries a row for a position that holds no value"
|
||||
);
|
||||
assert_eq!(grid.timestamps().len(), expected.len());
|
||||
assert_eq!(grid.spec().len as usize, expected.len());
|
||||
assert_eq!(&actual.metric, metric);
|
||||
let bits = |dense: &Dense| -> Vec<Vec<Option<u64>>> {
|
||||
dense
|
||||
@@ -148,7 +148,7 @@ pub(crate) fn stitch(pieces: &[Series]) -> Dense {
|
||||
panic!("expected a grid series, got {:?}", piece.time);
|
||||
};
|
||||
let dense = dense
|
||||
.get_or_insert_with(|| vec![vec![None; piece.fields.len()]; grid.timestamps().len()]);
|
||||
.get_or_insert_with(|| vec![vec![None; piece.fields.len()]; grid.spec().len as usize]);
|
||||
let mut cursors = piece
|
||||
.fields
|
||||
.iter()
|
||||
|
||||
@@ -312,12 +312,7 @@ fn runs_never_bridge_positions_without_a_value() {
|
||||
let series = builder.finish(metric(0)).unwrap().expect("two rows");
|
||||
// One series over a million-position grid, with two runs, and the axis says where they sit.
|
||||
assert_eq!(
|
||||
series
|
||||
.time
|
||||
.grid()
|
||||
.expect("a grid series")
|
||||
.timestamps()
|
||||
.len(),
|
||||
series.time.grid().expect("a grid series").spec().len as usize,
|
||||
SPAN,
|
||||
"the grid really holds a million positions"
|
||||
);
|
||||
@@ -787,7 +782,7 @@ fn merge_of_many_segments_keeps_one_row_per_value() {
|
||||
assert_eq!(merged.len(), 1);
|
||||
let series = &merged[0];
|
||||
assert_eq!(
|
||||
series.time.grid().unwrap().timestamps().len(),
|
||||
series.time.grid().unwrap().spec().len as usize,
|
||||
GRID,
|
||||
"the grid really holds a million positions"
|
||||
);
|
||||
@@ -971,7 +966,10 @@ mod acceptance {
|
||||
// The axis lends the grid's buffer instead of copying it: the fast path is not only a
|
||||
// label, it is the same rows without per-series timestamp storage.
|
||||
assert!(
|
||||
matches!(series.time.timestamps(), std::borrow::Cow::Borrowed(_)),
|
||||
matches!(
|
||||
series.time.timestamps().unwrap(),
|
||||
std::borrow::Cow::Borrowed(_)
|
||||
),
|
||||
"a single range lends its timestamps"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -40,9 +40,10 @@ use datatypes::series::time::{TimeAxis, union_positions};
|
||||
pub(crate) use aggregate::sum_or_avg;
|
||||
pub use aggregate::{HistogramAggregate, aggregate_histograms};
|
||||
|
||||
use crate::go_format::format_open_metrics_float;
|
||||
use crate::go_sort;
|
||||
use crate::grid_series::{
|
||||
GridSeriesBuilder, LabelSet, METADATA_LABELS, label_set, merge_same_labelset,
|
||||
GridSeriesBuilder, LabelSet, METADATA_LABELS, external, label_set, merge_same_labelset,
|
||||
};
|
||||
|
||||
const BUCKET_LABEL: &str = "le";
|
||||
@@ -56,6 +57,33 @@ pub enum HistogramFunction {
|
||||
Fraction { lower: f64, upper: f64 },
|
||||
}
|
||||
|
||||
/// `histogram_quantiles(v, label, q...)`: [`evaluate_histograms`] with `Quantile(q)` for each
|
||||
/// `q`, each result labelled with `label` set to `q` as Prometheus' `FormatOpenMetricsFloat`
|
||||
/// writes it. Results that then share a label set merge as [`merge_same_labelset`] does, so a
|
||||
/// repeated `q`, or a `label` that overwrites the label telling two histograms apart, fails at a
|
||||
/// point both hold.
|
||||
///
|
||||
/// Each `q` is a constant. Prometheus evaluates the quantile arguments at every step, so a
|
||||
/// quantile that varies over time is not this kernel's to evaluate.
|
||||
pub fn histogram_quantiles(
|
||||
label: &str,
|
||||
quantiles: &[f64],
|
||||
schema: &Arc<SeriesSchema>,
|
||||
series: &[Series],
|
||||
) -> Result<Vec<Series>> {
|
||||
let mut output = Vec::new();
|
||||
for &quantile in quantiles {
|
||||
let value: Arc<str> = Arc::from(format_open_metrics_float(quantile));
|
||||
for mut result in
|
||||
evaluate_histograms(HistogramFunction::Quantile(quantile), schema, series)?
|
||||
{
|
||||
result.metric.set(label, value.clone()).map_err(external)?;
|
||||
output.push(result);
|
||||
}
|
||||
}
|
||||
merge_same_labelset(schema, output)
|
||||
}
|
||||
|
||||
/// Evaluates `function` over the classic and native histograms in `series`.
|
||||
///
|
||||
/// Float series with the same labels other than `le` form a classic histogram; a float series
|
||||
@@ -237,6 +265,12 @@ fn has_histograms(series: &Series) -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
/// A native histogram as Go's `FloatHistogram.String` writes it, the label value `count_values`
|
||||
/// gives a histogram sample.
|
||||
pub(crate) fn histogram_string(h: &HistogramView<'_>) -> Result<String> {
|
||||
native::go_string(h)
|
||||
}
|
||||
|
||||
/// A function of one native histogram.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum NativeHistogramFunction {
|
||||
|
||||
@@ -19,6 +19,7 @@ use datafusion::error::{DataFusionError, Result};
|
||||
use datatypes::series::histogram::{BucketSideView, HistogramView};
|
||||
|
||||
use crate::functions::kahan_inc;
|
||||
use crate::go_format::format_float_g;
|
||||
use crate::histogram_eval::bounds::EXPONENTIAL_BOUNDS;
|
||||
|
||||
pub(super) const CUSTOM_BUCKETS_SCHEMA: i32 = -53;
|
||||
@@ -28,6 +29,8 @@ pub(super) struct NativeBucket {
|
||||
lower: f64,
|
||||
upper: f64,
|
||||
count: f64,
|
||||
/// The bucket's index on its side; meaningless for the zero bucket.
|
||||
index: i32,
|
||||
}
|
||||
|
||||
/// The buckets of `h` from the lowest to the highest, or the reverse: negative buckets, the zero
|
||||
@@ -57,6 +60,7 @@ pub(super) fn all_buckets(
|
||||
lower: -h.zero_threshold,
|
||||
upper: h.zero_threshold,
|
||||
count: h.zero_count,
|
||||
index: 0,
|
||||
});
|
||||
}
|
||||
side(h, high, !reverse, out)?;
|
||||
@@ -107,16 +111,74 @@ fn bucket(h: &HistogramView<'_>, idx: i32, count: f64, positive: bool) -> Result
|
||||
lower: bound(idx - 1)?,
|
||||
upper: bound(idx)?,
|
||||
count,
|
||||
index: idx,
|
||||
}
|
||||
} else {
|
||||
NativeBucket {
|
||||
lower: -bound(idx)?,
|
||||
upper: -bound(idx - 1)?,
|
||||
count,
|
||||
index: idx,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Go `FloatHistogram.String`, the form `count_values` writes a histogram value in: the count and
|
||||
/// sum, then the buckets holding observations from the lowest up -- negative, zero, positive --
|
||||
/// each as `[lower,upper):count` with its bounds' inclusiveness, every number in Go's `%g`.
|
||||
/// Unlike [`all_buckets`], the buckets keep the bounds their index gives, unclipped by the zero
|
||||
/// bucket.
|
||||
pub(super) fn go_string(h: &HistogramView<'_>) -> Result<String> {
|
||||
let mut text = format!(
|
||||
"{{count:{}, sum:{}",
|
||||
format_float_g(h.count),
|
||||
format_float_g(h.sum)
|
||||
);
|
||||
let custom = h.schema == CUSTOM_BUCKETS_SCHEMA;
|
||||
let mut buckets = Vec::new();
|
||||
side(h, h.negative, false, &mut buckets)?;
|
||||
for bucket in buckets.iter().rev().filter(|bucket| bucket.count != 0.0) {
|
||||
push_bucket(&mut text, bucket, custom);
|
||||
}
|
||||
if h.zero_count != 0.0 {
|
||||
if custom {
|
||||
return Err(DataFusionError::Execution(
|
||||
"histograms with custom buckets have no zero bucket".into(),
|
||||
));
|
||||
}
|
||||
text.push_str(&format!(
|
||||
", [{},{}]:{}",
|
||||
format_float_g(-h.zero_threshold),
|
||||
format_float_g(h.zero_threshold),
|
||||
format_float_g(h.zero_count)
|
||||
));
|
||||
}
|
||||
buckets.clear();
|
||||
side(h, h.positive, true, &mut buckets)?;
|
||||
for bucket in buckets.iter().filter(|bucket| bucket.count != 0.0) {
|
||||
push_bucket(&mut text, bucket, custom);
|
||||
}
|
||||
text.push('}');
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
/// Go `Bucket.String`, with the inclusiveness `baseBucketIterator.at` gives the bucket.
|
||||
fn push_bucket(text: &mut String, bucket: &NativeBucket, custom: bool) {
|
||||
let (lower_inclusive, upper_inclusive) = if custom {
|
||||
(bucket.index == 0, true)
|
||||
} else {
|
||||
(bucket.lower < 0.0, bucket.upper > 0.0)
|
||||
};
|
||||
text.push_str(&format!(
|
||||
", {}{},{}{}:{}",
|
||||
if lower_inclusive { '[' } else { '(' },
|
||||
format_float_g(bucket.lower),
|
||||
format_float_g(bucket.upper),
|
||||
if upper_inclusive { ']' } else { ')' },
|
||||
format_float_g(bucket.count)
|
||||
));
|
||||
}
|
||||
|
||||
/// Go `getBound`.
|
||||
pub(super) fn bound(idx: i32, schema: i32, custom_values: &[f64]) -> Result<f64> {
|
||||
if schema == CUSTOM_BUCKETS_SCHEMA {
|
||||
@@ -203,6 +265,7 @@ pub(super) fn quantile(
|
||||
lower: 0.0,
|
||||
upper: 0.0,
|
||||
count: 0.0,
|
||||
index: 0,
|
||||
};
|
||||
let mut count = 0.0;
|
||||
let mut visited = 0;
|
||||
|
||||
@@ -0,0 +1,425 @@
|
||||
# Generated by promql-compat cmd/narthex-golden-histogram built for amd64 (Prometheus
|
||||
# v3.15.0 engine, seed 1, 30 cases per function). Each case lists its operation and
|
||||
# tab-separated arguments, the grid, the input series and either the result series or
|
||||
# the error. Points are `grid index:hex bits`.
|
||||
case quantiles quantile 3fd0000000000000
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x ; 0:4026000000000000 1:4018000000000000
|
||||
series __name__=a_bucket|job=x|le=abc ; 0:404f000000000000 1:4057c00000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:403c000000000000 1:403c000000000000
|
||||
series __name__=a_bucket|job=y|le=0.1 ; 0:4051500000000000 1:4053b00000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:4058e00000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:405d300000000000 1:4069600000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:4065580000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:406c280000000000 1:4074240000000000
|
||||
series __name__=b_bucket|job=z|le=0.25 ; 0:404ba00000000000 1:4040200000000000
|
||||
series __name__=b_bucket|job=z|le=2.5 ; 0:4060580000000000 1:405a100000000000
|
||||
series __name__=b_bucket|job=z|le=inf ; 0:4065680000000000 1:4065180000000000
|
||||
result job=x|quantile=0.25 ; 0:7ff8000000000001 1:7ff8000000000001
|
||||
result job=y|quantile=0.25 ; 0:3fb4d13c711ede2d 1:3fbcf601dab51b18
|
||||
result job=z|quantile=0.25 ; 0:3fc8cbdf90cbdf91 1:3fe1f00000000000
|
||||
end
|
||||
case quantiles quantile bfe0000000000000,3fefae147ae147ae
|
||||
grid 100000 10000 5
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4024000000000000 1:4035000000000000 2:4000000000000000 3:4028000000000000 4:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=abc ; 1:4047800000000000 4:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=1e400 ; 0:0 1:4053800000000000 2:4052800000000000 3:7ff8000000000001 4:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:4044800000000000 1:4057800000000000 3:7ff8000000000001 4:7ff0000000000000
|
||||
series __name__=a_bucket|job=y ; 0:404a400000000000 1:4041400000000000 2:4017000000000000 3:4058600000000000 4:4050d00000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:405d800000000000 1:404de00000000000 2:405a700000000000 3:405db00000000000 4:405ad00000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 2:4063c80000000000 3:406af00000000000 4:4072440000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:406fd80000000000 1:4066280000000000 3:406c180000000000 4:4074440000000000
|
||||
series __name__=b_bucket|job=y ; 0:404f800000000000 2:4058300000000000
|
||||
series __name__=b_bucket|job=y|le=0 ; 0:4061480000000000 1:4064700000000000 2:4066800000000000 3:4058600000000000 4:4051900000000000
|
||||
series __name__=b_bucket|job=y|le=1 ; 0:4069300000000000 1:406ee00000000000 2:406a600000000000 3:4066180000000000 4:405a400000000000
|
||||
series __name__=b_bucket|job=y|le=+Inf ; 0:406a680000000000 2:4072380000000000 3:406b600000000000 4:4063680000000000
|
||||
series __name__=b_bucket|job=z|le=0.25 ; 0:4014000000000000 1:4020000000000000 2:4031000000000000 3:4043800000000000 4:4028000000000000
|
||||
series __name__=b_bucket|job=z|le=2.5 ; 0:4048000000000000 1:40200000000001c2 2:4030ffffffff4515 3:4055400000000000 4:4028000000000000
|
||||
series __name__=b_bucket|job=z|le=inf ; 1:40200000000001c2 2:4030ffffffff4336 3:405c400000000000 4:4028000000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 0,bfe0000000000000
|
||||
grid 100000 10000 3
|
||||
series __name__=a_bucket|job=x|le=0.1 ; 0:403e000000000000 1:4028000000000000 2:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:404c800000000000 1:4041000000000000 2:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4055400000000000 1:4045000000000000 2:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:4060000000000000 1:4056000000000000 2:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:7ff8000000000001 1:7ff0000000000000 2:7ff0000000000000
|
||||
series __name__=b_bucket|job=y|le=0.1 ; 0:4041800000000000 1:4038000000000000 2:4048000000000000
|
||||
series __name__=b_bucket|job=y|le=0.5 ; 0:4043000000000000 1:4037fffffffffd5c
|
||||
series __name__=b_bucket|job=y|le=1 ; 0:4054000000000000 1:4037ffffffffffff
|
||||
series __name__=b_bucket|job=y|le=5 ; 0:4053ffffffff2419 1:40380000000107e1 2:40553fffffff165b
|
||||
series __name__=b_bucket|job=y|le=+Inf ; 0:4054000000000000 2:40557fffffff165b
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:0 1:0 2:0
|
||||
series __name__=b_bucket|job=z|le=1.0 ; 0:0 2:0
|
||||
series __name__=b_bucket|job=z|le=2 ; 1:1 2:1a56e1fc2f8f359
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:1 1:1 2:0
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile bfe0000000000000,bfe0000000000000
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:402a000000000000 2:4010000000000000 3:4033000000000000 4:401c000000000000
|
||||
series __name__=a_bucket|job=y|le=2 ; 0:4035000000000000 1:4040ffffffff4515 2:4010000000000000 3:403d000000000000 4:401c000000000000 5:4056800000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:4034ffffffff191a 1:404a7fffffff4515 2:4047000000000000 4:401c000000000000 5:4058400000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4057c00000000000 1:4039800000000000 3:4028800000000000 4:404f600000000000 5:4044600000000000
|
||||
series __name__=b_bucket|job=z|le=2 ; 0:4065c00000000000 1:4055d00000000000 2:4060200000000000 3:4057500000000000 4:405f300000000000 5:4059600000000000
|
||||
series __name__=b_bucket|job=z|le=5 ; 0:4070e80000000000 1:405a900000000000 2:4069e80000000000 3:4062580000000000 4:4065500000000000 5:4062780000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile 3fd0000000000000,3ff8000000000000,3fe0000000000000
|
||||
grid 100000 10000 8
|
||||
series __name__=b_bucket|job=z ; 0:4044c00000000000 1:402e000000000000 2:4021000000000000 3:4048e00000000000 4:404ea00000000000 5:4056c00000000000 6:404ce00000000000 7:4047e00000000000
|
||||
series __name__=b_bucket|job=z|le=2.5 ; 1:403dc00000000000 2:4049000000000000 3:4050800000000000 4:4055b00000000000 5:4061300000000000 6:4055d00000000000
|
||||
series __name__=b_bucket|job=z|le=inf ; 0:4064180000000000 1:4056c00000000000 2:4060400000000000 3:4064100000000000 5:406bf00000000000 6:4057d00000000000 7:4051e00000000000
|
||||
result job=z|quantile=0.25 ; 0:7ff8000000000001 1:3ffe969696969696 2:3ffa000000000000 3:3ff851745d1745d1 4:7ff8000000000001 5:3ff04129e4129e41 6:3fe5d574a1601779 7:7ff8000000000001
|
||||
result job=z|quantile=0.5 ; 0:7ff8000000000001 1:4004000000000000 2:4004000000000000 3:4004000000000000 4:7ff8000000000001 5:40004129e4129e41 6:3ff5d574a1601779 7:7ff8000000000001
|
||||
result job=z|quantile=1.5 ; 0:7ff0000000000000 1:7ff0000000000000 2:7ff0000000000000 3:7ff0000000000000 4:7ff0000000000000 5:7ff0000000000000 6:7ff0000000000000 7:7ff0000000000000
|
||||
end
|
||||
case quantiles quantile 3ff8000000000000,3ff0000000000000
|
||||
grid 100000 10000 5
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 1:4052600000000000 3:404b000000000000 4:404fe00000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 1:0 2:0 3:0
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:404e800000000000 1:4052000000000000 2:4029800000000000 3:4054b00000000000 4:4052900000000000
|
||||
series __name__=b_bucket|job=z|le=abc ; 2:4056900000000000 3:4062380000000000 4:405f200000000000
|
||||
series __name__=b_bucket|job=z|le=1e400 ; 0:4053900000000000 1:4062f00000000000 2:405e800000000000 3:4064380000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4063080000000000 1:4067680000000000 2:4064480000000000 4:406ee80000000000
|
||||
result job=x|quantile=1.0 ; 1:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001
|
||||
result job=x|quantile=1.5 ; 1:7ff0000000000000 3:7ff0000000000000 4:7ff0000000000000
|
||||
result job=y|quantile=1.0 ; 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001
|
||||
result job=y|quantile=1.5 ; 1:7ff0000000000000 2:7ff0000000000000 3:7ff0000000000000
|
||||
result job=z|quantile=1.0 ; 0:3ff0000000000000 1:3ff0000000000000 2:3ff0000000000000 3:7ff8000000000001 4:3ff0000000000000
|
||||
result job=z|quantile=1.5 ; 0:7ff0000000000000 1:7ff0000000000000 2:7ff0000000000000 3:7ff0000000000000 4:7ff0000000000000
|
||||
end
|
||||
case quantiles quantile 3ff0000000000000
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x ; 0:4026000000000000 1:4040800000000000
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:7ff8000000000001 1:7ff0000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:7ff8000000000001 1:7ff8000000000001
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:7ff8000000000001 1:7ff8000000000001
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:7ff8000000000001
|
||||
result job=x|quantile=1.0 ; 0:4014000000000000 1:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 3feccccccccccccd,3feccccccccccccd,3fe0000000000000
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4024000000000000 1:4010000000000000
|
||||
series __name__=a_bucket|job=x|le=2 ; 0:4008000000000000 1:4058800000000000
|
||||
series __name__=a_bucket|job=x|le=5 ;
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile 3fefae147ae147ae,3fd0000000000000,3ff0000000000000
|
||||
grid 100000 10000 7
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 1:4047000000000000 2:4010000000000000 3:4024000000000000 4:4022000000000000 5:4047800000000000 6:4033000000000000
|
||||
series __name__=b_bucket|job=y|le=1 ; 0:4053000000000000 1:404d000000000000 2:4026000000000000 3:4054800000000000 4:4055400000000000 5:4050000000000000 6:403d000000000000
|
||||
series __name__=b_bucket|job=y|le=2 ; 0:4054400000000000 1:4045000000000000 2:402a000000000000 3:4032000000000000 4:403d000000000000 5:4054800000000000 6:4055400000000000
|
||||
series __name__=b_bucket|job=y|le=5 ; 0:4057800000000000 1:4056800000000000 2:4038000000000000 3:403a000000000000 4:404f800000000000 5:404a000000000000 6:4055000000000000
|
||||
result job=x|quantile=0.25 ; 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=x|quantile=0.99 ; 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=x|quantile=1.0 ; 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=y|quantile=0.25 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=y|quantile=0.99 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=y|quantile=1.0 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 0,3ff0000000000000,0
|
||||
grid 100000 10000 7
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4045000000000000 1:4034000000000000 2:403a000000000000 3:4056000000000000 5:401c000000000000
|
||||
series __name__=a_bucket|job=x|le=1.0 ; 0:4024000000000000 1:4044000000000000 2:4008000000000000 3:4054400000000000 4:4039000000000000 5:4055800000000000 6:4052c00000000000
|
||||
series __name__=a_bucket|job=x|le=2 ; 0:404b800000000000 1:4042800000000000 3:4055c00000000000 4:4018000000000000 5:404b800000000000 6:4054c00000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:4045800000000000 1:4054800000000000 2:4050400000000000 3:4043000000000000 4:4050800000000000
|
||||
series __name__=a_bucket|job=y|le=0.1 ; 0:4056900000000000 1:403bc00000000000 2:4035800000000000 3:403b400000000000 4:402f000000000000 5:4052600000000000 6:4047600000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:4063f00000000000 1:4045800000000000 2:405a100000000000 3:4057d00000000000 5:4061080000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:4069e00000000000 1:405ab00000000000 3:4066400000000000 6:4053200000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:4072180000000000 2:406b780000000000 3:406d000000000000 4:4067080000000000 5:4070700000000000 6:405a500000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:4074b00000000000 1:4067680000000000 2:4073c80000000000 3:4072340000000000 4:4069000000000000 5:4070e80000000000 6:4062c80000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 0
|
||||
grid 100000 10000 2
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4055300000000000 1:4036400000000000
|
||||
series __name__=b_bucket|job=z|le=2 ; 0:4062280000000000 1:4047800000000000
|
||||
series __name__=b_bucket|job=z|le=5 ; 1:404e600000000000
|
||||
result job=0.0 ; 0:7ff8000000000001 1:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 3feccccccccccccd,bfe0000000000000
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:0 1:0 4:0 5:0
|
||||
series __name__=b_bucket|job=z|le=0.25 ; 0:4031000000000000 1:404e000000000000 3:4047800000000000 4:402c000000000000 5:4053400000000000
|
||||
series __name__=b_bucket|job=z|le=2.5 ; 0:404a800000000000 1:4058c00000000000 2:4033000000000000 3:4058000000000000 4:4054c00000000000 5:4048000000000000
|
||||
series __name__=b_bucket|job=z ; 0:4055000000000000 1:404c000000000000 3:3ff0000000000000 4:4035000000000000 5:4045000000000000
|
||||
result job=x|quantile=-0.5 ; 0:fff0000000000000 1:fff0000000000000 4:fff0000000000000 5:fff0000000000000
|
||||
result job=x|quantile=0.9 ; 0:7ff8000000000001 1:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001
|
||||
result job=z|quantile=-0.5 ; 0:fff0000000000000 1:fff0000000000000 2:fff0000000000000 3:fff0000000000000 4:fff0000000000000 5:fff0000000000000
|
||||
result job=z|quantile=0.9 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001
|
||||
end
|
||||
case quantiles job 3fefae147ae147ae,3fefae147ae147ae
|
||||
grid 100000 10000 8
|
||||
series __name__=a_bucket|job=x ; 0:4050f00000000000 1:4047000000000000 2:4057500000000000 3:404aa00000000000 4:4025000000000000 5:4050c00000000000 6:4055500000000000 7:402b800000000000
|
||||
series __name__=a_bucket|job=x|le=1.0 ; 0:4059b00000000000 1:405ac00000000000 3:405e200000000000 4:4056a00000000000 5:4064280000000000 6:4064200000000000 7:4048600000000000
|
||||
series __name__=a_bucket|job=x|le=2 ; 0:4061d80000000000 1:405c000000000000 2:4062180000000000 4:4066180000000000 5:406fe00000000000 6:406fa80000000000 7:404b000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:4065380000000000 1:4061800000000000 2:4064080000000000 3:4066180000000000 4:4067080000000000 5:4075340000000000 6:4071f40000000000 7:4051b00000000000
|
||||
series __name__=b_bucket|job=y|le=-1 ; 0:3ff0000000000000 1:4010000000000000 2:4010000000000000 4:c014000000000000 5:c014000000000000 6:4028000000000000 7:403d000000000000
|
||||
series __name__=b_bucket|job=y|le=0 ; 0:0 1:403d000000000000 2:c014000000000000 3:4055000000000000 4:4032000000000000 5:403e000000000000 6:c014000000000000 7:4043800000000000
|
||||
series __name__=b_bucket|job=y|le=1 ; 1:4048000000000000 2:4008000000000000 3:405d400000000000 4:4040800000000000 5:4045000000000000 6:c014000000000000 7:4053c00000000000
|
||||
series __name__=b_bucket|job=y|le=+Inf ; 0:c014000000000000 1:4052000000000000 2:7ff0000000000000 3:4060800000000000 4:7ff0000000000000 5:404c000000000000 6:4044800000000000 7:4056000000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4053c00000000000 2:403f400000000000 3:4048a00000000000 4:4008000000000000 5:4052400000000000 6:404ec00000000000 7:4055600000000000
|
||||
series __name__=b_bucket|job=z|le=abc ; 0:405be00000000000 1:4059a00000000000 2:4059d00000000000 3:4054c00000000000 4:402d800000000000 5:4055b00000000000 6:4060680000000000 7:4057a00000000000
|
||||
series __name__=b_bucket|job=z|le=1e400 ; 0:4063500000000000 1:4065580000000000 2:4061600000000000 3:405ae00000000000 4:4049200000000000 5:4059c00000000000 6:4067900000000000 7:405d900000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:406c480000000000 1:4070800000000000 2:406d200000000000 3:4060400000000000 4:404ba00000000000 6:406a580000000000 7:4062c00000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 7ff8000000000001
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=x|le=0.1 ; 0:4053900000000000 1:402b000000000000 2:4045400000000000 4:4020000000000000 5:4053000000000000
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:405a400000000000 1:4048e00000000000 2:404aa00000000000 3:405e700000000000 4:4041600000000000 5:405e100000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:405d400000000000 1:4053000000000000 2:4054f00000000000 3:4067800000000000 5:405e500000000000
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:4069480000000000 1:4058200000000000 2:4064480000000000 3:4068d80000000000 4:4057f00000000000 5:4068c80000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:4070ec0000000000 2:4069000000000000 3:4071b40000000000 5:406a080000000000
|
||||
series __name__=b_bucket|job=z|le=0.25 ; 0:4046000000000000 1:403e000000000000 3:403c000000000000 4:403b000000000000 5:4045800000000000
|
||||
series __name__=b_bucket|job=z|le=2.5 ; 0:4053400000000000 1:4045000000000000 2:40390000000112e1 3:404e000000000000 4:403b000000000000 5:40457ffffffffda3
|
||||
series __name__=b_bucket|job=z|le=inf ; 0:40533fffffff2c58 1:4051000000000000 3:404dfffffffffcb3 5:40457ffffffffda3
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile 3fd0000000000000
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x|le=-1 ; 0:4048800000000000
|
||||
series __name__=a_bucket|job=x|le=0 ; 0:4055800000000000 1:4034000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4056800000000000 1:0
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:c014000000000000 1:4036000000000000
|
||||
result job=x|quantile=0.25 ; 0:bff0000000000000 1:0
|
||||
end
|
||||
case quantiles job 3fefae147ae147ae,3fd0000000000000
|
||||
grid 100000 10000 7
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4026000000000000 1:403a000000000000 5:4043000000000000 6:4000000000000000
|
||||
series __name__=a_bucket|job=x|le=2 ; 0:4046800000000000 1:4051800000000000 2:402e000000000000 3:4014000000000000 4:4040000000000000 5:403f000000000000 6:4041000000000000
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:4030000000000000 1:4028000000000000 2:4052400000000000 3:4056400000000000 4:4041800000000000 5:4055800000000000 6:4050000000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 1:4041800000000000 2:4056e00000000000 3:4048000000000000 4:4052c00000000000 5:4042a00000000000 6:401d000000000000
|
||||
series __name__=a_bucket|job=y|le=2 ; 1:4046800000000000 2:4062180000000000 3:404e400000000000 4:4065800000000000 5:4050e00000000000 6:4056e00000000000
|
||||
series __name__=a_bucket|job=y ; 0:405de00000000000 2:4068180000000000 3:4050000000000000 4:4069700000000000 5:4063d00000000000 6:4066980000000000
|
||||
series __name__=b_bucket|job=z|le=-1 ; 0:4014000000000000 1:4000000000000000 2:402c000000000000 3:403d000000000000 4:7ff8000000000001 5:3ff0000000000000 6:4043800000000000
|
||||
series __name__=b_bucket|job=z|le=0 ; 0:4039000000000000 1:4028000000000000 2:403f000000000000 4:7ff8000000000001 6:c014000000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:0 1:7ff0000000000000 2:4041800000000000 3:0 4:7ff8000000000001 5:c014000000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:7ff0000000000000 1:7ff0000000000000 4:7ff8000000000001 5:7ff0000000000000 6:4046000000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile 3feccccccccccccd
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x ; 0:0
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:0 1:0
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:0 1:0
|
||||
series __name__=a_bucket|job=x|le=5 ; 1:0
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:1a56e1fc2f8f359 1:0
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 1:4058800000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:0 1:0
|
||||
series __name__=b_bucket|job=z|le=abc ; 1:0
|
||||
series __name__=b_bucket|job=z|le=1e400 ; 0:0 1:1a56e1fc2f8f359
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 1:0
|
||||
result job=x|quantile=0.9 ; 0:3ff0000000000000 1:7ff8000000000001
|
||||
result job=y|quantile=0.9 ; 1:7ff8000000000001
|
||||
result job=z|quantile=0.9 ; 0:7ff8000000000001 1:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 3ff0000000000000
|
||||
grid 100000 10000 4
|
||||
series __name__=a_bucket|job=x|le=-1 ; 0:4045000000000000 1:4048800000000000 2:4036000000000000
|
||||
series __name__=a_bucket|job=x ; 0:4046800000000000 1:4053400000000000 2:4058c00000000000 3:4057800000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4051c00000000000 1:4044000000000000 2:4058c00000000000 3:403e000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:4018000000000000 1:4048000000000000 2:4041800000000000
|
||||
series __name__=a_bucket|job=y|le=0.1 ; 0:4022000000000000 2:402e000000000000 3:4042800000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:4024000000000000 2:4043800000000000 3:4042800000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:4023ffffffff2419 1:404a000000000000 2:4043800000000000 3:40427fffffff3497
|
||||
series __name__=a_bucket|job=y ; 0:4032ffffffff920c 1:404a000000000000 2:404380000000d668 3:40427fffffff3497
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:4032ffffffff920c 1:404a000000011de0 2:404380000000d443 3:40437fffffff3497
|
||||
series __name__=b_bucket|job=z|le=0.1 ; 0:7ff0000000000000 1:c014000000000000 2:4043000000000000 3:4008000000000000
|
||||
series __name__=b_bucket|job=z|le=0.5 ; 1:4030000000000000 2:4055800000000000 3:0
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:7ff0000000000000 1:4050400000000000 2:0 3:4037000000000000
|
||||
series __name__=b_bucket|job=z|le=5 ; 1:405a400000000000 2:403f000000000000 3:404b000000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:7ff8000000000001 1:4060600000000000 2:4041800000000000 3:7ff8000000000001
|
||||
result job=x|quantile=1.0 ; 0:3ff0000000000000 1:bff0000000000000 2:3ff0000000000000 3:7ff8000000000001
|
||||
result job=y|quantile=1.0 ; 0:3ff0000000000000 1:3ff0000000000000 2:3ff0000000000000 3:3ff0000000000000
|
||||
result job=z|quantile=1.0 ; 0:3ff0000000000000 1:4014000000000000 2:3fe0000000000000 3:4014000000000000
|
||||
end
|
||||
case quantiles quantile 3fefae147ae147ae
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=y|le=-1 ; 0:4051100000000000 1:4043800000000000 2:4050000000000000 4:4054300000000000 5:4055000000000000
|
||||
series __name__=a_bucket|job=y|le=0 ; 1:4047600000000000 2:4059a00000000000 3:405e000000000000 4:4055f00000000000 5:405a200000000000
|
||||
series __name__=a_bucket|job=y ; 0:4068e80000000000 1:4060900000000000 2:405fd00000000000 3:4063c00000000000 4:4061000000000000 5:4067b80000000000
|
||||
series __name__=b_bucket|job=z|le=-1 ; 0:4055000000000000 2:404a800000000000 3:4052000000000000 4:4024000000000000 5:4051400000000000
|
||||
series __name__=b_bucket|job=z|le=0 ; 1:4010000000000000 2:4053000000000000 3:4044800000000000 5:4032000000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4053c00000000000 1:4024000000000000 2:404a000000000000 3:4047000000000000 4:403e000000000000 5:4047000000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:404c000000000000 1:403d000000000000 2:3ff0000000000000 3:4054400000000000 4:404a000000000000 5:4024000000000000
|
||||
result job=y|quantile=0.99 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001
|
||||
result job=z|quantile=0.99 ; 0:bff0000000000000 1:3ff0000000000000 2:bfa0eb1324f3fad0 3:3ff0000000000000 4:3ff0000000000000 5:bff0000000000000
|
||||
end
|
||||
case quantiles quantile 3fefae147ae147ae
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=y|le=0.25 ; 0:4057800000000000 2:404f000000000000 3:403e000000000000 4:403d000000000000 5:4056c00000000000
|
||||
series __name__=a_bucket|job=y|le=2.5 ; 0:4055400000000000 1:4053400000000000 2:404a800000000000 3:404f000000000000 4:4055400000000000 5:4050c00000000000
|
||||
series __name__=a_bucket|job=y|le=inf ; 0:4044800000000000 1:4034000000000000 2:4041000000000000 3:404f000000000000 4:404f000000000000 5:402a000000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4043800000000000 1:4022000000000000 3:4042800000000000 4:4026000000000000 5:403e000000000000
|
||||
result job=y|quantile=0.99 ; 0:3fcfae147ae147ae 1:4003cccccccccccd 2:3fcfae147ae147ae 3:4003a6b851eb851f 4:4003ba0ea0ea0ea1 5:3fcfae147ae147ae
|
||||
result job=z|quantile=0.99 ; 0:7ff8000000000001 1:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 7ff8000000000001
|
||||
grid 100000 10000 7
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:0 1:0 2:0 3:0 4:0 5:0
|
||||
series __name__=a_bucket|job=x|le=abc ; 0:0 1:0 2:1 3:1 4:0 5:0
|
||||
series __name__=a_bucket|job=x|le=1e400 ; 0:1a56e1fc2f8f359 1:0 2:0 3:0 4:1a56e1fc2f8f359 5:1 6:0
|
||||
series __name__=a_bucket|job=x ; 0:0 1:1a56e1fc2f8f359 2:0 3:0 4:1a56e1fc2f8f359 5:0 6:0
|
||||
series __name__=a_bucket|job=y ; 0:402a000000000000 1:4035000000000000 3:4044000000000000 4:403f000000000000 5:3ff0000000000000 6:4034000000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:404d000000000000 1:4046800000000000 2:4043000000000000 4:4042800000000000 5:4040000000000000 6:403a000000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:405a400000000000 1:4049800000000000 3:405e800000000000 4:404e000000000000 5:4042000000000000 6:4045800000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:7ff0000000000000 1:404c000000000000 2:0 3:4061000000000000 4:4059000000000000 5:0 6:4056400000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 1:4059000000000000 2:4020000000000000 3:4062a00000000000 4:405bc00000000000 5:c014000000000000 6:405e800000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:404b800000000000 2:4052400000000000 4:4044000000000000 5:4047800000000000 6:4050800000000000
|
||||
result job=x|quantile=NaN ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001
|
||||
result job=y|quantile=NaN ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
result job=z|quantile=NaN ; 0:7ff8000000000001 2:7ff8000000000001 4:7ff8000000000001 5:7ff8000000000001 6:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 3fefae147ae147ae,3fefae147ae147ae,7ff8000000000001
|
||||
grid 100000 10000 2
|
||||
series __name__=a_bucket|job=x|le=0.25 ; 0:403a000000000000 1:4030000000000000
|
||||
series __name__=a_bucket|job=x|le=2.5 ; 0:4039fffffffffd24
|
||||
series __name__=a_bucket|job=x|le=inf ; 0:404afffffffffe92 1:404d800000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:4046800000000000
|
||||
series __name__=a_bucket|job=y|le=1.0 ; 0:7ff0000000000000 1:4018000000000000
|
||||
series __name__=a_bucket|job=y|le=2 ; 0:7ff0000000000000 1:4046000000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:7ff0000000000000 1:404b800000000000
|
||||
series __name__=b_bucket|job=z|le=-1 ; 0:404b800000000000 1:4043e00000000000
|
||||
series __name__=b_bucket|job=z|le=0 ; 0:405ec00000000000 1:4060900000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4062f80000000000
|
||||
series __name__=b_bucket|job=z ; 0:406e400000000000 1:406a380000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile 3ff0000000000000,7ff8000000000001,3ff8000000000000
|
||||
grid 100000 10000 1
|
||||
series __name__=a_bucket|job=x|le=0.1 ; 0:403d000000000000
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:403d000000000330
|
||||
series __name__=a_bucket|job=x ;
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:404ffffffffffe13
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:404ffffffffffe13
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:4055000000000000
|
||||
series __name__=a_bucket|job=y|le=abc ; 0:404f800000000000
|
||||
series __name__=a_bucket|job=y|le=1e400 ;
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:4053000000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:404bc00000000000
|
||||
series __name__=b_bucket|job=z|le=1.0 ; 0:4058900000000000
|
||||
series __name__=b_bucket|job=z|le=2 ;
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4065480000000000
|
||||
result job=x|quantile=1.0 ; 0:4014000000000000
|
||||
result job=x|quantile=1.5 ; 0:7ff0000000000000
|
||||
result job=x|quantile=NaN ; 0:7ff8000000000001
|
||||
result job=y|quantile=1.0 ; 0:3ff0000000000000
|
||||
result job=y|quantile=1.5 ; 0:7ff0000000000000
|
||||
result job=y|quantile=NaN ; 0:7ff8000000000001
|
||||
result job=z|quantile=1.0 ; 0:3ff0000000000000
|
||||
result job=z|quantile=1.5 ; 0:7ff0000000000000
|
||||
result job=z|quantile=NaN ; 0:7ff8000000000001
|
||||
end
|
||||
case quantiles quantile 3fd0000000000000,bfe0000000000000,3fefae147ae147ae
|
||||
grid 100000 10000 6
|
||||
series __name__=a_bucket|job=x|le=-1 ; 0:4045000000000000 1:4042000000000000 2:4046000000000000 3:403d000000000000 4:3ff0000000000000
|
||||
series __name__=a_bucket|job=x ; 1:404e800000000000 3:403d000000000330 5:4038000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:405d400000000000 1:4057c00000000000 2:404fffffffff0e1c 3:403d000000000330
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 1:4057c00000010522 2:4057ffffffff870e 3:4041000000000198 4:4030000000000000 5:4048000000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:0 1:0 2:0 3:0 4:0
|
||||
series __name__=a_bucket|job=y|le=2 ; 0:1 2:1a56e1fc2f8f359 3:1 4:0 5:0
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:1a56e1fc2f8f359 1:0 2:0 3:1 4:0
|
||||
series __name__=b_bucket|job=y|le=-1 ; 0:402c000000000000 1:4018000000000000 2:4014000000000000 3:4047000000000000 4:4024000000000000 5:4031000000000000
|
||||
series __name__=b_bucket|job=y|le=0 ; 0:404a800000000000 2:4040800000000000 4:4031000000000000 5:404c800000000000
|
||||
series __name__=b_bucket|job=y|le=1 ; 0:404e800000000000 1:4055000000000000 2:4053400000000000 3:4046fffffffffd78 4:4040800000000000 5:404c800000000322
|
||||
series __name__=b_bucket|job=y|le=+Inf ; 1:4054ffffffff191a 2:4053400000000000 3:4057bffffffffebc 4:404080000000b56b 5:404c800000013c7e
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4037000000000000 1:4048e00000000000 2:4050800000000000 3:404d400000000000 4:4057800000000000
|
||||
series __name__=b_bucket|job=z|le=abc ; 0:405a500000000000 1:405ea00000000000 2:4062a00000000000 4:405f800000000000
|
||||
series __name__=b_bucket|job=z|le=1e400 ; 0:405eb00000000000 1:4067b80000000000 2:406d300000000000 3:406ab00000000000 4:4064b80000000000 5:4062800000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4061000000000000 1:406df00000000000 2:4070100000000000 3:4072040000000000 4:4065280000000000 5:4064880000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 3fe0000000000000,3ff0000000000000,bfe0000000000000
|
||||
grid 100000 10000 1
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4012000000000000
|
||||
series __name__=a_bucket|job=x ; 0:403d000000000000
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:4055f00000000000
|
||||
series __name__=b_bucket|job=z|le=1 ;
|
||||
series __name__=b_bucket|job=z|le=1.0 ; 0:4042000000000000
|
||||
series __name__=b_bucket|job=z|le=2 ; 0:404b000000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:404b000000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 3ff8000000000000,0
|
||||
grid 100000 10000 3
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4041800000000000 1:4024000000000000 2:4000000000000000
|
||||
series __name__=b_bucket|job=z|le=abc ; 0:40417fffffff3f96 1:402e000000000000 2:4000000000000000
|
||||
series __name__=b_bucket|job=z|le=1e400 ; 0:40527fffffff9fcb 1:4045000000000000 2:40000000000001c2
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:405dbfffffff9fcb 1:4044ffffffff191a 2:400000000000b1ae
|
||||
result job=0.0 ; 0:0 1:0 2:0
|
||||
result job=1.5 ; 0:7ff0000000000000 1:7ff0000000000000 2:7ff0000000000000
|
||||
end
|
||||
case quantiles quantile 3ff0000000000000,3fe0000000000000,3fefae147ae147ae
|
||||
grid 100000 10000 4
|
||||
series __name__=a_bucket|job=x|le=0.1 ; 0:c014000000000000 1:4026000000000000 2:4010000000000000 3:0
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:4000000000000000 2:403e000000000000 3:403a000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:4038000000000000 1:4054000000000000 2:404a000000000000 3:4044800000000000
|
||||
series __name__=a_bucket|job=x|le=5 ; 1:4057000000000000 2:404b000000000000 3:4056000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:7ff8000000000001 1:4060400000000000 2:4055c00000000000 3:4061000000000000
|
||||
series __name__=a_bucket|job=y|le=0.1 ; 0:0 1:0 2:0 3:0
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:1 1:0 2:0 3:1
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:1 1:0 2:0 3:1
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:0 2:0 3:1
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:0 1:0 3:0
|
||||
result job=x|quantile=0.5 ; 0:3ff0000000000000 1:3fe9bd37a6f4de9c 2:3fe9d1745d1745d2 3:400a620ae4c415ca
|
||||
result job=x|quantile=0.99 ; 0:3ff0000000000000 1:4014000000000000 2:4014000000000000 3:4014000000000000
|
||||
result job=x|quantile=1.0 ; 0:3ff0000000000000 1:4014000000000000 2:4014000000000000 3:4014000000000000
|
||||
result job=y|quantile=0.5 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001
|
||||
result job=y|quantile=0.99 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001
|
||||
result job=y|quantile=1.0 ; 0:7ff8000000000001 1:7ff8000000000001 2:7ff8000000000001 3:7ff8000000000001
|
||||
end
|
||||
case quantiles job 3fefae147ae147ae
|
||||
grid 100000 10000 1
|
||||
series __name__=a_bucket|job=x|le=0.25 ; 0:4055e00000000000
|
||||
series __name__=a_bucket|job=x|le=2.5 ; 0:4059200000000000
|
||||
series __name__=a_bucket|job=x|le=inf ; 0:405b200000000000
|
||||
series __name__=a_bucket|job=y|le=1 ; 0:4051300000000000
|
||||
series __name__=a_bucket|job=y|le=2 ; 0:4057800000000000
|
||||
series __name__=a_bucket|job=y|le=5 ; 0:4058b00000000000
|
||||
series __name__=b_bucket|job=z|le=-1 ; 0:4051c00000000000
|
||||
series __name__=b_bucket|job=z|le=0 ; 0:4060400000000000
|
||||
series __name__=b_bucket|job=z|le=1 ; 0:4065280000000000
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4066880000000000
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles job 3fe0000000000000,3fe0000000000000
|
||||
grid 100000 10000 5
|
||||
series __name__=a_bucket|job=x|le=0.1 ; 0:4008000000000000 3:4022000000000000 4:403a000000000000
|
||||
series __name__=a_bucket|job=x|le=0.5 ; 0:4032000000000000 1:4053400000000000 3:40220000000001fa 4:4040000000000000
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:40320000000001fa 1:40533fffffff2c58 2:404200000000c5ea 3:404c80000000007e
|
||||
series __name__=a_bucket|job=x|le=5 ; 0:40320000000003f4 2:4042000000018bd4 4:4054000000000000
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 1:40533fffffff2c57 2:40420000000251be 3:404c7ffffffd8dc6 4:4053ffffffff2419
|
||||
series __name__=b_bucket|job=y|le=1 ; 0:4050c00000000000 2:4055500000000000 3:4044c00000000000 4:4047000000000000
|
||||
series __name__=b_bucket|job=y|le=1.0 ; 0:405b000000000000 1:405f400000000000 2:4066c00000000000 3:4054500000000000 4:404b800000000000
|
||||
series __name__=b_bucket|job=y|le=2 ; 0:405e400000000000 1:406a500000000000 2:406b980000000000 4:405d100000000000
|
||||
series __name__=b_bucket|job=y|le=+Inf ; 0:4060b80000000000 1:4072180000000000 2:406e400000000000 3:4067500000000000 4:4067580000000000
|
||||
series __name__=b_bucket|job=z|le=-1 ; 0:0 1:0 2:0 3:0 4:0
|
||||
series __name__=b_bucket|job=z|le=0 ; 0:1 1:0 3:1a56e1fc2f8f359 4:0
|
||||
series __name__=b_bucket|job=z ; 0:1 1:0 2:1a56e1fc2f8f359 3:0 4:0
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:0 1:1 3:0
|
||||
error vector cannot contain metrics with the same labelset
|
||||
end
|
||||
case quantiles quantile bfe0000000000000,0
|
||||
grid 100000 10000 8
|
||||
series __name__=a_bucket|job=x|le=1 ; 0:0 1:0 5:0 6:0
|
||||
series __name__=a_bucket|job=x ; 0:1a56e1fc2f8f359 1:0 2:0 3:0 4:0 5:0 6:0 7:1
|
||||
series __name__=a_bucket|job=x|le=2 ; 0:1 1:1a56e1fc2f8f359 2:0 3:1 4:0 5:0 6:1a56e1fc2f8f359 7:0
|
||||
series __name__=a_bucket|job=x|le=+Inf ; 0:0 1:0 2:1 3:0 4:1a56e1fc2f8f359 5:0 6:0 7:0
|
||||
series __name__=a_bucket|job=y|le=0.1 ; 0:403b000000000000 1:4040000000000000 2:4018000000000000 3:4008000000000000 4:4042000000000000 5:403c000000000000 6:4045800000000000 7:4032000000000000
|
||||
series __name__=a_bucket|job=y|le=0.5 ; 0:4052000000000000 1:404b800000000000 2:4017fffffffffd5c 3:4008000000000000 5:403c000000000000 6:4045800000000000 7:4032000000000000
|
||||
series __name__=a_bucket|job=y ; 0:405bc00000000000 1:404b800000012e5e 2:4017fffffffef57a 3:4030000000000000 4:4051400000000000 7:4045000000000000
|
||||
series __name__=a_bucket|job=y|le=+Inf ; 0:4061c0000000c32a 1:404e000000012e5e 2:402ffffffffef6cc 3:402ffffffffea028 4:405ac00000000000 5:40587fffffffff3b 6:40458000000004ba
|
||||
series __name__=b_bucket|job=z ; 0:4000000000000000 1:4043000000000000 2:402a000000000000 3:403c000000000000 4:4043800000000000 5:4043800000000000 6:4020000000000000 7:0
|
||||
series __name__=b_bucket|job=z|le=1.0 ; 0:4014000000000000 1:4050400000000000 2:402a000000000000 3:4046000000000000 4:4049800000000000 5:4055c00000000000 6:4039000000000000 7:403c000000000000
|
||||
series __name__=b_bucket|job=z|le=2 ; 0:4018000000000000 1:4056400000000000 2:402a0000000002db 3:4046000000000000 4:40497ffffffee7a0 5:405f800000000000 6:4038fffffffeed1f 7:403c0000000133dd
|
||||
series __name__=b_bucket|job=z|le=+Inf ; 0:4047800000000000 1:405640000000f4a4 2:40418000000000b7 3:404600000000f1e4 4:40497ffffffee7a0 5:405f800000000376 6:4038fffffffeed1f 7:403c000000000000
|
||||
result job=x|quantile=-0.5 ; 0:fff0000000000000 1:fff0000000000000 2:fff0000000000000 3:fff0000000000000 4:fff0000000000000 5:fff0000000000000 6:fff0000000000000 7:fff0000000000000
|
||||
result job=x|quantile=0.0 ; 0:7ff8000000000001 1:fff8000000000000 2:7ff8000000000001 3:0 4:fff8000000000000 5:7ff8000000000001 6:fff8000000000000 7:7ff8000000000001
|
||||
result job=y|quantile=-0.5 ; 0:fff0000000000000 1:fff0000000000000 2:fff0000000000000 3:fff0000000000000 4:fff0000000000000 5:fff0000000000000 6:fff0000000000000 7:fff0000000000000
|
||||
result job=y|quantile=0.0 ; 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:7ff8000000000001
|
||||
result job=z|quantile=-0.5 ; 0:fff0000000000000 1:fff0000000000000 2:fff0000000000000 3:fff0000000000000 4:fff0000000000000 5:fff0000000000000 6:fff0000000000000 7:fff0000000000000
|
||||
result job=z|quantile=0.0 ; 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
|
||||
end
|
||||
@@ -0,0 +1,47 @@
|
||||
0/0/4000000000000000/4014000000000000/400c000000000000/0/0_3/3ff0000000000000,0,4000000000000000/// {count:5, sum:3.5, [-0,0]:2, (0.5,1]:1, (2,4]:2}
|
||||
1/3f50624dd2f1a9fc/0/4132d68700000000/be7ad7f29abcaf48/0/20_1/0/-2_2,3_1/3ff0000000000000,4000000000000000,4132d68400000000/ {count:1.234567e+06, sum:-1e-07, [-2.82842712474619,-2):1.234564e+06, [-0.7071067811865475,-0.5):2, [-0.5,-0.35355339059327373):1}
|
||||
-53/0/0/4018000000000000/4028000000000000/0/0_3/3ff0000000000000,4000000000000000,4008000000000000///3ff0000000000000,4014000000000000 {count:6, sum:12, [-Inf,1]:1, (1,5]:2, (5,+Inf]:3}
|
||||
-53/0/0/4000000000000000/7ff8000000000001/0/1_2/3fe0000000000000,3ff8000000000000///bff0000000000000,0,3fe0000000000000 {count:2, sum:NaN, (-1,0]:0.5, (0,0.5]:1.5}
|
||||
3/0/0/7ff0000000000000/fff0000000000000/0///// {count:+Inf, sum:-Inf}
|
||||
0/0/0/4008000000000000/4000000000000000/0/0_1,0_0,2_1/3ff0000000000000,7ff8000000000001/// {count:3, sum:2, (0.5,1]:1, (4,8]:NaN}
|
||||
-2/3fe0000000000000/3ff0000000000000/4008000000000000/3ff0000000000000/0/-1_2/3ff0000000000000,3ff0000000000000/// {count:3, sum:1, [-0.5,0.5]:1, (0.00390625,0.0625]:1, (0.0625,1]:1}
|
||||
3/3fe0000000000000/0/4055001175c76074/40adb352ce0f464f/3/-2_1/0/-19_1,2_4/4042800000000000,3f2aa1d1faf86ea9,0,3f4c431a422a645b,4047800000000000/ {count:84.00106567831409, sum:3801.6617283604933, [-0.3242098886627524,-0.29730177875068026):47, [-0.29730177875068026,-0.2726269331663144):0.0008624914374478864, [-0.25,-0.22925101080116778):0.00020318687664732285, [-0.1927763531759926,-0.17677669529663687):37}
|
||||
-53/0/0/405bc00000000000/c068dcd4a171fac8/3/0_2/4040800000000000,4053800000000000///3ff0000000000000,4014000000000000,4024000000000000 {count:111, sum:-198.90095588933605, [-Inf,1]:33, (1,5]:78}
|
||||
0/255bba08cf8c979d/0/0/8000000000000000/1///// {count:0, sum:-0}
|
||||
5/255bba08cf8c979d/404c000000000000/404c000000000000/40a0bdb814b160c1/3///// {count:56, sum:2142.8595328741653, [-1e-128,1e-128]:56}
|
||||
-1/255bba08cf8c979d/3f36cdb2e4f0a26e/3f52159408fe4cb1/bf9324164071cb30/3/4_1/3f48c44e9f84482b/// {count:0.0011037774711198207, sum:-0.018692348178272755, [-1e-128,1e-128]:0.0003479539636282229, (64,256]:0.0007558235074915978}
|
||||
-1/3f847ae147ae147b/0/406308039013db44/40b9864b9433bdfa/3/-9_1,0_4/0,4053a00000000000,4014000000000000,3f3c809eda1bc597,4051300000000000/// {count:152.2504349124739, sum:6534.2952301348505, (3.814697265625e-06,1.52587890625e-05]:78.5, (1.52587890625e-05,6.103515625e-05]:5, (6.103515625e-05,0.000244140625]:0.0004349124738914577, (0.000244140625,0.0009765625]:68.75}
|
||||
-2/0/0/407a5c05888d260f/40d2aa08d6c06f03/1/-7_1,1_4,1_4/0,3f43aa8dcd5315f5,4053c00000000000,4051800000000000,4055800000000000,4059700000000000,0,4055400000000000,3f4899db6326e870/// {count:421.7513509286518, sum:19112.138107403192, (5.960464477539063e-08,9.5367431640625e-07]:0.0006001655953233308, (9.5367431640625e-07,1.52587890625e-05]:79, (1.52587890625e-05,0.000244140625]:70, (0.000244140625,0.00390625]:86, (0.0625,1]:101.75, (16,256]:85, (256,4096]:0.0007507630564795986}
|
||||
-4/3fe0000000000000/0/0/0/3///// {count:0, sum:0}
|
||||
-1/255bba08cf8c979d/4014000000000000/4073bc05b4869dbb/40d03bb730a8c0e4/0/-7_3/3f483e1bc460aff5,4053800000000000,0/-17_4,1_3/0,4044000000000000,3f4566192979cc81,4054700000000000,4043800000000000,402a000000000000,404d800000000000/ {count:315.75139286598613, sum:16622.862344921465, [-9.5367431640625e-07,-2.384185791015625e-07):59, [-2.384185791015625e-07,-5.960464477539063e-08):13, [-5.960464477539063e-08,-1.4901161193847656e-08):39, [-3.725290298461914e-09,-9.313225746154785e-10):81.75, [-9.313225746154785e-10,-2.3283064365386963e-10):0.0006530402051353608, [-2.3283064365386963e-10,-5.820766091346741e-11):40, [-1e-128,1e-128]:5, (1.52587890625e-05,6.103515625e-05]:0.0007398257810158337, (6.103515625e-05,0.000244140625]:78}
|
||||
0/3f50624dd2f1a9fc/404f000000000000/4076e803fd306cca/40ca9dc9569918a8/0///-4_1,2_2,0_4/403f600000000000,3f4fe983664f7c80,0,405d880000000000,4026000000000000,4056000000000000,404c000000000000/ {count:366.5009738818741, sum:13627.572955262222, [-16,-8):56, [-8,-4):88, [-4,-2):11, [-2,-1):118.125, [-0.5,-0.25):0.0009738818740706728, [-0.0625,-0.03125):31.375, [-0.001,0.001]:62}
|
||||
1/0/0/406c800000000000/40cb50bfd34524c0/0/3_2,1_1,1_1/404e000000000000,403d000000000000,404e000000000000,4053c00000000000/// {count:228, sum:13985.498634951538, (2,2.82842712474619]:60, (2.82842712474619,4]:29, (5.65685424949238,8]:60, (11.31370849898476,16]:79}
|
||||
2/0/0/4054480000000000/40b0f29c9a52a00d/0/-13_1/4054480000000000/// {count:81.125, sum:4338.611729778361, (0.08838834764831843,0.1051120519067143]:81.125}
|
||||
-4/0/3f4b4dbd2c7610e7/40826a048a221018/40a75eb15b3b6ea7/2/-15_3,2_2,2_1/3f3bc20af1adbdf4,4057c00000000000,0,4038000000000000,4018000000000000,3f4b85fd5a261456/-8_4,2_3,1_1/4058100000000000,404b800000000000,4045800000000000,4050800000000000,4050c00000000000,3f1f6b081088c64f,404a800000000000,4055000000000000/ {count:589.2522165928995, sum:2991.346399171121, [-4.294967296e+09,-65536):84, [-1,-1.52587890625e-05):53, [-1.52587890625e-05,-2.3283064365386963e-10):0.0001198505089028631, [-2.3283064365386963e-10,-3.552713678800501e-15):67, [-8.271806125530277e-25,-1.262177448353619e-29):66, [-1.262177448353619e-29,-1.925929944387236e-34):43, [-1.925929944387236e-34,-2.938735877055719e-39):55, [-2.938735877055719e-39,-4.484155085839415e-44):96.25, [-0,0]:0.0008332418155281546, (8.636168555094445e-78,5.659799424266695e-73]:0.00042355315388584104, (5.659799424266695e-73,3.7092061506874214e-68]:95, (1.044048714879764e-53,6.842277657836021e-49]:24, (6.842277657836021e-49,4.484155085839415e-44]:6, (1.925929944387236e-34,1.262177448353619e-29]:0.0008399474211705598}
|
||||
5/255bba08cf8c979d/0/40770c0000000000/40b5fa8d74c0ae79/1/4_3,0_1/404c800000000000,4043800000000000,404ef00000000000,4056380000000000/-4_1/405e800000000000/ {count:368.75, sum:5626.552562754246, [-0.9170040432046711,-0.8973545375015533):122, (1.0671404006768235,1.0905077326652577]:57, (1.0905077326652577,1.1143867425958924]:39, (1.1143867425958924,1.1387886347566916]:61.875, (1.1387886347566916,1.1637248587775775]:88.875}
|
||||
5/0/0/4064400710556858/40c19327f5a61c63/3/-2_3,2_2,2_2/3f4015141ed88149,4014000000000000,3f385883050fd155,0,0,4054000000000000,4053400000000000/// {count:162.00086228066607, sum:8998.312184108601, (0.9370838170551498,0.9576032806985735]:0.0004907940144143554, (0.9576032806985735,0.9785720620876999]:5, (0.9785720620876999,1]:0.0003714866516582223, (1.1387886347566916,1.1637248587775775]:80, (1.1637248587775775,1.189207115002721]:77}
|
||||
0/3f847ae147ae147b/0/4075b203e4a2a823/40b7d2e96b597241/0/-15_3,0_3/4052000000000000,405a480000000000,4058000000000000,3f4d7879838a47e5,4052800000000000,3f0ac9bbd92a4212/// {count:347.1259504655629, sum:6098.911794271845, (1.52587890625e-05,3.0517578125e-05]:72, (3.0517578125e-05,6.103515625e-05]:105.125, (6.103515625e-05,0.0001220703125]:96, (0.0001220703125,0.000244140625]:0.0008993714646570143, (0.000244140625,0.00048828125]:74, (0.00048828125,0.0009765625]:5.1094098258212245e-05}
|
||||
5/3f847ae147ae147b/4055c00000000000/4055c00000000000/4094695b49a7cbae/2///// {count:87, sum:1306.33914816075, [-0.01,0.01]:87}
|
||||
-2/3f50624dd2f1a9fc/0/4050000000000000/c074ba1db757b67f/1/4_1,0_1/401c000000000000,404c800000000000/// {count:64, sum:-331.6322549272808, (4096,65536]:7, (65536,1.048576e+06]:57}
|
||||
0/3fe0000000000000/0/4075ce01dd4d0692/40ce28744fb7f0d2/1/-11_4/4053b80000000000,405e400000000000,4052000000000000,4053400000000000/3_4/3f3dd4d069223133,0,0,0/ {count:348.8754551895919, sum:15440.908682816113, [-8,-4):0.00045518959189718154, (0.000244140625,0.00048828125]:78.875, (0.00048828125,0.0009765625]:121, (0.0009765625,0.001953125]:72, (0.001953125,0.00390625]:77}
|
||||
0/3fe0000000000000/3f20901594b4379e/406c14063509d9f0/40c9c9a9524a50e8/1/-6_2,1_2,1_1/4041000000000000,4042500000000000,4055c00000000000,3f44b0220295c4b2,4050c00000000000/// {count:224.6257577125939, sum:13203.322823800569, [-0.5,0.5]:0.00012636435926832457, (0.0078125,0.015625]:34, (0.015625,0.03125]:36.625, (0.0625,0.125]:87, (0.125,0.25]:0.0006313482346468528, (0.5,1]:67}
|
||||
-53/0/0/4024000000000000/4038a4e446df92d7/2/0_2/4024000000000000,0///3ff0000000000000,4014000000000000,4024000000000000 {count:10, sum:24.644108228286168, [-Inf,1]:10}
|
||||
-1/255bba08cf8c979d/0/4081a80000000000/c0c61d084603aa97/1/-16_4,0_4,1_3/404bd00000000000,403e000000000000,0,4041000000000000,4050800000000000,4046800000000000,403c600000000000,4057800000000000,4052800000000000,4049000000000000,4056000000000000/// {count:565, sum:-11322.064636667534, (5.820766091346741e-11,2.3283064365386963e-10]:55.625, (2.3283064365386963e-10,9.313225746154785e-10]:30, (3.725290298461914e-09,1.4901161193847656e-08]:34, (1.4901161193847656e-08,5.960464477539063e-08]:66, (5.960464477539063e-08,2.384185791015625e-07]:45, (2.384185791015625e-07,9.5367431640625e-07]:28.375, (9.5367431640625e-07,3.814697265625e-06]:94, (1.52587890625e-05,6.103515625e-05]:74, (6.103515625e-05,0.000244140625]:50, (0.000244140625,0.0009765625]:88}
|
||||
-4/255bba08cf8c979d/402e400000000000/407caa0113ee4a12/40de3cd98f49b46d/1/-3_3,2_4,1_2/403b000000000000,404a000000000000,0,4047800000000000,4028000000000000,0,4010000000000000,3f313ee4a128a361,4037000000000000/-16_4,2_1,2_2/402a000000000000,403c000000000000,4036800000000000,4054c00000000000,4049000000000000,4044000000000000,4045000000000000/ {count:458.62526314813397, sum:30963.399370599935, [-1.262177448353619e-29,-1.925929944387236e-34):42, [-1.925929944387236e-34,-2.938735877055719e-39):40, [-6.842277657836021e-49,-1.044048714879764e-53):50, [-2.4308653429145085e-63,-3.7092061506874214e-68):83, [-3.7092061506874214e-68,-5.659799424266695e-73):22.5, [-5.659799424266695e-73,-8.636168555094445e-78):28, [-8.636168555094445e-78,-1.3177747429038154e-82):13, [-1e-128,1e-128]:15.125, (5.421010862427522e-20,3.552713678800501e-15]:27, (3.552713678800501e-15,2.3283064365386963e-10]:52, (65536,4.294967296e+09]:47, (4.294967296e+09,2.81474976710656e+14]:12, (1.8446744073709552e+19,1.2089258196146292e+24]:4, (7.922816251426434e+28,5.192296858534828e+33]:0.00026314813399835017, (5.192296858534828e+33,3.402823669209385e+38]:23}
|
||||
5/3f50624dd2f1a9fc/0/0/8000000000000000/3///// {count:0, sum:-0}
|
||||
8/3f847ae147ae147b/4054800000000000/40716a00d423d86a/c088bf7fb4b025f5/2/-16_1,1_3/4039000000000000,3f2a847b0d4f57f6,4049800000000000,405e280000000000/// {count:278.6252023124895, sum:-791.9373563539108, [-0.01,0.01]:82, (0.9550139751351947,0.9576032806985735]:25, (0.9601996065815236,0.9628029718180622]:0.0002023124895049186, (0.9628029718180622,0.9654133954938133]:51, (0.9654133954938133,0.9680308967461471]:120.625}
|
||||
-1/255bba08cf8c979d/4055400000000000/4072be0000000000/40a8ecb19995b81f/2/-9_1,2_4/4048000000000000,0,403b600000000000,4053a00000000000,404e800000000000/// {count:299.875, sum:3190.346874884344, [-1e-128,1e-128]:85, (9.5367431640625e-07,3.814697265625e-06]:48, (0.000244140625,0.0009765625]:27.375, (0.0009765625,0.00390625]:78.5, (0.00390625,0.015625]:61}
|
||||
0/3f847ae147ae147b/404a000000000000/4063e8030e6cf560/4081209123fc57ba/3/-20_1,2_3/4036400000000000,4036000000000000,3f387367ab034118,404f000000000000/-13_2/3ff0000000000000,0/ {count:159.25037308960873, sum:548.0708694185516, [-0.0001220703125,-6.103515625e-05):1, [-0.01,0.01]:52, (4.76837158203125e-07,9.5367431640625e-07]:22.25, (3.814697265625e-06,7.62939453125e-06]:22, (7.62939453125e-06,1.52587890625e-05]:0.00037308960874062177, (1.52587890625e-05,3.0517578125e-05]:62}
|
||||
-4/3f50624dd2f1a9fc/0/408b9402008af180/c0b46b244e186d84/2/4_3,0_4/0,4010000000000000,404a000000000000,4053200000000000,404a800000000000,4052400000000000,4059000000000000/5_3,2_4,1_4/404e000000000000,4057400000000000,4028000000000000,4054400000000000,4053c00000000000,4055400000000000,4048000000000000,3f5004578c010910,0,404e000000000000,4018000000000000/ {count:882.5009775977087, sum:-5227.141816641557, [-4.9732323640978664e+86,-7.588550360256754e+81):6, [-7.588550360256754e+81,-1.157920892373162e+77):60, [-1.7668470647783843e+72,-2.695994666715064e+67):0.0009775977087121447, [-4.113761393303015e+62,-6.277101735386681e+57):48, [-6.277101735386681e+57,-9.578097130411805e+52):85, [-9.578097130411805e+52,-1.461501637330903e+48):79, [-1.461501637330903e+48,-2.2300745198530623e+43):81, [-5.192296858534828e+33,-7.922816251426434e+28):12, [-7.922816251426434e+28,-1.2089258196146292e+24):93, [-1.2089258196146292e+24,-1.8446744073709552e+19):60, (1.8446744073709552e+19,1.2089258196146292e+24]:4, (1.2089258196146292e+24,7.922816251426434e+28]:52, (7.922816251426434e+28,5.192296858534828e+33]:76.5, (5.192296858534828e+33,3.402823669209385e+38]:53, (3.402823669209385e+38,2.2300745198530623e+43]:73, (2.2300745198530623e+43,1.461501637330903e+48]:100}
|
||||
-53/0/0/0/0/0/////4000000000000000 {count:0, sum:0}
|
||||
-1/3fe0000000000000/3f451c23ba20a83c/4064d005d3d0315d/40a5598b12aa98db/1/-17_4,1_3/4056000000000000,4044800000000000,3ff8000000000000,0,4042000000000000,0,3f1198e85aa23b6b/// {count:166.50071135199224, sum:2732.771626788291, [-0.5,0.5]:0.0006442236374415498, (1.4551915228366852e-11,5.820766091346741e-11]:88, (5.820766091346741e-11,2.3283064365386963e-10]:41, (2.3283064365386963e-10,9.313225746154785e-10]:1.5, (1.4901161193847656e-08,5.960464477539063e-08]:36, (2.384185791015625e-07,9.5367431640625e-07]:6.712835480214361e-05}
|
||||
5/3f50624dd2f1a9fc/0/407da004bdf97b56/40bf8a2ca0dadb2a/3/9_3,0_1,0_2/0,402e000000000000,3f4d33d2909fe965,3f3177f2941fd20e,4010000000000000,0/-1_2,0_4,2_2/4056b00000000000,404f000000000000,404d000000000000,4043000000000000,4056000000000000,4047200000000000,0,4052000000000000/ {count:474.00115773633786, sum:8074.174329451091, [-1.189207115002721,-1.1637248587775775):72, [-1.0905077326652577,-1.0671404006768235):46.25, [-1.0671404006768235,-1.0442737824274138):88, [-1.0442737824274138,-1.0218971486541166):38, [-1.0218971486541166,-1):58, [-1,-0.9785720620876999):62, [-0.9785720620876999,-0.9576032806985735):90.75, (1.2152473599804687,1.241857812073484]:15, (1.241857812073484,1.2690509571917332]:0.0008911874913331395, (1.2690509571917332,1.2968395546510096]:0.0002665488465248008, (1.2968395546510096,1.325236643159741]:4}
|
||||
-53/0/0/0/0/2/1_1/0///3fe0000000000000,3ff0000000000000,4000000000000000,4014000000000000 {count:0, sum:0}
|
||||
-1/255bba08cf8c979d/0/0/0/2///// {count:0, sum:0}
|
||||
8/255bba08cf8c979d/0/0/0/3///// {count:0, sum:0}
|
||||
-53/0/0/0/0/0/////3ff0000000000000,4014000000000000,4024000000000000 {count:0, sum:0}
|
||||
-2/255bba08cf8c979d/0/407aec0000000000/c0c5f6b501bf6e81/3/-16_4,1_1,0_3/4047800000000000,0,4050000000000000,4059500000000000,4054400000000000,0,4034200000000000,0/-5_2/405c580000000000,4010000000000000/ {count:430.75, sum:-11245.41411583801, [-1.52587890625e-05,-9.5367431640625e-07):4, [-9.5367431640625e-07,-5.960464477539063e-08):113.375, (3.3881317890172014e-21,5.421010862427522e-20]:47, (8.673617379884035e-19,1.3877787807814457e-17]:64, (1.3877787807814457e-17,2.220446049250313e-16]:101.25, (3.552713678800501e-15,5.684341886080802e-14]:81, (9.094947017729282e-13,1.4551915228366852e-11]:20.125}
|
||||
-2/0/0/4064b80edc14cbae/40b53ea5f0cc1841/0/-10_1,0_3,0_2/4052000000000000,4050b80000000000,3f4262e9d0820b8b,4030e00000000000,0,0/-10_2,2_2/4024000000000000,3efb4dc57a9c81a3,3f3d19806b2a8e0f,3f49a63afccc53b9/ {count:165.75181392728467, sum:5438.64820552437, [-9.5367431640625e-07,-5.960464477539063e-08):0.0007827556633822542, [-5.960464477539063e-08,-3.725290298461914e-09):0.000444024898721391, [-1.4551915228366852e-11,-9.094947017729282e-13):2.6038927592898807e-05, [-9.094947017729282e-13,-5.684341886080802e-14):10, (5.684341886080802e-14,9.094947017729282e-13]:72, (9.094947017729282e-13,1.4551915228366852e-11]:66.875, (1.4551915228366852e-11,2.3283064365386963e-10]:0.0005611077949797411, (2.3283064365386963e-10,3.725290298461914e-09]:16.875}
|
||||
-1/3f847ae147ae147b/0/4052b80000000000/4095b92e3d3d165a/1/6_2/4051b80000000000,4010000000000000/// {count:74.875, sum:1390.295155481805, (1024,4096]:70.875, (4096,16384]:4}
|
||||
1/3fe0000000000000/0/40718a00b3c301d4/c0acbe7f64b2f3a9/1/-11_3/3f2678603a7aeac0,4050380000000000,403f000000000000/-12_3,2_2/0,4052f00000000000,402e000000000000,0,4057800000000000/ {count:280.62517143416085, sum:-3679.2488151476323, [-0.125,-0.08838834764831843):94, [-0.03125,-0.022097086912079608):15, [-0.022097086912079608,-0.015625):75.75, (0.015625,0.022097086912079608]:0.00017143416083860961, (0.022097086912079608,0.03125]:64.875, (0.03125,0.044194173824159216]:31}
|
||||
3/3f50624dd2f1a9fc/4044400000000000/406290088825a224/40a922533829a9d0/2/4_1,1_2,0_3/0,3f435c1278746a1d,3f3d89082035127a,403d000000000000,4036000000000000,404c800000000000/// {count:148.50104148245111, sum:3217.1625378627577, [-0.001,0.001]:40.5, (1.5422108254079407,1.6817928305074288]:0.0005908098400227466, (1.6817928305074288,1.8340080864093422]:0.00045067261108425844, (1.8340080864093422,2]:29, (2,2.1810154653305154]:22, (2.1810154653305154,2.378414230005442]:57}
|
||||
@@ -25,7 +25,7 @@ use datatypes::series::time::{EvalGrid, GridRange, GridSpec, TimeAxis};
|
||||
use super::float_histogram::{AddOutcome, FloatHistogram};
|
||||
use super::{
|
||||
HistogramAggregate, HistogramFunction, NativeHistogramFunction, aggregate_histograms,
|
||||
evaluate_histograms, map_native_histogram,
|
||||
evaluate_histograms, histogram_quantiles, histogram_string, map_native_histogram,
|
||||
};
|
||||
use crate::grid_series::merge_same_labelset;
|
||||
use crate::grid_series::reference::{
|
||||
@@ -130,6 +130,30 @@ fn classic_histograms_match_prometheus() {
|
||||
assert!(failed > 20, "{failed} failed");
|
||||
}
|
||||
|
||||
/// `histogram_quantiles` over classic histograms, evaluated by the Prometheus v3.15.0 engine with
|
||||
/// experimental functions enabled (promql-compat `narthex-golden-histogram -quantiles`).
|
||||
#[test]
|
||||
fn histogram_quantiles_match_prometheus() {
|
||||
let cases = golden_cases(include_str!("quantiles_cases.txt"));
|
||||
assert_eq!(cases.len(), 30);
|
||||
let mut failed = 0;
|
||||
for case in &cases {
|
||||
let schema = schema(case);
|
||||
let input: Vec<_> = case
|
||||
.inputs
|
||||
.iter()
|
||||
.filter_map(|(labels, points)| mixed_series(&schema, labels, points, &BTreeMap::new()))
|
||||
.collect();
|
||||
let quantiles: Vec<f64> = case.spec[2]
|
||||
.split(',')
|
||||
.map(|bits| f64::from_bits(u64::from_str_radix(bits, 16).unwrap()))
|
||||
.collect();
|
||||
let actual = histogram_quantiles(&case.spec[1], &quantiles, &schema, &input);
|
||||
failed += usize::from(assert_outcome(case, actual, &case.outcome));
|
||||
}
|
||||
assert_eq!(failed, 13, "{failed} failed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_histograms_match_prometheus() {
|
||||
let cases = golden_cases(include_str!("native_cases.txt"));
|
||||
@@ -838,3 +862,18 @@ fn native_histogram_functions_keep_the_runs_of_a_sparse_series() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Go `FloatHistogram.String` over fixed and random histograms (`narthex-golden-histstring`):
|
||||
/// bucket order, inclusiveness, empty buckets skipped, the zero bucket, custom bounds and `%g`.
|
||||
#[test]
|
||||
fn histogram_string_matches_prometheus() {
|
||||
for line in include_str!("string_cases.txt").lines() {
|
||||
let (encoded, expected) = line.split_once('\t').expect("tab-separated case");
|
||||
let histogram = OwnedHistogram::parse(encoded);
|
||||
assert_eq!(
|
||||
histogram_string(&histogram.view()).unwrap(),
|
||||
expected,
|
||||
"{encoded}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+7
-14
@@ -33,7 +33,9 @@ use datatypes::series::MetricName;
|
||||
use datatypes::series::samples::{SampleRef, Series, SeriesSchema};
|
||||
use regex::Regex;
|
||||
|
||||
use crate::grid_series::{GridSample, GridSeriesBuilder, LabelSet, external, label_set, owned};
|
||||
use crate::grid_series::{
|
||||
GridSample, GridSeriesBuilder, LabelSet, external, label_set, owned, render_label_set,
|
||||
};
|
||||
use crate::select::MatchOp;
|
||||
|
||||
/// The labels an info series is joined on, as in Prometheus.
|
||||
@@ -73,7 +75,7 @@ pub fn info(
|
||||
.eval_grid
|
||||
.as_ref()
|
||||
.ok_or_else(|| DataFusionError::Execution("info requires an evaluated grid".into()))?;
|
||||
let points = grid.timestamps().len();
|
||||
let points = grid.spec().len as usize;
|
||||
for series in base {
|
||||
if series.time.grid().map(|grid| grid.spec()) != Some(grid.spec())
|
||||
|| series.fields.len() != schema.fields.len()
|
||||
@@ -171,9 +173,9 @@ pub fn info(
|
||||
format!(
|
||||
"found duplicate series for info metric: existing {} @ {}, new \
|
||||
{} @ {}",
|
||||
render(&label_set(&info[existing_index].metric)),
|
||||
render_label_set(&label_set(&info[existing_index].metric)),
|
||||
existing as i64,
|
||||
render(&label_set(&series.metric)),
|
||||
render_label_set(&label_set(&series.metric)),
|
||||
timestamp as i64
|
||||
)
|
||||
});
|
||||
@@ -281,7 +283,7 @@ pub fn info(
|
||||
error.record(point, 2, || {
|
||||
format!(
|
||||
"vector cannot contain metrics with the same labelset {}",
|
||||
render(&label_set(metric))
|
||||
render_label_set(&label_set(metric))
|
||||
)
|
||||
});
|
||||
}
|
||||
@@ -340,15 +342,6 @@ impl FirstError {
|
||||
}
|
||||
}
|
||||
|
||||
/// A label set as Prometheus prints one.
|
||||
fn render(labels: &LabelSet) -> String {
|
||||
let labels: Vec<String> = labels
|
||||
.iter()
|
||||
.map(|(name, value)| format!("{name}=\"{value}\""))
|
||||
.collect();
|
||||
format!("{{{}}}", labels.join(", "))
|
||||
}
|
||||
|
||||
/// An info metric name and the identifying label values present on a series.
|
||||
type Signature = (Arc<str>, Vec<Option<Arc<str>>>);
|
||||
|
||||
|
||||
@@ -13,11 +13,16 @@
|
||||
// limitations under the License.
|
||||
|
||||
use std::ops::RangeInclusive;
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prometheus::is_prometheus_stale_nan;
|
||||
use datafusion::error::{DataFusionError, Result};
|
||||
use datatypes::series::MetricName;
|
||||
use datatypes::series::raw::RawTimestamps;
|
||||
use datatypes::series::samples::{SampleRef, SeriesFields};
|
||||
use datatypes::series::samples::{SampleRef, Series, SeriesFields, SeriesSchema};
|
||||
|
||||
use crate::grid_series::{GridSeriesBuilder, label_set, render_label_set};
|
||||
use crate::map::drop_metric_name;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
@@ -142,3 +147,176 @@ pub fn select_instants(
|
||||
}
|
||||
Ok((indices, evaluation_timestamps))
|
||||
}
|
||||
|
||||
/// `timestamp(v)` over an instant selector: at each point the selection holds, the original time of
|
||||
/// the selected sample in seconds, not the time it was looked up at, so an offset or `@` moves
|
||||
/// which sample is selected but not the time reported. The metric name, `__type__` and `__unit__`
|
||||
/// are dropped.
|
||||
///
|
||||
/// `indices` and `evaluation_timestamps` are a selection of `time` and `fields` as
|
||||
/// [`select_instants`] returns one: grid positions increasing, rows not decreasing. `schema` holds
|
||||
/// the evaluation grid the timestamps lie on and one field per field of `fields`. A field without a
|
||||
/// sample in the selected row has no result there.
|
||||
///
|
||||
/// `timestamp(x @ t)` selects at `t` for every point, so its selection repeats one row; Prometheus
|
||||
/// ignores an offset next to `@` here, while `info()` applies it.
|
||||
pub fn timestamp_of_selection(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
metric: &MetricName,
|
||||
time: &RawTimestamps,
|
||||
fields: &SeriesFields,
|
||||
indices: &[u64],
|
||||
evaluation_timestamps: &[i64],
|
||||
) -> Result<Option<Series>> {
|
||||
project_selection(
|
||||
schema,
|
||||
drop_metric_name(metric)?,
|
||||
time,
|
||||
fields,
|
||||
indices,
|
||||
evaluation_timestamps,
|
||||
|_, time| Ok(time as f64 / 1000.0),
|
||||
)
|
||||
}
|
||||
|
||||
/// The series `info()` reads its data labels from: at each point the selection holds, the original
|
||||
/// time of the selected sample in milliseconds, as Prometheus' `evalSeries` records it for `info`.
|
||||
/// The labels are kept. Info series hold float samples, so a selected native histogram fails.
|
||||
///
|
||||
/// Takes the same selection as [`timestamp_of_selection`].
|
||||
pub fn info_sample_times(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
metric: &MetricName,
|
||||
time: &RawTimestamps,
|
||||
fields: &SeriesFields,
|
||||
indices: &[u64],
|
||||
evaluation_timestamps: &[i64],
|
||||
) -> Result<Option<Series>> {
|
||||
project_selection(
|
||||
schema,
|
||||
metric.clone(),
|
||||
time,
|
||||
fields,
|
||||
indices,
|
||||
evaluation_timestamps,
|
||||
|sample, time| match sample {
|
||||
SampleRef::Histogram(_) => Err(DataFusionError::Execution(format!(
|
||||
"this should be an info metric, with float samples: {}",
|
||||
render_label_set(&label_set(metric))
|
||||
))),
|
||||
_ => Ok(time as f64),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// `timestamp(v)` over an evaluated `v`, such as `timestamp(abs(x))`: the evaluation time in
|
||||
/// seconds at each point a field of `series` holds a value. The metric name, `__type__` and
|
||||
/// `__unit__` are dropped.
|
||||
pub fn timestamp_of_evaluated(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
series: &Series,
|
||||
) -> Result<Option<Series>> {
|
||||
let grid = schema.eval_grid.as_ref().ok_or_else(|| {
|
||||
DataFusionError::Execution("timestamp requires an evaluation grid".into())
|
||||
})?;
|
||||
if series.time.grid().map(|grid| grid.spec()) != Some(grid.spec())
|
||||
|| series.fields.len() != schema.fields.len()
|
||||
{
|
||||
return Err(DataFusionError::Execution(
|
||||
"timestamp input does not match its schema".into(),
|
||||
));
|
||||
}
|
||||
let timestamps = grid.timestamps().map_err(crate::grid_series::external)?;
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone())?;
|
||||
let mut cursors: Vec<_> = series.fields.iter().map(|field| field.cursor()).collect();
|
||||
let mut row = Vec::with_capacity(cursors.len());
|
||||
for (point, index) in series.time.positions() {
|
||||
let seconds = timestamps[point] as f64 / 1000.0;
|
||||
row.clear();
|
||||
for cursor in &mut cursors {
|
||||
let sample = cursor
|
||||
.at(index)
|
||||
.expect("the axis and the fields are aligned");
|
||||
row.push((!matches!(sample, SampleRef::Missing)).then_some(seconds));
|
||||
}
|
||||
builder.push(point, &row)?;
|
||||
}
|
||||
builder.finish(drop_metric_name(&series.metric)?)
|
||||
}
|
||||
|
||||
/// One series over an instant selection, each field holding `value(sample, original time)` where
|
||||
/// the selected row has a sample for it.
|
||||
///
|
||||
/// The selection is `indices` and `evaluation_timestamps` as [`select_instants`] returns them: one
|
||||
/// source row and one grid timestamp per selected point, with positions increasing and rows not
|
||||
/// decreasing. An `@` selection, which reads one row at every point, repeats that row; it is the
|
||||
/// caller's to build. A selection off the grid, going back, or naming a row the series does not
|
||||
/// have fails.
|
||||
fn project_selection(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
metric: MetricName,
|
||||
time: &RawTimestamps,
|
||||
fields: &SeriesFields,
|
||||
indices: &[u64],
|
||||
evaluation_timestamps: &[i64],
|
||||
value: impl Fn(SampleRef<'_>, i64) -> Result<f64>,
|
||||
) -> Result<Option<Series>> {
|
||||
let grid = schema.eval_grid.as_ref().ok_or_else(|| {
|
||||
DataFusionError::Execution("an instant selection requires an evaluation grid".into())
|
||||
})?;
|
||||
if indices.len() != evaluation_timestamps.len()
|
||||
|| fields.len() != schema.fields.len()
|
||||
|| fields.iter().any(|field| field.len() != time.len())
|
||||
{
|
||||
return Err(DataFusionError::Execution(
|
||||
"instant selection does not match its series or schema".into(),
|
||||
));
|
||||
}
|
||||
let spec = grid.spec();
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone())?;
|
||||
let mut cursors: Vec<_> = fields.iter().map(|field| field.cursor()).collect();
|
||||
// Consecutive points often select the same row, which a cursor reads only once.
|
||||
let mut row: Vec<Option<f64>> = Vec::with_capacity(fields.len());
|
||||
let mut last: Option<(i128, usize)> = None;
|
||||
for (&index, ×tamp) in indices.iter().zip(evaluation_timestamps) {
|
||||
let offset = timestamp as i128 - spec.start_ms as i128;
|
||||
let position = offset / spec.step_ms as i128;
|
||||
let index = usize::try_from(index)
|
||||
.ok()
|
||||
.filter(|index| *index < time.len());
|
||||
let (Some(index), true) = (
|
||||
index,
|
||||
offset % spec.step_ms as i128 == 0
|
||||
&& (0..spec.len as i128).contains(&position)
|
||||
&& last.is_none_or(|(last_position, _)| last_position < position),
|
||||
) else {
|
||||
return Err(DataFusionError::Execution(
|
||||
"instant selection points off the grid or out of the series".into(),
|
||||
));
|
||||
};
|
||||
match last {
|
||||
Some((_, last_index)) if last_index == index => {}
|
||||
Some((_, last_index)) if last_index > index => {
|
||||
return Err(DataFusionError::Execution(
|
||||
"instant selection rows must not go back".into(),
|
||||
));
|
||||
}
|
||||
_ => {
|
||||
let time = time.timestamp(index).expect("checked against the axis");
|
||||
row.clear();
|
||||
for cursor in &mut cursors {
|
||||
let sample = cursor
|
||||
.at(index)
|
||||
.expect("rows increase and lie in fields as long as the axis");
|
||||
row.push(match sample {
|
||||
SampleRef::Missing => None,
|
||||
sample => Some(value(sample, time)?),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
last = Some((position, index));
|
||||
builder.push(position as usize, &row)?;
|
||||
}
|
||||
builder.finish(metric)
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::ops::RangeInclusive;
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -21,9 +22,14 @@ use datatypes::series::histogram::{
|
||||
BucketSideView, CounterResetHint, HistogramBatch, HistogramBatchBuilder, HistogramView,
|
||||
};
|
||||
use datatypes::series::raw::RawTimestamps;
|
||||
use datatypes::series::samples::{DenseSamplesRef, SampleRef, Samples, SeriesFields};
|
||||
use datatypes::series::samples::{
|
||||
DenseSamplesRef, FieldSpec, SampleRef, Samples, Series, SeriesFields, SeriesSchema,
|
||||
};
|
||||
use datatypes::series::time::{EvalGrid, GridSpec};
|
||||
|
||||
use super::select_instants;
|
||||
use super::{info_sample_times, select_instants, timestamp_of_evaluated, timestamp_of_selection};
|
||||
use crate::grid_series::GridSeriesBuilder;
|
||||
use crate::grid_series::reference::{Labels, labels_of, metric_of};
|
||||
|
||||
fn axis(times: &[i64]) -> RawTimestamps {
|
||||
RawTimestamps::try_new(ScalarBuffer::from(times.to_vec())).unwrap()
|
||||
@@ -250,3 +256,249 @@ fn empty_axis_returns_no_selections() {
|
||||
(vec![], vec![])
|
||||
);
|
||||
}
|
||||
|
||||
fn grid_schema(fields: usize, start: i64, step: i64, points: u64) -> Arc<SeriesSchema> {
|
||||
Arc::new(SeriesSchema {
|
||||
fields: (0..fields)
|
||||
.map(|field| FieldSpec {
|
||||
name: Arc::from(format!("f{field}")),
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.into(),
|
||||
eval_grid: Some(Arc::new(
|
||||
EvalGrid::try_new(GridSpec::try_new(start, step, points).unwrap()).unwrap(),
|
||||
)),
|
||||
})
|
||||
}
|
||||
|
||||
/// Each field's value per position, `None` where it has none.
|
||||
fn values_by_position(series: &Series) -> Vec<(usize, Vec<Option<u64>>)> {
|
||||
series
|
||||
.time
|
||||
.positions()
|
||||
.map(|(position, row)| {
|
||||
let values = series
|
||||
.fields
|
||||
.iter()
|
||||
.map(|field| match field.sample(row) {
|
||||
SampleRef::Float(value) => Some(value.to_bits()),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
(position, values)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `timestamp()` evaluated by the Prometheus v3.15.0 engine, over a selector and over an
|
||||
/// evaluated expression; see the header of `timestamp_cases.txt`.
|
||||
#[test]
|
||||
fn timestamp_matches_prometheus() {
|
||||
const LOOKBACK_MS: i64 = 25_000;
|
||||
let mut checked = 0;
|
||||
for case in include_str!("timestamp_cases.txt")
|
||||
.split("end\n")
|
||||
.filter(|case| case.contains("case "))
|
||||
{
|
||||
let lines: Vec<&str> = case.lines().filter(|line| !line.starts_with('#')).collect();
|
||||
let spec: Vec<&str> = lines[0].split_whitespace().skip(1).collect();
|
||||
let offset: i64 = spec[1].parse().unwrap();
|
||||
let grid: Vec<i64> = lines[2]
|
||||
.split_whitespace()
|
||||
.skip(1)
|
||||
.map(|value| value.parse().unwrap())
|
||||
.collect();
|
||||
let (start, step, points) = (grid[0], grid[1], grid[2] as u64);
|
||||
let schema = grid_schema(1, start, step, points);
|
||||
let end = start + (points as i64 - 1) * step;
|
||||
let mut actual = BTreeMap::new();
|
||||
let mut expected = BTreeMap::new();
|
||||
for line in &lines[3..] {
|
||||
let (kind, rest) = line.split_once(' ').unwrap();
|
||||
let (labels, values) = rest.split_once(" ;").unwrap();
|
||||
let labels: Labels = labels
|
||||
.split(',')
|
||||
.filter(|pair| *pair != "-")
|
||||
.map(|pair| {
|
||||
let (name, value) = pair.split_once('=').unwrap();
|
||||
(name.to_string(), value.to_string())
|
||||
})
|
||||
.collect();
|
||||
let points: Vec<(i64, u64)> = values
|
||||
.split_whitespace()
|
||||
.map(|point| {
|
||||
let (at, bits) = point.split_once(':').unwrap();
|
||||
(at.parse().unwrap(), u64::from_str_radix(bits, 16).unwrap())
|
||||
})
|
||||
.collect();
|
||||
match kind {
|
||||
"raw" => {
|
||||
let times: Vec<i64> = points.iter().map(|(at, _)| *at).collect();
|
||||
let fields = SeriesFields::one(Samples::from_float(Float64Array::from(
|
||||
points
|
||||
.iter()
|
||||
.map(|(_, bits)| f64::from_bits(*bits))
|
||||
.collect::<Vec<_>>(),
|
||||
)));
|
||||
let time = axis(×);
|
||||
let (indices, timestamps) =
|
||||
select_instants(&time, &fields, start..=end, step, LOOKBACK_MS, offset)
|
||||
.unwrap();
|
||||
let metric = metric_of(&labels);
|
||||
let result = if spec[0] == "selector" {
|
||||
timestamp_of_selection(
|
||||
&schema,
|
||||
&metric,
|
||||
&time,
|
||||
&fields,
|
||||
&indices,
|
||||
×tamps,
|
||||
)
|
||||
} else {
|
||||
// Any evaluated series holding the selected points: only where it holds
|
||||
// values matters to `timestamp`.
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone()).unwrap();
|
||||
for at in ×tamps {
|
||||
builder
|
||||
.push(((at - start) / step) as usize, &[Some(1.0)])
|
||||
.unwrap();
|
||||
}
|
||||
match builder.finish(metric).unwrap() {
|
||||
Some(evaluated) => timestamp_of_evaluated(&schema, &evaluated),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
.unwrap();
|
||||
if let Some(result) = result {
|
||||
actual.insert(labels_of(&result.metric), values_by_position(&result));
|
||||
}
|
||||
}
|
||||
"result" => {
|
||||
let values = points
|
||||
.into_iter()
|
||||
.map(|(position, bits)| (position as usize, vec![Some(bits)]))
|
||||
.collect::<Vec<_>>();
|
||||
expected.insert(labels, values);
|
||||
}
|
||||
other => panic!("unexpected line {other}"),
|
||||
}
|
||||
}
|
||||
assert_eq!(actual, expected, "{case}");
|
||||
checked += 1;
|
||||
}
|
||||
assert!(checked > 40, "{checked}");
|
||||
}
|
||||
|
||||
/// The series `info()` reads holds the original time in milliseconds, keeps its labels, and
|
||||
/// leaves a field without a sample in the selected row empty; a native histogram is not an info
|
||||
/// sample.
|
||||
#[test]
|
||||
fn info_sample_times_keep_milliseconds_labels_and_missing_fields() {
|
||||
let schema = grid_schema(2, 100, 10, 3);
|
||||
let time = axis(&[95, 107]);
|
||||
let fields = SeriesFields::try_from_vec(vec![
|
||||
Samples::from_float(Float64Array::from(vec![Some(1.0), Some(1.0)])),
|
||||
Samples::from_float(Float64Array::from(vec![Some(1.0), None])),
|
||||
])
|
||||
.unwrap();
|
||||
let metric = metric_of(&Labels::from([
|
||||
("__name__".to_string(), "target_info".to_string()),
|
||||
("job".to_string(), "x".to_string()),
|
||||
]));
|
||||
let (indices, timestamps) = select_instants(&time, &fields, 100..=120, 10, 20, 0).unwrap();
|
||||
let result = info_sample_times(&schema, &metric, &time, &fields, &indices, ×tamps)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(result.metric, metric);
|
||||
assert_eq!(
|
||||
values_by_position(&result),
|
||||
[
|
||||
(0, vec![Some(95f64.to_bits()), Some(95f64.to_bits())]),
|
||||
(1, vec![Some(107f64.to_bits()), None]),
|
||||
(2, vec![Some(107f64.to_bits()), None]),
|
||||
]
|
||||
);
|
||||
|
||||
let histograms = SeriesFields::one(
|
||||
Samples::try_from_histogram(UInt32Array::from(vec![Some(0)]), histogram_batch(&[1.0]))
|
||||
.unwrap(),
|
||||
);
|
||||
let schema = grid_schema(1, 100, 10, 1);
|
||||
let time = axis(&[95]);
|
||||
let (indices, timestamps) = select_instants(&time, &histograms, 100..=100, 10, 20, 0).unwrap();
|
||||
let error = info_sample_times(&schema, &metric, &time, &histograms, &indices, ×tamps)
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(
|
||||
error.contains(
|
||||
r#"this should be an info metric, with float samples: {__name__="target_info", job="x"}"#
|
||||
),
|
||||
"{error}"
|
||||
);
|
||||
assert!(
|
||||
timestamp_of_selection(&schema, &metric, &time, &histograms, &indices, ×tamps)
|
||||
.unwrap()
|
||||
.is_some(),
|
||||
"timestamp() reads the time of a histogram sample too"
|
||||
);
|
||||
}
|
||||
|
||||
/// A selection must lie on the grid in increasing points and rows.
|
||||
#[test]
|
||||
fn selections_off_the_grid_or_going_back_are_rejected() {
|
||||
let schema = grid_schema(1, 100, 10, 3);
|
||||
let time = axis(&[95, 107]);
|
||||
let fields = SeriesFields::one(Samples::from_float(Float64Array::from(vec![1.0, 2.0])));
|
||||
let metric = metric_of(&Labels::new());
|
||||
for (indices, timestamps) in [
|
||||
(vec![0], vec![105]),
|
||||
(vec![0], vec![130]),
|
||||
(vec![2], vec![100]),
|
||||
(vec![1, 0], vec![100, 110]),
|
||||
(vec![0, 0], vec![110, 100]),
|
||||
] {
|
||||
assert!(
|
||||
timestamp_of_selection(&schema, &metric, &time, &fields, &indices, ×tamps)
|
||||
.is_err(),
|
||||
"{indices:?} {timestamps:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Fields shorter than the time axis are refused, not read past their end.
|
||||
#[test]
|
||||
fn a_selection_over_fields_shorter_than_the_axis_fails() {
|
||||
let schema = grid_schema(1, 100, 10, 3);
|
||||
let time = axis(&[95, 107]);
|
||||
let fields = SeriesFields::one(Samples::from_float(Float64Array::from(vec![1.0])));
|
||||
let metric = metric_of(&Labels::new());
|
||||
assert!(timestamp_of_selection(&schema, &metric, &time, &fields, &[1], &[110]).is_err());
|
||||
}
|
||||
|
||||
/// The evaluation time is reported per field: a field without a value at a point has no result
|
||||
/// there, while the other field does.
|
||||
#[test]
|
||||
fn evaluated_timestamps_follow_each_field() {
|
||||
let schema = grid_schema(2, 100, 10, 3);
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone()).unwrap();
|
||||
builder.push(0, &[Some(1.0), None]).unwrap();
|
||||
builder.push(2, &[Some(1.0), Some(5.0)]).unwrap();
|
||||
let evaluated = builder
|
||||
.finish(metric_of(&Labels::from([(
|
||||
"__name__".to_string(),
|
||||
"a".to_string(),
|
||||
)])))
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
let result = timestamp_of_evaluated(&schema, &evaluated)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(result.metric.name().is_none());
|
||||
assert_eq!(
|
||||
values_by_position(&result),
|
||||
[
|
||||
(0, vec![Some(0.1f64.to_bits()), None]),
|
||||
(2, vec![Some(0.12f64.to_bits()), Some(0.12f64.to_bits())]),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
# Generated by promql-compat cmd/narthex-golden-timestamp built for amd64 (Prometheus v3.15.0
|
||||
# engine, seed 1, lookback 25000ms). `case <selector|evaluated> <offset ms>`, `raw` series as
|
||||
# `ms:hex value bits` samples, then the result series as `grid index:hex value bits`.
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=y ; 79059:c062d80000000000 96484:7ff0000000000002 102784:403e600000000000 108946:7ff0000000000002
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=x ; 81528:4086a80000000000 99475:7ff0000000000002 105265:4078980000000000 108673:c0855b0000000000 112102:c08a490000000000 115733:4078000000000000
|
||||
raw __name__=a,i=1,job=x ; 80563:c063380000000000 87641:c07b3e0000000000 91994:4087c90000000000 107183:7ff0000000000002
|
||||
raw __name__=a,i=2,job=x ; 79703:c077900000000000 86213:4082d10000000000 97479:c061e80000000000 107681:408def0000000000
|
||||
result i=0,job=x ; 0:405461cac083126f 2:405c06872b020c4a
|
||||
result i=1,job=x ; 0:4056ff9db22d0e56 1:4056ff9db22d0e56
|
||||
result i=2,job=x ; 0:40558da1cac08312 1:40585ea7ef9db22d 2:405aeb95810624dd
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=y ; 82463:4072b00000000000 99086:c0680c0000000000 105219:4088c50000000000 116252:4072120000000000 121254:7ff0000000000002 136800:4051a80000000000 151303:4070040000000000 166508:c0666c0000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 4:4061800000000000 5:4062c00000000000 6:4064000000000000 7:4065400000000000
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=y ; 83590:c011000000000000 90143:c08d650000000000 93527:4089fd0000000000 107664:4083b60000000000 113390:40690c0000000000 118469:c07f4a0000000000
|
||||
raw __name__=a,i=1,job=y ; 73981:40783a0000000000 81866:408dbb0000000000 93615:407fa60000000000 105999:7ff0000000000002 113546:c00a000000000000
|
||||
raw __name__=a,i=2,job=x ; 75786:4062040000000000 82426:405a480000000000 85790:4082930000000000 98591:c06ab00000000000 103849:407dfa0000000000 109427:4044600000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000
|
||||
result i=1,job=y ; 0:4059000000000000 2:405e000000000000
|
||||
result i=2,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 1
|
||||
raw __name__=a,i=0,job=x ; 74535:7ff0000000000002 90697:4048800000000000 98068:4071140000000000
|
||||
raw __name__=a,i=1,job=y ; 81720:c0730a0000000000 97704:c081b80000000000
|
||||
raw __name__=a,i=2,job=y ; 77920:40847f0000000000 81615:7ff0000000000002 99071:40818c0000000000
|
||||
result i=0,job=x ; 0:4056ac9ba5e353f8
|
||||
result i=1,job=y ; 0:40546e147ae147ae
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=y ; 75292:c089700000000000 82395:c059c80000000000 89151:40757a0000000000 98308:408c1e0000000000 113489:c068e40000000000 121906:c061d40000000000 138902:7ff0000000000002 154422:407c720000000000 164451:c08ab90000000000
|
||||
raw __name__=a,i=1,job=x ; 71464:c07f840000000000 85645:4084f20000000000 94245:4070800000000000 108778:4064f40000000000 118463:4072a60000000000 128182:7ff0000000000002 134844:7ff0000000000002 147128:4084d40000000000 151571:c064ac0000000000 165145:4088bd0000000000 169481:7ff0000000000002
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000 7:4065400000000000
|
||||
result i=1,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 6:4064000000000000 7:4065400000000000
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 5
|
||||
raw __name__=a,i=0,job=x ; 84648:40853f0000000000 96270:c07dda0000000000 102421:c08ec50000000000 113900:c088fd0000000000 126131:4082320000000000
|
||||
raw __name__=a,i=1,job=y ; 75166:4068cc0000000000 86574:7ff0000000000002 93227:c080f10000000000 103602:c072d60000000000 108139:4075700000000000 117253:c078ba0000000000 123799:408bfc0000000000 139386:40835e0000000000
|
||||
result i=0,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000
|
||||
result i=1,job=y ; 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 1
|
||||
raw __name__=a,i=0,job=x ; 77477:408bc60000000000 81528:408dbe0000000000 86317:c077140000000000
|
||||
raw __name__=a,i=1,job=x ; 74159:c054900000000000 84577:4059500000000000 90230:7ff0000000000002
|
||||
result i=0,job=x ; 0:4059000000000000
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=y ; 70580:c08ef90000000000 87847:4086370000000000 100840:7ff0000000000002 106471:4050b80000000000 112355:4079a40000000000 118517:c0749c0000000000
|
||||
result i=0,job=y ; 0:4051a51eb851eb85 1:4055f6353f7ced91 2:4055f6353f7ced91
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 1
|
||||
raw __name__=a,i=0,job=y ; 72574:407a960000000000 84105:c048500000000000 97507:4068640000000000
|
||||
raw __name__=a,i=1,job=y ; 81462:407b8a0000000000 92655:c074200000000000
|
||||
result i=0,job=y ; 0:40586072b020c49c
|
||||
result i=1,job=y ; 0:405729eb851eb852
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 7
|
||||
raw __name__=a,i=0,job=x ; 76420:4073b00000000000 88427:c085e50000000000 105280:c07f7a0000000000 119020:c08a200000000000 132565:406e800000000000 143696:c0854d0000000000 157325:c07fc60000000000
|
||||
raw __name__=a,i=1,job=y ; 74377:407d3c0000000000 90394:c052780000000000 102174:c056680000000000 117648:c063140000000000 121542:4076d00000000000 132116:c06c480000000000 135511:c07aac0000000000 141161:408eea0000000000 144806:7ff0000000000002
|
||||
result i=0,job=x ; 0:40531ae147ae147b 1:40561b53f7ced917 2:40561b53f7ced917 3:405a51eb851eb852 4:405dc147ae147ae1 5:405dc147ae147ae1 6:406092147ae147ae
|
||||
result i=1,job=y ; 0:40529820c49ba5e3 1:40529820c49ba5e3 2:405699374bc6a7f0 3:40598b22d0e56042 4:405d6978d4fdf3b6 5:405e62b020c49ba6 6:4060f05a1cac0831
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=x ; 75010:c06a840000000000 87311:7ff0000000000002 94231:4071da0000000000 105371:c08b210000000000
|
||||
result i=0,job=x ; 1:40578ec8b4395810
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=y ; 80325:4077720000000000 89569:c081b00000000000 94056:c080f70000000000 103981:4089050000000000
|
||||
raw __name__=a,i=1,job=x ; 78176:c08ed60000000000 90610:4080a70000000000 104523:406b7c0000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000
|
||||
result i=1,job=x ; 0:4059000000000000 1:405b800000000000
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 7
|
||||
raw __name__=a,i=0,job=x ; 79485:7ff0000000000002 97045:c05df00000000000 111627:4046100000000000 122143:c085020000000000 129698:c07c140000000000 143915:7ff0000000000002 154094:c04b900000000000
|
||||
result i=0,job=x ; 1:405842e147ae147b 2:405be820c49ba5e3 3:405e8926e978d4fe 4:4060365604189375 5:4060365604189375
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 1
|
||||
raw __name__=a,i=0,job=y ; 79969:c083fc0000000000 88978:c085590000000000
|
||||
raw __name__=a,i=1,job=y ; 72442:4071ac0000000000 87172:c07bb20000000000 98532:c061f00000000000
|
||||
raw __name__=a,i=2,job=y ; 77328:7ff0000000000002 86590:c0836a0000000000 90940:c08d980000000000 98432:c083c40000000000
|
||||
result i=0,job=y ; 0:4053fe04189374bc
|
||||
result i=1,job=y ; 0:40521c49ba5e353f
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=x ; 82283:c07b600000000000 95023:c082690000000000 104613:408b9b0000000000 121027:c08e1a0000000000 137295:c08b4c0000000000 149491:7ff0000000000002 155253:c076ba0000000000 167374:408d0b0000000000
|
||||
result i=0,job=x ; 0:4057c178d4fdf3b6 1:405a273b645a1cac 2:405a273b645a1cac 3:405e41ba5e353f7d 4:40612970a3d70a3d 6:406368189374bc6a 7:4064ebf7ced91687
|
||||
end
|
||||
case evaluated 20000
|
||||
expr timestamp(abs(a offset 20000ms))
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=x ; 73585:406cd80000000000 87303:c08b230000000000 96118:7ff0000000000002 106474:4056c00000000000
|
||||
result i=0,job=x ; 0:4059000000000000 1:405b800000000000
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 5
|
||||
raw __name__=a,i=0,job=x ; 84214:406b000000000000 91349:406ec80000000000 97389:4077f80000000000 110055:7ff0000000000002 116849:c08d8e0000000000 125515:c06d7c0000000000 139575:c06fbc0000000000
|
||||
raw __name__=a,i=1,job=y ; 70093:c07a7e0000000000 84311:c0869d0000000000 99499:c08bce0000000000 107741:4054680000000000 124025:c07f1a0000000000 129247:c065680000000000 139495:7ff0000000000002
|
||||
result i=0,job=x ; 1:40550db22d0e5604 2:405858e560418937 3:405858e560418937 4:405d365604189375
|
||||
result i=1,job=y ; 0:405185f3b645a1cb 1:405513e76c8b4396 2:4058dfef9db22d0e 3:405aef6c8b439581 4:405aef6c8b439581
|
||||
end
|
||||
case evaluated 20000
|
||||
expr timestamp(abs(a offset 20000ms))
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=y ; 77173:c0653c0000000000 83623:c088c70000000000 91922:c085140000000000 109537:4081d20000000000 112877:c07eba0000000000 126470:7ff0000000000002 136248:c051380000000000 147427:4084dd0000000000 160938:c079b80000000000 169031:4059380000000000
|
||||
raw __name__=a,i=1,job=x ; 82483:4073c60000000000 99095:c0872d0000000000 110238:40801f0000000000 114761:c083a40000000000 128951:408dd00000000000 143452:7ff0000000000002 148859:c04f400000000000 157955:4054500000000000 162543:7ff0000000000002
|
||||
raw __name__=a,i=2,job=y ; 80197:407a2a0000000000 89805:4072080000000000 96259:407bda0000000000 107660:7ff0000000000002 120748:c084510000000000 129419:c08e860000000000 135946:c0899b0000000000 152730:c08b0e0000000000 161728:c085580000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000 6:4064000000000000 7:4065400000000000
|
||||
result i=1,job=x ; 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000 5:4062c00000000000 6:4064000000000000 7:4065400000000000
|
||||
result i=2,job=y ; 1:405b800000000000 2:405e000000000000 5:4062c00000000000 6:4064000000000000 7:4065400000000000
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 4
|
||||
raw __name__=a,i=0,job=x ; 73495:4088fe0000000000 82484:407b9c0000000000 98399:c071c20000000000 102319:c07d2e0000000000 109123:4081310000000000 114002:40607c0000000000 123387:406ab00000000000
|
||||
result i=0,job=x ; 0:40525fae147ae148 1:40549ef9db22d0e5 2:40589989374bc6a8 3:405b47df3b645a1d
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 5
|
||||
raw __name__=a,i=0,job=x ; 76458:4015000000000000 88707:c077ea0000000000 100123:c087690000000000 117095:c04db00000000000 129728:c047100000000000
|
||||
result i=0,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 1
|
||||
raw __name__=a,i=0,job=y ; 72663:4053980000000000 77224:7ff0000000000002 84502:7ff0000000000002 88814:c0848b0000000000
|
||||
raw __name__=a,i=1,job=y ; 78158:c081500000000000 85741:c089da0000000000 94037:407c900000000000
|
||||
result i=0,job=y ; 0:4059000000000000
|
||||
result i=1,job=y ; 0:4059000000000000
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 4
|
||||
raw __name__=a,i=0,job=y ; 73784:407f5e0000000000 79895:7ff0000000000002 90545:40838d0000000000 97792:c0878c0000000000 107256:4089960000000000 112659:4066640000000000 124276:c085360000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=y ; 80763:4068cc0000000000 93710:407a140000000000 103920:40869c0000000000
|
||||
result i=0,job=y ; 0:40576d70a3d70a3d 1:4059fae147ae147b
|
||||
end
|
||||
case evaluated 20000
|
||||
expr timestamp(abs(a offset 20000ms))
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=y ; 72337:4087940000000000 89966:407f120000000000 93255:c086d80000000000 108513:7ff0000000000002
|
||||
raw __name__=a,i=1,job=y ; 75213:40897d0000000000 90123:c06d340000000000 93395:7ff0000000000002 106382:4070040000000000
|
||||
raw __name__=a,i=2,job=x ; 70983:c0867d0000000000 75257:c088a40000000000 85105:c08dee0000000000 98459:7ff0000000000002 115950:c07a300000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000
|
||||
result i=1,job=y ; 0:4059000000000000 1:405b800000000000
|
||||
result i=2,job=x ; 0:4059000000000000 1:405b800000000000
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 6
|
||||
raw __name__=a,i=0,job=x ; 76830:c084890000000000 83432:c077c40000000000 97404:c081a00000000000 110009:400c000000000000 113602:4081e70000000000 122418:40793e0000000000 133494:c087e30000000000 142135:4072a60000000000
|
||||
raw __name__=a,i=1,job=x ; 83237:c0760e0000000000 94648:7ff0000000000002 108659:4088840000000000 122841:4082c00000000000 135311:4080e80000000000 144496:407d1a0000000000
|
||||
result i=0,job=x ; 0:405859db22d0e560 1:405859db22d0e560 2:405c66872b020c4a 3:405e9ac083126e98 4:4060afced916872b 5:4061c451eb851eb8
|
||||
result i=1,job=x ; 1:405b2a2d0e560419 2:405b2a2d0e560419 3:405eb5d2f1a9fbe7 4:4060e9f3b645a1cb 5:40620fdf3b645a1d
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=x ; 73573:c083f80000000000 84358:c0797e0000000000 94517:7ff0000000000002 100117:4056f80000000000 113287:40744e0000000000 117896:4052000000000000
|
||||
raw __name__=a,i=1,job=y ; 74254:4080380000000000 84440:c08ea90000000000 96483:c086bb0000000000 101838:405f180000000000 116855:408a350000000000
|
||||
raw __name__=a,i=2,job=y ; 79882:c079de0000000000 86885:c0875e0000000000 102488:4083eb0000000000 108210:7ff0000000000002
|
||||
result i=0,job=x ; 1:405b800000000000 2:405e000000000000
|
||||
result i=1,job=y ; 0:4059000000000000 1:405b800000000000 2:405e000000000000
|
||||
result i=2,job=y ; 0:4059000000000000
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 7
|
||||
raw __name__=a,i=0,job=x ; 81134:4056880000000000 85959:4076fa0000000000 91975:4041600000000000 101146:c070620000000000 118601:4050c00000000000 124531:7ff0000000000002 133191:c08b7b0000000000 138507:c07dc00000000000 145636:405ed00000000000 152418:c062d80000000000
|
||||
raw __name__=a,i=1,job=x ; 73371:c083d70000000000 82765:4089390000000000 90378:40843c0000000000 94675:c0830b0000000000 105880:c0698c0000000000 121661:40710e0000000000 128521:c086fd0000000000 133192:4087b80000000000 139057:7ff0000000000002 147190:408d310000000000
|
||||
result i=0,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 5:4062c00000000000 6:4064000000000000
|
||||
result i=1,job=x ; 0:4059000000000000 1:405b800000000000 2:405e000000000000 3:4060400000000000 4:4061800000000000 6:4064000000000000
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 4
|
||||
raw __name__=a,i=0,job=y ; 80343:4067380000000000 95678:c0837b0000000000 98812:c0857f0000000000 115313:c060e40000000000
|
||||
result i=0,job=y ; 0:4058b3f7ced91687 1:4058b3f7ced91687 2:405cd4083126e979 3:405cd4083126e979
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 5
|
||||
raw __name__=a,i=0,job=x ; 70359:c053e00000000000 87803:7ff0000000000002 96519:c084350000000000 100451:c08bca0000000000 106415:7ff0000000000002 111246:c0756c0000000000 122772:40861b0000000000 126738:408d5e0000000000
|
||||
raw __name__=a,i=1,job=y ; 70521:40700c0000000000 82930:40661c0000000000 87118:4075a20000000000 98137:40888f0000000000 102042:c088c80000000000 115583:4069280000000000 126036:c07ac00000000000
|
||||
raw __name__=a,i=2,job=x ; 78261:408c2b0000000000 86619:4076d80000000000 90915:4084270000000000 101761:4089a90000000000 117458:c088ef0000000000 121861:4080b10000000000 137796:406a200000000000
|
||||
result i=0,job=x ; 0:405821374bc6a7f0 2:405bcfbe76c8b439 3:405faf3b645a1cac 4:405faf3b645a1cac
|
||||
result i=1,job=y ; 0:405888c49ba5e354 1:405982b020c49ba6 2:405ce54fdf3b645a 3:405f824dd2f1a9fc 4:405f824dd2f1a9fc
|
||||
result i=2,job=x ; 0:4056ba8f5c28f5c3 1:405970b439581062 2:405d5d4fdf3b645a 3:405e771a9fbe76c9 4:40613978d4fdf3b6
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,i=0,job=x ; 70692:c08d1e0000000000 78141:c07f6e0000000000 93314:4082fc0000000000 97579:c079500000000000 102613:408c750000000000 112101:c083000000000000
|
||||
result i=0,job=x ; 0:4058650e56041893 1:4059a73b645a1cac 2:405c0676c8b43958
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=x ; 70309:c06a6c0000000000 76161:4074a20000000000 87945:c06f600000000000 98986:c065fc0000000000 108988:c06c840000000000 118961:7ff0000000000002 131871:4062680000000000 148655:40846d0000000000 152936:4089a80000000000 161492:4087ab0000000000 165161:4084300000000000
|
||||
raw __name__=a,i=1,job=y ; 72192:7ff0000000000002 85093:40647c0000000000 98209:4086680000000000 108630:c03ea00000000000 122677:7ff0000000000002 128819:4073de0000000000 143534:c0811b0000000000 146909:405a580000000000 164059:4073f80000000000
|
||||
raw __name__=a,i=2,job=y ; 83490:c085880000000000 94498:40762c0000000000 97566:c07ca00000000000 111237:c06f500000000000 119775:406b480000000000 131001:4077a20000000000 141755:c085f40000000000 152222:c0717a0000000000 160874:4068e40000000000
|
||||
result i=0,job=x ; 0:4058bf1a9fbe76c9 1:405b3f3b645a1cac 4:40607bdf3b645a1d 5:406294f5c28f5c29 6:40631df3b645a1cb 7:4064a526e978d4fe
|
||||
result i=1,job=y ; 0:40588d604189374c 1:405b2851eb851eb8 2:405b2851eb851eb8 3:40601a353f7ced91 4:40601a353f7ced91 5:40625d16872b020c 6:40625d16872b020c 7:406481e353f7ced9
|
||||
result i=2,job=y ; 0:405864395810624e 1:405864395810624e 2:405df1999999999a 3:405df1999999999a 4:406060083126e979 5:4061b828f5c28f5c 6:4063071a9fbe76c9 7:40641bf7ced91687
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 8
|
||||
raw __name__=a,i=0,job=y ; 82336:4086df0000000000 97193:4086b60000000000 107022:c061540000000000 110754:4075420000000000 127175:4085550000000000 133841:408ace0000000000 144856:4080100000000000 151854:c084cb0000000000 155543:c084240000000000
|
||||
raw __name__=a,i=1,job=y ; 82716:407e2a0000000000 97270:c082920000000000 105884:c043200000000000 111713:4059180000000000 118035:4071f20000000000 128994:c0838f0000000000 135164:408e9c0000000000 148259:408da70000000000 157945:c08d340000000000 166446:4072ce0000000000
|
||||
raw __name__=a,i=2,job=x ; 74467:40751a0000000000 78531:c03ca00000000000 82485:406d140000000000 96626:c07c280000000000 108500:40811f0000000000 124811:407e3a0000000000 140915:c06ac80000000000 148542:c068900000000000 166451:407e500000000000
|
||||
result i=0,job=y ; 0:40584c5a1cac0831 1:405ac16872b020c5 2:405bb04189374bc7 3:405fcb3333333333 4:4060bae978d4fdf4 5:40621b645a1cac08 6:406371604189374c 7:406371604189374c
|
||||
result i=1,job=y ; 0:40585147ae147ae1 1:405a789374bc6a7f 2:405d823d70a3d70a 3:40601fced916872b 4:4060e53f7ced9168 5:40628849ba5e353f 6:4063be3d70a3d70a 7:4064ce45a1cac083
|
||||
result i=2,job=x ; 0:40582810624dd2f2 1:405b200000000000 2:405b200000000000 3:405f33e76c8b4396 4:405f33e76c8b4396 5:4062915810624dd3 6:4062915810624dd3 7:4064ce6e978d4fdf
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=x ; 81983:408af30000000000 86135:c077800000000000 89808:4085920000000000 98138:4076e00000000000
|
||||
raw __name__=a,i=1,job=y ; 75638:c077680000000000 84618:c08d680000000000 101016:4082580000000000
|
||||
result i=0,job=x ; 0:405673b645a1cac1 1:405888d4fdf3b646
|
||||
result i=1,job=y ; 0:4055278d4fdf3b64 1:4059410624dd2f1b
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 7
|
||||
raw __name__=a,i=0,job=x ; 80262:4084870000000000 97781:c085160000000000 104530:c08e230000000000 112820:4088160000000000 119829:c0709c0000000000 136289:c08c990000000000 148554:4062400000000000
|
||||
raw __name__=a,i=1,job=y ; 84412:40856a0000000000 91735:7ff0000000000002 107846:406e0c0000000000 113332:40822a0000000000 126462:407b3c0000000000 140435:408e9f0000000000 147416:c086b40000000000 150997:4085260000000000 158188:c079da0000000000
|
||||
raw __name__=a,i=2,job=x ; 70377:405a280000000000 88006:c030a00000000000 96180:c081360000000000 110797:4015000000000000 118425:c087eb0000000000 122156:c08eae0000000000 139863:408df90000000000 151463:7ff0000000000002 157803:c085760000000000
|
||||
result i=0,job=x ; 0:405871fbe76c8b44 1:405a21eb851eb852 2:405df50e56041893 3:405df50e56041893 4:4061093f7ced9168 5:406291ba5e353f7d 6:406291ba5e353f7d
|
||||
result i=1,job=y ; 1:405af624dd2f1aa0 2:405c553f7ced9168 3:405f9d916872b021 4:405f9d916872b021 5:40626d4fdf3b645a 6:4063c604189374bc
|
||||
result i=2,job=x ; 0:40580b851eb851ec 1:40580b851eb851ec 2:405d9b3333333333 3:405e89fbe76c8b44 4:40617b9db22d0e56 5:40617b9db22d0e56 6:4063b9b22d0e5604
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 4
|
||||
raw __name__=a,i=0,job=x ; 72060:c085290000000000 86966:c080740000000000 94720:c088140000000000 111684:c0886a0000000000 116860:c078300000000000
|
||||
result i=0,job=x ; 0:4055bdd2f1a9fbe7 1:4057ae147ae147ae 2:405bebc6a7ef9db2 3:405d370a3d70a3d7
|
||||
end
|
||||
case selector 20000
|
||||
expr timestamp(a offset 20000ms)
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=x ; 84133:407d4e0000000000 94059:4081990000000000 103268:4085f40000000000
|
||||
result i=0,job=x ; 1:40550883126e978d
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 5
|
||||
raw __name__=a,i=0,job=x ; 72030:c082640000000000 84232:c073240000000000 89684:c014000000000000 98005:c0784e0000000000 104332:40789a0000000000 115856:c070e80000000000 121883:407a020000000000 131354:7ff0000000000002
|
||||
result i=0,job=x ; 0:40566bc6a7ef9db2 1:40588051eb851eb8 2:405a153f7ced9168 3:405e7883126e978d
|
||||
end
|
||||
case evaluated 0
|
||||
expr timestamp(abs(a))
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=y ; 78383:4074ce0000000000 86252:40868f0000000000 92548:4080340000000000 105336:4080e40000000000
|
||||
result i=0,job=y ; 0:4059000000000000 1:405b800000000000
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 2
|
||||
raw __name__=a,i=0,job=y ; 83007:c081980000000000 99231:c08a9d0000000000 106872:c07cf60000000000
|
||||
result i=0,job=y ; 0:4058cec8b4395810 1:405ab7ced916872b
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,job=x ; 75000:3ff0000000000000 110000:4000000000000000
|
||||
result job=x ; 1:405b800000000000 2:405b800000000000
|
||||
end
|
||||
case selector 0
|
||||
expr timestamp(a)
|
||||
grid 100000 10000 4
|
||||
raw __name__=a,job=x ; 99000:3ff0000000000000 105000:7ff0000000000002 123000:4008000000000000
|
||||
result job=x ; 0:4058c00000000000 3:405ec00000000000
|
||||
end
|
||||
case selector 7000
|
||||
expr timestamp(a offset 7000ms)
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,job=x ; 93500:3ff0000000000000 103500:4000000000000000
|
||||
result job=x ; 1:4057600000000000 2:4059e00000000000
|
||||
end
|
||||
case evaluated 7000
|
||||
expr timestamp(abs(a offset 7000ms))
|
||||
grid 100000 10000 3
|
||||
raw __name__=a,job=x ; 93500:3ff0000000000000 103500:4000000000000000
|
||||
result job=x ; 1:405b800000000000 2:405e000000000000
|
||||
end
|
||||
@@ -17,6 +17,7 @@ pub mod error;
|
||||
pub mod evaluated;
|
||||
pub mod extension_plan;
|
||||
pub mod functions;
|
||||
mod go_format;
|
||||
mod go_sort;
|
||||
#[cfg(test)]
|
||||
mod golden;
|
||||
|
||||
+57
-18
@@ -25,8 +25,8 @@ mod tests;
|
||||
|
||||
pub(crate) use broadcast::validate_scalar;
|
||||
pub use broadcast::{
|
||||
ScalarFunction, ScalarSeriesOperand, map_arithmetic_series, map_compare_series,
|
||||
map_scalar_function,
|
||||
ParameterizedMath, ScalarFunction, ScalarSeriesOperand, map_arithmetic_series,
|
||||
map_compare_series, map_math_with_parameters, map_scalar_function,
|
||||
};
|
||||
pub use comparison::{Comparison, ComparisonOp, map_compare, scalar_arithmetic, scalar_compare};
|
||||
|
||||
@@ -42,6 +42,7 @@ use datatypes::series::samples::{
|
||||
use datatypes::series::time::{GridRange, TimeAxis};
|
||||
|
||||
use crate::functions::Round;
|
||||
use crate::grid_series::GridSeriesBuilder;
|
||||
|
||||
/// Float math supported by the evaluated-series kernel.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
@@ -117,9 +118,12 @@ pub enum ScalarOperand {
|
||||
///
|
||||
/// Eval must resolve raw staleness before calling this kernel. Evaluated samples use validity,
|
||||
/// not NaN payloads, for missingness: arithmetic can produce the reserved stale bit pattern too.
|
||||
/// Raw time axes and histogram/mixed segments are rejected. Unsupported query types must be
|
||||
/// admitted to the ordinary path before execution; this function neither falls back nor returns
|
||||
/// partial results.
|
||||
/// Raw time axes are rejected.
|
||||
///
|
||||
/// Native histogram samples are skipped, as Prometheus' math functions process only float
|
||||
/// samples: a point holding a histogram has no result in that field, and a series with no float
|
||||
/// sample left has none at all. Float-only fields keep their chunks; a field with histograms is
|
||||
/// rebuilt point by point.
|
||||
///
|
||||
/// Returns `None` for `clamp` with `max < min`, which yields no series in Prometheus.
|
||||
///
|
||||
@@ -128,9 +132,31 @@ pub enum ScalarOperand {
|
||||
/// `mergeSeriesWithSameLabelset`).
|
||||
pub fn map_math(schema: &SeriesSchema, series: &Series, op: MathOp) -> Result<Option<Series>> {
|
||||
if op.is_empty_clamp() {
|
||||
validate_float_series(schema, series)?;
|
||||
validate_grid_series(schema, series)?;
|
||||
return Ok(None);
|
||||
}
|
||||
if !is_float_series(series) {
|
||||
let time = validate_grid_series(schema, series)?;
|
||||
let mut builder = GridSeriesBuilder::try_new(Arc::new(schema.clone()))?;
|
||||
let mut cursors: Vec<_> = series.fields.iter().map(|field| field.cursor()).collect();
|
||||
let mut row = Vec::with_capacity(cursors.len());
|
||||
for (point, index) in time.positions() {
|
||||
row.clear();
|
||||
for cursor in &mut cursors {
|
||||
row.push(
|
||||
match cursor
|
||||
.at(index)
|
||||
.expect("the axis and the fields are aligned")
|
||||
{
|
||||
SampleRef::Float(value) => Some(op.calculate(value)),
|
||||
SampleRef::Missing | SampleRef::Histogram(_) => None,
|
||||
},
|
||||
);
|
||||
}
|
||||
builder.push(point, &row)?;
|
||||
}
|
||||
return builder.finish(drop_metric_name(&series.metric)?);
|
||||
}
|
||||
let mut output = map_float_fields(schema, series, |floats| op.apply(floats))?;
|
||||
output.metric = drop_metric_name(&output.metric)?;
|
||||
Ok(Some(output))
|
||||
@@ -372,6 +398,31 @@ fn series_error(error: datatypes::error::Error) -> DataFusionError {
|
||||
pub(crate) fn validate_float_series<'a>(
|
||||
schema: &SeriesSchema,
|
||||
series: &'a Series,
|
||||
) -> Result<&'a TimeAxis> {
|
||||
let time = validate_grid_series(schema, series)?;
|
||||
if !is_float_series(series) {
|
||||
return Err(DataFusionError::NotImplemented(
|
||||
"Series operators support only float samples; histogram/mixed samples are not supported"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
Ok(time)
|
||||
}
|
||||
|
||||
/// Whether every field of `series` holds float samples only.
|
||||
fn is_float_series(series: &Series) -> bool {
|
||||
series.fields.iter().all(|field| {
|
||||
field
|
||||
.segments()
|
||||
.iter()
|
||||
.all(|segment| matches!(segment.dense(), Some(DenseSamplesRef::Float(_))))
|
||||
})
|
||||
}
|
||||
|
||||
/// Checks `series` against `schema` and its evaluation grid, whatever its sample types.
|
||||
pub(crate) fn validate_grid_series<'a>(
|
||||
schema: &SeriesSchema,
|
||||
series: &'a Series,
|
||||
) -> Result<&'a TimeAxis> {
|
||||
let time = &series.time;
|
||||
if time.grid().is_none() {
|
||||
@@ -386,18 +437,6 @@ pub(crate) fn validate_float_series<'a>(
|
||||
series.fields.clone(),
|
||||
)
|
||||
.map_err(series_error)?;
|
||||
for field in series.fields.iter() {
|
||||
if field
|
||||
.segments()
|
||||
.iter()
|
||||
.any(|segment| !matches!(segment.dense(), Some(DenseSamplesRef::Float(_))))
|
||||
{
|
||||
return Err(DataFusionError::NotImplemented(
|
||||
"Series operators support only float samples; histogram/mixed samples are not supported"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(time)
|
||||
}
|
||||
|
||||
|
||||
@@ -16,11 +16,14 @@ use std::sync::Arc;
|
||||
|
||||
use datafusion::error::{DataFusionError, Result};
|
||||
use datatypes::series::MetricName;
|
||||
use datatypes::series::samples::{FieldSpec, Series, SeriesSchema};
|
||||
use datatypes::series::samples::{FieldSpec, SampleRef, Series, SeriesSchema};
|
||||
use datatypes::series::time::TimeAxis;
|
||||
|
||||
use crate::grid_series::GridSeriesBuilder;
|
||||
use crate::map::{ArithmeticOp, Comparison, drop_metric_name, float_sample, validate_float_series};
|
||||
use crate::map::{
|
||||
ArithmeticOp, Comparison, MathOp, drop_metric_name, float_sample, validate_float_series,
|
||||
validate_grid_series,
|
||||
};
|
||||
|
||||
/// A time-varying, unlabelled single-field scalar, with explicit operand direction.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
@@ -126,6 +129,98 @@ pub fn map_scalar_function(
|
||||
)
|
||||
}
|
||||
|
||||
/// A math function whose parameters are scalar expressions, which Prometheus evaluates at every
|
||||
/// step, with those parameters as time-varying scalars.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub enum ParameterizedMath<'a> {
|
||||
/// `round(v, to_nearest)`.
|
||||
Round { to_nearest: &'a Series },
|
||||
/// `clamp(v, min, max)`.
|
||||
Clamp { min: &'a Series, max: &'a Series },
|
||||
/// `clamp_min(v, min)`.
|
||||
ClampMin { min: &'a Series },
|
||||
/// `clamp_max(v, max)`.
|
||||
ClampMax { max: &'a Series },
|
||||
}
|
||||
|
||||
impl<'a> ParameterizedMath<'a> {
|
||||
/// The parameters in argument order.
|
||||
fn parameters(self) -> Vec<&'a Series> {
|
||||
match self {
|
||||
Self::Round { to_nearest } => vec![to_nearest],
|
||||
Self::Clamp { min, max } => vec![min, max],
|
||||
Self::ClampMin { min } => vec![min],
|
||||
Self::ClampMax { max } => vec![max],
|
||||
}
|
||||
}
|
||||
|
||||
/// The function with the parameter values of one point, in argument order.
|
||||
fn at(self, values: &[f64]) -> MathOp {
|
||||
match self {
|
||||
Self::Round { .. } => MathOp::Round { nearest: values[0] },
|
||||
Self::Clamp { .. } => MathOp::Clamp {
|
||||
min: values[0],
|
||||
max: values[1],
|
||||
},
|
||||
Self::ClampMin { .. } => MathOp::ClampMin { min: values[0] },
|
||||
Self::ClampMax { .. } => MathOp::ClampMax { max: values[0] },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Applies `function` to every field with its parameters read at each point from time-varying
|
||||
/// scalars, and drops the metric name, as [`crate::map::map_math`] does with constant parameters;
|
||||
/// native histogram samples are skipped the same way.
|
||||
///
|
||||
/// A point where a parameter has no value has no result. `clamp` with `max < min` at a point has
|
||||
/// no result there, as Prometheus returns an empty vector for that step. `None` means no point
|
||||
/// holds a result. Same-labelset merging is left to the caller.
|
||||
pub fn map_math_with_parameters(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
series: &Series,
|
||||
function: ParameterizedMath<'_>,
|
||||
) -> Result<Option<Series>> {
|
||||
let time = validate_grid_series(schema, series)?;
|
||||
let parameters = function.parameters();
|
||||
let mut parameter_axes = Vec::with_capacity(parameters.len());
|
||||
for parameter in ¶meters {
|
||||
parameter_axes.push(validate_scalar(schema, parameter)?);
|
||||
}
|
||||
// Points only increase, so each parameter's row does too and one cursor reads it.
|
||||
let mut parameter_cursors: Vec<_> = parameters
|
||||
.iter()
|
||||
.map(|parameter| parameter.fields.as_slice()[0].cursor())
|
||||
.collect();
|
||||
let mut field_cursors: Vec<_> = series.fields.iter().map(|field| field.cursor()).collect();
|
||||
let mut values = Vec::with_capacity(parameters.len());
|
||||
let mut row = Vec::with_capacity(field_cursors.len());
|
||||
let mut builder = GridSeriesBuilder::try_new(schema.clone())?;
|
||||
for (point, index) in time.positions() {
|
||||
values.clear();
|
||||
for (axis, cursor) in parameter_axes.iter().zip(&mut parameter_cursors) {
|
||||
let value = axis.row_of(point).and_then(|row| {
|
||||
float_sample(cursor.at(row).expect("parameter rows only increase"))
|
||||
});
|
||||
values.extend(value);
|
||||
}
|
||||
let op = (values.len() == parameters.len())
|
||||
.then(|| function.at(&values))
|
||||
.filter(|op| !op.is_empty_clamp());
|
||||
row.clear();
|
||||
for cursor in &mut field_cursors {
|
||||
let sample = cursor
|
||||
.at(index)
|
||||
.expect("the axis and the fields are aligned");
|
||||
row.push(match (op, sample) {
|
||||
(Some(op), SampleRef::Float(value)) => Some(op.calculate(value)),
|
||||
_ => None,
|
||||
});
|
||||
}
|
||||
builder.push(point, &row)?;
|
||||
}
|
||||
builder.finish(drop_metric_name(&series.metric)?)
|
||||
}
|
||||
|
||||
/// Calls `calculate(left, right, vector_sample)` at each index where both operands exist.
|
||||
fn map_grid_scalar(
|
||||
schema: &Arc<SeriesSchema>,
|
||||
|
||||
@@ -119,7 +119,13 @@ impl Comparison {
|
||||
}
|
||||
|
||||
impl ComparisonOp {
|
||||
fn apply(self, left: f64, right: f64) -> bool {
|
||||
/// Whether `left op right` holds, by the same IEEE rules the kernels use.
|
||||
///
|
||||
/// Public because a compiler that folds a constant comparison has to fold it with the same
|
||||
/// answer the kernel would compute: NaN satisfies only `NotEqual`, and signed zeros compare
|
||||
/// equal. Defining the six rules a second time is how the folded number and the evaluated one
|
||||
/// would start to disagree.
|
||||
pub fn apply(self, left: f64, right: f64) -> bool {
|
||||
match self {
|
||||
Self::Equal => left == right,
|
||||
Self::NotEqual => left != right,
|
||||
|
||||
@@ -0,0 +1,365 @@
|
||||
# Generated by promql-compat cmd/narthex-golden-mathparam built for amd64 (Prometheus v3.15.0
|
||||
# engine, seed 1). `case <function>`, the input series, `param <name>` lines in argument order,
|
||||
# then the result series. Points are `grid index:hex value bits`.
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 6
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77 1:c004000000000000 4:7ff8000000000001 5:3ff4000000000000
|
||||
param __name__=p0 ; 0:fff0000000000000 1:7ff0000000000000 2:4008000000000000 3:3fe0000000000000 4:c000000000000000 5:7ff8000000000001
|
||||
result i=0 ; 0:fff8000000000000 1:fff8000000000000 4:7ff8000000000001 5:7ff8000000000001
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:401c000000000000 1:0 2:3ff4000000000000
|
||||
series __name__=a,i=1 ; 2:7ff8000000000001
|
||||
series __name__=a,i=2 ; 0:c02b800000000000 1:c02b800000000000 2:c02b800000000000
|
||||
param __name__=p0 ; 0:7ff8000000000001 1:3ff0000000000000 2:3ff0000000000000
|
||||
param __name__=p1 ; 0:4000000000000000 1:0 2:4000000000000000
|
||||
result i=0 ; 0:7ff8000000000001 2:3ff4000000000000
|
||||
result i=1 ; 2:7ff8000000000001
|
||||
result i=2 ; 0:7ff8000000000001 2:3ff0000000000000
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:c02b800000000000 1:7ff0000000000000 2:0
|
||||
series __name__=a,i=1 ; 0:3ff4000000000000 2:c004000000000000 4:3e112e0be826d695
|
||||
series __name__=a,i=2 ; 0:fff0000000000000 1:fff0000000000000 2:7ff8000000000001 3:fff0000000000000 4:c004000000000000
|
||||
param __name__=p0 ; 0:fff0000000000000 1:4024000000000000 2:4024000000000000 3:fff0000000000000 4:0
|
||||
result i=0 ; 0:c02b800000000000 1:7ff0000000000000 2:4024000000000000
|
||||
result i=1 ; 0:3ff4000000000000 2:4024000000000000 4:3e112e0be826d695
|
||||
result i=2 ; 0:fff0000000000000 1:4024000000000000 2:7ff8000000000001 3:fff0000000000000 4:0
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:3ff4000000000000 1:405edd2f1a9fbe77
|
||||
series __name__=a,i=1 ; 0:fff0000000000000 1:c004000000000000
|
||||
param __name__=p0 ; 0:4024000000000000 1:fff0000000000000
|
||||
result i=0 ; 0:3ff4000000000000 1:fff0000000000000
|
||||
result i=1 ; 0:fff0000000000000 1:fff0000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 1
|
||||
series __name__=a,i=0 ;
|
||||
series __name__=a,i=1 ;
|
||||
param __name__=p0 ; 0:c000000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 6
|
||||
series __name__=a,i=0 ; 4:c004000000000000 5:3ff4000000000000
|
||||
series __name__=a,i=1 ; 0:7ff0000000000000 1:7ff0000000000000 2:c004000000000000 3:3e112e0be826d695 4:0 5:c004000000000000
|
||||
series __name__=a,i=2 ; 0:3ff4000000000000 1:7ff8000000000001 5:7ff0000000000000
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:c014000000000000 2:fff0000000000000 3:c014000000000000 4:4000000000000000 5:4024000000000000
|
||||
param __name__=p1 ; 0:4024000000000000 1:c014000000000000 2:fff0000000000000 3:4024000000000000 4:3ff0000000000000 5:4000000000000000
|
||||
result i=1 ; 0:4024000000000000 1:c014000000000000 2:fff0000000000000 3:3e112e0be826d695
|
||||
result i=2 ; 0:3ff4000000000000 1:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:8000000000000000 1:4004000000000000 2:c02b800000000000
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:3ff0000000000000 2:fff0000000000000
|
||||
result i=0 ; 0:3ff0000000000000 1:4004000000000000 2:c02b800000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 0:c004000000000000 1:c004000000000000 2:0 3:c004000000000000
|
||||
param __name__=p0 ; 0:fff0000000000000 1:c014000000000000 2:c014000000000000 3:3ff0000000000000
|
||||
result i=0 ; 0:fff0000000000000 1:c014000000000000 2:c014000000000000 3:c004000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:fff0000000000000
|
||||
result i=0 ; 0:405ec00000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 1:3ff4000000000000
|
||||
series __name__=a,i=1 ; 0:4004000000000000 2:8000000000000000
|
||||
series __name__=a,i=2 ; 0:401c000000000000 1:c02b800000000000 2:405edd2f1a9fbe77
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:c014000000000000 2:fff0000000000000
|
||||
param __name__=p1 ; 0:3ff0000000000000 1:fff0000000000000 2:4024000000000000
|
||||
result i=1 ; 2:8000000000000000
|
||||
result i=2 ; 2:4024000000000000
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:7ff8000000000001 1:7ff8000000000001 2:405edd2f1a9fbe77 3:7ff8000000000001
|
||||
series __name__=a,i=1 ; 0:3ff4000000000000 1:c02b800000000000 3:3e112e0be826d695 4:8000000000000000
|
||||
series __name__=a,i=2 ; 0:c02b800000000000 1:8000000000000000 3:c004000000000000 4:7ff8000000000001
|
||||
param __name__=p0 ; 0:7ff8000000000001 1:7ff8000000000001 2:4024000000000000 3:c014000000000000 4:fff0000000000000
|
||||
result i=0 ; 0:7ff8000000000001 1:7ff8000000000001 2:405edd2f1a9fbe77 3:7ff8000000000001
|
||||
result i=1 ; 0:7ff8000000000001 1:7ff8000000000001 3:3e112e0be826d695 4:8000000000000000
|
||||
result i=2 ; 0:7ff8000000000001 1:7ff8000000000001 3:c004000000000000 4:7ff8000000000001
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:8000000000000000
|
||||
series __name__=a,i=1 ; 1:fff0000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:4024000000000000
|
||||
result i=0 ; 0:8000000000000000
|
||||
result i=1 ; 1:fff0000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 1
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77
|
||||
param __name__=p0 ; 0:3fe0000000000000
|
||||
result i=0 ; 0:405ee00000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 1
|
||||
series __name__=a,i=0 ; 0:fff0000000000000
|
||||
series __name__=a,i=1 ; 0:8000000000000000
|
||||
param __name__=p0 ; 0:4000000000000000
|
||||
param __name__=p1 ; 0:7ff0000000000000
|
||||
result i=0 ; 0:4000000000000000
|
||||
result i=1 ; 0:4000000000000000
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 0:c004000000000000 1:7ff8000000000001 3:3ff4000000000000
|
||||
series __name__=a,i=1 ; 0:4004000000000000 1:3ff4000000000000 2:7ff0000000000000 3:4004000000000000
|
||||
param __name__=p0 ; 0:fff0000000000000 1:fff0000000000000 2:7ff8000000000001 3:7ff0000000000000
|
||||
result i=0 ; 0:c004000000000000 1:7ff8000000000001 3:7ff0000000000000
|
||||
result i=1 ; 0:4004000000000000 1:3ff4000000000000 2:7ff0000000000000 3:7ff0000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 0:7ff8000000000001 1:4004000000000000 2:8000000000000000
|
||||
series __name__=a,i=1 ; 0:7ff0000000000000 1:0 2:c004000000000000 3:c004000000000000
|
||||
param __name__=p0 ; 0:4024000000000000 1:4000000000000000 2:0 3:7ff0000000000000
|
||||
result i=0 ; 0:7ff8000000000001 1:4000000000000000 2:8000000000000000
|
||||
result i=1 ; 0:4024000000000000 1:0 2:c004000000000000 3:c004000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 0:3ff4000000000000 1:7ff8000000000001 2:7ff0000000000000 3:7ff8000000000001
|
||||
series __name__=a,i=1 ; 1:7ff0000000000000 2:3ff4000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:fff0000000000000 2:3fe0000000000000 3:3fb999999999999a
|
||||
result i=0 ; 0:fff8000000000000 1:7ff8000000000001 2:7ff0000000000000 3:7ff8000000000001
|
||||
result i=1 ; 1:fff8000000000000 2:3ff8000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:7ff0000000000000
|
||||
param __name__=p0 ; 0:7ff8000000000001 1:4024000000000000 2:7ff8000000000001
|
||||
param __name__=p1 ; 0:3ff0000000000000 1:4024000000000000 2:fff0000000000000
|
||||
result i=0 ; 0:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77 1:405edd2f1a9fbe77 2:401c000000000000 3:0
|
||||
param __name__=p0 ; 0:4024000000000000 1:3ff0000000000000 2:fff0000000000000 3:3ff0000000000000 4:0
|
||||
result i=0 ; 0:405edd2f1a9fbe77 1:405edd2f1a9fbe77 2:401c000000000000 3:3ff0000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77 1:401c000000000000 2:405edd2f1a9fbe77
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:4024000000000000 2:4000000000000000
|
||||
result i=0 ; 0:3ff0000000000000 1:401c000000000000 2:4000000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 1
|
||||
series __name__=a,i=0 ; 0:3e112e0be826d695
|
||||
series __name__=a,i=1 ;
|
||||
param __name__=p0 ; 0:7ff8000000000001
|
||||
result i=0 ; 0:7ff8000000000001
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:c02b800000000000 1:8000000000000000
|
||||
param __name__=p0 ; 0:7ff8000000000001 1:7ff0000000000000 2:7ff0000000000000
|
||||
param __name__=p1 ; 0:fff0000000000000 1:c014000000000000 2:4024000000000000
|
||||
result i=0 ; 0:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:8000000000000000 1:4004000000000000 2:405edd2f1a9fbe77 4:fff0000000000000
|
||||
param __name__=p0 ; 0:4000000000000000 1:7ff8000000000001 2:4000000000000000 3:4000000000000000 4:0
|
||||
result i=0 ; 0:4000000000000000 1:7ff8000000000001 2:405edd2f1a9fbe77 4:0
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 1:8000000000000000 3:c004000000000000
|
||||
series __name__=a,i=1 ; 0:3ff4000000000000 1:401c000000000000 3:c004000000000000
|
||||
param __name__=p0 ; 0:4000000000000000 1:7ff0000000000000 2:3ff0000000000000 3:0
|
||||
result i=0 ; 1:8000000000000000 3:c004000000000000
|
||||
result i=1 ; 0:3ff4000000000000 1:401c000000000000 3:c004000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 2:0
|
||||
series __name__=a,i=1 ; 0:7ff0000000000000 2:401c000000000000
|
||||
param __name__=p0 ; 0:4008000000000000 1:fff0000000000000 2:4008000000000000
|
||||
result i=0 ; 2:0
|
||||
result i=1 ; 0:7ff0000000000000 2:4018000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 6
|
||||
series __name__=a,i=0 ; 0:c004000000000000 1:3ff4000000000000 2:c02b800000000000 3:401c000000000000 4:c02b800000000000
|
||||
param __name__=p0 ; 0:4024000000000000 1:3ff0000000000000 2:4000000000000000 3:7ff0000000000000 4:c014000000000000 5:0
|
||||
param __name__=p1 ; 0:7ff0000000000000 1:fff0000000000000 2:7ff8000000000001 3:7ff0000000000000 4:0 5:4024000000000000
|
||||
result i=0 ; 0:4024000000000000 2:7ff8000000000001 3:7ff0000000000000 4:c014000000000000
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:7ff8000000000001 1:7ff0000000000000
|
||||
series __name__=a,i=1 ; 0:7ff8000000000001 1:3e112e0be826d695
|
||||
param __name__=p0 ; 0:4024000000000000 1:4000000000000000
|
||||
result i=0 ; 0:7ff8000000000001 1:7ff0000000000000
|
||||
result i=1 ; 0:7ff8000000000001 1:4000000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:4004000000000000 2:fff0000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:7ff8000000000001 2:7ff0000000000000
|
||||
result i=0 ; 0:4004000000000000 2:fff0000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 1:8000000000000000 3:405edd2f1a9fbe77
|
||||
series __name__=a,i=1 ; 0:4004000000000000 1:fff0000000000000 2:3ff4000000000000 3:3ff4000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:3fb999999999999a 2:fff0000000000000 3:c000000000000000
|
||||
result i=0 ; 1:0 3:405f000000000000
|
||||
result i=1 ; 0:fff8000000000000 1:fff0000000000000 2:fff8000000000000 3:4000000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:3e112e0be826d695 1:7ff8000000000001
|
||||
series __name__=a,i=1 ; 0:c02b800000000000 1:c004000000000000
|
||||
param __name__=p0 ; 0:c014000000000000 1:4024000000000000
|
||||
param __name__=p1 ; 0:3ff0000000000000 1:7ff8000000000001
|
||||
result i=0 ; 0:3e112e0be826d695 1:7ff8000000000001
|
||||
result i=1 ; 0:c014000000000000 1:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 2
|
||||
series __name__=a,i=0 ; 0:405edd2f1a9fbe77
|
||||
series __name__=a,i=1 ; 0:401c000000000000 1:c02b800000000000
|
||||
series __name__=a,i=2 ; 0:c02b800000000000 1:7ff0000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:4000000000000000
|
||||
result i=0 ; 0:7ff0000000000000
|
||||
result i=1 ; 0:7ff0000000000000 1:4000000000000000
|
||||
result i=2 ; 0:7ff0000000000000 1:7ff0000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 6
|
||||
series __name__=a,i=0 ; 2:c004000000000000 3:fff0000000000000 4:0 5:3ff4000000000000
|
||||
param __name__=p0 ; 0:4024000000000000 1:4000000000000000 2:4000000000000000 3:0 4:4024000000000000 5:7ff0000000000000
|
||||
result i=0 ; 2:c004000000000000 3:fff0000000000000 4:0 5:3ff4000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:fff0000000000000 1:3e112e0be826d695 2:401c000000000000
|
||||
series __name__=a,i=1 ; 0:8000000000000000 2:fff0000000000000
|
||||
param __name__=p0 ; 0:3fe0000000000000 1:3fe0000000000000 2:4024000000000000
|
||||
result i=0 ; 0:fff0000000000000 1:0 2:4024000000000000
|
||||
result i=1 ; 0:0 2:fff0000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 1:3e112e0be826d695 2:fff0000000000000 3:c02b800000000000
|
||||
series __name__=a,i=1 ; 0:fff0000000000000 1:405edd2f1a9fbe77 3:8000000000000000
|
||||
param __name__=p0 ; 0:0 1:3ff0000000000000 2:4000000000000000 3:7ff8000000000001
|
||||
param __name__=p1 ; 0:4000000000000000 1:0 2:4024000000000000 3:4024000000000000
|
||||
result i=0 ; 2:4000000000000000 3:7ff8000000000001
|
||||
result i=1 ; 0:0 3:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:0 2:fff0000000000000
|
||||
series __name__=a,i=1 ; 0:c004000000000000 1:0
|
||||
series __name__=a,i=2 ; 1:fff0000000000000 2:3e112e0be826d695
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:7ff8000000000001 2:3ff0000000000000
|
||||
result i=0 ; 0:3ff0000000000000 2:3ff0000000000000
|
||||
result i=1 ; 0:3ff0000000000000 1:7ff8000000000001
|
||||
result i=2 ; 1:7ff8000000000001 2:3ff0000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:7ff8000000000001 1:3e112e0be826d695 2:c004000000000000 3:405edd2f1a9fbe77 4:3e112e0be826d695
|
||||
series __name__=a,i=1 ; 0:7ff8000000000001 2:fff0000000000000
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:c014000000000000 2:3ff0000000000000 3:c014000000000000 4:fff0000000000000
|
||||
result i=0 ; 0:7ff8000000000001 1:c014000000000000 2:c004000000000000 3:c014000000000000 4:fff0000000000000
|
||||
result i=1 ; 0:7ff8000000000001 2:fff0000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 5
|
||||
series __name__=a,i=0 ; 0:4004000000000000 1:7ff0000000000000
|
||||
series __name__=a,i=1 ; 0:c004000000000000 1:405edd2f1a9fbe77 3:8000000000000000
|
||||
series __name__=a,i=2 ; 0:0 1:3e112e0be826d695 2:3e112e0be826d695 3:405edd2f1a9fbe77
|
||||
param __name__=p0 ; 0:7ff0000000000000 1:4008000000000000 2:4024000000000000 3:0 4:c000000000000000
|
||||
result i=0 ; 0:fff8000000000000 1:7ff0000000000000
|
||||
result i=1 ; 0:fff8000000000000 1:405ec00000000000 3:fff8000000000000
|
||||
result i=2 ; 0:fff8000000000000 1:0 2:0 3:fff8000000000000
|
||||
end
|
||||
case clamp
|
||||
expr clamp(a, scalar(p0), scalar(p1))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:0 1:4004000000000000 2:405edd2f1a9fbe77
|
||||
series __name__=a,i=1 ; 0:405edd2f1a9fbe77 1:405edd2f1a9fbe77 2:c02b800000000000
|
||||
param __name__=p0 ; 0:7ff8000000000001 1:0 2:7ff8000000000001
|
||||
param __name__=p1 ; 0:fff0000000000000 1:c014000000000000 2:fff0000000000000
|
||||
result i=0 ; 0:7ff8000000000001 2:7ff8000000000001
|
||||
result i=1 ; 0:7ff8000000000001 2:7ff8000000000001
|
||||
end
|
||||
case clamp_min
|
||||
expr clamp_min(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=0 ; 0:401c000000000000 1:8000000000000000 2:401c000000000000 3:fff0000000000000
|
||||
series __name__=a,i=1 ; 0:8000000000000000 1:401c000000000000 2:0 3:8000000000000000
|
||||
series __name__=a,i=2 ; 0:fff0000000000000 1:401c000000000000 3:3ff4000000000000
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:fff0000000000000 2:4024000000000000 3:3ff0000000000000
|
||||
result i=0 ; 0:401c000000000000 1:8000000000000000 2:4024000000000000 3:3ff0000000000000
|
||||
result i=1 ; 0:3ff0000000000000 1:401c000000000000 2:4024000000000000 3:3ff0000000000000
|
||||
result i=2 ; 0:3ff0000000000000 1:401c000000000000 3:3ff4000000000000
|
||||
end
|
||||
case clamp_max
|
||||
expr clamp_max(a, scalar(p0))
|
||||
grid 100000 10000 3
|
||||
series __name__=a,i=0 ; 0:3e112e0be826d695 1:3e112e0be826d695
|
||||
param __name__=p0 ; 0:4000000000000000 1:fff0000000000000 2:4000000000000000
|
||||
result i=0 ; 0:3e112e0be826d695 1:fff0000000000000
|
||||
end
|
||||
case round
|
||||
expr round(a, scalar(p0))
|
||||
grid 100000 10000 4
|
||||
series __name__=a,i=ties ; 0:4004000000000000 1:c004000000000000 2:3fe0000000000000 3:bfe0000000000000
|
||||
param __name__=p0 ; 0:3ff0000000000000 1:3ff0000000000000 2:3ff0000000000000 3:3ff0000000000000
|
||||
result i=ties ; 0:4008000000000000 1:c000000000000000 2:3ff0000000000000 3:0
|
||||
end
|
||||
+35
-16
@@ -31,7 +31,7 @@ use datatypes::series::histogram::{
|
||||
BucketSideView, CounterResetHint, HistogramBatch, HistogramBatchBuilder, HistogramView,
|
||||
};
|
||||
use datatypes::series::samples::{FieldSpec, HistogramSamples, SampleRef};
|
||||
use datatypes::series::time::{EvalGrid, GridRange, GridSpec};
|
||||
use datatypes::series::time::{EvalGrid, GridRange, GridSpan, GridSpec};
|
||||
use datatypes::series::{Label, MetricName};
|
||||
|
||||
use super::*;
|
||||
@@ -383,31 +383,50 @@ fn histogram_batch() -> Arc<HistogramBatch> {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn histogram_and_mixed_fields_are_rejected_without_partial_success() {
|
||||
fn histogram_samples_are_skipped_and_float_samples_mapped() {
|
||||
let batch = histogram_batch();
|
||||
let histogram = Samples::try_from_histogram(UInt32Array::from(vec![0]), batch.clone()).unwrap();
|
||||
let mixed = Samples::try_from_mixed(
|
||||
Float64Array::from(vec![Some(1.0), None]),
|
||||
Float64Array::from(vec![Some(-1.0), None]),
|
||||
HistogramSamples::try_new(UInt32Array::from(vec![None, Some(0)]), batch.clone()).unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
let null_histogram = Samples::try_from_histogram(UInt32Array::from(vec![None]), batch).unwrap();
|
||||
let chunks = Samples::try_from_chunks(vec![floats(&[Some(-1.0)]), histogram.clone()]).unwrap();
|
||||
let schema = schema(&["float", "unsupported"]);
|
||||
for unsupported in [histogram, mixed, null_histogram, chunks] {
|
||||
let prefix = floats(&vec![Some(-2.0); unsupported.len()]);
|
||||
let input = series(&schema, 2, vec![prefix, unsupported]);
|
||||
for op in [MathOp::Abs, MathOp::Clamp { min: 2.0, max: 1.0 }] {
|
||||
assert!(matches!(
|
||||
map_math(&schema, &input, op),
|
||||
Err(DataFusionError::NotImplemented(_))
|
||||
));
|
||||
}
|
||||
assert_floats(
|
||||
input.fields.get(0).unwrap(),
|
||||
&vec![Some(-2.0); input.time.len()],
|
||||
let two_fields = schema(&["float", "histograms"]);
|
||||
// Prometheus' math functions process only float samples: each histogram is a point without a
|
||||
// result, and the floats beside it, in its field or the other, are mapped as usual.
|
||||
for (samples, expected) in [
|
||||
(histogram.clone(), vec![None]),
|
||||
(mixed, vec![Some(1.0), None]),
|
||||
(null_histogram, vec![None]),
|
||||
(chunks, vec![Some(1.0), None]),
|
||||
] {
|
||||
let len = samples.len();
|
||||
let input = series(
|
||||
&two_fields,
|
||||
2,
|
||||
vec![floats(&vec![Some(-2.0); len]), samples],
|
||||
);
|
||||
let output = map_math(&two_fields, &input, MathOp::Abs).unwrap().unwrap();
|
||||
assert_eq!(
|
||||
run_floats(&output, 0),
|
||||
vec![((2, len), vec![Some(2.0); len])]
|
||||
);
|
||||
assert_eq!(run_floats(&output, 1), vec![((2, len), expected)]);
|
||||
assert!(
|
||||
map_math(&two_fields, &input, MathOp::Clamp { min: 2.0, max: 1.0 })
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
let only_histograms = schema(&["h"]);
|
||||
let input = series(&only_histograms, 2, vec![histogram]);
|
||||
assert!(
|
||||
map_math(&only_histograms, &input, MathOp::Abs)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -376,8 +376,8 @@ fn a_vector_scalar_operation_of_sparse_operands_keeps_the_shared_positions() {
|
||||
.eval_grid
|
||||
.as_ref()
|
||||
.expect("the fixture schema has a grid")
|
||||
.timestamps()
|
||||
.len();
|
||||
.spec()
|
||||
.len as usize;
|
||||
let mut expected: Dense = vec![vec![None]; points_on_grid];
|
||||
expected[5] = vec![Some(52.0)];
|
||||
expected[9] = vec![Some(93.0)];
|
||||
@@ -431,3 +431,184 @@ fn a_vector_scalar_operation_of_sparse_operands_keeps_the_shared_positions() {
|
||||
"nothing passes, so there is no result series"
|
||||
);
|
||||
}
|
||||
|
||||
/// `round` and the clamp functions with time-varying scalar parameters, evaluated by the
|
||||
/// Prometheus v3.15.0 engine; see the header of `scalar_parameter_cases.txt`.
|
||||
#[test]
|
||||
fn parameterized_math_matches_prometheus() {
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use crate::grid_series::reference::{Labels, Points, labels_of, points_of};
|
||||
|
||||
let parse = |line: &str| -> (Labels, Points) {
|
||||
let (labels, points) = line.split_once(" ;").unwrap();
|
||||
let labels = labels
|
||||
.split(',')
|
||||
.filter(|pair| *pair != "-")
|
||||
.map(|pair| {
|
||||
let (name, value) = pair.split_once('=').unwrap();
|
||||
(name.to_string(), value.to_string())
|
||||
})
|
||||
.collect();
|
||||
let points = points
|
||||
.split_whitespace()
|
||||
.map(|point| {
|
||||
let (index, bits) = point.split_once(':').unwrap();
|
||||
(
|
||||
index.parse().unwrap(),
|
||||
u64::from_str_radix(bits, 16).unwrap(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
(labels, points)
|
||||
};
|
||||
let mut checked = 0;
|
||||
for case in include_str!("../scalar_parameter_cases.txt")
|
||||
.split("end\n")
|
||||
.filter(|case| case.contains("case "))
|
||||
{
|
||||
let lines: Vec<&str> = case.lines().filter(|line| !line.starts_with('#')).collect();
|
||||
let name = lines[0].strip_prefix("case ").unwrap();
|
||||
let points: u64 = lines[2].split_whitespace().nth(3).unwrap().parse().unwrap();
|
||||
let grid = Arc::new(
|
||||
EvalGrid::try_new(GridSpec::try_new(100_000, 10_000, points).unwrap()).unwrap(),
|
||||
);
|
||||
let input_schema = Arc::new(SeriesSchema {
|
||||
fields: vec![FieldSpec {
|
||||
name: "value".into(),
|
||||
}]
|
||||
.into(),
|
||||
eval_grid: Some(grid.clone()),
|
||||
});
|
||||
let sparse_series = crate::grid_series::reference::sparse_series;
|
||||
let (mut inputs, mut parameters, mut expected) = (Vec::new(), Vec::new(), BTreeMap::new());
|
||||
for line in &lines[3..] {
|
||||
let (kind, rest) = line.split_once(' ').unwrap();
|
||||
let (labels, values) = parse(rest);
|
||||
match kind {
|
||||
"series" => inputs.extend(sparse_series(&input_schema, &labels, &values)),
|
||||
// A parameter is `scalar(p)`: unlabelled, with a value at every point.
|
||||
"param" => parameters.push(
|
||||
sparse_series(&input_schema, &Labels::new(), &values)
|
||||
.expect("a value at every point"),
|
||||
),
|
||||
"result" => {
|
||||
expected.insert(labels, values);
|
||||
}
|
||||
other => panic!("unexpected line {other}"),
|
||||
}
|
||||
}
|
||||
let function = match (name, parameters.as_slice()) {
|
||||
("round", [to_nearest]) => ParameterizedMath::Round { to_nearest },
|
||||
("clamp", [min, max]) => ParameterizedMath::Clamp { min, max },
|
||||
("clamp_min", [min]) => ParameterizedMath::ClampMin { min },
|
||||
("clamp_max", [max]) => ParameterizedMath::ClampMax { max },
|
||||
other => panic!("unknown case {other:?}"),
|
||||
};
|
||||
let mut actual = BTreeMap::new();
|
||||
for input in &inputs {
|
||||
if let Some(output) = map_math_with_parameters(&input_schema, input, function).unwrap()
|
||||
{
|
||||
actual.insert(labels_of(&output.metric), points_of(&output));
|
||||
}
|
||||
}
|
||||
// Go's NaN and Rust's differ in their payload; a NaN is a NaN.
|
||||
let canonical = |series: BTreeMap<Labels, Points>| -> BTreeMap<Labels, Points> {
|
||||
series
|
||||
.into_iter()
|
||||
.map(|(labels, points)| {
|
||||
let points = points
|
||||
.into_iter()
|
||||
.map(|(index, bits)| {
|
||||
(
|
||||
index,
|
||||
crate::grid_series::reference::canonical_bits(f64::from_bits(bits)),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
(labels, points)
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
assert_eq!(canonical(actual), canonical(expected), "{case}");
|
||||
checked += 1;
|
||||
}
|
||||
assert!(checked >= 40, "{checked}");
|
||||
}
|
||||
|
||||
/// Parameters are read at each point of the input, whether they cover the grid or not: a point
|
||||
/// where a parameter has no value, absent from its axis or null, has no result, and every field
|
||||
/// takes the parameters of its point.
|
||||
#[test]
|
||||
fn parameters_are_read_per_point_and_missing_ones_drop_the_point() {
|
||||
let schema = Arc::new(schema(&["a", "b"]));
|
||||
let input = series(
|
||||
&schema,
|
||||
2,
|
||||
vec![
|
||||
floats(&[Some(1.4), Some(2.6), Some(3.5), Some(4.4)]),
|
||||
floats(&[Some(-1.4), None, Some(-3.5), Some(9.0)]),
|
||||
],
|
||||
);
|
||||
// `min` covers 2..=3 and 5 with a null at 3; `max` covers 2..=5.
|
||||
let min = Series::try_new(
|
||||
&SeriesSchema {
|
||||
fields: vec![FieldSpec {
|
||||
name: "scalar".into(),
|
||||
}]
|
||||
.into(),
|
||||
eval_grid: schema.eval_grid.clone(),
|
||||
},
|
||||
MetricName::try_new(None, vec![]).unwrap(),
|
||||
TimeAxis::from_grid_spans(
|
||||
schema.eval_grid.clone().unwrap(),
|
||||
vec![GridSpan::new(2, 2), GridSpan::new(5, 1)],
|
||||
)
|
||||
.unwrap(),
|
||||
SeriesFields::one(floats(&[Some(0.0), None, Some(5.0)])),
|
||||
)
|
||||
.unwrap();
|
||||
let max = scalar(
|
||||
&schema,
|
||||
2,
|
||||
floats(&[Some(2.0), Some(2.0), Some(2.0), Some(4.0)]),
|
||||
);
|
||||
let output = map_math_with_parameters(
|
||||
&schema,
|
||||
&input,
|
||||
ParameterizedMath::Clamp {
|
||||
min: &min,
|
||||
max: &max,
|
||||
},
|
||||
)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
// Point 2 clamps both fields to [0, 2]; point 3 has a null `min`; point 4 has no `min`; point 5
|
||||
// has `max < min`, so no result.
|
||||
assert_eq!(run_floats(&output, 0), vec![((2, 1), vec![Some(1.4)])]);
|
||||
assert_eq!(run_floats(&output, 1), vec![((2, 1), vec![Some(0.0)])]);
|
||||
}
|
||||
|
||||
/// Like `map_math`, the parameterized functions skip native histogram samples.
|
||||
#[test]
|
||||
fn parameterized_math_skips_histogram_samples() {
|
||||
let schema = Arc::new(schema(&["mixed"]));
|
||||
let mixed = Samples::try_from_mixed(
|
||||
Float64Array::from(vec![Some(1.26), None]),
|
||||
HistogramSamples::try_new(UInt32Array::from(vec![None, Some(0)]), histogram_batch())
|
||||
.unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
let input = series(&schema, 2, vec![mixed]);
|
||||
let to_nearest = scalar(&schema, 2, floats(&[Some(0.5), Some(0.5)]));
|
||||
let output = map_math_with_parameters(
|
||||
&schema,
|
||||
&input,
|
||||
ParameterizedMath::Round {
|
||||
to_nearest: &to_nearest,
|
||||
},
|
||||
)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(run_floats(&output, 0), vec![((2, 1), vec![Some(1.5)])]);
|
||||
}
|
||||
|
||||
@@ -174,7 +174,7 @@ fn evaluate(case: &Case) -> Vec<Series> {
|
||||
}
|
||||
|
||||
fn grid_len(case: &Case) -> usize {
|
||||
case.schema.eval_grid.as_ref().unwrap().timestamps().len()
|
||||
case.schema.eval_grid.as_ref().unwrap().spec().len as usize
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -170,7 +170,12 @@ fn eval_time_series(schema: &SeriesSchema, value: impl Fn(i64) -> f64) -> Result
|
||||
let grid = schema.eval_grid.clone().ok_or_else(|| {
|
||||
DataFusionError::Execution("time functions require an evaluation grid".into())
|
||||
})?;
|
||||
let values: Float64Array = grid.timestamps().iter().map(|&ms| value(ms)).collect();
|
||||
let values: Float64Array = grid
|
||||
.timestamps()
|
||||
.map_err(external)?
|
||||
.iter()
|
||||
.map(|&ms| value(ms))
|
||||
.collect();
|
||||
let range = GridRange::try_new(grid.clone(), 0, values.len()).map_err(external)?;
|
||||
let fields = vec![Samples::from_float(values); schema.fields.len()];
|
||||
Series::try_new(
|
||||
|
||||
@@ -129,11 +129,15 @@ fn describe(node: &NativeNode) -> String {
|
||||
}
|
||||
},
|
||||
NativeExpr::Aggregate { op, .. } => format!(
|
||||
"{} {} {}",
|
||||
match op.kind {
|
||||
AggregateKind::Sum => "sum",
|
||||
"{} {} [{}] {}",
|
||||
aggregate_kind(op.kind),
|
||||
match &op.group {
|
||||
Grouping::By(_) => "by",
|
||||
Grouping::Without(_) => "without",
|
||||
},
|
||||
grouping(&op.group),
|
||||
label_list(match &op.group {
|
||||
Grouping::By(labels) | Grouping::Without(labels) => labels,
|
||||
}),
|
||||
match op.stage {
|
||||
AggregateStage::Complete => "complete",
|
||||
AggregateStage::Partial => "partial",
|
||||
@@ -171,6 +175,21 @@ fn fill(values: FillValues) -> String {
|
||||
)
|
||||
}
|
||||
|
||||
/// The PromQL name of an aggregate, with the constant argument it was bound to.
|
||||
fn aggregate_kind(kind: AggregateKind) -> String {
|
||||
match kind {
|
||||
AggregateKind::Sum => "sum".to_string(),
|
||||
AggregateKind::Avg => "avg".to_string(),
|
||||
AggregateKind::Count => "count".to_string(),
|
||||
AggregateKind::Min => "min".to_string(),
|
||||
AggregateKind::Max => "max".to_string(),
|
||||
AggregateKind::Group => "group".to_string(),
|
||||
AggregateKind::Stddev => "stddev".to_string(),
|
||||
AggregateKind::Stdvar => "stdvar".to_string(),
|
||||
AggregateKind::Quantile { quantile } => format!("quantile {quantile}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// A window: what it runs, over which range, and where it lands.
|
||||
fn window(op: &BoundWindow) -> String {
|
||||
format!(
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
//! as one chunk addressed on its own grid. Nothing is decided for a series before it ends, which is
|
||||
//! the price of the kernel taking one axis rather than a chunk of one.
|
||||
//!
|
||||
//! A vector–vector binary node is the one kind that reads two inputs: it retains the chunks of both
|
||||
//! A vector–vector binary node is the only kind that reads two inputs: it retains the chunks of both
|
||||
//! operands and runs the matching kernel once the input has ended, because the operation pairs two
|
||||
//! whole vectors point by point and nothing is decided for a point until both sides are complete.
|
||||
//! The chunks of one logical series are assembled into the single series the kernel matches before it
|
||||
@@ -44,13 +44,23 @@
|
||||
//! (`topk`, `limitk`, `limit_ratio`) reads its one input the same way, because which samples it
|
||||
//! keeps at a point depends on every series present there.
|
||||
//!
|
||||
//! A constant scalar is a value node rather than a stream: it has no input and hands over no chunk,
|
||||
//! because what it produces is the one value it was bound to and not a series. A binary node with a
|
||||
//! scalar operand therefore reads one input — the vector side — and applies the operation to each
|
||||
//! chunk it is handed as it arrives: the scalar is read off the node's binding rather than pulled,
|
||||
//! and no chunk of the vector is retained, because one vector value pairs with one constant and
|
||||
//! nothing is decided by what comes after it. That broadcast is a different mode from the vector
|
||||
//! matching above and keeps neither operand: what it hands out is the chunk it transformed.
|
||||
//!
|
||||
//! A node kind this runtime does not execute yet is rejected when the execution is built, naming
|
||||
//! the kind and its position. An unsupported tree therefore never starts, rather than answering
|
||||
//! half a query or panicking halfway through one.
|
||||
//! the kind and its position. That is an aggregate, and a binary node whose two operands are
|
||||
//! neither two evaluated vectors nor one evaluated vector and one constant. An unsupported tree
|
||||
//! therefore never starts, rather than answering half a query or panicking halfway through one.
|
||||
//!
|
||||
//! A node that rewrites labels — a window drops the metric name and the empty-valued labels, a map
|
||||
//! drops the name — can turn two distinct input series into one output label set. That pair is
|
||||
//! legal when the two hold values at disjoint positions, which is exactly what
|
||||
//! drops the name, and a binary node drops it for arithmetic and for a `bool` comparison — can turn
|
||||
//! two distinct input series into one output label set. That pair is legal when the two hold values
|
||||
//! at disjoint positions, which is exactly what
|
||||
//! [`crate::grid_series::merge_same_labelset`] merges: `is_last` closes the source series whose
|
||||
//! data the chunk carries, not the label set it shares with another series, so a second series may
|
||||
//! fill the positions the first one left empty. The node that rewrote the labels keeps which
|
||||
@@ -96,16 +106,21 @@ use crate::grid_series::{
|
||||
};
|
||||
use crate::histogram_eval::evaluate_histograms;
|
||||
use crate::instant_eval::select_instants;
|
||||
use crate::map::{MathOp, drop_metric_name, map_math};
|
||||
use crate::map::{
|
||||
ArithmeticOp, Comparison, ComparisonOp, MathOp, ScalarOperand, drop_metric_name,
|
||||
map_arithmetic, map_compare, map_math,
|
||||
};
|
||||
use crate::native::batch::EvaluatedChunk;
|
||||
use crate::native::metrics::{NodeMetrics, NodePath};
|
||||
use crate::native::plan::{
|
||||
BoundInstant, BoundVectorOp, Grouping, NativeExpr, NativeNode, NativeTree,
|
||||
BoundInstant, BoundScalar, BoundVectorOp, Grouping, NativeExpr, NativeNode, NativeTree,
|
||||
NativeType,
|
||||
};
|
||||
use crate::range_eval::RangeEvaluator;
|
||||
use crate::select::{self, SortOrder};
|
||||
use crate::vector_match::{
|
||||
BinaryMatching, binary_vectors_with_reservation, validate_binary_operand,
|
||||
BinaryMatching, BinaryOp, Cardinality, FillValues, MatchingLabels,
|
||||
binary_vectors_with_reservation, validate_binary_operand,
|
||||
};
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -377,13 +392,21 @@ fn buffer_bytes(buffer: &Buffer) -> usize {
|
||||
enum NodeTag {
|
||||
/// Forwards the chunks it is given.
|
||||
Input,
|
||||
/// A constant scalar: the value its consumer broadcasts, and no chunk of its own.
|
||||
///
|
||||
/// It is a value node and not a stream. Its output type is [`NativeType::Scalar`], and the
|
||||
/// runtime produces evaluated chunks only, so the node hands nothing over: what its consumer
|
||||
/// reads is the constant bound to it, which is why it has no input and no run-time state. A
|
||||
/// `pull` of it answers `None` for exactly that reason, and that is not a stall to fix.
|
||||
Scalar,
|
||||
/// Buffers one series and selects the last sample at or before each of its grid points.
|
||||
Instant,
|
||||
/// Feeds its input to its own [`RangeEvaluator`].
|
||||
Window,
|
||||
/// Applies point-wise operations to its input chunks.
|
||||
Map,
|
||||
/// Matches two evaluated vectors and applies one operation to every matched pair of series.
|
||||
/// Matches two evaluated vectors and applies one operation to every matched pair of series, or
|
||||
/// broadcasts one constant over the chunks of one evaluated vector.
|
||||
Binary,
|
||||
/// Keeps some samples of its whole input vector at each grid point.
|
||||
Vector,
|
||||
@@ -394,6 +417,7 @@ impl NodeTag {
|
||||
fn describe(self) -> &'static str {
|
||||
match self {
|
||||
Self::Input => "input",
|
||||
Self::Scalar => "scalar",
|
||||
Self::Instant => "instant selection",
|
||||
Self::Window => "window",
|
||||
Self::Map => "map",
|
||||
@@ -1934,10 +1958,9 @@ fn vector_workspace_bytes(op: &BoundVectorOp, schema: &SeriesSchema, series: &[S
|
||||
});
|
||||
}
|
||||
BoundVectorOp::Absent { .. } => {
|
||||
let points = schema
|
||||
.eval_grid
|
||||
.as_ref()
|
||||
.map_or(0, |grid| grid.timestamps().len());
|
||||
let points = schema.eval_grid.as_ref().map_or(0, |grid| {
|
||||
usize::try_from(grid.spec().len).unwrap_or(usize::MAX)
|
||||
});
|
||||
return points.saturating_mul(fields.saturating_mul(1 + std::mem::size_of::<f64>()));
|
||||
}
|
||||
BoundVectorOp::Histogram(_) => {
|
||||
@@ -1987,6 +2010,86 @@ fn vector_workspace_bytes(op: &BoundVectorOp, schema: &SeriesSchema, series: &[S
|
||||
})
|
||||
}
|
||||
|
||||
/// The operation a scalar broadcast applies, with the side the constant sits on already resolved.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
enum BroadcastOp {
|
||||
/// Float arithmetic between the vector and the constant.
|
||||
Arithmetic(ArithmeticOp),
|
||||
/// A comparison: `Filter` keeps the positions that passed, `Bool` reports `1`/`0` for them.
|
||||
Compare(Comparison),
|
||||
}
|
||||
|
||||
/// What a binary node holds when one operand is a constant: the operation, its scalar operand and
|
||||
/// the layout of the vector side.
|
||||
///
|
||||
/// A broadcast applies to each chunk as it arrives; unlike [`BinaryRuntime`], it does not retain
|
||||
/// both operands for whole-vector matching.
|
||||
struct BroadcastRuntime {
|
||||
/// The scalar and its position in the operation.
|
||||
operand: ScalarOperand,
|
||||
/// The operation applied to every value of the vector.
|
||||
op: BroadcastOp,
|
||||
/// The layout of the vector side: the fields a chunk of it carries, on the grid it is addressed
|
||||
/// on. A chunk of the vector side is checked against it by the kernel itself.
|
||||
sample_schema: Arc<SeriesSchema>,
|
||||
}
|
||||
|
||||
impl BroadcastRuntime {
|
||||
/// Whether the operation rewrites the label set of what it hands out.
|
||||
///
|
||||
/// Arithmetic and a `bool` comparison drop the metric name, so two input series can become one
|
||||
/// label set under this node and what it hands out is recorded in an [`Emitted`] table. A
|
||||
/// filtering comparison keeps the label set it read, so it keeps none. The answer is read off
|
||||
/// [`Self::op`] rather than stored beside it, so an operation and its label policy cannot
|
||||
/// disagree.
|
||||
fn drops_labels(&self) -> bool {
|
||||
match self.op {
|
||||
BroadcastOp::Arithmetic(_) | BroadcastOp::Compare(Comparison::Bool(_)) => true,
|
||||
BroadcastOp::Compare(Comparison::Filter(_)) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The label set a chunk of the vector side's `metric` becomes under this operation.
|
||||
///
|
||||
/// It is what a series closes under when the operation emptied every position: the labels say
|
||||
/// which series closed, and they are the labels the operation produces, not the ones it read.
|
||||
fn output_metric(&self, metric: &MetricName) -> Result<MetricName> {
|
||||
if self.drops_labels() {
|
||||
drop_metric_name(metric)
|
||||
} else {
|
||||
Ok(metric.clone())
|
||||
}
|
||||
}
|
||||
|
||||
/// Applies the operation to one chunk of the vector side.
|
||||
///
|
||||
/// `None` means the operation emptied every position and the chunk does not close its series:
|
||||
/// the input may still decide more, so nothing is handed on. A chunk that does close its series
|
||||
/// is answered even then — with a series carrying no row — because a consumer tells "nothing
|
||||
/// more here" from "no more input" by that chunk alone.
|
||||
fn apply(&self, chunk: &EvaluatedChunk) -> Result<Option<Series>> {
|
||||
let output = match self.op {
|
||||
BroadcastOp::Arithmetic(op) => Some(map_arithmetic(
|
||||
&self.sample_schema,
|
||||
&chunk.series,
|
||||
op,
|
||||
self.operand,
|
||||
)?),
|
||||
BroadcastOp::Compare(comparison) => {
|
||||
map_compare(&self.sample_schema, &chunk.series, comparison, self.operand)?
|
||||
}
|
||||
};
|
||||
match output {
|
||||
Some(output) => Ok(Some(output)),
|
||||
None if chunk.is_last => {
|
||||
let metric = self.output_metric(&chunk.series.metric)?;
|
||||
no_row_series(&self.sample_schema, &chunk.series, metric).map(Some)
|
||||
}
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The buffered chunks as the one source axis they make up: their timestamps in arrival order.
|
||||
///
|
||||
/// The axis type validates the result as strictly increasing, which is also what refuses a series
|
||||
@@ -2152,10 +2255,19 @@ struct NodeRuntime {
|
||||
window: Option<RangeEvaluator>,
|
||||
/// What a map node applies.
|
||||
map: Option<MapRuntime>,
|
||||
/// What a binary node holds.
|
||||
/// What a binary node holds when it matches two vectors: both operands, until the input ends.
|
||||
///
|
||||
/// `None` for every other node, and for a binary node that broadcasts a constant instead: that
|
||||
/// one holds no operand at all and is described by [`NodeRuntime::broadcast`].
|
||||
binary: Option<BinaryRuntime>,
|
||||
/// What a vector node holds.
|
||||
vector: Option<VectorRuntime>,
|
||||
/// What a binary node holds when one operand is a constant: the operation and the layout of the
|
||||
/// vector side.
|
||||
///
|
||||
/// The two modes are exclusive. A broadcast transforms each chunk of the vector side as it
|
||||
/// arrives and retains none of them, so what it holds is its own configuration and not data.
|
||||
broadcast: Option<BroadcastRuntime>,
|
||||
/// The layout the caller's chunks of an input node are checked against.
|
||||
///
|
||||
/// A tree with a binary node reads two inputs, and each of them is bound to its own layout, so
|
||||
@@ -2291,6 +2403,64 @@ impl NodeRuntime {
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// Broadcasts one chunk of the vector side with the constant this node was bound to.
|
||||
///
|
||||
/// `None` means the operation emptied every position and the chunk does not close its series:
|
||||
/// there is nothing to hand on, and the vector side may still decide more. Nothing of the chunk
|
||||
/// is retained — the operation pairs one value with one constant — so what this node holds
|
||||
/// afterwards is the label sets it has handed out and nothing else.
|
||||
fn broadcast(&mut self, path: &NodePath, chunk: NativeChunk) -> Result<Option<EvaluatedChunk>> {
|
||||
let NativeChunk::Evaluated(chunk) = chunk else {
|
||||
return Err(DataFusionError::Execution(format!(
|
||||
"the binary node at {path} broadcasts over an evaluated vector, not raw samples"
|
||||
)));
|
||||
};
|
||||
let (positions, closing) = (chunk.series.time.len(), chunk.is_last);
|
||||
// The chunk is held for as long as this node works on it.
|
||||
self.buffer.grow(evaluated_bytes(&chunk))?;
|
||||
let metrics = Arc::clone(&self.buffer.metrics);
|
||||
let outcome = {
|
||||
// The timer covers this node's own work only: it is dropped before the driver polls
|
||||
// the vector side again.
|
||||
let _timer = metrics.timer();
|
||||
match self.broadcast.as_ref() {
|
||||
Some(broadcast) => broadcast.apply(&chunk),
|
||||
None => Err(no_broadcast(path)),
|
||||
}
|
||||
};
|
||||
metrics.record_input(1, positions);
|
||||
if closing {
|
||||
// A series closes exactly once, on the chunk carrying `is_last`: the broadcast kept it
|
||||
// on every chunk it handed up, so the series it read is closed here.
|
||||
metrics.record_completed_series(1);
|
||||
}
|
||||
// An error leaves the reservation to the driver, which releases every node when it stops.
|
||||
let output = outcome?;
|
||||
let output = match output {
|
||||
Some(series) => Some(EvaluatedChunk::try_new(series, closing)?),
|
||||
None => None,
|
||||
};
|
||||
if let Some(output) = &output {
|
||||
metrics.record_grid_points(output.series.time.len());
|
||||
metrics.record_output(1, valid_values(output));
|
||||
// Only an operation that rewrites the label set is where two input series can become
|
||||
// one, and only that one keeps the table `track` records into. A filtering comparison
|
||||
// keeps the label set it read, so recording it would refuse two input series that
|
||||
// share it without this node having made them share it.
|
||||
if self
|
||||
.broadcast
|
||||
.as_ref()
|
||||
.is_some_and(BroadcastRuntime::drops_labels)
|
||||
{
|
||||
self.track(path, output)?;
|
||||
}
|
||||
}
|
||||
// The chunk is gone and the label sets this node has handed out are what is left.
|
||||
let emitted = self.emitted_bytes();
|
||||
self.buffer.resize(emitted)?;
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// Feeds one chunk to this node's window.
|
||||
///
|
||||
/// A window that collects its input keeps the whole series of every chunk it is handed and
|
||||
@@ -2702,6 +2872,7 @@ impl NodeRuntime {
|
||||
self.instant = None;
|
||||
self.binary = None;
|
||||
self.vector = None;
|
||||
self.broadcast = None;
|
||||
self.source_schema = None;
|
||||
self.emitted = None;
|
||||
self.merge = None;
|
||||
@@ -2788,15 +2959,18 @@ impl NativeExecutionState {
|
||||
NativeExpr::Window { .. } => NodeTag::Window,
|
||||
NativeExpr::Map { .. } => NodeTag::Map,
|
||||
// A binary node is built here and narrowed below: legality is a wider set of operand
|
||||
// types than the kernel takes, so the shape is checked where the operands' layouts
|
||||
// are read.
|
||||
// types than the kernels take, so which of its two modes it is -- a match between
|
||||
// two vectors, or a constant broadcast over one -- is decided where the operands'
|
||||
// layouts are read.
|
||||
NativeExpr::Binary { .. } => NodeTag::Binary,
|
||||
NativeExpr::Vector { .. } => NodeTag::Vector,
|
||||
// The kinds below are legal in a bound tree, and `validate` stays the authority on
|
||||
// that; this runtime simply does not execute them yet. Rejected before a single
|
||||
// chunk is read: a tree the runtime cannot execute must not start, because a
|
||||
// partial answer would be one the caller never asked for.
|
||||
NativeExpr::Scalar(_) => return Err(not_implemented("Scalar", &path)),
|
||||
// A scalar is a value node: it produces the constant it was bound to, hands over no
|
||||
// chunk and holds no state, so it is built like any other node and pulled like none.
|
||||
NativeExpr::Scalar(_) => NodeTag::Scalar,
|
||||
// An aggregate is legal in a bound tree, and `validate` stays the authority on that;
|
||||
// this runtime simply does not execute it yet. Rejected before a single chunk is
|
||||
// read: a tree the runtime cannot execute must not start, because a partial answer
|
||||
// would be one the caller never asked for.
|
||||
NativeExpr::Aggregate { .. } => return Err(not_implemented("Aggregate", &path)),
|
||||
};
|
||||
let buffer = NodeBuffer::new(&path, handles, &pool);
|
||||
@@ -2809,6 +2983,7 @@ impl NativeExecutionState {
|
||||
map: None,
|
||||
binary: None,
|
||||
vector: None,
|
||||
broadcast: None,
|
||||
source_schema: None,
|
||||
emitted: None,
|
||||
merge: None,
|
||||
@@ -2872,11 +3047,12 @@ impl NativeExecutionState {
|
||||
right,
|
||||
matching,
|
||||
} => {
|
||||
// The one shape this runtime executes is two evaluated vectors: a scalar side is
|
||||
// a broadcast the kernel does not take, and a raw side has no positions to be
|
||||
// matched at. Both are rejected as this node, before the pass reaches the
|
||||
// operand that produced them, so the position named is the one that asked for
|
||||
// them.
|
||||
// A binary node has two modes here, and the operands' layouts are what tell
|
||||
// them apart. Two evaluated vectors are matched by label set and retain both
|
||||
// operands; one evaluated vector and a constant is a broadcast, which pairs
|
||||
// each vector value with the constant as the chunk carrying it arrives. All
|
||||
// that is left is refused as this node, before the pass reaches the operands
|
||||
// that produced them, so the position named is the one that asked for them.
|
||||
match (left.output.samples(), right.output.samples()) {
|
||||
(Some(left), Some(right))
|
||||
if left.eval_grid.is_some() && right.eval_grid.is_some() =>
|
||||
@@ -2888,11 +3064,22 @@ impl NativeExecutionState {
|
||||
[Arc::clone(left), Arc::clone(right)],
|
||||
));
|
||||
}
|
||||
_ => return Err(not_implemented("Binary", &path)),
|
||||
_ => {
|
||||
let broadcast = broadcast_runtime(&path, left, right, matching)?;
|
||||
if broadcast.drops_labels() {
|
||||
// The operation rewrites the label set it hands out, so this node
|
||||
// is one where two input series can become one: it keeps the table
|
||||
// that refuses a second value at a position the first one filled.
|
||||
runtime.emitted = Some(Emitted::default());
|
||||
}
|
||||
runtime.broadcast = Some(broadcast);
|
||||
}
|
||||
}
|
||||
// A binary operation matches by label set, so a result label set may be
|
||||
// produced at points that do not collide; the kernel groups and merges them
|
||||
// itself, which is why this node keeps no ledger of what it handed out.
|
||||
// A binary operation over two vectors matches by label set, so a result label
|
||||
// set may be produced at points that do not collide; the kernel groups and
|
||||
// merges them itself, which is why that mode keeps no ledger of what it handed
|
||||
// out. A broadcast that keeps the labels it reads is not where two series
|
||||
// become one either, so it keeps none; the one that drops them does, above.
|
||||
}
|
||||
NativeExpr::Vector { op, .. } => {
|
||||
let schema = node.output.samples().cloned().ok_or_else(|| {
|
||||
@@ -2917,7 +3104,9 @@ impl NativeExecutionState {
|
||||
// label sets of its own, so it keeps no ledger of what it handed out.
|
||||
}
|
||||
NativeExpr::Scalar(_) | NativeExpr::Aggregate { .. } => {
|
||||
// Rejected above, where the pass has the position to name.
|
||||
// A scalar holds nothing: what its consumer reads is the constant the node was
|
||||
// bound to, which is on the tree rather than in this state. The aggregate is
|
||||
// rejected above, where the pass has the position to name.
|
||||
}
|
||||
}
|
||||
// The inputs this node reads, in the order the tree declares them: a unary node's one
|
||||
@@ -2950,6 +3139,16 @@ impl NativeExecutionState {
|
||||
runtimes.push(runtime);
|
||||
}
|
||||
|
||||
// A caller names a node by the position the metrics give it, and a chunk it queues is bound
|
||||
// to the input that position names, so every position is indexed once the pass has given the
|
||||
// whole tree one. The reverse pass below reads it as well, because an operand is not always
|
||||
// the node the spec's `input` names: a broadcast's vector side may be its right operand.
|
||||
let by_path: HashMap<NodePath, usize> = paths
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, path)| (path.clone(), index))
|
||||
.collect();
|
||||
|
||||
// One reverse pass sets where a merge has to happen, from the input upwards: the pass above
|
||||
// walked the root first, so a node's input is the node after it and has been decided by the
|
||||
// time the node itself is.
|
||||
@@ -2990,14 +3189,40 @@ impl NativeExecutionState {
|
||||
holds_shared[index] = input_holds_shared;
|
||||
}
|
||||
NodeTag::Map => holds_shared[index] = true,
|
||||
NodeTag::Binary => {
|
||||
// The kernel matches its operands by label set and files one result row per
|
||||
// label set and point, merging the rows of one set whatever piece they came
|
||||
// from: what it hands out is one series per label set, so nothing above has
|
||||
// to join pieces of a series. Its own operands are read whole -- it retains
|
||||
// them until the input ends -- so its flag is false whatever theirs are.
|
||||
NodeTag::Scalar => {
|
||||
// A scalar produces one value and no labels, so it is not a stream of series
|
||||
// and nothing above it can read two of them under one label set. The flag
|
||||
// is false rather than inherited: the node has no input at all.
|
||||
holds_shared[index] = false;
|
||||
}
|
||||
NodeTag::Binary => {
|
||||
// A vector match files one result row per label set and point, merging the
|
||||
// rows of one set whatever piece they came from: what it hands out is one
|
||||
// series per label set, so nothing above has to join pieces of a series. Its
|
||||
// own operands are read whole -- it retains them until the input ends -- so
|
||||
// its flag is false whatever theirs are.
|
||||
//
|
||||
// A broadcast hands each input chunk up transformed instead, so it merges
|
||||
// nothing: what its output can hold under one label set is what its vector
|
||||
// side held, plus the pair its own label rewrite can create. One that drops
|
||||
// the metric name -- arithmetic and a `bool` comparison -- is therefore where
|
||||
// two input series become one label set, and one that keeps the name only
|
||||
// inherits the pair its input already had. The flag of the *vector* side is
|
||||
// the one that matters: the constant is a node with no input at all.
|
||||
holds_shared[index] = match runtimes[index].broadcast.as_ref() {
|
||||
Some(broadcast) => {
|
||||
let vector = match broadcast.operand {
|
||||
ScalarOperand::Right(_) => nodes[index].input,
|
||||
ScalarOperand::Left(_) => {
|
||||
by_path.get(&paths[index].right()).copied()
|
||||
}
|
||||
};
|
||||
broadcast.drops_labels()
|
||||
|| vector.is_some_and(|vector| holds_shared[vector])
|
||||
}
|
||||
None => false,
|
||||
};
|
||||
}
|
||||
NodeTag::Vector => {
|
||||
// Most vector operations keep samples of the series they read, so pieces of one label
|
||||
// set stay pieces: they hold disjoint points, so ranking them apart picks
|
||||
@@ -3075,14 +3300,6 @@ impl NativeExecutionState {
|
||||
}
|
||||
}
|
||||
|
||||
// A caller names a node by the position the metrics give it, and a chunk it queues is bound
|
||||
// to the input that position names, so every position is indexed once the pass has given the
|
||||
// whole tree one.
|
||||
let by_path: HashMap<NodePath, usize> = paths
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, path)| (path.clone(), index))
|
||||
.collect();
|
||||
// Every input node is where a caller's chunks can enter, and each of them is named by its
|
||||
// own position when a caller queues a chunk. A tree of unary nodes has one input; a binary
|
||||
// node adds one per operand, so a tree of nested binary nodes has as many as it has leaves.
|
||||
@@ -3254,8 +3471,9 @@ impl NativeExecutionState {
|
||||
/// The driver owns the state and nothing else: every decision a node makes was bound before the
|
||||
/// tree existed, so a pull only reads the position and runs the bound operation.
|
||||
///
|
||||
/// A window directly over a source evaluates its chunks as they arrive, and a map at the root hands
|
||||
/// its chunks on as it produces them: both stream. A window whose input can hold two series under
|
||||
/// A window directly over a source evaluates its chunks as they arrive, a map at the root hands
|
||||
/// its chunks on as it produces them, and a binary node over a constant transforms each chunk of
|
||||
/// the vector side as it arrives: all three stream. A window whose input can hold two series under
|
||||
/// one canonical label set collects that input instead, merges each set once and evaluates it, so
|
||||
/// it reports at [`Self::finish`] rather than as its input arrives.
|
||||
pub struct NativeExecutor {
|
||||
@@ -3336,10 +3554,12 @@ impl NativeExecutor {
|
||||
///
|
||||
/// A window and a map take whole chunks from their input, so nothing beyond the chunk in hand
|
||||
/// is materialised on their edge -- except below a window that collects its input, which holds
|
||||
/// that input until [`Self::finish`]. A binary node is the other exception: it pairs two whole
|
||||
/// vectors point by point, so it retains every chunk each operand hands over and reports `None`
|
||||
/// until [`Self::finish`] ends the input, which is when it computes its result. A vector node does
|
||||
/// the same with its one input.
|
||||
/// that input until [`Self::finish`]. A binary node that matches two vectors is the other
|
||||
/// exception: it pairs two whole vectors point by point, so it retains every chunk each operand
|
||||
/// hands over and reports `None` until [`Self::finish`] ends the input, which is when it
|
||||
/// computes its result. A binary node that broadcasts a constant is not one: it pairs each
|
||||
/// value of the vector side with the constant and hands the transformed chunk up as it arrives. A vector node
|
||||
/// reads its one input whole the same way.
|
||||
///
|
||||
/// A chunk's `is_last` says the series whose data it carries ended. It does not say its label
|
||||
/// set is finished: two source series whose labels were rewritten into one set are two pieces
|
||||
@@ -3508,9 +3728,10 @@ impl NativeExecutor {
|
||||
/// Whether the node at `index` has produced everything it ever will.
|
||||
///
|
||||
/// A pull that finds nothing does not answer it: a collecting window reports nothing until it
|
||||
/// is flushed, and a binary node reports nothing until both nodes it reads from are finished.
|
||||
/// Each node therefore answers from its own state and from its children's, which is what keeps
|
||||
/// a binary node from computing on an operand a window below it has not handed over yet.
|
||||
/// is flushed, and a binary node that matches two vectors reports nothing until both nodes it
|
||||
/// reads from are finished. Each node therefore answers from its own state and from its
|
||||
/// children's, which is what keeps a binary node from computing on an operand a window below it
|
||||
/// has not handed over yet.
|
||||
fn settled(&self, index: usize) -> bool {
|
||||
let runtime = &self.state.runtimes[index];
|
||||
if !runtime.queued.is_empty() || runtime.merge.is_some() {
|
||||
@@ -3518,6 +3739,9 @@ impl NativeExecutor {
|
||||
}
|
||||
match self.state.nodes[index].tag {
|
||||
NodeTag::Input => true,
|
||||
// A scalar produces its value without reading anything, so it is settled from the
|
||||
// start: there is no chunk it owes its consumer.
|
||||
NodeTag::Scalar => true,
|
||||
NodeTag::Map | NodeTag::Window | NodeTag::Instant => self.state.nodes[index]
|
||||
.input
|
||||
.is_none_or(|child| self.settled(child)),
|
||||
@@ -3531,6 +3755,16 @@ impl NativeExecutor {
|
||||
.is_none_or(|child| self.settled(child))
|
||||
}
|
||||
NodeTag::Binary => {
|
||||
// A broadcast owes only what its vector side has not handed over yet: it answers
|
||||
// each chunk as it reads it and keeps nothing, so it is settled exactly when that
|
||||
// side is. A vector match is settled once its kernel ran and both operands are.
|
||||
if let Some(broadcast) = &runtime.broadcast {
|
||||
let vector = match broadcast.operand {
|
||||
ScalarOperand::Right(_) => self.state.nodes[index].input,
|
||||
ScalarOperand::Left(_) => self.state.right_input(index).ok(),
|
||||
};
|
||||
return vector.is_some_and(|vector| self.settled(vector));
|
||||
}
|
||||
runtime
|
||||
.binary
|
||||
.as_ref()
|
||||
@@ -3626,6 +3860,10 @@ impl NativeExecutor {
|
||||
let input = self.state.nodes[index].input;
|
||||
match tag {
|
||||
NodeTag::Input => self.state.runtimes[index].pop_source(),
|
||||
// A scalar owes its consumer no chunk: it produces one value and no series, and the
|
||||
// value is the constant on the tree, which the consumer reads off its own binding. A
|
||||
// pull of it is therefore empty by construction and says nothing has stalled.
|
||||
NodeTag::Scalar => Ok(None),
|
||||
NodeTag::Map => {
|
||||
let child = input.ok_or_else(|| missing_input(tag))?;
|
||||
// An operation that empties every position yields no chunk, and the map asks its
|
||||
@@ -3676,6 +3914,13 @@ impl NativeExecutor {
|
||||
// one is the input a spec of two fields would have had to carry.
|
||||
let left = input.ok_or_else(|| missing_input(tag))?;
|
||||
let right = self.state.right_input(index)?;
|
||||
// A broadcast reads the vector side only: the constant is on the node's binding, so
|
||||
// there is no second stream to pull and no chunk to retain. Each chunk is
|
||||
// transformed as it arrives and handed up, which is what keeps a broadcast over a
|
||||
// whole vector from costing the vector.
|
||||
if self.state.runtimes[index].broadcast.is_some() {
|
||||
return self.broadcast_node(index, left, right);
|
||||
}
|
||||
// Both operands are read to the end of what they hand over: before the input ends a
|
||||
// binary node only retains the chunks, because a pair is not decided until both
|
||||
// sides are whole, and after it ends this is the drain that empties the two.
|
||||
@@ -3716,6 +3961,45 @@ impl NativeExecutor {
|
||||
}
|
||||
}
|
||||
|
||||
/// Pulls the vector side of the broadcast at `index` until it hands over a chunk, and returns
|
||||
/// that chunk transformed.
|
||||
///
|
||||
/// Only the vector side is pulled: the scalar side is the constant the node was bound to. The
|
||||
/// chunk is transformed inside this call and nothing of it is kept, so what a broadcast holds
|
||||
/// does not grow with the vector it reads.
|
||||
fn broadcast_node(
|
||||
&mut self,
|
||||
index: usize,
|
||||
left: usize,
|
||||
right: usize,
|
||||
) -> Result<Option<NativeChunk>> {
|
||||
let vector = {
|
||||
let broadcast = self.state.runtimes[index]
|
||||
.broadcast
|
||||
.as_ref()
|
||||
.ok_or_else(|| no_broadcast(&self.state.paths[index]))?;
|
||||
match broadcast.operand {
|
||||
ScalarOperand::Right(_) => left,
|
||||
ScalarOperand::Left(_) => right,
|
||||
}
|
||||
};
|
||||
// An operation that empties every position yields no chunk, and the node asks its vector
|
||||
// side again rather than reporting a decision it did not make.
|
||||
loop {
|
||||
let Some(chunk) = self.pull(vector)? else {
|
||||
// Nothing is left in the vector side. Before the input ends that is a stall rather
|
||||
// than an end, and so is a vector side still holding collected input: the node has
|
||||
// answered everything it can and owes whatever that side has not produced yet.
|
||||
return Ok(None);
|
||||
};
|
||||
let path = &self.state.paths[index];
|
||||
let runtime = &mut self.state.runtimes[index];
|
||||
if let Some(output) = runtime.broadcast(path, chunk)? {
|
||||
return Ok(Some(NativeChunk::Evaluated(output)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Maps one chunk the node at `index` pulled from its input.
|
||||
fn map_node(&mut self, index: usize, chunk: NativeChunk) -> Result<Option<NativeChunk>> {
|
||||
let NativeChunk::Evaluated(chunk) = chunk else {
|
||||
@@ -3854,7 +4138,10 @@ fn map_chunk(
|
||||
// holds no value now, so there is no row to hand over.
|
||||
None => {
|
||||
return if chunk.is_last {
|
||||
let empty = no_row_series(schema, &chunk.series)?;
|
||||
// A map drops the metric name, so the empty series closes under the labels the
|
||||
// map produces.
|
||||
let metric = drop_metric_name(&chunk.series.metric)?;
|
||||
let empty = no_row_series(schema, &chunk.series, metric)?;
|
||||
EvaluatedChunk::try_new(empty, true).map(Some)
|
||||
} else {
|
||||
Ok(None)
|
||||
@@ -3865,11 +4152,13 @@ fn map_chunk(
|
||||
EvaluatedChunk::try_new(series, chunk.is_last).map(Some)
|
||||
}
|
||||
|
||||
/// A series carrying the identity of `series` and no row at all.
|
||||
/// A series carrying `metric` and no row at all.
|
||||
///
|
||||
/// It is what a closing chunk holds when the operation emptied its input: the labels say which
|
||||
/// series closed, and the axis is empty because no position holds a value.
|
||||
fn no_row_series(schema: &SeriesSchema, series: &Series) -> Result<Series> {
|
||||
/// series closed, and the axis is empty because no position holds a value. `metric` is the identity
|
||||
/// the operation produces — a map drops the name, a filtering comparison keeps it — so the caller
|
||||
/// says which, rather than this function deciding for it.
|
||||
fn no_row_series(schema: &SeriesSchema, series: &Series, metric: MetricName) -> Result<Series> {
|
||||
let grid = series.time.grid().ok_or_else(|| {
|
||||
DataFusionError::Execution("a mapped chunk must be addressed on a grid".to_string())
|
||||
})?;
|
||||
@@ -3877,7 +4166,6 @@ fn no_row_series(schema: &SeriesSchema, series: &Series) -> Result<Series> {
|
||||
.map(|_| Samples::from_float(Float64Array::from(Vec::<f64>::new())))
|
||||
.collect();
|
||||
let fields = SeriesFields::try_from_vec(fields).map_err(series_error)?;
|
||||
let metric = drop_metric_name(&series.metric)?;
|
||||
let at = series.time.last_position().map_or(0, |last| last + 1);
|
||||
let time = GridRange::try_new(Arc::clone(grid), at, 0).map_err(series_error)?;
|
||||
Series::try_new(schema, metric, TimeAxis::Grid(time), fields).map_err(series_error)
|
||||
@@ -3913,7 +4201,9 @@ fn evaluated_values(chunk: &NativeChunk) -> usize {
|
||||
/// The rejection of a node kind this runtime does not execute yet.
|
||||
///
|
||||
/// It names the kind and the node's position: the position is what the metrics and `EXPLAIN` call
|
||||
/// the same node, so a reader can find it in the tree.
|
||||
/// the same node, so a reader can find it in the tree. An aggregate is the one kind left here: a
|
||||
/// scalar executes as a value node, and a binary node executes both as a vector match and as a
|
||||
/// constant broadcast.
|
||||
fn not_implemented(kind: &str, path: &NodePath) -> DataFusionError {
|
||||
DataFusionError::NotImplemented(format!(
|
||||
"the native runtime does not execute {kind} nodes yet: {kind} at {path}"
|
||||
@@ -4010,6 +4300,154 @@ fn no_vector(path: &NodePath) -> DataFusionError {
|
||||
DataFusionError::Execution(format!("the node at {path} holds no vector operation"))
|
||||
}
|
||||
|
||||
/// The error for a node asked to broadcast without the broadcast its kind holds.
|
||||
fn no_broadcast(path: &NodePath) -> DataFusionError {
|
||||
DataFusionError::Execution(format!("the node at {path} holds no scalar broadcast"))
|
||||
}
|
||||
|
||||
/// The rejection of a scalar broadcast whose matching is not the one a constant can take.
|
||||
///
|
||||
/// A constant has no labels, so there is nothing to match it against: Prometheus does not consult
|
||||
/// the matching modifiers of a vector–scalar operation at all. A tree that carries them therefore
|
||||
/// describes an operation that is not the one it would compute, which is refused rather than run
|
||||
/// as if the modifiers were absent.
|
||||
fn matched_scalar(path: &NodePath) -> DataFusionError {
|
||||
DataFusionError::NotImplemented(format!(
|
||||
"the native runtime does not execute a scalar broadcast whose constant carries matching \
|
||||
modifiers: a constant has no labels to match, so the operation this Binary at {path} \
|
||||
describes is not the one it would compute"
|
||||
))
|
||||
}
|
||||
|
||||
/// The rejection of a binary node whose operands are neither two evaluated vectors nor one
|
||||
/// evaluated vector and one constant.
|
||||
///
|
||||
/// `operands` says which shape arrived and why that shape has no path here, so a tree a later
|
||||
/// stage compiles is refused for a reason a reader can act on. The kind and the position are named
|
||||
/// the way every other rejection in this runtime names them.
|
||||
fn unsupported_binary(path: &NodePath, operands: &str) -> DataFusionError {
|
||||
DataFusionError::NotImplemented(format!(
|
||||
"the native runtime does not execute this Binary node: it reads {operands}, and this \
|
||||
runtime reads two evaluated vectors or one evaluated vector and one constant, at {path}"
|
||||
))
|
||||
}
|
||||
|
||||
/// The broadcast a binary node with one constant operand is, read off its binding.
|
||||
///
|
||||
/// The two sides say which mode this is: one evaluated vector and one constant is a broadcast. The
|
||||
/// constant comes from the scalar node's own binding — the value the node produces — and the
|
||||
/// operation from the matching the compiler bound; [`ScalarOperand`] preserves the constant's side.
|
||||
///
|
||||
/// Of the rejection arms in the operand-type match below, only the two-constant one is a shape a
|
||||
/// tree that `validate` accepts can put here; the others are backstops that keep the match
|
||||
/// exhaustive. That is not the wide safety margin it looks like: [`NativeNode::validate`] refuses a
|
||||
/// binary operand that produces neither evaluated values nor a scalar before any of this runs, so a
|
||||
/// raw side or an aggregate-state side can never reach this function at all. The refusals after the
|
||||
/// match — a constant operand that is not a literal scalar node, and a constant carrying matching
|
||||
/// modifiers — are reachable through a validated tree.
|
||||
fn broadcast_runtime(
|
||||
path: &NodePath,
|
||||
left: &NativeNode,
|
||||
right: &NativeNode,
|
||||
matching: &BinaryMatching,
|
||||
) -> Result<BroadcastRuntime> {
|
||||
let (scalar_operand, vector, constant) = match (&left.output, &right.output) {
|
||||
(NativeType::Samples(vector), NativeType::Scalar(_)) if vector.eval_grid.is_some() => (
|
||||
ScalarOperand::Right as fn(f64) -> ScalarOperand,
|
||||
vector,
|
||||
right,
|
||||
),
|
||||
(NativeType::Scalar(_), NativeType::Samples(vector)) if vector.eval_grid.is_some() => (
|
||||
ScalarOperand::Left as fn(f64) -> ScalarOperand,
|
||||
vector,
|
||||
left,
|
||||
),
|
||||
// The two-constant pair is reachable through the state builder, which is what this arm
|
||||
// guards: `validate` accepts a binary node of two constants (it binds them to a scalar), and
|
||||
// a compiler that could not fold a scalar–scalar operation would emit one. No constant has
|
||||
// labels or fields to broadcast over, so there is no vector to apply the operation to. The
|
||||
// executor test `an_unimplemented_node_is_reported_not_panicked` builds this pair directly.
|
||||
(NativeType::Scalar(_), NativeType::Scalar(_)) => {
|
||||
return Err(unsupported_binary(
|
||||
path,
|
||||
"two constants, which broadcast over no vector",
|
||||
));
|
||||
}
|
||||
// Raw samples or aggregate state: `validate` already refused either as a binary operand, so
|
||||
// this arm is the exhaustiveness backstop rather than a reachable path.
|
||||
(_, _) => {
|
||||
return Err(unsupported_binary(
|
||||
path,
|
||||
"an operand that is neither evaluated samples nor a constant, which no validated \
|
||||
tree binds as a binary operand",
|
||||
));
|
||||
}
|
||||
};
|
||||
let NativeExpr::Scalar(BoundScalar { value }) = &constant.expr else {
|
||||
// NotImplemented rather than Execution: this is a shape the runtime does not support, not a
|
||||
// tree whose values are wrong. A caller that falls back to another planner on NotImplemented
|
||||
// then still can, which an Execution would deny it.
|
||||
return Err(DataFusionError::NotImplemented(format!(
|
||||
"the native runtime does not execute the Binary node at {path}, which broadcasts an \
|
||||
operand that is not a literal scalar: the constant side has to be a Scalar node, and a \
|
||||
computed constant such as `1 < bool 2` is not one until it is folded"
|
||||
)));
|
||||
};
|
||||
// A constant has no labels, so the modifiers a vector–vector operation would use have no
|
||||
// meaning here: a tree that carries them describes another operation than the one it computes.
|
||||
if matching.labels != MatchingLabels::Ignoring(Vec::new())
|
||||
|| matching.cardinality != Cardinality::OneToOne
|
||||
|| matching.fill != FillValues::default()
|
||||
{
|
||||
return Err(matched_scalar(path));
|
||||
}
|
||||
Ok(BroadcastRuntime {
|
||||
operand: scalar_operand(*value),
|
||||
op: broadcast_op(path, matching)?,
|
||||
sample_schema: Arc::clone(vector),
|
||||
})
|
||||
}
|
||||
|
||||
/// The operation a binary node's `matching` describes.
|
||||
///
|
||||
/// A comparison is a filter unless it was bound as `bool`, and a filter keeps the label set it
|
||||
/// read while every other operation drops the metric name. Arithmetic has no `bool` form, so one
|
||||
/// bound that way describes a node the compiler does not produce and is refused.
|
||||
fn broadcast_op(path: &NodePath, matching: &BinaryMatching) -> Result<BroadcastOp> {
|
||||
let arithmetic = |op: ArithmeticOp| {
|
||||
if matching.return_bool {
|
||||
Err(DataFusionError::Execution(format!(
|
||||
"the binary node at {path} carries `bool` on an arithmetic operator, which has no \
|
||||
bool form"
|
||||
)))
|
||||
} else {
|
||||
Ok(BroadcastOp::Arithmetic(op))
|
||||
}
|
||||
};
|
||||
let comparison = |op: ComparisonOp| {
|
||||
if matching.return_bool {
|
||||
Ok(BroadcastOp::Compare(Comparison::Bool(op)))
|
||||
} else {
|
||||
Ok(BroadcastOp::Compare(Comparison::Filter(op)))
|
||||
}
|
||||
};
|
||||
match matching.op {
|
||||
BinaryOp::Add => arithmetic(ArithmeticOp::Add),
|
||||
BinaryOp::Sub => arithmetic(ArithmeticOp::Subtract),
|
||||
BinaryOp::Mul => arithmetic(ArithmeticOp::Multiply),
|
||||
BinaryOp::Div => arithmetic(ArithmeticOp::Divide),
|
||||
BinaryOp::Mod => arithmetic(ArithmeticOp::Modulo),
|
||||
BinaryOp::Pow => arithmetic(ArithmeticOp::Power),
|
||||
BinaryOp::Atan2 => arithmetic(ArithmeticOp::Atan2),
|
||||
BinaryOp::Eq => comparison(ComparisonOp::Equal),
|
||||
BinaryOp::Ne => comparison(ComparisonOp::NotEqual),
|
||||
BinaryOp::Gt => comparison(ComparisonOp::Greater),
|
||||
BinaryOp::Lt => comparison(ComparisonOp::Less),
|
||||
BinaryOp::Ge => comparison(ComparisonOp::GreaterEqual),
|
||||
BinaryOp::Le => comparison(ComparisonOp::LessEqual),
|
||||
}
|
||||
}
|
||||
|
||||
/// The error for a reservation the platform cannot express.
|
||||
///
|
||||
/// The binary node reserves its retained operands and the kernel's live workspace together, and
|
||||
|
||||
@@ -485,6 +485,7 @@ fn source_runtime(pool: &Arc<dyn MemoryPool>) -> NodeRuntime {
|
||||
window: None,
|
||||
map: None,
|
||||
binary: None,
|
||||
broadcast: None,
|
||||
vector: None,
|
||||
source_schema: None,
|
||||
emitted: None,
|
||||
@@ -506,6 +507,7 @@ fn binary_runtime(pool: &Arc<dyn MemoryPool>, schema: &Arc<SeriesSchema>) -> Nod
|
||||
Arc::new(binary_matching(BinaryOp::Add)),
|
||||
[Arc::clone(schema), Arc::clone(schema)],
|
||||
)),
|
||||
broadcast: None,
|
||||
vector: None,
|
||||
source_schema: None,
|
||||
emitted: None,
|
||||
@@ -1082,12 +1084,15 @@ fn a_downstream_window_emits_before_its_input_series_ends() {
|
||||
}
|
||||
|
||||
/// Every node kind the runtime does not execute is rejected, naming the kind and its position.
|
||||
///
|
||||
/// A scalar is not one of them any more: it is a value node, and a binary node over a constant is
|
||||
/// executed as a broadcast. The binary shapes that stay refused are the ones no kernel here takes,
|
||||
/// and the aggregate is still a kind this runtime does not run at all.
|
||||
#[test]
|
||||
fn an_unimplemented_node_is_reported_not_panicked() {
|
||||
// One grid shared by the nodes that validate by allocation identity.
|
||||
let statement = || grid(0, 120_000, 2);
|
||||
|
||||
let scalar_tree = scalar(1.0, statement());
|
||||
let aggregate_tree = {
|
||||
let out = statement();
|
||||
NativeNode {
|
||||
@@ -1102,18 +1107,30 @@ fn an_unimplemented_node_is_reported_not_panicked() {
|
||||
output: NativeType::Samples(evaluated_schema(out)),
|
||||
}
|
||||
};
|
||||
let binary_tree = {
|
||||
// Two constants make a scalar, which is a legal node and a shape no binary here broadcasts
|
||||
// over: there is no vector to pair either of them with.
|
||||
let two_scalars_tree = {
|
||||
let out = statement();
|
||||
NativeNode {
|
||||
expr: NativeExpr::Binary {
|
||||
left: Box::new(scalar(1.0, Arc::clone(&out))),
|
||||
right: Box::new(scalar(2.0, Arc::clone(&out))),
|
||||
matching: binary_matching(BinaryOp::Add),
|
||||
},
|
||||
output: NativeType::Scalar(out),
|
||||
}
|
||||
};
|
||||
// A constant carries no labels, so the matching modifiers of a vector–scalar operation have no
|
||||
// meaning: the operation this tree names is not the one it would compute.
|
||||
let matched_scalar_tree = {
|
||||
let out = statement();
|
||||
NativeNode {
|
||||
expr: NativeExpr::Binary {
|
||||
left: Box::new(window(input(), Arc::clone(&out))),
|
||||
right: Box::new(scalar(1.0, Arc::clone(&out))),
|
||||
matching: BinaryMatching {
|
||||
op: BinaryOp::Add,
|
||||
return_bool: false,
|
||||
labels: MatchingLabels::Ignoring(Vec::new()),
|
||||
cardinality: Cardinality::OneToOne,
|
||||
fill: FillValues::default(),
|
||||
labels: MatchingLabels::On(vec!["job".into()]),
|
||||
..binary_matching(BinaryOp::Mul)
|
||||
},
|
||||
},
|
||||
output: NativeType::Samples(evaluated_schema(out)),
|
||||
@@ -1121,11 +1138,16 @@ fn an_unimplemented_node_is_reported_not_panicked() {
|
||||
};
|
||||
|
||||
let cases = [
|
||||
("Scalar", "root", scalar_tree),
|
||||
("Aggregate", "root", aggregate_tree),
|
||||
("Binary", "root", binary_tree),
|
||||
(
|
||||
"Aggregate",
|
||||
"root",
|
||||
"does not execute Aggregate",
|
||||
aggregate_tree,
|
||||
),
|
||||
("Binary", "root", "two constants", two_scalars_tree),
|
||||
("Binary", "root", "matching modifiers", matched_scalar_tree),
|
||||
];
|
||||
for (kind, path, root) in cases {
|
||||
for (kind, path, reason, root) in cases {
|
||||
// Legality stays `validate`'s: the plan accepts these trees, the runtime does not run them.
|
||||
let tree = NativeTree::try_new(root).expect("the bound tree is legal");
|
||||
let error = match NativeExecutionState::try_new(
|
||||
@@ -1144,6 +1166,7 @@ fn an_unimplemented_node_is_reported_not_panicked() {
|
||||
let message = error.to_string();
|
||||
assert!(message.contains(kind), "{kind}: {message}");
|
||||
assert!(message.contains(&format!("at {path}")), "{message}");
|
||||
assert!(message.contains(reason), "{reason}: {message}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6425,6 +6448,804 @@ fn an_all_missing_match_group_fits_its_pool() {
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Scalar broadcasts, driven through `NativeExecutor`.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// A binary node over `left` and `right` where one side is a constant, bound the way the compiler
|
||||
/// binds a broadcast: the output is the vector side's fields on the statement's grid.
|
||||
fn broadcast_of(left: NativeNode, right: NativeNode, matching: BinaryMatching) -> NativeNode {
|
||||
let vector = left
|
||||
.output
|
||||
.samples()
|
||||
.or_else(|| right.output.samples())
|
||||
.expect("a broadcast reads one evaluated vector")
|
||||
.clone();
|
||||
NativeNode {
|
||||
expr: NativeExpr::Binary {
|
||||
left: Box::new(left),
|
||||
right: Box::new(right),
|
||||
matching,
|
||||
},
|
||||
output: NativeType::Samples(Arc::new(SeriesSchema {
|
||||
fields: vector.fields.clone(),
|
||||
eval_grid: vector.eval_grid.clone(),
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
/// `vector <op> constant`, on the grid the vector is evaluated on.
|
||||
fn broadcast_right(
|
||||
vector: NativeNode,
|
||||
constant: f64,
|
||||
op: BinaryOp,
|
||||
return_bool: bool,
|
||||
) -> NativeNode {
|
||||
let grid = vector
|
||||
.output
|
||||
.grid()
|
||||
.expect("a broadcast reads a grid")
|
||||
.clone();
|
||||
let matching = BinaryMatching {
|
||||
return_bool,
|
||||
..binary_matching(op)
|
||||
};
|
||||
broadcast_of(vector, scalar(constant, grid), matching)
|
||||
}
|
||||
|
||||
/// `constant <op> vector`, on the grid the vector is evaluated on.
|
||||
fn broadcast_left(
|
||||
constant: f64,
|
||||
vector: NativeNode,
|
||||
op: BinaryOp,
|
||||
return_bool: bool,
|
||||
) -> NativeNode {
|
||||
let grid = vector
|
||||
.output
|
||||
.grid()
|
||||
.expect("a broadcast reads a grid")
|
||||
.clone();
|
||||
let matching = BinaryMatching {
|
||||
return_bool,
|
||||
..binary_matching(op)
|
||||
};
|
||||
broadcast_of(scalar(constant, grid), vector, matching)
|
||||
}
|
||||
|
||||
/// The metric name a chunk carries, or `None` when the operation dropped it.
|
||||
fn name_of(chunk: &EvaluatedChunk) -> Option<&str> {
|
||||
chunk.series.metric.name()
|
||||
}
|
||||
|
||||
/// A constant on the right multiplies every value of the vector and drops the metric name.
|
||||
///
|
||||
/// The chunk arrives as it is pushed, before the input ends: a broadcast answers a chunk from that
|
||||
/// chunk alone, so nothing waits for the vector to be complete.
|
||||
#[test]
|
||||
fn a_constant_on_the_right_multiplies_every_value() {
|
||||
let grid = grid(0, 60_000, 4);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
100.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![1.0, 2.0, 3.0, 4.0], true))
|
||||
.unwrap();
|
||||
let streamed = execution
|
||||
.next_chunk()
|
||||
.unwrap()
|
||||
.expect("a broadcast answers the chunk it read without waiting for the end of the input");
|
||||
assert!(streamed.is_last, "the closing the chunk carried is kept");
|
||||
assert_eq!(
|
||||
sample_values(&streamed),
|
||||
vec![Some(100.0), Some(200.0), Some(300.0), Some(400.0)]
|
||||
);
|
||||
assert!(
|
||||
name_of(&streamed).is_none(),
|
||||
"arithmetic drops the metric name, and the chunk still carries it"
|
||||
);
|
||||
|
||||
let out = run(&mut execution);
|
||||
assert!(
|
||||
out.is_empty(),
|
||||
"the chunk was already handed over, so the end of the input owes nothing"
|
||||
);
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
}
|
||||
|
||||
/// The side a constant is on decides which operand the vector's value is.
|
||||
///
|
||||
/// Division is not commutative, so `x / 100` and `100 / x` answering different values is what says
|
||||
/// the operand direction survived the broadcast.
|
||||
#[test]
|
||||
fn a_broadcast_keeps_the_operand_direction() {
|
||||
let grid = grid(0, 60_000, 3);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let values = || vec![100.0, 200.0, 400.0];
|
||||
let run_one = |root: NativeNode| -> Vec<Option<f64>> {
|
||||
let tree = NativeTree::try_new(root).unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, values(), true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(out.len(), 1, "one chunk in, one chunk out");
|
||||
sample_values(&out[0])
|
||||
};
|
||||
assert_eq!(
|
||||
run_one(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
100.0,
|
||||
BinaryOp::Div,
|
||||
false,
|
||||
)),
|
||||
vec![Some(1.0), Some(2.0), Some(4.0)],
|
||||
"x / 100 divides the vector's value"
|
||||
);
|
||||
assert_eq!(
|
||||
run_one(broadcast_left(
|
||||
100.0,
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
BinaryOp::Div,
|
||||
false,
|
||||
)),
|
||||
vec![Some(1.0), Some(0.5), Some(0.25)],
|
||||
"100 / x divides the constant by the vector's value"
|
||||
);
|
||||
assert_eq!(
|
||||
run_one(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
100.0,
|
||||
BinaryOp::Sub,
|
||||
false,
|
||||
)),
|
||||
vec![Some(0.0), Some(100.0), Some(300.0)],
|
||||
"x - 100 subtracts from the vector's value"
|
||||
);
|
||||
assert_eq!(
|
||||
run_one(broadcast_left(
|
||||
100.0,
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
BinaryOp::Sub,
|
||||
false,
|
||||
)),
|
||||
vec![Some(0.0), Some(-100.0), Some(-300.0)],
|
||||
"100 - x subtracts the vector's value"
|
||||
);
|
||||
}
|
||||
|
||||
/// A `bool` comparison reports `1`/`0` at every position and drops the metric name.
|
||||
#[test]
|
||||
fn a_bool_comparison_reports_one_and_zero() {
|
||||
let grid = grid(0, 60_000, 3);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
0.0,
|
||||
BinaryOp::Gt,
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![1.0, -1.0, 0.0], true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
addressed(&out),
|
||||
vec![(0, vec![Some(1.0), Some(0.0), Some(0.0)])],
|
||||
"`x > bool 0` answers 1 or 0 at every position it read"
|
||||
);
|
||||
assert!(
|
||||
name_of(&out[0]).is_none(),
|
||||
"a bool comparison drops the metric name"
|
||||
);
|
||||
}
|
||||
|
||||
/// A filtering comparison keeps only the positions that passed, and keeps the label set it read.
|
||||
#[test]
|
||||
fn a_filtering_comparison_keeps_the_positions_that_passed() {
|
||||
let grid = grid(0, 60_000, 3);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
0.0,
|
||||
BinaryOp::Gt,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![1.0, -1.0, 0.0], true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
decided(&out),
|
||||
vec![(0, Some(1.0))],
|
||||
"only the position that passed is handed over"
|
||||
);
|
||||
assert_eq!(
|
||||
name_of(&out[0]),
|
||||
Some("requests"),
|
||||
"a filter keeps the label set it read"
|
||||
);
|
||||
}
|
||||
|
||||
/// `-1 * x` is the negation shape: a constant on the left, multiplying every value.
|
||||
#[test]
|
||||
fn a_constant_of_minus_one_on_the_left_negates_the_vector() {
|
||||
let grid = grid(0, 60_000, 3);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_left(
|
||||
-1.0,
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![1.0, -2.0, 5.0], true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
addressed(&out),
|
||||
vec![(0, vec![Some(-1.0), Some(2.0), Some(-5.0)])],
|
||||
"-1 * x negates every value of the vector"
|
||||
);
|
||||
}
|
||||
|
||||
/// A position the vector held no value at stays missing, and a position it skipped stays skipped.
|
||||
///
|
||||
/// The chunk that is not last is answered as it arrives: a broadcast transforms a chunk and hands
|
||||
/// it up, so `is_last` is the only thing that says where a series ends.
|
||||
#[test]
|
||||
fn a_broadcast_preserves_missing_values_and_skipped_positions() {
|
||||
let grid = grid(0, 60_000, 6);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
2.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
|
||||
// Position 0 holds a value and position 1 holds none: the axis covers both, so the difference
|
||||
// is the field's validity rather than the axis's rows.
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
metric(),
|
||||
0,
|
||||
vec![Some(3.0), None],
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let first = execution
|
||||
.next_chunk()
|
||||
.unwrap()
|
||||
.expect("a broadcast answers a chunk that does not close its series");
|
||||
assert!(!first.is_last, "the chunk that did not close is not closed");
|
||||
assert_eq!(
|
||||
decided(&[first]),
|
||||
vec![(0, Some(6.0)), (1, None)],
|
||||
"the positions are unchanged and the missing one stayed missing"
|
||||
);
|
||||
|
||||
// The last chunk sits at position 5: positions 2 to 4 are skipped, not empty values.
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
metric(),
|
||||
5,
|
||||
vec![Some(4.0)],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let second = execution
|
||||
.next_chunk()
|
||||
.unwrap()
|
||||
.expect("the closing chunk is answered as it arrives");
|
||||
assert!(second.is_last);
|
||||
assert_eq!(decided(&[second]), vec![(5, Some(8.0))]);
|
||||
assert!(
|
||||
execution.finish().unwrap().is_empty(),
|
||||
"both chunks were handed over as they arrived, so the end of the input owes nothing"
|
||||
);
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
}
|
||||
|
||||
/// An empty non-closing filter result does not end a broadcast before a later chunk passes.
|
||||
#[test]
|
||||
fn a_filter_empty_nonclosing_chunk_does_not_end_the_broadcast() {
|
||||
let grid = grid(0, 60_000, 5);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
0.0,
|
||||
BinaryOp::Gt,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![-1.0, -2.0], false))
|
||||
.unwrap();
|
||||
assert!(
|
||||
execution.next_chunk().unwrap().is_none(),
|
||||
"a non-closing chunk with no passing positions emits no output but does not end the series"
|
||||
);
|
||||
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 2, vec![3.0, -4.0, 5.0], true))
|
||||
.unwrap();
|
||||
let passing = execution
|
||||
.next_chunk()
|
||||
.unwrap()
|
||||
.expect("the broadcast remains live and hands over a later passing chunk");
|
||||
assert!(
|
||||
passing.is_last,
|
||||
"only the later input chunk closes the series"
|
||||
);
|
||||
assert_eq!(
|
||||
decided(std::slice::from_ref(&passing)),
|
||||
vec![(2, Some(3.0)), (4, Some(5.0))]
|
||||
);
|
||||
assert_eq!(
|
||||
name_of(&passing),
|
||||
Some("requests"),
|
||||
"the filtering comparison preserves the input labels"
|
||||
);
|
||||
assert!(
|
||||
execution.finish().unwrap().is_empty(),
|
||||
"the sole closing chunk was handed over exactly once"
|
||||
);
|
||||
}
|
||||
|
||||
/// A filter that passes nothing on the chunk that closes a series still closes it.
|
||||
/// The chunk it hands over holds no row and carries the label set the operation produces, which is
|
||||
/// the only way the consumer downstream learns that the series ended rather than that no input has
|
||||
/// arrived yet.
|
||||
#[test]
|
||||
fn a_filter_that_passes_nothing_still_closes_the_series() {
|
||||
let grid = grid(0, 60_000, 3);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
|
||||
// A filtering comparison keeps the label set it read, so the closing chunk carries it.
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
0.0,
|
||||
BinaryOp::Gt,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![-1.0, -2.0], false))
|
||||
.unwrap();
|
||||
assert!(
|
||||
execution.next_chunk().unwrap().is_none(),
|
||||
"a chunk that passed nothing and closed nothing has nothing to hand over"
|
||||
);
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 2, vec![-3.0], true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
out.len(),
|
||||
1,
|
||||
"the closing chunk is handed over even though it holds no value"
|
||||
);
|
||||
assert!(out[0].is_last, "the empty chunk closes the series");
|
||||
assert_eq!(
|
||||
out[0].series.time.len(),
|
||||
0,
|
||||
"a filter that passed nothing hands over no row"
|
||||
);
|
||||
assert_eq!(
|
||||
name_of(&out[0]),
|
||||
Some("requests"),
|
||||
"a filter keeps the labels it read, so the closing chunk carries them"
|
||||
);
|
||||
assert_eq!(
|
||||
series_start(&out[0]),
|
||||
3,
|
||||
"the empty chunk closes at the end"
|
||||
);
|
||||
|
||||
// A `bool` comparison drops the name, so its empty closing chunk carries the dropped one.
|
||||
let bool_tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
0.0,
|
||||
BinaryOp::Gt,
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let mut execution = executor(&bool_tree, pool(), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
metric(),
|
||||
0,
|
||||
vec![None; 3],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
out.len(),
|
||||
1,
|
||||
"the closing chunk of an all-missing input arrives"
|
||||
);
|
||||
assert!(out[0].is_last);
|
||||
assert_eq!(out[0].series.time.len(), 0);
|
||||
assert!(
|
||||
name_of(&out[0]).is_none(),
|
||||
"a bool comparison drops the metric name, and the empty closing chunk carries that"
|
||||
);
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
}
|
||||
|
||||
/// A broadcast holds the chunk it is working on and nothing more, however long the vector is.
|
||||
///
|
||||
/// The same series arrives in one chunk and in many: what the node reserves has to be the chunk in
|
||||
/// hand and the label sets it handed out, not the vector it read, or a broadcast over a long series
|
||||
/// would cost the whole series.
|
||||
#[test]
|
||||
fn a_broadcast_does_not_retain_the_vector_it_reads() {
|
||||
// The most the node ever reserved, over the same series delivered in `count` chunks.
|
||||
let peak =
|
||||
|count: usize| -> usize {
|
||||
let grid = grid(0, 60_000, count as u64);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
2.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let pool = bounded_pool(32 << 10);
|
||||
let mut execution = executor(&tree, Arc::clone(&pool), &ExecutionPlanMetricsSet::new());
|
||||
let mut peak = 0;
|
||||
for index in 0..count {
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(
|
||||
&schema,
|
||||
index,
|
||||
vec![index as f64],
|
||||
index + 1 == count,
|
||||
))
|
||||
.unwrap_or_else(|error| {
|
||||
panic!("chunk {index} of {count} did not fit: {error}");
|
||||
});
|
||||
// The consumer drops every chunk: a reader that streams the vector never holds it, and
|
||||
// neither may the node.
|
||||
drop(execution.next_chunk().unwrap_or_else(|error| {
|
||||
panic!("chunk {index} of {count} did not fit: {error}")
|
||||
}));
|
||||
let held = execution.state().reserved_bytes();
|
||||
assert_eq!(
|
||||
held,
|
||||
pool.reserved(),
|
||||
"chunk {index} of {count}: the state and the pool disagree on what is held"
|
||||
);
|
||||
assert!(
|
||||
held <= (8 << 10),
|
||||
"chunk {index} of {count} holds {held} bytes of one float and one label set"
|
||||
);
|
||||
peak = peak.max(held);
|
||||
}
|
||||
// The series ends, and the node holds nothing afterwards.
|
||||
drop(run(&mut execution));
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
assert_eq!(pool.reserved(), 0);
|
||||
peak
|
||||
};
|
||||
assert_eq!(
|
||||
peak(64),
|
||||
peak(1024),
|
||||
"the reservation grew with the number of chunks the same vector arrived in"
|
||||
);
|
||||
}
|
||||
|
||||
/// A window above a broadcast that drops the metric name merges what that broadcast collapses.
|
||||
///
|
||||
/// The multiplication rewrites the label set, so two input series with different names and one set
|
||||
/// of tags become one label set under it. A window above therefore has to collect its input and
|
||||
/// merge the pieces before it evaluates: reporting each piece on its own answers two series where
|
||||
/// Prometheus answers one, and the second piece's positions are exactly the ones the first left
|
||||
/// empty. `holds_shared` is what tells the window to collect, so a window that streams here is the
|
||||
/// regression this test exists for.
|
||||
#[test]
|
||||
fn a_window_above_a_broadcast_merges_the_series_it_collapses() {
|
||||
let grid = grid(0, 60_000, 4);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(window_with(
|
||||
broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
2.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
),
|
||||
RangeFunction::LastOverTime,
|
||||
60_000,
|
||||
Some(Arc::clone(&grid)),
|
||||
Arc::clone(&grid),
|
||||
))
|
||||
.unwrap();
|
||||
let set = ExecutionPlanMetricsSet::new();
|
||||
let mut execution = executor(&tree, pool(), &set);
|
||||
// Two series that only differ by name, each holding the half of the grid the other leaves empty.
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
named_metric("requests", "x"),
|
||||
0,
|
||||
vec![Some(1.0), Some(2.0)],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
named_metric("requests_total", "x"),
|
||||
2,
|
||||
vec![Some(3.0), Some(4.0)],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
decided(&out),
|
||||
vec![
|
||||
(0, Some(2.0)),
|
||||
(1, Some(4.0)),
|
||||
(2, Some(6.0)),
|
||||
(3, Some(8.0))
|
||||
],
|
||||
"the window did not merge the two pieces the broadcast collapsed onto one label set"
|
||||
);
|
||||
assert_eq!(
|
||||
out.iter().filter(|chunk| chunk.is_last).count(),
|
||||
1,
|
||||
"the merged series closed more or less than once"
|
||||
);
|
||||
// The node counts the source series it collected, not the merged one: two fragments arrived
|
||||
// with `is_last`, so two source series ended. What proves the merge happened is the single
|
||||
// closing chunk above -- a window that streamed would have closed each piece on its own.
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root(), NodeMetric::COMPLETED_SERIES),
|
||||
2,
|
||||
"the window did not see both source series end"
|
||||
);
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
}
|
||||
|
||||
/// A broadcast hands one chunk over for every chunk it reads, and closes the series once.
|
||||
///
|
||||
/// The metrics are what says the chunk-per-chunk behaviour held: one input batch per chunk read, one
|
||||
/// output batch per chunk handed over, and one closed series for the series that ended.
|
||||
#[test]
|
||||
fn a_broadcast_reports_one_chunk_in_for_each_chunk_out() {
|
||||
let grid = grid(0, 60_000, 4);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
2.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let set = ExecutionPlanMetricsSet::new();
|
||||
let mut execution = executor(&tree, pool(), &set);
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 0, vec![1.0, 2.0], false))
|
||||
.unwrap();
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&schema, 2, vec![3.0, 4.0], true))
|
||||
.unwrap();
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(
|
||||
addressed(&out),
|
||||
vec![
|
||||
(0, vec![Some(2.0), Some(4.0)]),
|
||||
(2, vec![Some(6.0), Some(8.0)])
|
||||
],
|
||||
"each chunk is answered as it arrived, at the positions it held"
|
||||
);
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root(), NodeMetric::INPUT_BATCHES),
|
||||
2,
|
||||
"the node read two chunks"
|
||||
);
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root(), NodeMetric::OUTPUT_BATCHES),
|
||||
2,
|
||||
"the node handed two chunks over"
|
||||
);
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root(), NodeMetric::COMPLETED_SERIES),
|
||||
1,
|
||||
"the series closed once, on the chunk that said so"
|
||||
);
|
||||
// The scalar side is a node of the tree and reads nothing: it has no input at all.
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root().right(), NodeMetric::INPUT_BATCHES),
|
||||
0,
|
||||
"the constant was pulled as a stream"
|
||||
);
|
||||
}
|
||||
|
||||
/// A broadcast forwards the settled pieces of a streaming window and its eventual close.
|
||||
#[test]
|
||||
fn a_broadcast_over_a_streaming_window_hands_over_the_points_it_settles() {
|
||||
let inner_grid = grid(0, 60_000, 5);
|
||||
let outer_grid = grid(0, 120_000, 3);
|
||||
let inner_schema = evaluated_schema(Arc::clone(&inner_grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
window_with(
|
||||
evaluated_input(Arc::clone(&inner_grid)),
|
||||
RangeFunction::LastOverTime,
|
||||
240_000,
|
||||
Some(Arc::clone(&inner_grid)),
|
||||
outer_grid,
|
||||
),
|
||||
100.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let set = ExecutionPlanMetricsSet::new();
|
||||
let pool = pool();
|
||||
let mut execution = executor(&tree, Arc::clone(&pool), &set);
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(
|
||||
&inner_schema,
|
||||
0,
|
||||
vec![1.0, 2.0, 3.0],
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
|
||||
let mut out = Vec::new();
|
||||
while let Some(chunk) = execution.next_chunk().unwrap() {
|
||||
out.push(chunk);
|
||||
}
|
||||
assert_eq!(decided(&out), vec![(0, Some(100.0)), (1, Some(300.0))]);
|
||||
assert!(out.iter().all(|chunk| !chunk.is_last));
|
||||
assert_eq!(
|
||||
count(&set, &NodePath::root(), NodeMetric::COMPLETED_SERIES),
|
||||
0
|
||||
);
|
||||
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk(&inner_schema, 3, vec![4.0, 5.0], true))
|
||||
.unwrap();
|
||||
while let Some(chunk) = execution.next_chunk().unwrap() {
|
||||
out.push(chunk);
|
||||
}
|
||||
out.extend(execution.finish().unwrap());
|
||||
assert_eq!(
|
||||
decided(&out),
|
||||
vec![(0, Some(100.0)), (1, Some(300.0)), (2, Some(500.0))]
|
||||
);
|
||||
assert_eq!(out.iter().filter(|chunk| chunk.is_last).count(), 1);
|
||||
assert!(out.last().unwrap().is_last);
|
||||
assert!(out.iter().all(|chunk| name_of(chunk).is_none()));
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
assert_eq!(pool.reserved(), 0);
|
||||
}
|
||||
|
||||
/// A broadcast waits for a collecting window to flush before answering its series.
|
||||
#[test]
|
||||
fn a_broadcast_above_a_collecting_window_waits_for_its_flush() {
|
||||
let grid = grid(0, 60_000, 1);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
window_with(
|
||||
map(evaluated_input(Arc::clone(&grid)), vec![MathOp::Abs]),
|
||||
RangeFunction::LastOverTime,
|
||||
60_000,
|
||||
Some(Arc::clone(&grid)),
|
||||
Arc::clone(&grid),
|
||||
),
|
||||
10.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let pool = pool();
|
||||
let mut execution = executor(&tree, Arc::clone(&pool), &ExecutionPlanMetricsSet::new());
|
||||
execution
|
||||
.push_evaluated_at(
|
||||
&NodePath::root().left().input().input(),
|
||||
evaluated_chunk_of(&schema, metric(), 0, vec![Some(-1.0)], true),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(execution.next_chunk().unwrap().is_none());
|
||||
let out = run(&mut execution);
|
||||
assert_eq!(decided(&out), vec![(0, Some(10.0))]);
|
||||
assert_eq!(out.iter().filter(|chunk| chunk.is_last).count(), 1);
|
||||
assert_eq!(execution.state().reserved_bytes(), 0);
|
||||
assert_eq!(pool.reserved(), 0);
|
||||
}
|
||||
|
||||
/// A broadcast rejects distinct input names that become one output label set at occupied points.
|
||||
#[test]
|
||||
fn a_broadcast_refuses_two_series_that_share_a_label_set_and_releases_the_reservation() {
|
||||
let grid = grid(0, 60_000, 2);
|
||||
let schema = evaluated_schema(Arc::clone(&grid));
|
||||
let tree = NativeTree::try_new(broadcast_right(
|
||||
evaluated_input(Arc::clone(&grid)),
|
||||
100.0,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
.unwrap();
|
||||
let pool = pool();
|
||||
let mut execution = executor(&tree, Arc::clone(&pool), &ExecutionPlanMetricsSet::new());
|
||||
let first_metric =
|
||||
MetricName::try_new(Some("requests".to_string().into()), Vec::new()).unwrap();
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
first_metric,
|
||||
0,
|
||||
vec![Some(1.0), Some(2.0)],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let first = execution
|
||||
.next_chunk()
|
||||
.unwrap()
|
||||
.expect("first series output");
|
||||
assert_eq!(
|
||||
decided(std::slice::from_ref(&first)),
|
||||
vec![(0, Some(100.0)), (1, Some(200.0))]
|
||||
);
|
||||
|
||||
let second_metric =
|
||||
MetricName::try_new(Some("requests_total".to_string().into()), Vec::new()).unwrap();
|
||||
execution
|
||||
.push_evaluated(evaluated_chunk_of(
|
||||
&schema,
|
||||
second_metric,
|
||||
0,
|
||||
vec![Some(3.0), Some(4.0)],
|
||||
true,
|
||||
))
|
||||
.unwrap();
|
||||
let error = execution
|
||||
.next_chunk()
|
||||
.expect_err("the output label set and occupied positions conflict");
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains("that both hold a value of field"),
|
||||
"expected the collision the node produces, got: {message:?}"
|
||||
);
|
||||
assert_eq!(
|
||||
pool.reserved(),
|
||||
0,
|
||||
"error retained a pool reservation: {message}"
|
||||
);
|
||||
assert_eq!(
|
||||
execution.state().reserved_bytes(),
|
||||
0,
|
||||
"error retained execution bytes: {message}"
|
||||
);
|
||||
}
|
||||
|
||||
fn vector_op(inner: NativeNode, op: BoundVectorOp) -> NativeNode {
|
||||
let output = inner.output.clone();
|
||||
NativeNode {
|
||||
|
||||
@@ -180,10 +180,43 @@ pub struct BoundMap {
|
||||
}
|
||||
|
||||
/// Which aggregate a node computes.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
///
|
||||
/// Every kind here folds a group's values into one value per grid point. The aggregates that
|
||||
/// instead select some of the input's series (`topk`, `bottomk`, `limitk`, `limit_ratio`) or write
|
||||
/// one output label per distinct value (`count_values`) are a different shape and have no variant,
|
||||
/// so a node cannot claim to compute one of them.
|
||||
///
|
||||
/// A kind may be bound before its kernel exists: whether the tree can run is
|
||||
/// [`NativeTree::try_new`]'s and the runtime's question, not this enum's.
|
||||
///
|
||||
/// `Eq` is missing because [`AggregateKind::Quantile`] carries an `f64`, as
|
||||
/// [`RangeFunction::QuantileOverTime`] does.
|
||||
///
|
||||
/// [`RangeFunction::QuantileOverTime`]: crate::range_eval::RangeFunction::QuantileOverTime
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum AggregateKind {
|
||||
/// `sum` over the values of each group at each grid point.
|
||||
/// `sum`: the total of each group's values at each grid point.
|
||||
Sum,
|
||||
/// `avg`: the mean of each group's values at each grid point.
|
||||
Avg,
|
||||
/// `count`: how many values each group has at each grid point.
|
||||
Count,
|
||||
/// `min`: the smallest of each group's values at each grid point.
|
||||
Min,
|
||||
/// `max`: the largest of each group's values at each grid point.
|
||||
Max,
|
||||
/// `group`: `1` where a group has any value at a grid point.
|
||||
Group,
|
||||
/// `stddev`: the standard deviation of each group's values at each grid point.
|
||||
Stddev,
|
||||
/// `stdvar`: the variance of each group's values at each grid point.
|
||||
Stdvar,
|
||||
/// `quantile`: the φ-quantile of each group's values at each grid point, as written.
|
||||
Quantile {
|
||||
/// The φ the query wrote, which `0` to `1` does not bound: Prometheus reads a negative φ as
|
||||
/// `-Infinity` and one above `1` as `+Infinity`.
|
||||
quantile: f64,
|
||||
},
|
||||
}
|
||||
|
||||
/// Where an aggregate sits in the execution, which decides whether its output is a query value.
|
||||
@@ -389,11 +422,11 @@ impl NativeNode {
|
||||
}
|
||||
ensure_not_interleaved(input)?;
|
||||
// What is checked here and just below is that the declared input grid equals the
|
||||
// input's grid, and that the output grid matches the declared one. The grid is
|
||||
// not checked to cover the window's read range
|
||||
// `(out.start - offset - range, out.end - offset]`: only a subquery feeds an
|
||||
// evaluated input to a window, and this compiler does not accept subqueries yet,
|
||||
// so that coverage check waits for it.
|
||||
// input's grid, and that the output grid matches the declared one. That the grid
|
||||
// covers the window's read range
|
||||
// `(out.start - offset - range, out.end - offset]` is not checked here: only a
|
||||
// subquery feeds an evaluated input to a window, and deriving that coverage is the
|
||||
// compiler's, which builds the subquery's child grid to span it.
|
||||
match (op.input_grid.as_ref(), schema.eval_grid.as_ref()) {
|
||||
(None, None) => {}
|
||||
(Some(bound), Some(actual)) if bound.spec() == actual.spec() => {}
|
||||
|
||||
@@ -714,7 +714,7 @@ impl RangeEvaluator {
|
||||
}
|
||||
// The input may skip positions: it is ingested by the positions it carries, and the points
|
||||
// it skips hold no sample, which is exactly what a hole means.
|
||||
let timestamps = time.timestamps();
|
||||
let timestamps = time.timestamps().map_err(external)?;
|
||||
let last_ts = timestamps.last().copied();
|
||||
if !last && last_ts.is_none() {
|
||||
// An empty delivery inside a series decides nothing: it is indistinguishable from a
|
||||
@@ -1045,7 +1045,7 @@ impl RangeEvaluator {
|
||||
let points_read = self.points_read();
|
||||
let state = self.series.as_mut().expect("checked by check_chunk");
|
||||
let spec = self.grid.spec();
|
||||
let grid = self.grid.timestamps();
|
||||
let grid = self.grid.timestamps().map_err(external)?;
|
||||
let range = i128::from(self.window.range_ms);
|
||||
let mut empty_tail = false;
|
||||
while state.cursor < points_read {
|
||||
@@ -1212,7 +1212,7 @@ impl RangeEvaluator {
|
||||
/// only the first.
|
||||
fn points_read(&self) -> usize {
|
||||
match self.replay {
|
||||
Replay::PerPoint => self.grid.timestamps().len(),
|
||||
Replay::PerPoint => self.grid.spec().len as usize,
|
||||
Replay::Once | Replay::EveryPoint => 1,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -279,7 +279,7 @@ fn fold_chunks(chunks: &[EvaluatedChunk], schema: &Arc<SeriesSchema>) -> Option<
|
||||
.metric
|
||||
.clone();
|
||||
let width = chunks[0].series.fields.len();
|
||||
let grid_len = schema.eval_grid.as_ref().unwrap().timestamps().len();
|
||||
let grid_len = schema.eval_grid.as_ref().unwrap().spec().len as usize;
|
||||
let mut rows: Vec<Option<Vec<Option<GridSample>>>> = (0..grid_len).map(|_| None).collect();
|
||||
let mut closing = false;
|
||||
let mut previous_end = 0;
|
||||
@@ -495,6 +495,7 @@ fn window_end(grid: &EvalGrid, window: RangeWindow, eval_ts: i64) -> i64 {
|
||||
/// Rows of `rows`, already on the evaluation timeline, inside each grid point's window.
|
||||
fn windows_of(rows: &[Row], grid: &EvalGrid, window: RangeWindow) -> Vec<Range<usize>> {
|
||||
grid.timestamps()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|&eval_ts| {
|
||||
let end = window_end(grid, window, eval_ts);
|
||||
@@ -520,6 +521,7 @@ fn reference(
|
||||
) -> Dense {
|
||||
let (kept, windows) = reference_windows(rows, grid, window, drop_stale);
|
||||
grid.timestamps()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.zip(windows)
|
||||
.map(|(&eval_ts, rows)| {
|
||||
@@ -2610,7 +2612,7 @@ fn random_evaluated(
|
||||
fields: usize,
|
||||
host: u64,
|
||||
) -> (Option<Series>, Vec<Row>) {
|
||||
let mut dense = random_dense(rng, grid.timestamps().len(), fields);
|
||||
let mut dense = random_dense(rng, grid.spec().len as usize, fields);
|
||||
for value in dense.iter_mut().flatten().flatten() {
|
||||
if rng.random_bool(0.05) {
|
||||
*value = stale();
|
||||
@@ -2626,6 +2628,7 @@ fn random_evaluated(
|
||||
}
|
||||
let rows = grid
|
||||
.timestamps()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.zip(&dense)
|
||||
.filter(|(_, row)| row.iter().any(Option::is_some))
|
||||
@@ -2750,7 +2753,7 @@ fn subquery_grid_covers_every_step_multiple_in_the_windows() {
|
||||
let fold = fold_offset(&outer, window);
|
||||
let last_eval = match window.anchor_ms {
|
||||
Some(_) => spec.start_ms,
|
||||
None => *outer.timestamps().last().unwrap(),
|
||||
None => *outer.timestamps().unwrap().last().unwrap(),
|
||||
};
|
||||
let lower = spec.start_ms - fold - window.range_ms;
|
||||
let expected: Vec<i64> = ((lower.div_euclid(step) - 1) * step..=last_eval - fold)
|
||||
@@ -2759,7 +2762,13 @@ fn subquery_grid_covers_every_step_multiple_in_the_windows() {
|
||||
.collect();
|
||||
let actual = subquery_grid(spec, window, step)
|
||||
.unwrap()
|
||||
.map(|grid| EvalGrid::try_new(grid).unwrap().timestamps().to_vec())
|
||||
.map(|grid| {
|
||||
EvalGrid::try_new(grid)
|
||||
.unwrap()
|
||||
.timestamps()
|
||||
.unwrap()
|
||||
.to_vec()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
assert_eq!(actual, expected, "seed {seed}");
|
||||
}
|
||||
@@ -3764,17 +3773,12 @@ fn a_million_position_grid_with_two_values_is_one_series_of_two_rows() {
|
||||
|
||||
// What the axis does carry is the shared grid: every result of the query addresses rows on it.
|
||||
assert_eq!(
|
||||
series
|
||||
.time
|
||||
.grid()
|
||||
.expect("a grid series")
|
||||
.timestamps()
|
||||
.len(),
|
||||
series.time.grid().expect("a grid series").spec().len as usize,
|
||||
SPAN as usize,
|
||||
"the grid really holds a million positions"
|
||||
);
|
||||
assert_eq!(
|
||||
series.time.timestamps().len(),
|
||||
series.time.timestamps().unwrap().len(),
|
||||
2,
|
||||
"the axis hands out two timestamps, not one per grid position"
|
||||
);
|
||||
@@ -4148,6 +4152,7 @@ fn a_prefix_hands_out_its_rows_before_the_series_ends() {
|
||||
fn dense_sum_over_time(rows: &[Row], grid: &EvalGrid, window: RangeWindow) -> Dense {
|
||||
assert_eq!(window.offset_ms, 0, "this reference folds no offset");
|
||||
grid.timestamps()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|&eval_ts| {
|
||||
let inside: Vec<f64> = rows
|
||||
@@ -4462,7 +4467,7 @@ fn two_composed_evaluators_agree_with_the_dense_reference() {
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter_map(|(index, row)| {
|
||||
row[0].map(|value| (inner.timestamps()[index], vec![Some(value)]))
|
||||
row[0].map(|value| (inner.timestamps().unwrap()[index], vec![Some(value)]))
|
||||
})
|
||||
.collect();
|
||||
let expected = dense_sum_over_time(&stage_one_rows, &outer, outer_window);
|
||||
|
||||
@@ -646,7 +646,7 @@ fn grid_points(schema: &SeriesSchema, series: &[Series]) -> Result<usize> {
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(grid.timestamps().len())
|
||||
Ok(grid.spec().len as usize)
|
||||
}
|
||||
|
||||
/// The first field's value of `series` at grid `point`, looked up by position.
|
||||
|
||||
@@ -310,7 +310,7 @@ fn sort_drops_histogram_samples() {
|
||||
)))],
|
||||
)
|
||||
.unwrap();
|
||||
if schema.eval_grid.as_ref().unwrap().timestamps().len() > 1 {
|
||||
if schema.eval_grid.as_ref().unwrap().spec().len > 1 {
|
||||
builder
|
||||
.push_samples(1, &[Some(GridSample::Float(2.0))])
|
||||
.unwrap();
|
||||
|
||||
@@ -720,7 +720,10 @@ fn binary_field(
|
||||
Cardinality::ManyToOne(include) => (lhs, rhs, &include[..], false),
|
||||
Cardinality::OneToMany(include) => (rhs, lhs, &include[..], true),
|
||||
};
|
||||
let points = grid.timestamps().len();
|
||||
let points = grid.spec().len as usize;
|
||||
let timestamp_at = |point: usize| {
|
||||
(grid.spec().start_ms as i128 + point as i128 * grid.spec().step_ms as i128) as i64
|
||||
};
|
||||
let errors_before = errors.len();
|
||||
// The pairing phase is entered once, before the field allocates anything: its presence flags,
|
||||
// its match groups, the widest label set a pairing builds and the histogram payload one row of
|
||||
@@ -948,7 +951,7 @@ fn binary_field(
|
||||
operation at {} ms; many-to-many matching not allowed: matching labels must be \
|
||||
unique on one side",
|
||||
render(key),
|
||||
grid.timestamps()[point]
|
||||
timestamp_at(point)
|
||||
),
|
||||
));
|
||||
}
|
||||
@@ -961,7 +964,7 @@ fn binary_field(
|
||||
"multiple matches for labels {} at {} ms: many-to-one matching must be explicit \
|
||||
(group_left/group_right)",
|
||||
render(key),
|
||||
grid.timestamps()[point]
|
||||
timestamp_at(point)
|
||||
),
|
||||
));
|
||||
}
|
||||
@@ -975,14 +978,14 @@ fn binary_field(
|
||||
"multiple matches for labels {} at {} ms: grouping labels must ensure \
|
||||
unique matches",
|
||||
render(&duplicate.key),
|
||||
grid.timestamps()[duplicate.point]
|
||||
timestamp_at(duplicate.point)
|
||||
),
|
||||
)
|
||||
} else {
|
||||
(
|
||||
duplicate.point,
|
||||
2,
|
||||
same_labelset(&duplicate.key, grid.timestamps()[duplicate.point]),
|
||||
same_labelset(&duplicate.key, timestamp_at(duplicate.point)),
|
||||
)
|
||||
});
|
||||
}
|
||||
@@ -1151,7 +1154,9 @@ pub fn set_vectors(
|
||||
let rhs = Operand::try_new(rhs_schema, rhs, field)?;
|
||||
let mut results = set_field(op, labels, &lhs, &rhs)?;
|
||||
if let Some(duplicate) = results.sort_rows() {
|
||||
let timestamp = grid.timestamps()[duplicate.point];
|
||||
let timestamp = (grid.spec().start_ms as i128
|
||||
+ duplicate.point as i128 * grid.spec().step_ms as i128)
|
||||
as i64;
|
||||
errors.push((duplicate.point, 0, same_labelset(&duplicate.key, timestamp)));
|
||||
}
|
||||
fields.push(results);
|
||||
|
||||
@@ -130,7 +130,7 @@ fn random_side(rng: &mut StdRng, grid: &Arc<EvalGrid>, fields: usize) -> Side {
|
||||
}
|
||||
// Each label set starts its own series: only `begin_series` resets the builder.
|
||||
builder.begin_series();
|
||||
let values = random_dense(rng, grid.timestamps().len(), fields);
|
||||
let values = random_dense(rng, grid.spec().len as usize, fields);
|
||||
for (index, row) in values.iter().enumerate() {
|
||||
builder.push(index, row).unwrap();
|
||||
}
|
||||
@@ -489,7 +489,7 @@ fn binary_vectors_match_prometheus_reference() {
|
||||
fill: FillValues::default(),
|
||||
};
|
||||
let fields = lhs_fields.min(rhs_fields);
|
||||
let points = grid.timestamps().len();
|
||||
let points = grid.spec().len as usize;
|
||||
let expected = per_field(fields, points, |field| {
|
||||
reference_binary(
|
||||
&matching,
|
||||
@@ -561,7 +561,7 @@ fn set_vectors_match_prometheus_reference() {
|
||||
let rhs = random_side(&mut rng, &grid, fields);
|
||||
let op = [SetOp::And, SetOp::Or, SetOp::Unless][rng.random_range(0..3)];
|
||||
let labels = random_labels(&mut rng);
|
||||
let points = grid.timestamps().len();
|
||||
let points = grid.spec().len as usize;
|
||||
let expected = per_field(fields, points, |field| {
|
||||
reference_set(
|
||||
op,
|
||||
|
||||
@@ -19,24 +19,66 @@
|
||||
//! accepts is one recursive rule:
|
||||
//!
|
||||
//! ```text
|
||||
//! <selector> e.g. x
|
||||
//! <range function>(<selector>[<range>]) e.g. rate(x[5m])
|
||||
//! <range function>(<constant>, <selector>[<range>]) e.g. quantile_over_time(0.9, x[5m])
|
||||
//! <math function>(<expression>[, <constant>]*) e.g. abs(rate(x[5m])), clamp_min(x, 0)
|
||||
//! <sort>(<expression>[, <string>]*) e.g. sort_desc(x), sort_by_label(x, "job")
|
||||
//! absent(<expression>) e.g. absent(x{job="a"})
|
||||
//! <histogram>(<constant>+, <expression>) e.g. histogram_quantile(0.9, rate(x[5m]))
|
||||
//! <expression> one of, recursively:
|
||||
//! <selector> e.g. x, x offset -5m
|
||||
//! <range function>(<matrix>) e.g. rate(x[5m]), rate(x[5m:1m] @ 1234)
|
||||
//! quantile_over_time(<constant>, <matrix>) e.g. quantile_over_time(0.9, x[5m])
|
||||
//! predict_linear(<matrix>, <constant>) e.g. predict_linear(x[5m], 300)
|
||||
//! double_exponential_smoothing(<matrix>, <constant>, <constant>)
|
||||
//! e.g. double_exponential_smoothing(x[5m], .5, .1)
|
||||
//! <math function>(<expression>[, <constant>]*) e.g. abs(rate(x[5m])), clamp_min(x, 0)
|
||||
//! <sort>(<expression>[, <string>]*) e.g. sort_desc(x), sort_by_label(x, "job")
|
||||
//! absent(<expression>) e.g. absent(x{job="a"})
|
||||
//! <histogram>(<constant>+, <expression>) e.g. histogram_quantile(0.9, rate(x[5m]))
|
||||
//! <expression> <operator> <constant> e.g. x * 100, x > bool 0
|
||||
//! <constant> <operator> <expression> e.g. 1 / x, 0 < x
|
||||
//! -<expression> e.g. -x
|
||||
//! <aggregator> [(by|without)(<labels>)] (<expression>)
|
||||
//! e.g. avg by(job)(rate(x[5m]) * 100)
|
||||
//! <aggregator> (<expression>) [(by|without)(<labels>)]
|
||||
//! e.g. count (x) by(job)
|
||||
//! quantile(<constant>, <expression>) [(by|without)(<labels>)]
|
||||
//! e.g. quantile(0.9, x) by(job)
|
||||
//!
|
||||
//! <matrix> <selector>[<range>]<modifiers> a range selector: raw samples
|
||||
//! | <expression>[<range>:<step>]<modifiers>
|
||||
//! a subquery: <expression> evaluated on a grid;
|
||||
//! the step may be omitted as `[<range>:]`
|
||||
//! <modifiers> nothing, `offset <duration>`, `@ <anchor>`, or
|
||||
//! both, in either order — e.g. `x[5m] @ 1234
|
||||
//! offset 1m`
|
||||
//! ```
|
||||
//!
|
||||
//! which become `Input` (raw samples) → `Instant` (the last sample within the statement's lookback),
|
||||
//! `Input` (raw samples) → `Window` (the range function over the statement's grid) and
|
||||
//! `<expression>` → `Map` (a pointwise math function over what its argument evaluates to), and
|
||||
//! `<expression>` → `Vector` (a sort, `absent` or a classic histogram function, which read the
|
||||
//! whole vector). A
|
||||
//! selector may carry `offset` in either selection shape and, in the window shape, an `@` anchor.
|
||||
//! Everything else is a [`DataFusionError::NotImplemented`] that names what was found and quotes the
|
||||
//! shapes above, because rejecting a query the compiler does not understand is the only honest
|
||||
//! alternative to compiling it into something else.
|
||||
//! where `<aggregator>` is `sum`, `avg`, `count`, `min`, `max`, `group`, `stddev`, `stdvar` or
|
||||
//! `quantile`, the function names are [`range_signature`]'s and [`math_signature`]'s, and which side
|
||||
//! of the matrix a range function's constants go on is [`RangeSignature`]'s: before it for
|
||||
//! `quantile_over_time`, after it for `predict_linear` and `double_exponential_smoothing`. A math
|
||||
//! function's constants always follow its vector.
|
||||
//!
|
||||
//! Parentheses are transparent wherever they appear — `rate((x[5m]))` is `rate(x[5m])` — because
|
||||
//! they say how a statement was typed rather than what it asks for.
|
||||
//!
|
||||
//! A subquery's `<expression>` is any shape above, so a subquery nests inside a subquery on a grid
|
||||
//! of its own, as in `rate(rate(x[5m:1m])[10m:30s])`; a `<step>` left out means Prometheus's default
|
||||
//! evaluation interval, one minute. Its `offset` and its `@` anchor are its own, not the inner
|
||||
//! expression's, so `rate(x[5m:1m] @ end() offset 1m)` reads one fixed interval. A `<selector>`
|
||||
//! outside a `<matrix>` takes an `offset` and no anchor: the instant shape has no field for one, so
|
||||
//! `x @ 100` is refused and `rate(x[5m] @ 100)` is not.
|
||||
//!
|
||||
//! Those shapes become `Input` (raw samples) → `Instant` (the last sample within the statement's
|
||||
//! lookback), `Input` (raw samples) → `Window` (the range function over the statement's grid),
|
||||
//! `<expression>` → `Window` whose input is evaluated (the subquery, whose input grid is its own),
|
||||
//! `<expression>` → `Map` (a pointwise math function over what its argument evaluates to) and, over
|
||||
//! any of those, `Binary` (an arithmetic or comparison operation with a constant on one side),
|
||||
//! `Scalar` (that constant) and `Aggregate` (one of the nine aggregators that fold a group's values
|
||||
//! into one value per grid point, optionally grouped). Everything else is a
|
||||
//! [`DataFusionError::NotImplemented`] that names what was found and quotes the shapes above,
|
||||
//! because rejecting a query the compiler does not understand is the only honest alternative to
|
||||
//! compiling it into something else.
|
||||
//!
|
||||
//! The compiler binds a subquery's child grid and `RangeWindow` independently of runtime execution:
|
||||
//! compilation describes the anchored interval, while runtime support determines how that bound
|
||||
//! plan is consumed.
|
||||
//!
|
||||
//! Two decisions keep the recursion from growing a branch per query. A constant argument — a
|
||||
//! quantile, a rounding step, a clamp bound — is folded out of the statement by [`constant_value`],
|
||||
@@ -47,6 +89,28 @@
|
||||
//! alone, so the number this compiler binds and the number the kernel computes cannot disagree
|
||||
//! about `%`, `^` or `atan2`.
|
||||
//!
|
||||
//! Two more are about what a node means rather than about which node to build. A binary operation
|
||||
//! keeps its operands in the order they were written, because most operators are not commutative:
|
||||
//! `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them
|
||||
//! apart. `return_bool` is carried as its own field, so `x > 0` (which filters) and `x > bool 0`
|
||||
//! (which reports zero or one) are one node with one field changed. A negation then needs no node of
|
||||
//! its own: `-x` binds as the multiplication by `-1` that it is, which the multiplication kernel
|
||||
//! already computes.
|
||||
//!
|
||||
//! What a tree cannot yet express is deliberate too. The one input binding carries the ordering it
|
||||
//! guarantees and what it produces, but not the selector it reads, so a tree with two selectors
|
||||
//! cannot tell its inputs apart and an operation between two vectors is refused. A statement whose
|
||||
//! value is a scalar has nothing to read, so it is refused as well: a constant is bound next to a
|
||||
//! vector that gives it something to broadcast over.
|
||||
//!
|
||||
//! An aggregate is bound as [`AggregateStage::Complete`], which is a claim about the whole execution
|
||||
//! rather than about the node: it says that every contribution to every group is in this execution's
|
||||
//! input. The caller has to keep that claim. One tree per partition does not: handing every
|
||||
//! partition the same `Complete` tree sums each partition's part and reports it as the group's
|
||||
//! whole, so a partitioned caller has to feed whole groups to one execution or split the aggregation
|
||||
//! into `Partial` and `Final` instead. What a `Complete` node guarantees here is only the arithmetic
|
||||
//! — that the values it folded are all the values it was given.
|
||||
//!
|
||||
//! The statement's grid, the selector's lookback, window, offset and anchor are the only numbers
|
||||
//! derived here; the legality of the tree is not checked here. [`NativeTree::try_new`] is the
|
||||
//! single authority on that, so a node added later cannot be legal in one place and illegal in
|
||||
@@ -62,30 +126,52 @@ use datatypes::series::samples::SeriesSchema;
|
||||
use datatypes::series::time::{EvalGrid, GridSpec};
|
||||
use datatypes::series::{MetricName, SeriesOrdering};
|
||||
use promql::histogram_eval::HistogramFunction;
|
||||
use promql::map::{ArithmeticOp, MathOp};
|
||||
use promql::map::{ArithmeticOp, ComparisonOp, MathOp};
|
||||
use promql::native::plan::{
|
||||
BoundInput, BoundInstant, BoundMap, BoundVectorOp, BoundWindow, NativeExpr, NativeNode,
|
||||
NativeTree, NativeType,
|
||||
AggregateKind, AggregateStage, BoundAggregate, BoundInput, BoundInstant, BoundMap, BoundScalar,
|
||||
BoundVectorOp, BoundWindow, Grouping, NativeExpr, NativeNode, NativeTree, NativeType,
|
||||
};
|
||||
use promql::range_eval::{RangeFunction, RangeWindow};
|
||||
use promql::range_eval::{RangeFunction, RangeWindow, subquery_grid};
|
||||
use promql::select::{LabelMatcher, MatchOp, SortOrder, absent_labels};
|
||||
use promql::vector_match::{BinaryMatching, BinaryOp, Cardinality, FillValues, MatchingLabels};
|
||||
use promql_parser::label::MatchOp as ParserMatchOp;
|
||||
use promql_parser::parser::token::TokenType;
|
||||
use promql_parser::parser::value::ValueType;
|
||||
use promql_parser::parser::{
|
||||
AtModifier, Call, EvalStmt, Expr as PromExpr, MatrixSelector, NumberLiteral, Offset,
|
||||
StringLiteral, VectorSelector, token,
|
||||
AggregateExpr, AtModifier, BinModifier, BinaryExpr, Call, EvalStmt, Expr as PromExpr,
|
||||
LabelModifier, MatrixSelector, NumberLiteral, Offset, StringLiteral, SubqueryExpr,
|
||||
VectorMatchCardinality, VectorMatchFillValues, VectorSelector, token,
|
||||
};
|
||||
|
||||
use crate::promql::label_values::signed_millis_since_epoch;
|
||||
|
||||
/// The shapes this compiler accepts, quoted in every rejection.
|
||||
const SUPPORTED: &str = "a selector, a range function over a range selector — \
|
||||
`<range function>(<selector>[<range>])`, with its constants in front of \
|
||||
the selector as in `quantile_over_time(0.9, x[5m])` — or a math function, \
|
||||
a sort, `absent` or a histogram function over any of them, as in \
|
||||
`abs(rate(x[5m]))`: for example `x`, `rate(x[5m])`, \
|
||||
`quantile_over_time(0.9, x[5m])` or `abs(rate(x[5m]))`";
|
||||
///
|
||||
/// It is one sentence of grammar rather than a list of examples, because it is what a refused query
|
||||
/// reads: the shapes have to be true of every query the compiler binds, and the examples only show
|
||||
/// them. The two matrix shapes and the recursion through them are stated first, because that
|
||||
/// recursion is what a reader is most likely to get wrong.
|
||||
const SUPPORTED: &str = "a selector — `x`, optionally with an `offset` — a range function over \
|
||||
a matrix, where a matrix is `<selector>[<range>]` as in `rate(x[5m])` or \
|
||||
`<expression>[<range>:<step>]` as in `rate(x[5m:1m])`, with the step \
|
||||
optionally omitted as `[<range>:]`, where the subquery's `<expression>` \
|
||||
is any shape here, so a subquery nests on a grid of its own as in \
|
||||
`rate(rate(x[5m:1m])[10m:30s])`, and where either matrix shape carries the \
|
||||
`offset` and `@` anchor written on it as in `rate(x[5m] @ 1234)`; a math \
|
||||
function over any of them (`abs(rate(x[5m]))`, `clamp_min(x, 0)`), a binary \
|
||||
operation between one of them and a constant (`2 / (x + 1)`, \
|
||||
`x > bool 0`), a negation of one of them (`-x`), or one of the aggregators \
|
||||
that folds a group's values into one value per grid point — `sum`, `avg`, \
|
||||
`count`, `min`, `max`, `group`, `stddev`, `stdvar` and \
|
||||
`quantile(<constant>, <expression>)` — over any of them, optionally \
|
||||
grouped by `by` or `without` as in `avg by(job)(rate(x[5m]) * 100)`; a \
|
||||
sort, `absent` or `histogram_quantile` / `histogram_fraction` over any of \
|
||||
them (`sort_desc(x)`, `absent(x{job=\"a\"})`, \
|
||||
`histogram_quantile(0.9, rate(x[5m]))`); a range function's own \
|
||||
constants sit where that function takes them, before \
|
||||
the matrix for `quantile_over_time(0.9, x[5m])` and after it for \
|
||||
`predict_linear(x[5m], 300)` and \
|
||||
`double_exponential_smoothing(x[5m], 0.5, 0.1)`";
|
||||
|
||||
/// Compiles `stmt` into a tree that reads `input` and evaluates the statement's expression over it.
|
||||
///
|
||||
@@ -124,10 +210,25 @@ pub fn compile(
|
||||
series may not interleave",
|
||||
));
|
||||
}
|
||||
// A tree reads a scan and reports evaluated samples, so an expression whose value is not a
|
||||
// vector has nothing to compile into: `1 + 2` is a scalar, and the scalar node is bound as a
|
||||
// constant next to a vector that gives it something to broadcast over rather than on its own.
|
||||
// The check is on the statement's type rather than on the compiled node, so it does not depend on
|
||||
// which kinds of expression happen to compile to a scalar today.
|
||||
if stmt.expr.value_type() != ValueType::Vector {
|
||||
return Err(not_implemented(&format!(
|
||||
"the expression `{}`, which is a {} where a tree reports evaluated samples",
|
||||
stmt.expr,
|
||||
stmt.expr.value_type()
|
||||
)));
|
||||
}
|
||||
let context = Context {
|
||||
stmt,
|
||||
input,
|
||||
ordering,
|
||||
// The grid is built once for the whole compilation rather than per node: every node of a
|
||||
// tree without a subquery lands on this one, and a subquery replaces it for its own subtree.
|
||||
grid: statement_grid(stmt)?,
|
||||
};
|
||||
// The whole tree is validated once, here: a node compiled as another node's input is checked as
|
||||
// part of the tree rather than on its own, so the same node cannot be legal in one place and
|
||||
@@ -147,6 +248,13 @@ struct Context<'a> {
|
||||
input: Arc<SeriesSchema>,
|
||||
/// The ordering the caller's scan guarantees.
|
||||
ordering: SeriesOrdering,
|
||||
/// The grid the expression being compiled is evaluated on.
|
||||
///
|
||||
/// It is the statement's grid for everything except the inside of a subquery, which opens a grid
|
||||
/// of its own: `rate(x[5m:1m])` evaluates `x` at one-minute steps over the five-minute lookback,
|
||||
/// not at the statement's steps. Carrying it here is what lets [`compile_expr`] recurse into a
|
||||
/// subquery without every node builder having to ask which grid it is on.
|
||||
grid: Arc<EvalGrid>,
|
||||
}
|
||||
|
||||
/// Compiles `expr` into the node it denotes, recursing into whatever it is made of.
|
||||
@@ -155,16 +263,320 @@ struct Context<'a> {
|
||||
/// its argument, so a query nests as deeply as it was written. A kind this compiler has no node for
|
||||
/// is refused with what was found, which is why the rejection is useful without a coverage list.
|
||||
fn compile_expr(context: &Context, expr: &PromExpr) -> Result<NativeNode> {
|
||||
// A constant is a value rather than a computation, so it binds as the number it folds to, by the
|
||||
// same fold a call's parameters use: `x * (1 + 2)` carries `3` rather than an addition the tree
|
||||
// would have to know how to run. The fold is checked first so that no shape below has to say
|
||||
// what it does with a constant operand.
|
||||
if let Some(value) = constant_value(expr) {
|
||||
return Ok(scalar_node(context, value));
|
||||
}
|
||||
match expr {
|
||||
// Parentheses group, so they change how a statement was typed rather than what it asks for:
|
||||
// the expression inside them is the one that compiles.
|
||||
PromExpr::Paren(paren) => compile_expr(context, &paren.expr),
|
||||
PromExpr::Call(call) => compile_call(context, call),
|
||||
PromExpr::VectorSelector(selector) => instant_selection(context, selector),
|
||||
PromExpr::Unary(unary) => negate(context, &unary.expr),
|
||||
PromExpr::Binary(binary) => binary_operation(context, binary),
|
||||
PromExpr::Aggregate(aggregate) => aggregate_expr(context, aggregate),
|
||||
found => Err(not_implemented(&format!("the expression `{found}`"))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Binds `value` as a constant scalar on the grid of the expression being compiled.
|
||||
///
|
||||
/// The grid is the context's rather than the constant's own: a constant has no timestamps, so the
|
||||
/// only positions it can be broadcast over are the ones the expression it sits next to is evaluated
|
||||
/// at — the statement's grid, or a subquery's inside one. [`NativeTree::validate`] compares a binary
|
||||
/// operation's two grids by spec rather than by allocation.
|
||||
///
|
||||
/// Nothing here can fail: the grid is the one [`Context`] already carries, so a caller needs no `?`.
|
||||
fn scalar_node(context: &Context, value: f64) -> NativeNode {
|
||||
NativeNode {
|
||||
expr: NativeExpr::Scalar(BoundScalar { value }),
|
||||
output: NativeType::Scalar(Arc::clone(&context.grid)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compiles a negation as the multiplication it is: `-x` is `-1 * x`.
|
||||
///
|
||||
/// The tree has no negate node and needs none: broadcasting a constant over the expression's value
|
||||
/// is the same arithmetic, and the multiplication kernel is the one that already computes it. A
|
||||
/// dedicated negate kernel would be an optimization of this node's work rather than a different
|
||||
/// meaning, so binding the multiplication does not bind anything the query did not ask for.
|
||||
fn negate(context: &Context, operand: &PromExpr) -> Result<NativeNode> {
|
||||
// The constant goes on the left, where the minus was: `-x` subtracts nothing, but a reader of
|
||||
// the tree should see the same order the query was written in.
|
||||
Ok(broadcast(
|
||||
scalar_node(context, -1.0),
|
||||
compile_expr(context, operand)?,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
}
|
||||
|
||||
/// Compiles a binary operation whose operands are not both vectors.
|
||||
///
|
||||
/// The tree holds one input binding, and that binding carries the ordering it guarantees and what it
|
||||
/// produces but not the selector it reads, so two vectors cannot be told apart: `a + b` and `a + a`
|
||||
/// would compile to the same tree. That is why a vector–vector operation is refused until an input
|
||||
/// names its selector, and why an operation over one vector and one scalar is the shape the tree can
|
||||
/// bind: a scalar carries no labels, so broadcasting it is unambiguous.
|
||||
fn binary_operation(context: &Context, binary: &BinaryExpr) -> Result<NativeNode> {
|
||||
// A set operator combines two vectors' series rather than their values, and the tree has no node
|
||||
// for it at all. Its name is reported before the operands are looked at, so a set operation
|
||||
// between two vectors is refused as the set operation it is rather than as a binary operation
|
||||
// the compiler has no node for.
|
||||
if binary.op.is_set_operator() {
|
||||
return Err(not_implemented(&format!(
|
||||
"the set operator `{}` in `{binary}`, which has no native node: it combines two \
|
||||
vectors' series, and every binary node here is an arithmetic or comparison operation \
|
||||
over values",
|
||||
binary.op
|
||||
)));
|
||||
}
|
||||
let Some(op) = binary_op(&binary.op) else {
|
||||
return Err(not_implemented(&format!(
|
||||
"the binary operator `{}` in `{binary}`",
|
||||
binary.op
|
||||
)));
|
||||
};
|
||||
// Two vectors are refused before the modifier is looked at, so a vector–vector query with
|
||||
// matching says what is actually missing. The refusal names the missing identity rather than the
|
||||
// query, because that is what whoever adds it has to add.
|
||||
if binary.lhs.value_type() == ValueType::Vector && binary.rhs.value_type() == ValueType::Vector
|
||||
{
|
||||
return Err(not_implemented(&format!(
|
||||
"the binary operation `{binary}` between two vectors, which needs each input to name \
|
||||
the selector it reads: an input binding carries its ordering and what it produces, but \
|
||||
not which selector it was bound to"
|
||||
)));
|
||||
}
|
||||
// What is left has a constant on one side, and a modifier is refused rather than dropped. This
|
||||
// is deliberately narrower than Prometheus, which drops an `on` or `ignoring` written on a
|
||||
// scalar rather than refusing it; refusing is the conservative choice here, because the tree's
|
||||
// matching has no field that could carry the modifier, so binding the operation would compute
|
||||
// something the query did not ask for. A fill value is refused for the same reason: the tree's
|
||||
// matching has no field for one, and dropping it would compute the operation without the fill
|
||||
// the query asked for.
|
||||
if refuses_modifier(binary.modifier.as_ref()) {
|
||||
return Err(not_implemented(&format!(
|
||||
"the vector matching in `{binary}`, which a constant operand has nothing to match: \
|
||||
`on`, `ignoring`, `group_left` and `group_right` name labels of a vector, and a fill \
|
||||
value fills in a match group that could miss"
|
||||
)));
|
||||
}
|
||||
// Two constants are a value rather than an operation over a vector. A `bool` comparison of two
|
||||
// of them folded to its number before this point, so what arrives here is a comparison without
|
||||
// `bool`, which Prometheus refuses to parse at all. Binding it would produce a node that only
|
||||
// fails when it is run, so it is refused here with what was found.
|
||||
if binary.lhs.value_type() == ValueType::Scalar && binary.rhs.value_type() == ValueType::Scalar
|
||||
{
|
||||
return Err(not_implemented(&format!(
|
||||
"the binary operation `{binary}` between two constants, which reports no value: a \
|
||||
comparison of two scalars is written with `bool` to say whether it holds, and one \
|
||||
written that way is folded to the number it means"
|
||||
)));
|
||||
}
|
||||
// Both operands are compiled before the node is built, so `2 / (x + 1)` nests. The constant side
|
||||
// is bound by [`compile_expr`]'s fold, which is what makes it a scalar node rather than an
|
||||
// expression the tree would have to run.
|
||||
Ok(broadcast(
|
||||
compile_expr(context, &binary.lhs)?,
|
||||
compile_expr(context, &binary.rhs)?,
|
||||
op,
|
||||
binary.return_bool(),
|
||||
))
|
||||
}
|
||||
|
||||
/// Wraps two compiled operands in the binary node that applies `op` to them.
|
||||
///
|
||||
/// The output is the vector side's own binding, because a scalar carries no fields: a broadcast keeps
|
||||
/// the vector's fields and the grid it is evaluated on, which is the statement's grid on both sides.
|
||||
/// Two scalars are possible as a nested operand — a pair that [`constant_value`] refuses to fold, such
|
||||
/// as a comparison between two constants — and the node reports the left one's grid, which is the
|
||||
/// statement's too. The operands keep the order they were written in, because a non-commutative
|
||||
/// operator means what that order says.
|
||||
fn broadcast(left: NativeNode, right: NativeNode, op: BinaryOp, return_bool: bool) -> NativeNode {
|
||||
// `compile_expr` binds a constant to a scalar node and a selection or a function to a vector, so
|
||||
// this is where the side the operation reports is decided.
|
||||
let output = if matches!(left.output, NativeType::Scalar(_)) {
|
||||
right.output.clone()
|
||||
} else {
|
||||
left.output.clone()
|
||||
};
|
||||
NativeNode {
|
||||
expr: NativeExpr::Binary {
|
||||
left: Box::new(left),
|
||||
right: Box::new(right),
|
||||
matching: scalar_matching(op, return_bool),
|
||||
},
|
||||
output,
|
||||
}
|
||||
}
|
||||
|
||||
/// The matching a vector-and-constant operation records.
|
||||
///
|
||||
/// Nothing is matched: a scalar carries no labels, so [`NativeTree::validate`] broadcasts it and
|
||||
/// reads neither `labels` nor `cardinality`. They carry the default a vector–vector operation of the
|
||||
/// same operator would have, which is also the only matching PromQL allows here — it rejects
|
||||
/// `on`, `ignoring`, `group_left` and `group_right` on a scalar–vector operation. `return_bool` is
|
||||
/// the one modifier that changes what this operation computes: `x > 0` filters the vector and
|
||||
/// `x > bool 0` reports zero or one for each of its values.
|
||||
fn scalar_matching(op: BinaryOp, return_bool: bool) -> BinaryMatching {
|
||||
BinaryMatching {
|
||||
op,
|
||||
return_bool,
|
||||
labels: MatchingLabels::Ignoring(Vec::new()),
|
||||
cardinality: Cardinality::OneToOne,
|
||||
// A fill value fills in a match group that could miss, which a broadcast over a constant never
|
||||
// does: the constant exists at every grid point, so there is no missing side to fill.
|
||||
fill: FillValues::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The binary operator a PromQL token denotes, or `None` when there is no node for it.
|
||||
///
|
||||
/// A comparison is bound like an arithmetic operator, because the node carries `return_bool`
|
||||
/// separately: `x > 0` and `x > bool 0` are one operator with one field changed. [`arithmetic_op`]
|
||||
/// is not reused because it decides what a *fold* may do at compile time and a comparison folds
|
||||
/// nothing, while this map decides what a *node* computes once the query runs.
|
||||
fn binary_op(op: &TokenType) -> Option<BinaryOp> {
|
||||
match op.id() {
|
||||
token::T_ADD => Some(BinaryOp::Add),
|
||||
token::T_SUB => Some(BinaryOp::Sub),
|
||||
token::T_MUL => Some(BinaryOp::Mul),
|
||||
token::T_DIV => Some(BinaryOp::Div),
|
||||
token::T_MOD => Some(BinaryOp::Mod),
|
||||
token::T_POW => Some(BinaryOp::Pow),
|
||||
token::T_ATAN2 => Some(BinaryOp::Atan2),
|
||||
token::T_EQLC => Some(BinaryOp::Eq),
|
||||
token::T_NEQ => Some(BinaryOp::Ne),
|
||||
token::T_GTR => Some(BinaryOp::Gt),
|
||||
token::T_LSS => Some(BinaryOp::Lt),
|
||||
token::T_GTE => Some(BinaryOp::Ge),
|
||||
token::T_LTE => Some(BinaryOp::Le),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Compiles an aggregation into an aggregate node over the expression it groups.
|
||||
///
|
||||
/// Every aggregator that folds a group's values into one value per grid point binds, whether or not
|
||||
/// its kernel exists yet: `sum`, `avg`, `count`, `min`, `max`, `group`, `stddev`, `stdvar` and
|
||||
/// `quantile`. Binding a shape the compiler can describe and letting the runtime refuse a kind it
|
||||
/// cannot execute keeps this the one place that says what an aggregate means, so a kernel landing
|
||||
/// later changes no binding.
|
||||
///
|
||||
/// The rest of the parser's aggregates are a different shape, and are refused with the reason rather
|
||||
/// than folded into a kind that would compute something else: `topk`, `bottomk`, `limitk` and
|
||||
/// `limit_ratio` select some of the input's series, so they do not report one value per group per
|
||||
/// grid point, and `count_values` writes one output label per distinct value rather than folding a
|
||||
/// group's values at all.
|
||||
///
|
||||
/// The grouping is bound in the tree's own terms, because `by` and `without` are not each other's
|
||||
/// inverse: `by(job)` keeps `job` and drops every other label, while `without(instance)` keeps every
|
||||
/// label it does not name and drops `__name__` as well, so a list of the surviving labels could not
|
||||
/// say the same thing. A bare `sum(x)` groups by no label at all, which is Prometheus's reading of
|
||||
/// "no modifier": every label is dropped, `__name__` included, and every series of the input falls
|
||||
/// into one group.
|
||||
///
|
||||
/// The stage is [`AggregateStage::Complete`], which says that every contribution to each group is in
|
||||
/// this execution's input. The module documentation says what keeping that claim costs a caller.
|
||||
fn aggregate_expr(context: &Context, aggregate: &AggregateExpr) -> Result<NativeNode> {
|
||||
let AggregateExpr {
|
||||
op,
|
||||
expr,
|
||||
param,
|
||||
modifier,
|
||||
} = aggregate;
|
||||
let kind = match op.id() {
|
||||
token::T_SUM => AggregateKind::Sum,
|
||||
token::T_AVG => AggregateKind::Avg,
|
||||
token::T_COUNT => AggregateKind::Count,
|
||||
token::T_MIN => AggregateKind::Min,
|
||||
token::T_MAX => AggregateKind::Max,
|
||||
token::T_GROUP => AggregateKind::Group,
|
||||
token::T_STDDEV => AggregateKind::Stddev,
|
||||
token::T_STDVAR => AggregateKind::Stdvar,
|
||||
token::T_QUANTILE => {
|
||||
// `quantile` is the only one of the nine with a parameter, and the parser always gives
|
||||
// it one: a `quantile` with no quantile written is refused before the compiler sees it.
|
||||
// The quantile is the call's first argument, so a parameter that is not a constant is
|
||||
// refused under that name rather than answered from a number bound too early.
|
||||
let param = param.as_ref().ok_or_else(|| {
|
||||
not_implemented(&format!(
|
||||
"the aggregate `{op}` in `{aggregate}`, which takes its quantile as its first \
|
||||
argument"
|
||||
))
|
||||
})?;
|
||||
AggregateKind::Quantile {
|
||||
quantile: constant_argument(&op.to_string(), param, 0)?,
|
||||
}
|
||||
}
|
||||
// A ranking aggregate keeps some of the input's series, so what it reports is not one value
|
||||
// per group per grid point: `topk(3, x)` answers with three series, not with the groups'
|
||||
// folded values.
|
||||
token::T_TOPK | token::T_BOTTOMK | token::T_LIMITK | token::T_LIMIT_RATIO => {
|
||||
return Err(not_implemented(&format!(
|
||||
"the aggregate `{op}` in `{aggregate}`, which selects some of its input's series \
|
||||
rather than folding each group's values into one value per grid point"
|
||||
)));
|
||||
}
|
||||
// `count_values` folds nothing: it writes one output label per distinct value it saw, which
|
||||
// is a shape no folded value can spell.
|
||||
token::T_COUNT_VALUES => {
|
||||
return Err(not_implemented(&format!(
|
||||
"the aggregate `{op}` in `{aggregate}`, which writes one output label per distinct \
|
||||
value rather than folding each group's values into one value per grid point"
|
||||
)));
|
||||
}
|
||||
// The parser's `aggregate_op` rule lists exactly the fourteen names above, so this arm is
|
||||
// an operator this compiler was not taught rather than one it means to fold.
|
||||
_ => {
|
||||
return Err(not_implemented(&format!(
|
||||
"the aggregate `{op}` in `{aggregate}`"
|
||||
)));
|
||||
}
|
||||
};
|
||||
let input = compile_expr(context, expr)?;
|
||||
// A complete aggregate folds each group's values and reports them on its input's grid, one value
|
||||
// per input field, so its layout is its input's.
|
||||
let output = input.output.clone();
|
||||
Ok(NativeNode {
|
||||
expr: NativeExpr::Aggregate {
|
||||
input: Box::new(input),
|
||||
op: BoundAggregate {
|
||||
kind,
|
||||
group: grouping(modifier.as_ref()),
|
||||
stage: AggregateStage::Complete,
|
||||
},
|
||||
},
|
||||
output,
|
||||
})
|
||||
}
|
||||
|
||||
/// How a `by` or `without` modifier identifies a group, in the tree's own terms.
|
||||
///
|
||||
/// The labels keep the order they were written in: a grouping is a set, and the tree records the
|
||||
/// query rather than a key it derived from one.
|
||||
fn grouping(modifier: Option<&LabelModifier>) -> Grouping {
|
||||
/// The written labels, as the tree stores them.
|
||||
fn labels(labels: &[String]) -> Arc<[Arc<str>]> {
|
||||
labels
|
||||
.iter()
|
||||
.map(|label| Arc::<str>::from(label.as_str()))
|
||||
.collect()
|
||||
}
|
||||
match modifier {
|
||||
// No modifier and `by()` ask for the same grouping: no label is kept, so every series of the
|
||||
// input falls into one group.
|
||||
None => Grouping::By(Arc::new([])),
|
||||
Some(LabelModifier::Include(by)) => Grouping::By(labels(&by.labels)),
|
||||
Some(LabelModifier::Exclude(without)) => Grouping::Without(labels(&without.labels)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compiles a call: a range function opens a window over the selector it is given, and a math
|
||||
/// function maps the value of its first argument.
|
||||
///
|
||||
@@ -181,9 +593,8 @@ fn compile_call(context: &Context, call: &Call) -> Result<NativeNode> {
|
||||
return vector_call(context, func.name, op, vector);
|
||||
}
|
||||
// The name is looked up before the call's arguments are read, so a name this compiler has no
|
||||
// kernel for is refused as such whichever arguments it was given: `pi()` and `time()` have no
|
||||
// vector to map, and the rejection names the name rather than describing a vector they never
|
||||
// wrote.
|
||||
// kernel for is refused as such whichever argument it was given: `absent(x)` is refused as a
|
||||
// name with no kernel, not as a selector this compiler could not compile.
|
||||
let Some((op, vector)) = math_op(func.name, &args.args)? else {
|
||||
return Err(not_implemented(&format!(
|
||||
"a call to `{}`, which has no native kernel",
|
||||
@@ -193,10 +604,17 @@ fn compile_call(context: &Context, call: &Call) -> Result<NativeNode> {
|
||||
map_call(context, func.name, op, vector)
|
||||
}
|
||||
|
||||
/// Binds a range function over the range selector it is given, and the window that selector opens.
|
||||
/// Binds a range function over the matrix it is given, and the window that matrix opens.
|
||||
///
|
||||
/// The parameters come first and the range selector is last, for both signatures: a function without
|
||||
/// parameters has no `parameters` to read.
|
||||
/// Where the matrix sits is the signature's, not a convention: `quantile_over_time(0.9, x[5m])` writes
|
||||
/// its constant before the matrix, while `predict_linear(x[5m], 300)` and
|
||||
/// `double_exponential_smoothing(x[5m], 0.5, 0.1)` write theirs after it, and a function with no
|
||||
/// constants at all reads the matrix first. [`RangeSignature::selector_index`] is that position, and
|
||||
/// everything else the call wrote is a constant — which keeps the position it was written at, so a
|
||||
/// rejection names the argument the query wrote rather than the constant's offset among the constants.
|
||||
///
|
||||
/// The matrix is either shape: a range selector, which reads raw samples, or a subquery, which
|
||||
/// evaluates its own expression on a grid of its own.
|
||||
fn window_call(
|
||||
context: &Context,
|
||||
name: &str,
|
||||
@@ -205,7 +623,7 @@ fn window_call(
|
||||
) -> Result<NativeNode> {
|
||||
// The count is the signature's, so the rejection names what this function takes rather than
|
||||
// what one argument happens to be.
|
||||
let expected = signature.parameters() + 1;
|
||||
let expected = signature.arity();
|
||||
if args.len() != expected {
|
||||
return Err(not_implemented(&format!(
|
||||
"a call to `{name}` with {} argument(s), where it takes {expected}: {}",
|
||||
@@ -213,18 +631,24 @@ fn window_call(
|
||||
signature.arguments()
|
||||
)));
|
||||
}
|
||||
let (parameters, selector) = args.split_at(signature.parameters());
|
||||
let function = match signature {
|
||||
RangeSignature::Selector(function) => function,
|
||||
RangeSignature::Scalar(bind) => bind(constant_argument(name, ¶meters[0], 0)?),
|
||||
};
|
||||
let argument = &selector[0];
|
||||
let PromExpr::MatrixSelector(MatrixSelector { vs, range }) = unparen(argument) else {
|
||||
return Err(not_implemented(&format!(
|
||||
"a call to `{name}` whose argument is `{argument}`"
|
||||
)));
|
||||
};
|
||||
window_selection(context, function, vs, range)
|
||||
// The matrix sits where the signature says it does, and everything else is a constant. Reading it
|
||||
// by position rather than by "the last argument" is what lets `predict_linear(x[5m], 300)` and
|
||||
// `quantile_over_time(0.9, x[5m])` share one path.
|
||||
let selector_index = signature.selector_index();
|
||||
// The selector stays in its original argument position for signature binding, while either
|
||||
// matrix form can provide the samples folded by the range function.
|
||||
let argument = &args[selector_index];
|
||||
let function = signature.bind(name, args)?;
|
||||
match unparen(argument) {
|
||||
PromExpr::MatrixSelector(MatrixSelector { vs, range }) => {
|
||||
window_selection(context, function, vs, range)
|
||||
}
|
||||
PromExpr::Subquery(subquery) => subquery_window(context, name, function, subquery),
|
||||
other => Err(not_implemented(&format!(
|
||||
"a call to `{name}` whose matrix is `{other}`, where it has to be a range \
|
||||
selector like `x[5m]` or a subquery like `x[5m:1m]`"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Wraps `op` over the value of `vector`, which is the first argument of a math function.
|
||||
@@ -420,7 +844,7 @@ fn window_selection(
|
||||
let anchor_ms = selector_anchor_ms(&vs.at, context.stmt)?;
|
||||
// The grid is the last thing built: a query refused above, or a window that cannot exist,
|
||||
// never pays for a grid nothing would read.
|
||||
let output_grid = statement_grid(context.stmt)?;
|
||||
let output_grid = Arc::clone(&context.grid);
|
||||
|
||||
// A range function reports one value per input field, so the layout is the input's.
|
||||
let output = NativeType::Samples(Arc::new(SeriesSchema {
|
||||
@@ -458,6 +882,115 @@ fn window_selection(
|
||||
})
|
||||
}
|
||||
|
||||
/// The step a subquery names when it names none, as in `x[5m:]`.
|
||||
///
|
||||
/// Prometheus falls back to the global evaluation interval, whose default is a minute, rather than to
|
||||
/// the statement's own step. The existing planner folds the same default; the constant is restated
|
||||
/// here because that one is private to the planner, and it is a Prometheus protocol constant rather
|
||||
/// than a number either caller may choose.
|
||||
const DEFAULT_SUBQUERY_STEP_MS: i64 = 60_000;
|
||||
|
||||
/// Binds `function` over `subquery`, whose inner expression is evaluated on a grid of its own.
|
||||
///
|
||||
/// A range function folds a matrix, and PromQL writes a matrix two ways. A range selector reads raw
|
||||
/// samples from the scan; a subquery builds one by evaluating an expression. That is why this node
|
||||
/// declares `input_grid`: `rate(x[5m:1m])` samples `x` at one-minute steps over the five-minute
|
||||
/// lookback, and the window folds those points rather than raw samples. The inner expression is
|
||||
/// compiled against the subquery's grid for the same reason — a selector inside it has to land where
|
||||
/// the window will read it.
|
||||
///
|
||||
/// An `@` anchor makes the subquery step-invariant: every output point reads the same interval,
|
||||
/// `(anchor - offset - range, anchor - offset]`. The child grid is built for that interval, while
|
||||
/// the output grid remains the statement's grid; the anchor, range and offset are bound together in
|
||||
/// the window.
|
||||
fn subquery_window(
|
||||
context: &Context,
|
||||
name: &str,
|
||||
function: RangeFunction,
|
||||
subquery: &SubqueryExpr,
|
||||
) -> Result<NativeNode> {
|
||||
let SubqueryExpr {
|
||||
expr,
|
||||
range,
|
||||
step,
|
||||
offset,
|
||||
at,
|
||||
} = subquery;
|
||||
// The anchor is resolved by the same function a range selector's is, so `start()`, `end()` and a
|
||||
// literal — including a negative one — mean the same thing in both shapes.
|
||||
let anchor_ms = selector_anchor_ms(at, context.stmt)?;
|
||||
// A subquery evaluates an expression and hands the window one value per grid point per field, so
|
||||
// the expression inside it has to be a vector: a scalar has no fields for the window to fold.
|
||||
if expr.value_type() != ValueType::Vector {
|
||||
return Err(not_implemented(&format!(
|
||||
"a call to `{name}` over the subquery `{subquery}`, whose expression is a {}, where a \
|
||||
subquery has to evaluate a vector",
|
||||
expr.value_type()
|
||||
)));
|
||||
}
|
||||
let range_ms = range_millis(range.as_millis())?;
|
||||
// A subquery step of zero names no evaluation point at all, so it is reported as its own value
|
||||
// rather than as an empty grid.
|
||||
let step_ms = match step {
|
||||
Some(step) => millis(step.as_millis()).ok_or_else(|| {
|
||||
DataFusionError::Plan(format!(
|
||||
"the subquery's step of {}ms is out of range",
|
||||
step.as_millis()
|
||||
))
|
||||
})?,
|
||||
None => DEFAULT_SUBQUERY_STEP_MS,
|
||||
};
|
||||
if step_ms <= 0 {
|
||||
return Err(DataFusionError::Plan(format!(
|
||||
"the subquery's step is {step_ms}ms, and a step has to be longer than zero"
|
||||
)));
|
||||
}
|
||||
let offset_ms = selector_offset_ms(offset)?;
|
||||
let window = RangeWindow {
|
||||
range_ms,
|
||||
offset_ms,
|
||||
anchor_ms,
|
||||
};
|
||||
let child_spec = subquery_grid(context.grid.spec(), window, step_ms)?.ok_or_else(|| {
|
||||
not_implemented(&format!(
|
||||
"the subquery over {range_ms}ms at {step_ms}ms per step, which names no evaluation point \
|
||||
in its anchored interval or the statement's range"
|
||||
))
|
||||
})?;
|
||||
let child_grid = Arc::new(EvalGrid::try_deferred(child_spec).map_err(grid_error)?);
|
||||
let inner = compile_expr(
|
||||
&Context {
|
||||
stmt: context.stmt,
|
||||
input: Arc::clone(&context.input),
|
||||
ordering: context.ordering,
|
||||
grid: Arc::clone(&child_grid),
|
||||
},
|
||||
expr,
|
||||
)?;
|
||||
|
||||
// A range function reports one value per input field, and the inner expression reports one per
|
||||
// field of the same raw input, so the layout is the input's.
|
||||
let output = NativeType::Samples(Arc::new(SeriesSchema {
|
||||
fields: Arc::clone(&context.input.fields),
|
||||
eval_grid: Some(Arc::clone(&context.grid)),
|
||||
}));
|
||||
|
||||
Ok(NativeNode {
|
||||
expr: NativeExpr::Window {
|
||||
input: Box::new(inner),
|
||||
op: BoundWindow {
|
||||
function,
|
||||
window,
|
||||
// The one field that tells this window from a raw one: its input was evaluated on the
|
||||
// subquery's grid rather than read from the scan.
|
||||
input_grid: Some(child_grid),
|
||||
output_grid: Arc::clone(&context.grid),
|
||||
},
|
||||
},
|
||||
output,
|
||||
})
|
||||
}
|
||||
|
||||
/// Compiles a bare selector into an instant selection over the raw input.
|
||||
///
|
||||
/// The lookback, the offset and the grid are the whole of an instant selection: it reports the last
|
||||
@@ -475,7 +1008,7 @@ fn instant_selection(context: &Context, selector: &VectorSelector) -> Result<Nat
|
||||
}
|
||||
let lookback_ms = lookback_millis(context.stmt.lookback_delta.as_millis())?;
|
||||
let offset_ms = selector_offset_ms(&selector.offset)?;
|
||||
let output_grid = statement_grid(context.stmt)?;
|
||||
let output_grid = Arc::clone(&context.grid);
|
||||
|
||||
// An instant selection reports one value per input field, so the layout is the input's.
|
||||
let output = NativeType::Samples(Arc::new(SeriesSchema {
|
||||
@@ -526,7 +1059,7 @@ fn statement_grid(stmt: &EvalStmt) -> Result<Arc<EvalGrid>> {
|
||||
{error}"
|
||||
))
|
||||
})?;
|
||||
Ok(Arc::new(EvalGrid::try_new(spec).map_err(grid_error)?))
|
||||
Ok(Arc::new(EvalGrid::try_deferred(spec).map_err(grid_error)?))
|
||||
}
|
||||
|
||||
/// The window length of a range selector, which has to be longer than zero to select anything.
|
||||
@@ -602,34 +1135,74 @@ fn grid_error(error: datatypes::error::Error) -> DataFusionError {
|
||||
DataFusionError::Plan(format!("the statement's grid is invalid: {error}"))
|
||||
}
|
||||
|
||||
/// What a call to a range function has to be: the parameters it reads, and how to bind them.
|
||||
/// What a call to a range function has to be: where its range selector sits, how many constants it
|
||||
/// reads, and how they bind.
|
||||
///
|
||||
/// The selector's position is a variant rather than a convention because PromQL does not put it in
|
||||
/// one place: `quantile_over_time(0.9, x[5m])` writes its quantile before the selector, while
|
||||
/// `predict_linear(x[5m], 300)` and `double_exponential_smoothing(x[5m], 0.5, 0.1)` write theirs
|
||||
/// after it. A single "parameters come first" rule would read the selector as a constant in the
|
||||
/// second group and reject an ordinary query, so each shape states its own layout.
|
||||
#[derive(Clone, Copy)]
|
||||
enum RangeSignature {
|
||||
/// A function that reads only its range selector, as in `rate(x[5m])`.
|
||||
Selector(RangeFunction),
|
||||
/// A function that reads one constant before its range selector, and the function that constant
|
||||
/// binds, as in `quantile_over_time(0.9, x[5m])`.
|
||||
///
|
||||
/// The constant is kept out of the variant name because only the call site knows it.
|
||||
Scalar(fn(f64) -> RangeFunction),
|
||||
/// One constant, then the range selector, as in `quantile_over_time(0.9, x[5m])`.
|
||||
ConstantBeforeSelector(fn(f64) -> RangeFunction),
|
||||
/// The range selector, then one constant, as in `predict_linear(x[5m], 300)`.
|
||||
ConstantAfterSelector(fn(f64) -> RangeFunction),
|
||||
/// The range selector, then two constants, as in
|
||||
/// `double_exponential_smoothing(x[5m], 0.5, 0.1)`.
|
||||
TwoConstantsAfterSelector(fn(f64, f64) -> RangeFunction),
|
||||
}
|
||||
|
||||
impl RangeSignature {
|
||||
/// How many scalar parameters the function reads before its range selector.
|
||||
fn parameters(self) -> usize {
|
||||
/// The position of the range selector among the call's arguments.
|
||||
fn selector_index(self) -> usize {
|
||||
match self {
|
||||
Self::Selector(_)
|
||||
| Self::ConstantAfterSelector(_)
|
||||
| Self::TwoConstantsAfterSelector(_) => 0,
|
||||
Self::ConstantBeforeSelector(_) => 1,
|
||||
}
|
||||
}
|
||||
|
||||
/// How many constants the function reads.
|
||||
fn constants(self) -> usize {
|
||||
match self {
|
||||
Self::Selector(_) => 0,
|
||||
Self::Scalar(_) => 1,
|
||||
Self::ConstantBeforeSelector(_) | Self::ConstantAfterSelector(_) => 1,
|
||||
Self::TwoConstantsAfterSelector(_) => 2,
|
||||
}
|
||||
}
|
||||
|
||||
/// How many arguments the call writes, its range selector counted in.
|
||||
fn arity(self) -> usize {
|
||||
self.constants() + 1
|
||||
}
|
||||
|
||||
/// What the call's arguments are, for the arity rejection to name them.
|
||||
fn arguments(self) -> &'static str {
|
||||
match self {
|
||||
Self::Selector(_) => "its range selector",
|
||||
Self::Scalar(_) => "a number and its range selector",
|
||||
Self::ConstantBeforeSelector(_) => "a number and its range selector",
|
||||
Self::ConstantAfterSelector(_) => "its range selector and a number",
|
||||
Self::TwoConstantsAfterSelector(_) => "its range selector and two numbers",
|
||||
}
|
||||
}
|
||||
|
||||
/// Binds the function from the call's arguments, which have already passed the arity check.
|
||||
fn bind(self, name: &str, args: &[Box<PromExpr>]) -> Result<RangeFunction> {
|
||||
Ok(match self {
|
||||
Self::Selector(function) => function,
|
||||
Self::ConstantBeforeSelector(bind) => bind(constant_argument(name, &args[0], 0)?),
|
||||
Self::ConstantAfterSelector(bind) => bind(constant_argument(name, &args[1], 1)?),
|
||||
Self::TwoConstantsAfterSelector(bind) => bind(
|
||||
constant_argument(name, &args[1], 1)?,
|
||||
constant_argument(name, &args[2], 2)?,
|
||||
),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// The signature the PromQL name calls, or `None` when this compiler has no range kernel for it.
|
||||
@@ -640,7 +1213,17 @@ impl RangeSignature {
|
||||
/// its name alone cannot produce the variant the tree binds.
|
||||
fn range_signature(name: &str) -> Option<RangeSignature> {
|
||||
match name {
|
||||
"quantile_over_time" => Some(RangeSignature::Scalar(RangeFunction::QuantileOverTime)),
|
||||
"quantile_over_time" => Some(RangeSignature::ConstantBeforeSelector(
|
||||
RangeFunction::QuantileOverTime,
|
||||
)),
|
||||
// These two write their constants after the selector, which is the opposite order to
|
||||
// `quantile_over_time`.
|
||||
"predict_linear" => Some(RangeSignature::ConstantAfterSelector(
|
||||
RangeFunction::PredictLinear,
|
||||
)),
|
||||
"double_exponential_smoothing" => Some(RangeSignature::TwoConstantsAfterSelector(
|
||||
|smoothing, trend| RangeFunction::DoubleExponentialSmoothing { smoothing, trend },
|
||||
)),
|
||||
_ => range_function(name).map(RangeSignature::Selector),
|
||||
}
|
||||
}
|
||||
@@ -805,14 +1388,18 @@ fn arity_error(name: &str, signature: MathSignature, given: usize) -> DataFusion
|
||||
/// case — has nowhere to go. It is refused rather than answered from the wrong number, and the
|
||||
/// rejection names the expression, because that is what the query has to change.
|
||||
///
|
||||
/// `name` is whoever reads the argument: a call's function name, or an aggregator's, since
|
||||
/// `quantile(0.9, x)` reads a constant too. The rejection therefore names that and the position
|
||||
/// rather than calling everything a call, which would be false for an aggregate.
|
||||
///
|
||||
/// A quantile outside `[0, 1]` is not refused here: Prometheus binds it and the kernel reports
|
||||
/// `-Infinity` below zero and `+Infinity` above one, so refusing it would be this compiler inventing
|
||||
/// a rule the query engine does not have.
|
||||
fn constant_argument(name: &str, argument: &PromExpr, position: usize) -> Result<f64> {
|
||||
constant_value(argument).ok_or_else(|| {
|
||||
not_implemented(&format!(
|
||||
"a call to `{name}` whose {} is `{argument}`, where it has to be a constant expression \
|
||||
this compiler folds to a number literal",
|
||||
"`{name}`, whose {} is `{argument}`, where it has to be a constant expression this \
|
||||
compiler folds to a number literal",
|
||||
argument_position(position)
|
||||
))
|
||||
})
|
||||
@@ -820,14 +1407,30 @@ fn constant_argument(name: &str, argument: &PromExpr, position: usize) -> Result
|
||||
|
||||
/// How a rejection names the argument a parameter was read from.
|
||||
///
|
||||
/// The three positions are the ones a signature can read: the vector, and the two constants behind
|
||||
/// it. Every caller passes the literal position its signature reads, so there is no fourth position
|
||||
/// to name and no default to invent one for.
|
||||
/// The three positions are the ones a supported call can write: a math function's vector and the two
|
||||
/// constants behind it, or a range function's selector and the constants on either side of it. Every
|
||||
/// caller passes the literal position it read, so there is no fourth position to name and no default
|
||||
/// to invent one for.
|
||||
fn argument_position(position: usize) -> &'static str {
|
||||
const POSITIONS: [&str; 3] = ["first argument", "second argument", "third argument"];
|
||||
POSITIONS[position]
|
||||
}
|
||||
|
||||
/// Whether a binary operation carries a modifier this compiler refuses to bind.
|
||||
///
|
||||
/// A scalar carries no labels, so there is nothing to match or name, and the tree's matching has no
|
||||
/// field for a fill value: binding the operation would compute it without the modifier the query
|
||||
/// asked for. The fold and [`binary_operation`] share this test rather than restating it, because a
|
||||
/// fold that computed a constant without reading its modifier would answer a query the dispatch
|
||||
/// below refuses.
|
||||
fn refuses_modifier(modifier: Option<&BinModifier>) -> bool {
|
||||
modifier.is_some_and(|modifier| {
|
||||
modifier.matching.is_some()
|
||||
|| modifier.card != VectorMatchCardinality::OneToOne
|
||||
|| modifier.fill_values != VectorMatchFillValues::default()
|
||||
})
|
||||
}
|
||||
|
||||
/// The number `expr` folds to, or `None` when its value is not the statement's to know.
|
||||
///
|
||||
/// Only what the statement itself decides is folded: a number literal, a group around one, a negated
|
||||
@@ -847,10 +1450,28 @@ fn constant_value(expr: &PromExpr) -> Option<f64> {
|
||||
PromExpr::Paren(paren) => constant_value(&paren.expr),
|
||||
PromExpr::Unary(unary) => constant_value(&unary.expr).map(|value| -value),
|
||||
PromExpr::Binary(binary) => {
|
||||
let op = arithmetic_op(&binary.op)?;
|
||||
// A modifier is read before the operands are folded: `x * (1 + fill(0) 2)` asks for a
|
||||
// fill, and folding the inner addition would answer `x * 3` with the fill gone. The
|
||||
// expression is left unfolded so the dispatch below refuses it with the reason it
|
||||
// writes, which is also why this is a `None` rather than an error here.
|
||||
if refuses_modifier(binary.modifier.as_ref()) {
|
||||
return None;
|
||||
}
|
||||
let left = constant_value(&binary.lhs)?;
|
||||
let right = constant_value(&binary.rhs)?;
|
||||
Some(op.calculate(left, right))
|
||||
if let Some(op) = arithmetic_op(&binary.op) {
|
||||
return Some(op.calculate(left, right));
|
||||
}
|
||||
// A comparison of two constants is the number it reports when it is written with
|
||||
// `bool`: Prometheus reads `1 > bool 2` as `0`, and [`ComparisonOp::apply`] is the same
|
||||
// answer the kernel gives. Without `bool` there is no number to report -- `1 > 2` is a
|
||||
// parse error in Prometheus -- so it is left unfolded and the dispatch refuses it.
|
||||
if binary.return_bool()
|
||||
&& let Some(op) = comparison_op(&binary.op)
|
||||
{
|
||||
return Some(f64::from(op.apply(left, right)));
|
||||
}
|
||||
None
|
||||
}
|
||||
// Calls (`time()`, `scalar(v)`), selectors, range selectors, aggregates and subqueries all
|
||||
// read something this compiler cannot know before the query runs.
|
||||
@@ -862,8 +1483,9 @@ fn constant_value(expr: &PromExpr) -> Option<f64> {
|
||||
///
|
||||
/// The token is matched by its id, which is stable and is what the rest of the planner matches on,
|
||||
/// rather than by the text it displays as: a symbol this compiler does not recognise is not
|
||||
/// arithmetic here rather than a wrong operator. A comparison and a set operator are the cases that
|
||||
/// reach this and fold nothing, which is this batch's boundary rather than an accident of the map.
|
||||
/// arithmetic here rather than a wrong operator. A comparison is `None` here and is folded by
|
||||
/// [`comparison_op`] instead, because only a `bool` comparison of two constants has a number to
|
||||
/// report; a set operator folds nothing at all.
|
||||
fn arithmetic_op(op: &TokenType) -> Option<ArithmeticOp> {
|
||||
match op.id() {
|
||||
token::T_ADD => Some(ArithmeticOp::Add),
|
||||
@@ -877,6 +1499,22 @@ fn arithmetic_op(op: &TokenType) -> Option<ArithmeticOp> {
|
||||
}
|
||||
}
|
||||
|
||||
/// The comparison operator a PromQL token denotes, or `None` when the token is not a comparison.
|
||||
///
|
||||
/// This is the folding half of [`binary_op`], which binds the same tokens to a node: the fold and
|
||||
/// the node have to agree on which token is which comparison, so the two maps sit beside each other.
|
||||
fn comparison_op(op: &TokenType) -> Option<ComparisonOp> {
|
||||
match op.id() {
|
||||
token::T_EQLC => Some(ComparisonOp::Equal),
|
||||
token::T_NEQ => Some(ComparisonOp::NotEqual),
|
||||
token::T_GTR => Some(ComparisonOp::Greater),
|
||||
token::T_LSS => Some(ComparisonOp::Less),
|
||||
token::T_GTE => Some(ComparisonOp::GreaterEqual),
|
||||
token::T_LTE => Some(ComparisonOp::LessEqual),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The range function a PromQL name denotes, or `None` when this compiler has no kernel for it.
|
||||
///
|
||||
/// The map is explicit on purpose: a name that is not listed here is rejected, so supporting one
|
||||
@@ -901,6 +1539,13 @@ fn range_function(name: &str) -> Option<RangeFunction> {
|
||||
"resets" => Some(RangeFunction::Resets),
|
||||
"changes" => Some(RangeFunction::Changes),
|
||||
"deriv" => Some(RangeFunction::Deriv),
|
||||
// These functions are supported unconditionally by this native compiler.
|
||||
"first_over_time" => Some(RangeFunction::FirstOverTime),
|
||||
"mad_over_time" => Some(RangeFunction::MadOverTime),
|
||||
"ts_of_first_over_time" => Some(RangeFunction::TsOfFirstOverTime),
|
||||
"ts_of_last_over_time" => Some(RangeFunction::TsOfLastOverTime),
|
||||
"ts_of_min_over_time" => Some(RangeFunction::TsOfMinOverTime),
|
||||
"ts_of_max_over_time" => Some(RangeFunction::TsOfMaxOverTime),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user