trinity-1686a
9bcd99c396
rustfmt
2026-08-04 13:23:19 +02:00
trinity.pointard
d6666d09f3
extract per-field fast-path compatiblity check
2026-08-04 13:23:19 +02:00
trinity.pointard
8af115f802
decode column once per doc max
2026-08-04 13:23:19 +02:00
trinity.pointard
2f477e9414
more codex review
2026-08-04 13:23:19 +02:00
trinity.pointard
1fcf935f5c
fix handling of bytes for cardinality agg
2026-08-04 13:23:19 +02:00
trinity.pointard
73010f4d86
codex review
2026-08-04 13:23:19 +02:00
trinity.pointard
53d7af7b50
rustfmt
2026-08-04 13:23:19 +02:00
trinity.pointard
901af02fe0
fix bool serialization
2026-08-04 13:23:19 +02:00
trinity.pointard
9a3b4f3a8a
first pass of optimising multi-terms agg
2026-08-04 13:23:19 +02:00
trinity.pointard
261c0a11d4
allow extracting values from IntermediateMultiTermsBucketResult
2026-08-04 13:23:19 +02:00
trinity.pointard
ebd37278ca
first swab at multi-terms impl
2026-08-04 13:23:19 +02:00
Paul Masurel and Paul Masurel
3bb9a430dd
Refactoring TextFastFieldOptions ( #3020 )
...
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com >
2026-08-02 17:17:02 +02:00
Pascal Seitz
667132fa7a
rename to WithLanes
2026-07-29 08:57:54 +02:00
PSeitz
95f2c0c5b6
Optimize low-cardinality term aggregation counters
2026-07-29 08:57:54 +02:00
Pascal Seitz
277e59776b
Extend Vec term storage to 20k with eager/lazy bucket ids
...
Raise MAX_NUM_TERMS_FOR_VEC to 20_000 so the dense Vec term storage
(direct-indexed, no hashing/paging) is used for low/moderate cardinality
both with and without sub-aggregations, not just the <100 low-card case.
Split the old single threshold into MAX_NUM_TERMS_FOR_LOWCARD_SUBAGG (100,
still gates the Vec + LowCard-buffer pairing) and MAX_NUM_TERMS_FOR_VEC.
VecTermBuckets is now generic over a compile-time `const LAZY: bool`:
- eager (default): assigns all ids up front in `new`, branchless term_entry
- lazy: assigns on first occurrence, keeping the sub-agg bucket range down
to the terms that actually occur (no blowup for sparse ordinal spaces)
A new BucketIdSlot::ASSIGNS_ID const plus the LAZY const fully gate the
first-seen branch out at monomorphization, so the only branchy path is the
deliberately-lazy <BucketId, true> instantiation. The with-sub-agg Vec path
picks eager below MAX_NUM_TERMS_FOR_EAGER_BUCKET_IDS (4096) and lazy above,
avoiding the ~6-11% hot-loop regression eager fixes while keeping large
sparse cases compact.
Biggest impact on terms_zipf_1000_only with -40%
```
full
terms_7 Memory: 46.5 KB Avg: 2.3566ms (+4.29%) Median: 2.3566ms (+4.29%) [2.3566ms .. 2.3566ms]
terms_all_unique Memory: 11.5 MB Avg: 4.8532ms (-5.74%) Median: 4.8532ms (-5.74%) [4.8532ms .. 4.8532ms]
terms_all_unique_order_by_key Memory: 11.5 MB Avg: 4.8448ms (-0.53%) Median: 4.8448ms (-0.53%) [4.8448ms .. 4.8448ms]
terms_150_000 Memory: 2.7 MB Avg: 5.5295ms (+0.64%) Median: 5.5295ms (+0.64%) [5.5295ms .. 5.5295ms]
terms_many_top_1000 Memory: 5.3 MB Avg: 8.4738ms (+0.58%) Median: 8.4738ms (+0.58%) [8.4738ms .. 8.4738ms]
terms_many_order_by_term Memory: 2.7 MB (-0.01%) Avg: 4.5959ms (-0.15%) Median: 4.5959ms (-0.15%) [4.5959ms .. 4.5959ms]
terms_many_with_top_hits Memory: 48.9 MB (-0.00%) Avg: 85.8206ms (-5.46%) Median: 85.8206ms (-5.46%) [85.8206ms .. 85.8206ms]
terms_all_unique_with_avg_sub_agg Memory: 54.7 MB (-0.00%) Avg: 17.8747ms (+8.16%) Median: 17.8747ms (+8.16%) [17.8747ms .. 17.8747ms]
terms_many_with_avg_sub_agg Memory: 13.5 MB (-0.00%) Avg: 14.7797ms (+2.54%) Median: 14.7797ms (+2.54%) [14.7797ms .. 14.7797ms]
terms_status_with_avg_sub_agg Memory: 92.1 KB (-0.35%) Avg: 5.1467ms (+0.72%) Median: 5.1467ms (+0.72%) [5.1467ms .. 5.1467ms]
terms_status_with_terms_zipf_1000_sub_agg Memory: 213.2 KB Avg: 3.9576ms (-2.73%) Median: 3.9576ms (-2.73%) [3.9576ms .. 3.9576ms]
terms_zipf_1000_with_terms_status_sub_agg Memory: 726.5 KB (+0.28%) Avg: 12.1432ms (+3.30%) Median: 12.1432ms (+3.30%) [12.1432ms .. 12.1432ms]
terms_status_with_histogram Memory: 139.0 KB Avg: 2.4717ms (+2.45%) Median: 2.4717ms (+2.45%) [2.4717ms .. 2.4717ms]
terms_status_with_date_histogram Memory: 145.9 KB Avg: 2.3644ms (-0.44%) Median: 2.3644ms (-0.44%) [2.3644ms .. 2.3644ms]
terms_status_with_date_histogram_hard_bounds Memory: 145.4 KB Avg: 2.5705ms (+4.70%) Median: 2.5705ms (+4.70%) [2.5705ms .. 2.5705ms]
terms_status_with_date_histogram_and_sibling_terms Memory: 143.3 KB Avg: 3.8946ms (+1.61%) Median: 3.8946ms (+1.61%) [3.8946ms .. 3.8946ms]
terms_zipf_1000_only Memory: 75.8 KB (-0.02%) Avg: 1.2684ms (-41.50%) Median: 1.2684ms (-41.50%) [1.2684ms .. 1.2684ms]
terms_zipf_1000_with_histogram Memory: 1.2 MB (+0.17%) Avg: 20.9323ms (+4.08%) Median: 20.9323ms (+4.08%) [20.9323ms .. 20.9323ms]
terms_zipf_1000_with_avg_sub_agg Memory: 486.7 KB (+2.95%) Avg: 8.5297ms (-1.66%) Median: 8.5297ms (-1.66%) [8.5297ms .. 8.5297ms]
terms_many_json_mixed_type_with_avg_sub_agg Memory: 17.9 MB Avg: 27.0210ms (+1.81%) Median: 27.0210ms (+1.81%) [27.0210ms .. 27.0210ms]
terms_status_with_cardinality_agg Memory: 93.9 KB Avg: 3.2022ms (-0.64%) Median: 3.2022ms (-0.64%) [3.2022ms .. 3.2022ms]
terms_100_buckets_with_cardinality_agg Memory: 9.8 MB (+0.31%) Avg: 49.3188ms (-1.70%) Median: 49.3188ms (-1.70%) [49.3188ms .. 49.3188ms]
terms_many_with_single_term_order_by_card Memory: 48.9 MB Avg: 77.2127ms (-5.02%) Median: 77.2127ms (-5.02%) [77.2127ms .. 77.2127ms]
terms_many_with_single_term_2_order_by_card Memory: 40.6 MB (-0.02%) Avg: 45.8004ms (-17.57%) Median: 45.8004ms (-17.57%) [45.8004ms .. 45.8004ms]
```
2026-07-27 18:00:37 +02:00
PSeitz-dd
70f0b039f5
Merge pull request #2978 from quickwit-oss/mallets/finalize-docidmapping
...
feat: add custom doc id mapping finalization
2026-07-22 15:20:19 +02:00
trinity-1686a
057458bf14
use enum PruneMode instead of bool
2026-07-13 12:16:18 +02:00
trinity-1686a
f05ef0c4cc
cr
2026-07-13 12:16:18 +02:00
trinity-1686a
16dfddf31a
add method to prune intermediate agg results
2026-07-13 12:16:18 +02:00
Pascal Seitz
61693134be
fix cache flush in aggregations
...
fixes #2992
```
full
terms_7 Memory: 37.2 KB Avg: 2.3958ms (+0.31%) Median: 2.3896ms (+0.18%) [2.3573ms .. 2.5176ms]
terms_all_unique Memory: 10.8 MB Avg: 5.5144ms (-1.07%) Median: 5.4625ms (-1.98%) [5.3364ms .. 5.9712ms]
terms_all_unique_order_by_key Memory: 10.8 MB Avg: 5.2614ms (-0.85%) Median: 5.2177ms (-1.21%) [5.0823ms .. 5.6316ms]
terms_150_000 Memory: 2.7 MB Avg: 5.5335ms (-1.07%) Median: 5.5152ms (-1.06%) [5.4151ms .. 5.9654ms]
terms_many_top_1000 Memory: 5.2 MB Avg: 8.3579ms (-1.53%) Median: 8.3604ms (-0.95%) [8.2184ms .. 8.5421ms]
terms_many_order_by_term Memory: 2.7 MB Avg: 4.6713ms (-0.07%) Median: 4.6569ms (-0.15%) [4.5994ms .. 4.9115ms]
terms_all_unique_with_avg_sub_agg Memory: 54.0 MB Avg: 17.4981ms (-2.43%) Median: 17.6075ms (-1.75%) [15.8166ms .. 18.9250ms]
terms_status_with_avg_sub_agg Memory: 90.3 KB Avg: 5.6365ms (+7.77%) Median: 5.6255ms (+7.97%) [5.5489ms .. 5.8254ms]
terms_status_with_terms_zipf_1000_sub_agg Memory: 318.5 KB (+56.52%) Avg: 4.4504ms (+11.55%) Median: 4.4436ms (+11.59%) [4.3858ms .. 4.5692ms]
terms_zipf_1000_with_terms_status_sub_agg Memory: 684.9 KB Avg: 11.8606ms (+0.19%) Median: 11.8360ms (-0.02%) [11.7478ms .. 12.0609ms]
terms_status_with_histogram Memory: 139.5 KB Avg: 2.4524ms (-1.09%) Median: 2.4521ms (-0.23%) [2.4179ms .. 2.5049ms]
terms_status_with_date_histogram Memory: 136.7 KB Avg: 2.3407ms (-1.28%) Median: 2.3359ms (-1.06%) [2.3001ms .. 2.4310ms]
terms_status_with_date_histogram_hard_bounds Memory: 136.1 KB Avg: 2.5113ms (-2.04%) Median: 2.5073ms (-0.97%) [2.4455ms .. 2.7280ms]
terms_status_with_date_histogram_and_sibling_terms Memory: 137.3 KB Avg: 3.8695ms (-0.48%) Median: 3.8653ms (+0.11%) [3.8093ms .. 4.0528ms]
terms_zipf_1000 Memory: 69.8 KB Avg: 2.2022ms (-1.95%) Median: 2.2026ms (-1.12%) [2.1705ms .. 2.2859ms]
terms_zipf_1000_with_histogram Memory: 1.2 MB Avg: 20.4087ms (-0.02%) Median: 20.3665ms (+0.11%) [20.1912ms .. 20.7933ms]
terms_zipf_1000_with_avg_sub_agg Memory: 472.0 KB Avg: 8.7387ms (-3.48%) Median: 8.7043ms (-3.44%) [8.6466ms .. 9.1396ms]
terms_zipf_90 Memory: 55.3 KB Avg: 1.3784ms (-2.04%) Median: 1.3787ms (-1.61%) [1.3484ms .. 1.4611ms]
terms_zipf_90_with_sum_sub_agg Memory: 367.6 KB Avg: 4.8520ms (+8.43%) Median: 4.8326ms (+8.94%) [4.8058ms .. 5.1278ms]
terms_many_json_mixed_type_with_avg_sub_agg Memory: 17.8 MB Avg: 25.0853ms (-7.70%) Median: 25.0591ms (-7.12%) [24.8103ms .. 25.4936ms]
terms_status_with_cardinality_agg Memory: 91.8 KB Avg: 3.3667ms (+1.47%) Median: 3.3690ms (+1.66%) [3.3311ms .. 3.4070ms]
terms_100_buckets_with_cardinality_agg Memory: 9.9 MB Avg: 48.4768ms (-3.07%) Median: 48.3745ms (-3.38%) [48.1425ms .. 49.5503ms]
```
2026-07-12 16:40:05 +02:00
Pascal Seitz
7152d53182
clippy
2026-07-10 12:33:33 +02:00
Pascal Seitz
6b8bd7b884
reorder if block
2026-07-03 17:44:44 +02:00
Pascal Seitz
9db05b660e
fix overflow issue
2026-07-03 17:44:44 +02:00
Pascal Seitz
057e9d6618
make BucketId optional in aggregations
...
If term aggregations don't have sub-aggregations, we don't need to carry
BucketId(u32).
```
full
terms_7 Memory: 46.5 KB Avg: 2.4024ms (+1.96%) Median: 2.4024ms (+1.96%) [2.4024ms .. 2.4024ms]
terms_all_unique Memory: 11.5 MB (-9.93%) Avg: 4.7910ms (-22.62%) Median: 4.7910ms (-22.62%) [4.7910ms .. 4.7910ms]
terms_all_unique_order_by_key Memory: 11.5 MB (-9.94%) Avg: 4.8056ms (-21.31%) Median: 4.8056ms (-21.31%) [4.8056ms .. 4.8056ms]
terms_150_000 Memory: 2.7 MB (-9.90%) Avg: 5.7312ms (-5.96%) Median: 5.7312ms (-5.96%) [5.7312ms .. 5.7312ms]
terms_many_top_1000 Memory: 5.3 MB Avg: 8.5912ms (-3.34%) Median: 8.5912ms (-3.34%) [8.5912ms .. 8.5912ms]
terms_many_order_by_term Memory: 2.7 MB (-9.95%) Avg: 4.5581ms (-10.79%) Median: 4.5581ms (-10.79%) [4.5581ms .. 4.5581ms]
terms_many_with_top_hits Memory: 48.9 MB Avg: 102.3672ms (-2.30%) Median: 102.3672ms (-2.30%) [102.3672ms .. 102.3672ms]
terms_all_unique_with_avg_sub_agg Memory: 54.7 MB Avg: 18.4644ms (-0.11%) Median: 18.4644ms (-0.11%) [18.4644ms .. 18.4644ms]
terms_many_with_avg_sub_agg Memory: 13.5 MB Avg: 16.7345ms (-4.09%) Median: 16.7345ms (-4.09%) [16.7345ms .. 16.7345ms]
terms_status_with_avg_sub_agg Memory: 92.1 KB Avg: 5.1590ms (+0.33%) Median: 5.1590ms (+0.33%) [5.1590ms .. 5.1590ms]
terms_status_with_terms_zipf_1000_sub_agg Memory: 213.2 KB Avg: 3.9562ms (+1.49%) Median: 3.9562ms (+1.49%) [3.9562ms .. 3.9562ms]
terms_zipf_1000_with_terms_status_sub_agg Memory: 724.4 KB Avg: 11.7206ms (-4.68%) Median: 11.7206ms (-4.68%) [11.7206ms .. 11.7206ms]
terms_status_with_histogram Memory: 139.0 KB Avg: 2.3985ms (-1.07%) Median: 2.3985ms (-1.07%) [2.3985ms .. 2.3985ms]
terms_status_with_date_histogram Memory: 145.9 KB Avg: 2.2963ms (-1.86%) Median: 2.2963ms (-1.86%) [2.2963ms .. 2.2963ms]
terms_status_with_date_histogram_hard_bounds Memory: 145.4 KB Avg: 2.4312ms (-1.37%) Median: 2.4312ms (-1.37%) [2.4312ms .. 2.4312ms]
terms_status_with_date_histogram_and_sibling_terms Memory: 143.3 KB Avg: 3.8401ms (+0.44%) Median: 3.8401ms (+0.44%) [3.8401ms .. 3.8401ms]
terms_zipf_1000 Memory: 75.8 KB Avg: 2.1611ms (-2.70%) Median: 2.1611ms (-2.70%) [2.1611ms .. 2.1611ms]
terms_zipf_1000_with_histogram Memory: 1.2 MB Avg: 20.1369ms (-1.16%) Median: 20.1369ms (-1.16%) [20.1369ms .. 20.1369ms]
terms_zipf_1000_with_avg_sub_agg Memory: 472.7 KB Avg: 8.5851ms (-4.93%) Median: 8.5851ms (-4.93%) [8.5851ms .. 8.5851ms]
terms_many_json_mixed_type_with_avg_sub_agg Memory: 17.9 MB Avg: 26.3988ms (-6.61%) Median: 26.3988ms (-6.61%) [26.3988ms .. 26.3988ms]
```
2026-07-03 17:44:44 +02:00
trinity.pointard
715590b357
rename local var
2026-07-02 12:00:00 +00:00
trinity.pointard
d496e402ca
rustfmt
2026-07-02 07:12:04 +00:00
trinity.pointard
348ca1e309
don't count matching doc twice
2026-06-30 16:09:11 +00:00
trinity.pointard
5e4fe3520c
better handle sorted buckets
2026-06-30 14:56:24 +00:00
trinity.pointard
74a510cb56
try to use select-nth instead of full sort in segment level agg top-k selection
2026-06-29 09:13:21 +00:00
Luca Cominardi and Cursor
910861a3e9
feat: add custom doc id mapping finalization
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-06-26 14:45:57 +02:00
Pascal Seitz
1e859fd78d
fix term aggregation u32::MAX overflow issue
2026-06-18 17:07:43 +08:00
Pascal Seitz
c096b2ad89
aggregation/terms: charge fused term_counts to the memory limit
...
term_counts (one u32/term) was allocated but not charged to
AggregationLimitsGuard, so a memory limit could be exceeded silently.
Charge it, skip allocating it when unbounded, and add a regression test.
2026-06-16 21:23:23 +08:00
Pascal Seitz
ac7a3d347c
add comment, hoist variables
2026-06-16 21:23:23 +08:00
Pascal Seitz
03520a0719
add top level comment
2026-06-16 21:23:23 +08:00
Pascal Seitz
86a4c47bed
merge loops, histo with bounds may benefit from single vec opt
2026-06-16 21:23:23 +08:00
Pascal Seitz
3ca510dff0
aggregation/terms: tidy fused term×histogram grid construction
...
Rename the value threaded through build_segment_term_collector and
maybe_build_collector from max_term_id to col_max_val/max_column_val — it
is the column's max value, only later reused as the max term id. Make the
grid-size arithmetic overflow-/zero-safe (saturating_add, checked_div).
2026-06-16 21:23:23 +08:00
Pascal Seitz
3cb400c300
clarify counts/term_counts field docs
...
Spell out that `counts` is the flattened per-term × time-bucket grid (each
term's own contiguous slice) and that `term_counts` is only needed when the
per-term total can't be derived from that grid (i.e. with hard bounds).
2026-06-16 21:23:23 +08:00
Pascal Seitz
ef13489d63
skip hard_bounds that can't exclude any value
...
When a histogram's hard_bounds are wider than the column's value range, the
per-doc `bounds.contains` check can never fail. Collapse such bounds to the
unbounded sentinel in `normalize_histogram_req`, so both the general histogram
hot loop and the fused term×histogram path skip the check — the latter then
derives per-term counts from the grid (the ~17% win) instead of falling back to
per-doc counting just because `bounds != [MIN, MAX]`.
Only the collect-time filter is affected: empty-bucket emission reads
`req.hard_bounds` directly, and hard_bounds only ever clips that range, so a
wider-than-data bound leaves results unchanged. Covered by new tests on the
general and fused paths, including mid-interval (bucket-splitting) bounds.
Also tighten the fused-path u32-overflow guard to bound on `num_vals()` (the
per-value increment count) rather than `num_docs()`, and document why the fused
collector's hot-loop fields are hoisted into locals (re-reading them from memory
each iteration measured ~15% slower).
2026-06-16 21:23:23 +08:00
Pascal Seitz
9f7aea4765
derive term counts
2026-06-16 21:23:23 +08:00
Pascal Seitz
2c8536ab11
add specialized TermHistogram
2026-06-16 21:23:23 +08:00
Pascal Seitz
05f4c02ac5
add dense histogram, optional sub-buckets
2026-06-16 21:23:23 +08:00
Pascal Seitz
d137779219
add no sub-gg fastpath
2026-06-16 21:23:23 +08:00
Pascal Seitz
8f9846ac80
use get_range when possible
2026-06-16 21:23:23 +08:00
Mohammad Dashti
799f7b4646
Built SUM final result in each branch directly.
...
Keeps the empty-bucket coercion visible at the boundary instead of a
shared binding, following the reviewer's suggested shape.
2026-06-16 03:10:30 +08:00
Mohammad Dashti
fc88d80726
docs: drop downstream-specific name from none_if_no_match doc
...
The flag's purpose is described well enough by "SQL-style consumers";
no need to call out a specific downstream.
2026-06-16 03:10:30 +08:00
Mohammad Dashti
6a684e7c38
feat: opt-in none_if_no_match flag on SumAggregation for SQL-style null
...
Switch the default serialized output of `sum` on empty / all-missing
buckets back to `"value": 0` to match Elasticsearch, and gate the
SQL-style `"value": null` behavior behind a new
`none_if_no_match: Option<bool>` flag on `SumAggregation`.
`IntermediateSum::finalize` still returns `Option<f64>` internally so
the Rust API stays parallel to min/max/avg, but the ES-vs-SQL choice is
made at the boundary in `IntermediateMetricResult::into_final_metric_result`:
`None` is coerced to `Some(0.0)` unless `none_if_no_match` is set on the
aggregation request.
Adds `AggregationVariants::as_sum()` accessor for that boundary check
and two end-to-end tests covering both the default ES behavior and the
opt-in null behavior on an empty index.
2026-06-16 03:10:30 +08:00
Mohammad Dashti
94fe52cc67
docs: clarify SUM finalize returning None diverges from Elasticsearch
...
Surface the trade-off in the doc comment so future reviewers see why
this differs from ES (which returns "value": 0 for sum over
empty/all-missing buckets) and what consumers (ParadeDB SQL NULL) the
None variant is meant to serve.
2026-06-16 03:10:30 +08:00
Mohammad Dashti
2ff39f6f7f
fix: return None from SUM when no values were collected
...
IntermediateSum::finalize() returned Some(0.0) even when count==0
(all documents had missing/NULL values). This differs from MIN, MAX,
and AVG which all return None for count==0.
The 0.0 came from IntermediateStats' default sum initialization.
Consumers (like ParadeDB) that map None to SQL NULL were incorrectly
getting 0 for SUM on all-NULL groups.
Fixes paradedb/paradedb#4621
2026-06-16 03:10:30 +08:00
Pascal Seitz
b19f0ddc77
fix clippy
2026-06-09 23:14:12 +08:00
Pascal Seitz
b4acfcf881
cleanup AggregationsSegmentCtx
...
The metric/cardinality/histogram _mut getters had no callers needing
mutation; their two uses already pass the resulting reference as &T.
simplify req_data ownership: clone into collectors, Rc only for filter BitSet
Replace Vec<Option<Box<T>>> + take/put-back round-trip with Vec<T> +
direct clone into collector. Collectors now own their per-segment
request data outright, removing the borrow-checker dance that the
take/put-back pattern existed to satisfy.
The structural clones are cheap (Column<u64> is Arc-internal) except
for the filter aggregation, whose DocumentQueryEvaluator carries a
precomputed per-segment BitSet sized by max_doc. Wrap that in
Rc<DocumentQueryEvaluator> so FilterAggReqData::clone() bumps a
refcount instead of duplicating the BitSet. Move SegmentFilterCollector's
matching_docs_buffer out of FilterAggReqData so its pre-allocated
capacity is preserved per collector instead of being lost on every clone.
2026-06-09 23:14:12 +08:00
trinity-1686a
Paul MasurelandPaul Masurel
Pascal Seitz
PSeitz
PSeitz-dd
Luca CominardiandCursor
Mohammad Dashti