Commit Graph
3845 Commits
Author SHA1 Message Date
Luca Cominardi bd875fd734 Store the jitexpr term ord results in boxed slices
The slots of `TermOrdResults` are allocated once, when the source is
opened, and never grow: store them in boxed slices rather than vectors.
`vec!` of zeroes still allocates zeroed memory, and converting it does not
reallocate.

Also document why there is one slot more than terms: slot 0 holds the
result for the docs without a value, and comes first so that a corrupted
term ord is out of bounds instead of reading it.
2026-11-02 14:59:36 +01:00
Luca Cominardi 55ecc9c56e Rename the jitexpr term ord memo to TermOrdResults
`TermOrdMemo` named the technique, not what it holds: the results of the
expression per input term ord. Rename it `TermOrdResults`, the field
`term_ord_results`, the limit `TERM_ORD_RESULTS_MAX_NUM_TERMS`, and the
slot states `RESULT_UNKNOWN`/`RESULT_NULL`/`RESULT_VALUE`. No behavior
change.
2026-11-02 14:59:36 +01:00
Luca Cominardi 4880970cc2 Remove the jitexpr aggregation benchmark
Keep the PR focused on the memo; the benchmark numbers in the PR
description were measured with it.
2026-11-02 14:59:36 +01:00
Luca Cominardi 6faa2fecf6 Simplify the jitexpr term ord memo
- Replace `TermOrdMemo::get`/`set` by `get_or_eval`, and store the result
  for a doc without an input value in slot 0 (term ord `o` in slot `o + 1`)
  instead of a separate field.
- Index the memo directly: a term ord out of the input dictionary is a
  corrupted column, which panics like the rest of the module.
- Destructure the source once per block, and evaluate misses with a free
  `eval_single_input`, instead of a per-doc `memoized_result` method.
- Share the cardinality tail of the two `load_block` paths, and implement
  `InputColumn::load_block` on top of `load_block_values`.
- Drop the test-only memo threshold: the limit test builds a dictionary of
  `TERM_ORD_MEMO_MAX_NUM_TERMS + 1` terms, and the other tests exercise the
  memo for their small dictionaries.

jitexpr_agg_bench: terms on (REGEXP_EXTRACT service ..) 7.7 ms -> 6.8 ms,
on (UPPER service) 8.0 ms -> 6.8 ms; the other cases are unchanged.
2026-11-02 14:59:35 +01:00
Luca Cominardi 24f329fcaa Memoize jitexpr value sources per input term ord
A jitexpr value source evaluated its expression once per doc. For an
expression of a text input, e.g. `(REGEXP_EXTRACT service "^([^-]+)" 1u64)`,
the result only depends on the term of the doc, and text inputs are
dictionary encoded with typically few distinct terms per segment (service
names, hosts, statuses). Evaluating the expression per doc made a terms
aggregation over such a calculated field ~100x slower than over the
physical column; on a CloudPrem searcher, the expression accounted for 91%
of the CPU of such a query.

Memoize the results of an expression of a single text input whose
dictionary has at most 65,536 terms, per input term ord. The expression is
evaluated on the first doc of each term, and later docs read the encoded
result from two dense arrays allocated zeroed when the source is opened
(9 bytes per term), plus a slot for docs without a value. Terms are only
resolved for the term ords evaluated for the first time. jitexpr functions
are deterministic, and `Str` results are interned in the order of the docs
as before, so the values and term ords are unchanged. `to_f64` clears the
memo, whose results are encoded for the previous result type. Other
expressions keep the per-doc path.

benches/jitexpr_agg_bench.rs (2M docs, 4 segments, 200 services):

  terms on (REGEXP_EXTRACT service ..)     140.6 ms -> 7.7 ms   (18x)
  terms on (UPPER service)                 305.7 ms -> 8.0 ms   (38x)
  terms on (REGEXP_EXTRACT request_id ..)  1.441 s  -> 1.444 s  (unique terms, no memo)
  terms on physical service                1.08 ms  -> 1.07 ms
2026-11-02 14:59:35 +01:00
Paul Masurel 286179a1a7 jitexpr: parse string expressions as f64 in numeric aggregations
A jitexpr field that can only return a string (e.g. REGEXP_EXTRACT) used
to fail in aggregations requiring a number (stats, histogram, range,
percentiles, ...). It is now parsed as f64. The doc has no value if the
string cannot be parsed, or is not finite.

- jitexpr: lower the Str -> F64 coercion with a native parse function.
- jitexpr: accept Str in CompiledFn::coerce_result_to_f64.
- jitexpr: add infer_result_types.
- JitExprValueSourceProvider: coerce string-only expressions when the
  aggregation accepts f64 but not strings.
2026-11-02 14:59:35 +01:00
Paul Masurel 360a5ebbbb Add ExpUnrolledLinkedList::iter_chunks (#3218)
* Add ExpUnrolledLinkedList::iter_chunks

Adds an iterator over the chunks of an ExpUnrolledLinkedList, one slice
per block, in write order.

read_to_end now goes through ExpUnrolledLinkedListChunks::read_to_end,
which keeps the plain block loop instead of calling next() per chunk.
It also reserves the exact output length upfront, computed from the
block metadata.

* Export ExpUnrolledLinkedListChunks and make read_exact_into public

So callers can name the type returned by iter_chunks, and use
read_exact_into from outside the crate.
2026-10-27 16:04:24 +01:00
Paul Masurel 6bced0ba02 Add the plain codec format to the columnar (#3179)
* Add the plain codec format to the columnar

Adds StrColumnCodec, the column key code of plain columns, the V3
columnar version, and the plain column storage: OnPair16 blocks of at
most 65,536 values and 1 MiB, block index, and reader.

The columnar writer does not produce plain columns yet.

* less result moar panic

* CR Comment. Making PlainColumn Inner type private

* CR comment: checked sorted index in block index.
2026-10-26 11:09:47 +01:00
Yang Baiandybai08 66f8302256 Count chars, not bytes, for the snippet max_num_chars (#3199)
`SnippetGenerator::set_max_num_chars` and `search_fragments` document the
limit as a number of chars (not bytes), but fragments were cut as soon as
their byte length went over it. For non-ASCII text, snippets were therefore
much shorter than requested: about half for Cyrillic or Greek, a third for
CJK.

Count the chars of the fragment instead. Since a char is at least one byte,
the chars only need counting once the byte length exceeds the limit, so ASCII
text does no extra work.

Co-authored-by: ybai08 <271969568+ybai08@users.noreply.github.com>
2026-10-26 10:05:00 +01:00
palmoni5andKleiKodesh df47456ca8 Export PhraseScorer from tantivy::query (#3210)
* Export PhraseScorer from tantivy::query

PhraseScorer already has a public constructor and the Postings trait is public, but the type was only reachable inside the crate. Exporting it lets a custom Query/Weight score a phrase over any Postings, for example a union of several terms at one position, without copying the phrase matching code.

Also documents the struct, the constructor and phrase_count.

* Document that PhraseScorer::new needs two postings

Intersection::new asserts at least two docsets, so a custom Weight whose phrase degenerates to one term must branch before building the scorer.

---------

Co-authored-by: KleiKodesh <211884665+KleiKodesh@users.noreply.github.com>
2026-10-26 10:02:12 +01:00
Pascal Seitz b9c4ed1c62 Rename encoded postings buffer to encoded terminfo 2026-10-23 17:11:03 +02:00
Pascal Seitz 85c15c83f8 Avoid rescanning terms during automaton bitset warmup 2026-10-23 17:11:03 +02:00
Pascal Seitz 9f2f1b2f6c Fix snippet panic with overlapping tokens 2026-10-22 15:10:02 +02:00
Paul Masurel 67889269ca Version bump and changelog for tantivy 0.27 (#3197) 0.27.0 2026-10-17 08:32:20 +02:00
Adrien Guillo e1acb3e011 perf(aggregation): break serial Kahan/Welford chains with accumulator lanes (#3180)
The metric reducers folded values one at a time through recurrences that
carry state across every iteration: Kahan compensated summation in
`collect_stats` (sum/avg/min/max/count/stats) and Welford's online variance
in `extended_stats`. These strict dependency chains block both CPU
pipelining and auto-vectorization.

`collect_stats` now reduces each block of values as a slice into 4
independent Kahan (sum, delta) lanes plus 4 min/max lanes, combined at the
end of the block.

`extended_stats` accumulates each block into 4 independent
`IntermediateExtendedStats` lanes and folds them into the bucket with the
existing `merge_fruits` (Chan parallel-variance combination), the same
operation used to merge segment results. The lanes carry the bucket's sigma,
since `merge_fruits` replaces an empty bucket with the incoming lane.

Blocks with fewer than two values per lane are collected value by value,
where setting up and combining lanes costs more than it saves.

Only the floating-point summation order changes, as it already does across
segment merges. All aggregation tests pass within the existing `2e-12`
tolerance, including the rounding-sensitive `test_aggregation_level1`, and a
new test covers sigma through the lane path.
2026-10-15 10:20:13 -04:00
ZainnQureshiiandClaude Opus 5.5 8a4cfc40f5 Fix stale JSON range query note and ARCHITECTURE.md link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:47:29 +02:00
Paul MasurelandPaul Masurel 7fea2ab8d8 columnar: read datetime columns with the right type in sort_order (#3185)
The datetime hash map stores `ColumnWriter` values, but `sort_order`
read them as `NumericalColumnWriter`, a different and larger type.
`ArenaHashMap::get` reads raw bytes, so this read garbage and could read
past the end of the value. It is reachable through index sorting on a
date field.

Read a `ColumnWriter` and iterate its `NumericalValue` operations, as
`serialize` already does.

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-14 12:40:04 +02:00
Paul Masurel 0f8f6c5a9f term_agg: use bounds-checked indexing in Vec term maps (#3182)
`VecTermBuckets` and `LazyVecTermBuckets` used `get_unchecked_mut` on
the term id. The bucket vector is sized from the column's stored
`max_value`, which is read from disk and never checked against the
decoded values. A corrupt or crafted column could cause an
out-of-bounds write.

Use plain indexing instead. A corrupt column now panics, like
`PagedTermMap` already does.

agg_bench (terms without sub-aggregations, 4 index types, 2 rounds):
all differences vs main are within noise (-3.8% to +1.5%).
2026-10-14 09:05:12 +02:00
Paul Masurel 881fb3e02c filter_vec avx2: make target_feature helpers safe (#3178)
u32_to_i32_avx2, compact and compute_filter_bitset only use AVX2 intrinsics
on values, with no pointers. Since Rust 1.86, a #[target_feature] fn can be
safe, and calling it from a context with the same feature needs no unsafe
block. u32_to_i32_avx2 also gains its missing #[target_feature(enable = "avx2")].

The unsafe blocks in filter_vec_avx2_aux now only cover the pointer
load, store and offsets.
2026-10-13 15:39:12 +02:00
Raphaël Cohen a32be82a9e fix: Fall back to string for dates that don't fit in i64 nanoseconds (#3156)
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
2026-10-13 15:32:18 +02:00
Paul Masurel c6dc5cb90e BitUnpacker::get_range: compute range bounds in u64 (#3177)
On 32-bit targets, the usize bound computation could wrap. The range check
then passed and the fast path read out of bounds. In u64, it cannot
overflow. On 64-bit targets, the generated code is the same.
2026-10-13 15:31:50 +02:00
Paul MasurelandPaul Masurel 59daaef723 Split term bucket maps into eager (lanes) and lazy variants (#3174)
Replace the `LAZY_BUCKET_ID_GENERATION` const bool on `VecTermBuckets`
with two focused types:

- `VecTermBuckets<B, const LANES: usize = 1>`: eager bucket-id assignment,
  with `LANES` count lanes. `LANES == 1` is the former scalar-eager map;
  `LANES == 8` (via the `VecTermBucketsWithLanes` alias) the lane map.
- `LazyVecTermBuckets<B>`: lazy bucket-id assignment on first occurrence,
  single counter. Lazy only applies above
  `LAZY_BUCKET_ID_GENERATION_THRESHOLD`, which never overlaps the very-low
  cardinality lane regime, so it needs no lanes.

This removes the const bool and its awkward `LAZY && !B::ASSIGNS_ID`
edge case: each `term_entry` now does exactly one thing (eager rotates
across lanes; lazy assigns on first use), so both hot loops are simpler.

`VecTermBucketsWithLanes` is kept as a type alias
(`= VecTermBuckets<B, LANES>`) so the lane call site and the existing
lane unit test are unchanged. Dispatch in term_agg and multi_terms now
selects `LazyVecTermBuckets` for the lazy branch.

No behavior change: all term and multi-terms aggregation tests pass.

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-13 15:31:02 +02:00
Paul Masurel 3ecca8151a MemoryArena: panic when allocating more than a page (#3176)
add_page used to accept any length, leaving the page with a len larger than
its 1MB buffer. The module documentation already says that allocating more
than 1MB panics. The check is in add_page, which is off the hot path.
2026-10-13 15:21:22 +02:00
Paul Masurel 6fc5a7d9b0 Unsafe hardening: fix unsound unsafe sites (#3175)
* Hardening unsafe in substring_count

* Remove unsafe in StrColumn::ord_to_str

ord_to_term may leave arbitrary bytes in the buffer when the term is not
found or on error, which left an invalid UTF-8 String behind.
The buffer is now only put back into the String after UTF-8 validation.
On not found or error, the output is left empty.

* Make CompactDoc::node_data crate private

extract_str relies on string values in node_data being valid UTF-8.
With a public field, safe code could break this invariant.

* Remove unsafe in Term::get_json_path

The public Term API (append_bytes, set_bytes, Term::wrap, serde, ...) can
put arbitrary bytes in the json path. Invalid UTF-8 sequences are now
replaced by U+FFFD.

* LZ4 doc store: never hand back a buffer with uninitialized bytes

compress/decompress now work on a local Vec while it contains
uninitialized bytes, and clear it on error. On error or panic, the
caller's buffer is left empty. Buffer reuse is preserved.
2026-10-13 14:52:34 +02:00
Paul MasurelandPaul Masurel 9f9030a5fa Adding a Jitexpr ValueSource to allow jitexpr in aggregations. (#3162)
* Adding a jitexpr ValueSource

* CR comments

* Bugfix

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-13 10:59:52 +02:00
Paul MasurelandPaul Masurel 0ae0df6b8c Rename ValueSource::as_column into as_physical_column (#3171)
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-12 11:08:16 +02:00
nadav-govari 6783439906 Oversized dictionaries return error (#3169)
* Oversized dictionaries return error

* Preexisting test fixes
2026-10-09 17:20:45 -04:00
Paul MasurelandPaul Masurel d6f0e2c320 Prepare ValueSource for computed text sources (#3160)
* Prepare ValueSource for computed text sources

- `ValueSource::term_dictionary()` lets a source resolve its own term
  ords (`ValueSourceDictionary`, implemented by the sstable dictionary).
  Terms and cardinality resolve ords through it; a registered name never
  uses the physical dictionary of the same field.
- `ValueSourceProvider::for_segment` receives the column types the
  aggregation accepts.
- composite and multi_terms reject registered sources explicitly.

* CR comment

* CR comment

* CR comment

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-09 18:07:04 +02:00
Paul MasurelandPaul Masurel 17ddaa08c5 Better rank for sparse columns (#3159)
* Faster lookups in sparse optional index blocks

- Branchless binary search in SparseBlock, using
  std::hint::select_unpredictable. Indices are usize: with u16, LLVM
  emits a branch on x86_64, even with the hint.
- SparseBlock::rank_if_exists_batch checks the next value first, then
  uses an exponential search. The search is bounded by the doc id gap:
  values are distinct and sorted, so vals[c + k] >= vals[c] + k.
- OptionalIndex::rank_if_exists_batch reserves its output vectors.
- MSRV 1.88, for select_unpredictable.
- Add a rank_if_exists_batch group to bench_optional_index, unit tests
  and proptests.

* CR comments

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-09 10:11:56 +02:00
Paul MasurelandPaul Masurel c12cc2d54b Add VersatileBuffer, a reusable allocation lent as a Vec (#3166)
`VersatileBuffer<A>` (tantivy-common) keeps an allocation and lends it
as an empty `Vec<T>`, for any `T` with the size and alignment of `A`.
The allocation goes back to the buffer when the guard is dropped,
including on panic. If the guard is leaked, the buffer starts again
from an empty `Vec`.

This makes it possible to reuse a buffer of values borrowing data with
a short lifetime, without self-referential structs.

The jitexpr predicate uses it for its argument slots. It replaces the
`ClearOnDrop` guard, which transmuted a `Vec<VariableValue<'static>>`
to a shorter lifetime.

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-08 18:47:36 +02:00
Paul Masurel fc5770124e Fix clippy warnings (#3165)
* Fix clippy warnings
2026-10-08 18:23:57 +02:00
Paul Masurel c091be8597 Remove allocation in Intersection::seek (#3164)
`Intersection::seek` used to build a `Vec<&mut dyn DocSet>` on every call
to reuse `go_to_first_doc`. It now shares the leapfrog loop of `advance`,
which uses `seek_danger` on the secondary docsets and does not allocate.

Added a proptest mixing `seek` and `advance` on a 3-way intersection.
2026-10-08 18:15:40 +02:00
Pascal Seitz 83a1faaca2 Preserve document ordering for nested aggregations 2026-10-08 17:48:01 +02:00
Pascal Seitz ef8e047db0 Return matching document bitset from automaton warmup 2026-10-08 17:21:43 +02:00
Pascal Seitz 4dfcac7628 Return matching term infos from automaton warmup 2026-10-08 17:21:43 +02:00
Paul MasurelandPaul Masurel e988ecb7a1 Add without_keys() to term dictionary streamers (#3155)
* Add keyless streaming to the term dictionary streamers

The sstable streamer rebuilds the key of each entry it reads. It does
this also for the entries that the automaton rejects. Many callers do
not read the key. They only use term infos or term ordinals.

Add `without_keys()` to the streamer builders. The resulting streamer
does not rebuild keys. It gives term ordinals and values only.

- `KeyTracking` (in tantivy-common) is the buffer for the current key.
  `Vec<u8>` keeps the key. `WithoutKeys` does nothing.
- The streamer types get a generic parameter `K = Vec<u8>`. Existing
  type names keep the keyed behavior. `key()` and `next()` exist only
  on the keyed streamer. The keyless streamer has `next_without_key()`.
- At the lower bound, the streamer does not know the previous key. It
  uses the lower bound prefix that the comparator found as the
  previous key. It then checks the upper bound on this prefix and the
  suffix, with the new `DeltaKeyComparator::compare_prefix_and_suffix`.
- The FST streamer uses the same `KeyTracking`. Thus the
  `TermDictionary` facade has `without_keys()` on both backends.

These callers now use `without_keys()`: AutomatonWeight,
InvertedIndexRangeWeight, the quickwit warmup paths and the terms
aggregation include/exclude regex.

* CR comment

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-07 14:54:51 +02:00
Luca Cominardi 2e87c2eb0c fix: make fmt 2026-10-07 11:47:25 +02:00
Luca Cominardi 7daaa5dfec chore: tiebreaker cleanup 2026-10-07 11:47:25 +02:00
Luca Cominardi c3e40c63f7 chore: tiebreaker tests cleanup 2026-10-07 11:47:25 +02:00
Luca Cominardi 042a1b3ed8 Generate tie-breakers directly in i32 range 2026-10-07 11:47:25 +02:00
Luca Cominardi 224f16b60c Generate tie-breakers as i64 values 2026-10-07 11:47:25 +02:00
David Yaffe 1783018f9e Pin GitHub Actions runner images (#3153)
Replace floating runner labels with the images they resolve to today
(`ubuntu-latest` -> `ubuntu-24.04`), so OS upgrades happen in an explicit
change instead of whenever GitHub moves a `-latest` label. GitHub moves
`ubuntu-latest` to Ubuntu 26.04 between Oct 19 and Nov 19, 2026.
2026-10-05 16:19:24 -04:00
Paul MasurelandPaul Masurel 45bbb16542 Refactoring to make ValueSource::load_block take &mut self (#3147)
* Make ValueSource::load_block take &mut self

Value sources were held as `Arc<dyn ValueSource>` and cloned into their
collector, so `load_block` could only take `&self`. A computed source
could not keep per-segment state (caches, scratch buffers) across blocks.

Collectors were built in two phases: `build_nodes` pushed each
`XxxAggReqData` into per-kind vectors of `AggregationsSegmentCtx`, and
each collector builder cloned its request data out of those vectors by
index. The ctx copy had to stay around for name lookups, memory
accounting, and for the collectors that read their data back by index.

Each request node is turned into exactly one collector, so nothing
actually shares a value source. This change makes that ownership
explicit:

- `ValueSource::load_block` takes `&mut self`,
  `ValueSourceProvider::for_segment` returns `Box<dyn ValueSource>`.
- The request tree (`AggNode { data: AggNodeData, children }`) owns the
  request data. Building collectors consumes the tree, moving the
  request data into the collectors instead of cloning it.
- `AggregationsSegmentCtx` only keeps the shared collect-time state
  (context and block accessor). `PerRequestAggSegCtx`, `AggKind`,
  `idx_in_req_data` and the `push_*`/`get_*_req_data` helpers are gone.
- Cardinality, percentiles, top_hits and missing_term collectors own
  their request data instead of reading it from the ctx by index.
- Histogram requests are normalized once, when building the tree. The
  flattened terms x histogram path is split into a borrow-only plan step
  and a consuming build step, and no longer clones the histogram request
  at finalization.
- Request data memory is charged exactly once per node. Previously it
  was charged at every nesting level, and range, histogram, filter and
  composite builders charged their request data a second time.
- `FilterAggReqData::evaluator` no longer needs an `Rc`, and the
  filter, composite and multi_terms request data no longer derive Clone.

* CR comment

* CR comment

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-05 15:13:44 +02:00
Pascal Seitz 3db020e707 Clarify write position when tracking multivalued documents 2026-10-05 14:21:30 +02:00
Pascal Seitz 98fba246db Extract per-document sorting for deduplication 2026-10-05 14:21:30 +02:00
Pascal Seitz f18eb5df53 Use shared value fetching entry point for bucket collectors 2026-10-05 14:21:30 +02:00
Pascal Seitz 36884928c5 Cache block multivalued state during collection 2026-10-05 14:21:30 +02:00
Pascal Seitz 045b210e3b Deduplicate multivalued documents within histogram buckets 2026-10-05 14:21:30 +02:00
Pascal Seitz 7122be03e6 Deduplicate multivalued documents within range buckets 2026-10-05 14:21:30 +02:00
Pascal Seitz 8e5f1cef03 Document aggregation collector call contracts 2026-10-05 14:21:30 +02:00