Commit Graph
92 Commits
Author SHA1 Message Date
Pascal Seitz 333b0d4948 Inline short postings in SSTable term dictionaries 2026-10-27 08:56:57 +01:00
Paul Masurel 67889269ca Version bump and changelog for tantivy 0.27 (#3197) 2026-10-17 08:32:20 +02:00
Paul Masurel fc5770124e Fix clippy warnings (#3165)
* Fix clippy warnings
2026-10-08 18:23:57 +02:00
Paul MasurelandPaul Masurel e988ecb7a1 Add without_keys() to term dictionary streamers (#3155)
* Add keyless streaming to the term dictionary streamers

The sstable streamer rebuilds the key of each entry it reads. It does
this also for the entries that the automaton rejects. Many callers do
not read the key. They only use term infos or term ordinals.

Add `without_keys()` to the streamer builders. The resulting streamer
does not rebuild keys. It gives term ordinals and values only.

- `KeyTracking` (in tantivy-common) is the buffer for the current key.
  `Vec<u8>` keeps the key. `WithoutKeys` does nothing.
- The streamer types get a generic parameter `K = Vec<u8>`. Existing
  type names keep the keyed behavior. `key()` and `next()` exist only
  on the keyed streamer. The keyless streamer has `next_without_key()`.
- At the lower bound, the streamer does not know the previous key. It
  uses the lower bound prefix that the comparator found as the
  previous key. It then checks the upper bound on this prefix and the
  suffix, with the new `DeltaKeyComparator::compare_prefix_and_suffix`.
- The FST streamer uses the same `KeyTracking`. Thus the
  `TermDictionary` facade has `without_keys()` on both backends.

These callers now use `without_keys()`: AutomatonWeight,
InvertedIndexRangeWeight, the quickwit warmup paths and the terms
aggregation include/exclude regex.

* CR comment

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-07 14:54:51 +02:00
trinity-1686a 171340677f address cr 2026-09-29 12:04:38 +02:00
trinity-1686a 6c311b311d use will_always_match hint 2026-09-24 19:09:25 +02:00
trinity-1686a 2bb52f1460 hide some branches behind const generics 2026-09-24 18:46:21 +02:00
trinity-1686a bec1b08bdf use delta-based prefix matcher
it does less slice comparisons overall, which improves throughput
2026-09-24 18:46:21 +02:00
trinity-1686a aac95014b3 remove lower_bound check from streamer hotloop 2026-09-24 18:46:21 +02:00
trinity-1686a 95dfcd3ada add bench for automaton sstable streaming 2026-09-24 18:46:15 +02:00
Pascal Seitz bbc1b9e1d1 Restore TerminatingWrite API 2026-09-16 23:39:47 +08:00
Pascal Seitz 98d91f0436 Remove intermediate flush from directory contract
Intermediate writes are an implementation detail. From Tantivy side we only decide when a file is finished. This change is done to remove flush overhead for VecWriter. On finalization we just move the Vec to the directory.

Rename TerminatingWrite to FinishableWrite to reflect these semantics.
2026-09-16 23:39:47 +08:00
dependabot[bot] dd0bc23360 Update zstd requirement from 0.13 to 0.14 (#3085)
Updates the requirements on [zstd](https://github.com/gyscos/zstd-rs) to permit the latest version.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: zstd
  dependency-version: 0.14.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-10 18:36:24 +02:00
Paul MasurelandPaul Masurel ed6a90e840 Minor change: removing BooleanWeight::new as it is quite the footgun (#3071)
BooleanWeight::new just default min_should_match to 1.
This is misleading because I think someone who passes a single must clause would expect the result
to match that clause.

Due to min_should_match, it actually returns nothing.

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-09-02 13:47:51 +02:00
ildis 7f6685f5d6 fix(sstable): report the real term ordinal when an automaton prunes blocks
`Streamer::term_ord()` derived the ordinal by counting `delta_reader.advance()`
calls from a seed taken only when the stream had a key lower bound. An automaton
search has no key bounds, so the seed was 0 — but the reader it is handed *is*
block-pruned by that automaton (`get_block_iterator_for_range_and_automaton`).
Every block skipped ahead of the first match went uncounted, so `term_ord()`
returned the term's position among the blocks actually scanned rather than its
ordinal in the dictionary.

The error is silent and grows with how deep the first match sits: on a 262k-term
dictionary, a regex matching only the last key reported ordinal 999 instead of
262142. Callers that resolve those ordinals back to terms therefore act on a
different term entirely — `build_allowed_term_ids_for_str` builds the terms
aggregation's allowed-ordinal bitset this way, so an `include` regex could make
the aggregation count unrelated terms while returning the expected bucket count.
Broad patterns matching from the start of the dictionary hid it, since nothing is
pruned ahead of the first match.

Each slice handed to the reader now carries the ordinal of its first term, and
the streamer resets to it on entering a slice instead of incrementing across the
gap. Blocks merged into one slice stay contiguous, so counting within a slice is
still correct.

`Dictionary::sorted_ords_to_term_cb` already tracked `BlockAddr::first_ordinal`
explicitly, which is why ordinal->term resolution was unaffected.
2026-08-04 21:04:17 +03:00
trinity-1686a 95d8a3989a cr 2026-05-19 11:38:48 +02:00
trinity-1686a ea61a68db4 skip sstable index binary search when ordinal is in same block 2026-05-16 11:35:38 +02:00
trinity-1686a c367df37c1 refactor sstable index 2026-05-16 11:30:02 +02:00
dependabot[bot] 2f5a48e8b1 Update criterion requirement from 0.5 to 0.8 (#2873)
Updates the requirements on [criterion](https://github.com/criterion-rs/criterion.rs) to permit the latest version.
- [Release notes](https://github.com/criterion-rs/criterion.rs/releases)
- [Changelog](https://github.com/criterion-rs/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/criterion-rs/criterion.rs/compare/0.5.0...criterion-v0.8.2)

---
updated-dependencies:
- dependency-name: criterion
  dependency-version: 0.8.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-25 14:15:53 +02:00
Paul Masurel 04beab3b29 Performance improvement for nested cardinality aggregation
When a string cardinality aggregation is nested it end up being applied to different buckets.
Dictionary encoding relies on a different dictionaries for each segment.

As a result, during segment collection, we only collect term ordinals in a HashSet, and decode them in the
term dictionary at the end of collection.

Before this PR, this decoding phase was done once for each bucket, causing the same work to be done over and over. This PR introduce a coupon cache. The HLL sketch relies on a hash of the string values.

We populate the cache before bucket collection, and get our values from it.

This PR also rename "caching" "buffering" in aggregation (it was never caching), and does several cleanups.
2026-04-10 14:51:00 +02:00
Pascal Seitz 5c344db1bf chore: Release 2026-03-31 17:15:34 +08:00
545169c0d8 Composite agg merge (#2856)
Add composite aggregation

Co-authored-by: Remi Dettai <remi.dettai@sekoia.io>
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-03-18 17:28:59 +01:00
PSeitz 57fe659fff make serializer pub (#2835)
some changes on the posting list serializer to make it usable in
other contexts.

Improve errors

Signed-off-by: Pascal Seitz <pascal.seitz@gmail.com>
2026-02-11 14:37:42 +01:00
trinity-1686a 12977bc7c4 upgrade some dependancies (#2802)
including rand, which had a few breaking changes
2026-01-14 10:19:09 +01:00
PSeitz-dd c6912ce89a Handle JSON fields and columnar in space_usage (#2761)
return field names in space_usage instead of `Field`
more detailed info for columns
2025-12-10 20:33:33 +08:00
Ang 08a92675dc Fix typos again (#2753)
Found via `codespell -S benches,stopwords.rs -L
womens,parth,abd,childs,ond,ser,ue,mot,hel,atleast,pris,claus,allo`
2025-12-01 12:15:41 +01:00
PSeitzandPascal Seitz 43a784671a clippy (#2741)
Co-authored-by: Pascal Seitz <pascal.seitz@datadoghq.com>
2025-11-21 18:07:03 +01:00
PSeitzandPascal Seitz 85010b589a clippy (#2700)
* clippy

* clippy

* clippy

* clippy + fmt

---------

Co-authored-by: Pascal Seitz <pascal.seitz@datadoghq.com>
2025-09-19 18:04:25 +02:00
PSeitz-dd 2340dca628 fix compiler warnings (#2699)
* fix compiler warnings

* fix import
2025-09-19 15:55:04 +02:00
PSeitzandPascal Seitz 33794a114c chore: Release (#2686)
Co-authored-by: Pascal Seitz <pascal.seitz@datadoghq.com>
2025-08-20 18:29:37 +08:00
Stu Hood d9eb093368 Attempt to clarify sorted_ords_to_term_cb. 2025-07-29 21:56:31 -07:00
Eric Ridge 2f01152a3c adjust Dictionary::sorted_ords_to_term_cb() to allow duplicates 2025-07-16 13:38:43 -07:00
PSeitz 080fa4d1f4 add docs/example and Vec<u32> values to sstable (#2660) 2025-07-01 15:40:02 +02:00
PSeitzandPascal Seitz 4a6123d3ff release tantivy: bump versions (#2625)
* chore: Release

* chore: Release

---------

Co-authored-by: Pascal Seitz <pascal.seitz@datadoghq.com>
2025-06-10 15:34:39 +02:00
Parth 5a2fe42c24 make zstd optional in sstable (#2633)
* make zstd truly optional

* changelog notes

* make sure we write

* resolve comments

* make this a default feature

* remove changelog notes
2025-05-14 17:16:41 +02:00
PSeitz 5379c99ea2 update edition to 2024 (#2620)
* update common to edition 2024

* update bitpacker to edition 2024

* update stacker to edition 2024

* update query-grammar to edition 2024

* update sstable to edition 2024 + fmt

* fmt

* update columnar to edition 2024

* cargo fmt

* use None instead of _
2025-04-18 04:56:31 +02:00
Paul Masurel 519e5d2ed1 clippy warnings 2025-03-05 11:15:06 +01:00
dependabot[bot] 43c89b4360 Update itertools requirement from 0.13.0 to 0.14.0 (#2563)
Updates the requirements on [itertools](https://github.com/rust-itertools/itertools) to permit the latest version.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-08 17:11:46 +01:00
trinity-1686a d281ca3e65 Merge pull request #2559 from quickwit-oss/trinity/sstable-partial-automaton
allow warming partially an sstable for an automaton
2025-01-08 16:35:35 +01:00
Remi Dettai 71cf19870b Exist queries match subpath fields (#2558)
* Exist queries match subpath fields

* Make subpath check optional

* Add async subpath listing
2025-01-06 10:17:39 +01:00
trinity Pointard fe0c7c5408 change rangebound style 2025-01-02 11:56:05 +01:00
trinity Pointard dfff5f3bcb rename merge_holes_under => merge_holes_under_bytes 2024-12-23 16:17:44 +01:00
trinity-1686a ebf4d84553 add comment about cpu-intensive operation in async context 2024-12-20 12:23:49 +01:00
trinity-1686a 42efc7f7c8 clippy 2024-12-20 11:00:11 +01:00
trinity-1686a 192395c311 attempt at simplifying can_block_match_automaton 2024-12-20 10:25:38 +01:00
trinity-1686a a1447cc9c2 remove breaking change in sstable public api 2024-12-19 17:30:05 +01:00
trinity-1686a 24c5dc2398 allow warming up automaton 2024-12-10 13:32:12 +01:00
trinity-1686a 9e2ddec4b3 merge adjacent block when building delta for automaton 2024-12-10 13:32:12 +01:00
trinity-1686a 1f6a8e74bb support iterating over partially loaded sstable 2024-12-10 13:32:12 +01:00
trinity-1686a 7e901f523b get iter for blocks of sstable matching automaton 2024-12-10 13:32:12 +01:00