Files
tantivy/common/src/key_tracking.rs
T
Paul MasurelandPaul Masurel e988ecb7a1 Add without_keys() to term dictionary streamers (#3155)
* Add keyless streaming to the term dictionary streamers

The sstable streamer rebuilds the key of each entry it reads. It does
this also for the entries that the automaton rejects. Many callers do
not read the key. They only use term infos or term ordinals.

Add `without_keys()` to the streamer builders. The resulting streamer
does not rebuild keys. It gives term ordinals and values only.

- `KeyTracking` (in tantivy-common) is the buffer for the current key.
  `Vec<u8>` keeps the key. `WithoutKeys` does nothing.
- The streamer types get a generic parameter `K = Vec<u8>`. Existing
  type names keep the keyed behavior. `key()` and `next()` exist only
  on the keyed streamer. The keyless streamer has `next_without_key()`.
- At the lower bound, the streamer does not know the previous key. It
  uses the lower bound prefix that the comparator found as the
  previous key. It then checks the upper bound on this prefix and the
  suffix, with the new `DeltaKeyComparator::compare_prefix_and_suffix`.
- The FST streamer uses the same `KeyTracking`. Thus the
  `TermDictionary` facade has `without_keys()` on both backends.

These callers now use `without_keys()`: AutomatonWeight,
InvertedIndexRangeWeight, the quickwit warmup paths and the terms
aggregation include/exclude regex.

* CR comment

---------

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
2026-10-07 14:54:51 +02:00

87 lines
2.5 KiB
Rust

/// Initial capacity of the `Vec<u8>` key buffer, large enough for most keys.
const DEFAULT_KEY_CAPACITY: usize = 100;
/// Buffer in which a term dictionary streamer keeps track of the current key.
///
/// - `Vec<u8>` keeps track of the current key.
/// - [`WithoutKeys`] skips this work, for callers that only need term ordinals or values.
pub trait KeyTracking {
/// Returns the buffer a streamer starts with.
fn make_default() -> Self;
/// Sets the current key to `key`.
fn set_key(&mut self, key: &[u8]);
/// Sets the current key to its first `common_prefix_len` bytes, followed by `suffix`.
///
/// The current key is not always a key of the dictionary. For instance, when positioning
/// itself on its lower bound, the sstable streamer sets the current key to a prefix of
/// the lower bound, and then applies its first entry with this method.
fn update_with_prefix(&mut self, common_prefix_len: usize, suffix: &[u8]);
}
impl KeyTracking for Vec<u8> {
#[inline]
fn make_default() -> Self {
Vec::with_capacity(DEFAULT_KEY_CAPACITY)
}
#[inline(always)]
fn set_key(&mut self, key: &[u8]) {
self.clear();
self.extend_from_slice(key);
}
#[inline(always)]
fn update_with_prefix(&mut self, common_prefix_len: usize, suffix: &[u8]) {
self.truncate(common_prefix_len);
self.extend_from_slice(suffix);
}
}
/// [`KeyTracking`] that does not keep track of keys.
///
/// A streamer using it only gives access to term ordinals and values.
#[derive(Default)]
pub struct WithoutKeys;
impl KeyTracking for WithoutKeys {
#[inline]
fn make_default() -> Self {
WithoutKeys
}
#[inline(always)]
fn set_key(&mut self, _key: &[u8]) {}
#[inline(always)]
fn update_with_prefix(&mut self, _common_prefix_len: usize, _suffix: &[u8]) {}
}
#[cfg(test)]
mod tests {
use super::KeyTracking;
#[test]
fn test_vec_key_tracking() {
let mut key: Vec<u8> = Vec::make_default();
key.update_with_prefix(0, b"abc");
assert_eq!(key, b"abc");
key.update_with_prefix(2, b"xy");
assert_eq!(key, b"abxy");
key.update_with_prefix(0, b"z");
assert_eq!(key, b"z");
}
#[test]
fn test_vec_set_key() {
let mut key: Vec<u8> = Vec::make_default();
key.set_key(b"abc");
assert_eq!(key, b"abc");
key.set_key(b"de");
assert_eq!(key, b"de");
key.set_key(b"");
assert!(key.is_empty());
}
}