rust/tantivy
1
0
Fork 0
mirror of https://github.com/quickwit-oss/tantivy.git synced 2026-11-14 14:47:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,657 Commits 297 Branches 83 Tags
cc0fad0214098dba2b2ec9c2aaa96ec9b48bb080
Commit Graph
3657 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Pascal Seitz cc0fad0214 refactor 2026-07-31 17:07:51 +02:00
Pascal Seitz e81daa93da simplify 2026-07-31 14:57:44 +02:00
Pascal Seitz b5522e97f1 merge multi-terms code 2026-07-30 17:48:06 +02:00
Pascal Seitz 7d0353c577 refactor benches 2026-07-30 13:56:16 +02:00
Pascal Seitz 31b2f3c07e fix merge 2026-07-29 17:42:32 +02:00
Pascal Seitz 2590f660ca move impl 2026-07-29 17:32:41 +02:00
Pascal Seitz 0f1d29a1fb improve naming 2026-07-29 17:32:41 +02:00
Pascal Seitz f2fd29d0b2 Optimize sparse multi-terms collection 
Filter documents without values after each single-valued field so later
decoding and key construction only process viable documents. Use exact
block lengths to validate aligned results and keep unsupported missing-value
combinations on the general path.
 2026-07-29 17:32:41 +02:00
Pascal Seitz ccaedc30d1 improve missing performance 2026-07-29 17:32:41 +02:00
Pascal Seitz 0166ab8a17 add missing to benchmark 2026-07-29 17:32:41 +02:00
Pascal Seitz ff7de278da fmt 2026-07-29 17:32:41 +02:00
Pascal Seitz b370404130 cleanup 2026-07-29 17:32:41 +02:00
Pascal Seitz 08ea8c9d04 Simplify optional multi-terms document filtering 
Track valid document positions in a reusable bitset instead of compacting
parallel document and index buffers after each optional field. Decode each
field against the original block and consult the mask before writing keys or
populating buckets.

This favors simpler collection logic over shrinking later sparse decodes.
 2026-07-29 17:32:41 +02:00
Pascal Seitz ac0d4e46fd Refactor multi-terms collection around typed key codecs 
Fan out dynamic/JSON physical column combinations during aggregation tree
construction so each segment collector handles one typed accessor per
field and applies missing values exactly once.

Use a generic key codec and shared term bucket maps for both packed and
unpacked keys. Keep optional and multivalued fields eligible for packed
storage, short-circuit sparse documents, and account for spilled composite
keys.
 2026-07-29 17:32:41 +02:00
Pascal Seitz b902acf12b columnar storage - add batched rank 2026-07-29 17:32:41 +02:00
Pascal Seitz 1cde792c03 add filtered benchmark 2026-07-29 17:32:41 +02:00
Pascal Seitz 48959111d9 more benchmarks to compare 2026-07-29 17:32:41 +02:00
Pascal Seitz 41b2018433 rename vars 2026-07-29 17:32:40 +02:00
Pascal Seitz 044c24243e move term req data to collector 2026-07-29 17:32:40 +02:00
trinity-1686a 8ff7e6d2e8 rename agg benches 2026-07-29 17:32:40 +02:00
trinity-1686a 9ae914e3eb rustfmt 2026-07-29 17:32:40 +02:00
trinity.pointard d738590053 extract per-field fast-path compatiblity check 2026-07-29 17:32:40 +02:00
trinity.pointard bd2e54c0cb decode column once per doc max 2026-07-29 17:32:40 +02:00
trinity.pointard e580c7725c more codex review 2026-07-29 17:32:40 +02:00
trinity.pointard 349325bf6e fix handling of bytes for cardinality agg 2026-07-29 17:32:40 +02:00
trinity.pointard a158d20344 codex review 2026-07-29 17:32:39 +02:00
trinity.pointard 993800e2ad rustfmt 2026-07-29 17:32:39 +02:00
trinity.pointard 2700fed93e fix bool serialization 2026-07-29 17:32:39 +02:00
trinity.pointard 22b6bbf2eb first pass of optimising multi-terms agg 2026-07-29 17:32:39 +02:00
trinity.pointard b03213e528 allow extracting values from IntermediateMultiTermsBucketResult 2026-07-29 17:32:38 +02:00
trinity.pointard 6455da5dba first swab at multi-terms impl 2026-07-29 17:32:38 +02:00
Pascal Seitz 667132fa7a rename to WithLanes 2026-07-29 08:57:54 +02:00
PSeitz 95f2c0c5b6 Optimize low-cardinality term aggregation counters 2026-07-29 08:57:54 +02:00
PSeitz bb049ee714 default REUSE_AGG_BENCH_INDEX to true 2026-07-29 08:57:54 +02:00
Pascal Seitz 277e59776b Extend Vec term storage to 20k with eager/lazy bucket ids 
Raise MAX_NUM_TERMS_FOR_VEC to 20_000 so the dense Vec term storage
(direct-indexed, no hashing/paging) is used for low/moderate cardinality
both with and without sub-aggregations, not just the <100 low-card case.
Split the old single threshold into MAX_NUM_TERMS_FOR_LOWCARD_SUBAGG (100,
still gates the Vec + LowCard-buffer pairing) and MAX_NUM_TERMS_FOR_VEC.

VecTermBuckets is now generic over a compile-time `const LAZY: bool`:
- eager (default): assigns all ids up front in `new`, branchless term_entry
- lazy: assigns on first occurrence, keeping the sub-agg bucket range down
  to the terms that actually occur (no blowup for sparse ordinal spaces)

A new BucketIdSlot::ASSIGNS_ID const plus the LAZY const fully gate the
first-seen branch out at monomorphization, so the only branchy path is the
deliberately-lazy <BucketId, true> instantiation. The with-sub-agg Vec path
picks eager below MAX_NUM_TERMS_FOR_EAGER_BUCKET_IDS (4096) and lazy above,
avoiding the ~6-11% hot-loop regression eager fixes while keeping large
sparse cases compact.

Biggest impact on terms_zipf_1000_only with -40%
```
full
terms_7                                               Memory: 46.5 KB              Avg: 2.3566ms (+4.29%)      Median: 2.3566ms (+4.29%)      [2.3566ms .. 2.3566ms]
terms_all_unique                                      Memory: 11.5 MB              Avg: 4.8532ms (-5.74%)      Median: 4.8532ms (-5.74%)      [4.8532ms .. 4.8532ms]
terms_all_unique_order_by_key                         Memory: 11.5 MB              Avg: 4.8448ms (-0.53%)      Median: 4.8448ms (-0.53%)      [4.8448ms .. 4.8448ms]
terms_150_000                                         Memory: 2.7 MB               Avg: 5.5295ms (+0.64%)      Median: 5.5295ms (+0.64%)      [5.5295ms .. 5.5295ms]
terms_many_top_1000                                   Memory: 5.3 MB               Avg: 8.4738ms (+0.58%)      Median: 8.4738ms (+0.58%)      [8.4738ms .. 8.4738ms]
terms_many_order_by_term                              Memory: 2.7 MB (-0.01%)      Avg: 4.5959ms (-0.15%)      Median: 4.5959ms (-0.15%)      [4.5959ms .. 4.5959ms]
terms_many_with_top_hits                              Memory: 48.9 MB (-0.00%)     Avg: 85.8206ms (-5.46%)     Median: 85.8206ms (-5.46%)     [85.8206ms .. 85.8206ms]
terms_all_unique_with_avg_sub_agg                     Memory: 54.7 MB (-0.00%)     Avg: 17.8747ms (+8.16%)     Median: 17.8747ms (+8.16%)     [17.8747ms .. 17.8747ms]
terms_many_with_avg_sub_agg                           Memory: 13.5 MB (-0.00%)     Avg: 14.7797ms (+2.54%)     Median: 14.7797ms (+2.54%)     [14.7797ms .. 14.7797ms]
terms_status_with_avg_sub_agg                         Memory: 92.1 KB (-0.35%)     Avg: 5.1467ms (+0.72%)      Median: 5.1467ms (+0.72%)      [5.1467ms .. 5.1467ms]
terms_status_with_terms_zipf_1000_sub_agg             Memory: 213.2 KB             Avg: 3.9576ms (-2.73%)      Median: 3.9576ms (-2.73%)      [3.9576ms .. 3.9576ms]
terms_zipf_1000_with_terms_status_sub_agg             Memory: 726.5 KB (+0.28%)    Avg: 12.1432ms (+3.30%)     Median: 12.1432ms (+3.30%)     [12.1432ms .. 12.1432ms]
terms_status_with_histogram                           Memory: 139.0 KB             Avg: 2.4717ms (+2.45%)      Median: 2.4717ms (+2.45%)      [2.4717ms .. 2.4717ms]
terms_status_with_date_histogram                      Memory: 145.9 KB             Avg: 2.3644ms (-0.44%)      Median: 2.3644ms (-0.44%)      [2.3644ms .. 2.3644ms]
terms_status_with_date_histogram_hard_bounds          Memory: 145.4 KB             Avg: 2.5705ms (+4.70%)      Median: 2.5705ms (+4.70%)      [2.5705ms .. 2.5705ms]
terms_status_with_date_histogram_and_sibling_terms    Memory: 143.3 KB             Avg: 3.8946ms (+1.61%)      Median: 3.8946ms (+1.61%)      [3.8946ms .. 3.8946ms]
terms_zipf_1000_only                                  Memory: 75.8 KB (-0.02%)     Avg: 1.2684ms (-41.50%)     Median: 1.2684ms (-41.50%)     [1.2684ms .. 1.2684ms]
terms_zipf_1000_with_histogram                        Memory: 1.2 MB (+0.17%)      Avg: 20.9323ms (+4.08%)     Median: 20.9323ms (+4.08%)     [20.9323ms .. 20.9323ms]
terms_zipf_1000_with_avg_sub_agg                      Memory: 486.7 KB (+2.95%)    Avg: 8.5297ms (-1.66%)      Median: 8.5297ms (-1.66%)      [8.5297ms .. 8.5297ms]
terms_many_json_mixed_type_with_avg_sub_agg           Memory: 17.9 MB              Avg: 27.0210ms (+1.81%)     Median: 27.0210ms (+1.81%)     [27.0210ms .. 27.0210ms]
terms_status_with_cardinality_agg                     Memory: 93.9 KB              Avg: 3.2022ms (-0.64%)      Median: 3.2022ms (-0.64%)      [3.2022ms .. 3.2022ms]
terms_100_buckets_with_cardinality_agg                Memory: 9.8 MB (+0.31%)      Avg: 49.3188ms (-1.70%)     Median: 49.3188ms (-1.70%)     [49.3188ms .. 49.3188ms]
terms_many_with_single_term_order_by_card             Memory: 48.9 MB              Avg: 77.2127ms (-5.02%)     Median: 77.2127ms (-5.02%)     [77.2127ms .. 77.2127ms]
terms_many_with_single_term_2_order_by_card           Memory: 40.6 MB (-0.02%)     Avg: 45.8004ms (-17.57%)    Median: 45.8004ms (-17.57%)    [45.8004ms .. 45.8004ms]
```
 2026-07-27 18:00:37 +02:00
Pascal Seitz fc84c0f299 fix REUSE_AGG_BENCH_INDEX 2026-07-27 18:00:37 +02:00
PSeitz-dd 70f0b039f5 Merge pull request #2978 from quickwit-oss/mallets/finalize-docidmapping 
feat: add custom doc id mapping finalization
 2026-07-22 15:20:19 +02:00
Luca CominardiandCursor 2554072e8f fix: address manual mapping review comments 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-22 14:50:29 +02:00
Luca Cominardi fe1bdf4a54 fix: make fmt 2026-07-22 11:48:17 +02:00
Luca CominardiandCursor 2c967e5d20 refactor: share doc id mapping construction 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-22 11:46:39 +02:00
Luca CominardiandPSeitz 2350c8e645 Update src/indexer/segment_writer.rs 
Co-authored-by: PSeitz <PSeitz@users.noreply.github.com>
 2026-07-22 10:22:51 +02:00
Luca CominardiandCursor 0a071c55f7 fix: clarify manual doc mapping serialization test 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-21 12:41:32 +02:00
Luca CominardiandCursor 56bd23bafe fix: move doc mapping tests to doc mapping module 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-21 12:40:08 +02:00
dependabot[bot] 31ca1a8ba2 Update lz4_flex requirement from 0.13 to 0.14 
Updates the requirements on [lz4_flex](https://github.com/pseitz/lz4_flex) to permit the latest version.
- [Release notes](https://github.com/pseitz/lz4_flex/releases)
- [Changelog](https://github.com/PSeitz/lz4_flex/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pseitz/lz4_flex/compare/0.13.0...0.14.0)

---
updated-dependencies:
- dependency-name: lz4_flex
  dependency-version: 0.14.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-20 09:51:23 +02:00
trinity-1686a 057458bf14 use enum PruneMode instead of bool 2026-07-13 12:16:18 +02:00
trinity-1686a f05ef0c4cc cr 2026-07-13 12:16:18 +02:00
trinity-1686a 16dfddf31a add method to prune intermediate agg results 2026-07-13 12:16:18 +02:00
Pascal Seitz 61693134be fix cache flush in aggregations 
fixes #2992

```
full
terms_7                                               Memory: 37.2 KB               Avg: 2.3958ms (+0.31%)     Median: 2.3896ms (+0.18%)     [2.3573ms .. 2.5176ms]
terms_all_unique                                      Memory: 10.8 MB               Avg: 5.5144ms (-1.07%)     Median: 5.4625ms (-1.98%)     [5.3364ms .. 5.9712ms]
terms_all_unique_order_by_key                         Memory: 10.8 MB               Avg: 5.2614ms (-0.85%)     Median: 5.2177ms (-1.21%)     [5.0823ms .. 5.6316ms]
terms_150_000                                         Memory: 2.7 MB                Avg: 5.5335ms (-1.07%)     Median: 5.5152ms (-1.06%)     [5.4151ms .. 5.9654ms]
terms_many_top_1000                                   Memory: 5.2 MB                Avg: 8.3579ms (-1.53%)     Median: 8.3604ms (-0.95%)     [8.2184ms .. 8.5421ms]
terms_many_order_by_term                              Memory: 2.7 MB                Avg: 4.6713ms (-0.07%)     Median: 4.6569ms (-0.15%)     [4.5994ms .. 4.9115ms]
terms_all_unique_with_avg_sub_agg                     Memory: 54.0 MB               Avg: 17.4981ms (-2.43%)    Median: 17.6075ms (-1.75%)    [15.8166ms .. 18.9250ms]
terms_status_with_avg_sub_agg                         Memory: 90.3 KB               Avg: 5.6365ms (+7.77%)     Median: 5.6255ms (+7.97%)     [5.5489ms .. 5.8254ms]
terms_status_with_terms_zipf_1000_sub_agg             Memory: 318.5 KB (+56.52%)    Avg: 4.4504ms (+11.55%)    Median: 4.4436ms (+11.59%)    [4.3858ms .. 4.5692ms]
terms_zipf_1000_with_terms_status_sub_agg             Memory: 684.9 KB              Avg: 11.8606ms (+0.19%)    Median: 11.8360ms (-0.02%)    [11.7478ms .. 12.0609ms]
terms_status_with_histogram                           Memory: 139.5 KB              Avg: 2.4524ms (-1.09%)     Median: 2.4521ms (-0.23%)     [2.4179ms .. 2.5049ms]
terms_status_with_date_histogram                      Memory: 136.7 KB              Avg: 2.3407ms (-1.28%)     Median: 2.3359ms (-1.06%)     [2.3001ms .. 2.4310ms]
terms_status_with_date_histogram_hard_bounds          Memory: 136.1 KB              Avg: 2.5113ms (-2.04%)     Median: 2.5073ms (-0.97%)     [2.4455ms .. 2.7280ms]
terms_status_with_date_histogram_and_sibling_terms    Memory: 137.3 KB              Avg: 3.8695ms (-0.48%)     Median: 3.8653ms (+0.11%)     [3.8093ms .. 4.0528ms]
terms_zipf_1000                                       Memory: 69.8 KB               Avg: 2.2022ms (-1.95%)     Median: 2.2026ms (-1.12%)     [2.1705ms .. 2.2859ms]
terms_zipf_1000_with_histogram                        Memory: 1.2 MB                Avg: 20.4087ms (-0.02%)    Median: 20.3665ms (+0.11%)    [20.1912ms .. 20.7933ms]
terms_zipf_1000_with_avg_sub_agg                      Memory: 472.0 KB              Avg: 8.7387ms (-3.48%)     Median: 8.7043ms (-3.44%)     [8.6466ms .. 9.1396ms]
terms_zipf_90                                         Memory: 55.3 KB               Avg: 1.3784ms (-2.04%)     Median: 1.3787ms (-1.61%)     [1.3484ms .. 1.4611ms]
terms_zipf_90_with_sum_sub_agg                        Memory: 367.6 KB              Avg: 4.8520ms (+8.43%)     Median: 4.8326ms (+8.94%)     [4.8058ms .. 5.1278ms]
terms_many_json_mixed_type_with_avg_sub_agg           Memory: 17.8 MB               Avg: 25.0853ms (-7.70%)    Median: 25.0591ms (-7.12%)    [24.8103ms .. 25.4936ms]
terms_status_with_cardinality_agg                     Memory: 91.8 KB               Avg: 3.3667ms (+1.47%)     Median: 3.3690ms (+1.66%)     [3.3311ms .. 3.4070ms]
terms_100_buckets_with_cardinality_agg                Memory: 9.9 MB                Avg: 48.4768ms (-3.07%)    Median: 48.3745ms (-3.38%)    [48.1425ms .. 49.5503ms]
```
 2026-07-12 16:40:05 +02:00
Pascal Seitz 7152d53182 clippy 2026-07-10 12:33:33 +02:00
Luca Cominardi 1c5af9489a fix: cargo fmt 2026-07-07 15:25:35 +02:00