ChangRui-Ryan
db2ecc6057
fix Column.first method parameter type ( #2792 )
2026-01-05 10:03:01 +01:00
PSeitz
5379c99ea2
update edition to 2024 ( #2620 )
...
* update common to edition 2024
* update bitpacker to edition 2024
* update stacker to edition 2024
* update query-grammar to edition 2024
* update sstable to edition 2024 + fmt
* fmt
* update columnar to edition 2024
* cargo fmt
* use None instead of _
2025-04-18 04:56:31 +02:00
PSeitz
d5d2d41264
merge column: small refactors ( #2579 )
...
* merge column: small refactors
* make ord dependency more explicit
* add columnar merge crashtest proptest
* fix naming
2025-03-07 18:52:34 +08:00
Paul Masurel
519e5d2ed1
clippy warnings
2025-03-05 11:15:06 +01:00
PSeitz
59084143ef
use optional index in multivalued index ( #2439 )
...
* use optional index in multivalued index
For mostly empty multivalued indices there was a large overhead during
creation when iterating all docids. This is alleviated by placing an
optional index in the multivalued index to mark documents that have values.
There's some performance overhead when accessing values in a multivalued
index. The accessing cost is now optional index + multivalue index. The
sparse codec performs relatively bad with the binary_search when accessing
data. This is reflected in the benchmarks below.
This changes the format of columnar to v2, but code is added to handle the v1
formats.
```
Running benches/bench_access.rs (/home/pascal/Development/tantivy/optional_multivalues/target/release/deps/bench_access-ea323c028db88db4)
multi sparse 1/13
access_values_for_doc Avg: 42.8946ms (+241.80%) Median: 42.8869ms (+244.10%) [42.7484ms .. 43.1074ms]
access_first_vals Avg: 42.8022ms (+421.93%) Median: 42.7553ms (+439.84%) [42.6794ms .. 43.7404ms]
multi 2x
access_values_for_doc Avg: 31.1244ms (+24.17%) Median: 30.8339ms (+23.46%) [30.7192ms .. 33.6059ms]
access_first_vals Avg: 24.3070ms (+70.92%) Median: 24.0966ms (+70.18%) [23.9328ms .. 26.4851ms]
sparse 1/13
access_values_for_doc Avg: 42.2490ms (+0.61%) Median: 42.2346ms (+2.28%) [41.8988ms .. 43.7821ms]
access_first_vals Avg: 43.6272ms (+0.23%) Median: 43.6197ms (+1.78%) [43.4920ms .. 43.9009ms]
dense 1/12
access_values_for_doc Avg: 8.6184ms (+23.18%) Median: 8.6126ms (+23.78%) [8.5843ms .. 8.7527ms]
access_first_vals Avg: 6.8112ms (+4.47%) Median: 6.8002ms (+4.55%) [6.7887ms .. 6.8991ms]
full
access_values_for_doc Avg: 9.4073ms (-5.09%) Median: 9.4023ms (-2.23%) [9.3694ms .. 9.4568ms]
access_first_vals Avg: 4.9531ms (+6.24%) Median: 4.9502ms (+7.85%) [4.9423ms .. 4.9718ms]
```
```
Running benches/bench_merge.rs (/home/pascal/Development/tantivy/optional_multivalues/target/release/deps/bench_merge-475697dfceb3639f)
merge_multi 2x_and_multi 2x Avg: 20.2280ms (+34.33%) Median: 20.1829ms (+35.33%) [19.9933ms .. 20.8806ms]
merge_multi sparse 1/13_and_multi sparse 1/13 Avg: 0.8961ms (-78.04%) Median: 0.8943ms (-77.61%) [0.8899ms .. 0.9272ms]
merge_dense 1/12_and_dense 1/12 Avg: 0.6619ms (-1.26%) Median: 0.6616ms (+2.20%) [0.6473ms .. 0.6837ms]
merge_sparse 1/13_and_sparse 1/13 Avg: 0.5508ms (-0.85%) Median: 0.5508ms (+2.80%) [0.5420ms .. 0.5634ms]
merge_sparse 1/13_and_dense 1/12 Avg: 0.6046ms (-4.64%) Median: 0.6038ms (+2.80%) [0.5939ms .. 0.6296ms]
merge_multi sparse 1/13_and_dense 1/12 Avg: 0.9111ms (-83.48%) Median: 0.9063ms (-83.50%) [0.9047ms .. 0.9663ms]
merge_multi sparse 1/13_and_sparse 1/13 Avg: 0.8451ms (-89.49%) Median: 0.8428ms (-89.43%) [0.8411ms .. 0.8563ms]
merge_multi 2x_and_dense 1/12 Avg: 10.6624ms (-4.82%) Median: 10.6568ms (-4.49%) [10.5738ms .. 10.8353ms]
merge_multi 2x_and_sparse 1/13 Avg: 10.6336ms (-22.95%) Median: 10.5925ms (-22.33%) [10.5149ms .. 11.5657ms]
```
* Update columnar/src/columnar/format_version.rs
Co-authored-by: Paul Masurel <paul@quickwit.io >
* Update columnar/src/column_index/mod.rs
Co-authored-by: Paul Masurel <paul@quickwit.io >
---------
Co-authored-by: Paul Masurel <paul@quickwit.io >
2024-06-19 14:54:12 +08:00
PSeitz
72f61ff89c
remove index sorting ( #2434 )
...
closes https://github.com/quickwit-oss/tantivy/issues/2352
2024-06-13 15:51:53 +08:00
trinity-1686a
9ebc5ed053
use fst for sstable index ( #2268 )
...
* read path for new fst based index
* implement BlockAddrStoreWriter
* extract slop/derivation computation
* use better linear approximator and allow negative correction to approximator
* document format and reorder some fields
* optimize single block sstable size
* plug backward compat
2023-12-04 15:13:15 +01:00
PSeitz
4feeb2323d
fix clippy ( #2223 )
2023-10-24 10:05:22 +02:00
Paul Masurel
910b0b0c61
Cargo fmt
2023-07-03 22:03:31 +09:00
trinity-1686a
780e26331d
sstable compression ( #1946 )
...
* compress sstable with zstd
* add some details to sstable readme
* compress only block which benefit from it
* multiple changes to sstable
make compression optional
use OwnedBytes instead of impl Read in sstable, required for next point
use zstd bulk api, which is much faster on small records
* cleanup and use bulk api for compression
* use dedicated byte for compression
* switch block len and compression flag
* change default zstd level in sstable
2023-04-14 16:25:50 +02:00
trinity-1686a
205e8a0a92
encode dictionary type in fst footer ( #1968 )
...
* encode additional footer for dictionary kind in fst
2023-04-12 09:43:01 +02:00
Paul Masurel
5eb12173d6
Proptest merge columnar ( #1976 )
...
* Added proptest on columnar merge with a shuffle
Made column serialization more explicit.
Bugfix when a bytes column is missing, and with a shuffle.
Improved the cardinality detection logic / column detection.
* Code review
* CR comments
* Following CR
2023-04-04 11:28:42 +09:00
PSeitz
571735c5f7
Fix index sort by on optional/multicolumn ( #1972 )
...
Fix index sort by on optional/multicolumn
add optional columns to proptest
extend proptests for sort
add columnar sort tests
2023-03-31 04:24:11 +02:00
Paul Masurel
2955e34452
Added proptests for building/merging columnar. ( #1963 )
2023-03-27 14:56:02 +09:00
Paul Masurel
a2e3c2ed5b
Renaming Column::idx -> Column::index ( #1961 )
...
There was some variable name ghosting happening.
2023-03-26 13:58:50 +09:00
trinity-1686a
482b4155e8
fix bug with new sstable index format ( #1953 )
2023-03-22 10:22:36 +01:00
trinity-1686a
e5e50603a8
new sstable format ( #1943 )
...
* document a new sstable format
* add support for changing target block size
* use new format for sstable index
* handle sstable version errror
* use very small blocks for proptests
* add a footer structure
2023-03-21 15:03:52 +01:00
PSeitz
1cfb9ce59a
improve range query performance ( #1864 )
...
fix RowId vs DocId naming
fixes #1863
2023-02-14 13:25:39 +09:00
Paul Masurel
bd5eea9852
Integrated columnar work.
2023-02-09 13:14:31 +01:00
Paul Masurel
08919a2900
Improvement on the scalar / random bitpacker code. ( #1781 )
...
* Improvement on the scalar / random bitpacker code.
Added proptesting
Added simple benchmark
Added assert and comments on the very non trivial hidden contract
Remove the need for an extra padding.
The last point introduces a small performance regression (~10%).
* Fixing unit tests
2023-01-19 18:09:13 +09:00
Paul Masurel
5a42c5aae9
Add support for multivalues ( #1809 )
2023-01-19 16:55:01 +09:00
Paul Masurel
a86b104a40
Differentiating between str and bytes, + unit test
2023-01-19 14:38:12 +09:00
PSeitz
f9abd256b7
add ip addr to columnar ( #1805 )
2023-01-19 05:36:06 +01:00
Paul Masurel
9f42b6440a
Completed unit test for dictionary encoded column
2023-01-19 12:15:27 +09:00
Paul Masurel
25bad784ad
Integrated fastfield codecs into columnar. ( #1782 )
...
Introduced asymetric OptionalCodec / SerializableOptionalCodec
Removed cardinality from the columnar sstable.
Added DynamicColumn
Reorganized all files
Change DenseCodec serialization logic.
Renamed methods to rank/select
Moved versioning footer to the columnar level
2023-01-16 17:24:49 +09:00
