rust/tantivy
1
0
Fork 0
mirror of https://github.com/quickwit-oss/tantivy.git synced 2026-09-06 21:39:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,681 Commits 262 Branches 81 Tags
bugfix-position-broken
Commit Graph
2681 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Paul Masurel 727d024a23 Bugfix position broken. 
For Field with several FieldValues, with a
value that contained no token at all, the token position
was reinitialized to 0.

As a result, PhraseQueries can show some false positives.
In addition, after the computation of the position delta, we can
underflow u32, and end up with gigantic delta.

We haven't been able to actually explain the bug in 1629, but it
is assumed that in some corner case these delta can cause a panic.

Closes #1629
 2022-10-20 10:19:41 +09:00
PSeitz 449f595832 Merge pull request #1628 from quickwit-oss/skip_index_deser 
faster skipindex deserialization, larger blocksize on sort
 2022-10-19 11:05:20 +08:00
PSeitz c9235df059 Merge pull request #1627 from quickwit-oss/ip_field_range_query 
add range query handling for ip via term dictionary
 2022-10-19 10:53:00 +08:00
Pascal Seitz a4485f7611 faster skipindex deserialization, larger blocksize on sort 2022-10-18 19:32:23 +08:00
Pascal Seitz 1082ff60f9 add range query handling for ip via term dictionary 
since IPs are mapped monotonically we can use the term dictionary for range queries
 2022-10-18 13:08:27 +08:00
PSeitz 491854155c Merge pull request #1625 from quickwit-oss/index_ip_field 
index ip field
 2022-10-18 11:18:17 +08:00
Christoph Herzog 96c3d54ac7 fix: Fix power of two computation on 32bit architectures (#1624) 
The current `compute_previous_power_of_two()` implementation used for
TermHashmap takes and returns `usize` , but actually only works
correclty on 64 bit architectures (aka usize == u64)

On other architectures the leading_zeros computation is run on the wrong
type (must be u64), and leads to overflows.

Fixed simply computing the leading_zeros based on a u64 value.
 2022-10-18 11:55:02 +09:00
Pascal Seitz 6800fdec9d add indexing for ip field 
Closes #1595
 2022-10-18 10:07:48 +08:00
PSeitz c9cf9c952a Merge pull request #1614 from quickwit-oss/remove_superfluous_steps 
refactor Term
 2022-10-17 18:25:31 +08:00
Pascal Seitz 024e53a99c remove truncate 2022-10-17 12:14:35 +08:00
Pascal Seitz 8d75e451bd fix truncate, remove mutable access from term 2022-10-17 12:14:35 +08:00
Pascal Seitz fcfd76ec55 refactor Term 
fixes some issues with Term
Remove duplicate calls to truncate or resize
Replace Magic Number 5 with constant
Enforce minimum size of 5 for metadata
Fix broken truncate docs
use constructor instead new + set calls
normalize constructor stack
replace assert on internal behavior fixes #1585
 2022-10-17 12:14:34 +08:00
PSeitz 6b7b1cc4fa Merge pull request #1623 from quickwit-oss/remove_unused_buffer 
remove unused buffer
 2022-10-14 20:36:00 +08:00
Pascal Seitz 129f7422f5 remove unused buffer 2022-10-14 20:01:10 +08:00
PSeitz f39cce2c8b Merge pull request #1622 from quickwit-oss/term_aggregation 
add term aggregation clarification
 2022-10-14 18:09:18 +08:00
PSeitz d2478fac8a Merge pull request #1621 from quickwit-oss/changelog 
update CHANGELOG
 2022-10-14 18:08:57 +08:00
Pascal Seitz 952b048341 add term aggregation clarification 2022-10-14 16:12:19 +08:00
PSeitz 80f9596ec8 Merge pull request #1611 from quickwit-oss/remove_token_stream_alloc 
remove tokenstream vec alloc
 2022-10-14 15:12:30 +08:00
Pascal Seitz 84f9e77e1d update CHANGELOG 2022-10-14 15:10:33 +08:00
PSeitz a602c248fb Merge pull request #1590 from waywardmonkeys/fix-doc-warnings-quickwit 
Fix missing doc warnings when enabling feature "quickwit".
 2022-10-14 14:09:25 +08:00
PSeitz 4b9d1fe828 Merge pull request #1620 from quickwit-oss/fix_fieldnorms_indexing 
Fix missing fieldnorm indexing
 2022-10-14 13:41:38 +08:00
Pascal Seitz 63bc390b02 Fix missing fieldnorm indexing 
Fixes broken search (no results) with BM25 for u64, i64, f64, bool, bytes and date after deletion and merge.
There were no fieldnorms recorded for those field. After merge InvertedIndexReader::total_num_tokens returns 0 (Sum over the fieldnorms is 0). BM25 does not work when total_num_tokens is 0.
Fixes #1617
 2022-10-14 12:44:40 +08:00
Paul Masurel 07393c2fa0 Attempt to fix race condition in test. (#1619) 
Close #1550
 2022-10-14 10:56:37 +09:00
PSeitz 77a415cbe4 rename NothingRecorder to DocIdRecorder (#1615) 2022-10-13 15:43:40 +09:00
PSeitz 4b4c231bba Merge pull request #1612 from quickwit-oss/no_panic_please 
return Error instead panic in fastfields
 2022-10-11 18:33:00 +08:00
PSeitz 11d3409286 add missing docs for fastfield_codecs crate (#1613) 
closes #1603
 2022-10-11 18:54:24 +09:00
Pascal Seitz 9cb8cfbea8 return Error instead panic in fastfields 
fixes #1572
 2022-10-11 14:15:22 +08:00
PSeitz 8b69aab0fc avoid prepare_doc allocation (#1610) 
avoid prepare_doc allocation, ~10% more thoughput best case
 2022-10-11 14:15:55 +09:00
PSeitz 3650d1f36a Merge pull request #1553 from quickwit-oss/ip_field 
ip field
 2022-10-11 13:09:47 +08:00
Pascal Seitz 2efebdb1bb remove tokenstream vec alloc 2022-10-11 10:30:56 +08:00
François Massot e443ca63aa Merge pull request #1608 from quickwit-oss/nigel/serialise-bytes-as-b64-#2042 
Serialise bytes as base64 strings instead of arrays.
 2022-10-10 11:51:23 +02:00
Pascal Seitz 5c9cbee29d handle IpV4 serialization case 2022-10-07 19:52:00 +08:00
Pascal Seitz b2ca83a93c switch to ipv6, add monotonic_mapping tests 2022-10-07 18:47:55 +08:00
Nigel Andrews 3b189080d4 Use raw string literals in tests 2022-10-07 12:28:25 +02:00
Nigel Andrews 00a6586efe Replaced String::serialize for serializer.serialize_str 2022-10-07 11:55:05 +02:00
Pascal Seitz b9b913510e fmt 2022-10-07 16:56:19 +08:00
PSeitzandPaul Masurel 534b1d33c3 use ipv6 
Co-authored-by: Paul Masurel <paul@quickwit.io>
 2022-10-07 16:56:00 +08:00
PSeitzandPaul Masurel f465173872 Apply suggestions from code review 
Co-authored-by: Paul Masurel <paul@quickwit.io>
 2022-10-07 16:55:53 +08:00
Pascal Seitz 96315df20d use idx part only for positions_to_docid 2022-10-07 16:54:04 +08:00
Pascal Seitz 9a1609d364 add test 2022-10-07 16:25:01 +08:00
Pascal Seitz 39f4e58450 improve comment 2022-10-07 16:25:01 +08:00
Pascal Seitz a8a36b62cd enable test 2022-10-07 16:25:01 +08:00
Pascal Seitz 226a49338f add StrictlyMonotonicFn 2022-10-07 16:25:01 +08:00
Pascal Seitz 2864bf7123 use serializer for u128 2022-10-07 16:25:01 +08:00
Pascal Seitz 5171ff611b serialize ip as u128, add test for positions_to_docid 2022-10-07 16:25:01 +08:00
Pascal Seitz e50e74acf8 remove u128 type 2022-10-07 16:25:01 +08:00
Pascal Seitz 0b86658389 rename ip addr, use buffer 2022-10-07 16:25:01 +08:00
Pascal Seitz 5d6602a8d9 mark null handling TODO 2022-10-07 16:25:01 +08:00
Pascal Seitz 4d29ff4d01 finalize ip addr rename 2022-10-07 16:25:01 +08:00
Pascal Seitz cdc8e3a8be group montonic mapping and inverse 
fix mapping inverse
remove ip indexing
add get_between_vals test
 2022-10-07 16:25:01 +08:00