rust/tantivy
1
0
Fork 0
mirror of https://github.com/quickwit-oss/tantivy.git synced 2026-10-26 21:46:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,904 Commits 291 Branches 83 Tags
9e2faecf5b5c75bd72cf9b8009a1e9bdb6fc25d0
Commit Graph
131 Commits
Author SHA1 Message Date
PSeitzandPaul Masurel 9e2faecf5b add memory limit for aggregations (#1942) 
* add memory limit for aggregations

introduce AggregationLimits to set memory consumption limit and bucket limits
memory limit is checked during aggregation, bucket limit is checked before returning the aggregation request.

* Apply suggestions from code review

Co-authored-by: Paul Masurel <paul@quickwit.io>

* add ByteCount with human readable format

---------

Co-authored-by: Paul Masurel <paul@quickwit.io>
 2023-03-16 06:21:07 +01:00
PSeitz b6703f1b3c fix validation in date histogram (#1936) 
fix validation in date histogram for parameters interval and date_interval
 2023-03-15 06:10:43 +01:00
PSeitz 2fb3740cb0 handle missing column for aggs (#1920) 
* handle missing column for aggs

add empty column fallback for missing column in aggs.
Fix sort for term agg on sub-agg with missing value (null is smallest)

* add error when field is not fast
 2023-03-15 06:09:59 +01:00
PSeitz 8459efa32c split term collection count and sub_agg (#1921) 
use unrolled ColumnValues::get_vals
 2023-03-13 04:37:41 +01:00
PSeitz 61cfd8dc57 fix clippy (#1927) 2023-03-13 03:12:02 +01:00
Paul Masurel 364e321415 Clippy fix (#1926) 2023-03-06 10:37:17 +09:00
PSeitz ca20bfa776 add date_histogram (#1900) 
* add date_histogram

* add return result
 2023-03-02 05:17:35 +01:00
PSeitz 850a0d7ae2 add agg benchmark for optional and multi value (#1916) 
closes #1870
 2023-03-01 17:01:52 +09:00
Paul MasurelandPSeitz 7fae4d98d7 Adapting for quickwit2 (#1912) 
* Adapting tantivy to make it possible to be plugged to quickwit.

* Apply suggestions from code review

Co-authored-by: PSeitz <PSeitz@users.noreply.github.com>

* Added unit test

---------

Co-authored-by: PSeitz <PSeitz@users.noreply.github.com>
 2023-03-01 16:27:46 +09:00
PSeitz bc36458334 move buffer in front of dynamic dispatch (#1915) 
dynamic dispatch seems to be really expensive, move the buffer in front of the dynamic dispatch, to reduce the number of calls into the dynamic dispatched collector.
 2023-02-28 13:07:50 +08:00
PSeitzandPaul Masurel e510f699c8 feat: add support for u64,i64,f64 fields in term aggregation (#1883) 
* feat: add support for u64,i64,f64 fields in term aggregation

* hash enum values

* fix build

* Apply suggestions from code review

Co-authored-by: Paul Masurel <paul@quickwit.io>

---------

Co-authored-by: Paul Masurel <paul@quickwit.io>
 2023-02-27 15:04:41 +08:00
Paul Masurel d25fc155b2 Making some of the column/termdict operations async-friendly (#1902) 2023-02-27 15:34:47 +09:00
Paul Masurel 8ea97e7d6b Minor refactoring preparing for getting columnar integrated in quickwit. (#1911) 2023-02-27 14:23:30 +09:00
Paul Masurel c838aa808b Removedc the extra nesting in unit test file (#1907) 2023-02-27 12:17:52 +09:00
Paul Masurel 06850719dc Renaming .values(DocId) to .values_for_doc(DocId) (#1906) 2023-02-27 12:15:13 +09:00
PSeitz 5f23bb7e65 switch to sparse collection for histogram (#1898) 
* switch to sparse collection for histogram

Replaces histogram vec collection with a hashmap. This approach works much better for sparse data and enables use cases like drill downs (filter + small interval).
It is slower for dense cases (1.3x-2x slower). This can be alleviated with a specialized hashmap in the future.
closes #1704
closes #1370

* refactor, clippy

* fix bucket_pos overflow issue
 2023-02-23 07:02:58 +01:00
PSeitz c7278b3258 remove schema in aggs (#1888) 
* switch to ColumnType, move tests

* remove Schema dependency in agg
 2023-02-22 04:50:28 +01:00
Paul Masurel e2aa5af075 Clippy warnings fixes (#1885) 2023-02-20 19:04:13 +09:00
PSeitz 74bf60b4f7 implement SegmentAggregationCollector on bucket aggs (#1878) 2023-02-17 12:53:29 +01:00
PSeitz 111f25a8f7 clippy (#1879) 
* fix clippy

* fix clippy

* fmt
 2023-02-17 11:34:21 +01:00
PSeitz 019db10e8e refactor aggregations (#1875) 
* add specialized version for full cardinality

Pre Columnar
test aggregation::tests::bench::bench_aggregation_average_u64                                                            ... bench:   6,681,850 ns/iter (+/- 1,217,385)
test aggregation::tests::bench::bench_aggregation_average_u64_and_f64                                                    ... bench:  10,576,327 ns/iter (+/- 494,380)

Current
test aggregation::tests::bench::bench_aggregation_average_u64                                                            ... bench:  11,562,084 ns/iter (+/- 3,678,682)
test aggregation::tests::bench::bench_aggregation_average_u64_and_f64                                                    ... bench:  18,925,790 ns/iter (+/- 17,616,771)

Post Change
test aggregation::tests::bench::bench_aggregation_average_u64                                                            ... bench:   9,123,811 ns/iter (+/- 399,720)
test aggregation::tests::bench::bench_aggregation_average_u64_and_f64                                                    ... bench:  13,111,825 ns/iter (+/- 273,547)

* refactor aggregation collection

* add buffering collector
 2023-02-16 13:15:16 +01:00
PSeitz 347614c841 test error for avg agg on ip field (#1873) 
closes #1835
 2023-02-14 23:22:56 +08:00
Paul Masurel 097fd6138d Fix clippy comments (#1872) 2023-02-14 23:12:45 +09:00
Paul Masurel 60cc2644d6 Fixing test_fail_on_flush_segment_but_one_worker_remains (#1869) 
The new fast field code, based on columnar, had a larger minimum memory
footprint, causing the first docuemnt to trigger a flush of the asegment
in this unit test.

This PR prevents the allocation of a large capacity for the different hashmap tables
using in the columnar writer.

Closes #1859
 2023-02-14 16:09:42 +09:00
Paul Masurel 10bccac61b Bugfix in parse_into_milliseconds (#1867) 2023-02-14 15:06:40 +09:00
PSeitz 7a9befd18d fix sort order test for term aggregation (#1858) 
fix sort order test for term aggregation
fix invalid request test
 2023-02-10 10:26:58 +01:00
Paul Masurel bd5eea9852 Integrated columnar work. 2023-02-09 13:14:31 +01:00
PSeitz a2ca12995e update aggregation docs (#1807) 2023-01-19 09:52:47 +01:00
PSeitz f687b3a5aa start migrate Field to &str (#1772) 
start migrate Field to &str in preparation of columnar
return Result for get_field
 2023-01-18 16:12:07 +09:00
Adrien Guillo c51d9f9f83 Fix some Clippy warnings 2023-01-17 10:17:51 -05:00
Adrien Guillo 0caaf13a90 Remove standard deviation from stats aggregation 2023-01-16 22:58:23 -05:00
Adrien Guillo f2dad194ea Add count, min, max, and sum aggregations 2023-01-16 12:22:20 -05:00
PSeitz 6ca9a477f3 reuse stats for average (#1785) 
* reuse stats for average

* fix count type
 2023-01-13 23:32:27 +08:00
Adam Reichold 8312c882a5 More cosmetic fixes for upcoming Clippy lints. (#1771) 2023-01-10 10:32:45 +01:00
PSeitz f9171a3981 fix clippy (#1725) 
* fix clippy

* fix clippy fastfield codecs

* fix clippy bitpacker

* fix clippy common

* fix clippy stacker

* fix clippy sstable

* fmt
 2022-12-20 07:30:06 +01:00
PSeitzandPaul Masurel 2c50b02eb3 Fix max bucket limit in histogram (#1703) 
* Fix max bucket limit in histogram

The max bucket limit in histogram was broken, since some code introduced temporary filtering of buckets, which then resulted into an incorrect increment on the bucket count.
The provided solution covers more scenarios, but there are still some scenarios unhandled (See #1702).

* Apply suggestions from code review

Co-authored-by: Paul Masurel <paul@quickwit.io>

Co-authored-by: Paul Masurel <paul@quickwit.io>
 2022-12-12 04:40:15 +01:00
PSeitz ee1f2c1f28 add aggregation support for date type (#1693) 
* add aggregation support for date type
fixes #1332

* serialize key_as_string as rfc3339 in date histogram
* update docs
* enable date for range aggregation
 2022-11-28 09:12:08 +09:00
Pascal Seitz 279b1b28d3 switch to fx hashmap 2022-10-27 16:19:59 +08:00
Adam Reichold bbb058d976 Replace FNV by rustc-hash 
Both construction have similar goals but rustc-hash ist better suited for
contemporary CPU as it works one word at a time instead of byte per byte.
 2022-10-27 00:35:09 +02:00
Pascal Seitz e772d3170d switch get_val() to u32 
Fixes #1638
 2022-10-24 19:05:57 +08:00
Pascal Seitz 952b048341 add term aggregation clarification 2022-10-14 16:12:19 +08:00
Bruce Mitchener b3bf9a5716 Documentation improvements. 2022-10-05 14:18:10 +07:00
Bruce Mitchener 44e03791f9 Fix warnings when doc'ing private items. (#1579) 
This also fixes a couple of typos, but plenty remain!
 2022-10-03 14:24:00 +09:00
Bruce Mitchener d231671fe2 clippy: Remove borrows that the compiler will do. 
This started showing up with clippy in rust 1.64.
 2022-09-22 22:38:23 +07:00
Bruce Mitchener cf02e32578 Improvements to doc linking, grammar, etc. 2022-09-19 18:10:22 +07:00
PSeitz 45924711fd improve docs (#1514) 
fix link alias after https://github.com/rust-lang/rustfmt/pull/5262 has been merged and released.
fix dead links
 2022-09-08 22:33:59 +09:00
Paul Masurel 26876d41d7 Moving the serialization logic to the fastfield_codecs crate. 2022-09-03 00:29:52 +09:00
Paul Masurel 8e775b6c3d Refactoring dyn Column (#1502) 2022-09-02 17:26:30 +09:00
Paul Masurel 5331be800b Introducing a column trait 2022-08-28 14:14:27 +02:00
Kian-Meng Ang 014b1adc3e cargo +nightly fmt 2022-08-17 22:33:44 +08:00