mirror of
https://github.com/quickwit-oss/tantivy.git
synced 2026-10-14 07:42:53 +00:00
François Massot
5c28c1a188
On OTel logs (~1 KB stored per doc, a stored JSON object of attributes): - Doc store: CompactDoc serializes its stored fields by walking node_data directly (strings and bytes are already VInt-len + bytes there, the doc store encoding after the type byte). New hidden Document:: serialize_stored_fields hook, default false: other document types keep the generic serializer. Byte-identical to BinaryDocumentSerializer (proptest on random nested values of every type, stored/unstored fields); IP addresses, pre-tokenized text and custom values fall back to the generic path. StoreWriter::store: 987 -> 583 ns/doc. - CompactDoc: decode ValueAddr / VInt from slices instead of io::Read; exact size_hint on object/array iterators (the generic serializer then streams entries instead of collecting them). - CompactDoc::add_value: one thread-local scratch stack for child addresses instead of a Vec per nested object; leaves skip it. Doc building: -7.7%. - SimpleTokenizer: byte lookup table for pure-ASCII text, same tokens as the char path (proptest on ASCII and arbitrary Unicode). Tokenizer: -14%. Indexing 2M OTel docs, one thread (harness): -6% CPU with zstd 8 / 1 MB blocks, -12% with zstd 3 / 4 MB blocks.