Luca Cominardi 24f329fcaa Memoize jitexpr value sources per input term ord
A jitexpr value source evaluated its expression once per doc. For an
expression of a text input, e.g. `(REGEXP_EXTRACT service "^([^-]+)" 1u64)`,
the result only depends on the term of the doc, and text inputs are
dictionary encoded with typically few distinct terms per segment (service
names, hosts, statuses). Evaluating the expression per doc made a terms
aggregation over such a calculated field ~100x slower than over the
physical column; on a CloudPrem searcher, the expression accounted for 91%
of the CPU of such a query.

Memoize the results of an expression of a single text input whose
dictionary has at most 65,536 terms, per input term ord. The expression is
evaluated on the first doc of each term, and later docs read the encoded
result from two dense arrays allocated zeroed when the source is opened
(9 bytes per term), plus a slot for docs without a value. Terms are only
resolved for the term ords evaluated for the first time. jitexpr functions
are deterministic, and `Str` results are interned in the order of the docs
as before, so the values and term ords are unchanged. `to_f64` clears the
memo, whose results are encoded for the previous result type. Other
expressions keep the per-doc path.

benches/jitexpr_agg_bench.rs (2M docs, 4 segments, 200 services):

  terms on (REGEXP_EXTRACT service ..)     140.6 ms -> 7.7 ms   (18x)
  terms on (UPPER service)                 305.7 ms -> 8.0 ms   (38x)
  terms on (REGEXP_EXTRACT request_id ..)  1.441 s  -> 1.444 s  (unique terms, no memo)
  terms on physical service                1.08 ms  -> 1.07 ms
2026-11-02 14:59:35 +01:00
2026-09-16 23:39:47 +08:00
2026-03-31 17:15:34 +08:00
2026-07-27 18:00:37 +02:00
2024-10-17 10:19:50 +09:00
2025-04-09 03:31:30 +02:00
2026-04-24 06:56:58 +00:00
2025-08-22 07:57:48 +08:00
2022-01-28 15:55:55 +09:00

Docs Build Status codecov OpenSSF Scorecard Join the chat at https://discord.gg/MT27AG5EVE License: MIT Crates.io

Tantivy, the fastest full-text search engine library written in Rust

Fast full-text search engine library written in Rust

If you are looking for an alternative to Elasticsearch or Apache Solr, check out Quickwit, our distributed search engine built on top of Tantivy.

Tantivy is closer to Apache Lucene than to Elasticsearch or Apache Solr in the sense it is not an off-the-shelf search engine server, but rather a crate that can be used to build such a search engine.

Tantivy is, in fact, strongly inspired by Lucene's design.

Benchmark

The following benchmark breaks down the performance for different types of queries/collections.

Your mileage WILL vary depending on the nature of queries and their load.

Details about the benchmark can be found at this repository.

Features

  • Full-text search
  • Configurable tokenizer (stemming available for 17 Latin languages) with third party support for Chinese (tantivy-jieba and cang-jie), Japanese (lindera, Vaporetto, and tantivy-tokenizer-tiny-segmenter) and Korean (lindera + lindera-ko-dic-builder)
  • Fast (check out the 🐎 ✨ benchmark ✨ 🐎)
  • Tiny startup time (<10ms), perfect for command-line tools
  • BM25 scoring (the same as Lucene)
  • Natural query language (e.g. (michael AND jackson) OR "king of pop")
  • Phrase queries search (e.g. "michael jackson")
  • Incremental indexing
  • Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop)
  • Mmap directory
  • SIMD integer compression when the platform/CPU includes the SSE2 instruction set
  • Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene)
  • &[u8] fast fields
  • Text, i64, u64, f64, dates, ip, bool, and hierarchical facet fields
  • Compressed document store (LZ4, Zstd, None)
  • Range queries
  • Faceted search
  • Configurable indexing (optional term frequency and position indexing)
  • JSON Field
  • Aggregation Collector: histogram, range buckets, average, and stats metrics
  • LogMergePolicy with deletes
  • Searcher Warmer API
  • Cheesy logo with a horse

Non-features

Distributed search is out of the scope of Tantivy, but if you are looking for this feature, check out Quickwit.

Getting started

Tantivy works on stable Rust and supports Linux, macOS, and Windows.

How can I support this project?

There are many ways to support this project.

  • Use Tantivy and tell us about your experience on Discord or by email (paul.masurel@gmail.com)
  • Report bugs
  • Write a blog post
  • Help with documentation by asking questions or submitting PRs
  • Contribute code (you can join our Discord server)
  • Talk about Tantivy around you

Contributing code

We use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR. Feel free to update CHANGELOG.md with your contribution.

Tokenizer

When implementing a tokenizer for tantivy depend on the tantivy-tokenizer-api crate.

Clone and build locally

Tantivy compiles on stable Rust. To check out and run tests, you can simply run:

git clone https://github.com/quickwit-oss/tantivy.git
cd tantivy
cargo test

Companies Using Tantivy

Etsy   ParadeDB   Nuclia   Humanfirst.ai Element.io Nuclia   Humanfirst.ai    Element.io

FAQ

Can I use Tantivy in other languages?

You can also find other bindings on GitHub but they may be less maintained.

What are some examples of Tantivy use?

  • seshat: A matrix message database/indexer
  • tantiny: Tiny full-text search for Ruby
  • lnx: adaptable, typo tolerant search engine with a REST API
  • Bichon: A lightweight, high-performance Rust email archiver with WebUI
  • and more!

On average, how much faster is Tantivy compared to Lucene?

Does tantivy support incremental indexing?

  • Yes.

How can I edit documents?

  • Data in tantivy is immutable. To edit a document, the document needs to be deleted and reindexed.

When will my documents be searchable during indexing?

  • Documents will be searchable after a commit is called on an IndexWriter. Existing IndexReaders will also need to be reloaded in order to reflect the changes. Finally, changes are only visible to newly acquired Searcher.
S
Description
Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust
Readme MIT Cite this repository
90 MiB
Languages
Rust 100%