mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-18 01:42:28 +00:00
dennis zhuang
5c4d9eb4f8
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
50 lines
1.8 KiB
Plaintext
50 lines
1.8 KiB
Plaintext
SELECT ts, host, val FROM t_downgrade_compatibility ORDER BY ts, host;
|
|
|
|
+---------------------+--------+-----+
|
|
| ts | host | val |
|
|
+---------------------+--------+-----+
|
|
| 2024-02-09T00:00:00 | host_a | 1 |
|
|
| 2024-02-09T00:01:00 | host_b | 2 |
|
|
+---------------------+--------+-----+
|
|
|
|
SELECT ts, host, val FROM t_preserve_sequence_downgrade ORDER BY ts, host;
|
|
|
|
+---------------------+--------+-----+
|
|
| ts | host | val |
|
|
+---------------------+--------+-----+
|
|
| 2024-02-09T00:00:00 | host_a | 1 |
|
|
| 2024-02-09T00:01:00 | host_b | 2 |
|
|
| 2024-02-09T00:02:00 | host_a | 3 |
|
|
| 2024-02-09T00:03:00 | host_c | 4 |
|
|
+---------------------+--------+-----+
|
|
|
|
SELECT ts, host, f, d
|
|
FROM t_sst_float_field_encoding_downgrade
|
|
ORDER BY ts, host;
|
|
|
|
+---------------------+--------+------+-------+
|
|
| ts | host | f | d |
|
|
+---------------------+--------+------+-------+
|
|
| 2024-02-10T00:00:00 | host_a | 1.25 | 10.5 |
|
|
| 2024-02-10T00:01:00 | host_b | -2.5 | 20.25 |
|
|
| 2024-02-10T00:02:00 | host_c | | |
|
|
+---------------------+--------+------+-------+
|
|
|
|
-- Older binaries don't recognize the bloom blob written by the current one and must
|
|
-- scan without pruning instead of probing it with their own tokens.
|
|
SELECT ts, msg FROM t_fulltext_bloom_v2_downgrade WHERE msg @@ 'hello_world' ORDER BY ts;
|
|
|
|
+-------------------------+-------------+
|
|
| ts | msg |
|
|
+-------------------------+-------------+
|
|
| 1970-01-01T00:00:00.001 | hello_world |
|
|
+-------------------------+-------------+
|
|
|
|
SELECT ts, msg FROM t_fulltext_bloom_v2_downgrade WHERE msg @@ '错误error日志' ORDER BY ts;
|
|
|
|
+-------------------------+---------------+
|
|
| ts | msg |
|
|
+-------------------------+---------------+
|
|
| 1970-01-01T00:00:00.002 | 错误error日志 |
|
|
+-------------------------+---------------+
|