mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-26 21:56:17 +00:00
jeremyhi
baa5630119
* perf(log-store): keep the object store WAL reclaim boundary independent of the region count The WAL actor recomputed the reclaim boundary after every watermark update, release and indexed object by scanning the footers of the objects from the previous boundary, checking each segment against the obsolete watermarks. Each scan started at the first segment of the boundary object, so when one object holds segments of many regions whose watermarks advance one region at a time, every update inspected the segments already known to be obsolete again, about R^2/2 footer entries in total for R regions. The catalog now keeps the obsolete watermarks itself and counts the live segments of every object. A watermark move visits only the segments of its region that it newly reaches, through the per-region index, and the boundary is the first object at or above the floor that still holds a live segment. The boundary is unchanged: segments of release records never hold it, a region without a watermark does, a watermark of EntryId::MAX releases every segment of its region, and it never moves below the floor. The store records each watermark in the catalog together with the one reads use. The per-region index is now keyed by the largest entry id of each segment, which is also object sequence order because the entry ranges of a region increase strictly with the sequence. Indexing an object already enforced that; it now checks the neighbours by entry id instead of by sequence, which accepts and rejects the same objects. A property test checks the boundary against a scan of the footers over random appends, watermark moves, releases, drops, collections and recoveries, and ignored measurement tests time watermark updates and tail reads through the store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): seek the object store WAL objects of a region by entry id A read found the objects of its region by filtering every object of the region that the catalog retains, so a read of the last few objects cost time in proportion to the whole retained history of the region, under the catalog read lock. The per-region index is keyed by the largest entry id of each segment, and the entry ranges of a region increase strictly with the object sequence, which the catalog enforces when it indexes an object, whatever entry ids the objects carry. So the lookup seeks to the first segment that ends at or after the start of the range and stops at the first that starts after its end, and the result is still in sequence order. The property test of the catalog now also checks range lookups against a scan of the footers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>