mirror of
https://github.com/neondatabase/neon.git
synced 2026-01-13 16:32:56 +00:00
2c544343e0
## Problem Currently, if one creates many shards they will all ingest all the data: not much use! We want them to ingest a proportional share of the data each. Closes: #6025 ## Summary of changes - WalIngest object gets a copy of the ShardIdentity for the Tenant it was created by. - While iterating the `blocks` part of a decoded record, blocks that do not match the current shard are ignored, apart from on shard zero where they are used to update relation sizes in `observe_decoded_block` (but not stored). - Before committing a `DataDirModificiation` from a WAL record, we check if it's empty, and drop the record if so. This check is necessary (rather than just looking at the `blocks` part) because certain record types may modify blocks in non-obvious ways (e.g. `ingest_heapam_record`). - Add WAL ingest metrics to record the total received, total committed, and total filtered out - Behaviour for unsharded tenants is unchanged: they will continue to ingest all blocks, and will take the fast path through `is_key_local` that doesn't bother calculating any hashes. After this change, shards store a subset of the tenant's total data, and accurate relation sizes are only maintained on shard zero. --------- Co-authored-by: Arpad Müller <arpad-m@users.noreply.github.com>