Christian Schwarz
f32ebb74a1
refactor: move the ldpreload setup code into a method on EvictionEnv
2023-03-29 15:55:13 +02:00
Christian Schwarz
c8784cba6b
wire up the statvfs ldpreload thing in an example test
2023-03-29 15:44:14 +02:00
Christian Schwarz
555ccb8c91
feat: add LD_PRELOADable library for mocking statvfs
...
use like so:
env RUST_LOG=pageserver=info,pageserver::disk_usage_eviction_task=debug LD_PRELOAD=$PWD/target/debug/libstatvfs_ldpreload.so NEON_STATVFS_LDPRELOAD_CONFIG="$(echo '{}' | jq '{magic: "foobar", mock: { type: "Failure", mocked_error: "EIO" }}')" ./target/debug/neon_local pageserver start
2023-03-29 15:01:45 +02:00
Christian Schwarz
216f613e24
Merge pull request #3890 from neondatabase/heikki/disk-usage-eviction
...
Rewrite parts of disk usage eviction implementation to make it more understandable (I hope).
2023-03-29 12:43:38 +02:00
Christian Schwarz
b47a02569f
tests: fully read-only warmup + wait for remote storage upload
2023-03-29 11:55:02 +02:00
Christian Schwarz
699ca672a4
test: refine test_pageserver_respects_overridden_resident_size
2023-03-29 11:22:53 +02:00
Christian Schwarz
a698ddb8a4
fix: avoid needless timeline.clone()
2023-03-29 10:58:59 +02:00
Christian Schwarz
57d215e6bb
fix: suggestions commited from GitHub web didn't compile
2023-03-29 10:55:57 +02:00
Christian Schwarz
83813f2cb1
fix: remove unneeded clippy allow
2023-03-29 10:55:45 +02:00
Christian Schwarz
bdc7f8d192
fix: remove now-unused is_sorted
2023-03-29 10:52:56 +02:00
Christian Schwarz
9a55e4f909
fix: structured logging of tenant_id
...
Co-authored-by: Joonas Koivunen <joonas@neon.tech >
2023-03-29 10:51:14 +02:00
Christian Schwarz
0b9a44a879
fix: structured logging of tenant_id
...
Co-authored-by: Joonas Koivunen <joonas@neon.tech >
2023-03-29 10:50:36 +02:00
Christian Schwarz
a6f9ebf178
fix: repeat tenant_id in debug message
2023-03-29 10:49:06 +02:00
Christian Schwarz
370b3637db
doc: add explainer to debug_assert
2023-03-29 10:47:15 +02:00
Christian Schwarz
88753b3325
doc: link follow-up issue in TODO comment
2023-03-29 10:36:55 +02:00
Christian Schwarz
bb5947afde
test: test_pageserver_respects_overridden_resident_size: use absolute wiggle room instead of percentage
...
Heikki added the `*0.75` in
commit 11b16614a3heikki@neon.tech >
Date: Tue Mar 28 01:13:33 2023 +0300
Fix test for change in behavior close to the min_resident_size boundary
This PR changed the behavior to match my expectation per my comment:
https://github.com/neondatabase/neon/pull/3809/files#r1149837135
Without it, the test fails because we fall back to global LRU, and we
have an assert on that.
The reason why it falls back to global LRU is that
`target = delta_between_small_and_big_tenant`
doesn't leave any wiggle-room to go over min_resident_size boundary.
But, we redefined min_resident_size to include up to 1 layer above it
in this branch.
Multiply that by two because we're dealing with 2 tenants here.
2023-03-28 19:16:59 +02:00
Christian Schwarz
d6c2867b46
doc: add debug_assert for self-documenting candidates.sort_unstable_by_kye()
2023-03-28 19:16:26 +02:00
Christian Schwarz
386c2d0112
refactor: go back to a single list
...
The MinResidentSizePartition is effectively what `overage` was earlier,
but more expressive and outside of EvictionCandidates.
So switch the code back to a single list,
but use (MinResidentSizePartition, EvictionCandidates) tuples.
That eliminates the need for iter_in_eviction_order() alltogether.
It consumes 8 bytes more memory per candidate, but, that doesn't matter
for now.
2023-03-28 18:25:44 +02:00
Christian Schwarz
704d4f4640
doc: improve comment on min_resident_size
2023-03-28 18:06:44 +02:00
Christian Schwarz
dc72a9534e
doc: update doc comment for collect_eviction_candidates
...
And move the impl of MinResidentSizePartitionedCandidates
below it because it makes sense when reading the code top-down.
2023-03-28 18:05:31 +02:00
Christian Schwarz
07c44f9151
doc: hint that usage_assumed is modified in the loop
2023-03-28 17:47:11 +02:00
Christian Schwarz
0c10e6d3e7
feat: demote info logs to debug
...
These would be per tenant, we don't want to emit thousands of log lines
when this code runs.
2023-03-28 17:36:59 +02:00
Christian Schwarz
85becb148f
feat: bring back min_resident=max(all layers) behavior
2023-03-28 17:36:59 +02:00
Christian Schwarz
ea3c76a9d6
refactor: instead of 'overage', have two separate lists
2023-03-28 17:36:59 +02:00
Christian Schwarz
799576ab1e
Merge branch 'problame/disk-usage-eviction' into heikki/disk-usage-eviction
2023-03-28 15:24:52 +02:00
Joonas Koivunen
b1d54024e7
doc: why tokio mutex instead of std mutex
2023-03-28 14:14:14 +03:00
Christian Schwarz
18ed0f9a06
enable disk-usage-based eviction in staging
...
We'll do some manual testing of the statvfs loop there.
2023-03-28 12:52:16 +02:00
Christian Schwarz
453c3fd2da
doc: fix typo
...
Co-authored-by: Heikki Linnakangas <heikki@neon.tech >
2023-03-28 12:22:26 +02:00
Christian Schwarz
0428b6822a
doc: more comment on lru_candidates to address questions from review
2023-03-28 11:53:48 +02:00
Christian Schwarz
42d63270a5
doc: add comment on extend_lru_candidates
2023-03-28 11:44:10 +02:00
Christian Schwarz
0056108c45
doc: remove stray comment
2023-03-28 11:44:10 +02:00
Christian Schwarz
54cc1d5064
doc: sub-headings for mechanics & policy in module comment
2023-03-28 11:44:10 +02:00
Joonas Koivunen
70c837a4b2
refactor: simplify max_layer_size as u64
2023-03-28 12:44:01 +03:00
Christian Schwarz
38d3061143
add comment on not-need to be 100% accurate about max_layer_size
2023-03-28 12:44:01 +03:00
Joonas Koivunen
75759f709f
doc: explain "bug" message, log layers
2023-03-28 12:44:01 +03:00
Joonas Koivunen
03ab5df081
chore: remove dead code
2023-03-28 12:44:01 +03:00
Joonas Koivunen
6e8d7b449f
refactor: combine nested macthes
2023-03-28 12:44:01 +03:00
Joonas Koivunen
17b5c8d1c4
refactor: get rid of ApproxAccurate
2023-03-28 12:44:01 +03:00
Joonas Koivunen
0943dd30eb
chore: clippy
2023-03-28 12:44:01 +03:00
Joonas Koivunen
b599755042
refactor: rename DiskUsageEvictionState => State
2023-03-28 12:44:01 +03:00
Joonas Koivunen
244185e6e6
doc: comment changes
...
Co-authored-by: Christian Schwarz <christian@neon.tech >
2023-03-28 12:44:01 +03:00
Joonas Koivunen
0a5043fae5
refactor: less static mutexes
2023-03-28 12:44:01 +03:00
Heikki Linnakangas
041b708dc6
rustfmt
2023-03-28 11:08:35 +03:00
Heikki Linnakangas
11b16614a3
Fix test for change in behavior close to the min_resident_size boundary
...
This PR changed the behavior to match my expectation per my comment:
https://github.com/neondatabase/neon/pull/3809/files#r1149837135
2023-03-28 01:28:27 +03:00
Heikki Linnakangas
b6b8265450
Rewrite to make the algorithm more understandable (I hope).
...
The algorithm is the same (with two small exceptions), but rewrite the
way it's implemented to make it easier to follow.
The exceptions:
1. 'min_resident_size' now protects at least that much data in the first
"respectful" phase of the algorithm. Previously, it would evict layers
until the resident size fell below min_resident_size. In other words,
we know protect one more layer of each tenant, so that the resident
size stays just above min_resident_size, while previously we would
evict enough to bring the resident size just under min_resident_size.
2. Previously, the "max layer size" that's used as the default
min_resident_size was calculated from *all* layers in the tenant,
including remote layers. Now it's only calculated across all
locally-present layers. I don't know if that was a deliberate choice,
but this is slightly simpler.
2023-03-28 01:28:27 +03:00
Christian Schwarz
0462563d31
doc: more explanation of the added config knobs
2023-03-27 19:52:48 +02:00
Christian Schwarz
ef39b3f067
rename: dedupe comment about allow(dead_code)
2023-03-27 19:27:41 +02:00
Christian Schwarz
e1ba2f96ab
fix: post-merge, file_size() returns u64
2023-03-27 19:07:34 +02:00
Christian Schwarz
de3a7470de
Merge remote-tracking branch 'origin/main' into problame/disk-usage-eviction
2023-03-27 19:04:55 +02:00
Christian Schwarz
504256a0cc
rename: serde_percent::{Value => Percent}
2023-03-27 18:54:32 +02:00
