rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-01-08 22:12:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
b492cedf514b87de72c9e543cd87a68f3ddf9f66
neon/libs/vm_monitor
History
Em Sharnoff acef742a6e vm-monitor: Remove dependency on workspace_hack (#5752) 
neondatabase/autoscaling builds libs/vm-monitor during CI because it's a
necessary component of autoscaling.

workspace_hack includes a lot of crates that are not necessary for
vm-monitor, which artificially inflates the build time on the
autoscaling side, so hopefully removing the dependency should speed
things up.

Co-authored-by: Joonas Koivunen <joonas@neon.tech>
2023-11-07 09:41:20 -08:00
..
src
refactor: remove unnecessary unsafe (#5802)
2023-11-07 10:26:25 +00:00
Cargo.toml
vm-monitor: Remove dependency on workspace_hack (#5752)
2023-11-07 09:41:20 -08:00
README.md
vm-monitor: Switch from memory.high to polling memory.stat (#5524)
2023-10-17 15:30:40 -07:00

README.md

vm-monitor

The vm-monitor (or just monitor) is a core component of the autoscaling system, along with the autoscale-scheduler and the autoscaler-agents. The monitor has two primary roles: 1) notifying agents when immediate upscaling is necessary due to memory conditions and 2) managing Postgres' file cache and a cgroup to carry out upscaling and downscaling decisions.

More on scaling

We scale CPU and memory using NeonVM, our in-house QEMU tool for use with Kubernetes. To control thresholds for receiving memory usage notifications, we start Postgres in the neon-postgres cgroup and set its memory.{max,high}.

Structure

The vm-monitor is loosely comprised of a few systems. These are:

  • the server: this is just a simple axum server that accepts requests and upgrades them to websocket connections. The server only allows one connection at a time. This means that upon receiving a new connection, the server will terminate and old one if it exists.
  • the filecache: a struct that allows communication with the Postgres file cache. On startup, we connect to the filecache and hold on to the connection for the entire monitor lifetime.
  • the cgroup watcher: the CgroupWatcher polls the neon-postgres cgroup's memory usage and sends rolling aggregates to the runner.
  • the runner: the runner marries the filecache and cgroup watcher together, communicating with the agent throught the Dispatcher, and then calling filecache and cgroup watcher functions as needed to upscale and downscale