rust/neon

mirror of https://github.com/neondatabase/neon.git synced 2026-01-10 06:52:55 +00:00

Files

Em Sharnoff 8d2a4aa5f8 vm-monitor: Add flag for when file cache on disk (#5130 )

Part 1 of 2, for moving the file cache onto disk.

Because VMs are created by the control plane (and that's where the
filesystem for the file cache is defined), we can't rely on any kind of
synchronization between releases, so the change needs to be
feature-gated (kind of), with the default remaining the same for now.

See also: neondatabase/cloud#6593

2023-08-29 12:44:48 -07:00

src

vm-monitor: Add flag for when file cache on disk (#5130 )

2023-08-29 12:44:48 -07:00

Cargo.toml

compute_ctl: add vm-monitor (#4946 )

2023-08-24 15:54:37 -04:00

README.md

monitor/compute_ctl: remove references to the informant (#5115 )

2023-08-29 02:59:27 +03:00

README.md

`vm-monitor`

The vm-monitor (or just monitor) is a core component of the autoscaling system, along with the autoscale-scheduler and the autoscaler-agents. The monitor has two primary roles: 1) notifying agents when immediate upscaling is necessary due to memory conditions and 2) managing Postgres' file cache and a cgroup to carry out upscaling and downscaling decisions.

More on scaling

We scale CPU and memory using NeonVM, our in-house QEMU tool for use with Kubernetes. To control thresholds for receiving memory usage notifications, we start Postgres in the neon-postgres cgroup and set its memory.{max,high}.

See also: neondatabase/autoscaling
See also: neondatabase/vm-monitor, where initial development of the monitor happened. The repository is no longer maintained but the commit history may be useful for debugging.

Structure

The vm-monitor is loosely comprised of a few systems. These are:

the server: this is just a simple axum server that accepts requests and upgrades them to websocket connections. The server only allows one connection at a time. This means that upon receiving a new connection, the server will terminate and old one if it exists.
the filecache: a struct that allows communication with the Postgres file cache. On startup, we connect to the filecache and hold on to the connection for the entire monitor lifetime.
the cgroup watcher: the CgroupWatcher manages the neon-postgres cgroup by listening for memory.high events and setting its memory.{high,max} values.
the runner: the runner marries the filecache and cgroup watcher together, communicating with the agent throught the Dispatcher, and then calling filecache and cgroup watcher functions as needed to upscale and downscale