rust/neon

mirror of https://github.com/neondatabase/neon.git synced 2026-05-17 13:10:38 +00:00

Files

Em Sharnoff 2c8741a5ed vm-monitor: Log full error on message handling failure (#5604 )

There's currently an issue with the vm-monitor on staging that's not
really feasible to debug because the current display impl gives no
context to the errors (just says "failed to downscale").

Logging the full error should help.

For communications with the autoscaler-agent, it's ok to only provide
the outermost cause, because we can cross-reference with the VM logs.
At some point in the future, we may want to change that.

2023-10-19 18:10:33 +02:00

src

vm-monitor: Log full error on message handling failure (#5604 )

2023-10-19 18:10:33 +02:00

Cargo.toml

compute_ctl: add vm-monitor (#4946 )

2023-08-24 15:54:37 -04:00

README.md

vm-monitor: Switch from memory.high to polling memory.stat (#5524 )

2023-10-17 15:30:40 -07:00

README.md

`vm-monitor`

The vm-monitor (or just monitor) is a core component of the autoscaling system, along with the autoscale-scheduler and the autoscaler-agents. The monitor has two primary roles: 1) notifying agents when immediate upscaling is necessary due to memory conditions and 2) managing Postgres' file cache and a cgroup to carry out upscaling and downscaling decisions.

More on scaling

We scale CPU and memory using NeonVM, our in-house QEMU tool for use with Kubernetes. To control thresholds for receiving memory usage notifications, we start Postgres in the neon-postgres cgroup and set its memory.{max,high}.

See also: neondatabase/autoscaling
See also: neondatabase/vm-monitor, where initial development of the monitor happened. The repository is no longer maintained but the commit history may be useful for debugging.

Structure

The vm-monitor is loosely comprised of a few systems. These are:

the server: this is just a simple axum server that accepts requests and upgrades them to websocket connections. The server only allows one connection at a time. This means that upon receiving a new connection, the server will terminate and old one if it exists.
the filecache: a struct that allows communication with the Postgres file cache. On startup, we connect to the filecache and hold on to the connection for the entire monitor lifetime.
the cgroup watcher: the CgroupWatcher polls the neon-postgres cgroup's memory usage and sends rolling aggregates to the runner.
the runner: the runner marries the filecache and cgroup watcher together, communicating with the agent throught the Dispatcher, and then calling filecache and cgroup watcher functions as needed to upscale and downscale