rust/neon

mirror of https://github.com/neondatabase/neon.git synced 2026-05-19 14:10:37 +00:00

Files

Heikki Linnakangas e67746c41c Implement new vm-monitor - controller autoscaling protocol

This is the VM monitor implementation of the RFC at
https://github.com/neondatabase/neon/pull/8111.

I tried to keep the user-visible behavior unchanged from what we have
today. Improving the autoscaling algorithm is a separate topic, the
point of this work is just to move the algorihm from the autoscaler
agent to the VM monitor. That lays the groundwork for improving it
later, based on more metrics and signals inside the VM.

Some notable changes:

- I removed all the cgroup managing stuff. Instead of polling the
  cgroup memory threshold, this polls the overall system memory usage.

- The scaling algorithm is based on sliding window of load average and
  memory usage over the last minute. I'm not sure how close that is to
  the algorithm used by the autoscaler agent, I couldn't find a
  description of what exactly the algorithm used there is. I think
  this is close, but if not, it can be changed to match the agent's
  current algorithm more closely. I copied the LoadAverageFractionTarget
  and MemoryUsageFractionTarget settings from the autoscaler agent, with
  the defaults I found in the repo, but I'm not sure if we use different
  settings in production.

- I also didn't fully understand how the memory history logging in VM
  monitor, which was used to trigger upscaling. There is only one
  memory scaling codepath now, based on the max over 1-minute sliding
  window.

2024-06-19 16:58:10 +03:00

src

Implement new vm-monitor - controller autoscaling protocol

2024-06-19 16:58:10 +03:00

Cargo.toml

Implement new vm-monitor - controller autoscaling protocol

2024-06-19 16:58:10 +03:00

README.md

vm-monitor: Switch from memory.high to polling memory.stat (#5524 )

2023-10-17 15:30:40 -07:00

README.md

`vm-monitor`

The vm-monitor (or just monitor) is a core component of the autoscaling system, along with the autoscale-scheduler and the autoscaler-agents. The monitor has two primary roles: 1) notifying agents when immediate upscaling is necessary due to memory conditions and 2) managing Postgres' file cache and a cgroup to carry out upscaling and downscaling decisions.

More on scaling

We scale CPU and memory using NeonVM, our in-house QEMU tool for use with Kubernetes. To control thresholds for receiving memory usage notifications, we start Postgres in the neon-postgres cgroup and set its memory.{max,high}.

See also: neondatabase/autoscaling
See also: neondatabase/vm-monitor, where initial development of the monitor happened. The repository is no longer maintained but the commit history may be useful for debugging.

Structure

The vm-monitor is loosely comprised of a few systems. These are:

the server: this is just a simple axum server that accepts requests and upgrades them to websocket connections. The server only allows one connection at a time. This means that upon receiving a new connection, the server will terminate and old one if it exists.
the filecache: a struct that allows communication with the Postgres file cache. On startup, we connect to the filecache and hold on to the connection for the entire monitor lifetime.
the cgroup watcher: the CgroupWatcher polls the neon-postgres cgroup's memory usage and sends rolling aggregates to the runner.
the runner: the runner marries the filecache and cgroup watcher together, communicating with the agent throught the Dispatcher, and then calling filecache and cgroup watcher functions as needed to upscale and downscale