rust/neon

mirror of https://github.com/neondatabase/neon.git synced 2026-01-07 13:32:57 +00:00

Files

Heikki Linnakangas 299db9d028 Simplify and clean up the $NEON_AUTH_TOKEN stuff in compute

- Remove the neon.safekeeper_token_env GUC. It was used to set the
  name of an environment variable, which was then used in pageserver
  and safekeeper connection strings to in place of the
  password. Instead, always look up the environment variable called
  NEON_AUTH_TOKEN. That's what neon.safekeeper_token_env was always
  set to in practice, and I don't see the need for the extra level of
  indirection or configurability.

- Instead of substituting $NEON_AUTH_TOKEN in the connection strings,
  pass $NEON_AUTH_TOKEN "out-of-band" as the password, when we connect
  to the pageserver or safekeepers. That's simpler.

- Also use the password from $NEON_AUTH_TOKEN in compute_ctl, when it
  connects to the pageserver to get the "base backup".

2023-03-21 00:15:04 +02:00

src

Simplify and clean up the $NEON_AUTH_TOKEN stuff in compute

2023-03-21 00:15:04 +02:00

tests

Fix escaping in postgresql.conf that we generate at compute startup

2023-03-10 14:59:21 +02:00

.dockerignore

Move compute_tools from console repo (zenithdb/console#383 )

2021-12-28 20:17:29 +03:00

.gitignore

Move compute_tools from console repo (zenithdb/console#383 )

2021-12-28 20:17:29 +03:00

Cargo.toml

Be able to get number of CPUs (#3774 )

2023-03-10 19:00:20 +02:00

README.md

Add VM informant to vm-compute-node (#3324 )

2023-01-16 07:05:29 -08:00

rustfmt.toml

Move compute_tools from console repo (zenithdb/console#383 )

2021-12-28 20:17:29 +03:00

README.md

Compute node tools

Postgres wrapper (compute_ctl) is intended to be run as a Docker entrypoint or as a systemd ExecStart option. It will handle all the Neon specifics during compute node initialization:

compute_ctl accepts cluster (compute node) specification as a JSON file.
Every start is a fresh start, so the data directory is removed and initialized again on each run.
Next it will put configuration files into the PGDATA directory.
Sync safekeepers and get commit LSN.
Get basebackup from pageserver using the returned on the previous step LSN.
Try to start postgres and wait until it is ready to accept connections.
Check and alter/drop/create roles and databases.
Hang waiting on the postmaster process to exit.

Also compute_ctl spawns two separate service threads:

compute-monitor checks the last Postgres activity timestamp and saves it into the shared ComputeNode;
http-endpoint runs a Hyper HTTP API server, which serves readiness and the last activity requests.

If the vm-informant binary is present at /bin/vm-informant, it will also be started. For VM compute nodes, vm-informant communicates with the VM autoscaling system. It coordinates downscaling and (eventually) will request immediate upscaling under resource pressure.

Usage example:

compute_ctl -D /var/db/postgres/compute \
            -C 'postgresql://cloud_admin@localhost/postgres' \
            -S /var/db/postgres/specs/current.json \
            -b /usr/local/bin/postgres

Tests

Cargo formatter:

cargo fmt

Run tests:

cargo test

Clippy linter:

cargo clippy --all --all-targets -- -Dwarnings -Drust-2018-idioms

Cross-platform compilation

Imaging that you are on macOS (x86) and you want a Linux GNU (x86_64-unknown-linux-gnu platform in rust terminology) executable.

Using docker

You can use a throw-away Docker container (rustlang/rust image) for doing that:

docker run --rm \
    -v $(pwd):/compute_tools \
    -w /compute_tools \
    -t rustlang/rust:nightly cargo build --release --target=x86_64-unknown-linux-gnu

or one-line:

docker run --rm -v $(pwd):/compute_tools -w /compute_tools -t rust:latest cargo build --release --target=x86_64-unknown-linux-gnu

Using rust native cross-compilation

Another way is to add x86_64-unknown-linux-gnu target on your host system:

rustup target add x86_64-unknown-linux-gnu

Install macOS cross-compiler toolchain:

brew tap SergioBenitez/osxct
brew install x86_64-unknown-linux-gnu

And finally run cargo build:

CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-unknown-linux-gnu-gcc cargo build --target=x86_64-unknown-linux-gnu --release