Exposes an endpoint "/profile/cpu" for profiling the postgres
processes (currently spawned and the new ones) using "perf".
Adds the corresponding python test to test the added endpoint
and confirm the output expected is the profiling data in the
expected format.
Add "perf" binary to the sudo list.
Fix python poetry ruff
Address the clippy lints
Document the code
Format python code
Address code review
Prettify
Embed profile_pb2.py and small code/test fixes.
Make the code slightly better.
1. Makes optional the sampling_frequency parameter for profiling.
2. Avoids using unsafe code when killing a child.
Better code, better tests
More tests
Separate start and stop of profiling
Correctly check for the exceptions
Address clippy lint
Final fixes.
1. Allows the perf to be found in $PATH instead of having the path
hardcoded.
2. Changes the path to perf in the sudoers file so that the compute
can run it properly.
3. Changes the way perf is invoked, now it is with sudo and the path
from $PATH.
4. Removes the authentication requirement from the /profile/cpu/
endpoint.
hakari thing
Python fixes
Fix python formatting
More python fixes
Update poetry lock
Fix ruff
Address the review comments
Fix the tests
Try fixing the flaky test for pg17?
Try fixing the flaky test for pg17?
PYTHON
Fix the tests
Remove the PROGRESS parameter
Remove unused
Increase the timeout due to concurrency
Increase the timeout to 60
Increase the profiling window timeout
Try this
Lets see the error
Just log all the errors
Add perf into the build environment
uijdfghjdf
Update tempfile to 3.20
Snapshot
Use bbc-profile
Update tempfile to 3.20
Provide bpfcc-tools in debian
Properly respond with status
Python check
Fix build-tools dockerfile
Add path probation for the bcc profile
Try err printing
Refactor
Add bpfcc-tools to the final image
Add error context
sudo not found?
Print more errors for verbosity
Remove procfs and use libproc
Update hakari
Debug sudo in CI
Rebase and adjust hakari
remove leftover
Add archiving support
Correct the paths to the perf binary
Try hardcoded sudo path
Add sudo into build-tools dockerfile
Minor cleanup
Print out the sudoers file from github
Stop the tests earlier
Add the sudoers entry for nonroot, install kmod for modprobe for bcc-profile
Try hacking the kernel headers for bcc-profile
Redeclare the kernel version argument
Try using the kernel of the runner
Try another way
Check bpfcc-tools
When a function is owned by a superuser (bootstrap user or otherwise),
we consider it safe to run it. Only a superuser could have installed it,
typically from CREATE EXTENSION script: we trust the code to execute.
## Problem
This is intended to solve running pg_graphql Event Triggers
graphql_watch_ddl and graphql_watch_drop which are executing the secdef
function graphql.increment_schema_version().
## Summary of changes
Allow executing Event Trigger function owned by a superuser and with
SECURITY DEFINER properties. The Event Trigger code runs with superuser
privileges, and we consider that it's fine.
---------
Co-authored-by: Tristan Partin <tristan.partin@databricks.com>
The current test was just SQL files only, but we also want to test a
remote extension which includes a loadable library. With both extensions
we should cover a larger portion of compute_ctl's remote extension code
paths.
Fixes: https://github.com/neondatabase/neon/issues/11146
Signed-off-by: Tristan Partin <tristan@neon.tech>
Previously, remote extensions were not fetched unless they were used in
some other manner. For instance, loading a BM25 index in pg_search
fetches the pg_search extension. However, if on a fresh compute with
pg_search 0.15.5 installed, the user ran `ALTER EXTENSION pg_search
UPDATE TO '0.15.6'` without first using the pg_search extension, we
would not fetch the extension and fail to find an update path.
Signed-off-by: Tristan Partin <tristan@neon.tech>
## Problem
In multi-character keys, the GIN index creates a CRC Hash of the first 3
bytes of the key.
The hash can have the first bit to be set or unset, needing to have a
consistent representation
of `char` across architectures for consistent results. GIN stores these
keys by their hashes
which determines the order in which the keys are obtained from the GIN
index.
By default, chars are signed in x86 and unsigned in arm, leading to
inconsistent behavior across different platform architectures. Adding
the `-fsigned-char` flag to the GCC compiler forces chars to be treated
as signed across platforms, ensuring the ordering in which the keys are
obtained consistent.
## Summary of changes
Added `-fsigned-char` to the `CFLAGS` to force GCC to use signed chars
across platforms.
Added a test to check this across platforms.
Fixes: https://github.com/neondatabase/cloud/issues/23199
This adds preliminary PG17 support to Neon, based on RC1 / 2024-09-04
07b828e9d4
NOTICE: The data produced by the included version of the PostgreSQL fork
may not be compatible with the future full release of PostgreSQL 17 due to
expected or unexpected future changes in magic numbers and internals.
DO NOT EXPECT DATA IN V17-TENANTS TO BE COMPATIBLE WITH THE 17.0
RELEASE!
Co-authored-by: Anastasia Lubennikova <anastasia@neon.tech>
Co-authored-by: Alexander Bayandin <alexander@neon.tech>
Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>
Co-authored-by: Heikki Linnakangas <heikki@neon.tech>
This adds PostgreSQL 16 as a vendored postgresql version, and adapts the
code to support this version.
The important changes to PostgreSQL 16 compared to the PostgreSQL 15
changeset include the addition of a neon_rmgr instead of altering Postgres's
original WAL format.
Co-authored-by: Alexander Bayandin <alexander@neon.tech>
Co-authored-by: Heikki Linnakangas <heikki@neon.tech>
Add infrastructure to dynamically load postgres extensions and shared libraries from remote extension storage.
Before postgres start downloads list of available remote extensions and libraries, and also downloads 'shared_preload_libraries'. After postgres is running, 'compute_ctl' listens for HTTP requests to load files.
Postgres has new GUC 'extension_server_port' to specify port on which 'compute_ctl' listens for requests.
When PostgreSQL requests a file, 'compute_ctl' downloads it.
See more details about feature design and remote extension storage layout in docs/rfcs/024-extension-loading.md
---------
Co-authored-by: Anastasia Lubennikova <anastasia@neon.tech>
Co-authored-by: Alek Westover <alek.westover@gmail.com>
