mirror of
https://github.com/neondatabase/neon.git
synced 2026-01-03 11:32:56 +00:00
Update zenith readmes:
- Move source tree overview into separate docs/sourcetree.md and update it. - Add glossary: docs/glossary.md - Add a draft of Architecture overview to main Readme.md
This commit is contained in:
anastasia
69
README.md
69
README.md
@@ -2,6 +2,22 @@
|
||||
|
||||
Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes
|
||||
|
||||
## Architecture overview
|
||||
|
||||
A Zenith installation consists of Compute nodes and Storage engine.
|
||||
|
||||
Compute nodes are stateles PostgreSQL nodes, backed by zenith storage.
|
||||
|
||||
Zenith storage engine consists of two major components:
|
||||
- Pageserver. Scalable storage backend for compute nodes.
|
||||
- WAL service. The service that receives WAL from compute node and ensures that it is stored durably.
|
||||
|
||||
Pageserver consists of:
|
||||
- Page cache repository - Zenith storage implementation.
|
||||
- WAL receiver - service that recieves WAL from WAL service and stores it in the page cache repository.
|
||||
- Page service - service that communicates with compute nodes and responds with pages from the repository.
|
||||
- WAL redo - service that builds pages from base images and WAL records on Page service request.
|
||||
|
||||
## Running local installation
|
||||
|
||||
1. Install build dependencies and other useful packages
|
||||
@@ -105,52 +121,13 @@ pytest
|
||||
|
||||
Now we use README files to cover design ideas and overall architecture for each module and `rustdoc` style documentation comments. See also [/docs/](/docs/) a top-level overview of all available markdown documentation.
|
||||
|
||||
- [/docs/sourcetree.md](/docs/sourcetree.md) contains overview of source tree layout.
|
||||
|
||||
To view your `rustdoc` documentation in a browser, try running `cargo doc --no-deps --open`
|
||||
|
||||
## Source tree layout
|
||||
## Join the development
|
||||
|
||||
`/control_plane`:
|
||||
|
||||
Local control plane.
|
||||
Functions to start, configure and stop pageserver and postgres instances running as a local processes.
|
||||
Intended to be used in integration tests and in CLI tools for local installations.
|
||||
|
||||
`/zenith`
|
||||
|
||||
Main entry point for the 'zenith' CLI utility.
|
||||
TODO: Doesn't it belong to control_plane?
|
||||
|
||||
`/postgres_ffi`:
|
||||
|
||||
Utility functions for interacting with PostgreSQL file formats.
|
||||
Misc constants, copied from PostgreSQL headers.
|
||||
|
||||
`/zenith_utils`:
|
||||
|
||||
Helpers that are shared between other crates in this repository.
|
||||
|
||||
`/walkeeper`:
|
||||
|
||||
WAL safekeeper (also known as WAL acceptor). Written in Rust.
|
||||
|
||||
`/pageserver`:
|
||||
|
||||
Page Server. Written in Rust.
|
||||
|
||||
Depends on the modified 'postgres' binary for WAL redo.
|
||||
|
||||
`/vendor/postgres`:
|
||||
|
||||
PostgreSQL source tree, with the modifications needed for Zenith.
|
||||
|
||||
`/vendor/postgres/contrib/zenith`:
|
||||
|
||||
PostgreSQL extension that implements storage manager API and network communications with remote page server.
|
||||
|
||||
`/test_runner`:
|
||||
|
||||
Integration tests, written in Python using the `pytest` framework.
|
||||
|
||||
`test_runner/zenith_regress`:
|
||||
|
||||
Quick way to add new SQL regression test to integration tests set.
|
||||
- Read `CONTRIBUTING.md` to learn about project code style and practices.
|
||||
- Use glossary in [/docs/glossary.md](/docs/glossary.md)
|
||||
- To get familiar with a source tree layout, use [/docs/sourcetree.md](/docs/sourcetree.md).
|
||||
- To learn more about PostgreSQL internals, check http://www.interdb.jp/pg/index.html
|
||||
|
||||
184
docs/glossary.md
Normal file
184
docs/glossary.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# Glossary
|
||||
|
||||
### Authentication
|
||||
|
||||
### Base image (page image)
|
||||
|
||||
### Basebackup
|
||||
|
||||
A tarball with files needed to bootstrap a compute node[] and a corresponding command to create it.
|
||||
NOTE:It has nothing to do with PostgreSQL pg_basebackup.
|
||||
|
||||
### Branch
|
||||
|
||||
We can create branch at certain LSN using `zenith branch` command.
|
||||
Each Branch lives in a corresponding timeline[] and has an ancestor[].
|
||||
|
||||
|
||||
### Checkpoint (PostgreSQL)
|
||||
|
||||
NOTE: This is an overloaded term.
|
||||
|
||||
A checkpoint record in the WAL marks a point in the WAL sequence at which it is guaranteed that all data files have been updated with all information from shared memory modified before that checkpoint;
|
||||
|
||||
### Checkpoint (Layered repository)
|
||||
|
||||
NOTE: This is an overloaded term.
|
||||
|
||||
Whenever enough WAL has been accumulated in memory, the page server []
|
||||
writes out the changes in memory into new snapshot files[]. This process
|
||||
is called "checkpointing". The page server only creates snapshot files for
|
||||
relations that have been modified since the last checkpoint.
|
||||
|
||||
### Compute node
|
||||
|
||||
Stateless Postgres node that stores data in pageserver.
|
||||
|
||||
### Garbage collection
|
||||
|
||||
### Fork
|
||||
|
||||
Each of the separate segmented file sets in which a relation is stored. The main fork is where the actual data resides. There also exist two secondary forks for metadata: the free space map and the visibility map.
|
||||
Each PostgreSQL fork is considered a separate relish.
|
||||
|
||||
### Layered repository
|
||||
|
||||
### LSN
|
||||
|
||||
|
||||
### Page (block)
|
||||
|
||||
The basic structure used to store relation data. All pages are of the same size.
|
||||
This is the unit of data exchange between compute node and pageserver.
|
||||
|
||||
### Pageserver
|
||||
|
||||
Zenith storage engine: page cache repositories + wal receiver + page service + wal redo.
|
||||
|
||||
### Page cache
|
||||
|
||||
This module acts as a switchboard to access different repositories managed by this pageserver.
|
||||
|
||||
### Page service
|
||||
|
||||
The Page Service listens for GetPage@LSN requests from the Compute Nodes,
|
||||
and responds with pages from the repository.
|
||||
|
||||
|
||||
### PITR (Point-in-time-recovery)
|
||||
|
||||
PostgreSQL's ability to restore up to a specified LSN.
|
||||
|
||||
### Primary node
|
||||
|
||||
|
||||
### Proxy
|
||||
|
||||
Postgres protocol proxy/router.
|
||||
This service listens psql port, can check auth via external service
|
||||
and create new databases and accounts (control plane API in our case).
|
||||
|
||||
### Relation
|
||||
|
||||
The generic term in PostgreSQL for all objects in a database that have a name and a list of attributes defined in a specific order.
|
||||
|
||||
### Relish
|
||||
|
||||
We call each relation and other file that is stored in the
|
||||
repository a "relish". It comes from "rel"-ish, as in "kind of a
|
||||
rel", because it covers relations as well as other things that are
|
||||
not relations, but are treated similarly for the purposes of the
|
||||
storage layer.
|
||||
|
||||
### Replication slot
|
||||
|
||||
|
||||
### Replica node
|
||||
|
||||
|
||||
### Repository
|
||||
|
||||
Repository stores multiple timelines, forked off from the same initial call to 'initdb'
|
||||
and has associated WAL redo service.
|
||||
One repository corresponds to one Tenant.
|
||||
|
||||
### SLRU
|
||||
|
||||
SLRUs include pg_clog, pg_multixact/members, and
|
||||
pg_multixact/offsets. There are other SLRUs in PostgreSQL, but
|
||||
they don't need to be stored permanently (e.g. pg_subtrans),
|
||||
or we do not support them in zenith yet (pg_commit_ts).
|
||||
Each SLRU segment is considered a separate relish[].
|
||||
|
||||
### Snapshot file
|
||||
|
||||
Layered repository on-disk format is based on immutable files.
|
||||
The files are called "snapshot files".
|
||||
Each snapshot file contains a full snapshot, that is, full copy of all
|
||||
pages in the relation, as of the "start LSN". It also contains all WAL
|
||||
records applicable to the relation between the start and end
|
||||
LSNs.
|
||||
Each snapshot file corresponds to one 10 MB slice of a PostgreSQL relation fork.
|
||||
|
||||
### Tenant (Multitenancy)
|
||||
Tenant represents a single customer, interacting with Zenith.
|
||||
Wal redo[] activity, timelines[], snapshots[] are managed for each tenant independently.
|
||||
One pageserver[] can serve multiple tenants at once.
|
||||
One safekeeper
|
||||
|
||||
See `docs/multitenancy.md` for more.
|
||||
|
||||
### Timeline
|
||||
|
||||
Timeline is a page cache workhorse that accepts page changes
|
||||
and serves get_page_at_lsn() and get_rel_size() requests.
|
||||
|
||||
NOTE: this has nothing to do with PostgreSQL WAL timelines.
|
||||
|
||||
### XLOG
|
||||
|
||||
PostgreSQL alias for WAL[].
|
||||
|
||||
### WAL (Write-ahead log)
|
||||
|
||||
The journal that keeps track of the changes in the database cluster as user- and system-invoked operations take place. It comprises many individual WAL records[] written sequentially to WAL files[].
|
||||
|
||||
### WAL acceptor, WAL proposer
|
||||
|
||||
In the context of the consensus algorithm, the Postgres
|
||||
compute node is also known as the WAL proposer, and the safekeeper is also known
|
||||
as the acceptor. Those are the standard terms in the Paxos algorithm.
|
||||
|
||||
### WAL receiver (WAL decoder)
|
||||
|
||||
The WAL receiver connects to the external WAL safekeeping service (or
|
||||
directly to the primary) using PostgreSQL physical streaming
|
||||
replication, and continuously receives WAL. It decodes the WAL records,
|
||||
and stores them to the page cache repository.
|
||||
|
||||
We keep one WAL receiver active per timeline.
|
||||
|
||||
### WAL record
|
||||
|
||||
A low-level description of an individual data change.
|
||||
|
||||
### WAL redo
|
||||
|
||||
A service that runs PostgreSQL in a special wal_redo mode
|
||||
to apply given WAL records over an old page image and return new page image.
|
||||
|
||||
### WAL safekeeper
|
||||
|
||||
One node that participates in the quorum. All the safekeepers
|
||||
together form the WAL service.
|
||||
|
||||
### WAL segment (WAL file)
|
||||
|
||||
Also known as WAL segment or WAL segment file. Each of the sequentially-numbered files that provide storage space for WAL. The files are all of the same predefined size and are written in sequential order, interspersing changes as they occur in multiple simultaneous sessions.
|
||||
|
||||
### WAL service
|
||||
|
||||
The service as whole that ensures that WAL is stored durably.
|
||||
|
||||
### Web console
|
||||
|
||||
81
docs/sourcetree.md
Normal file
81
docs/sourcetree.md
Normal file
@@ -0,0 +1,81 @@
|
||||
## Source tree layout
|
||||
|
||||
Below you will find a brief overview of each subdir in the source tree in alphabetical order.
|
||||
|
||||
`/control_plane`:
|
||||
|
||||
Local control plane.
|
||||
Functions to start, configure and stop pageserver and postgres instances running as a local processes.
|
||||
Intended to be used in integration tests and in CLI tools for local installations.
|
||||
|
||||
`/docs`:
|
||||
|
||||
Documentaion of the Zenith features and concepts.
|
||||
Now it is mostly dev documentation.
|
||||
|
||||
`monitoring`:
|
||||
|
||||
TODO
|
||||
|
||||
`/pageserver`:
|
||||
|
||||
Zenith storage service.
|
||||
The pageserver has a few different duties:
|
||||
|
||||
- Store and manage the data.
|
||||
- Generate a tarball with files needed to bootstrap ComputeNode.
|
||||
- Respond to GetPage@LSN requests from the Compute Nodes.
|
||||
- Receive WAL from the WAL service and decode it.
|
||||
- Replay WAL that's applicable to the chunks that the Page Server maintains
|
||||
|
||||
For more detailed info, see `/pageserver/README`
|
||||
|
||||
`/postgres_ffi`:
|
||||
|
||||
Utility functions for interacting with PostgreSQL file formats.
|
||||
Misc constants, copied from PostgreSQL headers.
|
||||
|
||||
`/proxy`:
|
||||
|
||||
Postgres protocol proxy/router.
|
||||
This service listens psql port, can check auth via external service
|
||||
and create new databases and accounts (control plane API in our case).
|
||||
|
||||
`/test_runner`:
|
||||
|
||||
Integration tests, written in Python using the `pytest` framework.
|
||||
|
||||
`/vendor/postgres`:
|
||||
|
||||
PostgreSQL source tree, with the modifications needed for Zenith.
|
||||
|
||||
`/vendor/postgres/contrib/zenith`:
|
||||
|
||||
PostgreSQL extension that implements storage manager API and network communications with remote page server.
|
||||
|
||||
`/vendor/postgres/contrib/zenith_test_utils`:
|
||||
|
||||
PostgreSQL extension that contains functions needed for testing and debugging.
|
||||
|
||||
`/walkeeper`:
|
||||
|
||||
The zenith WAL service that receives WAL from a primary compute nodes and streams it to the pageserver.
|
||||
It acts as a holding area and redistribution center for recently generated WAL.
|
||||
|
||||
For more detailed info, see `/walkeeper/README`
|
||||
|
||||
`/workspace_hack`:
|
||||
The workspace_hack crate exists only to pin down some dependencies.
|
||||
|
||||
`/zenith`
|
||||
|
||||
Main entry point for the 'zenith' CLI utility.
|
||||
TODO: Doesn't it belong to control_plane?
|
||||
|
||||
`zenith_metrics`:
|
||||
|
||||
TODO
|
||||
|
||||
`/zenith_utils`:
|
||||
|
||||
Helpers that are shared between other crates in this repository.
|
||||
@@ -1,16 +1,5 @@
|
||||
## Page server architecture
|
||||
|
||||
The Page Server is responsible for all operations on a number of
|
||||
"chunks" of relation data. A chunk corresponds to a PostgreSQL
|
||||
relation segment (i.e. one max. 1 GB file in the data directory), but
|
||||
it holds all the different versions of every page in the segment that
|
||||
are still needed by the system.
|
||||
|
||||
Currently we do not specifically organize data in chunks.
|
||||
All page images and corresponding WAL records are stored as entries in a key-value storage,
|
||||
where StorageKey is a zenith_timeline_id + BufferTag + LSN.
|
||||
|
||||
|
||||
The Page Server has a few different duties:
|
||||
|
||||
- Respond to GetPage@LSN requests from the Compute Nodes
|
||||
@@ -72,7 +61,7 @@ WAL Receiver
|
||||
The WAL receiver connects to the external WAL safekeeping service (or
|
||||
directly to the primary) using PostgreSQL physical streaming
|
||||
replication, and continuously receives WAL. It decodes the WAL records,
|
||||
and stores them to the page cache.
|
||||
and stores them to the page cache repository.
|
||||
|
||||
|
||||
Page Cache
|
||||
@@ -83,6 +72,11 @@ The Page Cache is a switchboard to access different Repositories.
|
||||
#### Repository
|
||||
Repository corresponds to one .zenith directory.
|
||||
Repository is needed to manage Timelines.
|
||||
Each repository has associated WAL redo service.
|
||||
|
||||
Now we have two implementations of Repository:
|
||||
- ObjectRepository uses RocksDB as a storage
|
||||
- LayeredRepository uses custom storage format, described in layered_repository/README.md
|
||||
|
||||
#### Timeline
|
||||
Timeline is a page cache workhorse that accepts page changes
|
||||
|
||||
Reference in New Issue
Block a user