rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-01-16 09:52:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,214 Commits 1,487 Branches 282 Tags
proxy-http2-console
Commit Graph

4214 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
anastasia
0969574d48 Use bindgen for various xlog structures and checkpoint. 
Implement encode/decode methods for them.

Some methods are unused now. This is a preparatory commit for nonrel_wal
 2021-06-09 01:00:42 +03:00
anastasia
05a681be2c add createuser test to test shared catalog restore 2021-06-09 00:31:09 +03:00
anastasia
507c1fbbac bump vendor/postgres 2021-06-08 17:46:43 +03:00
Arseny Sher
b2f51026aa Consolidate PG proto parsing-deparsing and backend code. 
Now postgres_backend communicates with the client, passing queries to the
provided handler; we have two currently, for wal_acceptor and pageserver.

Now BytesMut is again used for writing data to avoid manual message length
calculation.

ref #118
 2021-06-08 17:31:40 +03:00
anastasia
2b0193e6bf implement from_bytes for XLogPageHeader structs 2021-06-08 13:08:57 +03:00
anastasia
c31a5e2c8f move XLogPageHeader structs to xlog_utils 2021-06-08 13:08:57 +03:00
anastasia
d85d67a6f1 use constants defined in xlog_utils for waldecoder 2021-06-08 13:08:57 +03:00
anastasia
9b8e82c6cf Fix makefile for out-of-tree compiling of contrib/zenith 2021-06-07 19:24:20 +03:00
Heikki Linnakangas
434374ebb4 Turn encode/decode into methods 
Like in PR #208
 2021-06-04 23:05:30 +03:00
Heikki Linnakangas
a7ae552851 Use rust memoffset crate to replace C offsetof(). 
Cherry-picked from Eric's PR #208
 2021-06-04 23:05:28 +03:00
Heikki Linnakangas
8b5a061c8e Add comments on the unsafe use of transmute in encode/decode_pg_control 
Note the unsafety of the unsafe block, with a link to the ongoing
discussion. This doesn't try to solve the problem, but let's at least
document the status quo.
 2021-06-04 23:05:26 +03:00
Heikki Linnakangas
8147aa7e93 Use u8 slice instead of Bytes in function argument. 
Bytes is handy, but in decode_pg_control's case it's just complicating
things. Also, pass ControlFileData by ref to encode_pg_control().
 2021-06-04 23:05:20 +03:00
Heikki Linnakangas
d18cc8a3a8 Update 'postgres_ffi' module's readme file and comments. 
Explain the purpose of of the 'postgres_ffi' module, explain what the
PostgreSQL control file is, and some other minor cleanup.
 2021-06-04 23:05:11 +03:00
Heikki Linnakangas
762e9859d6 Move functions for reading/writing control file to separate source file. 
To follow the precedent of xlog_utils.rs and relfile_utils.rs.
 2021-06-04 23:05:05 +03:00
Heikki Linnakangas
924261f7db Remove unused ControlFile::new() constructor. 
It has never been used, AFAICS.
 2021-06-04 23:05:02 +03:00
Konstantin Knizhnik
063429aade Implement GC for new object_store API (#229) 
* Implement GC for new object_store API

* Add comments for GC

* Revert postgres module version reference
 2021-06-04 20:11:56 +03:00
anastasia
445e88f50b Minor fix: preserve log of regression tests in circleci 2021-06-03 15:54:59 +03:00
Patrick Insinger
47694ea4f5 zenith push 2021-06-02 17:20:49 -04:00
Patrick Insinger
3364a8d442 pageserver - timeline history api 2021-06-02 16:20:26 -04:00
Dmitry Ivanov
f2243d7459 Merge pull request #219 from zenithdb/tidy-up-tests 
Tidy up pytest-based tests
 2021-06-01 22:06:13 +03:00
Dmitry Ivanov
244fcffc50 Fix typos found by codespell 2021-06-01 21:43:26 +03:00
Dmitry Ivanov
8c3c9c3394 Update README.md 2021-06-01 21:31:29 +03:00
Dmitry Ivanov
00ce635da9 Reformat tests using yapf 2021-06-01 21:09:09 +03:00
Dmitry Ivanov
7d5f7462c1 Tidy up pytest-based tests 2021-06-01 21:09:09 +03:00
Alexey Kondratov
bed75f800f Bump vendor/postgres to use rebased on top of REL_14_BETA1 version 2021-06-01 20:04:22 +03:00
anastasia
0e423d481e Update rustdoc comments and README for pageserver crate 2021-06-01 19:38:42 +03:00
anastasia
0c74f6fa4e Update README about source tree layout 2021-06-01 19:38:42 +03:00
Heikki Linnakangas
fc01fae9b4 Remove leftover references to safekeeper_proxy. 
We don't use it anymore. The WAL proposer is now a background worker that
runs as part of the primary Postgres server.
 2021-06-01 18:50:24 +03:00
Heikki Linnakangas
558a2214bc Fix comment 2021-06-01 18:28:01 +03:00
Heikki Linnakangas
31815bccc0 Remove obsolete script. 
It was used to launch the page server a long time ago. Doesn't work,
and isn't needed anymore.
 2021-06-01 18:27:54 +03:00
Stas Kelvich
e12cab7c17 fix dockerfile comments 2021-06-01 17:07:10 +03:00
Stas Kelvich
002cd8ed5b Dockerfile for pageserver. 2021-06-01 16:08:32 +03:00
Stas Kelvich
588a030803 Clean up control_plane's Cargo.toml 2021-06-01 16:08:32 +03:00
Stas Kelvich
0b9bbdc4ec Remove libxml2 and curl from postgres dependencies. 
As lazyrestore was removed from our postgres main branch.
 2021-06-01 16:08:32 +03:00
Heikki Linnakangas
922715a923 Use Timeline::get_last_record_lsn() to find end-of-WAL. 
That's less hacky than scanning the WAL directory.
 2021-06-01 09:37:54 +03:00
Heikki Linnakangas
ac60b68d50 Handle VM and FSM truncation WAL records in the page server. 
Fixes issue #190.

Original patch by Konstantin Knizhnik.
 2021-05-31 23:36:17 +03:00
Konstantin Knizhnik
1aceea1bdd Shutdown socket in ReplicationConn 2021-05-31 21:37:07 +03:00
Konstantin Knizhnik
e0cc4dee4f [refer #182] Make walkeeper periodically send callme requests to pageserver 2021-05-31 21:37:07 +03:00
Arseny Sher
fd20101e5c Configure pipenv for python tests. 2021-05-31 16:43:01 +03:00
anastasia
5a73a6fdfc add -w flag to wait till pg_ctl actually finishes what was asked 2021-05-28 20:33:16 +03:00
Stas Kelvich
4608b1ec70 Set wal_log_hints=on 
That is mandatory to correctly maintain visibility map (see issue#192).
It also makes sense to check that wal_log_hints is enabled at the pageserver side,
but for now let just check that tests will pass with this on.
 2021-05-28 11:38:46 +03:00
anastasia
ccb2eea7fd Add test_isolation that runs pg_isolation_regress for zenith 2021-05-28 11:38:46 +03:00
Heikki Linnakangas
6b615cbde1 Remove Copy marker from large ServerInfo struct. 
We don't want to encourage passing it by value. Doesn't matter much in
practice, but let's be tidy.

Per discussion at https://github.com/zenithdb/zenith/pull/195#issuecomment-849897327
 2021-05-27 23:16:54 +03:00
Heikki Linnakangas
adc0e04205 Misc cleanup of the 'zenith_regress' tests 
- Remove serial_schedule. As was alluded to in the README, it's really
  quote pointless.
- Remove unused PORT/HOST variables
- Fix typos
 2021-05-27 23:12:02 +03:00
Heikki Linnakangas
78e3edf2e9 Add CONTRIBUTING.md with some ground rules for submitting PRs. 
Let's try to rachet up our code quality, but at the same time keep the
bureaucracy low.
 2021-05-27 23:07:37 +03:00
Heikki Linnakangas
4b6563a55d Little cleanup on save_decoded_record function 
Pass DecodedWALRecord by reference, seems nicer, and probably marginally
faster.
 2021-05-27 20:50:45 +03:00
Heikki Linnakangas
34f4207501 Refactoring of the Repository/Timeline stuff 
- All timelines are now stored in the same rocksdb repository. The GET
  functions have been taught to follow the ancestors.

- Change the way relation size is stored. Instead of inserting "tombstone"
  entries for blocks that are truncated away, store relation size as
  separate key-value entry for each relation

- Add an abstraction for the key-value store: ObjectStore. It allows
  swapping RocksDB with some other key-value store easily. Perhaps we
  will write our own storage implementation using that interface, or
  perhaps we'll need a different abstraction, but this is a small
  improvement over status quo in any case.

- Garbage Collection is broken and commented out. It's not clear where and
  how it should be implemented.
 2021-05-27 20:07:50 +03:00
Heikki Linnakangas
d1d2d5ce69 Make multixact test more robust 
There was no guarantee that the SELECT FOR KEY SHARE queries actually
run in parallel. With unlucky timing, one query might finish before
the next one starts, so that the server doesn't need to create a
multixact. I got a failure like that on the CI:

    batch_others/test_multixact.py:56: in test_multixact
        assert(int(next_multixact_id) > int(next_multixact_id_old))
    E   AssertionError: assert 1 > 1
    E    +  where 1 = int('1')
    E    +  and   1 = int('1')

This could be reproduced by adding a random sleep in the runQuery
function, to make each query run at different times.

To fix, keep the transactions open after running the queries, so that
they will surely be open concurrently. With that, we can run the
queries serially, and don't need the 'multiprocessing' module anymore.

Fixes https://github.com/zenithdb/zenith/issues/196
 2021-05-27 20:00:52 +03:00
Heikki Linnakangas
60ea26316e Add LICENSE and COPYRIGHT files. 
This declares the software as Apache 2.0 licensed, and notes that the
vendor/postgres submodule is licensed under the PostgreSQL license.
 2021-05-27 15:33:08 +03:00
Heikki Linnakangas
cffc979058 Fix a couple of typos in comments. 2021-05-27 14:32:04 +03:00