Commit Graph

3392 Commits

Author SHA1 Message Date
Anastasia Lubennikova
1104de0b9b refactoring
- enable CREATE EXTENSION and LOAD test
- change test_file_download to use mock_s3

- some code cleanup
- add caching of extensions_list

- WIP downloading of shared_preload_libraries (not tested yet)
2023-06-26 18:54:38 +03:00
Alek Westover
a8f848b5de test reals3 2023-06-23 15:51:49 -04:00
Alek Westover
ca59330df8 modify file names 2023-06-23 15:28:30 -04:00
Alek Westover
36bb5ad527 refactor 2023-06-23 14:47:48 -04:00
Alek Westover
6532daf528 add todos 2023-06-23 14:07:10 -04:00
Alek Westover
8bc128e474 real s3 tests 2023-06-23 13:41:08 -04:00
Alek Westover
c3994541eb add real s3 tests 2023-06-23 13:26:28 -04:00
Anastasia Lubennikova
7cdcc8a500 Fix downloading of sql files for extension and libraries.
Rust code refactoring and C code fixes.

Add test for CREATE EXTENSION and LOAD 'library'
2023-06-23 20:25:14 +03:00
Alek Westover
4201f4695f fix typo 2023-06-23 10:03:41 -04:00
Alek Westover
5776df15da try to add library extensions 2023-06-23 09:33:59 -04:00
Alek Westover
31aa0283b0 More Extension Features (#4555)
Added tenant specific extensions and more tests
2023-06-23 09:30:49 -04:00
Alek Westover
fd3dfe9d52 fix typo 2023-06-23 08:04:37 -04:00
Alek Westover
384e3ab1a8 fix code style 2023-06-22 15:10:08 -04:00
Alek Westover
4259464f72 fix typo 2023-06-22 15:07:28 -04:00
Alek Westover
152206211b turn remote extensions off by default 2023-06-22 15:02:51 -04:00
Alek Westover
9c35c06c58 small refactor 2023-06-22 14:24:59 -04:00
Alek Westover
245b4c9d72 remove env vars 2023-06-22 13:48:21 -04:00
Alek Westover
3d0f74fc0c fixing paths 2023-06-22 13:47:47 -04:00
Alek Westover
ce55f70cac added env vars 2023-06-22 12:23:30 -04:00
Alek Westover
053d592ddb fix minor bug 2023-06-22 12:08:28 -04:00
Alek Westover
b85416b58d satisfying mypy 2023-06-22 11:15:33 -04:00
Alek Westover
195838436c Merge branch 'extension_server' of github.com:neondatabase/neon into extension_server 2023-06-22 11:01:04 -04:00
Alek Westover
9313045de6 finished making the configuration optional 2023-06-22 11:00:38 -04:00
Alek Westover
44ac7a45be Merge branch 'main' into extension_server 2023-06-22 10:30:19 -04:00
Alek Westover
e35e8a7dcb None 2023-06-22 10:25:54 -04:00
Alek Westover
a79b0d69c4 made remote_ext_config an optional parameter 2023-06-22 10:21:07 -04:00
Anastasia Lubennikova
2f618f46be Use BUILD_TAG in compute_ctl binary. (#4541)
Pass BUILD_TAG to compute_ctl binary. 
We need it to access versioned extension storage.
2023-06-22 17:06:16 +03:00
Alek Westover
d475e901e5 fix more code style 2023-06-22 09:45:43 -04:00
Alek Westover
bf3b83b504 fix code style for clippy 2023-06-22 09:37:07 -04:00
Alek Westover
94781e8710 remote useless return 2023-06-21 17:26:20 -04:00
Alek Westover
4b83a206bf download extension from rust works 2023-06-21 17:22:21 -04:00
Alek Westover
f984f9e7d3 seems close to working 2023-06-21 15:25:06 -04:00
Alek Westover
6b42464c23 delete useless file 2023-06-21 14:34:37 -04:00
Alek Westover
605c30e5c5 fixed an issue where pgconfig was pointing at global installation of postgres rather than the correct local version 2023-06-21 14:34:24 -04:00
Alek Westover
0b11d8e836 replaced download_files function with more appropriate download_extensions function 2023-06-21 14:01:44 -04:00
Alek Westover
7602483af9 also need separate bucket if real s3 2023-06-21 13:25:35 -04:00
Alek Westover
5e1e859ab8 separate bucket for extensions 2023-06-21 13:18:35 -04:00
Alek Westover
85a7511700 remove extraneous argv passing 2023-06-21 12:42:17 -04:00
Alek Westover
89b8ea132e refactor more 2023-06-21 11:39:24 -04:00
Alek Westover
bfbae98f24 refactor 2023-06-21 11:32:44 -04:00
Alek Westover
02a1d4d8c1 refactoring a bit 2023-06-21 11:32:04 -04:00
Alexander Bayandin
d3aa8a48ea Update client libs for test_runner/pg_clients to their latest versions (#4547)
Resolves https://github.com/neondatabase/neon/security/dependabot/27
2023-06-21 16:20:35 +01:00
Alek Westover
4a35f29301 code style 2023-06-21 11:07:27 -04:00
Alek Westover
559e318328 remote dead imports 2023-06-21 11:01:45 -04:00
Alek Westover
a4d236b02f finishing cleanup debugging 2023-06-21 11:00:36 -04:00
Christian Schwarz
e4da76f021 update_gc_info: fix typo in timeline_id tracing field (#4546)
Commit

```
commit 472cc17b7a
Author: Dmitry Rodionov <dmitry@neon.tech>
Date:   Thu Jun 15 17:30:12 2023 +0300

    propagate lock guard to background deletion task (#4495)
```

did a drive-by fix, but, the drive-by had a typo.

```
gc_loop{tenant_id=2e2f2bff091b258ac22a4c4dd39bd25d}:update_gc_info{timline_id=837c688fd37c903639b9aa0a6dd3f1f1}:download_remote_layer{layer=000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000024DA0D1-000000000443FB51}:panic{thread=background op worker location=pageserver/src/tenant/timeline.rs:4843:25}: missing extractors: ["TimelineId"]

Stack backtrace:
   0: utils::logging::tracing_panic_hook
             at /libs/utils/src/logging.rs:166:21
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   5: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   6: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   7: pageserver::tenant::timeline::debug_assert_current_span_has_tenant_and_timeline_id
             at /pageserver/src/tenant/timeline.rs:4843:25
   8: <pageserver::tenant::timeline::Timeline>::download_remote_layer::{closure#0}::{closure#0}
             at /pageserver/src/tenant/timeline.rs:4368:9
   9: <tracing::instrument::Instrumented<<pageserver::tenant::timeline::Timeline>::download_remote_layer::{closure#0}::{closure#0}> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  10: <pageserver::tenant::timeline::Timeline>::download_remote_layer::{closure#0}
             at /pageserver/src/tenant/timeline.rs:4363:5
  11: <pageserver::tenant::timeline::Timeline>::get_reconstruct_data::{closure#0}
             at /pageserver/src/tenant/timeline.rs:2618:69
  12: <pageserver::tenant::timeline::Timeline>::get::{closure#0}
             at /pageserver/src/tenant/timeline.rs:565:13
  13: <pageserver::tenant::timeline::Timeline>::list_slru_segments::{closure#0}
             at /pageserver/src/pgdatadir_mapping.rs:427:42
  14: <pageserver::tenant::timeline::Timeline>::is_latest_commit_timestamp_ge_than::{closure#0}
             at /pageserver/src/pgdatadir_mapping.rs:390:13
  15: <pageserver::tenant::timeline::Timeline>::find_lsn_for_timestamp::{closure#0}
             at /pageserver/src/pgdatadir_mapping.rs:338:17
  16: <pageserver::tenant::timeline::Timeline>::update_gc_info::{closure#0}::{closure#0}
             at /pageserver/src/tenant/timeline.rs:3967:71
  17: <tracing::instrument::Instrumented<<pageserver::tenant::timeline::Timeline>::update_gc_info::{closure#0}::{closure#0}> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  18: <pageserver::tenant::timeline::Timeline>::update_gc_info::{closure#0}
             at /pageserver/src/tenant/timeline.rs:3948:5
  19: <pageserver::tenant::Tenant>::refresh_gc_info_internal::{closure#0}
             at /pageserver/src/tenant.rs:2687:21
  20: <pageserver::tenant::Tenant>::gc_iteration_internal::{closure#0}
             at /pageserver/src/tenant.rs:2551:13
  21: <pageserver::tenant::Tenant>::gc_iteration::{closure#0}
             at /pageserver/src/tenant.rs:1490:13
  22: pageserver::tenant::tasks::gc_loop::{closure#0}::{closure#0}
             at /pageserver/src/tenant/tasks.rs:187:21
  23: pageserver::tenant::tasks::gc_loop::{closure#0}
             at /pageserver/src/tenant/tasks.rs:208:5
```

## Problem

## Summary of changes

## Checklist before requesting a review

- [ ] I have performed a self-review of my code.
- [ ] If it is a core feature, I have added thorough tests.
- [ ] Do we need to implement analytics? if so did you add the relevant
metrics to the dashboard?
- [ ] If this PR requires public announcement, mark it with
/release-notes label and add several sentences in this section.

## Checklist before merging

- [ ] Do not forget to reformat commit message to not include the above
checklist
2023-06-21 18:00:14 +03:00
Alek Westover
8b9f72e117 removing debugging 2023-06-21 10:51:35 -04:00
Alek Westover
bb414e5a0a removing debugging 2023-06-21 10:45:37 -04:00
Alek Westover
32c03bc784 cleaning up some comments 2023-06-21 10:32:28 -04:00
Alek Westover
c99e203094 I think it's working 2023-06-21 10:10:02 -04:00