3 Commits

Author	SHA1	Message	Date
Arseny Sher	1bb9abebf2	Remove WAL segments from s3 in batches. ... Do list-delete operations in batches instead of doing full list first, to ensure deletion makes progress even if there are a lot of files to remove. To this end, add max_keys limit to remote storage list_files.	2024-02-09 22:11:53 +04:00
Arpad Müller	6ffdcfe6a4	remote_storage: unify azure and S3 tests (#6364) ... The remote_storage crate contains two copies of each test, one for azure and one for S3. The repetition is not necessary and makes the tests more prone to drift, so we remove it by moving the tests into a shared module. The module has a different name depending on where it is included, so that each test still has "s3" or "azure" in its full path, allowing you to just test the S3 test or just the azure tests. Earlier PR that removed some duplication already: #6176 Fixes #6146.	2024-01-16 18:45:19 +01:00
Arpad Müller	fbb979d5e3	remote_storage: move shared utilities for S3 and Azure into common module (#6176) ... The PR does two things: * move the util functions present in the remote_storage Azure and S3 test files into a shared one, deduplicating them. * add a `s3_upload_download_works` test as a copy of the Azure test The goal is mainly to fight duplication and make the code a little bit more generic (like removing mentions of s3 and azure from function names). This is a first step towards #6146.	2023-12-19 11:29:50 +01:00