scrubber: remove AWS region assumption, tolerate negative max_project_size (#9636)

## Problem

First issues noticed when trying to run scrubber find-garbage on Azure:
- Azure staging contains projects with -1 set for max_project_size:
apparently the control plane treats this as a signed field.
- Scrubber code assumed that listing projects should filter to
aws-$REGION. This is no longer needed (per comment in the code) because
we know hit region-local APIs.

This PR doesn't make it work all the way (`init_remote` still assumes
S3), but these are necessary precursors.

## Summary of changes

- Change max-project_size from unsigned to signed
- Remove region filtering in favor of simply using the right region's
API (which we already do)
This commit is contained in:
John Spray
2024-11-05 13:32:50 +00:00
committed by GitHub
parent 70ae8c16da
commit e30f5fb922
2 changed files with 4 additions and 6 deletions

View File

@@ -147,7 +147,7 @@ pub struct ProjectData {
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub pg_version: u32,
pub max_project_size: u64,
pub max_project_size: i64,
pub remote_storage_size: u64,
pub resident_size: u64,
pub synthetic_storage_size: u64,
@@ -261,7 +261,7 @@ impl CloudAdminApiClient {
}
}
pub async fn list_projects(&self, region_id: String) -> Result<Vec<ProjectData>, Error> {
pub async fn list_projects(&self) -> Result<Vec<ProjectData>, Error> {
let _permit = self
.request_limiter
.acquire()
@@ -318,7 +318,7 @@ impl CloudAdminApiClient {
pagination_offset += response.data.len();
result.extend(response.data.drain(..).filter(|t| t.region_id == region_id));
result.append(&mut response.data);
if pagination_offset >= response.total.unwrap_or(0) {
break;

View File

@@ -160,9 +160,7 @@ async fn find_garbage_inner(
// Build a set of console-known tenants, for quickly eliminating known-active tenants without having
// to issue O(N) console API requests.
let console_projects: HashMap<TenantId, ProjectData> = cloud_admin_api_client
// FIXME: we can't just assume that all console's region ids are aws-<something>. This hack
// will go away when we are talking to Control Plane APIs, which are per-region.
.list_projects(format!("aws-{}", bucket_config.region))
.list_projects()
.await?
.into_iter()
.map(|t| (t.tenant, t))