Introduce RequestContexts.

RequestContext is used to track each "operation" or "task" in a way
that's not tied to tokio tasks. It provides support for fine-grained
cancellation of individual operations, or all tasks working on an
active tenant or timeline. Most async functions now take a
RequestContext argument.

RequestContexts form a hierarchy, so that you have a top-level context
e.g.  for a TCP listener task, a child context for each task handling
a connection, and perhaps a grandchild context for each individual
client request. In addition to the hierarchy, each RequestContext can
be associated with a Tenant or Timeline object. This is used to
prevent a Tenant or Timeline from being deleted or detached while
there are still tasks accessing it. This fixes a long-standing race
conditions between GC/compaction and deletion (see issues #2914 and
compiler in any way, but the functions like `get_active_timeline`
make it easy to do the right thing.

This replaces most of the machinery in `task_mgr.rs`. We don't track
running tasks as such anymore, only RequestContexts. In practice,
every task holds onto a RequestContext.

In addition to supporting cancellation, the RequestContext specifies
the desired behavior if a remote layer is needed for the operation.
This replaces the `with_ondemand_download_sync` and
`no_ondemand_download` macros. The on-demand download now happens deep
in the call stack, in get_reconstruct_data(), and the caller is no
longer involved in the download, except by passing a RequestContext
that specifies whether to do on-demand download or not. The
PageReconstructResult type is gone but the
PageReconstructError::NeedsDownload variant remains. It's now returned
if the context specified "don't do on-demand download", and a layer
is missing.

TODO:
- Enforce better that you hold a RequestContext associated with a Tenant
  or Timeline.
- All the fields in RequestContext are currently 'pub', but things will
  break if you modify the tenant/timeline fields directly. Make that more
  safe.
- When you create a subcontext, should it inherit the Tenant / Timeline
  of its parent?
- Can the walreceiver::TaskHandle stuff be replaced with this?
- Extract smaller patches:
  - What else could we extract?

test_runner/fixtures/neon_fixtures.py

@@ -1914,9 +1914,9 @@ class NeonPageserver(PgProtocol):
             ".*Shutdown task error: walreceiver connection handling failure.*",
             ".*wal_connection_manager.*tcp connect error: Connection refused.*",
             ".*query handler for .* failed: Socket IO error: Connection reset by peer.*",
-            ".*serving compute connection task.*exited with error: Postgres connection error.*",
-            ".*serving compute connection task.*exited with error: Connection reset by peer.*",
-            ".*serving compute connection task.*exited with error: Postgres query error.*",
+            ".*connection handler exited with error: Postgres connection error.*",
+            ".*connection handler exited with error: Connection reset by peer.*",
+            ".*connection handler exited with error: Postgres query error.*",
             ".*Connection aborted: connection error: error communicating with the server: Broken pipe.*",
             ".*Connection aborted: connection error: error communicating with the server: Transport endpoint is not connected.*",
             ".*Connection aborted: connection error: error communicating with the server: Connection reset by peer.*",

test_runner/regress/test_ondemand_download.py

@@ -371,7 +371,7 @@ def test_download_remote_layers_api(
     env.pageserver.allowed_errors.extend(
         [
             f".*download_all_remote_layers.*{tenant_id}.*{timeline_id}.*layer download failed.*remote-storage-download-pre-rename failpoint",
-            f".*initial size calculation.*{tenant_id}.*{timeline_id}.*Failed to calculate logical size",
+            f".*initial size calculation for {tenant_id}/{timeline_id} failed.*remote-storage-download-pre-rename failpoint",
         ]
     )
 

test_runner/regress/test_remote_storage.py

@@ -543,6 +543,7 @@ def test_timeline_deletion_with_files_stuck_in_upload_queue(
     )
 
     env = neon_env_builder.init_start()
+    env.pageserver.allowed_errors.append(".*request cancelled while waiting on uploads to finish.*")
 
     # create tenant with config that will determinstically allow
     # compaction and gc