Reword comment and add comment on race condition (#4725)

The race condition that caused #4526 is still not fixed, so point it out
in a comment. Also, reword a comment in upload.rs.

Follow-up of #4694

pageserver/src/tenant/remote_timeline_client/upload.rs

@@ -62,12 +62,11 @@ pub(super) async fn upload_timeline_layer<'a>(
     let source_file = match source_file_res {
         Ok(source_file) => source_file,
         Err(e) if e.kind() == ErrorKind::NotFound => {
-            // In some situations we might run into the underlying file being deleted by
-            // e.g. compaction before the uploader gets to it. In that instance, we don't
-            // want to retry the error: a deleted file won't come back. In theory, the
-            // file might not have been written in the first place, which also indicates
-            // a bug. Still log the situation so that we can keep an eye on it.
-            // See https://github.com/neondatabase/neon/issues/4526
+            // If we encounter this arm, it wasn't intended, but it's also not
+            // a big problem, if it's because the file was deleted before an
+            // upload. However, a nonexistent file can also be indicative of
+            // something worse, like when a file is scheduled for upload before
+            // it has been written to disk yet.
             info!(path = %source_path.display(), "File to upload doesn't exist. Likely the file has been deleted and an upload is not required any more.");
             return Ok(());
         }

pageserver/src/tenant/timeline.rs

@@ -2724,6 +2724,12 @@ impl Timeline {
                 HashMap::from([(delta_path, metadata)])
             };
 
+        // FIXME: between create_delta_layer and the scheduling of the upload in `update_metadata_file`,
+        // a compaction can delete the file and then it won't be available for uploads any more.
+        // We still schedule the upload, resulting in an error, but ideally we'd somehow avoid this
+        // race situation.
+        // See https://github.com/neondatabase/neon/issues/4526
+
         pausable_failpoint!("flush-frozen-before-sync");
 
         // The new on-disk layers are now in the layer map. We can remove the