mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 00:09:22 +00:00
Hengfei Yang
6e9e1339f0
The move file job will be stuck when there are many search requests on the ingester because we need to check if the file is using by search before deleting a local parquet in WAL. This PR introduces a new concept: `PENDING_DELETE_FILES`. We will verify if the file is in use when deleting a local parquet from WAL after uploading it to S3. - If the file is in use, it will be added to `PENDING_DELETE_FILES`. - If the file is not in use, it will be deleted. Before starting each subsequent move job, we will first check `PENDING_DELETE_FILES` and delete files that are not being used by searches. We also add to local database to persistently store files added to `PENDING_DELETE_FILES`. If the node restarts but files remain in `PENDING_DELETE_FILES`, we will load them from the local database and continue checking. Once they are no longer used, we will delete them. This change will not block the file move when there are a lot of search requests. let the move file faster.