Skip to content

Fix WorkUnit shard cleanup even when in-memory WorkUnit was filtered #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wilsonbb merged 1 commit into from
Dec 10, 2025
Merged

Fix WorkUnit shard cleanup even when in-memory WorkUnit was filtered #77

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion src/kbmod_wf/task_impls/kbmod_search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ def run_search(self):
wu = WorkUnit.from_sharded_fits(wu_filename, directory_containing_shards, lazy=False)
self.logger.debug("Loaded work unit")

# We may filter out images for the in-memory WorkUnit later on in search, but we would
# still like to cleanup shards for any filtered out images, hence we need to store the
# original WorkUnit size.
orig_wu_size = len(wu)

#! Seems odd that we extract, modify, and reset the config in the workunit.
#! Can we just modify the config in the workunit directly?
if self.search_config_filepath is not None:
Expand Down Expand Up @@ -114,7 +119,7 @@ def run_search(self):
self.logger.warning(f"Failed to remove {self.input_wu_filepath}: {e}")

# Delete the individual shards for this WorkUnit, one existing for each image.
for i in range(len(wu)):
for i in range(orig_wu_size):
shard_path = os.path.join(directory_containing_shards, f"{i}_{wu_filename}")
try:
os.remove(shard_path)
Expand Down
Loading