Skip to content

feat(docker): add minute-level log retention to clean-logs script #61855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jscheffl merged 6 commits into from
Mar 11, 2026
Merged

feat(docker): add minute-level log retention to clean-logs script #61855

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0418328
feat(docker): add minute-level log retention support to clean-logs.sh
pugarte7 Feb 13, 2026
e8206b6
feat(docker): import coauthored changes
pugarte7 Mar 6, 2026
c93b924
update inlined scripts in Dockerfile
pugarte7 Mar 6, 2026
61d4562
comments fixed
pugarte7 Mar 11, 2026
d264e95
Merge branch 'main' into feat/docker-minute-log-retention
pugarte7 Mar 11, 2026
7d2945c
Merge branch 'main' into feat/docker-minute-log-retention
pugarte7 Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
feat(docker): add minute-level log retention support to clean-logs.sh
  • Loading branch information
@pugarte7
pugarte7 committed Mar 11, 2026
commit 0418328d055c49343bdcef29f6622e7900230752
19 changes: 14 additions & 5 deletions scripts/docker/clean-logs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ set -euo pipefail

readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
readonly RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
readonly RETENTION_MINUTES="${AIRFLOW__LOG_RETENTION_MINUTES:-0}"
readonly FREQUENCY="${AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES:-15}"
readonly MAX_PERCENT="${AIRFLOW__LOG_MAX_SIZE_PERCENT:-0}"

Expand All @@ -45,11 +46,19 @@ fi
retention_days="${RETENTION}"

@potiuk potiuk Mar 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bug here = RETENTION is sttill used here @Wastelander777

@pugarte7 pugarte7 Mar 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups, my bad 😣. What should be the approach? How can I fix that? Do I re-open the issue?

@n-badtke-cg n-badtke-cg Mar 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, went over my head too :/

Do I re-open the issue?

no, the PR is already merged, there needs to be a new PR unfortunately


while true; do
echo "Trimming airflow logs to ${retention_days} days."
find "${DIRECTORY}"/logs \
-type d -name 'lost+found' -prune -o \
-type f -mtime +"${retention_days}" -name '*.log' -print0 | \
xargs -0 rm -f || true
if (( RETENTION_MINUTES > 0 )); then
echo "Trimming airflow logs to ${RETENTION_MINUTES} minutes."
find "${DIRECTORY}"/logs \
-type d -name 'lost+found' -prune -o \
-type f -mmin +"${RETENTION_MINUTES}" -name '*.log' -print0 | \
xargs -0 rm -f || true
else
echo "Trimming airflow logs to ${RETENTION} days."
find "${DIRECTORY}"/logs \
-type d -name 'lost+found' -prune -o \
-type f -mtime +"${RETENTION}" -name '*.log' -print0 | \
xargs -0 rm -f || true
fi

if [[ "$MAX_SIZE_BYTES" -gt 0 && "$retention_days" -ge 0 ]]; then
current_size=$(df -k "${DIRECTORY}"/logs 2>/dev/null | tail -1 | awk '{print $3}' || echo "0")
Expand Down