DE: document bulk deletion controls

[lazyoldbear]

No description provided.

[lazyoldbear]

@eteubert would be lovely if you'd test the expiration job — now that you are comfortable running stuff in K8s.

[eteubert]

Looks working to me. Here's how I tested it:

• checked out the PR
• created my own config override /tmp/test-pr-values.yaml:

documentLifecycle:
  bulkDocumentDeletionEnabled: true
  expirationJob:
    enabled: true
    schedule: "*/5 * * * *" 
    keepHours: 24
    ttlSecondsAfterFinished: 300
    activeDeadlineSeconds: 1800
    startingDeadlineSeconds: 600
apiAuth:
  apiToken: "test-token-123"

• then upgrade helm using local helm-charts and my overrides

helm upgrade --install --namespace document-engine document-engine ./charts/document-engine \
    -f /tmp/de.dupa-values.yaml \
    -f /tmp/test-pr-values.yaml \
    --set cloudNativePG.clusterSpec.storage.size=128Mi \
    --set cloudNativePG.clusterSpec.storage.storageClass=local-path \
    --set "ingress.hosts[0].host=document-engine.k8s.orb.local"

• had to | quote the schedule value in document-expiration.CronJob.yaml to get it working, not sure if I'm just holding it wrong because that was there before

 spec:
-  schedule: {{ .Values.documentLifecycle.expirationJob.schedule }}
+  schedule: {{ .Values.documentLifecycle.expirationJob.schedule | quote }}

• using dashboard, created documents, let the job run, verified that nothing was deleted

[info] 2025-10-09 02:50:00.671 {"args":{"filters":{"allowed_documents":"any","created_before":"2025-10-08T02:50:00Z","tenant_id":"default"},"job_id":"7KPSA0193DVSGVTHC5YNF869SJ"},"id":6,"meta":{},"system_time":1759978200671942236,"max_attempts":3,"queue":"default","worker":"PS.AsyncJobs.Jobs.AsyncDocumentDelete","source":"oban","event":"job:start","tags":[],"attempt":1} pid=<0.4453.0>
[info] 2025-10-09 02:50:00.673 Starting async document deletion with filters: %{tenant_id: :default, allowed_documents: :any, created_before: ~U[2025-10-08 02:50:00Z]} pid=<0.4453.0>
[info] 2025-10-09 02:50:00.674 Starting bulk deletion with filters: %{tenant_id: :default, allowed_documents: :any, created_before: ~U[2025-10-08 02:50:00Z]} pid=<0.4453.0>
[info] 2025-10-09 02:50:00.674 Completed bulk deletion: %{errors: [], deleted_count: 0} pid=<0.4453.0>
[info] 2025-10-09 02:50:00.674 Completed async document deletion: %{errors: [], deleted_count: 0} pid=<0.4453.0>
[info] 2025-10-09 02:50:00.680 {"args":{"filters":{"allowed_documents":"any","created_before":"2025-10-08T02:50:00Z","tenant_id":"default"},"job_id":"7KPSA0193DVSGVTHC5YNF869SJ"},"id":6,"meta":{},"state":"success","max_attempts":3,"queue":"default","worker":"PS.AsyncJobs.Jobs.AsyncDocumentDelete","source":"oban","event":"job:stop","tags":[],"attempt":1,"duration":5549,"queue_time":8495} pid=<0.4453.0>

• via remote iex, made one of the documents older, let the job run, verified that the doc got deleted

[info] 2025-10-09 02:55:00.672 POST /api/async/delete_documents request_id=GGyzHyDv6rpQs_AAAFSC pid=<0.4975.0>
[info] 2025-10-09 02:55:00.676 Sent 202 in 3ms request_id=GGyzHyDv6rpQs_AAAFSC pid=<0.4975.0>
[info] 2025-10-09 02:55:00.684 {"args":{"filters":{"allowed_documents":"any","created_before":"2025-10-08T02:55:00Z","tenant_id":"default"},"job_id":"7KPSJY4X12GJ0BTZ9J6V89DRYD"},"id":7,"meta":{},"system_time":1759978500684523143,"max_attempts":3,"queue":"default","worker":"PS.AsyncJobs.Jobs.AsyncDocumentDelete","source":"oban","event":"job:start","tags":[],"attempt":1} pid=<0.4976.0>
[info] 2025-10-09 02:55:00.686 Starting async document deletion with filters: %{tenant_id: :default, allowed_documents: :any, created_before: ~U[2025-10-08 02:55:00Z]} pid=<0.4976.0>
[info] 2025-10-09 02:55:00.687 Starting bulk deletion with filters: %{tenant_id: :default, allowed_documents: :any, created_before: ~U[2025-10-08 02:55:00Z]} pid=<0.4976.0>
[warning] 2025-10-09 02:55:00.692 Failed to delete PDF asset as other documents or layers depend on it.(SHA256: ac864abdc641c1859196d17e139dd513c483399a1e3075814f6b7f06ad7303d4, Storage Backend: %{"postgres" => true}) pid=<0.4977.0>
[info] 2025-10-09 02:55:00.692 Processing batch of 1 documents pid=<0.4976.0>
[info] 2025-10-09 02:55:00.693 Completed bulk deletion: %{errors: [], deleted_count: 1} pid=<0.4976.0>
[info] 2025-10-09 02:55:00.693 Completed async document deletion: %{errors: [], deleted_count: 1} pid=<0.4976.0>
[info] 2025-10-09 02:55:00.704 {"args":{"filters":{"allowed_documents":"any","created_before":"2025-10-08T02:55:00Z","tenant_id":"default"},"job_id":"7KPSJY4X12GJ0BTZ9J6V89DRYD"},"id":7,"meta":{},"state":"success","max_attempts":3,"queue":"default","worker":"PS.AsyncJobs.Jobs.AsyncDocumentDelete","source":"oban","event":"job:stop","tags":[],"attempt":1,"duration":17842,"queue_time":9483} pid=<0.4976.0>

[lazyoldbear]

Thank you!