Tags: FoundatioFx/Foundatio
Tags
Fix InMemoryQueue retry: create fresh entry per attempt (entry isolat… …ion) (#454) * Fix race condition in InMemoryQueue retry with zero delay When RetryDelay is zero, AbandonAsync used Task.Run to fire-and-forget the retry enqueue. This created a race where RunUntilEmptyAsync's continuation callback could check queue stats (Queued + Working == 0) and stop the job loop before the Task.Run actually re-enqueued the items, leaving them stranded. The fix has three parts: 1. Zero-delay retry in AbandonAsync now enqueues inline instead of via Task.Run, ensuring the item is visible in the queue before AbandonAsync returns. 2. entry.Reset() is moved from RetryAsync to DequeueImplAsync. This is critical because Reset() clears IsAbandoned/IsCompleted on the same object reference. If Reset ran inside AbandonAsync, callers would see IsAbandoned=false after abandoning, breaking DisposeAsync guards and auto-complete logic in QueueJobBase. 3. RetryAsync no longer calls Reset(), since dequeue handles it. The AsyncAutoResetEvent uses RunContinuationsAsynchronously, so the inline _autoResetEvent.Set() in AbandonAsync cannot cause reentrancy (the dequeue waiter resumes on a separate thread pool work item). This restores the original design from the initial codebase where Reset was called at dequeue time, before it was moved into RetryAsync in commit c201d8a. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix InMemoryQueue retry by creating fresh entry per attempt Instead of reusing the same QueueEntry object across retry attempts (which caused race conditions and mutation leakage), create a new QueueEntry via CreateRetryEntry() from the pristine _original value. This matches how external queue providers (Redis, Azure, SQS) naturally isolate entries by deserializing fresh objects on each dequeue. Changes: - Add QueueEntry<T>.CreateRetryEntry() that creates a new entry from _original with reset flags, preserving Id/Attempts/Properties - Use CreateRetryEntry() in both zero-delay and delayed retry paths - Remove Reset() call from RetryAsync (entry is already fresh) - Restore original Task.Run decoupling for zero-delay retries - Add DequeueAsync_AfterAbandonWithMutatedValue_ReturnsOriginalValueAsync test to QueueTestBase and all provider test classes Co-authored-by: Cursor <cursoragent@cursor.com> * Inline zero-delay retry enqueue and strengthen test assertions - Replace Task.Run with synchronous enqueue for zero-delay retries to eliminate race condition where AbandonAsync returns before item is re-queued. - Add mid-abandon queue stats assertion (Queued == 1) to verify item is immediately available after abandon. - Add entry state assertions verifying IsAbandoned/IsCompleted flags on both the original and retry entries before and after the second dequeue, ensuring the original caller's reference is never mutated by retry logic. Co-authored-by: Cursor <cursoragent@cursor.com> * Use InMemoryMetrics WaitForCounterAsync for provider-agnostic stats assertions Replace direct GetQueueStatsAsync polling with OTel-based WaitForCounterAsync to reliably wait for abandoned/completed counters across all queue providers. Co-authored-by: Cursor <cursoragent@cursor.com> * Pass TestCancellationToken to WaitForCounterAsync and restore Queued assertion Co-authored-by: Cursor <cursoragent@cursor.com> * Fixes in-memory queue retry race condition Addresses a race condition in the in-memory queue retry mechanism. The previous implementation could potentially enqueue retry entries multiple times due to the asynchronous nature of the retry delay. This change ensures that the `Retry` method is called directly after the delay, preventing the race condition. This commit also improves logging by including the number of attempts in the log messages when adding an item to the wait list or back to the queue for retry. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes async Task return in ScheduledTimer test Updates the ScheduledTimer test to properly await the async Task. This prevents potential issues with the test not completing correctly.
PreviousNext