Async workflows introduce a problem that synchronous code avoids: two things can happen at the same time. A webhook arrives while a retry is in progress. Two tasks try to update the same order. A signal and a timeout fire within milliseconds of each other. These are race conditions, and they corrupt data without warning.

This guide shows you how to identify and prevent race conditions in async task processing.

Step 1: Recognize Common Race Conditions

Race condition 1: duplicate webhook delivery

A payment webhook is delivered twice in rapid succession. Both instances pass the “is this processed?” check before either writes the “processed” flag.

Handler A: Read order status -> "pending"
Handler B: Read order status -> "pending"  (A hasn't written yet)
Handler A: Charge customer, set status "paid"
Handler B: Charge customer, set status "paid"  (double charge)

Race condition 2: retry overlaps with original

A task times out (network issue), so the queue retries. But the original request was not lost - it was slow. Now two instances of the same task run in parallel.

Attempt 1: Start processing (slow network, no response to queue)
Queue:      Timeout, schedule retry
Attempt 2: Start processing (runs in parallel with Attempt 1)
Attempt 1: Complete, write result
Attempt 2: Complete, overwrite result (possibly different)

Race condition 3: signal vs timeout

A task with waitForSignal approaches its maxWaitTime. The external system sends a signal at almost the same moment the timeout watchdog fires.

Signal:   Read task status -> "waiting", resolve to "completed"
Timeout:  Read task status -> "waiting", resolve to "timeout"
Result:   Task ends up in whichever state wrote last

Race condition 4: concurrent status updates

Two webhook events for the same order arrive at the same time: order.paid and order.shipped. Both try to update the order record.

Event A: Read order -> status "pending"
Event B: Read order -> status "pending"
Event A: Update to "paid"
Event B: Update to "shipped" (skipped "paid" state entirely)

Step 2: Use Idempotency Keys to Prevent Duplicate Work

The simplest defense: track which events you have already processed.

app.post('/api/handle-event', async (req, res) => {
  const eventId = req.body.eventId;

  // Atomic check-and-insert
  const inserted = await db.processedEvents.insertIfNotExists({
    eventId,
    processedAt: new Date(),
  });

  if (!inserted) {
    // Already processed - safe to skip
    return res.json({ received: true, deduplicated: true });
  }

  // First time processing this event
  await processEvent(req.body);
  res.json({ received: true });
});

Key detail: The check and the insert must happen atomically. If you do a SELECT followed by an INSERT, another handler can slip in between. Use INSERT ... ON CONFLICT DO NOTHING or an equivalent construct.

-- Atomic deduplication in SQL
INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

-- If no row returned, event was already processed

Step 3: Use Database Locks for Critical Sections

When multiple handlers might operate on the same resource, use pessimistic locking:

// Process a payment - only one handler at a time per order
app.post('/api/process-payment', async (req, res) => {
  const { orderId, amount } = req.body;

  await db.transaction(async (tx) => {
    // Lock this specific order row
    const order = await tx.orders.findOne(
      { orderId },
      { forUpdate: true }  // SELECT ... FOR UPDATE
    );

    if (!order) {
      throw new Error('Order not found');
    }

    // Check current state under lock
    if (order.status !== 'pending') {
      // Already processed by another handler
      return;
    }

    // Safe to process - we hold the lock
    await tx.orders.update(orderId, {
      status: 'paid',
      paidAt: new Date(),
      amount,
    });

    await tx.ledger.insert({
      orderId,
      amount,
      type: 'payment',
    });
  });
  // Lock released when transaction commits

  res.json({ received: true });
});

When to use locks:

Financial operations (payments, refunds, transfers)
Inventory management (stock decrements)
Any operation that must execute once

When locks are overkill:

Logging and analytics (duplicates are tolerable)
Notifications (sending an email twice is annoying but not catastrophic)

Step 4: Design Status Transitions as State Machines

Instead of allowing any status to change to any other, define valid transitions with explicit rules:

const ORDER_TRANSITIONS = {
  pending:   ['paid', 'canceled'],
  paid:      ['shipped', 'refunded'],
  shipped:   ['delivered', 'returned'],
  canceled:  [],  // terminal state
  delivered: [],  // terminal state
  refunded:  [],  // terminal state
  returned:  ['refunded'],
};

async function transitionOrder(orderId, newStatus, metadata = {}) {
  return db.transaction(async (tx) => {
    const order = await tx.orders.findOne({ orderId }, { forUpdate: true });

    if (!order) throw new Error(`Order ${orderId} not found`);

    const allowed = ORDER_TRANSITIONS[order.status] || [];
    if (!allowed.includes(newStatus)) {
      console.log(
        `Rejected transition: ${order.status} -> ${newStatus} for ${orderId}`
      );
      return null; // Silently reject invalid transition
    }

    const updated = await tx.orders.update(orderId, {
      status: newStatus,
      updatedAt: new Date(),
      ...metadata,
    });

    // Log the transition for debugging
    await tx.statusLog.insert({
      orderId,
      fromStatus: order.status,
      toStatus: newStatus,
      timestamp: new Date(),
    });

    return updated;
  });
}

Benefits:

Impossible to skip states (e.g., jump from “pending” to “shipped”)
When events race, one wins and the other gets rejected - no corrupted state
The status log provides a full audit trail for debugging

Step 5: Test for Concurrency Issues

Race conditions are hard to reproduce because they depend on timing. Use deliberate delays to force overlapping execution:

// Test: verify concurrent webhook handling is safe
it('should handle duplicate webhooks without double-processing', async () => {
  const eventPayload = {
    eventId: 'evt_test_123',
    orderId: 'order_test_456',
    amount: 99.99,
  };

  // Fire two identical webhooks simultaneously
  const [res1, res2] = await Promise.all([
    fetch('/api/process-payment', {
      method: 'POST',
      body: JSON.stringify(eventPayload),
    }),
    fetch('/api/process-payment', {
      method: 'POST',
      body: JSON.stringify(eventPayload),
    }),
  ]);

  // Both should succeed (200)
  expect(res1.status).toBe(200);
  expect(res2.status).toBe(200);

  // But only one payment should exist
  const payments = await db.ledger.find({ orderId: 'order_test_456' });
  expect(payments.length).toBe(1);
});

Test the signal vs timeout race:

it('should handle signal and timeout racing', async () => {
  const { task, signalToken } = await aq.tasks.create({
    targetUrl: 'https://your-app.com/api/start',
    waitForSignal: true,
    maxWaitTime: 2, // Very short timeout
  });

  // Wait for "waiting" state
  await waitForStatus(task.id, 'waiting');

  // Send signal right at the timeout boundary
  const signalRes = await fetch(`/v1/signals/${signalToken}`, {
    method: 'POST',
    body: JSON.stringify({ status: 'completed' }),
  });

  // Either the signal wins (200) or the timeout won (404)
  // Both are valid outcomes, but the task must be in exactly one terminal state
  const finalTask = await aq.tasks.get(task.id);
  expect(['completed', 'timeout']).toContain(finalTask.status);
});

Checklist for race condition prevention:

Check	Question
Deduplication	Do you track processed event IDs atomically?
Locking	Do critical sections use database locks?
State machine	Are status transitions defined with explicit rules?
Atomic writes	Are check-then-act operations wrapped in transactions?
Concurrent tests	Do you test parallel execution of the same event?
Audit log	Can you trace what happened when two events collide?

Step 1: Recognize Common Race Conditions

Step 2: Use Idempotency Keys to Prevent Duplicate Work

Step 3: Use Database Locks for Critical Sections

Step 4: Design Status Transitions as State Machines

Step 5: Test for Concurrency Issues

Related Guides