logo

Fan-Out/Fan-In

Fan-out/fan-in is a concurrency pattern for parallel processing. Fan-out splits a job into multiple independent tasks that run concurrently. Fan-in collects and aggregates the results once every task completes.

How It Works

  1. A coordinator receives a request and breaks it into N independent subtasks
  2. Each subtask is dispatched to the queue (fan-out)
  3. Workers process the subtasks concurrently
  4. As each subtask completes, results are collected
  5. When all subtasks are done, results are aggregated into a final output (fan-in)

Example with AsyncQueue

// Fan-out: create parallel tasks
const batchId = crypto.randomUUID();

for (const url of urls) {
  await aq.tasks.create({
    callbackUrl: 'https://your-app.com/api/scrape-page',
    payload: { batchId, url },
    webhookUrl: 'https://your-app.com/api/on-page-scraped',
    retries: 2,
  });
}

// Fan-in: aggregate results as they arrive
app.post('/api/on-page-scraped', async (req, res) => {
  const { batchId, url, data } = req.body.result;
  const batch = await db.batches.addResult(batchId, { url, data });

  if (batch.completedTasks === batch.totalTasks) {
    // All done — build final output
    await buildReport(batch.results);
  }

  res.status(200).json({ received: true });
});

When to Use Fan-Out/Fan-In

  • Batch processing: Resize 50 images, transcode 10 videos
  • Data aggregation: Query multiple APIs and combine results
  • Parallel analysis: Analyze a portfolio of stocks simultaneously
  • Distributed testing: Run tests across multiple environments

Considerations

  • Track completion state atomically — use Redis INCR or database transactions to avoid race conditions when multiple webhooks arrive simultaneously
  • Handle partial failures — decide whether to fail the entire batch or continue with available results
  • Set a reasonable fan-out limit — creating 10,000 tasks at once may overwhelm downstream services