How to Process File Uploads in the Background

File processing is one of the most common causes of slow API responses. Image resizing, video transcoding, PDF generation — these operations take seconds to minutes. Users shouldn’t wait. Offload them as background jobs.

The Problem

// Slow: user waits for processing to complete
app.post('/upload', async (req, res) => {
  const file = await saveToS3(req.file);
  const processed = await processImage(file.url, { width: 800 }); // 5-30 seconds
  const thumbnail = await generateThumbnail(file.url);              // 2-10 seconds

  await saveToDatabase({ original: file.url, processed, thumbnail });
  res.json({ success: true }); // User waited 7-40 seconds
});

The Solution

Step 1: Accept the file upload and store it

Keep your upload endpoint fast — store the file and respond:

app.post('/upload', async (req, res) => {
  const file = await saveToS3(req.file);

  await saveToDatabase({
    id: file.id,
    originalUrl: file.url,
    status: 'processing',
  });

  // Offload processing to AsyncQueue (Step 2)
  await aq.tasks.create({
    targetUrl: 'https://your-app.com/api/process-file',
    payload: { fileId: file.id, fileUrl: file.url },
    webhookUrl: 'https://your-app.com/api/on-file-processed',
    retries: 3,
  });

  res.json({ fileId: file.id, status: 'processing' });
});

The endpoint now responds in milliseconds.

Step 2: Create an AsyncQueue task for processing

The task was created in Step 1 above. AsyncQueue calls your processing endpoint with the file URL payload and waits for completion — regardless of duration.

Step 3: Build the processing endpoint

app.post('/api/process-file', async (req, res) => {
  const { fileId, fileUrl } = req.body;

  const processed = await processImage(fileUrl, { width: 800 });
  const thumbnail = await generateThumbnail(fileUrl);

  res.json({
    fileId,
    processedUrl: processed.url,
    thumbnailUrl: thumbnail.url,
  });
});

Step 4: Receive the result via webhook

app.post('/api/on-file-processed', async (req, res) => {
  const { fileId, processedUrl, thumbnailUrl } = req.body.result;

  await updateDatabase(fileId, {
    processedUrl,
    thumbnailUrl,
    status: 'completed',
  });

  await notifyUser(fileId, 'Your file is ready!');

  res.status(200).json({ received: true });
});

Step 5: Update the user interface

On the frontend, use polling or WebSockets to notify the user when processing finishes:

// Simple polling
const checkStatus = async (fileId) => {
  const res = await fetch(`/api/files/${fileId}`);
  const data = await res.json();

  if (data.status === 'completed') {
    showProcessedFile(data.processedUrl);
  } else {
    setTimeout(() => checkStatus(fileId), 2000);
  }
};

Use Cases

Image uploads: Resize, crop, apply filters, generate thumbnails
Video uploads: Transcode to multiple formats and resolutions
Document uploads: Convert DOCX to PDF, extract text, generate previews
CSV/data imports: Parse large files and insert records into your database in bulk