A queue with limited concurrency

Promise.all starts everything at once, which is right for three calls and wrong for three thousand: every open request costs memory, the other side has a connection limit, and a rate limiter counts them. A pool runs a fixed number of jobs and starts the next one as soon as a slot is free.

The pattern in this lesson is a loop per worker over a shared index, which needs no library and no queue object. The lesson shows the problem first, then the module, then a program that uses it, the order of the results when the jobs finish at different speeds, and what a single failure does to the workers that are still running.

Everything at once

Four jobs started together print four starts before any end, which is the shape of the problem: nothing in the program decides how many run at a time. With a limit of two, the same four jobs print two starts and then one start for each end.

The file all.mjs:

const delay = (name, ms) => new Promise((resolve) => { setTimeout(() => resolve(name), ms); }); async function job(name) { console.log('start', name); const value = await delay(name, 10); console.log('end', name); return value; } const results = await Promise.all(['a', 'b', 'c', 'd'].map((name) => job(name))); console.log('all at once:', results.join(','));

node all.mjs

start a
start b
start c
start d
end a
end b
end c
end d
all at once: a,b,c,d

The four jobs were all in flight before the first one finished, and the map built the whole array before anything was awaited. That is fine here and not fine with a thousand items, where the requests would be opened faster than they can close.

The pool, written by hand

The module below is the whole pattern: a shared index, a loop that takes the next item, and a list of workers started together. The loop body of a worker runs between awaits, so two workers never take the same index: the read and the increment of next happen in one turn of the loop, before the next await.

The file queue.mjs:

export async function pool(items, limit, worker) { const results = new Array(items.length); let next = 0; async function run() { while (next < items.length) { const index = next; next += 1; results[index] = await worker(items[index]); } } const workers = []; for (let n = 0; n < limit; n += 1) { workers.push(run()); } await Promise.all(workers); return results; }

The program below imports the module, which prints nothing on its own, and runs five jobs with a limit of two. The log lines are the workers taking their turns.

The file run.mjs:

import { pool } from './queue.mjs'; const delay = (name, ms) => new Promise((resolve) => { setTimeout(() => resolve(name), ms); }); async function job(name) { console.log('start', name); const value = await delay(name, 10); console.log('end', name); return value; } const results = await pool(['a', 'b', 'c', 'd', 'e'], 2, job); console.log('the results:', results.join(','));

node run.mjs

start a
start b
end a
start c
end b
start d
end c
start e
end d
end e
the results: a,b,c,d,e

Two names were in flight at every moment, and each end was followed by the next start. The pattern costs nine lines and needs no dependency, and the same nine lines work for a queue of requests, of files or of rows of a table. A limit of one turns it into a sequential loop with the same results.

Results in the input order

The workers finish in whatever order the work allows, and the results are written at the index of their item, so the array that comes back has the order of the input. The program below gives the jobs different durations to make that visible.

The file order.mjs:

import { pool } from './queue.mjs'; const delay = (name, ms) => new Promise((resolve) => { setTimeout(() => resolve(name), ms); }); async function job({ name, ms }) { console.log('start', name); await delay(name, ms); console.log('end', name); return name; } const items = [ { name: 'slow', ms: 30 }, { name: 'fast', ms: 10 }, { name: 'middle', ms: 20 }, ]; const results = await pool(items, 2, job); console.log('the results keep the input order:', results.join(','));

node order.mjs

start slow
start fast
end fast
start middle
end slow
end middle
the results keep the input order: slow,fast,middle

The three jobs finished as fast, slow, middle, and the array came back as slow, fast, middle. That is the same promise as the one Promise.all makes, and it lets a caller pair each result with the record it came from.

What a failure does to the pool

A worker that throws fails its own promise, and Promise.all over the workers rejects at that moment. The other workers are not stopped: they finish their current item and take the next one, so a program that needs a clean stop passes the signal from the lesson on AbortController into the worker.

The file error.mjs:

import { pool } from './queue.mjs'; const delay = (name, ms) => new Promise((resolve) => { setTimeout(() => resolve(name), ms); }); async function job(name) { console.log('start', name); await delay(name, 10); if (name === 'c') { throw new Error(`the job ${name} failed`); } console.log('end', name); return name; } try { const results = await pool(['a', 'b', 'c', 'd'], 2, job); console.log('the results:', results.join(',')); } catch (error) { console.log('the pool stopped:', error.message); }

node error.mjs

start a
start b
end a
start c
end b
start d
the pool stopped: the job c failed
end d

The catch ran as soon as c failed, and d still ended after it, because the second worker was already running that job. That is the behaviour to know before using the pattern: the pool stops taking work, and it does not stop the work it has. The next lesson is about the other way a program learns that something happened, the event emitter.

Exercises

  1. Run the five jobs with a limit of one. Which line follows each end?
  2. Run them with a limit of five and compare the output with the first program.
  3. Make the pool return the results as an object keyed by the item instead of an array.
  4. Pass an AbortSignal into the worker and throw from it when it fires. Which jobs run after the failure?
  5. Count how many jobs are running at once with a counter that goes up at the start and down at the end. What is the largest value with a limit of two?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Why limit how many jobs run at once?

    Answer

    The memory, the sockets and the other side all have a limit

    A thousand open requests cost memory and connections, and many servers refuse them.

  2. What does the pool function take?

    Answer

    The items, the limit and the worker

    The list gives the work, the limit gives the number of workers, and the worker does one item.

  3. How do the workers share the work?

    Answer

    Each takes the next index from one shared counter

    The read and the increment happen in one turn of the loop, so no item is taken twice.

  4. Where does the result of an item go?

    Answer

    Into the results array at the index of the item

    Writing by index is what keeps the results in the order of the input.

  5. What does this program print? const items = ["a", "b", "c"]; const results = new Array(items.length); results[2] = "c"; results[0] = "a"; results[1] = "b"; console.log(results.join(","));

    Answer

    a,b,c

    An array keeps the position of each write, whatever order the writes happen in.

  6. What does this program print? const next = { value: 0 }; function take() { const index = next.value; next.value += 1; return index; } console.log(take(), take(), take());

    Answer

    0 1 2

    Each call reads the counter, then increments it, so the three calls return 0, 1 and 2.

  7. What happens to a pool built on Promise.all when one worker rejects?

    Answer

    The pool rejects at once, and the other workers keep running

    The combined promise settles at the first rejection, and a running job cannot be cancelled.

  8. What does a limit of one do?

    Answer

    One job at a time, in the order of the items

    One worker takes the items one after another, so the order of the results and of the work agree.

  9. What does this program print? let running = 0; let peak = 0; async function job() { running += 1; peak = Math.max(peak, running); await null; running -= 1; } await Promise.all([job(), job(), job()]); console.log(peak);

    Answer

    3

    All three calls start before the first await resumes, so the counter reaches three.

  10. Why is the shared index safe without a lock?

    Answer

    The loop body runs between awaits, so no two workers read it at once

    A turn of the loop is not interrupted, and the await comes after the index was taken.

The code on this page was run with Node.js 24.22.1 and linted with ESLint 10.14.0 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev