Streams: reading, writing and piping

Reading a whole file is fine for a config and impossible for a log of ten gigabytes. A stream reads a piece at a time: createReadStream hands out the file as chunks and createWriteStream takes them in, so the memory a program needs does not grow with the file. The chunks are Buffers by default, and an encoding on the stream makes them strings.

The lesson collects the chunks of a small file, copies a file with pipeline, counts lines with readline and ends with a stream whose file is not there, which is the error the copy has to expect. The programs build their files in a temporary folder and remove it, and the last one shows that a failed pipeline leaves nothing half-written.

Chunks and data events

A read stream emits data once per chunk until the file ends, and then end. The size of a chunk is the buffer size of the stream, sixty-four kilobytes unless the options say otherwise, so a small file arrives in one piece and a large one in many. The program waits for end with a promise, because an asynchronous loop needs something to wait for.

The program writes three lines, reads them as utf8 and joins the chunks back into one text. It prints how many chunks arrived, how many characters they made and what the last line was.

The file chunks.mjs:

import { createReadStream } from 'node:fs'; import { mkdtemp, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; const dir = await mkdtemp(join(tmpdir(), 'stream-')); const file = join(dir, 'notes.txt'); await writeFile(file, 'alpha\nbeta\ngamma\n'); const stream = createReadStream(file, { encoding: 'utf8' }); const pieces = []; stream.on('data', (chunk) => { pieces.push(chunk); }); await new Promise((resolve, reject) => { stream.on('end', resolve); stream.on('error', reject); }); const text = pieces.join(''); console.log('chunks:', pieces.length); console.log('characters:', text.length); console.log('the last line:', text.trimEnd().split('\n').at(-1)); await rm(dir, { force: true, recursive: true });

node chunks.mjs

chunks: 1
characters: 17
the last line: gamma

The seventeen characters came in one chunk, which is the whole file: nothing is gained by streaming something this small, and nothing is lost either, because the code does not care how many chunks there are. Without the encoding the same pieces would be Buffer objects and the join would have to be a concatenation of buffers.

pipe and pipeline

pipe sends the output of one stream into the input of another, and pipeline does the same and returns a promise, which it rejects after destroying every stream on either side when one of them fails. The second is the one to use in a program that has to know whether the copy worked, and it is the function the rest of this lesson uses.

The program copies a file, then compares the two sizes and the two texts. Nothing is printed about the folder, only the three facts a copy has to establish.

The file pipe.mjs:

import { createReadStream, createWriteStream } from 'node:fs'; import { mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { pipeline } from 'node:stream/promises'; const dir = await mkdtemp(join(tmpdir(), 'stream-')); const file = join(dir, 'notes.txt'); const copy = join(dir, 'copy.txt'); await writeFile(file, 'one\ntwo\nthree\n'); await pipeline(createReadStream(file), createWriteStream(copy)); const original = await stat(file); const duplicated = await stat(copy); console.log('the sizes match:', original.size === duplicated.size); console.log('the texts match:', (await readFile(file, 'utf8')) === (await readFile(copy, 'utf8'))); console.log('the size in bytes:', duplicated.size); await rm(dir, { force: true, recursive: true });

node pipe.mjs

the sizes match: true
the texts match: true
the size in bytes: 14

A copied file is a new file with the same bytes, not a link: changing one leaves the other alone. The pipeline call also waits for the write stream to finish, which a bare pipe does not promise, and that is what makes the comparison safe on the next line.

A line counter

readline.createInterface turns a stream into lines, so a program can count them or search them without ever holding the whole file. The interface is an async iterable, which makes the counting loop a for await of the kind the lesson on async iteration built by hand. The crlfDelay option joins a carriage return and a newline into one line ending, which matters for a file written on Windows.

The program writes a four-line log, counts the lines and counts the ones that begin with ERROR.

The file lines.mjs:

import { createReadStream } from 'node:fs'; import { mkdtemp, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { createInterface } from 'node:readline'; const dir = await mkdtemp(join(tmpdir(), 'stream-')); const file = join(dir, 'app.log'); await writeFile(file, 'INFO start\nERROR disk full\nINFO retry\nERROR timeout\n'); const lines = createInterface({ input: createReadStream(file), crlfDelay: Infinity }); let total = 0; let errors = 0; for await (const line of lines) { total += 1; if (line.startsWith('ERROR ')) { errors += 1; } } console.log('lines:', total); console.log('errors:', errors); await rm(dir, { force: true, recursive: true });

node lines.mjs

lines: 4
errors: 2

The line ending is removed by the interface, so the test compares the text of the line and not a piece of it. This loop is the shape a log reader has: one pass, a counter per fact, and memory that does not depend on the size of the file.

An error in the middle

A stream reports a missing file with an error event rather than a throw, and pipeline turns that event into a rejection. The promise form is what lets a program put the copy in a try and keep going, which is exactly what the reader of a backup tool wants: a message and an exit code, not a stack of internal frames.

The program copies a file that does not exist, catches the rejection, prints the code and removes the folder.

The file error.mjs:

import { createReadStream, createWriteStream } from 'node:fs'; import { mkdtemp, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { pipeline } from 'node:stream/promises'; const dir = await mkdtemp(join(tmpdir(), 'stream-')); try { await pipeline( createReadStream(join(dir, 'missing.txt')), createWriteStream(join(dir, 'copy.txt')), ); } catch (error) { console.log('the copy failed with:', error.code); } await rm(dir, { force: true, recursive: true }); console.log('the folder is gone');

node error.mjs

the copy failed with: ENOENT
the folder is gone

The rejection carried the same ENOENT the file lessons use, and the program ended with zero because the failure was expected and handled. A real tool would set an exit code here, and the lesson on process shows how.

Exercises

  1. Read a file without an encoding and print the type and the length of every chunk. How many chunks does a file of a hundred kilobytes give?
  2. Count the bytes of a file with a stream and compare the total with stat. Do the two agree?
  3. Copy a file with pipe and an error listener instead of pipeline. What happens to the write stream when the read fails?
  4. Count the lines that contain a word with readline and print the first five of them. How do you stop the loop early?
  5. Read the same file with a highWaterMark of sixteen bytes and count the chunks again. Which number changed, and which did not?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is a chunk of a read stream?

    Answer

    A piece of the file, a Buffer unless an encoding is set

    The stream hands out the file in pieces, and the encoding decides whether a piece is text.

  2. What does this program print? import { Readable } from "node:stream"; const stream = Readable.from(["one\n", "two\n"]); let text = ""; for await (const chunk of stream) { text += chunk; } console.log(text.trimEnd().split("\n").length);

    Answer

    2

    The two pieces make two lines once the last newline is removed.

  3. What does pipeline do when one of its streams fails?

    Answer

    It destroys the others and rejects the promise

    That cleanup is the reason to prefer pipeline over a bare pipe.

  4. What does this program print? import { Readable } from "node:stream"; let size = 0; for await (const chunk of Readable.from(["ab", "cd", "ef"])) { size += chunk.length; } console.log(size);

    Answer

    6

    Each piece is two characters long and there are three of them.

  5. What does readline give over a stream?

    Answer

    One line at a time, without its line ending

    The interface splits on the line ending and hands each line over as it arrives.

  6. What does this program print? import { createReadStream, createWriteStream } from "node:fs"; import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { pipeline } from "node:stream/promises"; const dir = await mkdtemp(join(tmpdir(), "q-")); try { await pipeline( createReadStream(join(dir, "missing.txt")), createWriteStream(join(dir, "copy.txt")), ); } catch (error) { console.log(error.code); } await rm(dir, { force: true, recursive: true });

    Answer

    ENOENT

    The read stream cannot open the file, and pipeline passes the failure to the caller.

  7. Does createReadStream read the whole file at once?

    Answer

    No, it reads ahead by one buffer and hands the pieces out

    The point of a stream is that memory does not grow with the file.

  8. What does this program print? import { Readable } from "node:stream"; const kinds = []; for await (const chunk of Readable.from([Buffer.from("ab")])) { kinds.push(Buffer.isBuffer(chunk)); } console.log(kinds.join(","));

    Answer

    true

    A stream without an encoding hands out Buffers, so the test is true.

  9. Which event comes after the last chunk?

    Answer

    end

    The end event closes the reading side, and the promise in the first program waits for it.

  10. Why does pipeline wait for the write stream as well?

    Answer

    So the last bytes are flushed before the program goes on

    A copy that is compared before the flush would compare a shorter file.

The code on this page was run with Node.js 24.22.1 and linted with ESLint 10.14.0 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev