Streams

Read chunks

Streams deliver data piece by piece instead of loading the whole file into memory. A readable stream emits a data event for every chunk, which keeps memory flat even for huge files.

Create a read stream from node:fs and listen for data with the on method. Each chunk arrives as it is read from disk, so processing can start before the file ends.

import { createReadStream } from "node:fs"; const input = createReadStream("notes.txt"); input.on("data", function (chunk): void { console.log(chunk); });

Pipe them

Piping connects a readable stream to a writable one with one method call. Data flows chunk by chunk from source to destination without manual buffering.

Create a write stream for the target file and call pipe on the reader. Backpressure and cleanup travel through the pipe, so ten lines replace pages of manual code.

import { createReadStream, createWriteStream } from "node:fs"; const reader = createReadStream("a.txt"); const writer = createWriteStream("b.txt"); reader.pipe(writer);

Transform inline

A transform stream sits in the middle of a pipe and rewrites each chunk on the way through. Upper casing text is the classic first example of an inline step.

Build a Transform from node:stream with a small transform function that converts the chunk and passes it to done. Then pipe the reader through it into the writer.

import { Transform } from "node:stream"; const upper = new Transform({ transform(chunk, _encoding, done): void { done(null, chunk.toString().toUpperCase()); } }); console.log("transform ready");

Respect backpressure

A fast reader can overwhelm a slow writer, so write returns a boolean hint after every call. When it returns false, pause the reader until the writer drains.

Listen once for the drain event and resume the reader inside the handler. This tiny pause and resume loop is the whole secret behind stable pipes.

import { createReadStream, createWriteStream } from "node:fs"; const fast = createReadStream("big.txt"); const slow = createWriteStream("big.copy.txt"); const ok: boolean = slow.write("start"); if (ok === false) { fast.pause(); slow.once("drain", function (): void { fast.resume(); }); }

Next: Workers

Article author: Arthur Isaev