Node.js streams by examples

Stream is a powerful concept but it’s not widely understood by developers.

I have found myself googling and learning the concept again every time I needed to work with streams in Node.js. This article is very much to remind my future self.

This article focuses on applying streams with text content. Code in this article is available in this GitHub repo.

Why streams?

A few cases where streams really shine:

  • Processing large files that do not fit into computer’s memory. Using streams allows you to read a file and process each chunk as the data arrive into your program.

Four types of streams

  • Readable: from which data can be read, e.g. fs.createReadStream()

Two reading modes

Readable streams operate in one of two modes: flowing and paused:

  1. In flowing mode, data are automatically read from the underlying system and provided to an application as quickly as possible using events via EventEmitter interface (i.e. each chunk is provided via a data event).
  • The stream implementor decides how often a data event is emitted, e.g. HTTP request may emit a data event once a few KBs of data are read.

2. In paused mode, the stream.read() method must be explicitly called each time to return a chunk of data from the stream.

  • A readable event is emitted every time a chunk of data is ready.

Notes:

  1. All Readable streams begin in paused mode but can be switched on flowing mode in one of the following ways:
  • Adding a data event handler

2. The Readable stream can switch back to paused with one of the following:

  • There is no pipe destination, by calling stream.pause() method. Removing all pipe destinations with stream.unpipe() method.

Node.js components

The examples will show you how to work with:

  • process.stdin as a Readable stream

Further reading

Examples

  1. Get an input from the user

2. A simple CLI to interact with the user

3. Read a csv file line by line from postcodes.csv, convert each line to an JSON object and write it down to postcodes.txt

4. Read a file using flowing mode, i.e. by attaching a data event handler. Notice in this program, the events (data and end) are emitted after last line, i.e. console.log('The fun has just begun') has been executed.

5. Read a stream in paused mode using readble event and stream.read() method

6. Utility to download a file. http.response is a Readable stream and can be read using 3 ways:

  • Using flowing mode with response.pipe() method ()

Using either flowing mode or paused mode, you can see the size of each data chunk (except the last one) to be 16384 Bytes = 16KB.

7. Utility to copy a file

Notice the size of each chunk is 65536 Bytes = 64KB.

8. Readable.from() method can create a readable stream from a string or a iterator (both synchronous and asynchronous).

From a string, no buffer is provided. So the whole string will be return in one chunk in a data event.
From a synchronous iterator
From an asynchronous iterator, e.g. by using an async generator function

9. The Readable object constructor can be used to create a new readable stream. Notice the readable.push(null) is to signal the end of the stream.

Software developer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store