Node.js streams by examples

Stream is a powerful concept but it’s not widely understood by developers.

I have found myself googling and learning the concept again every time I needed to work with streams in Node.js. This article is very much to remind my future self.

This article focuses on applying streams with text content. Code in this article is available in this GitHub repo.

Why streams?

A few cases where streams really shine:

  • Processing large files that do not fit into computer’s memory. Using streams allows you to read a file and process each chunk as the data arrive into your program.
  • All the input is not available just yet. When you write a command line program for users to interact with, input only becomes available over time.
  • Reduce latency and improve user experience. Thanks for streams, your Netflix movies play (almost) immediately. The Netflix app on your PC/TV/browser shows you the content whilst it’s still downloading the rest of the movie.
  • Reduce bandwidth. You can show the user the interim download result and terminate it if it’s not the right one. It would be a waste of bandwidth to download a whole large file only to find out it’s the wrong one.

Four types of streams

  • : from which data can be read, e.g.
  • : to which data can be written, e.g.
  • : That are both Readable and Writeable, e.g.
  • : That can modify data as they are written or read, e.g. read and write compressed data from/to a file.

Two reading modes

Readable streams operate in one of two modes: and :

  1. In , data are automatically read from the underlying system and provided to an application as quickly as possible using events via EventEmitter interface (i.e. each chunk is provided via a event).
  • The stream implementor decides how often a data event is emitted, e.g. HTTP request may emit a event once a few KBs of data are read.
  • When there is more data to read (i.e. the end of stream), the stream emits an event.
  • If there is an error, the stream will emit an event.

2. In , the method must be explicitly called each time to return a chunk of data from the stream.

  • A event is emitted every time a chunk of data is ready.
  • A event is emitted when the end of the stream is reached.
  • returns when the end of the stream is reached.

Notes:

  1. All streams begin in but can be switched on in one of the following ways:
  • Adding a event handler
  • Calling the method
  • Calling the method to send data to a stream

2. The stream can switch back to with one of the following:

  • There is no pipe destination, by calling method. Removing all pipe destinations with method.
  • Adding a event handler, which has higher priority than event.
  • A will not generate data until a mechanism for either consuming or ignoring data is provided. If that mechanism is taken away or disabled, the will attempt to stop generating data.

Node.js components

The examples will show you how to work with:

  • as a stream
  • as a stream
  • as a stream
  • as a stream
  • module, which provides an interface to work with data from a textual stream
  • as a stream

Further reading

Examples

  1. Get an input from the user

2. A simple CLI to interact with the user

3. Read a csv file line by line from , convert each line to an JSON object and write it down to

4. Read a file using , i.e. by attaching a event handler. Notice in this program, the events ( and ) are emitted after last line, i.e. has been executed.

5. Read a stream in using event and method

6. Utility to download a file. is a and can be read using 3 ways:

  • Using with method ()
  • Using with event
  • Using with event and method

Using either or , you can see the size of each data chunk (except the last one) to be = .

7. Utility to copy a file

Notice the size of each chunk is = .

8. method can create a stream from a string or a iterator (both synchronous and asynchronous).

From a string, no buffer is provided. So the whole string will be return in one chunk in a event.
From a synchronous iterator
From an asynchronous iterator, e.g. by using an async generator function

9. The object constructor can be used to create a new stream. Notice the is to signal the end of the stream.

Software developer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store