Stream is a powerful concept but it’s not widely understood by developers.
I have found myself googling and learning the concept again every time I needed to work with streams in Node.js. This article is very much to remind my future self.
This article focuses on applying streams with text content. Code in this article is available in this GitHub repo.
A few cases where streams really shine:
- Processing large files that do not fit into computer’s memory. Using streams allows you to read a file and process each chunk as the data arrive into your program.
- All the input is not available just yet. When you write a command line program for users to interact with, input only becomes available over time.
- Reduce latency and improve user experience. Thanks for streams, your Netflix movies play (almost) immediately. The Netflix app on your PC/TV/browser shows you the content whilst it’s still downloading the rest of the movie.
- Reduce bandwidth. You can show the user the interim download result and terminate it if it’s not the right one. It would be a waste of bandwidth to download a whole large file only to find out it’s the wrong one.
Four types of streams
Readable: from which data can be read, e.g.
Writable: to which data can be written, e.g.
Duplex: That are both Readable and Writeable, e.g.
Transform: That can modify data as they are written or read, e.g. read and write compressed data from/to a file.
Two reading modes
Readable streams operate in one of two modes:
flowing mode, data are automatically read from the underlying system and provided to an application as quickly as possible using events via EventEmitter interface (i.e. each chunk is provided via a
- The stream implementor decides how often a data event is emitted, e.g. HTTP request may emit a
dataevent once a few KBs of data are read.
- When there is more data to read (i.e. the end of stream), the stream emits an
- If there is an error, the stream will emit an
paused mode, the
stream.read() method must be explicitly called each time to return a chunk of data from the stream.
readableevent is emitted every time a chunk of data is ready.
endevent is emitted when the end of the stream is reached.
nullwhen the end of the stream is reached.
Readablestreams begin in
paused modebut can be switched on
flowing modein one of the following ways:
- Adding a
- Calling the
- Calling the
stream.pipe()method to send data to a
Readable stream can switch back to
paused with one of the following:
- There is no pipe destination, by calling
stream.pause()method. Removing all pipe destinations with
- Adding a
readableevent handler, which has higher priority than
Readablewill not generate data until a mechanism for either consuming or ignoring data is provided. If that mechanism is taken away or disabled, the
Readablewill attempt to stop generating data.
The examples will show you how to work with:
readlinemodule, which provides an interface to work with data from a textual
- This wonderful introduction into streams in Node.js
- Readline module’s official documentation
- Async iterator in Node.js streams
- Node.js streams cheat sheet
- Get an input from the user
2. A simple CLI to interact with the user
3. Read a csv file line by line from
postcodes.csv, convert each line to an JSON object and write it down to
4. Read a file using
flowing mode, i.e. by attaching a
data event handler. Notice in this program, the events (
end) are emitted after last line, i.e.
console.log('The fun has just begun') has been executed.
5. Read a stream in
paused mode using
readble event and
6. Utility to download a file.
http.response is a
Readable stream and can be read using 3 ways:
flowing mode or
paused mode, you can see the size of each data chunk (except the last one) to be
16384 Bytes =
7. Utility to copy a file
Notice the size of each chunk is
65536 Bytes =
Readable.from() method can create a
readable stream from a string or a iterator (both synchronous and asynchronous).
Readable object constructor can be used to create a new
readable stream. Notice the
readable.push(null) is to signal the end of the stream.