Byte Streams and Binary Data in Java

Java's byte streams move raw binary data, one byte at a time. That is a good fit for images, compressed archives, and other data whose meaning is not text. Many programs, however, care about text: names, chat messages, and the lines of a document. A program that handles text wants to work with characters, not with the raw numbers that happen to represent them.

The question this article answers is how a byte stream can be connected to a character stream, so that text can be read from and written to a source that only understands bytes. The short answer is that you wrap the byte stream. You hand an existing byte stream to a constructor and receive back a character stream that performs the conversion for you. The program then reads and writes characters, and the wrapper handles the bytes underneath.

InputStreamReader and OutputStreamWriter are the two wrappers that bridge the gap. An InputStreamReader wraps an InputStream so that the program reads characters from it. An OutputStreamWriter wraps an OutputStream so that the program writes characters into it. In both cases the wrapper is the only place where the conversion between characters and bytes happens. The byte stream itself still moves bytes, and the program does not touch those bytes directly once the wrapper is in place.

Read the program line by line and notice what it never does. It never calls a method that handles individual bytes when it writes or reads the text. It calls write with a String and read to receive characters. The byte values appear only in the stored array, which the program prints as a length. The count of 5 is a clue: the four characters of café took five bytes to store, because the letter é needed more than one byte under UTF-8. The charset section below explains why.

When data is read, the flow runs from the byte source upward. Bytes come out of the byte stream, the InputStreamReader turns them into characters, and your code receives characters. The diagram below traces that path for a single read. Follow the arrows and note at which step the value stops being a byte and becomes a char.

Writing runs the other way. The program supplies characters, the OutputStreamWriter turns them into bytes, and the byte stream delivers those bytes to its destination. In both directions the wrapper sits between the program and the byte stream, and the program has no direct handle on the bytes once they are wrapped. Everything it sees passes through the character stream.

System.in is the standard input stream, and its declared type is InputStream. That makes it a byte stream, which is an unusual arrangement for something a program reads from the keyboard as text. Because of that, it is common to wrap System.in in a Reader when you want to read character data from standard input. The wrapping is a single line of code.

The same pattern applies to network connections. The input and output streams tied to a network connection are byte streams, and wrapping them lets a program send and receive character data over the network without handling bytes directly. Network input and output is treated in a later section of the course, so this article focuses on the wrapping itself and on the charset that the wrapper relies on.

The bytes produced by a writer are only meaningful if a reader interprets them with the same charset. The charset is therefore the missing piece in any wrapped stream. It is not a detail to ignore when data crosses from one program or machine to another, because the bytes alone do not say which characters they were meant to be.

The three charsets store the same character in different amounts of space. The diagram groups each charset into a column and shows the unit or units that represent one character. Read the columns from left to right to see how the amount of storage changes with the encoding. The letter é is a useful test case because it is a plain Latin letter with an accent, which every one of these charsets can represent.

The three columns show the same character taking different amounts of storage. Under ISO-8859-1 the letter é is one byte. Under UTF-8 it takes two bytes, which is why the earlier café example stored five bytes for four characters. Under UTF-16 the letter is also written in a two-byte unit, shown here in big-endian order. The lesson is not to memorize these byte values, but to see that the charset, not the character, determines how much space it takes.

Readers and Writers use the default charset of the computer they run on unless you choose otherwise. You can name a different charset in the constructor. The wrapper in the demo passed the name of the charset as a second argument, which overrides the default for that stream only. The same two-argument form works for both wrappers.

Specifying the charset is a choice you make at the point where the stream is created. Once the wrapper exists, it converts characters to bytes, or bytes to characters, using that charset. The choice is fixed for that stream, so you make it when you build the wrapper rather than later.

If you do not pass a charset, the wrapper uses the default charset of the machine the code runs on. The result then depends on that machine. Code that writes bytes on one computer and reads them on another can produce different text if the two defaults differ. This is the reason to name the charset whenever the bytes will be read somewhere else.

A mismatch appears when the bytes were written with one charset and read with another. The reader does not know the bytes came from a different charset, so it interprets each byte using its own rules. The wrong text appears on the reading side, at the point where the reader's wrapper converts bytes to characters. The writing side may have produced perfectly correct bytes.