Network I/O Streams

InputStreamReader and OutputStreamWriter wrap an existing byte stream so that a program can read and write characters instead of raw bytes.

Programming

Why Programs Need Characters

Many programs care about text: names, chat messages, and the lines of a document. Java's byte streams, however, move raw binary data one byte at a time. They have no notion of a letter or a word. This article answers a practical question: how can a byte stream be connected to a character stream, so that text can be read from and written to a source that only understands bytes?

The short answer is that you wrap the byte stream. You hand an existing byte stream to a constructor and receive back a character stream that performs the conversion for you. Your code then works with characters, and the conversion between characters and bytes happens inside the wrapper.

What do you think happens?

The letter é is written as text. Before reading on, predict how many bytes it takes up under UTF-8, and how many under ISO-8859-1.

One byte under each charset
Two bytes under UTF-8 and one byte under ISO-8859-1
One byte under UTF-8 and two bytes under ISO-8859-1

Reveal answer

Answer:

Two bytes under UTF-8 and one byte under ISO-8859-1.

Under UTF-8, é is stored as the two bytes C3 and A9. Under ISO-8859-1, the same letter is the single byte E9. The charset decides how much space each character occupies, and this difference drives every example in the article.

Byte Streams and Character Streams

A byte stream works with raw binary data. A character stream works with characters rather than bytes. The two are different levels of abstraction, and a program often needs both: the bytes are what travels across a file or a network connection, while the characters are what the program actually manipulates.

Definition

InputStreamReader and OutputStreamWriter wrap an existing byte stream so that a program can read and write characters instead of raw bytes.

Two details follow from this picture. First, the wrapper needs something to read from or write to, so the byte stream must exist before the wrapper is built. Second, once the bytes are wrapped, the program has no direct handle on them. Everything it sees passes through the character stream. The ByteArrayOutputStream and ByteArrayInputStream in the examples below are byte streams; the two wrappers are the only places where conversion between characters and bytes happens.

When a String is written through an OutputStreamWriter into a byte stream, what path do the characters take, and where do they become bytes?

Read the diagram from left to right to follow a write. The program supplies characters. The OutputStreamWriter is the only step that converts them into bytes, and the byte stream then delivers those bytes to its destination. Reading runs the other way: bytes come out of the byte stream, the InputStreamReader turns them into characters, and the program receives characters.

Wrapping a Byte Stream in Practice

The standard input stream, System.in, is an InputStream. Its declared type is a byte stream, which is unusual for something a program reads from the keyboard as text. For that reason it is common to wrap System.in in a Reader when you want character data from standard input. The wrapping is a single line of code. The same pattern applies to network connections: their input and output streams are byte streams, and wrapping them lets a program send and receive character data without handling bytes directly.

java

import java.io.*;
import java.nio.charset.StandardCharsets;

public class WrapDemo {
    public static void main(String[] args) throws IOException {
        ByteArrayOutputStream bytes = new ByteArrayOutputStream();
        Writer writer = new OutputStreamWriter(bytes, StandardCharsets.UTF_8);
        writer.write("caf\u00e9");
        writer.flush();

        byte[] raw = bytes.toByteArray();
        System.out.println("Bytes written: " + raw.length);

        Reader reader = new InputStreamReader(
                new ByteArrayInputStream(raw), StandardCharsets.UTF_8);
        int ch;
        while ((ch = reader.read()) != -1) {
            System.out.println((char) ch);
        }
    }
}

import java.io.*;
import java.nio.charset.StandardCharsets;

public class WrapDemo {
    public static void main(String[] args) throws IOException {
        ByteArrayOutputStream bytes = new ByteArrayOutputStream();
        Writer writer = new OutputStreamWriter(bytes, StandardCharsets.UTF_8);
        writer.write("caf\u00e9");
        writer.flush();

        byte[] raw = bytes.toByteArray();
        System.out.println("Bytes written: " + raw.length);

        Reader reader = new InputStreamReader(
                new ByteArrayInputStream(raw), StandardCharsets.UTF_8);
        int ch;
        while ((ch = reader.read()) != -1) {
            System.out.println((char) ch);
        }
    }
}

Output

Bytes written: 5
c
a
f
é

Notes on the code

The String is written through the OutputStreamWriter, which converts it to bytes before they reach the ByteArrayOutputStream.
The writer is flushed so buffered characters are pushed through to the byte stream before the bytes are read.
The raw array holds 5 bytes: three for c, a, and f, and two for é under UTF-8.
The reader wraps a new ByteArrayInputStream over the same bytes and returns characters, not bytes, to the loop.
The output line for é shows the character the console displays when it decodes the text as UTF-8.

Trace the program line by line and note which values are characters and which are bytes at each point. The String literal is characters. The byte count and the array are bytes. The values printed by the loop are characters again. The program never touches an individual byte when it writes or reads the text; it calls write with a String and read to receive characters.

What a Charset Is

Definition

A charset is a named encoding that decides how characters map to binary data. Each charset defines a rule for turning characters into bytes and bytes back into characters.

Three common charsets are UTF-16, UTF-8, and ISO-8859-1. They store the same character in different amounts of space. The main lesson is that the charset decides the size of each stored character. A program that assumes one character always equals one byte will be wrong under some charsets, and a program that assumes one charset when the data used another will decode the same bytes into different text.

Charset	Storage unit	Space for one character
ISO-8859-1	8-bit byte	One byte per character
UTF-8	8-bit byte	One or more bytes, depending on the character; é takes two
UTF-16	16-bit code unit	Characters are stored as 16-bit units

How the three common charsets store characters. Read each row as the unit a charset uses to represent one character.

Default and Explicit Charsets

Readers and Writers use the default charset of the computer they run on unless you choose otherwise. You can name a different charset in the constructor, and the name you pass overrides the default for that stream only. Specifying the charset is a choice you make at the point where the stream is created. Once the wrapper exists, it converts characters to bytes using that charset.

java

// Relies on the machine's default charset
Reader implicit = new InputStreamReader(in);

// Names the charset, so the result does not depend on the machine
Reader explicit = new InputStreamReader(in, StandardCharsets.UTF_8);

// Relies on the machine's default charset
Reader implicit = new InputStreamReader(in);

// Names the charset, so the result does not depend on the machine
Reader explicit = new InputStreamReader(in, StandardCharsets.UTF_8);

Notes on the code

The first constructor takes no charset argument, so the default charset of the running machine is used.
The second constructor names UTF-8 explicitly, so the mapping between characters and bytes is fixed by the code.

Charset Mismatch Between Writer and Reader

The charset is the missing piece in any exchange of text. The bytes produced by a writer are only meaningful if the reader interprets them with the same charset. Suppose a sender writes the letter é using UTF-8, and a receiver reads the same bytes using ISO-8859-1. The receiver is not told that anything went wrong. It simply decodes the bytes it was given, and the result is different text.

If the sender encodes é as UTF-8 but the receiver decodes with ISO-8859-1, at which step does the wrong character appear?

The wrong character appears at the decoding step on the receiving side. The sender's encoding was correct, and the bytes were delivered intact. The receiver's reader, however, maps each byte to a single ISO-8859-1 character. The two UTF-8 bytes for é therefore become two separate characters, which is why the text no longer reads as é.

java

import java.io.*;
import java.nio.charset.StandardCharsets;

public class MismatchDemo {
    public static void main(String[] args) throws IOException {
        byte[] sent = "caf\u00e9".getBytes(StandardCharsets.UTF_8);

        Reader wrong = new InputStreamReader(
                new ByteArrayInputStream(sent), StandardCharsets.ISO_8859_1);
        int count = 0;
        while (wrong.read() != -1) {
            count++;
        }
        System.out.println("Characters read: " + count);
    }
}

import java.io.*;
import java.nio.charset.StandardCharsets;

public class MismatchDemo {
    public static void main(String[] args) throws IOException {
        byte[] sent = "caf\u00e9".getBytes(StandardCharsets.UTF_8);

        Reader wrong = new InputStreamReader(
                new ByteArrayInputStream(sent), StandardCharsets.ISO_8859_1);
        int count = 0;
        while (wrong.read() != -1) {
            count++;
        }
        System.out.println("Characters read: " + count);
    }
}

Output

Characters read: 5

Notes on the code

The sender encodes the text as UTF-8, so é becomes two bytes and the array holds five bytes in total.
The receiver names ISO-8859-1, which maps each byte to one character.
Each of the five bytes decodes to one character, so the receiver sees five characters where the sender wrote four.

Worked example

Tracing the letter é through a mismatch

A sender writes the four-character text café using UTF-8. A receiver reads the same bytes using ISO-8859-1. What does the receiver end up with, and at which step does it go wrong?

1
Encode on the sender side
c, a, and f each take one byte under UTF-8. The letter é takes two bytes, C3 and A9. The sender produces five bytes in total.
2
Deliver the bytes unchanged
The byte stream carries the five bytes exactly as they were written. Nothing about the bytes is altered in transit.
3
Decode on the receiver side
The receiver uses ISO-8859-1, which maps every byte to one character. The five bytes become five characters.
4
Locate the wrong result
The C3 and A9 bytes decode to two separate characters instead of the single letter é. The error appears in the decoding step on the receiving side.

Result

The receiver reads five characters instead of four, and the é is replaced by two unrelated characters. The fix is to use the same charset on both sides, which means naming it explicitly in both constructors.

Mistakes Beginners Make with Charsets

Common mistake
Assuming one character always equals one byte
What it looks like
Allocating a byte array whose length equals the number of characters in a UTF-8 string containing é.
Why it fails
The charset decides how much space each character occupies. Under UTF-8, é takes two bytes, so the byte count and character count differ.
Do this instead
Measure the bytes after encoding with the charset you intend to use, rather than counting characters.
Common mistake
Writing with one charset and reading with another
What it looks like
A writer uses UTF-8 and a reader uses ISO-8859-1, so é arrives as two characters.
Why it fails
The reader interprets the bytes with a mapping the writer did not use, producing different text.
Do this instead
Pass the same charset to the writer and the reader, such as StandardCharsets.UTF_8 on both sides.
Common mistake
Omitting the charset and assuming it is fixed
What it looks like
Creating new InputStreamReader(in) and expecting the same text on every computer.
Why it fails
Readers and Writers use the default charset of the machine they run on, so the result depends on the environment.
Do this instead
Name the charset in the constructor whenever the bytes will be read somewhere else.
Common mistake
Forgetting that the wrapper needs an existing byte stream before it is built

Practice: Find the Mismatch

MEDIUM

A server writes the text naïve through an OutputStreamWriter using UTF-8 and sends the bytes to a client. The client wraps the same bytes in an InputStreamReader with no charset argument, and the client machine's default charset is ISO-8859-1. Identify the step where the wrong text can first appear, and state the one-line change that would prevent it.

Hints

Ask which step converts characters to bytes and which step converts bytes back to characters.
Look at the constructor on the client side. Does it name a charset?
The fix belongs in the constructor that creates the reader, and it should match the writer's charset.

Your attempt (not saved)

Key Takeaways in Review

1
InputStreamReader and OutputStreamWriter wrap an existing byte stream so the program reads and writes characters instead of raw bytes.
2
When writing, the OutputStreamWriter converts characters to bytes; when reading, the InputStreamReader converts bytes to characters.
3
A charset decides how characters map to bytes, and the charset decides how much space each character occupies. UTF-16, UTF-8, and ISO-8859-1 are three examples.
4
Name the charset explicitly in the constructor, and use the same charset for writing and reading, because the default depends on the machine.

At a glance

InputStreamReader and OutputStreamWriter wrap an existing byte stream so the program works with characters instead of raw bytes.
Writing flows from characters to bytes through the OutputStreamWriter; reading flows from bytes to characters through the InputStreamReader.
A charset is a named encoding that decides how characters map to bytes, and it decides the space each character occupies.
Name the charset explicitly, and use the same charset for writing and reading, so a mismatch does not silently change the text.

Why Programs Need Characters

Byte Streams and Character Streams

Wrapping a Byte Stream in Practice

What a Charset Is

Default and Explicit Charsets

Charset Mismatch Between Writer and Reader

Mistakes Beginners Make with Charsets

Practice: Find the Mismatch

Key Takeaways in Review

At a glance

Related Topics