HTTP Requests and Response Objects
Binary files require byte-for-byte copying from a remote URL to your local disk.
From Remote Bytes to Local Storage
Downloading an image, video, or another non-text file means copying its bytes from a remote URL to a file on your local disk. The basic urllib pattern has two stages: read the complete response into memory, then write those bytes to a local file using binary write mode.
The Request and Response Sequence
The expression urllib.request.urlopen(url) sends the request associated with the URL and gives the program a response object. Calling read() on that response retrieves the entire file body. In this pattern, the body is then held in memory as data before it is written to disk.
import urllib.request response = urllib.request.urlopen(url) data = response.read() with open(filename, 'wb') as output_file: output_file.write(data)
Reading the Complete Response
The call response.read() is the point at which the complete file is brought into memory. The variable data represents the downloaded binary content that will later be passed to write(). This makes the pattern straightforward: retrieve all the bytes first, then save them.
Why wb Protects Binary Data
| Mode | Use in this pattern | Result for binary data |
|---|---|---|
| wb | Opens the local destination for binary writing | Preserves the downloaded bytes for the file copy |
| w | Text write mode | Can corrupt binary data |
The b in wb indicates binary mode, and the w indicates writing. For an image, video, or another non-text file, the downloaded content must be written byte for byte. The source specifically identifies wb as essential and warns that text mode, w, will corrupt binary data.
A Complete Download Pattern
Saving a Binary File Locally
Retrieve a binary file from url and save it using the local name filename.
Obtain the response: Call urllib.request.urlopen(url) to obtain the response object associated with the remote URL.
Read the response body: Call response.read() and store the complete downloaded content in data.
Open the destination: Open filename with wb so the local file is prepared for binary writing.
Write the bytes: Pass data to output_file.write() to copy the downloaded binary content to the local file.
The binary file has been copied from the URL into the local file, provided that the complete file fits within available RAM.
Trace the variables from left to right: url identifies the remote location, response refers to the returned response object, data holds the complete downloaded file in memory, and output_file represents the local destination opened for binary writing.
Mistakes That Damage Downloads
Opening the destination with w instead of wb
Text mode can corrupt binary data.
Fix:
Use open(filename, 'wb') when writing the downloaded binary content.Assuming read() avoids memory usage
This call downloads the entire file into memory before it is written.
Fix:
Use this simple pattern for files smaller than available RAM.Writing before reading the response body
The source pattern reads the response body into data before passing the downloaded content to write().
Fix:
Call data = response.read(), then write data to the binary destination.
Check Your Understanding
You need to retrieve a non-text file from url and save it as filename. Identify the correct mode for the local file and arrange the operations in the correct order.
Hints
- First obtain a response object with urllib.request.urlopen(url).
- Then call read() and store the complete result in a variable.
- Use binary write mode rather than text write mode.
What do you think happens?
What happens to memory when response.read() is used in this download pattern?
Reveal answer
Answer: The complete file is loaded into memory.
The source pattern uses response.read() to download the entire file into memory before writing it with open(filename, 'wb').write(data).
Key Takeaways
- urllib.request.urlopen(url) provides a response object for the remote URL.
- response.read() retrieves the complete binary file into memory.
- open(filename, 'wb') opens the local destination in binary write mode.
- Text mode, w, can corrupt binary data, so wb is essential for binary downloads.
- The complete-file-in-memory pattern is suitable for files smaller than available RAM.
Key Takeaways
- A binary download copies bytes from a remote URL to a local file.
- urllib.request.urlopen(url) returns a response object, and response.read() loads the complete response body into memory.
- The local destination must be opened with wb so binary data is written without text-mode corruption.
- Because read() loads the entire file at once, this pattern requires the file to fit within available RAM.