File Transfer Protocols (FTP)
curl and wget are command-line tools for retrieving files from the web on Unix-like systems using HTTP or FTP protocols.
Why Command-Line Downloads Matter
A graphical browser is not always available when you need a file. You may be working on a remote server, automating a task, or retrieving many files at once. curl and wget let you fetch webpages and files directly from a Unix-like command line. Both tools can use HTTP or FTP protocols to retrieve content.
curl and wget are command-line tools for retrieving files from the web on Unix-like systems using HTTP or FTP protocols.
The HTTP Request-Response Cycle
When you run curl or wget, the tool sends an HTTP GET request to the remote server. The request identifies the URL of the file or webpage you want. The server receives the request, locates the requested content, and sends a response back. The command-line tool then either saves the received data to disk or displays it in the terminal, depending on the command and its options.
| Server response | Meaning for the retrieval |
|---|---|
| 200 OK | The file is successfully retrieved. |
| 404 Not Found | The requested content was not found; the tool reports a failure and no file is saved. |
Basic response outcomes described in the source material.
Saving One File with curl
curl can retrieve content, but saving it to a file requires the -O flag in this lesson's workflow. With -O, curl uses the filename from the remote URL and saves the result in the current directory. Without -O, curl prints the received content to the terminal instead of saving it to disk.
Mapping a remote filename to a local file
You want curl to retrieve the remote file cover.jpg and save it in the current directory rather than print its contents.
Choose the save option: Add -O to curl. This tells curl to use the remote filename.
Send the request: curl sends a request for the URL that identifies cover.jpg.
Save the response: If the server successfully retrieves the file, curl saves the received data in the current directory under the remote filename.
The local file is named cover.jpg. The same process can handle plain-text or binary content.
Retrieving and Mirroring with wget
wget also downloads webpages and remote files. Unlike curl in the basic comparison here, wget automatically saves files with their original names, so it does not require curl's -O flag for that behavior.
wget's distinctive feature in this topic is recursive downloading. With -r, wget follows the website's link structure. When it encounters a page, it parses the HTML to find linked pages and resource references, then downloads those items and continues the process recursively. The result is a local mirror of the website structure. An additional -l option can limit how many levels deep wget follows links.
Choosing the Retrieval Tool
Both tools perform the basic task of downloading files through the same request-response pattern. curl is described as more flexible for general-purpose data transfer, while wget is especially useful when you want automatic filename saving or recursive website downloading. For one remote file, either tool can retrieve the content; the practical choice depends on the behavior you need after the response arrives.
| Retrieval task | Useful choice | Reason |
|---|---|---|
| Save one remote file with its remote filename | curl -O | The -O flag tells curl to use the remote filename. |
| Retrieve one webpage or file and save it automatically | wget | wget automatically saves files with their original names. |
| Transfer data with flexible general-purpose behavior | curl | The source describes curl as more flexible for general-purpose data transfer. |
| Download a website hierarchy | wget -r | wget follows links and resource references recursively. |
Tool selection based on the capabilities described in the source material.
Diagnosing Download Results
Debugging starts by separating the request outcome from the display or saving behavior. A 200 OK response indicates successful retrieval. A 404 Not Found response indicates that the requested content was not found; the tool reports the failure and no file is saved. If curl prints content in the terminal, check whether -O was omitted before concluding that the request failed.
Assuming curl always saves the retrieved content.
In the basic behavior described here, curl prints the content to the terminal when -O is not used.
Fix:
Add -O when the goal is to save the file using the remote filename.Treating terminal output as proof that a file was saved.
curl can display content instead of saving it.
Fix:
Check whether -O was included and verify the response indicates successful retrieval.Expecting a missing URL to produce a local file.
The source states that the tool reports the failure and no file is saved.
Fix:
Treat the server response as a failure and check the requested URL.Using a single-file mindset for a website-mirroring task.
Following links and resource references recursively is the capability provided by wget with -r.
Fix:
Use wget -r and apply a level limit with -l when you need to control depth.
For a successful saved download, verify two parts of the result: the server response should indicate 200 OK, and the command should have saved the content rather than merely displaying it. For curl, the -O option is the key check in this workflow. For a failed 404 response, do not expect a saved file.
Practice: Predict the Outcome
What do you think happens?
What should you expect from curl URL-for-cover.jpg when the server returns 200 OK but the command does not include -O?
Reveal answer
Answer: The content is printed to the terminal.
The source states that curl requires -O to save files in this workflow. Without -O, it prints the content to the terminal. The 200 OK response indicates successful retrieval, but it does not by itself select the local saving behavior.
Choose the better command-line approach for each situation: retrieving one file while preserving its remote filename, downloading a website hierarchy, and performing flexible general-purpose data transfer. Explain which option or capability led to each choice.
Hints
- Remember that curl needs -O to save with the remote filename.
- Remember that wget uses -r for recursive downloading.
- Compare the source's descriptions of curl's flexibility and wget's recursive behavior.
Key Takeaways
- curl and wget retrieve web content from the command line using HTTP or FTP protocols.
- Both tools follow the request-response cycle: a command sends a request, the server returns a response, and the tool displays or saves the result.
- curl needs -O to save a retrieved file with its remote filename; without -O, it prints the content to the terminal.
- wget automatically saves files with their original names and supports recursive website downloading with -r.
- A 200 OK response indicates successful retrieval, while a 404 Not Found response indicates failure and no saved file.
Key Takeaways
- curl and wget are command-line retrieval tools that use HTTP or FTP protocols.
- The server response is central to understanding whether a request succeeded.
- Use curl -O when curl should save the remote file under its original filename.
- Use wget for automatic filename saving and for recursive website downloading with -r.
- Debug by distinguishing terminal output, a failed status such as 404 Not Found, and a successful 200 OK response with a saved file.