Introduction to Web Scraping

The Response Before the Page

When a program connects to a web server and requests a file, the server does not send only the file data. It sends a structured HTTP response. The response begins with information about the document, continues with metadata in HTTP headers, and then contains the actual file content. A web scraper must recognize this structure before it can correctly separate metadata from the document it wants to process.

Tracing the Four-Part Layout

The response has a fixed sequence. First comes the status line. Next come the headers, which provide metadata describing the document. After the headers comes a blank line. Finally, the body contains the actual file content. Although the topic is often described as a three-part response, the blank line is an essential delimiter inside that structure: it marks the boundary between the headers and the body.

Reading Header Pairs

HTTP headers are metadata about the document in the response. They are organized as key-value pairs: the header name identifies the kind of information being supplied, and the associated value supplies that information. Headers appear after the status line and before the blank line. This position matters because it tells a parser that the pairs belong to the metadata section rather than to the body.

Finding the Body Boundary

The blank line is not empty information to discard casually. It is the delimiter that separates the header section from the response body. A parser that recognizes this boundary can treat everything before it as response metadata and everything after it as the actual file content. Ignoring the delimiter is a common source of parsing errors because metadata and document content may then be mixed together.

Common Header Meanings

Several headers appear frequently when examining HTTP responses. Content-Type describes the type of the document. Content-Length describes the document's length. Server provides information about the server sending the response. Date and Connection are also frequently encountered and provide information associated with the response date and connection. These values help a scraper understand the metadata that accompanies the body rather than mistaking that metadata for the document itself.

Parsing Mistakes to Avoid

• Treating the response as only the file content

  The server sends a structured response containing metadata before the actual file content.

  Fix: Identify the status line and headers before processing the body.

• Ignoring the blank line

  The blank line is the delimiter between headers and the body.

  Fix: Use the blank line to separate metadata from the actual file content.

• Treating headers as ordinary body content

  Headers describe the document; they are not the document body.

  Fix: Keep header metadata separate from the body after locating the delimiter.

Practice the Sequence

Key Takeaways

1. An HTTP response begins with a status line, continues with headers, includes a blank line, and ends with the body.
2. Headers are key-value pairs that provide metadata describing the document and response.
3. The blank line is the delimiter that separates headers from the actual file content.
4. Content-Type describes document type, Content-Length describes document length, and Server provides server information.
5. Correct parsing depends on preserving the distinction between metadata and body content.