Handling Errors in Web Scraping

From Page to Link List

A web page displays clickable links, but a scraper sees those links as HTML anchor tags. An anchor tag stores its destination in an href attribute. BeautifulSoup can parse downloaded HTML, find every anchor tag, and retrieve each href value. When something goes wrong, the reliable approach is to trace the data from downloaded bytes to the final printed values instead of guessing which step failed.

Tracing the Extraction Steps

1. urllib.request.urlopen() opens the address over the network.
2. read() obtains the downloaded HTML as bytes.
3. BeautifulSoup parses those bytes and builds an internal tree representing the HTML document.
4. soup('a') searches the parsed tree and returns all anchor tag elements.
5. tag.get('href', None) retrieves the href value from each anchor tag, or returns None when that attribute is absent.
6. print() sends each extracted value to the console.

The expression soup('a') searches the entire parsed document tree, not just one particular part of the page. It returns a list of all matching anchor tag elements. Each returned tag can contain attributes such as href. Calling tag.get('href', None) asks for the href attribute safely: an existing attribute produces its stored value, while a missing attribute produces None.

from bs4 import BeautifulSoup

html = b'<a href="https://example.com">Example</a><a>Missing destination</a>'
soup = BeautifulSoup(html, "html.parser")
tags = soup('a')

for tag in tags:
    href = tag.get('href', None)
    print(href)

https://example.com
None

Certificate Verification

When urllib.request.urlopen() opens a URL, Python checks the website's SSL/TLS certificate. The check is intended to verify that the connection is secure and that the connection is with the real server rather than an imposter. In some websites or network environments, this check can fail even when the website is legitimate.

import ssl
import urllib.request

context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

response = urllib.request.urlopen(url, context=context)
html = response.read()

Debugging the Data Flow

A link extractor can fail at several different points. The HTML might not have been downloaded or parsed correctly. The search might find fewer anchor tags than expected. Or the anchor tags might exist without href attributes. Add observations at each stage so the first unexpected value identifies the part of the pipeline that needs investigation.

html = response.read() print("downloaded bytes:", len(html)) soup = BeautifulSoup(html, "html.parser") print("soup created:", soup is not None) tags = soup('a') print("anchor tags:", len(tags)) for tag in tags: href = tag.get('href', None) print("href:", href)

Mistakes to Avoid

• Accessing href directly instead of using get()

  An anchor tag can exist without an href attribute.

  Fix: Use tag.get('href', None) so a missing attribute produces None.

• Forgetting to pass the SSL context

  The configured context is not used unless it is passed to urlopen().

  Fix: Pass the context through the context parameter when opening the URL.

• Debugging only the final printed list

  The failure may have occurred during download, parsing, or tag selection rather than attribute retrieval.

  Fix: Trace len(html), soup creation, len(tags), and each href value.

• Assuming every href is a complete web address

  The program prints the exact href stored in each anchor tag, including relative paths and fragments.

  Fix: Interpret the output as raw attribute values before deciding how to process them.

Practice the Trace

Reliable Extraction

1. The extraction pipeline moves from downloaded HTML bytes to a BeautifulSoup tree, then to anchor tags, href values, and printed output.
2. soup('a') searches the complete parsed document tree and returns all anchor tag elements it finds.
3. tag.get('href', None) safely retrieves an href value and returns None when the attribute is missing.
4. Certificate verification can be explicitly disabled with an SSL context, but doing so has security implications.
5. Debugging is most effective when you inspect download size, soup creation, tag count, and extracted href values in sequence.