Introduction to BeautifulSoup

Tag inspection in BeautifulSoup means accessing the individual components of a Tag object: its attributes (via get() or attrs), its text content (via contents or get_text()), and its child elements (via contents).

Programming

From HTML Text to a Navigable Tree

BeautifulSoup is a Python library designed to parse real-world HTML. Its central job is to transform raw HTML text into a navigable structure. Once parsing is complete, HTML elements become nodes in a tree, and you can inspect tags by their names, attributes, text, and position among parent, child, and sibling elements.

How does raw HTML text become a nested structure that BeautifulSoup can navigate?

Provide the raw HTML string and a parser specification to the BeautifulSoup constructor.
Let the parser recognize tag boundaries and extract attributes.
BeautifulSoup builds an internal tree of elements.
Navigate the resulting object to locate tags and inspect their parts.

Why Real HTML Needs Tolerance

HTML resembles XML because both use tags, attributes, and hierarchical nesting. Their parsing expectations are different, however. XML requires a strict, well-formed structure. Real web pages commonly contain missing closing tags, malformed attributes, or invalid nesting. A strict XML parser can reject such a page as malformed and stop.

BeautifulSoup takes a tolerant approach. When it encounters malformed HTML, it makes assumptions about the author's intended structure, such as inferring where a missing closing tag should occur or doing its best to extract a malformed attribute value. It then builds a navigable tree instead of rejecting the entire page.

What happens when malformed HTML is processed by a strict XML parser compared with BeautifulSoup?

Approach	Response to malformed HTML	Result
Strict XML parsing	Enforces well-formed structure	May reject the page and stop
BeautifulSoup tolerant parsing	Makes assumptions about the intended structure	Builds a navigable tree

Installing and Starting the Parser

BeautifulSoup is available on the Python Package Index. The source material identifies pip as the easiest way to install it. After installation, import BeautifulSoup into a Python script and provide HTML text together with a parser specification.

python

pip install beautifulsoup4

pip install beautifulsoup4

python

from bs4 import BeautifulSoup

html = '<p>Welcome</p>'
soup = BeautifulSoup(html, 'html.parser')

from bs4 import BeautifulSoup

html = '<p>Welcome</p>'
soup = BeautifulSoup(html, 'html.parser')

In this sequence, html is the raw input, html.parser is the parser specification, and soup is the resulting BeautifulSoup object. The object represents the parsed HTML as a structure that can be queried and traversed.

Inside a Tag Object

A parsed HTML element becomes a BeautifulSoup Tag object. A Tag is not merely a string representation of the original element. It is a structured object with separately inspectable parts: its name, its attributes, its inner text and child nodes, and its position in the document tree.

What does a Tag contain, and how are its name, attributes, text content, and child elements connected?

Part	Access pattern	What it provides
Attribute value	tag.get('href', None)	A requested attribute value, or a default when it is missing
All attributes	tag.attrs	The tag's attributes
Contents	tag.contents	A list containing text nodes and child Tag objects
Clean text	tag.get_text().strip()	Text content with surrounding whitespace removed

Common inspection targets for a BeautifulSoup Tag.

Finding and Inspecting Links

Anchor elements are represented by the a tag. After parsing, retrieve the anchor tags from the document, then inspect each Tag separately. The URL is stored in the href attribute, the user-visible text can be obtained with get_text(), and the complete attribute collection is available through attrs.

python

from bs4 import BeautifulSoup

html = '''
<html>
  <body>
    <a href="https://example.com">Example</a>
    <a href="/about" class="navigation">About</a>
    <a href="#contact"><span>Contact</span></a>
  </body>
</html>
'''

soup = BeautifulSoup(html, 'html.parser')
links = soup.find_all('a')

for link in links:
    url = link.get('href', None)
    text = link.get_text().strip()
    attributes = link.attrs
    print(url)
    print(text)
    print(attributes)

from bs4 import BeautifulSoup

html = '''
<html>
  <body>
    <a href="https://example.com">Example</a>
    <a href="/about" class="navigation">About</a>
    <a href="#contact"><span>Contact</span></a>
  </body>
</html>
'''

soup = BeautifulSoup(html, 'html.parser')
links = soup.find_all('a')

for link in links:
    url = link.get('href', None)
    text = link.get_text().strip()
    attributes = link.attrs
    print(url)
    print(text)
    print(attributes)

Output

https://example.com
Example
{'href': 'https://example.com'}
/about
About
{'href': '/about', 'class': ['navigation']}
#contact
Contact
{'href': '#contact'}

The loop treats every result as a structured Tag. It does not need to parse the original HTML text manually. Instead, it asks the Tag for the specific pieces needed: href for the destination, get_text().strip() for readable text, and attrs for the full attribute collection.

How does BeautifulSoup find every anchor tag and expose its URL, text, attributes, and child content?

Contents, Text, and Children

A Tag's contents are the items between its opening and closing tags. The contents property returns a list. Each item may be a NavigableString representing text or another Tag representing a nested element. Because the original HTML may contain formatting whitespace, contents can include newline characters as well as child tags.

python

from bs4 import BeautifulSoup

html = '<a href="/profile">\n  <span>Profile</span>\n</a>'
soup = BeautifulSoup(html, 'html.parser')
link = soup.find('a')

print(link.contents)
print(link.get_text().strip())

from bs4 import BeautifulSoup

html = '<a href="/profile">\n  <span>Profile</span>\n</a>'
soup = BeautifulSoup(html, 'html.parser')
link = soup.find('a')

print(link.contents)
print(link.get_text().strip())

Output

[\n  , <span>Profile</span>, \n]
Profile

The contents result preserves the structure and whitespace found between the anchor's tags. get_text() is the more convenient choice when the goal is readable text rather than the individual child nodes. Applying strip() removes surrounding whitespace from that extracted text.

Understanding Link Destinations

Real web pages contain more than one kind of href value. An absolute URL includes its complete destination, such as https://example.com/page. A relative path points to a location related to the current page, such as /about. An in-page reference begins with a number sign, such as #contact, and refers to a location within the current page. BeautifulSoup exposes these values, but it does not make them equivalent destinations.

How does each href value point to a different location in relation to the current page?

Always filter and validate href values before using them. A scraper should distinguish complete URLs, relative paths, and in-page references instead of treating every href string as a ready-to-request web address.

Treating every href as an absolute URL
Relative paths are related to the current page, while absolute URLs include a complete destination.
Fix: Classify and validate the href before using it.
Assuming every anchor contains an href attribute
Direct attribute access can raise a KeyError when the attribute is missing.
Fix: Use link.get('href', None) when the attribute may be absent.
Using contents as if it were clean text
contents can include whitespace and nested Tag objects.
Fix: Use get_text().strip() for clean text extraction.

A Complete Inspection Routine

Inspect each anchor safely

Given a parsed HTML document, retrieve every anchor tag and report its href value, clean text, and attributes.

Parse: Pass the HTML string and html.parser to BeautifulSoup so the raw input becomes a navigable object.

Retrieve: Search the parsed object for all a tags and receive a collection of Tag objects.

Inspect: For each Tag, use get('href', None), get_text().strip(), and attrs to access the requested components.

Validate: Classify the href as an absolute URL, relative path, or in-page reference before using it.

The routine separates parsing, tag retrieval, component extraction, and URL handling into clear steps.

EASY

Write a short routine that parses an HTML string containing three anchor tags: one with an absolute URL, one with a relative path, and one with an in-page reference. For each anchor, print its href value, cleaned text, and complete attributes.

Hints

Create the BeautifulSoup object with the HTML string and html.parser.
Retrieve all anchor tags with find_all('a').
Use get('href', None), get_text().strip(), and attrs.
Compare the beginning and shape of each href before treating it as a destination.

Your attempt (not saved)

What do you think happens?

If an anchor contains a nested span and newline characters, what should you expect from contents compared with get_text().strip()?

Both return only the same clean string
contents preserves text and child nodes, while get_text().strip() returns cleaned text
contents returns the href and get_text().strip() returns the attributes

Reveal answer

Answer: contents preserves text and child nodes, while get_text().strip() returns cleaned text

contents represents the items inside the tag, including whitespace and nested Tag objects. get_text() extracts the rendered text, and strip() removes surrounding whitespace.

Essential Takeaways

BeautifulSoup transforms raw HTML into a navigable tree of Tag objects.
A Tag exposes its name, attributes, contents, text, and relationships in the document tree.
Use get('href', None) for safe attribute access, attrs for the complete attribute collection, and get_text().strip() for clean text.
contents preserves a tag's child structure and may include whitespace or nested tags.
BeautifulSoup's tolerant parsing approach is practical for malformed real-world HTML, but extracted URLs should still be classified and validated.

Key Takeaways

BeautifulSoup parses raw HTML into a navigable tree rather than leaving it as unstructured text.
Anchor tags can be retrieved as Tag objects and inspected through href, get_text(), attrs, and contents.
Use safe attribute access and clean text extraction to handle missing attributes and source whitespace.
Absolute URLs, relative paths, and in-page references require different handling.
BeautifulSoup tolerates malformed HTML that strict XML parsers may reject.

From HTML Text to a Navigable Tree

Why Real HTML Needs Tolerance

Installing and Starting the Parser

Inside a Tag Object

Finding and Inspecting Links

Contents, Text, and Children

Understanding Link Destinations

A Complete Inspection Routine

Inspect each anchor safely

Essential Takeaways

Key Takeaways

Related Topics