Parsing HTML with BeautifulSoup

How it works

1. html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
2. Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
3. An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
4. soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
5. Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]