Understanding Graph Structures and Connectivity
spider.py systematically crawls web pages and stores them in a local database, recording both page content and the links between pages.
From Web Pages to a Graph
A web crawler turns a collection of live web pages into a structure that can be studied. It visits pages, records their content, and notes which pages link to which other pages. In the resulting graph, each page is represented as a node and each hyperlink is represented as a relationship between two nodes. The crawler is therefore the foundation for later analysis: without collected page and link data, there is no network on which to calculate connectivity or importance.
A page record describes a node. A link record describes a directed relationship from a source page to a target page. Together, these records form the graph.
A Page's Journey into Storage
spider.py begins with a starting URL and a requested crawl size. It fetches the starting page, extracts the links found on that page, and writes information to the SQLite database file spider.sqlite. The page is stored with a unique ID, its URL, and a count of outgoing links. The relationships created by those links are stored separately using the source page ID and target page ID.
| Database structure | What it records |
|---|---|
| Pages table | A unique page ID, the URL, and the count of outgoing links |
| Links table | The source page ID and the target page ID |
The two database structures described for spider.sqlite
A Three-Page Crawl
Imagine an empty spider.sqlite database, a starting URL of http://example.com/, and a request to crawl 3 pages. Trace what happens as pages are discovered.
Start: The crawler begins with the starting URL. It fetches that page and records it as a page with a unique ID. The links found on the page become unvisited links that may be visited later.
Continue: The crawler selects an unvisited link, fetches that page, records its page information, and stores the relationships from the new page to its outgoing links.
Reach the requested count: The same process continues until three pages have been crawled. The database now contains page records and link records connecting the pages and their discovered targets.
Retain future work: Links discovered but not yet visited remain available as future crawling targets. They are part of the growing collection of unvisited links.
The crawl produces a small graph in spider.sqlite: visited pages are stored as nodes, discovered relationships are stored as links, and remaining unvisited links provide possible next targets.
Choosing the Next Page
The crawler does not finish after storing the starting page. Each page can reveal more URLs, so the crawler adds those links to the collection of possible future targets. It then chooses an unvisited link and repeats the fetch, extraction, and storage process. On a restart, spider.py begins from a random unvisited link in its queue. This lets successive runs explore different branches instead of repeatedly following only the first path discovered.
The important idea is that discovery and visitation are different states. A link can be known because it was extracted from a page, while its target page can still be unvisited. The queue connects these two stages: it holds discovered targets that can become future pages in the stored graph.
Building on Earlier Sessions
The database is persistent, so the collected pages and links remain on disk between program runs. When spider.py is run again, it checks the existing database and skips pages that have already been recorded. If one session collects 10 pages and a later session is asked to collect 20 pages, the later session fetches only the 10 new pages needed to extend the collection rather than re-fetching the original 10.
Separate Webs in One Database
A single spider.sqlite database can contain pages discovered from more than one starting URL. For example, one session can begin at http://www.dr-chuck.com/ and another can begin at http://www.wikipedia.org/. These starting points are treated as separate webs, meaning they may form regions that are not directly connected to each other. Nevertheless, the crawler stores them in the same database and treats all unvisited links across the webs as one unified queue.
The database can therefore contain both connected clusters and loosely connected or isolated pages. The crawler does not maintain a separate visitation process for each starting point. When choosing the next page, it randomly selects from all unvisited links across the webs, so visits from different starting points can be interleaved.
Reading the Stored Network
The stored database is more than a list of URLs. Its page and link records support questions about the network: which pages link to a particular page, which pages a given page links to, and what statistics describe the network. The collected graph can be analyzed with algorithms such as a simplified page rank calculation. D3 can read the page and link data and render nodes and edges as an interactive graph, making clusters, highly connected hub pages, and isolated pages easier to see.
Collection and analysis are separate stages. spider.py gathers the page and link data; later queries, algorithms, or visualization tools use that stored graph to reveal connectivity.
Common Crawling Mistakes
Treating the crawler as a one-page downloader.
The crawler's purpose is to build page relationships, so discovered links are essential future targets.
Fix:
Track both the current page record and the outgoing link relationships that it produces.Assuming every discovered link has already been visited.
A link can be discovered and placed in the unvisited queue before its target page is fetched.
Fix:
Distinguish discovered links from pages that have actually been recorded in the database.Expecting a later run to fetch every page again.
The crawler is additive and skips pages already in the database.
Fix:
Treat the persistent database as the crawler's record of prior work.Assuming two starting URLs require two databases.
Multiple webs can coexist in one spider.sqlite database, and their unvisited links are handled as one queue.
Fix:
Use the shared database while remembering that the starting points may remain separate regions.Assuming every page belongs to one connected cluster.
Separate webs may not be directly connected, and the resulting database can contain clusters or isolated pages.
Fix:
Inspect the stored relationships rather than assuming connectivity from shared storage alone.
Practice the State Trace
Imagine that a database already contains pages collected from one starting URL. You start spider.py again with a different starting URL. Explain what can be shared, what can remain separate, and how the crawler chooses among future targets.
Hints
- Identify what is shared by both crawling sessions.
- Distinguish a stored page from an unvisited link.
- Remember how the crawler selects its next target across multiple webs.
Practice Answer
Explain the database behavior when a second independent starting URL is added.
Shared storage: Both starting points and the pages they lead to can be stored in the same spider.sqlite database.
Separate regions: The two starting points may produce webs that are not directly connected, so sharing a database does not guarantee one connected cluster.
Unified selection: The crawler considers all unvisited links across the webs together and randomly chooses the next page from that combined set.
Independent starting points can coexist in one persistent graph while retaining separate connectivity patterns and sharing one pool of unvisited targets.
Key Takeaways
- spider.py builds a graph by storing pages as records and hyperlinks as relationships between page IDs.
- A page moves from live web content into persistent storage through fetching, link extraction, page recording, and relationship recording.
- Unvisited links become future crawling targets, allowing the crawler to discover more pages systematically.
- The persistent spider.sqlite database lets later sessions skip stored pages and add only new work.
- Multiple starting URLs can create separate webs inside one database, while all unvisited links share one crawling queue.
Key Takeaways
- A web crawler converts pages and hyperlinks into a graph stored in a database.
- Each fetched page contributes a page record and link relationships, while discovered but unvisited URLs become future targets.
- Repeated sessions extend the existing graph and skip pages already stored.
- Independent starting points can share one database without necessarily forming one connected region.
- The stored graph supports later queries, algorithms, and visualizations of connectivity.