Storing and Querying Spider Data with SQLite

From Database to Diagnostic View

After a spider finishes crawling, its findings are stored in spider.sqlite. That database file is a binary SQLite file, so it is not a convenient human-readable report. The spdump.py tool provides that report: it reads the database and displays information about the pages and connections discovered by the spider.

The most important habit is to read spdump.py as a diagnostic view rather than as a complete list of every page encountered. Its output focuses on pages that participate in the crawl's incoming-link network. That makes the output useful for checking connectivity, interpreting PageRank-related values, and spotting results that do not match your expectations.

Reading the Five Positions

• incoming_links is the direct connectivity count: how many other pages in the crawl link to this page.
• old_rank is the earlier rank value. It is often None on the first run.
• new_rank is the newer rank value shown by the output after rank computation.
• page_id is the page identifier stored in the tuple.
• url is the page address associated with that record.

Tracking Rank Across Runs

The two rank fields let you notice whether rank information has an earlier value to compare with. On the first run, old_rank is often None. After PageRank has been computed multiple times, numeric values appear. Therefore, None in the old_rank position is not automatically evidence that the crawl failed; it can describe the state of an initial run.

Understanding the Output Filter

spdump.py does not display every page discovered by the spider. It includes only pages with at least one incoming link. A page with zero inbound connections is excluded by design.

This filter focuses the report on pages that participate in the connectivity network. A page can be encountered in the crawl while still having no other crawled page link to it. Such a page has zero incoming links and therefore does not appear in spdump.py output.

Using Connectivity as Evidence

The first tuple position is especially useful when validating a crawl. If a page you expected to be central has only one incoming link, that result may indicate that the spider did not crawl deeply enough or missed pages. The count also helps you inspect the site's structure: pages with many incoming links can be navigation hubs, homepages, or popular content pages.

Incoming-link counts also provide the foundation for PageRank. Pages with more incoming links tend to have higher PageRank, although the exact calculation also depends on the PageRank of the pages linking to them. Use the rank fields together with the connectivity count instead of treating the count as the entire ranking calculation.

• Check whether important pages have the incoming-link counts you expected.
• Use the URL and page_id fields to identify which record a surprising result belongs to.
• Inspect old_rank and new_rank to understand whether the output reflects an initial run or later PageRank computation.
• Remember that an absent page may have been filtered because its incoming-link count is zero.

Common Interpretation Mistakes

• Treating spdump.py as a complete list of all discovered pages.

  The tool filters out pages with zero incoming links.

  Fix: Interpret the output as a filtered connectivity report, not as an unfiltered discovery list.

• Reading the tuple fields in the wrong order.

  The format always begins with incoming_links and ends with url.

  Fix: Read every tuple using the fixed order: incoming_links, old_rank, new_rank, page_id, url.

• Assuming old_rank being None proves that the spider failed.

  old_rank is often None on the first run.

  Fix: Recognize None as an expected initial-run state and inspect the rest of the tuple.

• Treating incoming-link count as the complete PageRank calculation.

  The exact PageRank calculation also depends on the PageRank of the pages providing those links.

  Fix: Use incoming_links as a connectivity measure and examine rank fields separately.

Practice the Diagnostic Read

Key Takeaways

1. spdump.py reads spider.sqlite and presents crawl information in a human-readable five-element tuple.
2. The tuple order is incoming_links, old_rank, new_rank, page_id, url.
3. incoming_links directly measures how many other crawled pages link to the page.
4. Only pages with at least one incoming link appear in the output.
5. old_rank is often None on the first run, while numeric values appear after PageRank has been computed multiple times.
6. Unexpected counts, rank values, identifiers, URLs, or missing pages can help validate and debug the crawl.