Web Crawling and Data Collection with spider.py

From Crawled Links to Rankings

A crawler collects pages and the links between them, but that collection does not yet tell you which pages are most important. PageRank supplies that next step. When you run sprank.py, it examines the link structure stored in the database and transforms it into numerical importance scores. The calculation is iterative: each pass uses the previous scores to produce a more refined set of scores.

Starting an Iterative Run

To begin, invoke sprank.py and specify how many iterations you want. The program prompts you for that input, then performs the requested number of complete passes through the pages in the database. An iteration is not a single-page calculation. It is one pass through the collection, during which PageRank values are recalculated using the scores from the preceding iteration.

Reading Progress Output

Each progress line reports two pieces of information: the iteration number and the average change in PageRank score per page during that iteration. The average-change value is a diagnostic signal. A large value means the calculation is still making substantial corrections. A smaller value means the scores are changing less from one pass to the next.

Interpreting Stored Scores

After the run, PageRank scores are stored in the database and can be queried. The scores are relative: they describe the importance of pages compared with other pages in the same network. A higher score indicates greater importance according to the link structure. The calculation reflects both how many pages point to a page and how important those referring pages are.

A detailed database record adds context to the score. The source describes records containing a page ID, a version number of 1.0, the PageRank score, the number of outgoing links, and the URL. This lets you inspect both the ranking result and part of the page's link topology. For example, the source describes a record with a PageRank of 2.135 and four outgoing links.

Running the Cycle Again

PageRank is not necessarily a one-time operation. You can run sprank.py multiple times on the same database. A later run starts with the scores from the earlier run and iterates again, allowing further refinement when more convergence is needed.

The ranking workflow also changes when the crawler adds information. A practical cycle is to run spider.py, add newly crawled pages and their links to the database, and then run sprank.py again. The expanded network requires the scores to be recalculated so that the rankings account for the new pages and links.

Resetting Without Recrawling

Sometimes you want to restart the PageRank calculation without downloading the pages again. The spreset.py program resets all PageRank values to their initial state while preserving the page records and the link structure. After the reset, sprank.py can calculate scores again from the beginning using the same network.

Common Diagnostic Mistakes

• Treating the first iteration as the final ranking

  Initial estimates are rough, and later iterations refine them using the preceding scores.

  Fix: Inspect the average change across iterations and continue until the change is small enough for your acceptable convergence threshold.

• Assuming that a larger score is an absolute measure of importance

  PageRank scores are relative to the other pages in the same network.

  Fix: Use the scores to compare pages within the same database and link structure.

• Ignoring the average-change output

  The average change shows whether the calculation is still making substantial corrections or has begun to stabilize.

  Fix: Track whether the average change is decreasing and becoming very small.

• Recrawling when only a fresh ranking calculation is needed

  spreset.py can reset rank values while preserving the crawled pages and link structure.

  Fix: Use spreset.py, then run sprank.py again when the network itself should remain unchanged.

Check Your Understanding

Operational Takeaways

1. sprank.py converts the database's page-link structure into numerical PageRank scores through repeated passes.
2. Each progress line reports an iteration number and the average change in score per page.
3. A progressively smaller average change indicates that rankings are stabilizing; very small changes indicate convergence.
4. Scores are relative to the pages in the same network, and detailed records can include IDs, scores, outgoing-link counts, and URLs.
5. Run sprank.py again to refine or update rankings, use spider.py before ranking when new pages and links have been crawled, and use spreset.py when you need a fresh calculation without losing the network.