Over at the standards-schmandards blog I often test websites to gather statistics on specific HTML use, accessibility and other things. Each time I have written a web crawler to collect the data. In Python and Ruby this is a simple task but last time it was like a déjà vu and I decided to create a Ruby library that I could use in the future.
A minimal example:
For more details and examples see the SimpleCrawler wiki.