in Accessibility, Markup, Ruby

SimpleCrawler for your everyday web crawling needs

Over at the standards-schmandards blog I often test websites to gather statistics on specific HTML use, accessibility and other things. Each time I have written a web crawler to collect the data. In Python and Ruby this is a simple task but last time it was like a déjà vu and I decided to create a Ruby library that I could use in the future.

SimpleCrawler is a Ruby gem that covers basic web crawling needs. Coupled with Raakt and Ruport it can be used to create a basic accessibility report for a website.

A minimal example:

require 'simplecrawler'

# Set up a new crawler
sc = SimpleCrawler::Crawler.new("http://www.peterkrantz.com/")

sc.crawl { |document|
   # output the crawled uri
   puts document.uri
}

For more details and examples see the SimpleCrawler wiki.

  1. Andrei: Thank you. Forgot to set the wiki permissions. It should be available now.

Comments are closed.