http://www.peterkrantz.com/simplecrawler/wiki/ 2010-03-14T02:45:01+00:00 http://www.peterkrantz.com/simplecrawler/wiki/ http://www.peterkrantz.com/simplecrawler/wiki/lib/images/favicon.ico text/html 2009-05-04T21:40:44+00:00 Peter Krantz http://www.peterkrantz.com/simplecrawler/wiki/examples/find-broken-links Find broken links This is an example of how SimpleCrawler can be used to find broken links on a website (links with HTTP status 404). In the example the site in the command line argument is crawled. require 'simplecrawler' # Mute log messages module SimpleCrawler class Crawler def log(message) end end end # Set up a new crawler sc = SimpleCrawler::Crawler.new(ARGV[0]) # Crawl first 100 links sc.maxcount = 100 sc.crawl { |document| if document.http_status[0] != "200"… text/html 2009-05-04T21:03:53+00:00 Peter Krantz http://www.peterkrantz.com/simplecrawler/wiki/start With SimpleCrawler (SC) basic web crawling becomes easy in Ruby. Use SimpleCrawler as the foundation for your own crawling needs. SC is inspired by code in an article by Scott Nedderman (which didn't work properly for me). A minimal example (crawl a website and print page titles): text/html 2008-12-09T13:09:41+00:00 Peter Krantz http://www.peterkrantz.com/simplecrawler/wiki/principles Basic Principles of using SimpleCrawler Using SC involves the following steps: 1. Require the simplecrawler library require 'simplecrawler' 2. Create an instance of the SimpleCrawler::Crawler object Pass the website address as a parameter if you like. text/html 2008-01-26T13:37:55+00:00 Peter Krantz http://www.peterkrantz.com/simplecrawler/wiki/examples/accessibility-report Site Accessibility Report with Raakt and Ruport In this example SimpleCrawler can be used to crawl a website, test each page with the Ruby Accessibility Analysis Kit and format the result with Ruport. Please note that you must have a working installation of the Ruby programming language and the RubyGems package manager to use this code.