Here’s a web crawler I wrote awhile back. It’s pretty simple, but does the job. If you need something more, you might try rdig or Nutch.

You can run this as a stand-alone script and just pass in the URL to crawl as an argument.

[ruby]
require ‘net/http’
require ‘uri’

class SiteCrawler
def initialize(url)
@site_uri = URI.parse(url)
@site_uri.path = “/” if @site_uri.path == “”
@visited = Hash.new
@queue = Array.new
addPath(@site_uri.path)
puts “Initialized site crawl for #{@site_uri}”
end

def addPath(path)
@queue.push path
@visited[path] = false
end

def getPage(path)
begin
uri = @site_uri.clone
uri.path = uri.path + path if path != “/”
puts “getting #{uri}”
response = Net::HTTP.get_response(uri)
rescue Exception
puts “Error: #{$!}”
return “”
end
return response.body
end

def queueLocalLinks(html)
html.scan(/ uri = URI.parse("#{w}")
if !@visited.has_key?(uri.path) and
(uri.relative? or uri.host == @site_uri.host)
addPath(uri.path)
end
}
end