Comments on: Ruby Web Crawler

By: Andre Durao

Andre Durao — Thu, 21 Jan 2010 13:34:49 +0000

thanks for that!
Helped a lot.

By: Nik

Nik — Tue, 30 Jun 2009 20:02:22 +0000

Hello, thanks for this script. I have been wanting to experiment with making my own web indexes for a few websites. –

Do you happen to know that, now say we have the crawler, how we come up with a list of websites for it to crawl on if we were to index the whole world wide web. Should we run through all ips from 0.0.0.0 to 255.255.255.255 ?

Or is it that the idea is that you give one seed website, a rather big one, and let the spider go link after link until hopefully it includes all webpages? or fail somewhere and try to find another one seed website?

Thanks!

By: Manjunath

Manjunath — Tue, 05 May 2009 10:14:00 +0000

hi, i am new to the ruby can i know this

i run this code in SciTE when i execute i am getting this error

updateversionofsitecrawl.rb:6:in `initialize’: uninitialized constant WebCrawler::URL (NameError)
from updateversionofsitecrawl.rb:53:in `new’
from updateversionofsitecrawl.rb:53

can u please help me out this

By: Carsten

Carsten — Wed, 23 Jan 2008 11:04:38 +0000

Great code,

Will have to try it out.

Thanks man!

By: Shig

Shig — Thu, 13 Sep 2007 15:16:38 +0000

Sounds good. Thanks for the response.

By: scott

scott — Thu, 13 Sep 2007 15:09:06 +0000

While it would be easy to add this to a controller, it's probably not the best place for this. One possibility would be to create an Observer to handle it; however, I would be more inclined to create a task (in lib/tasks) to handle the crawling and set up cron to run it periodically. That would insure a constant level of resource usage and make it easier to process the requests serially.

By: Shig

Shig — Thu, 13 Sep 2007 03:46:58 +0000

Scott – I’m currently in the process of learning Ruby and RoR. Is this something that would integrate easily into an RoR app? I’m wondering how this would fit into the MVC architecture.

By: scott

scott — Sun, 27 May 2007 20:15:13 +0000

You could store it wherever you like. In this example, the contents of the page are in the page_text variable. You can see an updated version at RubyForge where I'm storing the results in a hashmap.

By: webug

webug — Sun, 27 May 2007 03:02:18 +0000

dear sir:
i am a ruby beginner,i wondere in your programe above, where do you store the download html pages ?
thanks