Comments on: Ruby Web Crawler http://www.netphase.com/2007/04/19/ruby-web-crawler/ for a connected world Thu, 20 Nov 2008 11:15:07 +0000 http://wordpress.org/?v=2.6.3 By: Carsten http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-53 Carsten Wed, 23 Jan 2008 11:04:38 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-53 Great code, Will have to try it out. Thanks man! Great code,

Will have to try it out.

Thanks man!

]]> By: Shig http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-26 Shig Thu, 13 Sep 2007 15:16:38 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-26 Sounds good. Thanks for the response. Sounds good. Thanks for the response.

]]> By: scott http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-25 scott Thu, 13 Sep 2007 15:09:06 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-25 While it would be easy to add this to a controller, it's probably not the best place for this. One possibility would be to create an <a href="http://www.railsmanual.org/class/ActiveRecord%3A%3AObserver" rel="nofollow">Observer</a> to handle it; however, I would be more inclined to create a task (in lib/tasks) to handle the crawling and set up cron to run it periodically. That would insure a constant level of resource usage and make it easier to process the requests serially. While it would be easy to add this to a controller, it’s probably not the best place for this. One possibility would be to create an Observer to handle it; however, I would be more inclined to create a task (in lib/tasks) to handle the crawling and set up cron to run it periodically. That would insure a constant level of resource usage and make it easier to process the requests serially.

]]> By: Shig http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-24 Shig Thu, 13 Sep 2007 03:46:58 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-24 Scott - I'm currently in the process of learning Ruby and RoR. Is this something that would integrate easily into an RoR app? I'm wondering how this would fit into the MVC architecture. Scott - I’m currently in the process of learning Ruby and RoR. Is this something that would integrate easily into an RoR app? I’m wondering how this would fit into the MVC architecture.

]]> By: scott http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-5 scott Sun, 27 May 2007 20:15:13 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-5 You could store it wherever you like. In this example, the contents of the page are in the page_text variable. You can see an <a href="http://rubyforge.org/snippet/detail.php?type=snippet&id=155" rel="nofollow">updated version</a> at RubyForge where I'm storing the results in a hashmap. You could store it wherever you like. In this example, the contents of the page are in the page_text variable. You can see an updated version at RubyForge where I’m storing the results in a hashmap.

]]> By: webug http://www.netphase.com/2007/04/19/ruby-web-crawler/#comment-4 webug Sun, 27 May 2007 03:02:18 +0000 http://blog.netphase.com/2007/04/19/ruby-web-crawler/#comment-4 dear sir: i am a ruby beginner,i wondere in your programe above, where do you store the download html pages ? thanks dear sir:
i am a ruby beginner,i wondere in your programe above, where do you store the download html pages ?
thanks

]]>