scRUBYt!

Scraping pricegrabber.com with scRUBYt!

You will see an example of:

• filling and submitting a textfield
• crawling to the detail pages of the results
• downloading images
• scraping desired data (text, price, user rating)

Goal:

• Go to http://cameras.pricegrabber.com.
• Enter ‘canon EOS 20D’ into the search textfield and submit the form.
• Download the item image from the detail page(s) and additional product information (like the lowest price) from the ‘Product Details’ tab.

Solution:

• Use the navigational commands ‘fetch’, ‘fill_textfield’, ’submit’ to navigate to the list page containing items related to the search term ‘canon EOS 20D’
• Crawl to the detail page by “clicking on” the link Canon EOS 20D Digital SLR Camera Body Only.
• Choose the detail tab ‘Product Details’ by cliking on it.
• Download the product image by providing its location’, ’src’ and finally setting a directory to download the images into using a ‘download’ pattern (note: you do not need to create the given directory, scRUBYt! does it for you! Look for it at the same location as your extractor is stored)
• Scrape the desired data (in this case Description and Lowest Price).
• Extract user rating with a regular expression.

The code:

require 'rubygems'
require 'scrubyt'

pricegrabber_data = Scrubyt:: Extractor.define do

  fetch 'http://cameras.pricegrabber.com'
  fill_textfield 'form_keyword', 'canon EOS 20D'
  submit

  camera 'Canon EOS 20D Digital SLR Camera Body Only', :generalize => false do
    camera_detail do
      detail_tab 'Product Details', :generalize => false do
        detail_detail do
          camera_record :generalize => false do
            image "http://ai.pricegrabber.com/pi/0/37/48/3748716_125.jpg" do
              image_url "src", :type => :attribute do
                download 'camera_images', :type => :download
              end
            end
            name 'EOS 20D Digital SLR Camera Body Only'
            lowestprice '$649.95'
            rating_and_reviews :contains => '(Read 31 Reviews)' do
              rating /(d.dd / d.dd)/
            end
          end
        end
      end
    end
  end
end

puts pricegrabber_data.to_xml

If you’d like to run, it cut’n'paste it from here (otherwise the code will be messed up).

If you are more of a visual type, check out this cart depicting the process.

Troubleshooting

Problem: “This extractor does not work!”

Solution: ‘Lowest Price’ changes quite frequently - check if the actual ‘Lowest Price’ at http://cameras.pricegrabber.com/digital/canon/m/3748716/details/ does match the one stated in the extractor. If not, simply “upgrade” the example with the actual correct data. Hmmm, this should work.

Problem: “The extractor returns data for the first listed item only. Where is the rest of data?”

Solution: The extractor was designed only to give a taste of how to solve similar scraping problems like it was stated in ‘Goals” therefore it is limited to the first item only. If you would like to scrape all the items on the list page simply remove :generalize => false from “camera ‘Canon EOS 20D Digital SLR Camera Body Only’, :generalize => false do”. However, it will return data from the first list page only, while no crawling to the next pages is defined in the scraper (in our case it would be kind of insane since no next pages exist for this type of camera).

Ranking Ruby/Rails Blogs with scRUBYt! Using Alexa and Technorati

As a (positive) side-effect of Ruby’s and RoR’s current popularity, we can witness the mushrooming of Ruby/Rails blogs of different size, shape, color and quality. While this is a great thing in general, it is easy to get overflooded with all this information pouring in from every side - much harder to pick a smaller set which you can monitor in your time devoted for reading Ruby blogs.

Fortunately there are some great Ruby/Rails blog aggregators like http://rubycorner.com/, http://www.planetrubyonrails.org/ - not to be confused with http://www.planetrubyonrails.com/ (here is why) or http://planet.caboo.se/. The problem is that it’s clumsy to monitor all these aggregators - they contain hundreds of blogs each with a lot of overlapping.

This issue was partially solved by this Ruby on Rails Blog Aggregator Dupe Prevention yahoo pipe. However, there is a fundamental problem with that: You still have several hundreds of blogs - nearly 500 of them, and new ones coming out day by day. If you have time to read all these periodically, all the power to you - however, the rest of us would need a way to pick the top n ones in order to be able to cope with their reading.

I am proposing a solution for this problem, by performing these steps:

1. Collecting all the blog URLs from the above 4 aggreagators
2. Performing some data cleaning (e.g. http://www.site.com, http://site.com, http://site.com/blog and other variants, with and without trailing slashes are pointing to the same site) and dupe removal
3. Querying http://www.alexa.com and http://www.technorati.com for the popularity of each of the blogs on the list (this already generated two rankings: “Alexa top 30″ and “Technorati top 30″.
Alexa required some intervention by hand (more on that later).
4. Putting it all together: based on the two top 30 lists, generate a final, all-in-one Ruby/Rails blog top 10!

Let’s see an overview of the process - then I will explain each step in detail!

Let’s begin with scraping http://www.planetrubyonrails.com/. This extractor was the easiest to construct of all the four:

    planet_ror_com_data = Scrubyt::Extractor.define do
      fetch("http://www.planetrubyonrails.com/pages/channels")

      link "http://weblog.jamisbuck.org/"
    end

This wasn’t too hard, was it? We have just extracted all the blogs URLs from planetrubyonrails.com! To observe the result, all we need is to run this snippet:

planet_ror_com_data.to_hash.each{|h| puts h[:link]}

http://weblog.jamisbuck.org/
http://www.therailsway.com/
http://weblog.rubyonrails.com/
http://blogs.relevancellc.com/
http://www.anarchogeek.com/
http://www.bencurtis.com/
http://slash7.com/
http://blog.lathi.net/
http://convergentarts.com/
http://blog.innerewut.de/
http://codefluency.com/rails-views
http://www.busyashell.com/blog/
http://curthibbs.wordpress.com/
http://blog.bleything.net/
http://www.loudthinking.com/
http://railstips.org/
http://null.in/
http://www.jvoorhis.com/
http://www.puneruby.com/blog
http://sitekreator.com/satishtalim/rubyblog.html
http://www.rubyinside.com/
http://www.chadfowler.com/index.cgi
http://dibya.wordpress.com/
http://rubymerchant.blogspot.com/index.html
http://www.rubyonrailsblog.com/
http://blog.caboo.se/
http://www.ryandaigle.com/
http://vinsol.com/
http://jroller.com/page/obie
http://www.infoq.com/
http://drnicwilliams.com/
http://www.danwebb.net/
http://www.lukeredpath.co.uk/
http://blog.testingrails.com/
http://blog.vixiom.com/
http://robertrevans.com/
http://redhanded.hobix.com/
http://benmyles.com/
http://weblog.techno-weenie.net/
http://www.clarkware.com/cgi/blosxom
http://mir.aculo.us/
http://www.oreillynet.com/ruby/blog/
http://www.robbyonrails.com/
http://blog.grayproductions.net/
http://www.rubyquiz.com/
http://www.codyfauser.com/
http://railsexpress.de/blog/
http://www.softiesonrails.com/
http://blog.ericgoodwin.com/
http://iamrice.org/
http://blog.hasmanythrough.com/
http://david.goodlad.ca/
http://nubyonrails.topfunky.com/
http://onrails.org/
http://www.railtie.net/
http://brainspl.at/
http://podcast.rubyonrails.org/
http://blog.leetsoft.com/
http://errtheblog.com/
http://jystewart.net/process
http://www.pluginaweek.org/
http://mephistoblog.com/
http://encytemedia.com/
http://evil.che.lu/
http://technomancy.us/
http://poocs.net/
http://www.opensoul.org/
http://www.mathewabonyi.com/
http://www.railslivecd.org/
http://blog.methodmissing.com/
http://sporkmonger.com/breakpoint_client
http://typo.onxen.info/
http://blog.inquirylabs.com/
http://rashkovskii.com/
http://www.infused.org/
http://interblah.net/
http://streamlinedframework.com/
http://toolmantim.com/
http://webonrails.wordpress.com/
http://www.sapphiresteel.com/
http://eigenclass.org/hiki.rb
http://www.pjhyett.com/
http://jonathan.tron.name/
http://amazing-development.com/
http://merubyyoujane.com/
http://www.fallenrogue.com/
http://troubleseeker.com/
http://blog.nicksieger.com/
http://blog.methodmissing.com/
http://schf.uc.org/
http://www.brianketelsen.com/
http://www.simplisticcomplexity.com/
http://fluctisonous.com/
http://www.pinupgeek.com/
http://web2withrubyonrails.gauldong.net/
http://www.koziarski.net/
http://www.notsostupid.com/
http://www.shanesbrain.net/
http://blog.evanweaver.com/
http://www.zerosum.org/devblog
http://blog.zenspider.com/
http://townx.org/taxonomy/term/5/0
http://www.fromdelhi.com/
http://myles.eftos.id.au/blog
http://www.jeremyhubert.com/
http://www.jason-palmer.com/
http://seanicus.blogspot.com/
http://ajaxonrails.wordpress.com/
nil
http://www.matthewbass.com/blog
http://rails.co.za/
http://www.shifteleven.com/
http://peepcode.com/
http://blog.wolfman.com/
http://blog.dannyburkes.com/
http://jayfields.blogspot.com/
http://www.freeonrails.com/
http://techcheatsheets.com/tag/ruby%20or%20rails
http://rails-engines.org/rss/news/
http://www.railsenvy.com/
http://www.eribium.org/
http://hobocentral.net/blog
http://maintainable.com/articles
http://railspikes.com/
http://radiantcms.org/
http://hackety.org/
http://adam.blogs.bitscribe.net/
http://www.rubyfleebie.com/
http://www.stephenbartholomew.co.uk/
http://izumi.plan99.net/blog
http://blog.davidchelimsky.net

Let’s move on to http://planet.caboo.se/ which is just slightly harder to scrape:

    planet_caboose_data = Scrubyt::Extractor.define do
      fetch("http://planet.caboo.se/")

      link_row do
        link "al3x/@href"
      end
    end

The next site is http://rubycorner.com/. The new thing here compared to the first two scrapers is that the blog URLs can be found on different pages - ergo we need to crawl to all of those pages. Fortunately this is no problem for scRUBYt!:

    rubycorner_data = Scrubyt::Extractor.define do
      fetch 'http://rubycorner.com/blogs/lang/en'

      link "Atlantic Dominion Solutions/@href"
      next_page ">"
    end

Last but not least, here is the scraping code for http://www.planetrubyonrails.org/

      fetch "http://www.planetrubyonrails.org/"

      link_container "div[Feeds]" do
        link "a work on process/@href", :generalize => true
      end
    end

I don’t have too much to say about the data cleaning part - if you are interested, you can download the code used for generating this tutorial and check it out there.

Scraping alexa was great fun - mainly because they don’t want to be scraped! However, if they would like to stop a halfway serious script junkie, they will have to do better than this. Here is the alexa extractor:

  def scrape(site)
    @hidden = open("http://client.alexa.com/common/css/scramble.css").read.scan(/^\.(.+) /).flatten unless @hidden
    alexa_data = Scrubyt:: Extractor.define do
        fetch "http://www.alexa.com/data/details/traffic_details?url=#{site}"

        td "td[Traffic Rank for]" do
          span '/span[3]' do
            html :type => :html_subtree
          end
        end
    end
    data = alexa_data.to_hash[0][:html]
    data ? descramble(data) : nil
  end

  def descramble(mess)
    data = mess.gsub!(//,”).split(//)
    data = data.reject{|d| d=="" || (@hidden.include? d.scan(/"(.+?)"/).flatten[0])}
    data.map!{|d| d.sub(/".+">/,”) }
    data.join(”)
  end

Here is the extractor I have used to scrape technorati:

  def scrape(site)
    techno_data = Scrubyt::Extractor.define do
      fetch "http://www.technorati.com/blogs/#{site}?reactions"
      rank "/html/body/div/div/div/div/div/div" do
        rank_number /Rank: (.+)$/
      end
    end
    techno_data.to_hash.reject{|h| h.empty?}[0][:rank_number]
  end

Nothing special here - what you did not see yet in the other extractors is the use of a regular expression pattern (rank_number), which gets the number out of the corresponding element and leaves the trash behind.

The only manual step I had to make in this whole process was to tweak the alexa results a bit - because according to alexa.com, “Alexa’s traffic rankings are for top level domains only (e.g. domain.com). We do not provide separate rankings for subpages within a domain (e.g. www.domain.com/subpage.html) or subdomains (e.g. subdomain.domain.com)“. This in practice meant that I have checked all blogger.com, wordpress.org, jroller.com and similar domains whether they concrete pages are really that high ranked, or just their top-level domain is. This blog also made it to the top 30, but I had to throw it away because about half of the traffic goes to the scRUBYt! forum, and the remaining visits were not enough to stay inside the top 30.

If you are still with me, I guess you had enough of scraping, screwing, cleaning and assembling by now - and your ordeal is over! Here are the results:

There were a few blogs which performed equally well on both lists - they were assembled into a top 10 list, which you can check out here.

I would really like to hear your opinion on this little experiment - whether you think it makes sense or it is completely off, how could it be improved in the future, what features could be added etc. If I’ll receive some positive feedback, I think I will work on the algorithm a bit more, and run it once in say every 3 months to see what’s happening around the Ruby/Rails blogosphere. Let me know what do you think!

Related Links

• The code used in the article
• The final result: Ruby/Rails blog top 10

scRUBYt! 0.3.0 released

Thanks to thescRUBYt! forum, the team has received a lot of feedback - we have tried to incorporate the most features requested by the most people, also fixing a few annoying bugs. Well, I guess no one is interested in this introductory babbling, so here you go: the CHANGELOG for 0.3.0:

[NEW] complete rewrite of the output system, creating a solid 
           foundation for more robust output functions 
           (credit: Neelance)
[NEW] logging - no annoying puts messages anymore! 
           (credit: Tim Fletcher)
[NEW] can index an example - e.g.
           link “more[5]”‘
           semantics: give me the 6th element with the text “more”
[NEW] can use XPath checking an attribute value, like
            “//div[@id=’content’]”
[NEW] default values for missing elements (first version was 
            done in 0.2.8 but it did not work for all cases)
[NEW] possibility to click button with it’s text (instead of it’s index) 
           (credit: Nick Merwin)
[NEW] clicking radio buttons
[NEW] can click on image buttons (by specifying the name of the 
           button)
[NEW] possibility to extract an URL with one step, like so:
          link “The Difference/@href”
           i.e. give me the href attribute of the element matched by the 
           example ‘The Difference’
[NEW] new way to match an element of the page:
           content “div[The Difference]”
           means ‘return the div which contains the string “The Difference”‘. 
           This is useful if the XPath of the element is non-constant across 
           the same site (e.g.sometimes a banner or add is added, sometimes
           not etc.)
[NEW] Clicking image maps; At the moment this is achieved by specifying
           an index, like
           click_image_map 3
          which means click the 4th link in the image map
[FIX] Replacing \240 ( ) with space in the preprocessing phase
         automatically
[FIX] Fixed: correctly downloading image if the src attribute had a leading
         space, as in
        

Ruby Web Scraping Tool Guide

Hpricot vs Mechanize vs ScrAPI vs Watir vs ScRUBYt! vs…? Which one should you pick to solve your upcoming Web scraping project and why?

Since I got this question more times recently, I will try to answer it here. This article is not about how cool scRUBYt! is and how much the others suck, concluding that you should use scRUBYt! only and nothing else. I will try to be as objective as possible and strive to discuss the more generic features of web scraping tools, rather than talk about scRUBYt! itself. (You can find plenty of that on this blog and by visiting the forum if you are interested.)

First of all, since Web scraping is hard to define precisely (it means a bit different thing to everyone, ranging from grabbing a few elements from a page to spidering and crawling whole sites and turning their content into structured format), I believe it is somewhat impossible to answer this question - however, since I am fond of hard-to-solve problems, I will give it a shot and try nevertheless :-).

There is a plethora of tools that can be used to aid Web scraping in Ruby - this article on web scraping describes most of the important ones. A quick recap of the packages mentioned in this article:

• The main focus of WWW::Mechanize and (Fire)Watir is simulating web navigation (fetching documents, clicking links, submitting forms etc.). For plain old pages (i.e. no AJAX, JavaScript) Mechanize is a clear winner beacuse of speed issues.
• Hpricot is a handy and extremely fast HTML parser which offers also (pseudo)XPath and CSS selectors making web scraping possible. Though you have the most alternatives here (HTML parsers/XPath evaluators), if you would like to stay on this level, I definitely recommend Hpricot.
• ScrAPI and scRUBYt! are high-level web scraping frameworks, suited for more complex tasks. They are trying to hide the details of HTML and lookup of the elements you would like to grab (and in scRUBYt’s case also navigation, like fetching pages, following next links, crawling to new pages) and figure out your intention which you can feed into them through the provided DSLs.

Your first choice to make is whether you would like to use a low-level tool (usually smaller projects) or a high-level one (more complex scenarios). I believe this problem is as old as the first software package ever written, so it is definitely not a Ruby web scraping tool question in the first place. The dilemma is the following:

• Using a low level tool you have maximal freedom: you can do everything as you like and you are not constrained by anyone other’s decisions or design. On the other hand, most probably you will ‘reinvent the wheel’, i.e. waste your time writing code that was already implemented a mozillion times by other people (who potentially have more expertise and/or time to make it better) or fiddle around with low-level quirks.
• A high level package offers you a lot of goodies which you can just take and use out-of-the box - depending on the implementation, possibly allowing you plenty of space to tweak and configure these prefabricated components. However, this is also the downside of this option: if the creator of the tool did not add a feature you need, it’s much harder to change things to your liking than with a low-level package.

Talking about Ruby web scraping tools, the low level ones give you great freedom for doing whatever you want - for the price of much more code and the need to treat messy HTML and other low-level stuff like different protocols, HTTP connections, and proxies for example. On the other hand, scRUBYt! can scrape a page in as few as 5-6 lines of code - however, if there is a bug or something is missing/not working properly, you might have a much bigger problem than in the first case (unless somebody fixes it for you, that is). So which is the right choice to take in this situation?

You should consider more parameters to answer this question:

• Scenario size - If you would like to accomplish something small, it’s possible that the flexibility of a low-level package outweighs the constraints imposed by the heavier frameworks. However, if you would like to do advanced things like spidering a whole site or match elements based on complicated criteria, you should consider to check out a high-level tool.
• Amount of low level stuff - How much of the code will be web scraping and how much pure Ruby would you like to use in between? If you would like to leave yourself plenty of space for customizing and low level tweaking, and/or you plan to use a lot of Ruby code besides the scraping, probably a low-level tool is better. However, if your code will be mostly (or fully) Web scraping, it’s possible you will be better off with scRUBYt! or scrAPI.
• The ‘NOW’ factor - In a lot of cases, this point is enough alone in itself to make the decision. Sometimes you just need to do something NOW - and framework X does not contain the feature you can not live without. In this case you have several options: you can either code up the missing functionality yourself (sometimes this is still faster if you would like to use the other powerful features of the framework), ask the maintainer of the package for this feature and hope he will respond ‘we are just hacking on it’ If neither of these alternatives apply, and you really want to move on, sometimes you just have to omit framework X, no matter how cool it looks like and how great it will be if the missing feature will be added. You need to accomplish your task NOW…
• Framework evolution - If you are planning to do a bigger scenario, you should also consider whether the framework you would like to use is an ongoing project, and if yes, what are the future plans of it’s developers. A ‘live’ framework is always a better bet - during it’s development, bugs will be fixed, features will be added and most probably some support will be provided.
• Community and Support - This can be also a deciding factor on it’s own. Since web scraping can be (read: is) a very tedious process, it is always good to have some support from the community around the framework to help you through the hardest times. I think scRUBYt! already has a great community and a nice support - check the scRUBYt! forum and see it yourself.
• The future: maintainability and robustness - Unless you are going to write a throwaway scraper, you should really consider the maintainability issue. Web sites change over time, and they change often. Using low-level scraping frameworks, this will bring in a lot of extra work: with every modification of the page, you will have to rewrite your scraper. At the moment, scrAPI is the most robust framework from this point of view - since it uses CSS selectors which are the most resistant to change. scRUBYt! extractors are also easy to maintain - if the page changes, you will have to rewrite just the examples (or if the examples are still there, not even those) and regenerate the extractors which will work automagically again. In the future we are planning to add CSS selectors to scRUBYt!, as well as different automatic adaptation techniques to be resistant to change.

The need for navigation (clicking links, submitting forms, crawling to new pages etc) is also an important factor - if you need it, you will have to either combine more of the above tools, or decide between Mechanize (which now contains Hpricot) or scRUBYt! (built on Mechanize and Hpricot, FireWatir integration coming soon) which are the only tools capable of both web site navigation and some serious scraping (O.K., FireWatir can also do some scraping, but it’s much less powerful as Hpricot’s or scRUBYt!’s).

Every project is different. If you have chosen X for a simple, small project, it does not necessarily mean you will reach after it again when you will be crawling a set of sites. Take your time to consider all the requirements and goals you would like to achieve - choosing wisely can save you a lot of time.

scRUBYt! Forum Goes Live

Since more people have requested a forum for better information exchange, I have installed beast to enhance the discussion. After all, talking via WordPress comments and direct emails is not really cool :-). Please find the brand new forum here:

If you have any problems, errors, suggestions, please report them here, (or at the forum, if the problem is other than ‘the forum does not work :-). Enjoy!