NAME
    Web::Scraper - Web Scraping Toolkit using HTML and CSS Selectors or
    XPath expressions

SYNOPSIS
      use URI;
      use Web::Scraper;
      use Encode;

      # First, create your scraper block
      my $authors = scraper {
          # Parse all TDs inside 'table[width="100%]"', store them into
          # an array 'authors'.  We embed other scrapers for each TD.
          process 'table[width="100%"] td', "authors[]" => scraper {
            # And, in each TD,
            # get the URI of "a" element
            process "a", uri => '@href';
            # get text inside "small" element
            process "small", fullname => 'TEXT';
          };
      };

      my $res = $authors->scrape( URI->new("http://search.cpan.org/author/?A") );

      # iterate the array 'authors'
      for my $author (@{$res->{authors}}) {
          # output is like:
          # Andy Adler      http://search.cpan.org/~aadler/
          # Aaron K Dancygier       http://search.cpan.org/~aakd/
          # Aamer Akhter    http://search.cpan.org/~aakhter/
          print Encode::encode("utf8", "$author->{fullname}\t$author->{uri}\n");
      }

    The structure would resemble this (visually) { authors => [ { fullname
    => $fullname, link => $uri }, { fullname => $fullname, link => $uri }, ]
    }

DESCRIPTION
    Web::Scraper is a web scraper toolkit, inspired by Ruby's equivalent
    Scrapi. It provides a DSL-ish interface for traversing HTML documents
    and returning a neatly arranged Perl data structure.

    The *scraper* and *process* blocks provide a method to define what
    segments of a document to extract. It understands HTML and CSS Selectors
    as well as XPath expressions.

METHODS
  scraper
      $scraper = scraper { ... };

    Creates a new Web::Scraper object by wrapping the DSL code that will be
    fired when *scrape* method is called.

  scrape
      $res = $scraper->scrape(URI->new($uri));
      $res = $scraper->scrape($html_content);
      $res = $scraper->scrape(\$html_content);
      $res = $scraper->scrape($http_response);
      $res = $scraper->scrape($html_element);

    Retrieves the HTML from URI, HTTP::Response, HTML::Tree or text strings
    and creates a DOM object, then fires the callback scraper code to
    retrieve the data structure.

    If you pass URI or HTTP::Response object, Web::Scraper will
    automatically guesses the encoding of the content by looking at
    Content-Type headers and META tags. Otherwise you need to decode the
    HTML to Unicode before passing it to *scrape* method.

    You can optionally pass the base URL when you pass the HTML content as a
    string instead of URI or HTTP::Response.

      $res = $scraper->scrape($html_content, "http://example.com/foo");

    This way Web::Scraper can resolve the relative links found in the
    document.

  process
      scraper {
          process "tag.class", key => 'TEXT';
          process '//tag[contains(@foo, "bar")]', key2 => '@attr';
          process '//comment()', 'comments[]' => 'TEXT';
      };

    *process* is the method to find matching elements from HTML with CSS
    selector or XPath expression, then extract text or attributes into the
    result stash.

    If the first argument begins with "//" or "id(" it's treated as an XPath
    expression and otherwise CSS selector.

      # <span class="date">2008/12/21</span>
      # date => "2008/12/21"
      process ".date", date => 'TEXT';

      # <div class="body"><a href="http://example.com/">foo</a></div>
      # link => URI->new("http://example.com/")
      process ".body > a", link => '@href';

      # <div class="body"><!-- HTML Comment here --><a href="http://example.com/">foo</a></div>
      # comment => " HTML Comment here "
      #
      # NOTES: A comment nodes are accessed when installed
      # the HTML::TreeBuilder::XPath (version >= 0.14) and/or
      # the HTML::TreeBuilder::LibXML (version >= 0.13)
      process "//div[contains(@class, 'body')]/comment()", comment => 'TEXT';

      # <div class="body"><a href="http://example.com/">foo</a></div>
      # link => URI->new("http://example.com/"), text => "foo"
      process ".body > a", link => '@href', text => 'TEXT';

      # <ul><li>foo</li><li>bar</li></ul>
      # list => [ "foo", "bar" ]
      process "li", "list[]" => "TEXT";

      # <ul><li id="1">foo</li><li id="2">bar</li></ul>
      # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ];
      process "li", "list[]" => { id => '@id', text => "TEXT" };

  process_first
    "process_first" is the same as "process" but stops when the first
    matching result is found.

      # <span class="date">2008/12/21</span>
      # <span class="date">2008/12/22</span>
      # date => "2008/12/21"
      process_first ".date", date => 'TEXT';

  result
    "result" allows to return not the default value after processing but a
    single value specified by a key or a hash reference built from several
    keys.

      process 'a', 'want[]' => 'TEXT';
      result 'want';

EXAMPLES
    There are many examples in the "eg/" dir packaged in this distribution.
    It is recommended to look through these.

NESTED SCRAPERS
    Scrapers can be nested thus allowing to scrape already captured data.

      # <ul>
      # <li class="foo"><a href="foo1">bar1</a></li>
      # <li class="bar"><a href="foo2">bar2</a></li>
      # <li class="foo"><a href="foo3">bar3</a></li>
      # </ul>
      # friends => [ {href => 'foo1'}, {href => 'foo2'} ];
      process 'li', 'friends[]' => scraper {
        process 'a', href => '@href',
      };

FILTERS
    Filters are applied to the result after processing. They can be declared
    as anonymous subroutines or as class names.

      process $exp, $key => [ 'TEXT', sub { s/foo/bar/ } ];
      process $exp, $key => [ 'TEXT', 'Something' ];
      process $exp, $key => [ 'TEXT', '+MyApp::Filter::Foo' ];

    Filters can be stacked

      process $exp, $key => [ '@href', 'Foo', '+MyApp::Filter::Bar', \&baz ];

    More about filters you can find in Web::Scraper::Filter documentation.

XML backends
    By default HTML::TreeBuilder::XPath is used, this can be replaces by a
    XML::LibXML backend using Web::Scraper::LibXML module.

      use Web::Scraper::LibXML;

      # same as Web::Scraper
      my $scraper = scraper { ... };

AUTHOR
    Tatsuhiko Miyagawa <miyagawa@bulknews.net>

LICENSE
    This library is free software; you can redistribute it and/or modify it
    under the same terms as Perl itself.

SEE ALSO
    <http://blog.labnotes.org/category/scrapi/>

    HTML::TreeBuilder::XPath