Scrapy Tutorial Learning Notes

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

Install Scrapy in Ubuntu

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
pip install Scrapy

Creating a project

scrapy startproject tutorial

project directory

tutorial/
scrapy.cfg            # deploy configuration file

tutorial/             # project's Python module, you'll import your code from here
    __init__.py

    items.py          # project items file

    pipelines.py      # project pipelines file

    settings.py       # project settings file

    spiders/          # a directory where you'll later put your spiders
        __init__.py
        ...

Defining our Item

Items are containers that will be loaded with the scraped data, they are declared by creating a scrapy.Item class and defining its attributes as scrapy.Field objects.

import scrapy

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

Spider

Spiders are classes that you define and Scrapy uses to scrape information from a domain (or group of domains).

import scrapy
    
    class DmozSpider(scrapy.Spider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]
    
        def parse(self, response):
            filename = response.url.split("/")[-2] + '.html'
            with open(filename, 'wb') as f:
                f.write(response.body)

To create a Spider, you must subclass scrapy.Spider and define some attributes:

• name: identifies the Spider. It must be unique.
• start_urls: a list of URLs where the Spider will begin to crawl from.
• parse(): a method of the spider, which will be called with the downloaded Response object of each start URL. The response is passed to the method as the first and only argument.This method is responsible for parsing the response data and extracting scraped data (as scraped items) and more URLs to follow(as Request objects).

Crawling

scrapy crawl dmoz

Scrapy creates scrapy.Request objects for each URL in the start_urls attribute of the Spider, and assigns them the parse method of the spider as their callback function.

These Requests are scheduled, then executed, and scrapy.http.Response objects are returned and then fed back to the spider, through the parse() method.

Extracting the Data

Scrapy uses a mechanism based on XPath or CSS expressions called Scrapy Selectors to extract data from web pages.You can see selectors as objects that represent nodes in the document structure.

Selectors have four basic methods:

• xpath(): returns a list of selectors, each of which represents the nodes selected by the xpath expression given as argument.
• css(): returns a list of selectors, each of which represents the nodes selected by the CSS expression given as argument.
• extract(): returns a unicode string with the selected data.
• re(): returns a list of unicode strings extracted by applying the regular expression given as argument.

scrapy.http.Response objects has a selector attribute which is an instance of Selector class. You can run queries on response by calling response.selector.xpath() or response.selector.css()or response.xpath() or response.css() for short.

Using our Item

Item objects are custom Python dicts; you can access the values of their fields using the standard dict syntax like:

def parse(self, response):
    for sel in response.xpath('//ul/li'):
        item = DmozItem()
        item['title'] = sel.xpath('a/text()').extract()
        item['link'] = sel.xpath('a/@href').extract()
        item['desc'] = sel.xpath('text()').extract()
        yield item

Following links

extract the links for the pages you are interested, follow them and then extract the data you want for all of them.

def parse(self, response):
    for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
        url = response.urljoin(href.extract())
        yield scrapy.Request(url, callback=self.parse_articles_follow_next_page)

def parse_articles_follow_next_page(self, response):
for article in response.xpath("//article"):
    item = ArticleItem()

    ... extract article data here

    yield item

next_page = response.css("ul.navigation > li.next-page > a::attr('href')")
if next_page:
    url = response.urljoin(next_page[0].extract())
    yield scrapy.Request(url, self.parse_articles_follow_next_page)

When you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes.

Storing the scraped data

scrapy crawl dmoz -o items.json

That will generate an items.json file containing all scraped items, serialized in JSON.If you want to perform more complex things with the scraped items, you can write an Item Pipeline.