3.1 KiB

Raw Permalink Blame History

SEP	17
Title	Spider Contracts
Author	Insophia Team
Created	2010-06-10
Status	Draft

SEP-017: Spider Contracts

The motivation for Spider Contracts is to build a lightweight mechanism for testing your spiders, and be able to run the tests quickly without having to wait for all the spider to run. It's partially based on the [https://en.wikipedia.org/wiki/Design_by_contract Design by contract] approach (hence its name) where you define certain conditions that spider callbacks must met, and you give example testing pages.

How it works

In the docstring of your spider callbacks, you write certain tags that define the spider contract. For example, the URL of a sample page for that callback, and what you expect to scrape from it.

Then you can run a command to check that the spider contracts are met.

Contract examples

gExample URL for simple callback

The parse_product callback must return items containing the fields given in @scrapes.

System Message: WARNING/2 (<stdin>, line 38)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class ProductSpider(BaseSpider):
       def parse_product(self, response):
           """
           @url http://www.example.com/store/product.php?id=123
           @scrapes name, price, description
           """

gChained callbacks

The following spider contains two callbacks, one for login to a site, and the other for scraping user profile info.

The contracts assert that the first callback returns a Request and the second one scrape user, name, email fields.

System Message: WARNING/2 (<stdin>, line 57)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class UserProfileSpider(BaseSpider):
       def parse_login_page(self, response):
           """
           @url http://www.example.com/login.php
           @returns_request
           """
           # returns Request with callback=self.parse_profile_page

       def parse_profile_page(self, response):
           """
           @after parse_login_page
           @scrapes user, name, email
           """
           # ...

Tags reference

Note that tags can also be extended by users, meaning that you can have your own custom contract tags in your Scrapy project.

`@url`	url of a sample page parsed by the callback
`@after`	the callback is called with the response generated by the specified callback
`@scrapes`	list of fields that must be present in the item(s) scraped by the callback
`@returns_request`	the callback must return one (and only one) Request

Some tag constraints:

a callback cannot contain @url and @after

Checking spider contracts

To check the contracts of a single spider:

scrapy-ctl.py check example.com

Or to check all spiders:

scrapy-ctl.py check

No need to wait for the whole spider to run.

</html>

3.1 KiB Raw Permalink Blame History