3.5 KiB
Spiders Contracts
Testing spiders can get particularly annoying and while nothing prevents you from writing unit tests the task gets cumbersome quickly. Scrapy offers an integrated way of testing your spiders by the means of contracts.
This allows you to test each callback of your spider by hardcoding a sample url and check various constraints for how the callback processes the response. Each contract is prefixed with an @ and included in the docstring. See the following example:
System Message: WARNING/2 (<stdin>, line 16)
Cannot analyze code. Pygments package not found.
.. code-block:: python
def parse(self, response):
"""
This function parses a sample response. Some contracts are mingled
with this docstring.
@url http://www.example.com/s?field-keywords=selfish+gene
@returns items 1 16
@returns requests 0 0
@scrapes Title Author Year Price
"""
You can use the following contracts:
System Message: ERROR/3 (<stdin>, line 31)
Unknown directive type "module".
.. module:: scrapy.contracts.default
System Message: ERROR/3 (<stdin>, line 33)
Unknown directive type "autoclass".
.. autoclass:: UrlContract
System Message: ERROR/3 (<stdin>, line 35)
Unknown directive type "autoclass".
.. autoclass:: CallbackKeywordArgumentsContract
System Message: ERROR/3 (<stdin>, line 37)
Unknown directive type "autoclass".
.. autoclass:: MetadataContract
System Message: ERROR/3 (<stdin>, line 39)
Unknown directive type "autoclass".
.. autoclass:: ReturnsContract
System Message: ERROR/3 (<stdin>, line 41)
Unknown directive type "autoclass".
.. autoclass:: ScrapesContract
Use the :command:`check` command to run the contract checks.
System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "command".Custom Contracts
If you find you need more power than the built-in Scrapy contracts you can create and load your own contracts in the project by using the :setting:`SPIDER_CONTRACTS` setting:
System Message: ERROR/3 (<stdin>, line 48); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 52)
Cannot analyze code. Pygments package not found.
.. code-block:: python
SPIDER_CONTRACTS = {
"myproject.contracts.ResponseCheck": 10,
"myproject.contracts.ItemValidate": 10,
}
Each contract must inherit from :class:`~scrapy.contracts.Contract` and can override three methods:
System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 62)
Unknown directive type "module".
.. module:: scrapy.contracts
System Message: ERROR/3 (<stdin>, line 64)
Unknown directive type "autoclass".
.. autoclass:: Contract
.. automethod:: adjust_request_args
.. method:: pre_process(response)
This allows hooking in various checks on the response received from the
sample request, before it's being passed to the callback.
.. method:: post_process(output)
This allows processing the output of the callback. Iterators are
converted to lists before being passed to this hook.
Raise :class:`~scrapy.exceptions.ContractFail` from :class:`~scrapy.contracts.Contract.pre_process` or :class:`~scrapy.contracts.Contract.post_process` if expectations are not met:
System Message: ERROR/3 (<stdin>, line 78); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 78); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 78); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 82)
Unknown directive type "autoclass".
.. autoclass:: scrapy.exceptions.ContractFail
Here is a demo contract which checks the presence of a custom header in the response received:
System Message: WARNING/2 (<stdin>, line 88)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.contracts import Contract
from scrapy.exceptions import ContractFail
class HasHeaderContract(Contract):
"""
Demo contract which checks the presence of a custom header
@has_header X-CustomHeader
"""
name = "has_header"
def pre_process(self, response):
for header in self.args:
if header not in response.headers:
raise ContractFail("X-CustomHeader not present")
Detecting check runs
When scrapy check is running, the SCRAPY_CHECK environment variable is set to the true string. You can use :data:`os.environ` to perform any change to your spiders or your settings when scrapy check is used:
System Message: ERROR/3 (<stdin>, line 112); backlink
Unknown interpreted text role "data".System Message: WARNING/2 (<stdin>, line 116)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import os
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
def __init__(self):
if os.environ.get("SCRAPY_CHECK"):
pass # Do some scraper adjustments when a check is running