mirror of https://github.com/scrapy/scrapy.git
128 lines
3.5 KiB
ReStructuredText
128 lines
3.5 KiB
ReStructuredText
.. _topics-contracts:
|
|
|
|
=================
|
|
Spiders Contracts
|
|
=================
|
|
|
|
Testing spiders can get particularly annoying and while nothing prevents you
|
|
from writing unit tests the task gets cumbersome quickly. Scrapy offers an
|
|
integrated way of testing your spiders by the means of contracts.
|
|
|
|
This allows you to test each callback of your spider by hardcoding a sample url
|
|
and check various constraints for how the callback processes the response. Each
|
|
contract is prefixed with an ``@`` and included in the docstring. See the
|
|
following example:
|
|
|
|
.. code-block:: python
|
|
|
|
def parse(self, response):
|
|
"""
|
|
This function parses a sample response. Some contracts are mingled
|
|
with this docstring.
|
|
|
|
@url http://www.example.com/s?field-keywords=selfish+gene
|
|
@returns items 1 16
|
|
@returns requests 0 0
|
|
@scrapes Title Author Year Price
|
|
"""
|
|
|
|
You can use the following contracts:
|
|
|
|
.. module:: scrapy.contracts.default
|
|
|
|
.. autoclass:: UrlContract
|
|
|
|
.. autoclass:: CallbackKeywordArgumentsContract
|
|
|
|
.. autoclass:: MetadataContract
|
|
|
|
.. autoclass:: ReturnsContract
|
|
|
|
.. autoclass:: ScrapesContract
|
|
|
|
Use the :command:`check` command to run the contract checks.
|
|
|
|
Custom Contracts
|
|
================
|
|
|
|
If you find you need more power than the built-in Scrapy contracts you can
|
|
create and load your own contracts in the project by using the
|
|
:setting:`SPIDER_CONTRACTS` setting:
|
|
|
|
.. code-block:: python
|
|
|
|
SPIDER_CONTRACTS = {
|
|
"myproject.contracts.ResponseCheck": 10,
|
|
"myproject.contracts.ItemValidate": 10,
|
|
}
|
|
|
|
Each contract must inherit from :class:`~scrapy.contracts.Contract` and can
|
|
override three methods:
|
|
|
|
.. module:: scrapy.contracts
|
|
|
|
.. autoclass:: Contract
|
|
|
|
.. automethod:: adjust_request_args
|
|
|
|
.. method:: pre_process(response)
|
|
|
|
This allows hooking in various checks on the response received from the
|
|
sample request, before it's being passed to the callback.
|
|
|
|
.. method:: post_process(output)
|
|
|
|
This allows processing the output of the callback. Iterators are
|
|
converted to lists before being passed to this hook.
|
|
|
|
Raise :class:`~scrapy.exceptions.ContractFail` from
|
|
:class:`~scrapy.contracts.Contract.pre_process` or
|
|
:class:`~scrapy.contracts.Contract.post_process` if expectations are not met:
|
|
|
|
.. autoclass:: scrapy.exceptions.ContractFail
|
|
|
|
Here is a demo contract which checks the presence of a custom header in the
|
|
response received:
|
|
|
|
.. skip: next
|
|
.. code-block:: python
|
|
|
|
from scrapy.contracts import Contract
|
|
from scrapy.exceptions import ContractFail
|
|
|
|
|
|
class HasHeaderContract(Contract):
|
|
"""
|
|
Demo contract which checks the presence of a custom header
|
|
@has_header X-CustomHeader
|
|
"""
|
|
|
|
name = "has_header"
|
|
|
|
def pre_process(self, response):
|
|
for header in self.args:
|
|
if header not in response.headers:
|
|
raise ContractFail("X-CustomHeader not present")
|
|
|
|
.. _detecting-contract-check-runs:
|
|
|
|
Detecting check runs
|
|
====================
|
|
|
|
When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is
|
|
set to the ``true`` string. You can use :data:`os.environ` to perform any change to
|
|
your spiders or your settings when ``scrapy check`` is used:
|
|
|
|
.. code-block:: python
|
|
|
|
import os
|
|
import scrapy
|
|
|
|
|
|
class ExampleSpider(scrapy.Spider):
|
|
name = "example"
|
|
|
|
def __init__(self):
|
|
if os.environ.get("SCRAPY_CHECK"):
|
|
pass # Do some scraper adjustments when a check is running
|