RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add and run pre-commit hook 'blacken-docs' 

Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
This commit is contained in:
pankaj1707k 2023-02-01 16:30:57 +05:30
parent b337c986ca
commit c1bbb299d7
No known key found for this signature in database
GPG Key ID: 6757E896F6BC635E
50 changed files with 1821 additions and 1260 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.pre-commit-config.yaml
View File

@ -16,3 +16,9 @@ repos:
rev: 5.12.0 rev: 5.12.0
hooks: hooks:
- id: isort - id: isort
- repo: https://github.com/adamchainz/blacken-docs
rev: 1.13.0
hooks:
- id: blacken-docs
additional_dependencies:
- black==22.12.0

44
docs/faq.rst
View File

@ -35,8 +35,9 @@ for parsing HTML responses in Scrapy callbacks.
You just have to feed the response's body into a ``BeautifulSoup`` object You just have to feed the response's body into a ``BeautifulSoup`` object
and extract whatever data you need from it. and extract whatever data you need from it.
Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser:: Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser:
.. code-block:: python
from bs4 import BeautifulSoup from bs4 import BeautifulSoup
import scrapy import scrapy
@ -45,17 +46,12 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML pars
class ExampleSpider(scrapy.Spider): class ExampleSpider(scrapy.Spider):
name = "example" name = "example"
allowed_domains = ["example.com"] allowed_domains = ["example.com"]
start_urls = ( start_urls = ("http://www.example.com/",)
'http://www.example.com/',
)
def parse(self, response): def parse(self, response):
# use lxml to get decent HTML parsing speed # use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, 'lxml') soup = BeautifulSoup(response.text, "lxml")
yield { yield {"url": response.url, "title": soup.h1.string}
"url": response.url,
"title": soup.h1.string
}
.. note:: .. note::
@ -109,11 +105,13 @@ basically means that it crawls in `DFO order`_. This order is more convenient
in most cases. in most cases.
If you do want to crawl in true `BFO order`_, you can do it by If you do want to crawl in true `BFO order`_, you can do it by
setting the following settings:: setting the following settings:
.. code-block:: python
DEPTH_PRIORITY = 1 DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue' SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"
While pending requests are below the configured values of While pending requests are below the configured values of
:setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or
@ -159,11 +157,13 @@ See also other suggestions at `StackOverflow`_.
.. note:: Remember to disable .. note:: Remember to disable
:class:`scrapy.spidermiddlewares.offsite.OffsiteMiddleware` when you enable :class:`scrapy.spidermiddlewares.offsite.OffsiteMiddleware` when you enable
your custom implementation:: your custom implementation:
.. code-block:: python
SPIDER_MIDDLEWARES = { SPIDER_MIDDLEWARES = {
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None, "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
'myproject.middlewares.CustomOffsiteMiddleware': 500, "myproject.middlewares.CustomOffsiteMiddleware": 500,
} }
.. _meet the installation requirements: https://github.com/andreasvc/pyre2#installation .. _meet the installation requirements: https://github.com/andreasvc/pyre2#installation
@ -235,11 +235,13 @@ What does the response status code 999 means?
999 is a custom response status code used by Yahoo sites to throttle requests. 999 is a custom response status code used by Yahoo sites to throttle requests.
Try slowing down the crawling speed by using a download delay of ``2`` (or Try slowing down the crawling speed by using a download delay of ``2`` (or
higher) in your spider:: higher) in your spider:
.. code-block:: python
class MySpider(CrawlSpider): class MySpider(CrawlSpider):
name = 'myspider' name = "myspider"
download_delay = 2 download_delay = 2
@ -351,19 +353,21 @@ How to split an item into multiple items in an item pipeline?
input item. :ref:`Create a spider middleware <custom-spider-middleware>` input item. :ref:`Create a spider middleware <custom-spider-middleware>`
instead, and use its instead, and use its
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method for this purpose. For example:: method for this purpose. For example:
.. code-block:: python
from copy import deepcopy from copy import deepcopy
from itemadapter import is_item, ItemAdapter from itemadapter import is_item, ItemAdapter
class MultiplyItemsMiddleware:
class MultiplyItemsMiddleware:
def process_spider_output(self, response, result, spider): def process_spider_output(self, response, result, spider):
for item in result: for item in result:
if is_item(item): if is_item(item):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
for _ in range(adapter['multiply_by']): for _ in range(adapter["multiply_by"]):
yield deepcopy(item) yield deepcopy(item)
Does Scrapy support IPv6 addresses? Does Scrapy support IPv6 addresses?

14
docs/intro/overview.rst
View File

@ -20,22 +20,24 @@ In order to show you what Scrapy brings to the table, we'll walk you through an
example of a Scrapy Spider using the simplest way to run a spider. example of a Scrapy Spider using the simplest way to run a spider.
Here's the code for a spider that scrapes famous quotes from website Here's the code for a spider that scrapes famous quotes from website
https://quotes.toscrape.com, following the pagination:: https://quotes.toscrape.com, following the pagination
.. code-block:: python
import scrapy import scrapy
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = 'quotes' name = "quotes"
start_urls = [ start_urls = [
'https://quotes.toscrape.com/tag/humor/', "https://quotes.toscrape.com/tag/humor/",
] ]
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'author': quote.xpath('span/small/text()').get(), "author": quote.xpath("span/small/text()").get(),
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
} }
next_page = response.css('li.next a::attr("href")').get() next_page = response.css('li.next a::attr("href")').get()

120
docs/intro/tutorial.rst
View File

@ -177,7 +177,9 @@ that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,
you can just define a :attr:`~scrapy.Spider.start_urls` class attribute you can just define a :attr:`~scrapy.Spider.start_urls` class attribute
with a list of URLs. This list will then be used by the default implementation with a list of URLs. This list will then be used by the default implementation
of :meth:`~scrapy.Spider.start_requests` to create the initial requests of :meth:`~scrapy.Spider.start_requests` to create the initial requests
for your spider:: for your spider.
.. code-block:: python
from pathlib import Path from pathlib import Path
@ -187,13 +189,13 @@ for your spider::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'https://quotes.toscrape.com/page/1/', "https://quotes.toscrape.com/page/1/",
'https://quotes.toscrape.com/page/2/', "https://quotes.toscrape.com/page/2/",
] ]
def parse(self, response): def parse(self, response):
page = response.url.split("/")[-2] page = response.url.split("/")[-2]
filename = f'quotes-{page}.html' filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body) Path(filename).write_bytes(response.body)
The :meth:`~scrapy.Spider.parse` method will be called to handle each The :meth:`~scrapy.Spider.parse` method will be called to handle each
@ -438,7 +440,9 @@ extraction logic above into our spider.
A Scrapy spider typically generates many dictionaries containing the data A Scrapy spider typically generates many dictionaries containing the data
extracted from the page. To do that, we use the ``yield`` Python keyword extracted from the page. To do that, we use the ``yield`` Python keyword
in the callback, as you can see below:: in the callback, as you can see below:
.. code-block:: python
import scrapy import scrapy
@ -446,16 +450,16 @@ in the callback, as you can see below::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'https://quotes.toscrape.com/page/1/', "https://quotes.toscrape.com/page/1/",
'https://quotes.toscrape.com/page/2/', "https://quotes.toscrape.com/page/2/",
] ]
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
'author': quote.css('small.author::text').get(), "author": quote.css("small.author::text").get(),
'tags': quote.css('div.tags a.tag::text').getall(), "tags": quote.css("div.tags a.tag::text").getall(),
} }
If you run this spider, it will output the extracted data with the log:: If you run this spider, it will output the extracted data with the log::
@ -543,7 +547,9 @@ There is also an ``attrib`` property available
'/page/2/' '/page/2/'
Let's see now our spider modified to recursively follow the link to the next Let's see now our spider modified to recursively follow the link to the next
page, extracting data from it:: page, extracting data from it:
.. code-block:: python
import scrapy import scrapy
@ -551,18 +557,18 @@ page, extracting data from it::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'https://quotes.toscrape.com/page/1/', "https://quotes.toscrape.com/page/1/",
] ]
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
'author': quote.css('small.author::text').get(), "author": quote.css("small.author::text").get(),
'tags': quote.css('div.tags a.tag::text').getall(), "tags": quote.css("div.tags a.tag::text").getall(),
} }
next_page = response.css('li.next a::attr(href)').get() next_page = response.css("li.next a::attr(href)").get()
if next_page is not None: if next_page is not None:
next_page = response.urljoin(next_page) next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse) yield scrapy.Request(next_page, callback=self.parse)
@ -594,7 +600,9 @@ A shortcut for creating Requests
-------------------------------- --------------------------------
As a shortcut for creating Request objects you can use As a shortcut for creating Request objects you can use
:meth:`response.follow <scrapy.http.TextResponse.follow>`:: :meth:`response.follow <scrapy.http.TextResponse.follow>`
.. code-block:: python
import scrapy import scrapy
@ -602,18 +610,18 @@ As a shortcut for creating Request objects you can use
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'https://quotes.toscrape.com/page/1/', "https://quotes.toscrape.com/page/1/",
] ]
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
'author': quote.css('span small::text').get(), "author": quote.css("span small::text").get(),
'tags': quote.css('div.tags a.tag::text').getall(), "tags": quote.css("div.tags a.tag::text").getall(),
} }
next_page = response.css('li.next a::attr(href)').get() next_page = response.css("li.next a::attr(href)").get()
if next_page is not None: if next_page is not None:
yield response.follow(next_page, callback=self.parse) yield response.follow(next_page, callback=self.parse)
@ -622,57 +630,67 @@ need to call urljoin. Note that ``response.follow`` just returns a Request
instance; you still have to yield this Request. instance; you still have to yield this Request.
You can also pass a selector to ``response.follow`` instead of a string; You can also pass a selector to ``response.follow`` instead of a string;
this selector should extract necessary attributes:: this selector should extract necessary attributes:
for href in response.css('ul.pager a::attr(href)'): .. code-block:: python
for href in response.css("ul.pager a::attr(href)"):
yield response.follow(href, callback=self.parse) yield response.follow(href, callback=self.parse)
For ``<a>`` elements there is a shortcut: ``response.follow`` uses their href For ``<a>`` elements there is a shortcut: ``response.follow`` uses their href
attribute automatically. So the code can be shortened further:: attribute automatically. So the code can be shortened further:
for a in response.css('ul.pager a'): .. code-block:: python
for a in response.css("ul.pager a"):
yield response.follow(a, callback=self.parse) yield response.follow(a, callback=self.parse)
To create multiple requests from an iterable, you can use To create multiple requests from an iterable, you can use
:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead:: :meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead:
anchors = response.css('ul.pager a') .. code-block:: python
anchors = response.css("ul.pager a")
yield from response.follow_all(anchors, callback=self.parse) yield from response.follow_all(anchors, callback=self.parse)
or, shortening it further:: or, shortening it further:
yield from response.follow_all(css='ul.pager a', callback=self.parse) .. code-block:: python
yield from response.follow_all(css="ul.pager a", callback=self.parse)
More examples and patterns More examples and patterns
-------------------------- --------------------------
Here is another spider that illustrates callbacks and following links, Here is another spider that illustrates callbacks and following links,
this time for scraping author information:: this time for scraping author information:
.. code-block:: python
import scrapy import scrapy
class AuthorSpider(scrapy.Spider): class AuthorSpider(scrapy.Spider):
name = 'author' name = "author"
start_urls = ['https://quotes.toscrape.com/'] start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response): def parse(self, response):
author_page_links = response.css('.author + a') author_page_links = response.css(".author + a")
yield from response.follow_all(author_page_links, self.parse_author) yield from response.follow_all(author_page_links, self.parse_author)
pagination_links = response.css('li.next a') pagination_links = response.css("li.next a")
yield from response.follow_all(pagination_links, self.parse) yield from response.follow_all(pagination_links, self.parse)
def parse_author(self, response): def parse_author(self, response):
def extract_with_css(query): def extract_with_css(query):
return response.css(query).get(default='').strip() return response.css(query).get(default="").strip()
yield { yield {
'name': extract_with_css('h3.author-title::text'), "name": extract_with_css("h3.author-title::text"),
'birthdate': extract_with_css('.author-born-date::text'), "birthdate": extract_with_css(".author-born-date::text"),
'bio': extract_with_css('.author-description::text'), "bio": extract_with_css(".author-description::text"),
} }
This spider will start from the main page, it will follow all the links to the This spider will start from the main page, it will follow all the links to the
@ -720,7 +738,9 @@ spider attributes by default.
In this example, the value provided for the ``tag`` argument will be available In this example, the value provided for the ``tag`` argument will be available
via ``self.tag``. You can use this to make your spider fetch only quotes via ``self.tag``. You can use this to make your spider fetch only quotes
with a specific tag, building the URL based on the argument:: with a specific tag, building the URL based on the argument:
.. code-block:: python
import scrapy import scrapy
@ -729,20 +749,20 @@ with a specific tag, building the URL based on the argument::
name = "quotes" name = "quotes"
def start_requests(self): def start_requests(self):
url = 'https://quotes.toscrape.com/' url = "https://quotes.toscrape.com/"
tag = getattr(self, 'tag', None) tag = getattr(self, "tag", None)
if tag is not None: if tag is not None:
url = url + 'tag/' + tag url = url + "tag/" + tag
yield scrapy.Request(url, self.parse) yield scrapy.Request(url, self.parse)
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
'author': quote.css('small.author::text').get(), "author": quote.css("small.author::text").get(),
} }
next_page = response.css('li.next a::attr(href)').get() next_page = response.css("li.next a::attr(href)").get()
if next_page is not None: if next_page is not None:
yield response.follow(next_page, self.parse) yield response.follow(next_page, self.parse)

6
docs/topics/asyncio.rst
View File

@ -106,12 +106,14 @@ Enforcing asyncio as a requirement
If you are writing a :ref:`component <topics-components>` that requires asyncio If you are writing a :ref:`component <topics-components>` that requires asyncio
to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to
:ref:`enforce it as a requirement <enforce-component-requirements>`. For :ref:`enforce it as a requirement <enforce-component-requirements>`. For
example:: example:
.. code-block:: python
from scrapy.utils.reactor import is_asyncio_reactor_installed from scrapy.utils.reactor import is_asyncio_reactor_installed
class MyComponent:
class MyComponent:
def __init__(self): def __init__(self):
if not is_asyncio_reactor_installed(): if not is_asyncio_reactor_installed():
raise ValueError( raise ValueError(

40
docs/topics/broad-crawls.rst
View File

@ -48,9 +48,11 @@ Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQ
It works best during single-domain crawl. It does not work well with crawling It works best during single-domain crawl. It does not work well with crawling
many different domains in parallel many different domains in parallel
To apply the recommended priority queue use:: To apply the recommended priority queue use:
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' .. code-block:: python
SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue"
.. _broad-crawls-concurrency: .. _broad-crawls-concurrency:
@ -71,7 +73,9 @@ many different domains in parallel, so you will want to increase it. How much
to increase it will depend on how much CPU and memory your crawler will have to increase it will depend on how much CPU and memory your crawler will have
available. available.
A good starting point is ``100``:: A good starting point is ``100``:
.. code-block:: python
CONCURRENT_REQUESTS = 100 CONCURRENT_REQUESTS = 100
@ -92,7 +96,9 @@ hitting DNS resolver timeouts. Possible solution to increase the number of
threads handling DNS queries. The DNS queue will be processed faster speeding threads handling DNS queries. The DNS queue will be processed faster speeding
up establishing of connection and crawling overall. up establishing of connection and crawling overall.
To increase maximum thread pool size use:: To increase maximum thread pool size use:
.. code-block:: python
REACTOR_THREADPOOL_MAXSIZE = 20 REACTOR_THREADPOOL_MAXSIZE = 20
@ -114,9 +120,11 @@ should not use ``DEBUG`` log level when preforming large broad crawls in
production. Using ``DEBUG`` level when developing your (broad) crawler may be production. Using ``DEBUG`` level when developing your (broad) crawler may be
fine though. fine though.
To set the log level use:: To set the log level use:
LOG_LEVEL = 'INFO' .. code-block:: python
LOG_LEVEL = "INFO"
Disable cookies Disable cookies
=============== ===============
@ -126,7 +134,9 @@ doing broad crawls (search engine crawlers ignore them), and they improve
performance by saving some CPU cycles and reducing the memory footprint of your performance by saving some CPU cycles and reducing the memory footprint of your
Scrapy crawler. Scrapy crawler.
To disable cookies use:: To disable cookies use:
.. code-block:: python
COOKIES_ENABLED = False COOKIES_ENABLED = False
@ -138,7 +148,9 @@ when sites causes are very slow (or fail) to respond, thus causing a timeout
error which gets retried many times, unnecessarily, preventing crawler capacity error which gets retried many times, unnecessarily, preventing crawler capacity
to be reused for other domains. to be reused for other domains.
To disable retries use:: To disable retries use:
.. code-block:: python
RETRY_ENABLED = False RETRY_ENABLED = False
@ -149,7 +161,9 @@ Unless you are crawling from a very slow connection (which shouldn't be the
case for broad crawls) reduce the download timeout so that stuck requests are case for broad crawls) reduce the download timeout so that stuck requests are
discarded quickly and free up capacity to process the next ones. discarded quickly and free up capacity to process the next ones.
To reduce the download timeout use:: To reduce the download timeout use:
.. code-block:: python
DOWNLOAD_TIMEOUT = 15 DOWNLOAD_TIMEOUT = 15
@ -162,7 +176,9 @@ revisiting the site at a later crawl. This also help to keep the number of
request constant per crawl batch, otherwise redirect loops may cause the request constant per crawl batch, otherwise redirect loops may cause the
crawler to dedicate too many resources on any specific domain. crawler to dedicate too many resources on any specific domain.
To disable redirects use:: To disable redirects use:
.. code-block:: python
REDIRECT_ENABLED = False REDIRECT_ENABLED = False
@ -179,7 +195,9 @@ Pages can indicate it in two ways:
"main", "index" website pages. "main", "index" website pages.
Scrapy handles (1) automatically; to handle (2) enable Scrapy handles (1) automatically; to handle (2) enable
:ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`:: :ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`:
.. code-block:: python
AJAXCRAWL_ENABLED = True AJAXCRAWL_ENABLED = True

17
docs/topics/commands.rst
View File

@ -617,7 +617,7 @@ Example:
.. code-block:: python .. code-block:: python
COMMANDS_MODULE = 'mybot.commands' COMMANDS_MODULE = "mybot.commands"
.. _Deploying your project: https://scrapyd.readthedocs.io/en/latest/deploy.html .. _Deploying your project: https://scrapyd.readthedocs.io/en/latest/deploy.html
@ -636,10 +636,11 @@ The following example adds ``my_command`` command:
from setuptools import setup, find_packages from setuptools import setup, find_packages
setup(name='scrapy-mymodule', setup(
entry_points={ name="scrapy-mymodule",
'scrapy.commands': [ entry_points={
'my_command=my_scrapy_module.commands:MyCommand', "scrapy.commands": [
], "my_command=my_scrapy_module.commands:MyCommand",
}, ],
) },
)

8
docs/topics/components.rst
View File

@ -66,16 +66,18 @@ version mismatch, while :exc:`ValueError` may be better if the issue is the
value of a setting. value of a setting.
If your requirement is a minimum Scrapy version, you may use If your requirement is a minimum Scrapy version, you may use
:attr:`scrapy.__version__` to enforce your requirement. For example:: :attr:`scrapy.__version__` to enforce your requirement. For example:
.. code-block:: python
from pkg_resources import parse_version from pkg_resources import parse_version
import scrapy import scrapy
class MyComponent:
class MyComponent:
def __init__(self): def __init__(self):
if parse_version(scrapy.__version__) < parse_version('2.7'): if parse_version(scrapy.__version__) < parse_version("2.7"):
raise RuntimeError( raise RuntimeError(
f"{MyComponent.__qualname__} requires Scrapy 2.7 or " f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
f"later, which allow defining the process_spider_output " f"later, which allow defining the process_spider_output "

38
docs/topics/contracts.rst
View File

@ -11,10 +11,13 @@ integrated way of testing your spiders by the means of contracts.
This allows you to test each callback of your spider by hardcoding a sample url This allows you to test each callback of your spider by hardcoding a sample url
and check various constraints for how the callback processes the response. Each and check various constraints for how the callback processes the response. Each
contract is prefixed with an ``@`` and included in the docstring. See the contract is prefixed with an ``@`` and included in the docstring. See the
following example:: following example:
.. code-block:: python
def parse(self, response): def parse(self, response):
""" This function parses a sample response. Some contracts are mingled """
This function parses a sample response. Some contracts are mingled
with this docstring. with this docstring.
@url http://www.amazon.com/s?field-keywords=selfish+gene @url http://www.amazon.com/s?field-keywords=selfish+gene
@ -64,11 +67,13 @@ Custom Contracts
If you find you need more power than the built-in Scrapy contracts you can If you find you need more power than the built-in Scrapy contracts you can
create and load your own contracts in the project by using the create and load your own contracts in the project by using the
:setting:`SPIDER_CONTRACTS` setting:: :setting:`SPIDER_CONTRACTS` setting:
.. code-block:: python
SPIDER_CONTRACTS = { SPIDER_CONTRACTS = {
'myproject.contracts.ResponseCheck': 10, "myproject.contracts.ResponseCheck": 10,
'myproject.contracts.ItemValidate': 10, "myproject.contracts.ItemValidate": 10,
} }
Each contract must inherit from :class:`~scrapy.contracts.Contract` and can Each contract must inherit from :class:`~scrapy.contracts.Contract` and can
@ -111,22 +116,26 @@ Raise :class:`~scrapy.exceptions.ContractFail` from
.. autoclass:: scrapy.exceptions.ContractFail .. autoclass:: scrapy.exceptions.ContractFail
Here is a demo contract which checks the presence of a custom header in the Here is a demo contract which checks the presence of a custom header in the
response received:: response received:
.. code-block:: python
from scrapy.contracts import Contract from scrapy.contracts import Contract
from scrapy.exceptions import ContractFail from scrapy.exceptions import ContractFail
class HasHeaderContract(Contract): class HasHeaderContract(Contract):
""" Demo contract which checks the presence of a custom header """
@has_header X-CustomHeader Demo contract which checks the presence of a custom header
@has_header X-CustomHeader
""" """
name = 'has_header' name = "has_header"
def pre_process(self, response): def pre_process(self, response):
for header in self.args: for header in self.args:
if header not in response.headers: if header not in response.headers:
raise ContractFail('X-CustomHeader not present') raise ContractFail("X-CustomHeader not present")
.. _detecting-contract-check-runs: .. _detecting-contract-check-runs:
@ -135,14 +144,17 @@ Detecting check runs
When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is
set to the ``true`` string. You can use :data:`os.environ` to perform any change to set to the ``true`` string. You can use :data:`os.environ` to perform any change to
your spiders or your settings when ``scrapy check`` is used:: your spiders or your settings when ``scrapy check`` is used:
.. code-block:: python
import os import os
import scrapy import scrapy
class ExampleSpider(scrapy.Spider): class ExampleSpider(scrapy.Spider):
name = 'example' name = "example"
def __init__(self): def __init__(self):
if os.environ.get('SCRAPY_CHECK'): if os.environ.get("SCRAPY_CHECK"):
pass # Do some scraper adjustments when a check is running pass # Do some scraper adjustments when a check is running

29
docs/topics/coroutines.rst
View File

@ -58,49 +58,58 @@ There are several use cases for coroutines in Scrapy.
Code that would return Deferreds when written for previous Scrapy versions, Code that would return Deferreds when written for previous Scrapy versions,
such as downloader middlewares and signal handlers, can be rewritten to be such as downloader middlewares and signal handlers, can be rewritten to be
shorter and cleaner:: shorter and cleaner:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
class DbPipeline: class DbPipeline:
def _update_item(self, data, item): def _update_item(self, data, item):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
adapter['field'] = data adapter["field"] = data
return item return item
def process_item(self, item, spider): def process_item(self, item, spider):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
dfd = db.get_some_data(adapter['id']) dfd = db.get_some_data(adapter["id"])
dfd.addCallback(self._update_item, item) dfd.addCallback(self._update_item, item)
return dfd return dfd
becomes:: becomes:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
class DbPipeline: class DbPipeline:
async def process_item(self, item, spider): async def process_item(self, item, spider):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
adapter['field'] = await db.get_some_data(adapter['id']) adapter["field"] = await db.get_some_data(adapter["id"])
return item return item
Coroutines may be used to call asynchronous code. This includes other Coroutines may be used to call asynchronous code. This includes other
coroutines, functions that return Deferreds and functions that return coroutines, functions that return Deferreds and functions that return
:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.
This means you can use many useful Python libraries providing such code:: This means you can use many useful Python libraries providing such code:
.. code-block:: python
class MySpiderDeferred(Spider): class MySpiderDeferred(Spider):
# ... # ...
async def parse(self, response): async def parse(self, response):
additional_response = await treq.get('https://additional.url') additional_response = await treq.get("https://additional.url")
additional_data = await treq.content(additional_response) additional_data = await treq.content(additional_response)
# ... use response and additional_data to yield items and requests # ... use response and additional_data to yield items and requests
class MySpiderAsyncio(Spider): class MySpiderAsyncio(Spider):
# ... # ...
async def parse(self, response): async def parse(self, response):
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response: async with session.get("https://additional.url") as additional_response:
additional_data = await additional_response.text() additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests # ... use response and additional_data to yield items and requests
@ -192,7 +201,9 @@ while maintaining support for older Scrapy versions, you may define
:term:`asynchronous generator` version of that method with an alternative name: :term:`asynchronous generator` version of that method with an alternative name:
``process_spider_output_async``. ``process_spider_output_async``.
For example:: For example:
.. code-block:: python
class UniversalSpiderMiddleware: class UniversalSpiderMiddleware:
def process_spider_output(self, response, result, spider): def process_spider_output(self, response, result, spider):

35
docs/topics/debug.rst
View File

@ -5,21 +5,24 @@ Debugging Spiders
================= =================
This document explains the most common techniques for debugging spiders. This document explains the most common techniques for debugging spiders.
Consider the following Scrapy spider below:: Consider the following Scrapy spider below:
.. code-block:: python
import scrapy import scrapy
from myproject.items import MyItem from myproject.items import MyItem
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
start_urls = ( start_urls = (
'http://example.com/page1', "http://example.com/page1",
'http://example.com/page2', "http://example.com/page2",
) )
def parse(self, response): def parse(self, response):
# <processing code not shown> # <processing code not shown>
# collect `item_urls` # collect `item_urls`
for item_url in item_urls: for item_url in item_urls:
yield scrapy.Request(item_url, self.parse_item) yield scrapy.Request(item_url, self.parse_item)
@ -28,7 +31,9 @@ Consider the following Scrapy spider below::
item = MyItem() item = MyItem()
# populate `item` fields # populate `item` fields
# and extract item_details_url # and extract item_details_url
yield scrapy.Request(item_details_url, self.parse_details, cb_kwargs={'item': item}) yield scrapy.Request(
item_details_url, self.parse_details, cb_kwargs={"item": item}
)
def parse_details(self, response, item): def parse_details(self, response, item):
# populate more `item` fields # populate more `item` fields
@ -103,10 +108,13 @@ showing the response received and the output. How to debug the situation when
.. highlight:: python .. highlight:: python
Fortunately, the :command:`shell` is your bread and butter in this case (see Fortunately, the :command:`shell` is your bread and butter in this case (see
:ref:`topics-shell-inspect-response`):: :ref:`topics-shell-inspect-response`):
.. code-block:: python
from scrapy.shell import inspect_response from scrapy.shell import inspect_response
def parse_details(self, response, item=None): def parse_details(self, response, item=None):
if item: if item:
# populate more `item` fields # populate more `item` fields
@ -121,10 +129,13 @@ Open in browser
Sometimes you just want to see how a certain response looks in a browser, you Sometimes you just want to see how a certain response looks in a browser, you
can use the ``open_in_browser`` function for that. Here is an example of how can use the ``open_in_browser`` function for that. Here is an example of how
you would use it:: you would use it:
.. code-block:: python
from scrapy.utils.response import open_in_browser from scrapy.utils.response import open_in_browser
def parse_details(self, response): def parse_details(self, response):
if "item name" not in response.body: if "item name" not in response.body:
open_in_browser(response) open_in_browser(response)
@ -138,14 +149,16 @@ Logging
Logging is another useful option for getting information about your spider run. Logging is another useful option for getting information about your spider run.
Although not as convenient, it comes with the advantage that the logs will be Although not as convenient, it comes with the advantage that the logs will be
available in all future runs should they be necessary again:: available in all future runs should they be necessary again:
.. code-block:: python
def parse_details(self, response, item=None): def parse_details(self, response, item=None):
if item: if item:
# populate more `item` fields # populate more `item` fields
return item return item
else: else:
self.logger.warning('No item received for %s', response.url) self.logger.warning("No item received for %s", response.url)
For more information, check the :ref:`topics-logging` section. For more information, check the :ref:`topics-logging` section.

17
docs/topics/developer-tools.rst
View File

@ -237,17 +237,19 @@ on the request and open ``Open in new tab`` to get a better overview.
:alt: JSON-object returned from the quotes.toscrape API :alt: JSON-object returned from the quotes.toscrape API
With this response we can now easily parse the JSON-object and With this response we can now easily parse the JSON-object and
also request each page to get every quote on the site:: also request each page to get every quote on the site:
.. code-block:: python
import scrapy import scrapy
import json import json
class QuoteSpider(scrapy.Spider): class QuoteSpider(scrapy.Spider):
name = 'quote' name = "quote"
allowed_domains = ['quotes.toscrape.com'] allowed_domains = ["quotes.toscrape.com"]
page = 1 page = 1
start_urls = ['https://quotes.toscrape.com/api/quotes?page=1'] start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]
def parse(self, response): def parse(self, response):
data = json.loads(response.text) data = json.loads(response.text)
@ -275,7 +277,9 @@ requests, as we could need to add ``headers`` or ``cookies`` to make it work.
In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_ In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
format, by right-clicking on each of them in the network tool and using the format, by right-clicking on each of them in the network tool and using the
:meth:`~scrapy.Request.from_curl()` method to generate an equivalent :meth:`~scrapy.Request.from_curl()` method to generate an equivalent
request:: request:
.. code-block:: python
from scrapy import Request from scrapy import Request
@ -286,7 +290,8 @@ request::
"-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM" "-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM"
"zEwZTAxLTk5MWUtNDFiNC1iZWRmLTJjNGI4M2ZiNDBmNDpAVEstMDMzMTBlMDEtOTkxZS00MW" "zEwZTAxLTk5MWUtNDFiNC1iZWRmLTJjNGI4M2ZiNDBmNDpAVEstMDMzMTBlMDEtOTkxZS00MW"
"I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http" "I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http"
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'") "://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'"
)
Alternatively, if you want to know the arguments needed to recreate that Alternatively, if you want to know the arguments needed to recreate that
request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs` request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`

50
docs/topics/downloader-middleware.rst
View File

@ -17,10 +17,12 @@ To activate a downloader middleware component, add it to the
:setting:`DOWNLOADER_MIDDLEWARES` setting, which is a dict whose keys are the :setting:`DOWNLOADER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class paths and their values are the middleware orders. middleware class paths and their values are the middleware orders.
Here's an example:: Here's an example:
.. code-block:: python
DOWNLOADER_MIDDLEWARES = { DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543, "myproject.middlewares.CustomDownloaderMiddleware": 543,
} }
The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
@ -42,11 +44,13 @@ previous (or subsequent) middleware being applied.
If you want to disable a built-in middleware (the ones defined in If you want to disable a built-in middleware (the ones defined in
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it :setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None`` in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None``
as its value. For example, if you want to disable the user-agent middleware:: as its value. For example, if you want to disable the user-agent middleware:
.. code-block:: python
DOWNLOADER_MIDDLEWARES = { DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543, "myproject.middlewares.CustomDownloaderMiddleware": 543,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
} }
Finally, keep in mind that some middlewares may need to be enabled through a Finally, keep in mind that some middlewares may need to be enabled through a
@ -226,20 +230,25 @@ There is support for keeping multiple cookie sessions per spider by using the
:reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar :reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar
(session), but you can pass an identifier to use different ones. (session), but you can pass an identifier to use different ones.
For example:: For example:
.. code-block:: python
for i, url in enumerate(urls): for i, url in enumerate(urls):
yield scrapy.Request(url, meta={'cookiejar': i}, yield scrapy.Request(url, meta={"cookiejar": i}, callback=self.parse_page)
callback=self.parse_page)
Keep in mind that the :reqmeta:`cookiejar` meta key is not "sticky". You need to keep Keep in mind that the :reqmeta:`cookiejar` meta key is not "sticky". You need to keep
passing it along on subsequent requests. For example:: passing it along on subsequent requests. For example:
.. code-block:: python
def parse_page(self, response): def parse_page(self, response):
# do some processing # do some processing
return scrapy.Request("http://www.example.com/otherpage", return scrapy.Request(
meta={'cookiejar': response.meta['cookiejar']}, "http://www.example.com/otherpage",
callback=self.parse_other_page) meta={"cookiejar": response.meta["cookiejar"]},
callback=self.parse_other_page,
)
.. setting:: COOKIES_ENABLED .. setting:: COOKIES_ENABLED
@ -339,16 +348,19 @@ HttpAuthMiddleware
domain of the first request, which will work for some spiders but not domain of the first request, which will work for some spiders but not
for others. In the future the middleware will produce an error instead. for others. In the future the middleware will produce an error instead.
Example:: Example:
.. code-block:: python
from scrapy.spiders import CrawlSpider from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider): class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser' http_user = "someuser"
http_pass = 'somepass' http_pass = "somepass"
http_auth_domain = 'intranet.example.com' http_auth_domain = "intranet.example.com"
name = 'intranet.example.com' name = "intranet.example.com"
# .. rest of the spider code omitted ... # .. rest of the spider code omitted ...
@ -792,7 +804,9 @@ If you want to handle some redirect status codes in your spider, you can
specify these in the ``handle_httpstatus_list`` spider attribute. specify these in the ``handle_httpstatus_list`` spider attribute.
For example, if you want the redirect middleware to ignore 301 and 302 For example, if you want the redirect middleware to ignore 301 and 302
responses (and pass them through to your spider) you can do this:: responses (and pass them through to your spider) you can do this:
.. code-block:: python
class MySpider(CrawlSpider): class MySpider(CrawlSpider):
handle_httpstatus_list = [301, 302] handle_httpstatus_list = [301, 302]

15
docs/topics/dynamic-content.rst
View File

@ -119,16 +119,20 @@ data from it depends on the type of response:
<topics-selectors>` as usual. <topics-selectors>` as usual.
- If the response is JSON, use :func:`json.loads` to load the desired data from - If the response is JSON, use :func:`json.loads` to load the desired data from
:attr:`response.text <scrapy.http.TextResponse.text>`:: :attr:`response.text <scrapy.http.TextResponse.text>`:
.. code-block:: python
data = json.loads(response.text) data = json.loads(response.text)
If the desired data is inside HTML or XML code embedded within JSON data, If the desired data is inside HTML or XML code embedded within JSON data,
you can load that HTML or XML code into a you can load that HTML or XML code into a
:class:`~scrapy.Selector` and then :class:`~scrapy.Selector` and then
:ref:`use it <topics-selectors>` as usual:: :ref:`use it <topics-selectors>` as usual:
selector = Selector(data['html']) .. code-block:: python
selector = Selector(data["html"])
- If the response is JavaScript, or HTML with a ``<script/>`` element - If the response is JavaScript, or HTML with a ``<script/>`` element
containing the desired data, see :ref:`topics-parsing-javascript`. containing the desired data, see :ref:`topics-parsing-javascript`.
@ -250,11 +254,14 @@ automation. By installing the :ref:`asyncio reactor <install-asyncio>`,
it is possible to integrate ``asyncio``-based libraries which handle headless browsers. it is possible to integrate ``asyncio``-based libraries which handle headless browsers.
One such library is `playwright-python`_ (an official Python port of `playwright`_). One such library is `playwright-python`_ (an official Python port of `playwright`_).
The following is a simple snippet to illustrate its usage within a Scrapy spider:: The following is a simple snippet to illustrate its usage within a Scrapy spider:
.. code-block:: python
import scrapy import scrapy
from playwright.async_api import async_playwright from playwright.async_api import async_playwright
class PlaywrightSpider(scrapy.Spider): class PlaywrightSpider(scrapy.Spider):
name = "playwright" name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader start_urls = ["data:,"] # avoid using the default Scrapy downloader

20
docs/topics/email.rst
View File

@ -19,19 +19,31 @@ Quick example
============= =============
There are two ways to instantiate the mail sender. You can instantiate it using There are two ways to instantiate the mail sender. You can instantiate it using
the standard ``__init__`` method:: the standard ``__init__`` method:
.. code-block:: python
from scrapy.mail import MailSender from scrapy.mail import MailSender
mailer = MailSender() mailer = MailSender()
Or you can instantiate it passing a Scrapy settings object, which will respect Or you can instantiate it passing a Scrapy settings object, which will respect
the :ref:`settings <topics-email-settings>`:: the :ref:`settings <topics-email-settings>`:
.. code-block:: python
mailer = MailSender.from_settings(settings) mailer = MailSender.from_settings(settings)
And here is how to use it to send an e-mail (without attachments):: And here is how to use it to send an e-mail (without attachments):
mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another@example.com"]) .. code-block:: python
mailer.send(
to=["someone@example.com"],
subject="Some subject",
body="Some body",
cc=["another@example.com"],
)
MailSender class reference MailSender class reference
========================== ==========================

8
docs/topics/exceptions.rst
View File

@ -26,11 +26,13 @@ CloseSpider
:param reason: the reason for closing :param reason: the reason for closing
:type reason: str :type reason: str
For example:: For example:
.. code-block:: python
def parse_page(self, response): def parse_page(self, response):
if 'Bandwidth exceeded' in response.body: if "Bandwidth exceeded" in response.body:
raise CloseSpider('bandwidth_exceeded') raise CloseSpider("bandwidth_exceeded")
DontCloseSpider DontCloseSpider
--------------- ---------------

35
docs/topics/exporters.rst
View File

@ -38,11 +38,14 @@ the end of the exporting process
Here you can see an :doc:`Item Pipeline <item-pipeline>` which uses multiple Here you can see an :doc:`Item Pipeline <item-pipeline>` which uses multiple
Item Exporters to group scraped items to different files according to the Item Exporters to group scraped items to different files according to the
value of one of their fields:: value of one of their fields:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
from scrapy.exporters import XmlItemExporter from scrapy.exporters import XmlItemExporter
class PerYearXmlExportPipeline: class PerYearXmlExportPipeline:
"""Distribute items across multiple XML files according to their 'year' field""" """Distribute items across multiple XML files according to their 'year' field"""
@ -56,9 +59,9 @@ value of one of their fields::
def _exporter_for_item(self, item): def _exporter_for_item(self, item):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
year = adapter['year'] year = adapter["year"]
if year not in self.year_to_exporter: if year not in self.year_to_exporter:
xml_file = open(f'{year}.xml', 'wb') xml_file = open(f"{year}.xml", "wb")
exporter = XmlItemExporter(xml_file) exporter = XmlItemExporter(xml_file)
exporter.start_exporting() exporter.start_exporting()
self.year_to_exporter[year] = (exporter, xml_file) self.year_to_exporter[year] = (exporter, xml_file)
@ -94,12 +97,16 @@ If you use :class:`~scrapy.Item` you can declare a serializer in the
:ref:`field metadata <topics-items-fields>`. The serializer must be :ref:`field metadata <topics-items-fields>`. The serializer must be
a callable which receives a value and returns its serialized form. a callable which receives a value and returns its serialized form.
Example:: Example:
.. code-block:: python
import scrapy import scrapy
def serialize_price(value): def serialize_price(value):
return f'$ {str(value)}' return f"$ {str(value)}"
class Product(scrapy.Item): class Product(scrapy.Item):
name = scrapy.Field() name = scrapy.Field()
@ -115,15 +122,17 @@ customize how your field value will be exported.
Make sure you call the base class :meth:`~BaseItemExporter.serialize_field()` method Make sure you call the base class :meth:`~BaseItemExporter.serialize_field()` method
after your custom code. after your custom code.
Example:: Example:
.. code-block:: python
from scrapy.exporters import XmlItemExporter from scrapy.exporters import XmlItemExporter
class ProductXmlExporter(XmlItemExporter):
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value): def serialize_field(self, field, name, value):
if name == 'price': if name == "price":
return f'$ {str(value)}' return f"$ {str(value)}"
return super().serialize_field(field, name, value) return super().serialize_field(field, name, value)
.. _topics-exporters-reference: .. _topics-exporters-reference:
@ -132,10 +141,12 @@ Built-in Item Exporters reference
================================= =================================
Here is a list of the Item Exporters bundled with Scrapy. Some of them contain Here is a list of the Item Exporters bundled with Scrapy. Some of them contain
output examples, which assume you're exporting these two items:: output examples, which assume you're exporting these two items:
Item(name='Color TV', price='1200') .. code-block:: python
Item(name='DVD player', price='200')
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
BaseItemExporter BaseItemExporter
---------------- ----------------

24
docs/topics/extensions.rst
View File

@ -31,11 +31,13 @@ initialization code must be performed in the class ``__init__`` method.
To make an extension available, add it to the :setting:`EXTENSIONS` setting in To make an extension available, add it to the :setting:`EXTENSIONS` setting in
your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented
by a string: the full Python path to the extension's class name. For example:: by a string: the full Python path to the extension's class name. For example:
.. code-block:: python
EXTENSIONS = { EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': 500, "scrapy.extensions.corestats.CoreStats": 500,
'scrapy.extensions.telnet.TelnetConsole': 500, "scrapy.extensions.telnet.TelnetConsole": 500,
} }
@ -64,10 +66,12 @@ Disabling an extension
In order to disable an extension that comes enabled by default (i.e. those In order to disable an extension that comes enabled by default (i.e. those
included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
``None``. For example:: ``None``. For example:
.. code-block:: python
EXTENSIONS = { EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': None, "scrapy.extensions.corestats.CoreStats": None,
} }
Writing your own extension Writing your own extension
@ -98,7 +102,9 @@ in the previous section. This extension will log a message every time:
The extension will be enabled through the ``MYEXT_ENABLED`` setting and the The extension will be enabled through the ``MYEXT_ENABLED`` setting and the
number of items will be specified through the ``MYEXT_ITEMCOUNT`` setting. number of items will be specified through the ``MYEXT_ITEMCOUNT`` setting.
Here is the code of such extension:: Here is the code of such extension:
.. code-block:: python
import logging import logging
from scrapy import signals from scrapy import signals
@ -106,8 +112,8 @@ Here is the code of such extension::
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class SpiderOpenCloseLogging:
class SpiderOpenCloseLogging:
def __init__(self, item_count): def __init__(self, item_count):
self.item_count = item_count self.item_count = item_count
self.items_scraped = 0 self.items_scraped = 0
@ -116,11 +122,11 @@ Here is the code of such extension::
def from_crawler(cls, crawler): def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise # first check if the extension should be enabled and raise
# NotConfigured otherwise # NotConfigured otherwise
if not crawler.settings.getbool('MYEXT_ENABLED'): if not crawler.settings.getbool("MYEXT_ENABLED"):
raise NotConfigured raise NotConfigured
# get the number of items from settings # get the number of items from settings
item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000) item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)
# instantiate the extension object # instantiate the extension object
ext = cls(item_count) ext = cls(item_count)

67
docs/topics/feed-exports.rst
View File

@ -290,10 +290,11 @@ class, which is the default value of the ``item_filter`` :ref:`feed option <feed
You can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'s You can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'s
method ``accepts`` and taking ``feed_options`` as an argument. method ``accepts`` and taking ``feed_options`` as an argument.
For instance:: For instance:
.. code-block:: python
class MyCustomFilter: class MyCustomFilter:
def __init__(self, feed_options): def __init__(self, feed_options):
self.feed_options = feed_options self.feed_options = feed_options
@ -594,23 +595,27 @@ For a complete list of available values, access the `Canned ACL`_ section on Ama
FEED_STORAGES_BASE FEED_STORAGES_BASE
------------------ ------------------
Default:: Default:
.. code-block:: python
{ {
'': 'scrapy.extensions.feedexport.FileFeedStorage', "": "scrapy.extensions.feedexport.FileFeedStorage",
'file': 'scrapy.extensions.feedexport.FileFeedStorage', "file": "scrapy.extensions.feedexport.FileFeedStorage",
'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage', "stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
's3': 'scrapy.extensions.feedexport.S3FeedStorage', "s3": "scrapy.extensions.feedexport.S3FeedStorage",
'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', "ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
} }
A dict containing the built-in feed storage backends supported by Scrapy. You A dict containing the built-in feed storage backends supported by Scrapy. You
can disable any of these backends by assigning ``None`` to their URI scheme in can disable any of these backends by assigning ``None`` to their URI scheme in
:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend :setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
(without replacement), place this in your ``settings.py``:: (without replacement), place this in your ``settings.py``:
.. code-block:: python
FEED_STORAGES = { FEED_STORAGES = {
'ftp': None, "ftp": None,
} }
.. setting:: FEED_EXPORTERS .. setting:: FEED_EXPORTERS
@ -628,26 +633,30 @@ serialization formats and the values are paths to :ref:`Item exporter
FEED_EXPORTERS_BASE FEED_EXPORTERS_BASE
------------------- -------------------
Default:: Default:
.. code-block:: python
{ {
'json': 'scrapy.exporters.JsonItemExporter', "json": "scrapy.exporters.JsonItemExporter",
'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', "jsonlines": "scrapy.exporters.JsonLinesItemExporter",
'jsonl': 'scrapy.exporters.JsonLinesItemExporter', "jsonl": "scrapy.exporters.JsonLinesItemExporter",
'jl': 'scrapy.exporters.JsonLinesItemExporter', "jl": "scrapy.exporters.JsonLinesItemExporter",
'csv': 'scrapy.exporters.CsvItemExporter', "csv": "scrapy.exporters.CsvItemExporter",
'xml': 'scrapy.exporters.XmlItemExporter', "xml": "scrapy.exporters.XmlItemExporter",
'marshal': 'scrapy.exporters.MarshalItemExporter', "marshal": "scrapy.exporters.MarshalItemExporter",
'pickle': 'scrapy.exporters.PickleItemExporter', "pickle": "scrapy.exporters.PickleItemExporter",
} }
A dict containing the built-in feed exporters supported by Scrapy. You can A dict containing the built-in feed exporters supported by Scrapy. You can
disable any of these exporters by assigning ``None`` to their serialization disable any of these exporters by assigning ``None`` to their serialization
format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
(without replacement), place this in your ``settings.py``:: (without replacement), place this in your ``settings.py``:
.. code-block:: python
FEED_EXPORTERS = { FEED_EXPORTERS = {
'csv': None, "csv": None,
} }
@ -677,7 +686,9 @@ generated:
number by introducing leading zeroes as needed, use ``%(batch_id)05d`` number by introducing leading zeroes as needed, use ``%(batch_id)05d``
(e.g. ``3`` becomes ``00003``, ``123`` becomes ``00123``). (e.g. ``3`` becomes ``00003``, ``123`` becomes ``00123``).
For instance, if your settings include:: For instance, if your settings include:
.. code-block:: python
FEED_EXPORT_BATCH_ITEM_COUNT = 100 FEED_EXPORT_BATCH_ITEM_COUNT = 100
@ -746,16 +757,20 @@ The function signature should be as follows:
For example, to include the :attr:`name <scrapy.Spider.name>` of the For example, to include the :attr:`name <scrapy.Spider.name>` of the
source spider in the feed URI: source spider in the feed URI:
#. Define the following function somewhere in your project:: #. Define the following function somewhere in your project:
.. code-block:: python
# myproject/utils.py # myproject/utils.py
def uri_params(params, spider): def uri_params(params, spider):
return {**params, 'spider_name': spider.name} return {**params, "spider_name": spider.name}
#. Point :setting:`FEED_URI_PARAMS` to that function in your settings:: #. Point :setting:`FEED_URI_PARAMS` to that function in your settings:
.. code-block:: python
# myproject/settings.py # myproject/settings.py
FEED_URI_PARAMS = 'myproject.utils.uri_params' FEED_URI_PARAMS = "myproject.utils.uri_params"
#. Use ``%(spider_name)s`` in your feed URI:: #. Use ``%(spider_name)s`` in your feed URI::

50
docs/topics/item-pipeline.rst
View File

@ -81,19 +81,23 @@ Price validation and dropping items with no prices
Let's take a look at the following hypothetical pipeline that adjusts the Let's take a look at the following hypothetical pipeline that adjusts the
``price`` attribute for those items that do not include VAT ``price`` attribute for those items that do not include VAT
(``price_excludes_vat`` attribute), and drops those items which don't (``price_excludes_vat`` attribute), and drops those items which don't
contain a price:: contain a price:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem from scrapy.exceptions import DropItem
class PricePipeline: class PricePipeline:
vat_factor = 1.15 vat_factor = 1.15
def process_item(self, item, spider): def process_item(self, item, spider):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
if adapter.get('price'): if adapter.get("price"):
if adapter.get('price_excludes_vat'): if adapter.get("price_excludes_vat"):
adapter['price'] = adapter['price'] * self.vat_factor adapter["price"] = adapter["price"] * self.vat_factor
return item return item
else: else:
raise DropItem(f"Missing price in {item}") raise DropItem(f"Missing price in {item}")
@ -104,16 +108,18 @@ Write items to a JSON lines file
The following pipeline stores all scraped items (from all spiders) into a The following pipeline stores all scraped items (from all spiders) into a
single ``items.jsonl`` file, containing one item per line serialized in JSON single ``items.jsonl`` file, containing one item per line serialized in JSON
format:: format:
.. code-block:: python
import json import json
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
class JsonWriterPipeline:
class JsonWriterPipeline:
def open_spider(self, spider): def open_spider(self, spider):
self.file = open('items.jsonl', 'w') self.file = open("items.jsonl", "w")
def close_spider(self, spider): def close_spider(self, spider):
self.file.close() self.file.close()
@ -135,14 +141,17 @@ MongoDB address and database name are specified in Scrapy settings;
MongoDB collection is named after item class. MongoDB collection is named after item class.
The main point of this example is to show how to use :meth:`from_crawler` The main point of this example is to show how to use :meth:`from_crawler`
method and how to clean up the resources properly.:: method and how to clean up the resources properly.
.. code-block:: python
import pymongo import pymongo
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
class MongoPipeline: class MongoPipeline:
collection_name = 'scrapy_items' collection_name = "scrapy_items"
def __init__(self, mongo_uri, mongo_db): def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri self.mongo_uri = mongo_uri
@ -151,8 +160,8 @@ method and how to clean up the resources properly.::
@classmethod @classmethod
def from_crawler(cls, crawler): def from_crawler(cls, crawler):
return cls( return cls(
mongo_uri=crawler.settings.get('MONGO_URI'), mongo_uri=crawler.settings.get("MONGO_URI"),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items') mongo_db=crawler.settings.get("MONGO_DATABASE", "items"),
) )
def open_spider(self, spider): def open_spider(self, spider):
@ -183,7 +192,7 @@ render a screenshot of the item URL. After the request response is downloaded,
the item pipeline saves the screenshot to a file and adds the filename to the the item pipeline saves the screenshot to a file and adds the filename to the
item. item.
:: .. code-block:: python
import hashlib import hashlib
from pathlib import Path from pathlib import Path
@ -231,23 +240,24 @@ Duplicates filter
A filter that looks for duplicate items, and drops those items that were A filter that looks for duplicate items, and drops those items that were
already processed. Let's say that our items have a unique id, but our spider already processed. Let's say that our items have a unique id, but our spider
returns multiples items with the same id:: returns multiples items with the same id:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem from scrapy.exceptions import DropItem
class DuplicatesPipeline:
class DuplicatesPipeline:
def __init__(self): def __init__(self):
self.ids_seen = set() self.ids_seen = set()
def process_item(self, item, spider): def process_item(self, item, spider):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
if adapter['id'] in self.ids_seen: if adapter["id"] in self.ids_seen:
raise DropItem(f"Duplicate item found: {item!r}") raise DropItem(f"Duplicate item found: {item!r}")
else: else:
self.ids_seen.add(adapter['id']) self.ids_seen.add(adapter["id"])
return item return item
@ -255,11 +265,13 @@ Activating an Item Pipeline component
===================================== =====================================
To activate an Item Pipeline component you must add its class to the To activate an Item Pipeline component you must add its class to the
:setting:`ITEM_PIPELINES` setting, like in the following example:: :setting:`ITEM_PIPELINES` setting, like in the following example:
.. code-block:: python
ITEM_PIPELINES = { ITEM_PIPELINES = {
'myproject.pipelines.PricePipeline': 300, "myproject.pipelines.PricePipeline": 300,
'myproject.pipelines.JsonWriterPipeline': 800, "myproject.pipelines.JsonWriterPipeline": 800,
} }
The integer values you assign to classes in this setting determine the The integer values you assign to classes in this setting determine the

30
docs/topics/items.rst
View File

@ -76,10 +76,13 @@ make it the most feature-complete item type:
:class:`Field` objects used in the :ref:`Item declaration :class:`Field` objects used in the :ref:`Item declaration
<topics-items-declaring>`. <topics-items-declaring>`.
Example:: Example:
.. code-block:: python
from scrapy.item import Item, Field from scrapy.item import Item, Field
class CustomItem(Item): class CustomItem(Item):
one_field = Field() one_field = Field()
another_field = Field() another_field = Field()
@ -102,10 +105,13 @@ Additionally, ``dataclass`` items also allow to:
* define custom field metadata through :func:`dataclasses.field`, which can be used to * define custom field metadata through :func:`dataclasses.field`, which can be used to
:ref:`customize serialization <topics-exporters-field-serialization>`. :ref:`customize serialization <topics-exporters-field-serialization>`.
Example:: Example:
.. code-block:: python
from dataclasses import dataclass from dataclasses import dataclass
@dataclass @dataclass
class CustomItem: class CustomItem:
one_field: str one_field: str
@ -133,10 +139,13 @@ Additionally, ``attr.s`` items also allow to:
In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed. In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed.
Example:: Example:
.. code-block:: python
import attr import attr
@attr.s @attr.s
class CustomItem: class CustomItem:
one_field = attr.ib() one_field = attr.ib()
@ -152,10 +161,13 @@ Declaring Item subclasses
------------------------- -------------------------
Item subclasses are declared using a simple class definition syntax and Item subclasses are declared using a simple class definition syntax and
:class:`Field` objects. Here is an example:: :class:`Field` objects. Here is an example:
.. code-block:: python
import scrapy import scrapy
class Product(scrapy.Item): class Product(scrapy.Item):
name = scrapy.Field() name = scrapy.Field()
price = scrapy.Field() price = scrapy.Field()
@ -347,17 +359,21 @@ Extending Item subclasses
You can extend Items (to add more fields or to change some metadata for some You can extend Items (to add more fields or to change some metadata for some
fields) by declaring a subclass of your original Item. fields) by declaring a subclass of your original Item.
For example:: For example:
.. code-block:: python
class DiscountedProduct(Product): class DiscountedProduct(Product):
discount_percent = scrapy.Field(serializer=str) discount_percent = scrapy.Field(serializer=str)
discount_expiration_date = scrapy.Field() discount_expiration_date = scrapy.Field()
You can also extend field metadata by using the previous field metadata and You can also extend field metadata by using the previous field metadata and
appending more values, or changing existing values, like this:: appending more values, or changing existing values, like this:
.. code-block:: python
class SpecificProduct(Product): class SpecificProduct(Product):
name = scrapy.Field(Product.fields['name'], serializer=my_serializer) name = scrapy.Field(Product.fields["name"], serializer=my_serializer)
That adds (or replaces) the ``serializer`` metadata key for the ``name`` field, That adds (or replaces) the ``serializer`` metadata key for the ``name`` field,
keeping all the previously existing metadata values. keeping all the previously existing metadata values.

6
docs/topics/jobs.rst
View File

@ -51,11 +51,13 @@ loading that attribute from the job directory, when the spider starts and
stops. stops.
Here's an example of a callback that uses the spider state (other spider code Here's an example of a callback that uses the spider state (other spider code
is omitted for brevity):: is omitted for brevity):
.. code-block:: python
def parse_item(self, response): def parse_item(self, response):
# parse item here # parse item here
self.state['items_count'] = self.state.get('items_count', 0) + 1 self.state["items_count"] = self.state.get("items_count", 0) + 1
Persistence gotchas Persistence gotchas
=================== ===================

8
docs/topics/link-extractors.rst
View File

@ -18,7 +18,9 @@ through a set of :class:`~scrapy.spiders.Rule` objects.
You can also use link extractors in regular spiders. For example, you can instantiate You can also use link extractors in regular spiders. For example, you can instantiate
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class
variable in your spider, and use it from your spider callbacks:: variable in your spider, and use it from your spider callbacks:
.. code-block:: python
def parse(self, response): def parse(self, response):
for link in self.link_extractor.extract_links(response): for link in self.link_extractor.extract_links(response):
@ -132,7 +134,9 @@ LxmlLinkExtractor
.. highlight:: python .. highlight:: python
You can use the following function in ``process_value``:: You can use the following function in ``process_value``:
.. code-block:: python
def process_value(value): def process_value(value):
m = re.search("javascript:goToPage\('(.*?)'", value) m = re.search("javascript:goToPage\('(.*?)'", value)

100
docs/topics/loaders.rst
View File

@ -46,18 +46,21 @@ using a proper processing function.
Here is a typical Item Loader usage in a :ref:`Spider <topics-spiders>`, using Here is a typical Item Loader usage in a :ref:`Spider <topics-spiders>`, using
the :ref:`Product item <topics-items-declaring>` declared in the :ref:`Items the :ref:`Product item <topics-items-declaring>` declared in the :ref:`Items
chapter <topics-items>`:: chapter <topics-items>`:
.. code-block:: python
from scrapy.loader import ItemLoader from scrapy.loader import ItemLoader
from myproject.items import Product from myproject.items import Product
def parse(self, response): def parse(self, response):
l = ItemLoader(item=Product(), response=response) l = ItemLoader(item=Product(), response=response)
l.add_xpath('name', '//div[@class="product_name"]') l.add_xpath("name", '//div[@class="product_name"]')
l.add_xpath('name', '//div[@class="product_title"]') l.add_xpath("name", '//div[@class="product_title"]')
l.add_xpath('price', '//p[@id="price"]') l.add_xpath("price", '//p[@id="price"]')
l.add_css('stock', 'p#stock') l.add_css("stock", "p#stock")
l.add_value('last_updated', 'today') # you can also use literal values l.add_value("last_updated", "today") # you can also use literal values
return l.load_item() return l.load_item()
By quickly looking at that code, we can see the ``name`` field is being By quickly looking at that code, we can see the ``name`` field is being
@ -93,11 +96,14 @@ will be populated incrementally using the loader's :meth:`~ItemLoader.add_xpath`
:meth:`~ItemLoader.add_css` and :meth:`~ItemLoader.add_value` methods. :meth:`~ItemLoader.add_css` and :meth:`~ItemLoader.add_value` methods.
One approach to overcome this is to define items using the One approach to overcome this is to define items using the
:func:`~dataclasses.field` function, with a ``default`` argument:: :func:`~dataclasses.field` function, with a ``default`` argument:
.. code-block:: python
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import Optional from typing import Optional
@dataclass @dataclass
class InventoryItem: class InventoryItem:
name: Optional[str] = field(default=None) name: Optional[str] = field(default=None)
@ -122,14 +128,16 @@ processor). The result of the output processor is the final value that gets
assigned to the item. assigned to the item.
Let's see an example to illustrate how the input and output processors are Let's see an example to illustrate how the input and output processors are
called for a particular field (the same applies for any other field):: called for a particular field (the same applies for any other field):
.. code-block:: python
l = ItemLoader(Product(), some_selector) l = ItemLoader(Product(), some_selector)
l.add_xpath('name', xpath1) # (1) l.add_xpath("name", xpath1) # (1)
l.add_xpath('name', xpath2) # (2) l.add_xpath("name", xpath2) # (2)
l.add_css('name', css) # (3) l.add_css("name", css) # (3)
l.add_value('name', 'test') # (4) l.add_value("name", "test") # (4)
return l.load_item() # (5) return l.load_item() # (5)
So what happens is: So what happens is:
@ -184,11 +192,14 @@ processors <itemloaders:built-in-processors>` built-in for convenience.
Declaring Item Loaders Declaring Item Loaders
====================== ======================
Item Loaders are declared using a class definition syntax. Here is an example:: Item Loaders are declared using a class definition syntax. Here is an example:
.. code-block:: python
from itemloaders.processors import TakeFirst, MapCompose, Join from itemloaders.processors import TakeFirst, MapCompose, Join
from scrapy.loader import ItemLoader from scrapy.loader import ItemLoader
class ProductLoader(ItemLoader): class ProductLoader(ItemLoader):
default_output_processor = TakeFirst() default_output_processor = TakeFirst()
@ -215,16 +226,20 @@ As seen in the previous section, input and output processors can be declared in
the Item Loader definition, and it's very common to declare input processors the Item Loader definition, and it's very common to declare input processors
this way. However, there is one more place where you can specify the input and this way. However, there is one more place where you can specify the input and
output processors to use: in the :ref:`Item Field <topics-items-fields>` output processors to use: in the :ref:`Item Field <topics-items-fields>`
metadata. Here is an example:: metadata. Here is an example:
.. code-block:: python
import scrapy import scrapy
from itemloaders.processors import Join, MapCompose, TakeFirst from itemloaders.processors import Join, MapCompose, TakeFirst
from w3lib.html import remove_tags from w3lib.html import remove_tags
def filter_price(value): def filter_price(value):
if value.isdigit(): if value.isdigit():
return value return value
class Product(scrapy.Item): class Product(scrapy.Item):
name = scrapy.Field( name = scrapy.Field(
input_processor=MapCompose(remove_tags), input_processor=MapCompose(remove_tags),
@ -263,10 +278,12 @@ declaring, instantiating or using Item Loader. They are used to modify the
behaviour of the input/output processors. behaviour of the input/output processors.
For example, suppose you have a function ``parse_length`` which receives a text For example, suppose you have a function ``parse_length`` which receives a text
value and extracts a length from it:: value and extracts a length from it:
.. code-block:: python
def parse_length(text, loader_context): def parse_length(text, loader_context):
unit = loader_context.get('unit', 'm') unit = loader_context.get("unit", "m")
# ... length parsing code goes here ... # ... length parsing code goes here ...
return parsed_length return parsed_length
@ -278,22 +295,28 @@ function (``parse_length`` in this case) can thus use them.
There are several ways to modify Item Loader context values: There are several ways to modify Item Loader context values:
1. By modifying the currently active Item Loader context 1. By modifying the currently active Item Loader context
(:attr:`~ItemLoader.context` attribute):: (:attr:`~ItemLoader.context` attribute):
.. code-block:: python
loader = ItemLoader(product) loader = ItemLoader(product)
loader.context['unit'] = 'cm' loader.context["unit"] = "cm"
2. On Item Loader instantiation (the keyword arguments of Item Loader 2. On Item Loader instantiation (the keyword arguments of Item Loader
``__init__`` method are stored in the Item Loader context):: ``__init__`` method are stored in the Item Loader context):
loader = ItemLoader(product, unit='cm') .. code-block:: python
loader = ItemLoader(product, unit="cm")
3. On Item Loader declaration, for those input/output processors that support 3. On Item Loader declaration, for those input/output processors that support
instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
them:: them:
.. code-block:: python
class ProductLoader(ItemLoader): class ProductLoader(ItemLoader):
length_out = MapCompose(parse_length, unit='cm') length_out = MapCompose(parse_length, unit="cm")
ItemLoader objects ItemLoader objects
@ -323,25 +346,29 @@ Example::
Without nested loaders, you need to specify the full xpath (or css) for each value Without nested loaders, you need to specify the full xpath (or css) for each value
that you wish to extract. that you wish to extract.
Example:: Example:
.. code-block:: python
loader = ItemLoader(item=Item()) loader = ItemLoader(item=Item())
# load stuff not in the footer # load stuff not in the footer
loader.add_xpath('social', '//footer/a[@class = "social"]/@href') loader.add_xpath("social", '//footer/a[@class = "social"]/@href')
loader.add_xpath('email', '//footer/a[@class = "email"]/@href') loader.add_xpath("email", '//footer/a[@class = "email"]/@href')
loader.load_item() loader.load_item()
Instead, you can create a nested loader with the footer selector and add values Instead, you can create a nested loader with the footer selector and add values
relative to the footer. The functionality is the same but you avoid repeating relative to the footer. The functionality is the same but you avoid repeating
the footer selector. the footer selector.
Example:: Example:
.. code-block:: python
loader = ItemLoader(item=Item()) loader = ItemLoader(item=Item())
# load stuff not in the footer # load stuff not in the footer
footer_loader = loader.nested_xpath('//footer') footer_loader = loader.nested_xpath("//footer")
footer_loader.add_xpath('social', 'a[@class = "social"]/@href') footer_loader.add_xpath("social", 'a[@class = "social"]/@href')
footer_loader.add_xpath('email', 'a[@class = "email"]/@href') footer_loader.add_xpath("email", 'a[@class = "email"]/@href')
# no need to call footer_loader.load_item() # no need to call footer_loader.load_item()
loader.load_item() loader.load_item()
@ -370,25 +397,32 @@ three dashes (e.g. ``---Plasma TV---``) and you don't want to end up scraping
those dashes in the final product names. those dashes in the final product names.
Here's how you can remove those dashes by reusing and extending the default Here's how you can remove those dashes by reusing and extending the default
Product Item Loader (``ProductLoader``):: Product Item Loader (``ProductLoader``):
.. code-block:: python
from itemloaders.processors import MapCompose from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader from myproject.ItemLoaders import ProductLoader
def strip_dashes(x): def strip_dashes(x):
return x.strip('-') return x.strip("-")
class SiteSpecificLoader(ProductLoader): class SiteSpecificLoader(ProductLoader):
name_in = MapCompose(strip_dashes, ProductLoader.name_in) name_in = MapCompose(strip_dashes, ProductLoader.name_in)
Another case where extending Item Loaders can be very helpful is when you have Another case where extending Item Loaders can be very helpful is when you have
multiple source formats, for example XML and HTML. In the XML version you may multiple source formats, for example XML and HTML. In the XML version you may
want to remove ``CDATA`` occurrences. Here's an example of how to do it:: want to remove ``CDATA`` occurrences. Here's an example of how to do it:
.. code-block:: python
from itemloaders.processors import MapCompose from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader from myproject.ItemLoaders import ProductLoader
from myproject.utils.xml import remove_cdata from myproject.utils.xml import remove_cdata
class XmlProductLoader(ProductLoader): class XmlProductLoader(ProductLoader):
name_in = MapCompose(remove_cdata, ProductLoader.name_in) name_in = MapCompose(remove_cdata, ProductLoader.name_in)

109
docs/topics/logging.rst
View File

@ -39,16 +39,22 @@ How to log messages
=================== ===================
Here's a quick example of how to log a message using the ``logging.WARNING`` Here's a quick example of how to log a message using the ``logging.WARNING``
level:: level:
.. code-block:: python
import logging import logging
logging.warning("This is a warning") logging.warning("This is a warning")
There are shortcuts for issuing log messages on any of the standard 5 levels, There are shortcuts for issuing log messages on any of the standard 5 levels,
and there's also a general ``logging.log`` method which takes a given level as and there's also a general ``logging.log`` method which takes a given level as
argument. If needed, the last example could be rewritten as:: argument. If needed, the last example could be rewritten as:
.. code-block:: python
import logging import logging
logging.log(logging.WARNING, "This is a warning") logging.log(logging.WARNING, "This is a warning")
On top of that, you can create different "loggers" to encapsulate messages. (For On top of that, you can create different "loggers" to encapsulate messages. (For
@ -59,24 +65,33 @@ constructions.
The previous examples use the root logger behind the scenes, which is a top level The previous examples use the root logger behind the scenes, which is a top level
logger where all messages are propagated to (unless otherwise specified). Using logger where all messages are propagated to (unless otherwise specified). Using
``logging`` helpers is merely a shortcut for getting the root logger ``logging`` helpers is merely a shortcut for getting the root logger
explicitly, so this is also an equivalent of the last snippets:: explicitly, so this is also an equivalent of the last snippets:
.. code-block:: python
import logging import logging
logger = logging.getLogger() logger = logging.getLogger()
logger.warning("This is a warning") logger.warning("This is a warning")
You can use a different logger just by getting its name with the You can use a different logger just by getting its name with the
``logging.getLogger`` function:: ``logging.getLogger`` function:
.. code-block:: python
import logging import logging
logger = logging.getLogger('mycustomlogger')
logger = logging.getLogger("mycustomlogger")
logger.warning("This is a warning") logger.warning("This is a warning")
Finally, you can ensure having a custom logger for any module you're working on Finally, you can ensure having a custom logger for any module you're working on
by using the ``__name__`` variable, which is populated with current module's by using the ``__name__`` variable, which is populated with current module's
path:: path:
.. code-block:: python
import logging import logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger.warning("This is a warning") logger.warning("This is a warning")
@ -94,33 +109,39 @@ Logging from Spiders
==================== ====================
Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider
instance, which can be accessed and used like this:: instance, which can be accessed and used like this:
.. code-block:: python
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
start_urls = ['https://scrapy.org'] start_urls = ["https://scrapy.org"]
def parse(self, response): def parse(self, response):
self.logger.info('Parse function called on %s', response.url) self.logger.info("Parse function called on %s", response.url)
That logger is created using the Spider's name, but you can use any custom That logger is created using the Spider's name, but you can use any custom
Python logger you want. For example:: Python logger you want. For example:
.. code-block:: python
import logging import logging
import scrapy import scrapy
logger = logging.getLogger('mycustomlogger') logger = logging.getLogger("mycustomlogger")
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
start_urls = ['https://scrapy.org'] start_urls = ["https://scrapy.org"]
def parse(self, response): def parse(self, response):
logger.info('Parse function called on %s', response.url) logger.info("Parse function called on %s", response.url)
.. _topics-logging-configuration: .. _topics-logging-configuration:
@ -229,7 +250,9 @@ the crawl.
Next, we can see that the message has INFO level. To hide it Next, we can see that the message has INFO level. To hide it
we should set logging level for ``scrapy.spidermiddlewares.httperror`` we should set logging level for ``scrapy.spidermiddlewares.httperror``
higher than INFO; next level after INFO is WARNING. It could be done higher than INFO; next level after INFO is WARNING. It could be done
e.g. in the spider's ``__init__`` method:: e.g. in the spider's ``__init__`` method:
.. code-block:: python
import logging import logging
import scrapy import scrapy
@ -238,7 +261,7 @@ e.g. in the spider's ``__init__`` method::
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
# ... # ...
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
logger = logging.getLogger('scrapy.spidermiddlewares.httperror') logger = logging.getLogger("scrapy.spidermiddlewares.httperror")
logger.setLevel(logging.WARNING) logger.setLevel(logging.WARNING)
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
@ -249,43 +272,53 @@ You can also filter log records by :class:`~logging.LogRecord` data. For
example, you can filter log records by message content using a substring or example, you can filter log records by message content using a substring or
a regular expression. Create a :class:`logging.Filter` subclass a regular expression. Create a :class:`logging.Filter` subclass
and equip it with a regular expression pattern to and equip it with a regular expression pattern to
filter out unwanted messages:: filter out unwanted messages:
.. code-block:: python
import logging import logging
import re import re
class ContentFilter(logging.Filter): class ContentFilter(logging.Filter):
def filter(self, record): def filter(self, record):
match = re.search(r'\d{3} [Ee]rror, retrying', record.message) match = re.search(r"\d{3} [Ee]rror, retrying", record.message)
if match: if match:
return False return False
A project-level filter may be attached to the root A project-level filter may be attached to the root
handler created by Scrapy, this is a wieldy way to handler created by Scrapy, this is a wieldy way to
filter all loggers in different parts of the project filter all loggers in different parts of the project
(middlewares, spider, etc.):: (middlewares, spider, etc.):
import logging .. code-block:: python
import scrapy
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
for handler in logging.root.handlers:
handler.addFilter(ContentFilter())
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
for handler in logging.root.handlers:
handler.addFilter(ContentFilter())
Alternatively, you may choose a specific logger Alternatively, you may choose a specific logger
and hide it without affecting other loggers:: and hide it without affecting other loggers:
.. code-block:: python
import logging import logging
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
# ... # ...
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
logger = logging.getLogger('my_logger') logger = logging.getLogger("my_logger")
logger.addFilter(ContentFilter()) logger.addFilter(ContentFilter())
scrapy.utils.log module scrapy.utils.log module
======================= =======================
@ -306,14 +339,14 @@ scrapy.utils.log module
so it is recommended to only use :func:`logging.basicConfig` together with so it is recommended to only use :func:`logging.basicConfig` together with
:class:`~scrapy.crawler.CrawlerRunner`. :class:`~scrapy.crawler.CrawlerRunner`.
This is an example on how to redirect ``INFO`` or higher messages to a file:: This is an example on how to redirect ``INFO`` or higher messages to a file:
.. code-block:: python
import logging import logging
logging.basicConfig( logging.basicConfig(
filename='log.txt', filename="log.txt", format="%(levelname)s: %(message)s", level=logging.INFO
format='%(levelname)s: %(message)s',
level=logging.INFO
) )
Refer to :ref:`run-from-script` for more details about using Scrapy this Refer to :ref:`run-from-script` for more details about using Scrapy this

177
docs/topics/media-pipeline.rst
View File

@ -87,13 +87,17 @@ Enabling your Media Pipeline
To enable your media pipeline you must first add it to your project To enable your media pipeline you must first add it to your project
:setting:`ITEM_PIPELINES` setting. :setting:`ITEM_PIPELINES` setting.
For Images Pipeline, use:: For Images Pipeline, use:
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1} .. code-block:: python
For Files Pipeline, use:: ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}
ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1} For Files Pipeline, use:
.. code-block:: python
ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}
.. note:: .. note::
You can also use both the Files and Images Pipeline at the same time. You can also use both the Files and Images Pipeline at the same time.
@ -103,13 +107,17 @@ Then, configure the target storage setting to a valid value that will be used
for storing the downloaded images. Otherwise the pipeline will remain disabled, for storing the downloaded images. Otherwise the pipeline will remain disabled,
even if you include it in the :setting:`ITEM_PIPELINES` setting. even if you include it in the :setting:`ITEM_PIPELINES` setting.
For the Files Pipeline, set the :setting:`FILES_STORE` setting:: For the Files Pipeline, set the :setting:`FILES_STORE` setting:
FILES_STORE = '/path/to/valid/dir' .. code-block:: python
For the Images Pipeline, set the :setting:`IMAGES_STORE` setting:: FILES_STORE = "/path/to/valid/dir"
IMAGES_STORE = '/path/to/valid/dir' For the Images Pipeline, set the :setting:`IMAGES_STORE` setting:
.. code-block:: python
IMAGES_STORE = "/path/to/valid/dir"
.. _topics-file-naming: .. _topics-file-naming:
@ -157,10 +165,11 @@ By overriding ``file_path`` like this:
import hashlib import hashlib
def file_path(self, request, response=None, info=None, *, item=None): def file_path(self, request, response=None, info=None, *, item=None):
image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5) image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)
image_perspective = request.url.split('/')[-2] image_perspective = request.url.split("/")[-2]
image_filename = f'{image_url_hash}_{image_perspective}.jpg' image_filename = f"{image_url_hash}_{image_perspective}.jpg"
return image_filename return image_filename
@ -233,30 +242,38 @@ If botocore_ >= 1.4.87 is installed, :setting:`FILES_STORE` and
:setting:`IMAGES_STORE` can represent an Amazon S3 bucket. Scrapy will :setting:`IMAGES_STORE` can represent an Amazon S3 bucket. Scrapy will
automatically upload the files to the bucket. automatically upload the files to the bucket.
For example, this is a valid :setting:`IMAGES_STORE` value:: For example, this is a valid :setting:`IMAGES_STORE` value:
IMAGES_STORE = 's3://bucket/images' .. code-block:: python
IMAGES_STORE = "s3://bucket/images"
You can modify the Access Control List (ACL) policy used for the stored files, You can modify the Access Control List (ACL) policy used for the stored files,
which is defined by the :setting:`FILES_STORE_S3_ACL` and which is defined by the :setting:`FILES_STORE_S3_ACL` and
:setting:`IMAGES_STORE_S3_ACL` settings. By default, the ACL is set to :setting:`IMAGES_STORE_S3_ACL` settings. By default, the ACL is set to
``private``. To make the files publicly available use the ``public-read`` ``private``. To make the files publicly available use the ``public-read``
policy:: policy:
IMAGES_STORE_S3_ACL = 'public-read' .. code-block:: python
IMAGES_STORE_S3_ACL = "public-read"
For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide. For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.
You can also use other S3-like storages. Storages like self-hosted `Minio`_ or You can also use other S3-like storages. Storages like self-hosted `Minio`_ or
`s3.scality`_. All you need to do is set endpoint option in you Scrapy `s3.scality`_. All you need to do is set endpoint option in you Scrapy
settings:: settings:
AWS_ENDPOINT_URL = 'http://minio.example.com:9000' .. code-block:: python
For self-hosting you also might feel the need not to use SSL and not to verify SSL connection:: AWS_ENDPOINT_URL = "http://minio.example.com:9000"
AWS_USE_SSL = False # or True (None by default) For self-hosting you also might feel the need not to use SSL and not to verify SSL connection:
AWS_VERIFY = False # or True (None by default)
.. code-block:: python
AWS_USE_SSL = False # or True (None by default)
AWS_VERIFY = False # or True (None by default)
.. _botocore: https://github.com/boto/botocore .. _botocore: https://github.com/boto/botocore
.. _canned ACLs: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl .. _canned ACLs: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
@ -277,10 +294,12 @@ bucket. Scrapy will automatically upload the files to the bucket. (requires `goo
.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python .. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings:: For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings:
IMAGES_STORE = 'gs://bucket/images/' .. code-block:: python
GCS_PROJECT_ID = 'project_id'
IMAGES_STORE = "gs://bucket/images/"
GCS_PROJECT_ID = "project_id"
For information about authentication, see this `documentation`_. For information about authentication, see this `documentation`_.
@ -291,9 +310,11 @@ which is defined by the :setting:`FILES_STORE_GCS_ACL` and
:setting:`IMAGES_STORE_GCS_ACL` settings. By default, the ACL is set to :setting:`IMAGES_STORE_GCS_ACL` settings. By default, the ACL is set to
``''`` (empty string) which means that Cloud Storage applies the bucket's default object ACL to the object. ``''`` (empty string) which means that Cloud Storage applies the bucket's default object ACL to the object.
To make the files publicly available use the ``publicRead`` To make the files publicly available use the ``publicRead``
policy:: policy:
IMAGES_STORE_GCS_ACL = 'publicRead' .. code-block:: python
IMAGES_STORE_GCS_ACL = "publicRead"
For more information, see `Predefined ACLs`_ in the Google Cloud Platform Developer Guide. For more information, see `Predefined ACLs`_ in the Google Cloud Platform Developer Guide.
@ -318,10 +339,13 @@ respectively), the pipeline will put the results under the respective field
When using :ref:`item types <item-types>` for which fields are defined beforehand, When using :ref:`item types <item-types>` for which fields are defined beforehand,
you must define both the URLs field and the results field. For example, when you must define both the URLs field and the results field. For example, when
using the images pipeline, items must define both the ``image_urls`` and the using the images pipeline, items must define both the ``image_urls`` and the
``images`` field. For instance, using the :class:`~scrapy.Item` class:: ``images`` field. For instance, using the :class:`~scrapy.Item` class:
.. code-block:: python
import scrapy import scrapy
class MyItem(scrapy.Item): class MyItem(scrapy.Item):
# ... other item fields ... # ... other item fields ...
image_urls = scrapy.Field() image_urls = scrapy.Field()
@ -331,16 +355,20 @@ If you want to use another field name for the URLs key or for the results key,
it is also possible to override it. it is also possible to override it.
For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or
:setting:`FILES_RESULT_FIELD` settings:: :setting:`FILES_RESULT_FIELD` settings:
FILES_URLS_FIELD = 'field_name_for_your_files_urls' .. code-block:: python
FILES_RESULT_FIELD = 'field_name_for_your_processed_files'
FILES_URLS_FIELD = "field_name_for_your_files_urls"
FILES_RESULT_FIELD = "field_name_for_your_processed_files"
For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
:setting:`IMAGES_RESULT_FIELD` settings:: :setting:`IMAGES_RESULT_FIELD` settings:
IMAGES_URLS_FIELD = 'field_name_for_your_images_urls' .. code-block:: python
IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images'
IMAGES_URLS_FIELD = "field_name_for_your_images_urls"
IMAGES_RESULT_FIELD = "field_name_for_your_processed_images"
If you need something more complex and want to override the custom pipeline If you need something more complex and want to override the custom pipeline
behaviour, see :ref:`topics-media-pipeline-override`. behaviour, see :ref:`topics-media-pipeline-override`.
@ -366,7 +394,9 @@ File expiration
The Image Pipeline avoids downloading files that were downloaded recently. To The Image Pipeline avoids downloading files that were downloaded recently. To
adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or
:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which :setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which
specifies the delay in number of days:: specifies the delay in number of days:
.. code-block:: python
# 120 days of delay for files expiration # 120 days of delay for files expiration
FILES_EXPIRES = 120 FILES_EXPIRES = 120
@ -400,11 +430,13 @@ images.
In order to use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary In order to use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
where the keys are the thumbnail names and the values are their dimensions. where the keys are the thumbnail names and the values are their dimensions.
For example:: For example:
.. code-block:: python
IMAGES_THUMBS = { IMAGES_THUMBS = {
'small': (50, 50), "small": (50, 50),
'big': (270, 270), "big": (270, 270),
} }
When you use this feature, the Images Pipeline will create thumbnails of the When you use this feature, the Images Pipeline will create thumbnails of the
@ -495,17 +527,19 @@ See here the methods that you can override in your custom Files Pipeline:
For example, if file URLs end like regular paths (e.g. For example, if file URLs end like regular paths (e.g.
``https://example.com/a/b/c/foo.png``), you can use the following ``https://example.com/a/b/c/foo.png``), you can use the following
approach to download all files into the ``files`` folder with their approach to download all files into the ``files`` folder with their
original filenames (e.g. ``files/foo.png``):: original filenames (e.g. ``files/foo.png``):
.. code-block:: python
from pathlib import PurePosixPath from pathlib import PurePosixPath
from urllib.parse import urlparse from urllib.parse import urlparse
from scrapy.pipelines.files import FilesPipeline from scrapy.pipelines.files import FilesPipeline
class MyFilesPipeline(FilesPipeline):
class MyFilesPipeline(FilesPipeline):
def file_path(self, request, response=None, info=None, *, item=None): def file_path(self, request, response=None, info=None, *, item=None):
return 'files/' + PurePosixPath(urlparse(request.url).path).name return "files/" + PurePosixPath(urlparse(request.url).path).name
Similarly, you can use the ``item`` to determine the file path based on some item Similarly, you can use the ``item`` to determine the file path based on some item
property. property.
@ -521,13 +555,16 @@ See here the methods that you can override in your custom Files Pipeline:
As seen on the workflow, the pipeline will get the URLs of the images to As seen on the workflow, the pipeline will get the URLs of the images to
download from the item. In order to do this, you can override the download from the item. In order to do this, you can override the
:meth:`~get_media_requests` method and return a Request for each :meth:`~get_media_requests` method and return a Request for each
file URL:: file URL:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
def get_media_requests(self, item, info): def get_media_requests(self, item, info):
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
for file_url in adapter['file_urls']: for file_url in adapter["file_urls"]:
yield scrapy.Request(file_url) yield scrapy.Request(file_url)
Those requests will be processed by the pipeline and, when they have finished Those requests will be processed by the pipeline and, when they have finished
@ -567,15 +604,22 @@ See here the methods that you can override in your custom Files Pipeline:
guaranteed to retain the same order of the requests returned from the guaranteed to retain the same order of the requests returned from the
:meth:`~get_media_requests` method. :meth:`~get_media_requests` method.
Here's a typical value of the ``results`` argument:: Here's a typical value of the ``results`` argument:
[(True, .. code-block:: python
{'checksum': '2b00042f7481c7b056c4b410d28f33cf',
'path': 'full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg', [
'url': 'http://www.example.com/files/product1.pdf', (
'status': 'downloaded'}), True,
(False, {
Failure(...))] "checksum": "2b00042f7481c7b056c4b410d28f33cf",
"path": "full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg",
"url": "http://www.example.com/files/product1.pdf",
"status": "downloaded",
},
),
(False, Failure(...)),
]
By default the :meth:`get_media_requests` method returns ``None`` which By default the :meth:`get_media_requests` method returns ``None`` which
means there are no files to download for the item. means there are no files to download for the item.
@ -592,17 +636,20 @@ See here the methods that you can override in your custom Files Pipeline:
Here is an example of the :meth:`~item_completed` method where we Here is an example of the :meth:`~item_completed` method where we
store the downloaded file paths (passed in results) in the ``file_paths`` store the downloaded file paths (passed in results) in the ``file_paths``
item field, and we drop the item if it doesn't contain any files:: item field, and we drop the item if it doesn't contain any files:
.. code-block:: python
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem from scrapy.exceptions import DropItem
def item_completed(self, results, item, info): def item_completed(self, results, item, info):
file_paths = [x['path'] for ok, x in results if ok] file_paths = [x["path"] for ok, x in results if ok]
if not file_paths: if not file_paths:
raise DropItem("Item contains no files") raise DropItem("Item contains no files")
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
adapter['file_paths'] = file_paths adapter["file_paths"] = file_paths
return item return item
By default, the :meth:`item_completed` method returns the item. By default, the :meth:`item_completed` method returns the item.
@ -634,17 +681,19 @@ See here the methods that you can override in your custom Images Pipeline:
For example, if file URLs end like regular paths (e.g. For example, if file URLs end like regular paths (e.g.
``https://example.com/a/b/c/foo.png``), you can use the following ``https://example.com/a/b/c/foo.png``), you can use the following
approach to download all files into the ``files`` folder with their approach to download all files into the ``files`` folder with their
original filenames (e.g. ``files/foo.png``):: original filenames (e.g. ``files/foo.png``):
.. code-block:: python
from pathlib import PurePosixPath from pathlib import PurePosixPath
from urllib.parse import urlparse from urllib.parse import urlparse
from scrapy.pipelines.images import ImagesPipeline from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
class MyImagesPipeline(ImagesPipeline):
def file_path(self, request, response=None, info=None, *, item=None): def file_path(self, request, response=None, info=None, *, item=None):
return 'files/' + PurePosixPath(urlparse(request.url).path).name return "files/" + PurePosixPath(urlparse(request.url).path).name
Similarly, you can use the ``item`` to determine the file path based on some item Similarly, you can use the ``item`` to determine the file path based on some item
property. property.
@ -700,33 +749,35 @@ Custom Images pipeline example
============================== ==============================
Here is a full example of the Images Pipeline whose methods are exemplified Here is a full example of the Images Pipeline whose methods are exemplified
above:: above:
.. code-block:: python
import scrapy import scrapy
from itemadapter import ItemAdapter from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info): def get_media_requests(self, item, info):
for image_url in item['image_urls']: for image_url in item["image_urls"]:
yield scrapy.Request(image_url) yield scrapy.Request(image_url)
def item_completed(self, results, item, info): def item_completed(self, results, item, info):
image_paths = [x['path'] for ok, x in results if ok] image_paths = [x["path"] for ok, x in results if ok]
if not image_paths: if not image_paths:
raise DropItem("Item contains no images") raise DropItem("Item contains no images")
adapter = ItemAdapter(item) adapter = ItemAdapter(item)
adapter['image_paths'] = image_paths adapter["image_paths"] = image_paths
return item return item
To enable your custom media pipeline component you must add its class import path to the To enable your custom media pipeline component you must add its class import path to the
:setting:`ITEM_PIPELINES` setting, like in the following example:: :setting:`ITEM_PIPELINES` setting, like in the following example:
ITEM_PIPELINES = { .. code-block:: python
'myproject.pipelines.MyImagesPipeline': 300
} ITEM_PIPELINES = {"myproject.pipelines.MyImagesPipeline": 300}
.. _MD5 hash: https://en.wikipedia.org/wiki/MD5 .. _MD5 hash: https://en.wikipedia.org/wiki/MD5

55
docs/topics/practices.rst
View File

@ -25,23 +25,27 @@ the one used by all Scrapy commands.
Here's an example showing how to run a single spider with it. Here's an example showing how to run a single spider with it.
:: .. code-block:: python
import scrapy import scrapy
from scrapy.crawler import CrawlerProcess from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
# Your spider definition # Your spider definition
... ...
process = CrawlerProcess(settings={
"FEEDS": { process = CrawlerProcess(
"items.json": {"format": "json"}, settings={
}, "FEEDS": {
}) "items.json": {"format": "json"},
},
}
)
process.crawl(MySpider) process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished process.start() # the script will block here until the crawling is finished
Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess` Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess`
documentation to get acquainted with its usage details. documentation to get acquainted with its usage details.
@ -55,7 +59,7 @@ instance with your project settings.
What follows is a working example of how to do that, using the `testspiders`_ What follows is a working example of how to do that, using the `testspiders`_
project as example. project as example.
:: .. code-block:: python
from scrapy.crawler import CrawlerProcess from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings from scrapy.utils.project import get_project_settings
@ -63,8 +67,8 @@ project as example.
process = CrawlerProcess(get_project_settings()) process = CrawlerProcess(get_project_settings())
# 'followall' is the name of one of the spiders of the project. # 'followall' is the name of one of the spiders of the project.
process.crawl('followall', domain='scrapy.org') process.crawl("followall", domain="scrapy.org")
process.start() # the script will block here until the crawling is finished process.start() # the script will block here until the crawling is finished
There's another Scrapy utility that provides more control over the crawling There's another Scrapy utility that provides more control over the crawling
process: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper process: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper
@ -84,23 +88,25 @@ returned by the :meth:`CrawlerRunner.crawl
Here's an example of its usage, along with a callback to manually stop the Here's an example of its usage, along with a callback to manually stop the
reactor after ``MySpider`` has finished running. reactor after ``MySpider`` has finished running.
:: .. code-block:: python
from twisted.internet import reactor from twisted.internet import reactor
import scrapy import scrapy
from scrapy.crawler import CrawlerRunner from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging from scrapy.utils.log import configure_logging
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
# Your spider definition # Your spider definition
... ...
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = CrawlerRunner() runner = CrawlerRunner()
d = runner.crawl(MySpider) d = runner.crawl(MySpider)
d.addBoth(lambda _: reactor.stop()) d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until the crawling is finished reactor.run() # the script will block here until the crawling is finished
.. seealso:: :doc:`twisted:core/howto/reactor-basics` .. seealso:: :doc:`twisted:core/howto/reactor-basics`
@ -115,29 +121,32 @@ the :ref:`internal API <topics-api>`.
Here is an example that runs multiple spiders simultaneously: Here is an example that runs multiple spiders simultaneously:
:: .. code-block:: python
import scrapy import scrapy
from scrapy.crawler import CrawlerProcess from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider): class MySpider1(scrapy.Spider):
# Your first spider definition # Your first spider definition
... ...
class MySpider2(scrapy.Spider): class MySpider2(scrapy.Spider):
# Your second spider definition # Your second spider definition
... ...
settings = get_project_settings() settings = get_project_settings()
process = CrawlerProcess(settings) process = CrawlerProcess(settings)
process.crawl(MySpider1) process.crawl(MySpider1)
process.crawl(MySpider2) process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished process.start() # the script will block here until all crawling jobs are finished
Same example using :class:`~scrapy.crawler.CrawlerRunner`: Same example using :class:`~scrapy.crawler.CrawlerRunner`:
:: .. code-block:: python
import scrapy import scrapy
from twisted.internet import reactor from twisted.internet import reactor
@ -145,14 +154,17 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:
from scrapy.utils.log import configure_logging from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider): class MySpider1(scrapy.Spider):
# Your first spider definition # Your first spider definition
... ...
class MySpider2(scrapy.Spider): class MySpider2(scrapy.Spider):
# Your second spider definition # Your second spider definition
... ...
configure_logging() configure_logging()
settings = get_project_settings() settings = get_project_settings()
runner = CrawlerRunner(settings) runner = CrawlerRunner(settings)
@ -161,37 +173,42 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:
d = runner.join() d = runner.join()
d.addBoth(lambda _: reactor.stop()) d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished reactor.run() # the script will block here until all crawling jobs are finished
Same example but running the spiders sequentially by chaining the deferreds: Same example but running the spiders sequentially by chaining the deferreds:
:: .. code-block:: python
from twisted.internet import reactor, defer from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider): class MySpider1(scrapy.Spider):
# Your first spider definition # Your first spider definition
... ...
class MySpider2(scrapy.Spider): class MySpider2(scrapy.Spider):
# Your second spider definition # Your second spider definition
... ...
settings = get_project_settings() settings = get_project_settings()
configure_logging(settings) configure_logging(settings)
runner = CrawlerRunner(settings) runner = CrawlerRunner(settings)
@defer.inlineCallbacks @defer.inlineCallbacks
def crawl(): def crawl():
yield runner.crawl(MySpider1) yield runner.crawl(MySpider1)
yield runner.crawl(MySpider2) yield runner.crawl(MySpider2)
reactor.stop() reactor.stop()
crawl() crawl()
reactor.run() # the script will block here until the last crawl call is finished reactor.run() # the script will block here until the last crawl call is finished
Different spiders can set different values for the same setting, but when they Different spiders can set different values for the same setting, but when they
run in the same process it may be impossible, by design or because of some run in the same process it may be impossible, by design or because of some

176
docs/topics/request-response.rst
View File

@ -78,23 +78,27 @@ Request objects
:param cookies: the request cookies. These can be sent in two forms. :param cookies: the request cookies. These can be sent in two forms.
1. Using a dict:: 1. Using a dict:
.. code-block:: python
request_with_cookies = Request( request_with_cookies = Request(
url="http://www.example.com", url="http://www.example.com",
cookies={'currency': 'USD', 'country': 'UY'}, cookies={"currency": "USD", "country": "UY"},
) )
2. Using a list of dicts:: 2. Using a list of dicts:
.. code-block:: python
request_with_cookies = Request( request_with_cookies = Request(
url="http://www.example.com", url="http://www.example.com",
cookies=[ cookies=[
{ {
'name': 'currency', "name": "currency",
'value': 'USD', "value": "USD",
'domain': 'example.com', "domain": "example.com",
'path': '/currency', "path": "/currency",
}, },
], ],
) )
@ -114,12 +118,14 @@ Request objects
in :attr:`request.meta <scrapy.Request.meta>`. in :attr:`request.meta <scrapy.Request.meta>`.
Example of a request that sends manually-defined cookies and ignores Example of a request that sends manually-defined cookies and ignores
cookie storage:: cookie storage:
.. code-block:: python
Request( Request(
url="http://www.example.com", url="http://www.example.com",
cookies={'currency': 'USD', 'country': 'UY'}, cookies={"currency": "USD", "country": "UY"},
meta={'dont_merge_cookies': True}, meta={"dont_merge_cookies": True},
) )
For more info see :ref:`cookies-mw`. For more info see :ref:`cookies-mw`.
@ -259,11 +265,15 @@ The callback of a request is a function that will be called when the response
of that request is downloaded. The callback function will be called with the of that request is downloaded. The callback function will be called with the
downloaded :class:`Response` object as its first argument. downloaded :class:`Response` object as its first argument.
Example:: Example:
.. code-block:: python
def parse_page1(self, response): def parse_page1(self, response):
return scrapy.Request("http://www.example.com/some_page.html", return scrapy.Request(
callback=self.parse_page2) "http://www.example.com/some_page.html", callback=self.parse_page2
)
def parse_page2(self, response): def parse_page2(self, response):
# this would log http://www.example.com/some_page.html # this would log http://www.example.com/some_page.html
@ -274,15 +284,18 @@ functions so you can receive the arguments later, in the second callback.
The following example shows how to achieve this by using the The following example shows how to achieve this by using the
:attr:`Request.cb_kwargs` attribute: :attr:`Request.cb_kwargs` attribute:
:: .. code-block:: python
def parse(self, response): def parse(self, response):
request = scrapy.Request('http://www.example.com/index.html', request = scrapy.Request(
callback=self.parse_page2, "http://www.example.com/index.html",
cb_kwargs=dict(main_url=response.url)) callback=self.parse_page2,
request.cb_kwargs['foo'] = 'bar' # add more arguments for the callback cb_kwargs=dict(main_url=response.url),
)
request.cb_kwargs["foo"] = "bar" # add more arguments for the callback
yield request yield request
def parse_page2(self, response, main_url, foo): def parse_page2(self, response, main_url, foo):
yield dict( yield dict(
main_url=main_url, main_url=main_url,
@ -308,7 +321,9 @@ It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can
be used to track connection establishment timeouts, DNS errors etc. be used to track connection establishment timeouts, DNS errors etc.
Here's an example spider logging all errors and catching some specific Here's an example spider logging all errors and catching some specific
errors if needed:: errors if needed
.. code-block:: python
import scrapy import scrapy
@ -316,24 +331,28 @@ errors if needed::
from twisted.internet.error import DNSLookupError from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError, TCPTimedOutError from twisted.internet.error import TimeoutError, TCPTimedOutError
class ErrbackSpider(scrapy.Spider): class ErrbackSpider(scrapy.Spider):
name = "errback_example" name = "errback_example"
start_urls = [ start_urls = [
"http://www.httpbin.org/", # HTTP 200 expected "http://www.httpbin.org/", # HTTP 200 expected
"http://www.httpbin.org/status/404", # Not found error "http://www.httpbin.org/status/404", # Not found error
"http://www.httpbin.org/status/500", # server issue "http://www.httpbin.org/status/500", # server issue
"http://www.httpbin.org:12345/", # non-responding host, timeout expected "http://www.httpbin.org:12345/", # non-responding host, timeout expected
"https://example.invalid/", # DNS error expected "https://example.invalid/", # DNS error expected
] ]
def start_requests(self): def start_requests(self):
for u in self.start_urls: for u in self.start_urls:
yield scrapy.Request(u, callback=self.parse_httpbin, yield scrapy.Request(
errback=self.errback_httpbin, u,
dont_filter=True) callback=self.parse_httpbin,
errback=self.errback_httpbin,
dont_filter=True,
)
def parse_httpbin(self, response): def parse_httpbin(self, response):
self.logger.info('Got successful response from {}'.format(response.url)) self.logger.info("Got successful response from {}".format(response.url))
# do something useful here... # do something useful here...
def errback_httpbin(self, failure): def errback_httpbin(self, failure):
@ -347,16 +366,16 @@ errors if needed::
# these exceptions come from HttpError spider middleware # these exceptions come from HttpError spider middleware
# you can get the non-200 response # you can get the non-200 response
response = failure.value.response response = failure.value.response
self.logger.error('HttpError on %s', response.url) self.logger.error("HttpError on %s", response.url)
elif failure.check(DNSLookupError): elif failure.check(DNSLookupError):
# this is the original request # this is the original request
request = failure.request request = failure.request
self.logger.error('DNSLookupError on %s', request.url) self.logger.error("DNSLookupError on %s", request.url)
elif failure.check(TimeoutError, TCPTimedOutError): elif failure.check(TimeoutError, TCPTimedOutError):
request = failure.request request = failure.request
self.logger.error('TimeoutError on %s', request.url) self.logger.error("TimeoutError on %s", request.url)
.. _errback-cb_kwargs: .. _errback-cb_kwargs:
@ -367,21 +386,27 @@ Accessing additional data in errback functions
In case of a failure to process the request, you may be interested in In case of a failure to process the request, you may be interested in
accessing arguments to the callback functions so you can process further accessing arguments to the callback functions so you can process further
based on the arguments in the errback. The following example shows how to based on the arguments in the errback. The following example shows how to
achieve this by using ``Failure.request.cb_kwargs``:: achieve this by using ``Failure.request.cb_kwargs``
.. code-block:: python
def parse(self, response): def parse(self, response):
request = scrapy.Request('http://www.example.com/index.html', request = scrapy.Request(
callback=self.parse_page2, "http://www.example.com/index.html",
errback=self.errback_page2, callback=self.parse_page2,
cb_kwargs=dict(main_url=response.url)) errback=self.errback_page2,
cb_kwargs=dict(main_url=response.url),
)
yield request yield request
def parse_page2(self, response, main_url): def parse_page2(self, response, main_url):
pass pass
def errback_page2(self, failure): def errback_page2(self, failure):
yield dict( yield dict(
main_url=failure.request.cb_kwargs['main_url'], main_url=failure.request.cb_kwargs["main_url"],
) )
@ -528,18 +553,20 @@ in your :meth:`fingerprint` method implementation:
.. autofunction:: scrapy.utils.request.fingerprint .. autofunction:: scrapy.utils.request.fingerprint
For example, to take the value of a request header named ``X-ID`` into For example, to take the value of a request header named ``X-ID`` into
account:: account:
.. code-block:: python
# my_project/settings.py # my_project/settings.py
REQUEST_FINGERPRINTER_CLASS = 'my_project.utils.RequestFingerprinter' REQUEST_FINGERPRINTER_CLASS = "my_project.utils.RequestFingerprinter"
# my_project/utils.py # my_project/utils.py
from scrapy.utils.request import fingerprint from scrapy.utils.request import fingerprint
class RequestFingerprinter:
class RequestFingerprinter:
def fingerprint(self, request): def fingerprint(self, request):
return fingerprint(request, include_headers=['X-ID']) return fingerprint(request, include_headers=["X-ID"])
You can also write your own fingerprinting logic from scratch. You can also write your own fingerprinting logic from scratch.
@ -555,13 +582,16 @@ you use :class:`~weakref.WeakKeyDictionary` to cache request fingerprints:
references to them in your cache dictionary. references to them in your cache dictionary.
For example, to take into account only the URL of a request, without any prior For example, to take into account only the URL of a request, without any prior
URL canonicalization or taking the request method or body into account:: URL canonicalization or taking the request method or body into account:
.. code-block:: python
from hashlib import sha1 from hashlib import sha1
from weakref import WeakKeyDictionary from weakref import WeakKeyDictionary
from scrapy.utils.python import to_bytes from scrapy.utils.python import to_bytes
class RequestFingerprinter: class RequestFingerprinter:
cache = WeakKeyDictionary() cache = WeakKeyDictionary()
@ -577,21 +607,25 @@ If you need to be able to override the request fingerprinting for arbitrary
requests from your spider callbacks, you may implement a request fingerprinter requests from your spider callbacks, you may implement a request fingerprinter
that reads fingerprints from :attr:`request.meta <scrapy.http.Request.meta>` that reads fingerprints from :attr:`request.meta <scrapy.http.Request.meta>`
when available, and then falls back to when available, and then falls back to
:func:`scrapy.utils.request.fingerprint`. For example:: :func:`scrapy.utils.request.fingerprint`. For example:
.. code-block:: python
from scrapy.utils.request import fingerprint from scrapy.utils.request import fingerprint
class RequestFingerprinter:
class RequestFingerprinter:
def fingerprint(self, request): def fingerprint(self, request):
if 'fingerprint' in request.meta: if "fingerprint" in request.meta:
return request.meta['fingerprint'] return request.meta["fingerprint"]
return fingerprint(request) return fingerprint(request)
If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6 If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6
without using the deprecated ``'2.6'`` value of the without using the deprecated ``'2.6'`` value of the
:setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following :setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following
request fingerprinter:: request fingerprinter:
.. code-block:: python
from hashlib import sha1 from hashlib import sha1
from weakref import WeakKeyDictionary from weakref import WeakKeyDictionary
@ -599,6 +633,7 @@ request fingerprinter::
from scrapy.utils.python import to_bytes from scrapy.utils.python import to_bytes
from w3lib.url import canonicalize_url from w3lib.url import canonicalize_url
class RequestFingerprinter: class RequestFingerprinter:
cache = WeakKeyDictionary() cache = WeakKeyDictionary()
@ -608,7 +643,7 @@ request fingerprinter::
fp = sha1() fp = sha1()
fp.update(to_bytes(request.method)) fp.update(to_bytes(request.method))
fp.update(to_bytes(canonicalize_url(request.url))) fp.update(to_bytes(canonicalize_url(request.url)))
fp.update(request.body or b'') fp.update(request.body or b"")
self.cache[request] = fp.digest() self.cache[request] = fp.digest()
return self.cache[request] return self.cache[request]
@ -737,7 +772,9 @@ Stopping the download of a Response
Raising a :exc:`~scrapy.exceptions.StopDownload` exception from a handler for the Raising a :exc:`~scrapy.exceptions.StopDownload` exception from a handler for the
:class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received` :class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received`
signals will stop the download of a given response. See the following example:: signals will stop the download of a given response. See the following example:
.. code-block:: python
import scrapy import scrapy
@ -749,7 +786,9 @@ signals will stop the download of a given response. See the following example::
@classmethod @classmethod
def from_crawler(cls, crawler): def from_crawler(cls, crawler):
spider = super().from_crawler(crawler) spider = super().from_crawler(crawler)
crawler.signals.connect(spider.on_bytes_received, signal=scrapy.signals.bytes_received) crawler.signals.connect(
spider.on_bytes_received, signal=scrapy.signals.bytes_received
)
return spider return spider
def parse(self, response): def parse(self, response):
@ -878,11 +917,17 @@ Using FormRequest to send data via HTTP POST
If you want to simulate a HTML Form POST in your spider and send a couple of If you want to simulate a HTML Form POST in your spider and send a couple of
key-value fields, you can return a :class:`FormRequest` object (from your key-value fields, you can return a :class:`FormRequest` object (from your
spider) like this:: spider) like this:
return [FormRequest(url="http://www.example.com/post/action", .. code-block:: python
formdata={'name': 'John Doe', 'age': '27'},
callback=self.after_post)] return [
FormRequest(
url="http://www.example.com/post/action",
formdata={"name": "John Doe", "age": "27"},
callback=self.after_post,
)
]
.. _topics-request-response-ref-request-userlogin: .. _topics-request-response-ref-request-userlogin:
@ -894,25 +939,28 @@ type="hidden">`` elements, such as session related data or authentication
tokens (for login pages). When scraping, you'll want these fields to be tokens (for login pages). When scraping, you'll want these fields to be
automatically pre-populated and only override a couple of them, such as the automatically pre-populated and only override a couple of them, such as the
user name and password. You can use the :meth:`FormRequest.from_response` user name and password. You can use the :meth:`FormRequest.from_response`
method for this job. Here's an example spider which uses it:: method for this job. Here's an example spider which uses it:
.. code-block:: python
import scrapy import scrapy
def authentication_failed(response): def authentication_failed(response):
# TODO: Check the contents of the response and return True if it failed # TODO: Check the contents of the response and return True if it failed
# or False if it succeeded. # or False if it succeeded.
pass pass
class LoginSpider(scrapy.Spider): class LoginSpider(scrapy.Spider):
name = 'example.com' name = "example.com"
start_urls = ['http://www.example.com/users/login.php'] start_urls = ["http://www.example.com/users/login.php"]
def parse(self, response): def parse(self, response):
return scrapy.FormRequest.from_response( return scrapy.FormRequest.from_response(
response, response,
formdata={'username': 'john', 'password': 'secret'}, formdata={"username": "john", "password": "secret"},
callback=self.after_login callback=self.after_login,
) )
def after_login(self, response): def after_login(self, response):
@ -952,13 +1000,15 @@ dealing with JSON requests.
JsonRequest usage example JsonRequest usage example
------------------------- -------------------------
Sending a JSON POST request with a JSON payload:: Sending a JSON POST request with a JSON payload:
.. code-block:: python
data = { data = {
'name1': 'value1', "name1": "value1",
'name2': 'value2', "name2": "value2",
} }
yield JsonRequest(url='http://www.example.com/post/action', data=data) yield JsonRequest(url="http://www.example.com/post/action", data=data)
Response objects Response objects

33
docs/topics/selectors.rst
View File

@ -981,25 +981,34 @@ Selector examples on HTML response
Here are some :class:`Selector` examples to illustrate several concepts. Here are some :class:`Selector` examples to illustrate several concepts.
In all cases, we assume there is already a :class:`Selector` instantiated with In all cases, we assume there is already a :class:`Selector` instantiated with
a :class:`~scrapy.http.HtmlResponse` object like this:: a :class:`~scrapy.http.HtmlResponse` object like this:
.. code-block:: python
sel = Selector(html_response) sel = Selector(html_response)
1. Select all ``<h1>`` elements from an HTML response body, returning a list of 1. Select all ``<h1>`` elements from an HTML response body, returning a list of
:class:`Selector` objects (i.e. a :class:`SelectorList` object):: :class:`Selector` objects (i.e. a :class:`SelectorList` object):
.. code-block:: python
sel.xpath("//h1") sel.xpath("//h1")
2. Extract the text of all ``<h1>`` elements from an HTML response body, 2. Extract the text of all ``<h1>`` elements from an HTML response body,
returning a list of strings:: returning a list of strings:
sel.xpath("//h1").getall() # this includes the h1 tag .. code-block:: python
sel.xpath("//h1").getall() # this includes the h1 tag
sel.xpath("//h1/text()").getall() # this excludes the h1 tag sel.xpath("//h1/text()").getall() # this excludes the h1 tag
3. Iterate over all ``<p>`` tags and print their class attribute:: 3. Iterate over all ``<p>`` tags and print their class attribute:
.. code-block:: python
for node in sel.xpath("//p"): for node in sel.xpath("//p"):
print(node.attrib['class']) print(node.attrib["class"])
.. _selector-examples-xml: .. _selector-examples-xml:
@ -1008,17 +1017,23 @@ Selector examples on XML response
--------------------------------- ---------------------------------
Here are some examples to illustrate concepts for :class:`Selector` objects Here are some examples to illustrate concepts for :class:`Selector` objects
instantiated with an :class:`~scrapy.http.XmlResponse` object:: instantiated with an :class:`~scrapy.http.XmlResponse` object:
.. code-block:: python
sel = Selector(xml_response) sel = Selector(xml_response)
1. Select all ``<product>`` elements from an XML response body, returning a list 1. Select all ``<product>`` elements from an XML response body, returning a list
of :class:`Selector` objects (i.e. a :class:`SelectorList` object):: of :class:`Selector` objects (i.e. a :class:`SelectorList` object):
.. code-block:: python
sel.xpath("//product") sel.xpath("//product")
2. Extract all prices from a `Google Base XML feed`_ which requires registering 2. Extract all prices from a `Google Base XML feed`_ which requires registering
a namespace:: a namespace:
.. code-block:: python
sel.register_namespace("g", "http://base.google.com/ns/1.0") sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall() sel.xpath("//g:price").getall()

202
docs/topics/settings.rst
View File

@ -67,13 +67,15 @@ Example::
Spiders (See the :ref:`topics-spiders` chapter for reference) can define their Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
own settings that will take precedence and override the project ones. They can own settings that will take precedence and override the project ones. They can
do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute:: do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute:
.. code-block:: python
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
custom_settings = { custom_settings = {
'SOME_SETTING': 'some value', "SOME_SETTING": "some value",
} }
3. Project settings module 3. Project settings module
@ -115,14 +117,17 @@ class or a function, there are two different ways you can specify that object:
- As the object itself - As the object itself
For example:: For example:
.. code-block:: python
from mybot.pipelines.validate import ValidateMyItem from mybot.pipelines.validate import ValidateMyItem
ITEM_PIPELINES = { ITEM_PIPELINES = {
# passing the classname... # passing the classname...
ValidateMyItem: 300, ValidateMyItem: 300,
# ...equals passing the class path # ...equals passing the class path
'mybot.pipelines.validate.ValidateMyItem': 300, "mybot.pipelines.validate.ValidateMyItem": 300,
} }
.. note:: Passing non-callable objects is not supported. .. note:: Passing non-callable objects is not supported.
@ -133,11 +138,13 @@ How to access settings
.. highlight:: python .. highlight:: python
In a spider, the settings are available through ``self.settings``:: In a spider, the settings are available through ``self.settings``:
.. code-block:: python
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
start_urls = ['http://example.com'] start_urls = ["http://example.com"]
def parse(self, response): def parse(self, response):
print(f"Existing settings: {self.settings.attributes.keys()}") print(f"Existing settings: {self.settings.attributes.keys()}")
@ -150,7 +157,9 @@ In a spider, the settings are available through ``self.settings``::
Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings` Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
attribute of the Crawler that is passed to ``from_crawler`` method in attribute of the Crawler that is passed to ``from_crawler`` method in
extensions, middlewares and item pipelines:: extensions, middlewares and item pipelines:
.. code-block:: python
class MyExtension: class MyExtension:
def __init__(self, log_is_enabled=False): def __init__(self, log_is_enabled=False):
@ -160,7 +169,7 @@ extensions, middlewares and item pipelines::
@classmethod @classmethod
def from_crawler(cls, crawler): def from_crawler(cls, crawler):
settings = crawler.settings settings = crawler.settings
return cls(settings.getbool('LOG_ENABLED')) return cls(settings.getbool("LOG_ENABLED"))
The settings object can be used like a dict (e.g., The settings object can be used like a dict (e.g.,
``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting ``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting
@ -365,11 +374,13 @@ Scrapy shell <topics-shell>`.
DEFAULT_REQUEST_HEADERS DEFAULT_REQUEST_HEADERS
----------------------- -----------------------
Default:: Default:
.. code-block:: python
{ {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
'Accept-Language': 'en', "Accept-Language": "en",
} }
The default headers used for Scrapy HTTP Requests. They're populated in the The default headers used for Scrapy HTTP Requests. They're populated in the
@ -404,9 +415,11 @@ Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``
An integer that is used to adjust the :attr:`~scrapy.Request.priority` of An integer that is used to adjust the :attr:`~scrapy.Request.priority` of
a :class:`~scrapy.Request` based on its depth. a :class:`~scrapy.Request` based on its depth.
The priority of a request is adjusted as follows:: The priority of a request is adjusted as follows:
request.priority = request.priority - ( depth * DEPTH_PRIORITY ) .. code-block:: python
request.priority = request.priority - (depth * DEPTH_PRIORITY)
As depth increases, positive values of ``DEPTH_PRIORITY`` decrease request As depth increases, positive values of ``DEPTH_PRIORITY`` decrease request
priority (BFO), while negative values increase request priority (DFO). See priority (BFO), while negative values increase request priority (DFO). See
@ -595,23 +608,25 @@ orders. For more info see :ref:`topics-downloader-middleware-setting`.
DOWNLOADER_MIDDLEWARES_BASE DOWNLOADER_MIDDLEWARES_BASE
--------------------------- ---------------------------
Default:: Default:
.. code-block:: python
{ {
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, "scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, "scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350, "scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400, "scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500, "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550, "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560, "scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware": 560,
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700, "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750, "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
'scrapy.downloadermiddlewares.stats.DownloaderStats': 850, "scrapy.downloadermiddlewares.stats.DownloaderStats": 850,
'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, "scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900,
} }
A dict containing the downloader middlewares enabled by default in Scrapy. Low A dict containing the downloader middlewares enabled by default in Scrapy. Low
@ -687,15 +702,17 @@ See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.
DOWNLOAD_HANDLERS_BASE DOWNLOAD_HANDLERS_BASE
---------------------- ----------------------
Default:: Default:
.. code-block:: python
{ {
'data': 'scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler', "data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",
'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler', "file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",
'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', "http": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",
'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', "https": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",
's3': 'scrapy.core.downloader.handlers.s3.S3DownloadHandler', "s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
'ftp': 'scrapy.core.downloader.handlers.ftp.FTPDownloadHandler', "ftp": "scrapy.core.downloader.handlers.ftp.FTPDownloadHandler",
} }
@ -705,10 +722,12 @@ You should never modify this setting in your project, modify
You can disable any of these download handlers by assigning ``None`` to their You can disable any of these download handlers by assigning ``None`` to their
URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
handler (without replacement), place this in your ``settings.py``:: handler (without replacement), place this in your ``settings.py``:
.. code-block:: python
DOWNLOAD_HANDLERS = { DOWNLOAD_HANDLERS = {
'ftp': None, "ftp": None,
} }
.. _http2: .. _http2:
@ -718,10 +737,12 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2:
#. Install ``Twisted[http2]>=17.9.0`` to install the packages required to #. Install ``Twisted[http2]>=17.9.0`` to install the packages required to
enable HTTP/2 support in Twisted. enable HTTP/2 support in Twisted.
#. Update :setting:`DOWNLOAD_HANDLERS` as follows:: #. Update :setting:`DOWNLOAD_HANDLERS` as follows:
.. code-block:: python
DOWNLOAD_HANDLERS = { DOWNLOAD_HANDLERS = {
'https': 'scrapy.core.downloader.handlers.http2.H2DownloadHandler', "https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
} }
.. warning:: .. warning::
@ -890,18 +911,20 @@ A dict containing the extensions enabled in your project, and their orders.
EXTENSIONS_BASE EXTENSIONS_BASE
--------------- ---------------
Default:: Default:
.. code-block:: python
{ {
'scrapy.extensions.corestats.CoreStats': 0, "scrapy.extensions.corestats.CoreStats": 0,
'scrapy.extensions.telnet.TelnetConsole': 0, "scrapy.extensions.telnet.TelnetConsole": 0,
'scrapy.extensions.memusage.MemoryUsage': 0, "scrapy.extensions.memusage.MemoryUsage": 0,
'scrapy.extensions.memdebug.MemoryDebugger': 0, "scrapy.extensions.memdebug.MemoryDebugger": 0,
'scrapy.extensions.closespider.CloseSpider': 0, "scrapy.extensions.closespider.CloseSpider": 0,
'scrapy.extensions.feedexport.FeedExporter': 0, "scrapy.extensions.feedexport.FeedExporter": 0,
'scrapy.extensions.logstats.LogStats': 0, "scrapy.extensions.logstats.LogStats": 0,
'scrapy.extensions.spiderstate.SpiderState': 0, "scrapy.extensions.spiderstate.SpiderState": 0,
'scrapy.extensions.throttle.AutoThrottle': 0, "scrapy.extensions.throttle.AutoThrottle": 0,
} }
A dict containing the extensions available by default in Scrapy, and their A dict containing the extensions available by default in Scrapy, and their
@ -988,11 +1011,13 @@ A dict containing the item pipelines to use, and their orders. Order values are
arbitrary, but it is customary to define them in the 0-1000 range. Lower orders arbitrary, but it is customary to define them in the 0-1000 range. Lower orders
process before higher orders. process before higher orders.
Example:: Example:
.. code-block:: python
ITEM_PIPELINES = { ITEM_PIPELINES = {
'mybot.pipelines.validate.ValidateMyItem': 300, "mybot.pipelines.validate.ValidateMyItem": 300,
'mybot.pipelines.validate.StoreMyItem': 800, "mybot.pipelines.validate.StoreMyItem": 800,
} }
.. setting:: ITEM_PIPELINES_BASE .. setting:: ITEM_PIPELINES_BASE
@ -1417,12 +1442,14 @@ testing spiders. For more info see :ref:`topics-contracts`.
SPIDER_CONTRACTS_BASE SPIDER_CONTRACTS_BASE
--------------------- ---------------------
Default:: Default:
.. code-block:: python
{ {
'scrapy.contracts.default.UrlContract' : 1, "scrapy.contracts.default.UrlContract": 1,
'scrapy.contracts.default.ReturnsContract': 2, "scrapy.contracts.default.ReturnsContract": 2,
'scrapy.contracts.default.ScrapesContract': 3, "scrapy.contracts.default.ScrapesContract": 3,
} }
A dict containing the Scrapy contracts enabled by default in Scrapy. You should A dict containing the Scrapy contracts enabled by default in Scrapy. You should
@ -1431,10 +1458,12 @@ instead. For more info see :ref:`topics-contracts`.
You can disable any of these contracts by assigning ``None`` to their class You can disable any of these contracts by assigning ``None`` to their class
path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in
``ScrapesContract``, place this in your ``settings.py``:: ``ScrapesContract``, place this in your ``settings.py``:
.. code-block:: python
SPIDER_CONTRACTS = { SPIDER_CONTRACTS = {
'scrapy.contracts.default.ScrapesContract': None, "scrapy.contracts.default.ScrapesContract": None,
} }
.. setting:: SPIDER_LOADER_CLASS .. setting:: SPIDER_LOADER_CLASS
@ -1483,14 +1512,16 @@ orders. For more info see :ref:`topics-spider-middleware-setting`.
SPIDER_MIDDLEWARES_BASE SPIDER_MIDDLEWARES_BASE
----------------------- -----------------------
Default:: Default:
.. code-block:: python
{ {
'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50, "scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500, "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,
'scrapy.spidermiddlewares.referer.RefererMiddleware': 700, "scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800, "scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
'scrapy.spidermiddlewares.depth.DepthMiddleware': 900, "scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
} }
A dict containing the spider middlewares enabled by default in Scrapy, and A dict containing the spider middlewares enabled by default in Scrapy, and
@ -1506,9 +1537,11 @@ Default: ``[]``
A list of modules where Scrapy will look for spiders. A list of modules where Scrapy will look for spiders.
Example:: Example:
SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev'] .. code-block:: python
SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"]
.. setting:: STATS_CLASS .. setting:: STATS_CLASS
@ -1597,60 +1630,65 @@ If a reactor is already installed,
third-party libraries will make Scrapy raise :exc:`Exception` when third-party libraries will make Scrapy raise :exc:`Exception` when
it checks which reactor is installed. it checks which reactor is installed.
In order to use the reactor installed by Scrapy:: In order to use the reactor installed by Scrapy:
.. code-block:: python
import scrapy import scrapy
from twisted.internet import reactor from twisted.internet import reactor
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = 'quotes' name = "quotes"
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop('timeout', '60')) self.timeout = int(kwargs.pop("timeout", "60"))
super(QuotesSpider, self).__init__(*args, **kwargs) super(QuotesSpider, self).__init__(*args, **kwargs)
def start_requests(self): def start_requests(self):
reactor.callLater(self.timeout, self.stop) reactor.callLater(self.timeout, self.stop)
urls = ['https://quotes.toscrape.com/page/1'] urls = ["https://quotes.toscrape.com/page/1"]
for url in urls: for url in urls:
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield {'text': quote.css('span.text::text').get()} yield {"text": quote.css("span.text::text").get()}
def stop(self): def stop(self):
self.crawler.engine.close_spider(self, 'timeout') self.crawler.engine.close_spider(self, "timeout")
which raises :exc:`Exception`, becomes:: which raises :exc:`Exception`, becomes:
.. code-block:: python
import scrapy import scrapy
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = 'quotes' name = "quotes"
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop('timeout', '60')) self.timeout = int(kwargs.pop("timeout", "60"))
super(QuotesSpider, self).__init__(*args, **kwargs) super(QuotesSpider, self).__init__(*args, **kwargs)
def start_requests(self): def start_requests(self):
from twisted.internet import reactor from twisted.internet import reactor
reactor.callLater(self.timeout, self.stop) reactor.callLater(self.timeout, self.stop)
urls = ['https://quotes.toscrape.com/page/1'] urls = ["https://quotes.toscrape.com/page/1"]
for url in urls: for url in urls:
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield {'text': quote.css('span.text::text').get()} yield {"text": quote.css("span.text::text").get()}
def stop(self): def stop(self):
self.crawler.engine.close_spider(self, 'timeout') self.crawler.engine.close_spider(self, "timeout")
The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which

5
docs/topics/shell.rst
View File

@ -242,7 +242,9 @@ getting there.
This can be achieved by using the ``scrapy.shell.inspect_response`` function. This can be achieved by using the ``scrapy.shell.inspect_response`` function.
Here's an example of how you would call it from your spider:: Here's an example of how you would call it from your spider:
.. code-block:: python
import scrapy import scrapy
@ -259,6 +261,7 @@ Here's an example of how you would call it from your spider::
# We want to inspect one specific response. # We want to inspect one specific response.
if ".org" in response.url: if ".org" in response.url:
from scrapy.shell import inspect_response from scrapy.shell import inspect_response
inspect_response(response, self) inspect_response(response, self)
# Rest of parsing code. # Rest of parsing code.

31
docs/topics/signals.rst
View File

@ -16,7 +16,9 @@ deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the You can connect to signals (or send your own) through the
:ref:`topics-api-signals`. :ref:`topics-api-signals`.
Here is a simple example showing how you can catch signals and perform some action:: Here is a simple example showing how you can catch signals and perform some action:
.. code-block:: python
from scrapy import signals from scrapy import signals
from scrapy import Spider from scrapy import Spider
@ -30,17 +32,14 @@ Here is a simple example showing how you can catch signals and perform some acti
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
] ]
@classmethod @classmethod
def from_crawler(cls, crawler, *args, **kwargs): def from_crawler(cls, crawler, *args, **kwargs):
spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs) spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed) crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider return spider
def spider_closed(self, spider): def spider_closed(self, spider):
spider.logger.info('Spider closed: %s', spider.name) spider.logger.info("Spider closed: %s", spider.name)
def parse(self, response): def parse(self, response):
pass pass
@ -56,11 +55,13 @@ you to run asynchronous code that does not block Scrapy. If a signal
handler returns one of these objects, Scrapy waits for that asynchronous handler returns one of these objects, Scrapy waits for that asynchronous
operation to finish. operation to finish.
Let's take an example using :ref:`coroutines <topics-coroutines>`:: Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. code-block:: python
class SignalSpider(scrapy.Spider): class SignalSpider(scrapy.Spider):
name = 'signals' name = "signals"
start_urls = ['https://quotes.toscrape.com/page/1/'] start_urls = ["https://quotes.toscrape.com/page/1/"]
@classmethod @classmethod
def from_crawler(cls, crawler, *args, **kwargs): def from_crawler(cls, crawler, *args, **kwargs):
@ -71,19 +72,19 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`::
async def item_scraped(self, item): async def item_scraped(self, item):
# Send the scraped item to the server # Send the scraped item to the server
response = await treq.post( response = await treq.post(
'http://example.com/post', "http://example.com/post",
json.dumps(item).encode('ascii'), json.dumps(item).encode("ascii"),
headers={b'Content-Type': [b'application/json']} headers={b"Content-Type": [b"application/json"]},
) )
return response return response
def parse(self, response): def parse(self, response):
for quote in response.css('div.quote'): for quote in response.css("div.quote"):
yield { yield {
'text': quote.css('span.text::text').get(), "text": quote.css("span.text::text").get(),
'author': quote.css('small.author::text').get(), "author": quote.css("small.author::text").get(),
'tags': quote.css('div.tags a.tag::text').getall(), "tags": quote.css("div.tags a.tag::text").getall(),
} }
See the :ref:`topics-signals-ref` below to know which signals support See the :ref:`topics-signals-ref` below to know which signals support

18
docs/topics/spider-middleware.rst
View File

@ -18,10 +18,12 @@ To activate a spider middleware component, add it to the
:setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the :setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class path and their values are the middleware orders. middleware class path and their values are the middleware orders.
Here's an example:: Here's an example:
.. code-block:: python
SPIDER_MIDDLEWARES = { SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543, "myproject.middlewares.CustomSpiderMiddleware": 543,
} }
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the The :setting:`SPIDER_MIDDLEWARES` setting is merged with the
@ -44,11 +46,13 @@ previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware (the ones defined in If you want to disable a builtin middleware (the ones defined in
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it :setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
value. For example, if you want to disable the off-site middleware:: value. For example, if you want to disable the off-site middleware:
.. code-block:: python
SPIDER_MIDDLEWARES = { SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543, "myproject.middlewares.CustomSpiderMiddleware": 543,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None, "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
} }
Finally, keep in mind that some middlewares may need to be enabled through a Finally, keep in mind that some middlewares may need to be enabled through a
@ -261,7 +265,9 @@ specify which response codes the spider is able to handle using the
:setting:`HTTPERROR_ALLOWED_CODES` setting. :setting:`HTTPERROR_ALLOWED_CODES` setting.
For example, if you want your spider to handle 404 responses you can do For example, if you want your spider to handle 404 responses you can do
this:: this:
.. code-block:: python
class MySpider(CrawlSpider): class MySpider(CrawlSpider):
handle_httpstatus_list = [404] handle_httpstatus_list = [404]

246
docs/topics/spiders.rst
View File

@ -157,15 +157,21 @@ scrapy.Spider
If you want to change the Requests used to start scraping a domain, this is If you want to change the Requests used to start scraping a domain, this is
the method to override. For example, if you need to start by logging in using the method to override. For example, if you need to start by logging in using
a POST request, you could do:: a POST request, you could do:
.. code-block:: python
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
def start_requests(self): def start_requests(self):
return [scrapy.FormRequest("http://www.example.com/login", return [
formdata={'user': 'john', 'pass': 'secret'}, scrapy.FormRequest(
callback=self.logged_in)] "http://www.example.com/login",
formdata={"user": "john", "pass": "secret"},
callback=self.logged_in,
)
]
def logged_in(self, response): def logged_in(self, response):
# here you would extract links to follow and return Requests for # here you would extract links to follow and return Requests for
@ -200,63 +206,71 @@ scrapy.Spider
Called when the spider closes. This method provides a shortcut to Called when the spider closes. This method provides a shortcut to
signals.connect() for the :signal:`spider_closed` signal. signals.connect() for the :signal:`spider_closed` signal.
Let's see an example:: Let's see an example:
.. code-block:: python
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
start_urls = [ start_urls = [
'http://www.example.com/1.html', "http://www.example.com/1.html",
'http://www.example.com/2.html', "http://www.example.com/2.html",
'http://www.example.com/3.html', "http://www.example.com/3.html",
] ]
def parse(self, response): def parse(self, response):
self.logger.info('A response from %s just arrived!', response.url) self.logger.info("A response from %s just arrived!", response.url)
Return multiple Requests and items from a single callback:: Return multiple Requests and items from a single callback:
.. code-block:: python
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
start_urls = [ start_urls = [
'http://www.example.com/1.html', "http://www.example.com/1.html",
'http://www.example.com/2.html', "http://www.example.com/2.html",
'http://www.example.com/3.html', "http://www.example.com/3.html",
] ]
def parse(self, response): def parse(self, response):
for h3 in response.xpath('//h3').getall(): for h3 in response.xpath("//h3").getall():
yield {"title": h3} yield {"title": h3}
for href in response.xpath('//a/@href').getall(): for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse) yield scrapy.Request(response.urljoin(href), self.parse)
Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly; Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
to give data more structure you can use :class:`~scrapy.Item` objects:: to give data more structure you can use :class:`~scrapy.Item` objects:
.. code-block:: python
import scrapy import scrapy
from myproject.items import MyItem from myproject.items import MyItem
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
def start_requests(self): def start_requests(self):
yield scrapy.Request('http://www.example.com/1.html', self.parse) yield scrapy.Request("http://www.example.com/1.html", self.parse)
yield scrapy.Request('http://www.example.com/2.html', self.parse) yield scrapy.Request("http://www.example.com/2.html", self.parse)
yield scrapy.Request('http://www.example.com/3.html', self.parse) yield scrapy.Request("http://www.example.com/3.html", self.parse)
def parse(self, response): def parse(self, response):
for h3 in response.xpath('//h3').getall(): for h3 in response.xpath("//h3").getall():
yield MyItem(title=h3) yield MyItem(title=h3)
for href in response.xpath('//a/@href').getall(): for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse) yield scrapy.Request(response.urljoin(href), self.parse)
.. _spiderargs: .. _spiderargs:
@ -274,34 +288,42 @@ Spider arguments are passed through the :command:`crawl` command using the
scrapy crawl myspider -a category=electronics scrapy crawl myspider -a category=electronics
Spiders can access arguments in their `__init__` methods:: Spiders can access arguments in their `__init__` methods:
.. code-block:: python
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
def __init__(self, category=None, *args, **kwargs): def __init__(self, category=None, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs) super(MySpider, self).__init__(*args, **kwargs)
self.start_urls = [f'http://www.example.com/categories/{category}'] self.start_urls = [f"http://www.example.com/categories/{category}"]
# ... # ...
The default `__init__` method will take any spider arguments The default `__init__` method will take any spider arguments
and copy them to the spider as attributes. and copy them to the spider as attributes.
The above example can also be written as follows:: The above example can also be written as follows:
.. code-block:: python
import scrapy import scrapy
class MySpider(scrapy.Spider): class MySpider(scrapy.Spider):
name = 'myspider' name = "myspider"
def start_requests(self): def start_requests(self):
yield scrapy.Request(f'http://www.example.com/categories/{self.category}') yield scrapy.Request(f"http://www.example.com/categories/{self.category}")
If you are :ref:`running Scrapy from a script <run-from-script>`, you can If you are :ref:`running Scrapy from a script <run-from-script>`, you can
specify spider arguments when calling specify spider arguments when calling
:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or :class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:: :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
.. code-block:: python
process = CrawlerProcess() process = CrawlerProcess()
process.crawl(MySpider, category="electronics") process.crawl(MySpider, category="electronics")
@ -337,10 +359,13 @@ common scraping cases, like following all links on a site based on certain
rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed. rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.
For the examples used in the following spiders, we'll assume you have a project For the examples used in the following spiders, we'll assume you have a project
with a ``TestItem`` declared in a ``myproject.items`` module:: with a ``TestItem`` declared in a ``myproject.items`` module:
.. code-block:: python
import scrapy import scrapy
class TestItem(scrapy.Item): class TestItem(scrapy.Item):
id = scrapy.Field() id = scrapy.Field()
name = scrapy.Field() name = scrapy.Field()
@ -436,38 +461,46 @@ Crawling rules
CrawlSpider example CrawlSpider example
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
Let's now take a look at an example CrawlSpider with rules:: Let's now take a look at an example CrawlSpider with rules:
.. code-block:: python
import scrapy import scrapy
from scrapy.spiders import CrawlSpider, Rule from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor from scrapy.linkextractors import LinkExtractor
class MySpider(CrawlSpider): class MySpider(CrawlSpider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
start_urls = ['http://www.example.com'] start_urls = ["http://www.example.com"]
rules = ( rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php') # Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default). # and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))), Rule(LinkExtractor(allow=("category\.php",), deny=("subsection\.php",))),
# Extract links matching 'item.php' and parse them with the spider's method parse_item # Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'), Rule(LinkExtractor(allow=("item\.php",)), callback="parse_item"),
) )
def parse_item(self, response): def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url) self.logger.info("Hi, this is an item page! %s", response.url)
item = scrapy.Item() item = scrapy.Item()
item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)")
item['name'] = response.xpath('//td[@id="item_name"]/text()').get() item["name"] = response.xpath('//td[@id="item_name"]/text()').get()
item['description'] = response.xpath('//td[@id="item_description"]/text()').get() item["description"] = response.xpath(
item['link_text'] = response.meta['link_text'] '//td[@id="item_description"]/text()'
).get()
item["link_text"] = response.meta["link_text"]
url = response.xpath('//td[@id="additional_data"]/@href').get() url = response.xpath('//td[@id="additional_data"]/@href').get()
return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item)) return response.follow(
url, self.parse_additional_page, cb_kwargs=dict(item=item)
)
def parse_additional_page(self, response, item): def parse_additional_page(self, response, item):
item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get() item["additional_data"] = response.xpath(
'//p[@id="additional_data"]/text()'
).get()
return item return item
@ -568,25 +601,30 @@ XMLFeedSpider
XMLFeedSpider example XMLFeedSpider example
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
These spiders are pretty easy to use, let's have a look at one example:: These spiders are pretty easy to use, let's have a look at one example:
.. code-block:: python
from scrapy.spiders import XMLFeedSpider from scrapy.spiders import XMLFeedSpider
from myproject.items import TestItem from myproject.items import TestItem
class MySpider(XMLFeedSpider): class MySpider(XMLFeedSpider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
start_urls = ['http://www.example.com/feed.xml'] start_urls = ["http://www.example.com/feed.xml"]
iterator = 'iternodes' # This is actually unnecessary, since it's the default value iterator = "iternodes" # This is actually unnecessary, since it's the default value
itertag = 'item' itertag = "item"
def parse_node(self, response, node): def parse_node(self, response, node):
self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.getall())) self.logger.info(
"Hi, this is a <%s> node!: %s", self.itertag, "".join(node.getall())
)
item = TestItem() item = TestItem()
item['id'] = node.xpath('@id').get() item["id"] = node.xpath("@id").get()
item['name'] = node.xpath('name').get() item["name"] = node.xpath("name").get()
item['description'] = node.xpath('description').get() item["description"] = node.xpath("description").get()
return item return item
Basically what we did up there was to create a spider that downloads a feed from Basically what we did up there was to create a spider that downloads a feed from
@ -627,26 +665,29 @@ CSVFeedSpider example
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
Let's see an example similar to the previous one, but using a Let's see an example similar to the previous one, but using a
:class:`CSVFeedSpider`:: :class:`CSVFeedSpider`:
.. code-block:: python
from scrapy.spiders import CSVFeedSpider from scrapy.spiders import CSVFeedSpider
from myproject.items import TestItem from myproject.items import TestItem
class MySpider(CSVFeedSpider): class MySpider(CSVFeedSpider):
name = 'example.com' name = "example.com"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
start_urls = ['http://www.example.com/feed.csv'] start_urls = ["http://www.example.com/feed.csv"]
delimiter = ';' delimiter = ";"
quotechar = "'" quotechar = "'"
headers = ['id', 'name', 'description'] headers = ["id", "name", "description"]
def parse_row(self, response, row): def parse_row(self, response, row):
self.logger.info('Hi, this is a row!: %r', row) self.logger.info("Hi, this is a row!: %r", row)
item = TestItem() item = TestItem()
item['id'] = row['id'] item["id"] = row["id"]
item['name'] = row['name'] item["name"] = row["name"]
item['description'] = row['description'] item["description"] = row["description"]
return item return item
@ -728,19 +769,22 @@ SitemapSpider
<lastmod>2005-01-01</lastmod> <lastmod>2005-01-01</lastmod>
</url> </url>
We can define a ``sitemap_filter`` function to filter ``entries`` by date:: We can define a ``sitemap_filter`` function to filter ``entries`` by date:
.. code-block:: python
from datetime import datetime from datetime import datetime
from scrapy.spiders import SitemapSpider from scrapy.spiders import SitemapSpider
class FilteredSitemapSpider(SitemapSpider): class FilteredSitemapSpider(SitemapSpider):
name = 'filtered_sitemap_spider' name = "filtered_sitemap_spider"
allowed_domains = ['example.com'] allowed_domains = ["example.com"]
sitemap_urls = ['http://example.com/sitemap.xml'] sitemap_urls = ["http://example.com/sitemap.xml"]
def sitemap_filter(self, entries): def sitemap_filter(self, entries):
for entry in entries: for entry in entries:
date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d') date_time = datetime.strptime(entry["lastmod"], "%Y-%m-%d")
if date_time.year >= 2005: if date_time.year >= 2005:
yield entry yield entry
@ -765,60 +809,72 @@ SitemapSpider examples
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
Simplest example: process all urls discovered through sitemaps using the Simplest example: process all urls discovered through sitemaps using the
``parse`` callback:: ``parse`` callback:
.. code-block:: python
from scrapy.spiders import SitemapSpider from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider): class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/sitemap.xml'] sitemap_urls = ["http://www.example.com/sitemap.xml"]
def parse(self, response): def parse(self, response):
pass # ... scrape item here ... pass # ... scrape item here ...
Process some urls with certain callback and other urls with a different Process some urls with certain callback and other urls with a different
callback:: callback:
.. code-block:: python
from scrapy.spiders import SitemapSpider from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider): class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/sitemap.xml'] sitemap_urls = ["http://www.example.com/sitemap.xml"]
sitemap_rules = [ sitemap_rules = [
('/product/', 'parse_product'), ("/product/", "parse_product"),
('/category/', 'parse_category'), ("/category/", "parse_category"),
] ]
def parse_product(self, response): def parse_product(self, response):
pass # ... scrape product ... pass # ... scrape product ...
def parse_category(self, response): def parse_category(self, response):
pass # ... scrape category ... pass # ... scrape category ...
Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps
whose url contains ``/sitemap_shop``:: whose url contains ``/sitemap_shop``:
.. code-block:: python
from scrapy.spiders import SitemapSpider from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider): class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/robots.txt'] sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [ sitemap_rules = [
('/shop/', 'parse_shop'), ("/shop/", "parse_shop"),
] ]
sitemap_follow = ['/sitemap_shops'] sitemap_follow = ["/sitemap_shops"]
def parse_shop(self, response): def parse_shop(self, response):
pass # ... scrape shop here ... pass # ... scrape shop here ...
Combine SitemapSpider with other sources of urls:: Combine SitemapSpider with other sources of urls:
.. code-block:: python
from scrapy.spiders import SitemapSpider from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider): class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/robots.txt'] sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [ sitemap_rules = [
('/shop/', 'parse_shop'), ("/shop/", "parse_shop"),
] ]
other_urls = ['http://www.example.com/about'] other_urls = ["http://www.example.com/about"]
def start_requests(self): def start_requests(self):
requests = list(super(MySpider, self).start_requests()) requests = list(super(MySpider, self).start_requests())
@ -826,10 +882,10 @@ Combine SitemapSpider with other sources of urls::
return requests return requests
def parse_shop(self, response): def parse_shop(self, response):
pass # ... scrape shop here ... pass # ... scrape shop here ...
def parse_other(self, response): def parse_other(self, response):
pass # ... scrape other here ... pass # ... scrape other here ...
.. _Sitemaps: https://www.sitemaps.org/index.html .. _Sitemaps: https://www.sitemaps.org/index.html
.. _Sitemap index files: https://www.sitemaps.org/protocol.html#index .. _Sitemap index files: https://www.sitemaps.org/protocol.html#index

29
docs/topics/stats.rst
View File

@ -30,10 +30,11 @@ Common Stats Collector uses
=========================== ===========================
Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats` Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats`
attribute. Here is an example of an extension that access stats:: attribute. Here is an example of an extension that access stats:
.. code-block:: python
class ExtensionThatAccessStats: class ExtensionThatAccessStats:
def __init__(self, stats): def __init__(self, stats):
self.stats = stats self.stats = stats
@ -41,21 +42,29 @@ attribute. Here is an example of an extension that access stats::
def from_crawler(cls, crawler): def from_crawler(cls, crawler):
return cls(crawler.stats) return cls(crawler.stats)
Set stat value:: Set stat value:
stats.set_value('hostname', socket.gethostname()) .. code-block:: python
Increment stat value:: stats.set_value("hostname", socket.gethostname())
stats.inc_value('custom_count') Increment stat value:
Set stat value only if greater than previous:: .. code-block:: python
stats.max_value('max_items_scraped', value) stats.inc_value("custom_count")
Set stat value only if lower than previous:: Set stat value only if greater than previous:
stats.min_value('min_free_memory_percent', value) .. code-block:: python
stats.max_value("max_items_scraped", value)
Set stat value only if lower than previous:
.. code-block:: python
stats.min_value("min_free_memory_percent", value)
Get stat value: Get stat value:

549
sep/sep-001.rst
View File

@ -1,274 +1,275 @@
======= ============================================ ======= ============================================
SEP 1 SEP 1
Title API for populating item fields (comparison) Title API for populating item fields (comparison)
Author Ismael Carnales, Pablo Hoffman, Daniel Grana Author Ismael Carnales, Pablo Hoffman, Daniel Grana
Created 2009-07-19 Created 2009-07-19
Status Obsoleted by :ref:`sep-008` Status Obsoleted by :ref:`sep-008`
======= ============================================ ======= ============================================
===================================================== =====================================================
SEP-001 - API for populating item fields (comparison) SEP-001 - API for populating item fields (comparison)
===================================================== =====================================================
This page shows different usage scenarios for the two new proposed API for This page shows different usage scenarios for the two new proposed API for
populating item field values (which will replace the old deprecated !RobustItem populating item field values (which will replace the old deprecated !RobustItem
API) and compares them. One of these will be chosen as the recommended (and API) and compares them. One of these will be chosen as the recommended (and
supported) mechanism in Scrapy 0.7. supported) mechanism in Scrapy 0.7.
Candidates and their API Candidates and their API
======================== ========================
RobustItem (old, deprecated) RobustItem (old, deprecated)
---------------------------- ----------------------------
- ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)`` - ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)``
.. note:: ``attribute()`` modifiers (like ``add=True``) are passed together .. note:: ``attribute()`` modifiers (like ``add=True``) are passed together
with adaptor args as keyword arguments (this is ugly) with adaptor args as keyword arguments (this is ugly)
ItemForm ItemForm
-------- --------
- ``__init__(response, item=None, **adaptor_args)`` - ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments - instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments
- ``__setitem__(field_name, selector_or_value)`` - ``__setitem__(field_name, selector_or_value)``
- set field value - set field value
- ``__getitem__(field_name)`` - ``__getitem__(field_name)``
- return the "computed" value of a field (the one that would be set to the item). - return the "computed" value of a field (the one that would be set to the item).
returns ``None`` if not set. returns ``None`` if not set.
- ``get_item()`` - ``get_item()``
- return the item populated with the data provided so far - return the item populated with the data provided so far
ItemBuilder ItemBuilder
----------- -----------
- ``__init__(response, item=None, **adaptor_args)`` - ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemBuilder`` with predefined adaptor arguments - instantiate an ``ItemBuilder`` with predefined adaptor arguments
- ``add_value(field_name, selector_or_value, **adaptor_args)`` - ``add_value(field_name, selector_or_value, **adaptor_args)``
- add value to field - add value to field
- ``replace_value(field_name, selector_or_value, **adaptor_args)`` - ``replace_value(field_name, selector_or_value, **adaptor_args)``
- replace existing field value - replace existing field value
- ``get_value(field_name)`` - ``get_value(field_name)``
- return the "computed" value of a field (the one that would be set to the - return the "computed" value of a field (the one that would be set to the
item). returns ``None`` if not set. item). returns ``None`` if not set.
- ``get_item()`` - ``get_item()``
- return the item populated with the data provided so far - return the item populated with the data provided so far
Pros and cons of each candidate Pros and cons of each candidate
=============================== ===============================
ItemForm ItemForm
-------- --------
Pros: Pros:
- same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html) - same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html)
- some people consider setitem API more elegant than methods API - some people consider setitem API more elegant than methods API
Cons: Cons:
- doesn't allow passing run-time arguments to adaptors on assign, you have to - doesn't allow passing run-time arguments to adaptors on assign, you have to
override the adaptors for your spider if you need specific parameters, which override the adaptors for your spider if you need specific parameters, which
can be an overhead. Example: can be an overhead. Example:
Neutral: Neutral:
- solves the add=True problem using standard ``__add__`` and ``list.append()`` method - solves the add=True problem using standard ``__add__`` and ``list.append()`` method
ItemBuilder ItemBuilder
----------- -----------
Pros: Pros:
- allows passing run-time arguments to adaptors on assigned - allows passing run-time arguments to adaptors on assigned
Cons: Cons:
- some people consider setitem API more elegant than methods API - some people consider setitem API more elegant than methods API
Neutral: Neutral:
- solves the "add=True" problem by implementing different methods per action - solves the "add=True" problem by implementing different methods per action
(replacing or adding) (replacing or adding)
Usage Scenarios for each candidate Usage Scenarios for each candidate
================================== ==================================
Defining adaptors Defining adaptors
----------------- -----------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
class NewsForm(ItemForm): class NewsForm(ItemForm):
item_class = NewsItem item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip) url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip) headline = adaptor(extract, remove_tags(), unquote(), strip)
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
class NewsBuilder(ItemBuilder): class NewsBuilder(ItemBuilder):
item_class = NewsItem item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip) url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip) headline = adaptor(extract, remove_tags(), unquote(), strip)
Creating an Item Creating an Item
---------------- ----------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
ia = NewsForm(response) ia = NewsForm(response)
ia['url'] = response.url ia["url"] = response.url
ia['headline'] = x.x('//h1[@class="headline"]') ia["headline"] = x.x('//h1[@class="headline"]')
# if we want to add another value to the same field # if we want to add another value to the same field
ia['headline'] += x.x('//h1[@class="headline2"]') ia["headline"] += x.x('//h1[@class="headline2"]')
# if we want to replace the field value other value to the same field # if we want to replace the field value other value to the same field
ia['headline'] = x.x('//h1[@class="headline3"]') ia["headline"] = x.x('//h1[@class="headline3"]')
return ia.get_item() return ia.get_item()
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
il = NewsBuilder(response) il = NewsBuilder(response)
il.add_value('url', response.url) il.add_value("url", response.url)
il.add_value('headline', x.x('//h1[@class="headline"]')) il.add_value("headline", x.x('//h1[@class="headline"]'))
# if we want to add another value to the same field # if we want to add another value to the same field
il.add_value('headline', x.x('//h1[@class="headline2"]')) il.add_value("headline", x.x('//h1[@class="headline2"]'))
# if we want to replace the field value other value to the same field # if we want to replace the field value other value to the same field
il.replace_value('headline', x.x('//h1[@class="headline3"]')) il.replace_value("headline", x.x('//h1[@class="headline3"]'))
return il.get_item() return il.get_item()
Using different adaptors per Spider/Site Using different adaptors per Spider/Site
---------------------------------------- ----------------------------------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
class SiteNewsFrom(NewsForm): class SiteNewsFrom(NewsForm):
published = adaptor(HtmlNewsForm.published, to_date('%d.%m.%Y')) published = adaptor(HtmlNewsForm.published, to_date("%d.%m.%Y"))
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
class SiteNewsBuilder(NewsBuilder): class SiteNewsBuilder(NewsBuilder):
published = adaptor(HtmlNewsBuilder.published, to_date('%d.%m.%Y')) published = adaptor(HtmlNewsBuilder.published, to_date("%d.%m.%Y"))
Check the value of an item being-extracted Check the value of an item being-extracted
------------------------------------------ ------------------------------------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
ia = NewsForm(response) ia = NewsForm(response)
ia['headline'] = x.x('//h1[@class="headline"]') ia["headline"] = x.x('//h1[@class="headline"]')
if not ia['headline']: if not ia["headline"]:
ia['headline'] = x.x('//h1[@class="title"]') ia["headline"] = x.x('//h1[@class="title"]')
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
il = NewsBuilder(response) il = NewsBuilder(response)
il.add_value('headline', x.x('//h1[@class="headline"]')) il.add_value("headline", x.x('//h1[@class="headline"]'))
if not nf.get_value('headline'): if not nf.get_value("headline"):
il.add_value('headline', x.x('//h1[@class="title"]')) il.add_value("headline", x.x('//h1[@class="title"]'))
Adding a value to a list attribute/field Adding a value to a list attribute/field
---------------------------------------- ----------------------------------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
ia['headline'] += x.x('//h1[@class="headline"]') ia["headline"] += x.x('//h1[@class="headline"]')
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
il.add_value('headline', x.x('//h1[@class="headline"]')) il.add_value("headline", x.x('//h1[@class="headline"]'))
Passing run-time arguments to adaptors Passing run-time arguments to adaptors
-------------------------------------- --------------------------------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
# Only approach is passing arguments when instantiating the form # Only approach is passing arguments when instantiating the form
ia = NewsForm(response, default_unit='cm') ia = NewsForm(response, default_unit="cm")
ia['width'] = x.x('//p[@class="width"]') ia["width"] = x.x('//p[@class="width"]')
ItemBuilder ItemBuilder
~~~~~~~~~~~ ~~~~~~~~~~~
:: .. code-block:: python
#!python #!python
il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm') il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")
# an alternative approach (more efficient) # an alternative approach (more efficient)
il = NewsBuilder(response, default_unit='cm') il = NewsBuilder(response, default_unit="cm")
il.add_value('width', x.x('//p[@class="width"]')) il.add_value("width", x.x('//p[@class="width"]'))
Passing run-time arguments to adaptors (same argument name) Passing run-time arguments to adaptors (same argument name)
----------------------------------------------------------- -----------------------------------------------------------
ItemForm ItemForm
~~~~~~~~ ~~~~~~~~
:: .. code-block:: python
#!python #!python
class MySiteForm(ItemForm): class MySiteForm(ItemForm):
width = adaptor(ItemForm.width, default_unit='cm') width = adaptor(ItemForm.width, default_unit="cm")
volume = adaptor(ItemForm.width, default_unit='lt') volume = adaptor(ItemForm.width, default_unit="lt")
ia['width'] = x.x('//p[@class="width"]')
ia['volume'] = x.x('//p[@class="volume"]') ia["width"] = x.x('//p[@class="width"]')
ia["volume"] = x.x('//p[@class="volume"]')
# another example passing parameters on instance
ia = NewsForm(response, encoding='utf-8') # another example passing parameters on instance
ia['name'] = x.x('//p[@class="name"]') ia = NewsForm(response, encoding="utf-8")
ia["name"] = x.x('//p[@class="name"]')
ItemBuilder
~~~~~~~~~~~ ItemBuilder
~~~~~~~~~~~
::
.. code-block:: python
#!python
il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm') #!python
il.add_value('volume', x.x('//p[@class="volume"]'), default_unit='lt') il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")
il.add_value("volume", x.x('//p[@class="volume"]'), default_unit="lt")

44
sep/sep-002.rst
View File

@ -16,18 +16,19 @@ called !ListField.
Proposed Implementation Proposed Implementation
======================= =======================
:: .. code-block:: python
#!python #!python
from scrapy.item.fields import BaseField from scrapy.item.fields import BaseField
class ListField(BaseField): class ListField(BaseField):
def __init__(self, field, default=None): def __init__(self, field, default=None):
self._field = field self._field = field
super(ListField, self).__init__(default) super(ListField, self).__init__(default)
def to_python(self, value): def to_python(self, value):
if hasattr(value, '__iter__'): # str/unicode not allowed if hasattr(value, "__iter__"): # str/unicode not allowed
return [self._field.to_python(v) for v in value] return [self._field.to_python(v) for v in value]
else: else:
raise TypeError("Expected iterable, got %s" % type(value).__name__) raise TypeError("Expected iterable, got %s" % type(value).__name__)
@ -42,12 +43,13 @@ Usage Scenarios
Defining a list field Defining a list field
--------------------- ---------------------
:: .. code-block:: python
#!python #!python
from scrapy.item.models import Item from scrapy.item.models import Item
from scrapy.item.fields import ListField, TextField, DateField, IntegerField from scrapy.item.fields import ListField, TextField, DateField, IntegerField
class Article(Item): class Article(Item):
categories = ListField(TextField) categories = ListField(TextField)
dates = ListField(DateField, default=[]) dates = ListField(DateField, default=[])
@ -56,57 +58,59 @@ Defining a list field
Another case of products and variants which highlights the fact that it's Another case of products and variants which highlights the fact that it's
important to instantiate !ListField with field instances, not classes: important to instantiate !ListField with field instances, not classes:
:: .. code-block:: python
#!python #!python
from scrapy.item.models import Item from scrapy.item.models import Item
from scrapy.item.fields import ListField, TextField from scrapy.item.fields import ListField, TextField
class Variant(Item): class Variant(Item):
name = TextField() name = TextField()
class Product(Variant): class Product(Variant):
variants = ListField(ItemField(Variant)) variants = ListField(ItemField(Variant))
Assigning a list field Assigning a list field
---------------------- ----------------------
:: .. code-block:: python
#!python #!python
i = Article() i = Article()
i['categories'] = [] i["categories"] = []
i['categories'] = ['politics', 'sport'] i["categories"] = ["politics", "sport"]
i['categories'] = ['test', 1] -> raises TypeError i["categories"] = ["test", 1] # -> raises TypeError
i['categories'] = asd -> raises TypeError i["categories"] = asd # -> raises TypeError
i['dates'] = [] i["dates"] = []
i['dates'] = ['2009-01-01'] # raises TypeError? (depends on TextField) i["dates"] = ["2009-01-01"] # raises TypeError? (depends on TextField)
i['numbers'] = ['1', 2, '3'] i["numbers"] = ["1", 2, "3"]
i['numbers'] # returns [1, 2, 3] i["numbers"] # returns [1, 2, 3]
Default values Default values
-------------- --------------
:: .. code-block:: python
#!python #!python
i = Article() i = Article()
i['categories'] # raises KeyError i["categories"] # raises KeyError
i.get('categories') # returns None i.get("categories") # returns None
i['numbers'] # returns [] i["numbers"] # returns []
Appending values Appending values
---------------- ----------------
:: .. code-block:: python
#!python #!python
i = Article() i = Article()
i['categories'] = ['one', 'two'] i["categories"] = ["one", "two"]
i['categories'].append(3) # XXX: should this fail? i["categories"].append(3) # XXX: should this fail?

90
sep/sep-003.rst
View File

@ -27,18 +27,21 @@ This API proposal relies on the following API:
Proposed Implementation of ItemField Proposed Implementation of ItemField
==================================== ====================================
:: .. code-block:: python
#!python #!python
from scrapy.item.fields import BaseField from scrapy.item.fields import BaseField
class ItemField(BaseField): class ItemField(BaseField):
def __init__(self, item_type, default=None): def __init__(self, item_type, default=None):
self._item_type = item_type self._item_type = item_type
super(ItemField, self).__init__(default) super(ItemField, self).__init__(default)
def to_python(self, value): def to_python(self, value):
return self._item_type(value) if not isinstance(value, self._item_type) else value return (
self._item_type(value) if not isinstance(value, self._item_type) else value
)
def get_default(self): def get_default(self):
# WARNING: returns default item instead of a copy - this must be # WARNING: returns default item instead of a copy - this must be
@ -54,25 +57,28 @@ Usage Scenarios
Defining an item containing ItemField's Defining an item containing ItemField's
--------------------------------------- ---------------------------------------
:: .. code-block:: python
#!python #!python
from scrapy.item.models import Item from scrapy.item.models import Item
from scrapy.item.fields import ListField, ItemField, TextField, UrlField, DecimalField from scrapy.item.fields import ListField, ItemField, TextField, UrlField, DecimalField
class Supplier(Item): class Supplier(Item):
name = TextField(default="anonymous supplier") name = TextField(default="anonymous supplier")
url = UrlField() url = UrlField()
class Variant(Item): class Variant(Item):
name = TextField(required=True) name = TextField(required=True)
url = UrlField() url = UrlField()
price = DecimalField() price = DecimalField()
class Product(Variant): class Product(Variant):
supplier = ItemField(Supplier, default=Supplier(name="default supplier") supplier = ItemField(Supplier, default=Supplier(name="default supplier"))
variants = ListField(ItemField(Variant)) variants = ListField(ItemField(Variant))
# these ones are used for documenting default value examples # these ones are used for documenting default value examples
supplier2 = ItemField(Supplier) supplier2 = ItemField(Supplier)
variants2 = ListField(ItemField(Variant), default=[]) variants2 = ListField(ItemField(Variant), default=[])
@ -81,16 +87,16 @@ It's important to note here that the (perhaps most intuitive) way of defining a
Product-Variant relationship (i.e. defining a recursive !ItemField) doesn't Product-Variant relationship (i.e. defining a recursive !ItemField) doesn't
work. For example, this fails to compile: work. For example, this fails to compile:
:: .. code-block:: python
#!python #!python
class Product(Item): class Product(Item):
variants = ItemField(Product) # Fails to compile variants = ItemField(Product) # Fails to compile
Assigning an item field Assigning an item field
----------------------- -----------------------
:: .. code-block:: python
#!python #!python
supplier = Supplier(name="Supplier 1", url="http://example.com") supplier = Supplier(name="Supplier 1", url="http://example.com")
@ -98,69 +104,69 @@ Assigning an item field
p = Product() p = Product()
# standard assignment # standard assignment
p['supplier'] = supplier p["supplier"] = supplier
# this also works as it tries to instantiate a Supplier with the given dict # this also works as it tries to instantiate a Supplier with the given dict
p['supplier'] = {'name': 'Supplier 1' url='http://example.com'} p["supplier"] = {"name": "Supplier 1", url: "http://example.com"}
# this fails because it can't instantiate a Supplier # this fails because it can't instantiate a Supplier
p['supplier'] = 'Supplier 1' p["supplier"] = "Supplier 1"
# this fails because url doesn't have the valid type # this fails because url doesn't have the valid type
p['supplier'] = {'name': 'Supplier 1' url=123} p["supplier"] = {"name": "Supplier 1", url: 123}
v1 = Variant() v1 = Variant()
v1['name'] = "lala" v1["name"] = "lala"
v1['price'] = Decimal("100") v1["price"] = Decimal("100")
v2 = Variant() v2 = Variant()
v2['name'] = "lolo" v2["name"] = "lolo"
v2['price'] = Decimal("150") v2["price"] = Decimal("150")
# standard assignment # standard assignment
p['variants'] = [v1, v2] # OK p["variants"] = [v1, v2] # OK
# can also instantiate at assignment time # can also instantiate at assignment time
p['variants'] = [v1, Variant(name="lolo", price=Decimal("150")] p["variants"] = [v1, Variant(name="lolo", price=Decimal("150"))]
# this also works as it tries to instantiate a Variant with the given dict # this also works as it tries to instantiate a Variant with the given dict
p['variants'] = [v1, {'name': 'lolo', 'price': Decimal("150")] p["variants"] = [v1, {"name": "lolo", "price": Decimal("150")}]
# this fails because it can't instantiate a Variant # this fails because it can't instantiate a Variant
p['variants'] = [v1, 'test'] p["variants"] = [v1, "test"]
# this fails because 'coco' is not a valid value for price # this fails because 'coco' is not a valid value for price
p['variants'] = [v1, {'name': 'lolo', 'price': 'coco'] p["variants"] = [v1, {"name": "lolo", "price": "coco"}]
Default values Default values
-------------- --------------
:: .. code-block:: python
#!python #!python
p = Product() p = Product()
p['supplier'] # returns: Supplier(name='default supplier') p["supplier"] # returns: Supplier(name='default supplier')
p['supplier2'] # raises KeyError p["supplier2"] # raises KeyError
p['supplier2'] = Supplier() p["supplier2"] = Supplier()
p['supplier2'] # returns: Supplier(name='anonymous supplier') p["supplier2"] # returns: Supplier(name='anonymous supplier')
p['variants'] # raises KeyError p["variants"] # raises KeyError
p['variants2'] # returns [] p["variants2"] # returns []
p['categories'] # raises KeyError p["categories"] # raises KeyError
p.get('categories') # returns None p.get("categories") # returns None
p['numbers'] # returns [] p["numbers"] # returns []
Accessing and changing nested item values Accessing and changing nested item values
---------------------------------------- ----------------------------------------
:: .. code-block:: python
#!python #!python
p = Product(supplier=Supplier(name="some name", url="http://example.com")) p = Product(supplier=Supplier(name="some name", url="http://example.com"))
p['supplier']['url'] # returns 'http://example.com' p["supplier"]["url"] # returns 'http://example.com'
p['supplier']['url'] = "http://www.other.com" # works as expected p["supplier"]["url"] = "http://www.other.com" # works as expected
p['supplier']['url'] = 123 # fails: wrong type for supplier url p["supplier"]["url"] = 123 # fails: wrong type for supplier url
p['variants'] = [v1, v2] p["variants"] = [v1, v2]
p['variants'][0]['name'] # returns v1 name p["variants"][0]["name"] # returns v1 name
p['variants'][1]['name'] # returns v2 name p["variants"][1]["name"] # returns v2 name
# XXX: decide what to do about these cases: # XXX: decide what to do about these cases:
p['variants'].append(v3) # works but doesn't check type of v3 p["variants"].append(v3) # works but doesn't check type of v3
p['variants'].append(1) # works but shouldn't? p["variants"].append(1) # works but shouldn't?

9
sep/sep-004.rst
View File

@ -26,7 +26,7 @@ Proposed API
Here's a simple proof-of-concept code of such script: Here's a simple proof-of-concept code of such script:
:: .. code-block:: python
#!/usr/bin/env python #!/usr/bin/env python
from scrapy.http import Request from scrapy.http import Request
@ -35,21 +35,24 @@ Here's a simple proof-of-concept code of such script:
# a container to hold scraped items # a container to hold scraped items
scraped_items = [] scraped_items = []
def parse_start_page(response): def parse_start_page(response):
# collect urls to follow into urls_to_follow list # collect urls to follow into urls_to_follow list
requests = [Request(url, callback=parse_other_page) for url in urls_to_follow] requests = [Request(url, callback=parse_other_page) for url in urls_to_follow]
return requests return requests
def parse_other_page(response): def parse_other_page(response):
# ... parse items from response content ... # ... parse items from response content ...
scraped_items.extend(parsed_items) scraped_items.extend(parsed_items)
start_urls = ["http://www.example.com/start_page.html"] start_urls = ["http://www.example.com/start_page.html"]
cr = Crawler(start_urls, callback=parse_start_page) cr = Crawler(start_urls, callback=parse_start_page)
cr.run() # blocking call - this populates scraped_items cr.run() # blocking call - this populates scraped_items
print "%d items scraped" % len(scraped_items) print("%d items scraped" % len(scraped_items))
# ... do something more interesting with scraped_items ... # ... do something more interesting with scraped_items ...
The behaviour of the Scrapy crawler would be controller by the Scrapy settings, The behaviour of the Scrapy crawler would be controller by the Scrapy settings,

33
sep/sep-005.rst
View File

@ -12,7 +12,7 @@ SEP-005: Detailed ``ItemBuilder`` API use
Item class for examples: Item class for examples:
:: .. code-block:: python
#!python #!python
class NewsItem(Item): class NewsItem(Item):
@ -25,7 +25,7 @@ Item class for examples:
gSetting expanders gSetting expanders
================== ==================
:: .. code-block:: python
#!python #!python
class NewsItemBuilder(ItemBuilder): class NewsItemBuilder(ItemBuilder):
@ -44,7 +44,7 @@ on their Item Field class:
gSetting reducers gSetting reducers
================= =================
:: .. code-block:: python
#!python #!python
class NewsItemBuilder(ItemBuilder): class NewsItemBuilder(ItemBuilder):
@ -60,7 +60,7 @@ content
gSetting expanders/reducers new way gSetting expanders/reducers new way
=================================== ===================================
:: .. code-block:: python
#!python #!python
class NewsItemBuilder(ItemBuilder): class NewsItemBuilder(ItemBuilder):
@ -76,28 +76,29 @@ gSetting expanders/reducers new way
gExtending ``ItemBuilder`` gExtending ``ItemBuilder``
========================== ==========================
:: .. code-block:: python
#!python #!python
class SiteNewsItemBuilder(NewsItemBuilder): class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(), published = reducers.Reducer(
strip, to_date('%d.%m.%Y')) extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
)
gExtending ``ItemBuilder`` using statich methods gExtending ``ItemBuilder`` using statich methods
================================================ ================================================
:: .. code-block:: python
#!python #!python
class SiteNewsItemBuilder(NewsItemBuilder): class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.published, to_date('%d.%m.%Y')) published = reducers.Reducer(NewsItemBuilder.published, to_date("%d.%m.%Y"))
gUsing default_builder gUsing default_builder
====================== ======================
:: .. code-block:: python
#!python #!python
class DefaultedNewsItemBuilder(ItemBuilder): class DefaultedNewsItemBuilder(ItemBuilder):
@ -112,7 +113,7 @@ As a reducer is not set reducers will be set based on Item Field classes.
gReset default_builder for a field gReset default_builder for a field
================================== ==================================
:: .. code-block:: python
#!python #!python
class DefaultedNewsItemBuilder(ItemBuilder): class DefaultedNewsItemBuilder(ItemBuilder):
@ -125,18 +126,20 @@ gReset default_builder for a field
gExtending default ``ItemBuilder`` gExtending default ``ItemBuilder``
================================== ==================================
:: .. code-block:: python
#!python #!python
class SiteNewsItemBuilder(NewsItemBuilder): class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(), strip, to_date('%d.%m.%Y')) published = reducers.Reducer(
extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
)
gExtending default ``ItemBuilder`` using static methods gExtending default ``ItemBuilder`` using static methods
======================================================= =======================================================
:: .. code-block:: python
#!python #!python
class SiteNewsItemBuilder(NewsItemBuilder): class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.default_builder, to_date('%d.%m.%Y')) published = reducers.Reducer(NewsItemBuilder.default_builder, to_date("%d.%m.%Y"))

5
sep/sep-008.rst
View File

@ -87,11 +87,12 @@ Alternative Public API Proposal
Usage example: declaring Item Parsers Usage example: declaring Item Parsers
===================================== =====================================
:: .. code-block:: python
#!python #!python
from scrapy.contrib.itemparser import XPathItemParser, parsers from scrapy.contrib.itemparser import XPathItemParser, parsers
class ProductParser(XPathItemParser): class ProductParser(XPathItemParser):
name_in = parsers.MapConcat(removetags, filterx) name_in = parsers.MapConcat(removetags, filterx)
price_in = parsers.MapConcat(...) price_in = parsers.MapConcat(...)
@ -101,7 +102,7 @@ Usage example: declaring Item Parsers
Usage example: declaring parsers in Fields Usage example: declaring parsers in Fields
========================================== ==========================================
:: .. code-block:: python
#!python #!python
class Product(Item): class Product(Item):

18
sep/sep-009.rst
View File

@ -76,28 +76,30 @@ which we haven't documented so far (partly because of this).
So, for a typical middleware ``__init__`` method code, instead of this: So, for a typical middleware ``__init__`` method code, instead of this:
:: .. code-block:: python
#!python #!python
from scrapy.core.exceptions import NotConfigured from scrapy.core.exceptions import NotConfigured
from scrapy.conf import settings from scrapy.conf import settings
class SomeMiddleware(object): class SomeMiddleware(object):
def __init__(self): def __init__(self):
if not settings.getbool('SOMEMIDDLEWARE_ENABLED'): if not settings.getbool("SOMEMIDDLEWARE_ENABLED"):
raise NotConfigured raise NotConfigured
We'd write this: We'd write this:
:: .. code-block:: python
#!python #!python
from scrapy.core.exceptions import NotConfigured from scrapy.core.exceptions import NotConfigured
class SomeMiddleware(object): class SomeMiddleware(object):
def __init__(self, crawler): def __init__(self, crawler):
if not crawler.settings.getbool('SOMEMIDDLEWARE_ENABLED'): if not crawler.settings.getbool("SOMEMIDDLEWARE_ENABLED"):
raise NotConfigured raise NotConfigured
Running from command line Running from command line
========================= =========================

6
sep/sep-012.rst
View File

@ -83,10 +83,10 @@ example:
$ cat project/spiders/google.py $ cat project/spiders/google.py
:: .. code-block:: python
class GooglecomSpider(BaseSpider): class GooglecomSpider(BaseSpider):
name = 'google' name = "google"
allowed_domains = ['google.com'] allowed_domains = ["google.com"]
.. note:: ``spider_allowed_domains`` becomes optional as only ``OffsiteMiddleware`` uses it. .. note:: ``spider_allowed_domains`` becomes optional as only ``OffsiteMiddleware`` uses it.

170
sep/sep-014.rst
View File

@ -92,7 +92,7 @@ Usage Examples
Basic Crawling Basic Crawling
-------------- --------------
:: .. code-block:: python
#!python #!python
# #
@ -101,20 +101,20 @@ Basic Crawling
class SampleSpider(CrawlSpider): class SampleSpider(CrawlSpider):
rules = [ rules = [
# The dispatcher uses first-match policy # The dispatcher uses first-match policy
Rule(UrlRegexMatch(r'product\.html\?id=\d+'), 'parse_item', follow=False), Rule(UrlRegexMatch(r"product\.html\?id=\d+"), "parse_item", follow=False),
# by default, if the first param is string is wrapped into UrlRegexMatch # by default, if the first param is string is wrapped into UrlRegexMatch
Rule(r'.+', 'parse_page'), Rule(r".+", "parse_page"),
] ]
request_extractors = [ request_extractors = [
# crawl all links looking for products and images # crawl all links looking for products and images
SgmlRequestExtractor(), SgmlRequestExtractor(),
] ]
request_processors = [ request_processors = [
# canonicalize all requests' urls # canonicalize all requests' urls
Canonicalize(), Canonicalize(),
] ]
def parse_item(self, response): def parse_item(self, response):
# parse and extract items from response # parse and extract items from response
@ -127,7 +127,7 @@ Basic Crawling
Custom Processor and External Callback Custom Processor and External Callback
-------------------------------------- --------------------------------------
:: .. code-block:: python
#!python #!python
# #
@ -137,30 +137,32 @@ Custom Processor and External Callback
# Custom Processor # Custom Processor
def filter_today_links(requests): def filter_today_links(requests):
# only crawl today links # only crawl today links
today = datetime.datetime.today().strftime('%Y-%m-%d') today = datetime.datetime.today().strftime("%Y-%m-%d")
return [r for r in requests if today in r.url] return [r for r in requests if today in r.url]
# Callback defined out of spider # Callback defined out of spider
def my_external_callback(response): def my_external_callback(response):
# process item # process item
pass pass
class SampleSpider(CrawlSpider): class SampleSpider(CrawlSpider):
rules = [ rules = [
# The dispatcher uses first-match policy # The dispatcher uses first-match policy
Rule(UrlRegexMatch(r'/news/(.+)/'), my_external_callback), Rule(UrlRegexMatch(r"/news/(.+)/"), my_external_callback),
] ]
request_extractors = [ request_extractors = [
RegexRequestExtractor(r'/sections/.+'), RegexRequestExtractor(r"/sections/.+"),
RegexRequestExtractor(r'/news/.+'), RegexRequestExtractor(r"/news/.+"),
] ]
request_processors = [ request_processors = [
# canonicalize all requests' urls # canonicalize all requests' urls
Canonicalize(), Canonicalize(),
filter_today_links, filter_today_links,
] ]
Implementation Implementation
============== ==============
@ -199,7 +201,7 @@ Package Structure
Request/Response Matchers Request/Response Matchers
------------------------- -------------------------
:: .. code-block:: python
#!python #!python
""" """
@ -208,6 +210,7 @@ Request/Response Matchers
Perform evaluation to Request or Response attributes Perform evaluation to Request or Response attributes
""" """
class BaseMatcher(object): class BaseMatcher(object):
"""Base matcher. Returns True by default.""" """Base matcher. Returns True by default."""
@ -229,11 +232,11 @@ Request/Response Matchers
def matches_url(self, url): def matches_url(self, url):
"""Returns True if given url is equal to matcher's url""" """Returns True if given url is equal to matcher's url"""
return self._url url return self._url == url
def matches_request(self, request): def matches_request(self, request):
"""Returns True if Request's url matches initial url""" """Returns True if Request's url matches initial url"""
return self.matches_url(request.url) return self.matches_url(request.url)
def matches_response(self, response): def matches_response(self, response):
"""REturns True if Response's url matches initial url""" """REturns True if Response's url matches initial url"""
@ -254,7 +257,7 @@ Request/Response Matchers
Request Extractor Request Extractor
----------------- -----------------
:: .. code-block:: python
#!python #!python
# #
@ -262,21 +265,21 @@ Request Extractor
# Extractors receive response and return list of Requests # Extractors receive response and return list of Requests
# #
class BaseSgmlRequestExtractor(FixedSGMLParser): class BaseSgmlRequestExtractor(FixedSGMLParser):
"""Base SGML Request Extractor""" """Base SGML Request Extractor"""
def __init__(self, tag='a', attr='href'): def __init__(self, tag="a", attr="href"):
"""Initialize attributes""" """Initialize attributes"""
FixedSGMLParser.__init__(self) FixedSGMLParser.__init__(self)
self.scan_tag = tag if callable(tag) else lambda t: t tag self.scan_tag = tag if callable(tag) else lambda t: t = tag
self.scan_attr = attr if callable(attr) else lambda a: a attr self.scan_attr = attr if callable(attr) else lambda a: a = attr
self.current_request = None self.current_request = None
def extract_requests(self, response): def extract_requests(self, response):
"""Returns list of requests extracted from response""" """Returns list of requests extracted from response"""
return self._extract_requests(response.body, response.url, return self._extract_requests(response.body, response.url, response.encoding)
response.encoding)
def _extract_requests(self, response_text, response_url, response_encoding): def _extract_requests(self, response_text, response_url, response_encoding):
"""Extract requests with absolute urls""" """Extract requests with absolute urls"""
@ -303,20 +306,19 @@ Request Extractor
def _fix_link_text_encoding(self, encoding): def _fix_link_text_encoding(self, encoding):
"""Convert link_text to unicode for each request""" """Convert link_text to unicode for each request"""
for req in self.requests: for req in self.requests:
req.meta.setdefault('link_text', '') req.meta.setdefault("link_text", "")
req.meta['link_text'] = str_to_unicode(req.meta['link_text'], req.meta["link_text"] = str_to_unicode(req.meta["link_text"], encoding)
encoding)
def reset(self): def reset(self):
"""Reset state""" """Reset state"""
FixedSGMLParser.reset(self) FixedSGMLParser.reset(self)
self.requests = [] self.requests = []
self.base_url = None self.base_url = None
def unknown_starttag(self, tag, attrs): def unknown_starttag(self, tag, attrs):
"""Process unknown start tag""" """Process unknown start tag"""
if 'base' tag: if "base" == tag:
self.base_url = dict(attrs).get('href') self.base_url = dict(attrs).get("href")
if self.scan_tag(tag): if self.scan_tag(tag):
for attr, value in attrs: for attr, value in attrs:
@ -333,8 +335,8 @@ Request Extractor
def handle_data(self, data): def handle_data(self, data):
"""Process data""" """Process data"""
current = self.current_request current = self.current_request
if current and not 'link_text' in current.meta: if current and not "link_text" in current.meta:
current.meta['link_text'] = data.strip() current.meta["link_text"] = data.strip()
class SgmlRequestExtractor(BaseSgmlRequestExtractor): class SgmlRequestExtractor(BaseSgmlRequestExtractor):
@ -343,8 +345,8 @@ Request Extractor
def __init__(self, tags=None, attrs=None): def __init__(self, tags=None, attrs=None):
"""Initialize with custom tag & attribute function checkers""" """Initialize with custom tag & attribute function checkers"""
# defaults # defaults
tags = tuple(tags) if tags else ('a', 'area') tags = tuple(tags) if tags else ("a", "area")
attrs = tuple(attrs) if attrs else ('href', ) attrs = tuple(attrs) if attrs else ("href",)
tag_func = lambda x: x in tags tag_func = lambda x: x in tags
attr_func = lambda x: x in attrs attr_func = lambda x: x in attrs
@ -362,25 +364,26 @@ Request Extractor
def extract_requests(self, response): def extract_requests(self, response):
"""Restrict to XPath regions""" """Restrict to XPath regions"""
hxs = HtmlXPathSelector(response) hxs = HtmlXPathSelector(response)
fragments = (''.join( fragments = (
html_frag for html_frag in hxs.select(xpath).extract() "".join(html_frag for html_frag in hxs.select(xpath).extract())
) for xpath in self.restrict_xpaths) for xpath in self.restrict_xpaths
html_slice = ''.join(html_frag for html_frag in fragments) )
return self._extract_requests(html_slice, response.url, html_slice = "".join(html_frag for html_frag in fragments)
response.encoding) return self._extract_requests(html_slice, response.url, response.encoding)
Request Processor Request Processor
----------------- -----------------
:: .. code-block:: python
#!python #!python
# #
# Request Processors # Request Processors
# Processors receive list of requests and return list of requests # Processors receive list of requests and return list of requests
# #
"""Request Processors""" """Request Processors"""
class Canonicalize(object): class Canonicalize(object):
"""Canonicalize Request Processor""" """Canonicalize Request Processor"""
@ -390,14 +393,14 @@ Request Processor
# replace in-place # replace in-place
req.url = canonicalize_url(req.url) req.url = canonicalize_url(req.url)
yield req yield req
class Unique(object): class Unique(object):
"""Filter duplicate Requests""" """Filter duplicate Requests"""
def __init__(self, *attributes): def __init__(self, *attributes):
"""Initialize comparison attributes""" """Initialize comparison attributes"""
self._attributes = attributes or ['url'] self._attributes = attributes or ["url"]
def _requests_equal(self, req1, req2): def _requests_equal(self, req1, req2):
"""Attribute comparison helper""" """Attribute comparison helper"""
@ -430,20 +433,24 @@ Request Processor
"""Filter request's domain""" """Filter request's domain"""
def __init__(self, allow=(), deny=()): def __init__(self, allow=(), deny=()):
"""Initialize allow/deny attributes""" """Initialize allow/deny attributes"""
self.allow = tuple(arg_to_iter(allow)) self.allow = tuple(arg_to_iter(allow))
self.deny = tuple(arg_to_iter(deny)) self.deny = tuple(arg_to_iter(deny))
def __call__(self, requests): def __call__(self, requests):
"""Filter domains""" """Filter domains"""
processed = (req for req in requests) processed = (req for req in requests)
if self.allow: if self.allow:
processed = (req for req in requests processed = (
if url_is_from_any_domain(req.url, self.allow)) req for req in requests if url_is_from_any_domain(req.url, self.allow)
)
if self.deny: if self.deny:
processed = (req for req in requests processed = (
if not url_is_from_any_domain(req.url, self.deny)) req
for req in requests
if not url_is_from_any_domain(req.url, self.deny)
)
return processed return processed
@ -453,24 +460,28 @@ Request Processor
def __init__(self, allow=(), deny=()): def __init__(self, allow=(), deny=()):
"""Initialize allow/deny attributes""" """Initialize allow/deny attributes"""
_re_type = type(re.compile('', 0)) _re_type = type(re.compile("", 0))
self.allow_res = [x if isinstance(x, _re_type) else re.compile(x) self.allow_res = [
for x in arg_to_iter(allow)] x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(allow)
self.deny_res = [x if isinstance(x, _re_type) else re.compile(x) ]
for x in arg_to_iter(deny)] self.deny_res = [
x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(deny)
]
def __call__(self, requests): def __call__(self, requests):
"""Filter request's url based on allow/deny rules""" """Filter request's url based on allow/deny rules"""
#TODO: filter valid urls here? # TODO: filter valid urls here?
processed = (req for req in requests) processed = (req for req in requests)
if self.allow_res: if self.allow_res:
processed = (req for req in requests processed = (
if self._matches(req.url, self.allow_res)) req for req in requests if self._matches(req.url, self.allow_res)
)
if self.deny_res: if self.deny_res:
processed = (req for req in requests processed = (
if not self._matches(req.url, self.deny_res)) req for req in requests if not self._matches(req.url, self.deny_res)
)
return processed return processed
@ -481,7 +492,7 @@ Request Processor
Rule Object Rule Object
----------- -----------
:: .. code-block:: python
#!python #!python
# #
@ -490,8 +501,10 @@ Rule Object
# #
class Rule(object): class Rule(object):
"""Crawler Rule""" """Crawler Rule"""
def __init__(self, matcher, callback=None, cb_args=None,
cb_kwargs=None, follow=True): def __init__(
self, matcher, callback=None, cb_args=None, cb_kwargs=None, follow=True
):
"""Store attributes""" """Store attributes"""
self.matcher = matcher self.matcher = matcher
self.callback = callback self.callback = callback
@ -499,12 +512,14 @@ Rule Object
self.cb_kwargs = cb_kwargs if cb_kwargs else {} self.cb_kwargs = cb_kwargs if cb_kwargs else {}
self.follow = follow self.follow = follow
# #
# Rules Manager takes list of Rule objects and normalize matcher and callback # Rules Manager takes list of Rule objects and normalize matcher and callback
# into CompiledRule # into CompiledRule
# #
class CompiledRule(object): class CompiledRule(object):
"""Compiled version of Rule""" """Compiled version of Rule"""
def __init__(self, matcher, callback=None, follow=False): def __init__(self, matcher, callback=None, follow=False):
"""Initialize attributes checking type""" """Initialize attributes checking type"""
assert isinstance(matcher, BaseMatcher) assert isinstance(matcher, BaseMatcher)
@ -518,15 +533,16 @@ Rule Object
Rules Manager Rules Manager
------------- -------------
:: .. code-block:: python
#!python #!python
# #
# Handles rules matcher/callbacks # Handles rules matcher/callbacks
# Resolve rule for given response # Resolve rule for given response
# #
class RulesManager(object): class RulesManager(object):
"""Rules Manager""" """Rules Manager"""
def __init__(self, rules, spider, default_matcher=UrlRegexMatcher): def __init__(self, rules, spider, default_matcher=UrlRegexMatcher):
"""Initialize rules using spider and default matcher""" """Initialize rules using spider and default matcher"""
self._rules = tuple() self._rules = tuple()
@ -542,8 +558,9 @@ Rules Manager
# instance default matcher # instance default matcher
matcher = default_matcher(rule.matcher) matcher = default_matcher(rule.matcher)
else: else:
raise ValueError('Not valid matcher given %r in %r' \ raise ValueError(
% (rule.matcher, rule)) "Not valid matcher given %r in %r" % (rule.matcher, rule)
)
# prepare callback # prepare callback
if callable(rule.callback): if callable(rule.callback):
@ -553,8 +570,9 @@ Rules Manager
callback = getattr(spider, rule.callback) callback = getattr(spider, rule.callback)
if not callable(callback): if not callable(callback):
raise AttributeError('Invalid callback %r can not be resolved' \ raise AttributeError(
% callback) "Invalid callback %r can not be resolved" % callback
)
else: else:
callback = None callback = None
@ -564,7 +582,7 @@ Rules Manager
# append compiled rule to rules list # append compiled rule to rules list
crule = CompiledRule(matcher, callback, follow=rule.follow) crule = CompiledRule(matcher, callback, follow=rule.follow)
self._rules += (crule, ) self._rules += (crule,)
def get_rule(self, response): def get_rule(self, response):
"""Returns first rule that matches response""" """Returns first rule that matches response"""
@ -575,7 +593,7 @@ Rules Manager
Request Generator Request Generator
----------------- -----------------
:: .. code-block:: python
#!python #!python
# #
@ -605,7 +623,7 @@ Request Generator
``CrawlSpider`` ``CrawlSpider``
----------------- -----------------
:: .. code-block:: python
#!python #!python
# #
@ -625,9 +643,9 @@ Request Generator
# wrap rules # wrap rules
self._rulesman = RulesManager(self.rules, spider=self) self._rulesman = RulesManager(self.rules, spider=self)
# generates new requests with given callback # generates new requests with given callback
self._reqgen = RequestGenerator(self.request_extractors, self._reqgen = RequestGenerator(
self.request_processors, self.request_extractors, self.request_processors, self.parse
self.parse) )
def parse(self, response): def parse(self, response):
"""Dispatch callback and generate requests""" """Dispatch callback and generate requests"""

49
sep/sep-016.rst
View File

@ -67,21 +67,21 @@ Regex (HTML) Link Extractor
A typical application of LegSpider's is to build Link Extractors. For example: A typical application of LegSpider's is to build Link Extractors. For example:
:: .. code-block:: python
#!python #!python
class RegexHtmlLinkExtractor(LegSpider): class RegexHtmlLinkExtractor(LegSpider):
def process_response(self, response): def process_response(self, response):
if isinstance(response, HtmlResponse): if isinstance(response, HtmlResponse):
allowed_regexes = self.spider.url_regexes_to_follow allowed_regexes = self.spider.url_regexes_to_follow
# extract urls to follow using allowed_regexes # extract urls to follow using allowed_regexes
return [Request(x) for x in urls_to_follow] return [Request(x) for x in urls_to_follow]
class MySpider(LegSpider): class MySpider(LegSpider):
legs = [RegexHtmlLinkExtractor()] legs = [RegexHtmlLinkExtractor()]
url_regexes_to_follow = ['/product.php?.*'] url_regexes_to_follow = ["/product.php?.*"]
def parse_response(self, response): def parse_response(self, response):
# parse response and extract items # parse response and extract items
@ -92,13 +92,12 @@ RSS2 link extractor
This is a Leg Spider that can be used for following links from RSS2 feeds. This is a Leg Spider that can be used for following links from RSS2 feeds.
:: .. code-block:: python
#!python #!python
class Rss2LinkExtractor(LegSpider): class Rss2LinkExtractor(LegSpider):
def process_response(self, response): def process_response(self, response):
if response.headers.get('Content-type') 'application/rss+xml': if response.headers.get("Content-type") == "application/rss+xml":
xs = XmlXPathSelector(response) xs = XmlXPathSelector(response)
urls = xs.select("//item/link/text()").extract() urls = xs.select("//item/link/text()").extract()
return [Request(x) for x in urls] return [Request(x) for x in urls]
@ -108,11 +107,10 @@ Callback dispatcher based on rules
Another example could be to build a callback dispatcher based on rules: Another example could be to build a callback dispatcher based on rules:
:: .. code-block:: python
#!python #!python
class CallbackRules(LegSpider): class CallbackRules(LegSpider):
def __init__(self, *a, **kw): def __init__(self, *a, **kw):
super(CallbackRules, self).__init__(*a, **kw) super(CallbackRules, self).__init__(*a, **kw)
for regex, method_name in self.spider.callback_rules.items(): for regex, method_name in self.spider.callback_rules.items():
@ -128,12 +126,13 @@ Another example could be to build a callback dispatcher based on rules:
return method(response) return method(response)
return [] return []
class MySpider(LegSpider): class MySpider(LegSpider):
legs = [CallbackRules()] legs = [CallbackRules()]
callback_rules = { callback_rules = {
'/product.php.*': 'parse_product', "/product.php.*": "parse_product",
'/category.php.*': 'parse_category', "/category.php.*": "parse_category",
} }
def parse_product(self, response): def parse_product(self, response):
@ -145,19 +144,19 @@ URL Canonicalizers
Another example could be for building URL canonicalizers: Another example could be for building URL canonicalizers:
:: .. code-block:: python
#!python #!python
class CanonicalizeUrl(LegSpider): class CanonicalizeUrl(LegSpider):
def process_request(self, request): def process_request(self, request):
curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules) curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules)
return request.replace(url=curl) return request.replace(url=curl)
class MySpider(LegSpider): class MySpider(LegSpider):
legs = [CanonicalizeUrl()] legs = [CanonicalizeUrl()]
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...] canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
# ... # ...
@ -167,22 +166,22 @@ Setting item identifier
Another example could be for setting a unique identifier to items, based on Another example could be for setting a unique identifier to items, based on
certain fields: certain fields:
:: .. code-block:: python
#!python #!python
class ItemIdSetter(LegSpider): class ItemIdSetter(LegSpider):
def process_item(self, item): def process_item(self, item):
id_field = self.spider.id_field id_field = self.spider.id_field
id_fields_to_hash = self.spider.id_fields_to_hash id_fields_to_hash = self.spider.id_fields_to_hash
item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash) item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
return item return item
class MySpider(LegSpider): class MySpider(LegSpider):
legs = [ItemIdSetter()] legs = [ItemIdSetter()]
id_field = 'guid' id_field = "guid"
id_fields_to_hash = ['supplier_name', 'supplier_id'] id_fields_to_hash = ["supplier_name", "supplier_id"]
def process_response(self, item): def process_response(self, item):
# extract item from response # extract item from response
@ -193,24 +192,24 @@ Combining multiple leg spiders
Here's an example that combines functionality from multiple leg spiders: Here's an example that combines functionality from multiple leg spiders:
:: .. code-block:: python
#!python #!python
class MySpider(LegSpider): class MySpider(LegSpider):
legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()] legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]
url_regexes_to_follow = ['/product.php?.*'] url_regexes_to_follow = ["/product.php?.*"]
parse_rules = { parse_rules = {
'/product.php.*': 'parse_product', "/product.php.*": "parse_product",
'/category.php.*': 'parse_category', "/category.php.*": "parse_category",
} }
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...] canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
id_field = 'guid' id_field = "guid"
id_fields_to_hash = ['supplier_name', 'supplier_id'] id_fields_to_hash = ["supplier_name", "supplier_id"]
def process_product(self, item): def process_product(self, item):
# extract item from response # extract item from response
@ -249,7 +248,7 @@ important to keep in mind their scope and limitations, such as:
Here's a proof-of-concept implementation of ``LegSpider``: Here's a proof-of-concept implementation of ``LegSpider``:
:: .. code-block:: python
#!python #!python
from scrapy.http import Request from scrapy.http import Request

10
sep/sep-017.rst
View File

@ -35,16 +35,15 @@ gExample URL for simple callback
The ``parse_product`` callback must return items containing the fields given in The ``parse_product`` callback must return items containing the fields given in
``@scrapes``. ``@scrapes``.
:: .. code-block:: python
#!python #!python
class ProductSpider(BaseSpider): class ProductSpider(BaseSpider):
def parse_product(self, response): def parse_product(self, response):
""" """
@url http://www.example.com/store/product.php?id=123 @url http://www.example.com/store/product.php?id=123
@scrapes name, price, description @scrapes name, price, description
"""" """
gChained callbacks gChained callbacks
------------------ ------------------
@ -55,11 +54,10 @@ other for scraping user profile info.
The contracts assert that the first callback returns a Request and the second The contracts assert that the first callback returns a Request and the second
one scrape ``user, name, email`` fields. one scrape ``user, name, email`` fields.
:: .. code-block:: python
#!python #!python
class UserProfileSpider(BaseSpider): class UserProfileSpider(BaseSpider):
def parse_login_page(self, response): def parse_login_page(self, response):
""" """
@url http://www.example.com/login.php @url http://www.example.com/login.php
@ -71,7 +69,7 @@ one scrape ``user, name, email`` fields.
""" """
@after parse_login_page @after parse_login_page
@scrapes user, name, email @scrapes user, name, email
"""" """
# ... # ...
Tags reference Tags reference

150
sep/sep-018.rst
View File

@ -166,27 +166,32 @@ written, it should work both globally and per spider.
Here's an example that combines functionality from multiple middlewares into Here's an example that combines functionality from multiple middlewares into
the same spider: the same spider:
:: .. code-block:: python
#!python #!python
class MySpider(BaseSpider): class MySpider(BaseSpider):
middlewares = [RegexLinkExtractor(), CallbackRules(), CanonicalizeUrl(), middlewares = [
ItemIdSetter(), OffsiteMiddleware()] RegexLinkExtractor(),
CallbackRules(),
CanonicalizeUrl(),
ItemIdSetter(),
OffsiteMiddleware(),
]
allowed_domains = ['example.com', 'sub.example.com'] allowed_domains = ["example.com", "sub.example.com"]
url_regexes_to_follow = ['/product.php?.*'] url_regexes_to_follow = ["/product.php?.*"]
callback_rules = { callback_rules = {
'/product.php.*': 'parse_product', "/product.php.*": "parse_product",
'/category.php.*': 'parse_category', "/category.php.*": "parse_category",
} }
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...] canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
id_field = 'guid' id_field = "guid"
id_fields_to_hash = ['supplier_name', 'supplier_id'] id_fields_to_hash = ["supplier_name", "supplier_id"]
def parse_product(self, item): def parse_product(self, item):
# extract item from response # extract item from response
@ -234,35 +239,34 @@ Regex (HTML) Link Extractor
A typical application of spider middlewares could be to build Link Extractors. A typical application of spider middlewares could be to build Link Extractors.
For example: For example:
:: .. code-block:: python
#!python #!python
class RegexHtmlLinkExtractor(object): class RegexHtmlLinkExtractor(object):
def process_response(self, response, request, spider): def process_response(self, response, request, spider):
if isinstance(response, HtmlResponse): if isinstance(response, HtmlResponse):
allowed_regexes = spider.url_regexes_to_follow allowed_regexes = spider.url_regexes_to_follow
# extract urls to follow using allowed_regexes # extract urls to follow using allowed_regexes
return [Request(x) for x in urls_to_follow] return [Request(x) for x in urls_to_follow]
# Example spider using this middleware # Example spider using this middleware
class MySpider(BaseSpider): class MySpider(BaseSpider):
middlewares = [RegexHtmlLinkExtractor()] middlewares = [RegexHtmlLinkExtractor()]
url_regexes_to_follow = ['/product.php?.*'] url_regexes_to_follow = ["/product.php?.*"]
# parsing callbacks below # parsing callbacks below
RSS2 link extractor RSS2 link extractor
------------------- -------------------
:: .. code-block:: python
#!python #!python
class Rss2LinkExtractor(object): class Rss2LinkExtractor(object):
def process_response(self, response, request, spider): def process_response(self, response, request, spider):
if response.headers.get('Content-type') 'application/rss+xml': if response.headers.get("Content-type") == "application/rss+xml":
xs = XmlXPathSelector(response) xs = XmlXPathSelector(response)
urls = xs.select("//item/link/text()").extract() urls = xs.select("//item/link/text()").extract()
return [Request(x) for x in urls] return [Request(x) for x in urls]
@ -272,11 +276,10 @@ Callback dispatcher based on rules
Another example could be to build a callback dispatcher based on rules: Another example could be to build a callback dispatcher based on rules:
:: .. code-block:: python
#!python #!python
class CallbackRules(object): class CallbackRules(object):
def __init__(self): def __init__(self):
self.rules = {} self.rules = {}
dispatcher.connect(signals.spider_opened, self.spider_opened) dispatcher.connect(signals.spider_opened, self.spider_opened)
@ -300,13 +303,14 @@ Another example could be to build a callback dispatcher based on rules:
return method(response) return method(response)
return [] return []
# Example spider using this middleware # Example spider using this middleware
class MySpider(BaseSpider): class MySpider(BaseSpider):
middlewares = [CallbackRules()] middlewares = [CallbackRules()]
callback_rules = { callback_rules = {
'/product.php.*': 'parse_product', "/product.php.*": "parse_product",
'/category.php.*': 'parse_category', "/category.php.*": "parse_category",
} }
def parse_product(self, response): def parse_product(self, response):
@ -318,22 +322,20 @@ URL Canonicalizers
Another example could be for building URL canonicalizers: Another example could be for building URL canonicalizers:
:: .. code-block:: python
#!python #!python
class CanonicalizeUrl(object): class CanonicalizeUrl(object):
def process_request(self, request, response, spider): def process_request(self, request, response, spider):
curl = canonicalize_url(request.url, curl = canonicalize_url(request.url, rules=spider.canonicalization_rules)
rules=spider.canonicalization_rules)
return request.replace(url=curl) return request.replace(url=curl)
# Example spider using this middleware # Example spider using this middleware
class MySpider(BaseSpider): class MySpider(BaseSpider):
middlewares = [CanonicalizeUrl()] middlewares = [CanonicalizeUrl()]
canonicalization_rules = ['sort-query-args', canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
'normalize-percent-encoding', ...]
# ... # ...
@ -343,23 +345,23 @@ Setting item identifier
Another example could be for setting a unique identifier to items, based on Another example could be for setting a unique identifier to items, based on
certain fields: certain fields:
:: .. code-block:: python
#!python #!python
class ItemIdSetter(object): class ItemIdSetter(object):
def process_item(self, item, response, spider): def process_item(self, item, response, spider):
id_field = spider.id_field id_field = spider.id_field
id_fields_to_hash = spider.id_fields_to_hash id_fields_to_hash = spider.id_fields_to_hash
item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash) item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
return item return item
# Example spider using this middleware # Example spider using this middleware
class MySpider(BaseSpider): class MySpider(BaseSpider):
middlewares = [ItemIdSetter()] middlewares = [ItemIdSetter()]
id_field = 'guid' id_field = "guid"
id_fields_to_hash = ['supplier_name', 'supplier_id'] id_fields_to_hash = ["supplier_name", "supplier_id"]
def parse(self, response): def parse(self, response):
# extract item from response # extract item from response
@ -370,11 +372,10 @@ robots.txt exclusion
A spider middleware to avoid visiting pages forbidden by robots.txt: A spider middleware to avoid visiting pages forbidden by robots.txt:
:: .. code-block:: python
#!python #!python
class SpiderInfo(object): class SpiderInfo(object):
def __init__(self, useragent): def __init__(self, useragent):
self.useragent = useragent self.useragent = useragent
self.parsers = {} self.parsers = {}
@ -382,7 +383,6 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
class AllowAllParser(object): class AllowAllParser(object):
def can_fetch(useragent, url): def can_fetch(useragent, url):
return True return True
@ -397,7 +397,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
dispatcher.connect(self.spider_closed, signal=signals.spider_closed) dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
def process_request(self, request, response, spider): def process_request(self, request, response, spider):
return self.process_start_request(self, request) return self.process_start_request(request)
def process_start_request(self, request, spider): def process_start_request(self, request, spider):
info = self.spiders[spider] info = self.spiders[spider]
@ -415,17 +415,21 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
res = None res = None
else: else:
robotsurl = "%s://%s/robots.txt" % (url.scheme, netloc) robotsurl = "%s://%s/robots.txt" % (url.scheme, netloc)
meta = {'spider': spider, {'handle_httpstatus_list': [403, 404, 500]} meta = {"spider": spider, "handle_httpstatus_list": [403, 404, 500]}
res = Request(robotsurl, callback=self.parse_robots, res = Request(
meta=meta, priority=self.REQUEST_PRIORITY) robotsurl,
callback=self.parse_robots,
meta=meta,
priority=self.REQUEST_PRIORITY,
)
info.pending[netloc].append(request) info.pending[netloc].append(request)
return res return res
def parse_robots(self, response): def parse_robots(self, response):
spider = response.request.meta['spider'] spider = response.request.meta["spider"]
netloc urlparse_cached(response).netloc netloc = urlparse_cached(response).netloc
info = self.spiders[spider] info = self.spiders[spider]
if response.status 200; if response.status == 200:
rp = robotparser.RobotFileParser(response.url) rp = robotparser.RobotFileParser(response.url)
rp.parse(response.body.splitlines()) rp.parse(response.body.splitlines())
info.parsers[netloc] = rp info.parsers[netloc] = rp
@ -434,7 +438,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
return info.pending[netloc] return info.pending[netloc]
def spider_opened(self, spider): def spider_opened(self, spider):
ua = getattr(spider, 'user_agent', None) or settings['USER_AGENT'] ua = getattr(spider, "user_agent", None) or settings["USER_AGENT"]
self.spiders[spider] = SpiderInfo(ua) self.spiders[spider] = SpiderInfo(ua)
def spider_closed(self, spider): def spider_closed(self, spider):
@ -445,18 +449,16 @@ Offsite middleware
This is a port of the Offsite middleware to the new spider middleware API: This is a port of the Offsite middleware to the new spider middleware API:
:: .. code-block:: python
#!python #!python
class SpiderInfo(object): class SpiderInfo(object):
def __init__(self, host_regex): def __init__(self, host_regex):
self.host_regex = host_regex self.host_regex = host_regex
self.hosts_seen = set() self.hosts_seen = set()
class OffsiteMiddleware(object): class OffsiteMiddleware(object):
def __init__(self): def __init__(self):
self.spiders = {} self.spiders = {}
dispatcher.connect(self.spider_opened, signal=signals.spider_opened) dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
@ -472,19 +474,19 @@ This is a port of the Offsite middleware to the new spider middleware API:
info = self.spiders[spider] info = self.spiders[spider]
host = urlparse_cached(x).hostname host = urlparse_cached(x).hostname
if host and host not in info.hosts_seen: if host and host not in info.hosts_seen:
spider.log("Filtered offsite request to %r: %s" % (host, request)) spider.log("Filtered offsite request to %r: %s" % (host, request))
info.hosts_seen.add(host) info.hosts_seen.add(host)
def should_follow(self, request, spider): def should_follow(self, request, spider):
info = self.spiders[spider] info = self.spiders[spider]
# hostname can be None for wrong urls (like javascript links) # hostname can be None for wrong urls (like javascript links)
host = urlparse_cached(request).hostname or '' host = urlparse_cached(request).hostname or ""
return bool(info.regex.search(host)) return bool(info.regex.search(host))
def get_host_regex(self, spider): def get_host_regex(self, spider):
"""Override this method to implement a different offsite policy""" """Override this method to implement a different offsite policy"""
domains = [d.replace('.', r'\.') for d in spider.allowed_domains] domains = [d.replace(".", r"\.") for d in spider.allowed_domains]
regex = r'^(.*\.)?(%s)$' % '|'.join(domains) regex = r"^(.*\.)?(%s)$" % "|".join(domains)
return re.compile(regex) return re.compile(regex)
def spider_opened(self, spider): def spider_opened(self, spider):
@ -499,35 +501,36 @@ Limit URL length
A middleware to filter out requests with long urls: A middleware to filter out requests with long urls:
:: .. code-block:: python
#!python #!python
class LimitUrlLength(object):
class LimitUrlLength(object):
def __init__(self): def __init__(self):
self.maxlength = settings.getint('URLLENGTH_LIMIT') self.maxlength = settings.getint("URLLENGTH_LIMIT")
def process_request(self, request, response, spider): def process_request(self, request, response, spider):
return self.process_start_request(self, request) return self.process_start_request(self, request)
def process_start_request(self, request, spider): def process_start_request(self, request, spider):
if len(request.url) <= self.maxlength: if len(request.url) <= self.maxlength:
return request return request
spider.log("Ignoring request (url length > %d): %s " % (self.maxlength, request.url)) spider.log(
"Ignoring request (url length > %d): %s " % (self.maxlength, request.url)
)
Set Referer Set Referer
----------- -----------
A middleware to set the Referer: A middleware to set the Referer:
:: .. code-block:: python
#!python #!python
class SetReferer(object): class SetReferer(object):
def process_request(self, request, response, spider): def process_request(self, request, response, spider):
request.headers.setdefault('Referer', response.url) request.headers.setdefault("Referer", response.url)
return request return request
Set and limit crawling depth Set and limit crawling depth
@ -536,23 +539,22 @@ Set and limit crawling depth
A middleware to set (and limit) the request/response depth, taken from the A middleware to set (and limit) the request/response depth, taken from the
start requests: start requests:
:: .. code-block:: python
#!python #!python
class SetLimitDepth(object): class SetLimitDepth(object):
def __init__(self, maxdepth=0): def __init__(self, maxdepth=0):
self.maxdepth = maxdepth or settings.getint('DEPTH_LIMIT') self.maxdepth = maxdepth or settings.getint("DEPTH_LIMIT")
def process_request(self, request, response, spider): def process_request(self, request, response, spider):
depth = response.request.meta['depth'] + 1 depth = response.request.meta["depth"] + 1
request.meta['depth'] = depth request.meta["depth"] = depth
if not self.maxdepth or depth <= self.maxdepth: if not self.maxdepth or depth <= self.maxdepth:
return request return request
spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request) spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request))
def process_start_request(self, request, spider): def process_start_request(self, request, spider):
request.meta['depth'] = 0 request.meta["depth"] = 0
return request return request
Filter duplicate requests Filter duplicate requests
@ -560,17 +562,16 @@ Filter duplicate requests
A middleware to filter out requests already seen: A middleware to filter out requests already seen:
:: .. code-block:: python
#!python #!python
class FilterDuplicates(object): class FilterDuplicates(object):
def __init__(self): def __init__(self):
clspath = settings.get('DUPEFILTER_CLASS') clspath = settings.get("DUPEFILTER_CLASS")
self.dupefilter = load_object(clspath)() self.dupefilter = load_object(clspath)()
dispatcher.connect(self.spider_opened, signal=signals.spider_opened) dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
dispatcher.connect(self.spider_closed, signal=signals.spider_closed) dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
def enqueue_request(self, spider, request): def enqueue_request(self, spider, request):
seen = self.dupefilter.request_seen(spider, request) seen = self.dupefilter.request_seen(spider, request)
if not seen or request.dont_filter: if not seen or request.dont_filter:
@ -587,22 +588,25 @@ Scrape data using Parsley
A middleware to Scrape data using Parsley as described in UsingParsley A middleware to Scrape data using Parsley as described in UsingParsley
:: .. code-block:: python
#!python #!python
from pyparsley import PyParsley from pyparsley import PyParsley
class ParsleyExtractor(object):
class ParsleyExtractor(object):
def __init__(self, parsley_json_code): def __init__(self, parsley_json_code):
parsley = json.loads(parselet_json_code) parsley = json.loads(parselet_json_code)
class ParsleyItem(Item): class ParsleyItem(Item):
def __init__(self, *a, **kw): def __init__(self, *a, **kw):
for name in parsley.keys(): for name in parsley.keys():
self.fields[name] = Field() self.fields[name] = Field()
super(ParsleyItem, self).__init__(*a, **kw) super(ParsleyItem, self).__init__(*a, **kw)
self.item_class = ParsleyItem
self.parsley = PyParsley(parsley, output='python') self.item_class = ParsleyItem
self.parsley = PyParsley(parsley, output="python")
def process_response(self, response, request, spider): def process_response(self, response, request, spider):
return self.item_class(self.parsley.parse(string=response.body)) return self.item_class(self.parsley.parse(string=response.body))

10
sep/sep-019.rst
View File

@ -15,10 +15,11 @@ consistent way, while taking the chance to refactor the settings population
and whole crawl workflow. and whole crawl workflow.
In short, you will be able to overwrite settings (on a per-spider basis) by In short, you will be able to overwrite settings (on a per-spider basis) by
implementing a class method in your spider:: implementing a class method in your spider:
.. code-block:: python
class MySpider(Spider): class MySpider(Spider):
@classmethod @classmethod
def custom_settings(cls): def custom_settings(cls):
return { return {
@ -197,10 +198,11 @@ Spiders
A new class method ``custom_settings`` is proposed, that could be use to A new class method ``custom_settings`` is proposed, that could be use to
override project and default settings before they're used to instantiate the override project and default settings before they're used to instantiate the
crawler:: crawler:
.. code-block:: python
class MySpider(Spider): class MySpider(Spider):
@classmethod @classmethod
def custom_settings(cls): def custom_settings(cls):
return { return {

50
sep/sep-020.rst
View File

@ -54,18 +54,18 @@ required.
Before Before
------ ------
:: .. code-block:: python
xpath = '//div[@class="geeks"]/dl/dt[contains(text(),"%s")]/following-sibling::dd[1]//text()' xpath = '//div[@class="geeks"]/dl/dt[contains(text(),"%s")]/following-sibling::dd[1]//text()'
gl = XPathItemLoader(response=response, item=dict()) gl = XPathItemLoader(response=response, item=dict())
gl.default_output_processor = Compose(TakeFirst(), lambda v: v.strip()) gl.default_output_processor = Compose(TakeFirst(), lambda v: v.strip())
gl.add_xpath('hacker', xpath % 'hacker') gl.add_xpath("hacker", xpath % "hacker")
gl.add_xpath('nerd', xpath % 'nerd') gl.add_xpath("nerd", xpath % "nerd")
After After
----- -----
:: .. code-block:: python
bil = BulkItemLoader(response=response) bil = BulkItemLoader(response=response)
bil.parse_dl('//div[@class="geeks"]/dl') bil.parse_dl('//div[@class="geeks"]/dl')
@ -75,33 +75,34 @@ Code Proposal
This is a working code sample that covers just the basics. This is a working code sample that covers just the basics.
:: .. code-block:: python
from scrapy.contrib.loader import XPathItemLoader from scrapy.contrib.loader import XPathItemLoader
from scrapy.contrib.loader.processor import MapCompose from scrapy.contrib.loader.processor import MapCompose
class BulkItemLoader(XPathItemLoader): class BulkItemLoader(XPathItemLoader):
""" Item loader based on specified pattern recognition """Item loader based on specified pattern recognition"""
"""
default_item_class = dict default_item_class = dict
base_xpath = '//body' base_xpath = "//body"
ignore = () ignore = ()
def _get_label(self, entity): def _get_label(self, entity):
""" Pull the text label out of selected markup """Pull the text label out of selected markup
:param entity: Found markup :param entity: Found markup
:type entity: Selector :type entity: Selector
""" """
label = ' '.join(entity.xpath('.//text()').extract()) label = " ".join(entity.xpath(".//text()").extract())
label = label.encode('ascii', 'xmlcharrefreplace') if label else '' label = label.encode("ascii", "xmlcharrefreplace") if label else ""
label = label.strip('&#160;') if '&#160;' in label else label label = label.strip("&#160;") if "&#160;" in label else label
label = label.strip(':') if ':' in label else label label = label.strip(":") if ":" in label else label
label = label.strip() label = label.strip()
return label return label
def _get_entities(self, xpath): def _get_entities(self, xpath):
""" Retrieve the list of selectors for a given sub-pattern """Retrieve the list of selectors for a given sub-pattern
:param xpath: The xpath to select :param xpath: The xpath to select
:type xpath: String :type xpath: String
@ -110,20 +111,21 @@ This is a working code sample that covers just the basics.
""" """
return self.selector.xpath(self.base_xpath + xpath) return self.selector.xpath(self.base_xpath + xpath)
def parse_dl(self, xpath=u'//dl'): def parse_dl(self, xpath="//dl"):
""" Look for the specified definition list pattern and store all found """Look for the specified definition list pattern and store all found
values for the enclosed terms and descriptions. values for the enclosed terms and descriptions.
:param xpath: The xpath to select :param xpath: The xpath to select
:type xpath: String :type xpath: String
""" """
for term in self._get_entities(xpath + '/dt'): for term in self._get_entities(xpath + "/dt"):
label = self._get_label(term) label = self._get_label(term)
if label and label not in self.ignore: if label and label not in self.ignore:
value = term.xpath('following-sibling::dd[1]//text()') value = term.xpath("following-sibling::dd[1]//text()")
if value: if value:
self.add_value(label, value.extract(), self.add_value(
MapCompose(lambda v: v.strip())) label, value.extract(), MapCompose(lambda v: v.strip())
)
Example Spider Example Spider
============== ==============
@ -133,22 +135,24 @@ This spider uses the bulk loader above.
Spider code Spider code
----------- -----------
:: .. code-block:: python
from scrapy.spider import BaseSpider from scrapy.spider import BaseSpider
from scrapy.contrib.loader.bulk import BulkItemLoader from scrapy.contrib.loader.bulk import BulkItemLoader
class W3cSpider(BaseSpider): class W3cSpider(BaseSpider):
name = "w3c" name = "w3c"
allowed_domains = ["w3.org"] allowed_domains = ["w3.org"]
start_urls = ('http://www.w3.org/TR/html401/struct/lists.html',) start_urls = ("http://www.w3.org/TR/html401/struct/lists.html",)
def parse(self, response): def parse(self, response):
el = BulkItemLoader(response=response) el = BulkItemLoader(response=response)
el.parse_dl('//dl[2]') el.parse_dl("//dl[2]")
item = el.load_item() item = el.load_item()
from pprint import pprint from pprint import pprint
pprint(item) pprint(item)
Log Output Log Output

20
sep/sep-021.rst
View File

@ -76,14 +76,18 @@ addon_configure
Receives the Settings object and modifies it to enable the required components. Receives the Settings object and modifies it to enable the required components.
If it raises an exception, Scrapy will print it and exit. If it raises an exception, Scrapy will print it and exit.
Examples:: Examples:
.. code-block:: python
def addon_configure(settings): def addon_configure(settings):
settings.overrides['DOWNLOADER_MIDDLEWARES'].update({ settings.overrides["DOWNLOADER_MIDDLEWARES"].update(
'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900, {
}) "scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware": 900,
}
)
:: .. code-block:: python
def addon_configure(settings): def addon_configure(settings):
try: try:
@ -100,8 +104,10 @@ is meant to be used to perform post-initialization checks like making sure the
extension and its dependencies were configured properly. If it raises an extension and its dependencies were configured properly. If it raises an
exception, Scrapy will print and exit. exception, Scrapy will print and exit.
Examples:: Examples:
.. code-block:: python
def crawler_ready(crawler): def crawler_ready(crawler):
if 'some.other.addon' not in crawler.extensions.enabled: if "some.other.addon" not in crawler.extensions.enabled:
raise RuntimeError("Some other addon is required to use this addon") raise RuntimeError("Some other addon is required to use this addon")