RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add and run pre-commit hook 'blacken-docs' 

Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
This commit is contained in:
pankaj1707k 2023-02-01 16:30:57 +05:30
parent b337c986ca
commit c1bbb299d7
No known key found for this signature in database
GPG Key ID: 6757E896F6BC635E
50 changed files with 1821 additions and 1260 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.pre-commit-config.yaml
View File

@ -16,3 +16,9 @@ repos:
rev: 5.12.0
hooks:
- id: isort
- repo: https://github.com/adamchainz/blacken-docs
rev: 1.13.0
hooks:
- id: blacken-docs
additional_dependencies:
- black==22.12.0

44
docs/faq.rst
View File

@ -35,8 +35,9 @@ for parsing HTML responses in Scrapy callbacks.
You just have to feed the response's body into a ``BeautifulSoup`` object
and extract whatever data you need from it.
Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser::
Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser:
.. code-block:: python
from bs4 import BeautifulSoup
import scrapy
@ -45,17 +46,12 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML pars
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = (
'http://www.example.com/',
)
start_urls = ("http://www.example.com/",)
def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, 'lxml')
yield {
"url": response.url,
"title": soup.h1.string
}
soup = BeautifulSoup(response.text, "lxml")
yield {"url": response.url, "title": soup.h1.string}
.. note::
@ -109,11 +105,13 @@ basically means that it crawls in `DFO order`_. This order is more convenient
in most cases.
If you do want to crawl in true `BFO order`_, you can do it by
setting the following settings::
setting the following settings:
.. code-block:: python
DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"
While pending requests are below the configured values of
:setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or
@ -159,11 +157,13 @@ See also other suggestions at `StackOverflow`_.
.. note:: Remember to disable
:class:`scrapy.spidermiddlewares.offsite.OffsiteMiddleware` when you enable
your custom implementation::
your custom implementation:
.. code-block:: python
SPIDER_MIDDLEWARES = {
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,
'myproject.middlewares.CustomOffsiteMiddleware': 500,
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
"myproject.middlewares.CustomOffsiteMiddleware": 500,
}
.. _meet the installation requirements: https://github.com/andreasvc/pyre2#installation
@ -235,11 +235,13 @@ What does the response status code 999 means?
999 is a custom response status code used by Yahoo sites to throttle requests.
Try slowing down the crawling speed by using a download delay of ``2`` (or
higher) in your spider::
higher) in your spider:
.. code-block:: python
class MySpider(CrawlSpider):
name = 'myspider'
name = "myspider"
download_delay = 2
@ -351,19 +353,21 @@ How to split an item into multiple items in an item pipeline?
input item. :ref:`Create a spider middleware <custom-spider-middleware>`
instead, and use its
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method for this purpose. For example::
method for this purpose. For example:
.. code-block:: python
from copy import deepcopy
from itemadapter import is_item, ItemAdapter
class MultiplyItemsMiddleware:
class MultiplyItemsMiddleware:
def process_spider_output(self, response, result, spider):
for item in result:
if is_item(item):
adapter = ItemAdapter(item)
for _ in range(adapter['multiply_by']):
for _ in range(adapter["multiply_by"]):
yield deepcopy(item)
Does Scrapy support IPv6 addresses?

14
docs/intro/overview.rst
View File

@ -20,22 +20,24 @@ In order to show you what Scrapy brings to the table, we'll walk you through an
example of a Scrapy Spider using the simplest way to run a spider.
Here's the code for a spider that scrapes famous quotes from website
https://quotes.toscrape.com, following the pagination::
https://quotes.toscrape.com, following the pagination
.. code-block:: python
import scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes'
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/tag/humor/',
"https://quotes.toscrape.com/tag/humor/",
]
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'author': quote.xpath('span/small/text()').get(),
'text': quote.css('span.text::text').get(),
"author": quote.xpath("span/small/text()").get(),
"text": quote.css("span.text::text").get(),
}
next_page = response.css('li.next a::attr("href")').get()

120
docs/intro/tutorial.rst
View File

@ -177,7 +177,9 @@ that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,
you can just define a :attr:`~scrapy.Spider.start_urls` class attribute
with a list of URLs. This list will then be used by the default implementation
of :meth:`~scrapy.Spider.start_requests` to create the initial requests
for your spider::
for your spider.
.. code-block:: python
from pathlib import Path
@ -187,13 +189,13 @@ for your spider::
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
'https://quotes.toscrape.com/page/2/',
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
def parse(self, response):
page = response.url.split("/")[-2]
filename = f'quotes-{page}.html'
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
The :meth:`~scrapy.Spider.parse` method will be called to handle each
@ -438,7 +440,9 @@ extraction logic above into our spider.
A Scrapy spider typically generates many dictionaries containing the data
extracted from the page. To do that, we use the ``yield`` Python keyword
in the callback, as you can see below::
in the callback, as you can see below:
.. code-block:: python
import scrapy
@ -446,16 +450,16 @@ in the callback, as you can see below::
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
'https://quotes.toscrape.com/page/2/',
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
If you run this spider, it will output the extracted data with the log::
@ -543,7 +547,9 @@ There is also an ``attrib`` property available
'/page/2/'
Let's see now our spider modified to recursively follow the link to the next
page, extracting data from it::
page, extracting data from it:
.. code-block:: python
import scrapy
@ -551,18 +557,18 @@ page, extracting data from it::
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
"https://quotes.toscrape.com/page/1/",
]
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
next_page = response.css('li.next a::attr(href)').get()
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
@ -594,7 +600,9 @@ A shortcut for creating Requests
--------------------------------
As a shortcut for creating Request objects you can use
:meth:`response.follow <scrapy.http.TextResponse.follow>`::
:meth:`response.follow <scrapy.http.TextResponse.follow>`
.. code-block:: python
import scrapy
@ -602,18 +610,18 @@ As a shortcut for creating Request objects you can use
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'https://quotes.toscrape.com/page/1/',
"https://quotes.toscrape.com/page/1/",
]
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('span small::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
"text": quote.css("span.text::text").get(),
"author": quote.css("span small::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
next_page = response.css('li.next a::attr(href)').get()
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
@ -622,57 +630,67 @@ need to call urljoin. Note that ``response.follow`` just returns a Request
instance; you still have to yield this Request.
You can also pass a selector to ``response.follow`` instead of a string;
this selector should extract necessary attributes::
this selector should extract necessary attributes:
for href in response.css('ul.pager a::attr(href)'):
.. code-block:: python
for href in response.css("ul.pager a::attr(href)"):
yield response.follow(href, callback=self.parse)
For ``<a>`` elements there is a shortcut: ``response.follow`` uses their href
attribute automatically. So the code can be shortened further::
attribute automatically. So the code can be shortened further:
for a in response.css('ul.pager a'):
.. code-block:: python
for a in response.css("ul.pager a"):
yield response.follow(a, callback=self.parse)
To create multiple requests from an iterable, you can use
:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead::
:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead:
anchors = response.css('ul.pager a')
.. code-block:: python
anchors = response.css("ul.pager a")
yield from response.follow_all(anchors, callback=self.parse)
or, shortening it further::
or, shortening it further:
yield from response.follow_all(css='ul.pager a', callback=self.parse)
.. code-block:: python
yield from response.follow_all(css="ul.pager a", callback=self.parse)
More examples and patterns
--------------------------
Here is another spider that illustrates callbacks and following links,
this time for scraping author information::
this time for scraping author information:
.. code-block:: python
import scrapy
class AuthorSpider(scrapy.Spider):
name = 'author'
name = "author"
start_urls = ['https://quotes.toscrape.com/']
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
author_page_links = response.css('.author + a')
author_page_links = response.css(".author + a")
yield from response.follow_all(author_page_links, self.parse_author)
pagination_links = response.css('li.next a')
pagination_links = response.css("li.next a")
yield from response.follow_all(pagination_links, self.parse)
def parse_author(self, response):
def extract_with_css(query):
return response.css(query).get(default='').strip()
return response.css(query).get(default="").strip()
yield {
'name': extract_with_css('h3.author-title::text'),
'birthdate': extract_with_css('.author-born-date::text'),
'bio': extract_with_css('.author-description::text'),
"name": extract_with_css("h3.author-title::text"),
"birthdate": extract_with_css(".author-born-date::text"),
"bio": extract_with_css(".author-description::text"),
}
This spider will start from the main page, it will follow all the links to the
@ -720,7 +738,9 @@ spider attributes by default.
In this example, the value provided for the ``tag`` argument will be available
via ``self.tag``. You can use this to make your spider fetch only quotes
with a specific tag, building the URL based on the argument::
with a specific tag, building the URL based on the argument:
.. code-block:: python
import scrapy
@ -729,20 +749,20 @@ with a specific tag, building the URL based on the argument::
name = "quotes"
def start_requests(self):
url = 'https://quotes.toscrape.com/'
tag = getattr(self, 'tag', None)
url = "https://quotes.toscrape.com/"
tag = getattr(self, "tag", None)
if tag is not None:
url = url + 'tag/' + tag
url = url + "tag/" + tag
yield scrapy.Request(url, self.parse)
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
}
next_page = response.css('li.next a::attr(href)').get()
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, self.parse)

6
docs/topics/asyncio.rst
View File

@ -106,12 +106,14 @@ Enforcing asyncio as a requirement
If you are writing a :ref:`component <topics-components>` that requires asyncio
to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to
:ref:`enforce it as a requirement <enforce-component-requirements>`. For
example::
example:
.. code-block:: python
from scrapy.utils.reactor import is_asyncio_reactor_installed
class MyComponent:
class MyComponent:
def __init__(self):
if not is_asyncio_reactor_installed():
raise ValueError(

40
docs/topics/broad-crawls.rst
View File

@ -48,9 +48,11 @@ Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQ
It works best during single-domain crawl. It does not work well with crawling
many different domains in parallel
To apply the recommended priority queue use::
To apply the recommended priority queue use:
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'
.. code-block:: python
SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue"
.. _broad-crawls-concurrency:
@ -71,7 +73,9 @@ many different domains in parallel, so you will want to increase it. How much
to increase it will depend on how much CPU and memory your crawler will have
available.
A good starting point is ``100``::
A good starting point is ``100``:
.. code-block:: python
CONCURRENT_REQUESTS = 100
@ -92,7 +96,9 @@ hitting DNS resolver timeouts. Possible solution to increase the number of
threads handling DNS queries. The DNS queue will be processed faster speeding
up establishing of connection and crawling overall.
To increase maximum thread pool size use::
To increase maximum thread pool size use:
.. code-block:: python
REACTOR_THREADPOOL_MAXSIZE = 20
@ -114,9 +120,11 @@ should not use ``DEBUG`` log level when preforming large broad crawls in
production. Using ``DEBUG`` level when developing your (broad) crawler may be
fine though.
To set the log level use::
To set the log level use:
LOG_LEVEL = 'INFO'
.. code-block:: python
LOG_LEVEL = "INFO"
Disable cookies
===============
@ -126,7 +134,9 @@ doing broad crawls (search engine crawlers ignore them), and they improve
performance by saving some CPU cycles and reducing the memory footprint of your
Scrapy crawler.
To disable cookies use::
To disable cookies use:
.. code-block:: python
COOKIES_ENABLED = False
@ -138,7 +148,9 @@ when sites causes are very slow (or fail) to respond, thus causing a timeout
error which gets retried many times, unnecessarily, preventing crawler capacity
to be reused for other domains.
To disable retries use::
To disable retries use:
.. code-block:: python
RETRY_ENABLED = False
@ -149,7 +161,9 @@ Unless you are crawling from a very slow connection (which shouldn't be the
case for broad crawls) reduce the download timeout so that stuck requests are
discarded quickly and free up capacity to process the next ones.
To reduce the download timeout use::
To reduce the download timeout use:
.. code-block:: python
DOWNLOAD_TIMEOUT = 15
@ -162,7 +176,9 @@ revisiting the site at a later crawl. This also help to keep the number of
request constant per crawl batch, otherwise redirect loops may cause the
crawler to dedicate too many resources on any specific domain.
To disable redirects use::
To disable redirects use:
.. code-block:: python
REDIRECT_ENABLED = False
@ -179,7 +195,9 @@ Pages can indicate it in two ways:
"main", "index" website pages.
Scrapy handles (1) automatically; to handle (2) enable
:ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`::
:ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`:
.. code-block:: python
AJAXCRAWL_ENABLED = True

17
docs/topics/commands.rst
View File

@ -617,7 +617,7 @@ Example:
.. code-block:: python
COMMANDS_MODULE = 'mybot.commands'
COMMANDS_MODULE = "mybot.commands"
.. _Deploying your project: https://scrapyd.readthedocs.io/en/latest/deploy.html
@ -636,10 +636,11 @@ The following example adds ``my_command`` command:
from setuptools import setup, find_packages
setup(name='scrapy-mymodule',
entry_points={
'scrapy.commands': [
'my_command=my_scrapy_module.commands:MyCommand',
],
},
)
setup(
name="scrapy-mymodule",
entry_points={
"scrapy.commands": [
"my_command=my_scrapy_module.commands:MyCommand",
],
},
)

8
docs/topics/components.rst
View File

@ -66,16 +66,18 @@ version mismatch, while :exc:`ValueError` may be better if the issue is the
value of a setting.
If your requirement is a minimum Scrapy version, you may use
:attr:`scrapy.__version__` to enforce your requirement. For example::
:attr:`scrapy.__version__` to enforce your requirement. For example:
.. code-block:: python
from pkg_resources import parse_version
import scrapy
class MyComponent:
class MyComponent:
def __init__(self):
if parse_version(scrapy.__version__) < parse_version('2.7'):
if parse_version(scrapy.__version__) < parse_version("2.7"):
raise RuntimeError(
f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
f"later, which allow defining the process_spider_output "

38
docs/topics/contracts.rst
View File

@ -11,10 +11,13 @@ integrated way of testing your spiders by the means of contracts.
This allows you to test each callback of your spider by hardcoding a sample url
and check various constraints for how the callback processes the response. Each
contract is prefixed with an ``@`` and included in the docstring. See the
following example::
following example:
.. code-block:: python
def parse(self, response):
""" This function parses a sample response. Some contracts are mingled
"""
This function parses a sample response. Some contracts are mingled
with this docstring.
@url http://www.amazon.com/s?field-keywords=selfish+gene
@ -64,11 +67,13 @@ Custom Contracts
If you find you need more power than the built-in Scrapy contracts you can
create and load your own contracts in the project by using the
:setting:`SPIDER_CONTRACTS` setting::
:setting:`SPIDER_CONTRACTS` setting:
.. code-block:: python
SPIDER_CONTRACTS = {
'myproject.contracts.ResponseCheck': 10,
'myproject.contracts.ItemValidate': 10,
"myproject.contracts.ResponseCheck": 10,
"myproject.contracts.ItemValidate": 10,
}
Each contract must inherit from :class:`~scrapy.contracts.Contract` and can
@ -111,22 +116,26 @@ Raise :class:`~scrapy.exceptions.ContractFail` from
.. autoclass:: scrapy.exceptions.ContractFail
Here is a demo contract which checks the presence of a custom header in the
response received::
response received:
.. code-block:: python
from scrapy.contracts import Contract
from scrapy.exceptions import ContractFail
class HasHeaderContract(Contract):
""" Demo contract which checks the presence of a custom header
@has_header X-CustomHeader
"""
Demo contract which checks the presence of a custom header
@has_header X-CustomHeader
"""
name = 'has_header'
name = "has_header"
def pre_process(self, response):
for header in self.args:
if header not in response.headers:
raise ContractFail('X-CustomHeader not present')
raise ContractFail("X-CustomHeader not present")
.. _detecting-contract-check-runs:
@ -135,14 +144,17 @@ Detecting check runs
When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is
set to the ``true`` string. You can use :data:`os.environ` to perform any change to
your spiders or your settings when ``scrapy check`` is used::
your spiders or your settings when ``scrapy check`` is used:
.. code-block:: python
import os
import scrapy
class ExampleSpider(scrapy.Spider):
name = 'example'
name = "example"
def __init__(self):
if os.environ.get('SCRAPY_CHECK'):
if os.environ.get("SCRAPY_CHECK"):
pass # Do some scraper adjustments when a check is running

29
docs/topics/coroutines.rst
View File

@ -58,49 +58,58 @@ There are several use cases for coroutines in Scrapy.
Code that would return Deferreds when written for previous Scrapy versions,
such as downloader middlewares and signal handlers, can be rewritten to be
shorter and cleaner::
shorter and cleaner:
.. code-block:: python
from itemadapter import ItemAdapter
class DbPipeline:
def _update_item(self, data, item):
adapter = ItemAdapter(item)
adapter['field'] = data
adapter["field"] = data
return item
def process_item(self, item, spider):
adapter = ItemAdapter(item)
dfd = db.get_some_data(adapter['id'])
dfd = db.get_some_data(adapter["id"])
dfd.addCallback(self._update_item, item)
return dfd
becomes::
becomes:
.. code-block:: python
from itemadapter import ItemAdapter
class DbPipeline:
async def process_item(self, item, spider):
adapter = ItemAdapter(item)
adapter['field'] = await db.get_some_data(adapter['id'])
adapter["field"] = await db.get_some_data(adapter["id"])
return item
Coroutines may be used to call asynchronous code. This includes other
coroutines, functions that return Deferreds and functions that return
:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.
This means you can use many useful Python libraries providing such code::
This means you can use many useful Python libraries providing such code:
.. code-block:: python
class MySpiderDeferred(Spider):
# ...
async def parse(self, response):
additional_response = await treq.get('https://additional.url')
additional_response = await treq.get("https://additional.url")
additional_data = await treq.content(additional_response)
# ... use response and additional_data to yield items and requests
class MySpiderAsyncio(Spider):
# ...
async def parse(self, response):
async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response:
async with session.get("https://additional.url") as additional_response:
additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests
@ -192,7 +201,9 @@ while maintaining support for older Scrapy versions, you may define
:term:`asynchronous generator` version of that method with an alternative name:
``process_spider_output_async``.
For example::
For example:
.. code-block:: python
class UniversalSpiderMiddleware:
def process_spider_output(self, response, result, spider):

35
docs/topics/debug.rst
View File

@ -5,21 +5,24 @@ Debugging Spiders
=================
This document explains the most common techniques for debugging spiders.
Consider the following Scrapy spider below::
Consider the following Scrapy spider below:
.. code-block:: python
import scrapy
from myproject.items import MyItem
class MySpider(scrapy.Spider):
name = 'myspider'
name = "myspider"
start_urls = (
'http://example.com/page1',
'http://example.com/page2',
)
"http://example.com/page1",
"http://example.com/page2",
)
def parse(self, response):
# <processing code not shown>
# collect `item_urls`
# collect `item_urls`
for item_url in item_urls:
yield scrapy.Request(item_url, self.parse_item)
@ -28,7 +31,9 @@ Consider the following Scrapy spider below::
item = MyItem()
# populate `item` fields
# and extract item_details_url
yield scrapy.Request(item_details_url, self.parse_details, cb_kwargs={'item': item})
yield scrapy.Request(
item_details_url, self.parse_details, cb_kwargs={"item": item}
)
def parse_details(self, response, item):
# populate more `item` fields
@ -103,10 +108,13 @@ showing the response received and the output. How to debug the situation when
.. highlight:: python
Fortunately, the :command:`shell` is your bread and butter in this case (see
:ref:`topics-shell-inspect-response`)::
:ref:`topics-shell-inspect-response`):
.. code-block:: python
from scrapy.shell import inspect_response
def parse_details(self, response, item=None):
if item:
# populate more `item` fields
@ -121,10 +129,13 @@ Open in browser
Sometimes you just want to see how a certain response looks in a browser, you
can use the ``open_in_browser`` function for that. Here is an example of how
you would use it::
you would use it:
.. code-block:: python
from scrapy.utils.response import open_in_browser
def parse_details(self, response):
if "item name" not in response.body:
open_in_browser(response)
@ -138,14 +149,16 @@ Logging
Logging is another useful option for getting information about your spider run.
Although not as convenient, it comes with the advantage that the logs will be
available in all future runs should they be necessary again::
available in all future runs should they be necessary again:
.. code-block:: python
def parse_details(self, response, item=None):
if item:
# populate more `item` fields
return item
else:
self.logger.warning('No item received for %s', response.url)
self.logger.warning("No item received for %s", response.url)
For more information, check the :ref:`topics-logging` section.

17
docs/topics/developer-tools.rst
View File

@ -237,17 +237,19 @@ on the request and open ``Open in new tab`` to get a better overview.
:alt: JSON-object returned from the quotes.toscrape API
With this response we can now easily parse the JSON-object and
also request each page to get every quote on the site::
also request each page to get every quote on the site:
.. code-block:: python
import scrapy
import json
class QuoteSpider(scrapy.Spider):
name = 'quote'
allowed_domains = ['quotes.toscrape.com']
name = "quote"
allowed_domains = ["quotes.toscrape.com"]
page = 1
start_urls = ['https://quotes.toscrape.com/api/quotes?page=1']
start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]
def parse(self, response):
data = json.loads(response.text)
@ -275,7 +277,9 @@ requests, as we could need to add ``headers`` or ``cookies`` to make it work.
In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
format, by right-clicking on each of them in the network tool and using the
:meth:`~scrapy.Request.from_curl()` method to generate an equivalent
request::
request:
.. code-block:: python
from scrapy import Request
@ -286,7 +290,8 @@ request::
"-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM"
"zEwZTAxLTk5MWUtNDFiNC1iZWRmLTJjNGI4M2ZiNDBmNDpAVEstMDMzMTBlMDEtOTkxZS00MW"
"I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http"
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'"
)
Alternatively, if you want to know the arguments needed to recreate that
request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`

50
docs/topics/downloader-middleware.rst
View File

@ -17,10 +17,12 @@ To activate a downloader middleware component, add it to the
:setting:`DOWNLOADER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class paths and their values are the middleware orders.
Here's an example::
Here's an example:
.. code-block:: python
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543,
"myproject.middlewares.CustomDownloaderMiddleware": 543,
}
The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
@ -42,11 +44,13 @@ previous (or subsequent) middleware being applied.
If you want to disable a built-in middleware (the ones defined in
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None``
as its value. For example, if you want to disable the user-agent middleware::
as its value. For example, if you want to disable the user-agent middleware:
.. code-block:: python
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
"myproject.middlewares.CustomDownloaderMiddleware": 543,
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
}
Finally, keep in mind that some middlewares may need to be enabled through a
@ -226,20 +230,25 @@ There is support for keeping multiple cookie sessions per spider by using the
:reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar
(session), but you can pass an identifier to use different ones.
For example::
For example:
.. code-block:: python
for i, url in enumerate(urls):
yield scrapy.Request(url, meta={'cookiejar': i},
callback=self.parse_page)
yield scrapy.Request(url, meta={"cookiejar": i}, callback=self.parse_page)
Keep in mind that the :reqmeta:`cookiejar` meta key is not "sticky". You need to keep
passing it along on subsequent requests. For example::
passing it along on subsequent requests. For example:
.. code-block:: python
def parse_page(self, response):
# do some processing
return scrapy.Request("http://www.example.com/otherpage",
meta={'cookiejar': response.meta['cookiejar']},
callback=self.parse_other_page)
return scrapy.Request(
"http://www.example.com/otherpage",
meta={"cookiejar": response.meta["cookiejar"]},
callback=self.parse_other_page,
)
.. setting:: COOKIES_ENABLED
@ -339,16 +348,19 @@ HttpAuthMiddleware
domain of the first request, which will work for some spiders but not
for others. In the future the middleware will produce an error instead.
Example::
Example:
.. code-block:: python
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
http_auth_domain = 'intranet.example.com'
name = 'intranet.example.com'
http_user = "someuser"
http_pass = "somepass"
http_auth_domain = "intranet.example.com"
name = "intranet.example.com"
# .. rest of the spider code omitted ...
@ -792,7 +804,9 @@ If you want to handle some redirect status codes in your spider, you can
specify these in the ``handle_httpstatus_list`` spider attribute.
For example, if you want the redirect middleware to ignore 301 and 302
responses (and pass them through to your spider) you can do this::
responses (and pass them through to your spider) you can do this:
.. code-block:: python
class MySpider(CrawlSpider):
handle_httpstatus_list = [301, 302]

15
docs/topics/dynamic-content.rst
View File

@ -119,16 +119,20 @@ data from it depends on the type of response:
<topics-selectors>` as usual.
- If the response is JSON, use :func:`json.loads` to load the desired data from
:attr:`response.text <scrapy.http.TextResponse.text>`::
:attr:`response.text <scrapy.http.TextResponse.text>`:
.. code-block:: python
data = json.loads(response.text)
If the desired data is inside HTML or XML code embedded within JSON data,
you can load that HTML or XML code into a
:class:`~scrapy.Selector` and then
:ref:`use it <topics-selectors>` as usual::
:ref:`use it <topics-selectors>` as usual:
selector = Selector(data['html'])
.. code-block:: python
selector = Selector(data["html"])
- If the response is JavaScript, or HTML with a ``<script/>`` element
containing the desired data, see :ref:`topics-parsing-javascript`.
@ -250,11 +254,14 @@ automation. By installing the :ref:`asyncio reactor <install-asyncio>`,
it is possible to integrate ``asyncio``-based libraries which handle headless browsers.
One such library is `playwright-python`_ (an official Python port of `playwright`_).
The following is a simple snippet to illustrate its usage within a Scrapy spider::
The following is a simple snippet to illustrate its usage within a Scrapy spider:
.. code-block:: python
import scrapy
from playwright.async_api import async_playwright
class PlaywrightSpider(scrapy.Spider):
name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader

20
docs/topics/email.rst
View File

@ -19,19 +19,31 @@ Quick example
=============
There are two ways to instantiate the mail sender. You can instantiate it using
the standard ``__init__`` method::
the standard ``__init__`` method:
.. code-block:: python
from scrapy.mail import MailSender
mailer = MailSender()
Or you can instantiate it passing a Scrapy settings object, which will respect
the :ref:`settings <topics-email-settings>`::
the :ref:`settings <topics-email-settings>`:
.. code-block:: python
mailer = MailSender.from_settings(settings)
And here is how to use it to send an e-mail (without attachments)::
And here is how to use it to send an e-mail (without attachments):
mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another@example.com"])
.. code-block:: python
mailer.send(
to=["someone@example.com"],
subject="Some subject",
body="Some body",
cc=["another@example.com"],
)
MailSender class reference
==========================

8
docs/topics/exceptions.rst
View File

@ -26,11 +26,13 @@ CloseSpider
:param reason: the reason for closing
:type reason: str
For example::
For example:
.. code-block:: python
def parse_page(self, response):
if 'Bandwidth exceeded' in response.body:
raise CloseSpider('bandwidth_exceeded')
if "Bandwidth exceeded" in response.body:
raise CloseSpider("bandwidth_exceeded")
DontCloseSpider
---------------

35
docs/topics/exporters.rst
View File

@ -38,11 +38,14 @@ the end of the exporting process
Here you can see an :doc:`Item Pipeline <item-pipeline>` which uses multiple
Item Exporters to group scraped items to different files according to the
value of one of their fields::
value of one of their fields:
.. code-block:: python
from itemadapter import ItemAdapter
from scrapy.exporters import XmlItemExporter
class PerYearXmlExportPipeline:
"""Distribute items across multiple XML files according to their 'year' field"""
@ -56,9 +59,9 @@ value of one of their fields::
def _exporter_for_item(self, item):
adapter = ItemAdapter(item)
year = adapter['year']
year = adapter["year"]
if year not in self.year_to_exporter:
xml_file = open(f'{year}.xml', 'wb')
xml_file = open(f"{year}.xml", "wb")
exporter = XmlItemExporter(xml_file)
exporter.start_exporting()
self.year_to_exporter[year] = (exporter, xml_file)
@ -94,12 +97,16 @@ If you use :class:`~scrapy.Item` you can declare a serializer in the
:ref:`field metadata <topics-items-fields>`. The serializer must be
a callable which receives a value and returns its serialized form.
Example::
Example:
.. code-block:: python
import scrapy
def serialize_price(value):
return f'$ {str(value)}'
return f"$ {str(value)}"
class Product(scrapy.Item):
name = scrapy.Field()
@ -115,15 +122,17 @@ customize how your field value will be exported.
Make sure you call the base class :meth:`~BaseItemExporter.serialize_field()` method
after your custom code.
Example::
Example:
.. code-block:: python
from scrapy.exporters import XmlItemExporter
class ProductXmlExporter(XmlItemExporter):
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if name == 'price':
return f'$ {str(value)}'
if name == "price":
return f"$ {str(value)}"
return super().serialize_field(field, name, value)
.. _topics-exporters-reference:
@ -132,10 +141,12 @@ Built-in Item Exporters reference
=================================
Here is a list of the Item Exporters bundled with Scrapy. Some of them contain
output examples, which assume you're exporting these two items::
output examples, which assume you're exporting these two items:
Item(name='Color TV', price='1200')
Item(name='DVD player', price='200')
.. code-block:: python
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
BaseItemExporter
----------------

24
docs/topics/extensions.rst
View File

@ -31,11 +31,13 @@ initialization code must be performed in the class ``__init__`` method.
To make an extension available, add it to the :setting:`EXTENSIONS` setting in
your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented
by a string: the full Python path to the extension's class name. For example::
by a string: the full Python path to the extension's class name. For example:
.. code-block:: python
EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': 500,
'scrapy.extensions.telnet.TelnetConsole': 500,
"scrapy.extensions.corestats.CoreStats": 500,
"scrapy.extensions.telnet.TelnetConsole": 500,
}
@ -64,10 +66,12 @@ Disabling an extension
In order to disable an extension that comes enabled by default (i.e. those
included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
``None``. For example::
``None``. For example:
.. code-block:: python
EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': None,
"scrapy.extensions.corestats.CoreStats": None,
}
Writing your own extension
@ -98,7 +102,9 @@ in the previous section. This extension will log a message every time:
The extension will be enabled through the ``MYEXT_ENABLED`` setting and the
number of items will be specified through the ``MYEXT_ITEMCOUNT`` setting.
Here is the code of such extension::
Here is the code of such extension:
.. code-block:: python
import logging
from scrapy import signals
@ -106,8 +112,8 @@ Here is the code of such extension::
logger = logging.getLogger(__name__)
class SpiderOpenCloseLogging:
class SpiderOpenCloseLogging:
def __init__(self, item_count):
self.item_count = item_count
self.items_scraped = 0
@ -116,11 +122,11 @@ Here is the code of such extension::
def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise
# NotConfigured otherwise
if not crawler.settings.getbool('MYEXT_ENABLED'):
if not crawler.settings.getbool("MYEXT_ENABLED"):
raise NotConfigured
# get the number of items from settings
item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000)
item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)
# instantiate the extension object
ext = cls(item_count)

67
docs/topics/feed-exports.rst
View File

@ -290,10 +290,11 @@ class, which is the default value of the ``item_filter`` :ref:`feed option <feed
You can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'s
method ``accepts`` and taking ``feed_options`` as an argument.
For instance::
For instance:
.. code-block:: python
class MyCustomFilter:
def __init__(self, feed_options):
self.feed_options = feed_options
@ -594,23 +595,27 @@ For a complete list of available values, access the `Canned ACL`_ section on Ama
FEED_STORAGES_BASE
------------------
Default::
Default:
.. code-block:: python
{
'': 'scrapy.extensions.feedexport.FileFeedStorage',
'file': 'scrapy.extensions.feedexport.FileFeedStorage',
'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',
's3': 'scrapy.extensions.feedexport.S3FeedStorage',
'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
"": "scrapy.extensions.feedexport.FileFeedStorage",
"file": "scrapy.extensions.feedexport.FileFeedStorage",
"stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
"s3": "scrapy.extensions.feedexport.S3FeedStorage",
"ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
}
A dict containing the built-in feed storage backends supported by Scrapy. You
can disable any of these backends by assigning ``None`` to their URI scheme in
:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
(without replacement), place this in your ``settings.py``::
(without replacement), place this in your ``settings.py``:
.. code-block:: python
FEED_STORAGES = {
'ftp': None,
"ftp": None,
}
.. setting:: FEED_EXPORTERS
@ -628,26 +633,30 @@ serialization formats and the values are paths to :ref:`Item exporter
FEED_EXPORTERS_BASE
-------------------
Default::
Default:
.. code-block:: python
{
'json': 'scrapy.exporters.JsonItemExporter',
'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
'jsonl': 'scrapy.exporters.JsonLinesItemExporter',
'jl': 'scrapy.exporters.JsonLinesItemExporter',
'csv': 'scrapy.exporters.CsvItemExporter',
'xml': 'scrapy.exporters.XmlItemExporter',
'marshal': 'scrapy.exporters.MarshalItemExporter',
'pickle': 'scrapy.exporters.PickleItemExporter',
"json": "scrapy.exporters.JsonItemExporter",
"jsonlines": "scrapy.exporters.JsonLinesItemExporter",
"jsonl": "scrapy.exporters.JsonLinesItemExporter",
"jl": "scrapy.exporters.JsonLinesItemExporter",
"csv": "scrapy.exporters.CsvItemExporter",
"xml": "scrapy.exporters.XmlItemExporter",
"marshal": "scrapy.exporters.MarshalItemExporter",
"pickle": "scrapy.exporters.PickleItemExporter",
}
A dict containing the built-in feed exporters supported by Scrapy. You can
disable any of these exporters by assigning ``None`` to their serialization
format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
(without replacement), place this in your ``settings.py``::
(without replacement), place this in your ``settings.py``:
.. code-block:: python
FEED_EXPORTERS = {
'csv': None,
"csv": None,
}
@ -677,7 +686,9 @@ generated:
number by introducing leading zeroes as needed, use ``%(batch_id)05d``
(e.g. ``3`` becomes ``00003``, ``123`` becomes ``00123``).
For instance, if your settings include::
For instance, if your settings include:
.. code-block:: python
FEED_EXPORT_BATCH_ITEM_COUNT = 100
@ -746,16 +757,20 @@ The function signature should be as follows:
For example, to include the :attr:`name <scrapy.Spider.name>` of the
source spider in the feed URI:
#. Define the following function somewhere in your project::
#. Define the following function somewhere in your project:
.. code-block:: python
# myproject/utils.py
def uri_params(params, spider):
return {**params, 'spider_name': spider.name}
return {**params, "spider_name": spider.name}
#. Point :setting:`FEED_URI_PARAMS` to that function in your settings::
#. Point :setting:`FEED_URI_PARAMS` to that function in your settings:
.. code-block:: python
# myproject/settings.py
FEED_URI_PARAMS = 'myproject.utils.uri_params'
FEED_URI_PARAMS = "myproject.utils.uri_params"
#. Use ``%(spider_name)s`` in your feed URI::

50
docs/topics/item-pipeline.rst
View File

@ -81,19 +81,23 @@ Price validation and dropping items with no prices
Let's take a look at the following hypothetical pipeline that adjusts the
``price`` attribute for those items that do not include VAT
(``price_excludes_vat`` attribute), and drops those items which don't
contain a price::
contain a price:
.. code-block:: python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class PricePipeline:
vat_factor = 1.15
def process_item(self, item, spider):
adapter = ItemAdapter(item)
if adapter.get('price'):
if adapter.get('price_excludes_vat'):
adapter['price'] = adapter['price'] * self.vat_factor
if adapter.get("price"):
if adapter.get("price_excludes_vat"):
adapter["price"] = adapter["price"] * self.vat_factor
return item
else:
raise DropItem(f"Missing price in {item}")
@ -104,16 +108,18 @@ Write items to a JSON lines file
The following pipeline stores all scraped items (from all spiders) into a
single ``items.jsonl`` file, containing one item per line serialized in JSON
format::
format:
.. code-block:: python
import json
from itemadapter import ItemAdapter
class JsonWriterPipeline:
class JsonWriterPipeline:
def open_spider(self, spider):
self.file = open('items.jsonl', 'w')
self.file = open("items.jsonl", "w")
def close_spider(self, spider):
self.file.close()
@ -135,14 +141,17 @@ MongoDB address and database name are specified in Scrapy settings;
MongoDB collection is named after item class.
The main point of this example is to show how to use :meth:`from_crawler`
method and how to clean up the resources properly.::
method and how to clean up the resources properly.
.. code-block:: python
import pymongo
from itemadapter import ItemAdapter
class MongoPipeline:
collection_name = 'scrapy_items'
collection_name = "scrapy_items"
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
@ -151,8 +160,8 @@ method and how to clean up the resources properly.::
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
mongo_uri=crawler.settings.get("MONGO_URI"),
mongo_db=crawler.settings.get("MONGO_DATABASE", "items"),
)
def open_spider(self, spider):
@ -183,7 +192,7 @@ render a screenshot of the item URL. After the request response is downloaded,
the item pipeline saves the screenshot to a file and adds the filename to the
item.
::
.. code-block:: python
import hashlib
from pathlib import Path
@ -231,23 +240,24 @@ Duplicates filter
A filter that looks for duplicate items, and drops those items that were
already processed. Let's say that our items have a unique id, but our spider
returns multiples items with the same id::
returns multiples items with the same id:
.. code-block:: python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class DuplicatesPipeline:
class DuplicatesPipeline:
def __init__(self):
self.ids_seen = set()
def process_item(self, item, spider):
adapter = ItemAdapter(item)
if adapter['id'] in self.ids_seen:
if adapter["id"] in self.ids_seen:
raise DropItem(f"Duplicate item found: {item!r}")
else:
self.ids_seen.add(adapter['id'])
self.ids_seen.add(adapter["id"])
return item
@ -255,11 +265,13 @@ Activating an Item Pipeline component
=====================================
To activate an Item Pipeline component you must add its class to the
:setting:`ITEM_PIPELINES` setting, like in the following example::
:setting:`ITEM_PIPELINES` setting, like in the following example:
.. code-block:: python
ITEM_PIPELINES = {
'myproject.pipelines.PricePipeline': 300,
'myproject.pipelines.JsonWriterPipeline': 800,
"myproject.pipelines.PricePipeline": 300,
"myproject.pipelines.JsonWriterPipeline": 800,
}
The integer values you assign to classes in this setting determine the

30
docs/topics/items.rst
View File

@ -76,10 +76,13 @@ make it the most feature-complete item type:
:class:`Field` objects used in the :ref:`Item declaration
<topics-items-declaring>`.
Example::
Example:
.. code-block:: python
from scrapy.item import Item, Field
class CustomItem(Item):
one_field = Field()
another_field = Field()
@ -102,10 +105,13 @@ Additionally, ``dataclass`` items also allow to:
* define custom field metadata through :func:`dataclasses.field`, which can be used to
:ref:`customize serialization <topics-exporters-field-serialization>`.
Example::
Example:
.. code-block:: python
from dataclasses import dataclass
@dataclass
class CustomItem:
one_field: str
@ -133,10 +139,13 @@ Additionally, ``attr.s`` items also allow to:
In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed.
Example::
Example:
.. code-block:: python
import attr
@attr.s
class CustomItem:
one_field = attr.ib()
@ -152,10 +161,13 @@ Declaring Item subclasses
-------------------------
Item subclasses are declared using a simple class definition syntax and
:class:`Field` objects. Here is an example::
:class:`Field` objects. Here is an example:
.. code-block:: python
import scrapy
class Product(scrapy.Item):
name = scrapy.Field()
price = scrapy.Field()
@ -347,17 +359,21 @@ Extending Item subclasses
You can extend Items (to add more fields or to change some metadata for some
fields) by declaring a subclass of your original Item.
For example::
For example:
.. code-block:: python
class DiscountedProduct(Product):
discount_percent = scrapy.Field(serializer=str)
discount_expiration_date = scrapy.Field()
You can also extend field metadata by using the previous field metadata and
appending more values, or changing existing values, like this::
appending more values, or changing existing values, like this:
.. code-block:: python
class SpecificProduct(Product):
name = scrapy.Field(Product.fields['name'], serializer=my_serializer)
name = scrapy.Field(Product.fields["name"], serializer=my_serializer)
That adds (or replaces) the ``serializer`` metadata key for the ``name`` field,
keeping all the previously existing metadata values.

6
docs/topics/jobs.rst
View File

@ -51,11 +51,13 @@ loading that attribute from the job directory, when the spider starts and
stops.
Here's an example of a callback that uses the spider state (other spider code
is omitted for brevity)::
is omitted for brevity):
.. code-block:: python
def parse_item(self, response):
# parse item here
self.state['items_count'] = self.state.get('items_count', 0) + 1
self.state["items_count"] = self.state.get("items_count", 0) + 1
Persistence gotchas
===================

8
docs/topics/link-extractors.rst
View File

@ -18,7 +18,9 @@ through a set of :class:`~scrapy.spiders.Rule` objects.
You can also use link extractors in regular spiders. For example, you can instantiate
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class
variable in your spider, and use it from your spider callbacks::
variable in your spider, and use it from your spider callbacks:
.. code-block:: python
def parse(self, response):
for link in self.link_extractor.extract_links(response):
@ -132,7 +134,9 @@ LxmlLinkExtractor
.. highlight:: python
You can use the following function in ``process_value``::
You can use the following function in ``process_value``:
.. code-block:: python
def process_value(value):
m = re.search("javascript:goToPage\('(.*?)'", value)

100
docs/topics/loaders.rst
View File

@ -46,18 +46,21 @@ using a proper processing function.
Here is a typical Item Loader usage in a :ref:`Spider <topics-spiders>`, using
the :ref:`Product item <topics-items-declaring>` declared in the :ref:`Items
chapter <topics-items>`::
chapter <topics-items>`:
.. code-block:: python
from scrapy.loader import ItemLoader
from myproject.items import Product
def parse(self, response):
l = ItemLoader(item=Product(), response=response)
l.add_xpath('name', '//div[@class="product_name"]')
l.add_xpath('name', '//div[@class="product_title"]')
l.add_xpath('price', '//p[@id="price"]')
l.add_css('stock', 'p#stock')
l.add_value('last_updated', 'today') # you can also use literal values
l.add_xpath("name", '//div[@class="product_name"]')
l.add_xpath("name", '//div[@class="product_title"]')
l.add_xpath("price", '//p[@id="price"]')
l.add_css("stock", "p#stock")
l.add_value("last_updated", "today") # you can also use literal values
return l.load_item()
By quickly looking at that code, we can see the ``name`` field is being
@ -93,11 +96,14 @@ will be populated incrementally using the loader's :meth:`~ItemLoader.add_xpath`
:meth:`~ItemLoader.add_css` and :meth:`~ItemLoader.add_value` methods.
One approach to overcome this is to define items using the
:func:`~dataclasses.field` function, with a ``default`` argument::
:func:`~dataclasses.field` function, with a ``default`` argument:
.. code-block:: python
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class InventoryItem:
name: Optional[str] = field(default=None)
@ -122,14 +128,16 @@ processor). The result of the output processor is the final value that gets
assigned to the item.
Let's see an example to illustrate how the input and output processors are
called for a particular field (the same applies for any other field)::
called for a particular field (the same applies for any other field):
.. code-block:: python
l = ItemLoader(Product(), some_selector)
l.add_xpath('name', xpath1) # (1)
l.add_xpath('name', xpath2) # (2)
l.add_css('name', css) # (3)
l.add_value('name', 'test') # (4)
return l.load_item() # (5)
l.add_xpath("name", xpath1) # (1)
l.add_xpath("name", xpath2) # (2)
l.add_css("name", css) # (3)
l.add_value("name", "test") # (4)
return l.load_item() # (5)
So what happens is:
@ -184,11 +192,14 @@ processors <itemloaders:built-in-processors>` built-in for convenience.
Declaring Item Loaders
======================
Item Loaders are declared using a class definition syntax. Here is an example::
Item Loaders are declared using a class definition syntax. Here is an example:
.. code-block:: python
from itemloaders.processors import TakeFirst, MapCompose, Join
from scrapy.loader import ItemLoader
class ProductLoader(ItemLoader):
default_output_processor = TakeFirst()
@ -215,16 +226,20 @@ As seen in the previous section, input and output processors can be declared in
the Item Loader definition, and it's very common to declare input processors
this way. However, there is one more place where you can specify the input and
output processors to use: in the :ref:`Item Field <topics-items-fields>`
metadata. Here is an example::
metadata. Here is an example:
.. code-block:: python
import scrapy
from itemloaders.processors import Join, MapCompose, TakeFirst
from w3lib.html import remove_tags
def filter_price(value):
if value.isdigit():
return value
class Product(scrapy.Item):
name = scrapy.Field(
input_processor=MapCompose(remove_tags),
@ -263,10 +278,12 @@ declaring, instantiating or using Item Loader. They are used to modify the
behaviour of the input/output processors.
For example, suppose you have a function ``parse_length`` which receives a text
value and extracts a length from it::
value and extracts a length from it:
.. code-block:: python
def parse_length(text, loader_context):
unit = loader_context.get('unit', 'm')
unit = loader_context.get("unit", "m")
# ... length parsing code goes here ...
return parsed_length
@ -278,22 +295,28 @@ function (``parse_length`` in this case) can thus use them.
There are several ways to modify Item Loader context values:
1. By modifying the currently active Item Loader context
(:attr:`~ItemLoader.context` attribute)::
(:attr:`~ItemLoader.context` attribute):
.. code-block:: python
loader = ItemLoader(product)
loader.context['unit'] = 'cm'
loader.context["unit"] = "cm"
2. On Item Loader instantiation (the keyword arguments of Item Loader
``__init__`` method are stored in the Item Loader context)::
``__init__`` method are stored in the Item Loader context):
loader = ItemLoader(product, unit='cm')
.. code-block:: python
loader = ItemLoader(product, unit="cm")
3. On Item Loader declaration, for those input/output processors that support
instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
them::
them:
.. code-block:: python
class ProductLoader(ItemLoader):
length_out = MapCompose(parse_length, unit='cm')
length_out = MapCompose(parse_length, unit="cm")
ItemLoader objects
@ -323,25 +346,29 @@ Example::
Without nested loaders, you need to specify the full xpath (or css) for each value
that you wish to extract.
Example::
Example:
.. code-block:: python
loader = ItemLoader(item=Item())
# load stuff not in the footer
loader.add_xpath('social', '//footer/a[@class = "social"]/@href')
loader.add_xpath('email', '//footer/a[@class = "email"]/@href')
loader.add_xpath("social", '//footer/a[@class = "social"]/@href')
loader.add_xpath("email", '//footer/a[@class = "email"]/@href')
loader.load_item()
Instead, you can create a nested loader with the footer selector and add values
relative to the footer. The functionality is the same but you avoid repeating
the footer selector.
Example::
Example:
.. code-block:: python
loader = ItemLoader(item=Item())
# load stuff not in the footer
footer_loader = loader.nested_xpath('//footer')
footer_loader.add_xpath('social', 'a[@class = "social"]/@href')
footer_loader.add_xpath('email', 'a[@class = "email"]/@href')
footer_loader = loader.nested_xpath("//footer")
footer_loader.add_xpath("social", 'a[@class = "social"]/@href')
footer_loader.add_xpath("email", 'a[@class = "email"]/@href')
# no need to call footer_loader.load_item()
loader.load_item()
@ -370,25 +397,32 @@ three dashes (e.g. ``---Plasma TV---``) and you don't want to end up scraping
those dashes in the final product names.
Here's how you can remove those dashes by reusing and extending the default
Product Item Loader (``ProductLoader``)::
Product Item Loader (``ProductLoader``):
.. code-block:: python
from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader
def strip_dashes(x):
return x.strip('-')
return x.strip("-")
class SiteSpecificLoader(ProductLoader):
name_in = MapCompose(strip_dashes, ProductLoader.name_in)
Another case where extending Item Loaders can be very helpful is when you have
multiple source formats, for example XML and HTML. In the XML version you may
want to remove ``CDATA`` occurrences. Here's an example of how to do it::
want to remove ``CDATA`` occurrences. Here's an example of how to do it:
.. code-block:: python
from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader
from myproject.utils.xml import remove_cdata
class XmlProductLoader(ProductLoader):
name_in = MapCompose(remove_cdata, ProductLoader.name_in)

109
docs/topics/logging.rst
View File

@ -39,16 +39,22 @@ How to log messages
===================
Here's a quick example of how to log a message using the ``logging.WARNING``
level::
level:
.. code-block:: python
import logging
logging.warning("This is a warning")
There are shortcuts for issuing log messages on any of the standard 5 levels,
and there's also a general ``logging.log`` method which takes a given level as
argument. If needed, the last example could be rewritten as::
argument. If needed, the last example could be rewritten as:
.. code-block:: python
import logging
logging.log(logging.WARNING, "This is a warning")
On top of that, you can create different "loggers" to encapsulate messages. (For
@ -59,24 +65,33 @@ constructions.
The previous examples use the root logger behind the scenes, which is a top level
logger where all messages are propagated to (unless otherwise specified). Using
``logging`` helpers is merely a shortcut for getting the root logger
explicitly, so this is also an equivalent of the last snippets::
explicitly, so this is also an equivalent of the last snippets:
.. code-block:: python
import logging
logger = logging.getLogger()
logger.warning("This is a warning")
You can use a different logger just by getting its name with the
``logging.getLogger`` function::
``logging.getLogger`` function:
.. code-block:: python
import logging
logger = logging.getLogger('mycustomlogger')
logger = logging.getLogger("mycustomlogger")
logger.warning("This is a warning")
Finally, you can ensure having a custom logger for any module you're working on
by using the ``__name__`` variable, which is populated with current module's
path::
path:
.. code-block:: python
import logging
logger = logging.getLogger(__name__)
logger.warning("This is a warning")
@ -94,33 +109,39 @@ Logging from Spiders
====================
Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider
instance, which can be accessed and used like this::
instance, which can be accessed and used like this:
.. code-block:: python
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://scrapy.org']
name = "myspider"
start_urls = ["https://scrapy.org"]
def parse(self, response):
self.logger.info('Parse function called on %s', response.url)
self.logger.info("Parse function called on %s", response.url)
That logger is created using the Spider's name, but you can use any custom
Python logger you want. For example::
Python logger you want. For example:
.. code-block:: python
import logging
import scrapy
logger = logging.getLogger('mycustomlogger')
logger = logging.getLogger("mycustomlogger")
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://scrapy.org']
name = "myspider"
start_urls = ["https://scrapy.org"]
def parse(self, response):
logger.info('Parse function called on %s', response.url)
logger.info("Parse function called on %s", response.url)
.. _topics-logging-configuration:
@ -229,7 +250,9 @@ the crawl.
Next, we can see that the message has INFO level. To hide it
we should set logging level for ``scrapy.spidermiddlewares.httperror``
higher than INFO; next level after INFO is WARNING. It could be done
e.g. in the spider's ``__init__`` method::
e.g. in the spider's ``__init__`` method:
.. code-block:: python
import logging
import scrapy
@ -238,7 +261,7 @@ e.g. in the spider's ``__init__`` method::
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
logger = logging.getLogger('scrapy.spidermiddlewares.httperror')
logger = logging.getLogger("scrapy.spidermiddlewares.httperror")
logger.setLevel(logging.WARNING)
super().__init__(*args, **kwargs)
@ -249,43 +272,53 @@ You can also filter log records by :class:`~logging.LogRecord` data. For
example, you can filter log records by message content using a substring or
a regular expression. Create a :class:`logging.Filter` subclass
and equip it with a regular expression pattern to
filter out unwanted messages::
filter out unwanted messages:
.. code-block:: python
import logging
import re
class ContentFilter(logging.Filter):
def filter(self, record):
match = re.search(r'\d{3} [Ee]rror, retrying', record.message)
match = re.search(r"\d{3} [Ee]rror, retrying", record.message)
if match:
return False
A project-level filter may be attached to the root
handler created by Scrapy, this is a wieldy way to
filter all loggers in different parts of the project
(middlewares, spider, etc.)::
(middlewares, spider, etc.):
import logging
import scrapy
.. code-block:: python
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
for handler in logging.root.handlers:
handler.addFilter(ContentFilter())
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
for handler in logging.root.handlers:
handler.addFilter(ContentFilter())
Alternatively, you may choose a specific logger
and hide it without affecting other loggers::
and hide it without affecting other loggers:
.. code-block:: python
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
logger = logging.getLogger('my_logger')
logger = logging.getLogger("my_logger")
logger.addFilter(ContentFilter())
scrapy.utils.log module
=======================
@ -306,14 +339,14 @@ scrapy.utils.log module
so it is recommended to only use :func:`logging.basicConfig` together with
:class:`~scrapy.crawler.CrawlerRunner`.
This is an example on how to redirect ``INFO`` or higher messages to a file::
This is an example on how to redirect ``INFO`` or higher messages to a file:
.. code-block:: python
import logging
logging.basicConfig(
filename='log.txt',
format='%(levelname)s: %(message)s',
level=logging.INFO
filename="log.txt", format="%(levelname)s: %(message)s", level=logging.INFO
)
Refer to :ref:`run-from-script` for more details about using Scrapy this

177
docs/topics/media-pipeline.rst
View File

@ -87,13 +87,17 @@ Enabling your Media Pipeline
To enable your media pipeline you must first add it to your project
:setting:`ITEM_PIPELINES` setting.
For Images Pipeline, use::
For Images Pipeline, use:
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
.. code-block:: python
For Files Pipeline, use::
ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}
ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1}
For Files Pipeline, use:
.. code-block:: python
ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}
.. note::
You can also use both the Files and Images Pipeline at the same time.
@ -103,13 +107,17 @@ Then, configure the target storage setting to a valid value that will be used
for storing the downloaded images. Otherwise the pipeline will remain disabled,
even if you include it in the :setting:`ITEM_PIPELINES` setting.
For the Files Pipeline, set the :setting:`FILES_STORE` setting::
For the Files Pipeline, set the :setting:`FILES_STORE` setting:
FILES_STORE = '/path/to/valid/dir'
.. code-block:: python
For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::
FILES_STORE = "/path/to/valid/dir"
IMAGES_STORE = '/path/to/valid/dir'
For the Images Pipeline, set the :setting:`IMAGES_STORE` setting:
.. code-block:: python
IMAGES_STORE = "/path/to/valid/dir"
.. _topics-file-naming:
@ -157,10 +165,11 @@ By overriding ``file_path`` like this:
import hashlib
def file_path(self, request, response=None, info=None, *, item=None):
image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)
image_perspective = request.url.split('/')[-2]
image_filename = f'{image_url_hash}_{image_perspective}.jpg'
image_perspective = request.url.split("/")[-2]
image_filename = f"{image_url_hash}_{image_perspective}.jpg"
return image_filename
@ -233,30 +242,38 @@ If botocore_ >= 1.4.87 is installed, :setting:`FILES_STORE` and
:setting:`IMAGES_STORE` can represent an Amazon S3 bucket. Scrapy will
automatically upload the files to the bucket.
For example, this is a valid :setting:`IMAGES_STORE` value::
For example, this is a valid :setting:`IMAGES_STORE` value:
IMAGES_STORE = 's3://bucket/images'
.. code-block:: python
IMAGES_STORE = "s3://bucket/images"
You can modify the Access Control List (ACL) policy used for the stored files,
which is defined by the :setting:`FILES_STORE_S3_ACL` and
:setting:`IMAGES_STORE_S3_ACL` settings. By default, the ACL is set to
``private``. To make the files publicly available use the ``public-read``
policy::
policy:
IMAGES_STORE_S3_ACL = 'public-read'
.. code-block:: python
IMAGES_STORE_S3_ACL = "public-read"
For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.
You can also use other S3-like storages. Storages like self-hosted `Minio`_ or
`s3.scality`_. All you need to do is set endpoint option in you Scrapy
settings::
settings:
AWS_ENDPOINT_URL = 'http://minio.example.com:9000'
.. code-block:: python
For self-hosting you also might feel the need not to use SSL and not to verify SSL connection::
AWS_ENDPOINT_URL = "http://minio.example.com:9000"
AWS_USE_SSL = False # or True (None by default)
AWS_VERIFY = False # or True (None by default)
For self-hosting you also might feel the need not to use SSL and not to verify SSL connection:
.. code-block:: python
AWS_USE_SSL = False # or True (None by default)
AWS_VERIFY = False # or True (None by default)
.. _botocore: https://github.com/boto/botocore
.. _canned ACLs: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
@ -277,10 +294,12 @@ bucket. Scrapy will automatically upload the files to the bucket. (requires `goo
.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings::
For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings:
IMAGES_STORE = 'gs://bucket/images/'
GCS_PROJECT_ID = 'project_id'
.. code-block:: python
IMAGES_STORE = "gs://bucket/images/"
GCS_PROJECT_ID = "project_id"
For information about authentication, see this `documentation`_.
@ -291,9 +310,11 @@ which is defined by the :setting:`FILES_STORE_GCS_ACL` and
:setting:`IMAGES_STORE_GCS_ACL` settings. By default, the ACL is set to
``''`` (empty string) which means that Cloud Storage applies the bucket's default object ACL to the object.
To make the files publicly available use the ``publicRead``
policy::
policy:
IMAGES_STORE_GCS_ACL = 'publicRead'
.. code-block:: python
IMAGES_STORE_GCS_ACL = "publicRead"
For more information, see `Predefined ACLs`_ in the Google Cloud Platform Developer Guide.
@ -318,10 +339,13 @@ respectively), the pipeline will put the results under the respective field
When using :ref:`item types <item-types>` for which fields are defined beforehand,
you must define both the URLs field and the results field. For example, when
using the images pipeline, items must define both the ``image_urls`` and the
``images`` field. For instance, using the :class:`~scrapy.Item` class::
``images`` field. For instance, using the :class:`~scrapy.Item` class:
.. code-block:: python
import scrapy
class MyItem(scrapy.Item):
# ... other item fields ...
image_urls = scrapy.Field()
@ -331,16 +355,20 @@ If you want to use another field name for the URLs key or for the results key,
it is also possible to override it.
For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or
:setting:`FILES_RESULT_FIELD` settings::
:setting:`FILES_RESULT_FIELD` settings:
FILES_URLS_FIELD = 'field_name_for_your_files_urls'
FILES_RESULT_FIELD = 'field_name_for_your_processed_files'
.. code-block:: python
FILES_URLS_FIELD = "field_name_for_your_files_urls"
FILES_RESULT_FIELD = "field_name_for_your_processed_files"
For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
:setting:`IMAGES_RESULT_FIELD` settings::
:setting:`IMAGES_RESULT_FIELD` settings:
IMAGES_URLS_FIELD = 'field_name_for_your_images_urls'
IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images'
.. code-block:: python
IMAGES_URLS_FIELD = "field_name_for_your_images_urls"
IMAGES_RESULT_FIELD = "field_name_for_your_processed_images"
If you need something more complex and want to override the custom pipeline
behaviour, see :ref:`topics-media-pipeline-override`.
@ -366,7 +394,9 @@ File expiration
The Image Pipeline avoids downloading files that were downloaded recently. To
adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or
:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which
specifies the delay in number of days::
specifies the delay in number of days:
.. code-block:: python
# 120 days of delay for files expiration
FILES_EXPIRES = 120
@ -400,11 +430,13 @@ images.
In order to use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
where the keys are the thumbnail names and the values are their dimensions.
For example::
For example:
.. code-block:: python
IMAGES_THUMBS = {
'small': (50, 50),
'big': (270, 270),
"small": (50, 50),
"big": (270, 270),
}
When you use this feature, the Images Pipeline will create thumbnails of the
@ -495,17 +527,19 @@ See here the methods that you can override in your custom Files Pipeline:
For example, if file URLs end like regular paths (e.g.
``https://example.com/a/b/c/foo.png``), you can use the following
approach to download all files into the ``files`` folder with their
original filenames (e.g. ``files/foo.png``)::
original filenames (e.g. ``files/foo.png``):
.. code-block:: python
from pathlib import PurePosixPath
from urllib.parse import urlparse
from scrapy.pipelines.files import FilesPipeline
class MyFilesPipeline(FilesPipeline):
class MyFilesPipeline(FilesPipeline):
def file_path(self, request, response=None, info=None, *, item=None):
return 'files/' + PurePosixPath(urlparse(request.url).path).name
return "files/" + PurePosixPath(urlparse(request.url).path).name
Similarly, you can use the ``item`` to determine the file path based on some item
property.
@ -521,13 +555,16 @@ See here the methods that you can override in your custom Files Pipeline:
As seen on the workflow, the pipeline will get the URLs of the images to
download from the item. In order to do this, you can override the
:meth:`~get_media_requests` method and return a Request for each
file URL::
file URL:
.. code-block:: python
from itemadapter import ItemAdapter
def get_media_requests(self, item, info):
adapter = ItemAdapter(item)
for file_url in adapter['file_urls']:
for file_url in adapter["file_urls"]:
yield scrapy.Request(file_url)
Those requests will be processed by the pipeline and, when they have finished
@ -567,15 +604,22 @@ See here the methods that you can override in your custom Files Pipeline:
guaranteed to retain the same order of the requests returned from the
:meth:`~get_media_requests` method.
Here's a typical value of the ``results`` argument::
Here's a typical value of the ``results`` argument:
[(True,
{'checksum': '2b00042f7481c7b056c4b410d28f33cf',
'path': 'full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg',
'url': 'http://www.example.com/files/product1.pdf',
'status': 'downloaded'}),
(False,
Failure(...))]
.. code-block:: python
[
(
True,
{
"checksum": "2b00042f7481c7b056c4b410d28f33cf",
"path": "full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg",
"url": "http://www.example.com/files/product1.pdf",
"status": "downloaded",
},
),
(False, Failure(...)),
]
By default the :meth:`get_media_requests` method returns ``None`` which
means there are no files to download for the item.
@ -592,17 +636,20 @@ See here the methods that you can override in your custom Files Pipeline:
Here is an example of the :meth:`~item_completed` method where we
store the downloaded file paths (passed in results) in the ``file_paths``
item field, and we drop the item if it doesn't contain any files::
item field, and we drop the item if it doesn't contain any files:
.. code-block:: python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
def item_completed(self, results, item, info):
file_paths = [x['path'] for ok, x in results if ok]
file_paths = [x["path"] for ok, x in results if ok]
if not file_paths:
raise DropItem("Item contains no files")
adapter = ItemAdapter(item)
adapter['file_paths'] = file_paths
adapter["file_paths"] = file_paths
return item
By default, the :meth:`item_completed` method returns the item.
@ -634,17 +681,19 @@ See here the methods that you can override in your custom Images Pipeline:
For example, if file URLs end like regular paths (e.g.
``https://example.com/a/b/c/foo.png``), you can use the following
approach to download all files into the ``files`` folder with their
original filenames (e.g. ``files/foo.png``)::
original filenames (e.g. ``files/foo.png``):
.. code-block:: python
from pathlib import PurePosixPath
from urllib.parse import urlparse
from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
class MyImagesPipeline(ImagesPipeline):
def file_path(self, request, response=None, info=None, *, item=None):
return 'files/' + PurePosixPath(urlparse(request.url).path).name
return "files/" + PurePosixPath(urlparse(request.url).path).name
Similarly, you can use the ``item`` to determine the file path based on some item
property.
@ -700,33 +749,35 @@ Custom Images pipeline example
==============================
Here is a full example of the Images Pipeline whose methods are exemplified
above::
above:
.. code-block:: python
import scrapy
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
for image_url in item["image_urls"]:
yield scrapy.Request(image_url)
def item_completed(self, results, item, info):
image_paths = [x['path'] for ok, x in results if ok]
image_paths = [x["path"] for ok, x in results if ok]
if not image_paths:
raise DropItem("Item contains no images")
adapter = ItemAdapter(item)
adapter['image_paths'] = image_paths
adapter["image_paths"] = image_paths
return item
To enable your custom media pipeline component you must add its class import path to the
:setting:`ITEM_PIPELINES` setting, like in the following example::
:setting:`ITEM_PIPELINES` setting, like in the following example:
ITEM_PIPELINES = {
'myproject.pipelines.MyImagesPipeline': 300
}
.. code-block:: python
ITEM_PIPELINES = {"myproject.pipelines.MyImagesPipeline": 300}
.. _MD5 hash: https://en.wikipedia.org/wiki/MD5

55
docs/topics/practices.rst
View File

@ -25,23 +25,27 @@ the one used by all Scrapy commands.
Here's an example showing how to run a single spider with it.
::
.. code-block:: python
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = CrawlerProcess(settings={
"FEEDS": {
"items.json": {"format": "json"},
},
})
process = CrawlerProcess(
settings={
"FEEDS": {
"items.json": {"format": "json"},
},
}
)
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
process.start() # the script will block here until the crawling is finished
Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess`
documentation to get acquainted with its usage details.
@ -55,7 +59,7 @@ instance with your project settings.
What follows is a working example of how to do that, using the `testspiders`_
project as example.
::
.. code-block:: python
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
@ -63,8 +67,8 @@ project as example.
process = CrawlerProcess(get_project_settings())
# 'followall' is the name of one of the spiders of the project.
process.crawl('followall', domain='scrapy.org')
process.start() # the script will block here until the crawling is finished
process.crawl("followall", domain="scrapy.org")
process.start() # the script will block here until the crawling is finished
There's another Scrapy utility that provides more control over the crawling
process: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper
@ -84,23 +88,25 @@ returned by the :meth:`CrawlerRunner.crawl
Here's an example of its usage, along with a callback to manually stop the
reactor after ``MySpider`` has finished running.
::
.. code-block:: python
from twisted.internet import reactor
import scrapy
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
class MySpider(scrapy.Spider):
# Your spider definition
...
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = CrawlerRunner()
d = runner.crawl(MySpider)
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until the crawling is finished
reactor.run() # the script will block here until the crawling is finished
.. seealso:: :doc:`twisted:core/howto/reactor-basics`
@ -115,29 +121,32 @@ the :ref:`internal API <topics-api>`.
Here is an example that runs multiple spiders simultaneously:
::
.. code-block:: python
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
settings = get_project_settings()
process = CrawlerProcess(settings)
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished
process.start() # the script will block here until all crawling jobs are finished
Same example using :class:`~scrapy.crawler.CrawlerRunner`:
::
.. code-block:: python
import scrapy
from twisted.internet import reactor
@ -145,14 +154,17 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
configure_logging()
settings = get_project_settings()
runner = CrawlerRunner(settings)
@ -161,37 +173,42 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished
reactor.run() # the script will block here until all crawling jobs are finished
Same example but running the spiders sequentially by chaining the deferreds:
::
.. code-block:: python
from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
settings = get_project_settings()
configure_logging(settings)
runner = CrawlerRunner(settings)
@defer.inlineCallbacks
def crawl():
yield runner.crawl(MySpider1)
yield runner.crawl(MySpider2)
reactor.stop()
crawl()
reactor.run() # the script will block here until the last crawl call is finished
reactor.run() # the script will block here until the last crawl call is finished
Different spiders can set different values for the same setting, but when they
run in the same process it may be impossible, by design or because of some

176
docs/topics/request-response.rst
View File

@ -78,23 +78,27 @@ Request objects
:param cookies: the request cookies. These can be sent in two forms.
1. Using a dict::
1. Using a dict:
.. code-block:: python
request_with_cookies = Request(
url="http://www.example.com",
cookies={'currency': 'USD', 'country': 'UY'},
cookies={"currency": "USD", "country": "UY"},
)
2. Using a list of dicts::
2. Using a list of dicts:
.. code-block:: python
request_with_cookies = Request(
url="http://www.example.com",
cookies=[
{
'name': 'currency',
'value': 'USD',
'domain': 'example.com',
'path': '/currency',
"name": "currency",
"value": "USD",
"domain": "example.com",
"path": "/currency",
},
],
)
@ -114,12 +118,14 @@ Request objects
in :attr:`request.meta <scrapy.Request.meta>`.
Example of a request that sends manually-defined cookies and ignores
cookie storage::
cookie storage:
.. code-block:: python
Request(
url="http://www.example.com",
cookies={'currency': 'USD', 'country': 'UY'},
meta={'dont_merge_cookies': True},
cookies={"currency": "USD", "country": "UY"},
meta={"dont_merge_cookies": True},
)
For more info see :ref:`cookies-mw`.
@ -259,11 +265,15 @@ The callback of a request is a function that will be called when the response
of that request is downloaded. The callback function will be called with the
downloaded :class:`Response` object as its first argument.
Example::
Example:
.. code-block:: python
def parse_page1(self, response):
return scrapy.Request("http://www.example.com/some_page.html",
callback=self.parse_page2)
return scrapy.Request(
"http://www.example.com/some_page.html", callback=self.parse_page2
)
def parse_page2(self, response):
# this would log http://www.example.com/some_page.html
@ -274,15 +284,18 @@ functions so you can receive the arguments later, in the second callback.
The following example shows how to achieve this by using the
:attr:`Request.cb_kwargs` attribute:
::
.. code-block:: python
def parse(self, response):
request = scrapy.Request('http://www.example.com/index.html',
callback=self.parse_page2,
cb_kwargs=dict(main_url=response.url))
request.cb_kwargs['foo'] = 'bar' # add more arguments for the callback
request = scrapy.Request(
"http://www.example.com/index.html",
callback=self.parse_page2,
cb_kwargs=dict(main_url=response.url),
)
request.cb_kwargs["foo"] = "bar" # add more arguments for the callback
yield request
def parse_page2(self, response, main_url, foo):
yield dict(
main_url=main_url,
@ -308,7 +321,9 @@ It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can
be used to track connection establishment timeouts, DNS errors etc.
Here's an example spider logging all errors and catching some specific
errors if needed::
errors if needed
.. code-block:: python
import scrapy
@ -316,24 +331,28 @@ errors if needed::
from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError, TCPTimedOutError
class ErrbackSpider(scrapy.Spider):
name = "errback_example"
start_urls = [
"http://www.httpbin.org/", # HTTP 200 expected
"http://www.httpbin.org/status/404", # Not found error
"http://www.httpbin.org/status/500", # server issue
"http://www.httpbin.org:12345/", # non-responding host, timeout expected
"https://example.invalid/", # DNS error expected
"http://www.httpbin.org/", # HTTP 200 expected
"http://www.httpbin.org/status/404", # Not found error
"http://www.httpbin.org/status/500", # server issue
"http://www.httpbin.org:12345/", # non-responding host, timeout expected
"https://example.invalid/", # DNS error expected
]
def start_requests(self):
for u in self.start_urls:
yield scrapy.Request(u, callback=self.parse_httpbin,
errback=self.errback_httpbin,
dont_filter=True)
yield scrapy.Request(
u,
callback=self.parse_httpbin,
errback=self.errback_httpbin,
dont_filter=True,
)
def parse_httpbin(self, response):
self.logger.info('Got successful response from {}'.format(response.url))
self.logger.info("Got successful response from {}".format(response.url))
# do something useful here...
def errback_httpbin(self, failure):
@ -347,16 +366,16 @@ errors if needed::
# these exceptions come from HttpError spider middleware
# you can get the non-200 response
response = failure.value.response
self.logger.error('HttpError on %s', response.url)
self.logger.error("HttpError on %s", response.url)
elif failure.check(DNSLookupError):
# this is the original request
request = failure.request
self.logger.error('DNSLookupError on %s', request.url)
self.logger.error("DNSLookupError on %s", request.url)
elif failure.check(TimeoutError, TCPTimedOutError):
request = failure.request
self.logger.error('TimeoutError on %s', request.url)
self.logger.error("TimeoutError on %s", request.url)
.. _errback-cb_kwargs:
@ -367,21 +386,27 @@ Accessing additional data in errback functions
In case of a failure to process the request, you may be interested in
accessing arguments to the callback functions so you can process further
based on the arguments in the errback. The following example shows how to
achieve this by using ``Failure.request.cb_kwargs``::
achieve this by using ``Failure.request.cb_kwargs``
.. code-block:: python
def parse(self, response):
request = scrapy.Request('http://www.example.com/index.html',
callback=self.parse_page2,
errback=self.errback_page2,
cb_kwargs=dict(main_url=response.url))
request = scrapy.Request(
"http://www.example.com/index.html",
callback=self.parse_page2,
errback=self.errback_page2,
cb_kwargs=dict(main_url=response.url),
)
yield request
def parse_page2(self, response, main_url):
pass
def errback_page2(self, failure):
yield dict(
main_url=failure.request.cb_kwargs['main_url'],
main_url=failure.request.cb_kwargs["main_url"],
)
@ -528,18 +553,20 @@ in your :meth:`fingerprint` method implementation:
.. autofunction:: scrapy.utils.request.fingerprint
For example, to take the value of a request header named ``X-ID`` into
account::
account:
.. code-block:: python
# my_project/settings.py
REQUEST_FINGERPRINTER_CLASS = 'my_project.utils.RequestFingerprinter'
REQUEST_FINGERPRINTER_CLASS = "my_project.utils.RequestFingerprinter"
# my_project/utils.py
from scrapy.utils.request import fingerprint
class RequestFingerprinter:
class RequestFingerprinter:
def fingerprint(self, request):
return fingerprint(request, include_headers=['X-ID'])
return fingerprint(request, include_headers=["X-ID"])
You can also write your own fingerprinting logic from scratch.
@ -555,13 +582,16 @@ you use :class:`~weakref.WeakKeyDictionary` to cache request fingerprints:
references to them in your cache dictionary.
For example, to take into account only the URL of a request, without any prior
URL canonicalization or taking the request method or body into account::
URL canonicalization or taking the request method or body into account:
.. code-block:: python
from hashlib import sha1
from weakref import WeakKeyDictionary
from scrapy.utils.python import to_bytes
class RequestFingerprinter:
cache = WeakKeyDictionary()
@ -577,21 +607,25 @@ If you need to be able to override the request fingerprinting for arbitrary
requests from your spider callbacks, you may implement a request fingerprinter
that reads fingerprints from :attr:`request.meta <scrapy.http.Request.meta>`
when available, and then falls back to
:func:`scrapy.utils.request.fingerprint`. For example::
:func:`scrapy.utils.request.fingerprint`. For example:
.. code-block:: python
from scrapy.utils.request import fingerprint
class RequestFingerprinter:
class RequestFingerprinter:
def fingerprint(self, request):
if 'fingerprint' in request.meta:
return request.meta['fingerprint']
if "fingerprint" in request.meta:
return request.meta["fingerprint"]
return fingerprint(request)
If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6
without using the deprecated ``'2.6'`` value of the
:setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following
request fingerprinter::
request fingerprinter:
.. code-block:: python
from hashlib import sha1
from weakref import WeakKeyDictionary
@ -599,6 +633,7 @@ request fingerprinter::
from scrapy.utils.python import to_bytes
from w3lib.url import canonicalize_url
class RequestFingerprinter:
cache = WeakKeyDictionary()
@ -608,7 +643,7 @@ request fingerprinter::
fp = sha1()
fp.update(to_bytes(request.method))
fp.update(to_bytes(canonicalize_url(request.url)))
fp.update(request.body or b'')
fp.update(request.body or b"")
self.cache[request] = fp.digest()
return self.cache[request]
@ -737,7 +772,9 @@ Stopping the download of a Response
Raising a :exc:`~scrapy.exceptions.StopDownload` exception from a handler for the
:class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received`
signals will stop the download of a given response. See the following example::
signals will stop the download of a given response. See the following example:
.. code-block:: python
import scrapy
@ -749,7 +786,9 @@ signals will stop the download of a given response. See the following example::
@classmethod
def from_crawler(cls, crawler):
spider = super().from_crawler(crawler)
crawler.signals.connect(spider.on_bytes_received, signal=scrapy.signals.bytes_received)
crawler.signals.connect(
spider.on_bytes_received, signal=scrapy.signals.bytes_received
)
return spider
def parse(self, response):
@ -878,11 +917,17 @@ Using FormRequest to send data via HTTP POST
If you want to simulate a HTML Form POST in your spider and send a couple of
key-value fields, you can return a :class:`FormRequest` object (from your
spider) like this::
spider) like this:
return [FormRequest(url="http://www.example.com/post/action",
formdata={'name': 'John Doe', 'age': '27'},
callback=self.after_post)]
.. code-block:: python
return [
FormRequest(
url="http://www.example.com/post/action",
formdata={"name": "John Doe", "age": "27"},
callback=self.after_post,
)
]
.. _topics-request-response-ref-request-userlogin:
@ -894,25 +939,28 @@ type="hidden">`` elements, such as session related data or authentication
tokens (for login pages). When scraping, you'll want these fields to be
automatically pre-populated and only override a couple of them, such as the
user name and password. You can use the :meth:`FormRequest.from_response`
method for this job. Here's an example spider which uses it::
method for this job. Here's an example spider which uses it:
.. code-block:: python
import scrapy
def authentication_failed(response):
# TODO: Check the contents of the response and return True if it failed
# or False if it succeeded.
pass
class LoginSpider(scrapy.Spider):
name = 'example.com'
start_urls = ['http://www.example.com/users/login.php']
name = "example.com"
start_urls = ["http://www.example.com/users/login.php"]
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata={'username': 'john', 'password': 'secret'},
callback=self.after_login
formdata={"username": "john", "password": "secret"},
callback=self.after_login,
)
def after_login(self, response):
@ -952,13 +1000,15 @@ dealing with JSON requests.
JsonRequest usage example
-------------------------
Sending a JSON POST request with a JSON payload::
Sending a JSON POST request with a JSON payload:
.. code-block:: python
data = {
'name1': 'value1',
'name2': 'value2',
"name1": "value1",
"name2": "value2",
}
yield JsonRequest(url='http://www.example.com/post/action', data=data)
yield JsonRequest(url="http://www.example.com/post/action", data=data)
Response objects

33
docs/topics/selectors.rst
View File

@ -981,25 +981,34 @@ Selector examples on HTML response
Here are some :class:`Selector` examples to illustrate several concepts.
In all cases, we assume there is already a :class:`Selector` instantiated with
a :class:`~scrapy.http.HtmlResponse` object like this::
a :class:`~scrapy.http.HtmlResponse` object like this:
.. code-block:: python
sel = Selector(html_response)
1. Select all ``<h1>`` elements from an HTML response body, returning a list of
:class:`Selector` objects (i.e. a :class:`SelectorList` object)::
:class:`Selector` objects (i.e. a :class:`SelectorList` object):
.. code-block:: python
sel.xpath("//h1")
2. Extract the text of all ``<h1>`` elements from an HTML response body,
returning a list of strings::
returning a list of strings:
sel.xpath("//h1").getall() # this includes the h1 tag
.. code-block:: python
sel.xpath("//h1").getall() # this includes the h1 tag
sel.xpath("//h1/text()").getall() # this excludes the h1 tag
3. Iterate over all ``<p>`` tags and print their class attribute::
3. Iterate over all ``<p>`` tags and print their class attribute:
.. code-block:: python
for node in sel.xpath("//p"):
print(node.attrib['class'])
print(node.attrib["class"])
.. _selector-examples-xml:
@ -1008,17 +1017,23 @@ Selector examples on XML response
---------------------------------
Here are some examples to illustrate concepts for :class:`Selector` objects
instantiated with an :class:`~scrapy.http.XmlResponse` object::
instantiated with an :class:`~scrapy.http.XmlResponse` object:
.. code-block:: python
sel = Selector(xml_response)
1. Select all ``<product>`` elements from an XML response body, returning a list
of :class:`Selector` objects (i.e. a :class:`SelectorList` object)::
of :class:`Selector` objects (i.e. a :class:`SelectorList` object):
.. code-block:: python
sel.xpath("//product")
2. Extract all prices from a `Google Base XML feed`_ which requires registering
a namespace::
a namespace:
.. code-block:: python
sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall()

202
docs/topics/settings.rst
View File

@ -67,13 +67,15 @@ Example::
Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
own settings that will take precedence and override the project ones. They can
do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute::
do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute:
.. code-block:: python
class MySpider(scrapy.Spider):
name = 'myspider'
name = "myspider"
custom_settings = {
'SOME_SETTING': 'some value',
"SOME_SETTING": "some value",
}
3. Project settings module
@ -115,14 +117,17 @@ class or a function, there are two different ways you can specify that object:
- As the object itself
For example::
For example:
.. code-block:: python
from mybot.pipelines.validate import ValidateMyItem
ITEM_PIPELINES = {
# passing the classname...
ValidateMyItem: 300,
# ...equals passing the class path
'mybot.pipelines.validate.ValidateMyItem': 300,
"mybot.pipelines.validate.ValidateMyItem": 300,
}
.. note:: Passing non-callable objects is not supported.
@ -133,11 +138,13 @@ How to access settings
.. highlight:: python
In a spider, the settings are available through ``self.settings``::
In a spider, the settings are available through ``self.settings``:
.. code-block:: python
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['http://example.com']
name = "myspider"
start_urls = ["http://example.com"]
def parse(self, response):
print(f"Existing settings: {self.settings.attributes.keys()}")
@ -150,7 +157,9 @@ In a spider, the settings are available through ``self.settings``::
Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
attribute of the Crawler that is passed to ``from_crawler`` method in
extensions, middlewares and item pipelines::
extensions, middlewares and item pipelines:
.. code-block:: python
class MyExtension:
def __init__(self, log_is_enabled=False):
@ -160,7 +169,7 @@ extensions, middlewares and item pipelines::
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
return cls(settings.getbool('LOG_ENABLED'))
return cls(settings.getbool("LOG_ENABLED"))
The settings object can be used like a dict (e.g.,
``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting
@ -365,11 +374,13 @@ Scrapy shell <topics-shell>`.
DEFAULT_REQUEST_HEADERS
-----------------------
Default::
Default:
.. code-block:: python
{
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en",
}
The default headers used for Scrapy HTTP Requests. They're populated in the
@ -404,9 +415,11 @@ Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``
An integer that is used to adjust the :attr:`~scrapy.Request.priority` of
a :class:`~scrapy.Request` based on its depth.
The priority of a request is adjusted as follows::
The priority of a request is adjusted as follows:
request.priority = request.priority - ( depth * DEPTH_PRIORITY )
.. code-block:: python
request.priority = request.priority - (depth * DEPTH_PRIORITY)
As depth increases, positive values of ``DEPTH_PRIORITY`` decrease request
priority (BFO), while negative values increase request priority (DFO). See
@ -595,23 +608,25 @@ orders. For more info see :ref:`topics-downloader-middleware-setting`.
DOWNLOADER_MIDDLEWARES_BASE
---------------------------
Default::
Default:
.. code-block:: python
{
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
"scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
"scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
"scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
"scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400,
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500,
"scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
"scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware": 560,
"scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,
"scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,
"scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,
"scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
"scrapy.downloadermiddlewares.stats.DownloaderStats": 850,
"scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900,
}
A dict containing the downloader middlewares enabled by default in Scrapy. Low
@ -687,15 +702,17 @@ See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.
DOWNLOAD_HANDLERS_BASE
----------------------
Default::
Default:
.. code-block:: python
{
'data': 'scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler',
'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',
'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
's3': 'scrapy.core.downloader.handlers.s3.S3DownloadHandler',
'ftp': 'scrapy.core.downloader.handlers.ftp.FTPDownloadHandler',
"data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",
"file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",
"http": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",
"https": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",
"s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
"ftp": "scrapy.core.downloader.handlers.ftp.FTPDownloadHandler",
}
@ -705,10 +722,12 @@ You should never modify this setting in your project, modify
You can disable any of these download handlers by assigning ``None`` to their
URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
handler (without replacement), place this in your ``settings.py``::
handler (without replacement), place this in your ``settings.py``:
.. code-block:: python
DOWNLOAD_HANDLERS = {
'ftp': None,
"ftp": None,
}
.. _http2:
@ -718,10 +737,12 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2:
#. Install ``Twisted[http2]>=17.9.0`` to install the packages required to
enable HTTP/2 support in Twisted.
#. Update :setting:`DOWNLOAD_HANDLERS` as follows::
#. Update :setting:`DOWNLOAD_HANDLERS` as follows:
.. code-block:: python
DOWNLOAD_HANDLERS = {
'https': 'scrapy.core.downloader.handlers.http2.H2DownloadHandler',
"https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
}
.. warning::
@ -890,18 +911,20 @@ A dict containing the extensions enabled in your project, and their orders.
EXTENSIONS_BASE
---------------
Default::
Default:
.. code-block:: python
{
'scrapy.extensions.corestats.CoreStats': 0,
'scrapy.extensions.telnet.TelnetConsole': 0,
'scrapy.extensions.memusage.MemoryUsage': 0,
'scrapy.extensions.memdebug.MemoryDebugger': 0,
'scrapy.extensions.closespider.CloseSpider': 0,
'scrapy.extensions.feedexport.FeedExporter': 0,
'scrapy.extensions.logstats.LogStats': 0,
'scrapy.extensions.spiderstate.SpiderState': 0,
'scrapy.extensions.throttle.AutoThrottle': 0,
"scrapy.extensions.corestats.CoreStats": 0,
"scrapy.extensions.telnet.TelnetConsole": 0,
"scrapy.extensions.memusage.MemoryUsage": 0,
"scrapy.extensions.memdebug.MemoryDebugger": 0,
"scrapy.extensions.closespider.CloseSpider": 0,
"scrapy.extensions.feedexport.FeedExporter": 0,
"scrapy.extensions.logstats.LogStats": 0,
"scrapy.extensions.spiderstate.SpiderState": 0,
"scrapy.extensions.throttle.AutoThrottle": 0,
}
A dict containing the extensions available by default in Scrapy, and their
@ -988,11 +1011,13 @@ A dict containing the item pipelines to use, and their orders. Order values are
arbitrary, but it is customary to define them in the 0-1000 range. Lower orders
process before higher orders.
Example::
Example:
.. code-block:: python
ITEM_PIPELINES = {
'mybot.pipelines.validate.ValidateMyItem': 300,
'mybot.pipelines.validate.StoreMyItem': 800,
"mybot.pipelines.validate.ValidateMyItem": 300,
"mybot.pipelines.validate.StoreMyItem": 800,
}
.. setting:: ITEM_PIPELINES_BASE
@ -1417,12 +1442,14 @@ testing spiders. For more info see :ref:`topics-contracts`.
SPIDER_CONTRACTS_BASE
---------------------
Default::
Default:
.. code-block:: python
{
'scrapy.contracts.default.UrlContract' : 1,
'scrapy.contracts.default.ReturnsContract': 2,
'scrapy.contracts.default.ScrapesContract': 3,
"scrapy.contracts.default.UrlContract": 1,
"scrapy.contracts.default.ReturnsContract": 2,
"scrapy.contracts.default.ScrapesContract": 3,
}
A dict containing the Scrapy contracts enabled by default in Scrapy. You should
@ -1431,10 +1458,12 @@ instead. For more info see :ref:`topics-contracts`.
You can disable any of these contracts by assigning ``None`` to their class
path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in
``ScrapesContract``, place this in your ``settings.py``::
``ScrapesContract``, place this in your ``settings.py``:
.. code-block:: python
SPIDER_CONTRACTS = {
'scrapy.contracts.default.ScrapesContract': None,
"scrapy.contracts.default.ScrapesContract": None,
}
.. setting:: SPIDER_LOADER_CLASS
@ -1483,14 +1512,16 @@ orders. For more info see :ref:`topics-spider-middleware-setting`.
SPIDER_MIDDLEWARES_BASE
-----------------------
Default::
Default:
.. code-block:: python
{
'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
'scrapy.spidermiddlewares.referer.RefererMiddleware': 700,
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800,
'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
"scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
}
A dict containing the spider middlewares enabled by default in Scrapy, and
@ -1506,9 +1537,11 @@ Default: ``[]``
A list of modules where Scrapy will look for spiders.
Example::
Example:
SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev']
.. code-block:: python
SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"]
.. setting:: STATS_CLASS
@ -1597,60 +1630,65 @@ If a reactor is already installed,
third-party libraries will make Scrapy raise :exc:`Exception` when
it checks which reactor is installed.
In order to use the reactor installed by Scrapy::
In order to use the reactor installed by Scrapy:
.. code-block:: python
import scrapy
from twisted.internet import reactor
class QuotesSpider(scrapy.Spider):
name = 'quotes'
name = "quotes"
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop('timeout', '60'))
self.timeout = int(kwargs.pop("timeout", "60"))
super(QuotesSpider, self).__init__(*args, **kwargs)
def start_requests(self):
reactor.callLater(self.timeout, self.stop)
urls = ['https://quotes.toscrape.com/page/1']
urls = ["https://quotes.toscrape.com/page/1"]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for quote in response.css('div.quote'):
yield {'text': quote.css('span.text::text').get()}
for quote in response.css("div.quote"):
yield {"text": quote.css("span.text::text").get()}
def stop(self):
self.crawler.engine.close_spider(self, 'timeout')
self.crawler.engine.close_spider(self, "timeout")
which raises :exc:`Exception`, becomes::
which raises :exc:`Exception`, becomes:
.. code-block:: python
import scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes'
name = "quotes"
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop('timeout', '60'))
self.timeout = int(kwargs.pop("timeout", "60"))
super(QuotesSpider, self).__init__(*args, **kwargs)
def start_requests(self):
from twisted.internet import reactor
reactor.callLater(self.timeout, self.stop)
urls = ['https://quotes.toscrape.com/page/1']
urls = ["https://quotes.toscrape.com/page/1"]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for quote in response.css('div.quote'):
yield {'text': quote.css('span.text::text').get()}
for quote in response.css("div.quote"):
yield {"text": quote.css("span.text::text").get()}
def stop(self):
self.crawler.engine.close_spider(self, 'timeout')
self.crawler.engine.close_spider(self, "timeout")
The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which

5
docs/topics/shell.rst
View File

@ -242,7 +242,9 @@ getting there.
This can be achieved by using the ``scrapy.shell.inspect_response`` function.
Here's an example of how you would call it from your spider::
Here's an example of how you would call it from your spider:
.. code-block:: python
import scrapy
@ -259,6 +261,7 @@ Here's an example of how you would call it from your spider::
# We want to inspect one specific response.
if ".org" in response.url:
from scrapy.shell import inspect_response
inspect_response(response, self)
# Rest of parsing code.

31
docs/topics/signals.rst
View File

@ -16,7 +16,9 @@ deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the
:ref:`topics-api-signals`.
Here is a simple example showing how you can catch signals and perform some action::
Here is a simple example showing how you can catch signals and perform some action:
.. code-block:: python
from scrapy import signals
from scrapy import Spider
@ -30,17 +32,14 @@ Here is a simple example showing how you can catch signals and perform some acti
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
def spider_closed(self, spider):
spider.logger.info('Spider closed: %s', spider.name)
spider.logger.info("Spider closed: %s", spider.name)
def parse(self, response):
pass
@ -56,11 +55,13 @@ you to run asynchronous code that does not block Scrapy. If a signal
handler returns one of these objects, Scrapy waits for that asynchronous
operation to finish.
Let's take an example using :ref:`coroutines <topics-coroutines>`::
Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. code-block:: python
class SignalSpider(scrapy.Spider):
name = 'signals'
start_urls = ['https://quotes.toscrape.com/page/1/']
name = "signals"
start_urls = ["https://quotes.toscrape.com/page/1/"]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
@ -71,19 +72,19 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`::
async def item_scraped(self, item):
# Send the scraped item to the server
response = await treq.post(
'http://example.com/post',
json.dumps(item).encode('ascii'),
headers={b'Content-Type': [b'application/json']}
"http://example.com/post",
json.dumps(item).encode("ascii"),
headers={b"Content-Type": [b"application/json"]},
)
return response
def parse(self, response):
for quote in response.css('div.quote'):
for quote in response.css("div.quote"):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
See the :ref:`topics-signals-ref` below to know which signals support

18
docs/topics/spider-middleware.rst
View File

@ -18,10 +18,12 @@ To activate a spider middleware component, add it to the
:setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class path and their values are the middleware orders.
Here's an example::
Here's an example:
.. code-block:: python
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
"myproject.middlewares.CustomSpiderMiddleware": 543,
}
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the
@ -44,11 +46,13 @@ previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware (the ones defined in
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
value. For example, if you want to disable the off-site middleware::
value. For example, if you want to disable the off-site middleware:
.. code-block:: python
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,
"myproject.middlewares.CustomSpiderMiddleware": 543,
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
}
Finally, keep in mind that some middlewares may need to be enabled through a
@ -261,7 +265,9 @@ specify which response codes the spider is able to handle using the
:setting:`HTTPERROR_ALLOWED_CODES` setting.
For example, if you want your spider to handle 404 responses you can do
this::
this:
.. code-block:: python
class MySpider(CrawlSpider):
handle_httpstatus_list = [404]

246
docs/topics/spiders.rst
View File

@ -157,15 +157,21 @@ scrapy.Spider
If you want to change the Requests used to start scraping a domain, this is
the method to override. For example, if you need to start by logging in using
a POST request, you could do::
a POST request, you could do:
.. code-block:: python
class MySpider(scrapy.Spider):
name = 'myspider'
name = "myspider"
def start_requests(self):
return [scrapy.FormRequest("http://www.example.com/login",
formdata={'user': 'john', 'pass': 'secret'},
callback=self.logged_in)]
return [
scrapy.FormRequest(
"http://www.example.com/login",
formdata={"user": "john", "pass": "secret"},
callback=self.logged_in,
)
]
def logged_in(self, response):
# here you would extract links to follow and return Requests for
@ -200,63 +206,71 @@ scrapy.Spider
Called when the spider closes. This method provides a shortcut to
signals.connect() for the :signal:`spider_closed` signal.
Let's see an example::
Let's see an example:
.. code-block:: python
import scrapy
class MySpider(scrapy.Spider):
name = 'example.com'
allowed_domains = ['example.com']
name = "example.com"
allowed_domains = ["example.com"]
start_urls = [
'http://www.example.com/1.html',
'http://www.example.com/2.html',
'http://www.example.com/3.html',
"http://www.example.com/1.html",
"http://www.example.com/2.html",
"http://www.example.com/3.html",
]
def parse(self, response):
self.logger.info('A response from %s just arrived!', response.url)
self.logger.info("A response from %s just arrived!", response.url)
Return multiple Requests and items from a single callback::
Return multiple Requests and items from a single callback:
.. code-block:: python
import scrapy
class MySpider(scrapy.Spider):
name = 'example.com'
allowed_domains = ['example.com']
name = "example.com"
allowed_domains = ["example.com"]
start_urls = [
'http://www.example.com/1.html',
'http://www.example.com/2.html',
'http://www.example.com/3.html',
"http://www.example.com/1.html",
"http://www.example.com/2.html",
"http://www.example.com/3.html",
]
def parse(self, response):
for h3 in response.xpath('//h3').getall():
for h3 in response.xpath("//h3").getall():
yield {"title": h3}
for href in response.xpath('//a/@href').getall():
for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse)
Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
to give data more structure you can use :class:`~scrapy.Item` objects::
to give data more structure you can use :class:`~scrapy.Item` objects:
.. code-block:: python
import scrapy
from myproject.items import MyItem
class MySpider(scrapy.Spider):
name = 'example.com'
allowed_domains = ['example.com']
name = "example.com"
allowed_domains = ["example.com"]
def start_requests(self):
yield scrapy.Request('http://www.example.com/1.html', self.parse)
yield scrapy.Request('http://www.example.com/2.html', self.parse)
yield scrapy.Request('http://www.example.com/3.html', self.parse)
yield scrapy.Request("http://www.example.com/1.html", self.parse)
yield scrapy.Request("http://www.example.com/2.html", self.parse)
yield scrapy.Request("http://www.example.com/3.html", self.parse)
def parse(self, response):
for h3 in response.xpath('//h3').getall():
for h3 in response.xpath("//h3").getall():
yield MyItem(title=h3)
for href in response.xpath('//a/@href').getall():
for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse)
.. _spiderargs:
@ -274,34 +288,42 @@ Spider arguments are passed through the :command:`crawl` command using the
scrapy crawl myspider -a category=electronics
Spiders can access arguments in their `__init__` methods::
Spiders can access arguments in their `__init__` methods:
.. code-block:: python
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
name = "myspider"
def __init__(self, category=None, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs)
self.start_urls = [f'http://www.example.com/categories/{category}']
self.start_urls = [f"http://www.example.com/categories/{category}"]
# ...
The default `__init__` method will take any spider arguments
and copy them to the spider as attributes.
The above example can also be written as follows::
The above example can also be written as follows:
.. code-block:: python
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
name = "myspider"
def start_requests(self):
yield scrapy.Request(f'http://www.example.com/categories/{self.category}')
yield scrapy.Request(f"http://www.example.com/categories/{self.category}")
If you are :ref:`running Scrapy from a script <run-from-script>`, you can
specify spider arguments when calling
:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`::
:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
.. code-block:: python
process = CrawlerProcess()
process.crawl(MySpider, category="electronics")
@ -337,10 +359,13 @@ common scraping cases, like following all links on a site based on certain
rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.
For the examples used in the following spiders, we'll assume you have a project
with a ``TestItem`` declared in a ``myproject.items`` module::
with a ``TestItem`` declared in a ``myproject.items`` module:
.. code-block:: python
import scrapy
class TestItem(scrapy.Item):
id = scrapy.Field()
name = scrapy.Field()
@ -436,38 +461,46 @@ Crawling rules
CrawlSpider example
~~~~~~~~~~~~~~~~~~~
Let's now take a look at an example CrawlSpider with rules::
Let's now take a look at an example CrawlSpider with rules:
.. code-block:: python
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class MySpider(CrawlSpider):
name = 'example.com'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com']
name = "example.com"
allowed_domains = ["example.com"]
start_urls = ["http://www.example.com"]
rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
Rule(LinkExtractor(allow=("category\.php",), deny=("subsection\.php",))),
# Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
Rule(LinkExtractor(allow=("item\.php",)), callback="parse_item"),
)
def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url)
self.logger.info("Hi, this is an item page! %s", response.url)
item = scrapy.Item()
item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
item['name'] = response.xpath('//td[@id="item_name"]/text()').get()
item['description'] = response.xpath('//td[@id="item_description"]/text()').get()
item['link_text'] = response.meta['link_text']
item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)")
item["name"] = response.xpath('//td[@id="item_name"]/text()').get()
item["description"] = response.xpath(
'//td[@id="item_description"]/text()'
).get()
item["link_text"] = response.meta["link_text"]
url = response.xpath('//td[@id="additional_data"]/@href').get()
return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))
return response.follow(
url, self.parse_additional_page, cb_kwargs=dict(item=item)
)
def parse_additional_page(self, response, item):
item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()
item["additional_data"] = response.xpath(
'//p[@id="additional_data"]/text()'
).get()
return item
@ -568,25 +601,30 @@ XMLFeedSpider
XMLFeedSpider example
~~~~~~~~~~~~~~~~~~~~~
These spiders are pretty easy to use, let's have a look at one example::
These spiders are pretty easy to use, let's have a look at one example:
.. code-block:: python
from scrapy.spiders import XMLFeedSpider
from myproject.items import TestItem
class MySpider(XMLFeedSpider):
name = 'example.com'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com/feed.xml']
iterator = 'iternodes' # This is actually unnecessary, since it's the default value
itertag = 'item'
name = "example.com"
allowed_domains = ["example.com"]
start_urls = ["http://www.example.com/feed.xml"]
iterator = "iternodes" # This is actually unnecessary, since it's the default value
itertag = "item"
def parse_node(self, response, node):
self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.getall()))
self.logger.info(
"Hi, this is a <%s> node!: %s", self.itertag, "".join(node.getall())
)
item = TestItem()
item['id'] = node.xpath('@id').get()
item['name'] = node.xpath('name').get()
item['description'] = node.xpath('description').get()
item["id"] = node.xpath("@id").get()
item["name"] = node.xpath("name").get()
item["description"] = node.xpath("description").get()
return item
Basically what we did up there was to create a spider that downloads a feed from
@ -627,26 +665,29 @@ CSVFeedSpider example
~~~~~~~~~~~~~~~~~~~~~
Let's see an example similar to the previous one, but using a
:class:`CSVFeedSpider`::
:class:`CSVFeedSpider`:
.. code-block:: python
from scrapy.spiders import CSVFeedSpider
from myproject.items import TestItem
class MySpider(CSVFeedSpider):
name = 'example.com'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com/feed.csv']
delimiter = ';'
name = "example.com"
allowed_domains = ["example.com"]
start_urls = ["http://www.example.com/feed.csv"]
delimiter = ";"
quotechar = "'"
headers = ['id', 'name', 'description']
headers = ["id", "name", "description"]
def parse_row(self, response, row):
self.logger.info('Hi, this is a row!: %r', row)
self.logger.info("Hi, this is a row!: %r", row)
item = TestItem()
item['id'] = row['id']
item['name'] = row['name']
item['description'] = row['description']
item["id"] = row["id"]
item["name"] = row["name"]
item["description"] = row["description"]
return item
@ -728,19 +769,22 @@ SitemapSpider
<lastmod>2005-01-01</lastmod>
</url>
We can define a ``sitemap_filter`` function to filter ``entries`` by date::
We can define a ``sitemap_filter`` function to filter ``entries`` by date:
.. code-block:: python
from datetime import datetime
from scrapy.spiders import SitemapSpider
class FilteredSitemapSpider(SitemapSpider):
name = 'filtered_sitemap_spider'
allowed_domains = ['example.com']
sitemap_urls = ['http://example.com/sitemap.xml']
name = "filtered_sitemap_spider"
allowed_domains = ["example.com"]
sitemap_urls = ["http://example.com/sitemap.xml"]
def sitemap_filter(self, entries):
for entry in entries:
date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
date_time = datetime.strptime(entry["lastmod"], "%Y-%m-%d")
if date_time.year >= 2005:
yield entry
@ -765,60 +809,72 @@ SitemapSpider examples
~~~~~~~~~~~~~~~~~~~~~~
Simplest example: process all urls discovered through sitemaps using the
``parse`` callback::
``parse`` callback:
.. code-block:: python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/sitemap.xml']
sitemap_urls = ["http://www.example.com/sitemap.xml"]
def parse(self, response):
pass # ... scrape item here ...
pass # ... scrape item here ...
Process some urls with certain callback and other urls with a different
callback::
callback:
.. code-block:: python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/sitemap.xml']
sitemap_urls = ["http://www.example.com/sitemap.xml"]
sitemap_rules = [
('/product/', 'parse_product'),
('/category/', 'parse_category'),
("/product/", "parse_product"),
("/category/", "parse_category"),
]
def parse_product(self, response):
pass # ... scrape product ...
pass # ... scrape product ...
def parse_category(self, response):
pass # ... scrape category ...
pass # ... scrape category ...
Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps
whose url contains ``/sitemap_shop``::
whose url contains ``/sitemap_shop``:
.. code-block:: python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/robots.txt']
sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [
('/shop/', 'parse_shop'),
("/shop/", "parse_shop"),
]
sitemap_follow = ['/sitemap_shops']
sitemap_follow = ["/sitemap_shops"]
def parse_shop(self, response):
pass # ... scrape shop here ...
pass # ... scrape shop here ...
Combine SitemapSpider with other sources of urls::
Combine SitemapSpider with other sources of urls:
.. code-block:: python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ['http://www.example.com/robots.txt']
sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [
('/shop/', 'parse_shop'),
("/shop/", "parse_shop"),
]
other_urls = ['http://www.example.com/about']
other_urls = ["http://www.example.com/about"]
def start_requests(self):
requests = list(super(MySpider, self).start_requests())
@ -826,10 +882,10 @@ Combine SitemapSpider with other sources of urls::
return requests
def parse_shop(self, response):
pass # ... scrape shop here ...
pass # ... scrape shop here ...
def parse_other(self, response):
pass # ... scrape other here ...
pass # ... scrape other here ...
.. _Sitemaps: https://www.sitemaps.org/index.html
.. _Sitemap index files: https://www.sitemaps.org/protocol.html#index

29
docs/topics/stats.rst
View File

@ -30,10 +30,11 @@ Common Stats Collector uses
===========================
Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats`
attribute. Here is an example of an extension that access stats::
attribute. Here is an example of an extension that access stats:
.. code-block:: python
class ExtensionThatAccessStats:
def __init__(self, stats):
self.stats = stats
@ -41,21 +42,29 @@ attribute. Here is an example of an extension that access stats::
def from_crawler(cls, crawler):
return cls(crawler.stats)
Set stat value::
Set stat value:
stats.set_value('hostname', socket.gethostname())
.. code-block:: python
Increment stat value::
stats.set_value("hostname", socket.gethostname())
stats.inc_value('custom_count')
Increment stat value:
Set stat value only if greater than previous::
.. code-block:: python
stats.max_value('max_items_scraped', value)
stats.inc_value("custom_count")
Set stat value only if lower than previous::
Set stat value only if greater than previous:
stats.min_value('min_free_memory_percent', value)
.. code-block:: python
stats.max_value("max_items_scraped", value)
Set stat value only if lower than previous:
.. code-block:: python
stats.min_value("min_free_memory_percent", value)
Get stat value:

549
sep/sep-001.rst
View File

@ -1,274 +1,275 @@
======= ============================================
SEP 1
Title API for populating item fields (comparison)
Author Ismael Carnales, Pablo Hoffman, Daniel Grana
Created 2009-07-19
Status Obsoleted by :ref:`sep-008`
======= ============================================
=====================================================
SEP-001 - API for populating item fields (comparison)
=====================================================
This page shows different usage scenarios for the two new proposed API for
populating item field values (which will replace the old deprecated !RobustItem
API) and compares them. One of these will be chosen as the recommended (and
supported) mechanism in Scrapy 0.7.
Candidates and their API
========================
RobustItem (old, deprecated)
----------------------------
- ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)``
.. note:: ``attribute()`` modifiers (like ``add=True``) are passed together
with adaptor args as keyword arguments (this is ugly)
ItemForm
--------
- ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments
- ``__setitem__(field_name, selector_or_value)``
- set field value
- ``__getitem__(field_name)``
- return the "computed" value of a field (the one that would be set to the item).
returns ``None`` if not set.
- ``get_item()``
- return the item populated with the data provided so far
ItemBuilder
-----------
- ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemBuilder`` with predefined adaptor arguments
- ``add_value(field_name, selector_or_value, **adaptor_args)``
- add value to field
- ``replace_value(field_name, selector_or_value, **adaptor_args)``
- replace existing field value
- ``get_value(field_name)``
- return the "computed" value of a field (the one that would be set to the
item). returns ``None`` if not set.
- ``get_item()``
- return the item populated with the data provided so far
Pros and cons of each candidate
===============================
ItemForm
--------
Pros:
- same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html)
- some people consider setitem API more elegant than methods API
Cons:
- doesn't allow passing run-time arguments to adaptors on assign, you have to
override the adaptors for your spider if you need specific parameters, which
can be an overhead. Example:
Neutral:
- solves the add=True problem using standard ``__add__`` and ``list.append()`` method
ItemBuilder
-----------
Pros:
- allows passing run-time arguments to adaptors on assigned
Cons:
- some people consider setitem API more elegant than methods API
Neutral:
- solves the "add=True" problem by implementing different methods per action
(replacing or adding)
Usage Scenarios for each candidate
==================================
Defining adaptors
-----------------
ItemForm
~~~~~~~~
::
#!python
class NewsForm(ItemForm):
item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip)
ItemBuilder
~~~~~~~~~~~
::
#!python
class NewsBuilder(ItemBuilder):
item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip)
Creating an Item
----------------
ItemForm
~~~~~~~~
::
#!python
ia = NewsForm(response)
ia['url'] = response.url
ia['headline'] = x.x('//h1[@class="headline"]')
# if we want to add another value to the same field
ia['headline'] += x.x('//h1[@class="headline2"]')
# if we want to replace the field value other value to the same field
ia['headline'] = x.x('//h1[@class="headline3"]')
return ia.get_item()
ItemBuilder
~~~~~~~~~~~
::
#!python
il = NewsBuilder(response)
il.add_value('url', response.url)
il.add_value('headline', x.x('//h1[@class="headline"]'))
# if we want to add another value to the same field
il.add_value('headline', x.x('//h1[@class="headline2"]'))
# if we want to replace the field value other value to the same field
il.replace_value('headline', x.x('//h1[@class="headline3"]'))
return il.get_item()
Using different adaptors per Spider/Site
----------------------------------------
ItemForm
~~~~~~~~
::
#!python
class SiteNewsFrom(NewsForm):
published = adaptor(HtmlNewsForm.published, to_date('%d.%m.%Y'))
ItemBuilder
~~~~~~~~~~~
::
#!python
class SiteNewsBuilder(NewsBuilder):
published = adaptor(HtmlNewsBuilder.published, to_date('%d.%m.%Y'))
Check the value of an item being-extracted
------------------------------------------
ItemForm
~~~~~~~~
::
#!python
ia = NewsForm(response)
ia['headline'] = x.x('//h1[@class="headline"]')
if not ia['headline']:
ia['headline'] = x.x('//h1[@class="title"]')
ItemBuilder
~~~~~~~~~~~
::
#!python
il = NewsBuilder(response)
il.add_value('headline', x.x('//h1[@class="headline"]'))
if not nf.get_value('headline'):
il.add_value('headline', x.x('//h1[@class="title"]'))
Adding a value to a list attribute/field
----------------------------------------
ItemForm
~~~~~~~~
::
#!python
ia['headline'] += x.x('//h1[@class="headline"]')
ItemBuilder
~~~~~~~~~~~
::
#!python
il.add_value('headline', x.x('//h1[@class="headline"]'))
Passing run-time arguments to adaptors
--------------------------------------
ItemForm
~~~~~~~~
::
#!python
# Only approach is passing arguments when instantiating the form
ia = NewsForm(response, default_unit='cm')
ia['width'] = x.x('//p[@class="width"]')
ItemBuilder
~~~~~~~~~~~
::
#!python
il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm')
# an alternative approach (more efficient)
il = NewsBuilder(response, default_unit='cm')
il.add_value('width', x.x('//p[@class="width"]'))
Passing run-time arguments to adaptors (same argument name)
-----------------------------------------------------------
ItemForm
~~~~~~~~
::
#!python
class MySiteForm(ItemForm):
width = adaptor(ItemForm.width, default_unit='cm')
volume = adaptor(ItemForm.width, default_unit='lt')
ia['width'] = x.x('//p[@class="width"]')
ia['volume'] = x.x('//p[@class="volume"]')
# another example passing parameters on instance
ia = NewsForm(response, encoding='utf-8')
ia['name'] = x.x('//p[@class="name"]')
ItemBuilder
~~~~~~~~~~~
::
#!python
il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm')
il.add_value('volume', x.x('//p[@class="volume"]'), default_unit='lt')
======= ============================================
SEP 1
Title API for populating item fields (comparison)
Author Ismael Carnales, Pablo Hoffman, Daniel Grana
Created 2009-07-19
Status Obsoleted by :ref:`sep-008`
======= ============================================
=====================================================
SEP-001 - API for populating item fields (comparison)
=====================================================
This page shows different usage scenarios for the two new proposed API for
populating item field values (which will replace the old deprecated !RobustItem
API) and compares them. One of these will be chosen as the recommended (and
supported) mechanism in Scrapy 0.7.
Candidates and their API
========================
RobustItem (old, deprecated)
----------------------------
- ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)``
.. note:: ``attribute()`` modifiers (like ``add=True``) are passed together
with adaptor args as keyword arguments (this is ugly)
ItemForm
--------
- ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments
- ``__setitem__(field_name, selector_or_value)``
- set field value
- ``__getitem__(field_name)``
- return the "computed" value of a field (the one that would be set to the item).
returns ``None`` if not set.
- ``get_item()``
- return the item populated with the data provided so far
ItemBuilder
-----------
- ``__init__(response, item=None, **adaptor_args)``
- instantiate an ``ItemBuilder`` with predefined adaptor arguments
- ``add_value(field_name, selector_or_value, **adaptor_args)``
- add value to field
- ``replace_value(field_name, selector_or_value, **adaptor_args)``
- replace existing field value
- ``get_value(field_name)``
- return the "computed" value of a field (the one that would be set to the
item). returns ``None`` if not set.
- ``get_item()``
- return the item populated with the data provided so far
Pros and cons of each candidate
===============================
ItemForm
--------
Pros:
- same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html)
- some people consider setitem API more elegant than methods API
Cons:
- doesn't allow passing run-time arguments to adaptors on assign, you have to
override the adaptors for your spider if you need specific parameters, which
can be an overhead. Example:
Neutral:
- solves the add=True problem using standard ``__add__`` and ``list.append()`` method
ItemBuilder
-----------
Pros:
- allows passing run-time arguments to adaptors on assigned
Cons:
- some people consider setitem API more elegant than methods API
Neutral:
- solves the "add=True" problem by implementing different methods per action
(replacing or adding)
Usage Scenarios for each candidate
==================================
Defining adaptors
-----------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
class NewsForm(ItemForm):
item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip)
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
class NewsBuilder(ItemBuilder):
item_class = NewsItem
url = adaptor(extract, remove_tags(), unquote(), strip)
headline = adaptor(extract, remove_tags(), unquote(), strip)
Creating an Item
----------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
ia = NewsForm(response)
ia["url"] = response.url
ia["headline"] = x.x('//h1[@class="headline"]')
# if we want to add another value to the same field
ia["headline"] += x.x('//h1[@class="headline2"]')
# if we want to replace the field value other value to the same field
ia["headline"] = x.x('//h1[@class="headline3"]')
return ia.get_item()
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
il = NewsBuilder(response)
il.add_value("url", response.url)
il.add_value("headline", x.x('//h1[@class="headline"]'))
# if we want to add another value to the same field
il.add_value("headline", x.x('//h1[@class="headline2"]'))
# if we want to replace the field value other value to the same field
il.replace_value("headline", x.x('//h1[@class="headline3"]'))
return il.get_item()
Using different adaptors per Spider/Site
----------------------------------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
class SiteNewsFrom(NewsForm):
published = adaptor(HtmlNewsForm.published, to_date("%d.%m.%Y"))
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
class SiteNewsBuilder(NewsBuilder):
published = adaptor(HtmlNewsBuilder.published, to_date("%d.%m.%Y"))
Check the value of an item being-extracted
------------------------------------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
ia = NewsForm(response)
ia["headline"] = x.x('//h1[@class="headline"]')
if not ia["headline"]:
ia["headline"] = x.x('//h1[@class="title"]')
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
il = NewsBuilder(response)
il.add_value("headline", x.x('//h1[@class="headline"]'))
if not nf.get_value("headline"):
il.add_value("headline", x.x('//h1[@class="title"]'))
Adding a value to a list attribute/field
----------------------------------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
ia["headline"] += x.x('//h1[@class="headline"]')
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
il.add_value("headline", x.x('//h1[@class="headline"]'))
Passing run-time arguments to adaptors
--------------------------------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
# Only approach is passing arguments when instantiating the form
ia = NewsForm(response, default_unit="cm")
ia["width"] = x.x('//p[@class="width"]')
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")
# an alternative approach (more efficient)
il = NewsBuilder(response, default_unit="cm")
il.add_value("width", x.x('//p[@class="width"]'))
Passing run-time arguments to adaptors (same argument name)
-----------------------------------------------------------
ItemForm
~~~~~~~~
.. code-block:: python
#!python
class MySiteForm(ItemForm):
width = adaptor(ItemForm.width, default_unit="cm")
volume = adaptor(ItemForm.width, default_unit="lt")
ia["width"] = x.x('//p[@class="width"]')
ia["volume"] = x.x('//p[@class="volume"]')
# another example passing parameters on instance
ia = NewsForm(response, encoding="utf-8")
ia["name"] = x.x('//p[@class="name"]')
ItemBuilder
~~~~~~~~~~~
.. code-block:: python
#!python
il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")
il.add_value("volume", x.x('//p[@class="volume"]'), default_unit="lt")

44
sep/sep-002.rst
View File

@ -16,18 +16,19 @@ called !ListField.
Proposed Implementation
=======================
::
.. code-block:: python
#!python
from scrapy.item.fields import BaseField
class ListField(BaseField):
def __init__(self, field, default=None):
self._field = field
super(ListField, self).__init__(default)
def to_python(self, value):
if hasattr(value, '__iter__'): # str/unicode not allowed
if hasattr(value, "__iter__"): # str/unicode not allowed
return [self._field.to_python(v) for v in value]
else:
raise TypeError("Expected iterable, got %s" % type(value).__name__)
@ -42,12 +43,13 @@ Usage Scenarios
Defining a list field
---------------------
::
.. code-block:: python
#!python
from scrapy.item.models import Item
from scrapy.item.fields import ListField, TextField, DateField, IntegerField
class Article(Item):
categories = ListField(TextField)
dates = ListField(DateField, default=[])
@ -56,57 +58,59 @@ Defining a list field
Another case of products and variants which highlights the fact that it's
important to instantiate !ListField with field instances, not classes:
::
.. code-block:: python
#!python
from scrapy.item.models import Item
from scrapy.item.fields import ListField, TextField
class Variant(Item):
name = TextField()
class Product(Variant):
variants = ListField(ItemField(Variant))
Assigning a list field
----------------------
::
.. code-block:: python
#!python
i = Article()
i['categories'] = []
i['categories'] = ['politics', 'sport']
i['categories'] = ['test', 1] -> raises TypeError
i['categories'] = asd -> raises TypeError
i["categories"] = []
i["categories"] = ["politics", "sport"]
i["categories"] = ["test", 1] # -> raises TypeError
i["categories"] = asd # -> raises TypeError
i['dates'] = []
i['dates'] = ['2009-01-01'] # raises TypeError? (depends on TextField)
i["dates"] = []
i["dates"] = ["2009-01-01"] # raises TypeError? (depends on TextField)
i['numbers'] = ['1', 2, '3']
i['numbers'] # returns [1, 2, 3]
i["numbers"] = ["1", 2, "3"]
i["numbers"] # returns [1, 2, 3]
Default values
--------------
::
.. code-block:: python
#!python
i = Article()
i['categories'] # raises KeyError
i.get('categories') # returns None
i["categories"] # raises KeyError
i.get("categories") # returns None
i['numbers'] # returns []
i["numbers"] # returns []
Appending values
----------------
::
.. code-block:: python
#!python
i = Article()
i['categories'] = ['one', 'two']
i['categories'].append(3) # XXX: should this fail?
i["categories"] = ["one", "two"]
i["categories"].append(3) # XXX: should this fail?

90
sep/sep-003.rst
View File

@ -27,18 +27,21 @@ This API proposal relies on the following API:
Proposed Implementation of ItemField
====================================
::
.. code-block:: python
#!python
from scrapy.item.fields import BaseField
class ItemField(BaseField):
def __init__(self, item_type, default=None):
self._item_type = item_type
super(ItemField, self).__init__(default)
def to_python(self, value):
return self._item_type(value) if not isinstance(value, self._item_type) else value
return (
self._item_type(value) if not isinstance(value, self._item_type) else value
)
def get_default(self):
# WARNING: returns default item instead of a copy - this must be
@ -54,25 +57,28 @@ Usage Scenarios
Defining an item containing ItemField's
---------------------------------------
::
.. code-block:: python
#!python
from scrapy.item.models import Item
from scrapy.item.fields import ListField, ItemField, TextField, UrlField, DecimalField
class Supplier(Item):
name = TextField(default="anonymous supplier")
url = UrlField()
class Variant(Item):
name = TextField(required=True)
url = UrlField()
price = DecimalField()
class Product(Variant):
supplier = ItemField(Supplier, default=Supplier(name="default supplier")
supplier = ItemField(Supplier, default=Supplier(name="default supplier"))
variants = ListField(ItemField(Variant))
# these ones are used for documenting default value examples
supplier2 = ItemField(Supplier)
variants2 = ListField(ItemField(Variant), default=[])
@ -81,16 +87,16 @@ It's important to note here that the (perhaps most intuitive) way of defining a
Product-Variant relationship (i.e. defining a recursive !ItemField) doesn't
work. For example, this fails to compile:
::
.. code-block:: python
#!python
class Product(Item):
variants = ItemField(Product) # Fails to compile
variants = ItemField(Product) # Fails to compile
Assigning an item field
-----------------------
::
.. code-block:: python
#!python
supplier = Supplier(name="Supplier 1", url="http://example.com")
@ -98,69 +104,69 @@ Assigning an item field
p = Product()
# standard assignment
p['supplier'] = supplier
p["supplier"] = supplier
# this also works as it tries to instantiate a Supplier with the given dict
p['supplier'] = {'name': 'Supplier 1' url='http://example.com'}
# this fails because it can't instantiate a Supplier
p['supplier'] = 'Supplier 1'
p["supplier"] = {"name": "Supplier 1", url: "http://example.com"}
# this fails because it can't instantiate a Supplier
p["supplier"] = "Supplier 1"
# this fails because url doesn't have the valid type
p['supplier'] = {'name': 'Supplier 1' url=123}
p["supplier"] = {"name": "Supplier 1", url: 123}
v1 = Variant()
v1['name'] = "lala"
v1['price'] = Decimal("100")
v1["name"] = "lala"
v1["price"] = Decimal("100")
v2 = Variant()
v2['name'] = "lolo"
v2['price'] = Decimal("150")
v2["name"] = "lolo"
v2["price"] = Decimal("150")
# standard assignment
p['variants'] = [v1, v2] # OK
p["variants"] = [v1, v2] # OK
# can also instantiate at assignment time
p['variants'] = [v1, Variant(name="lolo", price=Decimal("150")]
p["variants"] = [v1, Variant(name="lolo", price=Decimal("150"))]
# this also works as it tries to instantiate a Variant with the given dict
p['variants'] = [v1, {'name': 'lolo', 'price': Decimal("150")]
# this fails because it can't instantiate a Variant
p['variants'] = [v1, 'test']
p["variants"] = [v1, {"name": "lolo", "price": Decimal("150")}]
# this fails because it can't instantiate a Variant
p["variants"] = [v1, "test"]
# this fails because 'coco' is not a valid value for price
p['variants'] = [v1, {'name': 'lolo', 'price': 'coco']
p["variants"] = [v1, {"name": "lolo", "price": "coco"}]
Default values
--------------
::
.. code-block:: python
#!python
p = Product()
p['supplier'] # returns: Supplier(name='default supplier')
p['supplier2'] # raises KeyError
p['supplier2'] = Supplier()
p['supplier2'] # returns: Supplier(name='anonymous supplier')
p["supplier"] # returns: Supplier(name='default supplier')
p["supplier2"] # raises KeyError
p["supplier2"] = Supplier()
p["supplier2"] # returns: Supplier(name='anonymous supplier')
p['variants'] # raises KeyError
p['variants2'] # returns []
p["variants"] # raises KeyError
p["variants2"] # returns []
p['categories'] # raises KeyError
p.get('categories') # returns None
p["categories"] # raises KeyError
p.get("categories") # returns None
p['numbers'] # returns []
p["numbers"] # returns []
Accessing and changing nested item values
----------------------------------------
::
.. code-block:: python
#!python
p = Product(supplier=Supplier(name="some name", url="http://example.com"))
p['supplier']['url'] # returns 'http://example.com'
p['supplier']['url'] = "http://www.other.com" # works as expected
p['supplier']['url'] = 123 # fails: wrong type for supplier url
p["supplier"]["url"] # returns 'http://example.com'
p["supplier"]["url"] = "http://www.other.com" # works as expected
p["supplier"]["url"] = 123 # fails: wrong type for supplier url
p['variants'] = [v1, v2]
p['variants'][0]['name'] # returns v1 name
p['variants'][1]['name'] # returns v2 name
p["variants"] = [v1, v2]
p["variants"][0]["name"] # returns v1 name
p["variants"][1]["name"] # returns v2 name
# XXX: decide what to do about these cases:
p['variants'].append(v3) # works but doesn't check type of v3
p['variants'].append(1) # works but shouldn't?
p["variants"].append(v3) # works but doesn't check type of v3
p["variants"].append(1) # works but shouldn't?

9
sep/sep-004.rst
View File

@ -26,7 +26,7 @@ Proposed API
Here's a simple proof-of-concept code of such script:
::
.. code-block:: python
#!/usr/bin/env python
from scrapy.http import Request
@ -35,21 +35,24 @@ Here's a simple proof-of-concept code of such script:
# a container to hold scraped items
scraped_items = []
def parse_start_page(response):
# collect urls to follow into urls_to_follow list
requests = [Request(url, callback=parse_other_page) for url in urls_to_follow]
return requests
def parse_other_page(response):
# ... parse items from response content ...
scraped_items.extend(parsed_items)
start_urls = ["http://www.example.com/start_page.html"]
cr = Crawler(start_urls, callback=parse_start_page)
cr.run() # blocking call - this populates scraped_items
cr.run() # blocking call - this populates scraped_items
print "%d items scraped" % len(scraped_items)
print("%d items scraped" % len(scraped_items))
# ... do something more interesting with scraped_items ...
The behaviour of the Scrapy crawler would be controller by the Scrapy settings,

33
sep/sep-005.rst
View File

@ -12,7 +12,7 @@ SEP-005: Detailed ``ItemBuilder`` API use
Item class for examples:
::
.. code-block:: python
#!python
class NewsItem(Item):
@ -25,7 +25,7 @@ Item class for examples:
gSetting expanders
==================
::
.. code-block:: python
#!python
class NewsItemBuilder(ItemBuilder):
@ -44,7 +44,7 @@ on their Item Field class:
gSetting reducers
=================
::
.. code-block:: python
#!python
class NewsItemBuilder(ItemBuilder):
@ -60,7 +60,7 @@ content
gSetting expanders/reducers new way
===================================
::
.. code-block:: python
#!python
class NewsItemBuilder(ItemBuilder):
@ -76,28 +76,29 @@ gSetting expanders/reducers new way
gExtending ``ItemBuilder``
==========================
::
.. code-block:: python
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(),
strip, to_date('%d.%m.%Y'))
published = reducers.Reducer(
extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
)
gExtending ``ItemBuilder`` using statich methods
================================================
::
.. code-block:: python
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.published, to_date('%d.%m.%Y'))
published = reducers.Reducer(NewsItemBuilder.published, to_date("%d.%m.%Y"))
gUsing default_builder
======================
::
.. code-block:: python
#!python
class DefaultedNewsItemBuilder(ItemBuilder):
@ -112,7 +113,7 @@ As a reducer is not set reducers will be set based on Item Field classes.
gReset default_builder for a field
==================================
::
.. code-block:: python
#!python
class DefaultedNewsItemBuilder(ItemBuilder):
@ -125,18 +126,20 @@ gReset default_builder for a field
gExtending default ``ItemBuilder``
==================================
::
.. code-block:: python
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(), strip, to_date('%d.%m.%Y'))
published = reducers.Reducer(
extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
)
gExtending default ``ItemBuilder`` using static methods
=======================================================
::
.. code-block:: python
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.default_builder, to_date('%d.%m.%Y'))
published = reducers.Reducer(NewsItemBuilder.default_builder, to_date("%d.%m.%Y"))

5
sep/sep-008.rst
View File

@ -87,11 +87,12 @@ Alternative Public API Proposal
Usage example: declaring Item Parsers
=====================================
::
.. code-block:: python
#!python
from scrapy.contrib.itemparser import XPathItemParser, parsers
class ProductParser(XPathItemParser):
name_in = parsers.MapConcat(removetags, filterx)
price_in = parsers.MapConcat(...)
@ -101,7 +102,7 @@ Usage example: declaring Item Parsers
Usage example: declaring parsers in Fields
==========================================
::
.. code-block:: python
#!python
class Product(Item):

18
sep/sep-009.rst
View File

@ -76,28 +76,30 @@ which we haven't documented so far (partly because of this).
So, for a typical middleware ``__init__`` method code, instead of this:
::
.. code-block:: python
#!python
from scrapy.core.exceptions import NotConfigured
from scrapy.conf import settings
class SomeMiddleware(object):
def __init__(self):
if not settings.getbool('SOMEMIDDLEWARE_ENABLED'):
raise NotConfigured
def __init__(self):
if not settings.getbool("SOMEMIDDLEWARE_ENABLED"):
raise NotConfigured
We'd write this:
::
.. code-block:: python
#!python
from scrapy.core.exceptions import NotConfigured
class SomeMiddleware(object):
def __init__(self, crawler):
if not crawler.settings.getbool('SOMEMIDDLEWARE_ENABLED'):
raise NotConfigured
def __init__(self, crawler):
if not crawler.settings.getbool("SOMEMIDDLEWARE_ENABLED"):
raise NotConfigured
Running from command line
=========================

6
sep/sep-012.rst
View File

@ -83,10 +83,10 @@ example:
$ cat project/spiders/google.py
::
.. code-block:: python
class GooglecomSpider(BaseSpider):
name = 'google'
allowed_domains = ['google.com']
name = "google"
allowed_domains = ["google.com"]
.. note:: ``spider_allowed_domains`` becomes optional as only ``OffsiteMiddleware`` uses it.

170
sep/sep-014.rst
View File

@ -92,7 +92,7 @@ Usage Examples
Basic Crawling
--------------
::
.. code-block:: python
#!python
#
@ -101,20 +101,20 @@ Basic Crawling
class SampleSpider(CrawlSpider):
rules = [
# The dispatcher uses first-match policy
Rule(UrlRegexMatch(r'product\.html\?id=\d+'), 'parse_item', follow=False),
Rule(UrlRegexMatch(r"product\.html\?id=\d+"), "parse_item", follow=False),
# by default, if the first param is string is wrapped into UrlRegexMatch
Rule(r'.+', 'parse_page'),
]
Rule(r".+", "parse_page"),
]
request_extractors = [
# crawl all links looking for products and images
SgmlRequestExtractor(),
]
]
request_processors = [
# canonicalize all requests' urls
Canonicalize(),
]
]
def parse_item(self, response):
# parse and extract items from response
@ -127,7 +127,7 @@ Basic Crawling
Custom Processor and External Callback
--------------------------------------
::
.. code-block:: python
#!python
#
@ -137,30 +137,32 @@ Custom Processor and External Callback
# Custom Processor
def filter_today_links(requests):
# only crawl today links
today = datetime.datetime.today().strftime('%Y-%m-%d')
today = datetime.datetime.today().strftime("%Y-%m-%d")
return [r for r in requests if today in r.url]
# Callback defined out of spider
def my_external_callback(response):
# process item
# process item
pass
class SampleSpider(CrawlSpider):
rules = [
# The dispatcher uses first-match policy
Rule(UrlRegexMatch(r'/news/(.+)/'), my_external_callback),
]
Rule(UrlRegexMatch(r"/news/(.+)/"), my_external_callback),
]
request_extractors = [
RegexRequestExtractor(r'/sections/.+'),
RegexRequestExtractor(r'/news/.+'),
]
RegexRequestExtractor(r"/sections/.+"),
RegexRequestExtractor(r"/news/.+"),
]
request_processors = [
# canonicalize all requests' urls
Canonicalize(),
filter_today_links,
]
]
Implementation
==============
@ -199,7 +201,7 @@ Package Structure
Request/Response Matchers
-------------------------
::
.. code-block:: python
#!python
"""
@ -208,6 +210,7 @@ Request/Response Matchers
Perform evaluation to Request or Response attributes
"""
class BaseMatcher(object):
"""Base matcher. Returns True by default."""
@ -229,11 +232,11 @@ Request/Response Matchers
def matches_url(self, url):
"""Returns True if given url is equal to matcher's url"""
return self._url url
return self._url == url
def matches_request(self, request):
"""Returns True if Request's url matches initial url"""
return self.matches_url(request.url)
return self.matches_url(request.url)
def matches_response(self, response):
"""REturns True if Response's url matches initial url"""
@ -254,7 +257,7 @@ Request/Response Matchers
Request Extractor
-----------------
::
.. code-block:: python
#!python
#
@ -262,21 +265,21 @@ Request Extractor
# Extractors receive response and return list of Requests
#
class BaseSgmlRequestExtractor(FixedSGMLParser):
"""Base SGML Request Extractor"""
def __init__(self, tag='a', attr='href'):
def __init__(self, tag="a", attr="href"):
"""Initialize attributes"""
FixedSGMLParser.__init__(self)
self.scan_tag = tag if callable(tag) else lambda t: t tag
self.scan_attr = attr if callable(attr) else lambda a: a attr
self.scan_tag = tag if callable(tag) else lambda t: t = tag
self.scan_attr = attr if callable(attr) else lambda a: a = attr
self.current_request = None
def extract_requests(self, response):
"""Returns list of requests extracted from response"""
return self._extract_requests(response.body, response.url,
response.encoding)
return self._extract_requests(response.body, response.url, response.encoding)
def _extract_requests(self, response_text, response_url, response_encoding):
"""Extract requests with absolute urls"""
@ -303,20 +306,19 @@ Request Extractor
def _fix_link_text_encoding(self, encoding):
"""Convert link_text to unicode for each request"""
for req in self.requests:
req.meta.setdefault('link_text', '')
req.meta['link_text'] = str_to_unicode(req.meta['link_text'],
encoding)
req.meta.setdefault("link_text", "")
req.meta["link_text"] = str_to_unicode(req.meta["link_text"], encoding)
def reset(self):
"""Reset state"""
FixedSGMLParser.reset(self)
self.requests = []
self.base_url = None
def unknown_starttag(self, tag, attrs):
"""Process unknown start tag"""
if 'base' tag:
self.base_url = dict(attrs).get('href')
if "base" == tag:
self.base_url = dict(attrs).get("href")
if self.scan_tag(tag):
for attr, value in attrs:
@ -333,8 +335,8 @@ Request Extractor
def handle_data(self, data):
"""Process data"""
current = self.current_request
if current and not 'link_text' in current.meta:
current.meta['link_text'] = data.strip()
if current and not "link_text" in current.meta:
current.meta["link_text"] = data.strip()
class SgmlRequestExtractor(BaseSgmlRequestExtractor):
@ -343,8 +345,8 @@ Request Extractor
def __init__(self, tags=None, attrs=None):
"""Initialize with custom tag & attribute function checkers"""
# defaults
tags = tuple(tags) if tags else ('a', 'area')
attrs = tuple(attrs) if attrs else ('href', )
tags = tuple(tags) if tags else ("a", "area")
attrs = tuple(attrs) if attrs else ("href",)
tag_func = lambda x: x in tags
attr_func = lambda x: x in attrs
@ -362,25 +364,26 @@ Request Extractor
def extract_requests(self, response):
"""Restrict to XPath regions"""
hxs = HtmlXPathSelector(response)
fragments = (''.join(
html_frag for html_frag in hxs.select(xpath).extract()
) for xpath in self.restrict_xpaths)
html_slice = ''.join(html_frag for html_frag in fragments)
return self._extract_requests(html_slice, response.url,
response.encoding)
fragments = (
"".join(html_frag for html_frag in hxs.select(xpath).extract())
for xpath in self.restrict_xpaths
)
html_slice = "".join(html_frag for html_frag in fragments)
return self._extract_requests(html_slice, response.url, response.encoding)
Request Processor
-----------------
::
.. code-block:: python
#!python
#
# Request Processors
# Request Processors
# Processors receive list of requests and return list of requests
#
"""Request Processors"""
class Canonicalize(object):
"""Canonicalize Request Processor"""
@ -390,14 +393,14 @@ Request Processor
# replace in-place
req.url = canonicalize_url(req.url)
yield req
class Unique(object):
"""Filter duplicate Requests"""
def __init__(self, *attributes):
"""Initialize comparison attributes"""
self._attributes = attributes or ['url']
self._attributes = attributes or ["url"]
def _requests_equal(self, req1, req2):
"""Attribute comparison helper"""
@ -430,20 +433,24 @@ Request Processor
"""Filter request's domain"""
def __init__(self, allow=(), deny=()):
"""Initialize allow/deny attributes"""
self.allow = tuple(arg_to_iter(allow))
self.deny = tuple(arg_to_iter(deny))
"""Initialize allow/deny attributes"""
self.allow = tuple(arg_to_iter(allow))
self.deny = tuple(arg_to_iter(deny))
def __call__(self, requests):
"""Filter domains"""
processed = (req for req in requests)
if self.allow:
processed = (req for req in requests
if url_is_from_any_domain(req.url, self.allow))
processed = (
req for req in requests if url_is_from_any_domain(req.url, self.allow)
)
if self.deny:
processed = (req for req in requests
if not url_is_from_any_domain(req.url, self.deny))
processed = (
req
for req in requests
if not url_is_from_any_domain(req.url, self.deny)
)
return processed
@ -453,24 +460,28 @@ Request Processor
def __init__(self, allow=(), deny=()):
"""Initialize allow/deny attributes"""
_re_type = type(re.compile('', 0))
_re_type = type(re.compile("", 0))
self.allow_res = [x if isinstance(x, _re_type) else re.compile(x)
for x in arg_to_iter(allow)]
self.deny_res = [x if isinstance(x, _re_type) else re.compile(x)
for x in arg_to_iter(deny)]
self.allow_res = [
x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(allow)
]
self.deny_res = [
x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(deny)
]
def __call__(self, requests):
"""Filter request's url based on allow/deny rules"""
#TODO: filter valid urls here?
# TODO: filter valid urls here?
processed = (req for req in requests)
if self.allow_res:
processed = (req for req in requests
if self._matches(req.url, self.allow_res))
processed = (
req for req in requests if self._matches(req.url, self.allow_res)
)
if self.deny_res:
processed = (req for req in requests
if not self._matches(req.url, self.deny_res))
processed = (
req for req in requests if not self._matches(req.url, self.deny_res)
)
return processed
@ -481,7 +492,7 @@ Request Processor
Rule Object
-----------
::
.. code-block:: python
#!python
#
@ -490,8 +501,10 @@ Rule Object
#
class Rule(object):
"""Crawler Rule"""
def __init__(self, matcher, callback=None, cb_args=None,
cb_kwargs=None, follow=True):
def __init__(
self, matcher, callback=None, cb_args=None, cb_kwargs=None, follow=True
):
"""Store attributes"""
self.matcher = matcher
self.callback = callback
@ -499,12 +512,14 @@ Rule Object
self.cb_kwargs = cb_kwargs if cb_kwargs else {}
self.follow = follow
#
# Rules Manager takes list of Rule objects and normalize matcher and callback
# into CompiledRule
#
class CompiledRule(object):
"""Compiled version of Rule"""
def __init__(self, matcher, callback=None, follow=False):
"""Initialize attributes checking type"""
assert isinstance(matcher, BaseMatcher)
@ -518,15 +533,16 @@ Rule Object
Rules Manager
-------------
::
.. code-block:: python
#!python
#
# Handles rules matcher/callbacks
# Resolve rule for given response
#
#
class RulesManager(object):
"""Rules Manager"""
def __init__(self, rules, spider, default_matcher=UrlRegexMatcher):
"""Initialize rules using spider and default matcher"""
self._rules = tuple()
@ -542,8 +558,9 @@ Rules Manager
# instance default matcher
matcher = default_matcher(rule.matcher)
else:
raise ValueError('Not valid matcher given %r in %r' \
% (rule.matcher, rule))
raise ValueError(
"Not valid matcher given %r in %r" % (rule.matcher, rule)
)
# prepare callback
if callable(rule.callback):
@ -553,8 +570,9 @@ Rules Manager
callback = getattr(spider, rule.callback)
if not callable(callback):
raise AttributeError('Invalid callback %r can not be resolved' \
% callback)
raise AttributeError(
"Invalid callback %r can not be resolved" % callback
)
else:
callback = None
@ -564,7 +582,7 @@ Rules Manager
# append compiled rule to rules list
crule = CompiledRule(matcher, callback, follow=rule.follow)
self._rules += (crule, )
self._rules += (crule,)
def get_rule(self, response):
"""Returns first rule that matches response"""
@ -575,7 +593,7 @@ Rules Manager
Request Generator
-----------------
::
.. code-block:: python
#!python
#
@ -605,7 +623,7 @@ Request Generator
``CrawlSpider``
-----------------
::
.. code-block:: python
#!python
#
@ -625,9 +643,9 @@ Request Generator
# wrap rules
self._rulesman = RulesManager(self.rules, spider=self)
# generates new requests with given callback
self._reqgen = RequestGenerator(self.request_extractors,
self.request_processors,
self.parse)
self._reqgen = RequestGenerator(
self.request_extractors, self.request_processors, self.parse
)
def parse(self, response):
"""Dispatch callback and generate requests"""

49
sep/sep-016.rst
View File

@ -67,21 +67,21 @@ Regex (HTML) Link Extractor
A typical application of LegSpider's is to build Link Extractors. For example:
::
.. code-block:: python
#!python
class RegexHtmlLinkExtractor(LegSpider):
def process_response(self, response):
if isinstance(response, HtmlResponse):
allowed_regexes = self.spider.url_regexes_to_follow
# extract urls to follow using allowed_regexes
return [Request(x) for x in urls_to_follow]
class MySpider(LegSpider):
legs = [RegexHtmlLinkExtractor()]
url_regexes_to_follow = ['/product.php?.*']
url_regexes_to_follow = ["/product.php?.*"]
def parse_response(self, response):
# parse response and extract items
@ -92,13 +92,12 @@ RSS2 link extractor
This is a Leg Spider that can be used for following links from RSS2 feeds.
::
.. code-block:: python
#!python
class Rss2LinkExtractor(LegSpider):
def process_response(self, response):
if response.headers.get('Content-type') 'application/rss+xml':
if response.headers.get("Content-type") == "application/rss+xml":
xs = XmlXPathSelector(response)
urls = xs.select("//item/link/text()").extract()
return [Request(x) for x in urls]
@ -108,11 +107,10 @@ Callback dispatcher based on rules
Another example could be to build a callback dispatcher based on rules:
::
.. code-block:: python
#!python
class CallbackRules(LegSpider):
def __init__(self, *a, **kw):
super(CallbackRules, self).__init__(*a, **kw)
for regex, method_name in self.spider.callback_rules.items():
@ -128,12 +126,13 @@ Another example could be to build a callback dispatcher based on rules:
return method(response)
return []
class MySpider(LegSpider):
legs = [CallbackRules()]
callback_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
"/product.php.*": "parse_product",
"/category.php.*": "parse_category",
}
def parse_product(self, response):
@ -145,19 +144,19 @@ URL Canonicalizers
Another example could be for building URL canonicalizers:
::
.. code-block:: python
#!python
class CanonicalizeUrl(LegSpider):
def process_request(self, request):
curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules)
return request.replace(url=curl)
class MySpider(LegSpider):
legs = [CanonicalizeUrl()]
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
# ...
@ -167,22 +166,22 @@ Setting item identifier
Another example could be for setting a unique identifier to items, based on
certain fields:
::
.. code-block:: python
#!python
class ItemIdSetter(LegSpider):
def process_item(self, item):
id_field = self.spider.id_field
id_fields_to_hash = self.spider.id_fields_to_hash
item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
return item
class MySpider(LegSpider):
legs = [ItemIdSetter()]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
id_field = "guid"
id_fields_to_hash = ["supplier_name", "supplier_id"]
def process_response(self, item):
# extract item from response
@ -193,24 +192,24 @@ Combining multiple leg spiders
Here's an example that combines functionality from multiple leg spiders:
::
.. code-block:: python
#!python
class MySpider(LegSpider):
legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]
url_regexes_to_follow = ['/product.php?.*']
url_regexes_to_follow = ["/product.php?.*"]
parse_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
"/product.php.*": "parse_product",
"/category.php.*": "parse_category",
}
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
id_field = "guid"
id_fields_to_hash = ["supplier_name", "supplier_id"]
def process_product(self, item):
# extract item from response
@ -249,7 +248,7 @@ important to keep in mind their scope and limitations, such as:
Here's a proof-of-concept implementation of ``LegSpider``:
::
.. code-block:: python
#!python
from scrapy.http import Request

10
sep/sep-017.rst
View File

@ -35,16 +35,15 @@ gExample URL for simple callback
The ``parse_product`` callback must return items containing the fields given in
``@scrapes``.
::
.. code-block:: python
#!python
class ProductSpider(BaseSpider):
def parse_product(self, response):
"""
@url http://www.example.com/store/product.php?id=123
@scrapes name, price, description
""""
"""
gChained callbacks
------------------
@ -55,11 +54,10 @@ other for scraping user profile info.
The contracts assert that the first callback returns a Request and the second
one scrape ``user, name, email`` fields.
::
.. code-block:: python
#!python
class UserProfileSpider(BaseSpider):
def parse_login_page(self, response):
"""
@url http://www.example.com/login.php
@ -71,7 +69,7 @@ one scrape ``user, name, email`` fields.
"""
@after parse_login_page
@scrapes user, name, email
""""
"""
# ...
Tags reference

150
sep/sep-018.rst
View File

@ -166,27 +166,32 @@ written, it should work both globally and per spider.
Here's an example that combines functionality from multiple middlewares into
the same spider:
::
.. code-block:: python
#!python
class MySpider(BaseSpider):
middlewares = [RegexLinkExtractor(), CallbackRules(), CanonicalizeUrl(),
ItemIdSetter(), OffsiteMiddleware()]
middlewares = [
RegexLinkExtractor(),
CallbackRules(),
CanonicalizeUrl(),
ItemIdSetter(),
OffsiteMiddleware(),
]
allowed_domains = ['example.com', 'sub.example.com']
allowed_domains = ["example.com", "sub.example.com"]
url_regexes_to_follow = ['/product.php?.*']
url_regexes_to_follow = ["/product.php?.*"]
callback_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
"/product.php.*": "parse_product",
"/category.php.*": "parse_category",
}
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
id_field = "guid"
id_fields_to_hash = ["supplier_name", "supplier_id"]
def parse_product(self, item):
# extract item from response
@ -234,35 +239,34 @@ Regex (HTML) Link Extractor
A typical application of spider middlewares could be to build Link Extractors.
For example:
::
.. code-block:: python
#!python
class RegexHtmlLinkExtractor(object):
def process_response(self, response, request, spider):
if isinstance(response, HtmlResponse):
allowed_regexes = spider.url_regexes_to_follow
# extract urls to follow using allowed_regexes
return [Request(x) for x in urls_to_follow]
# Example spider using this middleware
class MySpider(BaseSpider):
middlewares = [RegexHtmlLinkExtractor()]
url_regexes_to_follow = ['/product.php?.*']
url_regexes_to_follow = ["/product.php?.*"]
# parsing callbacks below
RSS2 link extractor
-------------------
::
.. code-block:: python
#!python
class Rss2LinkExtractor(object):
def process_response(self, response, request, spider):
if response.headers.get('Content-type') 'application/rss+xml':
if response.headers.get("Content-type") == "application/rss+xml":
xs = XmlXPathSelector(response)
urls = xs.select("//item/link/text()").extract()
return [Request(x) for x in urls]
@ -272,11 +276,10 @@ Callback dispatcher based on rules
Another example could be to build a callback dispatcher based on rules:
::
.. code-block:: python
#!python
class CallbackRules(object):
def __init__(self):
self.rules = {}
dispatcher.connect(signals.spider_opened, self.spider_opened)
@ -300,13 +303,14 @@ Another example could be to build a callback dispatcher based on rules:
return method(response)
return []
# Example spider using this middleware
class MySpider(BaseSpider):
middlewares = [CallbackRules()]
callback_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
"/product.php.*": "parse_product",
"/category.php.*": "parse_category",
}
def parse_product(self, response):
@ -318,22 +322,20 @@ URL Canonicalizers
Another example could be for building URL canonicalizers:
::
.. code-block:: python
#!python
class CanonicalizeUrl(object):
def process_request(self, request, response, spider):
curl = canonicalize_url(request.url,
rules=spider.canonicalization_rules)
curl = canonicalize_url(request.url, rules=spider.canonicalization_rules)
return request.replace(url=curl)
# Example spider using this middleware
class MySpider(BaseSpider):
middlewares = [CanonicalizeUrl()]
canonicalization_rules = ['sort-query-args',
'normalize-percent-encoding', ...]
canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]
# ...
@ -343,23 +345,23 @@ Setting item identifier
Another example could be for setting a unique identifier to items, based on
certain fields:
::
.. code-block:: python
#!python
class ItemIdSetter(object):
def process_item(self, item, response, spider):
id_field = spider.id_field
id_fields_to_hash = spider.id_fields_to_hash
item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
return item
# Example spider using this middleware
class MySpider(BaseSpider):
middlewares = [ItemIdSetter()]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
id_field = "guid"
id_fields_to_hash = ["supplier_name", "supplier_id"]
def parse(self, response):
# extract item from response
@ -370,11 +372,10 @@ robots.txt exclusion
A spider middleware to avoid visiting pages forbidden by robots.txt:
::
.. code-block:: python
#!python
class SpiderInfo(object):
def __init__(self, useragent):
self.useragent = useragent
self.parsers = {}
@ -382,7 +383,6 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
class AllowAllParser(object):
def can_fetch(useragent, url):
return True
@ -397,7 +397,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
def process_request(self, request, response, spider):
return self.process_start_request(self, request)
return self.process_start_request(request)
def process_start_request(self, request, spider):
info = self.spiders[spider]
@ -415,17 +415,21 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
res = None
else:
robotsurl = "%s://%s/robots.txt" % (url.scheme, netloc)
meta = {'spider': spider, {'handle_httpstatus_list': [403, 404, 500]}
res = Request(robotsurl, callback=self.parse_robots,
meta=meta, priority=self.REQUEST_PRIORITY)
meta = {"spider": spider, "handle_httpstatus_list": [403, 404, 500]}
res = Request(
robotsurl,
callback=self.parse_robots,
meta=meta,
priority=self.REQUEST_PRIORITY,
)
info.pending[netloc].append(request)
return res
def parse_robots(self, response):
spider = response.request.meta['spider']
netloc urlparse_cached(response).netloc
spider = response.request.meta["spider"]
netloc = urlparse_cached(response).netloc
info = self.spiders[spider]
if response.status 200;
if response.status == 200:
rp = robotparser.RobotFileParser(response.url)
rp.parse(response.body.splitlines())
info.parsers[netloc] = rp
@ -434,7 +438,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:
return info.pending[netloc]
def spider_opened(self, spider):
ua = getattr(spider, 'user_agent', None) or settings['USER_AGENT']
ua = getattr(spider, "user_agent", None) or settings["USER_AGENT"]
self.spiders[spider] = SpiderInfo(ua)
def spider_closed(self, spider):
@ -445,18 +449,16 @@ Offsite middleware
This is a port of the Offsite middleware to the new spider middleware API:
::
.. code-block:: python
#!python
class SpiderInfo(object):
def __init__(self, host_regex):
self.host_regex = host_regex
self.hosts_seen = set()
class OffsiteMiddleware(object):
def __init__(self):
self.spiders = {}
dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
@ -472,19 +474,19 @@ This is a port of the Offsite middleware to the new spider middleware API:
info = self.spiders[spider]
host = urlparse_cached(x).hostname
if host and host not in info.hosts_seen:
spider.log("Filtered offsite request to %r: %s" % (host, request))
info.hosts_seen.add(host)
spider.log("Filtered offsite request to %r: %s" % (host, request))
info.hosts_seen.add(host)
def should_follow(self, request, spider):
info = self.spiders[spider]
# hostname can be None for wrong urls (like javascript links)
host = urlparse_cached(request).hostname or ''
host = urlparse_cached(request).hostname or ""
return bool(info.regex.search(host))
def get_host_regex(self, spider):
"""Override this method to implement a different offsite policy"""
domains = [d.replace('.', r'\.') for d in spider.allowed_domains]
regex = r'^(.*\.)?(%s)$' % '|'.join(domains)
domains = [d.replace(".", r"\.") for d in spider.allowed_domains]
regex = r"^(.*\.)?(%s)$" % "|".join(domains)
return re.compile(regex)
def spider_opened(self, spider):
@ -499,35 +501,36 @@ Limit URL length
A middleware to filter out requests with long urls:
::
.. code-block:: python
#!python
class LimitUrlLength(object):
class LimitUrlLength(object):
def __init__(self):
self.maxlength = settings.getint('URLLENGTH_LIMIT')
self.maxlength = settings.getint("URLLENGTH_LIMIT")
def process_request(self, request, response, spider):
return self.process_start_request(self, request)
def process_start_request(self, request, spider):
if len(request.url) <= self.maxlength:
if len(request.url) <= self.maxlength:
return request
spider.log("Ignoring request (url length > %d): %s " % (self.maxlength, request.url))
spider.log(
"Ignoring request (url length > %d): %s " % (self.maxlength, request.url)
)
Set Referer
-----------
A middleware to set the Referer:
::
.. code-block:: python
#!python
class SetReferer(object):
def process_request(self, request, response, spider):
request.headers.setdefault('Referer', response.url)
request.headers.setdefault("Referer", response.url)
return request
Set and limit crawling depth
@ -536,23 +539,22 @@ Set and limit crawling depth
A middleware to set (and limit) the request/response depth, taken from the
start requests:
::
.. code-block:: python
#!python
class SetLimitDepth(object):
def __init__(self, maxdepth=0):
self.maxdepth = maxdepth or settings.getint('DEPTH_LIMIT')
self.maxdepth = maxdepth or settings.getint("DEPTH_LIMIT")
def process_request(self, request, response, spider):
depth = response.request.meta['depth'] + 1
request.meta['depth'] = depth
depth = response.request.meta["depth"] + 1
request.meta["depth"] = depth
if not self.maxdepth or depth <= self.maxdepth:
return request
spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request)
spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request))
def process_start_request(self, request, spider):
request.meta['depth'] = 0
request.meta["depth"] = 0
return request
Filter duplicate requests
@ -560,17 +562,16 @@ Filter duplicate requests
A middleware to filter out requests already seen:
::
.. code-block:: python
#!python
class FilterDuplicates(object):
def __init__(self):
clspath = settings.get('DUPEFILTER_CLASS')
clspath = settings.get("DUPEFILTER_CLASS")
self.dupefilter = load_object(clspath)()
dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
def enqueue_request(self, spider, request):
seen = self.dupefilter.request_seen(spider, request)
if not seen or request.dont_filter:
@ -587,22 +588,25 @@ Scrape data using Parsley
A middleware to Scrape data using Parsley as described in UsingParsley
::
.. code-block:: python
#!python
from pyparsley import PyParsley
class ParsleyExtractor(object):
class ParsleyExtractor(object):
def __init__(self, parsley_json_code):
parsley = json.loads(parselet_json_code)
class ParsleyItem(Item):
def __init__(self, *a, **kw):
for name in parsley.keys():
self.fields[name] = Field()
super(ParsleyItem, self).__init__(*a, **kw)
self.item_class = ParsleyItem
self.parsley = PyParsley(parsley, output='python')
self.item_class = ParsleyItem
self.parsley = PyParsley(parsley, output="python")
def process_response(self, response, request, spider):
return self.item_class(self.parsley.parse(string=response.body))

10
sep/sep-019.rst
View File

@ -15,10 +15,11 @@ consistent way, while taking the chance to refactor the settings population
and whole crawl workflow.
In short, you will be able to overwrite settings (on a per-spider basis) by
implementing a class method in your spider::
implementing a class method in your spider:
.. code-block:: python
class MySpider(Spider):
@classmethod
def custom_settings(cls):
return {
@ -197,10 +198,11 @@ Spiders
A new class method ``custom_settings`` is proposed, that could be use to
override project and default settings before they're used to instantiate the
crawler::
crawler:
.. code-block:: python
class MySpider(Spider):
@classmethod
def custom_settings(cls):
return {

50
sep/sep-020.rst
View File

@ -54,18 +54,18 @@ required.
Before
------
::
.. code-block:: python
xpath = '//div[@class="geeks"]/dl/dt[contains(text(),"%s")]/following-sibling::dd[1]//text()'
gl = XPathItemLoader(response=response, item=dict())
gl.default_output_processor = Compose(TakeFirst(), lambda v: v.strip())
gl.add_xpath('hacker', xpath % 'hacker')
gl.add_xpath('nerd', xpath % 'nerd')
gl.add_xpath("hacker", xpath % "hacker")
gl.add_xpath("nerd", xpath % "nerd")
After
-----
::
.. code-block:: python
bil = BulkItemLoader(response=response)
bil.parse_dl('//div[@class="geeks"]/dl')
@ -75,33 +75,34 @@ Code Proposal
This is a working code sample that covers just the basics.
::
.. code-block:: python
from scrapy.contrib.loader import XPathItemLoader
from scrapy.contrib.loader.processor import MapCompose
class BulkItemLoader(XPathItemLoader):
""" Item loader based on specified pattern recognition
"""
"""Item loader based on specified pattern recognition"""
default_item_class = dict
base_xpath = '//body'
base_xpath = "//body"
ignore = ()
def _get_label(self, entity):
""" Pull the text label out of selected markup
"""Pull the text label out of selected markup
:param entity: Found markup
:type entity: Selector
"""
label = ' '.join(entity.xpath('.//text()').extract())
label = label.encode('ascii', 'xmlcharrefreplace') if label else ''
label = label.strip('&#160;') if '&#160;' in label else label
label = label.strip(':') if ':' in label else label
label = " ".join(entity.xpath(".//text()").extract())
label = label.encode("ascii", "xmlcharrefreplace") if label else ""
label = label.strip("&#160;") if "&#160;" in label else label
label = label.strip(":") if ":" in label else label
label = label.strip()
return label
def _get_entities(self, xpath):
""" Retrieve the list of selectors for a given sub-pattern
"""Retrieve the list of selectors for a given sub-pattern
:param xpath: The xpath to select
:type xpath: String
@ -110,20 +111,21 @@ This is a working code sample that covers just the basics.
"""
return self.selector.xpath(self.base_xpath + xpath)
def parse_dl(self, xpath=u'//dl'):
""" Look for the specified definition list pattern and store all found
def parse_dl(self, xpath="//dl"):
"""Look for the specified definition list pattern and store all found
values for the enclosed terms and descriptions.
:param xpath: The xpath to select
:type xpath: String
"""
for term in self._get_entities(xpath + '/dt'):
for term in self._get_entities(xpath + "/dt"):
label = self._get_label(term)
if label and label not in self.ignore:
value = term.xpath('following-sibling::dd[1]//text()')
value = term.xpath("following-sibling::dd[1]//text()")
if value:
self.add_value(label, value.extract(),
MapCompose(lambda v: v.strip()))
self.add_value(
label, value.extract(), MapCompose(lambda v: v.strip())
)
Example Spider
==============
@ -133,22 +135,24 @@ This spider uses the bulk loader above.
Spider code
-----------
::
.. code-block:: python
from scrapy.spider import BaseSpider
from scrapy.contrib.loader.bulk import BulkItemLoader
class W3cSpider(BaseSpider):
name = "w3c"
allowed_domains = ["w3.org"]
start_urls = ('http://www.w3.org/TR/html401/struct/lists.html',)
start_urls = ("http://www.w3.org/TR/html401/struct/lists.html",)
def parse(self, response):
el = BulkItemLoader(response=response)
el.parse_dl('//dl[2]')
el.parse_dl("//dl[2]")
item = el.load_item()
from pprint import pprint
pprint(item)
Log Output

20
sep/sep-021.rst
View File

@ -76,14 +76,18 @@ addon_configure
Receives the Settings object and modifies it to enable the required components.
If it raises an exception, Scrapy will print it and exit.
Examples::
Examples:
.. code-block:: python
def addon_configure(settings):
settings.overrides['DOWNLOADER_MIDDLEWARES'].update({
'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
})
settings.overrides["DOWNLOADER_MIDDLEWARES"].update(
{
"scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware": 900,
}
)
::
.. code-block:: python
def addon_configure(settings):
try:
@ -100,8 +104,10 @@ is meant to be used to perform post-initialization checks like making sure the
extension and its dependencies were configured properly. If it raises an
exception, Scrapy will print and exit.
Examples::
Examples:
.. code-block:: python
def crawler_ready(crawler):
if 'some.other.addon' not in crawler.extensions.enabled:
if "some.other.addon" not in crawler.extensions.enabled:
raise RuntimeError("Some other addon is required to use this addon")