mirror of https://github.com/scrapy/scrapy.git
Document all import paths and use the shortest in examples (#5099)
This commit is contained in:
Marlena Chatzigrigoriou
GitHub
parent
fcc6becc58
commit
d7deba7e89
|
|
@ -149,7 +149,7 @@ How can I prevent memory errors due to many allowed domains?
|
|||
------------------------------------------------------------
|
||||
|
||||
If you have a spider with a long list of
|
||||
:attr:`~scrapy.spiders.Spider.allowed_domains` (e.g. 50,000+), consider
|
||||
:attr:`~scrapy.Spider.allowed_domains` (e.g. 50,000+), consider
|
||||
replacing the default
|
||||
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` spider middleware
|
||||
with a :ref:`custom spider middleware <custom-spider-middleware>` that requires
|
||||
|
|
@ -157,7 +157,7 @@ less memory. For example:
|
|||
|
||||
- If your domain names are similar enough, use your own regular expression
|
||||
instead joining the strings in
|
||||
:attr:`~scrapy.spiders.Spider.allowed_domains` into a complex regular
|
||||
:attr:`~scrapy.Spider.allowed_domains` into a complex regular
|
||||
expression.
|
||||
|
||||
- If you can `meet the installation requirements`_, use pyre2_ instead of
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ Our first Spider
|
|||
|
||||
Spiders are classes that you define and that Scrapy uses to scrape information
|
||||
from a website (or a group of websites). They must subclass
|
||||
:class:`~scrapy.spiders.Spider` and define the initial requests to make,
|
||||
:class:`~scrapy.Spider` and define the initial requests to make,
|
||||
optionally how to follow links in the pages, and how to parse the downloaded
|
||||
page content to extract data.
|
||||
|
||||
|
|
@ -107,26 +107,26 @@ This is the code for our first Spider. Save it in a file named
|
|||
self.log(f'Saved file {filename}')
|
||||
|
||||
|
||||
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>`
|
||||
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`
|
||||
and defines some attributes and methods:
|
||||
|
||||
* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
|
||||
* :attr:`~scrapy.Spider.name`: identifies the Spider. It must be
|
||||
unique within a project, that is, you can't set the same name for different
|
||||
Spiders.
|
||||
|
||||
* :meth:`~scrapy.spiders.Spider.start_requests`: must return an iterable of
|
||||
* :meth:`~scrapy.Spider.start_requests`: must return an iterable of
|
||||
Requests (you can return a list of requests or write a generator function)
|
||||
which the Spider will begin to crawl from. Subsequent requests will be
|
||||
generated successively from these initial requests.
|
||||
|
||||
* :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
|
||||
* :meth:`~scrapy.Spider.parse`: a method that will be called to handle
|
||||
the response downloaded for each of the requests made. The response parameter
|
||||
is an instance of :class:`~scrapy.http.TextResponse` that holds
|
||||
the page content and has further helpful methods to handle it.
|
||||
|
||||
The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
|
||||
The :meth:`~scrapy.Spider.parse` method usually parses the response, extracting
|
||||
the scraped data as dicts and also finding new URLs to
|
||||
follow and creating new requests (:class:`~scrapy.http.Request`) from them.
|
||||
follow and creating new requests (:class:`~scrapy.Request`) from them.
|
||||
|
||||
How to run our spider
|
||||
---------------------
|
||||
|
|
@ -162,7 +162,7 @@ for the respective URLs, as our ``parse`` method instructs.
|
|||
What just happened under the hood?
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
|
||||
Scrapy schedules the :class:`scrapy.Request <scrapy.Request>` objects
|
||||
returned by the ``start_requests`` method of the Spider. Upon receiving a
|
||||
response for each one, it instantiates :class:`~scrapy.http.Response` objects
|
||||
and calls the callback method associated with the request (in this case, the
|
||||
|
|
@ -171,11 +171,11 @@ and calls the callback method associated with the request (in this case, the
|
|||
|
||||
A shortcut to the start_requests method
|
||||
---------------------------------------
|
||||
Instead of implementing a :meth:`~scrapy.spiders.Spider.start_requests` method
|
||||
that generates :class:`scrapy.Request <scrapy.http.Request>` objects from URLs,
|
||||
you can just define a :attr:`~scrapy.spiders.Spider.start_urls` class attribute
|
||||
Instead of implementing a :meth:`~scrapy.Spider.start_requests` method
|
||||
that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,
|
||||
you can just define a :attr:`~scrapy.Spider.start_urls` class attribute
|
||||
with a list of URLs. This list will then be used by the default implementation
|
||||
of :meth:`~scrapy.spiders.Spider.start_requests` to create the initial requests
|
||||
of :meth:`~scrapy.Spider.start_requests` to create the initial requests
|
||||
for your spider::
|
||||
|
||||
import scrapy
|
||||
|
|
@ -194,9 +194,9 @@ for your spider::
|
|||
with open(filename, 'wb') as f:
|
||||
f.write(response.body)
|
||||
|
||||
The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each
|
||||
The :meth:`~scrapy.Spider.parse` method will be called to handle each
|
||||
of the requests for those URLs, even though we haven't explicitly told Scrapy
|
||||
to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's
|
||||
to do so. This happens because :meth:`~scrapy.Spider.parse` is Scrapy's
|
||||
default callback method, which is called for requests without an explicitly
|
||||
assigned callback.
|
||||
|
||||
|
|
@ -248,7 +248,7 @@ object:
|
|||
|
||||
The result of running ``response.css('title')`` is a list-like object called
|
||||
:class:`~scrapy.selector.SelectorList`, which represents a list of
|
||||
:class:`~scrapy.selector.Selector` objects that wrap around XML/HTML elements
|
||||
:class:`~scrapy.Selector` objects that wrap around XML/HTML elements
|
||||
and allow you to run further queries to fine-grain the selection or extract the
|
||||
data.
|
||||
|
||||
|
|
@ -670,7 +670,7 @@ the pagination links with the ``parse`` callback as we saw before.
|
|||
Here we're passing callbacks to
|
||||
:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` as positional
|
||||
arguments to make the code shorter; it also works for
|
||||
:class:`~scrapy.http.Request`.
|
||||
:class:`~scrapy.Request`.
|
||||
|
||||
The ``parse_author`` callback defines a helper function to extract and cleanup the
|
||||
data from a CSS query and yields the Python dict with the author data.
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ how you :ref:`configure the downloader middlewares
|
|||
.. class:: Crawler(spidercls, settings)
|
||||
|
||||
The Crawler object must be instantiated with a
|
||||
:class:`scrapy.spiders.Spider` subclass and a
|
||||
:class:`scrapy.Spider` subclass and a
|
||||
:class:`scrapy.settings.Settings` object.
|
||||
|
||||
.. attribute:: settings
|
||||
|
|
@ -196,7 +196,7 @@ SpiderLoader API
|
|||
match the request's url against the domains of the spiders.
|
||||
|
||||
:param request: queried request
|
||||
:type request: :class:`~scrapy.http.Request` instance
|
||||
:type request: :class:`~scrapy.Request` instance
|
||||
|
||||
.. _topics-api-signals:
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ This callback is tested using three built-in contracts:
|
|||
|
||||
.. class:: CallbackKeywordArgumentsContract
|
||||
|
||||
This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.http.Request.cb_kwargs>`
|
||||
This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.Request.cb_kwargs>`
|
||||
attribute for the sample request. It must be a valid JSON dictionary.
|
||||
::
|
||||
|
||||
|
|
@ -88,7 +88,7 @@ override three methods:
|
|||
.. method:: Contract.adjust_request_args(args)
|
||||
|
||||
This receives a ``dict`` as an argument containing default arguments
|
||||
for request object. :class:`~scrapy.http.Request` is used by default,
|
||||
for request object. :class:`~scrapy.Request` is used by default,
|
||||
but this can be changed with the ``request_cls`` attribute.
|
||||
If multiple contracts in chain have this attribute defined, the last one is used.
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Supported callables
|
|||
The following callables may be defined as coroutines using ``async def``, and
|
||||
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||
|
||||
- :class:`~scrapy.http.Request` callbacks.
|
||||
- :class:`~scrapy.Request` callbacks.
|
||||
|
||||
.. note:: The callback output is not processed until the whole callback
|
||||
finishes.
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ Consider the following Scrapy spider below::
|
|||
|
||||
Basically this is a simple spider which parses two pages of items (the
|
||||
start_urls). Items also have a details page with additional information, so we
|
||||
use the ``cb_kwargs`` functionality of :class:`~scrapy.http.Request` to pass a
|
||||
use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a
|
||||
partially populated item.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -274,7 +274,7 @@ In more complex websites, it could be difficult to easily reproduce the
|
|||
requests, as we could need to add ``headers`` or ``cookies`` to make it work.
|
||||
In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
|
||||
format, by right-clicking on each of them in the network tool and using the
|
||||
:meth:`~scrapy.http.Request.from_curl()` method to generate an equivalent
|
||||
:meth:`~scrapy.Request.from_curl()` method to generate an equivalent
|
||||
request::
|
||||
|
||||
from scrapy import Request
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
middleware.
|
||||
|
||||
:meth:`process_request` should either: return ``None``, return a
|
||||
:class:`~scrapy.http.Response` object, return a :class:`~scrapy.http.Request`
|
||||
:class:`~scrapy.Response` object, return a :class:`~scrapy.http.Request`
|
||||
object, or raise :exc:`~scrapy.exceptions.IgnoreRequest`.
|
||||
|
||||
If it returns ``None``, Scrapy will continue processing this request, executing all
|
||||
|
|
@ -88,7 +88,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
or the appropriate download function; it'll return that response. The :meth:`process_response`
|
||||
methods of installed middleware is always called on every response.
|
||||
|
||||
If it returns a :class:`~scrapy.http.Request` object, Scrapy will stop calling
|
||||
If it returns a :class:`~scrapy.Request` object, Scrapy will stop calling
|
||||
process_request methods and reschedule the returned request. Once the newly returned
|
||||
request is performed, the appropriate middleware chain will be called on
|
||||
the downloaded response.
|
||||
|
|
@ -100,22 +100,22 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
ignored and not logged (unlike other exceptions).
|
||||
|
||||
:param request: the request being processed
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider for which this request is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_response(request, response, spider)
|
||||
|
||||
:meth:`process_response` should either: return a :class:`~scrapy.http.Response`
|
||||
object, return a :class:`~scrapy.http.Request` object or
|
||||
object, return a :class:`~scrapy.Request` object or
|
||||
raise a :exc:`~scrapy.exceptions.IgnoreRequest` exception.
|
||||
|
||||
If it returns a :class:`~scrapy.http.Response` (it could be the same given
|
||||
response, or a brand-new one), that response will continue to be processed
|
||||
with the :meth:`process_response` of the next middleware in the chain.
|
||||
|
||||
If it returns a :class:`~scrapy.http.Request` object, the middleware chain is
|
||||
If it returns a :class:`~scrapy.Request` object, the middleware chain is
|
||||
halted and the returned request is rescheduled to be downloaded in the future.
|
||||
This is the same behavior as if a request is returned from :meth:`process_request`.
|
||||
|
||||
|
|
@ -124,13 +124,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
exception, it is ignored and not logged (unlike other exceptions).
|
||||
|
||||
:param request: the request that originated the response
|
||||
:type request: is a :class:`~scrapy.http.Request` object
|
||||
:type request: is a :class:`~scrapy.Request` object
|
||||
|
||||
:param response: the response being processed
|
||||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param spider: the spider for which this response is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_exception(request, exception, spider)
|
||||
|
||||
|
|
@ -139,7 +139,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
exception (including an :exc:`~scrapy.exceptions.IgnoreRequest` exception)
|
||||
|
||||
:meth:`process_exception` should return: either ``None``,
|
||||
a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request` object.
|
||||
a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.Request` object.
|
||||
|
||||
If it returns ``None``, Scrapy will continue processing this exception,
|
||||
executing any other :meth:`process_exception` methods of installed middleware,
|
||||
|
|
@ -149,19 +149,19 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
method chain of installed middleware is started, and Scrapy won't bother calling
|
||||
any other :meth:`process_exception` methods of middleware.
|
||||
|
||||
If it returns a :class:`~scrapy.http.Request` object, the returned request is
|
||||
If it returns a :class:`~scrapy.Request` object, the returned request is
|
||||
rescheduled to be downloaded in the future. This stops the execution of
|
||||
:meth:`process_exception` methods of the middleware the same as returning a
|
||||
response would.
|
||||
|
||||
:param request: the request that generated the exception
|
||||
:type request: is a :class:`~scrapy.http.Request` object
|
||||
:type request: is a :class:`~scrapy.Request` object
|
||||
|
||||
:param exception: the raised exception
|
||||
:type exception: an ``Exception`` object
|
||||
|
||||
:param spider: the spider for which this request is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: from_crawler(cls, crawler)
|
||||
|
||||
|
|
@ -203,13 +203,13 @@ CookiesMiddleware
|
|||
browsers do.
|
||||
|
||||
.. caution:: When non-UTF8 encoded byte sequences are passed to a
|
||||
:class:`~scrapy.http.Request`, the ``CookiesMiddleware`` will log
|
||||
:class:`~scrapy.Request`, the ``CookiesMiddleware`` will log
|
||||
a warning. Refer to :ref:`topics-logging-advanced-customization`
|
||||
to customize the logging behaviour.
|
||||
|
||||
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
|
||||
:ref:`cookies-mw`. If you need to set cookies for a request, use the
|
||||
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
|
||||
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
|
||||
current limitation that is being worked on.
|
||||
|
||||
The following settings can be used to configure the cookie middleware:
|
||||
|
|
@ -258,7 +258,7 @@ web server and received cookies in :class:`~scrapy.http.Response` will
|
|||
**not** be merged with the existing cookies.
|
||||
|
||||
For more detailed information see the ``cookies`` parameter in
|
||||
:class:`~scrapy.http.Request`.
|
||||
:class:`~scrapy.Request`.
|
||||
|
||||
.. setting:: COOKIES_DEBUG
|
||||
|
||||
|
|
@ -501,7 +501,7 @@ defines the methods described below.
|
|||
the :signal:`open_spider <spider_opened>` signal.
|
||||
|
||||
:param spider: the spider which has been opened
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: close_spider(spider)
|
||||
|
||||
|
|
@ -509,27 +509,27 @@ defines the methods described below.
|
|||
the :signal:`close_spider <spider_closed>` signal.
|
||||
|
||||
:param spider: the spider which has been closed
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: retrieve_response(spider, request)
|
||||
|
||||
Return response if present in cache, or ``None`` otherwise.
|
||||
|
||||
:param spider: the spider which generated the request
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param request: the request to find cached response for
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
.. method:: store_response(spider, request, response)
|
||||
|
||||
Store the given response in the cache.
|
||||
|
||||
:param spider: the spider for which the response is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param request: the corresponding request the spider generated
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param response: the response to store in the cache
|
||||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
|
@ -722,7 +722,7 @@ HttpProxyMiddleware
|
|||
.. class:: HttpProxyMiddleware
|
||||
|
||||
This middleware sets the HTTP proxy to use for requests, by setting the
|
||||
``proxy`` meta value for :class:`~scrapy.http.Request` objects.
|
||||
``proxy`` meta value for :class:`~scrapy.Request` objects.
|
||||
|
||||
Like the Python standard library module :mod:`urllib.request`, it obeys
|
||||
the following environment variables:
|
||||
|
|
@ -749,12 +749,12 @@ RedirectMiddleware
|
|||
.. reqmeta:: redirect_urls
|
||||
|
||||
The urls which the request goes through (while being redirected) can be found
|
||||
in the ``redirect_urls`` :attr:`Request.meta <scrapy.http.Request.meta>` key.
|
||||
in the ``redirect_urls`` :attr:`Request.meta <scrapy.Request.meta>` key.
|
||||
|
||||
.. reqmeta:: redirect_reasons
|
||||
|
||||
The reason behind each redirect in :reqmeta:`redirect_urls` can be found in the
|
||||
``redirect_reasons`` :attr:`Request.meta <scrapy.http.Request.meta>` key. For
|
||||
``redirect_reasons`` :attr:`Request.meta <scrapy.Request.meta>` key. For
|
||||
example: ``[301, 302, 307, 'meta refresh']``.
|
||||
|
||||
The format of a reason depends on the middleware that handled the corresponding
|
||||
|
|
@ -770,7 +770,7 @@ settings (see the settings documentation for more info):
|
|||
|
||||
.. reqmeta:: dont_redirect
|
||||
|
||||
If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_redirect``
|
||||
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_redirect``
|
||||
key set to True, the request will be ignored by this middleware.
|
||||
|
||||
If you want to handle some redirect status codes in your spider, you can
|
||||
|
|
@ -783,7 +783,7 @@ responses (and pass them through to your spider) you can do this::
|
|||
handle_httpstatus_list = [301, 302]
|
||||
|
||||
The ``handle_httpstatus_list`` key of :attr:`Request.meta
|
||||
<scrapy.http.Request.meta>` can also be used to specify which response codes to
|
||||
<scrapy.Request.meta>` can also be used to specify which response codes to
|
||||
allow on a per-request basis. You can also set the meta key
|
||||
``handle_httpstatus_all`` to ``True`` if you want to allow any response code
|
||||
for a request.
|
||||
|
|
@ -889,7 +889,7 @@ settings (see the settings documentation for more info):
|
|||
|
||||
.. reqmeta:: dont_retry
|
||||
|
||||
If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_retry`` key
|
||||
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
|
||||
set to True, the request will be ignored by this middleware.
|
||||
|
||||
To retry requests from a spider callback, you can use the
|
||||
|
|
@ -919,7 +919,7 @@ Default: ``2``
|
|||
Maximum number of times to retry, in addition to the first download.
|
||||
|
||||
Maximum number of retries can also be specified per-request using
|
||||
:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.http.Request.meta>`.
|
||||
:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.Request.meta>`.
|
||||
When initialized, the :reqmeta:`max_retry_times` meta key takes higher
|
||||
precedence over the :setting:`RETRY_TIMES` setting.
|
||||
|
||||
|
|
@ -986,7 +986,7 @@ RobotsTxtMiddleware
|
|||
|
||||
.. reqmeta:: dont_obey_robotstxt
|
||||
|
||||
If :attr:`Request.meta <scrapy.http.Request.meta>` has
|
||||
If :attr:`Request.meta <scrapy.Request.meta>` has
|
||||
``dont_obey_robotstxt`` key set to True
|
||||
the request will be ignored by this middleware even if
|
||||
:setting:`ROBOTSTXT_OBEY` is enabled.
|
||||
|
|
|
|||
|
|
@ -62,9 +62,9 @@ download the webpage with an HTTP client like curl_ or wget_ and see if the
|
|||
information can be found in the response they get.
|
||||
|
||||
If they get a response with the desired data, modify your Scrapy
|
||||
:class:`~scrapy.http.Request` to match that of the other HTTP client. For
|
||||
:class:`~scrapy.Request` to match that of the other HTTP client. For
|
||||
example, try using the same user-agent string (:setting:`USER_AGENT`) or the
|
||||
same :attr:`~scrapy.http.Request.headers`.
|
||||
same :attr:`~scrapy.Request.headers`.
|
||||
|
||||
If they also get a response without the desired data, you’ll need to take
|
||||
steps to make your request more similar to that of the web browser. See
|
||||
|
|
@ -81,14 +81,14 @@ Use the :ref:`network tool <topics-network-tool>` of your web browser to see
|
|||
how your web browser performs the desired request, and try to reproduce that
|
||||
request with Scrapy.
|
||||
|
||||
It might be enough to yield a :class:`~scrapy.http.Request` with the same HTTP
|
||||
It might be enough to yield a :class:`~scrapy.Request` with the same HTTP
|
||||
method and URL. However, you may also need to reproduce the body, headers and
|
||||
form parameters (see :class:`~scrapy.http.FormRequest`) of that request.
|
||||
form parameters (see :class:`~scrapy.FormRequest`) of that request.
|
||||
|
||||
As all major browsers allow to export the requests in `cURL
|
||||
<https://curl.haxx.se/>`_ format, Scrapy incorporates the method
|
||||
:meth:`~scrapy.http.Request.from_curl()` to generate an equivalent
|
||||
:class:`~scrapy.http.Request` from a cURL command. To get more information
|
||||
:meth:`~scrapy.Request.from_curl()` to generate an equivalent
|
||||
:class:`~scrapy.Request` from a cURL command. To get more information
|
||||
visit :ref:`request from curl <requests-from-curl>` inside the network
|
||||
tool section.
|
||||
|
||||
|
|
@ -125,7 +125,7 @@ data from it depends on the type of response:
|
|||
|
||||
If the desired data is inside HTML or XML code embedded within JSON data,
|
||||
you can load that HTML or XML code into a
|
||||
:class:`~scrapy.selector.Selector` and then
|
||||
:class:`~scrapy.Selector` and then
|
||||
:ref:`use it <topics-selectors>` as usual::
|
||||
|
||||
selector = Selector(data['html'])
|
||||
|
|
|
|||
|
|
@ -90,7 +90,7 @@ described next.
|
|||
1. Declaring a serializer in the field
|
||||
--------------------------------------
|
||||
|
||||
If you use :class:`~.Item` you can declare a serializer in the
|
||||
If you use :class:`~scrapy.Item` you can declare a serializer in the
|
||||
:ref:`field metadata <topics-items-fields>`. The serializer must be
|
||||
a callable which receives a value and returns its serialized form.
|
||||
|
||||
|
|
@ -172,7 +172,7 @@ BaseItemExporter
|
|||
:param field: the field being serialized. If the source :ref:`item object
|
||||
<item-types>` does not define field metadata, *field* is an empty
|
||||
:class:`dict`.
|
||||
:type field: :class:`~scrapy.item.Field` object or a :class:`dict` instance
|
||||
:type field: :class:`~scrapy.Field` object or a :class:`dict` instance
|
||||
|
||||
:param name: the name of the field being serialized
|
||||
:type name: str
|
||||
|
|
|
|||
|
|
@ -674,9 +674,9 @@ The function signature should be as follows:
|
|||
:type params: dict
|
||||
|
||||
:param spider: source spider of the feed items
|
||||
:type spider: scrapy.spiders.Spider
|
||||
:type spider: scrapy.Spider
|
||||
|
||||
For example, to include the :attr:`name <scrapy.spiders.Spider.name>` of the
|
||||
For example, to include the :attr:`name <scrapy.Spider.name>` of the
|
||||
source spider in the feed URI:
|
||||
|
||||
#. Define the following function somewhere in your project::
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ Each item pipeline component is a Python class that must implement the following
|
|||
:type item: :ref:`item object <item-types>`
|
||||
|
||||
:param spider: the spider which scraped the item
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
Additionally, they may also implement the following methods:
|
||||
|
||||
|
|
@ -51,14 +51,14 @@ Additionally, they may also implement the following methods:
|
|||
This method is called when the spider is opened.
|
||||
|
||||
:param spider: the spider which was opened
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: close_spider(self, spider)
|
||||
|
||||
This method is called when the spider is closed.
|
||||
|
||||
:param spider: the spider which was closed
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: from_crawler(cls, crawler)
|
||||
|
||||
|
|
|
|||
|
|
@ -42,7 +42,8 @@ Item objects
|
|||
:class:`Item` provides a :class:`dict`-like API plus additional features that
|
||||
make it the most feature-complete item type:
|
||||
|
||||
.. class:: Item([arg])
|
||||
.. class:: scrapy.item.Item([arg])
|
||||
.. class:: scrapy.Item([arg])
|
||||
|
||||
:class:`Item` objects replicate the standard :class:`dict` API, including
|
||||
its ``__init__`` method.
|
||||
|
|
@ -199,7 +200,8 @@ It's important to note that the :class:`Field` objects used to declare the item
|
|||
do not stay assigned as class attributes. Instead, they can be accessed through
|
||||
the :attr:`Item.fields` attribute.
|
||||
|
||||
.. class:: Field([arg])
|
||||
.. class:: scrapy.item.Field([arg])
|
||||
.. class:: scrapy.Field([arg])
|
||||
|
||||
The :class:`Field` class is just an alias to the built-in :class:`dict` class and
|
||||
doesn't provide any extra functionality or attributes. In other words,
|
||||
|
|
@ -317,11 +319,11 @@ If that is not the desired behavior, use a deep copy instead.
|
|||
See :mod:`copy` for more information.
|
||||
|
||||
To create a shallow copy of an item, you can either call
|
||||
:meth:`~scrapy.item.Item.copy` on an existing item
|
||||
:meth:`~scrapy.Item.copy` on an existing item
|
||||
(``product2 = product.copy()``) or instantiate your item class from an existing
|
||||
item (``product2 = Product(product)``).
|
||||
|
||||
To create a deep copy, call :meth:`~scrapy.item.Item.deepcopy` instead
|
||||
To create a deep copy, call :meth:`~scrapy.Item.deepcopy` instead
|
||||
(``product2 = product.deepcopy()``).
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -74,10 +74,10 @@ on cookies.
|
|||
Request serialization
|
||||
---------------------
|
||||
|
||||
For persistence to work, :class:`~scrapy.http.Request` objects must be
|
||||
For persistence to work, :class:`~scrapy.Request` objects must be
|
||||
serializable with :mod:`pickle`, except for the ``callback`` and ``errback``
|
||||
values passed to their ``__init__`` method, which must be methods of the
|
||||
running :class:`~scrapy.spiders.Spider` class.
|
||||
running :class:`~scrapy.Spider` class.
|
||||
|
||||
If you wish to log the requests that couldn't be serialized, you can set the
|
||||
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ Common causes of memory leaks
|
|||
|
||||
It happens quite often (sometimes by accident, sometimes on purpose) that the
|
||||
Scrapy developer passes objects referenced in Requests (for example, using the
|
||||
:attr:`~scrapy.http.Request.cb_kwargs` or :attr:`~scrapy.http.Request.meta`
|
||||
:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta`
|
||||
attributes or the request callback function) and that effectively bounds the
|
||||
lifetime of those referenced objects to the lifetime of the Request. This is,
|
||||
by far, the most common cause of memory leaks in Scrapy projects, and a quite
|
||||
|
|
@ -48,9 +48,9 @@ Too Many Requests?
|
|||
------------------
|
||||
|
||||
By default Scrapy keeps the request queue in memory; it includes
|
||||
:class:`~scrapy.http.Request` objects and all objects
|
||||
referenced in Request attributes (e.g. in :attr:`~scrapy.http.Request.cb_kwargs`
|
||||
and :attr:`~scrapy.http.Request.meta`).
|
||||
:class:`~scrapy.Request` objects and all objects
|
||||
referenced in Request attributes (e.g. in :attr:`~scrapy.Request.cb_kwargs`
|
||||
and :attr:`~scrapy.Request.meta`).
|
||||
While not necessarily a leak, this can take a lot of memory. Enabling
|
||||
:ref:`persistent job queue <topics-jobs>` could help keeping memory usage
|
||||
in control.
|
||||
|
|
@ -90,11 +90,11 @@ Which objects are tracked?
|
|||
The objects tracked by ``trackrefs`` are all from these classes (and all its
|
||||
subclasses):
|
||||
|
||||
* :class:`scrapy.http.Request`
|
||||
* :class:`scrapy.Request`
|
||||
* :class:`scrapy.http.Response`
|
||||
* :class:`scrapy.item.Item`
|
||||
* :class:`scrapy.selector.Selector`
|
||||
* :class:`scrapy.spiders.Spider`
|
||||
* :class:`scrapy.Item`
|
||||
* :class:`scrapy.Selector`
|
||||
* :class:`scrapy.Spider`
|
||||
|
||||
A real example
|
||||
--------------
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ path::
|
|||
Logging from Spiders
|
||||
====================
|
||||
|
||||
Scrapy provides a :data:`~scrapy.spiders.Spider.logger` within each Spider
|
||||
Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider
|
||||
instance, which can be accessed and used like this::
|
||||
|
||||
import scrapy
|
||||
|
|
|
|||
|
|
@ -259,7 +259,7 @@ respectively), the pipeline will put the results under the respective field
|
|||
When using :ref:`item types <item-types>` for which fields are defined beforehand,
|
||||
you must define both the URLs field and the results field. For example, when
|
||||
using the images pipeline, items must define both the ``image_urls`` and the
|
||||
``images`` field. For instance, using the :class:`~scrapy.item.Item` class::
|
||||
``images`` field. For instance, using the :class:`~scrapy.Item` class::
|
||||
|
||||
import scrapy
|
||||
|
||||
|
|
@ -424,7 +424,7 @@ See here the methods that you can override in your custom Files Pipeline:
|
|||
In addition to ``response``, this method receives the original
|
||||
:class:`request <scrapy.Request>`,
|
||||
:class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and
|
||||
:class:`item <scrapy.item.Item>`
|
||||
:class:`item <scrapy.Item>`
|
||||
|
||||
You can override this method to customize the download path of each file.
|
||||
|
||||
|
|
@ -563,7 +563,7 @@ See here the methods that you can override in your custom Images Pipeline:
|
|||
In addition to ``response``, this method receives the original
|
||||
:class:`request <scrapy.Request>`,
|
||||
:class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and
|
||||
:class:`item <scrapy.item.Item>`
|
||||
:class:`item <scrapy.Item>`
|
||||
|
||||
You can override this method to customize the download path of each file.
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ Request objects
|
|||
request (once it's downloaded) as its first parameter. For more information
|
||||
see :ref:`topics-request-response-ref-request-callback-arguments` below.
|
||||
If a Request doesn't specify a callback, the spider's
|
||||
:meth:`~scrapy.spiders.Spider.parse` method will be used.
|
||||
:meth:`~scrapy.Spider.parse` method will be used.
|
||||
Note that if exceptions are raised during processing, errback is called instead.
|
||||
|
||||
:type callback: collections.abc.Callable
|
||||
|
|
@ -60,7 +60,7 @@ Request objects
|
|||
|
||||
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
|
||||
:ref:`cookies-mw`. If you need to set cookies for a request, use the
|
||||
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
|
||||
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
|
||||
current limitation that is being worked on.
|
||||
|
||||
:type headers: dict
|
||||
|
|
@ -92,7 +92,7 @@ Request objects
|
|||
|
||||
To create a request that does not send stored cookies and does not
|
||||
store received cookies, set the ``dont_merge_cookies`` key to ``True``
|
||||
in :attr:`request.meta <scrapy.http.Request.meta>`.
|
||||
in :attr:`request.meta <scrapy.Request.meta>`.
|
||||
|
||||
Example of a request that sends manually-defined cookies and ignores
|
||||
cookie storage::
|
||||
|
|
@ -107,7 +107,7 @@ Request objects
|
|||
|
||||
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
|
||||
:ref:`cookies-mw`. If you need to set cookies for a request, use the
|
||||
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
|
||||
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
|
||||
current limitation that is being worked on.
|
||||
|
||||
:type cookies: dict or list
|
||||
|
|
@ -495,7 +495,9 @@ fields with form data from :class:`Response` objects.
|
|||
|
||||
.. _lxml.html forms: https://lxml.de/lxmlhtml.html#forms
|
||||
|
||||
.. class:: FormRequest(url, [formdata, ...])
|
||||
.. class:: scrapy.http.request.form.FormRequest
|
||||
.. class:: scrapy.http.FormRequest
|
||||
.. class:: scrapy.FormRequest(url, [formdata, ...])
|
||||
|
||||
The :class:`FormRequest` class adds a new keyword parameter to the ``__init__`` method. The
|
||||
remaining arguments are the same as for the :class:`Request` class and are
|
||||
|
|
@ -694,7 +696,7 @@ Response objects
|
|||
|
||||
:param request: the initial value of the :attr:`Response.request` attribute.
|
||||
This represents the :class:`Request` that generated this response.
|
||||
:type request: scrapy.http.Request
|
||||
:type request: scrapy.Request
|
||||
|
||||
:param certificate: an object representing the server's SSL certificate.
|
||||
:type certificate: twisted.internet.ssl.Certificate
|
||||
|
|
@ -920,7 +922,7 @@ TextResponse objects
|
|||
|
||||
.. attribute:: TextResponse.selector
|
||||
|
||||
A :class:`~scrapy.selector.Selector` instance using the response as
|
||||
A :class:`~scrapy.Selector` instance using the response as
|
||||
target. The selector is lazily instantiated on first access.
|
||||
|
||||
:class:`TextResponse` objects support the following methods in addition to
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ Constructing selectors
|
|||
|
||||
.. highlight:: python
|
||||
|
||||
Response objects expose a :class:`~scrapy.selector.Selector` instance
|
||||
Response objects expose a :class:`~scrapy.Selector` instance
|
||||
on ``.selector`` attribute:
|
||||
|
||||
>>> response.selector.xpath('//span/text()').get()
|
||||
|
|
@ -62,7 +62,7 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
|
|||
>>> response.css('span::text').get()
|
||||
'good'
|
||||
|
||||
Scrapy selectors are instances of :class:`~scrapy.selector.Selector` class
|
||||
Scrapy selectors are instances of :class:`~scrapy.Selector` class
|
||||
constructed by passing either :class:`~scrapy.http.TextResponse` object or
|
||||
markup as a string (in ``text`` argument).
|
||||
|
||||
|
|
@ -175,7 +175,7 @@ of ``None``:
|
|||
'not-found'
|
||||
|
||||
Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes
|
||||
using ``.attrib`` property of a :class:`~scrapy.selector.Selector`:
|
||||
using ``.attrib`` property of a :class:`~scrapy.Selector`:
|
||||
|
||||
>>> [img.attrib['src'] for img in response.css('img')]
|
||||
['image1_thumb.jpg',
|
||||
|
|
@ -383,7 +383,7 @@ ID, or when selecting an unique element on a page):
|
|||
Using selectors with regular expressions
|
||||
----------------------------------------
|
||||
|
||||
:class:`~scrapy.selector.Selector` also has a ``.re()`` method for extracting
|
||||
:class:`~scrapy.Selector` also has a ``.re()`` method for extracting
|
||||
data using regular expressions. However, unlike using ``.xpath()`` or
|
||||
``.css()`` methods, ``.re()`` returns a list of strings. So you
|
||||
can't construct nested ``.re()`` calls.
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ Example::
|
|||
|
||||
Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
|
||||
own settings that will take precedence and override the project ones. They can
|
||||
do so by setting their :attr:`~scrapy.spiders.Spider.custom_settings` attribute::
|
||||
do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute::
|
||||
|
||||
class MySpider(scrapy.Spider):
|
||||
name = 'myspider'
|
||||
|
|
@ -142,7 +142,7 @@ In a spider, the settings are available through ``self.settings``::
|
|||
The ``settings`` attribute is set in the base Spider class after the spider
|
||||
is initialized. If you want to use the settings before the initialization
|
||||
(e.g., in your spider's ``__init__()`` method), you'll need to override the
|
||||
:meth:`~scrapy.spiders.Spider.from_crawler` method.
|
||||
:meth:`~scrapy.Spider.from_crawler` method.
|
||||
|
||||
Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
|
||||
attribute of the Crawler that is passed to ``from_crawler`` method in
|
||||
|
|
@ -338,7 +338,7 @@ is non-zero, download delay is enforced per IP, not per domain.
|
|||
DEFAULT_ITEM_CLASS
|
||||
------------------
|
||||
|
||||
Default: ``'scrapy.item.Item'``
|
||||
Default: ``'scrapy.Item'``
|
||||
|
||||
The default class that will be used for instantiating items in the :ref:`the
|
||||
Scrapy shell <topics-shell>`.
|
||||
|
|
@ -360,7 +360,7 @@ The default headers used for Scrapy HTTP Requests. They're populated in the
|
|||
|
||||
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
|
||||
:ref:`cookies-mw`. If you need to set cookies for a request, use the
|
||||
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
|
||||
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
|
||||
current limitation that is being worked on.
|
||||
|
||||
.. setting:: DEPTH_LIMIT
|
||||
|
|
@ -384,8 +384,8 @@ Default: ``0``
|
|||
|
||||
Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``
|
||||
|
||||
An integer that is used to adjust the :attr:`~scrapy.http.Request.priority` of
|
||||
a :class:`~scrapy.http.Request` based on its depth.
|
||||
An integer that is used to adjust the :attr:`~scrapy.Request.priority` of
|
||||
a :class:`~scrapy.Request` based on its depth.
|
||||
|
||||
The priority of a request is adjusted as follows::
|
||||
|
||||
|
|
@ -816,14 +816,14 @@ The default (``RFPDupeFilter``) filters based on request fingerprint using
|
|||
the ``scrapy.utils.request.request_fingerprint`` function. In order to change
|
||||
the way duplicates are checked you could subclass ``RFPDupeFilter`` and
|
||||
override its ``request_fingerprint`` method. This method should accept
|
||||
scrapy :class:`~scrapy.http.Request` object and return its fingerprint
|
||||
scrapy :class:`~scrapy.Request` object and return its fingerprint
|
||||
(a string).
|
||||
|
||||
You can disable filtering of duplicate requests by setting
|
||||
:setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``.
|
||||
Be very careful about this however, because you can get into crawling loops.
|
||||
It's usually a better idea to set the ``dont_filter`` parameter to
|
||||
``True`` on the specific :class:`~scrapy.http.Request` that should not be
|
||||
``True`` on the specific :class:`~scrapy.Request` that should not be
|
||||
filtered.
|
||||
|
||||
.. setting:: DUPEFILTER_DEBUG
|
||||
|
|
|
|||
|
|
@ -118,7 +118,7 @@ Available Scrapy objects
|
|||
|
||||
The Scrapy shell automatically creates some convenient objects from the
|
||||
downloaded page, like the :class:`~scrapy.http.Response` object and the
|
||||
:class:`~scrapy.selector.Selector` objects (for both HTML and XML
|
||||
:class:`~scrapy.Selector` objects (for both HTML and XML
|
||||
content).
|
||||
|
||||
Those objects are:
|
||||
|
|
@ -126,12 +126,12 @@ Those objects are:
|
|||
- ``crawler`` - the current :class:`~scrapy.crawler.Crawler` object.
|
||||
|
||||
- ``spider`` - the Spider which is known to handle the URL, or a
|
||||
:class:`~scrapy.spiders.Spider` object if there is no spider found for the
|
||||
:class:`~scrapy.Spider` object if there is no spider found for the
|
||||
current URL
|
||||
|
||||
- ``request`` - a :class:`~scrapy.http.Request` object of the last fetched
|
||||
- ``request`` - a :class:`~scrapy.Request` object of the last fetched
|
||||
page. You can modify this request using
|
||||
:meth:`~scrapy.http.Request.replace` or fetch a new request (without
|
||||
:meth:`~scrapy.Request.replace` or fetch a new request (without
|
||||
leaving the shell) using the ``fetch`` shortcut.
|
||||
|
||||
- ``response`` - a :class:`~scrapy.http.Response` object containing the last
|
||||
|
|
|
|||
|
|
@ -155,7 +155,7 @@ item_scraped
|
|||
:type item: :ref:`item object <item-types>`
|
||||
|
||||
:param spider: the spider which scraped the item
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param response: the response from where the item was scraped
|
||||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
|
@ -175,7 +175,7 @@ item_dropped
|
|||
:type item: :ref:`item object <item-types>`
|
||||
|
||||
:param spider: the spider which scraped the item
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param response: the response from where the item was dropped
|
||||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
|
@ -203,7 +203,7 @@ item_error
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param spider: the spider which raised the exception
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param failure: the exception raised
|
||||
:type failure: twisted.python.failure.Failure
|
||||
|
|
@ -223,7 +223,7 @@ spider_closed
|
|||
This signal supports returning deferreds from its handlers.
|
||||
|
||||
:param spider: the spider which has been closed
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param reason: a string which describes the reason why the spider was closed. If
|
||||
it was closed because the spider has completed scraping, the reason
|
||||
|
|
@ -247,7 +247,7 @@ spider_opened
|
|||
This signal supports returning deferreds from its handlers.
|
||||
|
||||
:param spider: the spider which has been opened
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
spider_idle
|
||||
~~~~~~~~~~~
|
||||
|
|
@ -278,7 +278,7 @@ spider_idle
|
|||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
:param spider: the spider which has gone idle
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
|
||||
**not** guarantee that it can prevent the spider from being closed,
|
||||
|
|
@ -303,7 +303,7 @@ spider_error
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param spider: the spider which raised the exception
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
Request signals
|
||||
---------------
|
||||
|
|
@ -314,16 +314,16 @@ request_scheduled
|
|||
.. signal:: request_scheduled
|
||||
.. function:: request_scheduled(request, spider)
|
||||
|
||||
Sent when the engine schedules a :class:`~scrapy.http.Request`, to be
|
||||
Sent when the engine schedules a :class:`~scrapy.Request`, to be
|
||||
downloaded later.
|
||||
|
||||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
:param request: the request that reached the scheduler
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider that yielded the request
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
request_dropped
|
||||
~~~~~~~~~~~~~~~
|
||||
|
|
@ -331,16 +331,16 @@ request_dropped
|
|||
.. signal:: request_dropped
|
||||
.. function:: request_dropped(request, spider)
|
||||
|
||||
Sent when a :class:`~scrapy.http.Request`, scheduled by the engine to be
|
||||
Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
|
||||
downloaded later, is rejected by the scheduler.
|
||||
|
||||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
:param request: the request that reached the scheduler
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider that yielded the request
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
request_reached_downloader
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -348,15 +348,15 @@ request_reached_downloader
|
|||
.. signal:: request_reached_downloader
|
||||
.. function:: request_reached_downloader(request, spider)
|
||||
|
||||
Sent when a :class:`~scrapy.http.Request` reached downloader.
|
||||
Sent when a :class:`~scrapy.Request` reached downloader.
|
||||
|
||||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
:param request: the request that reached downloader
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider that yielded the request
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
request_left_downloader
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -366,16 +366,16 @@ request_left_downloader
|
|||
|
||||
.. versionadded:: 2.0
|
||||
|
||||
Sent when a :class:`~scrapy.http.Request` leaves the downloader, even in case of
|
||||
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
|
||||
failure.
|
||||
|
||||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
:param request: the request that reached the downloader
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider that yielded the request
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
bytes_received
|
||||
~~~~~~~~~~~~~~
|
||||
|
|
@ -402,10 +402,10 @@ bytes_received
|
|||
:type data: :class:`bytes` object
|
||||
|
||||
:param request: the request that generated the download
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider associated with the response
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
headers_received
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
|
@ -432,10 +432,10 @@ headers_received
|
|||
:type body_length: `int`
|
||||
|
||||
:param request: the request that generated the download
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider associated with the response
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
Response signals
|
||||
----------------
|
||||
|
|
@ -455,10 +455,10 @@ response_received
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param request: the request that generated the response
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider for which the response is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. note:: The ``request`` argument might not contain the original request that
|
||||
reached the downloader, if a :ref:`topics-downloader-middleware` modifies
|
||||
|
|
@ -479,7 +479,7 @@ response_downloaded
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param request: the request that generated the response
|
||||
:type request: :class:`~scrapy.http.Request` object
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider for which the response is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param spider: the spider for which this response is intended
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
|
||||
.. method:: process_spider_output(response, result, spider)
|
||||
|
|
@ -102,7 +102,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
it has processed the response.
|
||||
|
||||
:meth:`process_spider_output` must return an iterable of
|
||||
:class:`~scrapy.http.Request` objects and :ref:`item object
|
||||
:class:`~scrapy.Request` objects and :ref:`item object
|
||||
<topics-items>`.
|
||||
|
||||
:param response: the response which generated this output from the
|
||||
|
|
@ -110,11 +110,11 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
:type response: :class:`~scrapy.http.Response` object
|
||||
|
||||
:param result: the result returned by the spider
|
||||
:type result: an iterable of :class:`~scrapy.http.Request` objects and
|
||||
:type result: an iterable of :class:`~scrapy.Request` objects and
|
||||
:ref:`item object <topics-items>`
|
||||
|
||||
:param spider: the spider whose result is being processed
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_spider_exception(response, exception, spider)
|
||||
|
||||
|
|
@ -122,7 +122,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
method (from a previous spider middleware) raises an exception.
|
||||
|
||||
:meth:`process_spider_exception` should return either ``None`` or an
|
||||
iterable of :class:`~scrapy.http.Request` objects and :ref:`item object
|
||||
iterable of :class:`~scrapy.Request` objects and :ref:`item object
|
||||
<topics-items>`.
|
||||
|
||||
If it returns ``None``, Scrapy will continue processing this exception,
|
||||
|
|
@ -142,7 +142,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
:type exception: :exc:`Exception` object
|
||||
|
||||
:param spider: the spider which raised the exception
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_start_requests(start_requests, spider)
|
||||
|
||||
|
|
@ -152,7 +152,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
items).
|
||||
|
||||
It receives an iterable (in the ``start_requests`` parameter) and must
|
||||
return another iterable of :class:`~scrapy.http.Request` objects.
|
||||
return another iterable of :class:`~scrapy.Request` objects.
|
||||
|
||||
.. note:: When implementing this method in your spider middleware, you
|
||||
should always return an iterable (that follows the input one) and
|
||||
|
|
@ -164,10 +164,10 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
(like a time limit or item/page count).
|
||||
|
||||
:param start_requests: the start requests
|
||||
:type start_requests: an iterable of :class:`~scrapy.http.Request`
|
||||
:type start_requests: an iterable of :class:`~scrapy.Request`
|
||||
|
||||
:param spider: the spider to whom the start requests belong
|
||||
:type spider: :class:`~scrapy.spiders.Spider` object
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: from_crawler(cls, crawler)
|
||||
|
||||
|
|
@ -251,7 +251,7 @@ this::
|
|||
.. reqmeta:: handle_httpstatus_all
|
||||
|
||||
The ``handle_httpstatus_list`` key of :attr:`Request.meta
|
||||
<scrapy.http.Request.meta>` can also be used to specify which response codes to
|
||||
<scrapy.Request.meta>` can also be used to specify which response codes to
|
||||
allow on a per-request basis. You can also set the meta key ``handle_httpstatus_all``
|
||||
to ``True`` if you want to allow any response code for a request, and ``False`` to
|
||||
disable the effects of the ``handle_httpstatus_all`` key.
|
||||
|
|
@ -295,7 +295,7 @@ OffsiteMiddleware
|
|||
Filters out Requests for URLs outside the domains covered by the spider.
|
||||
|
||||
This middleware filters out every request whose host names aren't in the
|
||||
spider's :attr:`~scrapy.spiders.Spider.allowed_domains` attribute.
|
||||
spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
|
||||
All subdomains of any domain in the list are also allowed.
|
||||
E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
|
||||
but not ``www2.example.com`` nor ``example.com``.
|
||||
|
|
@ -313,10 +313,10 @@ OffsiteMiddleware
|
|||
will be printed (but only for the first request filtered).
|
||||
|
||||
If the spider doesn't define an
|
||||
:attr:`~scrapy.spiders.Spider.allowed_domains` attribute, or the
|
||||
:attr:`~scrapy.Spider.allowed_domains` attribute, or the
|
||||
attribute is empty, the offsite middleware will allow all requests.
|
||||
|
||||
If the request has the :attr:`~scrapy.http.Request.dont_filter` attribute
|
||||
If the request has the :attr:`~scrapy.Request.dont_filter` attribute
|
||||
set, the offsite middleware will allow the request even if its domain is not
|
||||
listed in allowed domains.
|
||||
|
||||
|
|
|
|||
|
|
@ -17,15 +17,15 @@ For spiders, the scraping cycle goes through something like this:
|
|||
those requests.
|
||||
|
||||
The first requests to perform are obtained by calling the
|
||||
:meth:`~scrapy.spiders.Spider.start_requests` method which (by default)
|
||||
generates :class:`~scrapy.http.Request` for the URLs specified in the
|
||||
:attr:`~scrapy.spiders.Spider.start_urls` and the
|
||||
:attr:`~scrapy.spiders.Spider.parse` method as callback function for the
|
||||
:meth:`~scrapy.Spider.start_requests` method which (by default)
|
||||
generates :class:`~scrapy.Request` for the URLs specified in the
|
||||
:attr:`~scrapy.Spider.start_urls` and the
|
||||
:attr:`~scrapy.Spider.parse` method as callback function for the
|
||||
Requests.
|
||||
|
||||
2. In the callback function, you parse the response (web page) and return
|
||||
:ref:`item objects <topics-items>`,
|
||||
:class:`~scrapy.http.Request` objects, or an iterable of these objects.
|
||||
:class:`~scrapy.Request` objects, or an iterable of these objects.
|
||||
Those Requests will also contain a callback (maybe
|
||||
the same) and will then be downloaded by Scrapy and then their
|
||||
response handled by the specified callback.
|
||||
|
|
@ -50,7 +50,8 @@ We will talk about those types here.
|
|||
scrapy.Spider
|
||||
=============
|
||||
|
||||
.. class:: Spider()
|
||||
.. class:: scrapy.spiders.Spider()
|
||||
.. class:: scrapy.Spider()
|
||||
|
||||
This is the simplest spider, and the one from which every other spider
|
||||
must inherit (including spiders that come bundled with Scrapy, as well as spiders
|
||||
|
|
@ -86,7 +87,7 @@ scrapy.Spider
|
|||
|
||||
A list of URLs where the spider will begin to crawl from, when no
|
||||
particular URLs are specified. So, the first pages downloaded will be those
|
||||
listed here. The subsequent :class:`~scrapy.http.Request` will be generated successively from data
|
||||
listed here. The subsequent :class:`~scrapy.Request` will be generated successively from data
|
||||
contained in the start URLs.
|
||||
|
||||
.. attribute:: custom_settings
|
||||
|
|
@ -179,7 +180,7 @@ scrapy.Spider
|
|||
the same requirements as the :class:`Spider` class.
|
||||
|
||||
This method, as well as any other Request callback, must return an
|
||||
iterable of :class:`~scrapy.http.Request` and/or :ref:`item objects
|
||||
iterable of :class:`~scrapy.Request` and/or :ref:`item objects
|
||||
<topics-items>`.
|
||||
|
||||
:param response: the response to parse
|
||||
|
|
@ -234,7 +235,7 @@ Return multiple Requests and items from a single callback::
|
|||
yield scrapy.Request(response.urljoin(href), self.parse)
|
||||
|
||||
Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
|
||||
to give data more structure you can use :class:`~scrapy.item.Item` objects::
|
||||
to give data more structure you can use :class:`~scrapy.Item` objects::
|
||||
|
||||
import scrapy
|
||||
from myproject.items import MyItem
|
||||
|
|
@ -373,7 +374,7 @@ CrawlSpider
|
|||
This method is called for each response produced for the URLs in
|
||||
the spider's ``start_urls`` attribute. It allows to parse
|
||||
the initial responses and must return either an
|
||||
:ref:`item object <topics-items>`, a :class:`~scrapy.http.Request`
|
||||
:ref:`item object <topics-items>`, a :class:`~scrapy.Request`
|
||||
object, or an iterable containing any of them.
|
||||
|
||||
Crawling rules
|
||||
|
|
@ -383,7 +384,7 @@ Crawling rules
|
|||
|
||||
``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
|
||||
defines how links will be extracted from each crawled page. Each produced link will
|
||||
be used to generate a :class:`~scrapy.http.Request` object, which will contain the
|
||||
be used to generate a :class:`~scrapy.Request` object, which will contain the
|
||||
link's text in its ``meta`` dictionary (under the ``link_text`` key).
|
||||
If omitted, a default link extractor created with no arguments will be used,
|
||||
resulting in all links being extracted.
|
||||
|
|
@ -392,9 +393,9 @@ Crawling rules
|
|||
object with that name will be used) to be called for each link extracted with
|
||||
the specified link extractor. This callback receives a :class:`~scrapy.http.Response`
|
||||
as its first argument and must return either a single instance or an iterable of
|
||||
:ref:`item objects <topics-items>` and/or :class:`~scrapy.http.Request` objects
|
||||
:ref:`item objects <topics-items>` and/or :class:`~scrapy.Request` objects
|
||||
(or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`
|
||||
object will contain the text of the link that produced the :class:`~scrapy.http.Request`
|
||||
object will contain the text of the link that produced the :class:`~scrapy.Request`
|
||||
in its ``meta`` dictionary (under the ``link_text`` key)
|
||||
|
||||
``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
|
||||
|
|
@ -411,7 +412,7 @@ Crawling rules
|
|||
|
||||
``process_request`` is a callable (or a string, in which case a method from
|
||||
the spider object with that name will be used) which will be called for every
|
||||
:class:`~scrapy.http.Request` extracted by this rule. This callable should
|
||||
:class:`~scrapy.Request` extracted by this rule. This callable should
|
||||
take said request as first argument and the :class:`~scrapy.http.Response`
|
||||
from which the request originated as second argument. It must return a
|
||||
``Request`` object or ``None`` (to filter out the request).
|
||||
|
|
@ -470,7 +471,7 @@ Let's now take a look at an example CrawlSpider with rules::
|
|||
This spider would start crawling example.com's home page, collecting category
|
||||
links, and item links, parsing the latter with the ``parse_item`` method. For
|
||||
each item response, some data will be extracted from the HTML using XPath, and
|
||||
an :class:`~scrapy.item.Item` will be filled with it.
|
||||
an :class:`~scrapy.Item` will be filled with it.
|
||||
|
||||
XMLFeedSpider
|
||||
-------------
|
||||
|
|
@ -493,11 +494,11 @@ XMLFeedSpider
|
|||
|
||||
- ``'iternodes'`` - a fast iterator based on regular expressions
|
||||
|
||||
- ``'html'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
|
||||
- ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
|
||||
Keep in mind this uses DOM parsing and must load all DOM in memory
|
||||
which could be a problem for big feeds
|
||||
|
||||
- ``'xml'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
|
||||
- ``'xml'`` - an iterator which uses :class:`~scrapy.Selector`.
|
||||
Keep in mind this uses DOM parsing and must load all DOM in memory
|
||||
which could be a problem for big feeds
|
||||
|
||||
|
|
@ -515,7 +516,7 @@ XMLFeedSpider
|
|||
available in that document that will be processed with this spider. The
|
||||
``prefix`` and ``uri`` will be used to automatically register
|
||||
namespaces using the
|
||||
:meth:`~scrapy.selector.Selector.register_namespace` method.
|
||||
:meth:`~scrapy.Selector.register_namespace` method.
|
||||
|
||||
You can then specify nodes with namespaces in the :attr:`itertag`
|
||||
attribute.
|
||||
|
|
@ -542,10 +543,10 @@ XMLFeedSpider
|
|||
|
||||
This method is called for the nodes matching the provided tag name
|
||||
(``itertag``). Receives the response and an
|
||||
:class:`~scrapy.selector.Selector` for each node. Overriding this
|
||||
:class:`~scrapy.Selector` for each node. Overriding this
|
||||
method is mandatory. Otherwise, you spider won't work. This method
|
||||
must return an :ref:`item object <topics-items>`, a
|
||||
:class:`~scrapy.http.Request` object, or an iterable containing any of
|
||||
:class:`~scrapy.Request` object, or an iterable containing any of
|
||||
them.
|
||||
|
||||
.. method:: process_results(response, results)
|
||||
|
|
@ -587,7 +588,7 @@ These spiders are pretty easy to use, let's have a look at one example::
|
|||
|
||||
Basically what we did up there was to create a spider that downloads a feed from
|
||||
the given ``start_urls``, and then iterates through each of its ``item`` tags,
|
||||
prints them out, and stores some random data in an :class:`~scrapy.item.Item`.
|
||||
prints them out, and stores some random data in an :class:`~scrapy.Item`.
|
||||
|
||||
CSVFeedSpider
|
||||
-------------
|
||||
|
|
|
|||
Loading…
Reference in New Issue