RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased

This commit is contained in:
Adrián Chaves 2021-07-16 14:32:15 +02:00 committed by GitHub
parent de69d967f9 ee2df97bbd
commit 8a2768561f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
46 changed files with 667 additions and 279 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.github/workflows/checks.yml vendored
View File

@ -5,6 +5,7 @@ jobs:
checks:
runs-on: ubuntu-18.04
strategy:
fail-fast: false
matrix:
include:
- python-version: 3.9
@ -18,6 +19,7 @@ jobs:
- python-version: 3.8
env:
TOXENV: pylint
TOX_PIP_VERSION: 20.3.3
- python-version: 3.9
env:
TOXENV: typing
@ -36,5 +38,8 @@ jobs:
- name: Run check
env: ${{ matrix.env }}
run: |
if [[ ! -z "$TOX_PIP_VERSION" ]]; then
pip install tox-pip-version
fi
pip install -U tox
tox

1
.github/workflows/tests-macos.yml vendored
View File

@ -5,6 +5,7 @@ jobs:
tests:
runs-on: macos-10.15
strategy:
fail-fast: false
matrix:
python-version: [3.6, 3.7, 3.8, 3.9]

8
.github/workflows/tests-ubuntu.yml vendored
View File

@ -5,6 +5,7 @@ jobs:
tests:
runs-on: ubuntu-18.04
strategy:
fail-fast: false
matrix:
include:
- python-version: 3.7
@ -39,6 +40,7 @@ jobs:
- python-version: 3.8
env:
TOXENV: extra-deps
TOX_PIP_VERSION: 20.3.3
- python-version: 3.9
env:
TOXENV: asyncio
@ -55,7 +57,8 @@ jobs:
if: matrix.python-version == 'pypy3' || contains(matrix.env.TOXENV, 'pinned')
run: |
sudo apt-get update
sudo apt-get install libxml2-dev libxslt-dev
# libxml2 2.9.12 from ondrej/php PPA breaks lxml so we pin it to the bionic-updates repo version
sudo apt-get install libxml2-dev/bionic-updates libxslt-dev
- name: Run tests
env: ${{ matrix.env }}
@ -67,6 +70,9 @@ jobs:
$PYPY_VERSION/bin/pypy3 -m venv "$HOME/virtualenvs/$PYPY_VERSION"
source "$HOME/virtualenvs/$PYPY_VERSION/bin/activate"
fi
if [[ ! -z "$TOX_PIP_VERSION" ]]; then
pip install tox-pip-version
fi
pip install -U tox
tox

1
.github/workflows/tests-windows.yml vendored
View File

@ -5,6 +5,7 @@ jobs:
tests:
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
include:
- python-version: 3.6

3
CODE_OF_CONDUCT.md
View File

@ -72,3 +72,6 @@ available at [http://contributor-covenant.org/version/1/4][version].
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

4
docs/faq.rst
View File

@ -149,7 +149,7 @@ How can I prevent memory errors due to many allowed domains?
------------------------------------------------------------
If you have a spider with a long list of
:attr:`~scrapy.spiders.Spider.allowed_domains` (e.g. 50,000+), consider
:attr:`~scrapy.Spider.allowed_domains` (e.g. 50,000+), consider
replacing the default
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` spider middleware
with a :ref:`custom spider middleware <custom-spider-middleware>` that requires
@ -157,7 +157,7 @@ less memory. For example:
- If your domain names are similar enough, use your own regular expression
instead joining the strings in
:attr:`~scrapy.spiders.Spider.allowed_domains` into a complex regular
:attr:`~scrapy.Spider.allowed_domains` into a complex regular
expression.
- If you can `meet the installation requirements`_, use pyre2_ instead of

32
docs/intro/tutorial.rst
View File

@ -78,7 +78,7 @@ Our first Spider
Spiders are classes that you define and that Scrapy uses to scrape information
from a website (or a group of websites). They must subclass
:class:`~scrapy.spiders.Spider` and define the initial requests to make,
:class:`~scrapy.Spider` and define the initial requests to make,
optionally how to follow links in the pages, and how to parse the downloaded
page content to extract data.
@ -107,26 +107,26 @@ This is the code for our first Spider. Save it in a file named
self.log(f'Saved file {filename}')
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>`
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`
and defines some attributes and methods:
* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
* :attr:`~scrapy.Spider.name`: identifies the Spider. It must be
unique within a project, that is, you can't set the same name for different
Spiders.
* :meth:`~scrapy.spiders.Spider.start_requests`: must return an iterable of
* :meth:`~scrapy.Spider.start_requests`: must return an iterable of
Requests (you can return a list of requests or write a generator function)
which the Spider will begin to crawl from. Subsequent requests will be
generated successively from these initial requests.
* :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
* :meth:`~scrapy.Spider.parse`: a method that will be called to handle
the response downloaded for each of the requests made. The response parameter
is an instance of :class:`~scrapy.http.TextResponse` that holds
the page content and has further helpful methods to handle it.
The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
The :meth:`~scrapy.Spider.parse` method usually parses the response, extracting
the scraped data as dicts and also finding new URLs to
follow and creating new requests (:class:`~scrapy.http.Request`) from them.
follow and creating new requests (:class:`~scrapy.Request`) from them.
How to run our spider
---------------------
@ -162,7 +162,7 @@ for the respective URLs, as our ``parse`` method instructs.
What just happened under the hood?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
Scrapy schedules the :class:`scrapy.Request <scrapy.Request>` objects
returned by the ``start_requests`` method of the Spider. Upon receiving a
response for each one, it instantiates :class:`~scrapy.http.Response` objects
and calls the callback method associated with the request (in this case, the
@ -171,11 +171,11 @@ and calls the callback method associated with the request (in this case, the
A shortcut to the start_requests method
---------------------------------------
Instead of implementing a :meth:`~scrapy.spiders.Spider.start_requests` method
that generates :class:`scrapy.Request <scrapy.http.Request>` objects from URLs,
you can just define a :attr:`~scrapy.spiders.Spider.start_urls` class attribute
Instead of implementing a :meth:`~scrapy.Spider.start_requests` method
that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,
you can just define a :attr:`~scrapy.Spider.start_urls` class attribute
with a list of URLs. This list will then be used by the default implementation
of :meth:`~scrapy.spiders.Spider.start_requests` to create the initial requests
of :meth:`~scrapy.Spider.start_requests` to create the initial requests
for your spider::
import scrapy
@ -194,9 +194,9 @@ for your spider::
with open(filename, 'wb') as f:
f.write(response.body)
The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each
The :meth:`~scrapy.Spider.parse` method will be called to handle each
of the requests for those URLs, even though we haven't explicitly told Scrapy
to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's
to do so. This happens because :meth:`~scrapy.Spider.parse` is Scrapy's
default callback method, which is called for requests without an explicitly
assigned callback.
@ -248,7 +248,7 @@ object:
The result of running ``response.css('title')`` is a list-like object called
:class:`~scrapy.selector.SelectorList`, which represents a list of
:class:`~scrapy.selector.Selector` objects that wrap around XML/HTML elements
:class:`~scrapy.Selector` objects that wrap around XML/HTML elements
and allow you to run further queries to fine-grain the selection or extract the
data.
@ -670,7 +670,7 @@ the pagination links with the ``parse`` callback as we saw before.
Here we're passing callbacks to
:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` as positional
arguments to make the code shorter; it also works for
:class:`~scrapy.http.Request`.
:class:`~scrapy.Request`.
The ``parse_author`` callback defines a helper function to extract and cleanup the
data from a CSS query and yields the Python dict with the author data.

4
docs/topics/api.rst
View File

@ -29,7 +29,7 @@ how you :ref:`configure the downloader middlewares
.. class:: Crawler(spidercls, settings)
The Crawler object must be instantiated with a
:class:`scrapy.spiders.Spider` subclass and a
:class:`scrapy.Spider` subclass and a
:class:`scrapy.settings.Settings` object.
.. attribute:: settings
@ -196,7 +196,7 @@ SpiderLoader API
match the request's url against the domains of the spiders.
:param request: queried request
:type request: :class:`~scrapy.http.Request` instance
:type request: :class:`~scrapy.Request` instance
.. _topics-api-signals:

4
docs/topics/contracts.rst
View File

@ -37,7 +37,7 @@ This callback is tested using three built-in contracts:
.. class:: CallbackKeywordArgumentsContract
This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.http.Request.cb_kwargs>`
This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.Request.cb_kwargs>`
attribute for the sample request. It must be a valid JSON dictionary.
::
@ -88,7 +88,7 @@ override three methods:
.. method:: Contract.adjust_request_args(args)
This receives a ``dict`` as an argument containing default arguments
for request object. :class:`~scrapy.http.Request` is used by default,
for request object. :class:`~scrapy.Request` is used by default,
but this can be changed with the ``request_cls`` attribute.
If multiple contracts in chain have this attribute defined, the last one is used.

2
docs/topics/coroutines.rst
View File

@ -15,7 +15,7 @@ Supported callables
The following callables may be defined as coroutines using ``async def``, and
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.http.Request` callbacks.
- :class:`~scrapy.Request` callbacks.
.. versionchanged:: VERSION
Output of async callbacks is now processed asynchronously instead of collecting

2
docs/topics/debug.rst
View File

@ -36,7 +36,7 @@ Consider the following Scrapy spider below::
Basically this is a simple spider which parses two pages of items (the
start_urls). Items also have a details page with additional information, so we
use the ``cb_kwargs`` functionality of :class:`~scrapy.http.Request` to pass a
use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a
partially populated item.

2
docs/topics/developer-tools.rst
View File

@ -274,7 +274,7 @@ In more complex websites, it could be difficult to easily reproduce the
requests, as we could need to add ``headers`` or ``cookies`` to make it work.
In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
format, by right-clicking on each of them in the network tool and using the
:meth:`~scrapy.http.Request.from_curl()` method to generate an equivalent
:meth:`~scrapy.Request.from_curl()` method to generate an equivalent
request::
from scrapy import Request

58
docs/topics/downloader-middleware.rst
View File

@ -76,7 +76,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
middleware.
:meth:`process_request` should either: return ``None``, return a
:class:`~scrapy.http.Response` object, return a :class:`~scrapy.http.Request`
:class:`~scrapy.Response` object, return a :class:`~scrapy.http.Request`
object, or raise :exc:`~scrapy.exceptions.IgnoreRequest`.
If it returns ``None``, Scrapy will continue processing this request, executing all
@ -88,7 +88,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
or the appropriate download function; it'll return that response. The :meth:`process_response`
methods of installed middleware is always called on every response.
If it returns a :class:`~scrapy.http.Request` object, Scrapy will stop calling
If it returns a :class:`~scrapy.Request` object, Scrapy will stop calling
process_request methods and reschedule the returned request. Once the newly returned
request is performed, the appropriate middleware chain will be called on
the downloaded response.
@ -100,22 +100,22 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
ignored and not logged (unlike other exceptions).
:param request: the request being processed
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider for which this request is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: process_response(request, response, spider)
:meth:`process_response` should either: return a :class:`~scrapy.http.Response`
object, return a :class:`~scrapy.http.Request` object or
object, return a :class:`~scrapy.Request` object or
raise a :exc:`~scrapy.exceptions.IgnoreRequest` exception.
If it returns a :class:`~scrapy.http.Response` (it could be the same given
response, or a brand-new one), that response will continue to be processed
with the :meth:`process_response` of the next middleware in the chain.
If it returns a :class:`~scrapy.http.Request` object, the middleware chain is
If it returns a :class:`~scrapy.Request` object, the middleware chain is
halted and the returned request is rescheduled to be downloaded in the future.
This is the same behavior as if a request is returned from :meth:`process_request`.
@ -124,13 +124,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
exception, it is ignored and not logged (unlike other exceptions).
:param request: the request that originated the response
:type request: is a :class:`~scrapy.http.Request` object
:type request: is a :class:`~scrapy.Request` object
:param response: the response being processed
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider for which this response is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: process_exception(request, exception, spider)
@ -139,7 +139,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
exception (including an :exc:`~scrapy.exceptions.IgnoreRequest` exception)
:meth:`process_exception` should return: either ``None``,
a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request` object.
a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.Request` object.
If it returns ``None``, Scrapy will continue processing this exception,
executing any other :meth:`process_exception` methods of installed middleware,
@ -149,19 +149,19 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
method chain of installed middleware is started, and Scrapy won't bother calling
any other :meth:`process_exception` methods of middleware.
If it returns a :class:`~scrapy.http.Request` object, the returned request is
If it returns a :class:`~scrapy.Request` object, the returned request is
rescheduled to be downloaded in the future. This stops the execution of
:meth:`process_exception` methods of the middleware the same as returning a
response would.
:param request: the request that generated the exception
:type request: is a :class:`~scrapy.http.Request` object
:type request: is a :class:`~scrapy.Request` object
:param exception: the raised exception
:type exception: an ``Exception`` object
:param spider: the spider for which this request is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: from_crawler(cls, crawler)
@ -203,13 +203,13 @@ CookiesMiddleware
browsers do.
.. caution:: When non-UTF8 encoded byte sequences are passed to a
:class:`~scrapy.http.Request`, the ``CookiesMiddleware`` will log
:class:`~scrapy.Request`, the ``CookiesMiddleware`` will log
a warning. Refer to :ref:`topics-logging-advanced-customization`
to customize the logging behaviour.
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
:ref:`cookies-mw`. If you need to set cookies for a request, use the
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
current limitation that is being worked on.
The following settings can be used to configure the cookie middleware:
@ -258,7 +258,7 @@ web server and received cookies in :class:`~scrapy.http.Response` will
**not** be merged with the existing cookies.
For more detailed information see the ``cookies`` parameter in
:class:`~scrapy.http.Request`.
:class:`~scrapy.Request`.
.. setting:: COOKIES_DEBUG
@ -501,7 +501,7 @@ defines the methods described below.
the :signal:`open_spider <spider_opened>` signal.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: close_spider(spider)
@ -509,27 +509,27 @@ defines the methods described below.
the :signal:`close_spider <spider_closed>` signal.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: retrieve_response(spider, request)
Return response if present in cache, or ``None`` otherwise.
:param spider: the spider which generated the request
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param request: the request to find cached response for
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
.. method:: store_response(spider, request, response)
Store the given response in the cache.
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param request: the corresponding request the spider generated
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param response: the response to store in the cache
:type response: :class:`~scrapy.http.Response` object
@ -722,7 +722,7 @@ HttpProxyMiddleware
.. class:: HttpProxyMiddleware
This middleware sets the HTTP proxy to use for requests, by setting the
``proxy`` meta value for :class:`~scrapy.http.Request` objects.
``proxy`` meta value for :class:`~scrapy.Request` objects.
Like the Python standard library module :mod:`urllib.request`, it obeys
the following environment variables:
@ -749,12 +749,12 @@ RedirectMiddleware
.. reqmeta:: redirect_urls
The urls which the request goes through (while being redirected) can be found
in the ``redirect_urls`` :attr:`Request.meta <scrapy.http.Request.meta>` key.
in the ``redirect_urls`` :attr:`Request.meta <scrapy.Request.meta>` key.
.. reqmeta:: redirect_reasons
The reason behind each redirect in :reqmeta:`redirect_urls` can be found in the
``redirect_reasons`` :attr:`Request.meta <scrapy.http.Request.meta>` key. For
``redirect_reasons`` :attr:`Request.meta <scrapy.Request.meta>` key. For
example: ``[301, 302, 307, 'meta refresh']``.
The format of a reason depends on the middleware that handled the corresponding
@ -770,7 +770,7 @@ settings (see the settings documentation for more info):
.. reqmeta:: dont_redirect
If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_redirect``
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_redirect``
key set to True, the request will be ignored by this middleware.
If you want to handle some redirect status codes in your spider, you can
@ -783,7 +783,7 @@ responses (and pass them through to your spider) you can do this::
handle_httpstatus_list = [301, 302]
The ``handle_httpstatus_list`` key of :attr:`Request.meta
<scrapy.http.Request.meta>` can also be used to specify which response codes to
<scrapy.Request.meta>` can also be used to specify which response codes to
allow on a per-request basis. You can also set the meta key
``handle_httpstatus_all`` to ``True`` if you want to allow any response code
for a request.
@ -889,7 +889,7 @@ settings (see the settings documentation for more info):
.. reqmeta:: dont_retry
If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_retry`` key
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
set to True, the request will be ignored by this middleware.
To retry requests from a spider callback, you can use the
@ -919,7 +919,7 @@ Default: ``2``
Maximum number of times to retry, in addition to the first download.
Maximum number of retries can also be specified per-request using
:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.http.Request.meta>`.
:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.Request.meta>`.
When initialized, the :reqmeta:`max_retry_times` meta key takes higher
precedence over the :setting:`RETRY_TIMES` setting.
@ -986,7 +986,7 @@ RobotsTxtMiddleware
.. reqmeta:: dont_obey_robotstxt
If :attr:`Request.meta <scrapy.http.Request.meta>` has
If :attr:`Request.meta <scrapy.Request.meta>` has
``dont_obey_robotstxt`` key set to True
the request will be ignored by this middleware even if
:setting:`ROBOTSTXT_OBEY` is enabled.

14
docs/topics/dynamic-content.rst
View File

@ -62,9 +62,9 @@ download the webpage with an HTTP client like curl_ or wget_ and see if the
information can be found in the response they get.
If they get a response with the desired data, modify your Scrapy
:class:`~scrapy.http.Request` to match that of the other HTTP client. For
:class:`~scrapy.Request` to match that of the other HTTP client. For
example, try using the same user-agent string (:setting:`USER_AGENT`) or the
same :attr:`~scrapy.http.Request.headers`.
same :attr:`~scrapy.Request.headers`.
If they also get a response without the desired data, you’ll need to take
steps to make your request more similar to that of the web browser. See
@ -81,14 +81,14 @@ Use the :ref:`network tool <topics-network-tool>` of your web browser to see
how your web browser performs the desired request, and try to reproduce that
request with Scrapy.
It might be enough to yield a :class:`~scrapy.http.Request` with the same HTTP
It might be enough to yield a :class:`~scrapy.Request` with the same HTTP
method and URL. However, you may also need to reproduce the body, headers and
form parameters (see :class:`~scrapy.http.FormRequest`) of that request.
form parameters (see :class:`~scrapy.FormRequest`) of that request.
As all major browsers allow to export the requests in `cURL
<https://curl.haxx.se/>`_ format, Scrapy incorporates the method
:meth:`~scrapy.http.Request.from_curl()` to generate an equivalent
:class:`~scrapy.http.Request` from a cURL command. To get more information
:meth:`~scrapy.Request.from_curl()` to generate an equivalent
:class:`~scrapy.Request` from a cURL command. To get more information
visit :ref:`request from curl <requests-from-curl>` inside the network
tool section.
@ -125,7 +125,7 @@ data from it depends on the type of response:
If the desired data is inside HTML or XML code embedded within JSON data,
you can load that HTML or XML code into a
:class:`~scrapy.selector.Selector` and then
:class:`~scrapy.Selector` and then
:ref:`use it <topics-selectors>` as usual::
selector = Selector(data['html'])

4
docs/topics/exporters.rst
View File

@ -90,7 +90,7 @@ described next.
1. Declaring a serializer in the field
--------------------------------------
If you use :class:`~.Item` you can declare a serializer in the
If you use :class:`~scrapy.Item` you can declare a serializer in the
:ref:`field metadata <topics-items-fields>`. The serializer must be
a callable which receives a value and returns its serialized form.
@ -172,7 +172,7 @@ BaseItemExporter
:param field: the field being serialized. If the source :ref:`item object
<item-types>` does not define field metadata, *field* is an empty
:class:`dict`.
:type field: :class:`~scrapy.item.Field` object or a :class:`dict` instance
:type field: :class:`~scrapy.Field` object or a :class:`dict` instance
:param name: the name of the field being serialized
:type name: str

61
docs/topics/feed-exports.rst
View File

@ -201,9 +201,10 @@ passed through the following settings:
- :setting:`AWS_ACCESS_KEY_ID`
- :setting:`AWS_SECRET_ACCESS_KEY`
You can also define a custom ACL for exported feeds using this setting:
You can also define a custom ACL and custom endpoint for exported feeds using this setting:
- :setting:`FEED_STORAGE_S3_ACL`
- :setting:`AWS_ENDPOINT_URL`
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
@ -268,6 +269,45 @@ in multiple files, with the specified maximum item count per file. That way, as
soon as a file reaches the maximum item count, that file is delivered to the
feed URI, allowing item delivery to start way before the end of the crawl.
.. _item-filter:
Item filtering
==============
.. versionadded:: VERSION
You can filter items that you want to allow for a particular feed by using the
``item_classes`` option in :ref:`feeds options <feed-options>`. Only items of
the specified types will be added to the feed.
The ``item_classes`` option is implemented by the :class:`~scrapy.extensions.feedexport.ItemFilter`
class, which is the default value of the ``item_filter`` :ref:`feed option <feed-options>`.
You can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'s
method ``accepts`` and taking ``feed_options`` as an argument.
For instance::
class MyCustomFilter:
def __init__(self, feed_options):
self.feed_options = feed_options
def accepts(self, item):
if "field1" in item and item["field1"] == "expected_data":
return True
return False
You can assign your custom filtering class to the ``item_filter`` :ref:`option of a feed <feed-options>`.
See :setting:`FEEDS` for examples.
ItemFilter
----------
.. autoclass:: scrapy.extensions.feedexport.ItemFilter
:members:
Settings
========
@ -311,6 +351,7 @@ For instance::
'format': 'json',
'encoding': 'utf8',
'store_empty': False,
'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'],
'fields': None,
'indent': 4,
'item_export_kwargs': {
@ -320,12 +361,14 @@ For instance::
'/home/user/documents/items.xml': {
'format': 'xml',
'fields': ['name', 'price'],
'item_filter': MyCustomFilter1,
'encoding': 'latin1',
'indent': 8,
},
pathlib.Path('items.csv'): {
'format': 'csv',
'fields': ['price', 'name'],
'item_filter': 'myproject.filters.MyCustomFilter2',
},
}
@ -347,6 +390,18 @@ as a fallback value if that key is not provided for a specific feed definition:
- ``fields``: falls back to :setting:`FEED_EXPORT_FIELDS`.
- ``item_classes``: list of :ref:`item classes <topics-items>` to export.
If undefined or empty, all items are exported.
.. versionadded:: VERSION
- ``item_filter``: a :ref:`filter class <item-filter>` to filter items to export.
:class:`~scrapy.extensions.feedexport.ItemFilter` is used be default.
.. versionadded:: VERSION
- ``indent``: falls back to :setting:`FEED_EXPORT_INDENT`.
- ``item_export_kwargs``: :class:`dict` with keyword arguments for the corresponding :ref:`item exporter class <topics-exporters>`.
@ -619,9 +674,9 @@ The function signature should be as follows:
:type params: dict
:param spider: source spider of the feed items
:type spider: scrapy.spiders.Spider
:type spider: scrapy.Spider
For example, to include the :attr:`name <scrapy.spiders.Spider.name>` of the
For example, to include the :attr:`name <scrapy.Spider.name>` of the
source spider in the feed URI:
#. Define the following function somewhere in your project::

6
docs/topics/item-pipeline.rst
View File

@ -42,7 +42,7 @@ Each item pipeline component is a Python class that must implement the following
:type item: :ref:`item object <item-types>`
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
Additionally, they may also implement the following methods:
@ -51,14 +51,14 @@ Additionally, they may also implement the following methods:
This method is called when the spider is opened.
:param spider: the spider which was opened
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: close_spider(self, spider)
This method is called when the spider is closed.
:param spider: the spider which was closed
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: from_crawler(cls, crawler)

10
docs/topics/items.rst
View File

@ -42,7 +42,8 @@ Item objects
:class:`Item` provides a :class:`dict`-like API plus additional features that
make it the most feature-complete item type:
.. class:: Item([arg])
.. class:: scrapy.item.Item([arg])
.. class:: scrapy.Item([arg])
:class:`Item` objects replicate the standard :class:`dict` API, including
its ``__init__`` method.
@ -199,7 +200,8 @@ It's important to note that the :class:`Field` objects used to declare the item
do not stay assigned as class attributes. Instead, they can be accessed through
the :attr:`Item.fields` attribute.
.. class:: Field([arg])
.. class:: scrapy.item.Field([arg])
.. class:: scrapy.Field([arg])
The :class:`Field` class is just an alias to the built-in :class:`dict` class and
doesn't provide any extra functionality or attributes. In other words,
@ -317,11 +319,11 @@ If that is not the desired behavior, use a deep copy instead.
See :mod:`copy` for more information.
To create a shallow copy of an item, you can either call
:meth:`~scrapy.item.Item.copy` on an existing item
:meth:`~scrapy.Item.copy` on an existing item
(``product2 = product.copy()``) or instantiate your item class from an existing
item (``product2 = Product(product)``).
To create a deep copy, call :meth:`~scrapy.item.Item.deepcopy` instead
To create a deep copy, call :meth:`~scrapy.Item.deepcopy` instead
(``product2 = product.deepcopy()``).

4
docs/topics/jobs.rst
View File

@ -74,10 +74,10 @@ on cookies.
Request serialization
---------------------
For persistence to work, :class:`~scrapy.http.Request` objects must be
For persistence to work, :class:`~scrapy.Request` objects must be
serializable with :mod:`pickle`, except for the ``callback`` and ``errback``
values passed to their ``__init__`` method, which must be methods of the
running :class:`~scrapy.spiders.Spider` class.
running :class:`~scrapy.Spider` class.
If you wish to log the requests that couldn't be serialized, you can set the
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.

16
docs/topics/leaks.rst
View File

@ -27,7 +27,7 @@ Common causes of memory leaks
It happens quite often (sometimes by accident, sometimes on purpose) that the
Scrapy developer passes objects referenced in Requests (for example, using the
:attr:`~scrapy.http.Request.cb_kwargs` or :attr:`~scrapy.http.Request.meta`
:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta`
attributes or the request callback function) and that effectively bounds the
lifetime of those referenced objects to the lifetime of the Request. This is,
by far, the most common cause of memory leaks in Scrapy projects, and a quite
@ -48,9 +48,9 @@ Too Many Requests?
------------------
By default Scrapy keeps the request queue in memory; it includes
:class:`~scrapy.http.Request` objects and all objects
referenced in Request attributes (e.g. in :attr:`~scrapy.http.Request.cb_kwargs`
and :attr:`~scrapy.http.Request.meta`).
:class:`~scrapy.Request` objects and all objects
referenced in Request attributes (e.g. in :attr:`~scrapy.Request.cb_kwargs`
and :attr:`~scrapy.Request.meta`).
While not necessarily a leak, this can take a lot of memory. Enabling
:ref:`persistent job queue <topics-jobs>` could help keeping memory usage
in control.
@ -90,11 +90,11 @@ Which objects are tracked?
The objects tracked by ``trackrefs`` are all from these classes (and all its
subclasses):
* :class:`scrapy.http.Request`
* :class:`scrapy.Request`
* :class:`scrapy.http.Response`
* :class:`scrapy.item.Item`
* :class:`scrapy.selector.Selector`
* :class:`scrapy.spiders.Spider`
* :class:`scrapy.Item`
* :class:`scrapy.Selector`
* :class:`scrapy.Spider`
A real example
--------------

2
docs/topics/logging.rst
View File

@ -93,7 +93,7 @@ path::
Logging from Spiders
====================
Scrapy provides a :data:`~scrapy.spiders.Spider.logger` within each Spider
Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider
instance, which can be accessed and used like this::
import scrapy

6
docs/topics/media-pipeline.rst
View File

@ -259,7 +259,7 @@ respectively), the pipeline will put the results under the respective field
When using :ref:`item types <item-types>` for which fields are defined beforehand,
you must define both the URLs field and the results field. For example, when
using the images pipeline, items must define both the ``image_urls`` and the
``images`` field. For instance, using the :class:`~scrapy.item.Item` class::
``images`` field. For instance, using the :class:`~scrapy.Item` class::
import scrapy
@ -424,7 +424,7 @@ See here the methods that you can override in your custom Files Pipeline:
In addition to ``response``, this method receives the original
:class:`request <scrapy.Request>`,
:class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and
:class:`item <scrapy.item.Item>`
:class:`item <scrapy.Item>`
You can override this method to customize the download path of each file.
@ -563,7 +563,7 @@ See here the methods that you can override in your custom Images Pipeline:
In addition to ``response``, this method receives the original
:class:`request <scrapy.Request>`,
:class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and
:class:`item <scrapy.item.Item>`
:class:`item <scrapy.Item>`
You can override this method to customize the download path of each file.

16
docs/topics/request-response.rst
View File

@ -35,7 +35,7 @@ Request objects
request (once it's downloaded) as its first parameter. For more information
see :ref:`topics-request-response-ref-request-callback-arguments` below.
If a Request doesn't specify a callback, the spider's
:meth:`~scrapy.spiders.Spider.parse` method will be used.
:meth:`~scrapy.Spider.parse` method will be used.
Note that if exceptions are raised during processing, errback is called instead.
:type callback: collections.abc.Callable
@ -60,7 +60,7 @@ Request objects
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
:ref:`cookies-mw`. If you need to set cookies for a request, use the
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
current limitation that is being worked on.
:type headers: dict
@ -92,7 +92,7 @@ Request objects
To create a request that does not send stored cookies and does not
store received cookies, set the ``dont_merge_cookies`` key to ``True``
in :attr:`request.meta <scrapy.http.Request.meta>`.
in :attr:`request.meta <scrapy.Request.meta>`.
Example of a request that sends manually-defined cookies and ignores
cookie storage::
@ -107,7 +107,7 @@ Request objects
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
:ref:`cookies-mw`. If you need to set cookies for a request, use the
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
current limitation that is being worked on.
:type cookies: dict or list
@ -495,7 +495,9 @@ fields with form data from :class:`Response` objects.
.. _lxml.html forms: https://lxml.de/lxmlhtml.html#forms
.. class:: FormRequest(url, [formdata, ...])
.. class:: scrapy.http.request.form.FormRequest
.. class:: scrapy.http.FormRequest
.. class:: scrapy.FormRequest(url, [formdata, ...])
The :class:`FormRequest` class adds a new keyword parameter to the ``__init__`` method. The
remaining arguments are the same as for the :class:`Request` class and are
@ -694,7 +696,7 @@ Response objects
:param request: the initial value of the :attr:`Response.request` attribute.
This represents the :class:`Request` that generated this response.
:type request: scrapy.http.Request
:type request: scrapy.Request
:param certificate: an object representing the server's SSL certificate.
:type certificate: twisted.internet.ssl.Certificate
@ -920,7 +922,7 @@ TextResponse objects
.. attribute:: TextResponse.selector
A :class:`~scrapy.selector.Selector` instance using the response as
A :class:`~scrapy.Selector` instance using the response as
target. The selector is lazily instantiated on first access.
:class:`TextResponse` objects support the following methods in addition to

8
docs/topics/selectors.rst
View File

@ -48,7 +48,7 @@ Constructing selectors
.. highlight:: python
Response objects expose a :class:`~scrapy.selector.Selector` instance
Response objects expose a :class:`~scrapy.Selector` instance
on ``.selector`` attribute:
>>> response.selector.xpath('//span/text()').get()
@ -62,7 +62,7 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
>>> response.css('span::text').get()
'good'
Scrapy selectors are instances of :class:`~scrapy.selector.Selector` class
Scrapy selectors are instances of :class:`~scrapy.Selector` class
constructed by passing either :class:`~scrapy.http.TextResponse` object or
markup as a string (in ``text`` argument).
@ -175,7 +175,7 @@ of ``None``:
'not-found'
Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes
using ``.attrib`` property of a :class:`~scrapy.selector.Selector`:
using ``.attrib`` property of a :class:`~scrapy.Selector`:
>>> [img.attrib['src'] for img in response.css('img')]
['image1_thumb.jpg',
@ -383,7 +383,7 @@ ID, or when selecting an unique element on a page):
Using selectors with regular expressions
----------------------------------------
:class:`~scrapy.selector.Selector` also has a ``.re()`` method for extracting
:class:`~scrapy.Selector` also has a ``.re()`` method for extracting
data using regular expressions. However, unlike using ``.xpath()`` or
``.css()`` methods, ``.re()`` returns a list of strings. So you
can't construct nested ``.re()`` calls.

16
docs/topics/settings.rst
View File

@ -67,7 +67,7 @@ Example::
Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
own settings that will take precedence and override the project ones. They can
do so by setting their :attr:`~scrapy.spiders.Spider.custom_settings` attribute::
do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute::
class MySpider(scrapy.Spider):
name = 'myspider'
@ -142,7 +142,7 @@ In a spider, the settings are available through ``self.settings``::
The ``settings`` attribute is set in the base Spider class after the spider
is initialized. If you want to use the settings before the initialization
(e.g., in your spider's ``__init__()`` method), you'll need to override the
:meth:`~scrapy.spiders.Spider.from_crawler` method.
:meth:`~scrapy.Spider.from_crawler` method.
Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
attribute of the Crawler that is passed to ``from_crawler`` method in
@ -338,7 +338,7 @@ is non-zero, download delay is enforced per IP, not per domain.
DEFAULT_ITEM_CLASS
------------------
Default: ``'scrapy.item.Item'``
Default: ``'scrapy.Item'``
The default class that will be used for instantiating items in the :ref:`the
Scrapy shell <topics-shell>`.
@ -360,7 +360,7 @@ The default headers used for Scrapy HTTP Requests. They're populated in the
.. caution:: Cookies set via the ``Cookie`` header are not considered by the
:ref:`cookies-mw`. If you need to set cookies for a request, use the
:class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
current limitation that is being worked on.
.. setting:: DEPTH_LIMIT
@ -384,8 +384,8 @@ Default: ``0``
Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``
An integer that is used to adjust the :attr:`~scrapy.http.Request.priority` of
a :class:`~scrapy.http.Request` based on its depth.
An integer that is used to adjust the :attr:`~scrapy.Request.priority` of
a :class:`~scrapy.Request` based on its depth.
The priority of a request is adjusted as follows::
@ -816,14 +816,14 @@ The default (``RFPDupeFilter``) filters based on request fingerprint using
the ``scrapy.utils.request.request_fingerprint`` function. In order to change
the way duplicates are checked you could subclass ``RFPDupeFilter`` and
override its ``request_fingerprint`` method. This method should accept
scrapy :class:`~scrapy.http.Request` object and return its fingerprint
scrapy :class:`~scrapy.Request` object and return its fingerprint
(a string).
You can disable filtering of duplicate requests by setting
:setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``.
Be very careful about this however, because you can get into crawling loops.
It's usually a better idea to set the ``dont_filter`` parameter to
``True`` on the specific :class:`~scrapy.http.Request` that should not be
``True`` on the specific :class:`~scrapy.Request` that should not be
filtered.
.. setting:: DUPEFILTER_DEBUG

8
docs/topics/shell.rst
View File

@ -118,7 +118,7 @@ Available Scrapy objects
The Scrapy shell automatically creates some convenient objects from the
downloaded page, like the :class:`~scrapy.http.Response` object and the
:class:`~scrapy.selector.Selector` objects (for both HTML and XML
:class:`~scrapy.Selector` objects (for both HTML and XML
content).
Those objects are:
@ -126,12 +126,12 @@ Those objects are:
- ``crawler`` - the current :class:`~scrapy.crawler.Crawler` object.
- ``spider`` - the Spider which is known to handle the URL, or a
:class:`~scrapy.spiders.Spider` object if there is no spider found for the
:class:`~scrapy.Spider` object if there is no spider found for the
current URL
- ``request`` - a :class:`~scrapy.http.Request` object of the last fetched
- ``request`` - a :class:`~scrapy.Request` object of the last fetched
page. You can modify this request using
:meth:`~scrapy.http.Request.replace` or fetch a new request (without
:meth:`~scrapy.Request.replace` or fetch a new request (without
leaving the shell) using the ``fetch`` shortcut.
- ``response`` - a :class:`~scrapy.http.Response` object containing the last

61
docs/topics/signals.rst
View File

@ -155,7 +155,7 @@ item_scraped
:type item: :ref:`item object <item-types>`
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param response: the response from where the item was scraped
:type response: :class:`~scrapy.http.Response` object
@ -175,7 +175,7 @@ item_dropped
:type item: :ref:`item object <item-types>`
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param response: the response from where the item was dropped
:type response: :class:`~scrapy.http.Response` object
@ -203,7 +203,7 @@ item_error
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
@ -223,7 +223,7 @@ spider_closed
This signal supports returning deferreds from its handlers.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
:param reason: a string which describes the reason why the spider was closed. If
it was closed because the spider has completed scraping, the reason
@ -247,7 +247,7 @@ spider_opened
This signal supports returning deferreds from its handlers.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
spider_idle
~~~~~~~~~~~
@ -268,10 +268,17 @@ spider_idle
You may raise a :exc:`~scrapy.exceptions.DontCloseSpider` exception to
prevent the spider from being closed.
Alternatively, you may raise a :exc:`~scrapy.exceptions.CloseSpider`
exception to provide a custom spider closing reason. An
idle handler is the perfect place to put some code that assesses
the final spider results and update the final closing reason
accordingly (e.g. setting it to 'too_few_results' instead of
'finished').
This signal does not support returning deferreds from its handlers.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
**not** guarantee that it can prevent the spider from being closed,
@ -296,7 +303,7 @@ spider_error
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
Request signals
---------------
@ -307,16 +314,16 @@ request_scheduled
.. signal:: request_scheduled
.. function:: request_scheduled(request, spider)
Sent when the engine schedules a :class:`~scrapy.http.Request`, to be
Sent when the engine schedules a :class:`~scrapy.Request`, to be
downloaded later.
This signal does not support returning deferreds from its handlers.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
request_dropped
~~~~~~~~~~~~~~~
@ -324,16 +331,16 @@ request_dropped
.. signal:: request_dropped
.. function:: request_dropped(request, spider)
Sent when a :class:`~scrapy.http.Request`, scheduled by the engine to be
Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
downloaded later, is rejected by the scheduler.
This signal does not support returning deferreds from its handlers.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
request_reached_downloader
~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -341,15 +348,15 @@ request_reached_downloader
.. signal:: request_reached_downloader
.. function:: request_reached_downloader(request, spider)
Sent when a :class:`~scrapy.http.Request` reached downloader.
Sent when a :class:`~scrapy.Request` reached downloader.
This signal does not support returning deferreds from its handlers.
:param request: the request that reached downloader
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
request_left_downloader
~~~~~~~~~~~~~~~~~~~~~~~
@ -359,16 +366,16 @@ request_left_downloader
.. versionadded:: 2.0
Sent when a :class:`~scrapy.http.Request` leaves the downloader, even in case of
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
failure.
This signal does not support returning deferreds from its handlers.
:param request: the request that reached the downloader
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
bytes_received
~~~~~~~~~~~~~~
@ -395,10 +402,10 @@ bytes_received
:type data: :class:`bytes` object
:param request: the request that generated the download
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
headers_received
~~~~~~~~~~~~~~~~
@ -425,10 +432,10 @@ headers_received
:type body_length: `int`
:param request: the request that generated the download
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
Response signals
----------------
@ -448,10 +455,10 @@ response_received
:type response: :class:`~scrapy.http.Response` object
:param request: the request that generated the response
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. note:: The ``request`` argument might not contain the original request that
reached the downloader, if a :ref:`topics-downloader-middleware` modifies
@ -472,7 +479,7 @@ response_downloaded
:type response: :class:`~scrapy.http.Response` object
:param request: the request that generated the response
:type request: :class:`~scrapy.http.Request` object
:type request: :class:`~scrapy.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object

26
docs/topics/spider-middleware.rst
View File

@ -93,7 +93,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider for which this response is intended
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: process_spider_output(response, result, spider)
@ -106,7 +106,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
it has processed the response.
:meth:`process_spider_output` must return an iterable of
:class:`~scrapy.http.Request` objects and :ref:`item objects
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
.. note:: When defined as a :ref:`coroutine <async>`, this method needs
@ -117,11 +117,11 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
:type response: :class:`~scrapy.http.Response` object
:param result: the result returned by the spider
:type result: an iterable of :class:`~scrapy.http.Request` objects and
:type result: an iterable of :class:`~scrapy.Request` objects and
:ref:`item objects <topics-items>`
:param spider: the spider whose result is being processed
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: process_spider_exception(response, exception, spider)
@ -129,7 +129,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
method (from a previous spider middleware) raises an exception.
:meth:`process_spider_exception` should return either ``None`` or an
iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects
iterable of :class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
If it returns ``None``, Scrapy will continue processing this exception,
@ -149,7 +149,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
:type exception: :exc:`Exception` object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: process_start_requests(start_requests, spider)
@ -159,7 +159,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
items).
It receives an iterable (in the ``start_requests`` parameter) and must
return another iterable of :class:`~scrapy.http.Request` objects.
return another iterable of :class:`~scrapy.Request` objects.
.. note:: When implementing this method in your spider middleware, you
should always return an iterable (that follows the input one) and
@ -171,10 +171,10 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
(like a time limit or item/page count).
:param start_requests: the start requests
:type start_requests: an iterable of :class:`~scrapy.http.Request`
:type start_requests: an iterable of :class:`~scrapy.Request`
:param spider: the spider to whom the start requests belong
:type spider: :class:`~scrapy.spiders.Spider` object
:type spider: :class:`~scrapy.Spider` object
.. method:: from_crawler(cls, crawler)
@ -258,7 +258,7 @@ this::
.. reqmeta:: handle_httpstatus_all
The ``handle_httpstatus_list`` key of :attr:`Request.meta
<scrapy.http.Request.meta>` can also be used to specify which response codes to
<scrapy.Request.meta>` can also be used to specify which response codes to
allow on a per-request basis. You can also set the meta key ``handle_httpstatus_all``
to ``True`` if you want to allow any response code for a request, and ``False`` to
disable the effects of the ``handle_httpstatus_all`` key.
@ -302,7 +302,7 @@ OffsiteMiddleware
Filters out Requests for URLs outside the domains covered by the spider.
This middleware filters out every request whose host names aren't in the
spider's :attr:`~scrapy.spiders.Spider.allowed_domains` attribute.
spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
All subdomains of any domain in the list are also allowed.
E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
but not ``www2.example.com`` nor ``example.com``.
@ -320,10 +320,10 @@ OffsiteMiddleware
will be printed (but only for the first request filtered).
If the spider doesn't define an
:attr:`~scrapy.spiders.Spider.allowed_domains` attribute, or the
:attr:`~scrapy.Spider.allowed_domains` attribute, or the
attribute is empty, the offsite middleware will allow all requests.
If the request has the :attr:`~scrapy.http.Request.dont_filter` attribute
If the request has the :attr:`~scrapy.Request.dont_filter` attribute
set, the offsite middleware will allow the request even if its domain is not
listed in allowed domains.

57
docs/topics/spiders.rst
View File

@ -17,15 +17,15 @@ For spiders, the scraping cycle goes through something like this:
those requests.
The first requests to perform are obtained by calling the
:meth:`~scrapy.spiders.Spider.start_requests` method which (by default)
generates :class:`~scrapy.http.Request` for the URLs specified in the
:attr:`~scrapy.spiders.Spider.start_urls` and the
:attr:`~scrapy.spiders.Spider.parse` method as callback function for the
:meth:`~scrapy.Spider.start_requests` method which (by default)
generates :class:`~scrapy.Request` for the URLs specified in the
:attr:`~scrapy.Spider.start_urls` and the
:attr:`~scrapy.Spider.parse` method as callback function for the
Requests.
2. In the callback function, you parse the response (web page) and return
:ref:`item objects <topics-items>`,
:class:`~scrapy.http.Request` objects, or an iterable of these objects.
:class:`~scrapy.Request` objects, or an iterable of these objects.
Those Requests will also contain a callback (maybe
the same) and will then be downloaded by Scrapy and then their
response handled by the specified callback.
@ -50,7 +50,8 @@ We will talk about those types here.
scrapy.Spider
=============
.. class:: Spider()
.. class:: scrapy.spiders.Spider()
.. class:: scrapy.Spider()
This is the simplest spider, and the one from which every other spider
must inherit (including spiders that come bundled with Scrapy, as well as spiders
@ -86,7 +87,7 @@ scrapy.Spider
A list of URLs where the spider will begin to crawl from, when no
particular URLs are specified. So, the first pages downloaded will be those
listed here. The subsequent :class:`~scrapy.http.Request` will be generated successively from data
listed here. The subsequent :class:`~scrapy.Request` will be generated successively from data
contained in the start URLs.
.. attribute:: custom_settings
@ -179,7 +180,7 @@ scrapy.Spider
the same requirements as the :class:`Spider` class.
This method, as well as any other Request callback, must return an
iterable of :class:`~scrapy.http.Request` and/or :ref:`item objects
iterable of :class:`~scrapy.Request` and/or :ref:`item objects
<topics-items>`.
:param response: the response to parse
@ -234,7 +235,7 @@ Return multiple Requests and items from a single callback::
yield scrapy.Request(response.urljoin(href), self.parse)
Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
to give data more structure you can use :class:`~scrapy.item.Item` objects::
to give data more structure you can use :class:`~scrapy.Item` objects::
import scrapy
from myproject.items import MyItem
@ -373,7 +374,7 @@ CrawlSpider
This method is called for each response produced for the URLs in
the spider's ``start_urls`` attribute. It allows to parse
the initial responses and must return either an
:ref:`item object <topics-items>`, a :class:`~scrapy.http.Request`
:ref:`item object <topics-items>`, a :class:`~scrapy.Request`
object, or an iterable containing any of them.
Crawling rules
@ -383,7 +384,7 @@ Crawling rules
``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
defines how links will be extracted from each crawled page. Each produced link will
be used to generate a :class:`~scrapy.http.Request` object, which will contain the
be used to generate a :class:`~scrapy.Request` object, which will contain the
link's text in its ``meta`` dictionary (under the ``link_text`` key).
If omitted, a default link extractor created with no arguments will be used,
resulting in all links being extracted.
@ -392,9 +393,9 @@ Crawling rules
object with that name will be used) to be called for each link extracted with
the specified link extractor. This callback receives a :class:`~scrapy.http.Response`
as its first argument and must return either a single instance or an iterable of
:ref:`item objects <topics-items>` and/or :class:`~scrapy.http.Request` objects
:ref:`item objects <topics-items>` and/or :class:`~scrapy.Request` objects
(or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`
object will contain the text of the link that produced the :class:`~scrapy.http.Request`
object will contain the text of the link that produced the :class:`~scrapy.Request`
in its ``meta`` dictionary (under the ``link_text`` key)
``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
@ -411,7 +412,7 @@ Crawling rules
``process_request`` is a callable (or a string, in which case a method from
the spider object with that name will be used) which will be called for every
:class:`~scrapy.http.Request` extracted by this rule. This callable should
:class:`~scrapy.Request` extracted by this rule. This callable should
take said request as first argument and the :class:`~scrapy.http.Response`
from which the request originated as second argument. It must return a
``Request`` object or ``None`` (to filter out the request).
@ -422,10 +423,9 @@ Crawling rules
It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
instance as first parameter.
.. warning:: Because of its internal implementation, you must explicitly set
callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
unexpected behaviour can occur otherwise.
.. warning:: Because of its internal implementation, you must explicitly set
callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
unexpected behaviour can occur otherwise.
.. versionadded:: 2.0
The *errback* parameter.
@ -471,7 +471,7 @@ Let's now take a look at an example CrawlSpider with rules::
This spider would start crawling example.com's home page, collecting category
links, and item links, parsing the latter with the ``parse_item`` method. For
each item response, some data will be extracted from the HTML using XPath, and
an :class:`~scrapy.item.Item` will be filled with it.
an :class:`~scrapy.Item` will be filled with it.
XMLFeedSpider
-------------
@ -494,11 +494,11 @@ XMLFeedSpider
- ``'iternodes'`` - a fast iterator based on regular expressions
- ``'html'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
- ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
Keep in mind this uses DOM parsing and must load all DOM in memory
which could be a problem for big feeds
- ``'xml'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
- ``'xml'`` - an iterator which uses :class:`~scrapy.Selector`.
Keep in mind this uses DOM parsing and must load all DOM in memory
which could be a problem for big feeds
@ -516,7 +516,7 @@ XMLFeedSpider
available in that document that will be processed with this spider. The
``prefix`` and ``uri`` will be used to automatically register
namespaces using the
:meth:`~scrapy.selector.Selector.register_namespace` method.
:meth:`~scrapy.Selector.register_namespace` method.
You can then specify nodes with namespaces in the :attr:`itertag`
attribute.
@ -543,10 +543,10 @@ XMLFeedSpider
This method is called for the nodes matching the provided tag name
(``itertag``). Receives the response and an
:class:`~scrapy.selector.Selector` for each node. Overriding this
:class:`~scrapy.Selector` for each node. Overriding this
method is mandatory. Otherwise, you spider won't work. This method
must return an :ref:`item object <topics-items>`, a
:class:`~scrapy.http.Request` object, or an iterable containing any of
:class:`~scrapy.Request` object, or an iterable containing any of
them.
.. method:: process_results(response, results)
@ -557,10 +557,9 @@ XMLFeedSpider
item IDs. It receives a list of results and the response which originated
those results. It must return a list of results (items or requests).
.. warning:: Because of its internal implementation, you must explicitly set
callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
unexpected behaviour can occur otherwise.
.. warning:: Because of its internal implementation, you must explicitly set
callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
unexpected behaviour can occur otherwise.
XMLFeedSpider example
@ -589,7 +588,7 @@ These spiders are pretty easy to use, let's have a look at one example::
Basically what we did up there was to create a spider that downloads a feed from
the given ``start_urls``, and then iterates through each of its ``item`` tags,
prints them out, and stores some random data in an :class:`~scrapy.item.Item`.
prints them out, and stores some random data in an :class:`~scrapy.Item`.
CSVFeedSpider
-------------

4
pylintrc
View File

@ -6,6 +6,7 @@ jobs=1 # >1 hides results
disable=abstract-method,
anomalous-backslash-in-string,
arguments-differ,
arguments-renamed,
attribute-defined-outside-init,
bad-classmethod-argument,
bad-continuation,
@ -21,6 +22,8 @@ disable=abstract-method,
cell-var-from-loop,
comparison-with-callable,
consider-iterating-dictionary,
consider-using-dict-items,
consider-using-from-import,
consider-using-in,
consider-using-set-comprehension,
consider-using-sys-exit,
@ -105,6 +108,7 @@ disable=abstract-method,
unsubscriptable-object,
unused-argument,
unused-import,
unused-private-member,
unused-variable,
unused-wildcard-import,
used-before-assignment,

16
scrapy/core/downloader/handlers/http11.py
View File

@ -7,7 +7,7 @@ import warnings
from contextlib import suppress
from io import BytesIO
from time import time
from urllib.parse import urldefrag
from urllib.parse import urldefrag, urlunparse
from twisted.internet import defer, protocol, ssl
from twisted.internet.endpoints import TCP4ClientEndpoint
@ -276,16 +276,16 @@ class ScrapyAgent:
bindaddress = request.meta.get('bindaddress') or self._bindAddress
proxy = request.meta.get('proxy')
if proxy:
_, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
scheme = _parse(request.url)[0]
proxyHost = to_unicode(proxyHost)
omitConnectTunnel = b'noconnect' in proxyParams
if omitConnectTunnel:
warnings.warn(
"Using HTTPS proxies in the noconnect mode is deprecated. "
"If you use Zyte Smart Proxy Manager (formerly Crawlera), "
"it doesn't require this mode anymore, so you should "
"update scrapy-crawlera to 1.3.0+ and remove '?noconnect' "
"If you use Zyte Smart Proxy Manager, it doesn't require "
"this mode anymore, so you should update scrapy-crawlera "
"to scrapy-zyte-smartproxy and remove '?noconnect' "
"from the Zyte Smart Proxy Manager URL.",
ScrapyDeprecationWarning,
)
@ -301,9 +301,13 @@ class ScrapyAgent:
pool=self._pool,
)
else:
proxyScheme = proxyScheme or b'http'
proxyHost = to_bytes(proxyHost, encoding='ascii')
proxyPort = to_bytes(str(proxyPort), encoding='ascii')
proxyURI = urlunparse((proxyScheme, proxyNetloc, proxyParams, '', '', ''))
return self._ProxyAgent(
reactor=reactor,
proxyURI=to_bytes(proxy, encoding='ascii'),
proxyURI=to_bytes(proxyURI, encoding='ascii'),
connectTimeout=timeout,
bindAddress=bindaddress,
pool=self._pool,

12
scrapy/core/downloader/handlers/http2.py
View File

@ -72,10 +72,14 @@ class ScrapyH2Agent:
proxy_host = proxy_host.decode()
omit_connect_tunnel = b'noconnect' in proxy_params
if omit_connect_tunnel:
warnings.warn("Using HTTPS proxies in the noconnect mode is not supported by the "
"downloader handler. If you use Crawlera, it doesn't require this "
"mode anymore, so you should update scrapy-crawlera to 1.3.0+ "
"and remove '?noconnect' from the Crawlera URL.")
warnings.warn(
"Using HTTPS proxies in the noconnect mode is not "
"supported by the downloader handler. If you use Zyte "
"Smart Proxy Manager, it doesn't require this mode "
"anymore, so you should update scrapy-crawlera to "
"scrapy-zyte-smartproxy and remove '?noconnect' from the "
"Zyte Smart Proxy Manager URL."
)
if scheme == b'https' and not omit_connect_tunnel:
# ToDo

3
scrapy/core/downloader/webclient.py
View File

@ -1,3 +1,4 @@
import re
from time import time
from urllib.parse import urlparse, urlunparse, urldefrag
@ -32,6 +33,8 @@ def _parse(url):
and is ascii-only.
"""
url = url.strip()
if not re.match(r'^\w+://', url):
url = '//' + url
parsed = urlparse(url)
return _parsed_url_args(parsed)

23
scrapy/core/engine.py
View File

@ -15,7 +15,11 @@ from twisted.python.failure import Failure
from scrapy import signals
from scrapy.core.scraper import Scraper
from scrapy.exceptions import DontCloseSpider, ScrapyDeprecationWarning
from scrapy.exceptions import (
CloseSpider,
DontCloseSpider,
ScrapyDeprecationWarning,
)
from scrapy.http import Response, Request
from scrapy.settings import BaseSettings
from scrapy.spiders import Spider
@ -325,14 +329,23 @@ class ExecutionEngine:
Called when a spider gets idle, i.e. when there are no remaining requests to download or schedule.
It can be called multiple times. If a handler for the spider_idle signal raises a DontCloseSpider
exception, the spider is not closed until the next loop and this function is guaranteed to be called
(at least) once again.
(at least) once again. A handler can raise CloseSpider to provide a custom closing reason.
"""
assert self.spider is not None # typing
res = self.signals.send_catch_log(signals.spider_idle, spider=self.spider, dont_log=DontCloseSpider)
if any(isinstance(x, Failure) and isinstance(x.value, DontCloseSpider) for _, x in res):
expected_ex = (DontCloseSpider, CloseSpider)
res = self.signals.send_catch_log(signals.spider_idle, spider=self.spider, dont_log=expected_ex)
detected_ex = {
ex: x.value
for _, x in res
for ex in expected_ex
if isinstance(x, Failure) and isinstance(x.value, ex)
}
if DontCloseSpider in detected_ex:
return None
if self.spider_is_idle():
self.close_spider(self.spider, reason='finished')
ex = detected_ex.get(CloseSpider, CloseSpider(reason='finished'))
assert isinstance(ex, CloseSpider) # typing
self.close_spider(self.spider, reason=ex.reason)
def close_spider(self, spider: Spider, reason: str = "cancelled") -> Deferred:
"""Close (cancel) spider and clear all its outstanding requests"""

2
scrapy/downloadermiddlewares/retry.py
View File

@ -98,7 +98,7 @@ def get_retry_request(
{'request': request, 'retry_times': retry_times, 'reason': reason},
extra={'spider': spider}
)
new_request = request.copy()
new_request: Request = request.copy()
new_request.meta['retry_times'] = retry_times
new_request.dont_filter = True
if priority_adjust is None:

58
scrapy/extensions/feedexport.py
View File

@ -11,6 +11,7 @@ import sys
import warnings
from datetime import datetime
from tempfile import NamedTemporaryFile
from typing import Any, Optional, Tuple
from urllib.parse import unquote, urlparse
from twisted.internet import defer, threads
@ -46,6 +47,40 @@ def build_storage(builder, uri, *args, feed_options=None, preargs=(), **kwargs):
return builder(*preargs, uri, *args, **kwargs)
class ItemFilter:
"""
This will be used by FeedExporter to decide if an item should be allowed
to be exported to a particular feed.
:param feed_options: feed specific options passed from FeedExporter
:type feed_options: dict
"""
feed_options: Optional[dict]
item_classes: Tuple
def __init__(self, feed_options: Optional[dict]) -> None:
self.feed_options = feed_options
if feed_options is not None:
self.item_classes = tuple(
load_object(item_class) for item_class in feed_options.get("item_classes") or ()
)
else:
self.item_classes = tuple()
def accepts(self, item: Any) -> bool:
"""
Return ``True`` if `item` should be exported or ``False`` otherwise.
:param item: scraped item which user wants to check if is acceptable
:type item: :ref:`Scrapy items <topics-items>`
:return: `True` if accepted, `False` otherwise
:rtype: bool
"""
if self.item_classes:
return isinstance(item, self.item_classes)
return True # accept all items by default
class IFeedStorage(Interface):
"""Interface that all Feed Storages must implement"""
@ -118,7 +153,7 @@ class FileFeedStorage:
class S3FeedStorage(BlockingFeedStorage):
def __init__(self, uri, access_key=None, secret_key=None, acl=None, *,
def __init__(self, uri, access_key=None, secret_key=None, acl=None, endpoint_url=None, *,
feed_options=None):
if not is_botocore_available():
raise NotConfigured('missing botocore library')
@ -128,11 +163,13 @@ class S3FeedStorage(BlockingFeedStorage):
self.secret_key = u.password or secret_key
self.keyname = u.path[1:] # remove first "/"
self.acl = acl
self.endpoint_url = endpoint_url
import botocore.session
session = botocore.session.get_session()
self.s3_client = session.create_client(
's3', aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key)
aws_secret_access_key=self.secret_key,
endpoint_url=self.endpoint_url)
if feed_options and feed_options.get('overwrite', True) is False:
logger.warning('S3 does not support appending to files. To '
'suppress this warning, remove the overwrite '
@ -146,6 +183,7 @@ class S3FeedStorage(BlockingFeedStorage):
access_key=crawler.settings['AWS_ACCESS_KEY_ID'],
secret_key=crawler.settings['AWS_SECRET_ACCESS_KEY'],
acl=crawler.settings['FEED_STORAGE_S3_ACL'] or None,
endpoint_url=crawler.settings['AWS_ENDPOINT_URL'] or None,
feed_options=feed_options,
)
@ -215,7 +253,7 @@ class FTPFeedStorage(BlockingFeedStorage):
class _FeedSlot:
def __init__(self, file, exporter, storage, uri, format, store_empty, batch_id, uri_template):
def __init__(self, file, exporter, storage, uri, format, store_empty, batch_id, uri_template, filter):
self.file = file
self.exporter = exporter
self.storage = storage
@ -225,6 +263,7 @@ class _FeedSlot:
self.store_empty = store_empty
self.uri_template = uri_template
self.uri = uri
self.filter = filter
# flags
self.itemcount = 0
self._exporting = False
@ -255,6 +294,7 @@ class FeedExporter:
self.settings = crawler.settings
self.feeds = {}
self.slots = []
self.filters = {}
if not self.settings['FEEDS'] and not self.settings['FEED_URI']:
raise NotConfigured
@ -269,12 +309,14 @@ class FeedExporter:
uri = str(self.settings['FEED_URI']) # handle pathlib.Path objects
feed_options = {'format': self.settings.get('FEED_FORMAT', 'jsonlines')}
self.feeds[uri] = feed_complete_default_values_from_settings(feed_options, self.settings)
self.filters[uri] = self._load_filter(feed_options)
# End: Backward compatibility for FEED_URI and FEED_FORMAT settings
# 'FEEDS' setting takes precedence over 'FEED_URI'
for uri, feed_options in self.settings.getdict('FEEDS').items():
uri = str(uri) # handle pathlib.Path objects
self.feeds[uri] = feed_complete_default_values_from_settings(feed_options, self.settings)
self.filters[uri] = self._load_filter(feed_options)
self.storages = self._load_components('FEED_STORAGES')
self.exporters = self._load_components('FEED_EXPORTERS')
@ -368,6 +410,7 @@ class FeedExporter:
store_empty=feed_options['store_empty'],
batch_id=batch_id,
uri_template=uri_template,
filter=self.filters[uri_template]
)
if slot.store_empty:
slot.start_exporting()
@ -376,6 +419,10 @@ class FeedExporter:
def item_scraped(self, item, spider):
slots = []
for slot in self.slots:
if not slot.filter.accepts(item):
slots.append(slot) # if slot doesn't accept item, continue with next slot
continue
slot.start_exporting()
slot.exporter.export_item(item)
slot.itemcount += 1
@ -486,3 +533,8 @@ class FeedExporter:
uripar_function = load_object(uri_params) if uri_params else lambda x, y: None
uripar_function(params, spider)
return params
def _load_filter(self, feed_options):
# load the item filter if declared else load the default filter class
item_filter_class = load_object(feed_options.get("item_filter", ItemFilter))
return item_filter_class(feed_options)

57
scrapy/http/request/__init__.py
View File

@ -5,7 +5,7 @@ requests in Scrapy.
See documentation in docs/topics/request-response.rst
"""
import inspect
from typing import Optional, Tuple
from typing import Callable, List, Optional, Tuple, Type, TypeVar, Union
from w3lib.url import safe_url_string
@ -18,6 +18,9 @@ from scrapy.utils.trackref import object_ref
from scrapy.utils.url import escape_ajax
RequestTypeVar = TypeVar("RequestTypeVar", bound="Request")
class Request(object_ref):
"""Represents an HTTP request, which is usually generated in a Spider and
executed by the Downloader, thus generating a :class:`Response`.
@ -36,10 +39,22 @@ class Request(object_ref):
:func:`~scrapy.utils.request.request_from_dict`.
"""
def __init__(self, url, callback=None, method='GET', headers=None, body=None,
cookies=None, meta=None, encoding='utf-8', priority=0,
dont_filter=False, errback=None, flags=None, cb_kwargs=None):
def __init__(
self,
url: str,
callback: Optional[Callable] = None,
method: str = "GET",
headers: Optional[dict] = None,
body: Optional[Union[bytes, str]] = None,
cookies: Optional[Union[dict, List[dict]]] = None,
meta: Optional[dict] = None,
encoding: str = "utf-8",
priority: int = 0,
dont_filter: bool = False,
errback: Optional[Callable] = None,
flags: Optional[List[str]] = None,
cb_kwargs: Optional[dict] = None,
) -> None:
self._encoding = encoding # this one has to be set first
self.method = str(method).upper()
self._set_url(url)
@ -64,23 +79,23 @@ class Request(object_ref):
self.flags = [] if flags is None else list(flags)
@property
def cb_kwargs(self):
def cb_kwargs(self) -> dict:
if self._cb_kwargs is None:
self._cb_kwargs = {}
return self._cb_kwargs
@property
def meta(self):
def meta(self) -> dict:
if self._meta is None:
self._meta = {}
return self._meta
def _get_url(self):
def _get_url(self) -> str:
return self._url
def _set_url(self, url):
def _set_url(self, url: str) -> None:
if not isinstance(url, str):
raise TypeError(f'Request url must be str or unicode, got {type(url).__name__}')
raise TypeError(f"Request url must be str, got {type(url).__name__}")
s = safe_url_string(url, self.encoding)
self._url = escape_ajax(s)
@ -94,31 +109,27 @@ class Request(object_ref):
url = property(_get_url, obsolete_setter(_set_url, 'url'))
def _get_body(self):
def _get_body(self) -> bytes:
return self._body
def _set_body(self, body):
if body is None:
self._body = b''
else:
self._body = to_bytes(body, self.encoding)
def _set_body(self, body: Optional[Union[str, bytes]]) -> None:
self._body = b"" if body is None else to_bytes(body, self.encoding)
body = property(_get_body, obsolete_setter(_set_body, 'body'))
@property
def encoding(self):
def encoding(self) -> str:
return self._encoding
def __str__(self):
def __str__(self) -> str:
return f"<{self.method} {self.url}>"
__repr__ = __str__
def copy(self):
"""Return a copy of this Request"""
def copy(self) -> "Request":
return self.replace()
def replace(self, *args, **kwargs):
def replace(self, *args, **kwargs) -> "Request":
"""Create a new Request with the same attributes except for those given new values"""
for x in self.attributes:
kwargs.setdefault(x, getattr(self, x))
@ -126,7 +137,9 @@ class Request(object_ref):
return cls(*args, **kwargs)
@classmethod
def from_curl(cls, curl_command, ignore_unknown_options=True, **kwargs):
def from_curl(
cls: Type[RequestTypeVar], curl_command: str, ignore_unknown_options: bool = True, **kwargs
) -> RequestTypeVar:
"""Create a Request object from a string containing a `cURL
<https://curl.haxx.se/>`_ command. It populates the HTTP method, the
URL, the headers, the cookies and the body. It accepts the same

92
scrapy/http/request/form.py
View File

@ -5,22 +5,28 @@ This module implements the FormRequest class which is a more convenient class
See documentation in docs/topics/request-response.rst
"""
from urllib.parse import urljoin, urlencode
from typing import Iterable, List, Optional, Tuple, Type, TypeVar, Union
from urllib.parse import urljoin, urlencode, urlsplit, urlunsplit
import lxml.html
from lxml.html import FormElement, HtmlElement, HTMLParser, SelectElement
from parsel.selector import create_root_node
from w3lib.html import strip_html5_whitespace
from scrapy.http.request import Request
from scrapy.http.response.text import TextResponse
from scrapy.utils.python import to_bytes, is_listlike
from scrapy.utils.response import get_base_url
FormRequestTypeVar = TypeVar("FormRequestTypeVar", bound="FormRequest")
FormdataType = Optional[Union[dict, List[Tuple[str, str]]]]
class FormRequest(Request):
valid_form_methods = ['GET', 'POST']
def __init__(self, *args, **kwargs):
formdata = kwargs.pop('formdata', None)
def __init__(self, *args, formdata: FormdataType = None, **kwargs) -> None:
if formdata and kwargs.get('method') is None:
kwargs['method'] = 'POST'
@ -28,17 +34,27 @@ class FormRequest(Request):
if formdata:
items = formdata.items() if isinstance(formdata, dict) else formdata
querystr = _urlencode(items, self.encoding)
form_query_str = _urlencode(items, self.encoding)
if self.method == 'POST':
self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
self._set_body(querystr)
self._set_body(form_query_str)
else:
self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)
self._set_url(urlunsplit(urlsplit(self.url)._replace(query=form_query_str)))
@classmethod
def from_response(cls, response, formname=None, formid=None, formnumber=0, formdata=None,
clickdata=None, dont_click=False, formxpath=None, formcss=None, **kwargs):
def from_response(
cls: Type[FormRequestTypeVar],
response: TextResponse,
formname: Optional[str] = None,
formid: Optional[str] = None,
formnumber: Optional[int] = 0,
formdata: FormdataType = None,
clickdata: Optional[dict] = None,
dont_click: bool = False,
formxpath: Optional[str] = None,
formcss: Optional[str] = None,
**kwargs,
) -> FormRequestTypeVar:
kwargs.setdefault('encoding', response.encoding)
if formcss is not None:
@ -46,7 +62,7 @@ class FormRequest(Request):
formxpath = HTMLTranslator().css_to_xpath(formcss)
form = _get_form(response, formname, formid, formnumber, formxpath)
formdata = _get_inputs(form, formdata, dont_click, clickdata, response)
formdata = _get_inputs(form, formdata, dont_click, clickdata)
url = _get_form_url(form, kwargs.pop('url', None))
method = kwargs.pop('method', form.method)
@ -58,7 +74,7 @@ class FormRequest(Request):
return cls(url=url, method=method, formdata=formdata, **kwargs)
def _get_form_url(form, url):
def _get_form_url(form: FormElement, url: Optional[str]) -> str:
if url is None:
action = form.get('action')
if action is None:
@ -67,17 +83,22 @@ def _get_form_url(form, url):
return urljoin(form.base_url, url)
def _urlencode(seq, enc):
def _urlencode(seq: Iterable, enc: str) -> str:
values = [(to_bytes(k, enc), to_bytes(v, enc))
for k, vs in seq
for v in (vs if is_listlike(vs) else [vs])]
return urlencode(values, doseq=True)
def _get_form(response, formname, formid, formnumber, formxpath):
"""Find the form element """
root = create_root_node(response.text, lxml.html.HTMLParser,
base_url=get_base_url(response))
def _get_form(
response: TextResponse,
formname: Optional[str],
formid: Optional[str],
formnumber: Optional[int],
formxpath: Optional[str],
) -> FormElement:
"""Find the wanted form element within the given response."""
root = create_root_node(response.text, HTMLParser, base_url=get_base_url(response))
forms = root.xpath('//form')
if not forms:
raise ValueError(f"No <form> element found in {response}")
@ -105,8 +126,7 @@ def _get_form(response, formname, formid, formnumber, formxpath):
break
raise ValueError(f'No <form> element found with {formxpath}')
# If we get here, it means that either formname was None
# or invalid
# If we get here, it means that either formname was None or invalid
if formnumber is not None:
try:
form = forms[formnumber]
@ -116,25 +136,32 @@ def _get_form(response, formname, formid, formnumber, formxpath):
return form
def _get_inputs(form, formdata, dont_click, clickdata, response):
def _get_inputs(
form: FormElement,
formdata: FormdataType,
dont_click: bool,
clickdata: Optional[dict],
) -> List[Tuple[str, str]]:
"""Return a list of key-value pairs for the inputs found in the given form."""
try:
formdata_keys = dict(formdata or ()).keys()
except (ValueError, TypeError):
raise ValueError('formdata should be a dict or iterable of tuples')
if not formdata:
formdata = ()
formdata = []
inputs = form.xpath('descendant::textarea'
'|descendant::select'
'|descendant::input[not(@type) or @type['
' not(re:test(., "^(?:submit|image|reset)$", "i"))'
' and (../@checked or'
' not(re:test(., "^(?:checkbox|radio)$", "i")))]]',
namespaces={
"re": "http://exslt.org/regular-expressions"})
values = [(k, '' if v is None else v)
for k, v in (_value(e) for e in inputs)
if k and k not in formdata_keys]
namespaces={"re": "http://exslt.org/regular-expressions"})
values = [
(k, '' if v is None else v)
for k, v in (_value(e) for e in inputs)
if k and k not in formdata_keys
]
if not dont_click:
clickable = _get_clickable(clickdata, form)
@ -142,13 +169,13 @@ def _get_inputs(form, formdata, dont_click, clickdata, response):
values.append(clickable)
if isinstance(formdata, dict):
formdata = formdata.items()
formdata = formdata.items() # type: ignore[assignment]
values.extend((k, v) for k, v in formdata if v is not None)
return values
def _value(ele):
def _value(ele: HtmlElement):
n = ele.name
v = ele.value
if ele.tag == 'select':
@ -156,7 +183,7 @@ def _value(ele):
return n, v
def _select_value(ele, n, v):
def _select_value(ele: SelectElement, n: str, v: str):
multiple = ele.multiple
if v is None and not multiple:
# Match browser behaviour on simple select tag without options selected
@ -167,11 +194,12 @@ def _select_value(ele, n, v):
# This is a workround to bug in lxml fixed 2.3.1
# fix https://github.com/lxml/lxml/commit/57f49eed82068a20da3db8f1b18ae00c1bab8b12#L1L1139
selected_options = ele.xpath('.//option[@selected]')
v = [(o.get('value') or o.text or '').strip() for o in selected_options]
values = [(o.get('value') or o.text or '').strip() for o in selected_options]
return n, values
return n, v
def _get_clickable(clickdata, form):
def _get_clickable(clickdata: Optional[dict], form: FormElement) -> Optional[Tuple[str, str]]:
"""
Returns the clickable element specified in clickdata,
if the latter is given. If not, it returns the first
@ -183,7 +211,7 @@ def _get_clickable(clickdata, form):
namespaces={"re": "http://exslt.org/regular-expressions"}
))
if not clickables:
return
return None
# If we don't have clickdata, we just use the first clickable element
if clickdata is None:

15
scrapy/http/request/json_request.py
View File

@ -8,7 +8,7 @@ See documentation in docs/topics/request-response.rst
import copy
import json
import warnings
from typing import Tuple
from typing import Optional, Tuple
from scrapy.http.request import Request
from scrapy.utils.deprecate import create_deprecated_class
@ -18,8 +18,8 @@ class JsonRequest(Request):
attributes: Tuple[str, ...] = Request.attributes + ("dumps_kwargs",)
def __init__(self, *args, **kwargs):
dumps_kwargs = copy.deepcopy(kwargs.pop('dumps_kwargs', {}))
def __init__(self, *args, dumps_kwargs: Optional[dict] = None, **kwargs) -> None:
dumps_kwargs = copy.deepcopy(dumps_kwargs) if dumps_kwargs is not None else {}
dumps_kwargs.setdefault('sort_keys', True)
self._dumps_kwargs = dumps_kwargs
@ -29,10 +29,8 @@ class JsonRequest(Request):
if body_passed and data_passed:
warnings.warn('Both body and data passed. data will be ignored')
elif not body_passed and data_passed:
kwargs['body'] = self._dumps(data)
if 'method' not in kwargs:
kwargs['method'] = 'POST'
@ -41,23 +39,22 @@ class JsonRequest(Request):
self.headers.setdefault('Accept', 'application/json, text/javascript, */*; q=0.01')
@property
def dumps_kwargs(self):
def dumps_kwargs(self) -> dict:
return self._dumps_kwargs
def replace(self, *args, **kwargs):
def replace(self, *args, **kwargs) -> Request:
body_passed = kwargs.get('body', None) is not None
data = kwargs.pop('data', None)
data_passed = data is not None
if body_passed and data_passed:
warnings.warn('Both body and data passed. data will be ignored')
elif not body_passed and data_passed:
kwargs['body'] = self._dumps(data)
return super().replace(*args, **kwargs)
def _dumps(self, data):
def _dumps(self, data: dict) -> str:
"""Convert to JSON """
return json.dumps(data, **self._dumps_kwargs)

4
scrapy/http/request/rpc.py
View File

@ -5,6 +5,7 @@ This module implements the XmlRpcRequest class which is a more convenient class
See documentation in docs/topics/request-response.rst
"""
import xmlrpc.client as xmlrpclib
from typing import Optional
from scrapy.http.request import Request
from scrapy.utils.python import get_func_args
@ -15,8 +16,7 @@ DUMPS_ARGS = get_func_args(xmlrpclib.dumps)
class XmlRpcRequest(Request):
def __init__(self, *args, **kwargs):
encoding = kwargs.get('encoding', None)
def __init__(self, *args, encoding: Optional[str] = None, **kwargs):
if 'body' not in kwargs and 'params' in kwargs:
kw = dict((k, kwargs.pop(k)) for k in DUMPS_ARGS if k in kwargs)
kwargs['body'] = xmlrpclib.dumps(**kw)

2
scrapy/utils/curl.py
View File

@ -54,7 +54,7 @@ def _parse_headers_and_cookies(parsed_args):
return headers, cookies
def curl_to_request_kwargs(curl_command, ignore_unknown_options=True):
def curl_to_request_kwargs(curl_command: str, ignore_unknown_options: bool = True) -> dict:
"""Convert a cURL command syntax to Request kwargs.
:param str curl_command: string containing the curl command

10
scrapy/utils/signal.py
View File

@ -1,5 +1,5 @@
"""Helper functions for working with signals"""
import collections
import logging
from twisted.internet.defer import DeferredList, Deferred
@ -16,15 +16,13 @@ from scrapy.utils.log import failure_to_exc_info
logger = logging.getLogger(__name__)
class _IgnoredException(Exception):
pass
def send_catch_log(signal=Any, sender=Anonymous, *arguments, **named):
"""Like pydispatcher.robust.sendRobust but it also logs errors and returns
Failures instead of exceptions.
"""
dont_log = (named.pop('dont_log', _IgnoredException), StopDownload)
dont_log = named.pop('dont_log', ())
dont_log = tuple(dont_log) if isinstance(dont_log, collections.Sequence) else (dont_log,)
dont_log += (StopDownload, )
spider = named.get('spider', None)
responses = []
for receiver in liveReceivers(getAllReceivers(sender, signal)):

10
tests/test_downloader_handlers.py
View File

@ -764,6 +764,16 @@ class Http11ProxyTestCase(HttpProxyTestCase):
timeout = yield self.assertFailure(d, error.TimeoutError)
self.assertIn(domain, timeout.osError)
def test_download_with_proxy_without_http_scheme(self):
def _test(response):
self.assertEqual(response.status, 200)
self.assertEqual(response.url, request.url)
self.assertEqual(response.body, self.expected_http_proxy_request_body)
http_proxy = self.getURL('').replace('http://', '')
request = Request('http://example.com', meta={'proxy': http_proxy})
return self.download_request(request, Spider('foo')).addCallback(_test)
class HttpDownloadHandlerMock:

21
tests/test_engine.py
View File

@ -26,7 +26,7 @@ from twisted.web import server, static, util
from scrapy import signals
from scrapy.core.engine import ExecutionEngine
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.exceptions import CloseSpider, ScrapyDeprecationWarning
from scrapy.http import Request
from scrapy.item import Item, Field
from scrapy.linkextractors import LinkExtractor
@ -113,6 +113,18 @@ class ItemZeroDivisionErrorSpider(TestSpider):
}
class ChangeCloseReasonSpider(TestSpider):
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = cls(*args, **kwargs)
spider._set_crawler(crawler)
crawler.signals.connect(spider.spider_idle, signals.spider_idle)
return spider
def spider_idle(self):
raise CloseSpider(reason="custom_reason")
def start_test_site(debug=False):
root_dir = os.path.join(tests_datadir, "test_site")
r = static.File(root_dir)
@ -251,6 +263,13 @@ class EngineTest(unittest.TestCase):
yield self.run.run()
self._assert_items_error()
@defer.inlineCallbacks
def test_crawler_change_close_reason_on_idle(self):
self.run = CrawlerRun(ChangeCloseReasonSpider)
yield self.run.run()
self.assertEqual({'spider': self.run.spider, 'reason': 'custom_reason'},
self.run.signals_caught[signals.spider_closed])
def _assert_visited_urls(self):
must_be_visited = ["/", "/redirect", "/redirected",
"/item1.html", "/item2.html", "/item999.html"]

162
tests/test_feedexport.py
View File

@ -326,6 +326,17 @@ class S3FeedStorageTest(unittest.TestCase):
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.acl, 'custom-acl')
def test_init_with_endpoint_url(self):
storage = S3FeedStorage(
's3://mybucket/export.csv',
'access_key',
'secret_key',
endpoint_url='https://example.com'
)
self.assertEqual(storage.access_key, 'access_key')
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.endpoint_url, 'https://example.com')
def test_from_crawler_without_acl(self):
settings = {
'AWS_ACCESS_KEY_ID': 'access_key',
@ -340,6 +351,20 @@ class S3FeedStorageTest(unittest.TestCase):
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.acl, None)
def test_without_endpoint_url(self):
settings = {
'AWS_ACCESS_KEY_ID': 'access_key',
'AWS_SECRET_ACCESS_KEY': 'secret_key',
}
crawler = get_crawler(settings_dict=settings)
storage = S3FeedStorage.from_crawler(
crawler,
's3://mybucket/export.csv',
)
self.assertEqual(storage.access_key, 'access_key')
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.endpoint_url, None)
def test_from_crawler_with_acl(self):
settings = {
'AWS_ACCESS_KEY_ID': 'access_key',
@ -355,6 +380,21 @@ class S3FeedStorageTest(unittest.TestCase):
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.acl, 'custom-acl')
def test_from_crawler_with_endpoint_url(self):
settings = {
'AWS_ACCESS_KEY_ID': 'access_key',
'AWS_SECRET_ACCESS_KEY': 'secret_key',
'AWS_ENDPOINT_URL': 'https://example.com',
}
crawler = get_crawler(settings_dict=settings)
storage = S3FeedStorage.from_crawler(
crawler,
's3://mybucket/export.csv'
)
self.assertEqual(storage.access_key, 'access_key')
self.assertEqual(storage.secret_key, 'secret_key')
self.assertEqual(storage.endpoint_url, 'https://example.com')
@defer.inlineCallbacks
def test_store_botocore_without_acl(self):
skip_if_no_boto()
@ -561,6 +601,10 @@ class FeedExportTestBase(ABC, unittest.TestCase):
egg = scrapy.Field()
baz = scrapy.Field()
class MyItem2(scrapy.Item):
foo = scrapy.Field()
hello = scrapy.Field()
def _random_temp_filename(self, inter_dir=''):
chars = [random.choice(ascii_letters + digits) for _ in range(15)]
filename = ''.join(chars)
@ -888,13 +932,9 @@ class FeedExportTest(FeedExportTestBase):
@defer.inlineCallbacks
def test_export_multiple_item_classes(self):
class MyItem2(scrapy.Item):
foo = scrapy.Field()
hello = scrapy.Field()
items = [
self.MyItem({'foo': 'bar1', 'egg': 'spam1'}),
MyItem2({'hello': 'world2', 'foo': 'bar2'}),
self.MyItem2({'hello': 'world2', 'foo': 'bar2'}),
self.MyItem({'foo': 'bar3', 'egg': 'spam3', 'baz': 'quux3'}),
{'hello': 'world4', 'egg': 'spam4'},
]
@ -929,6 +969,114 @@ class FeedExportTest(FeedExportTestBase):
yield self.assertExported(items, header, rows,
settings=settings, ordered=True)
@defer.inlineCallbacks
def test_export_based_on_item_classes(self):
items = [
self.MyItem({'foo': 'bar1', 'egg': 'spam1'}),
self.MyItem2({'hello': 'world2', 'foo': 'bar2'}),
{'hello': 'world3', 'egg': 'spam3'},
]
formats = {
'csv': b'baz,egg,foo\r\n,spam1,bar1\r\n',
'json': b'[\n{"hello": "world2", "foo": "bar2"}\n]',
'jsonlines': (
b'{"foo": "bar1", "egg": "spam1"}\n'
b'{"hello": "world2", "foo": "bar2"}\n'
),
'xml': (
b'<?xml version="1.0" encoding="utf-8"?>\n<items>\n<item>'
b'<foo>bar1</foo><egg>spam1</egg></item>\n<item><hello>'
b'world2</hello><foo>bar2</foo></item>\n<item><hello>world3'
b'</hello><egg>spam3</egg></item>\n</items>'
),
}
settings = {
'FEEDS': {
self._random_temp_filename(): {
'format': 'csv',
'item_classes': [self.MyItem],
},
self._random_temp_filename(): {
'format': 'json',
'item_classes': [self.MyItem2],
},
self._random_temp_filename(): {
'format': 'jsonlines',
'item_classes': [self.MyItem, self.MyItem2],
},
self._random_temp_filename(): {
'format': 'xml',
},
},
}
data = yield self.exported_data(items, settings)
for fmt, expected in formats.items():
self.assertEqual(expected, data[fmt])
@defer.inlineCallbacks
def test_export_based_on_custom_filters(self):
items = [
self.MyItem({'foo': 'bar1', 'egg': 'spam1'}),
self.MyItem2({'hello': 'world2', 'foo': 'bar2'}),
{'hello': 'world3', 'egg': 'spam3'},
]
MyItem = self.MyItem
class CustomFilter1:
def __init__(self, feed_options):
pass
def accepts(self, item):
return isinstance(item, MyItem)
class CustomFilter2(scrapy.extensions.feedexport.ItemFilter):
def accepts(self, item):
if 'foo' not in item.fields:
return False
return True
class CustomFilter3(scrapy.extensions.feedexport.ItemFilter):
def accepts(self, item):
if isinstance(item, tuple(self.item_classes)) and item['foo'] == "bar1":
return True
return False
formats = {
'json': b'[\n{"foo": "bar1", "egg": "spam1"}\n]',
'xml': (
b'<?xml version="1.0" encoding="utf-8"?>\n<items>\n<item>'
b'<foo>bar1</foo><egg>spam1</egg></item>\n<item><hello>'
b'world2</hello><foo>bar2</foo></item>\n</items>'
),
'jsonlines': b'{"foo": "bar1", "egg": "spam1"}\n',
}
settings = {
'FEEDS': {
self._random_temp_filename(): {
'format': 'json',
'item_filter': CustomFilter1,
},
self._random_temp_filename(): {
'format': 'xml',
'item_filter': CustomFilter2,
},
self._random_temp_filename(): {
'format': 'jsonlines',
'item_classes': [self.MyItem, self.MyItem2],
'item_filter': CustomFilter3,
},
},
}
data = yield self.exported_data(items, settings)
for fmt, expected in formats.items():
self.assertEqual(expected, data[fmt])
@defer.inlineCallbacks
def test_export_dicts(self):
# When dicts are used, only keys from the first row are used as
@ -1809,8 +1957,8 @@ class FileFeedStoragePreFeedOptionsTest(unittest.TestCase):
class S3FeedStorageWithoutFeedOptions(S3FeedStorage):
def __init__(self, uri, access_key, secret_key, acl):
super().__init__(uri, access_key, secret_key, acl)
def __init__(self, uri, access_key, secret_key, acl, endpoint_url):
super().__init__(uri, access_key, secret_key, acl, endpoint_url)
class S3FeedStorageWithoutFeedOptionsWithFromCrawler(S3FeedStorage):

14
tests/test_http_request.py
View File

@ -379,6 +379,20 @@ class FormRequestTest(RequestTest):
r1 = self.request_class("http://www.example.com", formdata={})
self.assertEqual(r1.body, b'')
def test_formdata_overrides_querystring(self):
data = (('a', 'one'), ('a', 'two'), ('b', '2'))
url = self.request_class('http://www.example.com/?a=0&b=1&c=3#fragment',
method='GET', formdata=data).url.split('#')[0]
fs = _qs(self.request_class(url, method='GET', formdata=data))
self.assertEqual(set(fs[b'a']), {b'one', b'two'})
self.assertEqual(fs[b'b'], [b'2'])
self.assertIsNone(fs.get(b'c'))
data = {'a': '1', 'b': '2'}
fs = _qs(self.request_class('http://www.example.com/', method='GET', formdata=data))
self.assertEqual(fs[b'a'], [b'1'])
self.assertEqual(fs[b'b'], [b'2'])
def test_default_encoding_bytes(self):
# using default encoding (utf-8)
data = {b'one': b'two', b'price': b'\xc2\xa3 100'}