RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into asyncio-startrequests-asyncgen

This commit is contained in:
Adrián Chaves 2020-08-28 09:55:59 +02:00 committed by GitHub
parent 7839aee998 64905e3397
commit d568423b68
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 19 additions and 132 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/contributing.rst
View File

@ -199,6 +199,17 @@ In any case, if something is covered in a docstring, use the
documentation instead of duplicating the docstring in files within the
``docs/`` directory.
Documentation updates that cover new or modified features must use Sphinx’s
:rst:dir:`versionadded` and :rst:dir:`versionchanged` directives. Use
``VERSION`` as version, we will replace it with the actual version right before
the corresponding release. When we release a new major or minor version of
Scrapy, we remove these directives if they are older than 3 years.
Documentation about deprecated features must be removed as those features are
deprecated, so that new readers do not run into it. New deprecations and
deprecation removals are documented in the :ref:`release notes <news>`.
Tests
=====

2
docs/topics/api.rst
View File

@ -4,8 +4,6 @@
Core API
========
.. versionadded:: 0.15
This section documents the Scrapy core API, and it's intended for developers of
extensions and middlewares.

2
docs/topics/autothrottle.rst
View File

@ -128,8 +128,6 @@ The maximum download delay (in seconds) to be set in case of high latencies.
AUTOTHROTTLE_TARGET_CONCURRENCY
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 1.1
Default: ``1.0``
Average number of requests Scrapy should be sending in parallel to remote

2
docs/topics/benchmarking.rst
View File

@ -4,8 +4,6 @@
Benchmarking
============
.. versionadded:: 0.17
Scrapy comes with a simple benchmarking suite that spawns a local HTTP server
and crawls it at the maximum possible speed. The goal of this benchmarking is
to get an idea of how Scrapy performs in your hardware, in order to have a

4
docs/topics/commands.rst
View File

@ -6,8 +6,6 @@
Command line tool
=================
.. versionadded:: 0.10
Scrapy is controlled through the ``scrapy`` command-line tool, to be referred
here as the "Scrapy tool" to differentiate it from the sub-commands, which we
just call "commands" or "Scrapy commands".
@ -566,8 +564,6 @@ and Platform info, which is useful for bug reports.
bench
-----
.. versionadded:: 0.17
* Syntax: ``scrapy bench``
* Requires project: *no*

2
docs/topics/contracts.rst
View File

@ -4,8 +4,6 @@
Spiders Contracts
=================
.. versionadded:: 0.15
Testing spiders can get particularly annoying and while nothing prevents you
from writing unit tests the task gets cumbersome quickly. Scrapy offers an
integrated way of testing your spiders by the means of contracts.

36
docs/topics/downloader-middleware.rst
View File

@ -217,8 +217,6 @@ The following settings can be used to configure the cookie middleware:
Multiple cookie sessions per spider
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 0.15
There is support for keeping multiple cookie sessions per spider by using the
:reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar
(session), but you can pass an identifier to use different ones.
@ -475,8 +473,6 @@ DBM storage backend
.. class:: DbmCacheStorage
.. versionadded:: 0.13
A DBM_ storage backend is also available for the HTTP cache middleware.
By default, it uses the :mod:`dbm`, but you can change it with the
@ -549,15 +545,10 @@ settings:
HTTPCACHE_ENABLED
^^^^^^^^^^^^^^^^^
.. versionadded:: 0.11
Default: ``False``
Whether the HTTP cache will be enabled.
.. versionchanged:: 0.11
Before 0.11, :setting:`HTTPCACHE_DIR` was used to enable cache.
.. setting:: HTTPCACHE_EXPIRATION_SECS
HTTPCACHE_EXPIRATION_SECS
@ -570,9 +561,6 @@ Expiration time for cached requests, in seconds.
Cached requests older than this time will be re-downloaded. If zero, cached
requests will never expire.
.. versionchanged:: 0.11
Before 0.11, zero meant cached requests always expire.
.. setting:: HTTPCACHE_DIR
HTTPCACHE_DIR
@ -589,8 +577,6 @@ project data dir. For more info see: :ref:`topics-project-structure`.
HTTPCACHE_IGNORE_HTTP_CODES
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.10
Default: ``[]``
Don't cache response with these HTTP codes.
@ -609,8 +595,6 @@ If enabled, requests not found in the cache will be ignored instead of downloade
HTTPCACHE_IGNORE_SCHEMES
^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.10
Default: ``['file']``
Don't cache responses with these URI schemes.
@ -629,8 +613,6 @@ The class which implements the cache storage backend.
HTTPCACHE_DBM_MODULE
^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.13
Default: ``'dbm'``
The database module to use in the :ref:`DBM storage backend
@ -641,8 +623,6 @@ The database module to use in the :ref:`DBM storage backend
HTTPCACHE_POLICY
^^^^^^^^^^^^^^^^
.. versionadded:: 0.18
Default: ``'scrapy.extensions.httpcache.DummyPolicy'``
The class which implements the cache policy.
@ -652,8 +632,6 @@ The class which implements the cache policy.
HTTPCACHE_GZIP
^^^^^^^^^^^^^^
.. versionadded:: 1.0
Default: ``False``
If enabled, will compress all cached data with gzip.
@ -664,8 +642,6 @@ This setting is specific to the Filesystem backend.
HTTPCACHE_ALWAYS_STORE
^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 1.1
Default: ``False``
If enabled, will cache pages unconditionally.
@ -684,8 +660,6 @@ responses you feed to the cache middleware.
HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 1.1
Default: ``[]``
List of Cache-Control directives in responses to be ignored.
@ -735,8 +709,6 @@ HttpProxyMiddleware
.. module:: scrapy.downloadermiddlewares.httpproxy
:synopsis: Http Proxy Middleware
.. versionadded:: 0.8
.. reqmeta:: proxy
.. class:: HttpProxyMiddleware
@ -817,8 +789,6 @@ RedirectMiddleware settings
REDIRECT_ENABLED
^^^^^^^^^^^^^^^^
.. versionadded:: 0.13
Default: ``True``
Whether the Redirect middleware will be enabled.
@ -860,8 +830,6 @@ MetaRefreshMiddleware settings
METAREFRESH_ENABLED
^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.17
Default: ``True``
Whether the Meta Refresh middleware will be enabled.
@ -924,8 +892,6 @@ RetryMiddleware Settings
RETRY_ENABLED
^^^^^^^^^^^^^
.. versionadded:: 0.13
Default: ``True``
Whether the Retry middleware will be enabled.
@ -1179,8 +1145,6 @@ AjaxCrawlMiddleware Settings
AJAXCRAWL_ENABLED
^^^^^^^^^^^^^^^^^
.. versionadded:: 0.21
Default: ``False``
Whether the AjaxCrawlMiddleware will be enabled. You may want to

4
docs/topics/extensions.rst
View File

@ -288,8 +288,6 @@ If zero (or non set), spiders won't be closed by number of passed items.
CLOSESPIDER_PAGECOUNT
"""""""""""""""""""""
.. versionadded:: 0.11
Default: ``0``
An integer which specifies the maximum number of responses to crawl. If the spider
@ -302,8 +300,6 @@ number of crawled responses.
CLOSESPIDER_ERRORCOUNT
""""""""""""""""""""""
.. versionadded:: 0.11
Default: ``0``
An integer which specifies the maximum number of errors to receive before

2
docs/topics/feed-exports.rst
View File

@ -4,8 +4,6 @@
Feed exports
============
.. versionadded:: 0.10
One of the most frequently required features when implementing scrapers is
being able to store the scraped data properly and, quite often, that means
generating an "export file" with the scraped data (commonly called "export

12
docs/topics/request-response.rst
View File

@ -553,18 +553,6 @@ fields with form data from :class:`Response` objects.
The other parameters of this class method are passed directly to the
:class:`FormRequest` ``__init__`` method.
.. versionadded:: 0.10.3
The ``formname`` parameter.
.. versionadded:: 0.17
The ``formxpath`` parameter.
.. versionadded:: 1.1.0
The ``formcss`` parameter.
.. versionadded:: 1.1.0
The ``formid`` parameter.
Request usage examples
----------------------

4
docs/topics/settings.rst
View File

@ -1076,8 +1076,6 @@ See :ref:`topics-extensions-ref-memusage`.
MEMUSAGE_CHECK_INTERVAL_SECONDS
-------------------------------
.. versionadded:: 1.1
Default: ``60.0``
Scope: ``scrapy.extensions.memusage``
@ -1358,8 +1356,6 @@ The class that will be used for loading spiders, which must implement the
SPIDER_LOADER_WARN_ONLY
-----------------------
.. versionadded:: 1.3.3
Default: ``False``
By default, when Scrapy tries to import spider classes from :setting:`SPIDER_MODULES`,

9
docs/topics/spider-middleware.rst
View File

@ -146,9 +146,8 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. method:: process_start_requests(start_requests, spider)
.. versionadded:: 0.15
.. versionchanged:: 2.3
Since 2.3 this can take and return an :term:`python:asynchronous
.. versionchanged:: VERSION
Since VERSION this can take and return an :term:`python:asynchronous
iterable`.
This method is called with the start requests of the spider, and works
@ -351,8 +350,6 @@ RefererMiddleware settings
REFERER_ENABLED
^^^^^^^^^^^^^^^
.. versionadded:: 0.15
Default: ``True``
Whether to enable referer middleware.
@ -362,8 +359,6 @@ Whether to enable referer middleware.
REFERRER_POLICY
^^^^^^^^^^^^^^^
.. versionadded:: 1.4
Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``
.. reqmeta:: referrer_policy

5
scrapy/spiders/__init__.py
View File

@ -5,6 +5,7 @@ See documentation in docs/topics/spiders.rst
"""
import logging
import warnings
from typing import Optional
from scrapy import signals
from scrapy.http import Request
@ -18,8 +19,8 @@ class Spider(object_ref):
class.
"""
name = None
custom_settings = None
name: Optional[str] = None
custom_settings: Optional[dict] = None
def __init__(self, name=None, **kwargs):
if name is not None:

3
scrapy/spiders/crawl.py
View File

@ -7,6 +7,7 @@ See documentation in docs/topics/spiders.rst
import copy
import warnings
from typing import Sequence
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.http import Request, HtmlResponse
@ -72,7 +73,7 @@ class Rule:
class CrawlSpider(Spider):
rules = ()
rules: Sequence[Rule] = ()
def __init__(self, *a, **kw):
super().__init__(*a, **kw)

51
setup.cfg
View File

@ -16,9 +16,6 @@ ignore_errors = True
[mypy-scrapy.commands]
ignore_errors = True
[mypy-scrapy.commands.bench]
ignore_errors = True
[mypy-scrapy.commands.parse]
ignore_errors = True
@ -28,9 +25,6 @@ ignore_errors = True
[mypy-scrapy.contracts]
ignore_errors = True
[mypy-scrapy.core.spidermw]
ignore_errors = True
[mypy-scrapy.interfaces]
ignore_errors = True
@ -70,15 +64,6 @@ ignore_errors = True
[mypy-tests.mocks.dummydbm]
ignore_errors = True
[mypy-tests.spiders]
ignore_errors = True
[mypy-tests.test_cmdline_crawl_with_pipeline.test_spider.spiders.exception]
ignore_errors = True
[mypy-tests.test_cmdline_crawl_with_pipeline.test_spider.spiders.normal]
ignore_errors = True
[mypy-tests.test_command_fetch]
ignore_errors = True
@ -94,9 +79,6 @@ ignore_errors = True
[mypy-tests.test_contracts]
ignore_errors = True
[mypy-tests.test_crawler]
ignore_errors = True
[mypy-tests.test_downloader_handlers]
ignore_errors = True
@ -127,53 +109,20 @@ ignore_errors = True
[mypy-tests.test_pipeline_images]
ignore_errors = True
[mypy-tests.test_pipelines]
ignore_errors = True
[mypy-tests.test_request_attribute_binding]
ignore_errors = True
[mypy-tests.test_request_cb_kwargs]
ignore_errors = True
[mypy-tests.test_request_left]
ignore_errors = True
[mypy-tests.test_scheduler]
ignore_errors = True
[mypy-tests.test_signals]
ignore_errors = True
[mypy-tests.test_spiderloader.test_spiders.nested.spider4]
ignore_errors = True
[mypy-tests.test_spiderloader.test_spiders.spider1]
ignore_errors = True
[mypy-tests.test_spiderloader.test_spiders.spider2]
ignore_errors = True
[mypy-tests.test_spiderloader.test_spiders.spider3]
ignore_errors = True
[mypy-tests.test_spidermiddleware_httperror]
ignore_errors = True
[mypy-tests.test_spidermiddleware_output_chain]
ignore_errors = True
[mypy-tests.test_spidermiddleware_referer]
ignore_errors = True
[mypy-tests.test_utils_reqser]
ignore_errors = True
[mypy-tests.test_utils_serialize]
ignore_errors = True
[mypy-tests.test_utils_spider]
ignore_errors = True
[mypy-tests.test_utils_url]
ignore_errors = True

2
tests/spiders.py
View File

@ -304,7 +304,7 @@ class CrawlSpiderWithParseMethod(MockServerSpider, CrawlSpider):
A CrawlSpider which overrides the 'parse' method
"""
name = 'crawl_spider_with_parse_method'
custom_settings = {
custom_settings: dict = {
'RETRY_HTTP_CODES': [], # no need to retry
}
rules = (