RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into typing-utils

This commit is contained in:
Andrey Rakhmatullin 2023-06-24 18:31:56 +04:00
parent 6998e1c905 85f5122965
commit 712ee98848
50 changed files with 780 additions and 238 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.bumpversion.cfg
View File

@ -1,5 +1,5 @@
[bumpversion]
current_version = 2.8.0
current_version = 2.9.0
commit = True
tag = True
tag_name = {new_version}

2
.github/workflows/tests-macos.yml vendored
View File

@ -7,7 +7,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v3

18
.github/workflows/tests-ubuntu.yml vendored
View File

@ -8,9 +8,6 @@ jobs:
fail-fast: false
matrix:
include:
- python-version: 3.8
env:
TOXENV: py
- python-version: 3.9
env:
TOXENV: py
@ -28,19 +25,28 @@ jobs:
TOXENV: pypy3
# pinned deps
- python-version: 3.7.13
- python-version: 3.8.17
env:
TOXENV: pinned
- python-version: 3.7.13
- python-version: 3.8.17
env:
TOXENV: asyncio-pinned
- python-version: pypy3.7
- python-version: pypy3.8
env:
TOXENV: pypy3-pinned
- python-version: 3.8.17
env:
TOXENV: extra-deps-pinned
- python-version: 3.8.17
env:
TOXENV: botocore-pinned
- python-version: "3.11"
env:
TOXENV: extra-deps
- python-version: "3.11"
env:
TOXENV: botocore
steps:
- uses: actions/checkout@v3

5
.github/workflows/tests-windows.yml vendored
View File

@ -8,12 +8,9 @@ jobs:
fail-fast: false
matrix:
include:
- python-version: 3.7
env:
TOXENV: windows-pinned
- python-version: 3.8
env:
TOXENV: py
TOXENV: windows-pinned
- python-version: 3.9
env:
TOXENV: py

4
.pre-commit-config.yaml
View File

@ -5,7 +5,7 @@ repos:
- id: bandit
args: [-r, -c, .bandit.yml]
- repo: https://github.com/PyCQA/flake8
rev: 5.0.4 # 6.0.0 drops Python 3.7 support
rev: 6.0.0
hooks:
- id: flake8
- repo: https://github.com/psf/black.git
@ -13,7 +13,7 @@ repos:
hooks:
- id: black
- repo: https://github.com/pycqa/isort
rev: 5.11.5 # 5.12 drops Python 3.7 support
rev: 5.12.0
hooks:
- id: isort
- repo: https://github.com/adamchainz/blacken-docs

2
README.rst
View File

@ -58,7 +58,7 @@ including a list of features.
Requirements
============
* Python 3.7+
* Python 3.8+
* Works on Linux, Windows, macOS, BSD
Install

12
docs/contributing.rst
View File

@ -265,15 +265,15 @@ To run a specific test (say ``tests/test_loader.py``) use:
To run the tests on a specific :doc:`tox <tox:index>` environment, use
``-e <name>`` with an environment name from ``tox.ini``. For example, to run
the tests with Python 3.7 use::
the tests with Python 3.10 use::
tox -e py37
tox -e py310
You can also specify a comma-separated list of environments, and use :ref:`tox’s
parallel mode <tox:parallel_mode>` to run the tests on multiple environments in
parallel::
tox -e py37,py38 -p auto
tox -e py39,py310 -p auto
To pass command-line options to :doc:`pytest <pytest:index>`, add them after
``--`` in your call to :doc:`tox <tox:index>`. Using ``--`` overrides the
@ -283,9 +283,9 @@ default positional arguments (``scrapy tests``) after ``--`` as well::
tox -- scrapy tests -x # stop after first failure
You can also use the `pytest-xdist`_ plugin. For example, to run all tests on
the Python 3.7 :doc:`tox <tox:index>` environment using all your CPU cores::
the Python 3.10 :doc:`tox <tox:index>` environment using all your CPU cores::
tox -e py37 -- scrapy tests -n auto
tox -e py310 -- scrapy tests -n auto
To see coverage report install :doc:`coverage <coverage:index>`
(``pip install coverage``) and run:
@ -322,4 +322,4 @@ And their unit-tests are in::
.. _pytest-xdist: https://github.com/pytest-dev/pytest-xdist
.. _good first issues: https://github.com/scrapy/scrapy/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
.. _help wanted issues: https://github.com/scrapy/scrapy/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
.. _test coverage: https://app.codecov.io/gh/scrapy/scrapy
.. _test coverage: https://app.codecov.io/gh/scrapy/scrapy

2
docs/intro/install.rst
View File

@ -9,7 +9,7 @@ Installation guide
Supported Python versions
=========================
Scrapy requires Python 3.7+, either the CPython implementation (default) or
Scrapy requires Python 3.8+, either the CPython implementation (default) or
the PyPy implementation (see :ref:`python:implementations`).
.. _intro-install-scrapy:

125
docs/news.rst
View File

@ -3,6 +3,128 @@
Release notes
=============
.. _release-2.9.0:
Scrapy 2.9.0 (2023-05-08)
-------------------------
Highlights:
- Per-domain download settings.
- Compatibility with new cryptography_ and new parsel_.
- JMESPath selectors from the new parsel_.
- Bug fixes.
Deprecations
~~~~~~~~~~~~
- :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to
:class:`scrapy.extensions.feedexport.FeedSlot` and the old name is
deprecated. (:issue:`5876`)
New features
~~~~~~~~~~~~
- Settings correponding to :setting:`DOWNLOAD_DELAY`,
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
:setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis
via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`)
- Added :meth:`TextResponse.jmespath`, a shortcut for JMESPath selectors
available since parsel_ 1.8.1. (:issue:`5894`, :issue:`5915`)
- Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
signals. (:issue:`5876`)
- Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a
curl command from a :class:`~scrapy.Request` object. (:issue:`5892`)
- Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be
:class:`pathlib.Path` instances. (:issue:`5801`)
Bug fixes
~~~~~~~~~
- Fixed a warning with Parsel 1.8.1+. (:issue:`5903`, :issue:`5918`)
- Fixed an error when using feed postprocessing with S3 storage.
(:issue:`5500`, :issue:`5581`)
- Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method.
(:issue:`5811`, :issue:`5821`)
- Fixed an error when using cryptography_ 40.0.0+ and
:setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled.
(:issue:`5857`, :issue:`5858`)
- The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline`
for files on Google Cloud Storage are no longer Base64-encoded.
(:issue:`5874`, :issue:`5891`)
- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed
string values for the curl ``--data-raw`` argument, which are produced by
browsers for data that includes certain symbols. (:issue:`5899`,
:issue:`5901`)
- The :command:`parse` command now also works with async generator callbacks.
(:issue:`5819`, :issue:`5824`)
- The :command:`genspider` command now properly works with HTTPS URLs.
(:issue:`3553`, :issue:`5808`)
- Improved handling of asyncio loops. (:issue:`5831`, :issue:`5832`)
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
now skips certain malformed URLs instead of raising an exception.
(:issue:`5881`)
- :func:`scrapy.utils.python.get_func_args` now supports more types of
callables. (:issue:`5872`, :issue:`5885`)
- Fixed an error when processing non-UTF8 values of ``Content-Type`` headers.
(:issue:`5914`, :issue:`5917`)
- Fixed an error breaking user handling of send failures in
:meth:`scrapy.mail.MailSender.send()`. (:issue:`1611`, :issue:`5880`)
Documentation
~~~~~~~~~~~~~
- Expanded contributing docs. (:issue:`5109`, :issue:`5851`)
- Added blacken-docs_ to pre-commit and reformatted the docs with it.
(:issue:`5813`, :issue:`5816`)
- Fixed a JS issue. (:issue:`5875`, :issue:`5877`)
- Fixed ``make htmlview``. (:issue:`5878`, :issue:`5879`)
- Fixed typos and other small errors. (:issue:`5827`, :issue:`5839`,
:issue:`5883`, :issue:`5890`, :issue:`5895`, :issue:`5904`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Extended typing hints. (:issue:`5805`, :issue:`5889`, :issue:`5896`)
- Tests for most of the examples in the docs are now run as a part of CI,
found problems were fixed. (:issue:`5816`, :issue:`5826`, :issue:`5919`)
- Removed usage of deprecated Python classes. (:issue:`5849`)
- Silenced ``include-ignored`` warnings from coverage. (:issue:`5820`)
- Fixed a random failure of the ``test_feedexport.test_batch_path_differ``
test. (:issue:`5855`, :issue:`5898`)
- Updated docstrings to match output produced by parsel_ 1.8.1 so that they
don't cause test failures. (:issue:`5902`, :issue:`5919`)
- Other CI and pre-commit improvements. (:issue:`5802`, :issue:`5823`,
:issue:`5908`)
.. _blacken-docs: https://github.com/adamchainz/blacken-docs
.. _release-2.8.0:
Scrapy 2.8.0 (2023-02-02)
@ -4207,8 +4329,6 @@ Relocations
+ Note: telnet is not enabled on Python 3
(https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)
.. _parsel: https://github.com/scrapy/parsel
Bugfixes
~~~~~~~~
@ -5638,6 +5758,7 @@ First release of Scrapy.
.. _LevelDB: https://github.com/google/leveldb
.. _lxml: https://lxml.de/
.. _marshal: https://docs.python.org/2/library/marshal.html
.. _parsel: https://github.com/scrapy/parsel
.. _parsel.csstranslator.GenericTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.GenericTranslator
.. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator
.. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr

2
docs/topics/components.rst
View File

@ -70,7 +70,7 @@ If your requirement is a minimum Scrapy version, you may use
.. code-block:: python
from pkg_resources import parse_version
from packaging.version import parse as parse_version
import scrapy

57
docs/topics/coroutines.rst
View File

@ -134,6 +134,63 @@ Common use cases for asynchronous code include:
.. _aio-libs: https://github.com/aio-libs
.. _inline-requests:
Inline requests
===============
The spider below shows how to send a request and await its response all from
within a spider callback:
.. code-block:: python
from scrapy import Spider, Request
from scrapy.utils.defer import maybe_deferred_to_future
class SingleRequestSpider(Spider):
name = "single"
start_urls = ["https://example.org/product"]
async def parse(self, response, **kwargs):
additional_request = Request("https://example.org/price")
deferred = self.crawler.engine.download(additional_request)
additional_response = await maybe_deferred_to_future(deferred)
yield {
"h1": response.css("h1").get(),
"price": additional_response.css("#price").get(),
}
You can also send multiple requests in parallel:
.. code-block:: python
from scrapy import Spider, Request
from scrapy.utils.defer import maybe_deferred_to_future
from twisted.internet.defer import DeferredList
class MultipleRequestsSpider(Spider):
name = "multiple"
start_urls = ["https://example.com/product"]
async def parse(self, response, **kwargs):
additional_requests = [
Request("https://example.com/price"),
Request("https://example.com/color"),
]
deferreds = []
for r in additional_requests:
deferred = self.crawler.engine.download(r)
deferreds.append(deferred)
responses = await maybe_deferred_to_future(DeferredList(deferreds))
yield {
"h1": response.css("h1::text").get(),
"price": responses[0][1].css(".price::text").get(),
"price2": responses[1][1].css(".color::text").get(),
}
.. _sync-async-spider-middleware:
Mixing synchronous and asynchronous spider middlewares

32
docs/topics/downloader-middleware.rst
View File

@ -915,6 +915,7 @@ settings (see the settings documentation for more info):
* :setting:`RETRY_ENABLED`
* :setting:`RETRY_TIMES`
* :setting:`RETRY_HTTP_CODES`
* :setting:`RETRY_EXCEPTIONS`
.. reqmeta:: dont_retry
@ -966,6 +967,37 @@ In some cases you may want to add 400 to :setting:`RETRY_HTTP_CODES` because
it is a common code used to indicate server overload. It is not included by
default because HTTP specs say so.
.. setting:: RETRY_EXCEPTIONS
RETRY_EXCEPTIONS
^^^^^^^^^^^^^^^^
Default::
[
'twisted.internet.defer.TimeoutError',
'twisted.internet.error.TimeoutError',
'twisted.internet.error.DNSLookupError',
'twisted.internet.error.ConnectionRefusedError',
'twisted.internet.error.ConnectionDone',
'twisted.internet.error.ConnectError',
'twisted.internet.error.ConnectionLost',
'twisted.internet.error.TCPTimedOutError',
'twisted.web.client.ResponseFailed',
IOError,
'scrapy.core.downloader.handlers.http11.TunnelError',
]
List of exceptions to retry.
Each list entry may be an exception type or its import path as a string.
An exception will not be caught when the exception type is not in
:setting:`RETRY_EXCEPTIONS` or when the maximum number of retries for a request
has been exceeded (see :setting:`RETRY_TIMES`). To learn about uncaught
exception propagation, see
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`.
.. setting:: RETRY_PRIORITY_ADJUST
RETRY_PRIORITY_ADJUST

32
docs/topics/feed-exports.rst
View File

@ -101,12 +101,12 @@ The storages backends supported out of the box are:
- :ref:`topics-feed-storage-fs`
- :ref:`topics-feed-storage-ftp`
- :ref:`topics-feed-storage-s3` (requires botocore_)
- :ref:`topics-feed-storage-s3` (requires boto3_)
- :ref:`topics-feed-storage-gcs` (requires `google-cloud-storage`_)
- :ref:`topics-feed-storage-stdout`
Some storage backends may be unavailable if the required external libraries are
not available. For example, the S3 backend is only available if the botocore_
not available. For example, the S3 backend is only available if the boto3_
library is installed.
@ -156,8 +156,8 @@ The feeds are stored in the local filesystem.
- Required external libraries: none
Note that for the local filesystem storage (only) you can omit the scheme if
you specify an absolute path like ``/tmp/export.csv``. This only works on Unix
systems though.
you specify an absolute path like ``/tmp/export.csv`` (Unix systems only).
Alternatively you can also use a :class:`pathlib.Path` object.
.. _topics-feed-storage-ftp:
@ -175,6 +175,12 @@ FTP supports two different connection modes: `active or passive
mode by default. To use the active connection mode instead, set the
:setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.
The default value for the ``overwrite`` key in the :setting:`FEEDS` for this
storage backend is: ``True``.
.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
previous version of your data.
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
@ -193,7 +199,7 @@ The feeds are stored on `Amazon S3`_.
- ``s3://aws_key:aws_secret@mybucket/path/to/export.csv``
- Required external libraries: `botocore`_ >= 1.4.87
- Required external libraries: `boto3`_ >= 1.20.0
The AWS credentials can be passed as user/password in the URI, or they can be
passed through the following settings:
@ -209,6 +215,12 @@ You can also define a custom ACL and custom endpoint for exported feeds using th
- :setting:`FEED_STORAGE_S3_ACL`
- :setting:`AWS_ENDPOINT_URL`
The default value for the ``overwrite`` key in the :setting:`FEEDS` for this
storage backend is: ``True``.
.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
previous version of your data.
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
@ -236,6 +248,12 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following
- :setting:`FEED_STORAGE_GCS_ACL`
- :setting:`GCS_PROJECT_ID`
The default value for the ``overwrite`` key in the :setting:`FEEDS` for this
storage backend is: ``True``.
.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
previous version of your data.
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
@ -488,6 +506,8 @@ as a fallback value if that key is not provided for a specific feed definition:
- :ref:`topics-feed-storage-s3`: ``True`` (appending `is not supported
<https://forums.aws.amazon.com/message.jspa?messageID=540395>`_)
- :ref:`topics-feed-storage-gcs`: ``True`` (appending is not supported)
- :ref:`topics-feed-storage-stdout`: ``False`` (overwriting is not supported)
.. versionadded:: 2.4.0
@ -779,6 +799,6 @@ source spider in the feed URI:
.. _URIs: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier
.. _Amazon S3: https://aws.amazon.com/s3/
.. _botocore: https://github.com/boto/botocore
.. _boto3: https://github.com/boto/boto3
.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
.. _Google Cloud Storage: https://cloud.google.com/storage/

7
docs/topics/request-response.rst
View File

@ -1103,9 +1103,10 @@ Response objects
through all :ref:`Downloader Middlewares <topics-downloader-middleware>`.
In particular, this means that:
- HTTP redirections will cause the original request (to the URL before
redirection) to be assigned to the redirected response (with the final
URL after redirection).
- HTTP redirections will create a new request from the request before
redirection. It has the majority of the same metadata and original
request attributes and gets assigned to the redirected response
instead of the propagation of the original request.
- Response.request.url doesn't always equal Response.url

2
docs/topics/settings.rst
View File

@ -783,7 +783,7 @@ DOWNLOAD_SLOTS
Default: ``{}``
Allows to define concurrency/delay parameters on per slot(domain) basis:
Allows to define concurrency/delay parameters on per slot (domain) basis:
.. code-block:: python

2
docs/utils/linkfix.py
View File

@ -30,7 +30,7 @@ def main():
try:
with Path("build/linkcheck/output.txt").open(encoding="utf-8") as out:
output_lines = out.readlines()
except IOError:
except OSError:
print("linkcheck output not found; please run linkcheck first.")
sys.exit(1)

2
scrapy/VERSION
View File

@ -1 +1 @@
2.8.0
2.9.0

4
scrapy/__init__.py
View File

@ -34,8 +34,8 @@ twisted_version = (_txv.major, _txv.minor, _txv.micro)
# Check minimum required Python version
if sys.version_info < (3, 7):
print(f"Scrapy {__version__} requires Python 3.7+")
if sys.version_info < (3, 8):
print(f"Scrapy {__version__} requires Python 3.8+")
sys.exit(1)

5
scrapy/cmdline.py
View File

@ -3,8 +3,7 @@ import cProfile
import inspect
import os
import sys
import pkg_resources
from importlib.metadata import entry_points
import scrapy
from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
@ -49,7 +48,7 @@ def _get_commands_from_module(module, inproject):
def _get_commands_from_entry_points(inproject, group="scrapy.commands"):
cmds = {}
for entry_point in pkg_resources.iter_entry_points(group):
for entry_point in entry_points().get(group, {}):
obj = entry_point.load()
if inspect.isclass(obj):
cmds[entry_point.name] = obj()

4
scrapy/downloadermiddlewares/decompression.py
View File

@ -63,7 +63,7 @@ class DecompressionMiddleware:
archive = BytesIO(response.body)
try:
body = gzip.GzipFile(fileobj=archive).read()
except IOError:
except OSError:
return
respcls = responsetypes.from_args(body=body)
@ -72,7 +72,7 @@ class DecompressionMiddleware:
def _is_bzip2(self, response):
try:
body = bz2.decompress(response.body)
except IOError:
except OSError:
return
respcls = responsetypes.from_args(body=body)

2
scrapy/downloadermiddlewares/httpcache.py
View File

@ -37,7 +37,7 @@ class HttpCacheMiddleware:
ConnectionLost,
TCPTimedOutError,
ResponseFailed,
IOError,
OSError,
)
def __init__(self, settings: Settings, stats: StatsCollector) -> None:

63
scrapy/downloadermiddlewares/retry.py
View File

@ -9,31 +9,36 @@ RETRY_HTTP_CODES - which HTTP response codes to retry
Failed pages are collected on the scraping process and rescheduled at the end,
once the spider has finished crawling all regular (non failed) pages.
"""
import warnings
from logging import Logger, getLogger
from typing import Optional, Union
from twisted.internet import defer
from twisted.internet.error import (
ConnectError,
ConnectionDone,
ConnectionLost,
ConnectionRefusedError,
DNSLookupError,
TCPTimedOutError,
TimeoutError,
)
from twisted.web.client import ResponseFailed
from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.exceptions import NotConfigured
from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
from scrapy.http.request import Request
from scrapy.settings import Settings
from scrapy.spiders import Spider
from scrapy.utils.misc import load_object
from scrapy.utils.python import global_object_name
from scrapy.utils.response import response_status_message
retry_logger = getLogger(__name__)
class BackwardsCompatibilityMetaclass(type):
@property
def EXCEPTIONS_TO_RETRY(cls):
warnings.warn(
"Attribute RetryMiddleware.EXCEPTIONS_TO_RETRY is deprecated. "
"Use the RETRY_EXCEPTIONS setting instead.",
ScrapyDeprecationWarning,
stacklevel=2,
)
return tuple(
load_object(x) if isinstance(x, str) else x
for x in Settings().getlist("RETRY_EXCEPTIONS")
)
def get_retry_request(
request: Request,
*,
@ -121,23 +126,7 @@ def get_retry_request(
return None
class RetryMiddleware:
# IOError is raised by the HttpCompression middleware when trying to
# decompress an empty response
EXCEPTIONS_TO_RETRY = (
defer.TimeoutError,
TimeoutError,
DNSLookupError,
ConnectionRefusedError,
ConnectionDone,
ConnectError,
ConnectionLost,
TCPTimedOutError,
ResponseFailed,
IOError,
TunnelError,
)
class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
def __init__(self, settings):
if not settings.getbool("RETRY_ENABLED"):
raise NotConfigured
@ -147,6 +136,16 @@ class RetryMiddleware:
)
self.priority_adjust = settings.getint("RETRY_PRIORITY_ADJUST")
if not hasattr(
self, "EXCEPTIONS_TO_RETRY"
): # If EXCEPTIONS_TO_RETRY is not "overriden"
self.exceptions_to_retry = tuple(
load_object(x) if isinstance(x, str) else x
for x in settings.getlist("RETRY_EXCEPTIONS")
)
else:
self.exceptions_to_retry = self.EXCEPTIONS_TO_RETRY
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings)
@ -160,7 +159,7 @@ class RetryMiddleware:
return response
def process_exception(self, request, exception, spider):
if isinstance(exception, self.EXCEPTIONS_TO_RETRY) and not request.meta.get(
if isinstance(exception, self.exceptions_to_retry) and not request.meta.get(
"dont_retry", False
):
return self._retry(request, exception, spider)

3
scrapy/exporters.py
View File

@ -255,6 +255,9 @@ class CsvItemExporter(BaseItemExporter):
values = list(self._build_row(x for _, x in fields))
self.csv_writer.writerow(values)
def finish_exporting(self):
self.stream.detach() # Avoid closing the wrapped file.
def _build_row(self, values):
for s in values:
try:

68
scrapy/extensions/feedexport.py
View File

@ -33,6 +33,13 @@ from scrapy.utils.python import get_func_args, without_none_values
logger = logging.getLogger(__name__)
try:
import boto3 # noqa: F401
IS_BOTO3_AVAILABLE = True
except ImportError:
IS_BOTO3_AVAILABLE = False
def build_storage(builder, uri, *args, feed_options=None, preargs=(), **kwargs):
argument_names = get_func_args(builder)
@ -176,16 +183,38 @@ class S3FeedStorage(BlockingFeedStorage):
self.keyname = u.path[1:] # remove first "/"
self.acl = acl
self.endpoint_url = endpoint_url
import botocore.session
session = botocore.session.get_session()
self.s3_client = session.create_client(
"s3",
aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key,
aws_session_token=self.session_token,
endpoint_url=self.endpoint_url,
)
if IS_BOTO3_AVAILABLE:
import boto3.session
session = boto3.session.Session()
self.s3_client = session.client(
"s3",
aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key,
aws_session_token=self.session_token,
endpoint_url=self.endpoint_url,
)
else:
warnings.warn(
"`botocore` usage has been deprecated for S3 feed "
"export, please use `boto3` to avoid problems",
category=ScrapyDeprecationWarning,
)
import botocore.session
session = botocore.session.get_session()
self.s3_client = session.create_client(
"s3",
aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key,
aws_session_token=self.session_token,
endpoint_url=self.endpoint_url,
)
if feed_options and feed_options.get("overwrite", True) is False:
logger.warning(
"S3 does not support appending to files. To "
@ -208,10 +237,16 @@ class S3FeedStorage(BlockingFeedStorage):
def _store_in_thread(self, file):
file.seek(0)
kwargs = {"ACL": self.acl} if self.acl else {}
self.s3_client.put_object(
Bucket=self.bucketname, Key=self.keyname, Body=file, **kwargs
)
if IS_BOTO3_AVAILABLE:
kwargs = {"ExtraArgs": {"ACL": self.acl}} if self.acl else {}
self.s3_client.upload_fileobj(
Bucket=self.bucketname, Key=self.keyname, Fileobj=file, **kwargs
)
else:
kwargs = {"ACL": self.acl} if self.acl else {}
self.s3_client.put_object(
Bucket=self.bucketname, Key=self.keyname, Body=file, **kwargs
)
file.close()
@ -347,7 +382,9 @@ class FeedExporter:
category=ScrapyDeprecationWarning,
stacklevel=2,
)
uri = str(self.settings["FEED_URI"]) # handle pathlib.Path objects
uri = self.settings["FEED_URI"]
# handle pathlib.Path objects
uri = str(uri) if not isinstance(uri, Path) else uri.absolute().as_uri()
feed_options = {"format": self.settings.get("FEED_FORMAT", "jsonlines")}
self.feeds[uri] = feed_complete_default_values_from_settings(
feed_options, self.settings
@ -357,7 +394,8 @@ class FeedExporter:
# 'FEEDS' setting takes precedence over 'FEED_URI'
for uri, feed_options in self.settings.getdict("FEEDS").items():
uri = str(uri) # handle pathlib.Path objects
# handle pathlib.Path objects
uri = str(uri) if not isinstance(uri, Path) else uri.absolute().as_uri()
self.feeds[uri] = feed_complete_default_values_from_settings(
feed_options, self.settings
)

4
scrapy/http/headers.py
View File

@ -2,7 +2,7 @@ from collections.abc import Mapping
from w3lib.http import headers_dict_to_raw
from scrapy.utils.datatypes import CaselessDict
from scrapy.utils.datatypes import CaseInsensitiveDict, CaselessDict
from scrapy.utils.python import to_unicode
@ -88,7 +88,7 @@ class Headers(CaselessDict):
"""Return headers as a CaselessDict with unicode keys
and unicode values. Multiple values are joined with ','.
"""
return CaselessDict(
return CaseInsensitiveDict(
(
to_unicode(key, encoding=self.encoding),
to_unicode(b",".join(value), encoding=self.encoding),

4
scrapy/pipelines/files.py
View File

@ -28,7 +28,7 @@ from scrapy.http.request import NO_CALLBACK
from scrapy.pipelines.media import MediaPipeline
from scrapy.settings import Settings
from scrapy.utils.boto import is_botocore_available
from scrapy.utils.datatypes import CaselessDict
from scrapy.utils.datatypes import CaseInsensitiveDict
from scrapy.utils.ftp import ftp_store_file
from scrapy.utils.log import failure_to_exc_info
from scrapy.utils.misc import md5sum
@ -155,7 +155,7 @@ class S3FilesStore:
def _headers_to_botocore_kwargs(self, headers):
"""Convert headers to botocore keyword arguments."""
# This is required while we need to support both boto and botocore.
mapping = CaselessDict(
mapping = CaseInsensitiveDict(
{
"Content-Type": "ContentType",
"Cache-Control": "CacheControl",

14
scrapy/settings/__init__.py
View File

@ -75,6 +75,8 @@ class BaseSettings(MutableMapping):
highest priority will be retrieved.
"""
__default = object()
def __init__(self, values=None, priority="project"):
self.frozen = False
self.attributes = {}
@ -445,6 +447,18 @@ class BaseSettings(MutableMapping):
else:
p.text(pformat(self.copy_to_dict()))
def pop(self, name, default=__default):
try:
value = self.attributes[name]
except KeyError:
if default is self.__default:
raise
return SettingsAttribute(default, get_settings_priority("project"))
else:
self.__delitem__(name)
return value
class Settings(BaseSettings):
"""

15
scrapy/settings/default_settings.py
View File

@ -258,6 +258,21 @@ RETRY_ENABLED = True
RETRY_TIMES = 2 # initial response + 2 retries = 3 requests
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]
RETRY_PRIORITY_ADJUST = -1
RETRY_EXCEPTIONS = [
"twisted.internet.defer.TimeoutError",
"twisted.internet.error.TimeoutError",
"twisted.internet.error.DNSLookupError",
"twisted.internet.error.ConnectionRefusedError",
"twisted.internet.error.ConnectionDone",
"twisted.internet.error.ConnectError",
"twisted.internet.error.ConnectionLost",
"twisted.internet.error.TCPTimedOutError",
"twisted.web.client.ResponseFailed",
# OSError is raised by the HttpCompression middleware when trying to
# decompress an empty response
OSError,
"scrapy.core.downloader.handlers.http11.TunnelError",
]
ROBOTSTXT_OBEY = False
ROBOTSTXT_PARSER = "scrapy.robotstxt.ProtegoRobotParser"

60
scrapy/utils/datatypes.py
View File

@ -5,9 +5,13 @@ Python Standard Library.
This module must not depend on any module outside the Standard Library.
"""
import collections
import warnings
import weakref
from collections.abc import Mapping
from typing import Any, Optional, OrderedDict, Sequence, TypeVar
from typing import Any, AnyStr, Optional, OrderedDict, Sequence, TypeVar
from scrapy.exceptions import ScrapyDeprecationWarning
_KT = TypeVar("_KT")
_VT = TypeVar("_VT")
@ -16,6 +20,18 @@ _VT = TypeVar("_VT")
class CaselessDict(dict):
__slots__ = ()
def __new__(cls, *args, **kwargs):
from scrapy.http.headers import Headers
if issubclass(cls, CaselessDict) and not issubclass(cls, Headers):
warnings.warn(
"scrapy.utils.datatypes.CaselessDict is deprecated,"
" please use scrapy.utils.datatypes.CaseInsensitiveDict instead",
category=ScrapyDeprecationWarning,
stacklevel=2,
)
return super().__new__(cls, *args, **kwargs)
def __init__(self, seq=None):
super().__init__()
if seq:
@ -67,6 +83,48 @@ class CaselessDict(dict):
return dict.pop(self, self.normkey(key), *args)
class CaseInsensitiveDict(collections.UserDict):
"""A dict-like structure that accepts strings or bytes
as keys and allows case-insensitive lookups.
"""
def __init__(self, *args, **kwargs) -> None:
self._keys: dict = {}
super().__init__(*args, **kwargs)
def __getitem__(self, key: AnyStr) -> Any:
normalized_key = self._normkey(key)
return super().__getitem__(self._keys[normalized_key.lower()])
def __setitem__(self, key: AnyStr, value: Any) -> None:
normalized_key = self._normkey(key)
try:
lower_key = self._keys[normalized_key.lower()]
del self[lower_key]
except KeyError:
pass
super().__setitem__(normalized_key, self._normvalue(value))
self._keys[normalized_key.lower()] = normalized_key
def __delitem__(self, key: AnyStr) -> None:
normalized_key = self._normkey(key)
stored_key = self._keys.pop(normalized_key.lower())
super().__delitem__(stored_key)
def __contains__(self, key: AnyStr) -> bool: # type: ignore[override]
normalized_key = self._normkey(key)
return normalized_key.lower() in self._keys
def __repr__(self) -> str:
return f"<{self.__class__.__name__}: {super().__repr__()}>"
def _normkey(self, key: AnyStr) -> AnyStr:
return key
def _normvalue(self, value: Any) -> Any:
return value
class LocalCache(OrderedDict[_KT, _VT]):
"""Dictionary with a finite number of keys.

10
scrapy/utils/defer.py
View File

@ -357,8 +357,9 @@ def deferred_to_future(d: Deferred) -> Future:
class MySpider(Spider):
...
async def parse(self, response):
d = treq.get('https://example.com/additional')
additional_response = await deferred_to_future(d)
additional_request = scrapy.Request('https://example.org/price')
deferred = self.crawler.engine.download(additional_request)
additional_response = await deferred_to_future(deferred)
"""
return d.asFuture(_get_asyncio_event_loop())
@ -385,8 +386,9 @@ def maybe_deferred_to_future(d: Deferred) -> Union[Deferred, Future]:
class MySpider(Spider):
...
async def parse(self, response):
d = treq.get('https://example.com/additional')
extra_response = await maybe_deferred_to_future(d)
additional_request = scrapy.Request('https://example.org/price')
deferred = self.crawler.engine.download(additional_request)
additional_response = await maybe_deferred_to_future(deferred)
"""
if not is_asyncio_reactor_installed():
return d

2
scrapy/utils/gz.py
View File

@ -18,7 +18,7 @@ def gunzip(data: bytes) -> bytes:
try:
chunk = f.read1(8196)
output_list.append(chunk)
except (IOError, EOFError, struct.error):
except (OSError, EOFError, struct.error):
# complete only if there is some data, otherwise re-raise
# see issue 87 about catching struct.error
# some pages are quite small so output_list is empty

37
setup.py
View File

@ -1,26 +1,13 @@
from pathlib import Path
from pkg_resources import parse_version
from setuptools import __version__ as setuptools_version
from setuptools import find_packages, setup
version = (Path(__file__).parent / "scrapy/VERSION").read_text("ascii").strip()
def has_environment_marker_platform_impl_support():
"""Code extracted from 'pytest/setup.py'
https://github.com/pytest-dev/pytest/blob/7538680c/setup.py#L31
The first known release to support environment marker with range operators
it is 18.5, see:
https://setuptools.readthedocs.io/en/latest/history.html#id235
"""
return parse_version(setuptools_version) >= parse_version("18.5")
install_requires = [
"Twisted>=18.9.0",
"cryptography>=3.4.6",
"cryptography>=36.0.0",
"cssselect>=0.9.1",
"itemloaders>=1.0.1",
"parsel>=1.5.0",
@ -34,21 +21,12 @@ install_requires = [
"setuptools",
"packaging",
"tldextract",
"lxml>=4.3.0",
"lxml>=4.4.1",
]
extras_require = {}
cpython_dependencies = [
"PyDispatcher>=2.0.5",
]
if has_environment_marker_platform_impl_support():
extras_require[
':platform_python_implementation == "CPython"'
] = cpython_dependencies
extras_require[':platform_python_implementation == "PyPy"'] = [
"PyPyDispatcher>=2.1.0",
]
else:
install_requires.extend(cpython_dependencies)
extras_require = {
':platform_python_implementation == "CPython"': ["PyDispatcher>=2.0.5"],
':platform_python_implementation == "PyPy"': ["PyPyDispatcher>=2.1.0"],
}
setup(
@ -80,7 +58,6 @@ setup(
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
@ -91,7 +68,7 @@ setup(
"Topic :: Software Development :: Libraries :: Application Frameworks",
"Topic :: Software Development :: Libraries :: Python Modules",
],
python_requires=">=3.7",
python_requires=">=3.8",
install_requires=install_requires,
extras_require=extras_require,
)

19
tests/mockserver.py
View File

@ -1,11 +1,13 @@
import argparse
import json
import os
import random
import sys
from pathlib import Path
from shutil import rmtree
from subprocess import PIPE, Popen
from tempfile import mkdtemp
from typing import Dict
from urllib.parse import urlencode
from OpenSSL import SSL
@ -20,7 +22,6 @@ from twisted.web.static import File
from twisted.web.util import redirectTo
from scrapy.utils.python import to_bytes, to_unicode
from scrapy.utils.test import get_testenv
def getarg(request, name, default=None, type=None):
@ -32,6 +33,16 @@ def getarg(request, name, default=None, type=None):
return default
def get_mockserver_env() -> Dict[str, str]:
"""Return a OS environment dict suitable to run mockserver processes."""
tests_path = Path(__file__).parent.parent
pythonpath = str(tests_path) + os.pathsep + os.environ.get("PYTHONPATH", "")
env = os.environ.copy()
env["PYTHONPATH"] = pythonpath
return env
# most of the following resources are copied from twisted.web.test.test_webclient
class ForeverTakingResource(resource.Resource):
"""
@ -264,7 +275,7 @@ class MockServer:
self.proc = Popen(
[sys.executable, "-u", "-m", "tests.mockserver", "-t", "http"],
stdout=PIPE,
env=get_testenv(),
env=get_mockserver_env(),
)
http_address = self.proc.stdout.readline().strip().decode("ascii")
https_address = self.proc.stdout.readline().strip().decode("ascii")
@ -308,7 +319,7 @@ class MockDNSServer:
self.proc = Popen(
[sys.executable, "-u", "-m", "tests.mockserver", "-t", "dns"],
stdout=PIPE,
env=get_testenv(),
env=get_mockserver_env(),
)
self.host = "127.0.0.1"
self.port = int(
@ -331,7 +342,7 @@ class MockFTPServer:
self.proc = Popen(
[sys.executable, "-u", "-m", "tests.ftpserver", "-d", str(self.path)],
stderr=PIPE,
env=get_testenv(),
env=get_mockserver_env(),
)
for line in self.proc.stderr:
if b"starting FTP server" in line:

3
tests/test_cmdline/__init__.py
View File

@ -1,4 +1,5 @@
import json
import os
import pstats
import shutil
import sys
@ -14,6 +15,8 @@ from scrapy.utils.test import get_testenv
class CmdlineTest(unittest.TestCase):
def setUp(self):
self.env = get_testenv()
tests_path = Path(__file__).parent.parent
self.env["PYTHONPATH"] += os.pathsep + str(tests_path.parent)
self.env["SCRAPY_SETTINGS_MODULE"] = "tests.test_cmdline.settings"
def _execute(self, *new_args, **kwargs):

13
tests/test_crawler.py
View File

@ -1,11 +1,12 @@
import logging
import os
import platform
import subprocess
import sys
import warnings
from pathlib import Path
from pkg_resources import parse_version
from packaging.version import parse as parse_version
from pytest import mark, raises
from twisted import version as twisted_version
from twisted.internet import defer
@ -23,8 +24,8 @@ from scrapy.spiderloader import SpiderLoader
from scrapy.utils.log import configure_logging, get_scrapy_root_handler
from scrapy.utils.misc import load_object
from scrapy.utils.spider import DefaultSpider
from scrapy.utils.test import get_crawler, get_testenv
from tests.mockserver import MockServer
from scrapy.utils.test import get_crawler
from tests.mockserver import MockServer, get_mockserver_env
class BaseCrawlerTest(unittest.TestCase):
@ -289,12 +290,16 @@ class CrawlerRunnerHasSpider(unittest.TestCase):
class ScriptRunnerMixin:
script_dir: Path
cwd = os.getcwd()
def run_script(self, script_name: str, *script_args):
script_path = self.script_dir / script_name
args = [sys.executable, str(script_path)] + list(script_args)
p = subprocess.Popen(
args, env=get_testenv(), stdout=subprocess.PIPE, stderr=subprocess.PIPE
args,
env=get_mockserver_env(),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
stdout, stderr = p.communicate()
return stderr.decode("utf-8")

2
tests/test_downloader_handlers.py
View File

@ -129,7 +129,7 @@ class FileTestCase(unittest.TestCase):
def test_non_existent(self):
request = Request(f"file://{self.mktemp()}")
d = self.download_request(request, Spider("foo"))
return self.assertFailure(d, IOError)
return self.assertFailure(d, OSError)
class ContentLengthHeaderResource(resource.Resource):

4
tests/test_downloadermiddleware.py
View File

@ -70,7 +70,7 @@ class DefaultsTest(ManagerTestCase):
In particular when some website returns a 30x response with header
'Content-Encoding: gzip' giving as result the error below:
exceptions.IOError: Not a gzipped file
BadGzipFile: Not a gzipped file (...)
"""
req = Request("http://example.com")
@ -108,7 +108,7 @@ class DefaultsTest(ManagerTestCase):
"Location": "http://example.com/login",
},
)
self.assertRaises(IOError, self._download, request=req, response=resp)
self.assertRaises(OSError, self._download, request=req, response=resp)
class ResponseFromProcessRequestTest(ManagerTestCase):

39
tests/test_downloadermiddleware_retry.py
View File

@ -1,5 +1,6 @@
import logging
import unittest
import warnings
from testfixtures import LogCapture
from twisted.internet import defer
@ -15,6 +16,7 @@ from twisted.web.client import ResponseFailed
from scrapy.downloadermiddlewares.retry import RetryMiddleware, get_retry_request
from scrapy.exceptions import IgnoreRequest
from scrapy.http import Request, Response
from scrapy.settings.default_settings import RETRY_EXCEPTIONS
from scrapy.spiders import Spider
from scrapy.utils.test import get_crawler
@ -110,19 +112,48 @@ class RetryTest(unittest.TestCase):
== 2
)
def _test_retry_exception(self, req, exception):
def test_exception_to_retry_added(self):
exc = ValueError
settings_dict = {
"RETRY_EXCEPTIONS": list(RETRY_EXCEPTIONS) + [exc],
}
crawler = get_crawler(Spider, settings_dict=settings_dict)
mw = RetryMiddleware.from_crawler(crawler)
req = Request(f"http://www.scrapytest.org/{exc.__name__}")
self._test_retry_exception(req, exc("foo"), mw)
def test_exception_to_retry_customMiddleware(self):
exc = ValueError
with warnings.catch_warnings(record=True) as warns:
class MyRetryMiddleware(RetryMiddleware):
EXCEPTIONS_TO_RETRY = RetryMiddleware.EXCEPTIONS_TO_RETRY + (exc,)
self.assertEqual(len(warns), 1)
mw2 = MyRetryMiddleware.from_crawler(self.crawler)
req = Request(f"http://www.scrapytest.org/{exc.__name__}")
req = mw2.process_exception(req, exc("foo"), self.spider)
assert isinstance(req, Request)
self.assertEqual(req.meta["retry_times"], 1)
def _test_retry_exception(self, req, exception, mw=None):
if mw is None:
mw = self.mw
# first retry
req = self.mw.process_exception(req, exception, self.spider)
req = mw.process_exception(req, exception, self.spider)
assert isinstance(req, Request)
self.assertEqual(req.meta["retry_times"], 1)
# second retry
req = self.mw.process_exception(req, exception, self.spider)
req = mw.process_exception(req, exception, self.spider)
assert isinstance(req, Request)
self.assertEqual(req.meta["retry_times"], 2)
# discard it
req = self.mw.process_exception(req, exception, self.spider)
req = mw.process_exception(req, exception, self.spider)
self.assertEqual(req, None)

16
tests/test_exporters.py
View File

@ -90,6 +90,10 @@ class BaseItemExporterTest(unittest.TestCase):
if self.ie.__class__ is not BaseItemExporter:
raise
self.ie.finish_exporting()
# Delete the item exporter object, so that if it causes the output
# file handle to be closed, which should not be the case, follow-up
# interactions with the output file handle will surface the issue.
del self.ie
self._check_output()
def test_export_item(self):
@ -243,6 +247,7 @@ class PickleItemExporterTest(BaseItemExporterTest):
ie.export_item(i1)
ie.export_item(i2)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
f.seek(0)
self.assertEqual(self.item_class(**pickle.load(f)), i1)
self.assertEqual(self.item_class(**pickle.load(f)), i2)
@ -254,6 +259,7 @@ class PickleItemExporterTest(BaseItemExporterTest):
ie.start_exporting()
ie.export_item(item)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
self.assertEqual(pickle.loads(fp.getvalue()), item)
@ -279,6 +285,7 @@ class MarshalItemExporterTest(BaseItemExporterTest):
ie.start_exporting()
ie.export_item(item)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
fp.seek(0)
self.assertEqual(marshal.load(fp), item)
@ -314,6 +321,7 @@ class CsvItemExporterTest(BaseItemExporterTest):
ie.start_exporting()
ie.export_item(item)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
self.assertCsvEqual(fp.getvalue(), expected)
def test_header_export_all(self):
@ -345,6 +353,7 @@ class CsvItemExporterTest(BaseItemExporterTest):
ie.export_item(item)
ie.export_item(item)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
self.assertCsvEqual(
output.getvalue(), b"age,name\r\n22,John\xc2\xa3\r\n22,John\xc2\xa3\r\n"
)
@ -429,6 +438,7 @@ class XmlItemExporterTest(BaseItemExporterTest):
ie.start_exporting()
ie.export_item(item)
ie.finish_exporting()
del ie # See the first “del self.ie” in this file for context.
self.assertXmlEquivalent(fp.getvalue(), expected_value)
def _check_output(self):
@ -536,6 +546,7 @@ class JsonLinesItemExporterTest(BaseItemExporterTest):
self.ie.start_exporting()
self.ie.export_item(i3)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
self.assertEqual(exported, self._expected_nested)
@ -550,6 +561,7 @@ class JsonLinesItemExporterTest(BaseItemExporterTest):
self.ie.start_exporting()
self.ie.export_item(item)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
item["time"] = str(item["time"])
self.assertEqual(exported, item)
@ -575,6 +587,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
self.ie.export_item(item)
self.ie.export_item(item)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
self.assertEqual(
exported, [ItemAdapter(item).asdict(), ItemAdapter(item).asdict()]
@ -593,6 +606,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
self.ie.start_exporting()
self.ie.export_item(i3)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
expected = {
"name": "Jesus",
@ -607,6 +621,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
self.ie.start_exporting()
self.ie.export_item(i3)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
expected = {"name": "Jesus", "age": {"name": "Maria", "age": i1}}
self.assertEqual(exported, [expected])
@ -616,6 +631,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
self.ie.start_exporting()
self.ie.export_item(item)
self.ie.finish_exporting()
del self.ie # See the first “del self.ie” in this file for context.
exported = json.loads(to_unicode(self.output.getvalue()))
item["time"] = str(item["time"])
self.assertEqual(exported, [item])

120
tests/test_feedexport.py
View File

@ -36,6 +36,7 @@ from scrapy import signals
from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
from scrapy.exporters import CsvItemExporter, JsonItemExporter
from scrapy.extensions.feedexport import (
IS_BOTO3_AVAILABLE,
BlockingFeedStorage,
FeedExporter,
FeedSlot,
@ -236,8 +237,10 @@ class BlockingFeedStorageTest(unittest.TestCase):
class S3FeedStorageTest(unittest.TestCase):
def test_parse_credentials(self):
def setUp(self):
skip_if_no_boto()
def test_parse_credentials(self):
aws_credentials = {
"AWS_ACCESS_KEY_ID": "settings_key",
"AWS_SECRET_ACCESS_KEY": "settings_secret",
@ -273,8 +276,6 @@ class S3FeedStorageTest(unittest.TestCase):
@defer.inlineCallbacks
def test_store(self):
skip_if_no_boto()
settings = {
"AWS_ACCESS_KEY_ID": "access_key",
"AWS_SECRET_ACCESS_KEY": "secret_key",
@ -286,30 +287,39 @@ class S3FeedStorageTest(unittest.TestCase):
verifyObject(IFeedStorage, storage)
file = mock.MagicMock()
from botocore.stub import Stubber
with Stubber(storage.s3_client) as stub:
stub.add_response(
"put_object",
expected_params={
"Body": file,
"Bucket": bucket,
"Key": key,
},
service_response={},
)
if IS_BOTO3_AVAILABLE:
storage.s3_client = mock.MagicMock()
yield storage.store(file)
stub.assert_no_pending_responses()
self.assertEqual(
file.method_calls,
[
mock.call.seek(0),
# The call to read does not happen with Stubber
mock.call.close(),
],
storage.s3_client.upload_fileobj.call_args,
mock.call(Bucket=bucket, Key=key, Fileobj=file),
)
else:
from botocore.stub import Stubber
with Stubber(storage.s3_client) as stub:
stub.add_response(
"put_object",
expected_params={
"Body": file,
"Bucket": bucket,
"Key": key,
},
service_response={},
)
yield storage.store(file)
stub.assert_no_pending_responses()
self.assertEqual(
file.method_calls,
[
mock.call.seek(0),
# The call to read does not happen with Stubber
mock.call.close(),
],
)
def test_init_without_acl(self):
storage = S3FeedStorage("s3://mybucket/export.csv", "access_key", "secret_key")
@ -392,8 +402,7 @@ class S3FeedStorageTest(unittest.TestCase):
self.assertEqual(storage.endpoint_url, "https://example.com")
@defer.inlineCallbacks
def test_store_botocore_without_acl(self):
skip_if_no_boto()
def test_store_without_acl(self):
storage = S3FeedStorage(
"s3://mybucket/export.csv",
"access_key",
@ -405,11 +414,18 @@ class S3FeedStorageTest(unittest.TestCase):
storage.s3_client = mock.MagicMock()
yield storage.store(BytesIO(b"test file"))
self.assertNotIn("ACL", storage.s3_client.put_object.call_args[1])
if IS_BOTO3_AVAILABLE:
acl = (
storage.s3_client.upload_fileobj.call_args[1]
.get("ExtraArgs", {})
.get("ACL")
)
else:
acl = storage.s3_client.put_object.call_args[1].get("ACL")
self.assertIsNone(acl)
@defer.inlineCallbacks
def test_store_botocore_with_acl(self):
skip_if_no_boto()
def test_store_with_acl(self):
storage = S3FeedStorage(
"s3://mybucket/export.csv", "access_key", "secret_key", "custom-acl"
)
@ -419,9 +435,11 @@ class S3FeedStorageTest(unittest.TestCase):
storage.s3_client = mock.MagicMock()
yield storage.store(BytesIO(b"test file"))
self.assertEqual(
storage.s3_client.put_object.call_args[1].get("ACL"), "custom-acl"
)
if IS_BOTO3_AVAILABLE:
acl = storage.s3_client.upload_fileobj.call_args[1]["ExtraArgs"]["ACL"]
else:
acl = storage.s3_client.put_object.call_args[1]["ACL"]
self.assertEqual(acl, "custom-acl")
def test_overwrite_default(self):
with LogCapture() as log:
@ -889,15 +907,10 @@ class FeedExportTest(FeedExportTestBase):
@defer.inlineCallbacks
def test_stats_multiple_file(self):
settings = {
"AWS_ACCESS_KEY_ID": "access_key",
"AWS_SECRET_ACCESS_KEY": "secret_key",
"FEEDS": {
printf_escape(path_to_url(str(self._random_temp_filename()))): {
"format": "json",
},
"s3://bucket/key/foo.csv": {
"format": "csv",
},
"stdout:": {
"format": "xml",
},
@ -909,18 +922,12 @@ class FeedExportTest(FeedExportTestBase):
self.assertIn(
"feedexport/success_count/FileFeedStorage", crawler.stats.get_stats()
)
self.assertIn(
"feedexport/success_count/S3FeedStorage", crawler.stats.get_stats()
)
self.assertIn(
"feedexport/success_count/StdoutFeedStorage", crawler.stats.get_stats()
)
self.assertEqual(
crawler.stats.get_value("feedexport/success_count/FileFeedStorage"), 1
)
self.assertEqual(
crawler.stats.get_value("feedexport/success_count/S3FeedStorage"), 1
)
self.assertEqual(
crawler.stats.get_value("feedexport/success_count/StdoutFeedStorage"), 1
)
@ -2587,7 +2594,6 @@ class BatchDeliveriesTest(FeedExportTestBase):
@defer.inlineCallbacks
def test_s3_export(self):
skip_if_no_boto()
bucket = "mybucket"
items = [
self.MyItem({"foo": "bar1", "egg": "spam1"}),
@ -2752,6 +2758,31 @@ class FeedExportInitTest(unittest.TestCase):
with self.assertRaises(NotConfigured):
FeedExporter.from_crawler(crawler)
def test_absolute_pathlib_as_uri(self):
with tempfile.NamedTemporaryFile(suffix="json") as tmp:
settings = {
"FEEDS": {
Path(tmp.name).resolve(): {
"format": "json",
},
},
}
crawler = get_crawler(settings_dict=settings)
exporter = FeedExporter.from_crawler(crawler)
self.assertIsInstance(exporter, FeedExporter)
def test_relative_pathlib_as_uri(self):
settings = {
"FEEDS": {
Path("./items.json"): {
"format": "json",
},
},
}
crawler = get_crawler(settings_dict=settings)
exporter = FeedExporter.from_crawler(crawler)
self.assertIsInstance(exporter, FeedExporter)
class StdoutFeedStorageWithoutFeedOptions(StdoutFeedStorage):
def __init__(self, uri):
@ -2836,6 +2867,9 @@ class S3FeedStoragePreFeedOptionsTest(unittest.TestCase):
maxDiff = None
def setUp(self):
skip_if_no_boto()
def test_init(self):
settings_dict = {
"FEED_URI": "file:///tmp/foobar",

2
tests/test_mail.py
View File

@ -1,5 +1,3 @@
# coding=utf-8
import unittest
from email.charset import Charset
from io import BytesIO

12
tests/test_pipeline_files.py
View File

@ -226,12 +226,16 @@ class FilesPipelineTestCase(unittest.TestCase):
class FilesPipelineTestCaseFieldsMixin:
def setUp(self):
self.tempdir = mkdtemp()
def tearDown(self):
rmtree(self.tempdir)
def test_item_fields_default(self):
url = "http://www.example.com/files/1.txt"
item = self.item_class(name="item1", file_urls=[url])
pipeline = FilesPipeline.from_settings(
Settings({"FILES_STORE": "s3://example/files/"})
)
pipeline = FilesPipeline.from_settings(Settings({"FILES_STORE": self.tempdir}))
requests = list(pipeline.get_media_requests(item, None))
self.assertEqual(requests[0].url, url)
results = [(True, {"url": url})]
@ -246,7 +250,7 @@ class FilesPipelineTestCaseFieldsMixin:
pipeline = FilesPipeline.from_settings(
Settings(
{
"FILES_STORE": "s3://example/files/",
"FILES_STORE": self.tempdir,
"FILES_URLS_FIELD": "custom_file_urls",
"FILES_RESULT_FIELD": "custom_files",
}

3
tests/test_proxy_connect.py
View File

@ -21,8 +21,6 @@ class MitmProxy:
auth_pass = "scrapy"
def start(self):
from scrapy.utils.test import get_testenv
script = """
import sys
from mitmproxy.tools.main import mitmdump
@ -46,7 +44,6 @@ sys.exit(mitmdump())
"--ssl-insecure",
],
stdout=PIPE,
env=get_testenv(),
)
line = self.proc.stdout.readline().decode("utf-8")
host_port = re.search(r"listening at http://([^:]+:\d+)", line).group(1)

1
tests/test_robotstxt_interface.py
View File

@ -1,4 +1,3 @@
# coding=utf-8
from twisted.trial import unittest

29
tests/test_settings/__init__.py
View File

@ -451,6 +451,35 @@ class SettingsTest(unittest.TestCase):
self.assertIsInstance(myhandler_instance, FileDownloadHandler)
self.assertTrue(hasattr(myhandler_instance, "download_request"))
def test_pop_item_with_default_value(self):
settings = Settings()
with self.assertRaises(KeyError):
settings.pop("DUMMY_CONFIG")
dummy_config = settings.pop("DUMMY_CONFIG", "dummy_value")
self.assertEqual(
repr(dummy_config), "<SettingsAttribute value='dummy_value' priority=20>"
)
self.assertEqual(dummy_config.value, "dummy_value")
def test_pop_item_with_immutable_settings(self):
settings = Settings(
{"DUMMY_CONFIG": "dummy_value", "OTHER_DUMMY_CONFIG": "other_dummy_value"}
)
self.assertEqual(settings.pop("DUMMY_CONFIG").value, "dummy_value")
settings.freeze()
with self.assertRaises(TypeError) as error:
settings.pop("OTHER_DUMMY_CONFIG")
self.assertEqual(
str(error.exception), "Trying to modify an immutable Settings object"
)
if __name__ == "__main__":
unittest.main()

5
tests/test_utils_asyncio.py
View File

@ -4,6 +4,7 @@ from unittest import TestCase
from pytest import mark
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.reactor import (
install_reactor,
is_asyncio_reactor_installed,
@ -29,6 +30,8 @@ class AsyncioTest(TestCase):
assert original_reactor == reactor
@mark.only_asyncio()
@deferred_f_from_coro_f
async def test_set_asyncio_event_loop(self):
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
assert set_asyncio_event_loop() is asyncio.get_running_loop()
assert set_asyncio_event_loop(None) is asyncio.get_running_loop()

86
tests/test_utils_datatypes.py
View File

@ -1,9 +1,13 @@
import copy
import unittest
import warnings
from collections.abc import Mapping, MutableMapping
from typing import Iterator
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.http import Request
from scrapy.utils.datatypes import (
CaseInsensitiveDict,
CaselessDict,
LocalCache,
LocalWeakReferencedCache,
@ -14,16 +18,16 @@ from scrapy.utils.python import garbage_collect
__doctests__ = ["scrapy.utils.datatypes"]
class CaselessDictTest(unittest.TestCase):
class CaseInsensitiveDictMixin:
def test_init_dict(self):
seq = {"red": 1, "black": 3}
d = CaselessDict(seq)
d = self.dict_class(seq)
self.assertEqual(d["red"], 1)
self.assertEqual(d["black"], 3)
def test_init_pair_sequence(self):
seq = (("red", 1), ("black", 3))
d = CaselessDict(seq)
d = self.dict_class(seq)
self.assertEqual(d["red"], 1)
self.assertEqual(d["black"], 3)
@ -42,7 +46,7 @@ class CaselessDictTest(unittest.TestCase):
return len(self._d)
seq = MyMapping(red=1, black=3)
d = CaselessDict(seq)
d = self.dict_class(seq)
self.assertEqual(d["red"], 1)
self.assertEqual(d["black"], 3)
@ -67,12 +71,12 @@ class CaselessDictTest(unittest.TestCase):
return len(self._d)
seq = MyMutableMapping(red=1, black=3)
d = CaselessDict(seq)
d = self.dict_class(seq)
self.assertEqual(d["red"], 1)
self.assertEqual(d["black"], 3)
def test_caseless(self):
d = CaselessDict()
d = self.dict_class()
d["key_Lower"] = 1
self.assertEqual(d["KEy_loWer"], 1)
self.assertEqual(d.get("KEy_loWer"), 1)
@ -82,7 +86,7 @@ class CaselessDictTest(unittest.TestCase):
self.assertEqual(d.get("key_Lower"), 3)
def test_delete(self):
d = CaselessDict({"key_lower": 1})
d = self.dict_class({"key_lower": 1})
del d["key_LOWER"]
self.assertRaises(KeyError, d.__getitem__, "key_LOWER")
self.assertRaises(KeyError, d.__getitem__, "key_lower")
@ -107,15 +111,15 @@ class CaselessDictTest(unittest.TestCase):
def test_fromkeys(self):
keys = ("a", "b")
d = CaselessDict.fromkeys(keys)
d = self.dict_class.fromkeys(keys)
self.assertEqual(d["A"], None)
self.assertEqual(d["B"], None)
d = CaselessDict.fromkeys(keys, 1)
d = self.dict_class.fromkeys(keys, 1)
self.assertEqual(d["A"], 1)
self.assertEqual(d["B"], 1)
instance = CaselessDict()
instance = self.dict_class()
d = instance.fromkeys(keys)
self.assertEqual(d["A"], None)
self.assertEqual(d["B"], None)
@ -125,31 +129,35 @@ class CaselessDictTest(unittest.TestCase):
self.assertEqual(d["B"], 1)
def test_contains(self):
d = CaselessDict()
d = self.dict_class()
d["a"] = 1
assert "a" in d
assert "A" in d
def test_pop(self):
d = CaselessDict()
d = self.dict_class()
d["a"] = 1
self.assertEqual(d.pop("A"), 1)
self.assertRaises(KeyError, d.pop, "A")
def test_normkey(self):
class MyDict(CaselessDict):
def normkey(self, key):
class MyDict(self.dict_class):
def _normkey(self, key):
return key.title()
normkey = _normkey # deprecated CaselessDict class
d = MyDict()
d["key-one"] = 2
self.assertEqual(list(d.keys()), ["Key-One"])
def test_normvalue(self):
class MyDict(CaselessDict):
def normvalue(self, value):
class MyDict(self.dict_class):
def _normvalue(self, value):
if value is not None:
return value + 1
normvalue = _normvalue # deprecated CaselessDict class
d = MyDict({"key": 1})
self.assertEqual(d["key"], 2)
self.assertEqual(d.get("key"), 2)
@ -174,11 +182,51 @@ class CaselessDictTest(unittest.TestCase):
self.assertEqual(d.get("key"), 2)
def test_copy(self):
h1 = CaselessDict({"header1": "value"})
h1 = self.dict_class({"header1": "value"})
h2 = copy.copy(h1)
assert isinstance(h2, self.dict_class)
self.assertEqual(h1, h2)
self.assertEqual(h1.get("header1"), h2.get("header1"))
assert isinstance(h2, CaselessDict)
self.assertEqual(h1.get("header1"), h2.get("HEADER1"))
h3 = h1.copy()
assert isinstance(h3, self.dict_class)
self.assertEqual(h1, h3)
self.assertEqual(h1.get("header1"), h3.get("header1"))
self.assertEqual(h1.get("header1"), h3.get("HEADER1"))
class CaseInsensitiveDictTest(CaseInsensitiveDictMixin, unittest.TestCase):
dict_class = CaseInsensitiveDict
def test_repr(self):
d1 = self.dict_class({"foo": "bar"})
self.assertEqual(repr(d1), "<CaseInsensitiveDict: {'foo': 'bar'}>")
d2 = self.dict_class({"AsDf": "QwErTy", "FoO": "bAr"})
self.assertEqual(
repr(d2), "<CaseInsensitiveDict: {'AsDf': 'QwErTy', 'FoO': 'bAr'}>"
)
def test_iter(self):
d = self.dict_class({"AsDf": "QwErTy", "FoO": "bAr"})
iterkeys = iter(d)
self.assertIsInstance(iterkeys, Iterator)
self.assertEqual(list(iterkeys), ["AsDf", "FoO"])
class CaselessDictTest(CaseInsensitiveDictMixin, unittest.TestCase):
dict_class = CaselessDict
def test_deprecation_message(self):
with warnings.catch_warnings(record=True) as caught:
self.dict_class({"foo": "bar"})
self.assertEqual(len(caught), 1)
self.assertTrue(issubclass(caught[0].category, ScrapyDeprecationWarning))
self.assertEqual(
"scrapy.utils.datatypes.CaselessDict is deprecated,"
" please use scrapy.utils.datatypes.CaseInsensitiveDict instead",
str(caught[0].message),
)
class SequenceExcludeTest(unittest.TestCase):

2
tests/test_utils_gz.py
View File

@ -28,7 +28,7 @@ class GunzipTest(unittest.TestCase):
def test_gunzip_no_gzip_file_raises(self):
self.assertRaises(
IOError, gunzip, (SAMPLEDIR / "feed-sample1.xml").read_bytes()
OSError, gunzip, (SAMPLEDIR / "feed-sample1.xml").read_bytes()
)
def test_gunzip_truncated_short(self):

4
tests/test_utils_iterators.py
View File

@ -346,8 +346,8 @@ class UtilsCsvTestCase(unittest.TestCase):
# explicit type check cuz' we no like stinkin' autocasting! yarrr
for result_row in result:
self.assertTrue(all((isinstance(k, str) for k in result_row.keys())))
self.assertTrue(all((isinstance(v, str) for v in result_row.values())))
self.assertTrue(all(isinstance(k, str) for k in result_row.keys()))
self.assertTrue(all(isinstance(v, str) for v in result_row.values()))
def test_csviter_delimiter(self):
body = get_testdata("feeds", "feed-sample3.csv").replace(b",", b"\t")

55
tox.ini
View File

@ -16,10 +16,6 @@ deps =
#mitmproxy >= 5.3.0; python_version >= '3.9' and implementation_name != 'pypy'
# The tests hang with mitmproxy 8.0.0: https://github.com/scrapy/scrapy/issues/5454
mitmproxy >= 4.0.4, < 8; python_version < '3.9' and implementation_name != 'pypy'
# newer markupsafe is incompatible with deps of old mitmproxy (which we get on Python 3.7 and lower)
markupsafe < 2.1.0; python_version < '3.8' and implementation_name != 'pypy'
# Extras
botocore>=1.4.87
passenv =
S3_TEST_FILE_URI
AWS_ACCESS_KEY_ID
@ -73,7 +69,7 @@ commands =
[pinned]
deps =
cryptography==3.4.6
cryptography==36.0.0
cssselect==0.9.1
h2==3.0
itemadapter==0.1.0
@ -85,16 +81,11 @@ deps =
Twisted[http2]==18.9.0
w3lib==1.17.0
zope.interface==5.1.0
lxml==4.3.0
lxml==4.4.1
-rtests/requirements.txt
# mitmproxy 4.0.4+ requires upgrading some of the pinned dependencies
# above, hence we do not install it in pinned environments at the moment
# Extras
botocore==1.4.87
google-cloud-storage==1.29.0
Pillow==7.1.0
setenv =
_SCRAPY_PINNED=true
install_command =
@ -103,7 +94,7 @@ commands =
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 scrapy tests}
[testenv:pinned]
basepython = python3.7
basepython = python3.8
deps =
{[pinned]deps}
PyDispatcher==2.0.5
@ -126,14 +117,27 @@ commands = {[pinned]commands}
basepython = python3
deps =
{[testenv]deps}
boto
boto3
google-cloud-storage
# Twisted[http2] currently forces old mitmproxy because of h2 version
# restrictions in their deps, so we need to pin old markupsafe here too.
markupsafe < 2.1.0
robotexclusionrulesparser
Pillow>=4.0.0
Twisted[http2]>=17.9.0
Pillow
Twisted[http2]
[testenv:extra-deps-pinned]
basepython = python3.8
deps =
{[pinned]deps}
boto3==1.20.0
google-cloud-storage==1.29.0
Pillow==7.1.0
robotexclusionrulesparser==1.6.2
install_command = {[pinned]install_command}
setenv =
{[pinned]setenv}
commands = {[pinned]commands}
[testenv:asyncio]
commands =
@ -193,3 +197,24 @@ deps = {[docs]deps}
setenv = {[docs]setenv}
commands =
sphinx-build -W -b linkcheck . {envtmpdir}/linkcheck
# Run S3 tests with botocore installed but without boto3.
[testenv:botocore]
deps =
{[testenv]deps}
botocore>=1.4.87
commands =
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -k s3}
[testenv:botocore-pinned]
basepython = python3.8
deps =
{[pinned]deps}
botocore==1.4.87
install_command = {[pinned]install_command}
setenv =
{[pinned]setenv}
commands =
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -k s3}