RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased

This commit is contained in:
Andrey Rakhmatullin 2021-08-16 20:20:43 +05:00
parent 7306a81188 2814e0e197
commit 5d4b232de8
23 changed files with 292 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.github/workflows/tests-ubuntu.yml vendored
View File

@ -41,7 +41,12 @@ jobs:
env:
TOXENV: extra-deps
TOX_PIP_VERSION: 20.3.3
- python-version: 3.9
# 3.10-pre
- python-version: "3.10.0-beta.4"
env:
TOXENV: py
- python-version: "3.10.0-beta.4"
env:
TOXENV: asyncio
@ -54,7 +59,7 @@ jobs:
python-version: ${{ matrix.python-version }}
- name: Install system libraries
if: matrix.python-version == 'pypy3' || contains(matrix.env.TOXENV, 'pinned')
if: matrix.python-version == 'pypy3' || contains(matrix.env.TOXENV, 'pinned') || matrix.python-version == '3.10.0-beta.4'
run: |
sudo apt-get update
# libxml2 2.9.12 from ondrej/php PPA breaks lxml so we pin it to the bionic-updates repo version

40
docs/topics/dynamic-content.rst
View File

@ -246,24 +246,46 @@ Using a headless browser
========================
A `headless browser`_ is a special web browser that provides an API for
automation.
automation. By installing the :ref:`asyncio reactor <install-asyncio>`,
it is possible to integrate ``asyncio``-based libraries which handle headless browsers.
The easiest way to use a headless browser with Scrapy is to use Selenium_,
along with `scrapy-selenium`_ for seamless integration.
One such library is `playwright-python`_ (an official Python port of `playwright`_).
The following is a simple snippet to illustrate its usage within a Scrapy spider::
import scrapy
from playwright.async_api import async_playwright
class PlaywrightSpider(scrapy.Spider):
name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader
async def parse(self, response):
async with async_playwright() as pw:
browser = await pw.chromium.launch()
page = await browser.new_page()
await page.goto("https:/example.org")
title = await page.title()
return {"title": title}
However, using `playwright-python`_ directly as in the above example
circumvents most of the Scrapy components (middlewares, dupefilter, etc).
We recommend using `scrapy-playwright`_ for a better integration.
.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29
.. _chompjs: https://github.com/Nykakin/chompjs
.. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _Splash: https://github.com/scrapinghub/splash
.. _chompjs: https://github.com/Nykakin/chompjs
.. _curl: https://curl.haxx.se/
.. _headless browser: https://en.wikipedia.org/wiki/Headless_browser
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _js2xml: https://github.com/scrapinghub/js2xml
.. _playwright-python: https://github.com/microsoft/playwright-python
.. _playwright: https://github.com/microsoft/playwright
.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/
.. _pytesseract: https://github.com/madmaze/pytesseract
.. _scrapy-selenium: https://github.com/clemfromspace/scrapy-selenium
.. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
.. _Selenium: https://www.selenium.dev/
.. _Splash: https://github.com/scrapinghub/splash
.. _tabula-py: https://github.com/chezou/tabula-py
.. _wget: https://www.gnu.org/software/wget/
.. _wgrep: https://github.com/stav/wgrep
.. _wgrep: https://github.com/stav/wgrep

5
docs/topics/extensions.rst
View File

@ -323,6 +323,11 @@ domain has finished scraping, including the Scrapy stats collected. The email
will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS`
setting.
Emails can be sent using the :class:`~scrapy.mail.MailSender` class. To see a
full list of parameters, including examples on how to instantiate
:class:`~scrapy.mail.MailSender` and use mail settings, see
:ref:`topics-email`.
.. module:: scrapy.extensions.debug
:synopsis: Extensions for debugging Scrapy

7
docs/topics/feed-exports.rst
View File

@ -135,7 +135,7 @@ Here are some examples to illustrate:
- ``s3://mybucket/scraping/feeds/%(name)s/%(time)s.json``
.. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
.. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
they can also be used as storage URI parameters.
@ -200,6 +200,9 @@ passed through the following settings:
- :setting:`AWS_ACCESS_KEY_ID`
- :setting:`AWS_SECRET_ACCESS_KEY`
- :setting:`AWS_SESSION_TOKEN` (only needed for `temporary security credentials`_)
.. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys
You can also define a custom ACL and custom endpoint for exported feeds using this setting:
@ -357,7 +360,7 @@ For instance::
'item_export_kwargs': {
'export_empty_fields': True,
},
},
},
'/home/user/documents/items.xml': {
'format': 'xml',
'fields': ['name', 'price'],

88
docs/topics/media-pipeline.rst
View File

@ -111,25 +111,82 @@ For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::
IMAGES_STORE = '/path/to/valid/dir'
.. _topics-file-naming:
File Naming
===========
Default File Naming
-------------------
By default, files are stored using an `SHA-1 hash`_ of their URLs for the file names.
For example, the following image URL::
http://www.example.com/image.jpg
Whose ``SHA-1 hash`` is::
3afec3b4765f8f0a07b78f98c07b83f013567a0a
Will be downloaded and stored using your chosen :ref:`storage method <topics-supported-storage>` and the following file name::
3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
Custom File Naming
-------------------
You may wish to use a different calculated file name for saved files.
For example, classifying an image by including meta in the file name.
Customize file names by overriding the ``file_path`` method of your
media pipeline.
For example, an image pipeline with image URL::
http://www.example.com/product/images/large/front/0000000004166
Can be processed into a file name with a condensed hash and the perspective
``front``::
00b08510e4_front.jpg
By overriding ``file_path`` like this:
.. code-block:: python
import hashlib
from os.path import splitext
def file_path(self, request, response=None, info=None, *, item=None):
image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)
image_perspective = request.url.split('/')[-2]
image_filename = f'{image_url_hash}_{image_perspective}.jpg'
return image_filename
.. warning::
If your custom file name scheme relies on meta data that can vary between
scrapes it may lead to unexpected re-downloading of existing media using
new file names.
For example, if your custom file name scheme uses a product title and the
site changes an item's product title between scrapes, Scrapy will re-download
the same media using updated file names.
For more information about the ``file_path`` method, see :ref:`topics-media-pipeline-override`.
.. _topics-supported-storage:
Supported Storage
=================
File system storage
-------------------
The files are stored using a `SHA1 hash`_ of their URLs for the file names.
File system storage will save files to the following path::
For example, the following image URL::
http://www.example.com/image.jpg
Whose ``SHA1 hash`` is::
3afec3b4765f8f0a07b78f98c07b83f013567a0a
Will be downloaded and stored in the following file::
<IMAGES_STORE>/full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
<IMAGES_STORE>/full/<FILE_NAME>
Where:
@ -139,6 +196,9 @@ Where:
* ``full`` is a sub-directory to separate full images from thumbnails (if
used). For more info see :ref:`topics-images-thumbnails`.
* ``<FILE_NAME>`` is the file name assigned to the file. For more info see :ref:`topics-file-naming`.
.. _media-pipeline-ftp:
FTP server storage
@ -353,9 +413,9 @@ Where:
* ``<size_name>`` is the one specified in the :setting:`IMAGES_THUMBS`
dictionary keys (``small``, ``big``, etc)
* ``<image_id>`` is the `SHA1 hash`_ of the image url
* ``<image_id>`` is the `SHA-1 hash`_ of the image url
.. _SHA1 hash: https://en.wikipedia.org/wiki/SHA_hash_functions
.. _SHA-1 hash: https://en.wikipedia.org/wiki/SHA_hash_functions
Example of image files stored using ``small`` and ``big`` thumbnail names::

7
docs/topics/request-response.rst
View File

@ -670,9 +670,6 @@ Response objects
.. autoclass:: Response
A :class:`Response` object represents an HTTP response, which is usually
downloaded (by the Downloader) and fed to the Spiders for processing.
:param url: the URL of this response
:type url: str
@ -829,6 +826,8 @@ Response objects
handlers, i.e. for ``http(s)`` responses. For other handlers,
:attr:`protocol` is always ``None``.
.. autoattribute:: Response.attributes
.. method:: Response.copy()
Returns a new Response which is a copy of this Response.
@ -925,6 +924,8 @@ TextResponse objects
A :class:`~scrapy.Selector` instance using the response as
target. The selector is lazily instantiated on first access.
.. autoattribute:: TextResponse.attributes
:class:`TextResponse` objects support the following methods in addition to
the standard :class:`Response` ones:

23
docs/topics/settings.rst
View File

@ -204,6 +204,19 @@ Default: ``None``
The AWS secret key used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
.. setting:: AWS_SESSION_TOKEN
AWS_SESSION_TOKEN
-----------------
Default: ``None``
The AWS security token used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
`temporary security credentials`_.
.. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys
.. setting:: AWS_ENDPOINT_URL
AWS_ENDPOINT_URL
@ -976,6 +989,16 @@ Default: ``{}``
A dict containing the pipelines enabled by default in Scrapy. You should never
modify this setting in your project, modify :setting:`ITEM_PIPELINES` instead.
.. setting:: JOBDIR
JOBDIR
------
Default: ``''``
A string indicating the directory for storing the state of a crawl when
:ref:`pausing and resuming crawls <topics-jobs>`.
.. setting:: LOG_ENABLED
LOG_ENABLED

19
scrapy/core/downloader/handlers/s3.py
View File

@ -1,5 +1,3 @@
from urllib.parse import unquote
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
from scrapy.exceptions import NotConfigured
from scrapy.utils.boto import is_botocore_available
@ -12,6 +10,7 @@ class S3DownloadHandler:
def __init__(self, settings, *,
crawler=None,
aws_access_key_id=None, aws_secret_access_key=None,
aws_session_token=None,
httpdownloadhandler=HTTPDownloadHandler, **kw):
if not is_botocore_available():
raise NotConfigured('missing botocore library')
@ -20,6 +19,8 @@ class S3DownloadHandler:
aws_access_key_id = settings['AWS_ACCESS_KEY_ID']
if not aws_secret_access_key:
aws_secret_access_key = settings['AWS_SECRET_ACCESS_KEY']
if not aws_session_token:
aws_session_token = settings['AWS_SESSION_TOKEN']
# If no credentials could be found anywhere,
# consider this an anonymous connection request by default;
@ -38,7 +39,7 @@ class S3DownloadHandler:
if not self.anon:
SignerCls = botocore.auth.AUTH_TYPE_MAPS['s3']
self._signer = SignerCls(botocore.credentials.Credentials(
aws_access_key_id, aws_secret_access_key))
aws_access_key_id, aws_secret_access_key, aws_session_token))
_http_handler = create_instance(
objcls=httpdownloadhandler,
@ -59,7 +60,7 @@ class S3DownloadHandler:
url = f'{scheme}://{bucket}.s3.amazonaws.com{path}'
if self.anon:
request = request.replace(url=url)
elif self._signer is not None:
else:
import botocore.awsrequest
awsrequest = botocore.awsrequest.AWSRequest(
method=request.method,
@ -69,14 +70,4 @@ class S3DownloadHandler:
self._signer.add_auth(awsrequest)
request = request.replace(
url=url, headers=awsrequest.headers.items())
else:
signed_headers = self.conn.make_request(
method=request.method,
bucket=bucket,
key=unquote(p.path),
query_args=unquote(p.query),
headers=request.headers,
data=request.body,
)
request = request.replace(url=url, headers=signed_headers)
return self._download_http(request, spider)

5
scrapy/extensions/feedexport.py
View File

@ -154,13 +154,14 @@ class FileFeedStorage:
class S3FeedStorage(BlockingFeedStorage):
def __init__(self, uri, access_key=None, secret_key=None, acl=None, endpoint_url=None, *,
feed_options=None):
feed_options=None, session_token=None):
if not is_botocore_available():
raise NotConfigured('missing botocore library')
u = urlparse(uri)
self.bucketname = u.hostname
self.access_key = u.username or access_key
self.secret_key = u.password or secret_key
self.session_token = session_token
self.keyname = u.path[1:] # remove first "/"
self.acl = acl
self.endpoint_url = endpoint_url
@ -169,6 +170,7 @@ class S3FeedStorage(BlockingFeedStorage):
self.s3_client = session.create_client(
's3', aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key,
aws_session_token=self.session_token,
endpoint_url=self.endpoint_url)
if feed_options and feed_options.get('overwrite', True) is False:
logger.warning('S3 does not support appending to files. To '
@ -182,6 +184,7 @@ class S3FeedStorage(BlockingFeedStorage):
uri,
access_key=crawler.settings['AWS_ACCESS_KEY_ID'],
secret_key=crawler.settings['AWS_SECRET_ACCESS_KEY'],
session_token=crawler.settings['AWS_SESSION_TOKEN'],
acl=crawler.settings['FEED_STORAGE_S3_ACL'] or None,
endpoint_url=crawler.settings['AWS_ENDPOINT_URL'] or None,
feed_options=feed_options,

23
scrapy/http/response/__init__.py
View File

@ -4,7 +4,7 @@ responses in Scrapy.
See documentation in docs/topics/request-response.rst
"""
from typing import Generator
from typing import Generator, Tuple
from urllib.parse import urljoin
from scrapy.exceptions import NotSupported
@ -16,6 +16,19 @@ from scrapy.utils.trackref import object_ref
class Response(object_ref):
"""An object that represents an HTTP response, which is usually
downloaded (by the Downloader) and fed to the Spiders for processing.
"""
attributes: Tuple[str, ...] = (
"url", "status", "headers", "body", "flags", "request", "certificate", "ip_address", "protocol",
)
"""A tuple of :class:`str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
``__init__`` method.
Currently used by :meth:`Response.replace`.
"""
def __init__(
self,
@ -97,12 +110,8 @@ class Response(object_ref):
return self.replace()
def replace(self, *args, **kwargs):
"""Create a new Response with the same attributes except for those
given new values.
"""
for x in [
"url", "status", "headers", "body", "request", "flags", "certificate", "ip_address", "protocol",
]:
"""Create a new Response with the same attributes except for those given new values"""
for x in self.attributes:
kwargs.setdefault(x, getattr(self, x))
cls = kwargs.pop('cls', self.__class__)
return cls(*args, **kwargs)

8
scrapy/http/response/text.py
View File

@ -8,7 +8,7 @@ See documentation in docs/topics/request-response.rst
import json
import warnings
from contextlib import suppress
from typing import Generator
from typing import Generator, Tuple
from urllib.parse import urljoin
import parsel
@ -30,6 +30,8 @@ class TextResponse(Response):
_DEFAULT_ENCODING = 'ascii'
_cached_decoded_json = _NONE
attributes: Tuple[str, ...] = Response.attributes + ("encoding",)
def __init__(self, *args, **kwargs):
self._encoding = kwargs.pop('encoding', None)
self._cached_benc = None
@ -53,10 +55,6 @@ class TextResponse(Response):
else:
super()._set_body(body)
def replace(self, *args, **kwargs):
kwargs.setdefault('encoding', self.encoding)
return Response.replace(self, *args, **kwargs)
@property
def encoding(self):
return self._declared_encoding() or self._body_inferred_encoding()

3
scrapy/pipelines/files.py
View File

@ -79,6 +79,7 @@ class FSFilesStore:
class S3FilesStore:
AWS_ACCESS_KEY_ID = None
AWS_SECRET_ACCESS_KEY = None
AWS_SESSION_TOKEN = None
AWS_ENDPOINT_URL = None
AWS_REGION_NAME = None
AWS_USE_SSL = None
@ -98,6 +99,7 @@ class S3FilesStore:
's3',
aws_access_key_id=self.AWS_ACCESS_KEY_ID,
aws_secret_access_key=self.AWS_SECRET_ACCESS_KEY,
aws_session_token=self.AWS_SESSION_TOKEN,
endpoint_url=self.AWS_ENDPOINT_URL,
region_name=self.AWS_REGION_NAME,
use_ssl=self.AWS_USE_SSL,
@ -349,6 +351,7 @@ class FilesPipeline(MediaPipeline):
s3store = cls.STORE_SCHEMES['s3']
s3store.AWS_ACCESS_KEY_ID = settings['AWS_ACCESS_KEY_ID']
s3store.AWS_SECRET_ACCESS_KEY = settings['AWS_SECRET_ACCESS_KEY']
s3store.AWS_SESSION_TOKEN = settings['AWS_SESSION_TOKEN']
s3store.AWS_ENDPOINT_URL = settings['AWS_ENDPOINT_URL']
s3store.AWS_REGION_NAME = settings['AWS_REGION_NAME']
s3store.AWS_USE_SSL = settings['AWS_USE_SSL']

1
scrapy/pipelines/images.py
View File

@ -92,6 +92,7 @@ class ImagesPipeline(FilesPipeline):
s3store = cls.STORE_SCHEMES['s3']
s3store.AWS_ACCESS_KEY_ID = settings['AWS_ACCESS_KEY_ID']
s3store.AWS_SECRET_ACCESS_KEY = settings['AWS_SECRET_ACCESS_KEY']
s3store.AWS_SESSION_TOKEN = settings['AWS_SESSION_TOKEN']
s3store.AWS_ENDPOINT_URL = settings['AWS_ENDPOINT_URL']
s3store.AWS_REGION_NAME = settings['AWS_REGION_NAME']
s3store.AWS_USE_SSL = settings['AWS_USE_SSL']

2
scrapy/utils/misc.py
View File

@ -138,7 +138,7 @@ def md5sum(file):
def rel_has_nofollow(rel):
"""Return True if link rel attribute has nofollow type"""
return rel is not None and 'nofollow' in rel.split()
return rel is not None and 'nofollow' in rel.replace(',', ' ').split()
def create_instance(objcls, settings, crawler, *args, **kwargs):

4
scrapy/utils/signal.py
View File

@ -1,5 +1,5 @@
"""Helper functions for working with signals"""
import collections
import collections.abc
import logging
from twisted.internet.defer import DeferredList, Deferred
@ -21,7 +21,7 @@ def send_catch_log(signal=Any, sender=Anonymous, *arguments, **named):
Failures instead of exceptions.
"""
dont_log = named.pop('dont_log', ())
dont_log = tuple(dont_log) if isinstance(dont_log, collections.Sequence) else (dont_log,)
dont_log = tuple(dont_log) if isinstance(dont_log, collections.abc.Sequence) else (dont_log,)
dont_log += (StopDownload, )
spider = named.get('spider', None)
responses = []

2
tests/CrawlerProcess/asyncio_deferred_signal.py
View File

@ -1,5 +1,6 @@
import asyncio
import sys
from typing import Optional
from scrapy import Spider
from scrapy.crawler import CrawlerProcess
@ -31,6 +32,7 @@ class UrlSpider(Spider):
if __name__ == "__main__":
ASYNCIO_EVENT_LOOP: Optional[str]
try:
ASYNCIO_EVENT_LOOP = sys.argv[1]
except IndexError:

0
tests/requirements-py3.txt → tests/requirements.txt
View File

29
tests/test_feedexport.py
View File

@ -244,7 +244,8 @@ class S3FeedStorageTest(unittest.TestCase):
def test_parse_credentials(self):
skip_if_no_boto()
aws_credentials = {'AWS_ACCESS_KEY_ID': 'settings_key',
'AWS_SECRET_ACCESS_KEY': 'settings_secret'}
'AWS_SECRET_ACCESS_KEY': 'settings_secret',
'AWS_SESSION_TOKEN': 'settings_token'}
crawler = get_crawler(settings_dict=aws_credentials)
# Instantiate with crawler
storage = S3FeedStorage.from_crawler(
@ -253,12 +254,15 @@ class S3FeedStorageTest(unittest.TestCase):
)
self.assertEqual(storage.access_key, 'settings_key')
self.assertEqual(storage.secret_key, 'settings_secret')
self.assertEqual(storage.session_token, 'settings_token')
# Instantiate directly
storage = S3FeedStorage('s3://mybucket/export.csv',
aws_credentials['AWS_ACCESS_KEY_ID'],
aws_credentials['AWS_SECRET_ACCESS_KEY'])
aws_credentials['AWS_SECRET_ACCESS_KEY'],
session_token=aws_credentials['AWS_SESSION_TOKEN'])
self.assertEqual(storage.access_key, 'settings_key')
self.assertEqual(storage.secret_key, 'settings_secret')
self.assertEqual(storage.session_token, 'settings_token')
# URI priority > settings priority
storage = S3FeedStorage('s3://uri_key:uri_secret@mybucket/export.csv',
aws_credentials['AWS_ACCESS_KEY_ID'],
@ -1928,14 +1932,15 @@ class FileFeedStoragePreFeedOptionsTest(unittest.TestCase):
maxDiff = None
def test_init(self):
settings_dict = {
'FEED_URI': 'file:///tmp/foobar',
'FEED_STORAGES': {
'file': FileFeedStorageWithoutFeedOptions
},
}
crawler = get_crawler(settings_dict=settings_dict)
feed_exporter = FeedExporter.from_crawler(crawler)
with tempfile.NamedTemporaryFile() as temp:
settings_dict = {
'FEED_URI': f'file:///{temp.name}',
'FEED_STORAGES': {
'file': FileFeedStorageWithoutFeedOptions
},
}
crawler = get_crawler(settings_dict=settings_dict)
feed_exporter = FeedExporter.from_crawler(crawler)
spider = scrapy.Spider("default")
with warnings.catch_warnings(record=True) as w:
feed_exporter.open_spider(spider)
@ -1957,8 +1962,8 @@ class FileFeedStoragePreFeedOptionsTest(unittest.TestCase):
class S3FeedStorageWithoutFeedOptions(S3FeedStorage):
def __init__(self, uri, access_key, secret_key, acl, endpoint_url):
super().__init__(uri, access_key, secret_key, acl, endpoint_url)
def __init__(self, uri, access_key, secret_key, acl, endpoint_url, **kwargs):
super().__init__(uri, access_key, secret_key, acl, endpoint_url, **kwargs)
class S3FeedStorageWithoutFeedOptionsWithFromCrawler(S3FeedStorage):

59
tests/test_http_response.py
View File

@ -820,3 +820,62 @@ class XmlResponseTest(TextResponseTest):
response.xpath("//s1:elem/text()", namespaces={'s1': 'http://scrapy.org'}).getall(),
response.selector.xpath("//s2:elem/text()").getall(),
)
class CustomResponse(TextResponse):
attributes = TextResponse.attributes + ("foo", "bar")
def __init__(self, *args, **kwargs) -> None:
self.foo = kwargs.pop("foo", None)
self.bar = kwargs.pop("bar", None)
self.lost = kwargs.pop("lost", None)
super().__init__(*args, **kwargs)
class CustomResponseTest(TextResponseTest):
response_class = CustomResponse
def test_copy(self):
super().test_copy()
r1 = self.response_class(url="https://example.org", status=200, foo="foo", bar="bar", lost="lost")
r2 = r1.copy()
self.assertIsInstance(r2, self.response_class)
self.assertEqual(r1.foo, r2.foo)
self.assertEqual(r1.bar, r2.bar)
self.assertEqual(r1.lost, "lost")
self.assertIsNone(r2.lost)
def test_replace(self):
super().test_replace()
r1 = self.response_class(url="https://example.org", status=200, foo="foo", bar="bar", lost="lost")
r2 = r1.replace(foo="new-foo", bar="new-bar", lost="new-lost")
self.assertIsInstance(r2, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r2.foo, "new-foo")
self.assertEqual(r2.bar, "new-bar")
self.assertEqual(r2.lost, "new-lost")
r3 = r1.replace(foo="new-foo", bar="new-bar")
self.assertIsInstance(r3, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r3.foo, "new-foo")
self.assertEqual(r3.bar, "new-bar")
self.assertIsNone(r3.lost)
r4 = r1.replace(foo="new-foo")
self.assertIsInstance(r4, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r4.foo, "new-foo")
self.assertEqual(r4.bar, "bar")
self.assertIsNone(r4.lost)
with self.assertRaises(TypeError) as ctx:
r1.replace(unknown="unknown")
self.assertTrue(str(ctx.exception).endswith("__init__() got an unexpected keyword argument 'unknown'"))

14
tests/test_request_cb_kwargs.py
View File

@ -158,12 +158,14 @@ class CallbackKeywordArgumentsTestCase(TestCase):
if key in line.getMessage():
exceptions[key] = line
self.assertEqual(exceptions['takes_less'].exc_info[0], TypeError)
self.assertEqual(
str(exceptions['takes_less'].exc_info[1]),
"parse_takes_less() got an unexpected keyword argument 'number'"
self.assertTrue(
str(exceptions['takes_less'].exc_info[1]).endswith(
"parse_takes_less() got an unexpected keyword argument 'number'"
)
)
self.assertEqual(exceptions['takes_more'].exc_info[0], TypeError)
self.assertEqual(
str(exceptions['takes_more'].exc_info[1]),
"parse_takes_more() missing 1 required positional argument: 'other'"
self.assertTrue(
str(exceptions['takes_more'].exc_info[1]).endswith(
"parse_takes_more() missing 1 required positional argument: 'other'"
)
)

5
tests/test_spiderloader/__init__.py
View File

@ -118,6 +118,11 @@ class SpiderLoaderTest(unittest.TestCase):
settings = Settings({'SPIDER_MODULES': [module],
'SPIDER_LOADER_WARN_ONLY': True})
spider_loader = SpiderLoader.from_settings(settings)
if str(w[0].message).startswith("_SixMetaPathImporter"):
# needed on 3.10 because of https://github.com/benjaminp/six/issues/349,
# at least until all six versions we can import (including botocore.vendored.six)
# are updated to 1.16.0+
w.pop(0)
self.assertIn("Could not load spiders from module", str(w[0].message))
spiders = spider_loader.list()

11
tests/test_utils_misc/__init__.py
View File

@ -4,7 +4,7 @@ import unittest
from unittest import mock
from scrapy.item import Item, Field
from scrapy.utils.misc import arg_to_iter, create_instance, load_object, set_environ, walk_modules
from scrapy.utils.misc import arg_to_iter, create_instance, load_object, rel_has_nofollow, set_environ, walk_modules
__doctests__ = ['scrapy.utils.misc']
@ -162,6 +162,15 @@ class UtilsMiscTestCase(unittest.TestCase):
assert os.environ.get('some_test_environ') == 'test_value'
assert os.environ.get('some_test_environ') == 'test'
def test_rel_has_nofollow(self):
assert rel_has_nofollow('ugc nofollow') is True
assert rel_has_nofollow('ugc,nofollow') is True
assert rel_has_nofollow('ugc') is False
assert rel_has_nofollow('nofollow') is True
assert rel_has_nofollow('nofollowfoo') is False
assert rel_has_nofollow('foonofollow') is False
assert rel_has_nofollow('ugc, , nofollow') is True
if __name__ == "__main__":
unittest.main()

10
tox.ini
View File

@ -9,7 +9,7 @@ minversion = 1.7.0
[testenv]
deps =
-rtests/requirements-py3.txt
-rtests/requirements.txt
# mitmproxy does not support PyPy
# mitmproxy does not support Windows when running Python < 3.7
# Python 3.9+ requires https://github.com/mitmproxy/mitmproxy/commit/8e5e43de24c9bc93092b63efc67fbec029a9e7fe
@ -23,6 +23,7 @@ passenv =
S3_TEST_FILE_URI
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
GCS_TEST_FILE_URI
GCS_PROJECT_ID
#allow tox virtualenv to upgrade pip/wheel/setuptools
@ -35,7 +36,10 @@ install_command =
[testenv:typing]
basepython = python3
deps =
mypy==0.780
lxml-stubs==0.2.0
mypy==0.910
types-pyOpenSSL==20.0.3
types-setuptools==57.0.0
commands =
mypy --show-error-codes {posargs: scrapy tests}
@ -78,7 +82,7 @@ deps =
Twisted[http2]==17.9.0
w3lib==1.17.0
zope.interface==4.1.3
-rtests/requirements-py3.txt
-rtests/requirements.txt
# mitmproxy 4.0.4+ requires upgrading some of the pinned dependencies
# above, hence we do not install it in pinned environments at the moment