RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased

This commit is contained in:
Andrey Rakhmatullin 2021-10-22 21:52:47 +05:00
parent a642b73e1a 144d1eb834
commit c105a7a85f
46 changed files with 227 additions and 94 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.github/workflows/checks.yml vendored
View File

@ -8,10 +8,10 @@ jobs:
fail-fast: false
matrix:
include:
- python-version: 3.9
- python-version: "3.10"
env:
TOXENV: security
- python-version: 3.9
- python-version: "3.10"
env:
TOXENV: flake8
# Pylint requires installing reppy, which does not support Python 3.9
@ -20,10 +20,10 @@ jobs:
env:
TOXENV: pylint
TOX_PIP_VERSION: 20.3.3
- python-version: 3.9
- python-version: 3.6
env:
TOXENV: typing
- python-version: 3.8 # Keep in sync with .readthedocs.yml
- python-version: "3.10" # Keep in sync with .readthedocs.yml
env:
TOXENV: docs

4
.github/workflows/publish.yml vendored
View File

@ -9,10 +9,10 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.9
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.9
python-version: "3.10"
- name: Check Tag
id: check-release-tag

2
.github/workflows/tests-macos.yml vendored
View File

@ -7,7 +7,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [3.6, 3.7, 3.8, 3.9]
python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
steps:
- uses: actions/checkout@v2

14
.github/workflows/tests-ubuntu.yml vendored
View File

@ -17,6 +17,12 @@ jobs:
- python-version: 3.9
env:
TOXENV: py
- python-version: "3.10"
env:
TOXENV: py
- python-version: "3.10"
env:
TOXENV: asyncio
- python-version: pypy3
env:
TOXENV: pypy3
@ -42,14 +48,6 @@ jobs:
TOXENV: extra-deps
TOX_PIP_VERSION: 20.3.3
# 3.10-pre
- python-version: "3.10.0-beta.4"
env:
TOXENV: py
- python-version: "3.10.0-beta.4"
env:
TOXENV: asyncio
steps:
- uses: actions/checkout@v2

9
.readthedocs.yml
View File

@ -5,12 +5,13 @@ sphinx:
fail_on_warning: true
build:
image: latest
os: ubuntu-20.04
tools:
# For available versions, see:
# https://docs.readthedocs.io/en/stable/config-file/v2.html#build-tools-python
python: "3.10" # Keep in sync with .github/workflows/checks.yml
python:
# For available versions, see:
# https://docs.readthedocs.io/en/stable/config-file/v2.html#build-image
version: 3.8 # Keep in sync with .github/workflows/checks.yml
install:
- requirements: docs/requirements.txt
- path: .

2
docs/intro/overview.rst
View File

@ -4,7 +4,7 @@
Scrapy at a glance
==================
Scrapy is an application framework for crawling web sites and extracting
Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting
structured data which can be used for a wide range of useful applications, like
data mining, information processing or historical archival.

16
docs/intro/tutorial.rst
View File

@ -277,9 +277,19 @@ As an alternative, you could've written:
>>> response.css('title::text')[0].get()
'Quotes to Scrape'
However, using ``.get()`` directly on a :class:`~scrapy.selector.SelectorList`
instance avoids an ``IndexError`` and returns ``None`` when it doesn't
find any element matching the selection.
Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will
raise an :exc:`IndexError` exception if there are no results::
>>> response.css('noelement')[0].get()
Traceback (most recent call last):
...
IndexError: list index out of range
You might want to use ``.get()`` directly on the
:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``
if there are no results::
>>> response.css("noelement").get()
There's a lesson here: for most scraping code, you want it to be resilient to
errors due to things not being found on a page, so that even if some parts fail

10
docs/news.rst
View File

@ -1830,7 +1830,7 @@ New features
* A new scheduler priority queue,
``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
:ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
scheduling improvement on crawls targetting multiple web domains, at the
scheduling improvement on crawls targeting multiple web domains, at the
cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
* A new :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>` attribute
@ -2868,7 +2868,7 @@ Bug fixes
- Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse <parse>`
(:issue:`2225`).
- Fix for invalid JSON and XML files when spider yields no items (:issue:`872`).
- Implement ``flush()`` fpr ``StreamLogger`` avoiding a warning in logs (:issue:`2125`).
- Implement ``flush()`` for ``StreamLogger`` avoiding a warning in logs (:issue:`2125`).
Refactoring
~~~~~~~~~~~
@ -3731,7 +3731,7 @@ Scrapy 0.24.3 (2014-08-09)
- adding some xpath tips to selectors docs (:commit:`2d103e0`)
- fix tests to account for https://github.com/scrapy/w3lib/pull/23 (:commit:`f8d366a`)
- get_func_args maximum recursion fix #728 (:commit:`81344ea`)
- Updated input/ouput processor example according to #560. (:commit:`f7c4ea8`)
- Updated input/output processor example according to #560. (:commit:`f7c4ea8`)
- Fixed Python syntax in tutorial. (:commit:`db59ed9`)
- Add test case for tunneling proxy (:commit:`f090260`)
- Bugfix for leaking Proxy-Authorization header to remote host when using tunneling (:commit:`d8793af`)
@ -4393,7 +4393,7 @@ Scrapyd changes
~~~~~~~~~~~~~~~
- Scrapyd now uses one process per spider
- It stores one log file per spider run, and rotate them keeping the lastest 5 logs per spider (by default)
- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default)
- A minimal web ui was added, available at http://localhost:6800 by default
- There is now a ``scrapy server`` command to start a Scrapyd server of the current project
@ -4429,7 +4429,7 @@ New features and improvements
- Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195)
- Support for overriding default request headers per spider (#181)
- Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186)
- Splitted Debian package into two packages - the library and the service (#187)
- Split Debian package into two packages - the library and the service (#187)
- Scrapy log refactoring (#188)
- New extension for keeping persistent spider contexts among different runs (#203)
- Added ``dont_redirect`` request.meta key for avoiding redirects (#233)

20
docs/topics/asyncio.rst
View File

@ -47,20 +47,14 @@ Awaiting on Deferreds
When the asyncio reactor isn't installed, you can await on Deferreds in the
coroutines directly. When it is installed, this is not possible anymore, due to
specifics of the Scrapy coroutine integration (the coroutines are wrapped into
asyncio Futures, not into Deferreds directly), and you need to wrap them into
:class:`asyncio.Future` objects, not into
:class:`~twisted.internet.defer.Deferred` directly), and you need to wrap them into
Futures. Scrapy provides two helpers for this:
.. autofunction:: scrapy.utils.defer.deferred_to_future
.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future
If you want to write universal code that works on any reactors,
you should use ``maybe_deferred_to_future`` on all Deferreds::
from scrapy.utils.defer import maybe_deferred_to_future
class MySpider(Spider):
# ...
async def parse_with_deferred(self, response):
additional_response = await maybe_deferred_to_future(treq.get('https://additional.url'))
additional_data = await maybe_deferred_to_future(treq.content(additional_response))
# ... use response and additional_data to yield items and requests
.. tip:: If you need to use these functions in code that aims to be compatible
with lower versions of Scrapy that do not provide these functions,
down to Scrapy 2.0 (earlier versions do not support
:mod:`asyncio`), you can copy the implementation of these functions
into your own code.

4
docs/topics/coroutines.rst
View File

@ -96,8 +96,8 @@ This means you can use many useful Python libraries providing such code::
:mod:`asyncio` loop and to use them you need to
:doc:`enable asyncio support in Scrapy<asyncio>`.
.. note:: If you want to ``await`` on Deferreds, you may need to
:ref:`wrap them<asyncio-await-dfd>`.
.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,
you need to :ref:`wrap them<asyncio-await-dfd>`.
Common use cases for asynchronous code include:

2
docs/topics/exporters.rst
View File

@ -122,7 +122,7 @@ Example::
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if field == 'price':
if name == 'price':
return f'$ {str(value)}'
return super().serialize_field(field, name, value)

4
docs/topics/item-pipeline.rst
View File

@ -190,6 +190,8 @@ item.
import scrapy
from itemadapter import ItemAdapter
from scrapy.utils.defer import maybe_deferred_to_future
class ScreenshotPipeline:
"""Pipeline that uses Splash to render screenshot of
@ -202,7 +204,7 @@ item.
encoded_item_url = quote(adapter["url"])
screenshot_url = self.SPLASH_URL.format(encoded_item_url)
request = scrapy.Request(screenshot_url)
response = await spider.crawler.engine.download(request, spider)
response = await maybe_deferred_to_future(spider.crawler.engine.download(request, spider))
if response.status != 200:
# Error happened, return item.

2
docs/topics/loaders.rst
View File

@ -56,7 +56,7 @@ chapter <topics-items>`::
l.add_xpath('name', '//div[@class="product_name"]')
l.add_xpath('name', '//div[@class="product_title"]')
l.add_xpath('price', '//p[@id="price"]')
l.add_css('stock', 'p#stock]')
l.add_css('stock', 'p#stock')
l.add_value('last_updated', 'today') # you can also use literal values
return l.load_item()

5
docs/topics/logging.rst
View File

@ -143,6 +143,7 @@ Logging settings
These settings can be used to configure the logging:
* :setting:`LOG_FILE`
* :setting:`LOG_FILE_APPEND`
* :setting:`LOG_ENABLED`
* :setting:`LOG_ENCODING`
* :setting:`LOG_LEVEL`
@ -155,7 +156,9 @@ The first couple of settings define a destination for log messages. If
:setting:`LOG_FILE` is set, messages sent through the root logger will be
redirected to a file named :setting:`LOG_FILE` with encoding
:setting:`LOG_ENCODING`. If unset and :setting:`LOG_ENABLED` is ``True``, log
messages will be displayed on the standard error. Lastly, if
messages will be displayed on the standard error. If :setting:`LOG_FILE` is set
and :setting:`LOG_FILE_APPEND` is ``False``, the file will be overwritten
(discarding the output from previous runs, if any). Lastly, if
:setting:`LOG_ENABLED` is ``False``, there won't be any visible log output.
:setting:`LOG_LEVEL` determines the minimum level of severity to display, those

3
docs/topics/media-pipeline.rst
View File

@ -383,6 +383,9 @@ class name. E.g. given pipeline class called MyPipeline you can set setting key:
and pipeline class MyPipeline will have expiration time set to 180.
The last modified time from the file is used to determine the age of the file in days,
which is then compared to the set expiration time to determine if the file is expired.
.. _topics-images-thumbnails:
Thumbnail generation for images

29
docs/topics/settings.rst
View File

@ -1026,6 +1026,16 @@ Default: ``None``
File name to use for logging output. If ``None``, standard error will be used.
.. setting:: LOG_FILE_APPEND
LOG_FILE_APPEND
---------------
Default: ``True``
If ``False``, the log file specified with :setting:`LOG_FILE` will be
overwritten (discarding the output from previous runs, if any).
.. setting:: LOG_FORMAT
LOG_FORMAT
@ -1566,7 +1576,7 @@ If a reactor is already installed,
:meth:`CrawlerRunner.__init__ <scrapy.crawler.CrawlerRunner.__init__>` raises
:exc:`Exception` if the installed reactor does not match the
:setting:`TWISTED_REACTOR` setting; therfore, having top-level
:setting:`TWISTED_REACTOR` setting; therefore, having top-level
:mod:`~twisted.internet.reactor` imports in project files and imported
third-party libraries will make Scrapy raise :exc:`Exception` when
it checks which reactor is installed.
@ -1645,8 +1655,19 @@ Default: ``2083``
Scope: ``spidermiddlewares.urllength``
The maximum URL length to allow for crawled URLs. For more information about
the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
The maximum URL length to allow for crawled URLs.
This setting can act as a stopping condition in case of URLs of ever-increasing
length, which may be caused for example by a programming error either in the
target server or in your code. See also :setting:`REDIRECT_MAX_TIMES` and
:setting:`DEPTH_LIMIT`.
Use ``0`` to allow URLs of any length.
The default value is copied from the `Microsoft Internet Explorer maximum URL
length`_, even though this setting exists for different reasons.
.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
.. setting:: USER_AGENT
@ -1658,7 +1679,7 @@ Default: ``"Scrapy/VERSION (+https://scrapy.org)"``
The default User-Agent to use when crawling, unless overridden. This user agent is
also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and
there is no overridding User-Agent header specified for the request.
there is no overriding User-Agent header specified for the request.
Settings documented elsewhere:

2
docs/topics/shell.rst
View File

@ -99,7 +99,7 @@ Available Shortcuts
shortcuts
- ``fetch(url[, redirect=True])`` - fetch a new response from the given URL
and update all related objects accordingly. You can optionaly ask for HTTP
and update all related objects accordingly. You can optionally ask for HTTP
3xx redirections to not be followed by passing ``redirect=False``
- ``fetch(request)`` - fetch a new response from the given request and update

1
docs/topics/spider-middleware.rst
View File

@ -447,4 +447,3 @@ UrlLengthMiddleware
settings (see the settings documentation for more info):
* :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.

4
docs/topics/spiders.rst
View File

@ -372,7 +372,7 @@ CrawlSpider
described below. If multiple rules match the same link, the first one
will be used, according to the order they're defined in this attribute.
This spider also exposes an overrideable method:
This spider also exposes an overridable method:
.. method:: parse_start_url(response, **kwargs)
@ -534,7 +534,7 @@ XMLFeedSpider
itertag = 'n:url'
# ...
Apart from these new attributes, this spider has the following overrideable
Apart from these new attributes, this spider has the following overridable
methods too:
.. method:: adapt_response(response)

2
docs/versioning.rst
View File

@ -13,7 +13,7 @@ There are 3 numbers in a Scrapy version: *A.B.C*
large changes.
* *B* is the release number. This will include many changes including features
and things that possibly break backward compatibility, although we strive to
keep theses cases at a minimum.
keep these cases at a minimum.
* *C* is the bugfix release number.
Backward-incompatibilities are explicitly mentioned in the :ref:`release notes <news>`,

2
extras/qpsclient.py
View File

@ -1,5 +1,5 @@
"""
A spider that generate light requests to meassure QPS troughput
A spider that generate light requests to meassure QPS throughput
usage:

2
scrapy/downloadermiddlewares/retry.py
View File

@ -2,7 +2,7 @@
An extension to retry failed requests that are potentially caused by temporary
problems such as a connection timeout or HTTP 500 error.
You can change the behaviour of this middleware by modifing the scraping settings:
You can change the behaviour of this middleware by modifying the scraping settings:
RETRY_TIMES - how many times to retry a failed page
RETRY_HTTP_CODES - which HTTP response codes to retry

2
scrapy/exporters.py
View File

@ -30,7 +30,7 @@ class BaseItemExporter:
self._configure(kwargs, dont_fail=dont_fail)
def _configure(self, options, dont_fail=False):
"""Configure the exporter by poping options from the ``options`` dict.
"""Configure the exporter by popping options from the ``options`` dict.
If dont_fail is set, it won't raise an exception on unexpected options
(useful for using with keyword arguments in subclasses ``__init__`` methods)
"""

2
scrapy/linkextractors/lxmlhtml.py
View File

@ -88,7 +88,7 @@ class LxmlParserLinkExtractor:
def _process_links(self, links):
""" Normalize and filter extracted links
The subclass should override it if neccessary
The subclass should override it if necessary
"""
return self._deduplicate_if_needed(links)

5
scrapy/loader/__init__.py
View File

@ -83,6 +83,9 @@ class ItemLoader(itemloaders.ItemLoader):
def __init__(self, item=None, selector=None, response=None, parent=None, **context):
if selector is None and response is not None:
selector = self.default_selector_class(response)
try:
selector = self.default_selector_class(response)
except AttributeError:
selector = None
context.update(response=response)
super().__init__(item=item, selector=selector, parent=parent, **context)

8
scrapy/pipelines/files.py
View File

@ -85,7 +85,7 @@ class S3FilesStore:
AWS_USE_SSL = None
AWS_VERIFY = None
POLICY = 'private' # Overriden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settings
POLICY = 'private' # Overridden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settings
HEADERS = {
'Cache-Control': 'max-age=172800',
}
@ -142,7 +142,7 @@ class S3FilesStore:
**extra)
def _headers_to_botocore_kwargs(self, headers):
""" Convert headers to botocore keyword agruments.
""" Convert headers to botocore keyword arguments.
"""
# This is required while we need to support both boto and botocore.
mapping = CaselessDict({
@ -190,7 +190,7 @@ class GCSFilesStore:
CACHE_CONTROL = 'max-age=172800'
# The bucket's default object ACL will be applied to the object.
# Overriden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings.
# Overridden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings.
POLICY = None
def __init__(self, uri):
@ -291,7 +291,7 @@ class FilesPipeline(MediaPipeline):
"""Abstract pipeline that implement the file downloading
This pipeline tries to minimize network transfers and file processing,
doing stat of the files and determining if file is new, uptodate or
doing stat of the files and determining if file is new, up-to-date or
expired.
``new`` files are those that pipeline never processed and needs to be

1
scrapy/settings/default_settings.py
View File

@ -207,6 +207,7 @@ LOG_DATEFORMAT = '%Y-%m-%d %H:%M:%S'
LOG_STDOUT = False
LOG_LEVEL = 'DEBUG'
LOG_FILE = None
LOG_FILE_APPEND = True
LOG_SHORT_NAMES = False
SCHEDULER_DEBUG = False

4
scrapy/spiders/feed.py
View File

@ -43,7 +43,7 @@ class XMLFeedSpider(Spider):
return response
def parse_node(self, response, selector):
"""This method must be overriden with your custom spider functionality"""
"""This method must be overridden with your custom spider functionality"""
if hasattr(self, 'parse_item'): # backward compatibility
return self.parse_item(response, selector)
raise NotImplementedError
@ -113,7 +113,7 @@ class CSVFeedSpider(Spider):
return response
def parse_row(self, response, row):
"""This method must be overriden with your custom spider functionality"""
"""This method must be overridden with your custom spider functionality"""
raise NotImplementedError
def parse_rows(self, response):

2
scrapy/utils/console.py
View File

@ -14,7 +14,7 @@ def _embed_ipython_shell(namespace={}, banner=''):
@wraps(_embed_ipython_shell)
def wrapper(namespace=namespace, banner=''):
config = load_default_config()
# Always use .instace() to ensure _instance propagation to all parents
# Always use .instance() to ensure _instance propagation to all parents
# this is needed for <TAB> completion works well for new imports
# and clear the instance to always have the fresh env
# on repeated breaks like with inspect_response()

2
scrapy/utils/datatypes.py
View File

@ -41,7 +41,7 @@ class CaselessDict(dict):
return key.lower()
def normvalue(self, value):
"""Method to normalize values prior to be setted"""
"""Method to normalize values prior to be set"""
return value
def get(self, key, def_val=None):

45
scrapy/utils/defer.py
View File

@ -46,7 +46,7 @@ def defer_succeed(result) -> Deferred:
"""Same as twisted.internet.defer.succeed but delay calling callback until
next reactor loop
It delays by 100ms so reactor has a chance to go trough readers and writers
It delays by 100ms so reactor has a chance to go through readers and writers
before attending pending delayed calls, so do not set delay to zero.
"""
from twisted.internet import reactor
@ -303,17 +303,50 @@ def maybeDeferred_coro(f: Callable, *args, **kw) -> Deferred:
def deferred_to_future(d: Deferred) -> Future:
""" Wraps a Deferred into a Future. Requires the asyncio reactor.
"""
.. versionadded:: VERSION
Return an :class:`asyncio.Future` object that wraps *d*.
When :ref:`using the asyncio reactor <install-asyncio>`, you cannot await
on :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy
callables defined as coroutines <coroutine-support>`, you can only await on
``Future`` objects. Wrapping ``Deferred`` objects into ``Future`` objects
allows you to wait on them::
class MySpider(Spider):
...
async def parse(self, response):
d = treq.get('https://example.com/additional')
additional_response = await deferred_to_future(d)
"""
return d.asFuture(asyncio.get_event_loop())
def maybe_deferred_to_future(d: Deferred) -> Union[Deferred, Future]:
""" Converts a Deferred to something that can be awaited in a callback or other user coroutine.
"""
.. versionadded:: VERSION
If the asyncio reactor is installed, coroutines are wrapped into Futures, and only Futures can be
awaited inside them. Otherwise, coroutines are wrapped into Deferreds and Deferreds can be awaited
directly inside them.
Return *d* as an object that can be awaited from a :ref:`Scrapy callable
defined as a coroutine <coroutine-support>`.
What you can await in Scrapy callables defined as coroutines depends on the
value of :setting:`TWISTED_REACTOR`:
- When not using the asyncio reactor, you can only await on
:class:`~twisted.internet.defer.Deferred` objects.
- When :ref:`using the asyncio reactor <install-asyncio>`, you can only
await on :class:`asyncio.Future` objects.
If you want to write code that uses ``Deferred`` objects but works with any
reactor, use this function on all ``Deferred`` objects::
class MySpider(Spider):
...
async def parse(self, response):
d = treq.get('https://example.com/additional')
extra_response = await maybe_deferred_to_future(d)
"""
if not is_asyncio_reactor_installed():
return d

3
scrapy/utils/log.py
View File

@ -124,8 +124,9 @@ def _get_handler(settings):
""" Return a log handler object according to settings """
filename = settings.get('LOG_FILE')
if filename:
mode = 'a' if settings.getbool('LOG_FILE_APPEND') else 'w'
encoding = settings.get('LOG_ENCODING')
handler = logging.FileHandler(filename, encoding=encoding)
handler = logging.FileHandler(filename, mode=mode, encoding=encoding)
elif settings.getbool('LOG_ENABLED'):
handler = logging.StreamHandler()
else:

4
scrapy/utils/request.py
View File

@ -48,7 +48,7 @@ def request_fingerprint(
the fingerprint.
For this reason, request headers are ignored by default when calculating
the fingeprint. If you want to include specific headers use the
the fingerprint. If you want to include specific headers use the
include_headers argument, which is a list of Request headers to include.
Also, servers usually ignore fragments in urls when handling requests,
@ -78,7 +78,7 @@ def request_fingerprint(
def request_authenticate(request: Request, username: str, password: str) -> None:
"""Autenticate the given request (in place) using the HTTP basic access
"""Authenticate the given request (in place) using the HTTP basic access
authentication mechanism (RFC 2617) and the given username and password
"""
request.headers['Authorization'] = basic_auth_header(username, password)

2
sep/sep-001.rst
View File

@ -260,7 +260,7 @@ ItemForm
ia['width'] = x.x('//p[@class="width"]')
ia['volume'] = x.x('//p[@class="volume"]')
# another example passing parametes on instance
# another example passing parameters on instance
ia = NewsForm(response, encoding='utf-8')
ia['name'] = x.x('//p[@class="name"]')

2
sep/sep-005.rst
View File

@ -107,7 +107,7 @@ gUsing default_builder
This will use default_builder as the builder for every field in the item class.
As a reducer is not set reducers will be set based on Item Field classess.
As a reducer is not set reducers will be set based on Item Field classes.
gReset default_builder for a field
==================================

2
sep/sep-014.rst
View File

@ -64,7 +64,7 @@ Request Processors takes requests objects and can perform any action to them,
like filtering or modifying on the fly.
The current ``LinkExtractor`` had integrated link processing, like
canonicalize. Request Processors can be reutilized and applied in serie.
canonicalize. Request Processors can be reutilized and applied in series.
Request Generator
-----------------

2
sep/sep-021.rst
View File

@ -22,7 +22,7 @@ Instead, the hooks are spread over:
* Downloader handlers (DOWNLOADER_HANDLERS)
* Item pipelines (ITEM_PIPELINES)
* Feed exporters and storages (FEED_EXPORTERS, FEED_STORAGES)
* Overrideable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc)
* Overridable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc)
* Generic extensions (EXTENSIONS)
* CLI commands (COMMANDS_MODULE)

1
setup.py
View File

@ -87,6 +87,7 @@ setup(
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
'Topic :: Internet :: WWW/HTTP',

29
tests/test_crawler.py
View File

@ -98,6 +98,8 @@ class CrawlerLoggingTestCase(unittest.TestCase):
def test_spider_custom_settings_log_level(self):
log_file = self.mktemp()
with open(log_file, 'wb') as fo:
fo.write('previous message\n'.encode('utf-8'))
class MySpider(scrapy.Spider):
name = 'spider'
@ -119,8 +121,9 @@ class CrawlerLoggingTestCase(unittest.TestCase):
logging.error('error message')
with open(log_file, 'rb') as fo:
logged = fo.read().decode('utf8')
logged = fo.read().decode('utf-8')
self.assertIn('previous message', logged)
self.assertNotIn('debug message', logged)
self.assertIn('info message', logged)
self.assertIn('warning message', logged)
@ -131,6 +134,30 @@ class CrawlerLoggingTestCase(unittest.TestCase):
crawler.stats.get_value('log_count/INFO') - info_count, 1)
self.assertEqual(crawler.stats.get_value('log_count/DEBUG', 0), 0)
def test_spider_custom_settings_log_append(self):
log_file = self.mktemp()
with open(log_file, 'wb') as fo:
fo.write('previous message\n'.encode('utf-8'))
class MySpider(scrapy.Spider):
name = 'spider'
custom_settings = {
'LOG_FILE': log_file,
'LOG_FILE_APPEND': False,
# disable telnet if not available to avoid an extra warning
'TELNETCONSOLE_ENABLED': telnet.TWISTED_CONCH_AVAILABLE,
}
configure_logging()
Crawler(MySpider, {})
logging.debug('debug message')
with open(log_file, 'rb') as fo:
logged = fo.read().decode('utf-8')
self.assertNotIn('previous message', logged)
self.assertIn('debug message', logged)
class SpiderLoaderWithWrongInterface:

4
tests/test_http_response.py
View File

@ -19,7 +19,7 @@ class BaseResponseTest(unittest.TestCase):
response_class = Response
def test_init(self):
# Response requires url in the consturctor
# Response requires url in the constructor
self.assertRaises(Exception, self.response_class)
self.assertTrue(isinstance(self.response_class('http://example.com/'), self.response_class))
self.assertRaises(TypeError, self.response_class, b"http://example.com")
@ -392,7 +392,7 @@ class TextResponseTest(BaseResponseTest):
def test_declared_encoding_invalid(self):
"""Check that unknown declared encodings are ignored"""
r = self.response_class("http://www.example.com",
headers={"Content-type": ["text/html; charset=UKNOWN"]},
headers={"Content-type": ["text/html; charset=UNKNOWN"]},
body=b"\xc2\xa3")
self.assertEqual(r._declared_encoding(), None)
self._assert_response_values(r, 'utf-8', "\xa3")

8
tests/test_loader.py
View File

@ -4,7 +4,7 @@ import attr
from itemadapter import ItemAdapter
from itemloaders.processors import Compose, Identity, MapCompose, TakeFirst
from scrapy.http import HtmlResponse
from scrapy.http import HtmlResponse, Response
from scrapy.item import Item, Field
from scrapy.loader import ItemLoader
from scrapy.selector import Selector
@ -304,6 +304,12 @@ class SelectortemLoaderTest(unittest.TestCase):
l.add_css('name', 'div::text')
self.assertEqual(l.get_output_value('name'), ['Marta'])
def test_init_method_with_base_response(self):
"""Selector should be None after initialization"""
response = Response("https://scrapy.org")
l = TestItemLoader(response=response)
self.assertIs(l.selector, None)
def test_init_method_with_response(self):
l = TestItemLoader(response=self.response)

30
tests/test_pipelines.py
View File

@ -6,6 +6,7 @@ from twisted.internet.defer import Deferred
from twisted.trial import unittest
from scrapy import Spider, signals, Request
from scrapy.utils.defer import maybe_deferred_to_future, deferred_to_future
from scrapy.utils.test import get_crawler, get_from_asyncio_queue
from tests.mockserver import MockServer
@ -31,18 +32,38 @@ class DeferredPipeline:
class AsyncDefPipeline:
async def process_item(self, item, spider):
await defer.succeed(42)
d = Deferred()
from twisted.internet import reactor
reactor.callLater(0, d.callback, None)
await maybe_deferred_to_future(d)
item['pipeline_passed'] = True
return item
class AsyncDefAsyncioPipeline:
async def process_item(self, item, spider):
d = Deferred()
from twisted.internet import reactor
reactor.callLater(0, d.callback, None)
await deferred_to_future(d)
await asyncio.sleep(0.2)
item['pipeline_passed'] = await get_from_asyncio_queue(True)
return item
class AsyncDefNotAsyncioPipeline:
async def process_item(self, item, spider):
d1 = Deferred()
from twisted.internet import reactor
reactor.callLater(0, d1.callback, None)
await d1
d2 = Deferred()
reactor.callLater(0, d2.callback, None)
await maybe_deferred_to_future(d2)
item['pipeline_passed'] = True
return item
class ItemSpider(Spider):
name = 'itemspider'
@ -99,3 +120,10 @@ class PipelineTestCase(unittest.TestCase):
crawler = self._create_crawler(AsyncDefAsyncioPipeline)
yield crawler.crawl(mockserver=self.mockserver)
self.assertEqual(len(self.items), 1)
@mark.only_not_asyncio()
@defer.inlineCallbacks
def test_asyncdef_not_asyncio_pipeline(self):
crawler = self._create_crawler(AsyncDefNotAsyncioPipeline)
yield crawler.crawl(mockserver=self.mockserver)
self.assertEqual(len(self.items), 1)

8
tests/test_request_attribute_binding.py
View File

@ -106,9 +106,9 @@ class CrawlTestCase(TestCase):
"""
Downloader middleware which returns a response with an specific 'request' attribute.
* The spider callback should receive the overriden response.request
* Handlers listening to the response_received signal should receive the overriden response.request
* The "crawled" log message should show the overriden response.request
* The spider callback should receive the overridden response.request
* Handlers listening to the response_received signal should receive the overridden response.request
* The "crawled" log message should show the overridden response.request
"""
signal_params = {}
@ -144,7 +144,7 @@ class CrawlTestCase(TestCase):
An exception is raised but caught by the next middleware, which
returns a Response with a specific 'request' attribute.
The spider callback should receive the overriden response.request
The spider callback should receive the overridden response.request
"""
url = self.mockserver.url("/status?n=200")
runner = CrawlerRunner(settings={

4
tests/test_utils_defer.py
View File

@ -29,7 +29,7 @@ class MustbeDeferredTest(unittest.TestCase):
dfd = mustbe_deferred(_append, 1)
dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred
steps.append(2) # add another value, that should be catched by assertEqual
steps.append(2) # add another value, that should be caught by assertEqual
return dfd
def test_unfired_deferred(self):
@ -43,7 +43,7 @@ class MustbeDeferredTest(unittest.TestCase):
dfd = mustbe_deferred(_append, 1)
dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred
steps.append(2) # add another value, that should be catched by assertEqual
steps.append(2) # add another value, that should be caught by assertEqual
return dfd

2
tests/test_utils_template.py
View File

@ -36,7 +36,7 @@ class UtilsRenderTemplateFileTestCase(unittest.TestCase):
self.assertEqual(result.read().decode('utf8'), rendered)
os.remove(render_path)
assert not os.path.exists(render_path) # Failure of test iself
assert not os.path.exists(render_path) # Failure of test itself
if '__main__' == __name__:

2
tox.ini
View File

@ -41,6 +41,7 @@ deps =
types-pyOpenSSL==20.0.3
types-setuptools==57.0.0
commands =
pip install types-dataclasses # remove once py36 support is dropped
mypy --show-error-codes {posargs: scrapy tests}
[testenv:security]
@ -57,6 +58,7 @@ deps =
# Twisted[http2] is required to import some files
Twisted[http2]>=17.9.0
pytest-flake8
flake8==3.9.2 # https://github.com/tholo/pytest-flake8/issues/81
commands =
py.test --flake8 {posargs:docs scrapy tests}