RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'scrapy:master' into docs-extension

This commit is contained in:
Aaron Tan 2021-07-30 20:36:58 +10:00 committed by GitHub
parent 209c1fce02 b406affd5b
commit 714aa3970e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
14 changed files with 227 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
docs/topics/dynamic-content.rst
View File

@ -246,24 +246,46 @@ Using a headless browser
========================
A `headless browser`_ is a special web browser that provides an API for
automation.
automation. By installing the :ref:`asyncio reactor <install-asyncio>`,
it is possible to integrate ``asyncio``-based libraries which handle headless browsers.
The easiest way to use a headless browser with Scrapy is to use Selenium_,
along with `scrapy-selenium`_ for seamless integration.
One such library is `playwright-python`_ (an official Python port of `playwright`_).
The following is a simple snippet to illustrate its usage within a Scrapy spider::
import scrapy
from playwright.async_api import async_playwright
class PlaywrightSpider(scrapy.Spider):
name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader
async def parse(self, response):
async with async_playwright() as pw:
browser = await pw.chromium.launch()
page = await browser.new_page()
await page.goto("https:/example.org")
title = await page.title()
return {"title": title}
However, using `playwright-python`_ directly as in the above example
circumvents most of the Scrapy components (middlewares, dupefilter, etc).
We recommend using `scrapy-playwright`_ for a better integration.
.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29
.. _chompjs: https://github.com/Nykakin/chompjs
.. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _Splash: https://github.com/scrapinghub/splash
.. _chompjs: https://github.com/Nykakin/chompjs
.. _curl: https://curl.haxx.se/
.. _headless browser: https://en.wikipedia.org/wiki/Headless_browser
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _js2xml: https://github.com/scrapinghub/js2xml
.. _playwright-python: https://github.com/microsoft/playwright-python
.. _playwright: https://github.com/microsoft/playwright
.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/
.. _pytesseract: https://github.com/madmaze/pytesseract
.. _scrapy-selenium: https://github.com/clemfromspace/scrapy-selenium
.. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
.. _Selenium: https://www.selenium.dev/
.. _Splash: https://github.com/scrapinghub/splash
.. _tabula-py: https://github.com/chezou/tabula-py
.. _wget: https://www.gnu.org/software/wget/
.. _wgrep: https://github.com/stav/wgrep
.. _wgrep: https://github.com/stav/wgrep

7
docs/topics/feed-exports.rst
View File

@ -135,7 +135,7 @@ Here are some examples to illustrate:
- ``s3://mybucket/scraping/feeds/%(name)s/%(time)s.json``
.. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
.. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
they can also be used as storage URI parameters.
@ -200,6 +200,9 @@ passed through the following settings:
- :setting:`AWS_ACCESS_KEY_ID`
- :setting:`AWS_SECRET_ACCESS_KEY`
- :setting:`AWS_SESSION_TOKEN` (only needed for `temporary security credentials`_)
.. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys
You can also define a custom ACL and custom endpoint for exported feeds using this setting:
@ -357,7 +360,7 @@ For instance::
'item_export_kwargs': {
'export_empty_fields': True,
},
},
},
'/home/user/documents/items.xml': {
'format': 'xml',
'fields': ['name', 'price'],

88
docs/topics/media-pipeline.rst
View File

@ -111,25 +111,82 @@ For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::
IMAGES_STORE = '/path/to/valid/dir'
.. _topics-file-naming:
File Naming
===========
Default File Naming
-------------------
By default, files are stored using an `SHA-1 hash`_ of their URLs for the file names.
For example, the following image URL::
http://www.example.com/image.jpg
Whose ``SHA-1 hash`` is::
3afec3b4765f8f0a07b78f98c07b83f013567a0a
Will be downloaded and stored using your chosen :ref:`storage method <topics-supported-storage>` and the following file name::
3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
Custom File Naming
-------------------
You may wish to use a different calculated file name for saved files.
For example, classifying an image by including meta in the file name.
Customize file names by overriding the ``file_path`` method of your
media pipeline.
For example, an image pipeline with image URL::
http://www.example.com/product/images/large/front/0000000004166
Can be processed into a file name with a condensed hash and the perspective
``front``::
00b08510e4_front.jpg
By overriding ``file_path`` like this:
.. code-block:: python
import hashlib
from os.path import splitext
def file_path(self, request, response=None, info=None, *, item=None):
image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)
image_perspective = request.url.split('/')[-2]
image_filename = f'{image_url_hash}_{image_perspective}.jpg'
return image_filename
.. warning::
If your custom file name scheme relies on meta data that can vary between
scrapes it may lead to unexpected re-downloading of existing media using
new file names.
For example, if your custom file name scheme uses a product title and the
site changes an item's product title between scrapes, Scrapy will re-download
the same media using updated file names.
For more information about the ``file_path`` method, see :ref:`topics-media-pipeline-override`.
.. _topics-supported-storage:
Supported Storage
=================
File system storage
-------------------
The files are stored using a `SHA1 hash`_ of their URLs for the file names.
File system storage will save files to the following path::
For example, the following image URL::
http://www.example.com/image.jpg
Whose ``SHA1 hash`` is::
3afec3b4765f8f0a07b78f98c07b83f013567a0a
Will be downloaded and stored in the following file::
<IMAGES_STORE>/full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
<IMAGES_STORE>/full/<FILE_NAME>
Where:
@ -139,6 +196,9 @@ Where:
* ``full`` is a sub-directory to separate full images from thumbnails (if
used). For more info see :ref:`topics-images-thumbnails`.
* ``<FILE_NAME>`` is the file name assigned to the file. For more info see :ref:`topics-file-naming`.
.. _media-pipeline-ftp:
FTP server storage
@ -353,9 +413,9 @@ Where:
* ``<size_name>`` is the one specified in the :setting:`IMAGES_THUMBS`
dictionary keys (``small``, ``big``, etc)
* ``<image_id>`` is the `SHA1 hash`_ of the image url
* ``<image_id>`` is the `SHA-1 hash`_ of the image url
.. _SHA1 hash: https://en.wikipedia.org/wiki/SHA_hash_functions
.. _SHA-1 hash: https://en.wikipedia.org/wiki/SHA_hash_functions
Example of image files stored using ``small`` and ``big`` thumbnail names::

7
docs/topics/request-response.rst
View File

@ -670,9 +670,6 @@ Response objects
.. autoclass:: Response
A :class:`Response` object represents an HTTP response, which is usually
downloaded (by the Downloader) and fed to the Spiders for processing.
:param url: the URL of this response
:type url: str
@ -829,6 +826,8 @@ Response objects
handlers, i.e. for ``http(s)`` responses. For other handlers,
:attr:`protocol` is always ``None``.
.. autoattribute:: Response.attributes
.. method:: Response.copy()
Returns a new Response which is a copy of this Response.
@ -925,6 +924,8 @@ TextResponse objects
A :class:`~scrapy.Selector` instance using the response as
target. The selector is lazily instantiated on first access.
.. autoattribute:: TextResponse.attributes
:class:`TextResponse` objects support the following methods in addition to
the standard :class:`Response` ones:

13
docs/topics/settings.rst
View File

@ -204,6 +204,19 @@ Default: ``None``
The AWS secret key used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
.. setting:: AWS_SESSION_TOKEN
AWS_SESSION_TOKEN
-----------------
Default: ``None``
The AWS security token used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
`temporary security credentials`_.
.. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys
.. setting:: AWS_ENDPOINT_URL
AWS_ENDPOINT_URL

19
scrapy/core/downloader/handlers/s3.py
View File

@ -1,5 +1,3 @@
from urllib.parse import unquote
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
from scrapy.exceptions import NotConfigured
from scrapy.utils.boto import is_botocore_available
@ -12,6 +10,7 @@ class S3DownloadHandler:
def __init__(self, settings, *,
crawler=None,
aws_access_key_id=None, aws_secret_access_key=None,
aws_session_token=None,
httpdownloadhandler=HTTPDownloadHandler, **kw):
if not is_botocore_available():
raise NotConfigured('missing botocore library')
@ -20,6 +19,8 @@ class S3DownloadHandler:
aws_access_key_id = settings['AWS_ACCESS_KEY_ID']
if not aws_secret_access_key:
aws_secret_access_key = settings['AWS_SECRET_ACCESS_KEY']
if not aws_session_token:
aws_session_token = settings['AWS_SESSION_TOKEN']
# If no credentials could be found anywhere,
# consider this an anonymous connection request by default;
@ -38,7 +39,7 @@ class S3DownloadHandler:
if not self.anon:
SignerCls = botocore.auth.AUTH_TYPE_MAPS['s3']
self._signer = SignerCls(botocore.credentials.Credentials(
aws_access_key_id, aws_secret_access_key))
aws_access_key_id, aws_secret_access_key, aws_session_token))
_http_handler = create_instance(
objcls=httpdownloadhandler,
@ -59,7 +60,7 @@ class S3DownloadHandler:
url = f'{scheme}://{bucket}.s3.amazonaws.com{path}'
if self.anon:
request = request.replace(url=url)
elif self._signer is not None:
else:
import botocore.awsrequest
awsrequest = botocore.awsrequest.AWSRequest(
method=request.method,
@ -69,14 +70,4 @@ class S3DownloadHandler:
self._signer.add_auth(awsrequest)
request = request.replace(
url=url, headers=awsrequest.headers.items())
else:
signed_headers = self.conn.make_request(
method=request.method,
bucket=bucket,
key=unquote(p.path),
query_args=unquote(p.query),
headers=request.headers,
data=request.body,
)
request = request.replace(url=url, headers=signed_headers)
return self._download_http(request, spider)

5
scrapy/extensions/feedexport.py
View File

@ -154,13 +154,14 @@ class FileFeedStorage:
class S3FeedStorage(BlockingFeedStorage):
def __init__(self, uri, access_key=None, secret_key=None, acl=None, endpoint_url=None, *,
feed_options=None):
feed_options=None, session_token=None):
if not is_botocore_available():
raise NotConfigured('missing botocore library')
u = urlparse(uri)
self.bucketname = u.hostname
self.access_key = u.username or access_key
self.secret_key = u.password or secret_key
self.session_token = session_token
self.keyname = u.path[1:] # remove first "/"
self.acl = acl
self.endpoint_url = endpoint_url
@ -169,6 +170,7 @@ class S3FeedStorage(BlockingFeedStorage):
self.s3_client = session.create_client(
's3', aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key,
aws_session_token=self.session_token,
endpoint_url=self.endpoint_url)
if feed_options and feed_options.get('overwrite', True) is False:
logger.warning('S3 does not support appending to files. To '
@ -182,6 +184,7 @@ class S3FeedStorage(BlockingFeedStorage):
uri,
access_key=crawler.settings['AWS_ACCESS_KEY_ID'],
secret_key=crawler.settings['AWS_SECRET_ACCESS_KEY'],
session_token=crawler.settings['AWS_SESSION_TOKEN'],
acl=crawler.settings['FEED_STORAGE_S3_ACL'] or None,
endpoint_url=crawler.settings['AWS_ENDPOINT_URL'] or None,
feed_options=feed_options,

23
scrapy/http/response/__init__.py
View File

@ -4,7 +4,7 @@ responses in Scrapy.
See documentation in docs/topics/request-response.rst
"""
from typing import Generator
from typing import Generator, Tuple
from urllib.parse import urljoin
from scrapy.exceptions import NotSupported
@ -16,6 +16,19 @@ from scrapy.utils.trackref import object_ref
class Response(object_ref):
"""An object that represents an HTTP response, which is usually
downloaded (by the Downloader) and fed to the Spiders for processing.
"""
attributes: Tuple[str, ...] = (
"url", "status", "headers", "body", "flags", "request", "certificate", "ip_address", "protocol",
)
"""A tuple of :class:`str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
``__init__`` method.
Currently used by :meth:`Response.replace`.
"""
def __init__(
self,
@ -97,12 +110,8 @@ class Response(object_ref):
return self.replace()
def replace(self, *args, **kwargs):
"""Create a new Response with the same attributes except for those
given new values.
"""
for x in [
"url", "status", "headers", "body", "request", "flags", "certificate", "ip_address", "protocol",
]:
"""Create a new Response with the same attributes except for those given new values"""
for x in self.attributes:
kwargs.setdefault(x, getattr(self, x))
cls = kwargs.pop('cls', self.__class__)
return cls(*args, **kwargs)

8
scrapy/http/response/text.py
View File

@ -8,7 +8,7 @@ See documentation in docs/topics/request-response.rst
import json
import warnings
from contextlib import suppress
from typing import Generator
from typing import Generator, Tuple
from urllib.parse import urljoin
import parsel
@ -30,6 +30,8 @@ class TextResponse(Response):
_DEFAULT_ENCODING = 'ascii'
_cached_decoded_json = _NONE
attributes: Tuple[str, ...] = Response.attributes + ("encoding",)
def __init__(self, *args, **kwargs):
self._encoding = kwargs.pop('encoding', None)
self._cached_benc = None
@ -53,10 +55,6 @@ class TextResponse(Response):
else:
super()._set_body(body)
def replace(self, *args, **kwargs):
kwargs.setdefault('encoding', self.encoding)
return Response.replace(self, *args, **kwargs)
@property
def encoding(self):
return self._declared_encoding() or self._body_inferred_encoding()

3
scrapy/pipelines/files.py
View File

@ -79,6 +79,7 @@ class FSFilesStore:
class S3FilesStore:
AWS_ACCESS_KEY_ID = None
AWS_SECRET_ACCESS_KEY = None
AWS_SESSION_TOKEN = None
AWS_ENDPOINT_URL = None
AWS_REGION_NAME = None
AWS_USE_SSL = None
@ -98,6 +99,7 @@ class S3FilesStore:
's3',
aws_access_key_id=self.AWS_ACCESS_KEY_ID,
aws_secret_access_key=self.AWS_SECRET_ACCESS_KEY,
aws_session_token=self.AWS_SESSION_TOKEN,
endpoint_url=self.AWS_ENDPOINT_URL,
region_name=self.AWS_REGION_NAME,
use_ssl=self.AWS_USE_SSL,
@ -349,6 +351,7 @@ class FilesPipeline(MediaPipeline):
s3store = cls.STORE_SCHEMES['s3']
s3store.AWS_ACCESS_KEY_ID = settings['AWS_ACCESS_KEY_ID']
s3store.AWS_SECRET_ACCESS_KEY = settings['AWS_SECRET_ACCESS_KEY']
s3store.AWS_SESSION_TOKEN = settings['AWS_SESSION_TOKEN']
s3store.AWS_ENDPOINT_URL = settings['AWS_ENDPOINT_URL']
s3store.AWS_REGION_NAME = settings['AWS_REGION_NAME']
s3store.AWS_USE_SSL = settings['AWS_USE_SSL']

1
scrapy/pipelines/images.py
View File

@ -92,6 +92,7 @@ class ImagesPipeline(FilesPipeline):
s3store = cls.STORE_SCHEMES['s3']
s3store.AWS_ACCESS_KEY_ID = settings['AWS_ACCESS_KEY_ID']
s3store.AWS_SECRET_ACCESS_KEY = settings['AWS_SECRET_ACCESS_KEY']
s3store.AWS_SESSION_TOKEN = settings['AWS_SESSION_TOKEN']
s3store.AWS_ENDPOINT_URL = settings['AWS_ENDPOINT_URL']
s3store.AWS_REGION_NAME = settings['AWS_REGION_NAME']
s3store.AWS_USE_SSL = settings['AWS_USE_SSL']

12
tests/test_feedexport.py
View File

@ -244,7 +244,8 @@ class S3FeedStorageTest(unittest.TestCase):
def test_parse_credentials(self):
skip_if_no_boto()
aws_credentials = {'AWS_ACCESS_KEY_ID': 'settings_key',
'AWS_SECRET_ACCESS_KEY': 'settings_secret'}
'AWS_SECRET_ACCESS_KEY': 'settings_secret',
'AWS_SESSION_TOKEN': 'settings_token'}
crawler = get_crawler(settings_dict=aws_credentials)
# Instantiate with crawler
storage = S3FeedStorage.from_crawler(
@ -253,12 +254,15 @@ class S3FeedStorageTest(unittest.TestCase):
)
self.assertEqual(storage.access_key, 'settings_key')
self.assertEqual(storage.secret_key, 'settings_secret')
self.assertEqual(storage.session_token, 'settings_token')
# Instantiate directly
storage = S3FeedStorage('s3://mybucket/export.csv',
aws_credentials['AWS_ACCESS_KEY_ID'],
aws_credentials['AWS_SECRET_ACCESS_KEY'])
aws_credentials['AWS_SECRET_ACCESS_KEY'],
session_token=aws_credentials['AWS_SESSION_TOKEN'])
self.assertEqual(storage.access_key, 'settings_key')
self.assertEqual(storage.secret_key, 'settings_secret')
self.assertEqual(storage.session_token, 'settings_token')
# URI priority > settings priority
storage = S3FeedStorage('s3://uri_key:uri_secret@mybucket/export.csv',
aws_credentials['AWS_ACCESS_KEY_ID'],
@ -1957,8 +1961,8 @@ class FileFeedStoragePreFeedOptionsTest(unittest.TestCase):
class S3FeedStorageWithoutFeedOptions(S3FeedStorage):
def __init__(self, uri, access_key, secret_key, acl, endpoint_url):
super().__init__(uri, access_key, secret_key, acl, endpoint_url)
def __init__(self, uri, access_key, secret_key, acl, endpoint_url, **kwargs):
super().__init__(uri, access_key, secret_key, acl, endpoint_url, **kwargs)
class S3FeedStorageWithoutFeedOptionsWithFromCrawler(S3FeedStorage):

59
tests/test_http_response.py
View File

@ -820,3 +820,62 @@ class XmlResponseTest(TextResponseTest):
response.xpath("//s1:elem/text()", namespaces={'s1': 'http://scrapy.org'}).getall(),
response.selector.xpath("//s2:elem/text()").getall(),
)
class CustomResponse(TextResponse):
attributes = TextResponse.attributes + ("foo", "bar")
def __init__(self, *args, **kwargs) -> None:
self.foo = kwargs.pop("foo", None)
self.bar = kwargs.pop("bar", None)
self.lost = kwargs.pop("lost", None)
super().__init__(*args, **kwargs)
class CustomResponseTest(TextResponseTest):
response_class = CustomResponse
def test_copy(self):
super().test_copy()
r1 = self.response_class(url="https://example.org", status=200, foo="foo", bar="bar", lost="lost")
r2 = r1.copy()
self.assertIsInstance(r2, self.response_class)
self.assertEqual(r1.foo, r2.foo)
self.assertEqual(r1.bar, r2.bar)
self.assertEqual(r1.lost, "lost")
self.assertIsNone(r2.lost)
def test_replace(self):
super().test_replace()
r1 = self.response_class(url="https://example.org", status=200, foo="foo", bar="bar", lost="lost")
r2 = r1.replace(foo="new-foo", bar="new-bar", lost="new-lost")
self.assertIsInstance(r2, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r2.foo, "new-foo")
self.assertEqual(r2.bar, "new-bar")
self.assertEqual(r2.lost, "new-lost")
r3 = r1.replace(foo="new-foo", bar="new-bar")
self.assertIsInstance(r3, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r3.foo, "new-foo")
self.assertEqual(r3.bar, "new-bar")
self.assertIsNone(r3.lost)
r4 = r1.replace(foo="new-foo")
self.assertIsInstance(r4, self.response_class)
self.assertEqual(r1.foo, "foo")
self.assertEqual(r1.bar, "bar")
self.assertEqual(r1.lost, "lost")
self.assertEqual(r4.foo, "new-foo")
self.assertEqual(r4.bar, "bar")
self.assertIsNone(r4.lost)
with self.assertRaises(TypeError) as ctx:
r1.replace(unknown="unknown")
self.assertEqual(str(ctx.exception), "__init__() got an unexpected keyword argument 'unknown'")

1
tox.ini
View File

@ -23,6 +23,7 @@ passenv =
S3_TEST_FILE_URI
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
GCS_TEST_FILE_URI
GCS_PROJECT_ID
#allow tox virtualenv to upgrade pip/wheel/setuptools