diff --git a/docs/README.rst b/docs/README.rst deleted file mode 100644 index 4d2236b53..000000000 --- a/docs/README.rst +++ /dev/null @@ -1,68 +0,0 @@ -:orphan: - -====================================== -Scrapy documentation quick start guide -====================================== - -This file provides a quick guide on how to compile the Scrapy documentation. - - -Setup the environment ---------------------- - -To compile the documentation you need Sphinx Python library. To install it -and all its dependencies run the following command from this dir - -:: - - pip install -r requirements.txt - - -Compile the documentation -------------------------- - -To compile the documentation (to classic HTML output) run the following command -from this dir:: - - make html - -Documentation will be generated (in HTML format) inside the ``build/html`` dir. - - -View the documentation ----------------------- - -To view the documentation run the following command:: - - make htmlview - -This command will fire up your default browser and open the main page of your -(previously generated) HTML documentation. - - -Start over ----------- - -To clean up all generated documentation files and start from scratch run:: - - make clean - -Keep in mind that this command won't touch any documentation source files. - - -Recreating documentation on the fly ------------------------------------ - -There is a way to recreate the doc automatically when you make changes, you -need to install watchdog (``pip install watchdog``) and then use:: - - make watch - -Alternative method using tox ----------------------------- - -To compile the documentation to HTML run the following command:: - - tox -e docs - -Documentation will be generated inside the ``docs/_build/all`` dir. diff --git a/docs/contributing.rst b/docs/contributing.rst index c868a0ac4..b43d30796 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -323,9 +323,10 @@ deprecation removals are documented in the :ref:`release notes `. Tests ===== -Tests are implemented using the :doc:`Twisted unit-testing framework -`. Running tests requires -:doc:`tox `. +Tests are implemented using pytest_. Running tests requires :doc:`tox +`. + +.. _pytest: https://pytest.org .. _running-tests: diff --git a/docs/faq.rst b/docs/faq.rst index 6eeb81a31..a396a8212 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -285,7 +285,8 @@ consume a lot of memory. In order to avoid parsing all the entire feed at once in memory, you can use the :func:`~scrapy.utils.iterators.xmliter_lxml` and :func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what -:class:`~scrapy.spiders.XMLFeedSpider` uses. +:class:`~scrapy.spiders.XMLFeedSpider` and +:class:`~scrapy.spiders.CSVFeedSpider` use. .. autofunction:: scrapy.utils.iterators.xmliter_lxml @@ -360,10 +361,11 @@ method for this purpose. For example: def process_spider_output(self, response, result): for item_or_request in result: if isinstance(item_or_request, Request): + yield item_or_request continue - adapter = ItemAdapter(item) + adapter = ItemAdapter(item_or_request) for _ in range(adapter["multiply_by"]): - yield deepcopy(item) + yield deepcopy(item_or_request) Does Scrapy support IPv6 addresses? ----------------------------------- diff --git a/docs/index.rst b/docs/index.rst index 8e8624a22..688cab81b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -24,7 +24,7 @@ Having trouble? We'd like to help! * Ask or search questions in `StackOverflow using the scrapy tag`_. * Ask or search questions in the `Scrapy subreddit`_. * Search for questions on the archives of the `scrapy-users mailing list`_. -* Ask a question in the `#scrapy IRC channel`_, +* Ask a question in the `#scrapy IRC channel`_. * Report bugs with Scrapy in our `issue tracker`_. * Join the Discord community `Scrapy Discord`_. @@ -91,15 +91,15 @@ Basic concepts :doc:`topics/selectors` Extract the data from web pages using XPath. -:doc:`topics/shell` - Test your extraction code in an interactive environment. - :doc:`topics/items` Define the data you want to scrape. :doc:`topics/loaders` Populate your items with the extracted data. +:doc:`topics/shell` + Test your extraction code in an interactive environment. + :doc:`topics/item-pipeline` Post-process and store your scraped data. diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index ee91ce7ca..488e1c2e3 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -90,9 +90,10 @@ to figure these settings out automatically. .. note:: This is using :ref:`feed exports ` to generate the - JSON file, you can easily change the export format (XML or CSV, for example) or the - storage backend (FTP or `Amazon S3`_, for example). You can also write an - :ref:`item pipeline ` to store the items in a database. + JSON Lines file, you can easily change the export format (XML or CSV, for + example) or the storage backend (FTP or `Amazon S3`_, for example). You can + also write an :ref:`item pipeline ` to store the + items in a database. .. _topics-whatelse: diff --git a/docs/news.rst b/docs/news.rst index 7d2b07b09..63f0d69e4 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1035,7 +1035,7 @@ Deprecations New features ~~~~~~~~~~~~ -- Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing +- Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing supported referrer policies. Bug fixes @@ -6046,7 +6046,7 @@ The following changes may impact custom priority queue classes: * A new keyword parameter has been added: ``key``. It is a string that is always an empty string for memory queues and indicates the - :setting:`JOB_DIR` value for disk queues. + :setting:`JOBDIR` value for disk queues. * The parameter for disk queues that contains data from the previous crawl, ``startprios`` or ``slot_startprios``, is now passed as a @@ -6600,7 +6600,7 @@ New features ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be :ref:`enabled ` for a significant scheduling improvement on crawls targeting multiple web domains, at the - cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`) + cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:issue:`3520`) * A new :attr:`.Request.cb_kwargs` attribute provides a cleaner way to pass keyword arguments to callback methods @@ -9079,7 +9079,7 @@ New features and settings - In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`) - Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`) - ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by: - - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`CONCURRENT_REQUESTS_PER_IP` + - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP`` - check the documentation for more details - Added builtin caching DNS resolver (:rev:`2728`) - Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`) diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 1a151044f..17b322b5a 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -105,6 +105,9 @@ Scrapy API requires passing a Deferred to it) using the following helpers: .. autofunction:: scrapy.utils.defer.deferred_from_coro .. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f + +The following function helps with a reverse wrapping: + .. autofunction:: scrapy.utils.defer.ensure_awaitable @@ -190,7 +193,7 @@ in future Scrapy versions. The following features are not available: :class:`~scrapy.crawler.CrawlerProcess` (:class:`~scrapy.crawler.AsyncCrawlerProcess` and :class:`~scrapy.crawler.AsyncCrawlerRunner` are available) -* Twisted-specific DNS resolvers (the :setting:`DNS_RESOLVER` setting) +* Twisted-specific DNS resolvers (the :setting:`TWISTED_DNS_RESOLVER` setting) * User and 3rd-party code that requires a reactor (see :ref:`below ` for examples) @@ -315,8 +318,7 @@ implementations, :class:`~asyncio.ProactorEventLoop` (default) and :class:`~asyncio.SelectorEventLoop` works with Twisted. Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop` -automatically when you change the :setting:`TWISTED_REACTOR` setting or call -:func:`~scrapy.utils.reactor.install_reactor`. +automatically when installing the asyncio reactor. .. note:: Other libraries you use may require :class:`~asyncio.ProactorEventLoop`, e.g. because it supports diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 8d1351eb9..270d9859d 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -199,6 +199,7 @@ Global commands: * :command:`fetch` * :command:`view` * :command:`version` +* :command:`bench` Project-only commands: @@ -207,7 +208,6 @@ Project-only commands: * :command:`list` * :command:`edit` * :command:`parse` -* :command:`bench` .. command:: startproject @@ -476,7 +476,7 @@ Supported options: * ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider -* ``--a NAME=VALUE``: set spider argument (may be repeated) +* ``-a NAME=VALUE``: set spider argument (may be repeated) * ``--callback`` or ``-c``: spider method to use as callback for parsing the response diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 116aac323..8aaf0aa31 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -268,5 +268,5 @@ You can also send multiple requests in parallel: yield { "h1": response.css("h1::text").get(), "price": responses[0].css(".price::text").get(), - "price2": responses[1].css(".color::text").get(), + "color": responses[1].css(".color::text").get(), } diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 054c1130c..383f331b5 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -505,7 +505,7 @@ Filesystem storage backend (default) * ``response_body`` - the plain response body - * ``response_headers`` - the request headers (in raw HTTP format) + * ``response_headers`` - the response headers (in raw HTTP format) * ``meta`` - some metadata of this cache resource in Python ``repr()`` format (grep-friendly format) @@ -547,7 +547,7 @@ defines the methods described below. .. method:: open_spider(spider) This method gets called after a spider has been opened for crawling. It handles - the :signal:`open_spider ` signal. + the :signal:`spider_opened` signal. :param spider: the spider which has been opened :type spider: :class:`~scrapy.Spider` object @@ -555,7 +555,7 @@ defines the methods described below. .. method:: close_spider(spider) This method gets called after a spider has been closed. It handles - the :signal:`close_spider ` signal. + the :signal:`spider_closed` signal. :param spider: the spider which has been closed :type spider: :class:`~scrapy.Spider` object @@ -814,7 +814,6 @@ HttpProxyMiddleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. setting:: HTTPPROXY_ENABLED -.. setting:: HTTPPROXY_AUTH_ENCODING HTTPPROXY_ENABLED ^^^^^^^^^^^^^^^^^ @@ -823,6 +822,8 @@ Default: ``True`` Whether or not to enable the :class:`HttpProxyMiddleware`. +.. setting:: HTTPPROXY_AUTH_ENCODING + HTTPPROXY_AUTH_ENCODING ^^^^^^^^^^^^^^^^^^^^^^^ @@ -984,7 +985,7 @@ Whether the Meta Refresh middleware will be enabled. METAREFRESH_IGNORE_TAGS ^^^^^^^^^^^^^^^^^^^^^^^ -Default: ``[]`` +Default: ``["noscript"]`` Meta tags within these tags are ignored. @@ -1091,7 +1092,7 @@ Default:: 'twisted.internet.error.ConnectionDone', 'twisted.internet.error.ConnectError', 'twisted.internet.error.ConnectionLost', - IOError, + OSError, 'scrapy.core.downloader.handlers.http11.TunnelError', ] diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 5a399e094..30b6536c7 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -133,7 +133,7 @@ data from it depends on the type of response: .. code-block:: python - selector = Selector(data["html"]) + selector = Selector(text=data["html"]) - If the response is JavaScript, or HTML with a ``