diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index a190ce407..0629b9e19 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -78,9 +78,9 @@ Our first Spider Spiders are classes that you define and that Scrapy uses to scrape information from a website (or a group of websites). They must subclass -:class:`scrapy.Spider` and define the initial requests to make, optionally how -to follow links in the pages, and how to parse the downloaded page content to -extract data. +:class:`~scrapy.spiders.Spider` and define the initial requests to make, +optionally how to follow links in the pages, and how to parse the downloaded +page content to extract data. This is the code for our first Spider. Save it in a file named ``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:: diff --git a/docs/news.rst b/docs/news.rst index aac750601..c107e907c 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -71,7 +71,7 @@ New features ~~~~~~~~~~~~ * A new scheduler priority queue, - :class:`scrapy.pqueues.DownloaderAwarePriorityQueue`, may be + ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be :ref:`enabled ` for a significant scheduling improvement on crawls targetting multiple web domains, at the cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`) @@ -150,9 +150,9 @@ Bug fixes :setting:`AWS_REGION_NAME`, :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY` (:issue:`3625`) -* Fixed a memory leak in :class:`~scrapy.pipelines.media.MediaPipeline` - affecting, for example, non-200 responses and exceptions from custom - middlewares (:issue:`3813`) +* Fixed a memory leak in ``scrapy.pipelines.media.MediaPipeline`` affecting, + for example, non-200 responses and exceptions from custom middlewares + (:issue:`3813`) * Requests with private callbacks are now correctly unserialized from disk (:issue:`3790`) @@ -301,7 +301,7 @@ Deprecations * The ``queuelib.PriorityQueue`` value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is deprecated. Use - :class:`scrapy.pqueues.ScrapyPriorityQueue` instead. + ``scrapy.pqueues.ScrapyPriorityQueue`` instead. * ``process_request`` callbacks passed to :class:`~scrapy.spiders.Rule` that do not accept two arguments are deprecated. @@ -551,7 +551,7 @@ Scrapy 1.5.2 (2019-01-22) *The fix is backward incompatible*, it enables telnet user-password authentication by default with a random generated password. If you can't - upgrade right away, please consider setting :setting:`TELNET_CONSOLE_PORT` + upgrade right away, please consider setting :setting:`TELNETCONSOLE_PORT` out of its default value. See :ref:`telnet console ` documentation for more info @@ -758,7 +758,9 @@ Enjoy! (Or read on for the rest of changes in this release.) Deprecations and Backward Incompatible Changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor` +- Default to ``canonicalize=False`` in + :class:`scrapy.linkextractors.LinkExtractor + ` (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`): **warning, this is technically backward-incompatible** - Enable memusage extension by default (:issue:`2539`, fixes :issue:`2187`); @@ -794,10 +796,13 @@ New Features - New ``data:`` URI download handler (:issue:`2334`, fixes :issue:`2156`) - Log cache directory when HTTP Cache is used (:issue:`2611`, fixes :issue:`2604`) - Warn users when project contains duplicate spider names (fixes :issue:`2181`) -- :class:`CaselessDict` now accepts ``Mapping`` instances and not only dicts (:issue:`2646`) -- :ref:`Media downloads `, with :class:`FilesPipelines` - or :class:`ImagesPipelines`, can now optionally handle HTTP redirects - using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting (:issue:`2616`, fixes :issue:`2004`) +- ``scrapy.utils.datatypes.CaselessDict`` now accepts ``Mapping`` instances and + not only dicts (:issue:`2646`) +- :ref:`Media downloads `, with + :class:`~scrapy.pipelines.files.FilesPipeline` or + :class:`~scrapy.pipelines.images.ImagesPipeline`, can now optionally handle + HTTP redirects using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting + (:issue:`2616`, fixes :issue:`2004`) - Accept non-complete responses from websites using a new :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:issue:`2590`, fixes :issue:`2586`) - Optional pretty-printing of JSON and XML items via @@ -817,8 +822,8 @@ Bug fixes - LinkExtractor now strips leading and trailing whitespaces from attributes (:issue:`2547`, fixes :issue:`1614`) -- Properly handle whitespaces in action attribute in :class:`FormRequest` - (:issue:`2548`) +- Properly handle whitespaces in action attribute in + :class:`~scrapy.http.FormRequest` (:issue:`2548`) - Buffer CONNECT response bytes from proxy until all HTTP headers are received (:issue:`2495`, fixes :issue:`2491`) - FTP downloader now works on Python 3, provided you use Twisted>=17.1 @@ -851,7 +856,8 @@ Cleanups & Refactoring fixes :issue:`2560`) - Add omitted ``self`` arguments in default project middleware template (:issue:`2595`) - Remove redundant ``slot.add_request()`` call in ExecutionEngine (:issue:`2617`) -- Catch more specific ``os.error`` exception in :class:`FSFilesStore` (:issue:`2644`) +- Catch more specific ``os.error`` exception in + ``scrapy.pipelines.files.FSFilesStore`` (:issue:`2644`) - Change "localhost" test server certificate (:issue:`2720`) - Remove unused ``MEMUSAGE_REPORT`` setting (:issue:`2576`) @@ -868,7 +874,8 @@ Documentation (:issue:`2477`, fixes :issue:`2475`) - FAQ: rewrite note on Python 3 support on Windows (:issue:`2690`) - Rearrange selector sections (:issue:`2705`) -- Remove ``__nonzero__`` from :class:`SelectorList` docs (:issue:`2683`) +- Remove ``__nonzero__`` from :class:`~scrapy.selector.SelectorList` + docs (:issue:`2683`) - Mention how to disable request filtering in documentation of :setting:`DUPEFILTER_CLASS` setting (:issue:`2714`) - Add sphinx_rtd_theme to docs setup readme (:issue:`2668`) @@ -2327,7 +2334,7 @@ Scrapy 0.18.0 (released 2013-08-09) - Moved persistent (on disk) queues to a separate project (queuelib_) which scrapy now depends on - Add scrapy commands using external libraries (:issue:`260`) - Added ``--pdb`` option to ``scrapy`` command line tool -- Added :meth:`XPathSelector.remove_namespaces` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`. +- Added :meth:`XPathSelector.remove_namespaces ` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`. - Several improvements to spider contracts - New default middleware named MetaRefreshMiddldeware that handles meta-refresh html tag redirections, - MetaRefreshMiddldeware and RedirectMiddleware have different priorities to address #62 @@ -2448,7 +2455,7 @@ Scrapy changes: - added options ``-o`` and ``-t`` to the :command:`runspider` command - documented :doc:`topics/autothrottle` and added to extensions installed by default. You still need to enable it with :setting:`AUTOTHROTTLE_ENABLED` - major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (``stats_spider_opened``, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals. -- added :meth:`~scrapy.contrib.spidermiddleware.SpiderMiddleware.process_start_requests` method to spider middlewares +- added :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests` method to spider middlewares - dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info. - dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info. - dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info. @@ -2609,7 +2616,8 @@ The numbers like #NNN reference tickets in the old issue tracker (Trac) which is New features and improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Passed item is now sent in the ``item`` argument of the :signal:`item_passed` (#273) +- Passed item is now sent in the ``item`` argument of the :signal:`item_passed + ` (#273) - Added verbose option to ``scrapy version`` command, useful for bug reports (#298) - HTTP cache now stored by default in the project data dir (#279) - Added project data storage directory (#276, #277) diff --git a/docs/topics/items.rst b/docs/topics/items.rst index 260f5882c..fbc888f47 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -240,6 +240,10 @@ Item objects Items replicate the standard `dict API`_, including its constructor. The only additional attribute provided by Items is: + .. automethod:: copy + + .. automethod:: deepcopy + .. attribute:: fields A dictionary containing *all declared fields* for this Item, not only diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 75e0af63b..350c3c4b9 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -796,6 +796,7 @@ Default: ``True`` Whether or not to use passive mode when initiating FTP transfers. +.. reqmeta:: ftp_password .. setting:: FTP_PASSWORD FTP_PASSWORD @@ -814,6 +815,7 @@ in ``Request`` meta. .. _RFC 1635: https://tools.ietf.org/html/rfc1635 +.. reqmeta:: ftp_user .. setting:: FTP_USER FTP_USER