|
|
|
|
@ -678,7 +678,7 @@ Usability improvements
|
|
|
|
|
* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`)
|
|
|
|
|
* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`)
|
|
|
|
|
* non-zero exit code is returned from Scrapy commands when error happens
|
|
|
|
|
on spider inititalization (:issue:`3226`)
|
|
|
|
|
on spider initialization (:issue:`3226`)
|
|
|
|
|
* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`);
|
|
|
|
|
"flv" is added to common video extensions (:issue:`3165`)
|
|
|
|
|
* better error message when an exporter is disabled (:issue:`3358`);
|
|
|
|
|
@ -1156,7 +1156,7 @@ Bug fixes
|
|
|
|
|
- Fix :command:`view` command ; it was a regression in v1.3.0 (:issue:`2503`).
|
|
|
|
|
- Fix tests regarding ``*_EXPIRES settings`` with Files/Images pipelines (:issue:`2460`).
|
|
|
|
|
- Fix name of generated pipeline class when using basic project template (:issue:`2466`).
|
|
|
|
|
- Fix compatiblity with Twisted 17+ (:issue:`2496`, :issue:`2528`).
|
|
|
|
|
- Fix compatibility with Twisted 17+ (:issue:`2496`, :issue:`2528`).
|
|
|
|
|
- Fix ``scrapy.Item`` inheritance on Python 3.6 (:issue:`2511`).
|
|
|
|
|
- Enforce numeric values for components order in ``SPIDER_MIDDLEWARES``,
|
|
|
|
|
``DOWNLOADER_MIDDLEWARES``, ``EXTENIONS`` and ``SPIDER_CONTRACTS`` (:issue:`2420`).
|
|
|
|
|
@ -1164,7 +1164,7 @@ Bug fixes
|
|
|
|
|
Documentation
|
|
|
|
|
~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
- Reword Code of Coduct section and upgrade to Contributor Covenant v1.4
|
|
|
|
|
- Reword Code of Conduct section and upgrade to Contributor Covenant v1.4
|
|
|
|
|
(:issue:`2469`).
|
|
|
|
|
- Clarify that passing spider arguments converts them to spider attributes
|
|
|
|
|
(:issue:`2483`).
|
|
|
|
|
@ -1178,7 +1178,7 @@ Documentation
|
|
|
|
|
Cleanups
|
|
|
|
|
~~~~~~~~
|
|
|
|
|
|
|
|
|
|
- Remove reduntant check in ``MetaRefreshMiddleware`` (:issue:`2542`).
|
|
|
|
|
- Remove redundant check in ``MetaRefreshMiddleware`` (:issue:`2542`).
|
|
|
|
|
- Faster checks in ``LinkExtractor`` for allow/deny patterns (:issue:`2538`).
|
|
|
|
|
- Remove dead code supporting old Twisted versions (:issue:`2544`).
|
|
|
|
|
|
|
|
|
|
@ -1204,7 +1204,7 @@ New Features
|
|
|
|
|
- ``MailSender`` now accepts single strings as values for ``to`` and ``cc``
|
|
|
|
|
arguments (:issue:`2272`)
|
|
|
|
|
- ``scrapy fetch url``, ``scrapy shell url`` and ``fetch(url)`` inside
|
|
|
|
|
scrapy shell now follow HTTP redirections by default (:issue:`2290`);
|
|
|
|
|
Scrapy shell now follow HTTP redirections by default (:issue:`2290`);
|
|
|
|
|
See :command:`fetch` and :command:`shell` for details.
|
|
|
|
|
- ``HttpErrorMiddleware`` now logs errors with ``INFO`` level instead of ``DEBUG``;
|
|
|
|
|
this is technically **backward incompatible** so please check your log parsers.
|
|
|
|
|
@ -1705,7 +1705,7 @@ Scrapy 1.0.4 (2015-12-30)
|
|
|
|
|
- fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst (:commit:`ca8d60f`)
|
|
|
|
|
- Typos corrections (:commit:`7067117`)
|
|
|
|
|
- fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware (:commit:`32f115c`)
|
|
|
|
|
- Add note to ubuntu install section about debian compatibility (:commit:`23fda69`)
|
|
|
|
|
- Add note to Ubuntu install section about Debian compatibility (:commit:`23fda69`)
|
|
|
|
|
- Replace alternative OSX install workaround with virtualenv (:commit:`98b63ee`)
|
|
|
|
|
- Reference Homebrew's homepage for installation instructions (:commit:`1925db1`)
|
|
|
|
|
- Add oldest supported tox version to contributing docs (:commit:`5d10d6d`)
|
|
|
|
|
@ -1758,7 +1758,7 @@ Scrapy 1.0.1 (2015-07-01)
|
|
|
|
|
- include tests/ to source distribution in MANIFEST.in (:commit:`eca227e`)
|
|
|
|
|
- DOC Fix SelectJmes documentation (:commit:`b8567bc`)
|
|
|
|
|
- DOC Bring Ubuntu and Archlinux outside of Windows subsection (:commit:`392233f`)
|
|
|
|
|
- DOC remove version suffix from ubuntu package (:commit:`5303c66`)
|
|
|
|
|
- DOC remove version suffix from Ubuntu package (:commit:`5303c66`)
|
|
|
|
|
- DOC Update release date for 1.0 (:commit:`c89fa29`)
|
|
|
|
|
|
|
|
|
|
.. _release-1.0.0:
|
|
|
|
|
@ -2211,7 +2211,7 @@ Scrapy 0.24.2 (2014-07-08)
|
|
|
|
|
|
|
|
|
|
- Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute (:commit:`e5e8133`)
|
|
|
|
|
- there is not support for python3 yet (:commit:`3cd6146`)
|
|
|
|
|
- Update python compatible version set to debian packages (:commit:`fa5d76b`)
|
|
|
|
|
- Update python compatible version set to Debian packages (:commit:`fa5d76b`)
|
|
|
|
|
- DOC fix formatting in release notes (:commit:`c6a9e20`)
|
|
|
|
|
|
|
|
|
|
Scrapy 0.24.1 (2014-06-27)
|
|
|
|
|
@ -2229,12 +2229,12 @@ Enhancements
|
|
|
|
|
|
|
|
|
|
- Improve Scrapy top-level namespace (:issue:`494`, :issue:`684`)
|
|
|
|
|
- Add selector shortcuts to responses (:issue:`554`, :issue:`690`)
|
|
|
|
|
- Add new lxml based LinkExtractor to replace unmantained SgmlLinkExtractor
|
|
|
|
|
- Add new lxml based LinkExtractor to replace unmaintained SgmlLinkExtractor
|
|
|
|
|
(:issue:`559`, :issue:`761`, :issue:`763`)
|
|
|
|
|
- Cleanup settings API - part of per-spider settings **GSoC project** (:issue:`737`)
|
|
|
|
|
- Add UTF8 encoding header to templates (:issue:`688`, :issue:`762`)
|
|
|
|
|
- Telnet console now binds to 127.0.0.1 by default (:issue:`699`)
|
|
|
|
|
- Update debian/ubuntu install instructions (:issue:`509`, :issue:`549`)
|
|
|
|
|
- Update Debian/Ubuntu install instructions (:issue:`509`, :issue:`549`)
|
|
|
|
|
- Disable smart strings in lxml XPath evaluations (:issue:`535`)
|
|
|
|
|
- Restore filesystem based cache as default for http
|
|
|
|
|
cache middleware (:issue:`541`, :issue:`500`, :issue:`571`)
|
|
|
|
|
@ -2267,7 +2267,7 @@ Enhancements
|
|
|
|
|
- Tests and docs for ``request_fingerprint`` function (:issue:`597`)
|
|
|
|
|
- Update SEP-19 for GSoC project ``per-spider settings`` (:issue:`705`)
|
|
|
|
|
- Set exit code to non-zero when contracts fails (:issue:`727`)
|
|
|
|
|
- Add a setting to control what class is instanciated as Downloader component
|
|
|
|
|
- Add a setting to control what class is instantiated as Downloader component
|
|
|
|
|
(:issue:`738`)
|
|
|
|
|
- Pass response in ``item_dropped`` signal (:issue:`724`)
|
|
|
|
|
- Improve ``scrapy check`` contracts command (:issue:`733`, :issue:`752`)
|
|
|
|
|
@ -2276,7 +2276,7 @@ Enhancements
|
|
|
|
|
- Add a note about reporting security issues (:issue:`697`)
|
|
|
|
|
- Add LevelDB http cache storage backend (:issue:`626`, :issue:`500`)
|
|
|
|
|
- Sort spider list output of ``scrapy list`` command (:issue:`742`)
|
|
|
|
|
- Multiple documentation enhancemens and fixes
|
|
|
|
|
- Multiple documentation enhancements and fixes
|
|
|
|
|
(:issue:`575`, :issue:`587`, :issue:`590`, :issue:`596`, :issue:`610`,
|
|
|
|
|
:issue:`617`, :issue:`618`, :issue:`627`, :issue:`613`, :issue:`643`,
|
|
|
|
|
:issue:`654`, :issue:`675`, :issue:`663`, :issue:`711`, :issue:`714`)
|
|
|
|
|
@ -2321,19 +2321,19 @@ Scrapy 0.22.1 (released 2014-02-08)
|
|
|
|
|
- BaseSgmlLinkExtractor: Added unit test of a link with an inner tag (:commit:`c1cb418`)
|
|
|
|
|
- BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag (:commit:`7e4d627`)
|
|
|
|
|
- Fix tests for Travis-CI build (:commit:`76c7e20`)
|
|
|
|
|
- replace unencodeable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)
|
|
|
|
|
- replace unencodable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)
|
|
|
|
|
- RegexLinkExtractor: encode URL unicode value when creating Links (:commit:`d0ee545`)
|
|
|
|
|
- Updated the tutorial crawl output with latest output. (:commit:`8da65de`)
|
|
|
|
|
- Updated shell docs with the crawler reference and fixed the actual shell output. (:commit:`875b9ab`)
|
|
|
|
|
- PEP8 minor edits. (:commit:`f89efaf`)
|
|
|
|
|
- Expose current crawler in the scrapy shell. (:commit:`5349cec`)
|
|
|
|
|
- Expose current crawler in the Scrapy shell. (:commit:`5349cec`)
|
|
|
|
|
- Unused re import and PEP8 minor edits. (:commit:`387f414`)
|
|
|
|
|
- Ignore None's values when using the ItemLoader. (:commit:`0632546`)
|
|
|
|
|
- DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. (:commit:`cde9a8c`)
|
|
|
|
|
- show ubuntu setup instructions as literal code (:commit:`fb5c9c5`)
|
|
|
|
|
- show Ubuntu setup instructions as literal code (:commit:`fb5c9c5`)
|
|
|
|
|
- Update Ubuntu installation instructions (:commit:`70fb105`)
|
|
|
|
|
- Merge pull request #550 from stray-leone/patch-1 (:commit:`6f70b6a`)
|
|
|
|
|
- modify the version of scrapy ubuntu package (:commit:`725900d`)
|
|
|
|
|
- modify the version of Scrapy Ubuntu package (:commit:`725900d`)
|
|
|
|
|
- fix 0.22.0 release date (:commit:`af0219a`)
|
|
|
|
|
- fix typos in news.rst and remove (not released yet) header (:commit:`b7f58f4`)
|
|
|
|
|
|
|
|
|
|
@ -2354,7 +2354,7 @@ Enhancements
|
|
|
|
|
- Improve test coverage and forthcoming Python 3 support (:issue:`525`)
|
|
|
|
|
- Promote startup info on settings and middleware to INFO level (:issue:`520`)
|
|
|
|
|
- Support partials in ``get_func_args`` util (:issue:`506`, issue:`504`)
|
|
|
|
|
- Allow running indiviual tests via tox (:issue:`503`)
|
|
|
|
|
- Allow running individual tests via tox (:issue:`503`)
|
|
|
|
|
- Update extensions ignored by link extractors (:issue:`498`)
|
|
|
|
|
- Add middleware methods to get files/images/thumbs paths (:issue:`490`)
|
|
|
|
|
- Improve offsite middleware tests (:issue:`478`)
|
|
|
|
|
@ -2411,7 +2411,7 @@ Enhancements
|
|
|
|
|
- scrapy.mail.MailSender now can connect over TLS or upgrade using STARTTLS (:issue:`327`)
|
|
|
|
|
- New FilesPipeline with functionality factored out from ImagesPipeline (:issue:`370`, :issue:`409`)
|
|
|
|
|
- Recommend Pillow instead of PIL for image handling (:issue:`317`)
|
|
|
|
|
- Added debian packages for Ubuntu quantal and raring (:commit:`86230c0`)
|
|
|
|
|
- Added Debian packages for Ubuntu Quantal and raring (:commit:`86230c0`)
|
|
|
|
|
- Mock server (used for tests) can listen for HTTPS requests (:issue:`410`)
|
|
|
|
|
- Remove multi spider support from multiple core components
|
|
|
|
|
(:issue:`422`, :issue:`421`, :issue:`420`, :issue:`419`, :issue:`423`, :issue:`418`)
|
|
|
|
|
@ -2430,7 +2430,7 @@ Bugfixes
|
|
|
|
|
- Fix tests under Django 1.6 (:commit:`b6bed44c`)
|
|
|
|
|
- Lot of bugfixes to retry middleware under disconnections using HTTP 1.1 download handler
|
|
|
|
|
- Fix inconsistencies among Twisted releases (:issue:`406`)
|
|
|
|
|
- Fix scrapy shell bugs (:issue:`418`, :issue:`407`)
|
|
|
|
|
- Fix Scrapy shell bugs (:issue:`418`, :issue:`407`)
|
|
|
|
|
- Fix invalid variable name in setup.py (:issue:`429`)
|
|
|
|
|
- Fix tutorial references (:issue:`387`)
|
|
|
|
|
- Improve request-response docs (:issue:`391`)
|
|
|
|
|
@ -2512,15 +2512,15 @@ Scrapy 0.18.1 (released 2013-08-27)
|
|
|
|
|
- test PotentiaDataLoss errors on unbound responses (:commit:`b15470d`)
|
|
|
|
|
- Treat responses without content-length or Transfer-Encoding as good responses (:commit:`c4bf324`)
|
|
|
|
|
- do no include ResponseFailed if http11 handler is not enabled (:commit:`6cbe684`)
|
|
|
|
|
- New HTTP client wraps connection losts in ResponseFailed exception. fix #373 (:commit:`1a20bba`)
|
|
|
|
|
- New HTTP client wraps connection lost in ResponseFailed exception. fix #373 (:commit:`1a20bba`)
|
|
|
|
|
- limit travis-ci build matrix (:commit:`3b01bb8`)
|
|
|
|
|
- Merge pull request #375 from peterarenot/patch-1 (:commit:`fa766d7`)
|
|
|
|
|
- Fixed so it refers to the correct folder (:commit:`3283809`)
|
|
|
|
|
- added quantal & raring to support ubuntu releases (:commit:`1411923`)
|
|
|
|
|
- added Quantal & raring to support Ubuntu releases (:commit:`1411923`)
|
|
|
|
|
- fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 (:commit:`bb35ed0`)
|
|
|
|
|
- fix XmlItemExporter in Python 2.7.4 and 2.7.5 (:commit:`de3e451`)
|
|
|
|
|
- minor updates to 0.18 release notes (:commit:`c45e5f1`)
|
|
|
|
|
- fix contributters list format (:commit:`0b60031`)
|
|
|
|
|
- fix contributors list format (:commit:`0b60031`)
|
|
|
|
|
|
|
|
|
|
Scrapy 0.18.0 (released 2013-08-09)
|
|
|
|
|
-----------------------------------
|
|
|
|
|
@ -2617,7 +2617,7 @@ contributors sorted by number of commits::
|
|
|
|
|
Scrapy 0.16.5 (released 2013-05-30)
|
|
|
|
|
-----------------------------------
|
|
|
|
|
|
|
|
|
|
- obey request method when scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)
|
|
|
|
|
- obey request method when Scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)
|
|
|
|
|
- fix inaccurate downloader middleware documentation. refs #280 (:commit:`40667cb`)
|
|
|
|
|
- doc: remove links to diveintopython.org, which is no longer available. closes #246 (:commit:`bd58bfa`)
|
|
|
|
|
- Find form nodes in invalid html5 documents (:commit:`e3d6945`)
|
|
|
|
|
@ -2631,8 +2631,8 @@ Scrapy 0.16.4 (released 2013-01-23)
|
|
|
|
|
- Fixed error message formatting. log.err() doesn't support cool formatting and when error occurred, the message was: "ERROR: Error processing %(item)s" (:commit:`c16150c`)
|
|
|
|
|
- lint and improve images pipeline error logging (:commit:`56b45fc`)
|
|
|
|
|
- fixed doc typos (:commit:`243be84`)
|
|
|
|
|
- add documentation topics: Broad Crawls & Common Practies (:commit:`1fbb715`)
|
|
|
|
|
- fix bug in scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)
|
|
|
|
|
- add documentation topics: Broad Crawls & Common Practices (:commit:`1fbb715`)
|
|
|
|
|
- fix bug in Scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)
|
|
|
|
|
- Update docs/topics/commands.rst (:commit:`28eac7a`)
|
|
|
|
|
|
|
|
|
|
Scrapy 0.16.3 (released 2012-12-07)
|
|
|
|
|
@ -2651,11 +2651,11 @@ Scrapy 0.16.3 (released 2012-12-07)
|
|
|
|
|
Scrapy 0.16.2 (released 2012-11-09)
|
|
|
|
|
-----------------------------------
|
|
|
|
|
|
|
|
|
|
- scrapy contracts: python2.6 compat (:commit:`a4a9199`)
|
|
|
|
|
- scrapy contracts verbose option (:commit:`ec41673`)
|
|
|
|
|
- proper unittest-like output for scrapy contracts (:commit:`86635e4`)
|
|
|
|
|
- Scrapy contracts: python2.6 compat (:commit:`a4a9199`)
|
|
|
|
|
- Scrapy contracts verbose option (:commit:`ec41673`)
|
|
|
|
|
- proper unittest-like output for Scrapy contracts (:commit:`86635e4`)
|
|
|
|
|
- added open_in_browser to debugging doc (:commit:`c9b690d`)
|
|
|
|
|
- removed reference to global scrapy stats from settings doc (:commit:`dd55067`)
|
|
|
|
|
- removed reference to global Scrapy stats from settings doc (:commit:`dd55067`)
|
|
|
|
|
- Fix SpiderState bug in Windows platforms (:commit:`58998f4`)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@ -2665,7 +2665,7 @@ Scrapy 0.16.1 (released 2012-10-26)
|
|
|
|
|
- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release (:commit:`8c780fd`)
|
|
|
|
|
- better backward compatibility for scrapy.conf.settings (:commit:`3403089`)
|
|
|
|
|
- extended documentation on how to access crawler stats from extensions (:commit:`c4da0b5`)
|
|
|
|
|
- removed .hgtags (no longer needed now that scrapy uses git) (:commit:`d52c188`)
|
|
|
|
|
- removed .hgtags (no longer needed now that Scrapy uses git) (:commit:`d52c188`)
|
|
|
|
|
- fix dashes under rst headers (:commit:`fa4f7f9`)
|
|
|
|
|
- set release date for 0.16.0 in news (:commit:`e292246`)
|
|
|
|
|
|
|
|
|
|
@ -2680,8 +2680,7 @@ Scrapy changes:
|
|
|
|
|
- documented :doc:`topics/autothrottle` and added to extensions installed by default. You still need to enable it with :setting:`AUTOTHROTTLE_ENABLED`
|
|
|
|
|
- major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (``stats_spider_opened``, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.
|
|
|
|
|
- added :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests` method to spider middlewares
|
|
|
|
|
- dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info.
|
|
|
|
|
- dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info.
|
|
|
|
|
- dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.
|
|
|
|
|
- dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.
|
|
|
|
|
- documented :ref:`topics-api`
|
|
|
|
|
- ``lxml`` is now the default selectors backend instead of ``libxml2``
|
|
|
|
|
@ -2715,7 +2714,7 @@ Scrapy changes:
|
|
|
|
|
Scrapy 0.14.4
|
|
|
|
|
-------------
|
|
|
|
|
|
|
|
|
|
- added precise to supported ubuntu distros (:commit:`b7e46df`)
|
|
|
|
|
- added precise to supported Ubuntu distros (:commit:`b7e46df`)
|
|
|
|
|
- fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py (:commit:`340fbdb`)
|
|
|
|
|
- meta tag attributes for content-type http equiv can be in any order. #123 (:commit:`0cb68af`)
|
|
|
|
|
- replace "import Image" by more standard "from PIL import Image". closes #88 (:commit:`4d17048`)
|
|
|
|
|
@ -2728,11 +2727,11 @@ Scrapy 0.14.3
|
|
|
|
|
- include egg files used by testsuite in source distribution. #118 (:commit:`c897793`)
|
|
|
|
|
- update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (:commit:`2548dcc`)
|
|
|
|
|
- added note to docs/topics/firebug.rst about google directory being shut down (:commit:`668e352`)
|
|
|
|
|
- dont discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)
|
|
|
|
|
- don't discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)
|
|
|
|
|
- do not fail handling unicode xpaths in libxml2 backed selectors (:commit:`b830e95`)
|
|
|
|
|
- fixed minor mistake in Request objects documentation (:commit:`bf3c9ee`)
|
|
|
|
|
- fixed minor defect in link extractors documentation (:commit:`ba14f38`)
|
|
|
|
|
- removed some obsolete remaining code related to sqlite support in scrapy (:commit:`0665175`)
|
|
|
|
|
- removed some obsolete remaining code related to sqlite support in Scrapy (:commit:`0665175`)
|
|
|
|
|
|
|
|
|
|
Scrapy 0.14.2
|
|
|
|
|
-------------
|
|
|
|
|
|
Marc Hernandez Cabot