RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,505 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

102 Commits

Author SHA1 Message Date
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
hy 14f7f22555 fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware 2015-09-16 16:59:23 +08:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Artur Gaspar ca83a0b028 Support for returning deferreds in downloader middleware methods. 2015-09-01 13:22:43 -03:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Marven Sanchez 8771d1f79b Update HTTPCache middleware docs 2015-06-01 18:20:59 +08:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Julia Medina 7a92dae4c8 Change Scrapy log output through docs 2015-04-22 17:27:24 -03:00
Pablo Hoffman c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Julia Medina 4fb818a250 Run linkfix over current docs 2015-03-18 20:04:14 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Elias Dorneles 57a5ee0097 added example value to set for proxy meta key 2015-03-12 23:20:44 -03:00
Elias Dorneles f7031c08ff updating list of Request.meta special keys 2015-03-10 22:29:07 -03:00
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Pablo Hoffman 675fd5ba04 Merge pull request #898 from scrapy/download-timeout 
[MRG] DOC document download_timeout
 2014-10-24 16:52:42 -02:00
Pablo Hoffman 0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Mikhail Korobov 7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Daniel Graña a90fd81c97 add leveldb cache backend docs 2014-06-23 14:37:29 -03:00
Mikhail Korobov 2d3803672b DOC use top-level shortcuts in docs 2014-04-15 01:09:35 +06:00
Nikolaos-Digenis Karagiannis 43a797e2f7 downloaderMW doc typo (spiderMW doc copy remnant) 2014-02-11 22:30:00 +02:00
Rolando Espinoza a6279fe95b DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. 2014-01-30 11:53:42 -04:00
Mikhail Korobov 7b7a1d8dfd Make Filesystem storage backend default again. See GH-500. 2014-01-17 04:32:08 +06:00
Mikhail Korobov b03fe04999 Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware 2014-01-16 23:09:37 +06:00
Pablo Hoffman ed6fd4933f Merge pull request #524 from hobsonlane/master 
documentation code example corrections per pablohoffman
 2014-01-16 06:44:51 -08:00
Daniel Graña b9bb9bed6b Merge pull request #343 from kmike/ajax-crawlable 
[MRG] AjaxCrawlableMiddleware
 2014-01-16 05:07:39 -08:00
Hobson Lane 6ba0857a5c documentation code example correction corrections per pablohoffman 2014-01-10 10:37:27 -08:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Mikhail Korobov 943a0bd264 AjaxCrawlableMiddleware in Broad Crawl docs 2013-12-19 01:01:26 +06:00
Mikhail Korobov a87b3bd1c8 AjaxCrawlableMiddleware 2013-12-19 00:06:47 +06:00
Travis Briggs 3043a5ba37 DownloaderMiddleware docs: Update process_request, proper explanation of IgnoreRequest. 
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.

Further changes.
 2013-10-17 00:23:21 +00:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
Mikhail Korobov ac2fadf3ab DownloaderMiddleware.process_response docs fix 
"returns an exception" -> "raises an exception"
 2013-07-08 19:41:58 +06:00
Mikhail Korobov 39e5da5f66 improve docs for DownloaderMiddleware.process_response 2013-07-08 19:17:29 +06:00
nramirezuy 83b2774354 remove wrong default httpcache 2013-06-25 17:01:29 -03:00
Pablo Hoffman 96c2332e0e fix inaccurate downloader middleware documentation. refs #280 2013-04-02 11:35:32 -03:00