RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,590 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

4590 Commits

Author SHA1 Message Date
Alexandr N Zamaraev (aka tonal) 3d397de0e9 Possible set log-level in LogFormatter.dropped and LogFormatter.scraped 2012-12-28 13:40:27 +07:00
Alexandr N Zamaraev (aka tonal) 21de68b766 Remove firget imports 2012-12-28 13:01:39 +07:00
Hasnain Lakhani 93a1102189 Implemented policies for HTTP Cache 2012-12-26 16:29:48 -08:00
Pablo Hoffman 51b8feb4ce fixed doc typos 2012-12-26 16:16:53 -02:00
Pablo Hoffman 1e2ee76df2 add documentation topics: Broad Crawls & Common Practies 2012-12-26 14:02:13 -02:00
Pedro Faustino fdaa35f6e8 Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching. 2012-12-24 19:37:53 +01:00
Pedro Faustino b2d3f4dd1b Add cache storage support for real HTTP caching. Add real HTTP caching unit tests for both middleware and cache storage. 2012-12-24 16:15:04 +01:00
Pedro Faustino 0e435fb5f9 Add middleware support for the 'no-store' Cache-Control directive. 2012-12-24 13:27:13 +01:00
Pedro Faustino bb55f39aed Add middleware support for real HTTP caching. Add httpcache setting to allow either real or dummy HTTP caching (for backwards compatibility it's set to use dummy cache by default). 2012-12-24 01:52:45 +01:00
Pedro Faustino e396509f5b Removing plural from httpcache stats' value names. 2012-12-24 00:24:27 +01:00
Pablo Hoffman dabb064293 fix bug in scrapy parse command when spider is not specified explicitly. closes #209 2012-12-20 11:59:07 -02:00
Pablo Hoffman 12475fccbe Merge pull request #206 from dangra/downloader-enhancements 
AutoThrottle and Downloader enhancements
 2012-12-17 10:11:46 -08:00
Daniel Graña 22559833e7 core: drop download inactive slots and get slot key from meta 2012-12-17 16:09:01 -02:00
Daniel Graña eb2d87259f core: move download slot assignment post middleware evaluation 2012-12-17 16:09:01 -02:00
Daniel Graña d7daf836d5 Altering delay is enough to auto throttle downloads 2012-12-17 16:08:49 -02:00
Daniel Graña 85c9881724 fix pylint and pep8 warnings 2012-12-14 12:27:53 -02:00
joehillen c505b33672 PEP8 on pydispatch/robustapply.py 2012-12-12 08:53:32 -08:00
joehillen 8778af5c5b Explicitly check if an object is a class is pydispatch. 
* This is to fix a disagreement in between CPython and PyPy.
* https://gist.github.com/4220533
 2012-12-12 08:42:18 -08:00
Pablo Hoffman 0bc5e4f9ec Merge pull request #204 from LuanP/patch-1 
Update docs/topics/commands.rst
 2012-12-10 16:19:25 -08:00
Luan 5582ea28ec Update docs/topics/commands.rst 
A short change.
 2012-12-10 15:16:02 -02:00
Daniel Graña 0cc138c010 Add 0.16.3 release notes 2012-12-07 18:55:15 -02:00
Daniel Graña e676ad31d0 Support sending requests trough multiples slots in QPS exmaple spider 2012-12-05 17:06:38 -02:00
Daniel Graña fd58de493c Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced 2012-12-05 15:08:34 -02:00
Daniel Graña dcc0a540c0 QPS bench server and a useful spider spider to generate requests 2012-12-05 15:08:34 -02:00
Daniel Graña a958fb11f0 Merge pull request #202 from saxicek/master 
Image pipeline error improvement
 2012-12-03 03:06:47 -08:00
Libor Nenadál 5f899aa4ec add error details when image pipeline fails 2012-12-02 23:29:26 +01:00
Pablo Hoffman 9c9a18b3a3 Merge pull request #201 from alexcepoi/test-fixes-mac 
improve mac os compatibility
 2012-12-02 07:37:53 -08:00
Alex Cepoi fc405e98aa improve mac os compatibility 
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
  of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
 2012-12-01 16:39:58 +01:00
Pablo Hoffman b9a96147ed setup.py: use README.rst to populate long_description 2012-11-25 22:22:33 -02:00
Pablo Hoffman 39274a2457 doc: removed obsolete references to ClientForm 2012-11-23 19:06:47 -02:00
Pablo Hoffman 8ca2ee3d60 Merge pull request #196 from stav/master 
The default storage backend is now DbmCacheStorage
 2012-11-22 13:02:15 -08:00
stav 99f164fc87 correct docs for default storage backend 2012-11-22 14:05:47 -06:00
Pablo Hoffman 1f0d167037 doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:26 -02:00
Pablo Hoffman 0a00e0fd63 Merge pull request #195 from kalessin/floatdelay 
download delay in autothrottle was being casted as int, should be float
 2012-11-15 11:18:45 -08:00
Martin Olveyra 421dba98a7 download delay in autothrottle was being casted as int, should be float 2012-11-15 18:41:32 +00:00
Pablo Hoffman 3a60d42072 removed unused import 2012-11-15 16:29:07 -02:00
Pablo Hoffman 5ebaa40ed2 changed SSL version to use from SSLv23 to TLSv1. closes #194 but needs more testing against counter-effects 2012-11-15 16:28:39 -02:00
Pablo Hoffman cba3967667 Merge pull request #192 from vkrest/master 
Fix broken doc links.
 2012-11-12 13:37:52 -08:00
vkrest ac4c66f351 Fix broken doc links. 
All generated template files contain broken link to doc files.
 2012-11-12 22:43:42 +02:00
Pablo Hoffman 14179a16a1 Merge pull request #190 from pedrofaustino/master 
Adding support to 'nofollow' in the SGML link extractors
 2012-11-12 10:09:30 -08:00
Pedro Faustino 3d0e962cbb The Link object has a 'nofollow' attribute. Adding support in the SGML link extractors. 2012-11-10 16:25:41 +01:00
Pablo Hoffman 79a8bb6b42 Merge pull request #189 from coagulant/patch-1 
Fixed docs typo in SpiderOpenCloseLogging example
 2012-11-10 06:37:54 -08:00
Ilya Baryshev 097aea04a4 Fixed docs typo in SpiderOpenCloseLogging example 2012-11-10 12:24:53 +04:00
Daniel Graña da7e414fe9 Add 0.16.2 release notes 
Conflicts:

	docs/news.rst
 2012-11-09 13:03:04 -02:00
Pablo Hoffman db21bccf9a added 0.18 to release notes and mention spider contracts 2012-11-07 16:02:18 -02:00
Pablo Hoffman c01f81ab44 Merge pull request #177 from alexcepoi/contracts_fixes 
Improve feedback on spider contracts run
 2012-11-07 09:40:21 -08:00
notsobad a438be39f1 Add xpath suppot for FormRequest.from_response. 2012-11-07 02:08:38 +08:00
Pablo Hoffman aa0e02dc54 added open_in_browser to debugging doc 2012-11-04 19:58:06 -02:00
Pablo Hoffman 7a7c5d1334 removed reference to global scrapy stats from settings doc 2012-11-03 17:05:01 -02:00
Pablo Hoffman 626662f0f3 Fix SpiderState bug in Windows platforms 
The spider state file was not opened in binary mode.
 2012-11-03 16:57:20 -02:00