RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,288 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2527 Commits

Author SHA1 Message Date
Daniel Graña ff04480675 fix exporting nested items as xml. fixes #66 2013-01-24 14:57:53 -02:00
Pablo Hoffman 71d7df9273 simplify InitSpider implementation, fixing one bug (closes #228) and adding support generators return from start_requests() (which the previous version didn't) 2013-01-23 16:28:41 -02:00
Pablo Hoffman 27583922a7 remove unused imports 2013-01-21 14:01:12 -02:00
Alexandr N Zamaraev (aka tonal) 2c51266a40 Correct init bag for load old scrapy.contrib.httpcache.FilesystemCacheStorage 2013-01-21 13:54:36 +07:00
Pablo Hoffman 1ba04b1fc3 added remove_namespaces() method to XmlXPathSelector objects 2013-01-18 12:20:03 -02:00
Pablo Hoffman b7eeeff410 get rid of assertDictEqual (since it's python 2.7+ only) 2013-01-17 13:18:23 -02:00
Pablo Hoffman c31441a273 revert default HTTP cache policy to dummy (instead of RFC2616) 2013-01-17 13:08:29 -02:00
Daniel Graña 7527ef97ba Merge pull request #185 from notsobad/master 
Added xpath support in FormRequest.from_response
 2013-01-08 12:34:06 -08:00
Daniel Graña 3cbc4d0b94 django is an optional_features, its imports must not fail 2013-01-08 17:56:46 -02:00
Daniel Graña d8a760bf57 Merge branch 'http-cache-middleware' 
Conflicts:
	scrapy/contrib/downloadermiddleware/httpcache.py
	scrapy/contrib/httpcache.py
	scrapy/tests/test_downloadermiddleware_httpcache.py
 2013-01-08 17:34:48 -02:00
Daniel Graña 864a7aef87 More httpcache updates 
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
 2013-01-08 17:26:32 -02:00
Daniel Graña 487299e068 TakeFirst doc says it returns first non-null/non-empty value, zero is a valid value. closes #59 2013-01-08 15:47:33 -02:00
Daniel Graña 9527c5819a pep8ize settings 2013-01-08 11:48:36 -02:00
Daniel Graña defc4f89b5 update metarefresh settings 2013-01-08 11:41:19 -02:00
Daniel Graña 71db7f1b25 Split redirection into status and metarefresh middlewares, also changes httpcompression priority. closes #78 2013-01-08 09:59:38 -02:00
Rolando Espinoza La fuente fe5d0ce2e0 tests: added downloader middleware manager integration tests for gzipped redirection. 2013-01-08 09:59:38 -02:00
Pablo Hoffman acb7bad1ff Merge pull request #218 from Mimino666/django-item-validation 
Django item validation
 2013-01-04 10:35:20 -08:00
Daniel Graña 3f03a2ca50 requests with no-cache set must force revalidation of cached responses 2013-01-04 04:20:45 -02:00
Daniel Graña cdecc760ee default httpcache to rfc2616 policy and improve storage and policy tests 2013-01-04 03:43:25 -02:00
Daniel Graña 0a5586fafd move FilesystemCacheStorage to scrapy.contrib.httpcache 2013-01-03 09:23:18 -02:00
Michal Danilak 2cfbd13c17 Added "exclude" parameter testing to unittests. 2013-01-03 02:02:03 +01:00
Michal Danilak 035d1e99a9 Added model validation to DjangoItem. 2013-01-03 01:52:53 +01:00
Michal Danilak 8ea89b277c Fixed error message formatting. 
log.err() doesn't support cool formatting and when error occured, the message was:
	"ERROR: Error processing %(item)s"
 2013-01-02 22:55:04 +01:00
Pablo Hoffman ea0967562e Merge pull request #216 from Mimino666/shell-spider-option 
Added --spider option to "shell" command.
 2013-01-02 09:15:15 -08:00
Daniel Graña a6ef76ed88 lint and improve images pipeline error logging 2013-01-02 14:14:37 -02:00
Michal Danilak ea68250d77 Added --spider option to "shell" command. 2013-01-02 17:13:37 +01:00
Alexandr N Zamaraev (aka tonal) 71b071ffeb Log level return from LogFormatter methods 2012-12-29 11:43:41 +07:00
Pedro Faustino cf5f0203b7 Instead of extending from HttpCachePolicy, following the same approach used for storage selection 2012-12-28 16:11:47 +01:00
Pedro Faustino 3e31d06872 Implement single HTTP cache policy 2012-12-28 13:28:35 +01:00
Pedro Faustino 63d0b9f8c8 Remove plural from stat key. 2012-12-28 11:06:04 +01:00
Alexandr N Zamaraev (aka tonal) 3d397de0e9 Possible set log-level in LogFormatter.dropped and LogFormatter.scraped 2012-12-28 13:40:27 +07:00
Hasnain Lakhani 93a1102189 Implemented policies for HTTP Cache 2012-12-26 16:29:48 -08:00
Pedro Faustino b2d3f4dd1b Add cache storage support for real HTTP caching. Add real HTTP caching unit tests for both middleware and cache storage. 2012-12-24 16:15:04 +01:00
Pedro Faustino 0e435fb5f9 Add middleware support for the 'no-store' Cache-Control directive. 2012-12-24 13:27:13 +01:00
Pedro Faustino bb55f39aed Add middleware support for real HTTP caching. Add httpcache setting to allow either real or dummy HTTP caching (for backwards compatibility it's set to use dummy cache by default). 2012-12-24 01:52:45 +01:00
Pedro Faustino e396509f5b Removing plural from httpcache stats' value names. 2012-12-24 00:24:27 +01:00
Pablo Hoffman dabb064293 fix bug in scrapy parse command when spider is not specified explicitly. closes #209 2012-12-20 11:59:07 -02:00
Daniel Graña 22559833e7 core: drop download inactive slots and get slot key from meta 2012-12-17 16:09:01 -02:00
Daniel Graña eb2d87259f core: move download slot assignment post middleware evaluation 2012-12-17 16:09:01 -02:00
Daniel Graña d7daf836d5 Altering delay is enough to auto throttle downloads 2012-12-17 16:08:49 -02:00
Daniel Graña 85c9881724 fix pylint and pep8 warnings 2012-12-14 12:27:53 -02:00
Daniel Graña fd58de493c Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced 2012-12-05 15:08:34 -02:00
Libor Nenadál 5f899aa4ec add error details when image pipeline fails 2012-12-02 23:29:26 +01:00
Alex Cepoi fc405e98aa improve mac os compatibility 
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
  of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
 2012-12-01 16:39:58 +01:00
Pablo Hoffman 0a00e0fd63 Merge pull request #195 from kalessin/floatdelay 
download delay in autothrottle was being casted as int, should be float
 2012-11-15 11:18:45 -08:00
Martin Olveyra 421dba98a7 download delay in autothrottle was being casted as int, should be float 2012-11-15 18:41:32 +00:00
Pablo Hoffman 3a60d42072 removed unused import 2012-11-15 16:29:07 -02:00
Pablo Hoffman 5ebaa40ed2 changed SSL version to use from SSLv23 to TLSv1. closes #194 but needs more testing against counter-effects 2012-11-15 16:28:39 -02:00
vkrest ac4c66f351 Fix broken doc links. 
All generated template files contain broken link to doc files.
 2012-11-12 22:43:42 +02:00
Pedro Faustino 3d0e962cbb The Link object has a 'nofollow' attribute. Adding support in the SGML link extractors. 2012-11-10 16:25:41 +01:00