RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,289 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3289 Commits

Author SHA1 Message Date
Daniel Graña 8e77f27897 Find form nodes in invalid html5 documents 
lxml fails to parse invalid html5 documents
This error was reported in scrapy/loginform#3
 2013-01-24 17:47:53 -02:00
Daniel Graña ff04480675 fix exporting nested items as xml. fixes #66 2013-01-24 14:57:53 -02:00
Pablo Hoffman 71d7df9273 simplify InitSpider implementation, fixing one bug (closes #228) and adding support generators return from start_requests() (which the previous version didn't) 2013-01-23 16:28:41 -02:00
Daniel Graña 3cf7f4975b Add 0.16.4 to release notes 
Conflicts:
	docs/news.rst
 2013-01-23 11:29:38 -02:00
Daniel Graña c40f947dc1 Merge pull request #229 from christilden/master 
fixes spelling errors in documentation
 2013-01-22 20:27:40 -08:00
Chris Tilden aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman 27583922a7 remove unused imports 2013-01-21 14:01:12 -02:00
Pablo Hoffman 65258e3621 Merge pull request #227 from tonal/correct-old-cache 
Correct init bag for load FilesystemCacheStorage from old location
 2013-01-21 04:57:19 -08:00
Alexandr N Zamaraev (aka tonal) 2c51266a40 Correct init bag for load old scrapy.contrib.httpcache.FilesystemCacheStorage 2013-01-21 13:54:36 +07:00
Pablo Hoffman 6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman 1ba04b1fc3 added remove_namespaces() method to XmlXPathSelector objects 2013-01-18 12:20:03 -02:00
Pablo Hoffman b7eeeff410 get rid of assertDictEqual (since it's python 2.7+ only) 2013-01-17 13:18:23 -02:00
Pablo Hoffman c31441a273 revert default HTTP cache policy to dummy (instead of RFC2616) 2013-01-17 13:08:29 -02:00
Daniel Graña 897195186a document new FormRequest parameter named `formxpath` that matches forms using xpath 2013-01-08 18:36:20 -02:00
Daniel Graña 7527ef97ba Merge pull request #185 from notsobad/master 
Added xpath support in FormRequest.from_response
 2013-01-08 12:34:06 -08:00
Daniel Graña 75563b3f00 Add list of supported and missing RFC2616 caching features 2013-01-08 18:16:44 -02:00
Daniel Graña 3cbc4d0b94 django is an optional_features, its imports must not fail 2013-01-08 17:56:46 -02:00
Daniel Graña d8a760bf57 Merge branch 'http-cache-middleware' 
Conflicts:
	scrapy/contrib/downloadermiddleware/httpcache.py
	scrapy/contrib/httpcache.py
	scrapy/tests/test_downloadermiddleware_httpcache.py
 2013-01-08 17:34:48 -02:00
Daniel Graña 864a7aef87 More httpcache updates 
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
 2013-01-08 17:26:32 -02:00
Daniel Graña 487299e068 TakeFirst doc says it returns first non-null/non-empty value, zero is a valid value. closes #59 2013-01-08 15:47:33 -02:00
Daniel Graña 672d09ea2e add meta-refresh changes to release notes 2013-01-08 12:30:36 -02:00
Daniel Graña 9527c5819a pep8ize settings 2013-01-08 11:48:36 -02:00
Daniel Graña defc4f89b5 update metarefresh settings 2013-01-08 11:41:19 -02:00
Daniel Graña 6a2b23883a Add MetaRefreshMiddleware docs 2013-01-08 11:25:38 -02:00
Daniel Graña 076ba40404 update DOWNLOADER_MIDDLEWARES_BASE setting documentation 2013-01-08 10:50:27 -02:00
Daniel Graña 71db7f1b25 Split redirection into status and metarefresh middlewares, also changes httpcompression priority. closes #78 2013-01-08 09:59:38 -02:00
Rolando Espinoza La fuente fe5d0ce2e0 tests: added downloader middleware manager integration tests for gzipped redirection. 2013-01-08 09:59:38 -02:00
Pablo Hoffman 227a1d666b add doc about disabling an extension. refs #132 2013-01-07 13:16:19 -02:00
Pedro Faustino 5d3a4d755f Update downloader middleware documentation 2013-01-06 18:53:14 +00:00
Pedro Faustino 59dc71f394 Merge branch 'http-cache-middleware', remote-tracking branch 'dangra/http-cache-middleware' into http-cache-middleware 2013-01-06 17:53:25 +00:00
Pablo Hoffman 7f990a4af2 Merge pull request #221 from emschorsch/patch-1 
Proposed Changes to DjangoItem documentation
 2013-01-04 14:17:54 -08:00
Emanuel Schorsch f9b130da12 Proposed Changes 
I was very confused as to how you actually import DjangoItem.
I searched extensively on the internet looking for actual code so I could see how it worked.
I finally found http://blog.just2us.com/2012/07/setting-up-django-with-scrapy/. It is much easier to understand with full files instead of code fragments.
I also edited where it says "we can see that the model is already saved" as I don't see how it's already saved.
 2013-01-04 15:59:04 -05:00
Pablo Hoffman acb7bad1ff Merge pull request #218 from Mimino666/django-item-validation 
Django item validation
 2013-01-04 10:35:20 -08:00
Daniel Graña 3f03a2ca50 requests with no-cache set must force revalidation of cached responses 2013-01-04 04:20:45 -02:00
Daniel Graña cdecc760ee default httpcache to rfc2616 policy and improve storage and policy tests 2013-01-04 03:43:25 -02:00
Daniel Graña 0a5586fafd move FilesystemCacheStorage to scrapy.contrib.httpcache 2013-01-03 09:23:18 -02:00
Pablo Hoffman 9f003a73da Merge pull request #217 from Mimino666/item-processing-errormsg 
Fixed error message formatting.
 2013-01-02 21:04:40 -08:00
Michal Danilak 2cfbd13c17 Added "exclude" parameter testing to unittests. 2013-01-03 02:02:03 +01:00
Michal Danilak 035d1e99a9 Added model validation to DjangoItem. 2013-01-03 01:52:53 +01:00
Michal Danilak 8ea89b277c Fixed error message formatting. 
log.err() doesn't support cool formatting and when error occured, the message was:
	"ERROR: Error processing %(item)s"
 2013-01-02 22:55:04 +01:00
Pablo Hoffman ea0967562e Merge pull request #216 from Mimino666/shell-spider-option 
Added --spider option to "shell" command.
 2013-01-02 09:15:15 -08:00
Daniel Graña a6ef76ed88 lint and improve images pipeline error logging 2013-01-02 14:14:37 -02:00
Michal Danilak ea68250d77 Added --spider option to "shell" command. 2013-01-02 17:13:37 +01:00
Pablo Hoffman e9c5b76242 Merge pull request #215 from kuyan/patch-1 
Fixed typo
 2013-01-02 06:21:32 -08:00
Natan L d572f8945e Fixed typo 
'persitent' --> 'persistent'
 2012-12-31 11:14:01 -08:00
Pablo Hoffman 1aa25cdff2 Merge pull request #214 from tonal/log-level-dropped-item 
Make LogFormatter return the log level (and require it)
 2012-12-29 19:21:54 -08:00
Alexandr N Zamaraev (aka tonal) 71b071ffeb Log level return from LogFormatter methods 2012-12-29 11:43:41 +07:00
Pedro Faustino cf5f0203b7 Instead of extending from HttpCachePolicy, following the same approach used for storage selection 2012-12-28 16:11:47 +01:00
Pablo Hoffman ba257db6e2 Merge pull request #213 from tonal/forget-import 
Remove firget imports
 2012-12-28 06:42:40 -08:00
Pedro Faustino 492831fc6f Merge branch 'master' of git://github.com/scrapy/scrapy into http-cache-middleware 2012-12-28 15:27:45 +01:00