RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,322 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3322 Commits

Author SHA1 Message Date
Pablo Hoffman d5d944fa44 Log overriden Scrapy settings when Scrapy starts. 
This is useful for debugging, to quickly find out which settings where
used for a specific spider run. dict settings are omitted for brevity.
DEBUG level is used for consistency with "Enabled
extensions/middlewares" lines (which share a similar purpose).
 2013-02-28 11:35:11 -02:00
Pablo Hoffman 8f3a509d44 remove debugging code 2013-02-27 03:52:55 -02:00
Pablo Hoffman 3a6b80259d arg_to_iter: replaced double call to isinstance with a single one. refs #248 2013-02-27 03:51:23 -02:00
Pablo Hoffman 2bbd92742b arg_to_iter: treat items the same way as dicts (ie. non iterables). fixes #248 2013-02-27 02:39:31 -02:00
Pablo Hoffman 7400ceb1ed added 502 to RETRY_HTTP_CODES 2013-02-22 19:12:59 -02:00
Pablo Hoffman e3f50b97b2 made scrapy/item.py pep8 compliant 2013-02-20 11:38:50 -02:00
Pablo Hoffman a038f46859 doc: fixed rst title 2013-02-14 11:11:17 -02:00
Pablo Hoffman 22edc44c6c doc: remove links to diveintopython.org, which is no longer available. closes #246 2013-02-14 11:09:40 -02:00
Pablo Hoffman aeb7fbe221 Merge pull request #240 from llonchj/spider_log 
spider.log to pass keyword arguments into twisted log.msg
 2013-02-12 18:13:26 -08:00
Pablo Hoffman 07669947c7 Merge pull request #244 from morty/master 
scrapy genspider allows you to create a spider module with the same name as the project
 2013-02-12 18:11:49 -08:00
Tom Mortimer-Jones 2efd859525 Added check so that genspider cannot create a spider with the same name as the project. 2013-02-12 12:26:38 +00:00
Pablo Hoffman 1ff8b4f831 updated release notes with previous commit 2013-02-12 00:59:25 -02:00
Rolando Espinoza La fuente 5971333a93 added --pdb command option to enable pdb debugger on failure. 2013-02-11 12:34:29 -04:00
Pablo Hoffman d9043ffa6e reverted previous changed as it broke tests 2013-02-11 11:43:25 -02:00
Pablo Hoffman 2df010a972 Merge pull request #241 from zuhao/master 
Fix url_has_any_extension bug
 2013-02-11 05:29:14 -08:00
Zuhao Wan 27ca25472f Fix url_has_any_extension bug 2013-02-11 17:19:31 +08:00
Jordi Llonch 6d4b764f29 spider.log to pass keyword arguments into twisted log.msg 2013-02-11 05:00:06 +11:00
Daniel Graña c0a7040f16 add PHONYs so build does not match build/ path 2013-02-08 15:07:00 -02:00
Daniel Graña 910effd145 get scrapy version from package data 2013-02-06 11:44:26 -02:00
Daniel Graña 79cb0318d0 update travis-ci conf 2013-02-06 10:36:08 -02:00
Daniel Graña 5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
Daniel Graña 33ca295129 Merge pull request #235 from whodatninja/patch-1 
Fix typo labeling attrs type bool instead of list
 2013-02-05 17:27:10 -08:00
whodatninja 8e3b5baac5 Fix typo labeling attrs type bool instead of list 2013-02-05 15:10:41 -05:00
Daniel Graña f293d08e7a Merge pull request #234 from darkrho/httpcache-dbm-module-fullpath 
allow to use full path in HTTPCACHE_DBM_MODULE setting
 2013-02-01 09:01:27 -08:00
Rolando Espinoza La fuente d00e6cf1d7 tests: dummydbm.error should be KeyError. 2013-02-01 12:52:59 -04:00
Rolando Espinoza La fuente 6ed44a3a67 httpcache: added tests for custom dbm module. 
And removed the break in the line to improve readability as there
is no strict 80-chars line width convention.
 2013-02-01 12:41:52 -04:00
Rolando Espinoza La fuente a2602df90b httpcache: allow to import submodules within packages as HTTPCACHE_DBM_MODULE setting. 2013-01-31 23:49:36 -04:00
Daniel Graña e5edb8ec3e Merge branch 'dangra/issue-24' 2013-01-30 16:55:53 -02:00
Daniel Graña 3af240f5f9 Merge branch 'dangra/issue-12' 2013-01-30 16:55:16 -02:00
Daniel Graña 0d3e4b4c43 do not unquote slash and question mark in url paths. fix #24 2013-01-30 11:22:03 -02:00
Daniel Graña 872a22df68 pep8ize sgml link extractors 2013-01-29 18:08:32 -02:00
Daniel Graña cc69b3aa4c Fix #199 encoding error concatenating link text 2013-01-29 17:56:34 -02:00
Daniel Graña 95fde0a498 register namespaces when XMLFeedSpider uses iternodes mode. fixes #12 2013-01-29 12:00:13 -02:00
Daniel Graña 8e77f27897 Find form nodes in invalid html5 documents 
lxml fails to parse invalid html5 documents
This error was reported in scrapy/loginform#3
 2013-01-24 17:47:53 -02:00
Daniel Graña ff04480675 fix exporting nested items as xml. fixes #66 2013-01-24 14:57:53 -02:00
Pablo Hoffman 71d7df9273 simplify InitSpider implementation, fixing one bug (closes #228) and adding support generators return from start_requests() (which the previous version didn't) 2013-01-23 16:28:41 -02:00
Daniel Graña 3cf7f4975b Add 0.16.4 to release notes 
Conflicts:
	docs/news.rst
 2013-01-23 11:29:38 -02:00
Daniel Graña c40f947dc1 Merge pull request #229 from christilden/master 
fixes spelling errors in documentation
 2013-01-22 20:27:40 -08:00
Chris Tilden aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman 27583922a7 remove unused imports 2013-01-21 14:01:12 -02:00
Pablo Hoffman 65258e3621 Merge pull request #227 from tonal/correct-old-cache 
Correct init bag for load FilesystemCacheStorage from old location
 2013-01-21 04:57:19 -08:00
Alexandr N Zamaraev (aka tonal) 2c51266a40 Correct init bag for load old scrapy.contrib.httpcache.FilesystemCacheStorage 2013-01-21 13:54:36 +07:00
Pablo Hoffman 6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman 1ba04b1fc3 added remove_namespaces() method to XmlXPathSelector objects 2013-01-18 12:20:03 -02:00
Pablo Hoffman b7eeeff410 get rid of assertDictEqual (since it's python 2.7+ only) 2013-01-17 13:18:23 -02:00
Pablo Hoffman c31441a273 revert default HTTP cache policy to dummy (instead of RFC2616) 2013-01-17 13:08:29 -02:00
Daniel Graña 897195186a document new FormRequest parameter named `formxpath` that matches forms using xpath 2013-01-08 18:36:20 -02:00
Daniel Graña 7527ef97ba Merge pull request #185 from notsobad/master 
Added xpath support in FormRequest.from_response
 2013-01-08 12:34:06 -08:00
Daniel Graña 75563b3f00 Add list of supported and missing RFC2616 caching features 2013-01-08 18:16:44 -02:00
Daniel Graña 3cbc4d0b94 django is an optional_features, its imports must not fail 2013-01-08 17:56:46 -02:00