RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,591 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

3591 Commits

Author SHA1 Message Date
Pablo Hoffman 780ef128d2 Merge pull request #399 from irgmedeiros/patch-1 
Update practices.rst
 2013-09-27 14:03:51 -07:00
irgmedeiros d9e0fdc9aa Update practices.rst 
With this modification scrapy runs the spider with project settings. The previous example ran only with default settings resulting in ignoring all user settings as pipelines for example.
 2013-09-27 17:56:30 -03:00
Daniel Graña 265910aae6 Merge pull request #363 from taikano/sitemap_alternate 
also fetch alternate URLs from sitemaps, see #360
 2013-09-26 09:15:02 -07:00
Pablo Hoffman 12280c2a95 fix sphinx references in doc 2013-09-25 15:13:17 -03:00
Mikhail Korobov b74e5aa551 fix ITEM_PIPELINES setting handling 
* item_pipelines variable was unused so the fallback didn't work;
* added support for fallback in case of ITEM_PIPELINIES defined as set.
 2013-09-25 20:37:06 +06:00
Daniel Graña 61e89d8247 Merge pull request #393 from scrapy/itempipe-dict 
Make ITEM_PIPELINE setting a dict
 2013-09-23 14:12:26 -07:00
Pablo Hoffman fc388f4636 Make ITEM_PIPELINE setting a dict 
This is for consistency with how spider and downloader middlewares are
defined. ITEM_PIPELINE_BASE was also added and both remain empty.

Backwards compatibility is kept (with a warning) with list-based
ITEM_PIPELINES.
 2013-09-23 17:50:43 -03:00
Pablo Hoffman b1d1a36a1e add note about enclosing urls with quotes when running from command-line. closes GH-384 2013-09-18 18:01:28 -03:00
Mikhail Korobov bb8cb41777 Merge pull request #391 from amferraz/patch-2 
Update request-response.rst
 2013-09-18 07:59:46 -07:00
cacovsky 71b320914a Update request-response.rst 
Fix small doc typo (too many backticks)
 2013-09-18 11:45:25 -03:00
Pablo Hoffman ee8464b471 Merge pull request #390 from kmike/mock-in-tox 
mock is required for running tests
 2013-09-17 15:51:54 -07:00
Mikhail Korobov 7ffc786fb6 mock is required for running tests since 45ff6ec28a 
This fixes tox test running. Mock is available at Travis, but there is no harm in listing it explicitly.
 2013-09-18 00:02:23 +06:00
Pablo Hoffman 2a89265b96 Merge pull request #387 from ktharmal/patch-1 
Fixed directory location for dmoz_spider.py file
 2013-09-16 11:36:13 -07:00
Kumara Tharmalingam bbb0603091 Fixed directory location for dmoz_spider.py file 
It should be under 'tutorial/spiders' not 'dmoz/spiders'
 2013-09-15 21:55:52 -07:00
Stefan 855041ebfd added test case for comments, see #363 2013-09-14 12:17:34 +02:00
Daniel Graña 0400b18efa docs: list lxml as installation prerequisite 2013-09-09 12:44:26 -03:00
Stefan 6994959181 renamed to sitemap_alternate_links and added default value, see #360 2013-09-08 10:38:28 +02:00
Daniel Graña b326b87154 forms: do not submit reset inputs 2013-09-07 21:53:27 -03:00
Stefan 8ed2d0cda1 improved changes to allow retrieval of alternate links in sitemaps, see #360 2013-09-07 12:56:30 +02:00
Daniel Graña 65135db70c increase unittest timeouts to decrease travis false positive failures 2013-09-05 21:53:50 -03:00
Daniel Graña 801564c2c9 exporters linting 2013-09-05 17:00:16 -03:00
Daniel Graña 2be3f7132a Do not fail on when json exporter are instanciated with extra keywords. closes #379 2013-09-05 16:57:27 -03:00
Daniel Graña 92d14d4a03 Fix permission and set umask before generating sdist tarball 2013-09-03 23:22:17 -03:00
Daniel Graña e6b3ca0180 Add 0.18.2 release notes 2013-09-03 14:29:40 -03:00
Daniel Graña 9b821df482 Merge pull request #330 from alexcepoi/contracts_fix 2013-09-03 11:41:04 -03:00
Daniel Graña f54d5c5896 Merge pull request #370 from loucash/master 
FilesPipeline extracted and separated from ImagesPipeline
 2013-09-03 07:29:30 -07:00
Alex Cepoi 50b653e5ae don't start reactor if no crawler is scheduled 2013-09-02 18:43:25 +02:00
Lukasz Biedrycki 0a8bf2c9e4 after additional tests with actual s3: k.set_contents_from_string is working 2013-08-28 18:00:54 +02:00
Daniel Graña 0f00b1602a merge 0.18 release notes 2013-08-27 18:47:46 -03:00
Daniel Graña ac218ed418 no need to test latest versions of dependencies on python 2.6 2013-08-27 17:38:28 -03:00
Daniel Graña c5222b4b8a remove phanton build row from travis matrix due to travis-ci/travis-core#1027 2013-08-27 17:19:45 -03:00
Daniel Graña ad140a2763 another try to limit travis build matrix 2013-08-27 16:53:54 -03:00
Daniel Graña 7394969236 test pypy with travis 2013-08-27 15:31:08 -03:00
Daniel Graña 401e888ced test lucid and precise python packages under their respective python versions 2013-08-27 15:30:10 -03:00
Daniel Graña 588a262b73 fix crawling tests under twisted pre 11.0.0 2013-08-27 14:05:37 -03:00
Daniel Graña 92826586d5 py26 can not format zero length fields {} 2013-08-27 12:12:50 -03:00
Daniel Graña caa0f90263 test PotentiaDataLoss errors on unbound responses 2013-08-27 11:53:18 -03:00
Daniel Graña 6720acc1af Treat responses without content-length or Transfer-Encoding as good responses 
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
 2013-08-27 11:10:20 -03:00
Pablo Hoffman da1f6d3144 remove unused imports and some assorted pylint-ing 2013-08-23 13:03:34 -03:00
Pablo Hoffman 19ff9ac4f9 url/body attributes of Request/Response objects are now immutable 2013-08-23 12:43:22 -03:00
Pablo Hoffman 86230c0ab8 added quantal & raring to support ubuntu releases 2013-08-22 21:49:55 -03:00
Lukasz Biedrycki 83bd151c57 backwards compatibility: image_key & image_downloaded 2013-08-22 15:13:59 +02:00
Lukasz Biedrycki 2b11e7da93 backwards compatibility: image_key 2013-08-22 14:44:04 +02:00
Lukasz Biedrycki 02d08722e7 typo 2013-08-22 14:33:28 +02:00
Lukasz Biedrycki 4c89d8b298 remarks 2013-08-22 12:26:56 +02:00
Daniel Graña efb4f32d80 do no include ResponseFailed if http11 handler is not enabled 2013-08-21 19:04:12 -03:00
Daniel Graña 3c64a989ca New HTTP client wraps connection losts in ResponseFailed exception. fix #373 2013-08-21 17:28:26 -03:00
Pablo Hoffman 071172cbd6 fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 2013-08-21 14:32:33 -03:00
Pablo Hoffman 6e4ed8b004 Merge pull request #372 from kmike/xml-exporter-fix-for-2.7.4 
fix XmlItemExporter in Python 2.7.4 and 2.7.5
 2013-08-21 09:55:51 -07:00
Mikhail Korobov a6e6ca06b9 fix XmlItemExporter in Python 2.7.4 and 2.7.5 2013-08-21 06:05:40 +06:00