RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,212 Commits 49 Branches 132 Tags 153 MiB
Commit Graph

3212 Commits

Author SHA1 Message Date
Pablo Hoffman 9c9a18b3a3 Merge pull request #201 from alexcepoi/test-fixes-mac 
improve mac os compatibility
 2012-12-02 07:37:53 -08:00
Alex Cepoi fc405e98aa improve mac os compatibility 
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
  of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
 2012-12-01 16:39:58 +01:00
Pablo Hoffman b9a96147ed setup.py: use README.rst to populate long_description 2012-11-25 22:22:33 -02:00
Pablo Hoffman 39274a2457 doc: removed obsolete references to ClientForm 2012-11-23 19:06:47 -02:00
Pablo Hoffman 8ca2ee3d60 Merge pull request #196 from stav/master 
The default storage backend is now DbmCacheStorage
 2012-11-22 13:02:15 -08:00
stav 99f164fc87 correct docs for default storage backend 2012-11-22 14:05:47 -06:00
Pablo Hoffman 1f0d167037 doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:26 -02:00
Pablo Hoffman 0a00e0fd63 Merge pull request #195 from kalessin/floatdelay 
download delay in autothrottle was being casted as int, should be float
 2012-11-15 11:18:45 -08:00
Martin Olveyra 421dba98a7 download delay in autothrottle was being casted as int, should be float 2012-11-15 18:41:32 +00:00
Pablo Hoffman 3a60d42072 removed unused import 2012-11-15 16:29:07 -02:00
Pablo Hoffman 5ebaa40ed2 changed SSL version to use from SSLv23 to TLSv1. closes #194 but needs more testing against counter-effects 2012-11-15 16:28:39 -02:00
Pablo Hoffman cba3967667 Merge pull request #192 from vkrest/master 
Fix broken doc links.
 2012-11-12 13:37:52 -08:00
vkrest ac4c66f351 Fix broken doc links. 
All generated template files contain broken link to doc files.
 2012-11-12 22:43:42 +02:00
Pablo Hoffman 14179a16a1 Merge pull request #190 from pedrofaustino/master 
Adding support to 'nofollow' in the SGML link extractors
 2012-11-12 10:09:30 -08:00
Pedro Faustino 3d0e962cbb The Link object has a 'nofollow' attribute. Adding support in the SGML link extractors. 2012-11-10 16:25:41 +01:00
Pablo Hoffman 79a8bb6b42 Merge pull request #189 from coagulant/patch-1 
Fixed docs typo in SpiderOpenCloseLogging example
 2012-11-10 06:37:54 -08:00
Ilya Baryshev 097aea04a4 Fixed docs typo in SpiderOpenCloseLogging example 2012-11-10 12:24:53 +04:00
Daniel Graña da7e414fe9 Add 0.16.2 release notes 
Conflicts:

	docs/news.rst
 2012-11-09 13:03:04 -02:00
Pablo Hoffman db21bccf9a added 0.18 to release notes and mention spider contracts 2012-11-07 16:02:18 -02:00
Pablo Hoffman c01f81ab44 Merge pull request #177 from alexcepoi/contracts_fixes 
Improve feedback on spider contracts run
 2012-11-07 09:40:21 -08:00
Pablo Hoffman aa0e02dc54 added open_in_browser to debugging doc 2012-11-04 19:58:06 -02:00
Pablo Hoffman 7a7c5d1334 removed reference to global scrapy stats from settings doc 2012-11-03 17:05:01 -02:00
Pablo Hoffman 626662f0f3 Fix SpiderState bug in Windows platforms 
The spider state file was not opened in binary mode.
 2012-11-03 16:57:20 -02:00
Alex Cepoi 0350920f1d scrapy contracts: python2.6 compat 2012-10-28 18:23:21 +01:00
Alex Cepoi c6fad05756 scrapy contracts verbose option 2012-10-28 17:53:38 +01:00
Alex Cepoi 94629a1f08 proper unittest-like output for scrapy contracts 2012-10-27 23:26:40 +02:00
Daniel Graña c0542838d3 update news file with 0.16.1 release notes 2012-10-26 18:53:59 -02:00
Pablo Hoffman b01be8619d fixed LogStats extension, which got broken after a wrong merge before the 0.16 release 2012-10-25 15:24:53 -02:00
Pablo Hoffman e8e5a62c20 better backwards compatibility for scrapy.conf.settings 
Althought this backwards compatibility is more complex, it avoid modules
from failing when importing scrapy.conf, if they are not run through
"scrapy" command (such as when running tests on scrapy projects code).
 2012-10-25 11:41:40 -02:00
Pablo Hoffman 8f4c879b58 extended documentation on how to access crawler stats from extensions 2012-10-25 11:28:23 -02:00
Pablo Hoffman 17d958469f removed .hgtags (no longer needed now that scrapy uses git) 2012-10-24 17:59:00 -02:00
Daniel Graña a9616f387e fix dashes under rst headers 2012-10-18 20:14:30 -02:00
Daniel Graña 4852c6102d set release date for 0.16.0 in news 2012-10-18 20:07:36 -02:00
Daniel Graña d4b8e2633e bumped version to 0.17.0 2012-10-18 17:11:33 -02:00
Pablo Hoffman c3e91ba73e updated scrapy_bash_completion 2012-10-12 20:02:03 -02:00
Pablo Hoffman e9cef3ae69 better description for scrapy check command 2012-10-12 20:01:43 -02:00
Pablo Hoffman a0e2b6e3ff simplified backwards compatibility of scrapy.conf 2012-10-12 17:28:02 -02:00
Pablo Hoffman 9d7106219a added deprecation warning when importing scrapy.conf module 2012-10-12 17:24:25 -02:00
Pablo Hoffman b038bb8504 minor message update: spider stats -> Scrapy stats 2012-10-12 16:53:44 -02:00
Artem Bogomyagkov 1d5967cee6 httpcompression middleware improvement 2012-10-10 20:20:13 +03:00
Pablo Hoffman 1a905d62f5 removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway) 2012-10-09 16:05:19 -02:00
Pablo Hoffman 1f89eb59fe fixed doc reference to topics-contracts 2012-10-09 16:02:12 -02:00
Pablo Hoffman 9e4c6aba2f updated release notes with new stats recently added 2012-10-09 12:50:42 -02:00
Pablo Hoffman bd3a109299 Track number of log messages via Scrapy stats 
Log messages count are grouped per LEVEL like:

    'log_count/DEBUG': 8,
    'log_count/ERROR': 1,
    'log_count/INFO': 4,

This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
 2012-10-09 12:41:31 -02:00
Pablo Hoffman 1ae1a3ea52 don't reset scrapy stats when spider is opened 2012-10-09 12:40:51 -02:00
Pablo Hoffman ceeea27d21 added response_received_count to scrapy stats 2012-10-09 12:39:30 -02:00
Pablo Hoffman 452595790e fix test-scrapyd.sh after changes made to website 2012-10-08 15:30:19 -02:00
Daniel Graña 3c3c0c1f3f notify #scrapy room of travis build results 2012-10-06 15:16:58 -03:00
Daniel Graña 05ce20dab3 more cases where response.meta is prefered 2012-10-06 15:16:58 -03:00
Pablo Hoffman 1e1cc76fb7 Merge pull request #179 from artem-dev/errback_fix 
This fixes the case where failure.request wasn't available when the failure was generated in a spider middleware (for example, the HttpError middleware).
 2012-10-06 08:47:26 -07:00