RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,212 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

3212 Commits

Author SHA1 Message Date
Daniel Graña 56b45fcf0f lint and improve images pipeline error logging 2013-01-02 14:16:17 -02:00
Pablo Hoffman 243be84eb8 fixed doc typos 2012-12-26 16:17:06 -02:00
Pablo Hoffman 1fbb715228 add documentation topics: Broad Crawls & Common Practies 2012-12-26 14:02:31 -02:00
Pablo Hoffman c72e6823fe fix bug in scrapy parse command when spider is not specified explicitly. closes #209 2012-12-20 11:59:38 -02:00
Luan 28eac7a773 Update docs/topics/commands.rst 
A short change.
 2012-12-10 22:20:03 -02:00
Daniel Graña 4ae7385743 Add 0.16.3 release notes 2012-12-07 18:52:42 -02:00
Daniel Graña 44b5c5d861 bumped version to 0.16.3 2012-12-07 18:51:24 -02:00
Daniel Graña 487b9b5147 Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced 2012-12-07 18:50:16 -02:00
Libor Nenadál 823256998f add error details when image pipeline fails 2012-12-03 16:03:18 -02:00
Alex Cepoi 8dcf8aae37 improve mac os compatibility 
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
  of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
 2012-12-02 13:39:05 -02:00
Pablo Hoffman 7b5310d8a8 setup.py: use README.rst to populate long_description 2012-11-25 22:22:24 -02:00
Pablo Hoffman 80f9bb6b80 doc: removed obsolete references to ClientForm 2012-11-23 19:06:57 -02:00
stav 2aa491bc31 correct docs for default storage backend 2012-11-22 19:03:09 -02:00
Pablo Hoffman bdf61c403d doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:39 -02:00
Pablo Hoffman d5087b0afa Merge branch '0.16' of github.com:scrapy/scrapy into 0.16 2012-11-10 12:38:59 -02:00
Ilya Baryshev 7184094658 Fixed docs typo in SpiderOpenCloseLogging example 2012-11-10 12:38:47 -02:00
Daniel Graña 1787011d4b update __version__ to 0.16.2 2012-11-09 13:12:47 -02:00
Daniel Graña 629f6ca36f Add 0.16.2 release notes 2012-11-09 13:01:20 -02:00
Daniel Graña 44b4a2aa17 bumped version to 0.16.2 2012-11-09 12:58:57 -02:00
Alex Cepoi a4a9199eb8 scrapy contracts: python2.6 compat 2012-11-09 11:27:07 -02:00
Alex Cepoi ec41673720 scrapy contracts verbose option 2012-11-09 11:27:02 -02:00
Alex Cepoi 86635e4afe proper unittest-like output for scrapy contracts 2012-11-09 11:27:00 -02:00
Pablo Hoffman c9b690deb2 added open_in_browser to debugging doc 2012-11-04 19:58:15 -02:00
Pablo Hoffman dd55067acc removed reference to global scrapy stats from settings doc 2012-11-03 17:05:12 -02:00
Pablo Hoffman 58998f4bc6 Fix SpiderState bug in Windows platforms 
The spider state file was not opened in binary mode.
 2012-11-03 16:58:31 -02:00
Daniel Graña 7284db1809 update news file with 0.16.1 release notes 2012-10-26 18:54:58 -02:00
Daniel Graña 9c9ac99a45 bumped version to 0.16.1 2012-10-26 18:51:10 -02:00
Pablo Hoffman 8c780fd9b5 fixed LogStats extension, which got broken after a wrong merge before the 0.16 release 2012-10-26 18:45:09 -02:00
Pablo Hoffman 3403089929 better backwards compatibility for scrapy.conf.settings 
Althought this backwards compatibility is more complex, it avoid modules
from failing when importing scrapy.conf, if they are not run through
"scrapy" command (such as when running tests on scrapy projects code).
 2012-10-26 18:45:05 -02:00
Pablo Hoffman c4da0b5240 extended documentation on how to access crawler stats from extensions 2012-10-26 18:44:58 -02:00
Pablo Hoffman d52c188ab5 removed .hgtags (no longer needed now that scrapy uses git) 2012-10-26 18:44:43 -02:00
Daniel Graña fa4f7f9d1a fix dashes under rst headers 2012-10-18 20:14:20 -02:00
Daniel Graña e292246b83 set release date for 0.16.0 in news 2012-10-18 20:07:52 -02:00
Daniel Graña 1477ab47fc bumped version to 0.16.0 2012-10-18 17:05:42 -02:00
Pablo Hoffman c3e91ba73e updated scrapy_bash_completion 2012-10-12 20:02:03 -02:00
Pablo Hoffman e9cef3ae69 better description for scrapy check command 2012-10-12 20:01:43 -02:00
Pablo Hoffman a0e2b6e3ff simplified backwards compatibility of scrapy.conf 2012-10-12 17:28:02 -02:00
Pablo Hoffman 9d7106219a added deprecation warning when importing scrapy.conf module 2012-10-12 17:24:25 -02:00
Pablo Hoffman b038bb8504 minor message update: spider stats -> Scrapy stats 2012-10-12 16:53:44 -02:00
Artem Bogomyagkov 1d5967cee6 httpcompression middleware improvement 2012-10-10 20:20:13 +03:00
Pablo Hoffman 1a905d62f5 removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway) 2012-10-09 16:05:19 -02:00
Pablo Hoffman 1f89eb59fe fixed doc reference to topics-contracts 2012-10-09 16:02:12 -02:00
Pablo Hoffman 9e4c6aba2f updated release notes with new stats recently added 2012-10-09 12:50:42 -02:00
Pablo Hoffman bd3a109299 Track number of log messages via Scrapy stats 
Log messages count are grouped per LEVEL like:

    'log_count/DEBUG': 8,
    'log_count/ERROR': 1,
    'log_count/INFO': 4,

This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
 2012-10-09 12:41:31 -02:00
Pablo Hoffman 1ae1a3ea52 don't reset scrapy stats when spider is opened 2012-10-09 12:40:51 -02:00
Pablo Hoffman ceeea27d21 added response_received_count to scrapy stats 2012-10-09 12:39:30 -02:00
Pablo Hoffman 452595790e fix test-scrapyd.sh after changes made to website 2012-10-08 15:30:19 -02:00
Daniel Graña 3c3c0c1f3f notify #scrapy room of travis build results 2012-10-06 15:16:58 -03:00
Daniel Graña 05ce20dab3 more cases where response.meta is prefered 2012-10-06 15:16:58 -03:00
Pablo Hoffman 1e1cc76fb7 Merge pull request #179 from artem-dev/errback_fix 
This fixes the case where failure.request wasn't available when the failure was generated in a spider middleware (for example, the HttpError middleware).
 2012-10-06 08:47:26 -07:00