RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,550 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2550 Commits

Author SHA1 Message Date
Daniel Graña fac5e5ea21 migrate hgignore to gitignore 2011-09-18 01:05:31 -03:00
Pablo Hoffman 2fcb7097bd removed documentation header notifying about other documentation versions, as that's provided by readthedocs already 2011-09-14 02:41:01 -03:00
Pablo Hoffman 3b00b9cb12 added support for generating version from git revision 2011-09-11 11:24:12 -03:00
Pablo Hoffman bc2d2183e9 fixed import in doc 2011-08-11 09:11:08 -03:00
Pablo Hoffman 09af0866c7 scrapy.utils.python: fixed bug introduced when adding support for new IPython 0.11. refs #335 2011-08-09 01:41:48 -03:00
Pablo Hoffman 3191a08560 fix start_python_console() to work with IPython >= 0.11. closes #335 2011-08-02 22:45:51 -03:00
Pablo Hoffman 18cb4ff1d8 added natty to list of supporte ubuntu distros 2011-07-13 18:43:52 -03:00
Pablo Hoffman 4a83167698 fixed small doc typo 2011-04-30 01:35:30 -03:00
Pablo Hoffman 678f08bc1b added warning about using 'parse' as callback in crawl spider rules 2011-04-25 09:30:42 -03:00
Pablo Hoffman ecb4f44cbc Added clarification on how to work with local settings and scrapy deploy 2011-04-14 12:36:09 -03:00
Pablo Hoffman 36431a1439 Silenced confusing sqlite3.ProgrammingError exception. For more info see: http://twistedmatrix.com/trac/ticket/4040 2011-03-09 12:39:24 -02:00
Pablo Hoffman 8f7e163b04 Fixed wrong method name in downloader middleware documentation 2011-02-24 15:26:32 -02:00
Shane Evans 74413ff989 prevent incorrect assertion error possible when spiders are closing 2011-02-16 18:30:50 -02:00
Pablo Hoffman 16d9a33951 added FAQ entry about working with big data feeds 2011-02-15 07:24:52 -02:00
Pablo Hoffman 3dc677725e Fixed scrapy.utils.python unittests 2011-02-10 17:27:40 -02:00
Pablo Hoffman ce8137b738 Replace unknown characters in sgml link extractor, to deal more gracefully with encoding errors in the page. Closes #309 2011-02-10 17:12:03 -02:00
Pablo Hoffman 181d1c09ae Fixed typo and code indentation in the doc. Closes #307 and #308 2011-02-09 11:19:46 -02:00
Pablo Hoffman c5499ead73 Clarified behaviour when multiple rules match the same link in CrawlSpider 2011-02-04 13:39:12 -02:00
Pablo Hoffman 65fc2fbd1f Set CONCURRENT_SPIDERS=1 in Scrapyd to force one spider per process 2011-02-04 13:30:01 -02:00
Pablo Hoffman b1c89508f5 fixed wrong changes commited in previous changeset 2011-01-28 16:22:39 -02:00
Pablo Hoffman 4361150494 added missing scrapyd/default_scrapyd.conf file to MANIFEST.in 2011-01-28 16:21:00 -02:00
Pablo Hoffman 09f084c220 simplified scrapy shell code after recent changes. refs #306 2011-01-13 13:11:39 -02:00
Pablo Hoffman 0aac226b42 Fixed bug in Scrapy shell's fetch() which wasn't updating local variables properly. Closes #306 2011-01-13 13:08:11 -02:00
LucianU 0c5f605b0e The xmlfeed.tmpl file didn't use the naming convention specific of the XMLFeedSpider. Namely, it used parse_item (which has been deprecated) instead of parse_node and it didn't show the iterator and itertag attributes. 2011-01-11 17:23:46 -02:00
Pablo Hoffman 7dc521c56e make scrapyd package depend on specific scrapy version 2011-01-11 17:03:51 -02:00
Pablo Hoffman 048044c1f8 A couple of changes to fix #303: 
* improved detection of inside-project environments
* make list command faster (by only instantiating the spider manger)
* print a warning when extensions (middlewares, etc) are disabled with a message on NotConfigured exception
* assert that scrapy configuration hasn't been loaded in scrapyd.runner
* simplified IgnoreRequest exception, to avoid loading settings when importing scrapy.exceptions
* added test to make sure certain modules don't cause scrapy.conf module to be
  loaded, to ensure the scrapyd runner bootstraping performs properly
 2011-01-05 15:59:43 -02:00
Pablo Hoffman 48b30ba939 fixed compatibility with python 2.5 and removed unused code 2011-01-05 12:03:54 -02:00
Pablo Hoffman ac8166251e make scrapy*-0.12 packages conflict with scrapy*-0.11 packages 2011-01-04 13:56:28 -02:00
Pablo Hoffman b56e933be9 bumped version to 0.12 in documentation 2011-01-02 17:28:33 -02:00
Pablo Hoffman 5879389ad0 Bumped version to 0.12 2011-01-02 16:16:40 -02:00
Vikas Dhiman fe218abbfd scrapy.utils.memory.get_vmvalue_from_procfs() 
causes test case failure on SunOS 5.10 i86pc. Modified it to support SunOS 5.10
 2011-01-02 15:46:11 -02:00
Shane Evans aebe5d5073 make inside_project work with SCRAPY_SETTINGS_MODULE. Closes #300 2010-12-28 15:34:39 -02:00
Pablo Hoffman 3d8b368fc6 scrapyd: use runner from config (if not specified) on get_spider_list() 2010-12-28 11:16:58 -02:00
Pablo Hoffman fa644f7a5e Some simplifications to Scrapyd architecture and internals: 
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
  get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
  doesn't necesarilly run eggs (though it does in the default case)

--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
 2010-12-27 16:22:32 -02:00
Pablo Hoffman 9cd649b3a0 scrapyd: populate SCRAPY_SPIDER and SCRAPY_JOB environment variables 2010-12-26 19:32:56 -02:00
Pablo Hoffman 1c8d74eb5b scrapyd: populate SCRAPY_SLOT environment variable with the scrapyd slot number 2010-12-24 12:47:59 -02:00
Martin Olveyra d9a3df45c6 Remove deprecated match common prefix feature from IBL code 2010-12-23 14:40:22 -02:00
Pablo Hoffman 633ebc4c43 minor indentation improvement 2010-12-23 13:04:49 -02:00
Pablo Hoffman db07a9a938 Added notice to documentation, pointing dev to stable versions and viceversa 2010-12-23 13:03:40 -02:00
Pablo Hoffman 544308d6d0 updated ubuntu repos doc, in preparation for the 0.11 release 2010-12-21 11:02:56 -02:00
Pablo Hoffman fff22c269a Added test to make sure custom mime.types shipped with Scrapy is loaded, and made Scrapy more egg-friendly by using pkutil.get_data() 2010-12-20 15:16:21 -02:00
Pablo Hoffman 87aa63da2b moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
Pablo Hoffman 432251bedd Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
Pablo Hoffman 63c97f65f4 fixed csviter bug when called with a Response, not TextResponse 2010-12-14 18:21:07 -02:00
Pablo Hoffman 55ec7a0554 deploy command: cleanup temporary files before doing a new build 2010-12-14 17:20:31 -02:00
Pablo Hoffman 3df9070ffd fixed bin\runtests.bat windows script 2010-12-14 12:20:38 -02:00
Pablo Hoffman 002abf204f Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273 2010-12-13 14:05:47 -02:00
Pablo Hoffman f984d438a0 updated docs to use scrapy version on aptitude install lines 2010-12-13 14:02:42 -02:00
Pablo Hoffman 60bdb7dc0a Removed some deprecated code and modules 2010-12-13 10:34:07 -02:00
Pablo Hoffman 97a790d659 improved mechanism for tagging version with hg revision 2010-12-13 10:21:39 -02:00