RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,148 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

3148 Commits

Author SHA1 Message Date
Pablo Hoffman 4ca35dba87 Merge pull request #175 from euphoris/master 
add options -o and -t to the runspider command
 2012-09-26 09:15:40 -07:00
Jae-Myoung Yu e94bd8187c add options -o and -t to the runspider command 
copy the codes from commands/crawl.py to commands/runspider.py
 2012-09-25 09:38:00 +09:00
Pablo Hoffman a6eacf2c9b Merge pull request #172 from artem-dev/new_git_version 
nicer deploy versions for git repos if tags are available
 2012-09-21 07:39:47 -07:00
Artem Bogomyagkov 7685cb14a4 nicer deploy versions for git repos if tags are available 2012-09-21 12:34:45 +03:00
Pablo Hoffman b46b5a6ef0 Documented AutoThrottle extension and added to extensions available by 
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
 2012-09-20 18:52:57 -03:00
Pablo Hoffman c1cbc5de3f renamed CONTRIBUTING to CONTRIBUTING.md so that links are rendered as links in github 2012-09-19 13:58:58 -03:00
Pablo Hoffman f3516aed69 added CONTRIBUTING file, for github: https://github.com/blog/1184-contributing-guidelines 2012-09-19 13:54:16 -03:00
Pablo Hoffman e8eacd6f9e minor code simplification 2012-09-19 13:28:34 -03:00
Pablo Hoffman 7afedb89e9 backwards compatibility support for scrapy.conf.settings singleton for when scrapy.conf is imported *before* calling scrapy.cmdline.execute(), such as in custom bot runners 2012-09-19 12:44:59 -03:00
Pablo Hoffman 768a483953 merge adding missing changes from previous commit 2012-09-19 03:19:03 -03:00
Pablo Hoffman c7f8219901 - removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes, 
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
 2012-09-19 03:03:34 -03:00
Pablo Hoffman aadf7ff143 - removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes, 
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
 2012-09-19 02:56:38 -03:00
Pablo Hoffman 391cc06063 Merge pull request #174 from stav/master 
Selector documentation typo fixes
 2012-09-18 11:19:23 -07:00
stav 303e13f616 selector documentation typos 2012-09-18 12:56:52 -05:00
Pablo Hoffman 3d736e657f fixed typo in doc 2012-09-18 10:51:01 -03:00
Pablo Hoffman eed6eb49da make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses) 2012-09-17 10:11:07 -03:00
Pablo Hoffman 8f2dda12cc removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead 2012-09-16 21:21:44 -03:00
Pablo Hoffman cd823018aa removed another instance of scrapy.conf.settings singleton, this time from scrapy.cmdline (which manages scrapy command line tool), by moving the get_project_settings() function to scrapy.utils.project 2012-09-16 20:53:52 -03:00
Pablo Hoffman 9685c24059 restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode 2012-09-14 12:38:00 -03:00
Pablo Hoffman 81ed2d2d0b major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API. 2012-09-14 12:31:33 -03:00
Pablo Hoffman 8b48420559 better names for scheduler stats 2012-09-13 20:07:11 -03:00
Pablo Hoffman 263e302b6a added more scheduler stats 2012-09-13 19:46:45 -03:00
Pablo Hoffman d6867f7984 removed sphinx warnings about duplicate reference names 'this page' 2012-09-13 15:26:45 -03:00
Pablo Hoffman a874964ad4 renamed 'XPath Selectors' title to just 'Selectors' 2012-09-13 15:24:44 -03:00
Pablo Hoffman acb8895e1a changed note in scrapyd doc to use sphinx notes 2012-09-13 15:22:59 -03:00
Pablo Hoffman 26f1d5cb48 Merge pull request #171 from artem-dev/scrapyd_job_times 
added start and stop times for scrapyd list jobs web service
 2012-09-13 11:16:01 -07:00
Artem Bogomyagkov 4d8f253912 commited doc file missed from prev commit 2012-09-12 11:12:59 +03:00
Artem Bogomyagkov 5dde26d3c8 simplified code for finished jobs times, updated docs for scrapyd 2012-09-12 11:10:49 +03:00
Pablo Hoffman 7ace23c385 removed another instance of scrapy.conf singleton, this time from test_cmdline 2012-09-11 16:37:55 -03:00
Pablo Hoffman 8642645109 removed another instance of scrapy.conf singleton 2012-09-11 16:34:58 -03:00
Pablo Hoffman 7ef593c5c2 refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal 2012-09-11 16:27:19 -03:00
Artem Bogomyagkov a42a8cafef added start and stop times for scrapyd list jobs web service 2012-09-11 21:06:36 +03:00
Pablo Hoffman 1e2efe5664 scrapy deploy: retry setup.py bdist_egg on EINTR. closes #9 2012-09-07 15:40:26 -03:00
Pablo Hoffman 3c05fbb6a9 removed redundant comment 2012-09-05 15:54:23 -03:00
Pablo Hoffman 44fe4c6b78 set missing 'settings' attribute in FeedExports 2012-09-05 15:54:13 -03:00
Pablo Hoffman 0135a5364e removed debugging print 2012-09-05 15:53:30 -03:00
Pablo Hoffman 7bf0af7a58 fixed some bugs in get_func_args() and added more test cases 2012-09-04 19:01:20 -03:00
Pablo Hoffman 251d9a9033 backwards compatibility support for FeedExport's that don't receive a settings object in their constructor 2012-09-04 18:13:27 -03:00
Pablo Hoffman caa64908d2 added tests for get_func_args() and support for more cases 2012-09-04 18:07:44 -03:00
Pablo Hoffman fff2871828 added doc section (and FAQ) about spider arguments 2012-09-04 14:49:30 -03:00
Pablo Hoffman 241abcb8aa scrapyd: log errors in API calls to the scrapyd log 2012-09-04 14:22:21 -03:00
Pablo Hoffman f4a17ec272 removed references to Scrapy Snippets site 2012-09-03 22:19:15 -03:00
Pablo Hoffman b901e64044 replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161 2012-09-03 19:28:16 -03:00
Pablo Hoffman e2f9daac67 fixed formatting in scrapyd release notes 2012-09-03 16:58:58 -03:00
Pablo Hoffman dd1398b280 removed another instance of scrapy.conf.settings singleton, this time from download handlers 2012-09-03 16:44:18 -03:00
Daniel Graña 3b768ae989 oops. remove santas debug line 2012-08-31 20:10:32 -03:00
Pablo Hoffman c6435c5aa7 restored scrapyd log message 2012-08-31 18:49:20 -03:00
Pablo Hoffman 7c8af83138 added issue role to documentation 2012-08-31 18:39:30 -03:00
Pablo Hoffman 3891c20840 Merge pull request #164 from dangra/lazy-log-formatting 
format log lines lazily in case they are dropped by loglevels
 2012-08-31 14:33:41 -07:00
Daniel Graña a2d22307ad update news.rst 2012-08-31 17:38:40 -03:00