RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,016 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

301 Commits

Author SHA1 Message Date
Pablo Hoffman dd662e09d8 some minor fixes to scheduler middleware doc 2009-11-19 12:23:54 -02:00
Pablo Hoffman f4e93700bd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-19 10:44:02 -02:00
Pablo Hoffman c4f77c4da0 minor fixes to images doc (thanks amccloud) 2009-11-16 11:15:25 -02:00
Pablo Hoffman 0d6aee1f12 updated wrong documentation 2009-11-13 20:03:56 -02:00
Pablo Hoffman aeab5370cb StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method 2009-11-14 20:28:59 -02:00
Pablo Hoffman c4c6e7c8cd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-13 20:04:39 -02:00
Pablo Hoffman 07655d05ea renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER 2009-11-13 14:38:22 -02:00
Pablo Hoffman 564abd10ad Refactored HttpCache middleware: 
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure

Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
 2009-11-13 14:25:47 -02:00
Pablo Hoffman db7fec1fef fixed doc typo 2009-11-12 12:17:39 -02:00
Pablo Hoffman 415dec4e16 made offsite middleware log messages when filtering out requests 2009-11-12 10:17:21 -02:00
Pablo Hoffman 49e39bf1ba fixed typo 2009-11-06 16:49:48 -02:00
Pablo Hoffman 791f4932dd added clarification about versioning and api stability 2009-11-06 16:28:51 -02:00
Pablo Hoffman 74d0e82dbe renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_* 
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
 2009-11-06 15:54:17 -02:00
Pablo Hoffman 919cd5b789 renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS 2009-11-06 15:44:11 -02:00
Pablo Hoffman d604dca96d renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER 2009-11-06 15:42:11 -02:00
Pablo Hoffman 7728a23e99 Changed item pipeline API to pass spider references (instead of domain names) to process_item() method 2009-11-06 13:46:36 -02:00
Pablo Hoffman a432c1ee40 updated logging doc to include new spider argument in log functions 2009-11-04 14:49:24 -02:00
Pablo Hoffman 97c322707a * Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle} 
* Changed them to pass spider instances only (no domains) (refs #105)
 2009-11-03 00:39:02 -02:00
Pablo Hoffman 904cde6513 added clarification about new dont_click argument of FormRequest.from_response() method 2009-10-29 13:47:10 -02:00
Ismael Carnales a244d23b89 added dont_click attr to FormRequest 2009-10-29 13:18:13 -02:00
Pablo Hoffman b41c5b5d5b fixed typo in intro/install doc (thanks phaithful) 2009-10-29 10:41:20 -02:00
Pablo Hoffman 9b5fef4f48 fixed typo in intro/install doc (thanks phaithful) 2009-10-28 09:34:31 -02:00
Pablo Hoffman 7296a7b889 added DEFAULT_RESPONSE_ENCODING setting 2009-10-21 16:13:41 -02:00
Pablo Hoffman 720bc166cf updated new clickdata argument doc 2009-10-20 17:21:56 -02:00
Daniel Grana 6abb3c17ee Improve FormRequest.from_response method to pass click data arguments to ClientForm library 2009-10-20 15:51:41 -02:00
Pablo Hoffman 2712d55cb9 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 23:58:38 -02:00
Pablo Hoffman bd481751d8 moved images pipeline documentation to stable doc 
--HG--
rename : docs/experimental/images.rst => docs/topics/images.rst
 2009-10-07 22:57:25 -02:00
Pablo Hoffman b4d202a6b0 added note about memory usage extension not working on windows 2009-10-07 22:57:10 -02:00
Pablo Hoffman 937acd91d1 improved documentation of http proxy middleware 2009-10-07 21:00:34 -02:00
Pablo Hoffman d03bf1f27f Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 20:38:46 -02:00
Pablo Hoffman 37d9e015bb minor fix to tutorial 2009-10-07 20:15:49 -02:00
Daniel Grana bc64ca3e13 Add support to set http proxies per request, and obey enviroment variables http_proxy and no_proxy by default. 2009-10-05 04:10:22 -02:00
Daniel Grana 8aa7d153ae rewrote of downloader handlers 
* add REQUEST_HANDLERS setting with defaults for file, http and https schemes
* add documentation of new setting
* add unittests for all the builtin handlers
* remove unused getPage function
 2009-10-05 04:10:22 -02:00
Pablo Hoffman a0eec7eaf6 some typos fixes and updates to install doc 2009-09-29 09:44:02 -03:00
Ismael Carnales 1646482bef reformatted installation guide 2009-09-29 08:41:34 -03:00
Pablo Hoffman 4a9d9282bc removed obsolete scrapy architecture dia diagram 2009-09-28 23:54:01 -03:00
Ismael Carnales 5862ba7db7 modified doc to reflect the new spider callback return policy (lists not needed) 2009-09-22 11:25:40 -03:00
Ismael Carnales 802f918b69 removed obsolete doc static file 2009-09-22 11:25:38 -03:00
Pablo Hoffman 6e93872955 updated installation guide for using releases 2009-09-17 11:06:55 -03:00
Pablo Hoffman 132557dd14 some deployment changes in preparation for the 0.7.0 release candidate 2009-09-16 22:40:36 -03:00
Ismael Carnales fd41f06056 added doc on how to enable an Item Pipeline component 2009-09-16 14:19:16 -03:00
Ismael Carnales 404e7e09d7 changed spider doc references in BaseSpider class 2009-09-16 14:10:11 -03:00
Daniel Grana 062730cbd8 fix csv exporter documentation 2009-09-16 00:17:50 -03:00
Pablo Hoffman 56b292e057 XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience) 2009-09-14 22:05:52 -03:00
Pablo Hoffman e8960bf616 added runspider command to run spiders directly, without having to create a project 2009-09-14 22:05:14 -03:00
Pablo Hoffman 99467d4e6e Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method 2009-09-13 20:51:43 -03:00
Pablo Hoffman 921fc4f3bf Big Scrapy core refactoring to pass around spider references instead of domains. 
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.

This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.

However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.

--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
 2009-09-12 14:34:18 -03:00
Pablo Hoffman 8d49dc2fb5 changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc 2009-09-11 17:36:00 -03:00
Pablo Hoffman e20f766792 fixed some typos 2009-09-11 16:55:37 -03:00
Pablo Hoffman c2fe350f72 more changes to images pipeline doc 2009-09-11 16:53:36 -03:00
Ismael Carnales ada46a2dbb styled imagesp doc 2009-09-11 15:30:46 -03:00
Pablo Hoffman be0f2beef0 more cleanup to scheduler middelware doc, and permanentely moved to experimental doc 2009-09-11 13:27:31 -03:00
Pablo Hoffman 0af052b68f removed confusing title 2009-09-11 12:19:18 -03:00
Pablo Hoffman f3240748cb changed link to scheduler middleware doc, now in experimental 2009-09-11 12:03:23 -03:00
Ismael Carnales 3998a0cb58 added more scheduler middleware documentation, and moved it to experimental 
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
 2009-09-11 11:58:53 -03:00
Pablo Hoffman d242a20573 updated images pipeline doc 2009-09-11 11:47:12 -03:00
Pablo Hoffman f1bb8dc2a3 first cleanup of spider manager api 
- removed asdict() and reload() methods
- added list() method
- removed default spider
 2009-09-10 19:06:46 -03:00
Pablo Hoffman f85813cd94 added FAQ entry about scrapy recipes and community spiders 2009-09-10 18:32:50 -03:00
Pablo Hoffman 269724a2b7 added Debugger extension, removed StackTraceDump from extensions available by default 2009-09-08 22:32:17 -03:00
Ismael Carnales 4ddfa9a2a3 stlyed downloaded middleware doc 2009-09-07 12:18:57 -03:00
Ismael Carnales e3df11e5bb added module directive to spidermw documentation 2009-09-07 12:03:24 -03:00
Pablo Hoffman 827aa19c6e removed obsolete scrapy.utils.db module 2009-09-04 17:38:14 -03:00
Pablo Hoffman 861a803cc3 removed obsolete RestrictMiddleware 2009-09-04 17:22:56 -03:00
Ismael Carnales 7e2587169b added missing middleware docs 2009-09-04 12:39:02 -03:00
Pablo Hoffman aefb94063a more updates to spider middleware doc 2009-09-04 13:46:04 -03:00
Pablo Hoffman d04640be5c some improvements to spider middleware doc 2009-09-04 13:29:16 -03:00
Pablo Hoffman 96bb223c13 removed (pretty useless) DebugMiddleware 2009-09-04 12:59:58 -03:00
Pablo Hoffman 8a715701ec fixed another doc typo 2009-09-03 14:31:00 -03:00
Ismael Carnales 3c1bb7bc40 fixed typo in djangoitems doc (thanks anibal) 2009-09-03 11:23:25 -03:00
Daniel Grana 0e7b2a6da5 write header line by default when using csv exporter 
--HG--
extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e
 2009-09-03 13:58:39 -03:00
Pablo Hoffman 596d2c4479 moved CoreStats extension to scrapy.contrib.corestats 
--HG--
rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py
 2009-09-01 23:00:49 -03:00
Pablo Hoffman 6a50af05d7 removed useless SpiderReloader extension 2009-09-01 22:49:15 -03:00
Pablo Hoffman 79851aefa6 moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation 
--HG--
rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py
 2009-09-01 22:38:37 -03:00
Pablo Hoffman d3c51fd6f2 improved images pipeline documentation 2009-09-01 21:07:47 -03:00
Pablo Hoffman 18fd635124 another doc typo 2009-09-01 12:52:40 -03:00
Pablo Hoffman 538cc9803a fixed doc typo 2009-09-01 12:47:53 -03:00
Pablo Hoffman df0e1f005f exporters doc: fixed example and some typos 2009-09-01 08:56:54 -03:00
Pablo Hoffman ac8f46ce9e added File Export Pipeline reference to Exporters doc 2009-08-31 21:01:35 -03:00
Pablo Hoffman 8d006e9ea1 moved item exporters doc to stable doc 
--HG--
rename : docs/experimental/exporters.rst => docs/topics/exporters.rst
 2009-08-31 20:47:12 -03:00
Pablo Hoffman 0b152c99b5 added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines 2009-08-31 20:40:41 -03:00
Pablo Hoffman 8fab524978 moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console 2009-08-31 12:44:32 -03:00
Pablo Hoffman 884f0c878f Stats collectin: fixed race condition between stats persistance and population of stats on domain close 2009-08-29 19:44:13 -03:00
Pablo Hoffman 895c70e036 doc: fixed some links to scrapy-ctl topic 2009-08-29 18:23:55 -03:00
Pablo Hoffman 64572124e0 added doc about SCRAPY_SETTINGS_MODULE 2009-08-29 18:20:13 -03:00
Pablo Hoffman 9dc7a3a373 some minor adjustments to logging doc 2009-08-29 18:10:28 -03:00
Pablo Hoffman 60cbf24c89 more cleanups to startproject and project templates 
--HG--
rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py
 2009-08-29 04:29:47 -03:00
Pablo Hoffman c87216f02b doc: added missing :synopsis: to some modules 2009-08-29 03:46:58 -03:00
Pablo Hoffman 2f55d7932b replaced :ref: by :doc: links in doc index 2009-08-29 03:37:59 -03:00
Pablo Hoffman 924219dd5a - added reference documentation about scrapy-ctl.py script 
- yet another refactor to cmdline module
- removed help command (use -h to help now)

--HG--
rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst
 2009-08-28 20:32:55 -03:00
Pablo Hoffman 609aed4553 added inspect_response() function for inspecting responses from spiders 2009-08-27 19:33:33 -03:00
Pablo Hoffman ea4f16bae6 refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut 2009-08-27 18:24:08 -03:00
Ismael Carnales fb39bca24f added djangoitem doc 2009-08-26 08:44:22 -03:00
Pablo Hoffman 65ab611fb0 more updates to HttpErrorMiddleware doc 2009-08-26 08:30:18 -03:00
Pablo Hoffman 44783a3a06 minor improvements to FAQ entry 2009-08-26 00:18:58 -03:00
Pablo Hoffman 4dba75a73f updated HttpErrorMiddleware doc 2009-08-25 20:13:58 -03:00
Pablo Hoffman 8a074c9cb5 removed scrapy-admin.py command, and left only scrapy-ctl as the only scrapy command 2009-08-24 15:43:36 -03:00
Ismael Carnales 39540b188a changed torrent in overview doc 2009-08-24 15:11:04 -03:00
Ismael Carnales 4b5aa30867 minor update to tutorial 2009-08-24 14:34:17 -03:00
Pablo Hoffman 0363040884 doc: added FAQ entry about Accept-Language 2009-08-24 13:56:44 -03:00
Ismael Carnales 85282a4b76 added scrapy commandline scripts doc 2009-08-24 12:02:44 -03:00