RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,796 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

508 Commits

Author SHA1 Message Date
Pablo Hoffman 031eb1e5ed removed no longer used SpiderScheduler (obsoleted by ExecutionQueue) 2010-05-28 17:27:15 -03:00
Ismael Carnales a71dc295af Some mail improvements and tests. 
* Add mail_sent signal and use it in MailSender
* Add MAIL_DEBUG setting to not send mails when testing
* Add MailSender tests
 2010-05-28 16:51:47 -03:00
Ping Yin 6059221716 Compose: stop process on None value by default 
By doing this, we can use str.lower as a processor safely without
checking whether the given value is None.

By passing stop_on_none=False as keyword argument, this behaviour can be changed.

Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-04-08 10:59:47 +08:00
Ping Yin 15b879f845 ItemLoader: Update docs for {add,replace,get}_{value,xpath} 
Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-05-18 17:54:25 +08:00
Pablo Hoffman bfd9cb42e5 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-05-17 20:11:27 -03:00
Pablo Hoffman 076cdfd585 Added documentation about contributing to Scrapy 2010-05-17 20:10:46 -03:00
Pablo Hoffman 7a55158fed fixed documentation bug (thanks rhill for reporting) 2010-05-11 11:25:03 -03:00
Steven Almeroth 5d03405cac FormRequest.from_response doc fix. closes #155 
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
 2010-04-26 22:28:07 -03:00
Pablo Hoffman 2121a30c74 added note about installing Zope.Interface in windows platforms 2010-04-24 18:19:52 -03:00
Daniel Grana 6c12106803 Remove shpinx warning introduced by shorter title overline 2010-04-18 23:42:56 -03:00
Lucian Ursu 2f8c052484 #154: Language fixes to the documentation 2010-04-18 23:39:54 -03:00
Pablo Hoffman dfdac356af added missing default values to file xporter doc 2010-04-02 02:49:18 -03:00
Pablo Hoffman f19c939925 fixed doc typo 2010-03-26 08:28:32 -03:00
Pablo Hoffman 99a876754c Improved "What else?" section of "Scrapy at a glance" overview 2010-03-20 20:24:18 -03:00
Pablo Hoffman 234fd709ad fixed doc typo (thanks Victor) 2010-03-19 10:32:17 -03:00
Daniel Grana 184cf6684f Remove HttpException references from docs. Since 0.7, scrapy returns non-200 as Response objects and does not raise HttpException anymore 2010-03-18 10:05:33 -03:00
Daniel Grana 17091902f3 Explicity say where to save item class in "Defining our item" section of tutorial 2010-03-12 14:12:49 -02:00
Daniel Grana c925c9e9a0 Notify spider when requests are ignored by HttpErrorMiddleware, and generally when any call to process_spider_input raises an exception 2010-05-12 16:41:06 -03:00
Daniel Grana c0d45846b8 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-26 22:29:45 -03:00
Pablo Hoffman 81f6502e37 Automated merge with http://hg.scrapy.org/scrapy-0.8/ 2010-04-24 18:22:13 -03:00
Daniel Grana 658e6f15e9 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-18 23:44:59 -03:00
Daniel Grana 68a875edb0 update ENCODING_ALIASES setting default value in settings documentation topic 2010-04-07 10:54:54 -03:00
Pablo Hoffman de32612c99 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-04-02 02:49:51 -03:00
Rolando Espinoza La fuente db5c3df679 SEP12 implementation 
* Rename BaseSpider.domain_name to BaseSpider.name

    This patch implements the domain_name to name change in BaseSpider class and
    change all spider instantiations to use the new attribute.

  * Add allowed_domains to spider

    This patch implements the merging of spider.domain_name and
    spider.extra_domain_names in spider.allowed_domains for offsite checking
    purposes.

    Note that spider.domain_name is not touched by this patch, only not used.

  * Remove spider.domain_name references from scrapy.stats

    * Rename domain_stats to spider_stats in MemoryStatsCollector
    * Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
    * Rename domain_stats_history table to spider_data_history and rename domain
    field to spider in MysqlStatsCollector

  * Refactor genspider command

    The new signature for genspider is: genspider [options] <domain_name>.

    Genspider uses domain_name for spider name and for the module name.

  * Remove spider.domain_name references

  * Update crawl command signature <spider|url>

  * docs: updated references to domain_name

  * examples/experimental: use spider.name

  * genspider: require <name> <domain>

  * spidermanager: renamed crawl_domain to crawl_spider_name

  * spiderctl: updated references of *domain* to spider

  * added backward compatiblity with legacy spider's attributes
    'domain_name' and 'extra_domain_names'
 2010-04-01 18:27:22 -03:00
Pablo Hoffman 2299deda66 updated wrong link in doc 2010-03-26 14:02:33 -03:00
Pablo Hoffman 7cf2f87e27 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-26 08:29:34 -03:00
Pablo Hoffman 1330697c3d Some improvements to Response encoding support: 
* added encoding aliases, configurable through a new ENCODING_ALIASES setting
* Response.encoding now returns the real encoding detected for the body
* simplified TextResponse API by removing body_encoding() and
  headers_encoding() methods
* Response.encoding now tries to infer the encoding from the body always (it
  was done before only on HtmlResponse and TextResponse)
* removed scrapy.utils.encoding.add_encoding_alias() function
* updated implementation of scrapy.utils.response function to reflect these API
  changes
* updated documentation to reflect API changes
 2010-03-25 15:47:10 -03:00
Pablo Hoffman 9ddcd1095d sort setting alphabetically 2010-03-25 11:45:06 -03:00
Pablo Hoffman 4fa833c849 Added LOG_ENCODING setting 2010-03-24 12:13:38 -03:00
Pablo Hoffman 87e68e7438 Made MailSender non IO-blocking, and improved MailSender documentation 2010-03-22 13:37:37 -03:00
Pablo Hoffman 1dfc79b5d0 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-20 20:48:11 -03:00
Pablo Hoffman 264cd2e035 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-19 10:32:42 -03:00
Pablo Hoffman d12cd22d5e switched default scheduler order to DFO, which consumes less memory by default 2010-03-04 10:15:58 -02:00
Pablo Hoffman 180c091fb2 Fixed encoding issue (reported in #135) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation. 2010-02-24 14:01:29 -02:00
Pablo Hoffman bbef0fe870 Automated merge with http://hg.scrapy.org/users/rolando/scrapy/ 2010-02-20 11:12:37 -02:00
Pablo Hoffman a3d22c7240 Automated merge with http://hg.scrapy.org/scrapy-0.8/ 2010-02-19 23:11:24 -02:00
Pablo Hoffman 60961e5499 minor documentation fix (refs #135) 2010-02-19 23:09:48 -02:00
Pablo Hoffman c1f8198639 Added RANDOMIZE_DOWNLOAD_DELAY setting 2010-02-19 21:53:18 -02:00
Rolando Espinoza La fuente a6a3f085a7 docs: added crawlspider v2 outline documentation 
Sign-Off: Rolando Espinoza La fuente
 2010-02-19 18:22:38 -04:00
Rolando Espinoza La fuente 7235040936 merged upstream 2010-02-19 17:41:45 -04:00
Daniel Grana 91f4d6dc51 docs: adds another spider example that yields multiples requests/items from a single callback 2010-02-18 16:51:05 -02:00
Pablo Hoffman 57d60eae39 sort settings doc alphabetically by setting name 2010-01-31 18:11:13 -02:00
Pablo Hoffman 67858af83c fixed doc typo 2010-01-18 18:16:58 -02:00
Pablo Hoffman 08eeaf98a2 fixed description of LOG_STDOUT setting 2010-01-13 15:51:08 -02:00
Pablo Hoffman 48739ae60c install.rst: added explanation about why libxml2 2.6.28 or above is required 2010-01-13 12:20:24 -02:00
Rolando Espinoza La fuente 1402da31c5 docs: fixed typos and updated code examples 2010-01-11 12:28:22 -04:00
Pablo Hoffman d60412ce19 titlecased Scrapy easy_install and some fixes to sign_release.sh script 2009-12-13 14:23:31 -02:00
Pablo Hoffman 422d6facb2 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-12-12 16:52:07 -02:00
Pablo Hoffman 9d50604d24 added |version| to documentation title 2009-12-12 16:51:59 -02:00
Pablo Hoffman a953efd8e5 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-12-12 15:40:16 -02:00
Ismael Carnales 4ecc909bc1 Fix RobotsTxtMiddleware reference in doc 2009-12-04 15:37:24 -02:00
Ismael Carnales 07344666e2 Move webconsole extensions doc to webconsole topic 2009-12-01 10:47:11 -02:00
Ismael Carnales e694c8ed02 Remove domain references in close spider extension doc 2009-11-30 11:38:56 -02:00
Ismael Carnales 12a7ff7312 Rename Close domain to close spider in extensions doc 2009-11-30 11:36:18 -02:00
Ismael Carnales 8d9cedd88b Reorder signals doc to respect alphabetical order 2009-11-30 11:29:19 -02:00
Ismael Carnales 93cc3d2715 Correct param formatting in item pipelines doc 2009-11-30 11:04:15 -02:00
Pablo Hoffman 6084be3b2e added iter_all() function to scrapy.util.trackref module and improved memory leaks documentation. also added a new FAQ antry about memory issues 2009-11-28 16:21:59 -02:00
Pablo Hoffman dd662e09d8 some minor fixes to scheduler middleware doc 2009-11-19 12:23:54 -02:00
Pablo Hoffman f4e93700bd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-19 10:44:02 -02:00
Pablo Hoffman c4f77c4da0 minor fixes to images doc (thanks amccloud) 2009-11-16 11:15:25 -02:00
Pablo Hoffman 0d6aee1f12 updated wrong documentation 2009-11-13 20:03:56 -02:00
Pablo Hoffman aeab5370cb StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method 2009-11-14 20:28:59 -02:00
Pablo Hoffman c4c6e7c8cd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-13 20:04:39 -02:00
Pablo Hoffman 07655d05ea renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER 2009-11-13 14:38:22 -02:00
Pablo Hoffman 564abd10ad Refactored HttpCache middleware: 
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure

Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
 2009-11-13 14:25:47 -02:00
Pablo Hoffman db7fec1fef fixed doc typo 2009-11-12 12:17:39 -02:00
Pablo Hoffman 415dec4e16 made offsite middleware log messages when filtering out requests 2009-11-12 10:17:21 -02:00
Pablo Hoffman 49e39bf1ba fixed typo 2009-11-06 16:49:48 -02:00
Pablo Hoffman 791f4932dd added clarification about versioning and api stability 2009-11-06 16:28:51 -02:00
Pablo Hoffman 74d0e82dbe renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_* 
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
 2009-11-06 15:54:17 -02:00
Pablo Hoffman 919cd5b789 renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS 2009-11-06 15:44:11 -02:00
Pablo Hoffman d604dca96d renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER 2009-11-06 15:42:11 -02:00
Pablo Hoffman 7728a23e99 Changed item pipeline API to pass spider references (instead of domain names) to process_item() method 2009-11-06 13:46:36 -02:00
Pablo Hoffman a432c1ee40 updated logging doc to include new spider argument in log functions 2009-11-04 14:49:24 -02:00
Pablo Hoffman 97c322707a * Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle} 
* Changed them to pass spider instances only (no domains) (refs #105)
 2009-11-03 00:39:02 -02:00
Pablo Hoffman 904cde6513 added clarification about new dont_click argument of FormRequest.from_response() method 2009-10-29 13:47:10 -02:00
Ismael Carnales a244d23b89 added dont_click attr to FormRequest 2009-10-29 13:18:13 -02:00
Pablo Hoffman b41c5b5d5b fixed typo in intro/install doc (thanks phaithful) 2009-10-29 10:41:20 -02:00
Pablo Hoffman 9b5fef4f48 fixed typo in intro/install doc (thanks phaithful) 2009-10-28 09:34:31 -02:00
Pablo Hoffman 7296a7b889 added DEFAULT_RESPONSE_ENCODING setting 2009-10-21 16:13:41 -02:00
Pablo Hoffman 720bc166cf updated new clickdata argument doc 2009-10-20 17:21:56 -02:00
Daniel Grana 6abb3c17ee Improve FormRequest.from_response method to pass click data arguments to ClientForm library 2009-10-20 15:51:41 -02:00
Pablo Hoffman 2712d55cb9 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 23:58:38 -02:00
Pablo Hoffman bd481751d8 moved images pipeline documentation to stable doc 
--HG--
rename : docs/experimental/images.rst => docs/topics/images.rst
 2009-10-07 22:57:25 -02:00
Pablo Hoffman b4d202a6b0 added note about memory usage extension not working on windows 2009-10-07 22:57:10 -02:00
Pablo Hoffman 937acd91d1 improved documentation of http proxy middleware 2009-10-07 21:00:34 -02:00
Pablo Hoffman d03bf1f27f Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 20:38:46 -02:00
Pablo Hoffman 37d9e015bb minor fix to tutorial 2009-10-07 20:15:49 -02:00
Daniel Grana bc64ca3e13 Add support to set http proxies per request, and obey enviroment variables http_proxy and no_proxy by default. 2009-10-05 04:10:22 -02:00
Daniel Grana 8aa7d153ae rewrote of downloader handlers 
* add REQUEST_HANDLERS setting with defaults for file, http and https schemes
* add documentation of new setting
* add unittests for all the builtin handlers
* remove unused getPage function
 2009-10-05 04:10:22 -02:00
Pablo Hoffman a0eec7eaf6 some typos fixes and updates to install doc 2009-09-29 09:44:02 -03:00
Ismael Carnales 1646482bef reformatted installation guide 2009-09-29 08:41:34 -03:00
Pablo Hoffman 4a9d9282bc removed obsolete scrapy architecture dia diagram 2009-09-28 23:54:01 -03:00
Ismael Carnales 5862ba7db7 modified doc to reflect the new spider callback return policy (lists not needed) 2009-09-22 11:25:40 -03:00
Ismael Carnales 802f918b69 removed obsolete doc static file 2009-09-22 11:25:38 -03:00
Pablo Hoffman 6e93872955 updated installation guide for using releases 2009-09-17 11:06:55 -03:00
Pablo Hoffman 132557dd14 some deployment changes in preparation for the 0.7.0 release candidate 2009-09-16 22:40:36 -03:00
Ismael Carnales fd41f06056 added doc on how to enable an Item Pipeline component 2009-09-16 14:19:16 -03:00
Ismael Carnales 404e7e09d7 changed spider doc references in BaseSpider class 2009-09-16 14:10:11 -03:00
Daniel Grana 062730cbd8 fix csv exporter documentation 2009-09-16 00:17:50 -03:00
Pablo Hoffman 56b292e057 XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience) 2009-09-14 22:05:52 -03:00
Pablo Hoffman e8960bf616 added runspider command to run spiders directly, without having to create a project 2009-09-14 22:05:14 -03:00
Pablo Hoffman 99467d4e6e Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method 2009-09-13 20:51:43 -03:00
Pablo Hoffman 921fc4f3bf Big Scrapy core refactoring to pass around spider references instead of domains. 
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.

This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.

However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.

--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
 2009-09-12 14:34:18 -03:00
Pablo Hoffman 8d49dc2fb5 changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc 2009-09-11 17:36:00 -03:00
Pablo Hoffman e20f766792 fixed some typos 2009-09-11 16:55:37 -03:00
Pablo Hoffman c2fe350f72 more changes to images pipeline doc 2009-09-11 16:53:36 -03:00
Ismael Carnales ada46a2dbb styled imagesp doc 2009-09-11 15:30:46 -03:00
Pablo Hoffman be0f2beef0 more cleanup to scheduler middelware doc, and permanentely moved to experimental doc 2009-09-11 13:27:31 -03:00
Pablo Hoffman 0af052b68f removed confusing title 2009-09-11 12:19:18 -03:00
Pablo Hoffman f3240748cb changed link to scheduler middleware doc, now in experimental 2009-09-11 12:03:23 -03:00
Ismael Carnales 3998a0cb58 added more scheduler middleware documentation, and moved it to experimental 
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
 2009-09-11 11:58:53 -03:00
Pablo Hoffman d242a20573 updated images pipeline doc 2009-09-11 11:47:12 -03:00
Pablo Hoffman f1bb8dc2a3 first cleanup of spider manager api 
- removed asdict() and reload() methods
- added list() method
- removed default spider
 2009-09-10 19:06:46 -03:00
Pablo Hoffman f85813cd94 added FAQ entry about scrapy recipes and community spiders 2009-09-10 18:32:50 -03:00
Pablo Hoffman 269724a2b7 added Debugger extension, removed StackTraceDump from extensions available by default 2009-09-08 22:32:17 -03:00
Ismael Carnales 4ddfa9a2a3 stlyed downloaded middleware doc 2009-09-07 12:18:57 -03:00
Ismael Carnales e3df11e5bb added module directive to spidermw documentation 2009-09-07 12:03:24 -03:00
Pablo Hoffman 827aa19c6e removed obsolete scrapy.utils.db module 2009-09-04 17:38:14 -03:00
Pablo Hoffman 861a803cc3 removed obsolete RestrictMiddleware 2009-09-04 17:22:56 -03:00
Ismael Carnales 7e2587169b added missing middleware docs 2009-09-04 12:39:02 -03:00
Pablo Hoffman aefb94063a more updates to spider middleware doc 2009-09-04 13:46:04 -03:00
Pablo Hoffman d04640be5c some improvements to spider middleware doc 2009-09-04 13:29:16 -03:00
Pablo Hoffman 96bb223c13 removed (pretty useless) DebugMiddleware 2009-09-04 12:59:58 -03:00
Pablo Hoffman 8a715701ec fixed another doc typo 2009-09-03 14:31:00 -03:00
Ismael Carnales 3c1bb7bc40 fixed typo in djangoitems doc (thanks anibal) 2009-09-03 11:23:25 -03:00
Daniel Grana 0e7b2a6da5 write header line by default when using csv exporter 
--HG--
extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e
 2009-09-03 13:58:39 -03:00
Pablo Hoffman 596d2c4479 moved CoreStats extension to scrapy.contrib.corestats 
--HG--
rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py
 2009-09-01 23:00:49 -03:00
Pablo Hoffman 6a50af05d7 removed useless SpiderReloader extension 2009-09-01 22:49:15 -03:00
Pablo Hoffman 79851aefa6 moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation 
--HG--
rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py
 2009-09-01 22:38:37 -03:00
Pablo Hoffman d3c51fd6f2 improved images pipeline documentation 2009-09-01 21:07:47 -03:00
Pablo Hoffman 18fd635124 another doc typo 2009-09-01 12:52:40 -03:00
Pablo Hoffman 538cc9803a fixed doc typo 2009-09-01 12:47:53 -03:00
Pablo Hoffman df0e1f005f exporters doc: fixed example and some typos 2009-09-01 08:56:54 -03:00
Pablo Hoffman ac8f46ce9e added File Export Pipeline reference to Exporters doc 2009-08-31 21:01:35 -03:00
Pablo Hoffman 8d006e9ea1 moved item exporters doc to stable doc 
--HG--
rename : docs/experimental/exporters.rst => docs/topics/exporters.rst
 2009-08-31 20:47:12 -03:00
Pablo Hoffman 0b152c99b5 added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines 2009-08-31 20:40:41 -03:00
Pablo Hoffman 8fab524978 moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console 2009-08-31 12:44:32 -03:00
Pablo Hoffman 884f0c878f Stats collectin: fixed race condition between stats persistance and population of stats on domain close 2009-08-29 19:44:13 -03:00
Pablo Hoffman 895c70e036 doc: fixed some links to scrapy-ctl topic 2009-08-29 18:23:55 -03:00
Pablo Hoffman 64572124e0 added doc about SCRAPY_SETTINGS_MODULE 2009-08-29 18:20:13 -03:00
Pablo Hoffman 9dc7a3a373 some minor adjustments to logging doc 2009-08-29 18:10:28 -03:00
Pablo Hoffman 60cbf24c89 more cleanups to startproject and project templates 
--HG--
rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py
 2009-08-29 04:29:47 -03:00
Pablo Hoffman c87216f02b doc: added missing :synopsis: to some modules 2009-08-29 03:46:58 -03:00
Pablo Hoffman 2f55d7932b replaced :ref: by :doc: links in doc index 2009-08-29 03:37:59 -03:00
Pablo Hoffman 924219dd5a - added reference documentation about scrapy-ctl.py script 
- yet another refactor to cmdline module
- removed help command (use -h to help now)

--HG--
rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst
 2009-08-28 20:32:55 -03:00
Pablo Hoffman 609aed4553 added inspect_response() function for inspecting responses from spiders 2009-08-27 19:33:33 -03:00
Pablo Hoffman ea4f16bae6 refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut 2009-08-27 18:24:08 -03:00
Ismael Carnales fb39bca24f added djangoitem doc 2009-08-26 08:44:22 -03:00
Pablo Hoffman 65ab611fb0 more updates to HttpErrorMiddleware doc 2009-08-26 08:30:18 -03:00
Pablo Hoffman 44783a3a06 minor improvements to FAQ entry 2009-08-26 00:18:58 -03:00
Pablo Hoffman 4dba75a73f updated HttpErrorMiddleware doc 2009-08-25 20:13:58 -03:00
Pablo Hoffman 8a074c9cb5 removed scrapy-admin.py command, and left only scrapy-ctl as the only scrapy command 2009-08-24 15:43:36 -03:00
Ismael Carnales 39540b188a changed torrent in overview doc 2009-08-24 15:11:04 -03:00
Ismael Carnales 4b5aa30867 minor update to tutorial 2009-08-24 14:34:17 -03:00
Pablo Hoffman 0363040884 doc: added FAQ entry about Accept-Language 2009-08-24 13:56:44 -03:00
Ismael Carnales 85282a4b76 added scrapy commandline scripts doc 2009-08-24 12:02:44 -03:00
Pablo Hoffman ead8cb9fac removed documentation about ugly DontCloseDomain exception (which will be removed in the future) 2009-08-24 11:57:21 -03:00
Pablo Hoffman 74f706b356 renamed "parse_item" method of XMLFeedSpider to "parse_node", keeping backwards compatibility 2009-08-24 10:54:34 -03:00
Pablo Hoffman 31693eb90f dropped "cache" attribute of Request and Response objects 2009-08-24 10:34:05 -03:00
Pablo Hoffman 0186c6937a HTTP auth middleware: added doc and unittest 2009-08-24 08:07:20 -03:00
Pablo Hoffman 46aaea9f32 doc: improved documentation about debugging leaks 2009-08-23 20:36:00 -03:00
Pablo Hoffman 0ed849248f some improvements to item exporters 
- passed previous class attributes to instances attributes
- better handling of constructor arguments
- better coverage on unittets (including encoding)
- updated documentation with new changes
 2009-08-23 05:48:35 -03:00
Pablo Hoffman 7461710ef1 added some missing dots 2009-08-21 21:54:10 -03:00
Pablo Hoffman 9635a7839c rearranged documentation into a better organization 
--HG--
rename : docs/topics/index.rst => docs/index.rst
 2009-08-21 21:49:54 -03:00
Pablo Hoffman 4761e0c88d minor doc correction 2009-08-21 19:11:59 -03:00
Pablo Hoffman 012fd1cc2f moved api-stability.rst doc to root and updated it 
--HG--
rename : docs/misc/api-stability.rst => docs/api-stability.rst
 2009-08-21 16:29:23 -03:00
Pablo Hoffman ef6f04eb06 moved doc about debugging memory leaks to its own topic and added doc about trackref module 2009-08-21 16:07:16 -03:00
Pablo Hoffman 4980eb4968 added titles to signals doc 2009-08-21 15:07:52 -03:00
Pablo Hoffman daea3f8a0c sphinx docs: replaced custom :exception: xref by standard :exc: 2009-08-21 15:05:06 -03:00
Ismael Carnales c08d3aa9cc updated tutorial to use new items api 2009-08-21 14:16:27 -03:00
Pablo Hoffman 37760eb13f improved consistency of logging settings to use LOG_* 2009-08-21 08:54:12 -03:00
Ismael Carnales f0689a9ee8 fixed error in link extractors doc, thanks tarasm 2009-08-21 08:34:53 -03:00
Pablo Hoffman 41f1972e47 removed unused TRACE log level and improved logging documentation 2009-08-20 18:17:48 -03:00
Pablo Hoffman 20e82335e2 rename some exporter methods and complete exporter tests refactoring 2009-08-20 14:09:24 -03:00
Pablo Hoffman 3152e66fec deprecate domain_open signal and handle stats domain open/close directly from the engine 2009-08-20 10:54:58 -03:00
Pablo Hoffman 33b53c59d5 moved scrapy.xpath to scrapy.selector 
--HG--
rename : scrapy/xpath/__init__.py => scrapy/selector/__init__.py
rename : scrapy/xpath/document.py => scrapy/selector/document.py
rename : scrapy/xpath/factories.py => scrapy/selector/factories.py
 2009-08-19 21:50:52 -03:00
Pablo Hoffman dcc90fc196 declared loaders api stable and updated example project to use them 
--HG--
rename : docs/experimental/loaders.rst => docs/topics/loaders.rst
 2009-08-19 21:39:59 -03:00
Pablo Hoffman e8504a054c moved scrapy.newitem to scrapy.item and declared newitem api officially stable. updated docs and example project. deprecated old ScrapedItem 2009-08-19 21:39:58 -03:00
Ismael Carnales 314e8dea43 added new item exporter tests, introduced some api changes 2009-08-19 19:05:33 -03:00
Pablo Hoffman 907fad6da8 make sure input processors always receive iterables as input 2009-08-19 16:49:49 -03:00
Pablo Hoffman 741a6d7863 item exporters refactoring 
--HG--
rename : scrapy/contrib/exporter/jsonexporter.py => scrapy/contrib/exporter/jsonlines.py
 2009-08-19 13:09:39 -03:00
Pablo Hoffman 82e4b6adcf merge with ismael repo 2009-08-18 15:38:20 -03:00
Pablo Hoffman e01d31c498 another improvement to doc navbar 2009-08-18 15:35:53 -03:00
Ismael Carnales be4226cec5 merge 2009-08-18 15:21:39 -03:00
Ismael Carnales f88fb27851 fixed error in xpath selectors doc 2009-08-18 15:18:49 -03:00
Ismael Carnales 67c0c6a9e4 corrected indentation in xpath selectors doc 2009-08-18 15:13:23 -03:00
Pablo Hoffman ff837e5a45 doc: improved top navbar 2009-08-18 15:12:44 -03:00
Ismael Carnales 428dfe0d4a corrected the style of spiders documentation 2009-08-18 15:06:33 -03:00
Pablo Hoffman 0192282d07 reorganized doc and moved robotstxt doc inside downloader middlewares doc 2009-08-18 14:36:18 -03:00
Ismael Carnales 33089d287d merged topics and reference doc 2009-08-18 14:05:15 -03:00
Pablo Hoffman 7f30461410 added support for defining EXTENSIONS setting using dicts, like middleware settings 
--HG--
rename : scrapy/tests/test_utils_middleware.py => scrapy/tests/test_utils_conf.py
rename : scrapy/utils/middleware.py => scrapy/utils/conf.py
 2009-08-18 11:05:36 -03:00
Ismael Carnales 1bbe7991dc added documentation for ImagesPipeline 2009-08-18 09:35:32 -03:00
Ismael Carnales 48b40bd620 renamed x method of selectors to select 2009-08-17 15:58:06 -03:00
Pablo Hoffman 59e0a83ad4 removed more obsolete adaptors code 2009-08-17 14:48:11 -03:00
Ismael Carnales 8c2f62ba9c added Item Exporters documentation 2009-08-14 09:16:29 -03:00
Pablo Hoffman 02b01c3d2b loaders doc: fixed outdated line 2009-08-13 23:24:24 -03:00
Daniel Grana 6a74b51371 remove dupes words in loaders doc, and unused import in example 
--HG--
extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e
 2009-08-13 23:23:08 -03:00
Ismael Carnales ecca7f6478 fixes to Item Loader doc 2009-08-13 13:30:50 -03:00
Pablo Hoffman 8131d674d4 fixed outdated documentation (refs #97) 2009-08-13 09:24:41 -03:00
Pablo Hoffman 5aeab5b291 converted scrapy.item package to module 
--HG--
rename : scrapy/item/models.py => scrapy/item.py
 2009-08-12 21:31:50 -03:00
Pablo Hoffman 991afdf6b3 some minor fixes to loaders doc 2009-08-12 19:23:04 -03:00
Pablo Hoffman e6dd4d0955 renamed ApplyConcat processor to MapCompose 2009-08-12 18:43:08 -03:00
Pablo Hoffman 4e74f32401 renamed Pipe processor to Compose and documented it 2009-08-12 18:09:36 -03:00
Pablo Hoffman 958c3d9940 fixed some links to item loaders doc 2009-08-12 17:42:01 -03:00
Pablo Hoffman d94b633565 renamed ItemLoader method populate_item() to load_item() 2009-08-12 17:40:27 -03:00
Pablo Hoffman 1dc592882b Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests 
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
 2009-08-12 16:49:07 -03:00
Pablo Hoffman 7cbbc3ffb0 Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated. 
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
 2009-08-12 16:49:05 -03:00
Pablo Hoffman d4aa72d2bb removed obsolete documentation about Robust Scraped Item and Adaptors 2009-08-12 13:50:15 -03:00
Pablo Hoffman e69311ef38 added missing text to new stats collector methods 2009-08-11 16:37:52 -03:00
Daniel Grana 45cbe1333f fix typo in stats docs 2009-08-11 16:30:04 -03:00
Daniel Grana 8f16e7f9fa remove `default` parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector 2009-08-11 16:23:23 -03:00
Daniel Grana cd8eed77c5 stats collector gains two new methods to store values only if greater/lower than previous value 2009-08-11 15:46:44 -03:00
Ismael Carnales 3614998053 fixed error in doc 2009-08-11 09:23:45 -03:00
Pablo Hoffman 7ac7597950 removed obsolete scrapy.contrib.cluster 2009-08-10 21:02:46 -03:00
Pablo Hoffman c0532712f9 removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support 2009-08-10 21:02:24 -03:00
Pablo Hoffman d334c035c0 removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references 2009-08-10 20:52:43 -03:00
Pablo Hoffman 49bab4777b XPathSelector: added 're' argument to add_xpath method, exposed selector attribute 2009-08-10 19:42:20 -03:00
Pablo Hoffman 50fcc67278 improved reducers examples 2009-08-10 10:13:30 -03:00
Pablo Hoffman 5be0cee577 added TreeExpander example 2009-08-09 20:54:22 -03:00
Pablo Hoffman 38f82e3993 loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements 2009-08-09 18:06:12 -03:00
Pablo Hoffman b296d4169e minor doc update for making it more windows-friendly 2009-08-09 17:08:42 -03:00
Pablo Hoffman 91eea82eef added XPathLoader for working with XPath Selectors more conveniently 2009-08-08 07:26:20 -03:00
Pablo Hoffman db90e26a8b renamed ItemLoader class to Loader 2009-08-07 14:39:30 -03:00
Pablo Hoffman e585c6cac4 relocated experimental newitems/loaders doc, and added example for extending fields metadata 
--HG--
rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst
rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst
 2009-08-07 14:28:58 -03:00
Pablo Hoffman d95e99f585 Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation 
--HG--
rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst
 2009-08-07 03:50:09 -03:00
Ismael Carnales 32894643a0 added ListField documentation, ordered field reference alphabetically 2009-08-03 15:00:04 -03:00
Pablo Hoffman c3427e075c added domain_stats parameter to stats_domain_closed signal 2009-07-31 16:36:35 -03:00
Pablo Hoffman 73172b244d added from_unicode_list() method to Field objects 2009-07-30 16:58:24 -03:00
Pablo Hoffman 8ec7c9e01c WEBCONSOLE_PORT setting now defaults to 6080 2009-07-29 18:59:34 -03:00
Pablo Hoffman fcc91901eb finished cleaning up closedomain documentation, and updated default settings 2009-07-27 15:42:35 -03:00
Pablo Hoffman 09ba6927d7 Some changes to CloseDomain extension: 
- added support for closing by item passed count (CLOSEDOMAIN_ITEMPASSED)
- removed support for sending notification emails (since that's the job of
  another extension)
 2009-07-27 15:23:50 -03:00
Pablo Hoffman 9da66698f3 moved httprepr() method (from Request and Response objects) to scrapy.utils functions 2009-07-25 18:56:12 -03:00
Pablo Hoffman c615ace6bd doc: updated google directory links in firebug guide 2009-07-24 13:14:36 -03:00
Daniel Grana 96c3cdbec2 improve OffsiteMiddleware reference docs 
--HG--
extra : rebase_source : 3ed3f23fc1ec63b521ead029c5749898f3ab05d7
 2009-07-24 12:59:38 -03:00
Pablo Hoffman 38c3f7d0b4 Some changes to logging of scraped items: 
1. "Scraped Item" log level changed to DEBUG
2. "Dropped Item" log level changed to WARNING
3. added "Passed Item" log message with INFO level
 2009-07-23 11:49:48 -03:00
Pablo Hoffman e43e28bf1d minimal doc improvement 2009-07-23 09:12:49 -03:00
Ismael Carnales 6d24ae5920 added reference to working with relative xpaths in the tutorial 2009-07-23 09:05:14 -03:00
Pablo Hoffman 9baa6bb2a8 minor selectors doc fix 2009-07-23 01:56:56 -03:00
Ismael Carnales 3c4afb23be moved newitem from scrapy.contrib_exp to scrapy.newitem 2009-07-22 15:13:36 -03:00
Ismael Carnales 202894dd8f fixes to newitem doc 2009-07-22 10:25:22 -03:00
Pablo Hoffman 7d8ba0542c fixed typo in doc 2009-07-21 17:38:46 -03:00
Pablo Hoffman aa345e116a Added spider middleware documentation 2009-07-21 17:19:19 -03:00
Ismael Carnales 90f1d9e489 Added Scheduler middleware reference documentation 2009-07-21 16:52:27 -03:00
Ismael Carnales 00d9ce6608 fixes to newitem doc 2009-07-21 12:22:49 -03:00
Ismael Carnales 894a5d81d3 changed the newitem API to a dict-like interface 2009-07-21 12:20:49 -03:00
Pablo Hoffman 9867e7b2d6 removed references to obsolete ENABLED_SPIDERS_FILE setting 2009-07-21 01:11:48 -03:00
Pablo Hoffman 92b746d866 SimpledbStatsCollector: moved domain creation to constructor 2009-07-17 12:49:53 -03:00
Pablo Hoffman 292757f312 added link to architecture overview and fixed old link 2009-07-16 19:15:19 -03:00
Pablo Hoffman 136014d718 Added section about relative xpaths to XPathSelectors doc 2009-07-16 17:29:29 -03:00