RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,590 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

1009 Commits

Author SHA1 Message Date
Pablo Hoffman 9686f97242 added precise to supported ubuntu distros 2012-05-12 19:54:36 -03:00
Pablo Hoffman 58e88ed246 scrapyd: do not set SCRAPY_FEED_URI/SCRAPY_LOG_FILE if items_dir/logs_dir settings are not set 2012-05-08 17:43:00 -03:00
Pablo Hoffman 9c3b9f2968 fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py 2012-05-03 12:05:40 -03:00
Pablo Hoffman abcac4fcbd updated maintainer to scrapinghub 2012-05-02 03:25:35 -03:00
stav 86dba76d1f documentation indentation 2012-04-30 13:09:34 -05:00
Pablo Hoffman c7f33c534b removed redundant lines from release notes 2012-04-20 09:09:55 -03:00
Daniel Graña 15c8c01828 add 0.14.3 release notes 2012-04-19 23:25:15 -03:00
Pablo Hoffman 97f362d64e removed deprecated/undocumented class: HTMLImageLinkExtractor 2012-04-19 12:07:46 -03:00
Pablo Hoffman 2fb5e62c39 doc: update overview page to point to the genspider command. refs #107 2012-04-19 02:37:22 -03:00
Pablo Hoffman d567d8efbe added note to docs/topics/firebug.rst about google directory being shut down 2012-04-19 01:34:20 -03:00
Pablo Hoffman 30ddbf624e mention about some scrapy.xlib modules removed in the release notes 2012-04-17 12:31:18 -03:00
Pablo Hoffman d99ee6deb9 added some missing entries to release notes 2012-04-17 12:29:48 -03:00
Daniel Graña d8ebf16fe5 Merge pull request #114 from stav/master 
Scrapy DOC changes
 2012-04-11 12:08:45 -07:00
Pablo Hoffman 7cca916ed5 added release notes to official documentation, including all release notes since Scrapy 0.7 2012-04-11 15:53:23 -03:00
stav f1802289cd small doc typo change to get the fork rolling 2012-04-11 12:05:39 -05:00
Pablo Hoffman 4f28ffcb2c removed no longer needed dependency on simplejson 2012-04-10 16:01:36 -03:00
Pablo Hoffman 6e8edbd72e switched default selectors backend to lxml 2012-04-10 15:52:14 -03:00
Pablo Hoffman 27018fced7 changed default user agent to Scrapy/0.15 (+http://scrapy.org) and removed no longer needed BOT_VERSION setting 2012-03-23 13:45:21 -03:00
Pablo Hoffman 8933e2f2be added REFERER_ENABLED setting, to control referer middleware 2012-03-22 16:35:14 -03:00
Jason Yeo da826aa13d fixed minor mistake in Request objects documentation 2012-03-21 10:25:41 +08:00
Pablo Hoffman 175c70ad44 fixed minor defect in link extractors documentation 2012-03-20 22:56:45 -03:00
Pablo Hoffman 35fb01156e removed some obsolete remaining code related to sqlite support in scrapy 2012-03-16 11:55:55 -03:00
Pablo Hoffman b6ae266546 Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman 2b16ebdc11 added minor clarification on cookiejar request meta key usage 2012-02-29 07:19:01 -02:00
lostsnow 5afe4f50c1 scrapyd: support bind to a specific ip address 2012-02-29 13:47:40 +08:00
Pablo Hoffman 81abb45000 fixed bug in new cookiejar documentation 2012-02-28 11:08:25 -02:00
Pablo Hoffman 26c8004125 added documentation for the new cookiejar Request.meta key 2012-02-27 19:58:58 -02:00
Pablo Hoffman 7fe7c3f3b1 MemoryUsage extension: close the spiders (instead of stopping the engine) when the limit is exceeded, providing a descriptive reason for the close. Also fixed default value of MEMUSAGE_ENABLED setting to match the documentation. 2012-02-23 17:05:06 -02:00
Pablo Hoffman 7b8942a648 updated StackTraceDump extension doc 2012-02-16 15:14:17 -02:00
Pablo Hoffman ea77342b55 updated versioning doc according to recent changes 2012-01-05 11:50:28 -02:00
Pablo Hoffman 0b0bce7f3c scrapyd: added cancel.json and listjobs.json api methods to documentation 2012-01-05 11:23:25 -02:00
Pablo Hoffman 8f42633a94 scrapyd: added clarification about how to disable items feeds generation 2012-01-05 11:20:50 -02:00
Pablo Hoffman dbda33efa6 scrapyd: added support for storing items by default 
Items are stored the same way as logs, in jsonlines format.

Also renamed logs_to_keep setting to jobs_to_keep.
 2012-01-03 23:08:54 -02:00
Pablo Hoffman 0be421fbf0 fixed reference to tutorial directory 2011-12-23 18:57:11 -02:00
Pablo Hoffman 41fd3c4f6c doc: removed duplicated callback argument from Request.replace() 2011-12-23 15:55:46 -02:00
Pablo Hoffman 0eeff76227 fixed formatting of scrapyd doc 2011-12-20 03:18:37 -02:00
Daniel Graña bcb31988f2 change tutorial to follow changes on dmoz site 2011-12-14 13:03:31 -02:00
Pablo Hoffman 992af8d38f ubuntu repos: added support for oneiric release 2011-10-25 14:26:38 -02:00
Pablo Hoffman c38c49d56a fixed PickeItemExporter bug, added unittest, and added pickle to suported feed exports formats 2011-10-25 02:36:51 -02:00
Pablo Hoffman 8bdf288428 made scrapyd doc more version agnostic 2011-10-23 05:29:54 -02:00
Pablo Hoffman ade5efdc61 added -o option to scrapy crawl, a convenient shortcut for using feed exports 2011-10-22 20:53:49 -02:00
Pablo Hoffman 431441cb52 updated documentation to remove references to old issue tracker and mercurial repos 2011-09-25 13:06:24 -03:00
Pablo Hoffman ce03ccd4ec updated documentation about DEPTH_PRIORITY and DFO/BFO crawls 2011-09-23 13:22:25 -03:00
Julien Duponchelle b7c436343a scrapy deploy support git version 2011-09-21 22:17:08 +02:00
Pablo Hoffman ab1c9cfc56 removed documentation header notifying about other documentation versions, as that's provided by readthedocs already 2011-09-14 02:39:32 -03:00
Daniel Grana 5f1b1c05f8 Do not filter requests with dont_filter attribute set in OffsiteMiddleware 2011-09-08 15:18:10 -03:00
Pablo Hoffman bff3d31469 scrapyd: updated schedule.json response format 2011-09-04 09:29:24 -03:00
Pablo Hoffman a1dbc62b45 removed CONCURRENT_SPIDERS setting (use scrapyd maxproc instead) 2011-09-02 18:27:39 -03:00
Pablo Hoffman 40f7075f11 added initial documentation about suspend and resume crawls 2011-09-02 13:12:27 -03:00
Pablo Hoffman 27dd68a690 added SpiderState extension 2011-09-02 13:06:59 -03:00
Pablo Hoffman 6a31ab667d minor fix to doc 2011-09-01 15:08:23 -03:00
Pablo Hoffman d98b058c21 no longer recommend using labmda's in the doc, as they're not friendly with scheduler persistence 2011-09-01 15:06:49 -03:00
Pablo Hoffman 76af0cdd44 updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
Pablo Hoffman 98b68ca89d scrapyd: documented support for passing setting to spiders in schedule.json 2011-08-27 01:31:12 -03:00
Pablo Hoffman 5c6b0631e2 minor doc fix 2011-08-19 11:42:03 -03:00
Pablo Hoffman 9d97e73a24 fixed priority handling on the new scheduler so that it's backwards compatible (ie. bigger priorities are higher). also fixed a few documentation bugs related to requests priority 2011-08-19 08:26:41 -03:00
Pablo Hoffman a3697421c0 some minor updates to documentation 2011-08-11 09:19:59 -03:00
Pablo Hoffman 5da6ffb57b Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-08-11 09:11:19 -03:00
Pablo Hoffman bc2d2183e9 fixed import in doc 2011-08-11 09:11:08 -03:00
Pablo Hoffman 19e6da59d8 added new downloader middleware: ChunkedTransferMiddleware 2011-08-09 03:03:25 -03:00
Pablo Hoffman 984be35461 Some telnet console changes: 
* renamed manager alias to crawler
* added aliases: spider, slot
* fixed est() function
 2011-08-08 15:01:08 -03:00
Pablo Hoffman f7c0aeccc6 added note about engine_started signal 2011-08-07 03:57:09 -03:00
Pablo Hoffman 9f60c27612 added setting to support disabling DNS cache: DNSCACHE_ENABLED 2011-08-05 20:41:59 -03:00
Pablo Hoffman cb95d7a5af added marshal to formats supported by feed exports 2011-08-03 16:16:48 -03:00
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 2011-08-02 11:57:55 -03:00
Pablo Hoffman f354a49d0f added FAQ about preventing bots getting banned 2011-07-28 00:40:30 -03:00
Pablo Hoffman ce7a787970 Big downloader refactoring to support real concurrency limits per domain/ip, 
instead of global limits per spider which were a bit useless.

This removes the setting CONCURRENT_REQUESTS_PER_SPIDER and adds thre new
settings:

* CONCURRENT_REQUESTS
* CONCURRENT_REQUESTS_PER_DOMAIN
* CONCURRENT_REQUESTS_PER_IP (overrides per domain)

The AutoThrottle extension had to be disabled, but will be ported and
re-enabled soon.
 2011-07-27 13:38:09 -03:00
Pablo Hoffman c59340150f Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled. 2011-07-27 03:45:15 -03:00
Pablo Hoffman 2ac08a713d downloader: renamed SpiderInfo to Slot, for consistency with engine and scraper names 2011-07-22 02:06:10 -03:00
Pablo Hoffman 0e008268e1 removed SimpledbStatsCollector from scrapy code, it was moved to https://github.com/scrapinghub/scaws 2011-07-20 10:38:16 -03:00
Pablo Hoffman b6b0a54d9f removed FAQ entry 2011-07-20 01:31:36 -03:00
Pablo Hoffman e3f640c7bf added FAQ entry about scrapy deploy issue on Mac + Python 2.5 2011-07-19 19:53:32 -03:00
Pablo Hoffman 84f518fc5e More core changes: 
* removed execution queue (replaced by newer spider queues)
* added real support for returning iterators in Spider.start_requests()
* removed support for passing urls to 'scrapy crawl' command
 2011-07-15 15:18:39 -03:00
Pablo Hoffman dbad1373f1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-07-13 18:44:54 -03:00
Pablo Hoffman 18cb4ff1d8 added natty to list of supporte ubuntu distros 2011-07-13 18:43:52 -03:00
Pablo Hoffman 39a2ea97c8 redirect mw: added REDIRECT_ENABLED setting and documented the other settings 2011-07-13 14:18:15 -03:00
Pablo Hoffman 541ed3913b retry middleware: added RETRY_ENABLED setting and documented the other settings more properly, also improved messages when no longer retrying requests 2011-07-13 11:55:05 -03:00
Pablo Hoffman 763f3dc628 minor update to doc 2011-07-12 19:56:39 -03:00
Pablo Hoffman bfda9ec319 added clarification about scrapy versioning including the recently adopted odd/even versioning scheme 
--HG--
rename : docs/api-stability.rst => docs/versioning.rst
 2011-07-12 19:53:23 -03:00
Pablo Hoffman 4fde1ef94d added CloseSpider exception, to manually close spiders 2011-07-12 14:24:10 -03:00
Pablo Hoffman db5cae7c03 SitemapSpider: added support for filtering which sitemaps to follow (patch contributed by Rolando Espinoza). closes #330 2011-06-23 18:18:29 -03:00
Pablo Hoffman 57c43fdce6 added SitemapSpider, with tests and doc 2011-06-15 11:54:34 -03:00
Pablo Hoffman 91dc46539f added LogStats extension for periodically logging basic stats (like crawled pages and scraped items) 2011-06-14 00:50:05 -03:00
Pablo Hoffman 841e9913db renamed CLOSESPIDER_ITEMPASSED setting to CLOSESPIDER_ITEMCOUNT, to follow the refactoring done in r2630 2011-06-13 16:58:51 -03:00
Pablo Hoffman 474cba512c simplified MemoryDebugger extension to use stats for dumping memory debugging info 2011-06-06 03:13:28 -03:00
Pablo Hoffman 5fbc32c015 call stats collector engine_stopped() after the engine is closed (to make sure all data from extensions has been collected), and added that method to documented api 2011-06-06 03:12:40 -03:00
Pablo Hoffman 9d9c8877da added 'scrapy edit' command 2011-06-05 22:02:56 -03:00
Pablo Hoffman 03ae481cad removed experimental crawlspider v2 2011-06-03 18:23:23 -03:00
Pablo Hoffman 5bf733b6f6 Changed default representation of items to pretty-printed dicts. This improves 
default logging by making log more readable in the default case, for both Scraped and Dropped lines.

Projects can still customize how items are represented by overriding the item's __str__ method, as usual.
 2011-06-03 01:13:01 -03:00
Pablo Hoffman 1bc2339bb8 Merged item passed and item scraped concepts, as they have often proved 
confusing in the past.

This means:

* original item_scraped signal was removed
* original item_passed signal was renamed to item_scraped
* old log lines "Scraped Item..." removed
* old log lines "Passed Item..." renamed to "Scraped Item..."
 2011-06-03 01:13:00 -03:00
Pablo Hoffman e6091df551 fixed doc typo 2011-05-30 09:04:31 -03:00
Pablo Hoffman 1d98fc8fb5 added spider_error signal 2011-05-29 22:38:17 -03:00
Pablo Hoffman 2fa0f75f2d added COOKIES_ENABLED setting to support disabling the cookies middleware 2011-05-27 00:35:34 -03:00
Pablo Hoffman d72d3f4607 stack trace dump extension: also dump engine status, and support triggering it with SIGQUIT, besides SIGUSR2 2011-05-20 03:25:00 -03:00
Pablo Hoffman 951ba507f9 Removed support for default values in Scrapy items, which have proven confusing in the past 2011-05-19 21:42:46 -03:00
Pablo Hoffman 503f302010 removed remaining references to scheduler middleware from doc, as it will be removed on next release 2011-05-18 19:48:48 -03:00
Pablo Hoffman 3fd17432cf fixed outdated documentation 2011-05-18 14:46:20 -03:00
Pablo Hoffman 9016e7e993 added role to link to scrapy source code (not yet used) 2011-05-18 14:43:34 -03:00
Pablo Hoffman cd85c12c33 Some Link extractor improvements: 
* added support for ignoring common file extensions that are not followed if
  they occur in links
* fixed link extractor documentation issues
* slighly improved performance of applying filters
* added link to link extractors doc from documentation index
 2011-05-18 12:32:34 -03:00
Pablo Hoffman 495152bd50 disabled verbose depth stats collection by default, added DEPTH_STATS_VERBOSE setting to enable it 2011-05-18 11:04:48 -03:00
Pablo Hoffman accb6ed830 dump stats to log by default (ie. change default value of STATS_DUMP to True) 2011-05-17 22:42:05 -03:00
Pablo Hoffman 7f97259ba7 added w3lib to requirements, in installation guide 2011-05-01 11:14:57 -03:00
Pablo Hoffman 4a83167698 fixed small doc typo 2011-04-30 01:35:30 -03:00
Pablo Hoffman bb2b67c862 updated tutorial to use 'dmoz' as the name of the spider instead of 'dmoz.org', so that it's more similar to the dirbot example project 2011-04-28 09:31:57 -03:00
Pablo Hoffman bf73002428 removed googledir example, replaced by dirbot project on github. updated docs accordingly 2011-04-28 02:28:39 -03:00
Pablo Hoffman b12dd76bb8 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-04-25 09:31:18 -03:00
Pablo Hoffman 678f08bc1b added warning about using 'parse' as callback in crawl spider rules 2011-04-25 09:30:42 -03:00
Pablo Hoffman ad496eb3b6 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-04-14 12:36:27 -03:00
Pablo Hoffman ecb4f44cbc Added clarification on how to work with local settings and scrapy deploy 2011-04-14 12:36:09 -03:00
Pablo Hoffman 3ee2c94e93 Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it 2011-04-06 14:54:48 -03:00
Pablo Hoffman 8a5c08a6bc added join_multivalued parameter to CsvItemExporter 2011-03-24 13:15:52 -03:00
Pablo Hoffman 3954e600ca added DBM storage backend for HTTP cache 2011-03-23 21:32:02 -03:00
Pablo Hoffman cfd11df539 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-24 15:28:57 -02:00
Pablo Hoffman 8f7e163b04 Fixed wrong method name in downloader middleware documentation 2011-02-24 15:26:32 -02:00
Daniel Grana c55355642c fix FAQ typos reported by marlun_ at #scrapy IRC channel 2011-02-16 08:57:42 -02:00
Pablo Hoffman 1fb55bdaf0 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-15 07:25:12 -02:00
Pablo Hoffman 16d9a33951 added FAQ entry about working with big data feeds 2011-02-15 07:24:52 -02:00
Pablo Hoffman 936353d5f1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-09 11:20:46 -02:00
Pablo Hoffman 181d1c09ae Fixed typo and code indentation in the doc. Closes #307 and #308 2011-02-09 11:19:46 -02:00
Pablo Hoffman c91f0d9ea1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-04 13:39:54 -02:00
Pablo Hoffman c5499ead73 Clarified behaviour when multiple rules match the same link in CrawlSpider 2011-02-04 13:39:12 -02:00
Pablo Hoffman d7f193cbea bumped version to 0.13 in documentation 2011-01-02 17:29:43 -02:00
Pablo Hoffman b56e933be9 bumped version to 0.12 in documentation 2011-01-02 17:28:33 -02:00
Pablo Hoffman 5879389ad0 Bumped version to 0.12 2011-01-02 16:16:40 -02:00
Pablo Hoffman fa644f7a5e Some simplifications to Scrapyd architecture and internals: 
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
  get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
  doesn't necesarilly run eggs (though it does in the default case)

--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
 2010-12-27 16:22:32 -02:00
Pablo Hoffman 633ebc4c43 minor indentation improvement 2010-12-23 13:04:49 -02:00
Pablo Hoffman db07a9a938 Added notice to documentation, pointing dev to stable versions and viceversa 2010-12-23 13:03:40 -02:00
Pablo Hoffman 544308d6d0 updated ubuntu repos doc, in preparation for the 0.11 release 2010-12-21 11:02:56 -02:00
Pablo Hoffman 002abf204f Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273 2010-12-13 14:05:47 -02:00
Pablo Hoffman f984d438a0 updated docs to use scrapy version on aptitude install lines 2010-12-13 14:02:42 -02:00
Pablo Hoffman 119fd20e91 Added verbose option to 'version' command. Closes #298 2010-12-13 00:32:44 -02:00
Pablo Hoffman 6a1b69c93f renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands 
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
 2010-11-30 20:23:27 -02:00
Pablo Hoffman df54ed0041 Some Scrapyd enhancements: 
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
 2010-11-30 02:26:31 -02:00
Pablo Hoffman bbffa59497 Some changes to Scrapyd: 
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
 2010-11-29 17:19:05 -02:00
Pablo Hoffman 2557777c39 Updated doc referring to HTTP cache middleware 2010-11-24 13:27:44 -02:00
Pablo Hoffman 426b6fa100 docs/intro/install.rst: added -U flag to easy_install command 2010-11-22 13:50:19 -02:00
Pablo Hoffman 91a7c25797 * Made Response.meta attribute map to Request.meta attribute. Closes #290 
* Record redirected URLs in redirect middleware. Closes #291
 2010-11-18 12:51:54 -02:00
Pablo Hoffman ac007802d6 Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280 2010-11-17 21:32:23 -02:00
Pablo Hoffman 5a5364d0c1 Updated documentation to point out that simplejson is now required if using Python 2.5, and to recommended switching to Python 2.6 2010-11-16 03:31:04 -02:00
Pablo Hoffman d988ca1ec2 Some changes to scrapy deploy command: 
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options

* updated documentation accordingly
 2010-11-08 17:01:06 -02:00
Pablo Hoffman 0f69e7a191 Some changes to HTTP Cache middleware: 
* made it use the project data storage by default (closes #279)
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
  dropped support for negative numbers
* other minor doc improvements
 2010-11-01 02:38:15 -02:00
Pablo Hoffman 3c94c6cb9b fixed sphinx doc id 2010-11-01 02:31:20 -02:00
dfdeshom 130276605b Bind the web server and telnet server to a configurable interface (WEBSERVICE_HOST). The default is to bind to all interfaces. Also add documentation for WEBSERVICE_HOST and TELNETCONSOLE_HOST. 2010-11-01 00:59:04 -02:00
Pablo Hoffman b76c5c597f * Added support for project data storage (closes #276) 
* Documented project file structure
* Moved default location of SQLite database to project data storage dir (closes #277)
 2010-10-31 03:25:37 -02:00
Pablo Hoffman dfa6745e91 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 16:05:53 -02:00
Pablo Hoffman a0d9b43031 fixed typo in scrapyd doc 2010-10-30 16:05:32 -02:00
Pablo Hoffman d67152ab0f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 01:56:12 -02:00
Pablo Hoffman 75451cbe84 scrapyd doc: fixed delversion.json example 2010-10-30 01:56:00 -02:00
Pablo Hoffman a59bfb539d * Added lxml backend for XPath selectors. Closes #147 
* Added new setting (SELECTORS_BACKEND) to choose which backend to use
* Deprecated the extract_unquoted() function from selectors
* Made libxml2 optional by adding a dummy selector backend. Closes #260

--HG--
rename : scrapy/tests/test_selector.py => scrapy/tests/test_selector_libxml2.py
 2010-10-25 14:47:10 -02:00
Pablo Hoffman 6c921896a5 Expanded documentation on deploy command and versions. Refs #261 2010-10-19 00:11:45 -02:00
Pablo Hoffman 1d567cdce6 Added new 'deploy' command. Closes #261 2010-10-18 22:38:46 -02:00
Pablo Hoffman 7d8f922df9 Added documentation for CLOSESPIDER_ERRORCOUNT setting. Refs #254 2010-10-18 22:36:30 -02:00
Pablo Hoffman c96f17c43d Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-18 03:21:21 -02:00
Pablo Hoffman 98662e53ea Formatting fix in Scrapyd doc 2010-10-17 03:20:23 -02:00
Pablo Hoffman a3d85da96f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-16 19:54:24 -02:00
Pablo Hoffman 5f65c26080 Some minor improvements to feature list in Scrapy at a Glance documentation page 2010-10-16 19:02:08 -02:00
Pablo Hoffman d5c8caf07b Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-10 20:31:38 -02:00
Pablo Hoffman b4fbc6c5fa Updated Scrapy Tutorial to reference feed exports, instead a custom written pipeline, and extended item pipeline documentation to include a JSON writer. 2010-10-10 20:31:05 -02:00
Pablo Hoffman aa4142e4ba Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-07 18:23:48 -02:00
Pablo Hoffman f4accb6c7f Updated dmoz xpaths of Scrapy tutorial 2010-10-07 18:22:01 -02:00
Pablo Hoffman 7826869cb2 Added missing colon 2010-09-28 16:44:53 -03:00
Martin Santos 0bf9e4627c added support to CloseSpider extension, for close the spider after N pages have been crawled. Using the CLOSESPIDER_PAGECOUNT setting. closes #253 2010-09-28 16:29:37 -03:00
Pablo Hoffman 279dcc245f Fixed role name in Sphinx doc 2010-09-26 01:01:06 -03:00
Pablo Hoffman 9599bde3e9 Removed RequestLimitMiddleware 2010-09-22 16:09:13 -03:00
Pablo Hoffman ed4aec187f Ported code to use new unified access to spider settings, keeping backwards compatibility for old spider attributes. Refs #245 2010-09-22 16:09:13 -03:00
Pablo Hoffman b6c2b55e5b Splitted settings classes from settings singleton. Closes #244 
--HG--
rename : scrapy/conf/__init__.py => scrapy/conf.py
rename : scrapy/conf/default_settings.py => scrapy/settings/default_settings.py
rename : scrapy/tests/test_conf.py => scrapy/tests/test_settings.py
 2010-09-22 15:47:33 -03:00
Pablo Hoffman 2ebfa7e68d Removed unneeded code (since autodoc is not used in Sphinx doc) 2010-09-22 10:52:02 -03:00
Shuaib 9288f622f9 Added formname parameter for FormRequest.from_response 2010-09-20 08:33:24 -03:00
Pablo Hoffman bf467fc37a Check 'dont_merge_cookies' membership in request.meta, instead of getting its value 2010-09-10 15:29:15 -03:00
Pablo Hoffman 7d14a52234 Reference dont_merge_cookies in list of special Request.meta keys 2010-09-09 21:54:26 -03:00
Pablo Hoffman 7f21a6384f Documented handle_httpstatus_list request.meta key 2010-09-09 21:50:40 -03:00
Pablo Hoffman f1c943543a Added dont_retry request.meta key to make RetryMiddleware ignore requests. Closes #234 2010-09-09 21:43:44 -03:00
Pablo Hoffman 9f01e3e79e Added dont_redirect request.meta key to make RedirectMiddleware ignore requests. Closes #233 2010-09-09 21:37:35 -03:00
Pablo Hoffman 7da79b90fe Make url/body attributes of Request/Response objects read-only - use replace() to change them. Deprecation warning left for backwards compatibilty. 2010-09-08 00:15:11 -03:00
Pablo Hoffman c1aab2f58e Copy callback/errback attributes when copying Requests 2010-09-08 00:15:09 -03:00
Pablo Hoffman e9ebebb230 Removed UrlFilterMiddleware from scrapy.contrib - see this snippet for an alternative: http://snippets.scrapy.org/snippets/12/ 2010-09-07 17:51:02 -03:00
Daniel Grana 12b04b068f make download_timeout configurable by request. closes #229 
--HG--
extra : rebase_source : e57dfd4aeb98d48b04fc4d0c6469e9a85e4b33a8
 2010-09-07 13:01:40 -03:00
Pablo Hoffman 9158e9d682 Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options 
--HG--
rename : debian/scrapyd.cfg => debian/000-default
rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
 2010-09-07 09:17:25 -03:00
Daniel Grana 3414bf13ee remove request_uploaded signal and move response_received and response_downloaded to downloader manager. closes #228 
--HG--
extra : rebase_source : 4af0d2a01b34de8a21048bb7f4a66bfc484b3b8f
 2010-09-06 23:23:14 -03:00
Pablo Hoffman 3c5ab10688 Added FAQ entry about __VIEWSTATE parameter 2010-09-06 13:17:08 -03:00
Pablo Hoffman e3d67d74f7 docs/intro/overview.rst: add example of scraped data and introduce loaders 2010-09-06 10:04:00 -03:00
Pablo Hoffman 00d55fbbd1 Updated 'Scrapy at a glance' document replacing item pipeline example by a simpler usage of feed exports 2010-09-05 23:38:37 -03:00
Pablo Hoffman 766f2d910d Renamed Request Handlers to Download Handlers 2010-09-05 19:35:53 -03:00
Pablo Hoffman a5cf71cb06 Updated Ubuntu package signing key location 2010-09-05 19:04:15 -03:00
Pablo Hoffman 6bf52fb50e Make telnet console and web service try a range of ports for binding, instead of just one. Closes #226 2010-09-05 06:48:08 -03:00
Pablo Hoffman 14e985b076 Updated Command line tool documentation 2010-09-05 05:29:58 -03:00
Pablo Hoffman 1190f97944 Updated settings documentation 2010-09-05 04:58:14 -03:00
Pablo Hoffman ebdb733e95 Updated some old messages in Scrapy shell doc 2010-09-05 04:45:43 -03:00
Pablo Hoffman 2f12618890 Post reference to Scrapyd in FAQ 2010-09-05 04:35:27 -03:00
Pablo Hoffman bf34094e5a Added versionadded:: notice to new documentation topics 2010-09-04 03:30:45 -03:00
Daniel Grana 9f4b1e47a4 damn, really fix httpcache docs 2010-09-04 03:26:41 -03:00
Daniel Grana 7ad901640b fix httpcache docs 2010-09-04 03:23:08 -03:00
Daniel Grana 1abaa79469 Make ignored schemes configurable in HttpCacheMiddleware. closes #224 
--HG--
extra : rebase_source : 2e6e8b93c642290f9bd6eb634eb4c8cd6da07c75
 2010-09-04 02:58:43 -03:00
Pablo Hoffman 7b9fa7fbaa Don't filter out requests coming from spiders that don't define allowed_domains. Closes #225 2010-09-04 02:23:04 -03:00
Pablo Hoffman 37e9c5d78e Added new Scrapy service with support for: 
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API

Documentation is added along with the code.

Closes #218.

--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
 2010-09-03 15:54:42 -03:00
Pablo Hoffman 758d21b2f9 Simplified images pipeline by allowing it to be used without having to override it in your project. Closes #217 2010-08-31 16:03:08 -03:00
Pablo Hoffman e7b3247a18 Updated some missing references to scrapy-ws script 2010-08-27 01:05:59 -03:00
Pablo Hoffman e2ed27e4fd Added documentation for Ubuntu packages. Refs #211 2010-08-23 21:28:32 -03:00
Pablo Hoffman 58b4cc2c32 Some minor fixes to contribution Contributing documentation 2010-08-23 00:23:14 -03:00
Pablo Hoffman 6585c1a28f removed (somewhat hacky) MAIL_DEBUG setting 2010-08-22 22:42:00 -03:00
Pablo Hoffman 6da1162839 minor fixes to FAQ 2010-08-22 19:08:45 -03:00
Pablo Hoffman b3753d34eb Added FAQ entry about feed exports 2010-08-22 05:59:30 -03:00
Pablo Hoffman cbfec4bb0e Renamed webservice ManagerResource to CrawlerResource 
--HG--
rename : scrapy/contrib/webservice/manager.py => scrapy/contrib/webservice/crawler.py
 2010-08-22 05:48:03 -03:00
Pablo Hoffman 7546a0805c Removed webservice Spiders and Extensions resources since they can now be accessed through the Execution Manager (aka. Crawler) resource 2010-08-22 05:38:46 -03:00
Pablo Hoffman c1225e0f45 "parse" command refactoring. This fixes #173 and renders #106 invalid. 2010-08-22 05:04:17 -03:00
Pablo Hoffman 9fccc11363 Moved scrapy.extension.extensions singleton to a "extensions" attribute of the scrapy.project.crawler singleton. Refs #189 2010-08-22 02:15:11 -03:00
Pablo Hoffman faf7a7da83 Moved scrapymanager singleton to scrapy.project module. Refs #189 
Detail of changes:

* Moved scrapy.core.manager.ExecutionManager class to scrapy.crawler.Crawler
* Added scrapy.project.crawler singleton to reference a singleton instance of
  Crawler class (previously known as scrapymanager)
* Left an alias scrapy.core.manager.scrapymanager to scrapy.project.crawler for
  backwards compatibility (to be removed in Scrapy 0.11)
 2010-08-22 02:10:53 -03:00
Pablo Hoffman 053d45e79f Splitted stats collector classes from stats collection facility (#204) 
* moved scrapy.stats.collector.__init__ module to scrapy.statscol
* moved scrapy.stats.collector.simpledb module to scrapy.contrib.statscol
* moved signals from scrapy.stats.signals to scrapy.signals
* moved scrapy/stats/__init__.py to scrapy/stats.py
* updated documentation and tests accordingly

--HG--
rename : scrapy/stats/collector/simpledb.py => scrapy/contrib/statscol.py
rename : scrapy/stats/__init__.py => scrapy/stats.py
rename : scrapy/stats/collector/__init__.py => scrapy/statscol.py
 2010-08-22 01:24:07 -03:00
Pablo Hoffman c276c48c91 Added settings to Scrapy shell variables 2010-08-21 05:10:06 -03:00
Pablo Hoffman 68f9fcffe8 genspider command refactoring. Also updated tests and doc 2010-08-21 04:46:48 -03:00
Pablo Hoffman 0da6132136 Made command-line too output more concise 2010-08-21 03:37:59 -03:00
Pablo Hoffman 50621b7ef3 Renamed command "start" to "runserver". Closes #209 
--HG--
rename : scrapy/commands/start.py => scrapy/commands/runserver.py
 2010-08-21 01:42:55 -03:00
Pablo Hoffman 9aefa242d5 Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
Pablo Hoffman f782245c5a Removed obsolete files 2010-08-21 01:24:39 -03:00
Pablo Hoffman 1d3b9e2ca8 Scrapy shell refactoring 2010-08-20 11:26:14 -03:00
Pablo Hoffman 7858244dca Scrapy shell: moved python console starting code to scrapy.utils.console and get rid of noisy console banners 2010-08-20 01:33:02 -03:00
Pablo Hoffman 30e2404d8f updated FAQ entry to recommend using higher download delays 2010-08-19 17:59:52 -03:00
Pablo Hoffman 3d8151bb26 Added FAQ entry about response code 999 2010-08-19 16:51:51 -03:00
Pablo Hoffman 2ff5a83b7a Added persistent execution queue (based on SQLite), and a new 'queue' command to control it. Closes #198 2010-08-19 02:55:52 -03:00
Pablo Hoffman 94ead94bf6 Improved documentation of Scrapy command-line tool 
--HG--
rename : docs/topics/cmdline.rst => docs/topics/commands.rst
 2010-08-19 00:04:52 -03:00
Pablo Hoffman 34554da201 Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199. Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows. 
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
 2010-08-18 19:48:32 -03:00
Pablo Hoffman a71521bfba Default per-command settings are now specified in the default_settings attribute of the command object. Closes #201 2010-08-17 18:30:13 -03:00
Pablo Hoffman ad3fd0afe8 fixed minor formatting issue with new feed exports doc 2010-08-17 14:37:59 -03:00
Pablo Hoffman e741a807d2 Added new Feed exports extension with documentation and storage tests. Closes #197. 
Also deprecated File export pipeline (to be removed in Scrapy 0.11).

Still need to add tests for FeedExport main extension code.
 2010-08-17 14:27:48 -03:00
Pablo Hoffman 3e3a66620b Added support for returning deferreds from (some) signal handlers. Closes #193 2010-08-14 21:10:37 -03:00
Pablo Hoffman 1df2c17b78 updated old documentation references 2010-08-12 20:45:11 -03:00
Pablo Hoffman 43d47e5d9b Some improvements to Item Pipeline (closes #195): 
* Made Item Pipeline Manager a subclass of scrapy.middleware.MiddlewareManager
* Added open_spider/close_spider methods with support for returning deferreds from them
* Inverted the process_item() arguments to be more friendly with deferred
  callbacks (backwards compatibility kept through arguments introspection)
* Updated documentation with new methods and process_item() arguments change
 2010-08-12 10:48:37 -03:00
Pablo Hoffman 9d38a99aa8 updated missing doc reference from previous commit 2010-08-10 17:47:04 -03:00
Pablo Hoffman 784722774b moved scrapy.core.signals to scrapy.signals, keeping backwards compatibility 2010-08-10 17:40:53 -03:00
Pablo Hoffman c359a34d7d moved scrapy.core.exceptions to scrapy.exceptions, keeping backwards compatibility 
--HG--
rename : scrapy/core/exceptions.py => scrapy/exceptions.py
 2010-08-10 17:36:48 -03:00
Pablo Hoffman b1c0280616 removed scheduler middleware doc, as scheduler middleware will be removed soon 2010-08-10 16:59:49 -03:00
Pablo Hoffman c7d9f6e270 Added JSON item exporter with doc and unittests (closes #192), and also: 
* put all json exporters in scrapy.contrib.exporters and deprecated
  scrapy.contrib.exporters.jsonlines to reduce module nesting
* use JSON exporter with EXPORT_FORMAT=json in file export pipeline
 2010-08-07 15:52:59 -03:00
Pablo Hoffman 49851d7f55 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-08-02 17:20:55 -03:00
Pablo Hoffman 6c68e4ce15 fixed documentation typo 2010-08-02 17:20:13 -03:00
Pablo Hoffman 453e7bf38c Scrapy logging refactoring (closes #188): 
* added Twisted log observer for Scrapy, with unittests
 * use numeric values from Python logging module for log levels
 * removed scrapy.log.exc() function - use scrapy.log.err() instead
 * removed logmessage_received signal - write a (twisted) log observer instead
 * dropped support for obsolete `domain` argument
 * dropped support for old setting names: LOGLEVEL, LOGFILE (replaced by LOG_LEVEL, LOG_FILE)
 * deprecated `component` argument
 2010-08-02 08:49:14 -03:00
Ismael Carnales e145ec686c Replaced default spider manager (TwistedPluginSpiderManger) with a simpler one that doesn't depend on Twisted Plugins infrastructure. 2010-07-30 17:30:32 -03:00
Pablo Hoffman e2290a5359 Some changes to Crawl spider: 
* added process_request attribute to rules
* removed docstrings, since it duplicates documentation
 2010-07-22 18:40:35 -03:00
Daniel Grana 3e013f564b update docs for defaultheaders middleware and change spider attribute to match global setting name 2010-07-16 16:17:08 -03:00
Daniel Grana 6883a99c1e Automated merge with ssh://hg.scrapy.org/scrapy-0.9 2010-07-16 14:56:00 -03:00
Pablo Hoffman b91d40ba78 Fixed grammar error in doc (patch by stav) - closes #176 2010-07-16 11:34:18 -03:00
Pablo Hoffman 90a04f0530 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-07-13 19:47:55 -03:00
Pablo Hoffman 9e37ec4230 fixed documentation typo (closes #151) 2010-07-13 19:03:02 -03:00
Ping Yin b3a65d3313 HTTPCACHE: Don't cache response with codes in HTTPCACHE_IGNORE_HTTP_CODES 2010-07-09 13:14:25 -03:00
Ismael Carnales 2571e1b7aa docs: Some DjangoItem docs improvements, closes #134. Thanks tn! 2010-06-27 09:09:54 -03:00
Pablo Hoffman 115e9f2162 Added FAQ entry about running Scrapy deployment. 2010-06-14 18:21:12 -03:00
Pablo Hoffman ede1df4b4f updated copyright year, and indentation space 2010-06-14 07:16:51 -03:00
Pablo Hoffman bd16d1cd48 Added SMTP-AUTH support to scrapy.mail (closes #149) 2010-06-13 17:14:46 -03:00
Pablo Hoffman 6a33d6c4d0 * Added Scrapy Web Service with documentation and tests. 
* Marked Web Console as deprecated.
* Removed Web Console documentation to discourage its use.
 2010-06-09 13:46:22 -03:00
Pablo Hoffman 73305b1eb3 Added support for Requests without callbacks (#166) - the Spider.parse() method 
is used in those cases.

Also removed Request.deferred attribute.
 2010-06-08 18:18:02 -03:00
Pablo Hoffman 38b5793152 Some changes to telnet console: 
* moved module from scrapy.management.telnet to scrapy.telnet (to minimize
  nested modules)
* added signal for updating telnet console variables (fixes #165)

--HG--
rename : scrapy/management/telnet.py => scrapy/telnet.py
 2010-06-02 17:49:18 -03:00
Pablo Hoffman 031eb1e5ed removed no longer used SpiderScheduler (obsoleted by ExecutionQueue) 2010-05-28 17:27:15 -03:00
Ismael Carnales a71dc295af Some mail improvements and tests. 
* Add mail_sent signal and use it in MailSender
* Add MAIL_DEBUG setting to not send mails when testing
* Add MailSender tests
 2010-05-28 16:51:47 -03:00
Ping Yin 6059221716 Compose: stop process on None value by default 
By doing this, we can use str.lower as a processor safely without
checking whether the given value is None.

By passing stop_on_none=False as keyword argument, this behaviour can be changed.

Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-04-08 10:59:47 +08:00
Ping Yin 15b879f845 ItemLoader: Update docs for {add,replace,get}_{value,xpath} 
Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-05-18 17:54:25 +08:00
Pablo Hoffman bfd9cb42e5 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-05-17 20:11:27 -03:00
Pablo Hoffman 076cdfd585 Added documentation about contributing to Scrapy 2010-05-17 20:10:46 -03:00
Pablo Hoffman 7a55158fed fixed documentation bug (thanks rhill for reporting) 2010-05-11 11:25:03 -03:00
Steven Almeroth 5d03405cac FormRequest.from_response doc fix. closes #155 
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
 2010-04-26 22:28:07 -03:00
Pablo Hoffman 2121a30c74 added note about installing Zope.Interface in windows platforms 2010-04-24 18:19:52 -03:00
Daniel Grana 6c12106803 Remove shpinx warning introduced by shorter title overline 2010-04-18 23:42:56 -03:00
Lucian Ursu 2f8c052484 #154: Language fixes to the documentation 2010-04-18 23:39:54 -03:00
Pablo Hoffman dfdac356af added missing default values to file xporter doc 2010-04-02 02:49:18 -03:00
Pablo Hoffman f19c939925 fixed doc typo 2010-03-26 08:28:32 -03:00
Pablo Hoffman 99a876754c Improved "What else?" section of "Scrapy at a glance" overview 2010-03-20 20:24:18 -03:00
Pablo Hoffman 234fd709ad fixed doc typo (thanks Victor) 2010-03-19 10:32:17 -03:00
Daniel Grana 184cf6684f Remove HttpException references from docs. Since 0.7, scrapy returns non-200 as Response objects and does not raise HttpException anymore 2010-03-18 10:05:33 -03:00
Daniel Grana 17091902f3 Explicity say where to save item class in "Defining our item" section of tutorial 2010-03-12 14:12:49 -02:00
Daniel Grana c925c9e9a0 Notify spider when requests are ignored by HttpErrorMiddleware, and generally when any call to process_spider_input raises an exception 2010-05-12 16:41:06 -03:00
Daniel Grana c0d45846b8 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-26 22:29:45 -03:00
Pablo Hoffman 81f6502e37 Automated merge with http://hg.scrapy.org/scrapy-0.8/ 2010-04-24 18:22:13 -03:00
Daniel Grana 658e6f15e9 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-18 23:44:59 -03:00
Daniel Grana 68a875edb0 update ENCODING_ALIASES setting default value in settings documentation topic 2010-04-07 10:54:54 -03:00
Pablo Hoffman de32612c99 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-04-02 02:49:51 -03:00
Rolando Espinoza La fuente db5c3df679 SEP12 implementation 
* Rename BaseSpider.domain_name to BaseSpider.name

    This patch implements the domain_name to name change in BaseSpider class and
    change all spider instantiations to use the new attribute.

  * Add allowed_domains to spider

    This patch implements the merging of spider.domain_name and
    spider.extra_domain_names in spider.allowed_domains for offsite checking
    purposes.

    Note that spider.domain_name is not touched by this patch, only not used.

  * Remove spider.domain_name references from scrapy.stats

    * Rename domain_stats to spider_stats in MemoryStatsCollector
    * Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
    * Rename domain_stats_history table to spider_data_history and rename domain
    field to spider in MysqlStatsCollector

  * Refactor genspider command

    The new signature for genspider is: genspider [options] <domain_name>.

    Genspider uses domain_name for spider name and for the module name.

  * Remove spider.domain_name references

  * Update crawl command signature <spider|url>

  * docs: updated references to domain_name

  * examples/experimental: use spider.name

  * genspider: require <name> <domain>

  * spidermanager: renamed crawl_domain to crawl_spider_name

  * spiderctl: updated references of *domain* to spider

  * added backward compatiblity with legacy spider's attributes
    'domain_name' and 'extra_domain_names'
 2010-04-01 18:27:22 -03:00
Pablo Hoffman 2299deda66 updated wrong link in doc 2010-03-26 14:02:33 -03:00
Pablo Hoffman 7cf2f87e27 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-26 08:29:34 -03:00
Pablo Hoffman 1330697c3d Some improvements to Response encoding support: 
* added encoding aliases, configurable through a new ENCODING_ALIASES setting
* Response.encoding now returns the real encoding detected for the body
* simplified TextResponse API by removing body_encoding() and
  headers_encoding() methods
* Response.encoding now tries to infer the encoding from the body always (it
  was done before only on HtmlResponse and TextResponse)
* removed scrapy.utils.encoding.add_encoding_alias() function
* updated implementation of scrapy.utils.response function to reflect these API
  changes
* updated documentation to reflect API changes
 2010-03-25 15:47:10 -03:00
Pablo Hoffman 9ddcd1095d sort setting alphabetically 2010-03-25 11:45:06 -03:00
Pablo Hoffman 4fa833c849 Added LOG_ENCODING setting 2010-03-24 12:13:38 -03:00
Pablo Hoffman 87e68e7438 Made MailSender non IO-blocking, and improved MailSender documentation 2010-03-22 13:37:37 -03:00
Pablo Hoffman 1dfc79b5d0 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-20 20:48:11 -03:00
Pablo Hoffman 264cd2e035 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-03-19 10:32:42 -03:00
Pablo Hoffman d12cd22d5e switched default scheduler order to DFO, which consumes less memory by default 2010-03-04 10:15:58 -02:00
Pablo Hoffman 180c091fb2 Fixed encoding issue (reported in #135) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation. 2010-02-24 14:01:29 -02:00
Pablo Hoffman bbef0fe870 Automated merge with http://hg.scrapy.org/users/rolando/scrapy/ 2010-02-20 11:12:37 -02:00
Pablo Hoffman a3d22c7240 Automated merge with http://hg.scrapy.org/scrapy-0.8/ 2010-02-19 23:11:24 -02:00
Pablo Hoffman 60961e5499 minor documentation fix (refs #135) 2010-02-19 23:09:48 -02:00
Pablo Hoffman c1f8198639 Added RANDOMIZE_DOWNLOAD_DELAY setting 2010-02-19 21:53:18 -02:00
Rolando Espinoza La fuente a6a3f085a7 docs: added crawlspider v2 outline documentation 
Sign-Off: Rolando Espinoza La fuente
 2010-02-19 18:22:38 -04:00
Rolando Espinoza La fuente 7235040936 merged upstream 2010-02-19 17:41:45 -04:00
Daniel Grana 91f4d6dc51 docs: adds another spider example that yields multiples requests/items from a single callback 2010-02-18 16:51:05 -02:00
Pablo Hoffman 57d60eae39 sort settings doc alphabetically by setting name 2010-01-31 18:11:13 -02:00
Pablo Hoffman 67858af83c fixed doc typo 2010-01-18 18:16:58 -02:00
Pablo Hoffman 08eeaf98a2 fixed description of LOG_STDOUT setting 2010-01-13 15:51:08 -02:00
Pablo Hoffman 48739ae60c install.rst: added explanation about why libxml2 2.6.28 or above is required 2010-01-13 12:20:24 -02:00
Rolando Espinoza La fuente 1402da31c5 docs: fixed typos and updated code examples 2010-01-11 12:28:22 -04:00
Pablo Hoffman d60412ce19 titlecased Scrapy easy_install and some fixes to sign_release.sh script 2009-12-13 14:23:31 -02:00
Pablo Hoffman 422d6facb2 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-12-12 16:52:07 -02:00
Pablo Hoffman 9d50604d24 added |version| to documentation title 2009-12-12 16:51:59 -02:00
Pablo Hoffman a953efd8e5 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-12-12 15:40:16 -02:00
Ismael Carnales 4ecc909bc1 Fix RobotsTxtMiddleware reference in doc 2009-12-04 15:37:24 -02:00
Ismael Carnales 07344666e2 Move webconsole extensions doc to webconsole topic 2009-12-01 10:47:11 -02:00
Ismael Carnales e694c8ed02 Remove domain references in close spider extension doc 2009-11-30 11:38:56 -02:00
Ismael Carnales 12a7ff7312 Rename Close domain to close spider in extensions doc 2009-11-30 11:36:18 -02:00
Ismael Carnales 8d9cedd88b Reorder signals doc to respect alphabetical order 2009-11-30 11:29:19 -02:00
Ismael Carnales 93cc3d2715 Correct param formatting in item pipelines doc 2009-11-30 11:04:15 -02:00
Pablo Hoffman 6084be3b2e added iter_all() function to scrapy.util.trackref module and improved memory leaks documentation. also added a new FAQ antry about memory issues 2009-11-28 16:21:59 -02:00
Pablo Hoffman dd662e09d8 some minor fixes to scheduler middleware doc 2009-11-19 12:23:54 -02:00
Pablo Hoffman f4e93700bd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-19 10:44:02 -02:00
Pablo Hoffman c4f77c4da0 minor fixes to images doc (thanks amccloud) 2009-11-16 11:15:25 -02:00
Pablo Hoffman 0d6aee1f12 updated wrong documentation 2009-11-13 20:03:56 -02:00
Pablo Hoffman aeab5370cb StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method 2009-11-14 20:28:59 -02:00
Pablo Hoffman c4c6e7c8cd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-13 20:04:39 -02:00
Pablo Hoffman 07655d05ea renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER 2009-11-13 14:38:22 -02:00
Pablo Hoffman 564abd10ad Refactored HttpCache middleware: 
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure

Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
 2009-11-13 14:25:47 -02:00
Pablo Hoffman db7fec1fef fixed doc typo 2009-11-12 12:17:39 -02:00
Pablo Hoffman 415dec4e16 made offsite middleware log messages when filtering out requests 2009-11-12 10:17:21 -02:00
Pablo Hoffman 49e39bf1ba fixed typo 2009-11-06 16:49:48 -02:00
Pablo Hoffman 791f4932dd added clarification about versioning and api stability 2009-11-06 16:28:51 -02:00
Pablo Hoffman 74d0e82dbe renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_* 
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
 2009-11-06 15:54:17 -02:00
Pablo Hoffman 919cd5b789 renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS 2009-11-06 15:44:11 -02:00
Pablo Hoffman d604dca96d renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER 2009-11-06 15:42:11 -02:00
Pablo Hoffman 7728a23e99 Changed item pipeline API to pass spider references (instead of domain names) to process_item() method 2009-11-06 13:46:36 -02:00
Pablo Hoffman a432c1ee40 updated logging doc to include new spider argument in log functions 2009-11-04 14:49:24 -02:00
Pablo Hoffman 97c322707a * Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle} 
* Changed them to pass spider instances only (no domains) (refs #105)
 2009-11-03 00:39:02 -02:00
Pablo Hoffman 904cde6513 added clarification about new dont_click argument of FormRequest.from_response() method 2009-10-29 13:47:10 -02:00
Ismael Carnales a244d23b89 added dont_click attr to FormRequest 2009-10-29 13:18:13 -02:00
Pablo Hoffman b41c5b5d5b fixed typo in intro/install doc (thanks phaithful) 2009-10-29 10:41:20 -02:00
Pablo Hoffman 9b5fef4f48 fixed typo in intro/install doc (thanks phaithful) 2009-10-28 09:34:31 -02:00
Pablo Hoffman 7296a7b889 added DEFAULT_RESPONSE_ENCODING setting 2009-10-21 16:13:41 -02:00
Pablo Hoffman 720bc166cf updated new clickdata argument doc 2009-10-20 17:21:56 -02:00
Daniel Grana 6abb3c17ee Improve FormRequest.from_response method to pass click data arguments to ClientForm library 2009-10-20 15:51:41 -02:00
Pablo Hoffman 2712d55cb9 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 23:58:38 -02:00
Pablo Hoffman bd481751d8 moved images pipeline documentation to stable doc 
--HG--
rename : docs/experimental/images.rst => docs/topics/images.rst
 2009-10-07 22:57:25 -02:00
Pablo Hoffman b4d202a6b0 added note about memory usage extension not working on windows 2009-10-07 22:57:10 -02:00
Pablo Hoffman 937acd91d1 improved documentation of http proxy middleware 2009-10-07 21:00:34 -02:00
Pablo Hoffman d03bf1f27f Automated merge with http://hg.scrapy.org/scrapy-stable 2009-10-07 20:38:46 -02:00
Pablo Hoffman 37d9e015bb minor fix to tutorial 2009-10-07 20:15:49 -02:00
Daniel Grana bc64ca3e13 Add support to set http proxies per request, and obey enviroment variables http_proxy and no_proxy by default. 2009-10-05 04:10:22 -02:00
Daniel Grana 8aa7d153ae rewrote of downloader handlers 
* add REQUEST_HANDLERS setting with defaults for file, http and https schemes
* add documentation of new setting
* add unittests for all the builtin handlers
* remove unused getPage function
 2009-10-05 04:10:22 -02:00
Pablo Hoffman a0eec7eaf6 some typos fixes and updates to install doc 2009-09-29 09:44:02 -03:00
Ismael Carnales 1646482bef reformatted installation guide 2009-09-29 08:41:34 -03:00
Pablo Hoffman 4a9d9282bc removed obsolete scrapy architecture dia diagram 2009-09-28 23:54:01 -03:00
Ismael Carnales 5862ba7db7 modified doc to reflect the new spider callback return policy (lists not needed) 2009-09-22 11:25:40 -03:00
Ismael Carnales 802f918b69 removed obsolete doc static file 2009-09-22 11:25:38 -03:00
Pablo Hoffman 6e93872955 updated installation guide for using releases 2009-09-17 11:06:55 -03:00
Pablo Hoffman 132557dd14 some deployment changes in preparation for the 0.7.0 release candidate 2009-09-16 22:40:36 -03:00
Ismael Carnales fd41f06056 added doc on how to enable an Item Pipeline component 2009-09-16 14:19:16 -03:00
Ismael Carnales 404e7e09d7 changed spider doc references in BaseSpider class 2009-09-16 14:10:11 -03:00
Daniel Grana 062730cbd8 fix csv exporter documentation 2009-09-16 00:17:50 -03:00
Pablo Hoffman 56b292e057 XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience) 2009-09-14 22:05:52 -03:00
Pablo Hoffman e8960bf616 added runspider command to run spiders directly, without having to create a project 2009-09-14 22:05:14 -03:00
Pablo Hoffman 99467d4e6e Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method 2009-09-13 20:51:43 -03:00
Pablo Hoffman 921fc4f3bf Big Scrapy core refactoring to pass around spider references instead of domains. 
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.

This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.

However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.

--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
 2009-09-12 14:34:18 -03:00
Pablo Hoffman 8d49dc2fb5 changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc 2009-09-11 17:36:00 -03:00
Pablo Hoffman e20f766792 fixed some typos 2009-09-11 16:55:37 -03:00
Pablo Hoffman c2fe350f72 more changes to images pipeline doc 2009-09-11 16:53:36 -03:00
Ismael Carnales ada46a2dbb styled imagesp doc 2009-09-11 15:30:46 -03:00
Pablo Hoffman be0f2beef0 more cleanup to scheduler middelware doc, and permanentely moved to experimental doc 2009-09-11 13:27:31 -03:00
Pablo Hoffman 0af052b68f removed confusing title 2009-09-11 12:19:18 -03:00
Pablo Hoffman f3240748cb changed link to scheduler middleware doc, now in experimental 2009-09-11 12:03:23 -03:00
Ismael Carnales 3998a0cb58 added more scheduler middleware documentation, and moved it to experimental 
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
 2009-09-11 11:58:53 -03:00
Pablo Hoffman d242a20573 updated images pipeline doc 2009-09-11 11:47:12 -03:00
Pablo Hoffman f1bb8dc2a3 first cleanup of spider manager api 
- removed asdict() and reload() methods
- added list() method
- removed default spider
 2009-09-10 19:06:46 -03:00
Pablo Hoffman f85813cd94 added FAQ entry about scrapy recipes and community spiders 2009-09-10 18:32:50 -03:00
Pablo Hoffman 269724a2b7 added Debugger extension, removed StackTraceDump from extensions available by default 2009-09-08 22:32:17 -03:00
Ismael Carnales 4ddfa9a2a3 stlyed downloaded middleware doc 2009-09-07 12:18:57 -03:00
Ismael Carnales e3df11e5bb added module directive to spidermw documentation 2009-09-07 12:03:24 -03:00
Pablo Hoffman 827aa19c6e removed obsolete scrapy.utils.db module 2009-09-04 17:38:14 -03:00
Pablo Hoffman 861a803cc3 removed obsolete RestrictMiddleware 2009-09-04 17:22:56 -03:00
Ismael Carnales 7e2587169b added missing middleware docs 2009-09-04 12:39:02 -03:00
Pablo Hoffman aefb94063a more updates to spider middleware doc 2009-09-04 13:46:04 -03:00
Pablo Hoffman d04640be5c some improvements to spider middleware doc 2009-09-04 13:29:16 -03:00
Pablo Hoffman 96bb223c13 removed (pretty useless) DebugMiddleware 2009-09-04 12:59:58 -03:00
Pablo Hoffman 8a715701ec fixed another doc typo 2009-09-03 14:31:00 -03:00
Ismael Carnales 3c1bb7bc40 fixed typo in djangoitems doc (thanks anibal) 2009-09-03 11:23:25 -03:00
Daniel Grana 0e7b2a6da5 write header line by default when using csv exporter 
--HG--
extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e
 2009-09-03 13:58:39 -03:00
Pablo Hoffman 596d2c4479 moved CoreStats extension to scrapy.contrib.corestats 
--HG--
rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py
 2009-09-01 23:00:49 -03:00
Pablo Hoffman 6a50af05d7 removed useless SpiderReloader extension 2009-09-01 22:49:15 -03:00
Pablo Hoffman 79851aefa6 moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation 
--HG--
rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py
 2009-09-01 22:38:37 -03:00
Pablo Hoffman d3c51fd6f2 improved images pipeline documentation 2009-09-01 21:07:47 -03:00
Pablo Hoffman 18fd635124 another doc typo 2009-09-01 12:52:40 -03:00
Pablo Hoffman 538cc9803a fixed doc typo 2009-09-01 12:47:53 -03:00
Pablo Hoffman df0e1f005f exporters doc: fixed example and some typos 2009-09-01 08:56:54 -03:00
Pablo Hoffman ac8f46ce9e added File Export Pipeline reference to Exporters doc 2009-08-31 21:01:35 -03:00
Pablo Hoffman 8d006e9ea1 moved item exporters doc to stable doc 
--HG--
rename : docs/experimental/exporters.rst => docs/topics/exporters.rst
 2009-08-31 20:47:12 -03:00
Pablo Hoffman 0b152c99b5 added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines 2009-08-31 20:40:41 -03:00
Pablo Hoffman 8fab524978 moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console 2009-08-31 12:44:32 -03:00
Pablo Hoffman 884f0c878f Stats collectin: fixed race condition between stats persistance and population of stats on domain close 2009-08-29 19:44:13 -03:00
Pablo Hoffman 895c70e036 doc: fixed some links to scrapy-ctl topic 2009-08-29 18:23:55 -03:00
Pablo Hoffman 64572124e0 added doc about SCRAPY_SETTINGS_MODULE 2009-08-29 18:20:13 -03:00
Pablo Hoffman 9dc7a3a373 some minor adjustments to logging doc 2009-08-29 18:10:28 -03:00
Pablo Hoffman 60cbf24c89 more cleanups to startproject and project templates 
--HG--
rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py
 2009-08-29 04:29:47 -03:00
Pablo Hoffman c87216f02b doc: added missing :synopsis: to some modules 2009-08-29 03:46:58 -03:00
Pablo Hoffman 2f55d7932b replaced :ref: by :doc: links in doc index 2009-08-29 03:37:59 -03:00
Pablo Hoffman 924219dd5a - added reference documentation about scrapy-ctl.py script 
- yet another refactor to cmdline module
- removed help command (use -h to help now)

--HG--
rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst
 2009-08-28 20:32:55 -03:00
Pablo Hoffman 609aed4553 added inspect_response() function for inspecting responses from spiders 2009-08-27 19:33:33 -03:00
Pablo Hoffman ea4f16bae6 refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut 2009-08-27 18:24:08 -03:00
Ismael Carnales fb39bca24f added djangoitem doc 2009-08-26 08:44:22 -03:00
Pablo Hoffman 65ab611fb0 more updates to HttpErrorMiddleware doc 2009-08-26 08:30:18 -03:00
Pablo Hoffman 44783a3a06 minor improvements to FAQ entry 2009-08-26 00:18:58 -03:00
Pablo Hoffman 4dba75a73f updated HttpErrorMiddleware doc 2009-08-25 20:13:58 -03:00
Pablo Hoffman 8a074c9cb5 removed scrapy-admin.py command, and left only scrapy-ctl as the only scrapy command 2009-08-24 15:43:36 -03:00
Ismael Carnales 39540b188a changed torrent in overview doc 2009-08-24 15:11:04 -03:00
Ismael Carnales 4b5aa30867 minor update to tutorial 2009-08-24 14:34:17 -03:00
Pablo Hoffman 0363040884 doc: added FAQ entry about Accept-Language 2009-08-24 13:56:44 -03:00
Ismael Carnales 85282a4b76 added scrapy commandline scripts doc 2009-08-24 12:02:44 -03:00
Pablo Hoffman ead8cb9fac removed documentation about ugly DontCloseDomain exception (which will be removed in the future) 2009-08-24 11:57:21 -03:00
Pablo Hoffman 74f706b356 renamed "parse_item" method of XMLFeedSpider to "parse_node", keeping backwards compatibility 2009-08-24 10:54:34 -03:00
Pablo Hoffman 31693eb90f dropped "cache" attribute of Request and Response objects 2009-08-24 10:34:05 -03:00
Pablo Hoffman 0186c6937a HTTP auth middleware: added doc and unittest 2009-08-24 08:07:20 -03:00
Pablo Hoffman 46aaea9f32 doc: improved documentation about debugging leaks 2009-08-23 20:36:00 -03:00
Pablo Hoffman 0ed849248f some improvements to item exporters 
- passed previous class attributes to instances attributes
- better handling of constructor arguments
- better coverage on unittets (including encoding)
- updated documentation with new changes
 2009-08-23 05:48:35 -03:00
Pablo Hoffman 7461710ef1 added some missing dots 2009-08-21 21:54:10 -03:00
Pablo Hoffman 9635a7839c rearranged documentation into a better organization 
--HG--
rename : docs/topics/index.rst => docs/index.rst
 2009-08-21 21:49:54 -03:00
Pablo Hoffman 4761e0c88d minor doc correction 2009-08-21 19:11:59 -03:00
Pablo Hoffman 012fd1cc2f moved api-stability.rst doc to root and updated it 
--HG--
rename : docs/misc/api-stability.rst => docs/api-stability.rst
 2009-08-21 16:29:23 -03:00
Pablo Hoffman ef6f04eb06 moved doc about debugging memory leaks to its own topic and added doc about trackref module 2009-08-21 16:07:16 -03:00
Pablo Hoffman 4980eb4968 added titles to signals doc 2009-08-21 15:07:52 -03:00
Pablo Hoffman daea3f8a0c sphinx docs: replaced custom :exception: xref by standard :exc: 2009-08-21 15:05:06 -03:00
Ismael Carnales c08d3aa9cc updated tutorial to use new items api 2009-08-21 14:16:27 -03:00
Pablo Hoffman 37760eb13f improved consistency of logging settings to use LOG_* 2009-08-21 08:54:12 -03:00
Ismael Carnales f0689a9ee8 fixed error in link extractors doc, thanks tarasm 2009-08-21 08:34:53 -03:00
Pablo Hoffman 41f1972e47 removed unused TRACE log level and improved logging documentation 2009-08-20 18:17:48 -03:00
Pablo Hoffman 20e82335e2 rename some exporter methods and complete exporter tests refactoring 2009-08-20 14:09:24 -03:00
Pablo Hoffman 3152e66fec deprecate domain_open signal and handle stats domain open/close directly from the engine 2009-08-20 10:54:58 -03:00
Pablo Hoffman 33b53c59d5 moved scrapy.xpath to scrapy.selector 
--HG--
rename : scrapy/xpath/__init__.py => scrapy/selector/__init__.py
rename : scrapy/xpath/document.py => scrapy/selector/document.py
rename : scrapy/xpath/factories.py => scrapy/selector/factories.py
 2009-08-19 21:50:52 -03:00
Pablo Hoffman dcc90fc196 declared loaders api stable and updated example project to use them 
--HG--
rename : docs/experimental/loaders.rst => docs/topics/loaders.rst
 2009-08-19 21:39:59 -03:00
Pablo Hoffman e8504a054c moved scrapy.newitem to scrapy.item and declared newitem api officially stable. updated docs and example project. deprecated old ScrapedItem 2009-08-19 21:39:58 -03:00
Ismael Carnales 314e8dea43 added new item exporter tests, introduced some api changes 2009-08-19 19:05:33 -03:00
Pablo Hoffman 907fad6da8 make sure input processors always receive iterables as input 2009-08-19 16:49:49 -03:00
Pablo Hoffman 741a6d7863 item exporters refactoring 
--HG--
rename : scrapy/contrib/exporter/jsonexporter.py => scrapy/contrib/exporter/jsonlines.py
 2009-08-19 13:09:39 -03:00
Pablo Hoffman 82e4b6adcf merge with ismael repo 2009-08-18 15:38:20 -03:00
Pablo Hoffman e01d31c498 another improvement to doc navbar 2009-08-18 15:35:53 -03:00
Ismael Carnales be4226cec5 merge 2009-08-18 15:21:39 -03:00
Ismael Carnales f88fb27851 fixed error in xpath selectors doc 2009-08-18 15:18:49 -03:00
Ismael Carnales 67c0c6a9e4 corrected indentation in xpath selectors doc 2009-08-18 15:13:23 -03:00
Pablo Hoffman ff837e5a45 doc: improved top navbar 2009-08-18 15:12:44 -03:00
Ismael Carnales 428dfe0d4a corrected the style of spiders documentation 2009-08-18 15:06:33 -03:00
Pablo Hoffman 0192282d07 reorganized doc and moved robotstxt doc inside downloader middlewares doc 2009-08-18 14:36:18 -03:00
Ismael Carnales 33089d287d merged topics and reference doc 2009-08-18 14:05:15 -03:00
Pablo Hoffman 7f30461410 added support for defining EXTENSIONS setting using dicts, like middleware settings 
--HG--
rename : scrapy/tests/test_utils_middleware.py => scrapy/tests/test_utils_conf.py
rename : scrapy/utils/middleware.py => scrapy/utils/conf.py
 2009-08-18 11:05:36 -03:00
Ismael Carnales 1bbe7991dc added documentation for ImagesPipeline 2009-08-18 09:35:32 -03:00
Ismael Carnales 48b40bd620 renamed x method of selectors to select 2009-08-17 15:58:06 -03:00
Pablo Hoffman 59e0a83ad4 removed more obsolete adaptors code 2009-08-17 14:48:11 -03:00
Ismael Carnales 8c2f62ba9c added Item Exporters documentation 2009-08-14 09:16:29 -03:00
Pablo Hoffman 02b01c3d2b loaders doc: fixed outdated line 2009-08-13 23:24:24 -03:00
Daniel Grana 6a74b51371 remove dupes words in loaders doc, and unused import in example 
--HG--
extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e
 2009-08-13 23:23:08 -03:00
Ismael Carnales ecca7f6478 fixes to Item Loader doc 2009-08-13 13:30:50 -03:00
Pablo Hoffman 8131d674d4 fixed outdated documentation (refs #97) 2009-08-13 09:24:41 -03:00
Pablo Hoffman 5aeab5b291 converted scrapy.item package to module 
--HG--
rename : scrapy/item/models.py => scrapy/item.py
 2009-08-12 21:31:50 -03:00
Pablo Hoffman 991afdf6b3 some minor fixes to loaders doc 2009-08-12 19:23:04 -03:00
Pablo Hoffman e6dd4d0955 renamed ApplyConcat processor to MapCompose 2009-08-12 18:43:08 -03:00
Pablo Hoffman 4e74f32401 renamed Pipe processor to Compose and documented it 2009-08-12 18:09:36 -03:00
Pablo Hoffman 958c3d9940 fixed some links to item loaders doc 2009-08-12 17:42:01 -03:00
Pablo Hoffman d94b633565 renamed ItemLoader method populate_item() to load_item() 2009-08-12 17:40:27 -03:00
Pablo Hoffman 1dc592882b Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests 
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
 2009-08-12 16:49:07 -03:00
Pablo Hoffman 7cbbc3ffb0 Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated. 
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
 2009-08-12 16:49:05 -03:00
Pablo Hoffman d4aa72d2bb removed obsolete documentation about Robust Scraped Item and Adaptors 2009-08-12 13:50:15 -03:00
Pablo Hoffman e69311ef38 added missing text to new stats collector methods 2009-08-11 16:37:52 -03:00
Daniel Grana 45cbe1333f fix typo in stats docs 2009-08-11 16:30:04 -03:00
Daniel Grana 8f16e7f9fa remove `default` parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector 2009-08-11 16:23:23 -03:00
Daniel Grana cd8eed77c5 stats collector gains two new methods to store values only if greater/lower than previous value 2009-08-11 15:46:44 -03:00
Ismael Carnales 3614998053 fixed error in doc 2009-08-11 09:23:45 -03:00
Pablo Hoffman 7ac7597950 removed obsolete scrapy.contrib.cluster 2009-08-10 21:02:46 -03:00
Pablo Hoffman c0532712f9 removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support 2009-08-10 21:02:24 -03:00
Pablo Hoffman d334c035c0 removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references 2009-08-10 20:52:43 -03:00
Pablo Hoffman 49bab4777b XPathSelector: added 're' argument to add_xpath method, exposed selector attribute 2009-08-10 19:42:20 -03:00
Pablo Hoffman 50fcc67278 improved reducers examples 2009-08-10 10:13:30 -03:00
Pablo Hoffman 5be0cee577 added TreeExpander example 2009-08-09 20:54:22 -03:00
Pablo Hoffman 38f82e3993 loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements 2009-08-09 18:06:12 -03:00
Pablo Hoffman b296d4169e minor doc update for making it more windows-friendly 2009-08-09 17:08:42 -03:00
Pablo Hoffman 91eea82eef added XPathLoader for working with XPath Selectors more conveniently 2009-08-08 07:26:20 -03:00
Pablo Hoffman db90e26a8b renamed ItemLoader class to Loader 2009-08-07 14:39:30 -03:00
Pablo Hoffman e585c6cac4 relocated experimental newitems/loaders doc, and added example for extending fields metadata 
--HG--
rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst
rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst
 2009-08-07 14:28:58 -03:00
Pablo Hoffman d95e99f585 Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation 
--HG--
rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst
 2009-08-07 03:50:09 -03:00
Ismael Carnales 32894643a0 added ListField documentation, ordered field reference alphabetically 2009-08-03 15:00:04 -03:00
Pablo Hoffman c3427e075c added domain_stats parameter to stats_domain_closed signal 2009-07-31 16:36:35 -03:00
Pablo Hoffman 73172b244d added from_unicode_list() method to Field objects 2009-07-30 16:58:24 -03:00
Pablo Hoffman 8ec7c9e01c WEBCONSOLE_PORT setting now defaults to 6080 2009-07-29 18:59:34 -03:00
Pablo Hoffman fcc91901eb finished cleaning up closedomain documentation, and updated default settings 2009-07-27 15:42:35 -03:00
Pablo Hoffman 09ba6927d7 Some changes to CloseDomain extension: 
- added support for closing by item passed count (CLOSEDOMAIN_ITEMPASSED)
- removed support for sending notification emails (since that's the job of
  another extension)
 2009-07-27 15:23:50 -03:00
Pablo Hoffman 9da66698f3 moved httprepr() method (from Request and Response objects) to scrapy.utils functions 2009-07-25 18:56:12 -03:00
Pablo Hoffman c615ace6bd doc: updated google directory links in firebug guide 2009-07-24 13:14:36 -03:00
Daniel Grana 96c3cdbec2 improve OffsiteMiddleware reference docs 
--HG--
extra : rebase_source : 3ed3f23fc1ec63b521ead029c5749898f3ab05d7
 2009-07-24 12:59:38 -03:00
Pablo Hoffman 38c3f7d0b4 Some changes to logging of scraped items: 
1. "Scraped Item" log level changed to DEBUG
2. "Dropped Item" log level changed to WARNING
3. added "Passed Item" log message with INFO level
 2009-07-23 11:49:48 -03:00
Pablo Hoffman e43e28bf1d minimal doc improvement 2009-07-23 09:12:49 -03:00
Ismael Carnales 6d24ae5920 added reference to working with relative xpaths in the tutorial 2009-07-23 09:05:14 -03:00
Pablo Hoffman 9baa6bb2a8 minor selectors doc fix 2009-07-23 01:56:56 -03:00
Ismael Carnales 3c4afb23be moved newitem from scrapy.contrib_exp to scrapy.newitem 2009-07-22 15:13:36 -03:00
Ismael Carnales 202894dd8f fixes to newitem doc 2009-07-22 10:25:22 -03:00
Pablo Hoffman 7d8ba0542c fixed typo in doc 2009-07-21 17:38:46 -03:00
Pablo Hoffman aa345e116a Added spider middleware documentation 2009-07-21 17:19:19 -03:00
Ismael Carnales 90f1d9e489 Added Scheduler middleware reference documentation 2009-07-21 16:52:27 -03:00
Ismael Carnales 00d9ce6608 fixes to newitem doc 2009-07-21 12:22:49 -03:00
Ismael Carnales 894a5d81d3 changed the newitem API to a dict-like interface 2009-07-21 12:20:49 -03:00
Pablo Hoffman 9867e7b2d6 removed references to obsolete ENABLED_SPIDERS_FILE setting 2009-07-21 01:11:48 -03:00
Pablo Hoffman 92b746d866 SimpledbStatsCollector: moved domain creation to constructor 2009-07-17 12:49:53 -03:00
Pablo Hoffman 292757f312 added link to architecture overview and fixed old link 2009-07-16 19:15:19 -03:00