RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,191 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2958 Commits

Author SHA1 Message Date
Mikhail Korobov 91791cd329 DOC final changelog cleanups 2019-01-30 12:49:36 -03:00
Mikhail Korobov 2c8c8b2dd8 DOC fix after bad merge - remove duplicate entries in changelog 2019-01-30 12:49:36 -03:00
Mikhail Korobov 0fc9d705c2 DOC mention that telnet security improvements happened in 1.5.2 2019-01-30 12:49:36 -03:00
Mikhail Korobov 4cf4dd1d3e DOC add recent changes to changelog 2019-01-30 12:49:36 -03:00
Mikhail Korobov 638469f9ef DOC extract_first/extract matches get/getall better 
Thanks @Gallaecio!
 2019-01-30 12:49:36 -03:00
Mikhail Korobov e479f5aa15 DOC update changelog 
* changes from recently merged pull requests
* more highlights
* re-organized headers
* Selector API changes
 2019-01-30 12:49:36 -03:00
Mikhail Korobov 706910790b [wip] draft 1.6 release notes 2019-01-30 12:49:36 -03:00
Daniel Graña b5026b842c
Merge pull request #3544 from joaquingx/fix-item-pipeline-x 
Fix item-pipeline example
 2019-01-28 10:44:19 -03:00
Harry Moreno 8fca98616a fix grammar 2019-01-28 10:19:31 -03:00
Harry Moreno b828b5f8c8
fix grammar 2019-01-26 18:39:05 -05:00
Daniel Graña 71743a6546 Add release notes for v1.5.2 2019-01-22 18:56:47 -03:00
Eugenio Lacuesta bddfeaba4c Add Request.kwargs docs 2019-01-15 19:14:59 -03:00
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
Eugenio Lacuesta 9759112a3a Merge branch 'master' into process_spider_exception_generator 2019-01-03 11:34:07 -03:00
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00
Daniel Graña 76433feb4f
Merge pull request #3548 from scrapy/fix-aws-settings-docs 
fix docs for AWS_... settings. A follow-up to GH-2609.
 2018-12-26 11:10:57 -03:00
Daniel Graña 6a0ea0cf26
Merge pull request #3415 from scrapy/telnet-auth 
[MRG+1] Telnet console authentication
 2018-12-26 10:58:44 -03:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00
Mikhail Korobov 4306886ac6
Merge pull request #3527 from hsiaoyi0504/patch-1 
unify the quote style
 2018-12-26 14:45:06 +05:00
Mikhail Korobov 7c26701012 DOC warn about telnet console being insecure 2018-12-26 01:33:58 +05:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
Joaquin Garmendia Cabrera f85c915872
Update item-pipeline example 2018-12-23 00:26:58 -05:00
kasun Herath 24acc50d18 dumps_kwargs parameter in docs 2018-12-18 23:16:14 +05:30
kasun Herath 12ad06b7ac docs change 2018-12-17 23:17:13 +05:30
kasun Herath 8f1507a4a5 dumps_kwargs 2018-12-17 23:14:06 +05:30
Daniel Graña 665c04b0c1
Merge pull request #3518 from Gallaecio/scrapy-project-doc 
Document the SCRAPY_PROJECT environment variable
 2018-12-12 11:56:47 -03:00
hsiao yi cd9d8e28cd
unify the quote style 2018-12-11 19:21:07 +08:00
kasun Herath ecda69130e allow to send empty data values and docs changes 2018-12-10 22:34:49 +05:30
kasun Herath 3c981bf204 add documentation 2018-12-09 12:56:12 +05:30
Adrián Chaves 62f3349c1a Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2018-12-03 17:14:10 +01:00
Adrián Chaves 274b65dff4 Add a troubleshooting section to the installation instructions 
Its initial content covers the workaround for #2473.
 2018-12-03 16:36:05 +01:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 44f8e28b3c Fix headings' underlines 2018-10-16 19:53:20 -03:00
Henrique Coura 92b7955d75 Add Telnet console authentication docs 2018-10-16 14:50:00 -03:00
Eugenio Lacuesta 2396356a82 Merge branch 'master' into process_spider_exception_generator 2018-10-10 11:37:43 -03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements 
also, move "Selecting attributes" reference closer to `a::atr(href)` example
 2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example 
See https://github.com/scrapy/parsel/pull/119
 2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs 
Also, response.urljoin is added in a few places, for robustness.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov afce9716fa DOC mention .attrib in the tutorial 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials: 
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 09fd6c2a81 DOC unlink Firefox & Firebug sections from the tutorial for now. 
See https://github.com/scrapy/scrapy/issues/3373 and https://github.com/scrapy/scrapy/issues/3372 for motivation.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Vostretsov Nikita c02cfa574c remove comma 2018-08-29 11:21:55 +00:00
Raphael Wuillemier 79de3d569a Removed obsolete firebug-images 2018-08-23 16:19:13 +02:00
testingcan e98e7f8506
Added missing curly brace 2018-08-23 14:50:49 +02:00
Raphael Wuillemier 4d3aaabbca Updated code, added code snippets and improved readability 2018-08-23 12:40:31 +02:00
testingcan 3a71e7dbce
Increased length of "=" 2018-08-22 16:57:51 +02:00
Raphael Wuillemier af555cab23 Added general guide for developer tools instead of Firefox and Firebug-sections 2018-08-22 14:15:53 +02:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Vostretsov Nikita 597b8a97ad documentation for new signal 2018-08-17 14:39:42 +00:00
Stas Glubokiy b4b1e48343 Add ability to use FormRequest in contracts 2018-08-11 22:18:43 +03:00
Raphael Wuillemier d3aa1e8666 Updated tutorial.rst to include more and up-to-date beginner resources 2018-08-06 17:40:31 +02:00
Eugenio Lacuesta 40449fa0eb Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
Mikhail Korobov 895df937a3
Merge pull request #3350 from granadoho/master 
make amendments to grammer
 2018-07-25 23:50:06 +03:00
Malcolm Granado Ho Yong Liang 48866457b3 make amendments to grammer 2018-07-25 14:38:37 +08:00
Andrei Korigodski 7020c3e452
Doc: update copyright notice 
The years are updated. The hyphen is replaced with an en dash.
 2018-07-20 14:46:57 +03:00
Eugenio Lacuesta c493721b64 Merge branch 'master' into process_spider_exception_generator 2018-07-18 15:18:42 -03:00
Eugenio Lacuesta e96b7782bc Merge branch 'master' into process_spider_exception_generator_experiment 2018-07-15 18:17:37 -03:00
Valdir Stumm Junior c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Mikhail Korobov c86213317d 1.5.1 release notes 2018-07-12 02:17:08 +05:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Eugenio Lacuesta df75a0942e Update docs 2018-07-01 13:35:14 -03:00
Daniel Graña 594a441eb5
Merge pull request #3294 from rennerocha/feed-exporter-docs 
[MRG+1] Improve docs of S3 Storage to make Python version more explicit
 2018-06-27 11:16:07 -03:00
Eugenio Lacuesta ab48837f09 Merge branch 'master' 2018-06-24 20:31:37 -03:00
Leo 7a601d76de fix typo 
extractred --> extracted
 2018-06-19 10:51:55 +02:00
Renne Rocha e2bb218e9a Include Python version indication to each required library used in S3 storage 2018-06-13 18:11:43 -03:00
mugayoshi d4511667fb Update debugging memory leaks section in the docs 
Add Python3 tools description.
 2018-06-09 18:17:11 +09:00
Mikhail Korobov 847b50ce9c
Merge pull request #3201 from grammy-jiang/master 
[MRG+1] fix a mistake in topic spider-middleware.rst
 2018-06-02 01:09:59 +05:00
Mikhail Korobov 13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1 
Update spiders.rst
 2018-06-01 22:20:41 +05:00
Chris Slothouber ecdd888ff4
Minor edits to contributing.rst 
Corrected minor grammatical issues and increased clarity of instructions.
 2018-06-01 09:25:34 -04:00
Kevin Tewouda ffa7bede17
Update spiders.rst 
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
 2018-05-30 06:33:18 +02:00
Ryan P Kilby 2dfc5d128b Update DEPTH_STATS refs to DEPTH_STATS_VERBOSE 2018-05-17 11:00:21 -04:00
Konstantin Lopuhin c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc 
[MRG+1] Improve document about functions as processors
 2018-04-26 09:36:39 +03:00
Konstantin Lopuhin f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl 
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
 2018-04-25 19:42:19 +03:00
Lucy Wang 57b0e6b695 improve document about functions as processors 2018-04-19 13:35:46 +08:00
rhoboro 6ef6585b5a update docs 2018-04-13 19:06:29 +09:00
rhoboro 464973489e Using bucket's default object ACL 2018-04-13 12:06:39 +09:00
grammy-jiang cb76b88331 fix a mistake in topic spider-middleware.rst 2018-04-04 05:56:05 -04:00
rhoboro 74a9c65290 update docs for support gcs acl 2018-04-03 18:20:37 +09:00
Steven Almeroth c6d20bdd82 Doc: update wording for COOKIES_ENABLED 2018-03-27 16:21:07 -04:00
Arvind Prasanna 412f852602 A few typo fixes and some grammatical enhancements 2018-03-06 23:58:27 -05:00
Anjali Jain bbc2a3569f further edited 2018-02-16 23:33:10 +05:30
Anjali Jain 6954da1366
Updated contributing.rst 
Rectified grammatical errors
 2018-02-15 23:27:40 +05:30
Mikhail Korobov dc0304fde1 fix docs building with recent sphinx: don't use deprecated sphinx options and imports 2018-02-13 19:47:41 +05:00
Mikhail Korobov 786144e0c7
Merge pull request #3058 from jesuslosada/fix-link 
Fix link in news.rst
 2018-01-11 01:33:38 +05:00
Jesús Losada a0836b8fd9 Fix link in news.rst 2018-01-01 15:59:38 +00:00
Daniel Graña 2dee191374
Merge branch 'master' into extending-s3-files-store 2017-12-31 16:44:38 -03:00
Mikhail Korobov d07fe11981 set release date 2017-12-30 02:09:41 +05:00
Mikhail Korobov c107059ef8 DOC fix rst syntax 2017-12-29 07:07:43 +05:00
Mikhail Korobov d4e5671d07 make release docs more readable, add highlights 2017-12-29 07:06:00 +05:00
Mikhail Korobov 45b0e1a0e4 DOC draft 1.5 release notes 2017-12-28 07:33:43 +05:00
Paul Tremberth 461f9daff5 Update release notes for upcoming 1.4.1 version 2017-12-28 05:50:18 +05:00
Raphael a1cc5a63d3 Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999) (#3030) 
Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999)
 2017-12-27 17:54:17 -03:00
Konstantin Lopuhin bb1f311891 Add PyPy3 support to faq and install doc 2017-12-25 15:46:05 +03:00
Konstantin Lopuhin ea41114cf0 Mention PyPy support, add PyPy to install docs 2017-12-25 12:35:04 +03:00
Hugo cbcf80b98f
Fix typo 
[CI skip]
 2017-12-20 17:34:13 +02:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Mikhail Korobov 9aa9dd8d45 DOC mention an easier way to track pull requests locally. 
Thanks @eliasdorneles!
 2017-12-12 19:17:00 +05:00
Mikhail Korobov f716843a66 DOC update "Contributing" docs: 
* suggest Stack Overflow for Scrapy usage questions;
* encourage users to submit test-only pull requests with reproducable examples;
* encourage users to pick up stalled pull requests;
* we don't use AUTHORS file as a main acknowledgement source;
* suggest using Sphinx autodocs extension
 2017-12-12 16:19:43 +05:00
Daniel Graña 3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command 
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
 2017-11-29 16:26:48 -03:00
IAlwaysBeCoding 6af323d7c8
Fix spelling mistake on scrapy parse command docs 
Fixed spelling mistake from "will be pass" to "will be passed"
 2017-11-26 00:24:52 +01:00
Jesse Bakker 0b14cb44aa Added from_crawler to middleware docs 2017-11-23 15:25:43 +01:00
IAlwaysBeCoding 846fd83512 removed commented out code, wrapped line to pep-8 and removed backlashes 2017-11-11 18:30:01 -05:00
Daniel Graña b8870ee8a1
Merge pull request #2989 from colinmorris/ItemExporterDocsExample 
[MRG+1] Revise/modernize item exporter example in docs
 2017-11-03 13:15:31 -02:00
colinmorris 23e571e860 fix issues identified in review 2017-10-31 18:08:47 -04:00
colinmorris 8a7552370d revise/modernize item exporter example in docs 2017-10-31 17:14:53 -04:00
Aditya df7e0a4315 Use https link in default user agent 2017-10-28 23:37:44 +05:30
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Mikhail Korobov 98fb03e8bd Merge pull request #2958 from codeaditya/update-links 
Link "Debugging in Python" article to its new location
 2017-10-26 17:01:47 +05:00
Mikhail Korobov 496fc60b25 Merge pull request #2963 from djunzu/mention_request_meta_depth_on_depthmiddleware_doc 
Add note about request.meta['depth'] in DepthMiddleware
 2017-10-26 16:15:36 +05:00
Mikhail Korobov 73c40eb324 Merge pull request #2964 from weldon0405/patch-1 
Update tutorial.rst startproject files
 2017-10-26 16:05:31 +05:00
Weldon Malbrough 95815d27e8 updated file structure to include middlewares.py 2017-10-25 23:16:30 -04:00
Weldon Malbrough 169dc2860e Update tutorial.rst startproject files 
Added middlewares.py to accurately reflect the file structure created by "scrapy startproject tutorial"
 2017-10-16 22:46:32 -04:00
djunzu 8288f78a39 Add note about request.meta['depth'] in DepthMiddleware 2017-10-16 21:34:37 -02:00
Aditya 9cdf34b7c7 Link "Debugging in Python" article to its new location 
Reference: https://web.archive.org/web/20170203104051/http://www.ferg.org/papers/debugging_in_python.html
 2017-10-10 22:49:22 +05:30
Paul Tremberth 0d8a33fddc Update docs 2017-10-05 10:16:42 +02:00
Daniel Graña 5fac2d7b90 Merge pull request #2923 from rhoboro/fixes-685 
[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage.
 2017-10-02 13:52:20 -03:00
Mikhail Korobov 92fa74fb43 Merge pull request #2922 from stav/doc.response-body 
[Doc] Update Response.body type
 2017-10-02 10:26:54 +02:00
Mikhail Korobov 3431870b53 Merge pull request #2924 from superyyrrzz/patch-1 
minor fix typo
 2017-10-02 10:25:40 +02:00
rhoboro d4555b2bcc update docs for supporting google cloud storage 2017-09-29 13:04:17 +09:00
Renze Yu 088b80d41a minor fix typo 2017-09-13 23:29:22 +08:00
Steven Almeroth 3637b75a67 [Doc] Update Response.body type 2017-09-12 17:41:47 -04:00
cclauss b8fabeed86 ur'string' not needed in Py 2, syntax error in Py3 
This instance was missed in #2909 --> ur'Scrapy developers' --> u'Scrapy developers'
 2017-09-04 14:29:23 +02:00
cclauss b702236082 ur'string' not needed in Py 2, syntax error in Py3 
Convert `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`--> `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`to be compatible with both Python 2 and Python 3.  See #2891
 2017-09-01 11:56:09 +02:00
cclauss 9f16f040b6 ur'string' not needed in Py 2, syntax error in Py3 
Convert `ur'Scrapy Documentation'`--> `u'Scrapy Documentation'`to be compatible with both Python 2 and Python 3.  See #2891
 2017-09-01 11:53:59 +02:00
Pablo Hoffman a429d78019 update scrapinghub.com urls to use https 2017-08-24 16:03:36 -03:00
david watson 71d5b7d75a fix typo (#2867) 2017-08-01 14:49:22 -03:00
simik-ru a65fec050a Small fix in description of startproject arguments 2017-07-31 13:03:49 +02:00
Daniel Graña b6d036ede9 Merge pull request #2837 from dguo/patch-2 
Fix a typo in the Items documentation
 2017-07-24 15:55:30 -03:00
Daniel Graña 6de106e7a8 Merge pull request #2763 from scrapy/dataloss-typo 
DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs
 2017-07-24 15:48:03 -03:00
Daniel Graña d1e948cef8 Merge pull request #2791 from starrify/doc-DontCloseSpider 
[MRG+1] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
 2017-07-24 15:29:38 -03:00
Daniel Graña 8e0b640d2b Merge pull request #2826 from dguo/patch-1 
[MRG+1] Tweak the CSVFeedSpider documentation
 2017-07-24 15:28:15 -03:00
Danny Guo 26c488970c Fix a typo in the Items documentation 2017-07-18 19:56:51 -05:00
Claus Conrad 18b96dd82a Spelling mistake 2017-07-15 11:31:09 +02:00
Danny Guo dedc4a8b8f Tweak the CSVFeedSpider documentation 2017-07-13 22:58:10 -05:00
Pengyu CHEN f712513ed7 Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`. 2017-06-15 11:09:17 +08:00
Mikhail Korobov 5aebdac45d Merge pull request #2781 from crasker/patch-1 
use suggest method instead of DEPRECATED one
 2017-06-14 03:29:45 +05:00
Casker ae679f6499 Create item-pipeline.rst 2017-06-09 16:12:20 +08:00
Paul Tremberth 39ad0d0bdd Fix setting name reference 2017-06-06 10:48:30 +02:00
Paul Tremberth e7061f7a41 Reformat a bit 2017-06-06 10:47:43 +02:00
Chuan Jin 3f8542eb56 Update extensions.rst 
#2759
 2017-06-05 23:31:37 +02:00
Paul Tremberth fad6b70d92 Use https:// for readthedocs links 2017-06-01 16:41:52 +02:00
Mikhail Korobov 083880888b DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs 2017-05-30 00:44:11 +05:00
Valdir Stumm Junior 80b160d0d7 include references to scrapy subreddit in the docs 2017-05-29 14:56:49 -03:00
Kurt Peek af2963d0eb Update autothrottle.rst 
Added missing bullet point for the AUTOTHROTTLE_TARGET_CONCURRENCY setting.
 2017-05-24 19:59:36 +05:00
Paul Tremberth fc2846d637 Set release date for v1.4.0 2017-05-18 22:59:46 +02:00
Mikhail Korobov 76e5b0f65c DOC 1.4 deprecations and backwards incompatible changes, add recent commits to news. 2017-05-18 22:37:15 +02:00
Mikhail Korobov edcde7a2cf DOC tweak release notes: promote response.follow, mention logging/stats changes 2017-05-18 22:37:15 +02:00
Paul Tremberth a3d3cd4cb7 Update with latest merges 2017-05-18 22:37:15 +02:00
Paul Tremberth 432668acf7 Mention implementation of #667 2017-05-18 22:37:15 +02:00
Paul Tremberth 896c30a8eb Reference items pretty-printing issue number 2017-05-18 22:37:15 +02:00
Paul Tremberth 55d1082360 Reference recent fixes and commits 2017-05-18 22:37:15 +02:00
Paul Tremberth 7d72394794 Reword mention of new response.follow() shortcut 2017-05-18 22:37:15 +02:00
Paul Tremberth c6464cc4f5 Add verbose introduction to new features 2017-05-18 22:37:14 +02:00
Paul Tremberth e139d990fc Fix sphinx-build warning on deprecated latex_paper_size 2017-05-18 22:37:14 +02:00
Paul Tremberth cba55cd190 Rephrase other sections 2017-05-18 22:37:14 +02:00
Paul Tremberth 8729a91f7a Rephrase "New features" section 2017-05-18 22:37:14 +02:00
Paul Tremberth 851adcedf2 List merged pull requests since 1.3.3 2017-05-18 22:37:14 +02:00
Paul Tremberth 8aa2e4f997 Merge pull request #1829 from nyov/nyov/editor 
[MRG+1] Remove dependency on os.environ from default settings
 2017-05-18 21:32:51 +02:00
Daniel Graña b9d3b447a5 Merge pull request #2670 from qhuang872/master 
[MRG+1] Update spiders.rst
 2017-05-18 16:31:01 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Mikhail Korobov 1acacab93b Merge pull request #2741 from eliat123/2576_cleanup_MEMUSAGE_REPORT 
[FIX #2576] cleanup: removed unused MEMUSAGE_REPORT
 2017-05-17 13:48:00 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Harrison Gregg 45a323024c Add documentation for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Paul Tremberth 3a0a86ed31 Clarify FEED_EXPORT_INDENT section 2017-05-12 17:26:17 +02:00
Eugenio Lacuesta 25535dba9c Feed exports: edit note, fix typos 2017-05-10 16:45:15 -03:00
Kurt Peek 548a432951 Minor grammatical changes 2017-05-10 23:48:01 +05:00
yandongxu 4966dd7a7f Fix doc: open file with "wb" mode will get an error in python 3 2017-05-08 18:50:30 +08:00
Liu Siyuan 6c1cacb5d5 [MRG+1] doc: fix documentation error in link-extractor.rst (#2676) 
* fix doc error in link-extractor.rst

* remove the import clause

* update based on suggestion
 2017-05-06 02:47:06 +05:00
Martín Gaitán 4bc0c6b0f4 Update practices.rst 
fix a typo
 2017-04-26 14:12:57 +05:00
Paul Tremberth aa59cf4241 Merge pull request #2668 from harshasrinivas/docs-sphinx-rtd-theme 
[MRG+1] Add sphinx_rtd_theme to docs setup readme
 2017-04-24 12:35:40 +02:00
Tiago Cardoso 00ee9eaeaf Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting 2017-04-22 14:36:44 +01:00
Mikhail Korobov 9731077a54 Merge pull request #2683 from harshasrinivas/docs-SelectorList 
[MRG+1] Remove __nonzero__ from SelectorList docs
 2017-04-12 21:25:00 +05:00
Paul Tremberth 422b38f65c DOC Rearrange selector sections 2017-04-11 16:55:43 +02:00
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
harshasrinivas 2ff6b05723 Remove __nonzero__ from SelectorList docs 2017-03-24 20:43:28 +05:30
harshasrinivas 0298bcbe79 Update Makefile to open webbrowser in MacOS (#2661) 2017-03-24 18:13:08 +05:30
Qiwei Huang 8ecc307e8f Update spiders.rst 
Added note to allowed_domain attribute with an example explaining what goes in the list
 2017-03-20 19:37:07 -07:00
Qiwei Huang c5f74f7d1a Update spiders.rst 
Added a note to allowed_domains attribute, reminding users not to add urls into the list.
 2017-03-20 18:52:33 -07:00
harshasrinivas 83aa0c5e1a Clarify docs readme 2017-03-20 22:36:09 +05:30
harshasrinivas 4ec07ae764 Create docs/requirements.txt 2017-03-20 22:21:08 +05:30
harshasrinivas a57e49d55b Add sphinx_rtd_theme to docs setup readme 2017-03-20 19:49:31 +05:30
Oto Brglez 11cf6ad425 Comments for AWS_ENDPOINT_URL setting. 2017-03-19 12:48:06 +01:00
Oto Brglez 605691792f Updating media-pipeline docs for S3-like storage. 2017-03-19 12:35:39 +01:00
harshasrinivas 10741aca72 Update docs - improve clarity 2017-03-19 06:17:28 +05:30
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
harshasrinivas 9d97d788c0 Update docs for meta key 2017-03-15 04:13:47 +05:30
Eugenio Lacuesta 9c256cf693 Undocument _InvalidOutput exception 2017-03-10 15:41:57 -03:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00
jorenham ac63d3a3cf Removed contrib section; contrib is deprecated 2017-03-09 10:56:23 +01:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth a8b47d6c68 Update release notes for 1.0.7, 1.1.4 and 1.2.3 2017-03-06 14:25:52 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Bernardas 8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas 25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth 605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth 03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Eugenio Lacuesta 706ed0e049 Spider middleware: process_spider_exception on generators 2017-03-01 12:01:53 -03:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña e578be73b6 Merge pull request #2539 from scrapy/enable-memusage 
Enable memusage extension by default.
 2017-02-20 11:18:04 -03:00
Daniel Graña fab168bbfb Merge pull request #2572 from advarisk/warning-formrequest-from-response 
[MRG+1] document issue with FormRequest.from_response due to bug in lxml
 2017-02-20 11:14:28 -03:00
Daniel Graña 85ef6a6229 Merge pull request #2564 from elacuesta/docs_exporters 
[MRG+1] Doc: binary mode is required for exporters
 2017-02-20 11:12:48 -03:00
Paul Tremberth d35a01a103 Update default password 2017-02-20 14:23:23 +01:00
Paul Tremberth b80e1bb6c5 Document new FTP_* settings 2017-02-20 14:19:36 +01:00
Ashish Kulkarni 165e2cb8c9 document issue with FormRequest.from_response due to bug in lxml 
This can make the spider fail due to incorrect values being posted
server-side, which is extremely hard to debug because it is easy
to miss leading/trailing whitespace, even with a logging proxy.

The fix was merged for lxml 3.8 in lxml/lxml#228 so document that
as well.
 2017-02-17 14:54:22 +05:30
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d09eed7674 use w3lib.html.strip_html5_whitespace function; expand docs; strip consistently before calling process_value 2017-02-16 02:22:18 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov 5b79c6a679 DOC document response.follow methods; expand the tutorial 2017-02-16 00:06:52 +05:00
Mikhail Korobov 877057fac0 initial response.follow implementation 2017-02-15 01:22:53 +05:00
Eugenio Lacuesta 922d3fec54 Doc: binary mode is required for exporters 2017-02-14 12:51:03 -03:00
Mikhail Korobov afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
Daniel Graña 9315e944a2 Release notes for 1.3.2 2017-02-13 14:56:29 -03:00
Paul Tremberth 29e60213db Use consistent selectors for author field in tutorial 2017-02-09 10:41:21 +01:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth ff8a564b1a Set date for 1.3.1 release 2017-02-08 17:05:06 +01:00
Paul Tremberth 48c8c679de Update changelog for upcoming 1.3.1 release 2017-02-08 15:53:44 +01:00
Paul Tremberth d205206aaa Merge pull request #2345 from gustavodeandrade/master 
[MRG+1] Fix documentation about HTML entities decoding with selector extraction
 2017-02-08 13:21:15 +01:00
Paul Tremberth 7d0b89042f Merge pull request #2533 from djrobust/patch-1 
[MRG+1] Use yield with nested parsing of responses
 2017-02-08 13:02:50 +01:00
Mikhail Korobov 85a124970a Enable memusage extension by default. Fixes GH-2187. 2017-02-07 03:32:54 +05:00
Mikhail Korobov 3b8e6d4d82 Merge pull request #2531 from Lukas0907/patch-1 
Fix typo in downloader-middleware.rst.
 2017-02-06 13:09:35 +05:00
Takehiro Shiozaki fcb3daf4fa fix typo 2017-02-06 14:03:41 +09:00
djrobust 3021084f37 Use 'yield' when parsing multiple responses 
Use 'yield' consistently across examples of parse functions.
 2017-02-04 20:07:05 -08:00
Lukas Anzinger 09643796b4 Fix typo in downloader-middleware.rst. 2017-02-03 20:05:17 +01:00
Paul Tremberth 1c0b805357 DOC Mention XPath variables in Selectors section 2017-02-02 17:44:57 +01:00
Mikhail Korobov c305c46254 Merge pull request #2503 from redapple/view-no-redirect 
[MRG] Fix view command against new --no-redirect option
 2017-02-02 21:24:56 +05:00
Mikhail Korobov 55742c0392 DOC mention LevelDB cache storage backend 2017-02-01 22:43:28 +05:00
Paul Tremberth 4156a86148 Update docs on view command 2017-01-30 15:57:37 +01:00
Raul Gallegos 299544416a changing README to README.rst 2017-01-22 19:23:44 -05:00
Raul Gallegos df1a42419f adding formid to FormRequest documentation 2017-01-14 20:45:20 -05:00
Paulius Aleksiūnas 5586fc7e38 Update architecture.rst 
In the data flow image arrows are red.
 2017-01-10 10:40:27 +01:00
Paul Tremberth bf2277a028 Update spiders.rst 2017-01-10 10:27:38 +01:00
Νικόλαος-Διγενής Καραγιάννης 900b6710d3 Document copying of spider arguments to attributes 2017-01-09 22:12:38 +02:00
Elias Dorneles b6ab1ae9c3 docs: installation instructions, mention conda in the beginning (closes #2475) 2017-01-03 15:14:59 -02:00
nyov 2240f00a13 Remove dependency on os.environ from default settings 
Avoid loading settings from environment in scrapy core.
Instead it's better to populate them from the starting
shell or an embedding script.
 2016-12-26 20:49:16 +00:00
Paul Tremberth f8793e2460 Set release date for 1.3.0 2016-12-21 16:27:01 +01:00
Paul Tremberth b9e7ca044c Reword things a tiny bit 2016-12-21 16:11:19 +01:00
Paul Tremberth d60a6899df Merge logging changes into "New features" section 2016-12-21 16:01:36 +01:00
Paul Tremberth 4eeec3e42c Add preamble on why 1.3.0 comes so soon after 1.2.2 2016-12-21 15:59:18 +01:00
Paul Tremberth 49a84c2d41 Add note on HTTP redirects with shell and fetch 2016-12-21 14:49:16 +01:00
Paul Tremberth 9098001888 Add note on LOG_SHORT_NAMES 2016-12-21 14:32:59 +01:00
Paul Tremberth 9d5afd8c35 Add note on HttpErrorMiddleware new logging level 2016-12-21 14:32:59 +01:00
Paul Tremberth e9b3cf01f4 Update changelog for upcoming 1.3.0 release 2016-12-21 14:32:59 +01:00
Elias Dorneles d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Paul Tremberth 140a57d7b0 Amend note on --no-redirect option for shell tool 2016-12-19 17:54:54 +01:00
Mikhail Korobov da19f0b7b7 DOC how to override log level for a specific Scrapy component 2016-12-16 22:14:54 +05:00
Mikhail Korobov 0fc73a9d55 DOC update examples with long longger names 2016-12-16 21:47:58 +05:00
Mikhail Korobov 05cec0f2f3 fixed ReST syntax 2016-12-09 00:04:04 +05:00
Akhil Lb a75ad2bbc6 LOG_SHORT_NAMES option 2016-12-09 00:04:04 +05:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
Mikhail Korobov bd8c293a97 Merge pull request #2411 from redapple/remove-chunked-transfer-mw 
[MRG+1] Remove ChunkedTransferMiddleware from default settings
 2016-12-07 22:05:15 +05:00
Mikhail Korobov ff3aec6613 Merge pull request #2331 from moisesguimaraes/fixes-2272 
[MRG+1] Fixes issue #2272 using arg_to_iter() to wrap single values and list() to…
 2016-12-07 20:08:18 +05:00
Elias Dorneles a9c69458ff Merge pull request #2422 from rolando-contrib/nested-spiders-modules 
[MRG+1] DOC State explicitly that spiders are loaded recursively.
 2016-12-07 11:21:15 -02:00
Paul Tremberth 09e310d0b7 Set release date for 1.2.2 2016-12-06 15:03:38 +01:00
Paul Tremberth aa2e1b030d Add reference to fixed scheduler settings documentation 2016-12-06 14:59:11 +01:00
Paul Tremberth 89d5f5acd3 Update changelog for upcoming 1.2.2 release 2016-12-06 14:59:11 +01:00
Moisés Guimarães c08d278c0c removes note from docs. 2016-12-05 16:47:24 -03:00
Moisés Guimarães a4178f99da fixes params types in docs. 2016-12-05 15:24:26 -03:00
Moisés Guimarães c58ea021b8 fixes docs 2016-12-04 11:56:14 -03:00
Rolando Espinoza 6431e7a138 DOC State explicitly that spiders are loaded recursively. 2016-12-01 13:26:19 -03:00
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 059085b5b4 Remove docs for deprecated ChunkedTransfer middleware 2016-11-24 18:23:34 +01:00
Paul Tremberth 03cf5f1bd2 Remove ChunkedTransferMiddleware from default settings 2016-11-24 17:18:57 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Paul Tremberth 11fe3751cf DOC Remove "Ubuntu" section from sidebar/ToC 2016-11-16 11:55:09 +01:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Rolando Espinoza 76459e1969 Update conda channel to conda-forge and show conda version badge. 2016-11-08 21:30:27 -03:00
Rahul Kant f56aef99c2 Add closing tag in <tbody> 2016-11-07 17:49:22 +05:30
Gustavo de Andrade f2e49bc23c Update selectors.rst 
Decode instead escape, exceptions &lt; and &amp; (kmike)
Second sentence droped (Digenis)
 2016-11-01 21:32:17 -02:00
Mikhail Korobov 9755ef933a Merge pull request #2369 from jc0n/media-pipeline-docs-typo 
Fix typo in media pipeline docs
 2016-10-29 12:46:48 +06:00
John O'Connor d85da273be Fix typo in media pipeline docs 2016-10-28 19:44:46 -07:00
Elias Dorneles a9c74dbe42 Merge pull request #2339 from visued/master 
[MRG+1] Update documentation to explain the use of double quotes on Windows (fixes #2325)
 2016-10-25 09:53:24 -02:00
Steven Almeroth 99daea495b Doc: wording 2016-10-21 16:16:42 -07:00
Steven Almeroth a958e54954 Doc: remove trailing spaces 2016-10-21 16:16:37 -07:00
Randy Pen c7d245b90b update 
Thx for your advice.
 2016-10-21 10:52:50 +02:00
Randy Pen eacc5937e4 fix example code 
In the AuthorSpider, original css selector failed to get links of author pages
 2016-10-21 10:52:50 +02:00
Paul Tremberth c559a64c20 Set date for 1.2.1 release 2016-10-21 10:17:46 +02:00
Paul Tremberth 22772a61c8 Update release notes for upcoming 1.2.1 2016-10-21 10:17:04 +02:00
gustavodeandrade c7dfb5eb88 Fix issue 1704 2016-10-21 00:08:08 -02:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00
Victor Sued f74051e69e update documentation to explain the use of double quotes on Windows. 2016-10-18 16:36:43 -02:00
bopace fd016ee71b Fixed wording of documentation 2016-10-18 09:37:45 -06:00
Paul Tremberth 6cc83c041e Merge pull request #2330 from lfmattossch/note-python2-name 
[MRG+1] Added a note about invalid spider names in python 2 (fixes #2251)
 2016-10-18 16:35:19 +02:00
Jose Ricardo e12e364a40 Add details to the spider middlewares docs 
Document the effects of the middleware order in a more detailed way.
 2016-10-18 12:29:30 -02:00
Luiz Fernando Mattos Schlindwein 7c33e0cb55 added a note about invalid spider names in python 2 2016-10-18 11:55:51 -02:00
Jose Ricardo ea7bd39529 Make architecture overview references a little more clear on the docs 
Expliciting what actually happens by adding links to the respective methods
that are invoked in each processing phase.
 2016-10-18 11:50:51 -02:00
Jose Ricardo bebcd5081c Add downloader middleware ordering details to the docs 
Add more details, making it easier to understand what are the effects of
setting a downloader middleware order.
 2016-10-18 11:22:55 -02:00
Bo Pace bfe28ae707 Added documentation about accessing header values 2016-10-17 14:10:05 -06:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
Paul Tremberth 95c6b9dffd Merge pull request #2280 from scrapy/release-notes-1.2 
Update release notes for upcoming 1.2.0
 2016-10-03 13:00:14 +02:00
Paul Tremberth e2137d77ce Add release date for scrapy 1.2 2016-10-03 12:41:18 +02:00
Paul Tremberth a433126301 Merge pull request #2294 from redapple/release-headers 
Change release section titles to have correct links in ToC
 2016-10-03 12:38:44 +02:00
Wieland Hoffmann e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Paul Tremberth fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles 559b4edaec update release notes with changes from #1581 2016-09-30 11:42:50 -03:00
Elias Dorneles 8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Paul Tremberth fd0b6faecb Change release section titles to have correct links in ToC 2016-09-29 14:36:13 +02:00
Paul Tremberth c670e3cdf6 Move mention 2016-09-29 14:23:52 +02:00
Elias Dorneles 86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles 60559369b5 Merge pull request #2292 from stummjr/fix-example-section 
[MRG+1] Update examples section in documentation with quotes.toscrape spiders
 2016-09-29 09:11:45 -03:00
Paul Tremberth 33d04684e5 Fix typo 2016-09-29 10:25:21 +02:00
Elias Dorneles 3ac3ac4d92 docs: update data flow description and image (fixes: #2278) 
This fixes the explanation to use Requests instead of URLs,
which is what actually happens, and is also consistent with the
new tutorial, which already explains how URLs become Request objects.

I've also changed the "loop", jumping from 9 to step 2.
 2016-09-28 16:38:45 -03:00
Valdir Stumm Junior 5680c61063 Doc: update examples section with quotes.toscrape.com spiders 2016-09-28 15:45:01 -03:00
Paul Tremberth 23abf2d31b Fix title underline 2016-09-28 18:31:27 +02:00
Paul Tremberth 8d130b295a Update installation guide 2016-09-28 17:09:17 +02:00
Paul Tremberth c707c31345 Fix Debian number 2016-09-26 17:20:13 +02:00
pawelmhm 39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00
Paul Tremberth bba31b545d Mention new Debian Jessie baseline 2016-09-23 17:33:08 +02:00
Paul Tremberth 7eeee6f616 Mention \uXXXX escapes in regards to FEED_EXPORT_ENCODING 2016-09-23 16:39:09 +02:00
Paul Tremberth d52fbf2045 Merge remote-tracking branch 'origin/master' into release-notes-1.2 2016-09-23 16:34:06 +02:00
Paul Tremberth 80c1e5dc25 Set release date, fix typo and add tutorial improvement issue number 2016-09-23 16:11:50 +02:00
Paul Tremberth a0f87d2f45 Update release notes for upcoming 1.1.3 bugfix release 2016-09-23 16:11:50 +02:00
Pawel Miech b2bfd1e5c5 [docs] document that process_item can return Deferred 2016-09-23 10:36:03 +02:00
Paul Tremberth a975a50558 Merge pull request #2252 from eliasdorneles/tutorial-upgrades 
[MRG+2] Tutorial: rewrite tutorial seeking to improve learning path
 2016-09-22 16:39:21 +02:00
Elias Dorneles f4a2208916 addressing review comments and other minor editing 2016-09-22 11:04:45 -03:00
Paul Tremberth 2e08a9b412 Merge pull request #2271 from redapple/mailsender-lists 
[MRG+1] Add note on "to" and "cc" as lists for sending emails
 2016-09-22 12:00:14 +02:00
Elias Dorneles d636e5baa8 better description for start_requests expected return value 2016-09-21 18:54:12 -03:00
Elias Dorneles 32017a76f8 recommend learn python the hard way for beginners 2016-09-21 11:06:36 -03:00
Elias Dorneles 38266cc949 recommend Dive into Python and Python tutorial instead of LPTHW for non-beginners 2016-09-21 11:02:24 -03:00
Elias Dorneles c126c59361 address more review comments 2016-09-20 18:19:25 -03:00
Elias Dorneles a876ea5bd2 minor grammar fix 2016-09-20 15:10:49 -03:00
Elias Dorneles bc41fdf20e address review comments, add debug log to initial spider 2016-09-20 15:04:08 -03:00
Paul Tremberth 40293551b2 Remove mention of odd-numbered versions for development releases 
Fixes GH-1317
 2016-09-20 18:15:45 +02:00
Elias Dorneles 125b691102 more reviewing and editing, minor restructure, syntax fixes 2016-09-20 12:47:03 -03:00
Paul Tremberth e59d79bf37 Add note on "to" and "cc" as lists for sending emails 
Fixes GH-2244
 2016-09-20 17:18:49 +02:00
Elias Dorneles 8975371a57 Merge branch 'master' into tutorial-upgrades 2016-09-20 09:45:05 -03:00
Elias Dorneles f4f93c5c26 fix tox docs build, adjust title 2016-09-20 09:19:59 -03:00
Ashish Kulkarni 8893527a2f temporarily deprecate official Ubuntu packages 
They are not currently updated and fail to install on
Ubuntu 16.04. Also update the instructions to refer to
the earliest supported LTS (Ubuntu 12.04).

fixes #2137 and closes #2076
 2016-09-20 11:10:59 +02:00
Valdir Stumm Junior fee07835f2 Completing the data extraction section 2016-09-19 19:19:44 -03:00
Elias Dorneles 2a409d1d95 [wip] changing introduction to scraping with selectors 2016-09-19 17:13:04 -03:00
Elias Dorneles 063315258e Merge pull request #2202 from scrapy/doc-arch-overview2 
[MRG+1] DOC move Data Flow below the picture; add links to components
 2016-09-19 08:11:18 -03:00
Paul Tremberth bb8740a552 Update 1.2 release notes with latest changes 2016-09-19 12:26:36 +02:00
Wayne Lovely cc8497abb1 Fix a dict key in the tutorial 2016-09-17 11:09:28 +00:00
Valdir Stumm Junior 233b98d642 include section describing spider arguments 2016-09-16 18:08:10 -03:00
Elias Dorneles 31545a9f84 tutorial: updating extracting data section to introduce CSS and XPath equally 2016-09-16 17:13:24 -03:00
Elias Dorneles 147e75602d update after review comments (thanks @stummjr) 2016-09-16 16:47:24 -03:00
Elias Dorneles 31260cf02f mentions stackoverflow as help channel (fixes #2255) 2016-09-16 16:05:36 -03:00
Elias Dorneles 21de617c77 mention that spiders need to subclass scrapy.Spider 2016-09-16 15:55:14 -03:00
Elias Dorneles b2a5cddbb0 tutorial: update section about following links, expand examples 
adding an AuthorSpider to demonstrate further a different crawling
arrangement.
 2016-09-16 15:49:49 -03:00
Valdir Stumm Junior 0cd9dfcc85 small fixes on tutorial 2016-09-16 15:21:49 -03:00
Valdir Stumm Junior 0da497cf7a updates on the first section (our first spider) 2016-09-16 11:55:23 -03:00
Elias Dorneles c508f40689 use harcoded URLs, remove item reference on second spider 2016-09-15 18:05:09 -03:00
Elias Dorneles 2427791287 tutorial: remove item class definition and present start_requests first 
This changes the tutorial, removing the step of creating an item class
and also starts by presenting the start_requests method instead of
start_urls.
 2016-09-15 17:46:31 -03:00
Elias Dorneles 75531e409e use better condition in example spider 2016-09-15 16:56:13 -03:00
Elias Dorneles 1d159ae6f9 minor grammar fix 2016-09-15 15:37:03 -03:00
Elias Dorneles 18bd0b0886 docs: update overview spider code to use toscrape.com and minor changes 
So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
 2016-09-15 15:16:30 -03:00
Paul Tremberth 2f60f2a5a6 Merge pull request #2236 from stummjr/new-tutorial-toscrape 
[MRG+1] Update broken Scrapy tutorial to use quotes.toscrape.com
 2016-09-15 12:05:03 +02:00
Elias Dorneles a9a96bed8f updated tutorial as per review comments 2016-09-14 18:09:39 -03:00
Valdir Stumm Junior bc67cd9edd fix indentation issue 2016-09-14 12:39:29 -03:00
Valdir Stumm Junior 10f8c52f5d changed tutorial examples from dmoz to quotes.toscrape.com 2016-09-13 14:05:26 -03:00
Pengyu CHEN 114437c169 added: Doc for `scrapy.http.TextResponse.urljoin` 2016-09-10 10:38:36 +12:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Valdir Stumm Junior 9cea6f0730 Add Segment Analytics to Documentation 2016-09-02 14:51:07 -03:00
Paul Tremberth b188f61b95 Update release notes for upcoming 1.2.0 version 2016-09-01 17:38:38 +02:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00
Mikhail Korobov 495d322691 DOC move Data Flow below the picture; add links to components 2016-08-26 20:16:22 +05:00
Thom Dixon f68dc3026d Fix indentation 2016-08-24 09:11:27 -07:00
Thom Dixon 633abfbea1 Correct documentation about Response parameters 
This fixes issue #2196
 2016-08-24 08:47:52 -07:00
Valdir Stumm Junior d61650d843 fix RANDOMIZE_DOWNLOAD_DELAY description in the docs 2016-08-19 18:24:32 -03:00
Paul Tremberth f18c3e5ce5 Add release notes for 1.1.2 version 2016-08-19 17:01:57 +02:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Lorena 7d432872bf text updates to match graphic 2016-08-04 11:01:14 -07:00
Lorena 04f93e096c updated graph for organization/clarity 2016-08-04 10:04:47 -07:00
Paul Tremberth 5b1d98b8c8 Update 1.1.1 release date 2016-08-01 20:21:12 +02:00
Paul Tremberth 928e93f8f3 Update notes with latest 1.1 commits 2016-08-01 20:21:12 +02:00
Paul Tremberth e1d118d5ca Update release notes for upcoming 1.1.1 release 2016-08-01 20:21:12 +02:00
Elias Dorneles 34e7dadf38 Merge pull request #1610 from darshanime/scheduler_debug 
[MGR+1] Change, document `LOG_UNSERIALIZABLE_REQUESTS`
 2016-07-29 10:12:52 -03:00
darshanime d8e62e660d update log demo print 2016-07-26 20:46:12 +05:30
Paul Tremberth fe68a45c82 Merge pull request #2138 from jesuslosada/master 
[MRG+1] Apply the FILES_STORE_S3_ACL setting in ImagesPipeline
 2016-07-26 12:50:25 +02:00
Elias Dorneles b6375d3701 Merge pull request #1566 from darshanime/signal-example 
[MRG+1] Include example for signal docs
 2016-07-25 10:49:29 -03:00
Darshan Chaudhary a2e6452554 Include signal example 2016-07-25 18:41:25 +05:30
darshanime 0c77b6d033 update docs for settings 2016-07-25 18:05:51 +05:30
Darshan Chaudhary 472a8a47d0 Change name, log once 2016-07-25 18:03:43 +05:30
Jesús Losada 8221556089 Add documentation on Amazon S3 ACLs 2016-07-22 23:22:18 +02:00
Jesús Losada c6a2ca4e93 Document S3 capabilities in FilesPipeline and ImagesPipeline 2016-07-22 19:32:29 +02:00
Paul Tremberth ec1c61504a Merge pull request #2005 from feliperuhland/master 
[MRG+1] Included new optional parameter in startproject command line
 2016-07-19 12:31:06 +02:00
Kenneth Schnall 5fabed514e Update logging.rst 2016-07-15 23:14:11 -04:00
Mikhail Korobov 79639d0fec Merge pull request #1989 from pawelmhm/fix-images-pipeline-uppercase-other 
[MRG+1] [image_pipeline] bring back uppercase class attributes
 2016-07-13 14:44:00 +00:00
Kenneth Schnall 2489f84d39 Update practices.rst 2016-07-12 15:34:50 -04:00
Paul Tremberth c3109daa72 Merge pull request #2034 from dracony/master 
[MRG+1] Added option to turn off ensure_ascii for JSON exporters
 2016-07-12 17:01:09 +02:00
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Pawel Miech c22cc1096b [image_pipeline] style edits 
* 80 characters line limit
* shortening some code
* removed dead code
* add doctest for _key_for_pipe function
 2016-07-12 13:58:36 +02:00
Mikhail Korobov 5f261cf295 Merge pull request #2026 from Digenis/contributing-full_subjects 
[MRG+1] Encourage complete titles in pull requests
 2016-07-08 10:45:22 +06:00
Valdir Stumm Junior 081595a2e4 document new genspider behavior 2016-07-06 15:10:48 -03:00
Raul Gallegos 49ac7de231 prioritize default headers over user agent 2016-07-05 15:38:17 -05:00
Νικόλαος-Διγενής Καραγιάννης 2a92ffb409 Encourage descriptive PR titles 2016-07-05 14:21:28 +03:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Pawel Miech 73cc066caa [docs] add note about windows + python3 2016-06-17 09:21:58 +02:00
Pawel Miech 10b79c9b3e [files-pipeline] update docs with note about settings 
for subclasses.
 2016-06-15 15:49:11 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
Felipe Ruhland 089483aece
Updated docs for new option in command line 2016-05-23 23:16:15 -03:00
Paul Tremberth b3367c7acd DOC Add info and example on errbacks 2016-05-18 18:00:09 +02:00
Paul Tremberth c2c8036a97 DOC Update copyright notice 2016-05-18 16:52:09 +02:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Paul Tremberth 149c4cd41e [DOC][Overview] Use idiomatic .extract_first() 2016-05-17 17:53:40 +02:00
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Paul Tremberth 2849ebf4c6 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 14:07:33 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
nblock a3557dd34d Fix spelling mistake 2016-04-11 14:06:57 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
djunzu 6988e9cd4b Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-01 21:51:15 -03:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
Paul Tremberth 9d8c368ce8 Merge pull request #1879 from scrapy/scrapy-arch-docs 
DOC improved Architecture overview
 2016-03-31 12:09:24 +02:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Lele 7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Mikhail Korobov 4f335b5a01 DOC clarify Architecture docs 2016-03-25 17:03:41 +05:00
Mikhail Korobov 3ca977a8cb DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-25 07:11:33 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin e7c4806c5e Update feedstorage docs: add botocore, mention that boto is supported only on Python 2 2016-02-15 19:59:48 +03:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Nicolas Pennequin 6efc7a9de0 Update the email doc for the charset argument to send(). 2016-02-04 19:33:44 +01:00
Elias Dorneles ca08314654 makes more explicit source of problems in Python 3 limitations 2016-02-03 21:38:25 -02:00
Elias Dorneles 2167573b32 fix comment about disabling robots.txt 2016-02-03 21:31:27 -02:00
Elias Dorneles 3180abc800 applying more review comments 2016-02-03 21:28:05 -02:00
Elias Dorneles d40add7b27 add note about robots.txt waiting and make it explicit builtin extensions only are ported 2016-02-03 21:18:25 -02:00
Elias Dorneles 9cfefd5277 favor beta over basic to describe PY3 support 2016-02-03 20:59:56 -02:00
Elias Dorneles 2f697ce59f added note about how to revert to old behavior for robotstxt 2016-02-03 20:51:26 -02:00
Elias Dorneles 89a088b3c3 applying review comments 2016-02-03 20:45:55 -02:00
Paul Tremberth 972f192568 Blank line before CoC 2016-02-03 22:58:49 +01:00
Paul Tremberth 728e505e53 Merge remote-tracking branch 'origin/master' into release-notes-1.1 2016-02-03 22:47:15 +01:00
Valdir Stumm Jr dd7b0eb359 Updates to release notes 2016-02-03 19:23:02 -02:00
Paul Tremberth 51dc741fa2 Add 1.0.5 release notes 2016-02-03 21:07:21 +01:00
Valdir Stumm Jr 0f816524e8 edit API changes, Deprecations, and Relocations sections. 2016-02-03 16:21:32 -02:00
Paul Tremberth fd3193da86 Reword some of "New Features and Enhancements" 2016-02-03 18:19:38 +01:00
Paul Tremberth 3b2a6abc0b Format "Code refactoring" subsection 2016-02-03 17:37:58 +01:00
Paul Tremberth ed02455010 Format "Logging" subsection 2016-02-03 17:27:43 +01:00
Valdir Stumm Jr 1fd95c9c9f fixed references to issues and fit text to 80 cols in the first two sections 2016-02-03 12:45:48 -02:00
Valdir Stumm Jr a21c90bee7 edits on Py3 Support and New features sections 2016-02-03 11:54:46 -02:00
Paul Tremberth 44d8df2060 Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS 2016-02-03 12:35:26 +01:00
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Paul Tremberth 142aa21737 Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note 2016-02-03 12:33:10 +01:00
Paul Tremberth db0697bc06 Add 1.1 release notes (draft) 2016-02-03 12:32:40 +01:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
Mikhail Korobov 7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Elias Dorneles d4c4ca8062 fix version number to appear new feature 2016-01-21 09:42:15 -02:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Daniel Graña b1e44436bc Merge pull request #1642 from orangain/doc-metarefresh 
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
 2016-01-19 13:33:50 -03:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
Daniel Graña 1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
orangain 4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Rolando Espinoza f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Pablo Hoffman eb4daa34a2 Merge pull request #1544 from rolando/conda-docs 
Added installation notes about using Conda for Windows.
 2015-11-26 14:11:28 -03:00
Rolando Espinoza cc8ddb68b0 Added installation notes about using Conda for Windows and other OSes. 2015-11-26 13:08:56 -04:00
Julia Medina 48539af163 Merge pull request #1601 from mvj3/minor-documentation-fixes 
Fixed minor grammar issues.
 2015-11-16 14:58:07 -03:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Jakob de Maeyer 18c7f3dbe2 Document built-in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 4ac6a83072 Fix class signatures in Extensions docs 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 2946b67414 Document add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d8af395d76 Introduce add-ons via AddonManager and Addon base class 2015-11-11 21:08:19 +01:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Valdir Stumm Jr d577c4702d fixed a typo in the documentation. 2015-10-26 00:00:20 -02:00
Christian Pedersen cc76fb3dff Version 1 now exists 2015-10-14 14:07:34 +02:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
hoatle 2869cf8dde fix another invalid xpath error 2015-10-07 16:03:43 +07:00
Hoat Le 4e66955411 fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst 
>>> response.xpath('//div/[id="not-exists"]/text()').extract_first() is None
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/http/response/text.py", line 109, in xpath
    return self.selector.xpath(query)
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/selector/unified.py", line 100, in xpath
    raise ValueError(msg if six.PY3 else msg.encode("unicode_escape"))
ValueError: Invalid XPath: //div/[id="not-exists"]/text()
 2015-10-07 15:43:02 +07:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
smirecki 8379bea7ed Typos corrections 
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.

 Please enter the commit message for your changes. Lines starting
 2015-10-02 23:48:27 -04:00
Marius Gedminas 0620e76433 Fix list formatting 2015-09-29 03:33:30 +05:00
Marius Gedminas eaad10facf Typo 2015-09-28 14:44:15 +05:00
Daniel Graña 3c596dcf46 Merge pull request #1467 from dacjames/master 
add support for a nested loaders
 2015-09-16 22:03:19 -03:00
hy 14f7f22555 fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware 2015-09-16 16:59:23 +08:00
Daniel Collins 036109e7de updte nested loader documentation 2015-09-15 23:49:35 -07:00
Mikhail Korobov 600f27c39b Merge pull request #1350 from scrapy/installation-docs-update 
Updates in Installation and Contributing sections in the docs
 2015-09-09 17:54:35 +05:00
Julia Medina eb7b9d506f Add note to ubuntu install section about debian compatibility 2015-09-08 18:49:44 -03:00
Daniel Collins 88c92cb68b provide documentation for nested loaders 2015-09-04 13:15:48 -07:00
Daniel Graña dd473145f2 Merge pull request #1473 from ArturGaspar/master 
[MRG+1] Support for returning deferreds in middlewares
 2015-09-02 13:04:26 -03:00
Robert Weindl 833efbfcbd Update tutorial.rst 
Add missing "scrapy." prefix.
 2015-09-02 20:35:23 +05:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Artur Gaspar ca83a0b028 Support for returning deferreds in downloader middleware methods. 2015-09-01 13:22:43 -03:00
Daniel Graña b4e382c867 Merge pull request #1318 from yarikoptic/fix-docs-explicit-rtd-theme-path 
[MRG+1] DOC(ENH): specify path to rtd theme explicitly
 2015-08-31 10:42:02 -03:00
David Tagatac 08162a15d8 minor: scrapy.Spider docs grammar 2015-08-27 17:37:16 -04:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Rick aa88fd797a Small grammatical change 2015-08-26 15:01:50 -03:00
Mikhail Korobov cfae62f9cc Merge pull request #1441 from aivarsk/fix-common-practices 
Make common practices sample code match the comments
 2015-08-23 17:36:09 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Aivars Kalvāns b8b1e8e544 Make common practices sample code match the comments 2015-08-19 16:54:10 +03:00
Julia Medina d3c3408cd4 Replace alternative OSX install workaround with virtualenv 2015-08-19 03:19:27 -03:00
Julia Medina 46ee1c6ee2 Reference Homebrew's homepage for installation instructions 2015-08-19 02:21:08 -03:00
Daniel Graña 54f9837804 Document how to get coverage report 2015-08-13 19:02:51 -03:00
Daniel Graña f67a808d9e Add 1.0.3 release notes 2015-08-11 13:49:49 -03:00
Daniel Graña 57fafc702d Add 1.0.2 release notes 2015-08-06 18:02:12 -03:00
Mikhail Lyundin ea4199f8e1 Minor method name fix 2015-08-05 22:42:53 +03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
David Tagatac 08123207c5 minor: scrapy.Spider grammar and clarity 2015-07-31 17:01:59 -04:00
Mikhail Korobov b6eb3404a2 Merge pull request #1335 from jdemaeyer/fix/docs-scrapy-path 
Make Sphinx autodoc use local, not system-wide Scrapy
 2015-07-30 15:33:02 +05:00
Jakob de Maeyer 9c12a3f23a Remove incorrect path for Sphinx autodoc 2015-07-30 12:12:08 +02:00
Andrew Murray 198be7d679 Fixed typos 2015-07-16 23:59:02 +10:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina 320d3a59b9 Add oldest supported tox version to contributing docs 
Better fix for #1337
 2015-07-10 20:34:44 -03:00
Julia Medina 439b376d5b Note in install docs about pip being already included in python>=2.7.9 2015-07-10 20:34:43 -03:00
Julia Medina 5723e6b457 Add non-python dependencies to Ubuntu install section in the docs 
Closes #1314 and closes #1198.
 2015-07-10 20:34:43 -03:00
Julia Medina 290ebee090 Add OS X installation section to docs 
Closes #1342 and (possibly) #1126
 2015-07-10 20:34:43 -03:00
Nikola Pavlović 7bef61dbbd Fix reference. 2015-07-10 02:35:27 +02:00
Jakob de Maeyer d438075a47 Make Sphinx autodoc use local, not system-wide Scrapy 2015-07-03 16:51:53 +02:00
Julia Medina 8b3ca4f250 Merge pull request #1302 from eliasdorneles/improving-access-settings-docs 
[MRG+1] Improvements for docs on how to access settings
 2015-07-03 00:56:32 -03:00
Daniel Graña 3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Daniel Graña 6fa3f2421c Add 1.0.1 release notes 2015-07-01 01:49:44 -03:00
Mikhail Korobov d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Mikhail Korobov 63317531f9 DOC fix authrottle docs 
see https://github.com/scrapy/scrapy/pull/502/files#r8574692
 2015-06-26 20:47:58 +05:00
Yaroslav Halchenko 5f3e1e9a16 ENH: make explicit build-ignore-errors to be used by linkfix 2015-06-25 21:30:35 -04:00
Yaroslav Halchenko 48582be9a6 DOC(ENH): specify path to rtd theme explicitly 2015-06-25 10:46:22 -04:00
Yaroslav Halchenko 49fe915798 BF: fail if docs failed to build 2015-06-25 10:12:56 -04:00
Marco DallaG 44c8ef2d47 DOC Bring Ubuntu and Archlinux outside of Windows subsection 
In the installation guide, in the "Platform specific installation notes" section, Ubuntu and ArchLinux are currently subsections of Windows, which does not make sense imho.
This commit changes the section tree from:

Platform specific installation notes
- Windows
  -- Archlinux
  -- Ubuntu

To:

Platform specific installation notes
- Windows
- Ubuntu
- Archlinux
 2015-06-24 11:44:48 +02:00
Pablo Hoffman 38e5bfb61c remove version suffix from ubuntu package 2015-06-22 10:57:24 -03:00
Julia Medina af97c373ea Update release date for 1.0 
(cherry picked from commit c89fa29c41)
 2015-06-19 18:53:54 -03:00
Elias Dorneles 2de5c66058 improvements for docs on how to access settings 2015-06-15 13:07:55 -03:00
Julia Medina f958fb9d62 Update release notes for 1.0.0rc3 2015-06-12 16:42:39 -03:00
Julia Medina fa1c25c840 Merge pull request #1286 from scrapy/configure_logging 
configure_logging: change the meaning of settings=None
 2015-06-12 13:22:42 -03:00
Julia Medina 36bc912cdd DOC indent additional docs for configure_logging 2015-06-12 13:00:31 -03:00
Bryan Crowe 6a4c475e87 Fix a couple typos 2015-06-11 19:47:30 -04:00
Daniel Graña 5bd0395be4 Merge pull request #1291 from scrapy/signalmanager-docstrings 
DOC SignalManager docstrings. See GH-713.
 2015-06-10 16:28:35 -03:00
Mikhail Korobov 6c9daf3a95 DOC remove unnecessary links; fix references in send_catch_log_deferred docstring 2015-06-10 01:44:19 +05:00
Mikhail Korobov a611f8dd2d DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring 2015-06-09 22:57:18 +05:00
Mikhail Korobov 790c67b643 DOC spider_error doesn't support deferreds 2015-06-09 02:20:10 +05:00
Mikhail Korobov 1740fcf1a6 DOC SignalManager docstrings. See GH-713. 
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
 2015-06-08 21:05:58 +05:00
Mikhail Korobov 9a787893e3 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
 2015-06-08 19:54:18 +05:00
Chris Nilsson 0c532baf4c Removed typo, and clarified time unit of setting 2015-06-06 11:18:13 +10:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Julia Medina 24d8a85269 Update release notes for 1.0.0rc2 
(cherry picked from commit 6e61d54168)
 2015-06-05 17:11:40 -03:00
Chris Nilsson eae25a04d9 Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. 
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
 2015-06-06 00:39:14 +10:00
Julia Medina 367ea81e71 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 04:11:23 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Marven Sanchez 8771d1f79b Update HTTPCache middleware docs 2015-06-01 18:20:59 +08:00
Mikhail Korobov 342cb622f1 DOC fix non-working link (by removing it). 
See https://github.com/scrapy/scrapy/pull/1260
 2015-05-27 23:04:58 +05:00
Julia Medina 343d20d791 Update 1.0 release notes 2015-05-27 11:53:54 -03:00
Pablo Hoffman 545c4224f9 update old crawlera link 2015-05-25 16:01:54 -03:00
Daniel Graña d439c26d76 update docstring and release notes 2015-05-22 20:00:58 -03:00
Julia Medina afcf70cdc6 Add 1.0 release notes 2015-05-22 12:53:11 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Julia Medina 7c61bd897c Fix in docs for error introduced in #1218 2015-05-14 20:09:19 -03:00
Daniel Graña 1195f906d6 Merge pull request #1218 from Curita/move-base-to-packages 
Move base classes to their packages
 2015-05-14 18:28:17 -03:00
Pablo Hoffman 65aa9ccc70 Merge pull request #1220 from eliasdorneles/building-settingslist-from-docs 
Building settings list from docs
 2015-05-13 14:43:19 -03:00
Elias Dorneles 6884b73e19 sort settings list by name and wrap it in a bullet list 2015-05-11 22:11:30 -03:00
Elias Dorneles 5753e498bf fixes referencing, and list only settings not documented in current document 2015-05-09 16:15:06 -03:00
Julia Medina 42c2c4b6ce Wrong topic link in docs/intro/overview.rst 2015-05-09 05:23:12 -03:00
Julia Medina c271d8f0b1 Title underline too short in docs/topics/selectors.rst 2015-05-09 05:20:54 -03:00
Julia Medina 6fd7d85448 Wrong bullet list indentation in docs/topics/media-pipeline.rst 2015-05-09 05:19:15 -03:00
Julia Medina acc13c9821 Delete tab used as indentation in docs/topics/loaders.rst 2015-05-09 05:15:17 -03:00
Julia Medina 819a8eceee Mark as orphan the doc topics not listed in the index 2015-05-09 05:12:35 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d72536688f Move scrapy/linkextractor.py to scrapy/linkextractors/__init__.py 2015-05-09 03:28:37 -03:00
bosnj 8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Elias Dorneles d1053d2a2f automatic settings list for docs [WIP] 2015-05-04 02:17:19 -03:00
Pablo Hoffman d00e43f39c restore wrongly commented line in sphinx conf.py 2015-05-01 05:24:49 -03:00
Pablo Hoffman b84b3585f2 remove empty experimental section from doc 2015-05-01 01:53:51 -03:00
Pablo Hoffman a5a8f82b06 move documentation about registering commands via setup.py to commands doc 2015-05-01 01:51:56 -03:00
Pablo Hoffman 9ee17fd537 add note to doc README about using 'make watch' 2015-05-01 01:39:03 -03:00
Julia Medina c1634e4914 Add CrawlerProcess to "Running multiple spiders[...]" doc section 2015-04-29 23:09:20 -03:00
Julia Medina d6a06a769f Add CrawlerProcess to "Run Scrapy from a script" doc section 2015-04-29 23:09:20 -03:00
Julia Medina 3a71504d35 Extend CrawlerProcess documentation 2015-04-29 23:09:20 -03:00
Julia Medina a2ce78db63 Use autoclass for CrawlerRunner in docs/topics/api.rst 2015-04-29 23:09:20 -03:00
Julia Medina 353672d292 Add autodocs Sphinx extension 2015-04-29 23:09:20 -03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Wieland Hoffmann de6501ed1b Remove a duplicate word 2015-04-29 22:31:48 +02:00
Pablo Hoffman 3d2b74a6ff Merge pull request #1188 from eliasdorneles/favoring_web_scraping_over_screen_scraping 
[MRG+1] Favoring web scraping over screen scraping in the descriptions
 2015-04-29 16:49:43 -03:00
Elias Dorneles 3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
Julia Medina 7a92dae4c8 Change Scrapy log output through docs 2015-04-22 17:27:24 -03:00
Julia Medina 69a3d58110 Basic example on manually configuring log handlers 2015-04-22 17:24:41 -03:00
Julia Medina bd0b639b21 Fix logging usage across docs 2015-04-22 17:24:41 -03:00
Julia Medina 4811d16f1d Update `logger` attr and `log` method in the Spiders topic on docs 2015-04-22 17:24:41 -03:00
Julia Medina d47a7edc65 Update Logging topic on docs 2015-04-22 17:24:40 -03:00
Julia Medina ccdd8bfbcc Parametrize log formatting strings 2015-04-22 17:24:40 -03:00
Pablo Hoffman 0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Pablo Hoffman 4c4eb4f77d Merge pull request #1180 from eliasdorneles/tutorial-improvements 
Some improvements for Scrapy tutorial
 2015-04-21 15:49:39 -03:00
Daniel Graña e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman 06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Pablo Hoffman e4122cdd79 Merge pull request #1150 from eliasdorneles/docs-files-pipeline 
Documenting Files Pipeline together with Images Pipeline
 2015-04-21 15:28:47 -03:00
Elias Dorneles d63567531d change data extraction in crawl example to be consistent with tutorial, removed statement implying mandatory usage of Item 2015-04-21 11:30:48 -03:00
Elias Dorneles f7da69d116 fixing example CSS expr 2015-04-21 11:19:10 -03:00
Elias Dorneles ff007afb9d expanded crawling primer with examples, and applied other suggestions from the review 2015-04-21 10:57:44 -03:00
Elias Dorneles 595146e158 some improvements for Scrapy tutorial 2015-04-20 21:10:07 -03:00
nyov 7ae37d61f9 remove scrapy deploy command 
(closes #1027, #1095, #1102)
 2015-04-20 20:35:06 -03:00
Julia Medina 6da091f162 Remove djangoitem topic from index 2015-04-20 20:19:28 -03:00
Daniel Graña 16a1a938c1 Add 0.24.6 release notes 2015-04-20 01:40:21 -03:00
Mikhail Korobov 1794a893f4 Merge pull request #1172 from bagratte/docs 
minor corrections in documentation.
 2015-04-19 21:41:29 +05:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
bagratte 1312bcd007 minor corrections in documentation. 2015-04-19 18:58:15 +04:00
bagratte beea9267a1 minor corrections in documentation. 2015-04-18 19:48:25 +04:00
rajathkumarmp 02629b5f7b Added link to ipython in doc. 2015-04-18 13:00:34 +05:30
Pablo Hoffman 757309efea Merge pull request #1169 from bagratte/docs 
add some minor stylistic and grammar corrections to tutorial.rst.
 2015-04-17 19:52:11 -03:00
José Ricardo dffc4549d7 Fix small typo in the docs 2015-04-17 23:37:12 +05:00
bagratte 8d339da4e5 add some minor stylistic and grammar corrections to tutorial.rst. 2015-04-17 20:55:02 +04:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
Elias Dorneles a1f3b3c7f0 images.rst -> media-pipeline.rst 2015-04-15 14:26:08 -03:00
Elias Dorneles fd1c688ad6 addressing issues from review at #1150 2015-04-15 14:21:48 -03:00
Pablo Hoffman fb85bd4b10 Merge pull request #1132 from sibiryakov/dns-options2 
[MRG+1] Dns cache size and timeout options
 2015-04-15 14:08:19 -03:00
Mikhail Korobov cb5280ab0d DOC tweak deployment docs 2015-04-15 19:25:22 +05:00
Nicolás Alejandro Ramírez Quiros e054b3c45f Merge pull request #1160 from scrapy/docs-theme 
Change local docs theme
 2015-04-15 10:34:19 -03:00
Mikhail Korobov a11bd3e89d DOC update docs readme file - sphinx >= 1.3 is now required. 2015-04-15 11:50:57 +05:00
Mikhail Korobov dd84f4bd9a DOC remove css file which isno longer needed 2015-04-15 03:44:49 +05:00
Mikhail Korobov 7dcd7f3ca7 DOC make local docs use the same theme as readthedocs.org 
sphinx_rtd_theme is builtin in recent Sphinx versions.
 2015-04-15 03:43:30 +05:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov 7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Daniel Graña e85679fa6b Merge pull request #1147 from Curita/pass-crawlers-to-runner-crawl-calls 
Allow passing Crawler instances directly to CrawlerRunner.crawl()
 2015-04-13 14:23:15 -03:00
Daniel Graña f71175a449 More replaces of references to old `sel` shortcut 2015-04-12 21:56:23 -03:00
Diana Huang 91a60d9f3d Remove references to the `sel` object in shell.rst 
The current documentation has references to the deprecated `sel` when interacting with the shell. I've removed them and replaced uses of `sel.xpath` with `response.xpath` instead.
 2015-04-12 13:44:32 -04:00
Elias Dorneles dce48b8654 documenting Files Pipeline together with Images Pipeline 2015-04-11 13:57:55 -03:00
Julia Medina 3dabde6706 Update docs for CrawlerRunner.crawl() new usage 2015-04-10 12:09:07 -03:00
Julia Medina 86b0951327 Delete `crawler_deferreds` doc in CrawlerRunner 
This attribute is now an internal one since it's main use-case was
covered by CrawlerRunner.stop().
 2015-04-10 12:06:27 -03:00
Pablo Hoffman 39fd2e62ef fix typo in deploy.rst 2015-04-09 18:01:11 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Pablo Hoffman 9ea309c354 rename deployment.rst -> deploy.rst (consist with others like debug.rst) 2015-04-09 16:56:35 -03:00
Pablo Hoffman d8184a7239 Merge pull request #1124 from rdowinton/deployment-doc 
Added deployment section covering scrapyd-deploy and shub
 2015-04-09 16:53:25 -03:00
Alexander Sibiryakov 85aa3c7596 Dns cache size and timeout options 2015-04-02 18:30:59 +02:00
nyov 92b574309e documentation build warning fixes 2015-04-01 19:46:21 +00:00
Daniel Graña 27591b55fc Merge pull request #1123 from sibiryakov/reactor-threadpool-size 
[MRG+1] Reactor threadpool max size setting
 2015-04-01 15:08:03 -03:00
Alexander Sibiryakov b794cdaf4b Broad crawls notes. 2015-04-01 12:07:03 +02:00
Alexander Sibiryakov e7b274edf3 Reformat to 80 characters per line. 2015-04-01 11:49:55 +02:00
Alexander Sibiryakov 94fceb4c15 Fixing underscore size. 2015-04-01 11:25:10 +02:00
Richard Dowinton 2d142d6401 Added deployment section covering scrapyd-deploy and shub 2015-03-31 12:17:31 +01:00
Alexander Sibiryakov 5864d291d4 Setting documentation. 2015-03-31 11:10:56 +02:00
Elias Dorneles 18c5bc75c2 some more minor grammar fixes 2015-03-27 21:00:21 -03:00
Elias Dorneles 7135c4e694 some minor grammar fixes 2015-03-27 17:56:56 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
Daniel Graña 55a23d102f Merge pull request #1086 from Curita/response-urljoin 
Add Response.urljoin() helper
 2015-03-27 15:17:54 -03:00
Julia Medina f4e241a018 Merge pull request #1106 from eliasdorneles/overview-page-improvements 
[MRG+1] some improvements to overview page
 2015-03-27 15:16:33 -03:00
Mikhail Korobov 39085ae18f Merge pull request #1098 from nyov/nyov/userconfig 
[+1 MRG]look in ~/.config/scrapy.cfg for user config
 2015-03-27 02:11:35 +05:00
nyov 1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
Peter Bronez 475766c73a Converted sel.xpath() calls to response.xpath() in Extracting the data 2015-03-26 15:34:30 -04:00
Elias Dorneles 4dcecc98f9 moved example data to a better place 2015-03-26 15:45:17 -03:00
Elias Dorneles 7402e27230 fix community link 2015-03-26 15:35:31 -03:00
Elias Dorneles 729861c864 fixing indentation 2015-03-26 15:31:42 -03:00
Elias Dorneles 13d0ecde77 addressing more review comments, to avoid ambiguity on desired reading flow 2015-03-26 15:26:16 -03:00
Elias Dorneles 76e3bf1250 addressing comments from the review plus further editing 2015-03-26 14:26:20 -03:00
Elias Dorneles 8f4a268f37 added bit about async requests, improved phrasing 2015-03-26 12:14:56 -03:00
Elias Dorneles 32423d4a33 some improvements to overview page 2015-03-25 19:27:52 -03:00
Ramiro Morales 933dbc6be6 Oops 2015-03-25 18:33:17 -03:00
Ramiro Morales ca2575001e Add missing callback arg in jobs topic example. 2015-03-25 18:32:20 -03:00
Mikhail Korobov 5ac91e4883 DOC remove Dynamic Creation of Item Classes section 
It was a hack, and dicts-as-items cover most use cases.

Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
 2015-03-23 18:11:35 +05:00
Julia Medina cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Pablo Hoffman c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Mikhail Korobov 8ac397670f DOC move .. module: declaration to a proper place 2015-03-19 21:41:36 +05:00
Faisal Anees 643984e1b4 Updated architecture.rst 
Added http://krondo.com/blog/?page_id=1327 as a resource
 2015-03-18 23:55:22 -03:00
Mikhail Korobov f16a33f34e DOC change structure of spider docs: 
* start with scrapy.Spider, then mention spider arguments,
  then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
  moved below Spider docs.
 2015-03-19 05:25:15 +05:00
Mikhail Korobov 817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Julia Medina 959aaad205 Document `re_first` 2015-03-18 21:11:08 -03:00
Mateusz Golewski 127c6c694a Fix extract_first() docs 2015-03-18 21:11:08 -03:00
Mateusz Golewski 012211accd Add docs for extract_first() 2015-03-18 21:11:08 -03:00
Daniel Graña 12eedd90d1 fix truncated 0.24.5 release notes. closes #1084 2015-03-18 20:46:28 -03:00
Julia Medina 4fb818a250 Run linkfix over current docs 2015-03-18 20:04:14 -03:00
Julia Medina ff987fb5a5 Add linkfix rule to docs Makefile 2015-03-18 20:00:03 -03:00
Shadab Zafar c05f5f175e Added linkfix script to docs/utils 
https://github.com/scrapy/scrapy/pull/1041#issuecomment-78143576
 2015-03-18 19:41:26 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov c1bf8365d5 Merge pull request #1022 from scrapy/docs-reorder 
DOC reorder topics
 2015-03-13 16:40:47 +05:00
Mikhail Korobov baf5c59386 Merge pull request #1071 from eliasdorneles/updating-request-meta-special-keys 
updating list of Request.meta special keys
 2015-03-13 16:38:19 +05:00
Elias Dorneles 57a5ee0097 added example value to set for proxy meta key 2015-03-12 23:20:44 -03:00
Elias Dorneles f7031c08ff updating list of Request.meta special keys 2015-03-10 22:29:07 -03:00
Daniel Graña c9d7386a32 Add 0.24.5 release notes 2015-02-25 19:36:40 -02:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Mikhail Korobov 6419f76bbc Merge pull request #1029 from dangra/contribute-to-master 
DOC contribute to master branch
 2015-02-03 02:10:47 +05:00
Nicolás Alejandro Ramírez Quiros 8a3b9b6131 Merge pull request #1011 from SudShekhar/master 
Extension example fix to something that makes more sense
 2015-01-30 15:45:52 -02:00
Sudhanshu Shekhar e42a1ac1a1 Reset items_scraped instead of item_count 
items_scraped is the counter that needs to be reset each time we have scraped a specific number of items in the code instead of item_count (which represents the specific number of items needed before a message is logged). Updating the source code to reflect this.
Removed some irrelevant words from the log message.
Signed-off-by: Sudhanshu Shekhar <sudshekhar02@gmail.com>
 2015-01-30 23:13:06 +05:30
Daniel Graña f0bdc14522 Tentative attention message about what document to read for contributions 2015-01-29 15:40:12 -02:00
Mikhail Korobov a07b4353d6 DOC reorder topics 
* duplicate "topics/commands" link is removed;
* Request/Response docs are moved to "Basic Concepts";
* settings docs are moved to "Basic Concepts";
* exceptions docs are moved to "Basic Concepts";
* "signals" and "exporters" docs are moved to "Extending Scrapy";
* "Reference" section is dropped because it is empty now.
 2015-01-22 22:58:10 +05:00
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Capi Etheriel 4bc14da59e Updates documentation on dynamic item classes. 
Fixes #398
 2015-01-19 17:21:56 -02:00
Mikhail Korobov 283d6a5344 DOC a couple more references are fixed 2015-01-19 22:07:03 +05:00
Mikhail Korobov 73e6b35622 DOC fix a reference 2015-01-19 22:02:46 +05:00
Artur Gaspar b0730a1d16 documentation for CSS support in link extractors 2014-12-11 18:22:08 -02:00
Stefan 3602fc4fcb fixed the variable types in mailsender documentation 2014-12-10 22:48:09 +01:00
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Jeff Paine b422312a38 Update docs copyright year range 2014-11-09 21:08:27 -05:00
Lazar-T 13f83f0da0 typo 2014-11-10 06:28:41 +05:00
HalfCrazy b21a28cc9a Afterwords->Afterwards 2014-11-10 06:28:09 +05:00
Daniel Graña 2c67bd6c57 pywin32 is required by Twisted. closes #937 
see:
* http://twistedmatrix.com/trac/ticket/6032
* https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028
 2014-11-05 23:05:54 -02:00
Daniel Graña 6cb8995731 Update install.rst 
fixes #937
 2014-11-05 22:50:52 -02:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Lazar-T 38dcf50cd6 comma instead of fullstop 2014-10-25 09:19:50 +06:00
Pablo Hoffman 675fd5ba04 Merge pull request #898 from scrapy/download-timeout 
[MRG] DOC document download_timeout
 2014-10-24 16:52:42 -02:00
Pablo Hoffman 0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Mikhail Korobov 7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Mikhail Korobov ea3b372b4f DOC typo fix in leaks.rst 2014-10-02 15:20:13 +06:00
Pablo Hoffman e7843d35de Merge pull request #894 from kmike/leaks-docs 
Leaks docs
 2014-10-02 01:14:54 -03:00
Pablo Hoffman 5835224eee Merge pull request #896 from scrapy/robotstxt-once 
[MRG] process robots.txt once
 2014-10-02 00:58:55 -03:00
Mikhail Korobov 6fcf9dce50 DOC document from_crawler method for item pipelines; add an example. 2014-09-25 03:13:51 +06:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov bdbca1e2d7 DOC request queue memory usage 2014-09-21 07:30:44 +06:00
Mikhail Korobov bc0f481a73 DOC bring back notes about multiple spiders per process because it is now documented how to do that 2014-09-21 07:12:01 +06:00
Mikhail Korobov a122fdbfea Update leaks.rst: there is now only a single spider in a process. 2014-09-21 06:54:00 +06:00
Mikhail Korobov e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
John-Scott Atlakson a312ebfb43 Update request-response.rst 
Fixed minor typo
 2014-09-14 22:06:31 +06:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Julia Medina 16e62e9c9b Per-spider settings documentation 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
Mikhail Korobov 774ab74ad2 Merge pull request #864 from younghz/master 
Duplicate comma in request-response.rst
 2014-08-28 18:52:51 +06:00
Uyounghz d49766a6ac Duplicate comma in request-response.rst 2014-08-28 19:58:58 +08:00
Daniel Graña 841dd5f1f5 Update webservice.rst 2014-08-18 17:48:01 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 3547ca6e61 Add example on running spiders outside projects 2014-08-14 11:50:33 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
Julia Medina d40273561d CrawlerProcess cleanup changes 2014-08-12 14:02:55 -03:00
Julia Medina 980e30a187 Crawler interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina 39c6a80f9d Both getdict and getlist return copies of the requested values 2014-08-12 14:02:55 -03:00
Julia Medina 3ae971468f Add Settings.copy, freeze and frozencopy method 2014-08-12 14:02:55 -03:00
Julia Medina 84fa004793 Add from_crawler class method to base Spider 2014-08-11 11:23:57 -03:00
Daniel Graña 8fece4b0b8 Add 0.24.4 release notes 2014-08-09 17:21:20 -03:00
Daniel Graña 4badcc077d Add 0.24.3 release notes 2014-08-09 00:31:26 -03:00
Daniel Graña 0772201ac8 Update installation docs 2014-08-08 18:07:38 -03:00
Daniel Graña 1a96cd5a3d update tox usage on contributing.rst 2014-07-31 12:28:35 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00
Lisimia b6b44a33f7 Update shell.rst typo 
There was a typo. 
"start" was mistyped as "star"
 2014-07-28 12:26:13 -04:00
Elias Dorneles c298eaf164 removes weird indentation in the shell results 2014-07-23 20:42:36 -03:00
Elias Dorneles 514e8582d0 improved explanations, clarified blog post as source, added link for XPath string functions in the spec 2014-07-23 19:43:57 -03:00
Elias Dorneles 80de43ddab adding some xpath tips to selectors docs 2014-07-22 23:20:53 -03:00
Rolando Espinoza e441a01818 Updated input/ouput processor example according to #560. 2014-07-16 09:04:48 -04:00
Roy Crihfield 1a950e8f5a Fix typo in commands.rst 2014-07-10 14:48:15 -03:00
Daniel Graña 379f6ee6b2 Add 0.24.2 release notes 2014-07-08 05:53:35 -03:00
Zbigniew Siciarz 0466e8cb7a Fixed Python syntax in tutorial. 2014-07-04 10:38:01 +02:00
Mikhail Korobov 00cd4f0fa5 deprecate SgmlLinkExtractor 2014-07-02 23:15:17 +06:00
Mikhail Korobov 173a9dfe86 DOC fix formatting in release notes 2014-06-28 13:28:57 +06:00
Daniel Graña 284a2a06c9 Add 0.24.1 release notes 2014-06-27 12:03:37 -03:00
Daniel Graña 77c8284fa5 set 0.24.0 release date 2014-06-26 11:45:32 -03:00
Daniel Graña ee33efac39 Add 0.24.0 release notes 2014-06-26 10:29:47 -03:00
Daniel Graña 436c1c8980 address latest comments 2014-06-25 15:12:58 -03:00
Daniel Graña 2ad8db6ae6 Merge pull request #761 from dangra/lxmlextractor 
Promote LxmlLinkExtractor as LxmlExtractor
 2014-06-25 15:07:02 -03:00
Daniel Graña a9ecef5662 promote LxmlLinkExtractor as default in docs 2014-06-25 14:34:30 -03:00
Daniel Graña 5b2faf61c3 recognize jl extension as jsonlines exporter and update docs 2014-06-25 13:55:15 -03:00
Daniel Graña 1b32ece918 Merge branch 'settings-cleanup' of https://github.com/Curita/scrapy into Curita-settings-cleanup 2014-06-25 02:55:20 -03:00
Daniel Graña 499438ec44 indent parsed-literal as part of ordered list 2014-06-24 10:32:56 -03:00
Pablo Hoffman ee5c3209af doc: use |version| substitution in ubuntu packages 2014-06-24 09:52:51 -03:00
Paul Tremberth 90e69141d3 Add doc on LxmlLinkExtractor class 2014-06-23 22:35:56 +02:00
Daniel Graña 05ffca2781 Merge pull request #626 from dangra/500-leveldb-cache 
[WIP] Add a LevelDB cache backend
 2014-06-23 15:30:13 -03:00
Daniel Graña a90fd81c97 add leveldb cache backend docs 2014-06-23 14:37:29 -03:00
Pablo Hoffman 7fd8a1a387 minor change to request_scheduled signal doc 2014-06-14 20:51:16 -03:00
Nikolaos-Digenis Karagiannis 5f79940295 Document signal "request_scheduled" 2014-06-14 11:08:18 +03:00
Pablo Hoffman ad11fc01cb crate release notes for 0.24 and #699 to it 2014-06-10 18:46:53 -03:00
Julia Medina ed033854e2 setmodule helper method on Settings class 2014-06-10 11:26:50 -03:00
Julia Medina bdca06240c Fix settings repr on the logs of the shell and tutorial docs topics 2014-06-10 11:26:50 -03:00
Julia Medina 8a3a6236d9 Settings topic updated 2014-06-10 11:26:50 -03:00
Julia Medina 00e6470ed0 Fix and extend the documentation of the new Settings api 2014-06-10 11:26:38 -03:00
Daniel Graña 9830efb946 Merge pull request #697 from allait/master 
Add a note about reporting security issues
 2014-06-03 11:13:13 -03:00
Daniel Graña 7752074c51 Merge pull request #719 from Digenis/master 
document spider.closed() shortcut
 2014-06-03 10:01:15 -03:00
Daniel Graña 0467ad7719 Merge pull request #699 from allait/telnet-port 
Bind telnet console and webservice to 127.0.0.1 by default
 2014-06-03 09:51:16 -03:00
Daniel Graña 8a1c67fdaa Merge pull request #724 from rafallo/master 
item_dropped signal should pass response arg as item_scraped does
 2014-06-03 09:48:06 -03:00
Alexander Chekunkov fa5a67729b DOWNLOADER setting 2014-06-02 13:11:02 +03:00
Alexander Chekunkov 1fba64d34e DOWNLOADER setting 2014-06-02 13:05:22 +03:00
Rafal Jagoda ad6e449c63 Merge remote-tracking branch 'upstream/master' 2014-05-19 16:18:33 +01:00
Nikolaos-Digenis Karagiannis 24073a2624 document spider.closed() shortcut 2014-05-12 22:25:18 +03:00
Rendaw 8bdb6e2e3e Elaborated request priority value. 2014-05-07 19:14:45 +09:00
Carlos Rivera 946b854ddf grammatical issue 2014-05-06 15:41:59 -05:00
Rafal Jagoda 9d226e6266 add response arg to item_dropped signal handlers 
#710
 2014-05-02 14:42:01 +01:00
Alexander Chekunkov 4aa6179af7 added short RFPDupeFilter.request_fingerprint interface description 2014-04-26 15:46:53 +03:00
Alexander Chekunkov baaa077725 added note about RFPDupeFilter.request_fingerprint overriding to the settings documentation 2014-04-26 15:43:12 +03:00
Pablo Hoffman 3932877bcd signals doc: make argument order more consistent with code (although it doesn't matter in practice) 2014-04-25 16:03:32 -03:00
Pablo Hoffman eb07e09166 Merge pull request #663 from pawl/patch-1 
fixed typo
 2014-04-24 17:59:36 -04:00
Mikhail Korobov bdea071af3 DOC selectors.rst cleanup 2014-04-24 22:54:17 +06:00
ncp1113 5f18816428 for loops have to have a : at the end of the line 
changed 3 instances
 2014-04-24 22:50:21 +06:00
Daniel Graña 18412d75b5 Ammend example nesting selectors 2014-04-24 10:32:17 -03:00
Daniel Graña b4593c2ae7 document shortcuts in TextResponse class 2014-04-24 00:15:00 -03:00
Daniel Graña 1117687c47 update docs 2014-04-23 23:39:58 -03:00
Alexey Bezhan 89159779d0 Bind telnet console and webservice to 127.0.0.1 by default 2014-04-17 15:16:58 +01:00
Alexey Bezhan 6d6282da30 Add a note about reporting security issues 2014-04-17 14:41:21 +01:00
Mikhail Korobov 2d3803672b DOC use top-level shortcuts in docs 2014-04-15 01:09:35 +06:00
Julia Medina 80081054a2 Fix broken links in documentation 2014-04-09 18:57:52 -03:00
Julia Medina f0a0d0bc22 Ignore known broken links in docs linkcheck 2014-04-09 18:46:21 -03:00
Ana Sabina Uban 73109bf950 Fixed SgmlLinkExtractor constructor to properly handle both string and list parameters (attrs, tags, deny_extensions) 2014-03-27 16:12:00 +02:00
Paul Brown a1ee354609 fixed typo 2014-03-20 15:16:48 -05:00
tpeng fd7b40e892 add message when raise IngoreReques; fix item_scraped document 2014-03-18 15:23:25 +08:00
Julia Medina b9e2aad874 Doc for disabling download handler 2014-03-12 23:21:33 -03:00
Julia Medina ca1c1a82b5 FormRequest doc improvements 
Clickdata doc enhancements:
 * Fix xml attributes mention
 * nr is 0-indexed reference
 2014-03-12 12:34:50 -03:00
Julia Medina e29ab4d112 New doc: clickdata in Formrequest.from_response 
Documentation about:
 * clickdata parameter in Formrequest.from_response
 * nr attribute in clickdata dict
 * default behaviour when clickdata is None
 2014-03-12 06:43:50 -03:00
Nikolaos-Digenis Karagiannis 8ac9d40665 ItemLoader doc: missing args in replace_value() 2014-03-09 15:16:57 +02:00
Pablo Hoffman 0bc2cbaf09 Merge pull request #613 from breno/master 
Update scrapy command line doc with additional scrapy parse options
 2014-03-07 10:35:58 -08:00
Mikhail Korobov d8fb4a35db Merge pull request #627 from Digenis/master 
SpiderMW doc typo: SWP request, response
 2014-03-07 12:41:39 +05:00
Capi Etheriel 72b6c96d9a Running lucasdemarchi/codespell to fix typos in docs 2014-03-06 12:40:55 -03:00
Nikolaos-Digenis Karagiannis 4335420f40 SpiderMW doc typo: SWP request, response 2014-03-06 16:09:37 +02:00
Alexey Bezhan 210a0a6fe1 Fix some typos, whitespace and small errors in docs 2014-02-27 18:02:22 +00:00
Alexey Bezhan 2e59d77580 Clarify MapCompose documentation 2014-02-27 17:56:14 +00:00
Cash Costello c532e13dcd Added missing word in practices.rst 2014-02-27 13:08:36 -02:00
Pablo Hoffman 6d8b7b29ef remove unused setting: DOWNLOADER_DEBUG 2014-02-27 12:16:05 -02:00
Breno Colom 551c1c5eae Update scrapy command line doc with additional scrapy parse options 2014-02-24 18:10:51 -05:00
Pablo Hoffman 822adb1d5f Merge pull request #602 from Blender3D/email 
Added a mimetype parameter to `MailSender.send`
 2014-02-20 12:05:14 -08:00
Capi Etheriel 1ec1100b27 fixes dynamic itemclass example usage of type() 2014-02-18 20:59:05 -03:00
Nikita Nikishin a676017fe4 Fixed #441. 2014-02-18 03:50:43 -05:00
Paul Tremberth 41765ca18d DupeFilter: add setting for verbose logging + stats counter for filtered requests 2014-02-17 13:42:42 +01:00
Pablo Hoffman d24212c8e5 remove references to deprecated scrapy-developers list 2014-02-16 21:44:49 -02:00
Rolando Espinoza 28f946b05f DOC Use pipelines module name instead of pipieline following default project files. 2014-02-15 11:01:26 -04:00
Daniel Graña 180fc98cd8 Add 0.22.2 release notes 2014-02-14 15:45:44 -02:00
Daniel Graña 6ca49ce76a Merge pull request #594 from dangra/593-engineslots 
fix a reference to unexistent engine.slots.
 2014-02-14 09:39:30 -08:00
Daniel Graña b58285b635 fix a reference to unexistent engine.slots. closes #593 2014-02-14 15:31:05 -02:00
Nikolaos-Digenis Karagiannis 43a797e2f7 downloaderMW doc typo (spiderMW doc copy remnant) 2014-02-11 22:30:00 +02:00
tracicot b2f4b296df Correct typos 2014-02-10 11:46:23 -02:00
Daniel Graña 783d8e3ae1 Add 0.22.1 release notes 2014-02-08 17:09:04 -02:00
Paul Tremberth 57f30bcb04 Docs: 4-space indent for final spider example 2014-02-01 23:34:55 +01:00
Rolando Espinoza a6279fe95b DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. 2014-01-30 11:53:42 -04:00
Rolando Espinoza 4255e12bc7 Updated the tutorial crawl output with latest output. 2014-01-23 18:18:56 -04:00
Rolando Espinoza 9aab9224cb Updated shell docs with the crawler reference and fixed the actual shell output. 
Also updated the shell example with a reproducible code example.
 2014-01-23 18:04:57 -04:00
Daniel Graña 677afe7e54 show ubuntu setup instructions as literal code 2014-01-20 16:22:53 -02:00
Daniel Graña ebfb5b7096 replace warning about updating package lists by a note on package upgrade 2014-01-20 15:18:34 -02:00
Daniel Graña 52c3ff9190 fix apt-get line 2014-01-20 14:39:26 -02:00
Daniel Graña eb73ddd301 Update Ubuntu installation instructions 2014-01-20 14:15:01 -02:00
stray-leone 7f30a671c3 modify the version of scrapy ubuntu package 
latest version is 0.22.
with scrapy-0.18, tutorial project provides error
relative issue : https://github.com/scrapy/scrapy/issues/511
 2014-01-20 10:24:34 -02:00
Daniel Graña 431f2d109f fix 0.22.0 release date 2014-01-17 17:54:01 -02:00
Mikhail Korobov dfd13f9941 fix typos in news.rst and remove (not released yet) header 2014-01-18 01:50:19 +06:00
Daniel Graña d1f1b074e1 Merge 0.22.0 release notes 2014-01-17 15:55:46 -02:00
Mikhail Korobov 7b7a1d8dfd Make Filesystem storage backend default again. See GH-500. 2014-01-17 04:32:08 +06:00
Mikhail Korobov b03fe04999 Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware 2014-01-16 23:09:37 +06:00
Pablo Hoffman ed6fd4933f Merge pull request #524 from hobsonlane/master 
documentation code example corrections per pablohoffman
 2014-01-16 06:44:51 -08:00
Pablo Hoffman 71ada5476e Merge pull request #472 from redapple/exslt 
Register EXSLT namespaces by default (resolves #470)
 2014-01-16 06:32:05 -08:00
Daniel Graña b9bb9bed6b Merge pull request #343 from kmike/ajax-crawlable 
[MRG] AjaxCrawlableMiddleware
 2014-01-16 05:07:39 -08:00
Hobson Lane 85a80d0752 remove "for brevity's sake" line and correct "Torrent item" 
Torrent item -> TorrentItem class
 2014-01-15 17:29:23 -08:00
Paul Tremberth 827c0cf51f Rename "regexp" prefix to "re" 2014-01-15 15:00:25 +01:00
Paul Tremberth 88c8a523a7 Add warning in docs on performance when using EXSLT regexp functions 2014-01-15 12:52:10 +01:00
Paul Tremberth a3eba68aca Drop EXSLT strings and math extensions 2014-01-15 12:28:25 +01:00
Pablo Hoffman ea2f897b81 Merge pull request #502 from scrapy/doc-fixes 
DOWNLOAD_DELAY docs clarification
 2014-01-14 21:07:42 -08:00
Hobson Lane a3db95985b another import name correction by pablo 2014-01-14 21:04:15 -08:00
Pablo Hoffman 3d308d7732 Merge pull request #517 from hoprocker/patch-1 
Clarify return value from extract_links
 2014-01-14 21:01:32 -08:00
Paul Tremberth 2cc26e6f56 Fix typo error 2014-01-14 13:09:18 +01:00
Paul Tremberth 29fc9f3466 Update selectors documentation and tests 2014-01-14 12:56:37 +01:00
Ferdy Rodriguez 807dd25324 fixed error on tor's name 2014-01-13 00:03:58 -06:00
Ferdy Rodriguez 8b9348cfaf Changed TOR Info as previous was removed from www.mininova.org 2014-01-12 23:46:04 -06:00
Hobson Lane 6ba0857a5c documentation code example correction corrections per pablohoffman 2014-01-10 10:37:27 -08:00
malcolm m 962e5ef702 Clarify return value from extract_links 2014-01-05 14:42:48 -08:00
Yuri Prezument 060891c01c Remove unused import from code sample 
Item pipeline docs - removed unused import from code sample
 2014-01-03 15:44:17 +02:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Mikhail Korobov e713733edf minor fixes to scrapy shell docs 
* better IPython links;
* MDC link instead of w3schools;
* small formatting fixes;
* show quoted URL in example
 2013-12-30 10:27:39 +06:00
Mikhail Korobov f18ac02987 remove duplicated link extractors link 
Check http://doc.scrapy.org/en/latest/topics/link-extractors.html - two menu items are highlighted at the left.
 2013-12-28 05:40:10 +05:00
Mikhail Korobov 9a999daa2a DOWNLOAD_DELAY docs clarification: 
* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
 2013-12-28 06:30:34 +06:00
Pablo Hoffman e42e3743fe quick documentation for #475 2013-12-24 12:19:15 -02:00
Mikhail Korobov e0cebbfc8f add a remark about 1% 2013-12-20 23:12:37 +06:00
Mikhail Korobov 943a0bd264 AjaxCrawlableMiddleware in Broad Crawl docs 2013-12-19 01:01:26 +06:00
Mikhail Korobov a87b3bd1c8 AjaxCrawlableMiddleware 2013-12-19 00:06:47 +06:00
RasPat1 ff21281b95 Note about selector class import 
This is the salient point of this code compared to the last example.  We have a selector now and this is how we use it.  Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
 2013-12-15 13:46:42 -05:00
Daniel Graña 8a7c5b5d81 Add 0.20.2 release notes 
Conflicts:
	docs/news.rst
 2013-12-09 18:33:46 -02:00
Pablo Hoffman f2741c413e fix method name in tutorial. closes GH-480 2013-12-02 13:24:12 -02:00
Daniel Graña e34ffc0f42 Add 0.20.1 release notes 
Conflicts:
	docs/news.rst
 2013-11-28 16:25:57 -02:00
Pablo Hoffman 339861367e Merge pull request #425 from audiodude/master 
DownloaderMiddleware docs: Update process_request and minor cleanups.
 2013-11-25 10:33:35 -08:00
Paul Tremberth 14f5817d6b Modify ItemLoader to support XPath and CSS selectors 
Deprecate XPathItemLoader (now an alias to the new ItemLoader)
 2013-11-21 18:05:24 +01:00
Pablo Hoffman f87be371a2 better names for HANDLE_* settings, and added doc 2013-11-21 14:33:17 -02:00
Brian Lange e4c1d8d37d Elaborate on use of order numbers 2013-11-19 17:51:50 -06:00
Brian Lange b878f60b5a Add note to item-pipeline documentation explaining order in the ITEM_PIPELINES setting. 2013-11-19 16:12:54 -06:00
Pablo Hoffman afe6eaa2fe Merge pull request #460 from tntC4stl3/master 
duplicate 'use' in line 87
 2013-11-15 04:10:49 -08:00
tntC4stl3 b51d5d81e4 duplicate 'use' in line 87 2013-11-15 13:56:44 +08:00
Daniel Graña 04ff7ecebf improve 0.20 release notes 
Conflicts:
	docs/news.rst
 2013-11-08 17:45:03 -02:00
Daniel Graña 3d18a3c49e bumped version to 0.21.0 2013-11-08 17:09:00 -02:00
Daniel Graña d0980e5c9b Merge 0.20 release notes 2013-11-08 17:06:10 -02:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Pablo Hoffman 911c8082b0 simplified description of crawl command 2013-10-21 14:42:51 -02:00
Pablo Hoffman e8ee449a2a Merge pull request #432 from darkrho/crawl-url 
Removed URL reference in crawl command and .tld suffix in docs for spider names
 2013-10-21 09:40:58 -07:00
Rolando Espinoza La fuente 34543c2b2e DOCS removed .tld suffix for spider names for the sake of consistency. 2013-10-19 23:03:20 -04:00
Daniel Graña 875b07aef8 fix references to old selector naming in docs 2013-10-17 09:33:15 -02:00
Travis Briggs 3043a5ba37 DownloaderMiddleware docs: Update process_request, proper explanation of IgnoreRequest. 
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.

Further changes.
 2013-10-17 00:23:21 +00:00
Mikhail Korobov 086b8a20d4 typo fix in TextResponse docs 2013-10-17 04:50:30 +06:00
Pablo Hoffman 951a9f3f4c Merge pull request #226 from scraperdragon/patch-1 
Parameters to Request() in wrong order
 2013-10-16 13:15:52 -07:00
Daniel Graña 1461363809 Replace `contenttype` references by `type` 
The type to choose from is the selector type, not the input type. A
content-type doesn't make sense in this context.
 2013-10-16 17:37:25 -02:00
Daniel Graña 155ea08ea1 use `sel` name for Selector's instances in docs, internals and shell 2013-10-15 15:58:42 -02:00
Daniel Graña 1abb1af0c6 fix typos and wording on selector's introduction 2013-10-15 10:13:43 -02:00
Dragon Dave a3b711bdea Move callback blob; mention errback 2013-10-15 12:19:42 +01:00
scraperdragon 0ba0d85685 Parameters to Request() in wrong order 
Implied that callback wasn't the first optional unnamed parameter.
 2013-10-15 11:50:43 +01:00
Daniel Graña 28999590fa update release notes 2013-10-14 16:41:04 -02:00
Daniel Graña ab9462a251 remove more references to libxml2 2013-10-14 16:37:14 -02:00
Daniel Graña 4645f9e03c Updates docs to reflect unified selectors api 2013-10-14 16:31:20 -02:00
Daniel Graña bf37f78572 Drop libxml2 selectors backend 2013-10-11 18:02:35 -02:00
Daniel Graña 6d598f0d94 Update selectors docs 2013-10-10 18:24:00 -02:00
Capi Etheriel bc17e9d412 Adds HtmlCSSSelector and XmlCSSSelector classes, cssselect as optional dependency. 
Ported .get() from _Element and .text_content() from HTMLMixin

Add CSS selectors to scrapy shell

Documenting CSS Selectors: Constructing selectors

Documenting CSS Selectors: Using Selectors

Make CSS Selectors a default feature.

Adds XPath powers to CSS Selectors and some syntactic sugar.

Removes methods copied over from lxml.html.HtmlMixin.

Updating docs to use new CSS Selector super powers.

Documenting CSS Selectors: Regular Expressions

Moving section after Nesting section, since it mentions it.

Documenting CSS Selectors: Nesting Selectors

Fix XPath specificity in lxml.selector.CSSSelectorMixin.text

Cleaning up unused stuff from cssel.py

Changing the behavior of lxml.selector.CSSSelectorMixin.text.

Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).

Documenting CSS Selectors: CSS Selector objects

Documenting CSS Selectors: CSSSelectorList objects

Documenting CSS Selectors: HtmlCSSSelector objects

Documenting CSS Selectors: XmlCSSSelector objects

Fixing some documentations typos and errors

Enforcing the 80-char width lines

Tidying up CSS selectors and CSSSelectorMixin objects

Adding some missing references in documentation.

Fixing lxml.selector.CSSSelectorList.text
 2013-10-10 18:23:15 -02:00
Daniel Graña 5b5dd679b0 Add 0.18.4 release notes 
Conflicts:
	docs/news.rst
 2013-10-10 01:04:35 -02:00
Pablo Hoffman e1683ddf9b fix doc typo 2013-10-09 17:24:12 -02:00
Pablo Hoffman 8b9526a8f6 Merge pull request #400 from irgmedeiros/patch-2 
Update the second code example
 2013-10-07 07:57:18 -07:00
Pablo Hoffman 86c6e9433f remove minor reference to 'scrapy server' command 2013-10-04 14:37:55 -03:00
Daniel Graña aad90ec5a2 Add 0.18.3 release notes 
Conflicts:
	docs/news.rst
 2013-10-03 12:56:25 -03:00
Pablo Hoffman 37c24e01d7 document bindaddress request meta 2013-10-02 17:13:17 -03:00
Pablo Hoffman a9c3519897 updated required twisted version to 10.0 2013-10-01 14:07:38 -03:00
Rolando Espinoza d6e3eae527 docs: added section regarding setting up django's settings. 2013-09-30 09:58:10 -04:00
Rolando Espinoza 0cc1d870db docs: minor tidy up sample code and missing shell prompts. 2013-09-30 09:58:10 -04:00
Loren Davie 8af0e89e85 Corrected typo. 2013-09-29 17:06:46 -04:00
Loren Davie f49f5724d5 Added dynamic creation of item classes to practices.rst. 2013-09-28 09:00:48 -04:00
irgmedeiros 9b50409986 Update the second code example 
Update the second code example to reflect the last change in the first example.
 2013-09-27 18:22:33 -03:00
irgmedeiros d9e0fdc9aa Update practices.rst 
With this modification scrapy runs the spider with project settings. The previous example ran only with default settings resulting in ignoring all user settings as pipelines for example.
 2013-09-27 17:56:30 -03:00
Daniel Graña 265910aae6 Merge pull request #363 from taikano/sitemap_alternate 
also fetch alternate URLs from sitemaps, see #360
 2013-09-26 09:15:02 -07:00
Pablo Hoffman 12280c2a95 fix sphinx references in doc 2013-09-25 15:13:17 -03:00
Pablo Hoffman fc388f4636 Make ITEM_PIPELINE setting a dict 
This is for consistency with how spider and downloader middlewares are
defined. ITEM_PIPELINE_BASE was also added and both remain empty.

Backwards compatibility is kept (with a warning) with list-based
ITEM_PIPELINES.
 2013-09-23 17:50:43 -03:00
Pablo Hoffman b1d1a36a1e add note about enclosing urls with quotes when running from command-line. closes GH-384 2013-09-18 18:01:28 -03:00
cacovsky 71b320914a Update request-response.rst 
Fix small doc typo (too many backticks)
 2013-09-18 11:45:25 -03:00
Kumara Tharmalingam bbb0603091 Fixed directory location for dmoz_spider.py file 
It should be under 'tutorial/spiders' not 'dmoz/spiders'
 2013-09-15 21:55:52 -07:00
Daniel Graña 0400b18efa docs: list lxml as installation prerequisite 2013-09-09 12:44:26 -03:00
Stefan 6994959181 renamed to sitemap_alternate_links and added default value, see #360 2013-09-08 10:38:28 +02:00
Stefan 8ed2d0cda1 improved changes to allow retrieval of alternate links in sitemaps, see #360 2013-09-07 12:56:30 +02:00
Daniel Graña e6b3ca0180 Add 0.18.2 release notes 2013-09-03 14:29:40 -03:00
Daniel Graña 0f00b1602a merge 0.18 release notes 2013-08-27 18:47:46 -03:00
Pablo Hoffman 19ff9ac4f9 url/body attributes of Request/Response objects are now immutable 2013-08-23 12:43:22 -03:00
Pablo Hoffman 86230c0ab8 added quantal & raring to support ubuntu releases 2013-08-22 21:49:55 -03:00
Mikhail Korobov 034ffae60f Recommend Pillow instead of PIL. Closes GH-317. 2013-08-18 00:44:01 +06:00
Berend Iwema 32b6364bcd #327 - Support STARTTLS / SSL option in email sender 2013-08-14 12:59:01 +02:00
Pablo Hoffman c0b26e3d49 minor updates to 0.18 release notes 2013-08-14 01:39:44 -03:00
Daniel Graña ed5b9068d2 fix contributters list format 2013-08-12 11:26:16 -03:00
Daniel Graña a6693c9a5c updated release notes and bumped version to 0.18.0 2013-08-09 19:02:28 -03:00
Hart c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
Dan 1ca31244b0 Fixed ordering of super argument call. 2013-07-16 14:50:10 -04:00
Dan e12b689c4f Updated documentation of spider arguments to include required super call. 2013-07-16 14:26:53 -04:00
Mikhail Korobov 1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Mikhail Korobov ac2fadf3ab DownloaderMiddleware.process_response docs fix 
"returns an exception" -> "raises an exception"
 2013-07-08 19:41:58 +06:00
Mikhail Korobov 39e5da5f66 improve docs for DownloaderMiddleware.process_response 2013-07-08 19:17:29 +06:00
Pablo Hoffman 0f4b70f582 remove no deprecated request_scheduled signal 
It will be replaced by more accurate scheduler signals (proposal will
come soon)
 2013-06-27 11:23:24 -03:00
nramirezuy bef8ade956 removed request_received and added request_scheduled 2013-06-26 16:45:46 -03:00
Pablo Hoffman 819b2776dd Merge pull request #326 from berendiwema/master 
Include example of how to stop the reactor from script
 2013-06-25 13:30:07 -07:00
nramirezuy 83b2774354 remove wrong default httpcache 2013-06-25 17:01:29 -03:00
Berend Iwema aec314db09 added a bit more documentation on how to close the reactor when running scrapy from a script 2013-06-25 16:08:22 +02:00
Pablo Hoffman bbde1d0e0b Merge pull request #275 from stav/doc 
doc: Response.replace() cannot take meta argument
 2013-06-24 11:09:28 -07:00
Capi Etheriel 50fa46d183 Document CrawlSpider.parse_start_urls method 2013-06-09 04:03:20 -03:00
Daniel Graña b4fca90bba merge 0.16.5 release notes 2013-05-30 18:49:00 -03:00
cacovsky 8007762890 Add FAQ entry referencing Request.meta usage 2013-05-27 13:02:17 -03:00
Pablo Hoffman 845c64b89d add benchmarking to 0.18 release notes 2013-05-17 10:38:42 -03:00
Pablo Hoffman ca12886acb update copyright notes 2013-05-16 15:05:52 -03:00
Pablo Hoffman 8e49fed918 minor improvements to benchmarking doc 2013-05-16 13:23:13 -03:00
Pablo Hoffman 76087e336a add scrapy bench command for benchmarking, with documentation 2013-05-16 13:15:25 -03:00
Pablo Hoffman 66311db23e mention crawlera in best practices, as a way to deal with bans 2013-05-04 18:20:23 -03:00
Pablo Hoffman 9361c89573 remove scrapyd doc, as it was moved to its own repo 2013-04-27 04:15:42 -03:00
Pablo Hoffman d02da2f31f ported code to use queuelib 2013-04-23 17:48:09 -03:00
Pablo Hoffman 7a1536f76e Merge pull request #290 from nramirezuy/item-copy 
added copy method to item
 2013-04-19 09:27:44 -07:00
Nicolás Ramírez 6df274bba5 added copy method to item 2013-04-19 13:23:53 -03:00
Mikhail Korobov b245d592aa Update faq.rst 
spider.DOWNLOAD_DELAY is deprecated
 2013-04-18 02:42:15 +06:00
Juan M Uys 4de3aa4932 Update overview.rst 2013-04-08 14:13:15 +02:00
Pablo Hoffman 96c2332e0e fix inaccurate downloader middleware documentation. refs #280 2013-04-02 11:35:32 -03:00
Steven Almeroth 70179c7c0c doc: remove trailing spaces 2013-03-21 13:57:39 -06:00
Steven Almeroth 0d7747d353 doc: Response.replace() cannot take meta argument 
>>> response.replace(meta={'foo':1})
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 45, in replace
    return Response.replace(self, *args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/__init__.py", line 77, in replace
    return cls(*args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 22, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'meta'
 2013-03-21 13:49:55 -06:00
Pablo Hoffman 2a5c7ed4da make Crawler.start() return a deferred that is fired when the crawl is finished 2013-03-20 14:48:59 -03:00
Pablo Hoffman b347c14b5f update engine status output on telnet console documentation 2013-03-18 19:12:12 -03:00
Shane Evans 5c2a82f1f7 fix typo 2013-03-17 19:34:55 +00:00
Pablo Hoffman bb20907254 minor updated to faq 2013-03-14 16:43:00 -03:00
Pablo Hoffman 098ccff862 added FAQ about error: "cannot import name crawler" 2013-03-14 12:57:59 -03:00
Pablo Hoffman 8391b36251 minor updates to contributing doc 2013-03-13 03:24:25 -03:00
Pablo Hoffman 51c301b3a2 added link to python binary libs, for windows installation 2013-03-13 03:18:33 -03:00
Pablo Hoffman 8e72730792 Merge pull request #261 from stav/allowed_domains 
allow spider allowed_domains to be set/tuple, #259
 2013-03-12 20:44:51 -07:00
Steven Almeroth 650eda68da doc: add comment about commit history cleanliness 2013-03-10 18:51:04 -06:00
Pablo Hoffman eeb69d2f70 added #260 to release notes 2013-03-08 11:59:38 -02:00
Jordi Llonch 5b118ff4ab added documentation (experimental feature) 2013-03-06 06:36:23 +11:00
Pablo Hoffman 3c8eef99cb docs/contributing: added note explaining what Scrapy contrib is 2013-03-04 01:35:17 -02:00
Steven Almeroth f62b6660d4 doc: fix typo in spider middleware 2013-03-02 19:46:31 -06:00
Pablo Hoffman 7400ceb1ed added 502 to RETRY_HTTP_CODES 2013-02-22 19:12:59 -02:00
Pablo Hoffman a038f46859 doc: fixed rst title 2013-02-14 11:11:17 -02:00
Pablo Hoffman 22edc44c6c doc: remove links to diveintopython.org, which is no longer available. closes #246 2013-02-14 11:09:40 -02:00
Pablo Hoffman 1ff8b4f831 updated release notes with previous commit 2013-02-12 00:59:25 -02:00
Daniel Graña 5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
whodatninja 8e3b5baac5 Fix typo labeling attrs type bool instead of list 2013-02-05 15:10:41 -05:00
Daniel Graña 3cf7f4975b Add 0.16.4 to release notes 
Conflicts:
	docs/news.rst
 2013-01-23 11:29:38 -02:00
Chris Tilden aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman 6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman 1ba04b1fc3 added remove_namespaces() method to XmlXPathSelector objects 2013-01-18 12:20:03 -02:00
Pablo Hoffman c31441a273 revert default HTTP cache policy to dummy (instead of RFC2616) 2013-01-17 13:08:29 -02:00
Daniel Graña 897195186a document new FormRequest parameter named `formxpath` that matches forms using xpath 2013-01-08 18:36:20 -02:00
Daniel Graña 75563b3f00 Add list of supported and missing RFC2616 caching features 2013-01-08 18:16:44 -02:00
Daniel Graña d8a760bf57 Merge branch 'http-cache-middleware' 
Conflicts:
	scrapy/contrib/downloadermiddleware/httpcache.py
	scrapy/contrib/httpcache.py
	scrapy/tests/test_downloadermiddleware_httpcache.py
 2013-01-08 17:34:48 -02:00
Daniel Graña 864a7aef87 More httpcache updates 
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
 2013-01-08 17:26:32 -02:00
Daniel Graña 672d09ea2e add meta-refresh changes to release notes 2013-01-08 12:30:36 -02:00
Daniel Graña defc4f89b5 update metarefresh settings 2013-01-08 11:41:19 -02:00
Daniel Graña 6a2b23883a Add MetaRefreshMiddleware docs 2013-01-08 11:25:38 -02:00
Daniel Graña 076ba40404 update DOWNLOADER_MIDDLEWARES_BASE setting documentation 2013-01-08 10:50:27 -02:00
Pablo Hoffman 227a1d666b add doc about disabling an extension. refs #132 2013-01-07 13:16:19 -02:00
Pedro Faustino 5d3a4d755f Update downloader middleware documentation 2013-01-06 18:53:14 +00:00
Emanuel Schorsch f9b130da12 Proposed Changes 
I was very confused as to how you actually import DjangoItem.
I searched extensively on the internet looking for actual code so I could see how it worked.
I finally found http://blog.just2us.com/2012/07/setting-up-django-with-scrapy/. It is much easier to understand with full files instead of code fragments.
I also edited where it says "we can see that the model is already saved" as I don't see how it's already saved.
 2013-01-04 15:59:04 -05:00
Natan L d572f8945e Fixed typo 
'persitent' --> 'persistent'
 2012-12-31 11:14:01 -08:00
Pedro Faustino 492831fc6f Merge branch 'master' of git://github.com/scrapy/scrapy into http-cache-middleware 2012-12-28 15:27:45 +01:00
Hasnain Lakhani 93a1102189 Implemented policies for HTTP Cache 2012-12-26 16:29:48 -08:00
Pablo Hoffman 51b8feb4ce fixed doc typos 2012-12-26 16:16:53 -02:00
Pablo Hoffman 1e2ee76df2 add documentation topics: Broad Crawls & Common Practies 2012-12-26 14:02:13 -02:00
Pedro Faustino fdaa35f6e8 Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching. 2012-12-24 19:37:53 +01:00
Pablo Hoffman 12475fccbe Merge pull request #206 from dangra/downloader-enhancements 
AutoThrottle and Downloader enhancements
 2012-12-17 10:11:46 -08:00
Daniel Graña d7daf836d5 Altering delay is enough to auto throttle downloads 2012-12-17 16:08:49 -02:00
Luan 5582ea28ec Update docs/topics/commands.rst 
A short change.
 2012-12-10 15:16:02 -02:00
Daniel Graña 0cc138c010 Add 0.16.3 release notes 2012-12-07 18:55:15 -02:00
Pablo Hoffman 39274a2457 doc: removed obsolete references to ClientForm 2012-11-23 19:06:47 -02:00
stav 99f164fc87 correct docs for default storage backend 2012-11-22 14:05:47 -06:00
Pablo Hoffman 1f0d167037 doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:26 -02:00
Ilya Baryshev 097aea04a4 Fixed docs typo in SpiderOpenCloseLogging example 2012-11-10 12:24:53 +04:00
Daniel Graña da7e414fe9 Add 0.16.2 release notes 
Conflicts:

	docs/news.rst
 2012-11-09 13:03:04 -02:00
Pablo Hoffman db21bccf9a added 0.18 to release notes and mention spider contracts 2012-11-07 16:02:18 -02:00
Pablo Hoffman aa0e02dc54 added open_in_browser to debugging doc 2012-11-04 19:58:06 -02:00
Pablo Hoffman 7a7c5d1334 removed reference to global scrapy stats from settings doc 2012-11-03 17:05:01 -02:00
Daniel Graña c0542838d3 update news file with 0.16.1 release notes 2012-10-26 18:53:59 -02:00
Pablo Hoffman 8f4c879b58 extended documentation on how to access crawler stats from extensions 2012-10-25 11:28:23 -02:00
Daniel Graña a9616f387e fix dashes under rst headers 2012-10-18 20:14:30 -02:00
Daniel Graña 4852c6102d set release date for 0.16.0 in news 2012-10-18 20:07:36 -02:00
Pablo Hoffman 1a905d62f5 removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway) 2012-10-09 16:05:19 -02:00
Pablo Hoffman 1f89eb59fe fixed doc reference to topics-contracts 2012-10-09 16:02:12 -02:00
Pablo Hoffman 9e4c6aba2f updated release notes with new stats recently added 2012-10-09 12:50:42 -02:00
Pablo Hoffman 7458092eef added spider contracts to release notes and warn that its API is still subject to change 2012-09-29 03:06:30 -03:00
Pablo Hoffman c380910b40 Merge pull request #167 from alexcepoi/sep-017 
Spider contracts (SEP-017)
 2012-09-28 13:57:07 -07:00
Pablo Hoffman a5f8943d3e added change introduced in previous commit to release notes 2012-09-26 13:17:01 -03:00
Alex Cepoi 11d29c7005 SEP-017 contracts: add tests and minor improvements 2012-09-21 00:12:46 +02:00
Pablo Hoffman b46b5a6ef0 Documented AutoThrottle extension and added to extensions available by 
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
 2012-09-20 18:52:57 -03:00
Pablo Hoffman c7f8219901 - removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes, 
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
 2012-09-19 03:03:34 -03:00
stav 303e13f616 selector documentation typos 2012-09-18 12:56:52 -05:00
Pablo Hoffman 3d736e657f fixed typo in doc 2012-09-18 10:51:01 -03:00
Pablo Hoffman eed6eb49da make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses) 2012-09-17 10:11:07 -03:00
Pablo Hoffman 8f2dda12cc removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead 2012-09-16 21:21:44 -03:00
Pablo Hoffman 9685c24059 restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode 2012-09-14 12:38:00 -03:00
Pablo Hoffman 81ed2d2d0b major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API. 2012-09-14 12:31:33 -03:00
Pablo Hoffman d6867f7984 removed sphinx warnings about duplicate reference names 'this page' 2012-09-13 15:26:45 -03:00
Pablo Hoffman a874964ad4 renamed 'XPath Selectors' title to just 'Selectors' 2012-09-13 15:24:44 -03:00
Pablo Hoffman acb8895e1a changed note in scrapyd doc to use sphinx notes 2012-09-13 15:22:59 -03:00
Pablo Hoffman 26f1d5cb48 Merge pull request #171 from artem-dev/scrapyd_job_times 
added start and stop times for scrapyd list jobs web service
 2012-09-13 11:16:01 -07:00
Artem Bogomyagkov 4d8f253912 commited doc file missed from prev commit 2012-09-12 11:12:59 +03:00
Pablo Hoffman 7ef593c5c2 refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal 2012-09-11 16:27:19 -03:00
Alex Cepoi bf8dc61fb7 SEP-017 contracts: pretty-printing and docs 2012-09-10 23:17:27 +02:00
Pablo Hoffman fff2871828 added doc section (and FAQ) about spider arguments 2012-09-04 14:49:30 -03:00
Pablo Hoffman f4a17ec272 removed references to Scrapy Snippets site 2012-09-03 22:19:15 -03:00
Pablo Hoffman b901e64044 replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161 2012-09-03 19:28:16 -03:00
Pablo Hoffman e2f9daac67 fixed formatting in scrapyd release notes 2012-09-03 16:58:58 -03:00
Pablo Hoffman dd1398b280 removed another instance of scrapy.conf.settings singleton, this time from download handlers 2012-09-03 16:44:18 -03:00
Pablo Hoffman 7c8af83138 added issue role to documentation 2012-08-31 18:39:30 -03:00
Daniel Graña a2d22307ad update news.rst 2012-08-31 17:38:40 -03:00
Pablo Hoffman be206ca5ab added process_start_requests method to spider middlewares 2012-08-31 16:41:50 -03:00
Pablo Hoffman b920674666 corrected minor issue with doc references 2012-08-31 16:41:49 -03:00
Pablo Hoffman 4ec99117d3 fixed minor doc typo 2012-08-30 11:56:30 -03:00
Pablo Hoffman 4a5f70278f minor tidy up to installation guide windows notes 2012-08-29 15:44:24 -03:00
Pablo Hoffman 098d892c03 simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users 2012-08-29 15:37:05 -03:00
Pablo Hoffman 70f8e517a1 promoted DjangoItem to main contrib 2012-08-29 11:23:11 -03:00
Pablo Hoffman da234af456 Merge branch 'singleton_removal' 2012-08-28 18:42:33 -03:00
Pablo Hoffman babfc6e79b Updated documentation after singleton removal changes. 
Also removed some unused code and made some minor additional
refactoring.
 2012-08-28 18:35:57 -03:00
Pablo Hoffman 5fa5c4545b use ScrapyJSONEncoder JsonItemExporter & JsonLinesItemExporter, to support nested items properly 2012-08-22 13:46:50 -03:00
Pablo Hoffman 36f47a4aec Removed per-spider settings concept, and scrapy.conf.settings singleton from many extensions and middlewares. There are some still remaining, that will be removed in future commits 2012-08-21 17:27:45 -03:00
Daniel Graña abcc8c9f63 Recommend pypi as single way to install on Windows 2012-08-06 10:21:13 -03:00
Daniel Graña b7b0c49520 append parse command to example code sections in docs. closes #162 2012-08-06 09:10:16 -03:00
Pablo Hoffman 832e45073b fixed typo in stats documentation. closes #159 2012-07-20 17:13:06 -03:00
Valentin-Costel Hăloiu 00bfb37e79 Update master 2012-07-04 06:55:01 +03:00
Daniel Graña 277ed0ae23 Merge pull request #145 from alexcepoi/cookies-changes 
domain and path support for request cookies
 2012-06-25 11:29:04 -07:00
Alexandru Cepoi 177c81745d domain and path support for request cookies 2012-06-25 20:17:59 +02:00
Pablo Hoffman 179e3810dc fixed links to doc. closes #150 2012-06-24 01:00:33 -03:00
Alexandru Cepoi f4faa19e31 added docs topic debugging spiders 2012-06-21 20:03:33 +02:00
Alexandru Cepoi 3e05a2ecf6 update docs for parse command 2012-06-12 18:28:10 +02:00
Pablo Hoffman e1be9c01bc updated FAQ about bot bans 2012-06-08 18:33:53 -03:00
Daniel Graña ae2ff4d33a update news file with 0.14.4 release notes 2012-05-15 16:16:02 -03:00
Pablo Hoffman 9686f97242 added precise to supported ubuntu distros 2012-05-12 19:54:36 -03:00
Pablo Hoffman 58e88ed246 scrapyd: do not set SCRAPY_FEED_URI/SCRAPY_LOG_FILE if items_dir/logs_dir settings are not set 2012-05-08 17:43:00 -03:00
Pablo Hoffman 9c3b9f2968 fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py 2012-05-03 12:05:40 -03:00
Pablo Hoffman abcac4fcbd updated maintainer to scrapinghub 2012-05-02 03:25:35 -03:00
stav 86dba76d1f documentation indentation 2012-04-30 13:09:34 -05:00
Pablo Hoffman c7f33c534b removed redundant lines from release notes 2012-04-20 09:09:55 -03:00
Daniel Graña 15c8c01828 add 0.14.3 release notes 2012-04-19 23:25:15 -03:00
Pablo Hoffman 97f362d64e removed deprecated/undocumented class: HTMLImageLinkExtractor 2012-04-19 12:07:46 -03:00
Pablo Hoffman 2fb5e62c39 doc: update overview page to point to the genspider command. refs #107 2012-04-19 02:37:22 -03:00
Pablo Hoffman d567d8efbe added note to docs/topics/firebug.rst about google directory being shut down 2012-04-19 01:34:20 -03:00
Pablo Hoffman 30ddbf624e mention about some scrapy.xlib modules removed in the release notes 2012-04-17 12:31:18 -03:00
Pablo Hoffman d99ee6deb9 added some missing entries to release notes 2012-04-17 12:29:48 -03:00
Daniel Graña d8ebf16fe5 Merge pull request #114 from stav/master 
Scrapy DOC changes
 2012-04-11 12:08:45 -07:00
Pablo Hoffman 7cca916ed5 added release notes to official documentation, including all release notes since Scrapy 0.7 2012-04-11 15:53:23 -03:00
stav f1802289cd small doc typo change to get the fork rolling 2012-04-11 12:05:39 -05:00
Pablo Hoffman 4f28ffcb2c removed no longer needed dependency on simplejson 2012-04-10 16:01:36 -03:00
Pablo Hoffman 6e8edbd72e switched default selectors backend to lxml 2012-04-10 15:52:14 -03:00
Pablo Hoffman 27018fced7 changed default user agent to Scrapy/0.15 (+http://scrapy.org) and removed no longer needed BOT_VERSION setting 2012-03-23 13:45:21 -03:00
Pablo Hoffman 8933e2f2be added REFERER_ENABLED setting, to control referer middleware 2012-03-22 16:35:14 -03:00
Jason Yeo da826aa13d fixed minor mistake in Request objects documentation 2012-03-21 10:25:41 +08:00
Pablo Hoffman 175c70ad44 fixed minor defect in link extractors documentation 2012-03-20 22:56:45 -03:00
Pablo Hoffman 35fb01156e removed some obsolete remaining code related to sqlite support in scrapy 2012-03-16 11:55:55 -03:00
Pablo Hoffman b6ae266546 Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman 2b16ebdc11 added minor clarification on cookiejar request meta key usage 2012-02-29 07:19:01 -02:00
lostsnow 5afe4f50c1 scrapyd: support bind to a specific ip address 2012-02-29 13:47:40 +08:00
Pablo Hoffman 81abb45000 fixed bug in new cookiejar documentation 2012-02-28 11:08:25 -02:00
Pablo Hoffman 26c8004125 added documentation for the new cookiejar Request.meta key 2012-02-27 19:58:58 -02:00
Pablo Hoffman 7fe7c3f3b1 MemoryUsage extension: close the spiders (instead of stopping the engine) when the limit is exceeded, providing a descriptive reason for the close. Also fixed default value of MEMUSAGE_ENABLED setting to match the documentation. 2012-02-23 17:05:06 -02:00
Pablo Hoffman 7b8942a648 updated StackTraceDump extension doc 2012-02-16 15:14:17 -02:00
Pablo Hoffman ea77342b55 updated versioning doc according to recent changes 2012-01-05 11:50:28 -02:00
Pablo Hoffman 0b0bce7f3c scrapyd: added cancel.json and listjobs.json api methods to documentation 2012-01-05 11:23:25 -02:00
Pablo Hoffman 8f42633a94 scrapyd: added clarification about how to disable items feeds generation 2012-01-05 11:20:50 -02:00
Pablo Hoffman dbda33efa6 scrapyd: added support for storing items by default 
Items are stored the same way as logs, in jsonlines format.

Also renamed logs_to_keep setting to jobs_to_keep.
 2012-01-03 23:08:54 -02:00
Pablo Hoffman 0be421fbf0 fixed reference to tutorial directory 2011-12-23 18:57:11 -02:00
Pablo Hoffman 41fd3c4f6c doc: removed duplicated callback argument from Request.replace() 2011-12-23 15:55:46 -02:00
Pablo Hoffman 0eeff76227 fixed formatting of scrapyd doc 2011-12-20 03:18:37 -02:00
Daniel Graña bcb31988f2 change tutorial to follow changes on dmoz site 2011-12-14 13:03:31 -02:00
Pablo Hoffman 992af8d38f ubuntu repos: added support for oneiric release 2011-10-25 14:26:38 -02:00
Pablo Hoffman c38c49d56a fixed PickeItemExporter bug, added unittest, and added pickle to suported feed exports formats 2011-10-25 02:36:51 -02:00
Pablo Hoffman 8bdf288428 made scrapyd doc more version agnostic 2011-10-23 05:29:54 -02:00
Pablo Hoffman ade5efdc61 added -o option to scrapy crawl, a convenient shortcut for using feed exports 2011-10-22 20:53:49 -02:00
Pablo Hoffman 431441cb52 updated documentation to remove references to old issue tracker and mercurial repos 2011-09-25 13:06:24 -03:00
Pablo Hoffman ce03ccd4ec updated documentation about DEPTH_PRIORITY and DFO/BFO crawls 2011-09-23 13:22:25 -03:00
Julien Duponchelle b7c436343a scrapy deploy support git version 2011-09-21 22:17:08 +02:00
Pablo Hoffman ab1c9cfc56 removed documentation header notifying about other documentation versions, as that's provided by readthedocs already 2011-09-14 02:39:32 -03:00
Daniel Grana 5f1b1c05f8 Do not filter requests with dont_filter attribute set in OffsiteMiddleware 2011-09-08 15:18:10 -03:00
Pablo Hoffman bff3d31469 scrapyd: updated schedule.json response format 2011-09-04 09:29:24 -03:00
Pablo Hoffman a1dbc62b45 removed CONCURRENT_SPIDERS setting (use scrapyd maxproc instead) 2011-09-02 18:27:39 -03:00
Pablo Hoffman 40f7075f11 added initial documentation about suspend and resume crawls 2011-09-02 13:12:27 -03:00
Pablo Hoffman 27dd68a690 added SpiderState extension 2011-09-02 13:06:59 -03:00
Pablo Hoffman 6a31ab667d minor fix to doc 2011-09-01 15:08:23 -03:00
Pablo Hoffman d98b058c21 no longer recommend using labmda's in the doc, as they're not friendly with scheduler persistence 2011-09-01 15:06:49 -03:00
Pablo Hoffman 76af0cdd44 updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
Pablo Hoffman 98b68ca89d scrapyd: documented support for passing setting to spiders in schedule.json 2011-08-27 01:31:12 -03:00
Pablo Hoffman 5c6b0631e2 minor doc fix 2011-08-19 11:42:03 -03:00
Pablo Hoffman 9d97e73a24 fixed priority handling on the new scheduler so that it's backwards compatible (ie. bigger priorities are higher). also fixed a few documentation bugs related to requests priority 2011-08-19 08:26:41 -03:00
Pablo Hoffman a3697421c0 some minor updates to documentation 2011-08-11 09:19:59 -03:00
Pablo Hoffman 5da6ffb57b Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-08-11 09:11:19 -03:00
Pablo Hoffman bc2d2183e9 fixed import in doc 2011-08-11 09:11:08 -03:00
Pablo Hoffman 19e6da59d8 added new downloader middleware: ChunkedTransferMiddleware 2011-08-09 03:03:25 -03:00
Pablo Hoffman 984be35461 Some telnet console changes: 
* renamed manager alias to crawler
* added aliases: spider, slot
* fixed est() function
 2011-08-08 15:01:08 -03:00
Pablo Hoffman f7c0aeccc6 added note about engine_started signal 2011-08-07 03:57:09 -03:00
Pablo Hoffman 9f60c27612 added setting to support disabling DNS cache: DNSCACHE_ENABLED 2011-08-05 20:41:59 -03:00
Pablo Hoffman cb95d7a5af added marshal to formats supported by feed exports 2011-08-03 16:16:48 -03:00
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 2011-08-02 11:57:55 -03:00
Pablo Hoffman f354a49d0f added FAQ about preventing bots getting banned 2011-07-28 00:40:30 -03:00
Pablo Hoffman ce7a787970 Big downloader refactoring to support real concurrency limits per domain/ip, 
instead of global limits per spider which were a bit useless.

This removes the setting CONCURRENT_REQUESTS_PER_SPIDER and adds thre new
settings:

* CONCURRENT_REQUESTS
* CONCURRENT_REQUESTS_PER_DOMAIN
* CONCURRENT_REQUESTS_PER_IP (overrides per domain)

The AutoThrottle extension had to be disabled, but will be ported and
re-enabled soon.
 2011-07-27 13:38:09 -03:00
Pablo Hoffman c59340150f Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled. 2011-07-27 03:45:15 -03:00
Pablo Hoffman 2ac08a713d downloader: renamed SpiderInfo to Slot, for consistency with engine and scraper names 2011-07-22 02:06:10 -03:00
Pablo Hoffman 0e008268e1 removed SimpledbStatsCollector from scrapy code, it was moved to https://github.com/scrapinghub/scaws 2011-07-20 10:38:16 -03:00
Pablo Hoffman b6b0a54d9f removed FAQ entry 2011-07-20 01:31:36 -03:00
Pablo Hoffman e3f640c7bf added FAQ entry about scrapy deploy issue on Mac + Python 2.5 2011-07-19 19:53:32 -03:00
Pablo Hoffman 84f518fc5e More core changes: 
* removed execution queue (replaced by newer spider queues)
* added real support for returning iterators in Spider.start_requests()
* removed support for passing urls to 'scrapy crawl' command
 2011-07-15 15:18:39 -03:00
Pablo Hoffman dbad1373f1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-07-13 18:44:54 -03:00
Pablo Hoffman 18cb4ff1d8 added natty to list of supporte ubuntu distros 2011-07-13 18:43:52 -03:00
Pablo Hoffman 39a2ea97c8 redirect mw: added REDIRECT_ENABLED setting and documented the other settings 2011-07-13 14:18:15 -03:00
Pablo Hoffman 541ed3913b retry middleware: added RETRY_ENABLED setting and documented the other settings more properly, also improved messages when no longer retrying requests 2011-07-13 11:55:05 -03:00
Pablo Hoffman 763f3dc628 minor update to doc 2011-07-12 19:56:39 -03:00
Pablo Hoffman bfda9ec319 added clarification about scrapy versioning including the recently adopted odd/even versioning scheme 
--HG--
rename : docs/api-stability.rst => docs/versioning.rst
 2011-07-12 19:53:23 -03:00
Pablo Hoffman 4fde1ef94d added CloseSpider exception, to manually close spiders 2011-07-12 14:24:10 -03:00
Pablo Hoffman db5cae7c03 SitemapSpider: added support for filtering which sitemaps to follow (patch contributed by Rolando Espinoza). closes #330 2011-06-23 18:18:29 -03:00
Pablo Hoffman 57c43fdce6 added SitemapSpider, with tests and doc 2011-06-15 11:54:34 -03:00
Pablo Hoffman 91dc46539f added LogStats extension for periodically logging basic stats (like crawled pages and scraped items) 2011-06-14 00:50:05 -03:00
Pablo Hoffman 841e9913db renamed CLOSESPIDER_ITEMPASSED setting to CLOSESPIDER_ITEMCOUNT, to follow the refactoring done in r2630 2011-06-13 16:58:51 -03:00
Pablo Hoffman 474cba512c simplified MemoryDebugger extension to use stats for dumping memory debugging info 2011-06-06 03:13:28 -03:00
Pablo Hoffman 5fbc32c015 call stats collector engine_stopped() after the engine is closed (to make sure all data from extensions has been collected), and added that method to documented api 2011-06-06 03:12:40 -03:00
Pablo Hoffman 9d9c8877da added 'scrapy edit' command 2011-06-05 22:02:56 -03:00
Pablo Hoffman 03ae481cad removed experimental crawlspider v2 2011-06-03 18:23:23 -03:00
Pablo Hoffman 5bf733b6f6 Changed default representation of items to pretty-printed dicts. This improves 
default logging by making log more readable in the default case, for both Scraped and Dropped lines.

Projects can still customize how items are represented by overriding the item's __str__ method, as usual.
 2011-06-03 01:13:01 -03:00
Pablo Hoffman 1bc2339bb8 Merged item passed and item scraped concepts, as they have often proved 
confusing in the past.

This means:

* original item_scraped signal was removed
* original item_passed signal was renamed to item_scraped
* old log lines "Scraped Item..." removed
* old log lines "Passed Item..." renamed to "Scraped Item..."
 2011-06-03 01:13:00 -03:00
Pablo Hoffman e6091df551 fixed doc typo 2011-05-30 09:04:31 -03:00
Pablo Hoffman 1d98fc8fb5 added spider_error signal 2011-05-29 22:38:17 -03:00
Pablo Hoffman 2fa0f75f2d added COOKIES_ENABLED setting to support disabling the cookies middleware 2011-05-27 00:35:34 -03:00
Pablo Hoffman d72d3f4607 stack trace dump extension: also dump engine status, and support triggering it with SIGQUIT, besides SIGUSR2 2011-05-20 03:25:00 -03:00
Pablo Hoffman 951ba507f9 Removed support for default values in Scrapy items, which have proven confusing in the past 2011-05-19 21:42:46 -03:00
Pablo Hoffman 503f302010 removed remaining references to scheduler middleware from doc, as it will be removed on next release 2011-05-18 19:48:48 -03:00
Pablo Hoffman 3fd17432cf fixed outdated documentation 2011-05-18 14:46:20 -03:00
Pablo Hoffman 9016e7e993 added role to link to scrapy source code (not yet used) 2011-05-18 14:43:34 -03:00
Pablo Hoffman cd85c12c33 Some Link extractor improvements: 
* added support for ignoring common file extensions that are not followed if
  they occur in links
* fixed link extractor documentation issues
* slighly improved performance of applying filters
* added link to link extractors doc from documentation index
 2011-05-18 12:32:34 -03:00
Pablo Hoffman 495152bd50 disabled verbose depth stats collection by default, added DEPTH_STATS_VERBOSE setting to enable it 2011-05-18 11:04:48 -03:00
Pablo Hoffman accb6ed830 dump stats to log by default (ie. change default value of STATS_DUMP to True) 2011-05-17 22:42:05 -03:00
Pablo Hoffman 7f97259ba7 added w3lib to requirements, in installation guide 2011-05-01 11:14:57 -03:00
Pablo Hoffman 4a83167698 fixed small doc typo 2011-04-30 01:35:30 -03:00
Pablo Hoffman bb2b67c862 updated tutorial to use 'dmoz' as the name of the spider instead of 'dmoz.org', so that it's more similar to the dirbot example project 2011-04-28 09:31:57 -03:00
Pablo Hoffman bf73002428 removed googledir example, replaced by dirbot project on github. updated docs accordingly 2011-04-28 02:28:39 -03:00
Pablo Hoffman b12dd76bb8 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-04-25 09:31:18 -03:00
Pablo Hoffman 678f08bc1b added warning about using 'parse' as callback in crawl spider rules 2011-04-25 09:30:42 -03:00
Pablo Hoffman ad496eb3b6 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-04-14 12:36:27 -03:00
Pablo Hoffman ecb4f44cbc Added clarification on how to work with local settings and scrapy deploy 2011-04-14 12:36:09 -03:00
Pablo Hoffman 3ee2c94e93 Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it 2011-04-06 14:54:48 -03:00
Pablo Hoffman 8a5c08a6bc added join_multivalued parameter to CsvItemExporter 2011-03-24 13:15:52 -03:00
Pablo Hoffman 3954e600ca added DBM storage backend for HTTP cache 2011-03-23 21:32:02 -03:00
Pablo Hoffman cfd11df539 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-24 15:28:57 -02:00
Pablo Hoffman 8f7e163b04 Fixed wrong method name in downloader middleware documentation 2011-02-24 15:26:32 -02:00
Daniel Grana c55355642c fix FAQ typos reported by marlun_ at #scrapy IRC channel 2011-02-16 08:57:42 -02:00
Pablo Hoffman 1fb55bdaf0 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-15 07:25:12 -02:00
Pablo Hoffman 16d9a33951 added FAQ entry about working with big data feeds 2011-02-15 07:24:52 -02:00
Pablo Hoffman 936353d5f1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-09 11:20:46 -02:00
Pablo Hoffman 181d1c09ae Fixed typo and code indentation in the doc. Closes #307 and #308 2011-02-09 11:19:46 -02:00
Pablo Hoffman c91f0d9ea1 Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12 2011-02-04 13:39:54 -02:00
Pablo Hoffman c5499ead73 Clarified behaviour when multiple rules match the same link in CrawlSpider 2011-02-04 13:39:12 -02:00
Pablo Hoffman d7f193cbea bumped version to 0.13 in documentation 2011-01-02 17:29:43 -02:00
Pablo Hoffman b56e933be9 bumped version to 0.12 in documentation 2011-01-02 17:28:33 -02:00
Pablo Hoffman 5879389ad0 Bumped version to 0.12 2011-01-02 16:16:40 -02:00
Pablo Hoffman fa644f7a5e Some simplifications to Scrapyd architecture and internals: 
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
  get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
  doesn't necesarilly run eggs (though it does in the default case)

--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
 2010-12-27 16:22:32 -02:00
Pablo Hoffman 633ebc4c43 minor indentation improvement 2010-12-23 13:04:49 -02:00
Pablo Hoffman db07a9a938 Added notice to documentation, pointing dev to stable versions and viceversa 2010-12-23 13:03:40 -02:00
Pablo Hoffman 544308d6d0 updated ubuntu repos doc, in preparation for the 0.11 release 2010-12-21 11:02:56 -02:00
Pablo Hoffman 002abf204f Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273 2010-12-13 14:05:47 -02:00
Pablo Hoffman f984d438a0 updated docs to use scrapy version on aptitude install lines 2010-12-13 14:02:42 -02:00
Pablo Hoffman 119fd20e91 Added verbose option to 'version' command. Closes #298 2010-12-13 00:32:44 -02:00
Pablo Hoffman 6a1b69c93f renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands 
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
 2010-11-30 20:23:27 -02:00
Pablo Hoffman df54ed0041 Some Scrapyd enhancements: 
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
 2010-11-30 02:26:31 -02:00
Pablo Hoffman bbffa59497 Some changes to Scrapyd: 
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
 2010-11-29 17:19:05 -02:00
Pablo Hoffman 2557777c39 Updated doc referring to HTTP cache middleware 2010-11-24 13:27:44 -02:00
Pablo Hoffman 426b6fa100 docs/intro/install.rst: added -U flag to easy_install command 2010-11-22 13:50:19 -02:00
Pablo Hoffman 91a7c25797 * Made Response.meta attribute map to Request.meta attribute. Closes #290 
* Record redirected URLs in redirect middleware. Closes #291
 2010-11-18 12:51:54 -02:00
Pablo Hoffman ac007802d6 Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280 2010-11-17 21:32:23 -02:00
Pablo Hoffman 5a5364d0c1 Updated documentation to point out that simplejson is now required if using Python 2.5, and to recommended switching to Python 2.6 2010-11-16 03:31:04 -02:00
Pablo Hoffman d988ca1ec2 Some changes to scrapy deploy command: 
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options

* updated documentation accordingly
 2010-11-08 17:01:06 -02:00
Pablo Hoffman 0f69e7a191 Some changes to HTTP Cache middleware: 
* made it use the project data storage by default (closes #279)
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
  dropped support for negative numbers
* other minor doc improvements
 2010-11-01 02:38:15 -02:00
Pablo Hoffman 3c94c6cb9b fixed sphinx doc id 2010-11-01 02:31:20 -02:00
dfdeshom 130276605b Bind the web server and telnet server to a configurable interface (WEBSERVICE_HOST). The default is to bind to all interfaces. Also add documentation for WEBSERVICE_HOST and TELNETCONSOLE_HOST. 2010-11-01 00:59:04 -02:00
Pablo Hoffman b76c5c597f * Added support for project data storage (closes #276) 
* Documented project file structure
* Moved default location of SQLite database to project data storage dir (closes #277)
 2010-10-31 03:25:37 -02:00
Pablo Hoffman dfa6745e91 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 16:05:53 -02:00
Pablo Hoffman a0d9b43031 fixed typo in scrapyd doc 2010-10-30 16:05:32 -02:00
Pablo Hoffman d67152ab0f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 01:56:12 -02:00
Pablo Hoffman 75451cbe84 scrapyd doc: fixed delversion.json example 2010-10-30 01:56:00 -02:00
Pablo Hoffman a59bfb539d * Added lxml backend for XPath selectors. Closes #147 
* Added new setting (SELECTORS_BACKEND) to choose which backend to use
* Deprecated the extract_unquoted() function from selectors
* Made libxml2 optional by adding a dummy selector backend. Closes #260

--HG--
rename : scrapy/tests/test_selector.py => scrapy/tests/test_selector_libxml2.py
 2010-10-25 14:47:10 -02:00
Pablo Hoffman 6c921896a5 Expanded documentation on deploy command and versions. Refs #261 2010-10-19 00:11:45 -02:00