RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

1544 Commits

Author SHA1 Message Date
Moisés Guimarães c58ea021b8 fixes docs 2016-12-04 11:56:14 -03:00
Rolando Espinoza 6431e7a138 DOC State explicitly that spiders are loaded recursively. 2016-12-01 13:26:19 -03:00
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 059085b5b4 Remove docs for deprecated ChunkedTransfer middleware 2016-11-24 18:23:34 +01:00
Paul Tremberth 03cf5f1bd2 Remove ChunkedTransferMiddleware from default settings 2016-11-24 17:18:57 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Paul Tremberth 11fe3751cf DOC Remove "Ubuntu" section from sidebar/ToC 2016-11-16 11:55:09 +01:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Rolando Espinoza 76459e1969 Update conda channel to conda-forge and show conda version badge. 2016-11-08 21:30:27 -03:00
Rahul Kant f56aef99c2 Add closing tag in <tbody> 2016-11-07 17:49:22 +05:30
Gustavo de Andrade f2e49bc23c Update selectors.rst 
Decode instead escape, exceptions &lt; and &amp; (kmike)
Second sentence droped (Digenis)
 2016-11-01 21:32:17 -02:00
Mikhail Korobov 9755ef933a Merge pull request #2369 from jc0n/media-pipeline-docs-typo 
Fix typo in media pipeline docs
 2016-10-29 12:46:48 +06:00
John O'Connor d85da273be Fix typo in media pipeline docs 2016-10-28 19:44:46 -07:00
Elias Dorneles a9c74dbe42 Merge pull request #2339 from visued/master 
[MRG+1] Update documentation to explain the use of double quotes on Windows (fixes #2325)
 2016-10-25 09:53:24 -02:00
Steven Almeroth 99daea495b Doc: wording 2016-10-21 16:16:42 -07:00
Steven Almeroth a958e54954 Doc: remove trailing spaces 2016-10-21 16:16:37 -07:00
Randy Pen c7d245b90b update 
Thx for your advice.
 2016-10-21 10:52:50 +02:00
Randy Pen eacc5937e4 fix example code 
In the AuthorSpider, original css selector failed to get links of author pages
 2016-10-21 10:52:50 +02:00
Paul Tremberth c559a64c20 Set date for 1.2.1 release 2016-10-21 10:17:46 +02:00
Paul Tremberth 22772a61c8 Update release notes for upcoming 1.2.1 2016-10-21 10:17:04 +02:00
gustavodeandrade c7dfb5eb88 Fix issue 1704 2016-10-21 00:08:08 -02:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00
Victor Sued f74051e69e update documentation to explain the use of double quotes on Windows. 2016-10-18 16:36:43 -02:00
bopace fd016ee71b Fixed wording of documentation 2016-10-18 09:37:45 -06:00
Paul Tremberth 6cc83c041e Merge pull request #2330 from lfmattossch/note-python2-name 
[MRG+1] Added a note about invalid spider names in python 2 (fixes #2251)
 2016-10-18 16:35:19 +02:00
Jose Ricardo e12e364a40 Add details to the spider middlewares docs 
Document the effects of the middleware order in a more detailed way.
 2016-10-18 12:29:30 -02:00
Luiz Fernando Mattos Schlindwein 7c33e0cb55 added a note about invalid spider names in python 2 2016-10-18 11:55:51 -02:00
Jose Ricardo ea7bd39529 Make architecture overview references a little more clear on the docs 
Expliciting what actually happens by adding links to the respective methods
that are invoked in each processing phase.
 2016-10-18 11:50:51 -02:00
Jose Ricardo bebcd5081c Add downloader middleware ordering details to the docs 
Add more details, making it easier to understand what are the effects of
setting a downloader middleware order.
 2016-10-18 11:22:55 -02:00
Bo Pace bfe28ae707 Added documentation about accessing header values 2016-10-17 14:10:05 -06:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
Paul Tremberth 95c6b9dffd Merge pull request #2280 from scrapy/release-notes-1.2 
Update release notes for upcoming 1.2.0
 2016-10-03 13:00:14 +02:00
Paul Tremberth e2137d77ce Add release date for scrapy 1.2 2016-10-03 12:41:18 +02:00
Paul Tremberth a433126301 Merge pull request #2294 from redapple/release-headers 
Change release section titles to have correct links in ToC
 2016-10-03 12:38:44 +02:00
Wieland Hoffmann e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Paul Tremberth fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles 559b4edaec update release notes with changes from #1581 2016-09-30 11:42:50 -03:00
Elias Dorneles 8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Paul Tremberth fd0b6faecb Change release section titles to have correct links in ToC 2016-09-29 14:36:13 +02:00
Paul Tremberth c670e3cdf6 Move mention 2016-09-29 14:23:52 +02:00
Elias Dorneles 86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles 60559369b5 Merge pull request #2292 from stummjr/fix-example-section 
[MRG+1] Update examples section in documentation with quotes.toscrape spiders
 2016-09-29 09:11:45 -03:00
Paul Tremberth 33d04684e5 Fix typo 2016-09-29 10:25:21 +02:00
Elias Dorneles 3ac3ac4d92 docs: update data flow description and image (fixes: #2278) 
This fixes the explanation to use Requests instead of URLs,
which is what actually happens, and is also consistent with the
new tutorial, which already explains how URLs become Request objects.

I've also changed the "loop", jumping from 9 to step 2.
 2016-09-28 16:38:45 -03:00
Valdir Stumm Junior 5680c61063 Doc: update examples section with quotes.toscrape.com spiders 2016-09-28 15:45:01 -03:00
Paul Tremberth 23abf2d31b Fix title underline 2016-09-28 18:31:27 +02:00
Paul Tremberth 8d130b295a Update installation guide 2016-09-28 17:09:17 +02:00
Paul Tremberth c707c31345 Fix Debian number 2016-09-26 17:20:13 +02:00
pawelmhm 39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00
Paul Tremberth bba31b545d Mention new Debian Jessie baseline 2016-09-23 17:33:08 +02:00
Paul Tremberth 7eeee6f616 Mention \uXXXX escapes in regards to FEED_EXPORT_ENCODING 2016-09-23 16:39:09 +02:00
Paul Tremberth d52fbf2045 Merge remote-tracking branch 'origin/master' into release-notes-1.2 2016-09-23 16:34:06 +02:00
Paul Tremberth 80c1e5dc25 Set release date, fix typo and add tutorial improvement issue number 2016-09-23 16:11:50 +02:00
Paul Tremberth a0f87d2f45 Update release notes for upcoming 1.1.3 bugfix release 2016-09-23 16:11:50 +02:00
Pawel Miech b2bfd1e5c5 [docs] document that process_item can return Deferred 2016-09-23 10:36:03 +02:00
Paul Tremberth a975a50558 Merge pull request #2252 from eliasdorneles/tutorial-upgrades 
[MRG+2] Tutorial: rewrite tutorial seeking to improve learning path
 2016-09-22 16:39:21 +02:00
Elias Dorneles f4a2208916 addressing review comments and other minor editing 2016-09-22 11:04:45 -03:00
Paul Tremberth 2e08a9b412 Merge pull request #2271 from redapple/mailsender-lists 
[MRG+1] Add note on "to" and "cc" as lists for sending emails
 2016-09-22 12:00:14 +02:00
Elias Dorneles d636e5baa8 better description for start_requests expected return value 2016-09-21 18:54:12 -03:00
Elias Dorneles 32017a76f8 recommend learn python the hard way for beginners 2016-09-21 11:06:36 -03:00
Elias Dorneles 38266cc949 recommend Dive into Python and Python tutorial instead of LPTHW for non-beginners 2016-09-21 11:02:24 -03:00
Elias Dorneles c126c59361 address more review comments 2016-09-20 18:19:25 -03:00
Elias Dorneles a876ea5bd2 minor grammar fix 2016-09-20 15:10:49 -03:00
Elias Dorneles bc41fdf20e address review comments, add debug log to initial spider 2016-09-20 15:04:08 -03:00
Paul Tremberth 40293551b2 Remove mention of odd-numbered versions for development releases 
Fixes GH-1317
 2016-09-20 18:15:45 +02:00
Elias Dorneles 125b691102 more reviewing and editing, minor restructure, syntax fixes 2016-09-20 12:47:03 -03:00
Paul Tremberth e59d79bf37 Add note on "to" and "cc" as lists for sending emails 
Fixes GH-2244
 2016-09-20 17:18:49 +02:00
Elias Dorneles 8975371a57 Merge branch 'master' into tutorial-upgrades 2016-09-20 09:45:05 -03:00
Elias Dorneles f4f93c5c26 fix tox docs build, adjust title 2016-09-20 09:19:59 -03:00
Ashish Kulkarni 8893527a2f temporarily deprecate official Ubuntu packages 
They are not currently updated and fail to install on
Ubuntu 16.04. Also update the instructions to refer to
the earliest supported LTS (Ubuntu 12.04).

fixes #2137 and closes #2076
 2016-09-20 11:10:59 +02:00
Valdir Stumm Junior fee07835f2 Completing the data extraction section 2016-09-19 19:19:44 -03:00
Elias Dorneles 2a409d1d95 [wip] changing introduction to scraping with selectors 2016-09-19 17:13:04 -03:00
Elias Dorneles 063315258e Merge pull request #2202 from scrapy/doc-arch-overview2 
[MRG+1] DOC move Data Flow below the picture; add links to components
 2016-09-19 08:11:18 -03:00
Paul Tremberth bb8740a552 Update 1.2 release notes with latest changes 2016-09-19 12:26:36 +02:00
Wayne Lovely cc8497abb1 Fix a dict key in the tutorial 2016-09-17 11:09:28 +00:00
Valdir Stumm Junior 233b98d642 include section describing spider arguments 2016-09-16 18:08:10 -03:00
Elias Dorneles 31545a9f84 tutorial: updating extracting data section to introduce CSS and XPath equally 2016-09-16 17:13:24 -03:00
Elias Dorneles 147e75602d update after review comments (thanks @stummjr) 2016-09-16 16:47:24 -03:00
Elias Dorneles 31260cf02f mentions stackoverflow as help channel (fixes #2255) 2016-09-16 16:05:36 -03:00
Elias Dorneles 21de617c77 mention that spiders need to subclass scrapy.Spider 2016-09-16 15:55:14 -03:00
Elias Dorneles b2a5cddbb0 tutorial: update section about following links, expand examples 
adding an AuthorSpider to demonstrate further a different crawling
arrangement.
 2016-09-16 15:49:49 -03:00
Valdir Stumm Junior 0cd9dfcc85 small fixes on tutorial 2016-09-16 15:21:49 -03:00
Valdir Stumm Junior 0da497cf7a updates on the first section (our first spider) 2016-09-16 11:55:23 -03:00
Elias Dorneles c508f40689 use harcoded URLs, remove item reference on second spider 2016-09-15 18:05:09 -03:00
Elias Dorneles 2427791287 tutorial: remove item class definition and present start_requests first 
This changes the tutorial, removing the step of creating an item class
and also starts by presenting the start_requests method instead of
start_urls.
 2016-09-15 17:46:31 -03:00
Elias Dorneles 75531e409e use better condition in example spider 2016-09-15 16:56:13 -03:00
Elias Dorneles 1d159ae6f9 minor grammar fix 2016-09-15 15:37:03 -03:00
Elias Dorneles 18bd0b0886 docs: update overview spider code to use toscrape.com and minor changes 
So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
 2016-09-15 15:16:30 -03:00
Paul Tremberth 2f60f2a5a6 Merge pull request #2236 from stummjr/new-tutorial-toscrape 
[MRG+1] Update broken Scrapy tutorial to use quotes.toscrape.com
 2016-09-15 12:05:03 +02:00
Elias Dorneles a9a96bed8f updated tutorial as per review comments 2016-09-14 18:09:39 -03:00
Valdir Stumm Junior bc67cd9edd fix indentation issue 2016-09-14 12:39:29 -03:00
Valdir Stumm Junior 10f8c52f5d changed tutorial examples from dmoz to quotes.toscrape.com 2016-09-13 14:05:26 -03:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Valdir Stumm Junior 9cea6f0730 Add Segment Analytics to Documentation 2016-09-02 14:51:07 -03:00
Paul Tremberth b188f61b95 Update release notes for upcoming 1.2.0 version 2016-09-01 17:38:38 +02:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00
Mikhail Korobov 495d322691 DOC move Data Flow below the picture; add links to components 2016-08-26 20:16:22 +05:00
Thom Dixon f68dc3026d Fix indentation 2016-08-24 09:11:27 -07:00
Thom Dixon 633abfbea1 Correct documentation about Response parameters 
This fixes issue #2196
 2016-08-24 08:47:52 -07:00
Valdir Stumm Junior d61650d843 fix RANDOMIZE_DOWNLOAD_DELAY description in the docs 2016-08-19 18:24:32 -03:00
Paul Tremberth f18c3e5ce5 Add release notes for 1.1.2 version 2016-08-19 17:01:57 +02:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Lorena 7d432872bf text updates to match graphic 2016-08-04 11:01:14 -07:00
Lorena 04f93e096c updated graph for organization/clarity 2016-08-04 10:04:47 -07:00
Paul Tremberth 5b1d98b8c8 Update 1.1.1 release date 2016-08-01 20:21:12 +02:00
Paul Tremberth 928e93f8f3 Update notes with latest 1.1 commits 2016-08-01 20:21:12 +02:00
Paul Tremberth e1d118d5ca Update release notes for upcoming 1.1.1 release 2016-08-01 20:21:12 +02:00
Elias Dorneles 34e7dadf38 Merge pull request #1610 from darshanime/scheduler_debug 
[MGR+1] Change, document `LOG_UNSERIALIZABLE_REQUESTS`
 2016-07-29 10:12:52 -03:00
darshanime d8e62e660d update log demo print 2016-07-26 20:46:12 +05:30
Paul Tremberth fe68a45c82 Merge pull request #2138 from jesuslosada/master 
[MRG+1] Apply the FILES_STORE_S3_ACL setting in ImagesPipeline
 2016-07-26 12:50:25 +02:00
Elias Dorneles b6375d3701 Merge pull request #1566 from darshanime/signal-example 
[MRG+1] Include example for signal docs
 2016-07-25 10:49:29 -03:00
Darshan Chaudhary a2e6452554 Include signal example 2016-07-25 18:41:25 +05:30
darshanime 0c77b6d033 update docs for settings 2016-07-25 18:05:51 +05:30
Darshan Chaudhary 472a8a47d0 Change name, log once 2016-07-25 18:03:43 +05:30
Jesús Losada 8221556089 Add documentation on Amazon S3 ACLs 2016-07-22 23:22:18 +02:00
Jesús Losada c6a2ca4e93 Document S3 capabilities in FilesPipeline and ImagesPipeline 2016-07-22 19:32:29 +02:00
Paul Tremberth ec1c61504a Merge pull request #2005 from feliperuhland/master 
[MRG+1] Included new optional parameter in startproject command line
 2016-07-19 12:31:06 +02:00
Kenneth Schnall 5fabed514e Update logging.rst 2016-07-15 23:14:11 -04:00
Mikhail Korobov 79639d0fec Merge pull request #1989 from pawelmhm/fix-images-pipeline-uppercase-other 
[MRG+1] [image_pipeline] bring back uppercase class attributes
 2016-07-13 14:44:00 +00:00
Kenneth Schnall 2489f84d39 Update practices.rst 2016-07-12 15:34:50 -04:00
Paul Tremberth c3109daa72 Merge pull request #2034 from dracony/master 
[MRG+1] Added option to turn off ensure_ascii for JSON exporters
 2016-07-12 17:01:09 +02:00
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Pawel Miech c22cc1096b [image_pipeline] style edits 
* 80 characters line limit
* shortening some code
* removed dead code
* add doctest for _key_for_pipe function
 2016-07-12 13:58:36 +02:00
Mikhail Korobov 5f261cf295 Merge pull request #2026 from Digenis/contributing-full_subjects 
[MRG+1] Encourage complete titles in pull requests
 2016-07-08 10:45:22 +06:00
Valdir Stumm Junior 081595a2e4 document new genspider behavior 2016-07-06 15:10:48 -03:00
Raul Gallegos 49ac7de231 prioritize default headers over user agent 2016-07-05 15:38:17 -05:00
Νικόλαος-Διγενής Καραγιάννης 2a92ffb409 Encourage descriptive PR titles 2016-07-05 14:21:28 +03:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Pawel Miech 73cc066caa [docs] add note about windows + python3 2016-06-17 09:21:58 +02:00
Pawel Miech 10b79c9b3e [files-pipeline] update docs with note about settings 
for subclasses.
 2016-06-15 15:49:11 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
Felipe Ruhland 089483aece
Updated docs for new option in command line 2016-05-23 23:16:15 -03:00
Paul Tremberth b3367c7acd DOC Add info and example on errbacks 2016-05-18 18:00:09 +02:00
Paul Tremberth c2c8036a97 DOC Update copyright notice 2016-05-18 16:52:09 +02:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Paul Tremberth 149c4cd41e [DOC][Overview] Use idiomatic .extract_first() 2016-05-17 17:53:40 +02:00
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Paul Tremberth 2849ebf4c6 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 14:07:33 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
nblock a3557dd34d Fix spelling mistake 2016-04-11 14:06:57 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
djunzu 6988e9cd4b Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-01 21:51:15 -03:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
Paul Tremberth 9d8c368ce8 Merge pull request #1879 from scrapy/scrapy-arch-docs 
DOC improved Architecture overview
 2016-03-31 12:09:24 +02:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Lele 7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Mikhail Korobov 4f335b5a01 DOC clarify Architecture docs 2016-03-25 17:03:41 +05:00
Mikhail Korobov 3ca977a8cb DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-25 07:11:33 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin e7c4806c5e Update feedstorage docs: add botocore, mention that boto is supported only on Python 2 2016-02-15 19:59:48 +03:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Nicolas Pennequin 6efc7a9de0 Update the email doc for the charset argument to send(). 2016-02-04 19:33:44 +01:00
Elias Dorneles ca08314654 makes more explicit source of problems in Python 3 limitations 2016-02-03 21:38:25 -02:00
Elias Dorneles 2167573b32 fix comment about disabling robots.txt 2016-02-03 21:31:27 -02:00
Elias Dorneles 3180abc800 applying more review comments 2016-02-03 21:28:05 -02:00
Elias Dorneles d40add7b27 add note about robots.txt waiting and make it explicit builtin extensions only are ported 2016-02-03 21:18:25 -02:00
Elias Dorneles 9cfefd5277 favor beta over basic to describe PY3 support 2016-02-03 20:59:56 -02:00
Elias Dorneles 2f697ce59f added note about how to revert to old behavior for robotstxt 2016-02-03 20:51:26 -02:00
Elias Dorneles 89a088b3c3 applying review comments 2016-02-03 20:45:55 -02:00
Paul Tremberth 972f192568 Blank line before CoC 2016-02-03 22:58:49 +01:00
Paul Tremberth 728e505e53 Merge remote-tracking branch 'origin/master' into release-notes-1.1 2016-02-03 22:47:15 +01:00
Valdir Stumm Jr dd7b0eb359 Updates to release notes 2016-02-03 19:23:02 -02:00
Paul Tremberth 51dc741fa2 Add 1.0.5 release notes 2016-02-03 21:07:21 +01:00
Valdir Stumm Jr 0f816524e8 edit API changes, Deprecations, and Relocations sections. 2016-02-03 16:21:32 -02:00
Paul Tremberth fd3193da86 Reword some of "New Features and Enhancements" 2016-02-03 18:19:38 +01:00
Paul Tremberth 3b2a6abc0b Format "Code refactoring" subsection 2016-02-03 17:37:58 +01:00
Paul Tremberth ed02455010 Format "Logging" subsection 2016-02-03 17:27:43 +01:00
Valdir Stumm Jr 1fd95c9c9f fixed references to issues and fit text to 80 cols in the first two sections 2016-02-03 12:45:48 -02:00
Valdir Stumm Jr a21c90bee7 edits on Py3 Support and New features sections 2016-02-03 11:54:46 -02:00
Paul Tremberth 44d8df2060 Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS 2016-02-03 12:35:26 +01:00
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Paul Tremberth 142aa21737 Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note 2016-02-03 12:33:10 +01:00
Paul Tremberth db0697bc06 Add 1.1 release notes (draft) 2016-02-03 12:32:40 +01:00