RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,575 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

826 Commits

Author SHA1 Message Date
Paul Tremberth 4ea2453ae0 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-06-08 17:24:33 +02:00
Panayiotis Lipiridis 28efd8dd35 Github >> GitHub 2016-05-11 17:10:12 +02:00
djunzu fd98e3c031 Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-11 18:06:03 +02:00
nblock 00792557ec Fix spelling mistake 2016-04-11 17:36:31 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
Mikhail Korobov 288f8c09f8 DOC clarify Architecture docs 2016-03-31 12:33:29 +02:00
Mikhail Korobov 1c70ece728 DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-31 12:33:29 +02:00
Aron Bordin 7b1dddceb3 small doc style fixes 2016-03-30 18:36:34 +02:00
nyov 718d386ebd Update documentation links 2016-03-30 18:36:34 +02:00
Paul Tremberth 844f040bc4 Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 17:49:14 +02:00
Lele 15ffb076ca Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-30 10:53:54 +02:00
Paul Tremberth c1337d74ab Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-27 15:12:56 +01:00
Konstantin Lopuhin 9083b68125 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 22:26:22 +01:00
Konstantin Lopuhin b9c6d67365 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 22:26:22 +01:00
Konstantin Lopuhin 25e481fd14 [backport][1.1.x] Py3 S3 botocore 
Originally from https://github.com/scrapy/scrapy/pull/1761
 2016-02-18 22:48:37 +01:00
Nicolas Pennequin 9cdbe177b7 Update the email doc for the charset argument to send(). 2016-02-09 19:52:06 +01:00
Paul Tremberth 44d8df2060 Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS 2016-02-03 12:35:26 +01:00
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Paul Tremberth 142aa21737 Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note 2016-02-03 12:33:10 +01:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
Mikhail Korobov 7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Elias Dorneles d4c4ca8062 fix version number to appear new feature 2016-01-21 09:42:15 -02:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Daniel Graña b1e44436bc Merge pull request #1642 from orangain/doc-metarefresh 
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
 2016-01-19 13:33:50 -03:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Julia Medina 48539af163 Merge pull request #1601 from mvj3/minor-documentation-fixes 
Fixed minor grammar issues.
 2015-11-16 14:58:07 -03:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Valdir Stumm Jr d577c4702d fixed a typo in the documentation. 2015-10-26 00:00:20 -02:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
hoatle 2869cf8dde fix another invalid xpath error 2015-10-07 16:03:43 +07:00