RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

1544 Commits

Author SHA1 Message Date
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Paul Tremberth 2849ebf4c6 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 14:07:33 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
nblock a3557dd34d Fix spelling mistake 2016-04-11 14:06:57 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
djunzu 6988e9cd4b Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-01 21:51:15 -03:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
Paul Tremberth 9d8c368ce8 Merge pull request #1879 from scrapy/scrapy-arch-docs 
DOC improved Architecture overview
 2016-03-31 12:09:24 +02:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Lele 7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Mikhail Korobov 4f335b5a01 DOC clarify Architecture docs 2016-03-25 17:03:41 +05:00
Mikhail Korobov 3ca977a8cb DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-25 07:11:33 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin e7c4806c5e Update feedstorage docs: add botocore, mention that boto is supported only on Python 2 2016-02-15 19:59:48 +03:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Nicolas Pennequin 6efc7a9de0 Update the email doc for the charset argument to send(). 2016-02-04 19:33:44 +01:00
Elias Dorneles ca08314654 makes more explicit source of problems in Python 3 limitations 2016-02-03 21:38:25 -02:00
Elias Dorneles 2167573b32 fix comment about disabling robots.txt 2016-02-03 21:31:27 -02:00
Elias Dorneles 3180abc800 applying more review comments 2016-02-03 21:28:05 -02:00
Elias Dorneles d40add7b27 add note about robots.txt waiting and make it explicit builtin extensions only are ported 2016-02-03 21:18:25 -02:00
Elias Dorneles 9cfefd5277 favor beta over basic to describe PY3 support 2016-02-03 20:59:56 -02:00
Elias Dorneles 2f697ce59f added note about how to revert to old behavior for robotstxt 2016-02-03 20:51:26 -02:00
Elias Dorneles 89a088b3c3 applying review comments 2016-02-03 20:45:55 -02:00
Paul Tremberth 972f192568 Blank line before CoC 2016-02-03 22:58:49 +01:00
Paul Tremberth 728e505e53 Merge remote-tracking branch 'origin/master' into release-notes-1.1 2016-02-03 22:47:15 +01:00
Valdir Stumm Jr dd7b0eb359 Updates to release notes 2016-02-03 19:23:02 -02:00
Paul Tremberth 51dc741fa2 Add 1.0.5 release notes 2016-02-03 21:07:21 +01:00
Valdir Stumm Jr 0f816524e8 edit API changes, Deprecations, and Relocations sections. 2016-02-03 16:21:32 -02:00
Paul Tremberth fd3193da86 Reword some of "New Features and Enhancements" 2016-02-03 18:19:38 +01:00
Paul Tremberth 3b2a6abc0b Format "Code refactoring" subsection 2016-02-03 17:37:58 +01:00
Paul Tremberth ed02455010 Format "Logging" subsection 2016-02-03 17:27:43 +01:00
Valdir Stumm Jr 1fd95c9c9f fixed references to issues and fit text to 80 cols in the first two sections 2016-02-03 12:45:48 -02:00
Valdir Stumm Jr a21c90bee7 edits on Py3 Support and New features sections 2016-02-03 11:54:46 -02:00
Paul Tremberth 44d8df2060 Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS 2016-02-03 12:35:26 +01:00
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Paul Tremberth 142aa21737 Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note 2016-02-03 12:33:10 +01:00
Paul Tremberth db0697bc06 Add 1.1 release notes (draft) 2016-02-03 12:32:40 +01:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
Mikhail Korobov 7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Elias Dorneles d4c4ca8062 fix version number to appear new feature 2016-01-21 09:42:15 -02:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Daniel Graña b1e44436bc Merge pull request #1642 from orangain/doc-metarefresh 
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
 2016-01-19 13:33:50 -03:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
Daniel Graña 1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
orangain 4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Rolando Espinoza f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Pablo Hoffman eb4daa34a2 Merge pull request #1544 from rolando/conda-docs 
Added installation notes about using Conda for Windows.
 2015-11-26 14:11:28 -03:00
Rolando Espinoza cc8ddb68b0 Added installation notes about using Conda for Windows and other OSes. 2015-11-26 13:08:56 -04:00
Julia Medina 48539af163 Merge pull request #1601 from mvj3/minor-documentation-fixes 
Fixed minor grammar issues.
 2015-11-16 14:58:07 -03:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Valdir Stumm Jr d577c4702d fixed a typo in the documentation. 2015-10-26 00:00:20 -02:00
Christian Pedersen cc76fb3dff Version 1 now exists 2015-10-14 14:07:34 +02:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
hoatle 2869cf8dde fix another invalid xpath error 2015-10-07 16:03:43 +07:00
Hoat Le 4e66955411 fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst 
>>> response.xpath('//div/[id="not-exists"]/text()').extract_first() is None
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/http/response/text.py", line 109, in xpath
    return self.selector.xpath(query)
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/selector/unified.py", line 100, in xpath
    raise ValueError(msg if six.PY3 else msg.encode("unicode_escape"))
ValueError: Invalid XPath: //div/[id="not-exists"]/text()
 2015-10-07 15:43:02 +07:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
smirecki 8379bea7ed Typos corrections 
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.

 Please enter the commit message for your changes. Lines starting
 2015-10-02 23:48:27 -04:00
Marius Gedminas 0620e76433 Fix list formatting 2015-09-29 03:33:30 +05:00
Marius Gedminas eaad10facf Typo 2015-09-28 14:44:15 +05:00
Daniel Graña 3c596dcf46 Merge pull request #1467 from dacjames/master 
add support for a nested loaders
 2015-09-16 22:03:19 -03:00
hy 14f7f22555 fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware 2015-09-16 16:59:23 +08:00
Daniel Collins 036109e7de updte nested loader documentation 2015-09-15 23:49:35 -07:00
Mikhail Korobov 600f27c39b Merge pull request #1350 from scrapy/installation-docs-update 
Updates in Installation and Contributing sections in the docs
 2015-09-09 17:54:35 +05:00
Julia Medina eb7b9d506f Add note to ubuntu install section about debian compatibility 2015-09-08 18:49:44 -03:00
Daniel Collins 88c92cb68b provide documentation for nested loaders 2015-09-04 13:15:48 -07:00
Daniel Graña dd473145f2 Merge pull request #1473 from ArturGaspar/master 
[MRG+1] Support for returning deferreds in middlewares
 2015-09-02 13:04:26 -03:00
Robert Weindl 833efbfcbd Update tutorial.rst 
Add missing "scrapy." prefix.
 2015-09-02 20:35:23 +05:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Artur Gaspar ca83a0b028 Support for returning deferreds in downloader middleware methods. 2015-09-01 13:22:43 -03:00
Daniel Graña b4e382c867 Merge pull request #1318 from yarikoptic/fix-docs-explicit-rtd-theme-path 
[MRG+1] DOC(ENH): specify path to rtd theme explicitly
 2015-08-31 10:42:02 -03:00
David Tagatac 08162a15d8 minor: scrapy.Spider docs grammar 2015-08-27 17:37:16 -04:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Rick aa88fd797a Small grammatical change 2015-08-26 15:01:50 -03:00
Mikhail Korobov cfae62f9cc Merge pull request #1441 from aivarsk/fix-common-practices 
Make common practices sample code match the comments
 2015-08-23 17:36:09 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Aivars Kalvāns b8b1e8e544 Make common practices sample code match the comments 2015-08-19 16:54:10 +03:00
Julia Medina d3c3408cd4 Replace alternative OSX install workaround with virtualenv 2015-08-19 03:19:27 -03:00
Julia Medina 46ee1c6ee2 Reference Homebrew's homepage for installation instructions 2015-08-19 02:21:08 -03:00
Daniel Graña 54f9837804 Document how to get coverage report 2015-08-13 19:02:51 -03:00
Daniel Graña f67a808d9e Add 1.0.3 release notes 2015-08-11 13:49:49 -03:00
Daniel Graña 57fafc702d Add 1.0.2 release notes 2015-08-06 18:02:12 -03:00
Mikhail Lyundin ea4199f8e1 Minor method name fix 2015-08-05 22:42:53 +03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
David Tagatac 08123207c5 minor: scrapy.Spider grammar and clarity 2015-07-31 17:01:59 -04:00
Mikhail Korobov b6eb3404a2 Merge pull request #1335 from jdemaeyer/fix/docs-scrapy-path 
Make Sphinx autodoc use local, not system-wide Scrapy
 2015-07-30 15:33:02 +05:00
Jakob de Maeyer 9c12a3f23a Remove incorrect path for Sphinx autodoc 2015-07-30 12:12:08 +02:00
Andrew Murray 198be7d679 Fixed typos 2015-07-16 23:59:02 +10:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina 320d3a59b9 Add oldest supported tox version to contributing docs 
Better fix for #1337
 2015-07-10 20:34:44 -03:00
Julia Medina 439b376d5b Note in install docs about pip being already included in python>=2.7.9 2015-07-10 20:34:43 -03:00
Julia Medina 5723e6b457 Add non-python dependencies to Ubuntu install section in the docs 
Closes #1314 and closes #1198.
 2015-07-10 20:34:43 -03:00
Julia Medina 290ebee090 Add OS X installation section to docs 
Closes #1342 and (possibly) #1126
 2015-07-10 20:34:43 -03:00
Nikola Pavlović 7bef61dbbd Fix reference. 2015-07-10 02:35:27 +02:00
Jakob de Maeyer d438075a47 Make Sphinx autodoc use local, not system-wide Scrapy 2015-07-03 16:51:53 +02:00
Julia Medina 8b3ca4f250 Merge pull request #1302 from eliasdorneles/improving-access-settings-docs 
[MRG+1] Improvements for docs on how to access settings
 2015-07-03 00:56:32 -03:00
Daniel Graña 3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Daniel Graña 6fa3f2421c Add 1.0.1 release notes 2015-07-01 01:49:44 -03:00
Mikhail Korobov d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Mikhail Korobov 63317531f9 DOC fix authrottle docs 
see https://github.com/scrapy/scrapy/pull/502/files#r8574692
 2015-06-26 20:47:58 +05:00
Yaroslav Halchenko 5f3e1e9a16 ENH: make explicit build-ignore-errors to be used by linkfix 2015-06-25 21:30:35 -04:00
Yaroslav Halchenko 48582be9a6 DOC(ENH): specify path to rtd theme explicitly 2015-06-25 10:46:22 -04:00
Yaroslav Halchenko 49fe915798 BF: fail if docs failed to build 2015-06-25 10:12:56 -04:00
Marco DallaG 44c8ef2d47 DOC Bring Ubuntu and Archlinux outside of Windows subsection 
In the installation guide, in the "Platform specific installation notes" section, Ubuntu and ArchLinux are currently subsections of Windows, which does not make sense imho.
This commit changes the section tree from:

Platform specific installation notes
- Windows
  -- Archlinux
  -- Ubuntu

To:

Platform specific installation notes
- Windows
- Ubuntu
- Archlinux
 2015-06-24 11:44:48 +02:00
Pablo Hoffman 38e5bfb61c remove version suffix from ubuntu package 2015-06-22 10:57:24 -03:00
Julia Medina af97c373ea Update release date for 1.0 
(cherry picked from commit c89fa29c41)
 2015-06-19 18:53:54 -03:00
Elias Dorneles 2de5c66058 improvements for docs on how to access settings 2015-06-15 13:07:55 -03:00
Julia Medina f958fb9d62 Update release notes for 1.0.0rc3 2015-06-12 16:42:39 -03:00
Julia Medina fa1c25c840 Merge pull request #1286 from scrapy/configure_logging 
configure_logging: change the meaning of settings=None
 2015-06-12 13:22:42 -03:00
Julia Medina 36bc912cdd DOC indent additional docs for configure_logging 2015-06-12 13:00:31 -03:00
Bryan Crowe 6a4c475e87 Fix a couple typos 2015-06-11 19:47:30 -04:00
Daniel Graña 5bd0395be4 Merge pull request #1291 from scrapy/signalmanager-docstrings 
DOC SignalManager docstrings. See GH-713.
 2015-06-10 16:28:35 -03:00
Mikhail Korobov 6c9daf3a95 DOC remove unnecessary links; fix references in send_catch_log_deferred docstring 2015-06-10 01:44:19 +05:00
Mikhail Korobov a611f8dd2d DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring 2015-06-09 22:57:18 +05:00
Mikhail Korobov 790c67b643 DOC spider_error doesn't support deferreds 2015-06-09 02:20:10 +05:00
Mikhail Korobov 1740fcf1a6 DOC SignalManager docstrings. See GH-713. 
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
 2015-06-08 21:05:58 +05:00
Mikhail Korobov 9a787893e3 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
 2015-06-08 19:54:18 +05:00
Chris Nilsson 0c532baf4c Removed typo, and clarified time unit of setting 2015-06-06 11:18:13 +10:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Julia Medina 24d8a85269 Update release notes for 1.0.0rc2 
(cherry picked from commit 6e61d54168)
 2015-06-05 17:11:40 -03:00
Chris Nilsson eae25a04d9 Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. 
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
 2015-06-06 00:39:14 +10:00
Julia Medina 367ea81e71 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 04:11:23 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Marven Sanchez 8771d1f79b Update HTTPCache middleware docs 2015-06-01 18:20:59 +08:00
Mikhail Korobov 342cb622f1 DOC fix non-working link (by removing it). 
See https://github.com/scrapy/scrapy/pull/1260
 2015-05-27 23:04:58 +05:00
Julia Medina 343d20d791 Update 1.0 release notes 2015-05-27 11:53:54 -03:00
Pablo Hoffman 545c4224f9 update old crawlera link 2015-05-25 16:01:54 -03:00
Daniel Graña d439c26d76 update docstring and release notes 2015-05-22 20:00:58 -03:00
Julia Medina afcf70cdc6 Add 1.0 release notes 2015-05-22 12:53:11 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Julia Medina 7c61bd897c Fix in docs for error introduced in #1218 2015-05-14 20:09:19 -03:00
Daniel Graña 1195f906d6 Merge pull request #1218 from Curita/move-base-to-packages 
Move base classes to their packages
 2015-05-14 18:28:17 -03:00
Pablo Hoffman 65aa9ccc70 Merge pull request #1220 from eliasdorneles/building-settingslist-from-docs 
Building settings list from docs
 2015-05-13 14:43:19 -03:00
Elias Dorneles 6884b73e19 sort settings list by name and wrap it in a bullet list 2015-05-11 22:11:30 -03:00
Elias Dorneles 5753e498bf fixes referencing, and list only settings not documented in current document 2015-05-09 16:15:06 -03:00
Julia Medina 42c2c4b6ce Wrong topic link in docs/intro/overview.rst 2015-05-09 05:23:12 -03:00
Julia Medina c271d8f0b1 Title underline too short in docs/topics/selectors.rst 2015-05-09 05:20:54 -03:00
Julia Medina 6fd7d85448 Wrong bullet list indentation in docs/topics/media-pipeline.rst 2015-05-09 05:19:15 -03:00
Julia Medina acc13c9821 Delete tab used as indentation in docs/topics/loaders.rst 2015-05-09 05:15:17 -03:00
Julia Medina 819a8eceee Mark as orphan the doc topics not listed in the index 2015-05-09 05:12:35 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d72536688f Move scrapy/linkextractor.py to scrapy/linkextractors/__init__.py 2015-05-09 03:28:37 -03:00
bosnj 8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Elias Dorneles d1053d2a2f automatic settings list for docs [WIP] 2015-05-04 02:17:19 -03:00
Pablo Hoffman d00e43f39c restore wrongly commented line in sphinx conf.py 2015-05-01 05:24:49 -03:00
Pablo Hoffman b84b3585f2 remove empty experimental section from doc 2015-05-01 01:53:51 -03:00
Pablo Hoffman a5a8f82b06 move documentation about registering commands via setup.py to commands doc 2015-05-01 01:51:56 -03:00
Pablo Hoffman 9ee17fd537 add note to doc README about using 'make watch' 2015-05-01 01:39:03 -03:00
Julia Medina c1634e4914 Add CrawlerProcess to "Running multiple spiders[...]" doc section 2015-04-29 23:09:20 -03:00
Julia Medina d6a06a769f Add CrawlerProcess to "Run Scrapy from a script" doc section 2015-04-29 23:09:20 -03:00
Julia Medina 3a71504d35 Extend CrawlerProcess documentation 2015-04-29 23:09:20 -03:00
Julia Medina a2ce78db63 Use autoclass for CrawlerRunner in docs/topics/api.rst 2015-04-29 23:09:20 -03:00
Julia Medina 353672d292 Add autodocs Sphinx extension 2015-04-29 23:09:20 -03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Wieland Hoffmann de6501ed1b Remove a duplicate word 2015-04-29 22:31:48 +02:00
Pablo Hoffman 3d2b74a6ff Merge pull request #1188 from eliasdorneles/favoring_web_scraping_over_screen_scraping 
[MRG+1] Favoring web scraping over screen scraping in the descriptions
 2015-04-29 16:49:43 -03:00
Elias Dorneles 3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
Julia Medina 7a92dae4c8 Change Scrapy log output through docs 2015-04-22 17:27:24 -03:00
Julia Medina 69a3d58110 Basic example on manually configuring log handlers 2015-04-22 17:24:41 -03:00
Julia Medina bd0b639b21 Fix logging usage across docs 2015-04-22 17:24:41 -03:00
Julia Medina 4811d16f1d Update `logger` attr and `log` method in the Spiders topic on docs 2015-04-22 17:24:41 -03:00
Julia Medina d47a7edc65 Update Logging topic on docs 2015-04-22 17:24:40 -03:00
Julia Medina ccdd8bfbcc Parametrize log formatting strings 2015-04-22 17:24:40 -03:00
Pablo Hoffman 0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Pablo Hoffman 4c4eb4f77d Merge pull request #1180 from eliasdorneles/tutorial-improvements 
Some improvements for Scrapy tutorial
 2015-04-21 15:49:39 -03:00
Daniel Graña e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman 06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Pablo Hoffman e4122cdd79 Merge pull request #1150 from eliasdorneles/docs-files-pipeline 
Documenting Files Pipeline together with Images Pipeline
 2015-04-21 15:28:47 -03:00
Elias Dorneles d63567531d change data extraction in crawl example to be consistent with tutorial, removed statement implying mandatory usage of Item 2015-04-21 11:30:48 -03:00
Elias Dorneles f7da69d116 fixing example CSS expr 2015-04-21 11:19:10 -03:00
Elias Dorneles ff007afb9d expanded crawling primer with examples, and applied other suggestions from the review 2015-04-21 10:57:44 -03:00
Elias Dorneles 595146e158 some improvements for Scrapy tutorial 2015-04-20 21:10:07 -03:00
nyov 7ae37d61f9 remove scrapy deploy command 
(closes #1027, #1095, #1102)
 2015-04-20 20:35:06 -03:00
Julia Medina 6da091f162 Remove djangoitem topic from index 2015-04-20 20:19:28 -03:00
Daniel Graña 16a1a938c1 Add 0.24.6 release notes 2015-04-20 01:40:21 -03:00
Mikhail Korobov 1794a893f4 Merge pull request #1172 from bagratte/docs 
minor corrections in documentation.
 2015-04-19 21:41:29 +05:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
bagratte 1312bcd007 minor corrections in documentation. 2015-04-19 18:58:15 +04:00
bagratte beea9267a1 minor corrections in documentation. 2015-04-18 19:48:25 +04:00
rajathkumarmp 02629b5f7b Added link to ipython in doc. 2015-04-18 13:00:34 +05:30
Pablo Hoffman 757309efea Merge pull request #1169 from bagratte/docs 
add some minor stylistic and grammar corrections to tutorial.rst.
 2015-04-17 19:52:11 -03:00
José Ricardo dffc4549d7 Fix small typo in the docs 2015-04-17 23:37:12 +05:00
bagratte 8d339da4e5 add some minor stylistic and grammar corrections to tutorial.rst. 2015-04-17 20:55:02 +04:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
Elias Dorneles a1f3b3c7f0 images.rst -> media-pipeline.rst 2015-04-15 14:26:08 -03:00
Elias Dorneles fd1c688ad6 addressing issues from review at #1150 2015-04-15 14:21:48 -03:00
Pablo Hoffman fb85bd4b10 Merge pull request #1132 from sibiryakov/dns-options2 
[MRG+1] Dns cache size and timeout options
 2015-04-15 14:08:19 -03:00
Mikhail Korobov cb5280ab0d DOC tweak deployment docs 2015-04-15 19:25:22 +05:00
Nicolás Alejandro Ramírez Quiros e054b3c45f Merge pull request #1160 from scrapy/docs-theme 
Change local docs theme
 2015-04-15 10:34:19 -03:00
Mikhail Korobov a11bd3e89d DOC update docs readme file - sphinx >= 1.3 is now required. 2015-04-15 11:50:57 +05:00
Mikhail Korobov dd84f4bd9a DOC remove css file which isno longer needed 2015-04-15 03:44:49 +05:00
Mikhail Korobov 7dcd7f3ca7 DOC make local docs use the same theme as readthedocs.org 
sphinx_rtd_theme is builtin in recent Sphinx versions.
 2015-04-15 03:43:30 +05:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov 7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Daniel Graña e85679fa6b Merge pull request #1147 from Curita/pass-crawlers-to-runner-crawl-calls 
Allow passing Crawler instances directly to CrawlerRunner.crawl()
 2015-04-13 14:23:15 -03:00
Daniel Graña f71175a449 More replaces of references to old `sel` shortcut 2015-04-12 21:56:23 -03:00
Diana Huang 91a60d9f3d Remove references to the `sel` object in shell.rst 
The current documentation has references to the deprecated `sel` when interacting with the shell. I've removed them and replaced uses of `sel.xpath` with `response.xpath` instead.
 2015-04-12 13:44:32 -04:00
Elias Dorneles dce48b8654 documenting Files Pipeline together with Images Pipeline 2015-04-11 13:57:55 -03:00
Julia Medina 3dabde6706 Update docs for CrawlerRunner.crawl() new usage 2015-04-10 12:09:07 -03:00
Julia Medina 86b0951327 Delete `crawler_deferreds` doc in CrawlerRunner 
This attribute is now an internal one since it's main use-case was
covered by CrawlerRunner.stop().
 2015-04-10 12:06:27 -03:00
Pablo Hoffman 39fd2e62ef fix typo in deploy.rst 2015-04-09 18:01:11 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Pablo Hoffman 9ea309c354 rename deployment.rst -> deploy.rst (consist with others like debug.rst) 2015-04-09 16:56:35 -03:00
Pablo Hoffman d8184a7239 Merge pull request #1124 from rdowinton/deployment-doc 
Added deployment section covering scrapyd-deploy and shub
 2015-04-09 16:53:25 -03:00
Alexander Sibiryakov 85aa3c7596 Dns cache size and timeout options 2015-04-02 18:30:59 +02:00
nyov 92b574309e documentation build warning fixes 2015-04-01 19:46:21 +00:00
Daniel Graña 27591b55fc Merge pull request #1123 from sibiryakov/reactor-threadpool-size 
[MRG+1] Reactor threadpool max size setting
 2015-04-01 15:08:03 -03:00
Alexander Sibiryakov b794cdaf4b Broad crawls notes. 2015-04-01 12:07:03 +02:00
Alexander Sibiryakov e7b274edf3 Reformat to 80 characters per line. 2015-04-01 11:49:55 +02:00
Alexander Sibiryakov 94fceb4c15 Fixing underscore size. 2015-04-01 11:25:10 +02:00
Richard Dowinton 2d142d6401 Added deployment section covering scrapyd-deploy and shub 2015-03-31 12:17:31 +01:00
Alexander Sibiryakov 5864d291d4 Setting documentation. 2015-03-31 11:10:56 +02:00
Elias Dorneles 18c5bc75c2 some more minor grammar fixes 2015-03-27 21:00:21 -03:00
Elias Dorneles 7135c4e694 some minor grammar fixes 2015-03-27 17:56:56 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
Daniel Graña 55a23d102f Merge pull request #1086 from Curita/response-urljoin 
Add Response.urljoin() helper
 2015-03-27 15:17:54 -03:00
Julia Medina f4e241a018 Merge pull request #1106 from eliasdorneles/overview-page-improvements 
[MRG+1] some improvements to overview page
 2015-03-27 15:16:33 -03:00
Mikhail Korobov 39085ae18f Merge pull request #1098 from nyov/nyov/userconfig 
[+1 MRG]look in ~/.config/scrapy.cfg for user config
 2015-03-27 02:11:35 +05:00
nyov 1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
Peter Bronez 475766c73a Converted sel.xpath() calls to response.xpath() in Extracting the data 2015-03-26 15:34:30 -04:00
Elias Dorneles 4dcecc98f9 moved example data to a better place 2015-03-26 15:45:17 -03:00
Elias Dorneles 7402e27230 fix community link 2015-03-26 15:35:31 -03:00
Elias Dorneles 729861c864 fixing indentation 2015-03-26 15:31:42 -03:00
Elias Dorneles 13d0ecde77 addressing more review comments, to avoid ambiguity on desired reading flow 2015-03-26 15:26:16 -03:00
Elias Dorneles 76e3bf1250 addressing comments from the review plus further editing 2015-03-26 14:26:20 -03:00
Elias Dorneles 8f4a268f37 added bit about async requests, improved phrasing 2015-03-26 12:14:56 -03:00
Elias Dorneles 32423d4a33 some improvements to overview page 2015-03-25 19:27:52 -03:00
Ramiro Morales 933dbc6be6 Oops 2015-03-25 18:33:17 -03:00
Ramiro Morales ca2575001e Add missing callback arg in jobs topic example. 2015-03-25 18:32:20 -03:00
Mikhail Korobov 5ac91e4883 DOC remove Dynamic Creation of Item Classes section 
It was a hack, and dicts-as-items cover most use cases.

Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
 2015-03-23 18:11:35 +05:00
Julia Medina cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Pablo Hoffman c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Mikhail Korobov 8ac397670f DOC move .. module: declaration to a proper place 2015-03-19 21:41:36 +05:00
Faisal Anees 643984e1b4 Updated architecture.rst 
Added http://krondo.com/blog/?page_id=1327 as a resource
 2015-03-18 23:55:22 -03:00
Mikhail Korobov f16a33f34e DOC change structure of spider docs: 
* start with scrapy.Spider, then mention spider arguments,
  then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
  moved below Spider docs.
 2015-03-19 05:25:15 +05:00
Mikhail Korobov 817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Julia Medina 959aaad205 Document `re_first` 2015-03-18 21:11:08 -03:00
Mateusz Golewski 127c6c694a Fix extract_first() docs 2015-03-18 21:11:08 -03:00
Mateusz Golewski 012211accd Add docs for extract_first() 2015-03-18 21:11:08 -03:00
Daniel Graña 12eedd90d1 fix truncated 0.24.5 release notes. closes #1084 2015-03-18 20:46:28 -03:00
Julia Medina 4fb818a250 Run linkfix over current docs 2015-03-18 20:04:14 -03:00
Julia Medina ff987fb5a5 Add linkfix rule to docs Makefile 2015-03-18 20:00:03 -03:00
Shadab Zafar c05f5f175e Added linkfix script to docs/utils 
https://github.com/scrapy/scrapy/pull/1041#issuecomment-78143576
 2015-03-18 19:41:26 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov c1bf8365d5 Merge pull request #1022 from scrapy/docs-reorder 
DOC reorder topics
 2015-03-13 16:40:47 +05:00
Mikhail Korobov baf5c59386 Merge pull request #1071 from eliasdorneles/updating-request-meta-special-keys 
updating list of Request.meta special keys
 2015-03-13 16:38:19 +05:00
Elias Dorneles 57a5ee0097 added example value to set for proxy meta key 2015-03-12 23:20:44 -03:00
Elias Dorneles f7031c08ff updating list of Request.meta special keys 2015-03-10 22:29:07 -03:00
Daniel Graña c9d7386a32 Add 0.24.5 release notes 2015-02-25 19:36:40 -02:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Mikhail Korobov 6419f76bbc Merge pull request #1029 from dangra/contribute-to-master 
DOC contribute to master branch
 2015-02-03 02:10:47 +05:00
Nicolás Alejandro Ramírez Quiros 8a3b9b6131 Merge pull request #1011 from SudShekhar/master 
Extension example fix to something that makes more sense
 2015-01-30 15:45:52 -02:00
Sudhanshu Shekhar e42a1ac1a1 Reset items_scraped instead of item_count 
items_scraped is the counter that needs to be reset each time we have scraped a specific number of items in the code instead of item_count (which represents the specific number of items needed before a message is logged). Updating the source code to reflect this.
Removed some irrelevant words from the log message.
Signed-off-by: Sudhanshu Shekhar <sudshekhar02@gmail.com>
 2015-01-30 23:13:06 +05:30
Daniel Graña f0bdc14522 Tentative attention message about what document to read for contributions 2015-01-29 15:40:12 -02:00
Mikhail Korobov a07b4353d6 DOC reorder topics 
* duplicate "topics/commands" link is removed;
* Request/Response docs are moved to "Basic Concepts";
* settings docs are moved to "Basic Concepts";
* exceptions docs are moved to "Basic Concepts";
* "signals" and "exporters" docs are moved to "Extending Scrapy";
* "Reference" section is dropped because it is empty now.
 2015-01-22 22:58:10 +05:00
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Capi Etheriel 4bc14da59e Updates documentation on dynamic item classes. 
Fixes #398
 2015-01-19 17:21:56 -02:00
Mikhail Korobov 283d6a5344 DOC a couple more references are fixed 2015-01-19 22:07:03 +05:00
Mikhail Korobov 73e6b35622 DOC fix a reference 2015-01-19 22:02:46 +05:00
Artur Gaspar b0730a1d16 documentation for CSS support in link extractors 2014-12-11 18:22:08 -02:00
Stefan 3602fc4fcb fixed the variable types in mailsender documentation 2014-12-10 22:48:09 +01:00
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Jeff Paine b422312a38 Update docs copyright year range 2014-11-09 21:08:27 -05:00
Lazar-T 13f83f0da0 typo 2014-11-10 06:28:41 +05:00
HalfCrazy b21a28cc9a Afterwords->Afterwards 2014-11-10 06:28:09 +05:00
Daniel Graña 2c67bd6c57 pywin32 is required by Twisted. closes #937 
see:
* http://twistedmatrix.com/trac/ticket/6032
* https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028
 2014-11-05 23:05:54 -02:00
Daniel Graña 6cb8995731 Update install.rst 
fixes #937
 2014-11-05 22:50:52 -02:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Lazar-T 38dcf50cd6 comma instead of fullstop 2014-10-25 09:19:50 +06:00
Pablo Hoffman 675fd5ba04 Merge pull request #898 from scrapy/download-timeout 
[MRG] DOC document download_timeout
 2014-10-24 16:52:42 -02:00
Pablo Hoffman 0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Mikhail Korobov 7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Mikhail Korobov ea3b372b4f DOC typo fix in leaks.rst 2014-10-02 15:20:13 +06:00
Pablo Hoffman e7843d35de Merge pull request #894 from kmike/leaks-docs 
Leaks docs
 2014-10-02 01:14:54 -03:00
Pablo Hoffman 5835224eee Merge pull request #896 from scrapy/robotstxt-once 
[MRG] process robots.txt once
 2014-10-02 00:58:55 -03:00
Mikhail Korobov 6fcf9dce50 DOC document from_crawler method for item pipelines; add an example. 2014-09-25 03:13:51 +06:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov bdbca1e2d7 DOC request queue memory usage 2014-09-21 07:30:44 +06:00
Mikhail Korobov bc0f481a73 DOC bring back notes about multiple spiders per process because it is now documented how to do that 2014-09-21 07:12:01 +06:00
Mikhail Korobov a122fdbfea Update leaks.rst: there is now only a single spider in a process. 2014-09-21 06:54:00 +06:00
Mikhail Korobov e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
John-Scott Atlakson a312ebfb43 Update request-response.rst 
Fixed minor typo
 2014-09-14 22:06:31 +06:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Julia Medina 16e62e9c9b Per-spider settings documentation 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
Mikhail Korobov 774ab74ad2 Merge pull request #864 from younghz/master 
Duplicate comma in request-response.rst
 2014-08-28 18:52:51 +06:00
Uyounghz d49766a6ac Duplicate comma in request-response.rst 2014-08-28 19:58:58 +08:00
Daniel Graña 841dd5f1f5 Update webservice.rst 2014-08-18 17:48:01 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 3547ca6e61 Add example on running spiders outside projects 2014-08-14 11:50:33 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00