RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,575 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

1241 Commits

Author SHA1 Message Date
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Elias Dorneles d4c4ca8062 fix version number to appear new feature 2016-01-21 09:42:15 -02:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Daniel Graña b1e44436bc Merge pull request #1642 from orangain/doc-metarefresh 
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
 2016-01-19 13:33:50 -03:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
Daniel Graña 1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
orangain 4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Rolando Espinoza f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Pablo Hoffman eb4daa34a2 Merge pull request #1544 from rolando/conda-docs 
Added installation notes about using Conda for Windows.
 2015-11-26 14:11:28 -03:00
Rolando Espinoza cc8ddb68b0 Added installation notes about using Conda for Windows and other OSes. 2015-11-26 13:08:56 -04:00
Julia Medina 48539af163 Merge pull request #1601 from mvj3/minor-documentation-fixes 
Fixed minor grammar issues.
 2015-11-16 14:58:07 -03:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Valdir Stumm Jr d577c4702d fixed a typo in the documentation. 2015-10-26 00:00:20 -02:00
Christian Pedersen cc76fb3dff Version 1 now exists 2015-10-14 14:07:34 +02:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
hoatle 2869cf8dde fix another invalid xpath error 2015-10-07 16:03:43 +07:00
Hoat Le 4e66955411 fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst 
>>> response.xpath('//div/[id="not-exists"]/text()').extract_first() is None
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/http/response/text.py", line 109, in xpath
    return self.selector.xpath(query)
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/selector/unified.py", line 100, in xpath
    raise ValueError(msg if six.PY3 else msg.encode("unicode_escape"))
ValueError: Invalid XPath: //div/[id="not-exists"]/text()
 2015-10-07 15:43:02 +07:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
smirecki 8379bea7ed Typos corrections 
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.

 Please enter the commit message for your changes. Lines starting
 2015-10-02 23:48:27 -04:00
Marius Gedminas 0620e76433 Fix list formatting 2015-09-29 03:33:30 +05:00
Marius Gedminas eaad10facf Typo 2015-09-28 14:44:15 +05:00
Daniel Graña 3c596dcf46 Merge pull request #1467 from dacjames/master 
add support for a nested loaders
 2015-09-16 22:03:19 -03:00
hy 14f7f22555 fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware 2015-09-16 16:59:23 +08:00
Daniel Collins 036109e7de updte nested loader documentation 2015-09-15 23:49:35 -07:00
Mikhail Korobov 600f27c39b Merge pull request #1350 from scrapy/installation-docs-update 
Updates in Installation and Contributing sections in the docs
 2015-09-09 17:54:35 +05:00
Julia Medina eb7b9d506f Add note to ubuntu install section about debian compatibility 2015-09-08 18:49:44 -03:00
Daniel Collins 88c92cb68b provide documentation for nested loaders 2015-09-04 13:15:48 -07:00
Daniel Graña dd473145f2 Merge pull request #1473 from ArturGaspar/master 
[MRG+1] Support for returning deferreds in middlewares
 2015-09-02 13:04:26 -03:00
Robert Weindl 833efbfcbd Update tutorial.rst 
Add missing "scrapy." prefix.
 2015-09-02 20:35:23 +05:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Artur Gaspar ca83a0b028 Support for returning deferreds in downloader middleware methods. 2015-09-01 13:22:43 -03:00
Daniel Graña b4e382c867 Merge pull request #1318 from yarikoptic/fix-docs-explicit-rtd-theme-path 
[MRG+1] DOC(ENH): specify path to rtd theme explicitly
 2015-08-31 10:42:02 -03:00
David Tagatac 08162a15d8 minor: scrapy.Spider docs grammar 2015-08-27 17:37:16 -04:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Rick aa88fd797a Small grammatical change 2015-08-26 15:01:50 -03:00
Mikhail Korobov cfae62f9cc Merge pull request #1441 from aivarsk/fix-common-practices 
Make common practices sample code match the comments
 2015-08-23 17:36:09 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Aivars Kalvāns b8b1e8e544 Make common practices sample code match the comments 2015-08-19 16:54:10 +03:00
Julia Medina d3c3408cd4 Replace alternative OSX install workaround with virtualenv 2015-08-19 03:19:27 -03:00
Julia Medina 46ee1c6ee2 Reference Homebrew's homepage for installation instructions 2015-08-19 02:21:08 -03:00
Daniel Graña 54f9837804 Document how to get coverage report 2015-08-13 19:02:51 -03:00
Daniel Graña f67a808d9e Add 1.0.3 release notes 2015-08-11 13:49:49 -03:00
Daniel Graña 57fafc702d Add 1.0.2 release notes 2015-08-06 18:02:12 -03:00
Mikhail Lyundin ea4199f8e1 Minor method name fix 2015-08-05 22:42:53 +03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
David Tagatac 08123207c5 minor: scrapy.Spider grammar and clarity 2015-07-31 17:01:59 -04:00
Mikhail Korobov b6eb3404a2 Merge pull request #1335 from jdemaeyer/fix/docs-scrapy-path 
Make Sphinx autodoc use local, not system-wide Scrapy
 2015-07-30 15:33:02 +05:00
Jakob de Maeyer 9c12a3f23a Remove incorrect path for Sphinx autodoc 2015-07-30 12:12:08 +02:00
Andrew Murray 198be7d679 Fixed typos 2015-07-16 23:59:02 +10:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina 320d3a59b9 Add oldest supported tox version to contributing docs 
Better fix for #1337
 2015-07-10 20:34:44 -03:00
Julia Medina 439b376d5b Note in install docs about pip being already included in python>=2.7.9 2015-07-10 20:34:43 -03:00
Julia Medina 5723e6b457 Add non-python dependencies to Ubuntu install section in the docs 
Closes #1314 and closes #1198.
 2015-07-10 20:34:43 -03:00
Julia Medina 290ebee090 Add OS X installation section to docs 
Closes #1342 and (possibly) #1126
 2015-07-10 20:34:43 -03:00
Nikola Pavlović 7bef61dbbd Fix reference. 2015-07-10 02:35:27 +02:00
Jakob de Maeyer d438075a47 Make Sphinx autodoc use local, not system-wide Scrapy 2015-07-03 16:51:53 +02:00
Julia Medina 8b3ca4f250 Merge pull request #1302 from eliasdorneles/improving-access-settings-docs 
[MRG+1] Improvements for docs on how to access settings
 2015-07-03 00:56:32 -03:00
Daniel Graña 3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Daniel Graña 6fa3f2421c Add 1.0.1 release notes 2015-07-01 01:49:44 -03:00
Mikhail Korobov d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Mikhail Korobov 63317531f9 DOC fix authrottle docs 
see https://github.com/scrapy/scrapy/pull/502/files#r8574692
 2015-06-26 20:47:58 +05:00
Yaroslav Halchenko 5f3e1e9a16 ENH: make explicit build-ignore-errors to be used by linkfix 2015-06-25 21:30:35 -04:00
Yaroslav Halchenko 48582be9a6 DOC(ENH): specify path to rtd theme explicitly 2015-06-25 10:46:22 -04:00
Yaroslav Halchenko 49fe915798 BF: fail if docs failed to build 2015-06-25 10:12:56 -04:00
Marco DallaG 44c8ef2d47 DOC Bring Ubuntu and Archlinux outside of Windows subsection 
In the installation guide, in the "Platform specific installation notes" section, Ubuntu and ArchLinux are currently subsections of Windows, which does not make sense imho.
This commit changes the section tree from:

Platform specific installation notes
- Windows
  -- Archlinux
  -- Ubuntu

To:

Platform specific installation notes
- Windows
- Ubuntu
- Archlinux
 2015-06-24 11:44:48 +02:00
Pablo Hoffman 38e5bfb61c remove version suffix from ubuntu package 2015-06-22 10:57:24 -03:00
Julia Medina af97c373ea Update release date for 1.0 
(cherry picked from commit c89fa29c41)
 2015-06-19 18:53:54 -03:00
Elias Dorneles 2de5c66058 improvements for docs on how to access settings 2015-06-15 13:07:55 -03:00
Julia Medina f958fb9d62 Update release notes for 1.0.0rc3 2015-06-12 16:42:39 -03:00
Julia Medina fa1c25c840 Merge pull request #1286 from scrapy/configure_logging 
configure_logging: change the meaning of settings=None
 2015-06-12 13:22:42 -03:00
Julia Medina 36bc912cdd DOC indent additional docs for configure_logging 2015-06-12 13:00:31 -03:00
Bryan Crowe 6a4c475e87 Fix a couple typos 2015-06-11 19:47:30 -04:00
Daniel Graña 5bd0395be4 Merge pull request #1291 from scrapy/signalmanager-docstrings 
DOC SignalManager docstrings. See GH-713.
 2015-06-10 16:28:35 -03:00
Mikhail Korobov 6c9daf3a95 DOC remove unnecessary links; fix references in send_catch_log_deferred docstring 2015-06-10 01:44:19 +05:00
Mikhail Korobov a611f8dd2d DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring 2015-06-09 22:57:18 +05:00
Mikhail Korobov 790c67b643 DOC spider_error doesn't support deferreds 2015-06-09 02:20:10 +05:00
Mikhail Korobov 1740fcf1a6 DOC SignalManager docstrings. See GH-713. 
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
 2015-06-08 21:05:58 +05:00
Mikhail Korobov 9a787893e3 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
 2015-06-08 19:54:18 +05:00
Chris Nilsson 0c532baf4c Removed typo, and clarified time unit of setting 2015-06-06 11:18:13 +10:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00