RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

1007 Commits

Author SHA1 Message Date
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña e578be73b6 Merge pull request #2539 from scrapy/enable-memusage 
Enable memusage extension by default.
 2017-02-20 11:18:04 -03:00
Daniel Graña fab168bbfb Merge pull request #2572 from advarisk/warning-formrequest-from-response 
[MRG+1] document issue with FormRequest.from_response due to bug in lxml
 2017-02-20 11:14:28 -03:00
Daniel Graña 85ef6a6229 Merge pull request #2564 from elacuesta/docs_exporters 
[MRG+1] Doc: binary mode is required for exporters
 2017-02-20 11:12:48 -03:00
Paul Tremberth d35a01a103 Update default password 2017-02-20 14:23:23 +01:00
Paul Tremberth b80e1bb6c5 Document new FTP_* settings 2017-02-20 14:19:36 +01:00
Ashish Kulkarni 165e2cb8c9 document issue with FormRequest.from_response due to bug in lxml 
This can make the spider fail due to incorrect values being posted
server-side, which is extremely hard to debug because it is easy
to miss leading/trailing whitespace, even with a logging proxy.

The fix was merged for lxml 3.8 in lxml/lxml#228 so document that
as well.
 2017-02-17 14:54:22 +05:30
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d09eed7674 use w3lib.html.strip_html5_whitespace function; expand docs; strip consistently before calling process_value 2017-02-16 02:22:18 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov 5b79c6a679 DOC document response.follow methods; expand the tutorial 2017-02-16 00:06:52 +05:00
Mikhail Korobov 877057fac0 initial response.follow implementation 2017-02-15 01:22:53 +05:00
Eugenio Lacuesta 922d3fec54 Doc: binary mode is required for exporters 2017-02-14 12:51:03 -03:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth d205206aaa Merge pull request #2345 from gustavodeandrade/master 
[MRG+1] Fix documentation about HTML entities decoding with selector extraction
 2017-02-08 13:21:15 +01:00
Paul Tremberth 7d0b89042f Merge pull request #2533 from djrobust/patch-1 
[MRG+1] Use yield with nested parsing of responses
 2017-02-08 13:02:50 +01:00
Mikhail Korobov 85a124970a Enable memusage extension by default. Fixes GH-2187. 2017-02-07 03:32:54 +05:00
Mikhail Korobov 3b8e6d4d82 Merge pull request #2531 from Lukas0907/patch-1 
Fix typo in downloader-middleware.rst.
 2017-02-06 13:09:35 +05:00
Takehiro Shiozaki fcb3daf4fa fix typo 2017-02-06 14:03:41 +09:00
djrobust 3021084f37 Use 'yield' when parsing multiple responses 
Use 'yield' consistently across examples of parse functions.
 2017-02-04 20:07:05 -08:00
Lukas Anzinger 09643796b4 Fix typo in downloader-middleware.rst. 2017-02-03 20:05:17 +01:00
Paul Tremberth 1c0b805357 DOC Mention XPath variables in Selectors section 2017-02-02 17:44:57 +01:00
Mikhail Korobov c305c46254 Merge pull request #2503 from redapple/view-no-redirect 
[MRG] Fix view command against new --no-redirect option
 2017-02-02 21:24:56 +05:00
Mikhail Korobov 55742c0392 DOC mention LevelDB cache storage backend 2017-02-01 22:43:28 +05:00
Paul Tremberth 4156a86148 Update docs on view command 2017-01-30 15:57:37 +01:00
Raul Gallegos df1a42419f adding formid to FormRequest documentation 2017-01-14 20:45:20 -05:00
Paulius Aleksiūnas 5586fc7e38 Update architecture.rst 
In the data flow image arrows are red.
 2017-01-10 10:40:27 +01:00
Paul Tremberth bf2277a028 Update spiders.rst 2017-01-10 10:27:38 +01:00
Νικόλαος-Διγενής Καραγιάννης 900b6710d3 Document copying of spider arguments to attributes 2017-01-09 22:12:38 +02:00
nyov 2240f00a13 Remove dependency on os.environ from default settings 
Avoid loading settings from environment in scrapy core.
Instead it's better to populate them from the starting
shell or an embedding script.
 2016-12-26 20:49:16 +00:00
Elias Dorneles d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Paul Tremberth 140a57d7b0 Amend note on --no-redirect option for shell tool 2016-12-19 17:54:54 +01:00
Mikhail Korobov da19f0b7b7 DOC how to override log level for a specific Scrapy component 2016-12-16 22:14:54 +05:00
Mikhail Korobov 0fc73a9d55 DOC update examples with long longger names 2016-12-16 21:47:58 +05:00
Mikhail Korobov 05cec0f2f3 fixed ReST syntax 2016-12-09 00:04:04 +05:00
Akhil Lb a75ad2bbc6 LOG_SHORT_NAMES option 2016-12-09 00:04:04 +05:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
Mikhail Korobov bd8c293a97 Merge pull request #2411 from redapple/remove-chunked-transfer-mw 
[MRG+1] Remove ChunkedTransferMiddleware from default settings
 2016-12-07 22:05:15 +05:00
Mikhail Korobov ff3aec6613 Merge pull request #2331 from moisesguimaraes/fixes-2272 
[MRG+1] Fixes issue #2272 using arg_to_iter() to wrap single values and list() to…
 2016-12-07 20:08:18 +05:00
Elias Dorneles a9c69458ff Merge pull request #2422 from rolando-contrib/nested-spiders-modules 
[MRG+1] DOC State explicitly that spiders are loaded recursively.
 2016-12-07 11:21:15 -02:00
Moisés Guimarães c08d278c0c removes note from docs. 2016-12-05 16:47:24 -03:00
Moisés Guimarães a4178f99da fixes params types in docs. 2016-12-05 15:24:26 -03:00
Moisés Guimarães c58ea021b8 fixes docs 2016-12-04 11:56:14 -03:00
Rolando Espinoza 6431e7a138 DOC State explicitly that spiders are loaded recursively. 2016-12-01 13:26:19 -03:00
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 059085b5b4 Remove docs for deprecated ChunkedTransfer middleware 2016-11-24 18:23:34 +01:00
Paul Tremberth 03cf5f1bd2 Remove ChunkedTransferMiddleware from default settings 2016-11-24 17:18:57 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Rahul Kant f56aef99c2 Add closing tag in <tbody> 2016-11-07 17:49:22 +05:30
Gustavo de Andrade f2e49bc23c Update selectors.rst 
Decode instead escape, exceptions &lt; and &amp; (kmike)
Second sentence droped (Digenis)
 2016-11-01 21:32:17 -02:00
Mikhail Korobov 9755ef933a Merge pull request #2369 from jc0n/media-pipeline-docs-typo 
Fix typo in media pipeline docs
 2016-10-29 12:46:48 +06:00
John O'Connor d85da273be Fix typo in media pipeline docs 2016-10-28 19:44:46 -07:00
Steven Almeroth 99daea495b Doc: wording 2016-10-21 16:16:42 -07:00
Steven Almeroth a958e54954 Doc: remove trailing spaces 2016-10-21 16:16:37 -07:00
gustavodeandrade c7dfb5eb88 Fix issue 1704 2016-10-21 00:08:08 -02:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00
bopace fd016ee71b Fixed wording of documentation 2016-10-18 09:37:45 -06:00
Paul Tremberth 6cc83c041e Merge pull request #2330 from lfmattossch/note-python2-name 
[MRG+1] Added a note about invalid spider names in python 2 (fixes #2251)
 2016-10-18 16:35:19 +02:00
Jose Ricardo e12e364a40 Add details to the spider middlewares docs 
Document the effects of the middleware order in a more detailed way.
 2016-10-18 12:29:30 -02:00
Luiz Fernando Mattos Schlindwein 7c33e0cb55 added a note about invalid spider names in python 2 2016-10-18 11:55:51 -02:00
Jose Ricardo ea7bd39529 Make architecture overview references a little more clear on the docs 
Expliciting what actually happens by adding links to the respective methods
that are invoked in each processing phase.
 2016-10-18 11:50:51 -02:00
Jose Ricardo bebcd5081c Add downloader middleware ordering details to the docs 
Add more details, making it easier to understand what are the effects of
setting a downloader middleware order.
 2016-10-18 11:22:55 -02:00
Bo Pace bfe28ae707 Added documentation about accessing header values 2016-10-17 14:10:05 -06:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
Wieland Hoffmann e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Paul Tremberth fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles 8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Elias Dorneles 86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles 3ac3ac4d92 docs: update data flow description and image (fixes: #2278) 
This fixes the explanation to use Requests instead of URLs,
which is what actually happens, and is also consistent with the
new tutorial, which already explains how URLs become Request objects.

I've also changed the "loop", jumping from 9 to step 2.
 2016-09-28 16:38:45 -03:00
pawelmhm 39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00
Pawel Miech b2bfd1e5c5 [docs] document that process_item can return Deferred 2016-09-23 10:36:03 +02:00
Paul Tremberth e59d79bf37 Add note on "to" and "cc" as lists for sending emails 
Fixes GH-2244
 2016-09-20 17:18:49 +02:00
Ashish Kulkarni 8893527a2f temporarily deprecate official Ubuntu packages 
They are not currently updated and fail to install on
Ubuntu 16.04. Also update the instructions to refer to
the earliest supported LTS (Ubuntu 12.04).

fixes #2137 and closes #2076
 2016-09-20 11:10:59 +02:00
Elias Dorneles 063315258e Merge pull request #2202 from scrapy/doc-arch-overview2 
[MRG+1] DOC move Data Flow below the picture; add links to components
 2016-09-19 08:11:18 -03:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00
Mikhail Korobov 495d322691 DOC move Data Flow below the picture; add links to components 2016-08-26 20:16:22 +05:00
Thom Dixon f68dc3026d Fix indentation 2016-08-24 09:11:27 -07:00
Thom Dixon 633abfbea1 Correct documentation about Response parameters 
This fixes issue #2196
 2016-08-24 08:47:52 -07:00