RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,360 Commits 49 Branches 132 Tags 158 MiB
Commit Graph

995 Commits

Author SHA1 Message Date
Daniel Graña b9d3b447a5 Merge pull request #2670 from qhuang872/master 
[MRG+1] Update spiders.rst
 2017-05-18 16:31:01 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Mikhail Korobov 1acacab93b Merge pull request #2741 from eliat123/2576_cleanup_MEMUSAGE_REPORT 
[FIX #2576] cleanup: removed unused MEMUSAGE_REPORT
 2017-05-17 13:48:00 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Harrison Gregg 45a323024c Add documentation for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Paul Tremberth 3a0a86ed31 Clarify FEED_EXPORT_INDENT section 2017-05-12 17:26:17 +02:00
Eugenio Lacuesta 25535dba9c Feed exports: edit note, fix typos 2017-05-10 16:45:15 -03:00
Kurt Peek 548a432951 Minor grammatical changes 2017-05-10 23:48:01 +05:00
yandongxu 4966dd7a7f Fix doc: open file with "wb" mode will get an error in python 3 2017-05-08 18:50:30 +08:00
Liu Siyuan 6c1cacb5d5 [MRG+1] doc: fix documentation error in link-extractor.rst (#2676) 
* fix doc error in link-extractor.rst

* remove the import clause

* update based on suggestion
 2017-05-06 02:47:06 +05:00
Martín Gaitán 4bc0c6b0f4 Update practices.rst 
fix a typo
 2017-04-26 14:12:57 +05:00
Tiago Cardoso 00ee9eaeaf Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting 2017-04-22 14:36:44 +01:00
Mikhail Korobov 9731077a54 Merge pull request #2683 from harshasrinivas/docs-SelectorList 
[MRG+1] Remove __nonzero__ from SelectorList docs
 2017-04-12 21:25:00 +05:00
Paul Tremberth 422b38f65c DOC Rearrange selector sections 2017-04-11 16:55:43 +02:00
harshasrinivas 2ff6b05723 Remove __nonzero__ from SelectorList docs 2017-03-24 20:43:28 +05:30
Qiwei Huang 8ecc307e8f Update spiders.rst 
Added note to allowed_domain attribute with an example explaining what goes in the list
 2017-03-20 19:37:07 -07:00
Qiwei Huang c5f74f7d1a Update spiders.rst 
Added a note to allowed_domains attribute, reminding users not to add urls into the list.
 2017-03-20 18:52:33 -07:00
harshasrinivas 10741aca72 Update docs - improve clarity 2017-03-19 06:17:28 +05:30
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
harshasrinivas 9d97d788c0 Update docs for meta key 2017-03-15 04:13:47 +05:30
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Bernardas 8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas 25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth 605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth 03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00