RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,344 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

1511 Commits

Author SHA1 Message Date
Mikhail Korobov 1acacab93b Merge pull request #2741 from eliat123/2576_cleanup_MEMUSAGE_REPORT 
[FIX #2576] cleanup: removed unused MEMUSAGE_REPORT
 2017-05-17 13:48:00 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Harrison Gregg 45a323024c Add documentation for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Paul Tremberth 3a0a86ed31 Clarify FEED_EXPORT_INDENT section 2017-05-12 17:26:17 +02:00
Eugenio Lacuesta 25535dba9c Feed exports: edit note, fix typos 2017-05-10 16:45:15 -03:00
Kurt Peek 548a432951 Minor grammatical changes 2017-05-10 23:48:01 +05:00
yandongxu 4966dd7a7f Fix doc: open file with "wb" mode will get an error in python 3 2017-05-08 18:50:30 +08:00
Liu Siyuan 6c1cacb5d5 [MRG+1] doc: fix documentation error in link-extractor.rst (#2676) 
* fix doc error in link-extractor.rst

* remove the import clause

* update based on suggestion
 2017-05-06 02:47:06 +05:00
Martín Gaitán 4bc0c6b0f4 Update practices.rst 
fix a typo
 2017-04-26 14:12:57 +05:00
Paul Tremberth aa59cf4241 Merge pull request #2668 from harshasrinivas/docs-sphinx-rtd-theme 
[MRG+1] Add sphinx_rtd_theme to docs setup readme
 2017-04-24 12:35:40 +02:00
Tiago Cardoso 00ee9eaeaf Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting 2017-04-22 14:36:44 +01:00
Mikhail Korobov 9731077a54 Merge pull request #2683 from harshasrinivas/docs-SelectorList 
[MRG+1] Remove __nonzero__ from SelectorList docs
 2017-04-12 21:25:00 +05:00
Paul Tremberth 422b38f65c DOC Rearrange selector sections 2017-04-11 16:55:43 +02:00
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
harshasrinivas 2ff6b05723 Remove __nonzero__ from SelectorList docs 2017-03-24 20:43:28 +05:30
harshasrinivas 0298bcbe79 Update Makefile to open webbrowser in MacOS (#2661) 2017-03-24 18:13:08 +05:30
harshasrinivas 83aa0c5e1a Clarify docs readme 2017-03-20 22:36:09 +05:30
harshasrinivas 4ec07ae764 Create docs/requirements.txt 2017-03-20 22:21:08 +05:30
harshasrinivas a57e49d55b Add sphinx_rtd_theme to docs setup readme 2017-03-20 19:49:31 +05:30
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00
jorenham ac63d3a3cf Removed contrib section; contrib is deprecated 2017-03-09 10:56:23 +01:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth a8b47d6c68 Update release notes for 1.0.7, 1.1.4 and 1.2.3 2017-03-06 14:25:52 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Bernardas 8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas 25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth 605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth 03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00