RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,312 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

214 Commits

Author SHA1 Message Date
Vostretsov Nikita 6bd88711f2 update documentation 2019-09-10 08:55:37 +00:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 92fa657f9e
Merge pull request #3730 from ankostis/doc_logstats_delay 
[MRG+1] doc: document LOGSTATS_INTERVAL setting
 2019-04-11 17:34:45 +02:00
Kostis Anagnostopoulos a8f83ab967 doc: document LOGSTATS_INTERVAL setting 2019-04-06 14:58:32 +02:00
Adrián Chaves 1ee99e1f42
Update docs/topics/settings.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:29:15 +00:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Anubhav Patel 66a502db8c
Merge branch 'master' into logFormatter-doc-patch 2019-03-15 00:28:54 +05:30
Mikhail Korobov 54d106033f
Merge pull request #3649 from Gallaecio/inline-code-formatting 
[MRG+1] Switch from ` to `` where inline code formatting is desired
 2019-03-14 22:18:57 +05:00
Anubhav Patel 82049e9c41 make suggested changes. 2019-03-10 20:17:07 +05:30
Adrián Chaves e108e3adbf Clarify the documentation of DEPTH_PRIORITY further 2019-03-08 15:13:11 +01:00
Anubhav Patel 924b67437b move api docs to source code 2019-03-07 16:40:59 +05:30
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Ryan P Kilby 2dfc5d128b Update DEPTH_STATS refs to DEPTH_STATS_VERBOSE 2018-05-17 11:00:21 -04:00
Daniel Graña 2dee191374
Merge branch 'master' into extending-s3-files-store 2017-12-31 16:44:38 -03:00
Aditya df7e0a4315 Use https link in default user agent 2017-10-28 23:37:44 +05:30
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Mikhail Korobov 083880888b DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs 2017-05-30 00:44:11 +05:00
Paul Tremberth 8aa2e4f997 Merge pull request #1829 from nyov/nyov/editor 
[MRG+1] Remove dependency on os.environ from default settings
 2017-05-18 21:32:51 +02:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Tiago Cardoso 00ee9eaeaf Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting 2017-04-22 14:36:44 +01:00
Oto Brglez 11cf6ad425 Comments for AWS_ENDPOINT_URL setting. 2017-03-19 12:48:06 +01:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Paul Tremberth d35a01a103 Update default password 2017-02-20 14:23:23 +01:00
Paul Tremberth b80e1bb6c5 Document new FTP_* settings 2017-02-20 14:19:36 +01:00
Mikhail Korobov 85a124970a Enable memusage extension by default. Fixes GH-2187. 2017-02-07 03:32:54 +05:00
nyov 2240f00a13 Remove dependency on os.environ from default settings 
Avoid loading settings from environment in scrapy core.
Instead it's better to populate them from the starting
shell or an embedding script.
 2016-12-26 20:49:16 +00:00
Mikhail Korobov 0fc73a9d55 DOC update examples with long longger names 2016-12-16 21:47:58 +05:00
Mikhail Korobov 05cec0f2f3 fixed ReST syntax 2016-12-09 00:04:04 +05:00
Akhil Lb a75ad2bbc6 LOG_SHORT_NAMES option 2016-12-09 00:04:04 +05:00