RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 151 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Vostretsov Nikita 7b33fa58fa
Update requirements-py2.txt 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:27 +03:00
Vostretsov Nikita 38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita e418554c21 use proper equal 2019-09-09 08:12:32 +00:00
Vostretsov Nikita 7af8c76649 add pinned versions 2019-09-09 08:10:09 +00:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Mikhail Korobov d4b8bf18b2
Merge pull request #3993 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.extensions
 2019-08-30 21:07:22 +05:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Eugenio Lacuesta eb0bd2daef
Revert backward-incompatible change (contract priorities) 2019-08-29 14:01:13 -03:00
Júlio César Batista b84f99ff5d Merge 2019-08-29 11:11:56 -03:00
Júlio César Batista 110bc92e6b Fix default value of FEED_STORAGE_GCS_ACL 2019-08-29 11:10:00 -03:00
Eugenio Lacuesta 97a7d775f7
Aplly suggestions by @victor-torres 2019-08-29 10:51:16 -03:00
Eugenio Lacuesta b6b76df057
CallbackKeywordArgumentsContract 2019-08-29 03:29:24 -03:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Mikhail Korobov b00f81c52b
Merge pull request #3978 from anubhavp28/doc-link-fix 
Fixes a link in docs
 2019-08-28 21:58:25 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta 3abe7e6e6d Add Bug report and Feature request templates (#3471) 2019-08-26 09:35:44 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
OmarFarrag 97d2f717ae Support extracting ftp settings in `ImagesPipeline` 2019-08-22 16:19:01 +02:00
OmarFarrag 2047124b35 Follow PEP8 .. Remove unnecessary comments 2019-08-22 16:18:14 +02:00
OmarFarrag bd22b25ef4 Make `stat_file` thread safe .. Refactor file storing.. Support act/psv 2019-08-22 01:30:15 +02:00
OmarFarrag 8c970c636e
port from str to int 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-08-21 18:28:36 +02:00
Mikhail Korobov 56948c446c
Merge pull request #3442 from wRAR/ciphers 
Support for overriding OpenSSL ciphers
 2019-08-19 23:15:09 +05:00
OmarFarrag 790bf90312 Make FTP persiting files thread safe 2019-08-19 19:16:47 +02:00
OmarFarrag 81ac1da381 Handle leading and trailing slashes 2019-08-19 17:17:21 +02:00
OmarFarrag 0a5cb7745b Fix reference mistake 2019-08-19 17:12:11 +02:00
OmarFarrag 9b1587ed1b Credentials from settings-Support custom paths-Remove close conenction 2019-08-19 16:13:56 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
OmarFarrag c025003da2 Add FTPFileStore 2019-08-18 04:44:09 +02:00
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin 3384db92b4 Add support for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov a95de71d8e
Merge pull request #3950 from elacuesta/version_updates 
Remove obsolete version checks
 2019-08-12 22:51:00 +05:00
Mikhail Korobov 2f0c46e762
Merge pull request #3946 from elacuesta/simplify_versions 
[MRG+1] Simplify version reporting
 2019-08-10 14:25:59 +05:00
Eugenio Lacuesta 26fb28b20f
PEP8-ify HTTP/1.1 downloader handler 
Signed-off-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2019-08-09 00:56:24 -03:00
Eugenio Lacuesta d5dcc5eaef
Import twisted.web.client.URI directly 2019-08-09 00:30:58 -03:00
Eugenio Lacuesta d3737d869b
Remove check for Twisted>=14.0 2019-08-09 00:21:43 -03:00
Eugenio Lacuesta e17c9a48fd
Remove check for Twisted>=15.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 23:59:17 -03:00
Eugenio Lacuesta d92f1b1858
Simplify import + assignment 2019-08-08 23:53:35 -03:00
Eugenio Lacuesta b404941e0d
Remove import check for service_identity 
service_identity.exceptions.CertificateError is available in the current minimum version (16.0.0)
 2019-08-08 23:18:47 -03:00