RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 49 Branches 132 Tags 158 MiB
Commit Graph

9472 Commits

Author SHA1 Message Date
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Mikhail Korobov d4b8bf18b2
Merge pull request #3993 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.extensions
 2019-08-30 21:07:22 +05:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Eugenio Lacuesta eb0bd2daef
Revert backward-incompatible change (contract priorities) 2019-08-29 14:01:13 -03:00
Júlio César Batista b84f99ff5d Merge 2019-08-29 11:11:56 -03:00
Júlio César Batista 110bc92e6b Fix default value of FEED_STORAGE_GCS_ACL 2019-08-29 11:10:00 -03:00
Eugenio Lacuesta 97a7d775f7
Aplly suggestions by @victor-torres 2019-08-29 10:51:16 -03:00
Eugenio Lacuesta b6b76df057
CallbackKeywordArgumentsContract 2019-08-29 03:29:24 -03:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Mikhail Korobov b00f81c52b
Merge pull request #3978 from anubhavp28/doc-link-fix 
Fixes a link in docs
 2019-08-28 21:58:25 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta 3abe7e6e6d Add Bug report and Feature request templates (#3471) 2019-08-26 09:35:44 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
OmarFarrag 97d2f717ae Support extracting ftp settings in `ImagesPipeline` 2019-08-22 16:19:01 +02:00
OmarFarrag 2047124b35 Follow PEP8 .. Remove unnecessary comments 2019-08-22 16:18:14 +02:00
OmarFarrag bd22b25ef4 Make `stat_file` thread safe .. Refactor file storing.. Support act/psv 2019-08-22 01:30:15 +02:00
OmarFarrag 8c970c636e
port from str to int 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-08-21 18:28:36 +02:00
Mikhail Korobov 56948c446c
Merge pull request #3442 from wRAR/ciphers 
Support for overriding OpenSSL ciphers
 2019-08-19 23:15:09 +05:00
OmarFarrag 790bf90312 Make FTP persiting files thread safe 2019-08-19 19:16:47 +02:00
OmarFarrag 81ac1da381 Handle leading and trailing slashes 2019-08-19 17:17:21 +02:00
OmarFarrag 0a5cb7745b Fix reference mistake 2019-08-19 17:12:11 +02:00
OmarFarrag 9b1587ed1b Credentials from settings-Support custom paths-Remove close conenction 2019-08-19 16:13:56 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
OmarFarrag c025003da2 Add FTPFileStore 2019-08-18 04:44:09 +02:00
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin 3384db92b4 Add support for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov a95de71d8e
Merge pull request #3950 from elacuesta/version_updates 
Remove obsolete version checks
 2019-08-12 22:51:00 +05:00
Mikhail Korobov 2f0c46e762
Merge pull request #3946 from elacuesta/simplify_versions 
[MRG+1] Simplify version reporting
 2019-08-10 14:25:59 +05:00
Eugenio Lacuesta 26fb28b20f
PEP8-ify HTTP/1.1 downloader handler 
Signed-off-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2019-08-09 00:56:24 -03:00
Eugenio Lacuesta d5dcc5eaef
Import twisted.web.client.URI directly 2019-08-09 00:30:58 -03:00
Eugenio Lacuesta d3737d869b
Remove check for Twisted>=14.0 2019-08-09 00:21:43 -03:00
Eugenio Lacuesta e17c9a48fd
Remove check for Twisted>=15.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 23:59:17 -03:00
Eugenio Lacuesta d92f1b1858
Simplify import + assignment 2019-08-08 23:53:35 -03:00
Eugenio Lacuesta b404941e0d
Remove import check for service_identity 
service_identity.exceptions.CertificateError is available in the current minimum version (16.0.0)
 2019-08-08 23:18:47 -03:00
Eugenio Lacuesta 3164543ed1
Remove fallback ScrapyClientContextFactory class (used in Twisted < 14.0.0) 
16.0.0 is currently the minimum supported version
 2019-08-08 23:15:03 -03:00
Eugenio Lacuesta a940a80f58
Remove check for pyOpenSSL>=0.16 
16.2.0 is currently the minimum supported version
 2019-08-08 23:05:52 -03:00
Eugenio Lacuesta fa9a9033f0
Remove check for Twisted>=14.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 22:58:59 -03:00
Eugenio Lacuesta da385b56b1
Move get_openssl_version function to scrapy.utils.ssl 2019-08-08 12:44:23 -03:00
Shivam Sandbhor 3040f77468 [MRG+1] Update project.py removed one 'hack', seems irrelevant. (#3910) 
* Update project.py removed one 'hack', seems irrelevant.

As mentioned by @Gallaecio in issue #3871, the 'hack'  is cleared. I also  double checked whether the environment variable "SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE" was ever set in our codebase and it turns out we didn't set it or used it anywhere else.So I guess the 'hack' was not used in the current version. Also the name of this environment variable rather doesn't suggest it was  a boolean(it is used in  an 'if' condition which has perplexed me )

* Update project.py

* Update project.py

How about this?

* Update project.py

* Update project.py

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update project.py
 2019-08-08 16:58:22 +05:00
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00