RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,313 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

7313 Commits

Author SHA1 Message Date
Mikhail Korobov 1364e92007
Merge pull request #4015 from elacuesta/remove_deprecated_xlib 
Remove deprecated xlib module
 2019-09-20 15:28:07 +01:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
Mikhail Korobov 874b005738
Merge pull request #4028 from Gallaecio/fix-dont-merge-cookies-docs 
Clarify the effects of dont_merge_cookies
 2019-09-20 14:51:37 +01:00
Adrián Chaves c26a9015ad Clarify the effects of dont_merge_cookies 2019-09-19 11:08:06 +02:00
Adrián Chaves 9b65f9aa5b
Fix the item exporter example (#4022) 2019-09-19 09:17:23 +02:00
watsta 0b52fa6ca9 LogFormatter: Add the ability to skip log messages (#3987) 2019-09-16 14:12:04 +02:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Mikhail Korobov d4b8bf18b2
Merge pull request #3993 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.extensions
 2019-08-30 21:07:22 +05:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Eugenio Lacuesta eb0bd2daef
Revert backward-incompatible change (contract priorities) 2019-08-29 14:01:13 -03:00
Eugenio Lacuesta 97a7d775f7
Aplly suggestions by @victor-torres 2019-08-29 10:51:16 -03:00
Eugenio Lacuesta b6b76df057
CallbackKeywordArgumentsContract 2019-08-29 03:29:24 -03:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Mikhail Korobov b00f81c52b
Merge pull request #3978 from anubhavp28/doc-link-fix 
Fixes a link in docs
 2019-08-28 21:58:25 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta 3abe7e6e6d Add Bug report and Feature request templates (#3471) 2019-08-26 09:35:44 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Mikhail Korobov 56948c446c
Merge pull request #3442 from wRAR/ciphers 
Support for overriding OpenSSL ciphers
 2019-08-19 23:15:09 +05:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin 3384db92b4 Add support for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov a95de71d8e
Merge pull request #3950 from elacuesta/version_updates 
Remove obsolete version checks
 2019-08-12 22:51:00 +05:00
Mikhail Korobov 2f0c46e762
Merge pull request #3946 from elacuesta/simplify_versions 
[MRG+1] Simplify version reporting
 2019-08-10 14:25:59 +05:00
Eugenio Lacuesta 26fb28b20f
PEP8-ify HTTP/1.1 downloader handler 
Signed-off-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2019-08-09 00:56:24 -03:00
Eugenio Lacuesta d5dcc5eaef
Import twisted.web.client.URI directly 2019-08-09 00:30:58 -03:00
Eugenio Lacuesta d3737d869b
Remove check for Twisted>=14.0 2019-08-09 00:21:43 -03:00
Eugenio Lacuesta e17c9a48fd
Remove check for Twisted>=15.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 23:59:17 -03:00
Eugenio Lacuesta d92f1b1858
Simplify import + assignment 2019-08-08 23:53:35 -03:00
Eugenio Lacuesta b404941e0d
Remove import check for service_identity 
service_identity.exceptions.CertificateError is available in the current minimum version (16.0.0)
 2019-08-08 23:18:47 -03:00
Eugenio Lacuesta 3164543ed1
Remove fallback ScrapyClientContextFactory class (used in Twisted < 14.0.0) 
16.0.0 is currently the minimum supported version
 2019-08-08 23:15:03 -03:00
Eugenio Lacuesta a940a80f58
Remove check for pyOpenSSL>=0.16 
16.2.0 is currently the minimum supported version
 2019-08-08 23:05:52 -03:00
Eugenio Lacuesta fa9a9033f0
Remove check for Twisted>=14.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 22:58:59 -03:00
Eugenio Lacuesta da385b56b1
Move get_openssl_version function to scrapy.utils.ssl 2019-08-08 12:44:23 -03:00
Shivam Sandbhor 3040f77468 [MRG+1] Update project.py removed one 'hack', seems irrelevant. (#3910) 
* Update project.py removed one 'hack', seems irrelevant.

As mentioned by @Gallaecio in issue #3871, the 'hack'  is cleared. I also  double checked whether the environment variable "SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE" was ever set in our codebase and it turns out we didn't set it or used it anywhere else.So I guess the 'hack' was not used in the current version. Also the name of this environment variable rather doesn't suggest it was  a boolean(it is used in  an 'if' condition which has perplexed me )

* Update project.py

* Update project.py

How about this?

* Update project.py

* Update project.py

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update project.py
 2019-08-08 16:58:22 +05:00
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves 9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Eugenio Lacuesta 595c995ee6
Simplify version reporting 2019-08-07 15:38:04 -03:00