RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,554 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

9554 Commits

Author SHA1 Message Date
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Andrey Rahmatullin 64e0ea4ee5
Merge pull request #4735 from wRAR/ciphers-tests 
Re-enable TLS 1.2 in cipher tests.
 2020-08-25 11:48:31 +05:00
drs-11 ecec5f9e51 Cleaned up code 2020-08-25 02:46:44 +05:30
drs-11 1f9598ada4 Update branch to latest 2020-08-25 02:14:22 +05:30
drs-11 8b84a65a6b cleaned up code relating to issue #3689 2020-08-25 00:30:17 +05:30
drs-11 0b3881d65e Reverted maxsize warning log message 2020-08-24 20:37:40 +05:30
WinterComes 2d8ec9d44f Change DOWNLOAD_MAXSIZE logger level from Error to Warning 2020-08-24 20:37:40 +05:30
Aditya 1432161477 fix: bump min typing-extensions version to 3.7.4 
- typing-extensions>=3.7.4 only supports TypedDict
 2020-08-24 15:40:01 +05:30
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
Eugenio Lacuesta f1250177dc
Remove Python 3.5 from CI (#4743) 2020-08-22 09:33:35 +02:00
Mikhail Korobov ba0e7f3c85
Merge pull request #4747 from Gallaecio/pylint-update 
Skip checks introduced in Pylint 2.6.0
 2020-08-21 23:18:51 +05:00
Adrián Chaves 7c076122eb Skip checks introduced in Pylint 2.6.0 2020-08-21 17:06:54 +02:00
Adrián Chaves afd3a4d116 Fix style issue 2020-08-21 17:04:02 +02:00
Adrián Chaves e90be0d8a5 Mark the new test as xfail for xmliter_lxml 2020-08-21 14:09:52 +02:00
Adrián Chaves d711eca4d9 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-21 13:52:52 +02:00
Andrey Rahmatullin 07b0c591e0
Merge pull request #4745 from Yogendra0Sharma/fix4741 
Removed appveyor.xml no longer needed
 2020-08-21 12:36:13 +05:00
yogendra0sharma 2fbfe2c214 Removed appveyor.xml no longer needed 2020-08-21 12:18:15 +05:30
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
Hugo van Kemenade a57db9e302
Bitbucket no longer supports Mercurial repositories (#4738) 2020-08-19 17:45:24 +02:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Aditya 2f00666d74 refactor: move agents & context-factory 2020-08-19 07:31:52 +05:30
Aditya 30eb005639 fix: InvalidNegotiatedProtocol __str__ method 2020-08-19 06:25:04 +05:30
Aditya e3233b79de refactor(h2-stream): alphabetical order of imports 2020-08-19 05:10:19 +05:30
Aditya a206ac5f6f tests: disable python 3.5 for travis and azure 2020-08-18 07:36:39 +05:30
Aditya a87ab71d10 refactor(http2): metadata for Stream 
- Add Note about HTTP/2 Cleartext not supported in settings.rst
 2020-08-18 05:52:40 +05:30
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 282a6d4fc1
Merge pull request #4727 from wRAR/mitmproxy-ca 
Use a longer key in mitmproxy-ca.pem.
 2020-08-17 14:06:58 +05:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Adrián Chaves 792d29380e
Merge Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory (#4165) 
* Port code from Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory

* Mention that ScrapyHTTPClientFactory has Twisted code
 2020-08-17 10:54:58 +02:00
Mikhail Korobov 5ac135f036
Merge pull request #4718 from Gallaecio/support-pinned-3.6 
Upgrade minimum dependencies for Python 3.6 support
 2020-08-17 13:54:45 +05:00
Grammy Jiang 55edf8d3b8
Add typing hint to httpcache downloadermiddlewares (#4243) 2020-08-17 10:50:52 +02:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
adityaa30 75fe3d1365 fix: increase timeout to 0.5 seconds 
- In Windows specifically the reactor was left unclean by the
  HostnameEndpoint due to the tearDown method of
  test_downloader_handlers.py::HttpTestCase due to
  which the following 2 tests were failing:
  1. test_timeout_download_from_spider_server_hangs
  2. test_timeout_download_from_spider_nodata_rcvd
- Increasing the timeout fixed the test (in local)
 2020-08-17 04:18:03 +05:30
Aditya 38d361792c fix: typing & pylint errors 
- Ignore typing check for http2 test files
 2020-08-16 17:55:16 +05:30
Aditya f9f008e935 test: add typing-extensions 2020-08-16 17:04:40 +05:30
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Aditya af73f141b2 refactor: move all http2 tests in separate files 2020-08-16 11:26:32 +05:30
Andrey Rakhmatullin 756c368a6b Use a longer key in mitmproxy-ca.pem. 2020-08-14 22:09:24 +05:00
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Mikhail Korobov cd0d3fd48d
Merge pull request #4705 from Gallaecio/deprecation-policy 
Cover our deprecation policy in the documentation
 2020-08-14 00:27:37 +05:00
Mikhail Korobov 1f44464a4a
Merge pull request #4723 from Gallaecio/windows 
test_utils_iterators.py: support Windows the right way
 2020-08-14 00:23:55 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Adrián Chaves 24ba5a71ac Maybe the problem is not in the code after all 2020-08-13 06:35:09 +02:00
Adrián Chaves 5f4df622a1 test_utils_iterators.py: support Windows the right way 2020-08-13 05:41:06 +02:00
Adrián Chaves 4c0afb606c Update permission expectations 2020-08-12 17:45:26 +02:00
Adrián Chaves 125a058340 Do not let umask affect the permissions of startproject-generated files 2020-08-12 17:07:21 +02:00
Adrián Chaves b1de55d37d Fix marker syntax 2020-08-12 12:34:40 +02:00