RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,790 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

1187 Commits

Author SHA1 Message Date
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
Mikhail Korobov 5f407cf657
Merge pull request #3961 from OmarFarrag/ftp_files#3928 
Add FTPFileStore to FilesPipeline
 2020-01-25 04:58:23 +05:00
Mikhail Korobov 7a62bd310c
Merge pull request #4126 from elacuesta/from_crawler_downloader_handlers 
Download handlers: from_crawler factory method, take crawler in __init__
 2020-01-25 04:52:56 +05:00
OmarFarrag 40e0a11aa8 Fix Flake8 errors 2020-01-24 15:51:48 +02:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Mikhail Korobov f80c7776ae
Merge pull request #4008 from elacuesta/docs_request_errback 
Request: remove restriction about errback without callback
 2020-01-23 23:12:44 +05:00
Mikhail Korobov c0a7dfbc01
Merge pull request #4057 from elacuesta/response_follow_all 
Response.follow_all
 2020-01-23 02:15:24 +05:00
Eugenio Lacuesta dee420a69c
Fix name resolvers tests 2020-01-16 23:48:16 -03:00
Eugenio Lacuesta d487498cff
Update name resolvers tests 2020-01-16 22:02:01 -03:00
Eugenio Lacuesta 90e3bd8715
[test] Name resolvers 2020-01-16 20:32:40 -03:00
Andrey Rakhmatullin 7d85984880 Use get_from_asyncio_queue in the pipeline test. 2020-01-09 14:49:16 +05:00
Andrey Rakhmatullin bdef948aae Mark the asyncio pipelines test as only_asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin bfdd552a32 Add a test for pipelines using asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin 1f9cef787d Add async def support to pipelines. 2020-01-09 14:19:02 +05:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
Andrey Rakhmatullin 2b9254c2bd Add a test function that uses asyncio.Queue(). 2019-12-31 17:54:41 +05:00
Andrey Rakhmatullin 16787f5bf4 Merge middleware tests back as we don't need to set the setting anymore. 2019-12-30 12:02:19 +05:00
Andrey Rakhmatullin 5cf1ac0005 Move the asyncio downloader mw test to a separate class. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 3603644552 Add a non-asyncio async def middleware test. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 21f50c795a Add async def support to downloader middlewares. 2019-12-30 11:46:45 +05:00
Mikhail Korobov bb991cd303
Merge pull request #4010 from scrapy/asyncio-base 
Base support for asyncio
 2019-12-30 00:51:28 +05:00
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Andrey Rakhmatullin 37ac47ff80 Fix a deprecation warning. 2019-12-26 20:46:54 +05:00
Eugenio Lacuesta 982a66f9fb
[test] Download handler: avoid passing settings if not necessary 2019-12-23 20:28:17 -03:00
Eugenio Lacuesta a6ec89251e
Downloader handlers: crawler=None in __init__ 2019-12-23 10:47:08 -03:00
Eugenio Lacuesta 8a567e98bb
Remove unnecessary __init__ methods in downloader handler tests 2019-12-23 10:36:46 -03:00
Eugenio Lacuesta 7e6387de40
Use create_instance in FTPDownloadHandler/DataURIDownloadHandler tests 2019-12-23 10:02:58 -03:00
Eugenio Lacuesta fa21d8687a
Use create_instance in S3DownloadHandler tests 2019-12-23 10:00:25 -03:00
Eugenio Lacuesta 9e5d945ef2
Use create_instance in downloader handler tests 2019-12-23 09:55:47 -03:00
Eugenio Lacuesta 931b7e68d3
Update FileDownloadHandler test 2019-12-23 09:50:28 -03:00
Eugenio Lacuesta 310cc081d0
Merge branch 'master' into from_crawler_downloader_handlers 2019-12-21 17:24:45 -03:00
Andrey Rakhmatullin 8de80f59db Raise an exception if ASYNCIO_ENABLED but the reactor is wrong. 2019-12-21 13:08:29 +05:00
Mikhail Korobov 5577d4d2be
Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks 
[MRG+1][Py3] Check for 'return' with arguments inside generators
 2019-12-20 22:46:46 +05:00
Andrey Rakhmatullin e342de5038 Remove a stray newline. 2019-12-20 19:37:50 +05:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 3bcb14fe4d
Merge pull request #4164 from Gallaecio/speed-up-tests-2 
Improve the performance of the DOWNLOAD_DELAY test
 2019-12-18 23:56:56 +05:00
Marc Hernandez Cabot a5de2c64e6 fix W291, W292, W293 (whitespaces) 2019-12-18 16:33:46 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Andrey Rakhmatullin 7ccb169a27 Split a long test in test_engine.py into three. 2019-12-18 19:41:16 +05:00
Mikhail Korobov e90f276903
Merge pull request #4198 from wRAR/deprecate-noconnect 
Deprecate the HTTPS proxy noconnect mode.
 2019-12-18 16:11:24 +05:00
Adrián Chaves 1f689b50c6 Merge remote-tracking branch 'origin/documentation-coverage' into documentation-coverage 2019-12-18 12:09:34 +01:00
Adrián Chaves 174769a3f0 Use a better name for the LxmlLinkExtractor subclassing test 2019-12-18 12:09:03 +01:00
Andrey Rakhmatullin ac302c3f61 Fix a flake8 problem. 2019-12-18 15:43:05 +05:00
Andrey Rakhmatullin bb2ff13e4c Skip Http10ProxyTestCase.test_download_with_proxy_https_noconnect 2019-12-18 15:39:08 +05:00
Andrey Rakhmatullin 2d92a39003 Restore test_download_with_proxy_https_noconnect, check for a warning there. 2019-12-18 12:07:08 +05:00
Adrián Chaves 607815dfc2
Merge branch 'master' into documentation-coverage 2019-12-17 16:54:27 +01:00
Andrey Rahmatullin bb3f164280
Merge pull request #4236 from wRAR/pipeline-tests 
Add simple tests for pipelines.
 2019-12-17 19:38:06 +05:00
Marc Hernández 63cf5c75c8 Fix E502: backslash is redundant between brackets (#4238) 2019-12-17 13:53:15 +01:00
Andrey Rakhmatullin 5980d3bbff Add simple tests for pipelines. 2019-12-16 22:41:28 +05:00