RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,020 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1283 Commits

Author SHA1 Message Date
Andrey Rahmatullin f558df2558
Merge pull request #4188 from elacuesta/logformatter-error-formatting 
LogFormatter error formatting
 2020-02-19 19:05:08 +05:00
Andrey Rahmatullin 528b894f28
Merge pull request #4321 from Gallaecio/link-extractor-encoding 
Use safe_url_string in link extraction
 2020-02-19 18:19:21 +05:00
Adrián Chaves 43b43654a1 Add tests for meta and cb_kwargs not being available 2020-02-13 22:39:58 +01:00
Adrián Chaves 5ff9eb90ea Add a test for the copy of cb_kwargs from Request to Response 2020-02-13 22:36:18 +01:00
Adrián Chaves b4958358e8 Update tests to account for link extractors escaping spaces 2020-02-12 19:00:04 +01:00
Mikhail Korobov a6ef065eb5
Merge pull request #4271 from wRAR/asyncio-signals 
async def support for signal handlers that already supported Deferreds
 2020-02-11 02:05:45 +05:00
Andrey Rakhmatullin 1f0f52cbf7 Improve async signal tests. 2020-02-11 01:05:45 +05:00
Andrey Rakhmatullin 36dcf90184 Also test non-default async callbacks. 2020-02-11 01:00:16 +05:00
Abhishek Pratap Singh 4626e90df8
Allow updating flags in follow and follow_all (#4279) 2020-02-10 19:48:31 +01:00
Adrián Chaves 35723d76c0 Use canonicalize_url in link extraction 2020-02-07 22:59:53 +01:00
Adrián Chaves c2f484de88
Merge branch 'master' into response_ip_address 2020-02-07 21:49:38 +01:00
Adrián Chaves 4c6993f264
Merge branch 'master' into asyncio-parse-asyncgen 2020-02-07 21:34:54 +01:00
Mikhail Korobov 957681bcfa
Merge pull request #4272 from elacuesta/spider-middleware 
Spider middleware: catch spider callback exceptions early
 2020-02-07 23:40:50 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Andrey Rakhmatullin 7323780c97 Support yield in async def callbacks. 2020-02-07 21:32:45 +05:00
Andrey Rakhmatullin 31f6c7112f Add a test for an async callbacks that returns requests. 2020-02-07 17:14:52 +05:00
Andrey Rakhmatullin 4a7c7340a0 Merge remote-tracking branch 'origin/master' into asyncio-signals 2020-02-07 16:58:59 +05:00
Eugenio Lacuesta 6c333491a8
Merge branch 'master' into response_ip_address 2020-02-06 18:53:11 -03:00
Lane Shaw 3263441fbc
Update RFPDupeFilter line separator for correct universal newlines mode usage (#4283) 2020-02-06 22:14:40 +01:00
Mikhail Korobov 042e71e2b8
Merge pull request #4311 from Gallaecio/metarefresh-ignore-tags 
Make METAREFRESH_IGNORE_TAGS an empty list by default
 2020-02-06 23:40:45 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Andrey Rakhmatullin 489ffcda51 Add a test for an async item_scraped handler. 2020-02-06 22:40:11 +05:00
Vostretsov Nikita 4be19e443e name signla catcher in accord with signal name 2020-02-06 13:46:23 +00:00
Vostretsov Nikita 4a91a5427d fix typo 2020-02-06 13:44:51 +00:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Vostretsov Nikita 3769f75386 pep8 E302 2020-02-05 12:08:08 +00:00
Vostretsov Nikita 9916f6e556 tests for new signal 2020-02-05 11:32:54 +00:00
Eugenio Lacuesta ad70497416
Remove unnecessary parentheses in class definition 2020-02-04 13:30:13 -03:00
Eugenio Lacuesta e0ef8ad2d6
CrawlerRunner test for Response.ip_address 2020-02-03 15:52:15 -03:00
Eugenio Lacuesta 4851efdfb0
Flake8 adjustments 2020-02-03 14:50:54 -03:00
Eugenio Lacuesta bb8f7dc609
Mock DNS server 2020-02-03 14:50:14 -03:00
Eugenio Lacuesta a2ae380efc
Remove unnecessary commas 2020-02-03 13:23:52 -03:00
Eugenio Lacuesta b9e3a6201b
Merge branch 'master' into response_ip_address 2020-02-03 04:07:44 -03:00
Mikhail Korobov 22f7934fcc
Merge pull request #4269 from wRAR/asyncio-parse 
Support for async def callbacks.
 2020-01-31 23:50:19 +05:00
Andrey Rakhmatullin 7687564c73 Merge remote-tracking branch 'origin/master' into asyncio-signals 2020-01-30 17:42:03 +05:00
Andrey Rakhmatullin cc825c21de Test returning items from an async def callback. 2020-01-30 16:17:06 +05:00
Evgeny Dorofeev 4e56571a19 [HttpCompressionMiddleware] fix delimiter for Accept-Encoding header 2020-01-29 15:49:22 +03:00
Eugenio Lacuesta e8da7e2966
Test DNS resolution using CrawlerProcess 2020-01-26 17:53:39 -03:00
Eugenio Lacuesta 80925ab845
Get server IP address for HTTP/1.1 responses 2020-01-26 16:57:04 -03:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
Mikhail Korobov 5f407cf657
Merge pull request #3961 from OmarFarrag/ftp_files#3928 
Add FTPFileStore to FilesPipeline
 2020-01-25 04:58:23 +05:00
Mikhail Korobov 7a62bd310c
Merge pull request #4126 from elacuesta/from_crawler_downloader_handlers 
Download handlers: from_crawler factory method, take crawler in __init__
 2020-01-25 04:52:56 +05:00
OmarFarrag 40e0a11aa8 Fix Flake8 errors 2020-01-24 15:51:48 +02:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Mikhail Korobov f80c7776ae
Merge pull request #4008 from elacuesta/docs_request_errback 
Request: remove restriction about errback without callback
 2020-01-23 23:12:44 +05:00
Mikhail Korobov c0a7dfbc01
Merge pull request #4057 from elacuesta/response_follow_all 
Response.follow_all
 2020-01-23 02:15:24 +05:00
Eugenio Lacuesta dee420a69c
Fix name resolvers tests 2020-01-16 23:48:16 -03:00
Eugenio Lacuesta d487498cff
Update name resolvers tests 2020-01-16 22:02:01 -03:00
Eugenio Lacuesta 90e3bd8715
[test] Name resolvers 2020-01-16 20:32:40 -03:00
Eugenio Lacuesta 6ce1ad3107
[test] Spider middleware: catch exceptions right after the spider callback 2020-01-10 04:20:37 -03:00
Andrey Rakhmatullin a91a13b443 Support for async def callbacks. 2020-01-09 15:59:19 +05:00
Andrey Rakhmatullin 3faef2d082 Add async def support to signal handlers that already supported Deferreds. 2020-01-09 15:17:45 +05:00
Andrey Rakhmatullin 7d85984880 Use get_from_asyncio_queue in the pipeline test. 2020-01-09 14:49:16 +05:00
Andrey Rakhmatullin bdef948aae Mark the asyncio pipelines test as only_asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin bfdd552a32 Add a test for pipelines using asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin 1f9cef787d Add async def support to pipelines. 2020-01-09 14:19:02 +05:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
Andrey Rakhmatullin 2b9254c2bd Add a test function that uses asyncio.Queue(). 2019-12-31 17:54:41 +05:00
Andrey Rakhmatullin 16787f5bf4 Merge middleware tests back as we don't need to set the setting anymore. 2019-12-30 12:02:19 +05:00
Andrey Rakhmatullin 5cf1ac0005 Move the asyncio downloader mw test to a separate class. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 3603644552 Add a non-asyncio async def middleware test. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 21f50c795a Add async def support to downloader middlewares. 2019-12-30 11:46:45 +05:00
Mikhail Korobov bb991cd303
Merge pull request #4010 from scrapy/asyncio-base 
Base support for asyncio
 2019-12-30 00:51:28 +05:00
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Andrey Rakhmatullin 37ac47ff80 Fix a deprecation warning. 2019-12-26 20:46:54 +05:00
Eugenio Lacuesta 982a66f9fb
[test] Download handler: avoid passing settings if not necessary 2019-12-23 20:28:17 -03:00
Eugenio Lacuesta a6ec89251e
Downloader handlers: crawler=None in __init__ 2019-12-23 10:47:08 -03:00
Eugenio Lacuesta 8a567e98bb
Remove unnecessary __init__ methods in downloader handler tests 2019-12-23 10:36:46 -03:00
Eugenio Lacuesta 7e6387de40
Use create_instance in FTPDownloadHandler/DataURIDownloadHandler tests 2019-12-23 10:02:58 -03:00
Eugenio Lacuesta fa21d8687a
Use create_instance in S3DownloadHandler tests 2019-12-23 10:00:25 -03:00
Eugenio Lacuesta 9e5d945ef2
Use create_instance in downloader handler tests 2019-12-23 09:55:47 -03:00
Eugenio Lacuesta 931b7e68d3
Update FileDownloadHandler test 2019-12-23 09:50:28 -03:00
Eugenio Lacuesta 310cc081d0
Merge branch 'master' into from_crawler_downloader_handlers 2019-12-21 17:24:45 -03:00
Andrey Rakhmatullin 8de80f59db Raise an exception if ASYNCIO_ENABLED but the reactor is wrong. 2019-12-21 13:08:29 +05:00
Mikhail Korobov 5577d4d2be
Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks 
[MRG+1][Py3] Check for 'return' with arguments inside generators
 2019-12-20 22:46:46 +05:00
Andrey Rakhmatullin e342de5038 Remove a stray newline. 2019-12-20 19:37:50 +05:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 3bcb14fe4d
Merge pull request #4164 from Gallaecio/speed-up-tests-2 
Improve the performance of the DOWNLOAD_DELAY test
 2019-12-18 23:56:56 +05:00
Marc Hernandez Cabot a5de2c64e6 fix W291, W292, W293 (whitespaces) 2019-12-18 16:33:46 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Andrey Rakhmatullin 7ccb169a27 Split a long test in test_engine.py into three. 2019-12-18 19:41:16 +05:00
Mikhail Korobov e90f276903
Merge pull request #4198 from wRAR/deprecate-noconnect 
Deprecate the HTTPS proxy noconnect mode.
 2019-12-18 16:11:24 +05:00
Adrián Chaves 1f689b50c6 Merge remote-tracking branch 'origin/documentation-coverage' into documentation-coverage 2019-12-18 12:09:34 +01:00
Adrián Chaves 174769a3f0 Use a better name for the LxmlLinkExtractor subclassing test 2019-12-18 12:09:03 +01:00
Andrey Rakhmatullin ac302c3f61 Fix a flake8 problem. 2019-12-18 15:43:05 +05:00
Andrey Rakhmatullin bb2ff13e4c Skip Http10ProxyTestCase.test_download_with_proxy_https_noconnect 2019-12-18 15:39:08 +05:00
Andrey Rakhmatullin 2d92a39003 Restore test_download_with_proxy_https_noconnect, check for a warning there. 2019-12-18 12:07:08 +05:00
Adrián Chaves 607815dfc2
Merge branch 'master' into documentation-coverage 2019-12-17 16:54:27 +01:00
Andrey Rahmatullin bb3f164280
Merge pull request #4236 from wRAR/pipeline-tests 
Add simple tests for pipelines.
 2019-12-17 19:38:06 +05:00
Marc Hernández 63cf5c75c8 Fix E502: backslash is redundant between brackets (#4238) 2019-12-17 13:53:15 +01:00
Andrey Rakhmatullin 5980d3bbff Add simple tests for pipelines. 2019-12-16 22:41:28 +05:00
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Adrián Chaves a4ef9750f9 Fix Flake8-reported issues 2019-12-13 14:32:06 +01:00
Adrián Chaves b5c4c2cae8
Keep 2 spaces between code and inline comments (#4195) 2019-12-13 14:20:48 +01:00
Andrey Rakhmatullin bfb78b8dea Add CrawlerProcess tests for ASYNCIO_ENABLED. 2019-12-13 18:12:07 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Andrey Rahmatullin 02cdc53fb8
Add a test for a CrawlerProcess script. (#4218) 
* Add a test for a CrawlerProcess script.

* Add tests/CrawlerProcess to collect_ignore.

* Remove an extra line.

* Fix/improve conftest.py.
 2019-12-13 18:04:05 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00