RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 		2025-05-28 18:55:44 +05:00
..
AsyncCrawlerProcess Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
AsyncCrawlerRunner Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
CrawlerProcess Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
CrawlerRunner Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
keys Commit mitmproxy-dhparam.pem. 2024-11-15 21:08:18 +05:00
mocks Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
sample_data Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
test_cmdline Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_cmdline_crawl_with_pipeline Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_settings Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
test_spiderloader Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_utils_misc Syntax Error Fixed (#6738) 2025-03-20 17:02:10 +05:00
utils Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
__init__.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
ftpserver.py adding black formatter to all the code 2022-11-29 11:30:46 -03:00
ignores.txt Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
mockserver.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
pipelines.py reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
spiders.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_addons.py Enable AsyncioSelectorReactor by default. (#6713) 2025-03-12 01:18:30 +05:00
test_closespider.py feat: Add count to spider_exceptions stats (#6740) 2025-03-20 15:49:44 +05:00
test_command_check.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_command_fetch.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_command_parse.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_command_shell.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_command_version.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_commands.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_contracts.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_core_downloader.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_crawl.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_crawler.py Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
test_dependencies.py Enable AsyncioSelectorReactor by default. (#6713) 2025-03-12 01:18:30 +05:00
test_downloader_handler_twisted_http2.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_downloader_handler_twisted_http10.py Restructure download handler tests. (#6821) 2025-05-27 18:19:47 +05:00
test_downloader_handler_twisted_http11.py Restructure download handler tests. (#6821) 2025-05-27 18:19:47 +05:00
test_downloader_handlers.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_downloader_handlers_http_base.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_downloadermiddleware.py Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
test_downloadermiddleware_ajaxcrawlable.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_cookies.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_defaultheaders.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_downloadtimeout.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_httpauth.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_httpcache.py refactor tests/test_downloadermiddleware_httpcache.py (#6769) 2025-05-12 13:04:34 +02:00
test_downloadermiddleware_httpcompression.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_httpproxy.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_offsite.py Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
test_downloadermiddleware_redirect.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_retry.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_robotstxt.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_downloadermiddleware_stats.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloadermiddleware_useragent.py Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
test_downloaderslotssettings.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_dupefilters.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_engine.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_engine_loop.py 2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
test_engine_stop_download_bytes.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_engine_stop_download_headers.py Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
test_exporters.py Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
test_extension_periodic_log.py Enable AsyncioSelectorReactor by default. (#6713) 2025-03-12 01:18:30 +05:00
test_extension_telnet.py Add flake8-pie rules to ruff. 2024-12-12 20:33:35 +05:00
test_extension_throttle.py Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
test_feedexport.py Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
test_http2_client_protocol.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_http_cookies.py Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
test_http_headers.py Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
test_http_request.py Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
test_http_response.py Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
test_item.py Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
test_link.py Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
test_linkextractors.py Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
test_loader.py Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
test_logformatter.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_logstats.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_mail.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_middleware.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_pipeline_crawl.py Enable AsyncioSelectorReactor by default. (#6713) 2025-03-12 01:18:30 +05:00
test_pipeline_files.py Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
test_pipeline_images.py Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
test_pipeline_media.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_pipelines.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_poet.py Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 2025-02-14 20:40:06 +05:00
test_pqueues.py 2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
test_proxy_connect.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_request_attribute_binding.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_request_cb_kwargs.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_request_dict.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_request_left.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_responsetypes.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_robotstxt_interface.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_scheduler.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_scheduler_base.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_scrapy__getattr__.py Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
test_selector.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_signals.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spider.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spider_start.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_base.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_depth.py Base class for universal spider middlewares (#6693) 2025-04-23 19:29:04 +05:00
test_spidermiddleware_httperror.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_offsite.py Base class for universal spider middlewares (#6693) 2025-04-23 19:29:04 +05:00
test_spidermiddleware_output_chain.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_process_start.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_referer.py Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_spidermiddleware_start.py Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
test_spidermiddleware_urllength.py Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_spiderstate.py Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_squeues.py Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
test_squeues_request.py Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_stats.py Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
test_toplevel.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_urlparse_monkeypatches.py Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
test_utils_asyncgen.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_asyncio.py Add is_asyncio_available(). (#6827) 2025-05-28 14:46:39 +05:00
test_utils_conf.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_console.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_curl.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_datatypes.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_defer.py Add is_asyncio_available(). (#6827) 2025-05-28 14:46:39 +05:00
test_utils_deprecate.py Add is_asyncio_available(). (#6827) 2025-05-28 14:46:39 +05:00
test_utils_display.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_gz.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_httpobj.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_iterators.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_log.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_project.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_python.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_reactor.py Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 2025-05-28 18:55:44 +05:00
test_utils_request.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_response.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_serialize.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_signal.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
test_utils_sitemap.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_spider.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_template.py Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
test_utils_trackref.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_utils_url.py Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
test_webclient.py Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
upper-constraints.txt upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00