RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
Fandu b2d4eedea8
Fix immediate delivery of full feed export batches (#7730) (#7733) 
* Store full feed batches before spider closes (#7730)

Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.

Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.

* Remove the issue reference

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 		2026-07-29 19:30:03 +05:00
..
AsyncCrawlerProcess Reactorless import hook improvements. (#7747) 2026-07-21 19:58:40 +05:00
AsyncCrawlerRunner Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
CrawlerProcess More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
CrawlerRunner More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
keys Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
mocks Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mockserver Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
sample_data Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
test_cmdline Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_cmdline_crawl_with_pipeline More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_settings Support iterable values in BaseSettings.update() (#7763) 2026-07-23 12:37:21 +02:00
test_spiderloader Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
test_utils_misc Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
utils Improve test coverage for scrapy.pipelines (#7798) 2026-07-29 14:32:18 +05:00
__init__.py TLS code modernization (#7353) 2026-03-25 15:42:43 +05:00
ignores.txt Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
pipelines.py Deprecate spider args of middlewares and pipelines. (#7006) 2025-08-21 13:23:10 +05:00
spiders.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_addons.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_closespider.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_command_check.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_command_crawl.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_command_fetch.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_genspider.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_command_parse.py fix(commands/parse): Restore request callback before invoking spider (#7803) 2026-07-29 16:00:24 +05:00
test_command_runspider.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_command_shell.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_command_startproject.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_command_version.py Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
test_commands.py Improve test coverage for scrapy.cmdline (#7795) 2026-07-28 17:04:54 +05:00
test_contracts.py Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
test_core_downloader.py Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 2026-06-25 16:44:51 +05:00
test_core_scraper.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_crawl.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_crawler.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_crawler_runners.py Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
test_crawler_subprocess.py Reactorless import hook improvements. (#7747) 2026-07-21 19:58:40 +05:00
test_dependencies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloader_handler_httpx.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_downloader_handler_twisted_ftp.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handler_twisted_http2.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_downloader_handler_twisted_http11.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_downloader_handlers.py Deprecate is_secure=False for s3://. (#7738) 2026-07-20 14:59:11 +02:00
test_downloadermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_cookies.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_downloadermiddleware_defaultheaders.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_downloadtimeout.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_httpauth.py Implement HTTP Auth settings and request metadata keys (#7590) 2026-06-11 15:56:53 +05:00
test_downloadermiddleware_httpcache.py Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
test_downloadermiddleware_httpcompression.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_downloadermiddleware_httpproxy.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_offsite.py Address recursion and media ignore-request reporting issues (#7673) 2026-07-27 20:34:10 +05:00
test_downloadermiddleware_redirect.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_downloadermiddleware_redirect_metarefresh.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_downloadermiddleware_retry.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_downloadermiddleware_robotstxt.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_downloadermiddleware_stats.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_downloadermiddleware_useragent.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloaderslotssettings.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_dupefilters.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_engine.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_engine_close_spider.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_engine_download.py Address recursion and media ignore-request reporting issues (#7673) 2026-07-27 20:34:10 +05:00
test_engine_loop.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_engine_stop_download_bytes.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_engine_stop_download_headers.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_exporters.py Fix silent data loss in CsvItemExporter (closes #4002) (#7651) 2026-07-08 17:49:29 +05:00
test_extension_debug.py Improve a flaky test_stacktracedump_dump_stacktrace(). (#7753) 2026-07-21 13:17:56 +05:00
test_extension_memdebug.py Tests for extensions.debug and extensions.memdebug. (#7749) 2026-07-20 15:21:28 +02:00
test_extension_memusage.py tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
test_extension_periodic_log.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_extension_statsmailer.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_extension_telnet.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_extension_throttle.py Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
test_feedexport.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_feedexport_batch.py Fix immediate delivery of full feed export batches (#7730) (#7733) 2026-07-29 19:30:03 +05:00
test_feedexport_postprocess.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_feedexport_storages.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_feedexport_uri_params.py Split long test files (#7329) 2026-03-13 10:30:25 +01:00
test_http2_client_protocol.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_http_cookies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_http_headers.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_request.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_request_form.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_request_json.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_response.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_response_text.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_item.py Keep Item fields in definition order (#7694) 2026-06-30 16:26:22 +02:00
test_link.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_linkextractors.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_loader.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
test_logformatter.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_logstats.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_mail.py Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
test_middleware.py Deprecate the spider arg of downloader middleware methods. (#7037) 2025-09-02 21:51:47 +05:00
test_pipeline_crawl.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_pipeline_files.py Improve test coverage for scrapy.pipelines (#7798) 2026-07-29 14:32:18 +05:00
test_pipeline_images.py Improve test coverage for scrapy.pipelines (#7798) 2026-07-29 14:32:18 +05:00
test_pipeline_media.py Improve test coverage for scrapy.pipelines (#7798) 2026-07-29 14:32:18 +05:00
test_pipelines.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_poet.py Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 2025-02-14 20:40:06 +05:00
test_pqueues.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_request_attribute_binding.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_request_cb_kwargs.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_request_dict.py Deprecate FormRequest in favor of form2request (#6438) 2026-05-04 22:38:29 +05:00
test_request_left.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_resolver.py Fix port issue with cached DNS (#7772) 2026-07-27 21:23:56 +05:00
test_responsetypes.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_robotstxt_interface.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_scheduler.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_scheduler_base.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_scrapy__getattr__.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_selector.py Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
test_signalmanager.py Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
test_signals.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spider.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_crawl.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_sitemap.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_base.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_depth.py Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
test_spidermiddleware_httperror.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_metacopy.py Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
test_spidermiddleware_output_chain.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_spidermiddleware_process_start.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_spidermiddleware_referer.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_spidermiddleware_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_urllength.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spiderstate.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_squeues.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_squeues_request.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_stats.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_toplevel.py Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
test_utils_asyncgen.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_asyncio.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_conf.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_console.py Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
test_utils_curl.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_datatypes.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_decorators.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_defer.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_deprecate.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_display.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_gz.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_httpobj.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_iterators.py Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
test_utils_log.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_utils_project.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_python.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_reactor.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_reactorless.py Reactorless import hook improvements. (#7747) 2026-07-21 19:58:40 +05:00
test_utils_request.py Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
test_utils_response.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_serialize.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_signal.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_sitemap.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_spider.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_template.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_trackref.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_url.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_zz_resources.py Assorted test fixes (#7585) 2026-06-10 11:01:06 +02:00