RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
max f5451d33ee Do not ignore sitemaps served with a query string 
SitemapSpider._get_sitemap_body checked response.url.endswith('.xml'),
which fails for sitemap URLs carrying a query string
(e.g. '.../sitemap_products_8.xml?from=1&to=2'), causing such sitemaps to
be silently ignored when the response was not already an XmlResponse.

Compare the extension against the URL path (via urlparse) instead of the
full URL, so query strings no longer defeat the check.

Closes #6293
 		2026-07-08 11:20:25 +03:00
..
AsyncCrawlerProcess More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
AsyncCrawlerRunner Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
CrawlerProcess More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
CrawlerRunner More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
keys Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
mocks Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mockserver Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
sample_data Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
test_cmdline Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_cmdline_crawl_with_pipeline More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_settings Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
test_spiderloader Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_misc More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
utils tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
__init__.py TLS code modernization (#7353) 2026-03-25 15:42:43 +05:00
ignores.txt Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
pipelines.py Deprecate spider args of middlewares and pipelines. (#7006) 2025-08-21 13:23:10 +05:00
spiders.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_addons.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_closespider.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_command_check.py Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
test_command_crawl.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_fetch.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_genspider.py Fix genspider --editor (#7683) 2026-06-30 12:05:42 +02:00
test_command_parse.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_runspider.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_command_shell.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_command_startproject.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_command_version.py Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
test_commands.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_contracts.py Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
test_core_downloader.py Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 2026-06-25 16:44:51 +05:00
test_core_scraper.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_crawl.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_crawler.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_crawler_subprocess.py Improve test coverage for crawler.py (#7682) 2026-07-01 11:53:07 +05:00
test_dependencies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloader_handler_httpx.py Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575) 2026-06-09 01:10:44 +05:00
test_downloader_handler_twisted_ftp.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handler_twisted_http2.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handler_twisted_http11.py Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
test_downloader_handlers.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloader_handlers_http_base.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_downloadermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_cookies.py Cleanup cookie handling in request_to_curl() (#7684) 2026-06-29 21:49:09 +05:00
test_downloadermiddleware_defaultheaders.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_downloadtimeout.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_httpauth.py Implement HTTP Auth settings and request metadata keys (#7590) 2026-06-11 15:56:53 +05:00
test_downloadermiddleware_httpcache.py Solve timing issues with HTTP cache tests? (#7692) 2026-06-29 17:21:20 +05:00
test_downloadermiddleware_httpcompression.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_httpproxy.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_offsite.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_redirect.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_redirect_base.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloadermiddleware_redirect_metarefresh.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_retry.py Allow configuring the log level of the retry give-up message (#7567) 2026-06-08 18:44:30 +05:00
test_downloadermiddleware_robotstxt.py Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
test_downloadermiddleware_stats.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_useragent.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloaderslotssettings.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_dupefilters.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_engine.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_engine_loop.py fix typos (#7564) 2026-06-02 11:04:38 +02:00
test_engine_stop_download_bytes.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_engine_stop_download_headers.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_exporters.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_extension_memusage.py tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
test_extension_periodic_log.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_extension_statsmailer.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_extension_telnet.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_extension_throttle.py Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
test_feedexport.py Keep Item fields in definition order (#7694) 2026-06-30 16:26:22 +02:00
test_feedexport_batch.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_feedexport_postprocess.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_feedexport_storages.py Close GCS feed temp files after upload (#7546) 2026-06-04 20:11:50 +05:00
test_feedexport_uri_params.py Split long test files (#7329) 2026-03-13 10:30:25 +01:00
test_http2_client_protocol.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_http_cookies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_http_headers.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_request.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_request_form.py Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
test_http_request_json.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_http_response.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_response_text.py fix-utf16-response-test-on-big-endian-systems (#7508) 2026-07-01 12:16:01 +05:00
test_item.py Keep Item fields in definition order (#7694) 2026-06-30 16:26:22 +02:00
test_link.py Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
test_linkextractors.py LxmlLinkExtractor: add deny_attrs and deny_tags (#7679) 2026-07-05 16:46:22 +05:00
test_loader.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
test_logformatter.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_logstats.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_mail.py Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
test_middleware.py Deprecate the spider arg of downloader middleware methods. (#7037) 2025-09-02 21:51:47 +05:00
test_pipeline_crawl.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_pipeline_files.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_pipeline_images.py Fix image store ACL settings for S3 and GCS (#7614) 2026-06-16 00:24:44 +05:00
test_pipeline_media.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_pipelines.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_poet.py Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 2025-02-14 20:40:06 +05:00
test_pqueues.py Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
test_request_attribute_binding.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_request_cb_kwargs.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_request_dict.py Deprecate FormRequest in favor of form2request (#6438) 2026-05-04 22:38:29 +05:00
test_request_left.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_resolver.py Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 2026-06-29 21:37:06 +05:00
test_responsetypes.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_robotstxt_interface.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_scheduler.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_scheduler_base.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_scrapy__getattr__.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_selector.py Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
test_signalmanager.py Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
test_signals.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spider.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_spider_crawl.py Document scrapy-lint, remove start_url check (#7627) 2026-06-22 20:12:16 +05:00
test_spider_sitemap.py Do not ignore sitemaps served with a query string 2026-07-08 11:20:25 +03:00
test_spider_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_base.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_depth.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_httperror.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_output_chain.py Remove support for sync process_spider_output(). (#7504) 2026-05-15 16:36:05 +05:00
test_spidermiddleware_process_start.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_spidermiddleware_referer.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_urllength.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spiderstate.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_squeues.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_squeues_request.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_stats.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_toplevel.py Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
test_utils_asyncgen.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_utils_asyncio.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_conf.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_console.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_curl.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_datatypes.py Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 2026-06-29 21:37:06 +05:00
test_utils_decorators.py Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
test_utils_defer.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_utils_deprecate.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_display.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_gz.py Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
test_utils_httpobj.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_utils_iterators.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_log.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_project.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_utils_python.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_utils_reactor.py Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
test_utils_request.py Cleanup cookie handling in request_to_curl() (#7684) 2026-06-29 21:49:09 +05:00
test_utils_response.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_serialize.py Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
test_utils_signal.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_sitemap.py Fix _get_tag_name() crash for non-string elem.tag (#7686) (#7687) 2026-06-30 17:49:47 +05:00
test_utils_spider.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_template.py Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
test_utils_trackref.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_url.py Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
test_zz_resources.py Assorted test fixes (#7585) 2026-06-10 11:01:06 +02:00