scrapy/scrapy
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830)
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
2026-07-31 11:59:52 +05:00
..
commands fix(commands/parse): Restore request callback before invoking spider (#7803) 2026-07-29 16:00:24 +05:00
contracts Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
core Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
downloadermiddlewares Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 2026-07-31 11:59:52 +05:00
extensions Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
http Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
linkextractors Release notes for 2.17.0 (#7723) 2026-07-07 14:57:42 +05:00
loader Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
pipelines Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
selector Clarify/cleanup Selector.type (#7704) 2026-07-01 08:50:32 +02:00
settings Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
spidermiddlewares Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
spiders Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
templates Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
utils Don't evaluate annotations when inspecting signatures (#7818) 2026-07-30 20:18:45 +05:00
VERSION Bump version: 2.16.0 → 2.17.0 2026-07-07 12:26:52 +02:00
__init__.py Remove code deprecated in 2.12.0 (#7126) 2025-10-27 17:14:38 +05:00
__main__.py Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
addons.py Assorted docstring fixes (#7698) 2026-06-30 18:27:32 +05:00
cmdline.py Improve test coverage for scrapy.cmdline (#7795) 2026-07-28 17:04:54 +05:00
crawler.py Reactorless import hook improvements. (#7747) 2026-07-21 19:58:40 +05:00
dupefilters.py Add Jobdir documentation (#5260) 2026-02-06 22:33:02 +04:00
exceptions.py Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
exporters.py Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
extension.py Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 2026-04-22 19:01:59 +05:00
interfaces.py Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
item.py Release notes for 2.17.0 (#7723) 2026-07-07 14:57:42 +05:00
link.py Assorted docstring fixes (#7698) 2026-06-30 18:27:32 +05:00
logformatter.py Assorted docstring fixes (#7698) 2026-06-30 18:27:32 +05:00
mail.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
middleware.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mime.types moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
pqueues.py Assorted docstring fixes (#7698) 2026-06-30 18:27:32 +05:00
py.typed Add py.typed 2023-09-21 13:06:12 +02:00
resolver.py Fix port issue with cached DNS (#7772) 2026-07-27 21:23:56 +05:00
responsetypes.py Improve test coverage for responsetypes.py (#7646) 2026-06-22 20:10:38 +05:00
robotstxt.py Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 2026-07-31 11:59:52 +05:00
shell.py Assorted docstring fixes (#7698) 2026-06-30 18:27:32 +05:00
signalmanager.py Move to mypy --strict with exceptions. (#7300) 2026-03-02 15:47:23 +05:00
signals.py Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 2026-07-31 11:59:52 +05:00
spiderloader.py Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
squeues.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
statscollectors.py Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00