PyPy environments explicitly disable coverage to avoid the significant
runtime overhead it adds. Without coverage data, delta has no accurate
mapping for these envs and falls back to the CPython mapping, causing:
- Unreliable test selection (CPython-mapped tests may fail on PyPy)
- Pre-existing PyPy/min-deps failures surfacing unexpectedly (exit code 1)
Affected envs: pypy3, pypy3-extra-deps, min-pypy3
This follows the same principle as the botocore/mitmproxy fix:
delta is only used where it has accurate coverage mapping.
These envs filter tests with -m requires_botocore / -m requires_mitmproxy.
When delta selects non-matching tests based on the diff, pytest finds no
tests to run for that marker and exits with code 5 (no tests collected).
Revert these three envs back to plain pytest since they are already
highly filtered and delta provides no benefit here.
- Replace all bare pytest invocations with delta run -- in tox.ini:
[min], [testenv:pypy3], [testenv:min-pypy3], [testenv:botocore],
[testenv:min-botocore], [testenv:mitmproxy]
- Add deltatest-cli to [min] and [testenv:min-pypy3] deps
- Add passenv DELTA_* GITHUB_* to [min] and [testenv:min-pypy3]
- Add Configure Delta Cloud step to tests job in tests-ubuntu.yml
(writes ~/.delta/config.toml from secrets before tox runs)
- Remove standalone delta pre-check job; delta now runs inline
in every matrix runner via tox
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.
Closes#6120
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* docs: switch `scrapy.Item` examples to dataclasses
* make serializer doc generic
* use modern type hints
* docs/spiders: switch TestItem consumer snippets to attribute access
Since the TestItem migration to @dataclass, the existing
item["id"] = ... assignments would raise TypeError on copy-paste.
Switch to item.id = ... to match the new dataclass declaration.
The snippets sit under .. skip: next so docs-tests still pass either
way, but the change keeps the examples runnable for readers.
aliowka
deltatest.dev
Adrian
Labib Bin Salam
Andrey Rakhmatullin
msn
Syncrain
Adnan Awan
Omkar Kabde
Ayush Singh
Ethan Kuo