RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,189 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

11189 Commits

Author SHA1 Message Date
aliowka 080e5fdecc t4 2026-06-18 00:04:18 +03:00
aliowka c58cc98da0 Fix conflicting pytest_runtest_setup hooks in conftest.py 2026-06-17 23:43:53 +03:00
aliowka 78367e8aa3 Merge remote-tracking branch 'upstream/master' 2026-06-17 20:30:53 +03:00
aliowka 95efb87809 Support target commit parameter in benchmark 2026-06-17 20:25:42 +03:00
aliowka 2fd48a6ec8 Increase benchmark commits limit to 10 2026-06-17 20:23:48 +03:00
aliowka aeaf98618c Remove --local flag from delta benchmark runner 2026-06-17 20:10:09 +03:00
aliowka c722ad8d59 Support context-aware subprocess tracking in tests 2026-06-17 19:36:51 +03:00
aliowka 086b1e4c31 Enable python subprocess coverage tracking in tests 2026-06-17 19:27:11 +03:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
aliowka 547cae3e75 t4 2026-06-17 12:38:30 +03:00
aliowka c2edc0fe6f t4 2026-06-17 11:09:46 +03:00
aliowka 233069fc7f Add delta push after every delta run in tox environments 
Envs that use {[testenv]commands} or {[min]commands} substitution with
extra pytest args (--reactor=default, -p no:twisted --reactor=none) were
expanded explicitly to prevent extra args being appended to delta push.

Affected environments:
- [testenv]              → delta run + delta push
- [min]                  → delta run + delta push
- [testenv:default-reactor]     → expanded, delta push added
- [testenv:min-default-reactor] → expanded, delta push added
- [testenv:no-reactor]          → expanded, delta push added
- [testenv:no-reactor-extra-deps] → expanded, delta push added
- [testenv:min-no-reactor]      → expanded, delta push added
 2026-06-17 11:08:08 +03:00
aliowka 2c0fb8fd6b t3 2026-06-17 10:59:42 +03:00
aliowka 4bf9c521bc t3 2026-06-17 10:59:39 +03:00
aliowka b905a6b2a3 Revert PyPy envs back to plain pytest 
PyPy environments explicitly disable coverage to avoid the significant
runtime overhead it adds. Without coverage data, delta has no accurate
mapping for these envs and falls back to the CPython mapping, causing:
- Unreliable test selection (CPython-mapped tests may fail on PyPy)
- Pre-existing PyPy/min-deps failures surfacing unexpectedly (exit code 1)

Affected envs: pypy3, pypy3-extra-deps, min-pypy3

This follows the same principle as the botocore/mitmproxy fix:
delta is only used where it has accurate coverage mapping.
 2026-06-17 10:40:03 +03:00
aliowka 699cdb93ed Fix exit code 5 in botocore/mitmproxy tox envs 
These envs filter tests with -m requires_botocore / -m requires_mitmproxy.
When delta selects non-matching tests based on the diff, pytest finds no
tests to run for that marker and exits with code 5 (no tests collected).

Revert these three envs back to plain pytest since they are already
highly filtered and delta provides no benefit here.
 2026-06-17 10:36:51 +03:00
aliowka bfc753dbee Integrate Delta into all tox environments and clean up workflow 
- Replace all bare pytest invocations with delta run -- in tox.ini:
  [min], [testenv:pypy3], [testenv:min-pypy3], [testenv:botocore],
  [testenv:min-botocore], [testenv:mitmproxy]
- Add deltatest-cli to [min] and [testenv:min-pypy3] deps
- Add passenv DELTA_* GITHUB_* to [min] and [testenv:min-pypy3]
- Add Configure Delta Cloud step to tests job in tests-ubuntu.yml
  (writes ~/.delta/config.toml from secrets before tox runs)
- Remove standalone delta pre-check job; delta now runs inline
  in every matrix runner via tox
 2026-06-17 10:29:07 +03:00
aliowka 3bce77074b Configure DELTA_BRANCH env var as master for test runners 2026-06-17 03:32:12 +03:00
aliowka 0dfff7301d Pass DELTA_API_KEY and DELTA_REPO_ID to tox test runners 2026-06-17 03:26:48 +03:00
aliowka 5150b80785 Integrate DeltaTest into tox configuration 2026-06-17 03:13:22 +03:00
aliowka 84d0ee13c8 t3 2026-06-17 02:58:31 +03:00
aliowka 6e6b0bf5c5 t2 2026-06-17 02:22:12 +03:00
aliowka 3ecca98ea2 t1 2026-06-17 02:16:41 +03:00
aliowka 161d4ef918 feat: introduce delta testing mechanism with benchmark reporting and workflow updates 2026-06-17 01:28:17 +03:00
aliowka c70798b1fc fix: update CI configuration to use echo for TOML generation and initialize Delta testing environment 2026-06-17 01:23:05 +03:00
aliowka aa54907a14 feat: implement delta-test benchmarking and test skipping infrastructure 2026-06-17 01:18:08 +03:00
aliowka 16587d9abf feat: add delta benchmark script and CI integration for testing efficiency monitoring 2026-06-17 01:12:55 +03:00
aliowka 3cc4cda13f feat: implement delta-based test selection and benchmarking workflow 2026-06-16 20:48:06 +03:00
aliowka 38cbf73cb5 feat: add benchmark script and automated workflow for delta test coverage evaluation 2026-06-16 20:40:18 +03:00
aliowka 852c692d20 Refactor DeltaTest workflow to conditionally install dependencies and use secrets for sensitive data 2026-06-16 20:29:47 +03:00
aliowka 2e2975eba1 Update DeltaTest workflow to install dependencies from docs/requirements.txt 2026-06-16 20:22:40 +03:00
aliowka c0e9ec76cd Merge branch 'master' of https://github.com/deltatest-org/scrapy 2026-06-16 10:41:26 +03:00
aliowka 906e9ee3df adding deltatest 2026-06-16 10:33:04 +03:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
deltatest.dev 5d07a95656
Create deltatest.yaml 2026-06-14 21:09:55 +03:00
deltatest.dev 0d8bbe9ca3
Create deltatest.properties.json 2026-06-14 21:09:20 +03:00
aliowka 851c1f72e9 Add .github/README.md with CI docs and Delta integration guide 2026-06-14 20:50:21 +03:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00
Adrian 983e6c1182
Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
Adrian b08ed1cf05
Add coverage for DummySpiderLoader.find_by_request() (#7608) 2026-06-12 13:13:17 +02:00
Adrian 7afc875081
Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00