RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,202 Commits 49 Branches 132 Tags 148 MiB
Commit Graph

11202 Commits

Author SHA1 Message Date
aliowka cd1aef17f6 feat: implement delta-based test selection and CI reporting for PRs 2026-06-21 20:18:13 +03:00
aliowka ac8b132605 Fix duplicate --reactor option registration in conftest.py 2026-06-21 20:17:03 +03:00
aliowka e6b785cd04 Merge commit 'c9f952c25' 2026-06-21 20:16:46 +03:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
aliowka 7d2f108630 chore: remove deltatest workflow and initialize delta test tracking files and reports 2026-06-18 01:06:39 +03:00
aliowka c87010d8a1 feat: initialize delta test impact evaluation infrastructure and update tox configuration 2026-06-18 00:16:10 +03:00
aliowka 080e5fdecc t4 2026-06-18 00:04:18 +03:00
aliowka c58cc98da0 Fix conflicting pytest_runtest_setup hooks in conftest.py 2026-06-17 23:43:53 +03:00
aliowka 78367e8aa3 Merge remote-tracking branch 'upstream/master' 2026-06-17 20:30:53 +03:00
aliowka 95efb87809 Support target commit parameter in benchmark 2026-06-17 20:25:42 +03:00
aliowka 2fd48a6ec8 Increase benchmark commits limit to 10 2026-06-17 20:23:48 +03:00
aliowka aeaf98618c Remove --local flag from delta benchmark runner 2026-06-17 20:10:09 +03:00
aliowka c722ad8d59 Support context-aware subprocess tracking in tests 2026-06-17 19:36:51 +03:00
aliowka 086b1e4c31 Enable python subprocess coverage tracking in tests 2026-06-17 19:27:11 +03:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
aliowka 547cae3e75 t4 2026-06-17 12:38:30 +03:00
aliowka c2edc0fe6f t4 2026-06-17 11:09:46 +03:00
aliowka 233069fc7f Add delta push after every delta run in tox environments 
Envs that use {[testenv]commands} or {[min]commands} substitution with
extra pytest args (--reactor=default, -p no:twisted --reactor=none) were
expanded explicitly to prevent extra args being appended to delta push.

Affected environments:
- [testenv]              → delta run + delta push
- [min]                  → delta run + delta push
- [testenv:default-reactor]     → expanded, delta push added
- [testenv:min-default-reactor] → expanded, delta push added
- [testenv:no-reactor]          → expanded, delta push added
- [testenv:no-reactor-extra-deps] → expanded, delta push added
- [testenv:min-no-reactor]      → expanded, delta push added
 2026-06-17 11:08:08 +03:00
aliowka 2c0fb8fd6b t3 2026-06-17 10:59:42 +03:00
aliowka 4bf9c521bc t3 2026-06-17 10:59:39 +03:00
aliowka b905a6b2a3 Revert PyPy envs back to plain pytest 
PyPy environments explicitly disable coverage to avoid the significant
runtime overhead it adds. Without coverage data, delta has no accurate
mapping for these envs and falls back to the CPython mapping, causing:
- Unreliable test selection (CPython-mapped tests may fail on PyPy)
- Pre-existing PyPy/min-deps failures surfacing unexpectedly (exit code 1)

Affected envs: pypy3, pypy3-extra-deps, min-pypy3

This follows the same principle as the botocore/mitmproxy fix:
delta is only used where it has accurate coverage mapping.
 2026-06-17 10:40:03 +03:00
aliowka 699cdb93ed Fix exit code 5 in botocore/mitmproxy tox envs 
These envs filter tests with -m requires_botocore / -m requires_mitmproxy.
When delta selects non-matching tests based on the diff, pytest finds no
tests to run for that marker and exits with code 5 (no tests collected).

Revert these three envs back to plain pytest since they are already
highly filtered and delta provides no benefit here.
 2026-06-17 10:36:51 +03:00
aliowka bfc753dbee Integrate Delta into all tox environments and clean up workflow 
- Replace all bare pytest invocations with delta run -- in tox.ini:
  [min], [testenv:pypy3], [testenv:min-pypy3], [testenv:botocore],
  [testenv:min-botocore], [testenv:mitmproxy]
- Add deltatest-cli to [min] and [testenv:min-pypy3] deps
- Add passenv DELTA_* GITHUB_* to [min] and [testenv:min-pypy3]
- Add Configure Delta Cloud step to tests job in tests-ubuntu.yml
  (writes ~/.delta/config.toml from secrets before tox runs)
- Remove standalone delta pre-check job; delta now runs inline
  in every matrix runner via tox
 2026-06-17 10:29:07 +03:00
aliowka 3bce77074b Configure DELTA_BRANCH env var as master for test runners 2026-06-17 03:32:12 +03:00
aliowka 0dfff7301d Pass DELTA_API_KEY and DELTA_REPO_ID to tox test runners 2026-06-17 03:26:48 +03:00
aliowka 5150b80785 Integrate DeltaTest into tox configuration 2026-06-17 03:13:22 +03:00
aliowka 84d0ee13c8 t3 2026-06-17 02:58:31 +03:00
aliowka 6e6b0bf5c5 t2 2026-06-17 02:22:12 +03:00
aliowka 3ecca98ea2 t1 2026-06-17 02:16:41 +03:00
aliowka 161d4ef918 feat: introduce delta testing mechanism with benchmark reporting and workflow updates 2026-06-17 01:28:17 +03:00
aliowka c70798b1fc fix: update CI configuration to use echo for TOML generation and initialize Delta testing environment 2026-06-17 01:23:05 +03:00
aliowka aa54907a14 feat: implement delta-test benchmarking and test skipping infrastructure 2026-06-17 01:18:08 +03:00
aliowka 16587d9abf feat: add delta benchmark script and CI integration for testing efficiency monitoring 2026-06-17 01:12:55 +03:00
aliowka 3cc4cda13f feat: implement delta-based test selection and benchmarking workflow 2026-06-16 20:48:06 +03:00
aliowka 38cbf73cb5 feat: add benchmark script and automated workflow for delta test coverage evaluation 2026-06-16 20:40:18 +03:00
aliowka 852c692d20 Refactor DeltaTest workflow to conditionally install dependencies and use secrets for sensitive data 2026-06-16 20:29:47 +03:00
aliowka 2e2975eba1 Update DeltaTest workflow to install dependencies from docs/requirements.txt 2026-06-16 20:22:40 +03:00
aliowka c0e9ec76cd Merge branch 'master' of https://github.com/deltatest-org/scrapy 2026-06-16 10:41:26 +03:00
aliowka 906e9ee3df adding deltatest 2026-06-16 10:33:04 +03:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00