RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

56 Commits

Author SHA1 Message Date
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin d59f9b644a
Replace _SettingsKeyT with str. (#7586) 2026-06-10 11:12:53 +02:00
Adrian 294abed138
Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
 2026-04-22 19:00:39 +05:00
Ryotaro 06fb87f7bb
Fix override behavior in getwithbase() issue #6912 (#6993) 
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly

* Test actual import paths from default_settings

* Improvements

* Remove unused logger

* Run pre-commit

* Restore and improve duplicate key handling and warning

* type → object

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-02-07 00:17:55 +04:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin c330a399dc
Merge pull request #6601 from wRAR/ruff-rules-5 
Add more Ruff rules, do some pylint cleanups
 2025-01-02 17:38:15 +04:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Andrey Rakhmatullin 043a24410b Disable pylint for broken code. 2023-07-17 00:13:49 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Jalil SA 8aca47e25d
Add setdefault to BaseSettings (#5821) 2023-03-07 10:26:59 +01:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 881b4f417f
Flake8: Remove E128 (tests, part 3) 2020-05-06 19:33:22 -03:00
Eugenio Lacuesta 07fa768d33
Merge branch 'remove-six-code' of github.com:scrapy/scrapy into remove-six-code 2019-11-08 22:17:12 -03:00
Eugenio Lacuesta 342e3b5bd5
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-08 12:29:15 -03:00
Marc Hernández c377c14e32 Fix W391 Blank line at end of file (#4137) 2019-11-07 17:47:35 +01:00
Eugenio Lacuesta 5797aefd4c
Remove six.assertCountEqual 2019-11-03 12:18:35 -03:00
Eugenio Lacuesta 1aba513693
Remove six.iter* occurrences 2019-11-03 00:26:44 -03:00
Andrey Rakhmatullin 75b1d051d9 Simplify some more imports. 2019-10-31 15:22:45 +05:00
Andrey Rakhmatullin bbd9f4be90 Remove six.PY2 and six.PY3 conditionals. 2019-10-31 15:07:10 +05:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Paul Tremberth ae6d8d728e Support 'True' and 'False' strings as boolean settings values 2017-01-30 16:33:08 +01:00
Paul Tremberth aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Jakob de Maeyer 44f6ada034 Overwrite, not update, dictionary-like settings 2015-11-11 17:40:42 +01:00
Jakob de Maeyer 52ecee6a62 Replace BaseSettings._getcomposite() with public .getwithbase() method 2015-11-11 17:39:57 +01:00
Jakob de Maeyer 32ff4cc1d7 PEP8ify scrapy.settings tests 2015-10-29 19:52:19 +01:00
Jakob de Maeyer 51ca84c9b4 Add missing tests for scrapy.settings module 2015-10-29 17:34:18 +01:00
Jakob de Maeyer 03f1720afb Fix backwards-compatibility for users who explicitly set _BASE settings 2015-10-27 14:22:56 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Julia Medina d3d61fde87 Fix a recent mock regression by not using it in a test 2015-07-10 19:58:38 -03:00