RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

44 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Albert Eduardovich N. ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` (#7388) 
* deprecate `walk_modules` in favour of `walk_modules_iter`

* remove deprecation line from docstring

* addressing review
 2026-04-01 22:11:35 +05:00
Andrey Rakhmatullin 7010985e4f
Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126) 
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
 2025-10-27 17:14:38 +05:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Andrey Rakhmatullin 7dd92e6e43
Add pylint rules to ruff, refresh the ignore list of pylint itself (#6584) 2024-12-16 09:44:36 +01:00
Andrey Rakhmatullin e7595837a6 Add flake8-raise rules to ruff. 2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin eda3a89b3f Remove build_from_settings(). 2024-11-11 16:44:47 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Andrey Rakhmatullin b71d0292d5 Add a test for processing partial callbacks. 2022-10-27 18:13:47 +06:00
Godson-Gnanaraj 830e1c5dd8 Add test for parsing decorated methods 2022-10-26 01:26:54 +05:30
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
Marc Hernández fffb0a5b6a
replace set constructor with set literals (#4573) 2020-05-15 15:23:09 +02:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
willbeaufoy cf9be5344a
Prevent create_instance() returning None (#4532) 
Currently create_instance() can return None if an extension is
incorrectly implemented, but the extension will still show up as
enabled in the logs. This can cause confusion, as in the linked bug.

This change prevents this occurring by throwing an error if
create_instance() will return None.
 2020-05-11 20:35:25 +02:00
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Mikhail Korobov 550d8046fd
Merge pull request #4157 from Gallaecio/flake8 
Add missing whitespace after ‘,’, ‘;’ or ‘:’
 2019-11-14 23:42:06 +05:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Adrián Chaves 33ef24c757 Add missing whitespace after ‘,’, ‘;’ or ‘:’ 2019-11-13 10:52:05 +01:00
Matthijs Vos f6485e6697 Restore alphabetic order and two lines between import and code 2019-05-03 13:55:18 +02:00
Matthijs Vos 6d52708579 Add reset case 2019-04-18 15:19:23 +02:00
Matthijs Vos 2973998947 Add set_environ test 2019-04-18 14:50:02 +02:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Jakob de Maeyer 815d6160cf Add from_crawler constructor for feed exporters and storages 2017-07-25 09:41:29 +02:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00