RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

124 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Adrian 6ad8a043ca
Fix genspider --editor (#7683) 2026-06-30 12:05:42 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259) 
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
 2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin e48a1bdde3
Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
Andrey Rakhmatullin 80beec41b5
Log when AsyncCrawlerProcess or CrawlerProcess is instantiated. (#7034) 2025-09-01 13:51:11 +02:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin a724541a71
Split tests/test_commands.py. (#6836) 2025-05-28 21:46:04 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Lidiane T 98a57e2418
Fix error when running `scrapy bench` (#6633) 2025-01-27 11:21:30 +01:00
anubhav 1c1e83895c
Fix _pop_command_name (#6606) 2025-01-14 16:40:24 +01:00
Ionut-Cezar Ciubotariu 402500b164
Change unknown cmd message when outside project (#3426) 
* Change unknown cmd message when outside project

* Simplification.

* Move the import to the top level.

* Reword the message.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-10 23:08:27 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Andrey Rakhmatullin 1c70d3e605 flake8-comprehensions 2024-06-19 23:36:36 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Adrián Chaves e72de11f55 Add super 2024-03-12 09:29:10 +01:00
Adrián Chaves 188d9a8bb3 Remove unnecessary comment 2024-03-11 23:28:57 +01:00
Adrián Chaves ab5ea32ffd Fix WindowsRunSpiderCommandTest skip outside Windows for older Twisted. 2024-03-11 23:26:19 +01:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Jon b0ef9a89a1
Remove usage of deprecated mktemp (#5285) 2024-02-26 19:53:06 +01:00
Andrey Rakhmatullin 77f39be407 Add integration tests for modifying settings via args. 2023-09-12 20:16:26 +04:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Laerte Pereira 7522aeed35
fix: -O/-o option does not work with absolute paths on Windows (#5971) 2023-07-24 09:53:41 +02:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Alex 8c8894f4be
Apply black formatting to templates (#5814) 2023-02-02 04:42:47 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Matthew Donoughe 065db7b566
fix some mypy issues 2022-10-19 08:28:26 -04:00
Matthew Donoughe 93d82648e5
resolve this path too 2022-10-19 08:08:21 -04:00
Matthew Donoughe fb26e6b650
resolve path to search in error 2022-10-18 20:21:11 -04:00
Matthew Donoughe 7daf735f45
downgrade type hint syntax 2022-10-18 20:07:29 -04:00