RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,909 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

132 Commits

Author SHA1 Message Date
Adrián Chaves 3a95feec75 Blind test #1 for Windows-specific test failure 2025-05-07 22:38:05 +02:00
Adrián Chaves 51c39308d8 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-05-07 21:16:13 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Adrián Chaves 76ccc50622 Add an example of higher-priority leaf requests 2025-04-01 13:16:14 +02:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 273620488c Enable PTH Ruff rules. 2025-01-01 22:21:51 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Adrián Chaves 21b9ba717c
Allow customizing logged software versions (#6582) 
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-12-16 14:46:23 +01:00
Swayam Gupta 8c23da943c
Integrating configs into pyproject.toml (#6547) 2024-11-19 19:21:15 +05:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin ceedb026f8
Remove top-level imports that install the reactor from scrapy.extensions.telnet. (#6432) 2024-07-09 08:34:58 +02:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Adrián Chaves fa0c598096
Add component getters to Crawler (#6181) 2024-01-15 13:14:02 +01:00
Adrián Chaves 1d81585612
Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-10-30 09:35:29 +01:00
Andrey Rakhmatullin 6b0c18e921 Improve signal sending in test_shutdown_forced. 2023-10-19 23:13:47 +04:00
Laerte Pereira ad4e8b64d4 fix test_spider_manager_verify_interface test 2023-10-17 18:58:27 -03:00
Andrey Rakhmatullin 2cb1e10c76 Make shutdown tests more robust. 2023-10-17 23:56:54 +04:00
Andrey Rakhmatullin 2f436c05e0 Use SIGBREAK in Windows tests. 2023-10-08 22:55:05 +04:00
Andrey Rakhmatullin eb5e2e79ba Use SIGINT instead of SIGTERM to support Windows. 2023-09-24 23:31:46 +04:00
Andrey Rakhmatullin 4b5fb9b5a6 Add a test for SIGTERM handling. 2023-09-24 22:52:17 +04:00
Andrey Rakhmatullin 3756216339 Rename methods. 2023-09-13 15:47:22 +04:00
Andrey Rakhmatullin 028a56b9a2 Improve and simplify tests. 2023-09-13 14:17:41 +04:00
Andrey Rakhmatullin 77f39be407 Add integration tests for modifying settings via args. 2023-09-12 20:16:26 +04:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Andrey Rakhmatullin b4380995da Merge remote-tracking branch 'origin/master' into change-init-order 2023-09-06 20:34:27 +04:00
Andrey Rakhmatullin b39d2d4353 Add a test. 2023-09-06 19:44:22 +04:00
Andrey Rakhmatullin 69282c5318 Merge branch 'master' into change-init-order 2023-09-05 19:23:36 +04:00
Andrey Rakhmatullin 34d050cfe5
Remove deprecated CrawlerRunner.spiders. (#6010) 2023-08-11 10:41:05 +02:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00