RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,718 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

118 Commits

Author SHA1 Message Date
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 273620488c Enable PTH Ruff rules. 2025-01-01 22:21:51 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Adrián Chaves 21b9ba717c
Allow customizing logged software versions (#6582) 
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-12-16 14:46:23 +01:00
Swayam Gupta 8c23da943c
Integrating configs into pyproject.toml (#6547) 2024-11-19 19:21:15 +05:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin ceedb026f8
Remove top-level imports that install the reactor from scrapy.extensions.telnet. (#6432) 2024-07-09 08:34:58 +02:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Adrián Chaves fa0c598096
Add component getters to Crawler (#6181) 2024-01-15 13:14:02 +01:00
Adrián Chaves 1d81585612
Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-10-30 09:35:29 +01:00
Andrey Rakhmatullin 6b0c18e921 Improve signal sending in test_shutdown_forced. 2023-10-19 23:13:47 +04:00
Laerte Pereira ad4e8b64d4 fix test_spider_manager_verify_interface test 2023-10-17 18:58:27 -03:00
Andrey Rakhmatullin 2cb1e10c76 Make shutdown tests more robust. 2023-10-17 23:56:54 +04:00
Andrey Rakhmatullin 2f436c05e0 Use SIGBREAK in Windows tests. 2023-10-08 22:55:05 +04:00
Andrey Rakhmatullin eb5e2e79ba Use SIGINT instead of SIGTERM to support Windows. 2023-09-24 23:31:46 +04:00
Andrey Rakhmatullin 4b5fb9b5a6 Add a test for SIGTERM handling. 2023-09-24 22:52:17 +04:00
Andrey Rakhmatullin 3756216339 Rename methods. 2023-09-13 15:47:22 +04:00
Andrey Rakhmatullin 028a56b9a2 Improve and simplify tests. 2023-09-13 14:17:41 +04:00
Andrey Rakhmatullin 77f39be407 Add integration tests for modifying settings via args. 2023-09-12 20:16:26 +04:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Andrey Rakhmatullin b4380995da Merge remote-tracking branch 'origin/master' into change-init-order 2023-09-06 20:34:27 +04:00
Andrey Rakhmatullin b39d2d4353 Add a test. 2023-09-06 19:44:22 +04:00
Andrey Rakhmatullin 69282c5318 Merge branch 'master' into change-init-order 2023-09-05 19:23:36 +04:00
Andrey Rakhmatullin 34d050cfe5
Remove deprecated CrawlerRunner.spiders. (#6010) 2023-08-11 10:41:05 +02:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Matthew Donoughe 065db7b566
fix some mypy issues 2022-10-19 08:28:26 -04:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
Adrián Chaves 960a7f68f6
Verify that the installed asyncio event loop matches ASYNCIO_EVENT_LOOP (#5529) 
Co-authored-by: Laerte Pereira <5853172+Laerte@users.noreply.github.com>
 2022-10-15 11:27:00 +02:00
Gábor Lipták 385acd5598
Match pyOpenSSL and service_identity to Twisted 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-24 21:33:02 -04:00
Laerte Pereira 1289422284
chore: Skip `test_ipv6_default_name_resolver` test if w3lib version >= 2.0.0 2022-09-06 08:17:58 -03:00
Laerte Pereira f4bcc3e67d
fix: failed tests 2022-09-06 01:15:41 -03:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Adrián Chaves 4ef71829b2
If TWISTED_REACTOR is None, reuse any pre-installed reactor (#5528) 2022-06-17 10:37:27 +02:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Adrián Chaves 96fc4dae15 CrawlerProcess: test a multi-spider scenario 2022-03-02 17:30:05 +01:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00