RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,969 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

2485 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 4279f2837c
Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 16f168b406
Deprecate mustbe_deferred() and defer_*(). (#6937) 2025-07-04 16:41:15 +05:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Andrey Rakhmatullin 5fbab843bd Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 16:24:09 +05:00
Andrey Rakhmatullin eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 13:05:02 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Thalison Fernandes 5e20b46e35 Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 22:26:04 +05:00
Thalison Fernandes 0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 18:56:29 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin 24a827c72e Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 13:09:44 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin d329eedfef Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 16:26:38 +05:00
Andrey Rakhmatullin 657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 13:02:15 +02:00
Andrey Rakhmatullin 54474ceb0d Fix a regression in errback result handling. (#6863) 2025-06-06 12:47:38 +05:00
Andrey Rakhmatullin d602f13e8c
Fix a regression in errback result handling. (#6863) 2025-06-05 18:02:31 +05:00
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin d400aa3e2d
Add _parallel_asyncio(). (#6852) 2025-06-03 11:19:15 +02:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Andrey Rakhmatullin a724541a71
Split tests/test_commands.py. (#6836) 2025-05-28 21:46:04 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821) 
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
 2025-05-27 18:19:47 +05:00
Adrián Chaves e3f82afaf1 Add a test for ScrapyPriorityQueue pop order 2025-05-27 10:01:00 +02:00
Adrián Chaves 9d92d16510 Prioritize other requests over start requests 2025-05-27 08:32:24 +02:00
Andrey Rakhmatullin b9caaf8a63 Simplify deferred_from_coro(), add more tests. 2025-05-16 17:16:45 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
Felipe Benevolo 128cb551eb
refactor tests/test_downloadermiddleware_httpcache.py (#6769) 2025-05-12 13:04:34 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Andrey Rakhmatullin 5dfe7cd7b8
Improve tests for start items. (#6770) 2025-05-05 09:36:52 +02:00
Andrey Rakhmatullin da9078c4bb Add tests for raising CloseSpider in callbacks. 2025-05-01 23:12:39 +05:00
Andrey Rakhmatullin 23c206af35 Improve test coverage of Scraper. 2025-05-01 22:59:18 +05:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
Andrey Rakhmatullin 02ed71d887
Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
Mehraz Hossain Rumman 044c3f69ed
Deprecate InitSpider (#6714) 2025-03-09 20:18:57 +01:00
Andrey Rakhmatullin 1469b2739e
Drop tests/test_loader_deprecated.py. (#6704) 2025-03-08 17:50:54 +01:00
Andrey Rakhmatullin 40833afc86
Work around a queuelib test file close problem. (#6703) 2025-03-07 19:33:41 +01:00
Andrey Rakhmatullin 3ded1dfe31
Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 35212ec5b0
Merge pull request #6696 from wRAR/deprecate-test-utils 
Move most of the test utils inside tests.
 2025-03-05 22:07:15 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin a5731c1944 Move most of the test utils inside tests. 2025-03-02 21:04:12 +05:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 391af6afcc
Unknown encoding handling in HttpCompressionMiddleware, restore x-gzip support (#6618) 
* Unknown encoding handling in HttpCompressionMiddleware.

* Implement the changes for unknown encoding handling.

* Restore support for Content-Encoding: x-gzip.

* Simplify the decoding logic.

* Add tests for the unsupported encoding warning.

* Add a test for the "no zstandard" warning.
 2025-02-27 22:37:01 +05:00
Matt Winter 8c34e6d9a4
curl: add support for parsing -b,--cookie (#6684) 2025-02-19 10:17:37 +01:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 7d5b189c11
Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 
* Fix getting annotations for _parse_sitemap() at the runtime.

* Split off the callback annotations test.
 2025-02-14 20:40:06 +05:00
Andrey Rakhmatullin d8978d405c Improve diagnostics for sync-only spider middlewares. 2025-02-09 01:33:27 +05:00
Andrey Rakhmatullin cc480680d7
Merge pull request #6650 from wRAR/cleanup-cache-tests 
Remove a duplicate test.
 2025-02-03 23:33:07 +04:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647) 
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
 2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin 4842bcbf1d Deprecate and disable escape_ajax(). 2025-02-03 14:08:05 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Mikhail Korobov b4c2531021
Merge pull request #6648 from wRAR/mockserver-calls 
Optimize mockserver calls
 2025-02-03 14:06:35 +05:00
Andrey Rakhmatullin df688910e0 Remove a duplicate test. 2025-02-02 18:48:26 +05:00
Andrey Rakhmatullin 783b98deda Make mockserver instances per-class. 2025-02-02 14:10:09 +05:00
Andrey Rakhmatullin 1a0dfbd32e Reuse mockserver instances in test_feedexport.py. 2025-02-02 13:28:34 +05:00
Andrey Rakhmatullin 200d76afa9 Refactor EngineTest tests. 2025-02-01 16:07:55 +05:00
Andrey Rakhmatullin 0a80871c3a Remove scrapy.core.downloader.webclient._parse(). 2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin 0d2d2892ba Silence the readBody warning. 2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin bc1aeeefc9 Deprecate overriding ScrapyClientContextFactory.getContext(). 2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Lidiane T 98a57e2418
Fix error when running `scrapy bench` (#6633) 2025-01-27 11:21:30 +01:00