RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,974 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

10974 Commits

Author SHA1 Message Date
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin 509b572efc
Migrate the build system to hatchling. (#6771) 2025-05-05 09:51:53 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Andrey Rakhmatullin 5dfe7cd7b8
Improve tests for start items. (#6770) 2025-05-05 09:36:52 +02:00
Adrián Chaves 8f059d4095
Merge pull request #6766 from wRAR/errback-tests 
Improve test coverage of Scraper.
 2025-05-02 10:37:28 +02:00
Andrey Rakhmatullin da9078c4bb Add tests for raising CloseSpider in callbacks. 2025-05-01 23:12:39 +05:00
Andrey Rakhmatullin 23c206af35 Improve test coverage of Scraper. 2025-05-01 22:59:18 +05:00
Andrey Rakhmatullin 6deae473d9
Merge pull request #6764 from wRAR/spider-arg-scraper 
Remove the unnecessary spider argument from Scraper and related code.
 2025-04-30 18:33:35 +04:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Andrey Rakhmatullin 4aba7e5f66 Mention the deprecation of TestSpider. 2025-04-25 20:16:26 +05:00
Andrey Rakhmatullin 095140f134 Cover the current master in the release notes (up to daf9db7). 2025-04-25 19:43:29 +05:00
Andrey Rakhmatullin b1f85b5a17 Release notes for 2.13.0, up to b4c2531021. 2025-04-24 20:03:36 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Suejung Shin e50914e0f5
Codecov: Add test analytics (#6741) 2025-03-21 12:28:47 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Andrey Rakhmatullin 8729247213
Fix running simple tests with --reactor=default. (#6735) 2025-03-15 20:33:10 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 26ecc93228
Run CI only on the main branch, on release branches and on PRs (#6720) 2025-03-11 14:12:11 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves faab15c3f2
Merge pull request #6722 from scrapy/remove-trailing-whitespace 
Remove trailing whitespace
 2025-03-11 12:03:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Andrey Rakhmatullin 2accaa4af4
Merge pull request #6716 from wRAR/better-queuelib-fix 
Better fix for test_non_pickable_object on Windows.
 2025-03-10 13:09:48 +04:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
Andrey Rakhmatullin 02ed71d887
Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
Mehraz Hossain Rumman 044c3f69ed
Deprecate InitSpider (#6714) 2025-03-09 20:18:57 +01:00
Andrey Rakhmatullin 1469b2739e
Drop tests/test_loader_deprecated.py. (#6704) 2025-03-08 17:50:54 +01:00
Andrey Rakhmatullin 40833afc86
Work around a queuelib test file close problem. (#6703) 2025-03-07 19:33:41 +01:00
Andrey Rakhmatullin 3ded1dfe31
Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
Andrey Rakhmatullin 18f912b78f
Merge pull request #6701 from wRAR/cleanup-splash 
Remove the Splash recommendation.
 2025-03-07 13:58:02 +04:00
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 35212ec5b0
Merge pull request #6696 from wRAR/deprecate-test-utils 
Move most of the test utils inside tests.
 2025-03-05 22:07:15 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin 93c076047b Add scrapy/utils/testproc.py to collect_ignore to silence a warning. 2025-03-02 21:19:24 +05:00