RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,169 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

111 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 4cb049cb15
Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
msn ad4549673b
Remove set_asyncio_event_loop call from Shell._schedule (#7594) 
Co-authored-by: Mayuresh <mayuresh@Mayureshs-MacBook-Air.local>
 2026-06-11 10:45:25 +02:00
msn 93a627ba1c
Prevent scrapy shell from starting spider requests (#7557) 
* Prevent scrapy shell from starting spider requests

* Run pre-commit fixes

* Update shell architecture notes

---------

Co-authored-by: Mayuresh <mayuresh@Mayureshs-MacBook-Air.local>
 2026-06-11 00:59:03 +05:00
Andrey Rakhmatullin f7db039d1c
Update the TLS code to Twisted 26.4.0 (#7347) 
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).

* Hide the _setAcceptableProtocols import.

* Silence ScrapyClientContextFactory HTTP/1.0 warnings.

* Update ScrapyClientTLSOptions for unreleased Twisted.

* Add twisted-trunk to CI.

* Update for new changes in Twisted trunk.

* Xfail a test failing with newer Twisted.

* Test Twisted trunk with extra-deps, update the ALPN code.

* Cleanup.

* Fix typing.

* Update relevant type hints for Twisted trunk.

* Update test_no_context_sharing().

* Silence a weird mypy error.

* Update Twisted versions, unpin pyOpenSSL.

* Update a comment.

* Silence pylint.

* Make a factory fixture.

* Improve the _setAcceptableProtocols comment.

* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.

* Add Twisted[http2] to extra-deps-pinned.
 2026-05-12 18:22:31 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 03d105ac92
Add warnings to components not working without a reactor (#7355) 2026-03-25 09:21:08 +01:00
Andrey Rakhmatullin e02ad08672
Move to mypy --strict with exceptions. (#7300) 
* Move to mypy --strict with exceptions.

* Fix typing-tests.
 2026-03-02 15:47:23 +05:00
Andrey Rakhmatullin 7010985e4f
Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin a0b766f9e1
More *_async APIs (#6997) 
* Add ExecutionEngine.close_spider_async().

* Add Scraper.{open,close}_spider_async().

* Fix double engine stopping via Crawler.stop().

* Add ExecutionEngine.stop_async().

* Add versionadded to new async APIs.

* Add ExecutionEngine.close_async().
 2025-08-08 22:15:41 +05:00
Andrey Rakhmatullin baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984) 
* Add a crawler argument to MiddlewareManager.__init__().

* Deprecate the spider argument in some MiddlewareManager subclasses.

* Remove the spider argument from ExecutionEngine.open_spider_async().

* Remove the spider argument from Scraper.open_spider().

* Fix the Scraper.close_spider() type hint.

* Remove the spider argument from Downloader internals.

* Deprecate the spider arg of download_func.

* Deprecate the spider arg of Downloader.fetch().

* Deprecate the spider arg of ItemPipelineManager.process_item().

* Fix pylint.

* Tests for MiddlewareManager spider code.

* Deprecated spider arg handling for custom pipeline managers.

* Add a test for the deprecated spider arg in a custom downloader.

* Remove TODO.

* Fix pylint.
 2025-08-05 12:21:08 +05:00
Neeraj Palliyali c47b5d049a
fix for formatting interactive shell (#6980) 2025-08-01 11:01:40 +02:00
Andrey Rakhmatullin 32f2aede82
Deprecate non-async counterparts of new async APIs. (#6979) 
* Deprecate non-async counterparts of new async APIs.

* Deprecate Scraper.call_spider().

* Add _schedule_coro().
 2025-08-01 00:58:48 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin e47110f9a5
Add parameteres to most Deferred instances. (#6414) 2024-06-26 10:01:43 +02:00
Andrey Rakhmatullin 4ed5c5ae91
Use ParamSpec for callables. (#6353) 2024-05-13 12:01:52 +02:00
Andrey Rakhmatullin 6ecc9e0a34
Add typing for scrapy/commands (#6268) 2024-03-06 13:21:08 +01:00
Adrián Chaves 2bfd9a2257 bandit: allow-list false positives 2024-02-29 11:11:42 +01:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan c5cdd0d30c reaplying black again and removing line length on pre-commit black config 2022-12-29 12:48:20 -03:00
Emmanuel Rondan 303f0a70fc re applying black to the code with default line length 2022-12-29 11:26:06 -03:00
Emmanuel Rondan 12b556a352
Merge branch 'master' into add-black-formatter 2022-12-26 12:25:32 -03:00
Alex 8b09b0e0d7 set asyncio event loop at start of new thread 2022-12-13 20:41:48 -08:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Eugenio Lacuesta 5256eae60d
Meta class to handle isinstance checks for BaseItem 2020-05-07 14:37:41 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Mikhail Korobov c4ee4b6075
Merge pull request #4347 from noviluni/deprecate_sel_shortcut 
Remove deprecated `sel` shortcut in scrapy shell
 2020-02-20 02:56:34 +05:00
Adrián Chaves 6972a19707
Remove unused imports 2020-02-19 18:59:09 +01:00
Marc Hernandez Cabot eb21dae524 deprecare sel shortcut in scrapy shell 2020-02-19 17:49:42 +01:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Andrey Rakhmatullin 69cd2e247e Move a bunch of "from twisted.internet import reactor" inside functions. 2019-12-13 18:07:55 +05:00
Eugenio Lacuesta 415526d922
Remove __future__ imports 2019-11-02 23:26:38 -03:00
Eugenio Lacuesta 793b2376f8 Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
Paul Tremberth 70a69d2199 Use built-in range() 2016-12-12 22:40:48 +01:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
Paul Tremberth 778bed07bf Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00