RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,526 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

10526 Commits

Author SHA1 Message Date
Adrián Chaves 4b58358fbb Test response override from downloader middlewares 2024-07-08 15:21:11 +02:00
Adrián Chaves 4479b97895 Add tests for request backout 2024-07-08 14:59:00 +02:00
Adrián Chaves 30c98924e3 Test RESPONSE_MAX_ACTIVE_SIZE and SCRAPER_SLOT_MAX_ACTIVE_SIZE setting behavior 2024-07-08 12:06:31 +02:00
Adrián Chaves fdc79f9536 Solve issues reported by pylint 2024-07-08 10:42:58 +02:00
Adrián Chaves 73ac493d4a Use stacklevel=2 for scraper warnings 2024-07-08 10:36:15 +02:00
Adrián Chaves 784179ad2b Remove debugging leftovers 2024-07-08 10:32:49 +02:00
Adrián Chaves f5779c9624 SCRAPER_MAX_ACTIVE_SIZE → SCRAPER_SLOT_MAX_ACTIVE_SIZE (again) 2024-07-08 10:30:22 +02:00
Adrián Chaves 035ae28886 Minor cleanup 2024-07-08 10:29:33 +02:00
Adrián Chaves c22b450d0c Address typing issues 2024-07-08 10:25:11 +02:00
Adrián Chaves 4b0c23194f Restore Python 3.8 support 2024-07-08 10:15:58 +02:00
Adrián Chaves 1d25494861 Restore backward compatibility in the scraper 2024-07-08 10:05:11 +02:00
Adrián Chaves 58302cab38 Add a request_backouts/total_per_second stat 2024-07-08 09:37:24 +02:00
Adrián Chaves 8e534f708a Move the implementation to the downloader 2024-07-08 09:17:24 +02:00
Adrián Chaves 2621b97c94 SCRAPER_MAX_ACTIVE_SIZE → SCRAPER_SLOT_MAX_ACTIVE_SIZE 2024-07-08 08:17:11 +02:00
Adrián Chaves 4bbe6f55d3 Initial implementation 2024-07-05 17:29:49 +02:00
mlmsmith d8ecd28c55
Documentation improvements (#6429) 2024-07-04 12:16:26 +02:00
Andrey Rakhmatullin 558b1d11d2
Use CallbackT for Request.callback. (#6422) 2024-07-01 09:30:49 +02:00
Andrey Rakhmatullin 41e15e93e7
Remove an obsolete import. (#6415) 2024-06-26 14:44:12 +02:00
Andrey Rakhmatullin 96d6519b25
Bump twine in twinecheck. (#6416) 2024-06-26 14:43:59 +02:00
Andrey Rakhmatullin e47110f9a5
Add parameteres to most Deferred instances. (#6414) 2024-06-26 10:01:43 +02:00
Andrey Rakhmatullin d08f559600
Add flake8-type-checking. (#6413) 2024-06-25 10:20:59 +02:00
mlmsmith 326e323e11
Apply grammar fixes (#6411) 2024-06-21 12:24:10 +02:00
Andrey Rakhmatullin 0e78ac609d
Merge pull request #6406 from wRAR/more-flake8 
Add some flake8 plugins
 2024-06-20 14:34:25 +05:00
Andrey Rakhmatullin 13d3b1af47 Split ignores into blocks. 2024-06-20 00:42:43 +05:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Andrey Rakhmatullin 1ef9c337ca flake8-docstrings 2024-06-19 23:57:40 +05:00
Andrey Rakhmatullin 1c70d3e605 flake8-comprehensions 2024-06-19 23:36:36 +05:00
Andrey Rakhmatullin a617e04d2e flake8-string-format 2024-06-19 23:28:58 +05:00
Andrey Rakhmatullin d132190625 flake8-debugger 2024-06-19 23:26:25 +05:00
Andrey Rakhmatullin a364560fad
Unpin markupsafe in extra-deps. (#6403) 2024-06-17 11:38:10 +02:00
Andrey Rakhmatullin 365c9e62ad
Removing empty example reference (#6402) 
Co-authored-by: Michael Duane Mooring <mikeumus@gmail.com>
 2024-06-17 11:37:11 +02:00
Andrey Rakhmatullin 1282ddf8f7
Add parameters to most Deferred in scrapy/core. (#6395) 2024-06-10 10:27:50 +02:00
Andrey Rakhmatullin ddc98fe91b
Deprecate scrapy.utils.defer.process_chain_both(). (#6397) 2024-06-10 10:16:26 +02:00
Andrey Rakhmatullin 163e7d925e
Merge pull request #6392 from wRAR/pyupgrade 
Add pyupgrade.
 2024-06-07 11:16:46 +04:00
Andrey Rakhmatullin 5850b8f3e6 Merge branch 'master' into pyupgrade 2024-06-07 11:20:31 +05:00
Andrey Rakhmatullin ed3a7acaf3 Remove the Python version check from scrapy/__init__.py. 2024-06-07 11:19:37 +05:00
Andrey Rakhmatullin a4778d2bdf
Merge pull request #6143 from wRAR/typing-request-response-cls 
Improve type hints for copy() and replace() in Request and Response.
 2024-06-06 22:58:14 +04:00
Andrey Rakhmatullin 1268b23304 Merge branch 'master' into typing-request-response-cls 2024-06-06 21:57:53 +05:00
Andrey Rakhmatullin b24ecca4d0
Merge pull request #6391 from wRAR/typing-generics-iterables 
Add parameters to iterable generics, replace generators with iterables.
 2024-06-06 20:54:16 +04:00
Andrey Rakhmatullin 23b1214e90 Add a comment about pyupgrade and scrapy/__init__.py. 2024-06-06 21:44:07 +05:00
Andrey Rakhmatullin d9d7bd170b
Merge pull request #6394 from Laerte/master 
Document missing parts of response.json method
 2024-06-06 09:55:56 +04:00
Laerte Pereira 144ff6c756 Document missing parts of response.json method 2024-06-05 21:09:10 -03:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Andrey Rakhmatullin 480a11b68b Add mssing __future__ imports. 2024-06-05 22:48:16 +05:00
Andrey Rakhmatullin 262c10d85b Use typing.Coroutine instead of types.CoroutineType. 2024-06-05 22:11:34 +05:00
Andrey Rakhmatullin de146ad7ce Bump typing deps. 2024-06-05 22:09:19 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Suvan Banerjee 3f76853bd2
Handle AttributeError: 'NoneType' in contract parsing (#6388) 2024-06-05 06:34:46 +02:00
Andrey Rakhmatullin e56b425198
Full typing for scrapy/pipelines. (#6387) 2024-06-05 06:33:45 +02:00
Adrián Chaves 2b9e32f1ca
Merge pull request #6385 from wRAR/typing-generics-collections 
Add parameters to various generics.
 2024-06-03 13:29:38 +02:00