RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,128 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11128 Commits

Author SHA1 Message Date
Adrián Chaves 8e25f8c157
Fix bad message 2023-11-23 14:12:59 +01:00
Adrián Chaves cf80e5670e Solve linting and typing issues 2023-11-23 12:07:15 +01:00
Adrián Chaves b53ed52a22 Update the release notes 2023-11-23 11:36:45 +01:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves 1087bb7b2e Update the docs 2023-11-23 09:11:14 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 3fda2fe103 Protect against gzip bomb sitemaps 2023-11-22 18:34:37 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Kiran 49b284ab85 Updated README.rst (#6144) 2023-11-16 19:17:52 +04:00
Jessica Allman-LaPorte 150f9d6d88 Make shell switching more clear in the tutorial (#6128) 2023-11-16 19:17:29 +04:00
Adrián Chaves 1045856a50 Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-11-16 19:16:52 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 538192916f Merge pull request #6064 from wRAR/signals-proper 
Refactor installing signals.
 2023-11-16 19:15:30 +04:00
Andrey Rakhmatullin 59cfdeaa5c Merge pull request #6083 from wRAR/py3.12-release 
Adapt to the Python 3.12 final release
 2023-11-16 19:14:47 +04:00
Andrey Rakhmatullin ffbf943e9d Merge pull request #6077 from 11-aryan/11-aryan 2023-11-16 19:13:02 +04:00
Andrey Rakhmatullin 75e99c75b3 Improve the docs about Crawler attributes and settings initialization. 2023-11-16 12:36:57 +04:00
Andrey Rakhmatullin 42b3a3a23b
Merge pull request #6147 from wRAR/crawler-init-docs 
Improve the docs about Crawler attributes and settings initialization.
 2023-11-16 12:35:43 +04:00
Andrey Rakhmatullin 603aa4924a Improve the docs about Crawler attributes and settings initialization. 2023-11-15 19:51:04 +04:00
Adrián Chaves 080fecd890 Drop the Authorization header on cross-domain redirect 2023-11-15 15:39:30 +01:00
Andrey Rakhmatullin 5fccf370b8 Update the RTD URL for coverage 2023-11-15 15:38:13 +01:00
Andrey Rakhmatullin ebdea4037a Update another output line. 2023-11-15 12:31:31 +04:00
Andrey Rakhmatullin db5a73f7bb Update the expected mypy output to match the old Python one. 2023-11-15 12:02:39 +04:00
Kiran 492584ec07
Updated README.rst (#6144) 2023-11-13 20:13:10 +01:00
Andrey Rakhmatullin 8776b4a6fb Fix env deps for typing-tests. 2023-11-12 20:52:29 +04:00
Andrey Rakhmatullin 204d6e180a Enable typing-tests in CI. 2023-11-12 20:47:52 +04:00
Andrey Rakhmatullin 5d55e4f56b Add mypy tests. 2023-11-12 20:15:06 +04:00
Andrey Rakhmatullin a6cee787dd Improve type hints for copy() and replace() in Request and Response. 2023-11-11 20:00:12 +04:00
Andrey Rakhmatullin b4acf5c827
Fix and remove most of the entries from the mypy ignore list (#6137) 2023-11-07 09:34:35 +01:00
Andrey Rakhmatullin c31e09d709
Merge pull request #6134 from wRAR/obsolete-setter 
Remove obsolete setters for body and url on Request and Response.
 2023-11-06 18:24:13 +04:00
Andrey Rakhmatullin 7c27c22a98
Full typing for scrapy/downloadermiddlewares (#6129) 2023-11-06 08:52:14 +01:00
cakemd eafe828484
scrapy.utils.data_path type hint change (#6133) 2023-11-06 08:37:18 +01:00
Andrey Rakhmatullin 2ac3ef73e6 Remove obsolete setters for body and url on Request and Response. 2023-11-03 20:12:18 +04:00
Jessica Allman-LaPorte 6587556af9
Make shell switching more clear in the tutorial (#6128) 2023-11-03 10:02:18 +01:00
Andrey Rakhmatullin f3561807a6
Merge pull request #6130 from wRAR/typing-spider-mw 
Full typing for scrapy/spidermiddlewares.
 2023-11-03 12:31:19 +04:00
Jeesang Kim dda6feb935
Improve assert readability (#6132) 2023-11-03 09:24:25 +01:00
Andrey Rakhmatullin e54dc59899 Full typing for scrapy/spidermiddlewares. 2023-11-02 23:40:14 +04:00
Andrey Rakhmatullin 593bfd895a
Merge pull request #6127 from wRAR/typing-response 
Full typing for Request and Response
 2023-11-02 17:46:42 +04:00
Andrey Rakhmatullin 24f21e96b9 Full typing for scrapy/http/response/text.py. 2023-11-01 01:53:25 +04:00
Andrey Rakhmatullin 01d9d28324 Full typing for scrapy/http/response/__init__.py. 2023-11-01 01:10:32 +04:00
Andrey Rakhmatullin 4cb2fc2c3b Update typing package versions. 2023-10-31 22:55:39 +04:00
Andrey Rakhmatullin 732557e698 Full typing for scrapy/http/request. 2023-10-31 22:48:22 +04:00
nihilisticneuralnet 04024f1e79
[Solved] JOBDIR= None for when Scheduler initializes disk queue even if JOBDIR is empty string (#6124) 
Co-authored-by: John Doe <johndoe@email.com>
 2023-10-31 17:37:26 +04:00
Andrey Rakhmatullin 7e6da37170
Merge pull request #6125 from Andy53/additional_ignored_extensions 
added_extensions
 2023-10-31 12:33:55 +04:00
andy53 8dff9633d0 added_extensions 2023-10-30 21:40:26 -06:00
Chenwei Niu 1f797d0fdb
Removed some deprecated functions and functionalities (#6116) 2023-10-30 09:59:11 +01:00