RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,453 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

10453 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 492c3bce9d
Don't run callbacks of requests from get_media_requests(). (#6386) 2024-06-03 13:28:20 +02:00
Andrey Rakhmatullin b4293e8f9e
Misc typing improvements. (#6384) 2024-05-31 08:50:36 +02:00
Andrey Rakhmatullin a011fa6f78
Merge pull request #6371 from wRAR/typing-contracts 
Full typing for scrapy/contracts.
 2024-05-30 20:59:50 +04:00
Andrey Rakhmatullin 469e8a23f8
Merge pull request #6381 from 0xdeb/referer-policy-tokens 
Add support for multiple referer policy tokens.
 2024-05-29 17:13:09 +04:00
Fabian Schneebauer 62a028b99d
Add spec link to scrapy/spidermiddlewares/referer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2024-05-29 13:19:27 +02:00
Fabian Schneebauer 0d58af8697 Add support for multiple referer policy tokens. 2024-05-29 10:59:32 +02:00
Sanchay Kumar cadb0dd707
Fix overridable methods in MediaPipeline (#6368) 2024-05-28 10:42:58 +02:00
Andrey Rakhmatullin 986d1ee1dd
Move CI from the decommissioned macos-11 to macos-latest. (#6372) 2024-05-28 10:37:19 +02:00
Andrey Rakhmatullin 9ba4dd311d
Install typing stubs for boto3 and botocore. (#6370) 2024-05-28 10:27:49 +02:00
Andrey Rakhmatullin f9a9860306
Merge pull request #6374 from Laerte/master 
docs: Remove top-level reactor imports from CrawlerProces/CrawlerRunner examples
 2024-05-27 14:36:34 +04:00
Laerte Pereira 6cd0857850 Move path 2024-05-26 19:57:16 -03:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Laerte Pereira 62c89aaf05
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:45 -03:00
Laerte Pereira 8ec67ca230
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:35 -03:00
Laerte Pereira 17e623cf0c
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:24 -03:00
Laerte Pereira 9d5a0d287b
Retrigger CI 2024-05-22 22:27:07 -03:00
Laerte Pereira e143dc7952
Update tests-macos.yml 2024-05-22 22:26:31 -03:00
Laerte Pereira 3f66b66e3f fix: checks 2024-05-22 22:01:55 -03:00
Laerte Pereira dc6a495fee
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-22 18:51:02 -03:00
Laerte Pereira 8210fae25a
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-22 18:50:50 -03:00
Laerte Pereira e676cd3ce0 docs: Remove top-level reactor imports from CrawlerProces/CrawlerRunner examples 2024-05-22 07:55:53 -03:00
Andrey Rakhmatullin b99526b740 Full typing for scrapy/contracts. 2024-05-19 15:45:51 +05:00
Andrey Rakhmatullin 631fc65fad
Update expectations of cookies after redirects. (#6367) 2024-05-16 16:42:09 +02:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Bagaudin Magomedov b88f22c6c5
Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 14:33:24 +02:00
Andrey Rakhmatullin b8e333c8ce
Full typing for scrapy/spiders. (#6356) 2024-05-13 12:07:51 +02:00
Andrey Rakhmatullin 4ed5c5ae91
Use ParamSpec for callables. (#6353) 2024-05-13 12:01:52 +02:00
Andrey Rakhmatullin 93f0628530
Improve typing for Spider.parse(). (#6274) 2024-05-13 11:55:45 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
Sanchay Kumar ae7bb849f5
Make certain args of ScrapyAgent and TunnelingAgent required (#6349) 2024-05-10 11:43:49 +02:00
Andrey Rakhmatullin 10a843ac1d
Merge pull request #6347 from aisha-partha/master 
Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default
 2024-05-10 11:47:40 +04:00
Adrián Chaves fe163d98ea
Fix typo 2024-05-08 20:41:20 +02:00
aisha-partha 2e13a9b8e1 Update METAREFRESH_IGNORE_TAGS default value in documentation 2024-05-08 23:33:13 +05:30
aisha-partha 3590a1f66b Closes #6342. Update documentation on METAREFRESH_IGNORE_TAGS default value change 2024-05-08 23:23:17 +05:30
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin 4300a1d240
Merge pull request #6344 from wRAR/fix-twinecheck 
Update MANIFEST.in.
 2024-05-08 08:46:28 +04:00
Andrey Rakhmatullin 6bbfb537f9 Update MANIFEST.in. 2024-05-08 00:39:05 +05:00
Andrey Rakhmatullin c9bac7a657
Merge pull request #6341 from wRAR/typing-downloader 
More typing for scrapy/core/downloader
 2024-05-07 12:24:25 +04:00
Andrey Rakhmatullin a828da98c3 Re-run pre-commit. 2024-05-06 22:34:36 +05:00
Andrey Rakhmatullin 045387e07f More typing for scrapy/core/downloader/handlers/http11.py. 2024-05-06 15:07:38 +05:00
Andrey Rakhmatullin af3e38ab1f Full typing for scrapy/core/downloader/handlers/http10.py. 2024-05-06 15:07:16 +05:00
Andrey Rakhmatullin e8e13ebb78 Full typing for scrapy/core/downloader/handlers/ftp.py. 2024-05-06 15:06:57 +05:00
Andrey Rakhmatullin ec4d407022 Full typing for smaller download handlers. 2024-05-06 15:06:33 +05:00
Andrey Rakhmatullin c4d2748ff5 Small typing improvements in scrapy/core/downloader/contextfactory.py. 2024-05-06 15:05:38 +05:00
Andrey Rakhmatullin 563ecbe966
Merge pull request #6333 from wRAR/typing-scrapy 
Full typing for scrapy/*.py
 2024-05-06 14:04:16 +04:00
Andrey Rakhmatullin d338982580
Merge pull request #6335 from wRAR/typing-self 
Use the Self type hint in from_crawler/from_settings.
 2024-05-06 14:02:10 +04:00
Andrey Rakhmatullin 8a083fb684
Merge pull request #6337 from wRAR/typing-linkextractors 
Full typing for scrapy/linkextractors.
 2024-05-06 14:02:03 +04:00
Andrey Rakhmatullin 07e31b9c93
Merge pull request #6336 from wRAR/typing-cookies 
Full typing for scrapy/http/cookies.py.
 2024-05-06 13:56:34 +04:00
Andrey Rakhmatullin 2cba7896d2 Small fix for _get_slot_key(). 2024-05-06 14:31:24 +05:00