RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,225 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11225 Commits

Author SHA1 Message Date
GeorgeA92 0ffc52a491 per slot settings: test added (delays for each download slots) 2022-05-11 10:40:12 +03:00
GeorgeA92 2006060688 per slot settings: codestyle(flake8) fix, code line length 2022-05-11 10:29:53 +03:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
AngelikiBoura b2afcbfe2b
Fix typos in three files for Flake8 check (#5487) 
* Fix typos in extensions files

Made some fixes in files memusage.py and statsmailer.py
in order to pass the flake8 check.

* Fix typos in twisted_reactor_custom_settings_same.py

A small change was needed in order for flake8 check
to pass.
 2022-05-05 15:49:52 +02:00
PluT00 cc16af35af Add deprecation warning test for scrapy.pipelines.images.NoimagesDrop 2022-05-03 11:29:21 +03:00
PluT00 9f659bd63c Fix deprecation of scrapy.pipelines.images.NoimagesDrop 2022-05-01 14:26:03 +03:00
PluT00 7de9ed5bd4 Deprecate scrapy.pipelines.images.NoimagesDrop 2022-05-01 01:34:35 +03:00
Andrey Rahmatullin b9b9422bb1
Merge pull request #5482 from alexpdev/parse_help_msg 
Fix TypeError in parse command
 2022-04-29 11:10:15 +05:00
Alex 915c288205 edit 2022-04-24 15:17:48 -07:00
Alex 56c9098d6a changed default depth to 1 2022-04-24 15:17:48 -07:00
Alex b0f5503cb8 Fixes Issue #5481 2022-04-24 15:17:48 -07:00
Alex 636127ec1e tests that all CLI help messages don't throw errors 2022-04-22 16:32:58 -07:00
Andrey Rahmatullin ded28f7bb9
Merge pull request #5461 from yash-fn/patch-1 
documentation update for multiple spiders
 2022-04-15 14:39:26 +05:00
Adrián Chaves aead27bcbd
Add release notes for 2.6.2 (#5448) 2022-04-14 15:06:22 +02:00
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin fa4c85f4ec
Merge pull request #5471 from Gallaecio/2.6-release-notes-request-serialization 
2.6 release notes request serialization
 2022-04-08 18:14:50 +05:00
Adrián Chaves bae3f87455 Cover a backward-incompatible Request serialization change in the 2.6 release notes 2022-04-08 12:04:02 +02:00
Andrey Rahmatullin 4af22bf157
Pin mitmproxy to < 8 for now (#5459) 2022-04-08 11:26:23 +02:00
Andrey Rahmatullin 35b44f3b98
Merge pull request #5436 from Gallaecio/fix-crawlerprocess-regression 
CrawlerProcess: initialize the reactor only once
 2022-03-25 13:21:21 +05:00
Andrey Rahmatullin 1e1bf353bf
Merge pull request #5440 from Gallaecio/fix-closespider-exception 
engine: prevent slot method call after unsetting the slot
 2022-03-24 19:43:31 +05:00
Silvio Pavanetto 0beed7055c fix: return unique_list only when link_extractor.unique is True 2022-03-23 17:28:55 +01:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
D00399830 fcf3d8e0a0 Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it 2022-03-21 14:09:31 -06:00
Adrián Chaves b95c634b86 Document how to enforce Scrapy versions on Scrapy components 2022-03-17 22:23:25 +01:00
FJMonteroInformatica 6a3f2ee687 HTML Conventions 2022-03-17 20:09:56 +01:00
Yann Defretin 0905d42e33
refactored url_has_any_extension function 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2022-03-17 11:19:09 +01:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Yann Defretin 5b4b8b6fb1
added test for new url_has_any_extension function 2022-03-16 22:32:05 +01:00
Adrián Chaves b78e6915c6 Clarify that without async-to-sync conversions items yielded before an exception are processed 2022-03-16 20:17:25 +01:00
Adrián Chaves c961438d5d tests: cover scenarios of bad results from process_spider_output 2022-03-16 18:45:56 +01:00
Adrián Chaves fd08bb6cd9 Refactor the asynchronous process_spider_output documentation 2022-03-16 18:10:57 +01:00
Adrián Chaves cb3afcf0e7 Merge remote-tracking branch 'upstream/master' into wrar-asyncio-parse-asyncgen-proper-rebased 2022-03-16 18:08:47 +01:00
Yann Defretin 78ba4b033b
fixed detection of extension like ".tar.gz" in URL 2022-03-16 15:14:24 +01:00
Adrián Chaves b59a69be17 Test that a low CLOSEPIDER_TIMEOUT does not raise an exception 2022-03-15 12:10:41 +01:00
Laerte Pereira c1d4be8cb5
Restore ‘-o -’ support (#5445) 2022-03-15 11:30:30 +01:00
Andrey Rahmatullin 2d6042b108
Merge pull request #5432 from Farsene1/master 
Recommend Common Crawl instead of Google Cache
 2022-03-11 20:05:43 +05:00
NaincyKumariKnoldus e264cc30d1
removed the pywin32 docs section (#5370) 2022-03-10 14:54:33 +01:00
Ali Rastegar d469214fe7
Update tutorial.rst (#5442) 
Fixed typo
 2022-03-08 10:29:22 +01:00
Adrián Chaves 62a0081266 engine: prevent slot method call after unsetting the slot 2022-03-07 12:00:44 +01:00
Adrián Chaves 3bf6baeaa7 initiated → initialized 2022-03-02 17:30:05 +01:00
Adrián Chaves 96fc4dae15 CrawlerProcess: test a multi-spider scenario 2022-03-02 17:30:05 +01:00
Adrián Chaves 3ecbea4b87 CrawlerProcess: initiate the reactor only once 2022-03-02 17:30:05 +01:00
Florentin ccdbb795ff Recommend Common Crawl instead of Google Cache 2022-03-01 22:01:55 +01:00
Adrián Chaves 50c8becbe0
Freeze and upgrade CI packages (#5429) 2022-03-01 17:29:08 +01:00
Adrián Chaves 23537a0f95 Bump version: 2.6.0 → 2.6.1 2022-03-01 13:48:40 +01:00
Adrián Chaves fab3e90729 Cover 2.6.1 in the release notes 2022-03-01 13:41:20 +01:00
Adrián Chaves d60636d0de Fix redirect handling regression 2022-03-01 13:06:58 +01:00
Adrián Chaves 84853c4fa6 bandit: allow-list B324 for the time being 2022-03-01 13:01:20 +01:00
Adrián Chaves 6b63e7c147 Bump version: 2.5.0 → 2.6.0 2022-03-01 12:43:11 +01:00