RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov bafe874ecc
Merge pull request #5358 from burakcank/signal_coroutine 
Document coroutines for signals.
 2022-05-28 13:38:05 +05:00
Mikhail Korobov 99cddecd28
Merge pull request #5497 from AngelikiBoura/issue_3264/command_parser_unhandled_error 
Issue #3264, fix error handling when spider is not matched
 2022-05-28 13:26:23 +05:00
Eugenio Lacuesta 2c65066ad9
Avoid exceptions on copy 2022-05-27 19:56:42 -03:00
Eugenio Lacuesta 1b1d518e2b
Merge remote-tracking branch 'upstream/master' into case-insensitive-dict 2022-05-27 18:45:10 -03:00
Mohammadtaher Abbasi 896f16f2de make thumb_path method backwards compatible 2022-05-25 23:59:25 +04:30
Mohammadtaher Abbasi 5c586d78f0 add tests 2022-05-25 23:58:09 +04:30
Mohammadtaher Abbasi f39def4492 add docs 2022-05-25 23:57:38 +04:30
Mohammadtaher Abbasi 8633190012 pass on item to thumb_path function as additional argument resolves #5504 2022-05-24 14:51:17 +04:30
Andreas Tziortziortziopoulos b5c15d87ff [issue3264] Separate test for not matched spider to a url 2022-05-22 12:19:20 +03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Andrey Rahmatullin 965fde24a4 Pin mitmproxy to < 8 for now (#5459) 2022-05-20 07:17:09 +02:00
Adrián Chaves 1c1cd5d8ea Update the 2.6.2 release notes 2022-05-20 07:05:26 +02:00
Adrián Chaves 84c29a286f
Unset the release date of still-unreleased 2.6.2 (#5503) 2022-05-20 06:45:38 +02:00
Eugenio Lacuesta 1c031b8a8d
Underscore CaseInsensitiveDict normkey/normvalue 2022-05-12 13:10:08 -03:00
Eugenio Lacuesta 7297ae566f
Merge remote-tracking branch 'upstream/master' into case-insensitive-dict 2022-05-12 12:43:41 -03:00
GeorgeA92 0ffc52a491 per slot settings: test added (delays for each download slots) 2022-05-11 10:40:12 +03:00
GeorgeA92 2006060688 per slot settings: codestyle(flake8) fix, code line length 2022-05-11 10:29:53 +03:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
AngelikiBoura b2afcbfe2b
Fix typos in three files for Flake8 check (#5487) 
* Fix typos in extensions files

Made some fixes in files memusage.py and statsmailer.py
in order to pass the flake8 check.

* Fix typos in twisted_reactor_custom_settings_same.py

A small change was needed in order for flake8 check
to pass.
 2022-05-05 15:49:52 +02:00
PluT00 cc16af35af Add deprecation warning test for scrapy.pipelines.images.NoimagesDrop 2022-05-03 11:29:21 +03:00
PluT00 9f659bd63c Fix deprecation of scrapy.pipelines.images.NoimagesDrop 2022-05-01 14:26:03 +03:00
PluT00 7de9ed5bd4 Deprecate scrapy.pipelines.images.NoimagesDrop 2022-05-01 01:34:35 +03:00
Andrey Rahmatullin b9b9422bb1
Merge pull request #5482 from alexpdev/parse_help_msg 
Fix TypeError in parse command
 2022-04-29 11:10:15 +05:00
Alex 915c288205 edit 2022-04-24 15:17:48 -07:00
Alex 56c9098d6a changed default depth to 1 2022-04-24 15:17:48 -07:00
Alex b0f5503cb8 Fixes Issue #5481 2022-04-24 15:17:48 -07:00
Alex 636127ec1e tests that all CLI help messages don't throw errors 2022-04-22 16:32:58 -07:00
Andrey Rahmatullin ded28f7bb9
Merge pull request #5461 from yash-fn/patch-1 
documentation update for multiple spiders
 2022-04-15 14:39:26 +05:00
Adrián Chaves aead27bcbd
Add release notes for 2.6.2 (#5448) 2022-04-14 15:06:22 +02:00
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin fa4c85f4ec
Merge pull request #5471 from Gallaecio/2.6-release-notes-request-serialization 
2.6 release notes request serialization
 2022-04-08 18:14:50 +05:00
Adrián Chaves bae3f87455 Cover a backward-incompatible Request serialization change in the 2.6 release notes 2022-04-08 12:04:02 +02:00
Andrey Rahmatullin 4af22bf157
Pin mitmproxy to < 8 for now (#5459) 2022-04-08 11:26:23 +02:00
Andrey Rahmatullin 35b44f3b98
Merge pull request #5436 from Gallaecio/fix-crawlerprocess-regression 
CrawlerProcess: initialize the reactor only once
 2022-03-25 13:21:21 +05:00
Andrey Rahmatullin 1e1bf353bf
Merge pull request #5440 from Gallaecio/fix-closespider-exception 
engine: prevent slot method call after unsetting the slot
 2022-03-24 19:43:31 +05:00
Silvio Pavanetto 0beed7055c fix: return unique_list only when link_extractor.unique is True 2022-03-23 17:28:55 +01:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
D00399830 fcf3d8e0a0 Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it 2022-03-21 14:09:31 -06:00
Adrián Chaves b95c634b86 Document how to enforce Scrapy versions on Scrapy components 2022-03-17 22:23:25 +01:00
FJMonteroInformatica 6a3f2ee687 HTML Conventions 2022-03-17 20:09:56 +01:00
Yann Defretin 0905d42e33
refactored url_has_any_extension function 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2022-03-17 11:19:09 +01:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Yann Defretin 5b4b8b6fb1
added test for new url_has_any_extension function 2022-03-16 22:32:05 +01:00
Adrián Chaves b78e6915c6 Clarify that without async-to-sync conversions items yielded before an exception are processed 2022-03-16 20:17:25 +01:00
Adrián Chaves c961438d5d tests: cover scenarios of bad results from process_spider_output 2022-03-16 18:45:56 +01:00
Adrián Chaves fd08bb6cd9 Refactor the asynchronous process_spider_output documentation 2022-03-16 18:10:57 +01:00