RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,195 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

9195 Commits

Author SHA1 Message Date
Adrián Chaves 42bca211d9 test_get_response_class_http: progress on documenting behavior changes 2022-06-30 21:28:02 +02:00
Adrián Chaves 0a77bfe2a0 get_response_class: use Response for compressed data 2022-06-30 20:39:34 +02:00
Adrián Chaves b7e88f4357 test_get_response_class_http: cover Apache bug scenarios 2022-06-30 19:12:13 +02:00
Adrián Chaves 3d7e2e5021 Add a test for the data URI handler now taking the body into account 2022-06-30 17:42:51 +02:00
Adrián Chaves 7ef061fc17 HttpCompressionMiddleware: clarify internal comment 2022-06-30 17:10:59 +02:00
Adrián Chaves 4759c68193 Address style issues 2022-06-30 16:57:17 +02:00
Adrián Chaves 983aa66687 Test ResponseTypes MIME types with the new implementation 2022-06-30 16:49:13 +02:00
Adrián Chaves 948b9ba125 Deprecate scrapy.responsetypes 2022-06-30 16:36:08 +02:00
Adrián Chaves 55105ddc7c Do not warn about internal usage of ResponseTypes.CLASSES 2022-06-30 16:29:18 +02:00
Adrián Chaves 54a0248301 Deprecate ResponseTypes.CLASSES 2022-06-30 16:24:42 +02:00
Adrián Chaves 1ea5a8e9e0 Deprecate ResponseTypes itself 2022-06-30 15:29:33 +02:00
Adrián Chaves fd2317bd6d Implement scrapy.utils.response.get_response_class 2022-06-30 15:15:52 +02:00
Adrián Chaves 19a00bd4ef Make xtractmime a dependency 2022-06-21 19:07:17 +02:00
Adrián Chaves d2accf2e39 Fix issues reported by static checks 2022-06-21 12:02:56 +02:00
Adrián Chaves 3fc9b732cb Refactor responsetypes 2022-06-21 11:12:10 +02:00
Adrián Chaves 9044ceb824 Refactor responsetypes 2022-06-21 10:27:20 +02:00
Akshay Sharma 8340394bcc removed prints 2022-06-21 02:45:39 -04:00
Akshay Sharma ef6290ef1b url ending fix 2022-06-21 02:44:18 -04:00
Akshay Sharma 3e9efe26ce fix 2022-06-20 21:12:34 -04:00
Akshay Sharma 0eb1de90e2 fix static checks 2022-06-20 21:10:06 -04:00
Akshay Sharma bedd4e51e1
Merge branch 'master' into integrate-mime 2022-06-20 20:19:04 -04:00
Akshay Sharma b9effde875 Refactored _guess_content_type 2022-06-20 20:15:33 -04:00
Emanuele d8223adfac
Typo: cleanup (verb) → clean up (#5538) 2022-06-20 11:54:05 +02:00
Michel Ace 6e878490e8
Support and prefer the .jsonl file extension (#4848) 2022-06-17 08:37:14 +02:00
Kromitvs 9e265a2c1f
Mind body to choose response class in cache, FTP and HTTP/1.0 (#4873) 2022-06-16 20:52:19 +02:00
Mikhail Korobov de0e2ccd7b
Merge pull request #5451 from FJMonteroInformatica/master 
make test files follow HTML conventions
 2022-06-12 00:46:45 +05:00
Mikhail Korobov e2769cfe48
Merge pull request #5524 from Laerte/master 
Remove minimal versions that Scrapy is tested against, cleanup
 2022-06-09 22:07:46 +05:00
Laerte Pereira 197aca2c94
docs: Remove leftover 2022-06-09 10:11:49 -03:00
Laerte Pereira c4c5c9f258
docs: Remove minimal versions paragraphs 2022-06-09 10:00:44 -03:00
Laerte Pereira 6770d1ec62
chore(tests): Remove validations for unsupported modules versions 2022-06-09 09:08:09 -03:00
Laerte Pereira 2e6721fd86
docs: Update minimal versions that Scrapy is tested against 2022-06-09 08:37:01 -03:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov bafe874ecc
Merge pull request #5358 from burakcank/signal_coroutine 
Document coroutines for signals.
 2022-05-28 13:38:05 +05:00
Mikhail Korobov 99cddecd28
Merge pull request #5497 from AngelikiBoura/issue_3264/command_parser_unhandled_error 
Issue #3264, fix error handling when spider is not matched
 2022-05-28 13:26:23 +05:00
Mohammadtaher Abbasi 896f16f2de make thumb_path method backwards compatible 2022-05-25 23:59:25 +04:30
Mohammadtaher Abbasi 5c586d78f0 add tests 2022-05-25 23:58:09 +04:30
Mohammadtaher Abbasi f39def4492 add docs 2022-05-25 23:57:38 +04:30
Mohammadtaher Abbasi 8633190012 pass on item to thumb_path function as additional argument resolves #5504 2022-05-24 14:51:17 +04:30
Andreas Tziortziortziopoulos b5c15d87ff [issue3264] Separate test for not matched spider to a url 2022-05-22 12:19:20 +03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Andrey Rahmatullin 965fde24a4 Pin mitmproxy to < 8 for now (#5459) 2022-05-20 07:17:09 +02:00
Akshay Sharma b2181d76d5 Merge branch 'master' of https://github.com/AKSHAYSHARMAJS/scrapy into integrate-mime 2022-05-08 17:48:57 -04:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
AngelikiBoura b2afcbfe2b
Fix typos in three files for Flake8 check (#5487) 
* Fix typos in extensions files

Made some fixes in files memusage.py and statsmailer.py
in order to pass the flake8 check.

* Fix typos in twisted_reactor_custom_settings_same.py

A small change was needed in order for flake8 check
to pass.
 2022-05-05 15:49:52 +02:00
Andrey Rahmatullin ded28f7bb9
Merge pull request #5461 from yash-fn/patch-1 
documentation update for multiple spiders
 2022-04-15 14:39:26 +05:00
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00