RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,581 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

5525 Commits

Author SHA1 Message Date
Adrián Chaves f8cdd092d5 Deprecate binary_is_text 2023-01-10 21:32:00 +01:00
Adrián Chaves 9ee2ebea54 Document TextResponse.base_url 2023-01-10 21:27:58 +01:00
Adrián Chaves ff37b274a6 Remove unused variable 2023-01-10 20:09:16 +01:00
Adrián Chaves fc2833d6bd Fix _get_encoding_or_mime_type_from_headers handling on comma-separated Content-Encoding values 2023-01-10 18:32:27 +01:00
Adrián Chaves ee6f5c56b2 Support multi-encoding Content-Encoding headers 2023-01-10 17:55:35 +01:00
Adrián Chaves c34853100a Revert changes on the deprecated decompression module 2023-01-10 17:00:33 +01:00
Adrián Chaves 2178521a79 Make sure that supplied “unknown” MIME types from Content-Type are ignored 2023-01-10 13:41:36 +01:00
Adrián Chaves fcc9e90553 Wrap XHTML with XmlResponse 2023-01-10 07:30:40 +01:00
Adrián Chaves 24e63f3294 Remove unused imports 2023-01-08 17:28:34 +01:00
Adrián Chaves 716561972b Fix typing issues 2023-01-08 17:27:55 +01:00
Adrián Chaves b2f2788bd4 Add tests for binary-vs-text body 2023-01-08 16:36:35 +01:00
Adrián Chaves f9f9d61457 Always prefer Content-Type over Content-Disposition 2023-01-08 15:46:32 +01:00
Adrián Chaves e7d36a305d Ignore file path extensions in HTTP responses 2023-01-08 14:24:54 +01:00
Adrián Chaves d319b62d16 Fix typing issues 2023-01-08 13:01:14 +01:00
Adrián Chaves f029be3aa7 Fix Apache bug handling 2023-01-08 12:55:17 +01:00
Adrián Chaves c6d416c69f Merge remote-tracking branch 'upstream/master' into integrate-mime 2023-01-08 11:25:00 +01:00
Adrián Chaves 724b033287
Fix CI issues related to asyncio (#5782) 2023-01-06 19:51:07 +01:00
Adrián Chaves b336980bd1 Merge remote-tracking branch 'upstream/master' into integrate-mime 2023-01-02 13:14:39 +01:00
Steve Niafas 7ae32ea38d
Fix transparency for processed webp images (#5767) 2022-12-22 14:33:14 +01:00
Alex 93ace91ace
Merge branch 'scrapy:master' into fix_shell_fetch_alt 2022-12-13 21:50:28 -08:00
Alex c2de9372a2 added unittest for scrapy shell command and fixed flake8 error 2022-12-13 21:37:17 -08:00
Alex 8b09b0e0d7 set asyncio event loop at start of new thread 2022-12-13 20:41:48 -08:00
Laerte Pereira e92e201b19
Silence `filelock` logger (#5754) 2022-12-12 11:55:15 +01:00
Andrey Rahmatullin 5bd27191a2
Bump mypy, flake8, and pylint (#5738) 2022-11-28 19:37:57 +01:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Hanzallah Burney b6e98ce6b6
Remove unnecessary backwards compatibility comments (#5732) 2022-11-24 15:01:15 +01:00
Andrey Rahmatullin 0417e3ede2
Merge pull request #5727 from JazzGlobal/Better-Language-Surrounding-Logging 
Better language surrounding logging
 2022-11-23 22:29:17 +05:00
Hanzallah Burney f6e9e6592a
Cleanup settings._DictProxy and scrapy.telnet (#5730) 2022-11-23 15:48:34 +01:00
Laerte Pereira e769532644
Remove `noconnect` deprecate code 2022-11-23 08:30:11 -03:00
Andrey Rahmatullin 45c2bd7d9c
Merge pull request #5150 from gustavo-bordin/master 
changing dunder-str to dunder-repr
 2022-11-23 13:12:06 +05:00
Andrey Rahmatullin c3b1700774
Restore lost typing 2022-11-23 12:17:30 +05:00
Andrey Rahmatullin 1fdd0a70a0
Restore lost typing 2022-11-23 12:16:48 +05:00
Christopher Gambrell bdc0bca5b1 Replace M occurrences with MiB for accuracy. 2022-11-22 12:10:49 -05:00
Christopher Gambrell fc8968672a renamed variables to clarify that we are using peak memory and not current memory utilization. 2022-11-22 11:49:28 -05:00
Laerte Pereira 8e0025f53d
Remove support for override settings with `SCRAPY_` environment variables 2022-11-22 09:38:54 -03:00
Andrey Rahmatullin aac6103179
Merge pull request #5722 from JazzGlobal/5717-memusage-logging-extension 
Adds get_virtual_size to _check_limit Logging
 2022-11-22 16:08:20 +05:00
Christopher Gambrell eb159c78f1 Use variable and convert to megabytes. 2022-11-22 03:36:00 -05:00
Christopher Gambrell 8f2adad7a7 Log self.get_virtual_size() on every call of _check_limit 2022-11-22 00:48:24 -05:00
Christopher Gambrell 24a18e9af1 Adds virtualsize property to _check_limit error log. 2022-11-21 17:41:06 -05:00
Laerte Pereira 1a6408c3fa
Remove `FilteringLinkExtractor` 2022-11-21 07:30:20 -03:00
Laerte Pereira d5b6c236a9
Remove deprecated code (#5719) 2022-11-21 09:32:26 +01:00
islem-esi 1200a54543 minor fix for readability 2022-11-15 16:28:45 +01:00
Laerte Pereira ae3fd01729
Fix stray “commands” (#5712) 2022-11-10 15:38:46 +01:00
Andrey Rahmatullin b03d84e9fb
Merge branch 'master' into 3689-update 2022-11-07 16:33:21 +05:00
Andrey Rahmatullin 77cd511a5e
Merge pull request #5489 from PluT00/deprecate-NoimagesDrop 
Deprecate scrapy.pipelines.images.NoimagesDrop
 2022-11-05 14:18:19 +05:00
Andrey Rahmatullin 8004075823
Merge pull request #5699 from elacuesta/crawlspider-callback-keyword-arguments 
CrawlSpider: pass cb_kwargs from process_request
 2022-11-02 17:35:03 +05:00
Andrey Rakhmatullin 6ded3cf4cd Bump version: 2.7.0 → 2.7.1 2022-11-02 17:00:47 +06:00
Andrey Rahmatullin a95a338eea
Merge pull request #5599 from tonal/patch-1 
Partial in is_generator_with_return_value
 2022-10-31 20:08:16 +05:00
Eugenio Lacuesta 3259a42525
CrawlSpider: pass cb_kwargs from process_request 2022-10-30 13:17:37 -03:00