RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,323 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

46 Commits

Author SHA1 Message Date
vishesh10 e208f82076
Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves f3064254ce Merge branch 'patch-2' of github.com:GeorgeA92/scrapy into patch-2 2020-11-06 16:42:58 +01:00
Adrián Chaves ee98771fa7 Remove unused variable 2020-11-06 16:42:32 +01:00
Adrián Chaves 5d2a9cf5b1
Merge branch 'master' into patch-2 2020-11-06 14:17:54 +01:00
Adrián Chaves a3e53027ec Test HttpCompressionMiddleware subclasses with custom, parameterless __init__ 2020-11-06 14:16:26 +01:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
GeorgeA92 c22e810658 httocompression tests added 2020-09-22 07:47:37 +03:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 25eeb77ba6
Merge remote-tracking branch 'upstream/master' into flake8-max-line-length 2020-05-06 16:52:17 -03:00
Eugenio Lacuesta 8643e8d355
Flake8: remove E123 (Closing bracket does not match indentation of opening bracket's line) 2020-05-06 12:26:04 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernandez Cabot 69a8648bef fix E251 2020-02-22 12:32:38 +01:00
Evgeny Dorofeev 4e56571a19 [HttpCompressionMiddleware] fix delimiter for Accept-Encoding header 2020-01-29 15:49:22 +03:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Pawel Miech 3daf473686 [httpcompression] skip test if no brotli 2017-02-06 13:49:41 +01:00
Pawel Miech f8f8bbe080 [httpcompression] import brotli when available 2017-02-06 12:19:00 +01:00
Pawel Miech 16c4b4e184 [httpcompression] add support for br - brotli content encoding 2017-02-06 11:41:08 +01:00
Paul Tremberth 10a2c46e12 [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-06-20 16:37:00 +02:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Konstantin Lopuhin 0b9336418e py3: port compression downloader middleware and tests 2016-01-18 16:43:58 +03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00