RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

61 Commits

Author SHA1 Message Date
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Rui Xi 813fd9f1ac
Restore brotlicffi support (#7160) 
* restore brotlicffi dependency

* pre-commit
 2025-11-25 12:06:55 +05:00
Rui Xi 14737e91ed
Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 14:21:50 +05:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037) 
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
 2025-09-02 21:51:47 +05:00
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00
NavpreetSSidhu d27d0a4ed9
Deprecate setting overrides through spider attributes (#6994) 2025-08-07 21:26:10 +02:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 391af6afcc
Unknown encoding handling in HttpCompressionMiddleware, restore x-gzip support (#6618) 
* Unknown encoding handling in HttpCompressionMiddleware.

* Implement the changes for unknown encoding handling.

* Restore support for Content-Encoding: x-gzip.

* Simplify the decoding logic.

* Add tests for the unsupported encoding warning.

* Add a test for the "no zstandard" warning.
 2025-02-27 22:37:01 +05:00
Andrey Rakhmatullin e7595837a6 Add flake8-raise rules to ruff. 2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Laerte Pereira cab1016bb6 Add brotlicffi support 2024-03-05 20:24:17 -03:00
vishesh10 e208f82076
Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves f3064254ce Merge branch 'patch-2' of github.com:GeorgeA92/scrapy into patch-2 2020-11-06 16:42:58 +01:00
Adrián Chaves ee98771fa7 Remove unused variable 2020-11-06 16:42:32 +01:00
Adrián Chaves 5d2a9cf5b1
Merge branch 'master' into patch-2 2020-11-06 14:17:54 +01:00
Adrián Chaves a3e53027ec Test HttpCompressionMiddleware subclasses with custom, parameterless __init__ 2020-11-06 14:16:26 +01:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
GeorgeA92 c22e810658 httocompression tests added 2020-09-22 07:47:37 +03:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 25eeb77ba6
Merge remote-tracking branch 'upstream/master' into flake8-max-line-length 2020-05-06 16:52:17 -03:00
Eugenio Lacuesta 8643e8d355
Flake8: remove E123 (Closing bracket does not match indentation of opening bracket's line) 2020-05-06 12:26:04 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernandez Cabot 69a8648bef fix E251 2020-02-22 12:32:38 +01:00
Evgeny Dorofeev 4e56571a19 [HttpCompressionMiddleware] fix delimiter for Accept-Encoding header 2020-01-29 15:49:22 +03:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00