RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

36 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 52147017b4
Cleanup cookie handling in request_to_curl() (#7684) 
* Adjust CookiesT.

* Drop list of plain cookie dicts support from request_to_curl().

* Extract _decode_cookie().

* Unify logging.

* Extract _to_verbose_cookies().

* Sync and type hint _cookie_to_set_cookie_value() in tests.

* Add tests for bytes in Request.cookies.

* Type hint assertCookieValEqual().
 2026-06-29 21:49:09 +05:00
greymoth 185d6b9a20
Fix request_to_curl() corrupting dict cookies with bytes keys/values (#7675) 
* Fix request_to_curl() corrupting dict cookies with bytes keys/values

PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.

Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: apply _cookie_value_to_unicode to list-branch cookie key/value

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-27 00:53:32 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Adrian b2d8b06be6
Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
Andrey Rakhmatullin 55c17a8985
Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126) 
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
 2025-10-27 17:14:38 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Andrey Rakhmatullin 8700a5b7a9 Deprecate build_from_crawler() calling from_settings(). 2024-11-13 00:20:10 +05:00
Rohit Kumar Singh d2156696c4
Deprecate unused scrapy utils (#6519) 
* Added deprecation warnings for unused Scrapy.utils

* Grammatical corrections

* Exceptions class connected

* Deprecation of ScrapyJSONDecoder

* request_authenticate function deprecation

* Making all warning similar

* Added ignore statements for deprecation warning in tests

* Missing stacklevel attr. added

* Added Deprecation message
 2024-11-04 21:09:45 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Laerte Pereira 7001193c80 Simplify the logic 2024-01-31 12:53:08 -03:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira 019443dd57 Remove settings from default implementation test 2024-01-31 10:08:07 -03:00
Laerte Pereira 2487e3cc03 Cleanup deprecated fingerprint code in scrapy.utils.request 2024-01-31 10:05:50 -03:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
Mikhail Korobov 56e2eeac10 fix typing issues by upgrading mypy 2022-07-27 09:41:12 +05:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Eugenio Lacuesta 881b4f417f
Flake8: Remove E128 (tests, part 3) 2020-05-06 19:33:22 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
Eugenio Lacuesta 714e0d3ac9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-20 12:41:38 -03:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Benjamin Ooghe-Tabanou 6d6da78eda Add a keep_fragments parameter to the request_fingerprint function (#4104) 2019-10-30 09:13:36 +01:00
Daniel Graña 8d45b3c481 Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
Mikhail Korobov 7874bb9f13 PY3 port utils.request 2015-07-25 17:14:56 +02:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00