RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,157 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

65 Commits

Author SHA1 Message Date
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Adrian b2d8b06be6
Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386) 
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
 2026-04-03 19:00:13 +05:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126) 
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
 2025-10-27 17:14:38 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Rohit Kumar Singh d2156696c4
Deprecate unused scrapy utils (#6519) 
* Added deprecation warnings for unused Scrapy.utils

* Grammatical corrections

* Exceptions class connected

* Deprecation of ScrapyJSONDecoder

* request_authenticate function deprecation

* Making all warning similar

* Added ignore statements for deprecation warning in tests

* Missing stacklevel attr. added

* Added Deprecation message
 2024-11-04 21:09:45 +05:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 5850b8f3e6 Merge branch 'master' into pyupgrade 2024-06-07 11:20:31 +05:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Andrey Rakhmatullin da42e8f124 Add parameters to typing.Dict. 2024-05-31 21:11:50 +05:00
Andrey Rakhmatullin b4293e8f9e
Misc typing improvements. (#6384) 2024-05-31 08:50:36 +02:00
Andrey Rakhmatullin 1f394306e1 Use the Self type hint in from_crawler/from_settings. 2024-05-05 22:05:11 +05:00
Adrián Chaves aa1bf69079
Mark hashlib usages as not intended for security (#6264) 2024-03-01 12:48:00 +01:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Laerte Pereira 7001193c80 Simplify the logic 2024-01-31 12:53:08 -03:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira 2487e3cc03 Cleanup deprecated fingerprint code in scrapy.utils.request 2024-01-31 10:05:50 -03:00
Rotzbua 0d445a3224
refactor(yield): use `yield from` syntax 2024-01-05 12:30:10 +01:00
Andrey Rakhmatullin 9df67a554e Add RequestFingerprinterProtocol. 2023-08-10 22:53:22 +04:00
Andrey Rakhmatullin e0dbc83bd2 More typing for scrapy/utils/request.py. 2023-05-07 00:54:23 +04:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Kian-Meng, Ang 3a263280ba Fix typos 2021-10-11 22:32:42 +08:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta ef09e0d10f
Some type hints 2020-12-02 10:39:10 -03:00
Eugenio Lacuesta 49e8a337f7
Flake8: remove E127 (continuation line over-indented for visual indent) 2020-05-06 09:37:01 -03:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Eugenio Lacuesta 714e0d3ac9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-20 12:41:38 -03:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
Eugenio Lacuesta df00389c16
Remove six.moves occurrences 2019-11-03 00:00:49 -03:00
Eugenio Lacuesta 415526d922
Remove __future__ imports 2019-11-02 23:26:38 -03:00
Andrey Rakhmatullin 397e883556 Replace to_native_str calls with to_unicode. 2019-10-31 15:22:53 +05:00
Benjamin Ooghe-Tabanou 6d6da78eda Add a keep_fragments parameter to the request_fingerprint function (#4104) 2019-10-30 09:13:36 +01:00
Adrián Chaves 447b3d9d81
Fix documentation typo: accesible → accessible (#4033) 2019-09-25 11:13:37 +02:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Mikhail Lyundin be7821a482 Optimization - avoid temporary list objects, unnecessary function call 2015-09-08 22:10:24 +03:00
Mikhail Korobov 642af00bb7 fix Referer logging 2015-08-26 02:19:33 +05:00