RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,189 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

71 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 00098cb596
Assorted docstring fixes (#7698) 
* Smaller fixes.

* Add more code blocks in docstrings.

* Queue stuff.

* Response stuff.

* Round 2.

* Address feedback.
 2026-06-30 18:27:32 +05:00
Ethan Kuo df2f3d708e
fix open_in_browser() logic issue plus add new tests (#7506) 2026-06-04 16:24:50 +02:00
Andrey Rakhmatullin fc4c57e795
Fix the substitution pattern in open_in_browser(). (#7459) 
* Fix the substitution pattern in open_in_browser().

* Formatting.
 2026-04-24 13:54:37 +05:00
Ashar Khan d239fcf936
Fix get_meta_refresh to correctly resolve relative URLs using <base> tag (#7047) 2025-09-10 15:17:16 +02:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin c2832ed131 Add flake8-pyi rules to ruff. 2024-12-12 20:44:28 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin b4293e8f9e
Misc typing improvements. (#6384) 2024-05-31 08:50:36 +02:00
Andrey Rakhmatullin 6ecc9e0a34
Add typing for scrapy/commands (#6268) 2024-03-06 13:21:08 +01:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves c5dad41190 Speed up tests, remove comments without regexps 2024-02-02 14:03:16 +01:00
Adrián Chaves 1533b69032 Test and address ReDoS attack vectors for open_in_browser 2023-12-13 12:01:35 +01:00
Adrián Chaves eb8b2c5197 Mention open_in_browser in the release notes 2023-11-29 12:13:04 +01:00
Adrián Chaves c947f51077 Set an arbitrary upper limit on ReDoS-vulnerable regexps 2023-11-29 11:54:08 +01:00
Chenwei Niu 1f797d0fdb
Removed some deprecated functions and functionalities (#6116) 2023-10-30 09:59:11 +01:00
Andrey Rakhmatullin f5f593e5f5 Remove a workaround for a w3lib typing bug. 2023-08-06 17:46:28 +04:00
Andrey Rakhmatullin d20f278882 Add a link to the w3lib typing fix. 2023-02-24 00:28:24 +04:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Andrey Rakhmatullin 232aab53b3 Fix most of the new typing issues in utils. 2023-01-25 22:28:15 +04:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Samuel Marchal f2c800c5c9 Improve open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
Eugenio Lacuesta ef09e0d10f
Some type hints 2020-12-02 10:39:10 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta 9e99be982a
Remove backslash 2020-07-09 09:51:28 -03:00
Eugenio Lacuesta a72f5aadcf
Flake8: Remove E128 (utils module) 2020-05-06 17:38:39 -03:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Mikhail Korobov e285b1d6c2 retry stats 2017-02-14 20:27:49 +05:00
Paul Tremberth cbb695d08c Merge pull request #1881 from nyov/dedupe 
[MRG+1] Remove duplicate code now handled by newer w3lib
 2016-04-06 15:47:06 +02:00
nyov 3787fec460 Remove duplicate code now handled by newer w3lib 
see f3029a6a10
 2016-03-30 14:58:03 +00:00
pawelmhm 65c7c05060 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-12 14:16:40 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Mikhail Korobov faf9265c91 fixed compatibility with Twisted 15.4.0 2015-09-04 20:50:48 +05:00
Mikhail Korobov 7f927f68e1 PY3 port scrapy.utils.response 2015-08-01 00:35:43 +05:00
Julia Medina 7a7c539116 Rename scrapy/utils/decorator.py to scrapy/utils/decorators.py 2015-04-29 21:27:19 -03:00
nyov 84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
Daniel Graña ec7833a910 Deprecate body_or_str helper function only used by xml iterators 2013-11-19 19:21:54 -02:00
Daniel Graña 5b9a7814a5 support TextResponse in open_in_browser util 2012-04-25 16:04:37 -03:00
Pablo Hoffman 36df87b4de ignore meta-refresh redirects embedded in <script> tags. related to issue 18 2011-11-14 16:54:13 -02:00
Pablo Hoffman ec1ef0235f ignore meta-refresh redirect when embedded inside <noscript> tag. closes issue 18 2011-11-14 16:25:22 -02:00