RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

51 Commits

Author SHA1 Message Date
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Andrey Rakhmatullin 584d99af30
Split long test files (#7329) 2026-03-13 10:30:25 +01:00
Adrian Chaves 8e28f938d2 Make test_no_warning_when_referer_middleware_present less brittle 2026-03-12 14:29:51 +01:00
Adrian Chaves 886131c7b2 Run pre-commit 2026-03-12 13:54:00 +01:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves 842d0becf0
Rename test function 2026-01-20 12:15:25 +01:00
Adrian Chaves f7f18123eb Remove body on 307/308 redirects and switch to GET on 301 2026-01-19 19:15:20 +01:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037) 
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
 2025-09-02 21:51:47 +05:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Adrián Chaves 080fecd890 Drop the Authorization header on cross-domain redirect 2023-11-15 15:39:30 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Eugenio Lacuesta be39f274b8
Merge branch 'master' into flake8-remove-e128 2020-05-08 18:33:29 -03:00
Antonio Gordillo Toledo cf09af787e
Remove Python 2 encoding header from files (#4553) 2020-05-08 15:45:19 +02:00
Eugenio Lacuesta 9c6d13075a
Flake8: Remove E128 (tests, part 1) 2020-05-06 18:56:14 -03:00
Eugenio Lacuesta 25eeb77ba6
Merge remote-tracking branch 'upstream/master' into flake8-max-line-length 2020-05-06 16:52:17 -03:00
Eugenio Lacuesta fe0c582ee0
Flake8: remove E127 in tests (continuation line over-indented for visual indent) 2020-05-06 09:49:10 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Amardeep Bhowmick 66cbceeb0a Fix redirection error when the Location header value starts with 3 slashes (#4042) 2019-10-30 09:09:12 +01:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 
* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import
 2016-10-19 23:26:12 -03:00
Elias Dorneles 430e90f4b2 minor refactor on metarefresh redirect mware test 2015-09-13 12:29:41 -03:00
Elias Dorneles 81950f773d made encoding conversion more explicit, added test for header with utf-8 encoding replicating what browsers do 2015-09-11 18:51:48 -03:00
Elias Dorneles 179c4588ca adding test for latin1 location 2015-09-10 19:36:47 -03:00
Elias Dorneles defa899135 PY3 port redirect middleware 2015-09-10 16:31:59 -03:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
Jakob de Maeyer c153217bfb Fix RedirectMiddleware test 2015-07-17 16:11:53 +02:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00