RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 131 MiB
Commit Graph

52 Commits

Author SHA1 Message Date
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin d08f559600
Add flake8-type-checking. (#6413) 2024-06-25 10:20:59 +02:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Andrey Rakhmatullin c0efb271a2 Ignore typing for twisted.internet.reactor globally. 2023-01-27 23:19:44 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Andrey Rakhmatullin 2e33fb812b Silence improper typing of twisted.internet.reactor in tests. 2023-01-25 22:15:07 +04:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Andrey Rakhmatullin 1e1cfc26db Copy resource classes from twisted.web.test.test_webclient. 2022-02-08 21:01:16 +05:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Adrián Chaves 3199048520 Complete Azure Pipelines CI setup 2020-07-03 00:56:24 +02:00
Adrián Chaves 6e58da1dcd
Merge branch 'master' into azure-pipelines 2020-07-02 17:49:42 +02:00
Adrián Chaves 94ee68695a Mock server: use 127.0.0.1 also for HTTPS 
Windows throws an error about 0.0.0.0 being external:
https://stackoverflow.com/a/23857995/939364
 2020-04-16 15:37:06 +02:00
Eugenio Lacuesta ad70497416
Remove unnecessary parentheses in class definition 2020-02-04 13:30:13 -03:00
Eugenio Lacuesta e0ef8ad2d6
CrawlerRunner test for Response.ip_address 2020-02-03 15:52:15 -03:00
Eugenio Lacuesta 4851efdfb0
Flake8 adjustments 2020-02-03 14:50:54 -03:00
Eugenio Lacuesta bb8f7dc609
Mock DNS server 2020-02-03 14:50:14 -03:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Eugenio Lacuesta 342e3b5bd5
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-08 12:29:15 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Adrián Chaves f02c3d1dcf
Use communicate() instead of wait() after killing the mock server (#4095) 2019-10-31 13:31:33 +01:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Kostis Anagnostopoulos 3a493b6066 fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
Daniel Graña 0e532e3dd8 Creating a connection to 0.0.0.0 fails on windows but not on linux nor mac 2018-08-15 01:35:01 -03:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Paul Tremberth 6d14e392f1 Remove old test certificate+key 2017-04-27 23:35:01 +02:00
Paul Tremberth b1a0a6e258 Make mockserver runnable outside of tox 
Add POST support for Echo resource
 2017-04-25 17:01:54 +02:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Mikhail Korobov 04b2f79e7a Remove code required to support ancient twisted versions. See GH-1887. 2017-02-07 22:30:58 +05:00
Paul Tremberth cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Konstantin Lopuhin bf5f54fa33 py3: fix getarg 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 0680950b98 py3: pass CrawlTestCase.test_referer_header, fixing Echo resource in mockserver and json decoding in test 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 20b839b44b py3: pass first crawl test (test_follow_all): fix mock server 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin 81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin 2aa6c92ffc py3 fixes in tests.mockserver 2016-01-14 17:52:50 +03:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
nyov dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
tpeng cd19382754 attemp to fix travis fails 2014-11-25 14:20:25 +01:00