RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,758 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

793 Commits

Author SHA1 Message Date
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Vostretsov Nikita afb1458bd3 tests for new signal 2018-08-17 14:39:54 +00:00
Mikhail Korobov c7654f7cb1
Merge pull request #3379 from StasDeep/fix/issue-3378 
Use inspect.getmembers in tested_methods_from_spidercls
 2018-08-17 19:25:41 +05:00
Daniel Graña 38608bc249 Use ignore_errors option from rmtree 2018-08-15 11:59:09 -03:00
Daniel Graña 4eaf8690b1 Twisted's unittest.Testcase assertRaiess can't be used as context manager 2018-08-15 08:54:18 -03:00
Daniel Grana a304d6b692 Workaround to pass tests/test_feedexporter.py under windows 2018-08-15 02:23:07 -07:00
Daniel Grana e7fe243c3e Fix test_crawler under windows 2018-08-15 01:09:23 -07:00
Daniel Grana 96517cb7de Fix test_command_parse under windows 2018-08-15 01:08:40 -07:00
Daniel Graña d93d960319 Fix test_utils_project under Windows 2018-08-15 01:53:20 -03:00
Jakob de Maeyer cb28175750 Fix csviter tests by explicitly using newline only 2018-08-15 01:35:01 -03:00
Daniel Graña ca53a8699a Fix presentation of template directory in startproject command 2018-08-15 01:35:01 -03:00
Daniel Graña 0e532e3dd8 Creating a connection to 0.0.0.0 fails on windows but not on linux nor mac 2018-08-15 01:35:01 -03:00
Daniel Graña a21abac743 fix ftp tests on windows 2018-08-15 01:35:01 -03:00
Daniel Graña fb09148c91 Fix bad merge on ParseCommandTest 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 22505a34a9 Fix cmdline profiling test on Windows by using proper path composing 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 034152961d Fix Feedexport test in Windows by using proper file URI 2018-08-15 01:35:01 -03:00
Daniel Graña ed8255bde0 Fix merge issues with stderr/out fixes for windows buffering 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 57a1d66c61 Fix test issues caused by Windows pipe buffer filling up 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 152fde70b1 Fix FTPTestCase by using Windows-friendly temporary file name 2018-08-15 01:33:38 -03:00
Daniel Graña 19ad94105f pywin32 is required to run tests under windows 2018-08-15 01:33:38 -03:00
Daniel Graña 4c53957f5b Skip leveldb tests on windows 2018-08-15 01:33:38 -03:00
Stas Glubokiy 76220e8733 Use inspect.getmembers in tested_methods_from_spidercls 2018-08-11 18:49:12 +03:00
Stas Glubokiy 16dad81715 Fix contract errback 2018-08-09 21:07:25 +03:00
Daniel Graña db714f5a07
Merge pull request #3283 from CCInCharge/issue3247 
Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys
 2018-08-02 14:49:14 -03:00
Eugenio Lacuesta 9e14f8c7e4 Fix test for dupefilter 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 0089a4ab31 Add test for direct creation of dupefilter (no from_crawler/from_settings) 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge 782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta 98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin 92b504eae5 Fix telnet warnings in tests 
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
 2018-07-09 13:44:02 +03:00
Konstantin Lopuhin 722e1afcdb Update ancient pytest on python 3 
2.9 gives collection errors on python 3.7 due to PEP 479.
 2018-07-09 12:21:19 +03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Mikhail Korobov f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Daniel Graña 64f48efa26
Merge pull request #3308 from scrapy/fix-build 
TST exclude lxml==4.2.2 from tests
 2018-06-27 23:42:32 -03:00
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Mikhail Korobov fac1b2f351 TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00
Grammy Jiang 9ad3af9d88
Update requirements.txt 
make the version of ipython less than 6.0 in python 2.7
 2018-06-23 17:31:54 +08:00
Grammy Jiang 88bd067912 fix the test case name of HttpProxyMiddleware 2018-06-20 16:56:46 +08:00
Vostretsov Nikita 72d0899bce Return non-zero exit code from scrapy commands in case of spider bootstrap errors 
* method to detect spider creation in crawler

* correct method name

* method to know if crawlers has spiders

* we do not need to issue requests

* set exit code accordingly to spiders in crawlers

* more portable way to check ofr exceptions

* more clear way

* test cases for several spiders per crawler

* grammatically correct name for method

* method is private

* grammatically correct name for method

* method is private

* remove unused import

* correct order of imports

* changes mechanism of obtaining spider status from method to object member

* rename tests
 2018-06-14 19:58:48 +05:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Konstantin Lopuhin bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0 
[MRG+1] Blacklist twisted version with regression
 2018-05-17 13:21:42 +03:00
Vostretsov Nikita c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
rhoboro 5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00