RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

2349 Commits

Author SHA1 Message Date
Victor Torres ea8be627d1 botocore is not supported on debian jessie 2019-02-13 20:32:36 -02:00
Victor Torres dc0b643832 refactoring tests to avoid mocking private method 2019-02-13 19:44:50 -02:00
Adrián Chaves 03e61b9908 Check that spidercls arguments in scrapy.crawler classes are not spider objects 2019-02-12 14:46:27 +01:00
Victor Torres 1eac2a163c simplifying how we deal with threads.deferToThread calls 2019-02-08 16:50:39 -02:00
Victor Torres f824f5b2d1 testing public method store instead of private method _store_in_thread 
need to mock deferToThread function
 2019-02-08 15:51:23 -02:00
Victor Torres 079af889e7 also testing without botocore 2019-02-08 15:51:23 -02:00
Victor Torres 126207fb7b PEP8: use short name for mock method 2019-02-08 15:51:23 -02:00
Victor Torres ad83ffdf1f refactoring 2019-02-08 15:51:23 -02:00
Júlio César Batista 2bbbd02bda Adding an option to set ACL while uploading the blob to GCS 2019-02-08 09:45:10 -02:00
Júlio César Batista a4059851e7 Refactoring tests 2019-01-31 18:29:15 -02:00
Júlio César Batista 5a55c4269d Adding GCSFeedStorage 2019-01-31 17:20:29 -02:00
Daniel Graña 65d631329a Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
Maram Sumanth 722a30ac2b
Update test_http_request.py 2019-01-19 13:20:05 +05:30
Júlio César Batista 8eade7d864 Testing stats and log messages from RFPDupeFilter 2019-01-18 11:39:35 -02:00
Maram Sumanth 6be73f06c3
Updated tests 2019-01-17 23:50:58 +05:30
Maram Sumanth 6f86c93f36
Increased test cases 2019-01-16 23:54:35 +05:30
Maram Sumanth 9f1f4df966
Update test_http_request.py 2019-01-16 22:59:41 +05:30
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
kasun Herath 3f914f6d8c made jsonrequest dump into private method 2019-01-14 23:03:14 +05:30
Maram Sumanth 023290dabc
Update test_http_request.py 2019-01-13 23:50:31 +05:30
Maram Sumanth 4abcdcb306
Update test_http_request.py 2019-01-13 23:22:53 +05:30
Maram Sumanth b5e454809e
Included test 2019-01-13 20:12:31 +05:30
Eugenio Lacuesta 57e7c76977 Test callback kwargs 2019-01-09 10:40:44 -03:00
Eugenio Lacuesta 770a501fb3 Test request kwargs (copy, serialization) 2019-01-09 10:40:03 -03:00
Eugenio Lacuesta 9759112a3a Merge branch 'master' into process_spider_exception_generator 2019-01-03 11:34:07 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
kasun Herath 8f1507a4a5 dumps_kwargs 2018-12-17 23:14:06 +05:30
kasun Herath ecda69130e allow to send empty data values and docs changes 2018-12-10 22:34:49 +05:30
kasun Herath c347acbff6 warning if body and data are provided 2018-12-09 11:27:09 +05:30
kasun Herath cd619c1d4f removed overriden replace method 2018-12-08 22:10:45 +05:30
kasun Herath 1b2b8b4bf0 fix tests under py3 2018-11-27 08:57:44 +05:30
kasun Herath 1ce6662a9d Implement Request subclass for json requests 2018-11-24 20:02:00 +05:30
Eugenio Lacuesta f97e3e90f2 Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
Eugenio Lacuesta e0360e5223 Add tests for MutableChain 2018-10-11 11:55:13 -03:00
Eugenio Lacuesta 2396356a82 Merge branch 'master' into process_spider_exception_generator 2018-10-10 11:37:43 -03:00
Henrique Coura 441e1e750f Style changes 2018-09-26 13:28:34 -03:00
Henrique Coura e57a629efc Generate only password, encode username/password only on login 2018-09-26 11:54:57 -03:00
Henrique Coura 37cfb49805 Randomly generate telnet credentials by default 2018-09-24 16:42:49 -03:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380 
[MRG+1] Use dont_filter=True for contracts requests
 2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Henrique Coura 167211ffb0 Default is lazy, load_object exception handling, code improvements 2018-08-20 15:54:04 -03:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Stas Glubokiy 57824600a8 Use six.get_unbound_function in test_same_url 2018-08-19 16:56:41 +03:00
Stas Glubokiy 0467737cf0 Fix mockserver usage 2018-08-18 15:43:46 +03:00
Stas Glubokiy 1311f6b536 Merge branch 'master' of github.com:StasDeep/scrapy into fix/issue-3380 2018-08-18 15:42:10 +03:00
Stas Glubokiy 11576f5c8f Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-18 15:24:07 +03:00
Henrique Coura 5bac436764 Make lazy loading Download Handlers optional 2018-08-17 15:07:37 -03:00
Vostretsov Nikita afb1458bd3 tests for new signal 2018-08-17 14:39:54 +00:00
Mikhail Korobov c7654f7cb1
Merge pull request #3379 from StasDeep/fix/issue-3378 
Use inspect.getmembers in tested_methods_from_spidercls
 2018-08-17 19:25:41 +05:00
Stas Glubokiy ddd69f4c10 Use MockServer in test_same_url 2018-08-15 20:39:43 +03:00
Stas Glubokiy 2cb4decb6a Move TestSameUrlSpider to test method 2018-08-15 20:36:10 +03:00
Stas Glubokiy 4de493efdd Add test_same_url 2018-08-15 20:24:00 +03:00
Daniel Graña 38608bc249 Use ignore_errors option from rmtree 2018-08-15 11:59:09 -03:00
Daniel Graña 4eaf8690b1 Twisted's unittest.Testcase assertRaiess can't be used as context manager 2018-08-15 08:54:18 -03:00
Daniel Grana a304d6b692 Workaround to pass tests/test_feedexporter.py under windows 2018-08-15 02:23:07 -07:00
Daniel Grana e7fe243c3e Fix test_crawler under windows 2018-08-15 01:09:23 -07:00
Daniel Grana 96517cb7de Fix test_command_parse under windows 2018-08-15 01:08:40 -07:00
Daniel Graña d93d960319 Fix test_utils_project under Windows 2018-08-15 01:53:20 -03:00
Jakob de Maeyer cb28175750 Fix csviter tests by explicitly using newline only 2018-08-15 01:35:01 -03:00
Daniel Graña ca53a8699a Fix presentation of template directory in startproject command 2018-08-15 01:35:01 -03:00
Daniel Graña 0e532e3dd8 Creating a connection to 0.0.0.0 fails on windows but not on linux nor mac 2018-08-15 01:35:01 -03:00
Daniel Graña a21abac743 fix ftp tests on windows 2018-08-15 01:35:01 -03:00
Daniel Graña fb09148c91 Fix bad merge on ParseCommandTest 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 22505a34a9 Fix cmdline profiling test on Windows by using proper path composing 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 034152961d Fix Feedexport test in Windows by using proper file URI 2018-08-15 01:35:01 -03:00
Daniel Graña ed8255bde0 Fix merge issues with stderr/out fixes for windows buffering 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 57a1d66c61 Fix test issues caused by Windows pipe buffer filling up 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 152fde70b1 Fix FTPTestCase by using Windows-friendly temporary file name 2018-08-15 01:33:38 -03:00
Daniel Graña 19ad94105f pywin32 is required to run tests under windows 2018-08-15 01:33:38 -03:00
Daniel Graña 4c53957f5b Skip leveldb tests on windows 2018-08-15 01:33:38 -03:00
Stas Glubokiy b4b1e48343 Add ability to use FormRequest in contracts 2018-08-11 22:18:43 +03:00
Stas Glubokiy 76220e8733 Use inspect.getmembers in tested_methods_from_spidercls 2018-08-11 18:49:12 +03:00
Stas Glubokiy 8bc536d88b Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-11 18:34:37 +03:00
Stas Glubokiy ebbde57eca Add custom contracts tests 2018-08-11 18:21:07 +03:00
Stas Glubokiy 16dad81715 Fix contract errback 2018-08-09 21:07:25 +03:00
Eugenio Lacuesta 40449fa0eb Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
Eugenio Lacuesta 8c55f5eb15 Simplify check for re-raised exception. Add tests. 2018-08-03 15:16:26 -03:00
Daniel Graña db714f5a07
Merge pull request #3283 from CCInCharge/issue3247 
Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys
 2018-08-02 14:49:14 -03:00
Eugenio Lacuesta 801d3c07b4 Fix bad exception handling, add tests 2018-07-27 15:06:37 -03:00
Eugenio Lacuesta d6d3e87e3a Rename test file 2018-07-27 14:47:52 -03:00
Eugenio Lacuesta 9e14f8c7e4 Fix test for dupefilter 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 0089a4ab31 Add test for direct creation of dupefilter (no from_crawler/from_settings) 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge 782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Eugenio Lacuesta 784eed1130 Improve test coverage (downloader middleware) 2018-07-20 19:08:46 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta 98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Eugenio Lacuesta 71a1406c99 Logging changes 2018-07-18 17:43:05 -03:00
Eugenio Lacuesta 6329441c82 ModuleNotFoundError was added in py3.6 2018-07-18 17:31:31 -03:00
Eugenio Lacuesta a3af0bfd56 More tests 2018-07-18 15:15:55 -03:00
Eugenio Lacuesta 56e92d90fd Update tests 2018-07-17 15:15:38 -03:00
Eugenio Lacuesta e96b7782bc Merge branch 'master' into process_spider_exception_generator_experiment 2018-07-15 18:17:37 -03:00
Eugenio Lacuesta b8e8922d54 Simplify stuff. Add more tests. 2018-07-15 18:17:00 -03:00
Eugenio Lacuesta cff9e87625 Fix tests 2018-07-15 16:21:08 -03:00
Eugenio Lacuesta e7e18db179 Fix tests 2018-07-13 16:40:30 -03:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin 92b504eae5 Fix telnet warnings in tests 
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
 2018-07-09 13:44:02 +03:00
Konstantin Lopuhin 722e1afcdb Update ancient pytest on python 3 
2.9 gives collection errors on python 3.7 due to PEP 479.
 2018-07-09 12:21:19 +03:00
Eugenio Lacuesta 0a0e62272e New tests 2018-07-04 16:19:19 -03:00
Eugenio Lacuesta 0b2870634a Do not inherit from object 2018-07-04 16:14:51 -03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Eugenio Lacuesta 4fca9aba85 Recover from a callback exception 2018-07-01 13:45:18 -03:00
Eugenio Lacuesta 6ed9440ed5 Tests for exceptions on spider callbacks 2018-07-01 13:45:18 -03:00
Eugenio Lacuesta 735de8167d Test for exceptions on process_spider_input 2018-07-01 13:45:18 -03:00
Mikhail Korobov f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Daniel Graña 64f48efa26
Merge pull request #3308 from scrapy/fix-build 
TST exclude lxml==4.2.2 from tests
 2018-06-27 23:42:32 -03:00
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Mikhail Korobov fac1b2f351 TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00
Eugenio Lacuesta ab48837f09 Merge branch 'master' 2018-06-24 20:31:37 -03:00
Grammy Jiang 9ad3af9d88
Update requirements.txt 
make the version of ipython less than 6.0 in python 2.7
 2018-06-23 17:31:54 +08:00
Grammy Jiang 88bd067912 fix the test case name of HttpProxyMiddleware 2018-06-20 16:56:46 +08:00
Vostretsov Nikita 72d0899bce Return non-zero exit code from scrapy commands in case of spider bootstrap errors 
* method to detect spider creation in crawler

* correct method name

* method to know if crawlers has spiders

* we do not need to issue requests

* set exit code accordingly to spiders in crawlers

* more portable way to check ofr exceptions

* more clear way

* test cases for several spiders per crawler

* grammatically correct name for method

* method is private

* grammatically correct name for method

* method is private

* remove unused import

* correct order of imports

* changes mechanism of obtaining spider status from method to object member

* rename tests
 2018-06-14 19:58:48 +05:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Konstantin Lopuhin bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0 
[MRG+1] Blacklist twisted version with regression
 2018-05-17 13:21:42 +03:00
Vostretsov Nikita c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
rhoboro 5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00
Daniel Graña 6c3970e672
Merge pull request #3153 from virmht/new_bug 
[MRG+1] Fixed bug FormRequest.from_response() clickdata ignores input[type=image]
 2018-03-21 16:32:12 -03:00
Lucy Wang 1a2f0193a3 fix tests on jessie 2018-03-13 19:14:52 +08:00
Lucy Wang d9e6c73fb3 revert wrong changes 2018-03-13 13:05:37 +08:00
Lucy Wang e487100987 add a test case 2018-03-13 08:59:03 +08:00
Viral Mehta 65744c2199 Corrected Test 2018-03-03 20:07:50 +05:30
Viral Mehta d5b7ebcfdc Fixed bug FormRequest.from_response() clickdata ignores input[type=image] 2018-03-03 18:17:49 +05:30
nctl144 4c05441450 add ftp to the scheme list 2018-03-03 00:00:03 -05:00
NewUserHa acd2b8d43b [MRG+1] Fix part of issue #3128 - None should not be a valid type for 'url' in Response.follow (#3131) 
* fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'

*  fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'
 2018-02-22 03:37:26 +05:00
Eugenio Lacuesta e4558cb27e Update test for unpickable objects 2018-02-08 15:03:57 -03:00
Eugenio Lacuesta a56540877c Do not serialize unpickable objects (py3) 2018-02-08 15:03:57 -03:00
Mikhail Korobov ba15b63ed6 TST fix tests to account for changes in w3lib 1.19 2018-01-26 02:11:49 +05:00
Konstantin Lopuhin 041308afe7 Fix get_func_args test for pypy3 
These built-in functions are exposed as methods in PyPy3.
For scrapy this does not matter as:
1) they do not work for CPython at all
2) get_func_args is checked for presense of an argument in scrapy,
   extra "self" does not matter.
But it still makes sense to leave these tests so that we know we
shouldn't use get_func_args for built-in functions/methods.
 2017-12-25 14:48:10 +03:00
Konstantin Lopuhin a21b800419
Merge pull request #3011 from Jane222/master 
[MRG+1] Issues a warning when user puts a URL into allowed_domains (#2250)
 2017-12-08 15:45:16 +03:00
Daniel Graña 3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command 
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
 2017-11-29 16:26:48 -03:00
Jana Cavojska 454d5e5733 checking for subclass of URLWarning instead of checking error message text when URL in allowed_domains 2017-11-26 20:07:04 +01:00
Jana Cavojska 8ec3b476b0 triggering a warning when user puts URL in allowed_domains now covered by test 2017-11-26 16:36:15 +01:00
IAlwaysBeCoding 846fd83512 removed commented out code, wrapped line to pep-8 and removed backlashes 2017-11-11 18:30:01 -05:00
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Daniel Graña 5fac2d7b90 Merge pull request #2923 from rhoboro/fixes-685 
[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage.
 2017-10-02 13:52:20 -03:00
Lucas Moauro 59c3f6f095 Fix typos in tests 2017-10-01 12:24:56 -03:00
Paul Tremberth e733f51d4b Fix test 2017-09-25 12:49:27 +02:00
rhoboro e5d4364b2a Add tests for GCS Storage 2017-09-13 16:24:16 +09:00
Mikhail Korobov 1ba77f0837 Merge pull request #2869 from cclauss/patch-3 
# noqa to close #2836
 2017-08-21 19:15:30 +05:00
Mikhail Korobov 885289f497 Merge pull request #2854 from jenya/2853-sitemap-follow-alternate-fix 
[MRG+1] Follow alternate link for all types of sitemaps #2853
 2017-08-21 19:15:03 +05:00
Daniel Graña d5f3543d77 Merge pull request #2865 from kirankoduru/2831-explicit-msg-for-scrapy-parse-callback 
[MRG+1] Explicit message for scrapy parse callback
 2017-08-09 07:56:57 -03:00
Chomba Ng'ang'a fd27cde24d Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Kiran Koduru 12409a0cf6 Fix broken encoding on text for py 3 2017-08-05 15:40:38 -04:00
kirankoduru 7adab61a7a Added test for NotImplemented Spider.parse method 2017-08-05 15:40:38 -04:00
cclauss 0a69a32b5a Force Travis CI to test again 2017-08-04 14:35:43 +02:00
cclauss c016a4309d # noqa to close #2836 
Marks #2836 as will not fix.
 2017-08-04 01:44:23 +02:00
Paul Tremberth 0cb3085f84 Add test for alternate links 2017-08-03 16:55:09 +02:00
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Jakob de Maeyer 4d77c3084e Add from_crawler constructor to S3FeedStorage 2017-07-25 09:41:51 +02:00
Jakob de Maeyer 815d6160cf Add from_crawler constructor for feed exporters and storages 2017-07-25 09:41:29 +02:00
Daniel Graña 2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña 45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Paul Tremberth 1f08d9a648 Add test for DNS cache disabling 2017-07-04 23:10:19 +02:00
Konstantin Lopuhin 7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin 5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin 6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00
Pengyu CHEN b33e0d5a54 Added: Now supporting <link> tags in Response.follow 2017-06-14 17:09:42 +08:00
Valdir Stumm Junior 4b6f68b9ee make reqser tests create Request with proper callback/errback 2017-06-05 17:26:52 -03:00
Valdir Stumm Junior 60727dedf6 verify if Request callback is callable 2017-05-31 15:00:38 -03:00
Daniel Graña 25f609e2a3 Merge pull request #2675 from simongartz/png-p-to-jpg-conversion-fix 
[MRG+1] Fixes conversion of transparent PNG with palette images to jpg #2452
 2017-05-18 16:34:02 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Harrison Gregg ffef828a8d Add test for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Eugenio Lacuesta 63b8caf5de Feed exports: rewrite indentation test without .strip() 2017-05-09 11:58:53 -03:00
Daniel Graña 362d6f2d20 Merge pull request #2622 from rolando-contrib/download-maxsize-abort 
[MRG+1] Abort connection earlier and avoid to buffer data when max size limit is reached
 2017-05-04 11:44:51 -03:00
Paul Tremberth 6d14e392f1 Remove old test certificate+key 2017-04-27 23:35:01 +02:00
Paul Tremberth e6ab8bc9a5 Change "localhost" test server certificate 2017-04-27 23:22:25 +02:00
Rolando Espinoza 2b34c6edff Abort connection earlier and avoid to buffer data 
A symptom of this issue was having the log message "Received (X) bytes
larger than download max size (Y)" several times printed, with increased
X values.
 2017-04-27 13:28:16 -03:00
Paul Tremberth c3d0f9b6c1 Add test for non-duplicated `Content-Length: 0` for bodyless POST 2017-04-25 17:03:41 +02:00
Paul Tremberth b1a0a6e258 Make mockserver runnable outside of tox 
Add POST support for Echo resource
 2017-04-25 17:01:54 +02:00
harshasrinivas 38e6857c95 Improvise the clarity of test cases 2017-03-23 19:45:04 +05:30
Simon Diviani Gartz 21d794d35a Fixes conversion of transparent PNG with palette images to jpg #2452 2017-03-22 13:29:15 +01:00
Mikhail Korobov b040df5ac0 TST cleanup spider middleware tests 2017-03-21 15:56:18 +05:00
Paul Tremberth 776129a951 Merge pull request #2649 from pawelmhm/logformatter-2647 
[MRG+2] [logformatter] 'flags' format spec backward compatibility
 2017-03-21 10:51:31 +01:00
harshasrinivas 0d9ebd6e1e Update tests for max_retry_times 2017-03-19 06:15:46 +05:30
Pawel Miech 4345eaf1b6 [logformatter] backward compat comments 2017-03-17 08:11:20 +01:00
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
woxcab a84652e775 Init tests are split by initializer' input 2017-03-15 12:39:48 +03:00
harshasrinivas e321ac9931 Update unittests for max_retry_times 2017-03-15 04:12:32 +05:30
harshasrinivas 966bd49c42 Update unittest for meta['max_retry_times'] 2017-03-14 16:23:47 +05:30
Pawel Miech 0f2a5cdb8e [logformatter] 'flags' format spec backward compatibility 
pass 'flags' kwarg to logger so that it is compatible with old
format of CRAWLEDMSG.
 2017-03-13 15:16:38 +01:00
woxcab fbb411a805 Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer 2017-03-13 14:16:39 +03:00
Paul Tremberth 871134ee22 Refactor to also test FilesPipeline 2017-03-12 17:30:24 +01:00
Paul Tremberth 708f1b009b Add integration tests for MEDIA_ALLOW_REDIRECTS 2017-03-10 21:36:33 +01:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Eugenio Lacuesta 9c256cf693 Undocument _InvalidOutput exception 2017-03-10 15:41:57 -03:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth c0cbaccb7b Merge pull request #2581 from lopuhin/respect-custom-log-level 
[MRG+1] Respect custom log level: fixes GH-1612
 2017-03-09 12:47:20 +01:00
Paul Tremberth 9cfe9ae098 Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
Eugenio Lacuesta 4090cc3990 Spider middleware: use Mockserver to test process_spider_exception 2017-03-08 18:11:20 -03:00
Paul Tremberth 7be773e14a Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
Mikhail Korobov d3f8f3d38a Merge pull request #2612 from redapple/dupe-spider-name-tests 
[WIP] Add warning on duplicate spider name
 2017-03-07 20:08:42 +05:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 0b9a18e1a1 Warn only once for all spiders 2017-03-07 15:41:17 +01:00
Paul Tremberth 978306a223 Fix dupe spider name warning string tests 2017-03-07 14:48:16 +01:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Konstantin Lopuhin 6f55ca4643 Revert unneeded test_crawl changes 2017-03-07 14:20:52 +03:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Paul Tremberth 768f3155e5 Fix referrer policy from response headers and support explicit empty string 2017-03-06 16:20:37 +01:00
Paul Tremberth 0a17f9b55c Merge pull request #2334 from ArturGaspar/data_uri 
[MRG+1] data URI download handler.
 2017-03-06 13:50:46 +01:00
Paul Tremberth c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Konstantin Lopuhin ef04cfd237 Respect log settings in custom_settings: fixes GH-1612 
A new root logger is installed when a crawler is created
if one was already installed before.
This allows to respect custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
 2017-03-03 18:01:11 +03:00
Paul Tremberth f7e11b198e Cleanup 2017-03-03 16:00:59 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Bernardas 11b31c9fbd fix redirect change 2017-03-03 15:52:05 +01:00
Bernardas 3cef1cd451 adjust variable wording and redirect logic 2017-03-03 15:52:05 +01:00
Bernardas 6a42214716 add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Artur Gaspar b50d0370f4 Test response attributes in data URI download handler. 2017-03-02 14:46:33 -03:00
Paul Tremberth 12a8ddecab Fix tests 2017-03-02 13:03:18 +01:00
Paul Tremberth e71803c833 Add tests for duplicate spider name warnings 2017-03-02 12:48:47 +01:00
Paul Tremberth 2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth 8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth 0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth 8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth 0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth 7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Eugenio Lacuesta 706ed0e049 Spider middleware: process_spider_exception on generators 2017-03-01 12:01:53 -03:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Mikhail Korobov 0e5ed21397 Merge pull request #2599 from redapple/py3-ignores-cleanup-ftp 
Fix FTP downloader and re-enable FTP tests on Python 3
 2017-02-28 15:39:42 +05:00
Paul Tremberth 0b90c3b43c Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
Mikhail Korobov c72ba07326 Merge pull request #2543 from scrapy/retry-stats 
[MRG+1] Retry stats
 2017-02-27 21:03:35 +05:00
Artur Gaspar 121a668a47 Rename data URI downloader module. 2017-02-22 12:30:56 -03:00
Artur Gaspar 3397d27574 Test for binary body content from data URI downloader. 2017-02-22 12:30:56 -03:00
Artur Gaspar d60642e175 data URI download handler. 2017-02-22 12:30:56 -03:00
Mikhail Korobov 26c1256f95 HttpErrorMiddleware stats 2017-02-22 17:36:25 +05:00
Mikhail Korobov 2b4d46315f TST fixed compatibility with new link extractor whitespace handling 2017-02-21 00:05:40 +05:00
Mikhail Korobov df446d167f fix deprecated link extractors 2017-02-20 23:01:14 +05:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña b15b4541d1 Merge pull request #2569 from scrapy/fix-project-util-test-osx 
TST fixed ProjectUtilsTest on OS X
 2017-02-20 11:15:21 -03:00
Daniel Graña 322fd68e4c Merge pull request #2548 from scrapy/formrequest-whitespaces 
[MRG+1] FormRequest: handle whitespaces in action attribute properly
 2017-02-20 11:09:50 -03:00
Daniel Graña 4a93be4ad8 Merge pull request #2547 from scrapy/linkextractor-strip-whitespaces 
[MRG+1] LinkExtractors: strip whitespaces
 2017-02-20 11:08:32 -03:00
Daniel Graña 58a18e3007 Merge pull request #2535 from pawelmhm/brotli 
[MRG+1] [httpcompression] add support for br - brotli content encoding
 2017-02-20 11:04:55 -03:00
Paul Tremberth f2e200120b Explicitly remove test directories in FTP tests 2017-02-20 14:50:05 +01:00
Paul Tremberth 26b7c039a9 Remove debug print statement 2017-02-20 14:19:36 +01:00
Paul Tremberth 3ffa2c5759 Support Anonymous FTP 
Fixes GH-2342
 2017-02-20 14:19:36 +01:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Mikhail Korobov 78cb46a079 TST fix a weird OS X testing issue 
pytest intercepts temp file creation to provide readable file/folder
names; path is built from method name; in case of conflicts
pytests uses increasing numbers, but it seems it doesn’t account
for case-insensitive (but case preserving) OS X filesystem. There
are methods named test_encoding, pytest thinks test_Encoding is
different and fails to create a test folder
 2017-02-16 18:22:04 +05:00
Mikhail Korobov b9928558b2 TST remove temporary files and folders 2017-02-16 17:50:38 +05:00
Mikhail Korobov f0c4e5dfd5 TST fixed ProjectUtilsTest on OS X 
Temp folder can be a symlink on OS X.
 2017-02-16 16:32:23 +05:00
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov 074caf434e FormRequest: handle whitespaces in action attribute properly 2017-02-16 02:13:39 +05:00
Mikhail Korobov fade5763af TST more response.follow tests 2017-02-16 02:02:50 +05:00
Mikhail Korobov 160da6abab fixed tests in Python 2 2017-02-15 04:41:53 +05:00
Mikhail Korobov 2674f317df Response.follow 2017-02-15 04:39:47 +05:00
Mikhail Korobov 608c3f0c45 handle whitespace in response.follow; add tests 2017-02-15 04:17:41 +05:00
Mikhail Korobov e285b1d6c2 retry stats 2017-02-14 20:27:49 +05:00
Mikhail Korobov afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
Paul Tremberth f32a229e3d Merge pull request #2510 from elacuesta/reqser_request_class 
[MRG+1] Preserve request class when converting to/from dicts (utils.reqser)
 2017-02-08 18:30:07 +01:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth a8893190a1 Merge pull request #2464 from elacuesta/component_order_integer 
[MRG+2] Validate numeric values (components order)
 2017-02-08 13:01:02 +01:00
Paul Tremberth 3becb861af Merge pull request #2544 from scrapy/twisted-version-cleanup 
[MRG+1] Remove code required to support ancient twisted versions.
 2017-02-08 12:59:45 +01:00
Rolando Espinoza 4e765acaed BUG: Fix __classcell__ propagation. 
Python 3.6 added simpler customization of class creation but this
requires to propagate correctly the __classcell__ attribute in custom
__new__ methods.

See https://docs.python.org/3.6/whatsnew/3.6.html#pep-487-simpler-
customization-of-class-creation
 2017-02-08 02:12:52 -04:00
Mikhail Korobov 04b2f79e7a Remove code required to support ancient twisted versions. See GH-1887. 2017-02-07 22:30:58 +05:00
Eugenio Lacuesta 24e82bfe75 Validate values for components order 2017-02-07 09:13:57 -03:00
Pawel Miech fb4ef21a1d [httpcompression] minor style edits 2017-02-07 10:22:42 +01:00
Pawel Miech af802bad14 [httpcompression] add brotlipy for python 3 2017-02-06 15:45:21 +01:00
Pawel Miech 3daf473686 [httpcompression] skip test if no brotli 2017-02-06 13:49:41 +01:00
Pawel Miech f8f8bbe080 [httpcompression] import brotli when available 2017-02-06 12:19:00 +01:00
Pawel Miech 16c4b4e184 [httpcompression] add support for br - brotli content encoding 2017-02-06 11:41:08 +01:00
Mikhail Korobov 814ce37dff Merge pull request #2528 from redapple/dns-retry-test 
Make DNS retry test compatible with Twisted 17+
 2017-02-03 02:43:46 +05:00
Paul Tremberth 02e1d2b1fd Add trailing dot 2017-02-02 22:28:37 +01:00
Paul Tremberth e604c0f3ab Remove unused imports 2017-02-02 18:26:41 +01:00
Paul Tremberth 3358254c5c Make DNS retry test compatible with Twisted 17+ 2017-02-02 17:53:28 +01:00
Paul Tremberth 803d8c4b57 Add tests for passing kwargs on response .xpath() shortcut 2017-02-02 17:44:57 +01:00
Paul Tremberth ae6d8d728e Support 'True' and 'False' strings as boolean settings values 2017-01-30 16:33:08 +01:00
Eugenio Lacuesta 53757e51e5 Preserve request class when converting to/from dicts (utils.reqser) 2017-01-24 17:13:21 -03:00
Paul Tremberth 07f9985a94 TST: Randomize FILES_EXPIRES above 90 days 2016-12-21 17:03:11 +01:00
Elias Dorneles d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Mikhail Korobov d19c4c1f80 Merge pull request #2433 from redapple/wrong-spidermodules-warning 
[MRG+1] Warn user instead of failing for wrong SPIDER_MODULES setting
 2016-12-19 21:46:56 +05:00
Paul Tremberth ed1e4d8df3 Merge pull request #1731 from scrapy/disable-toplevel-2 
[MRG+1] LOG_SHORT_NAMES option to disable TopLevelFormatter
 2016-12-19 16:02:55 +01:00
Elias Dorneles 97e82107b1 Merge pull request #2270 from redapple/httperror-log-info 
[MRG+1] Raise log level for HttpErrorMiddleware to INFO (from DEBUG)
 2016-12-14 12:18:04 -02:00
Paul Tremberth f7e4081414 Add tests for SequenceExclude container 2016-12-12 22:37:53 +01:00
Mikhail Korobov 05b4555f39 TST tests for LOG_SHORT_NAMES 2016-12-09 02:19:51 +05:00
Mikhail Korobov e46572d6f2 TST end-to-end test for LOG_LEVEL option 
there were no end-to-end tests for this option
 2016-12-09 02:19:33 +05:00
Mikhail Korobov 6eab59cbac TST cleanup runspider tests 2016-12-09 02:14:12 +05:00
Paul Tremberth 948e3cd003 Warn user instead of failing for wrong SPIDER_MODULES setting 2016-12-08 12:50:26 +01:00
Paul Tremberth 2cd579a774 Add test for fetch(url) within shell with and without redirect 2016-12-07 19:07:32 +01:00
Paul Tremberth 7e54de2455 Add tests for shell command with and without --no-redirect 2016-12-07 18:41:24 +01:00
Elias Dorneles cce631abec Merge pull request #1887 from nyov/twisted11 
[MRG+2] Bump Twisted dependency to 13.1.0
 2016-12-07 15:00:01 -02:00
Paul Tremberth 778bed07bf Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00
Mikhail Korobov ff3aec6613 Merge pull request #2331 from moisesguimaraes/fixes-2272 
[MRG+1] Fixes issue #2272 using arg_to_iter() to wrap single values and list() to…
 2016-12-07 20:08:18 +05:00
Elias Dorneles a9c69458ff Merge pull request #2422 from rolando-contrib/nested-spiders-modules 
[MRG+1] DOC State explicitly that spiders are loaded recursively.
 2016-12-07 11:21:15 -02:00
Paul Tremberth 5efd65255c TST: Randomize IMAGES_EXPIRES above 90 days 2016-12-06 18:49:53 +01:00
nyov 534772f6ea Import xlib.tx code from twisted proper 2016-12-02 21:21:51 +00:00
Elias Dorneles c4e67c0696 Merge pull request #2421 from rolando-contrib/tests-bug 
[MRG+1] TST: Fix duplicated test name.
 2016-12-01 18:43:48 -02:00
Elias Dorneles f3a4420750 Merge pull request #2388 from redapple/robotparser-native-str 
[MRG+1] Parse robots.txt content as native str
 2016-12-01 18:43:23 -02:00
Rolando Espinoza 923b974f0a TST Include nested a nested spider in spider loader test. 2016-12-01 13:26:19 -03:00
Rolando Espinoza d9f43e21ba TST: Fix duplicated test name. 2016-12-01 11:56:33 -03:00
Eugenio Lacuesta 5ff64ad015 handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
Paul Tremberth 9aefc0a886 Add test for fetch command with redirections disabled 2016-11-24 13:41:51 +01:00
Paul Tremberth 01142e2ae5 Print more dependencies versions in "scrapy version" verbose output 2016-11-22 14:48:33 +01:00
Paul Tremberth 6cd35c77da Pass user-agent as native str when checking URLs against robots.txt 2016-11-15 17:38:32 +01:00
Paul Tremberth de89b1b562 Merge pull request #2275 from scrapy/response-css-xpath-message 
[MRG+1] Add better messages for when response content isn't text (closes #2264)
 2016-11-10 11:38:22 +01:00
Paul Tremberth 28155dfccc Parse robots.txt content as native str 
Fixes #2373
 2016-11-09 12:20:06 +01:00
Paul Tremberth af2280e695 Update docstring 2016-11-08 13:30:51 +01:00
Paul Tremberth 27456996a9 Add assertion on crawler not running 2016-11-08 11:46:16 +01:00
Paul Tremberth 61efacdd1f Add testcase for catching exception from open_spider() from pipeline 2016-11-08 11:35:42 +01:00
Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 
* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import
 2016-10-19 23:26:12 -03:00
Moisés Guimarães 45e95b79ce (fixes #2272) using arg_to_iter() to wrap single values and list() to avoid consuming from generators. 2016-10-18 11:06:55 -03:00
Elias Dorneles 2d932c173c test abs path outside project as well 2016-09-30 15:07:58 -03:00
Elias Dorneles 25bd3b3fea add .scrapy when outside spider too, add tests 2016-09-29 18:30:42 -03:00
Elias Dorneles 9c9690c76c add better messages for when response content isn't text (closes #2264) 2016-09-21 10:30:35 -03:00
Paul Tremberth 81a0e3cd93 Raise log level for HttpErrorMiddleware to INFO (from DEBUG) 
Fixes GH-910
 2016-09-20 13:44:21 +02:00
Paul Tremberth 41cd9f401f Merge pull request #2243 from pawelmhm/image-pipeline-2198 
[MRG+1] [image & file pipeline] loading setting for user classes
 2016-09-19 18:43:52 +02:00
Mikhail Korobov 5657f6b8ef Merge pull request #2258 from redapple/feed-export-started 
[MRG+1] Feed exporter: start exporting only on first item
 2016-09-19 14:40:30 +06:00
Mikhail Korobov 552368727a Merge pull request #2225 from Tethik/parse_command_rules_fix 
[MRG+1] Two small fixes for when using the parse command and the '-r' flag (rules).
 2016-09-19 14:39:09 +06:00
Joakim Uddholm 8c38dde4e8 Moved parse command tests to its own file. Added some checks to check for logged errors. 2016-09-19 05:33:05 +02:00
Paul Tremberth 03ab077249 Feed exporter: start exporting only on first item 
Fixes GH-872
 2016-09-17 01:36:56 +02:00
Paul Tremberth b828facff4 Add shell test for using scrapy.Request() directly without importing scrapy 2016-09-15 19:25:20 +02:00
pawelmhm 7d88209543 [image & file pipeline] loading setting for user classes 
if user has some custom subclass of Image pipeline and no setting for
this pipeline, he should get default settings defined for Image Pipeline.

Fixes #2198
 2016-09-15 09:39:16 +02:00
Elias Dorneles 129421c7e3 Merge pull request #1503 from demelziraptor/amazon-json-response 
[MRG+1] interpreting json-amazonui-streaming as TextResponse
 2016-09-12 13:21:16 -03:00
Paul Tremberth fbb5559299 Add tests for crawl command non-default cases 2016-09-12 13:35:14 +02:00
Paul Tremberth 9de6f1ca75 Merge pull request #1905 from rootAvish/duplication-fix 
[MRG+1] Modified read failure recovery in utils/gz.py to read only the last f.extrasize bytes of f.extrabuf[ ]
 2016-08-17 14:51:30 +02:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Paul Tremberth 9a734e6759 Merge pull request #2058 from dalleng/serialize_set 
[MRG+1] Add set serialization to ScrapyJSONEncoder
 2016-08-12 18:28:34 +02:00
rootavish d9437fd3d9 Modifying existing gzip read failure recovery mechanism to patch read for broken archives 2016-08-11 18:21:42 +05:30
Mikhail Korobov 414857a593 Merge pull request #2140 from jesuslosada/images-expires 
[MRG+1] Fix IMAGES_EXPIRES default value
 2016-08-05 21:52:27 -04:00
Mikhail Korobov 2c9a38d1f5 Merge pull request #2153 from Digenis/Selector_bad_args 
[MRG+1] Selector should not receive both response and text
 2016-07-31 21:28:38 -04:00
Νικόλαος-Διγενής Καραγιάννης 643dbeffcf Selector should not receive both response and text 2016-07-30 10:35:16 +03:00
Diego Allen e17fdd7276 Add set serialization to ScrapyJSONEncoder 2016-07-22 17:20:03 -04:00
Jesús Losada 7c3e3b484e Fix ImagesPipeline test settings 2016-07-22 20:03:49 +02:00
Paul Tremberth ec1c61504a Merge pull request #2005 from feliperuhland/master 
[MRG+1] Included new optional parameter in startproject command line
 2016-07-19 12:31:06 +02:00
Mikhail Korobov 79639d0fec Merge pull request #1989 from pawelmhm/fix-images-pipeline-uppercase-other 
[MRG+1] [image_pipeline] bring back uppercase class attributes
 2016-07-13 14:44:00 +00:00
Mikhail Korobov 2dd1a9e3bc Merge pull request #2094 from redapple/dns-invalid-id 
Catch and ignore certification verification exception for IP-address hosts
 2016-07-13 10:48:08 +00:00
Paul Tremberth c3109daa72 Merge pull request #2034 from dracony/master 
[MRG+1] Added option to turn off ensure_ascii for JSON exporters
 2016-07-12 17:01:09 +02:00
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Paul Tremberth 778f1cf84c Merge remote-tracking branch 'origin/master' into octet-stream-no-decompress 2016-07-08 18:13:20 +02:00
Elias Dorneles d43a35735a Merge pull request #2050 from Tethik/is_gzipped_fix 
[MRG+1] Is_gzipped for application/x-gzip;charset=utf-8
 2016-07-08 08:47:56 -03:00
Mikhail Korobov b7553d921a Merge pull request #2038 from redapple/canonicalize-idna-failures 
[MRG] Do not fail on canonicalizing URLs with wrong netlocs
 2016-07-08 10:47:54 +06:00
Mikhail Korobov 52a52e2388 Merge pull request #2001 from matveinazaruk/issue-2000 
[MRG+1] Fixed choosing of response class.
 2016-07-08 10:44:24 +06:00
Valdir Stumm Junior 1779f5feca enable genspider command outside projects 2016-07-06 15:10:48 -03:00
Mikhail Korobov 759a555d28 Merge pull request #2069 from redapple/https-connect-host 
[MRG] Add "Host" header in CONNECT requests to HTTPS proxies
 2016-07-06 21:43:41 +06:00
Mikhail Korobov 4273734744 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
Paul Tremberth 37efdde3e3 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-06 14:20:13 +02:00
Paul Tremberth 6539277f99 Fix CONNECT request timeout (with an ugly hack) 2016-06-21 17:14:41 +02:00
Paul Tremberth 10a2c46e12 [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-06-20 16:37:00 +02:00
Pawel Miech fa4d0cdfe5 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-06-20 12:53:20 +02:00
Pawel Miech 539d34bce0 [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-06-15 15:39:11 +02:00
Pawel Miech acbfdc6184 [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-06-15 15:14:28 +02:00
Pawel Miech c6d1686d98 [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-06-15 14:48:25 +02:00
Pawel Miech 72e4d5f33e [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-06-15 14:07:17 +02:00
Pawel Miech ee39d11e45 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-06-15 11:25:38 +02:00
Joakim Uddholm 23f99e98c4 is_gzipped: Separated tests again. 2016-06-14 21:33:51 +02:00
Pawel Miech d715172528 [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-06-14 19:09:56 +02:00
Joakim Uddholm 124e218a3b Added new testcases suggested by @redapple. 2016-06-14 14:22:18 +02:00
Matvei Nazaruk b76b374648 Added test for http11 choosing response type without content-type header. 2016-06-13 23:21:38 +03:00
Joakim Uddholm 2c98a88a0e Separated tests based on case 2016-06-12 10:49:34 +02:00
Joakim Uddholm 989f6b8843 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-06-12 01:38:27 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Paul Tremberth 1aec5200bc Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-06-08 16:49:33 +02:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
Felipe Ruhland 9ad54b3815
Fix template description after create project 2016-05-24 13:03:33 -03:00
Felipe Ruhland b8a09d7ab7
Added tests for more or less parameters 2016-05-24 11:58:52 -03:00
Felipe Ruhland 2521f031d6
Created new tests for implementation 2016-05-23 23:15:53 -03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Pawel Miech 4cef1a1d00 [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-05-13 12:35:35 +02:00
Paul Tremberth 0e11b3e6f0 Add idempotence tests for canonicalize_url 2016-04-26 20:03:17 +02:00
Paul Tremberth 68dedf54cb Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-21 14:50:59 +02:00
Mikhail Korobov 73a5571c60 Merge pull request #1923 from scrapy/request-new-safe_url_string 
[MRG+1] Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests
 2016-04-20 19:33:45 +06:00
Paul Tremberth cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Paul Tremberth d42a98d3b5 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
Paul Tremberth 7b5243a263 Add link extractor test for non-ASCII characters in query part of URL 2016-04-09 15:15:01 +02:00
Paul Tremberth 1656fbcffa Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-08 23:25:50 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
djunzu 8228a0c491 Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-03-31 19:20:39 -03:00
djunzu e9d48f8a8e Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-03-31 19:19:49 -03:00
Paul Tremberth 3ba5671fbc Merge pull request #1851 from nyov/binary_or_text 
[MRG+1] Rename isbinarytext function to binary_is_text for clarity
 2016-03-31 11:55:09 +02:00
nyov e8ca467572 Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-30 15:44:15 +00:00
pawelmhm 65c7c05060 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-12 14:16:40 +01:00
Paul Tremberth ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Daniel Graña 513ba7a1fb Merge pull request #1800 from redapple/http11-post-content-length 
[MRG+1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-23 15:00:33 -03:00
Elias Dorneles 10bcdb49b0 Merge pull request #1787 from scrapy/improve-errors 
[MRG+1] Better tracebacks
 2016-02-23 13:16:47 -03:00
Paul Tremberth 6174192564 Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-20 23:27:04 +01:00
Paul Tremberth da36b7d386 Merge pull request #1761 from lopuhin/py3-s3-botocore 
[MRG+1] Py3 S3 botocore
 2016-02-18 15:11:03 +01:00
Paul Tremberth 104027d78d Minor change on quotes 
Trying to force Travis CI to build
 2016-02-18 11:45:03 +01:00
Konstantin Lopuhin d61fbcc8b5 Support headers in S3FilesStore.persist_file for botocore 2016-02-18 10:57:02 +03:00
Mikhail Korobov f766dd0ba8 Preserve tracebacks better. Fixes GH-1760. 2016-02-17 23:07:03 +05:00
Mikhail Korobov 06da7af9e2 TST clean up RunSpiderCommandTest 2016-02-17 23:03:12 +05:00
Paul Tremberth cabed6f183 More liberal Content-Disposition header parsing 
Fixes #1782
 2016-02-17 16:55:28 +01:00
Konstantin Lopuhin d1ecb8cd38 Fix S3TestCase for precise env: we reraise TypeError as NotConfigured in this case 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 77ebb13684 fix assertRaises for precise env 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 08bc41cc68 py3: reviewed s3 downloader handlers 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 3cb7a567ea py3 fix for TestS3FilesStore: checksum is a native string 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin cfc567f48e botocore support for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 32cd8c9165 add direct test for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin d1470e85a2 S3FeedStorageTest: pass on py3, add some non-ascii content to be sure 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 3ada45a9bb S3FeedStorageTest: add botocore support, and organize boto/botocore checks 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 5d2f067458 S3FeedStorageTest: delete key after test 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 19b2910ad1 Fix assert_aws_environ: check for botocore with boto fallback on PY2 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin bcb92b50dc check that no extra kwargs are silently discarded 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin d6bea3bf2e botocore not only does not allow passing our own Date header, but does not handle x-amz-date according to the spec 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 7748ee6bba mock date in s3 tests when using botocore 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin c3fec83e7e use botocore by default, boto is still used in "precise" env 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 467553cc29 fix anon test: in this case we do no signing, just change the url 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin eaf3a239e4 using botocore for s3 request signing: proof of concept 2016-02-15 19:59:46 +03:00
Paul Tremberth 41588397c0 Merge pull request #1765 from scrapy/add-deprecation-for-pydispatch 
[MRG+1] Add fallback and deprecation warning for pydispatch (fixes #1762)
 2016-02-11 19:29:53 +01:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Paul Tremberth c083935806 Merge pull request #1771 from orangain/secure-cookies 
[MRG+1] PY3: Implement some attributes of WrappedRequest required in Python 3
 2016-02-08 11:52:00 +01:00
orangain 1f743996ff PY3: Implement some attributes of WrappedRequest required in Python 3 
This will fix #1770.
 2016-02-07 14:19:27 +09:00
orangain 25c56159b8 Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-06 23:54:07 +09:00
Nicolas Pennequin 061c63592a MailSender.send: allow passing a charset. 
Resolves Issue #348
 2016-02-04 19:33:44 +01:00
Elias Dorneles a8a6f050e7 Merge pull request #1735 from ArturGaspar/master 
[MRG+1] Fix for KeyError in robots.txt middleware
 2016-02-03 13:30:06 -02:00
Mikhail Korobov 43a53aca12 Merge pull request #1746 from redapple/shell-settings-logging 
[MRG+1] Remove __str__ and __repr__ from settings, introduce copy_to_dict()
 2016-02-03 19:59:24 +05:00
stummjr bb2cf7c0d7 Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
Paul Tremberth d843a0aae8 Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
Paul Tremberth aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Mikhail Korobov a35aec71e9 Merge pull request #1710 from redapple/1550-shell_file-cont 
[MRG+1] shell command's ability to open local files + tests
 2016-01-29 21:03:40 +05:00
Paul Tremberth 78f00401cd Remove unused import in tests 2016-01-29 16:56:05 +01:00
stummjr cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 481e251775 Move guess_scheme() tests to relevant test module 2016-01-28 13:51:50 +01:00
Paul Tremberth cae268402d Move guess_scheme() to scrapy.utils.url 2016-01-28 13:42:04 +01:00
stummjr c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Artur Gaspar dc8701ea42 Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Daniel Graña a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles 2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles 23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Elias Dorneles 0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Konstantin Lopuhin 1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin 4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin 097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin 333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Konstantin Lopuhin 860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin 9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles 9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00
Paul Tremberth 5f09da60c1 Revert "Use pytest.mark.parametrize decorator" 
This reverts commit 1a30a7774b.
 2016-01-22 23:48:58 +01:00
Paul Tremberth 1a30a7774b Use pytest.mark.parametrize decorator 2016-01-22 18:22:19 +01:00
Paul Tremberth 61ca8d587a Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-22 18:16:36 +01:00
Paul Tremberth 7a51d370f3 Regex-based guess_scheme() + refactor tests 2016-01-22 17:16:27 +01:00
Elias Dorneles d0955fd083 add back test for latin-1 encoding 2016-01-22 10:07:55 -02:00
Paul Tremberth 6d73e057b5 Extract guess_scheme function and refactor tests 2016-01-22 13:07:42 +01:00
Mikhail Korobov 35ada10729 PY3 enable tests for scrapy parse command 
scrapy parse command is already ported
 2016-01-22 13:39:27 +05:00
Elias Dorneles e938752973 add test for PythonItemExporter binary mode 2016-01-21 21:51:59 -02:00
Elias Dorneles 2514973242 re-enable skipped feed export tests 2016-01-21 21:22:12 -02:00
Elias Dorneles b746d85f4c PY3 port csv exporter 2016-01-21 21:12:43 -02:00
Paul Tremberth 8bd5b60889 Remove relpath filepath 2016-01-21 23:23:50 +01:00
Paul Tremberth 240ecbf323 Add local file tests for scrapy shell command 
Continuation of #1579
 2016-01-21 22:59:48 +01:00
Elias Dorneles c76190d491 PY3: ported json(lines), xml exporters 2016-01-21 18:24:06 -02:00
Elias Dorneles b6ef1f19fd make BaseItemExporter export unicode, pushed down previous behavior for classes that need it 2016-01-21 15:36:14 -02:00
Elias Dorneles a76ecd4ef0 remove test_exporters from py3 ignores 2016-01-21 15:36:14 -02:00
Konstantin Lopuhin 5813de8838 py3: unskip test_closespider - it passes after fixing mockserver.Follow resouce on py3 2016-01-21 17:53:08 +03:00
Konstantin Lopuhin 4607f2843e py3: unskip test_crawl 2016-01-21 17:53:08 +03:00
Konstantin Lopuhin bf5f54fa33 py3: fix getarg 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin ad2b3321b9 py3 compat: use range, fixes CrawlTestCase.test_start_requests_bug_yielding 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 0680950b98 py3: pass CrawlTestCase.test_referer_header, fixing Echo resource in mockserver and json decoding in test 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 20b839b44b py3: pass first crawl test (test_follow_all): fix mock server 2016-01-21 17:52:12 +03:00
Daniel Graña 0f500a1f82 Merge pull request #1708 from scrapy/fix-scrapy-bench 
PY3 fixed scrapy bench command
 2016-01-21 11:51:27 -03:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Raul Gallegos f042ad0f39 py3 fix HttpProxy and Retry Middlewares 2016-01-21 09:35:04 -05:00
Paul Tremberth c9e046d11d Merge pull request #1657 from palego/os.mknod-fails-on-macos 
[MRG+1] change os.mknod() for open()
 2016-01-21 15:17:57 +01:00
Mikhail Korobov 80c55f19a1 PY3 fixed scrapy bench command 2016-01-21 18:31:58 +05:00
Paul Tremberth da83f2dca7 Merge pull request #1693 from lopuhin/py3-commands-fetch-shell 
[MRG+1] Py3: port commands fetch and shell
 2016-01-21 10:59:32 +01:00
Paul Tremberth 6d8620ffe7 Merge pull request #1691 from lopuhin/py3-test-engine 
[MRG+1] Py3: port test_engine
 2016-01-21 10:30:39 +01:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Konstantin Lopuhin a5da7531c4 py3 backout skipping test_closespider - it was fixed on another branch 2016-01-20 23:01:29 +03:00
Konstantin Lopuhin 47d3c63338 py3: port fetch and shell commands, and review + enable already passing test_closespider.py and tests/test_utils_template.py 2016-01-20 23:01:29 +03:00
Konstantin Lopuhin c1db60188a py3: unskip passing test_engine 2016-01-20 22:57:46 +03:00
Konstantin Lopuhin 0b08b4bfcf fix engine tests - this callback (spider_closed_callback) should accept one argument, but the error was hidden on py2 2016-01-20 22:57:46 +03:00
Konstantin Lopuhin a32b59ac75 py3: fix EngineTest.test_crawler 2016-01-20 22:57:46 +03:00
Elias Dorneles 8a1255d3ba Merge pull request #1680 from lopuhin/py3-downloader-middleware 
[MRG+1] Py3: port downloader cache and compression middlewares
 2016-01-20 17:05:54 -02:00
Daniel Graña 176610f910 optional_features has been removed 2016-01-19 15:34:26 -03:00
Daniel Graña 5ec4319885 Merge pull request #1358 from nyov/nyov/boto-anon-connection 
[MRG+1] Support anonymous S3DownloadHandler (boto) connections
 2016-01-19 13:35:23 -03:00
Konstantin Lopuhin 4c84172840 move leveldb to tests/requirements-py3.txt 2016-01-19 18:24:58 +03:00
Konstantin Lopuhin 07577c432e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:39:22 +03:00
Daniel Graña 49fe631d89 Merge pull request #1676 from lopuhin/py3-webclient 
[MRG+1] py3: fix webclient
 2016-01-19 11:07:54 -03:00
Konstantin Lopuhin de98d8d006 move comment about test skipped on py3 into a proper place 2016-01-18 19:27:31 +03:00
Konstantin Lopuhin 7ffcc58759 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 19:23:12 +03:00
Konstantin Lopuhin 7fdd3225b2 fix test skipping logic - this is (temporary) py2-only part 2016-01-18 19:09:09 +03:00
Konstantin Lopuhin 324f2c1a6f common test_downloadermiddleware.py also passes now due to fixes in compression middleware 2016-01-18 16:45:22 +03:00
Konstantin Lopuhin 0b9336418e py3: port compression downloader middleware and tests 2016-01-18 16:43:58 +03:00
Konstantin Lopuhin 6edd4dec33 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 15:51:26 +03:00
Konstantin Lopuhin f876654a0c Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 15:50:58 +03:00
Konstantin Lopuhin 4946434582 revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00
Konstantin Lopuhin a2efd389b0 clarify: rename r_transform to response_transform 2016-01-18 15:09:54 +03:00
Konstantin Lopuhin b940606b7e this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase 2016-01-18 15:06:15 +03:00
Konstantin Lopuhin 0f527849f2 https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3 2016-01-18 14:44:04 +03:00
Konstantin Lopuhin 98c060d0b2 py3: fix http 1.1 test with https, and use self.host everywhere 2016-01-18 12:42:21 +03:00
Konstantin Lopuhin 04f69fd184 add https 1.1 downloader test - localhost is a valid DNS-ID 2016-01-18 12:37:46 +03:00
Konstantin Lopuhin f403c88053 Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 12:25:51 +03:00
palego 6b905a9aec split-up the assertIn 
to deal with OS X intricacies (directories prefixed with /private)
 2016-01-18 10:18:31 +01:00
Konstantin Lopuhin 673df5e416 add webclient test - check that non-standart body encoding matches Content-Encoding header 2016-01-18 11:44:49 +03:00
Konstantin Lopuhin 7d44c5dcea py3: unskip tests/test_downloadermiddleware_httpcache.py and scrapy/downloadermiddlewares/httpcache.py 2016-01-15 15:41:31 +03:00
Konstantin Lopuhin 8330776c21 fix error reporting in test: we can fail in process_request too, so result should always be defined 2016-01-15 15:16:12 +03:00
Konstantin Lopuhin 4398d95a02 skip this file on py3 again - it has one compression test, sould be done separately 2016-01-15 14:54:12 +03:00
Konstantin Lopuhin ae6758ee97 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-15 14:39:27 +03:00
Konstantin Lopuhin a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin 96fcf4cea4 add a check that byte url is not accepted in http.Response on py3 2016-01-15 13:27:28 +03:00
Konstantin Lopuhin 87849780bc some py3 fixes for RFC2616Policy 2016-01-15 13:13:10 +03:00
Konstantin Lopuhin ddc91dda27 py3: fix _BaseTest in httpcache 2016-01-15 11:49:28 +03:00
Konstantin Lopuhin 94ab7bee6c py3: body to bytes in tests, unskip test file 2016-01-15 11:26:01 +03:00
Konstantin Lopuhin 99f1f2ad1d unskip tests and modules ported to py3 2016-01-14 19:00:48 +03:00
Konstantin Lopuhin 81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin 2aa6c92ffc py3 fixes in tests.mockserver 2016-01-14 17:52:50 +03:00
Konstantin Lopuhin f46a9d595d skip ftp tests on py3 for now 2016-01-14 17:31:58 +03:00
Konstantin Lopuhin 4950f5988e py3: pass http proxy tests 2016-01-14 17:24:08 +03:00
Konstantin Lopuhin c6f14a39de py3: fix http10 downloader - unicode host expected here 2016-01-14 16:50:16 +03:00
Konstantin Lopuhin 6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00
Konstantin Lopuhin 3509378b8b py3: pass first http downloader test, simple crawler works now, yay! 2016-01-14 16:29:19 +03:00
Konstantin Lopuhin 32bb5b682a fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests 2016-01-14 16:11:16 +03:00
Konstantin Lopuhin 5c2241ccc7 py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted 2016-01-14 15:30:28 +03:00
Konstantin Lopuhin 6a412d2503 all tests pass in test_webclient.py on py3 - removing from py3-ignores 2016-01-14 13:48:48 +03:00
Konstantin Lopuhin 0178356178 py3 fix testRedirect: url is bytes here 2016-01-14 13:46:46 +03:00
Konstantin Lopuhin 30c7b4e4cc py3 compat in test_timeoutTriggering cleanup 2016-01-14 13:44:14 +03:00
Konstantin Lopuhin 88f55312af py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted 2016-01-14 13:43:37 +03:00
Konstantin Lopuhin b5f9bc8499 py3 test fixes in test_webclient - expect bytes as page body 2016-01-14 13:04:45 +03:00
Konstantin Lopuhin ae4aa2c3b2 py3 test fix: putChild expects bytes as path 2016-01-14 13:04:10 +03:00
Konstantin Lopuhin 325b6af6c2 fix ScrapyHTTPPageGetterTests for py3 - we expect bytes here 2016-01-14 12:29:59 +03:00
Konstantin Lopuhin 945674eb8f pass test_externalUnicodeInterference - the logic for py3 is clearly inverse of what was expected in this test, as scrapy Request url must be unicode 2016-01-14 12:25:54 +03:00
Konstantin Lopuhin 1d5ab67183 pass test_getPageHead on py3 2016-01-14 12:04:26 +03:00
Konstantin Lopuhin 9f2be23a39 webclient tests, py3: fix setUp, pass test_getPage 2016-01-14 11:42:23 +03:00
Paul Tremberth 1347015a80 Refactored test code 2016-01-13 12:32:28 +01:00
Paul Tremberth d4c7d72b2b Add tests for input type in xmliter calls 2016-01-13 12:13:47 +01:00
Paul Tremberth 9fad25f3d1 Use explicit Unicode and bytes for XML body in tests 2016-01-13 11:42:41 +01:00
Paul Tremberth 2f2c2e8096 Merge remote-tracking branch 'origin/master' into xmliter-unicode 2016-01-13 11:42:01 +01:00
Paul Tremberth 6ddd814738 Support unicode tags in xml iterators (fixes #1665) 2016-01-12 10:48:45 +01:00
Paul Tremberth d4872940db PY3: port utils/iterators 2016-01-07 10:51:27 +01:00
palego c702c53015 change os.mknod() for open() 
os.mknod() is a privileged command on OS X, making the test fail
 2016-01-03 14:33:42 +01:00
Elias Dorneles 7d187735ff Merge pull request #1359 from nyov/nyov/drop-optional_features 
[MRG+1] drop deprecated "optional_features" set
 2015-12-04 14:08:44 -02:00
Daniel Graña 17aba44f16 Merge pull request #1565 from starrify/fix-1564 
[MRG+1] fixed: Issue #1564 (Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. )
 2015-12-04 10:31:13 -03:00
Daniel Graña dc65026395 Merge pull request #1597 from Digenis/various_formreq_fixes 
[MRG+1] Various FormRequest tests+fixes
 2015-12-04 09:55:12 -03:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Νικόλαος-Διγενής Καραγιάννης 395ef805eb FormRequest: test unicode xpath expr & exception 2015-11-13 18:54:02 +02:00
Νικόλαος-Διγενής Καραγιάννης 7a438a51b7 FormRequest: test default <input> type (is text) 2015-11-13 18:03:54 +02:00
Νικόλαος-Διγενής Καραγιάννης 881bf19e68 FormRequest: test case-insensitive type attribute 2015-11-13 17:57:25 +02:00
Jakob de Maeyer a65fc0db7d Drop support for providing file paths as add-ons 2015-11-12 18:37:26 +01:00
Jakob de Maeyer 33dfb3e167 Remove unused project path util function 2015-11-11 21:08:20 +01:00
Jakob de Maeyer b7b00fb956 PEP8ify add-ons and tests 2015-11-11 21:08:20 +01:00
Jakob de Maeyer e924d38238 Drop component configuration (copied from default_settings) from built-in add-ons 2015-11-11 21:08:20 +01:00
Jakob de Maeyer 388c5c4b78 Fix component exporting for Addon base class 2015-11-11 21:08:20 +01:00
Jakob de Maeyer 8e5d067af1 Drop BaseSettings.get_addon() prefix magic 2015-11-11 21:08:20 +01:00
Jakob de Maeyer b10caf91a1 Drop support for add-on configuration in scrapy.cfg 2015-11-11 21:08:20 +01:00
Jakob de Maeyer f7ed239fcb Replace INSTALLED_ADDONS tuple setting with ADDONS dictionary setting 2015-11-11 21:08:20 +01:00
Jakob de Maeyer 9f7fcf5582 Make update_classpath() util function return non-string objects 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d18b6a61d7 Add missing AddonManager tests 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 25498c3c21 Remove unused imports in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d91647c38b Add built-in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 07455b1883 Integrate add-ons into start-up process 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d8af395d76 Introduce add-ons via AddonManager and Addon base class 2015-11-11 21:08:19 +01:00
Jakob de Maeyer cfed9b6659 Allow passing Python objects to middleware dict settings 2015-11-11 21:08:19 +01:00
Julia Medina 54216d7afe Merge pull request #1586 from jdemaeyer/fix/backwards-compatible-per-key-priorities 
[MRG+1] Backwards compatible per key priorities
 2015-11-11 14:08:50 -03:00
Jakob de Maeyer 44f6ada034 Overwrite, not update, dictionary-like settings 2015-11-11 17:40:42 +01:00
Jakob de Maeyer 52ecee6a62 Replace BaseSettings._getcomposite() with public .getwithbase() method 2015-11-11 17:39:57 +01:00
Jakob de Maeyer b6a023ce98 Add backwards compatibility for build_component_list 2015-11-11 17:39:56 +01:00
Jakob de Maeyer 2629997a2f Make Spider.update_settings() an instance method 2015-11-10 23:48:20 +01:00
Mikhail Korobov 9548691fdd Merge pull request #1563 from starrify/master 
[MRG+1] fixed: Issue #1562 (Incorrectly picked URL in `scrapy.http.FormReques t.from_response` when there is a `<base>` tag)
 2015-11-10 21:33:24 +05:00
Jakob de Maeyer daec045711 Move spider settings tests 2015-11-09 18:07:05 +01:00
Jakob de Maeyer 380f76d35f Fix tests that had multiple calls to crawl() 2015-11-09 16:54:22 +01:00
Pengyu CHEN 94486bb294 added: Test case for the fix. 2015-11-02 23:00:42 +08:00
Pengyu CHEN e19bf4aecc added: Test case for the fix 2015-11-02 22:52:41 +08:00
Julia Medina 57f87b95d4 Merge pull request #1528 from scrapy/create-crawler 
public Crawler.create_crawler method
 2015-10-30 12:36:35 -03:00
Mikhail Korobov a49c82ad62 TST improve CrawlerRunner tests 
* use CrawlerRunner.create_crawler instead of get_crawler helper in test_crawl;
* add a test for loading spiders by name;
* add a test for passing Crawler objects instead of Spider objects;
* add a test for CrawlerRunner.join
 2015-10-30 20:24:37 +05:00
Jakob de Maeyer 32ff4cc1d7 PEP8ify scrapy.settings tests 2015-10-29 19:52:19 +01:00
Jakob de Maeyer 51ca84c9b4 Add missing tests for scrapy.settings module 2015-10-29 17:34:18 +01:00
Jakob de Maeyer 8307c1212f Add ExecutionEngine.close() method 2015-10-29 10:03:29 +01:00
Jakob de Maeyer 03f1720afb Fix backwards-compatibility for users who explicitly set _BASE settings 2015-10-27 14:22:56 +01:00
Jakob de Maeyer f249b309ab Move scrapy.utils.conf.remove_none_values to s.u.python.without_none_values 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Mikhail Korobov 215905bdb6 PY3 port bench, startproject, genspider, list and runspider commands 2015-10-12 17:59:49 +05:00
Νικόλαος-Διγενής Καραγιάννης d66efb13ba test xml nodename with dots 2015-10-07 14:43:47 +03:00
preetwinder 47c8e2ba78 Restructure tests for add_http_if_no_scheme function 2015-09-24 17:57:25 +00:00
preetwinder 8c629eee3e adds docstring, tests and correction 2015-09-24 17:28:44 +00:00
preetwinder c517951a48 add_scheme_if_missing for scrapy shell command 2015-09-24 17:28:44 +00:00
Mikhail Korobov 916141a08f TST pin pytest to 2.7.3 2015-09-21 21:12:52 +05:00
Demelziraptor 97a52665a0 interpreting json-amazonui-streaming as TextResponse 2015-09-18 17:16:43 +09:00
Daniel Graña 51b1249033 Revert "test suite requires recent queuelib to pass because it depends on queuelib test suite" 
This reverts commit a3390afc66.
 2015-09-18 02:27:07 -03:00
Daniel Graña a3390afc66 test suite requires recent queuelib to pass because it depends on queuelib test suite 2015-09-17 19:42:29 -03:00
Daniel Graña 3c596dcf46 Merge pull request #1467 from dacjames/master 
add support for a nested loaders
 2015-09-16 22:03:19 -03:00
nyov ecbfe4bd66 drop deprecated "optional_features" set 2015-09-15 09:00:08 +00:00
Elias Dorneles 430e90f4b2 minor refactor on metarefresh redirect mware test 2015-09-13 12:29:41 -03:00
Elias Dorneles 81950f773d made encoding conversion more explicit, added test for header with utf-8 encoding replicating what browsers do 2015-09-11 18:51:48 -03:00
Elias Dorneles 179c4588ca adding test for latin1 location 2015-09-10 19:36:47 -03:00
Elias Dorneles defa899135 PY3 port redirect middleware 2015-09-10 16:31:59 -03:00
Daniel Graña 9a64d8ff97 fix scrapy squeue tests after recent changes to queuelib 2015-09-09 15:55:55 -03:00
Daniel Collins 311d5cd495 split nested_loader into seperate methods 2015-09-04 13:15:48 -07:00
Daniel Collins 425e35ee90 removed commented code and add test for error handling 2015-09-04 13:15:48 -07:00
Daniel Collins 1f95af3c07 add support for a nested loaders 2015-09-04 13:15:48 -07:00
Daniel Graña dd473145f2 Merge pull request #1473 from ArturGaspar/master 
[MRG+1] Support for returning deferreds in middlewares
 2015-09-02 13:04:26 -03:00
Artur Gaspar dd83f6123d Test for same response object in downloader middleware test. 2015-09-02 11:36:15 -03:00
Artur Gaspar 2748b38592 Test for not calling the download function when downloader middleware returns a response in process_request. 2015-09-02 11:15:37 -03:00
Artur Gaspar 1f4af9d81e Test for robots.txt middleware for logger not being called when the request for robots.txt is ignored. 2015-09-02 10:33:53 -03:00
Artur Gaspar 668e5fd257 Test for robots.txt middleware for processing a request for which the robots.txt parser is ready. 2015-09-02 10:23:24 -03:00
Artur Gaspar a6a629e707 Call actual error logger in robots.txt middleware tests. 2015-09-02 10:23:24 -03:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Daniel Graña 3919ad64c5 Merge pull request #1470 from scrapy/py3-middlewares 
[MRG] port some downloader middlewares to Python 3
 2015-09-02 00:17:37 -03:00
Olaf Dietsche 2c28b53cc0 Add tests with incomplete buttons 2015-09-01 12:35:49 +02:00
Mikhail Korobov c44cafe4f5 PY3 fix UserAgentMiddleware tests 2015-09-01 04:11:33 +05:00
Mikhail Korobov 3a9c73bc5d PY3 fix DownloaderStats middleware tests 2015-09-01 04:09:15 +05:00
Mikhail Korobov 78a4cd0f1c PY3 fix HttpAuthMiddleware tests 2015-09-01 04:00:26 +05:00
Mikhail Korobov 179a4409fb PY3 enable DownloadTimeoutMiddleware tests 2015-09-01 03:58:25 +05:00
Mikhail Korobov 3cf1911a92 PY3 fix DefaultHeadersMiddleware tests 2015-09-01 03:57:05 +05:00
Mikhail Korobov 45101829a5 PY3 fix AjacCrawlable middleware tests 2015-09-01 03:49:52 +05:00
Olaf Dietsche e5f26078fa Add test: submit form with button control 2015-08-31 19:57:45 +02:00
Mikhail Korobov 44bfcbcf0f TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test 2015-08-31 00:49:38 +05:00
Mikhail Korobov f7052413e0 PY3 raise an exception if bytes are passed as url to Link constructor 2015-08-28 23:04:02 +05:00
Mikhail Korobov 9bfe6ece59 Merge branch 'master' into py3-linkextractors 
Conflicts:
	scrapy/linkextractors/lxmlhtml.py
	tests/test_linkextractors.py
 2015-08-28 04:53:32 +05:00
Mikhail Korobov ff24cbbc47 PY3 depth, offsite and referer spider middlewares; Crawler 2015-08-28 04:11:30 +05:00
Mikhail Korobov d5984bbea9 PY3 port scrapy.spiders 2015-08-28 04:11:30 +05:00
Mikhail Korobov f2edbd05de PY3 port LinkExtractor 
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
 2015-08-28 04:11:30 +05:00
Mikhail Korobov f46a450080 refactor test_linkextractors 
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
  and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
 2015-08-28 04:11:30 +05:00
Julia Medina aa31811cfd Merge pull request #1214 from rgtk/link-rel 
[MRG+1] Support link rel attribute with multiple values
 2015-08-27 19:57:54 -03:00
Rafał Gutkowski cb3007c066 support link rel attribute with multiple values 2015-08-27 20:13:47 +02:00
Mikhail Korobov 8ecc4544b3 enable console tests in PY3 2015-08-27 21:37:24 +05:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Daniel Graña 71bd79e70f Merge pull request #1212 from Digenis/test-fix-render_template_file 
fix utils.template.render_templatefile() bug +test
 2015-08-26 14:46:42 -03:00
Νικόλαος-Διγενής Καραγιάννης f5c54072ec Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
Mikhail Korobov 68a47ade46 PY3 port test_logformatter 2015-08-26 02:34:21 +05:00
Mikhail Korobov 7da769feb2 enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Andrew Scorpil de15fcdf33 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2015-08-15 00:16:39 +00:00
nyov 1b4fd3a8df Support anonymous connections in S3DownloadHandler 
Also consider any unknown keyword args for S3DownloadHandler as
arguments to pass on to S3Connection (e.g. proxy settings).
 2015-08-14 22:12:12 +00:00
Daniel Graña 27077d2b6c Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
Daniel Graña 43fc2f23b8 Increase coverage of version command 2015-08-13 19:30:06 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Daniel Graña 15c1300d35 Merge pull request #1409 from eliasdorneles/migrate-parsel 
[MRG+1] Migrating selectors to use parsel
 2015-08-11 16:22:58 -03:00
Elias Dorneles 766c255152 upgrade parsel and add shim for deprecated selectorlist methods 2015-08-11 15:20:33 -03:00
Daniel Graña 457b97c13c Merge pull request #1388 from scrapy/dupefilter-persist 
Dupefilter persistence
 2015-08-11 14:16:34 -03:00
Elias Dorneles e50610bd3a set base_url in kwargs to be fully backward compatible 2015-08-11 14:09:34 -03:00
Elias Dorneles 8ef5aa2ffc using bytes for response body in tests 2015-08-11 14:09:34 -03:00
Elias Dorneles 3a03ef7c08 cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
Elias Dorneles 12579b9afa warning when ambiguous root arguments and minor cleanups 2015-08-11 14:09:34 -03:00
Elias Dorneles 2fe6d128f5 upgrade parsel and using promoted root attribute 2015-08-11 14:09:16 -03:00
Elias Dorneles 94c3a345b7 remove deprecated module lxmldocument 2015-08-11 14:09:00 -03:00
Elias Dorneles 17d7347a36 update minimal parsel version, add deprecated classes for csstranslator modules, fix test 2015-08-11 14:09:00 -03:00
Elias Dorneles 01d948f0fd remove selector support for LxmlDocument DOM cache and add deprecation warning for _root argument 2015-08-11 14:08:43 -03:00
Elias Dorneles ce21884a97 migrating scrapy Selector to use Parsel 2015-08-11 14:08:43 -03:00
Mikhail Korobov fa123b335b Merge pull request #1421 from dangra/nyov/lazyload-downloadhandlers 
lazy-load downloadhandlers (continuation of #1357)
 2015-08-11 18:59:47 +05:00
Mikhail Korobov fa3d84b050 Merge pull request #1420 from scrapy/py3-trackrefs 
PY3: port scrapy.utils.trackref
 2015-08-11 18:21:34 +05:00
nyov 8c7997083f lazy-loading for DownloadHandlers 2015-08-10 16:52:44 -03:00
Daniel Graña accd28cc71 PY3: port scrapy.utils.trackref 2015-08-10 15:13:29 -03:00
Daniel Graña 8d45b3c481 Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
Mikhail Korobov 56be610e6e TST a test for --profile option 2015-08-08 04:54:47 +05:00
nyov 17b5e9fb86 PY3 response bodies as bytes 2015-08-08 04:23:32 +05:00
nyov 45d441d444 PY3 fix test loader 2015-08-08 04:23:16 +05:00
nyov 34eced0ee8 PY3 fix tests pipelines images 2015-08-08 04:23:06 +05:00
nyov 991197003b PY3 fix tests pipelines files 2015-08-08 04:22:58 +05:00
nyov 6e762ce25c PY3 renames (six types) 2015-08-08 04:21:58 +05:00
nyov 3e6d6c43ac PY3 fix test cmdline 2015-08-08 04:19:55 +05:00
nyov 4d41cc0dc4 PY3 split requirements into files 2015-08-08 04:19:37 +05:00
Daniel Graña c5ffa1a59f Twisted 15.3.0 does not raises expected exception serializing lambda functions 2015-08-06 16:40:21 -03:00
Mikhail Korobov 91991e0bdc PY port scrapy.utils.reqser 2015-08-04 02:17:46 +05:00
Mikhail Korobov 95e6bd2f8d Merge pull request #1398 from scrapy/py3-cookies 
PY3 port http cookies handling
 2015-08-04 01:23:37 +05:00
Daniel Graña f4fc05c2a1 Do not propagate cookie log messages in tests so TopLevelFormatter does not rewrite them 2015-08-03 17:18:08 -03:00
Daniel Graña c6adf648dc PY3 port COOKIES_DEBUG and add tests 2015-08-03 16:28:29 -03:00
Daniel Graña dba7e39f61 Do not break cookie parsing on non-utf8 headers 2015-08-03 10:53:40 -03:00
Daniel Graña cbfb24dbeb Merge pull request #1353 from nyov/test-no-proxies 
[MRG+1] Unset environment proxies for tests
 2015-08-03 09:49:50 -03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
Daniel Graña 5f02ef82e8 PY3 port http cookies handling 2015-08-02 22:28:45 -03:00
nyov b2fd9bdb63 lazy-load s3 boto 2015-08-01 03:43:03 +00:00
Daniel Graña 71c6bd920e Merge pull request #1396 from scrapy/py3-utils-response 
PY3 port scrapy.utils.response
 2015-07-31 17:53:53 -03:00
Mikhail Korobov 9316b2317b remove scrapy.utils.testsite from PY3 ignores 2015-08-01 00:43:13 +05:00
Mikhail Korobov 7f927f68e1 PY3 port scrapy.utils.response 2015-08-01 00:35:43 +05:00
Mikhail Korobov d05cf6e0af Merge branch 'responsetypes' 2015-07-31 23:50:06 +05:00
Mikhail Korobov 02b5182608 small ResponseTypes cleanup 2015-07-31 23:48:49 +05:00
Daniel Graña de6e013b9a fix form requests tests on py3 2015-07-31 14:54:07 -03:00
Daniel Graña 786f62664b Merge pull request #1137 from DharmeshPandav/patch-1 
Update form.py to improve existing capability
 2015-07-31 11:00:57 -03:00
Gregory Vigo Torres 06b91da943 using bytes for body constant 2015-07-31 14:31:11 +02:00
Gregory Vigo Torres 5d75d44f2c removed test_responsetypes from py3ignores 2015-07-30 13:07:42 +02:00
nyov e044bfa60f PY3 fix test downloadermiddleware decompression 2015-07-29 13:51:01 +00:00
nyov ec8afbc060 PY3 fix test pipeline media 2015-07-29 13:51:01 +00:00
nyov 8ab1648a36 PY3 fix test middleware 2015-07-29 13:51:01 +00:00
Mikhail Korobov 757f983a97 PY3: enable RobotsTxtMiddleware tests 2015-07-29 18:37:04 +05:00
nyov 1a1bc2c6b5 PY3 port scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware 2015-07-29 04:21:57 +00:00
Mikhail Korobov c78e2636f9 TST a test to show that dupefilter persistence is not working 2015-07-28 14:31:42 +05:00
Daniel Graña 2c937f3554 PY3 port scrapy.dupefilters 2015-07-25 18:10:29 +02:00
Daniel Graña 776064aee1 PY3 port signals 2015-07-25 18:04:55 +02:00
Mikhail Korobov c53d6d7460 Merge pull request #1383 from zoliszeredi/py3/fix_test_utils_defer 
Py3/fix test utils defer
 2015-07-25 17:43:50 +02:00
Mikhail Korobov 7874bb9f13 PY3 port utils.request 2015-07-25 17:14:56 +02:00
Daniel Graña f750ee4c00 Remove ignored passing tests from py3 branch 2015-07-25 16:51:22 +02:00
Daniel Graña ea56067e40 ignore moved files from py3 tests 2015-07-25 16:39:02 +02:00
Mikhail Korobov e438df7028 PY3 enable Request and Response tests 2015-07-25 13:10:13 +02:00
Mikhail Korobov e853d9e910 partial port of Request and Response 2015-07-25 13:08:44 +02:00
Zoltán Szeredi 1bf7a98000 Remove test_utils_defer from ignores. 2015-07-25 13:05:50 +02:00
Zoltán Szeredi 9aaa8f8125 Improve python3 support for test_utils_defer. 2015-07-25 12:58:39 +02:00
Mikhail Korobov f576b3ffee [tmp] improve python 3 support for scrapy.utils.url 2015-07-25 00:40:45 +02:00
Mikhail Korobov a7b4a3e7f9 cleanup 
* run test_utils_python in Python 3;
* make tests for 'errors' argument more explicit
* add missing test_ prefix utf_16_strings_contain_null_bytes;
* cleanup test names.
 2015-07-23 20:03:33 +02:00
Mikhail Korobov 4073498652 port utils.python 
* stringify_dict is deprecated
* is_writable is deprecated
* setattr_default is deprecated
* get_spec is untested
* re_rsearch is untested
* retry_on_eintr is untested
 2015-07-23 18:33:56 +02:00
Mikhail Korobov 41bcae5d00 TST fix to_bytes and to_unicode tests in Python 3.x 2015-07-23 17:35:23 +02:00
Mikhail Korobov 61cd27e5c7 Rename unicode_to_str and str_to_unicode method. Fixes GH-778. 2015-07-23 15:00:06 +02:00
Jakob de Maeyer c153217bfb Fix RedirectMiddleware test 2015-07-17 16:11:53 +02:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
nyov 4f56c6c79b Unset environment proxies for tests 
because urllib doesn't handle $no_proxy correctly
and the unittest webserver is always local.
 2015-07-11 11:30:08 +00:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina d3d61fde87 Fix a recent mock regression by not using it in a test 2015-07-10 19:58:38 -03:00
Demelziraptor 9a15fcf89a interpreting application/x-json as TextResponse 2015-07-02 19:56:30 +02:00
Daniel Graña 07f4f12e8b Merge pull request #1313 from eliasdorneles/support-empty-passwd-for-http-proxy 
[MRG+1] Support empty password for http_proxy config
 2015-06-30 19:58:16 -03:00
Mikhail Korobov ea4b175c2d Merge pull request #1254 from scrapy/bugfix-1232 
[MRG +1] Unquote request path before passing to FTPClient, it already escape paths
 2015-06-24 18:04:45 +05:00
Elias Dorneles eb8fed4ecd support empty password for http_proxy config 2015-06-23 00:43:15 -03:00
Berker Peksag 8a48d9c6a8 Ignore ScrapyDeprecationWarning warnings properly. 
Conflicts:

	tests/test_utils_deprecate.py
 2015-06-09 18:22:03 +03:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Mikhail Korobov cc2f3e1b46 TST a test case to show custom_settings doesn't always work 2015-06-03 04:26:20 +05:00
Julia Medina ffc7b7fd6c Add helper to update deprecated class paths 2015-06-01 17:01:33 -03:00
Marven Sanchez bb3ebf13f9 Add tests for RFC2616 policy enhancements 
Add `scrapy/downloadermiddlewares/httpcache.py` to `tests/py3-ignores.txt
 2015-06-01 18:20:12 +08:00
Mikhail Korobov aa6a72707d fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156
 2015-05-30 06:59:15 +05:00
Daniel Graña ebe889a663 Unquote request path before passing to FTPClient, it already escape paths 2015-05-23 20:50:30 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov 7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov 8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina 157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña 5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy 773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj 8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur) 667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina 6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina 6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina 1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina 6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
Julia Medina c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina 7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Daniel Graña 571bf68d7d Merge pull request #353 from nramirezuy/item-multi_inherit 
[MRG] Item multi inheritance fix
 2015-04-22 05:37:41 -03:00
nramirezuy 7871acd665 Item multi inherit fixed 2015-04-21 18:10:52 -03:00
Mikhail Korobov c3d3a94914 fixed backwards compatibility for SPIDER_MANAGER_CLASS option 2015-04-22 00:29:40 +05:00
Pablo Hoffman 0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Daniel Graña e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman 06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Nikolaos-Digenis Karagiannis 017fb25bee loader test with processors that use item's values 2015-04-20 11:04:31 +03:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
mrpandav 954c8fcecb changes `test_from_response_formname_notexists_fallback_formid` unit test for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 16:49:09 +05:30
mrpandav 020a32a3d9 Adding tests for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 11:23:25 +05:30
Julia Medina 526aa07f44 Merge pull request #1156 from kmike/crawlers-accept-dicts 
allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects
 2015-04-15 02:15:32 -03:00
Daniel Graña c013baa602 Merge pull request #1161 from scrapy/telnet-disable 
disable scrapy.telnet if twisted.conch is not available
 2015-04-15 00:32:42 -03:00
Mikhail Korobov 0b76487629 PY3 fix Python 3 syntax errors in ajaxcrawl.py 
Tests for _has_ajaxcrawlable_meta passin Python 3.
 2015-04-15 04:16:26 +05:00
Mikhail Korobov 378b6efc0e PY3 disable scrapy.telnet if twisted.conch is not available 2015-04-15 04:06:36 +05:00
Mikhail Korobov abeb8e3a01 TST skip a test in Python 3 because it imports some parts of Twisted which are not ported yet 2015-04-15 03:27:11 +05:00
Mikhail Korobov 647eeaea61 TST add a test for GH-1050. 2015-04-15 03:23:30 +05:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov 7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Mikhail Korobov 973c31f735 TST cleanup: use assertIn instead of assert_ 2015-04-14 20:11:46 +05:00
Pablo Hoffman 71c0aface7 Merge pull request #1148 from Curita/verify-spidermanager-interface 
Verify SPIDER_MANAGER_CLASS interface while loading it in CrawlerRunner
 2015-04-13 14:25:32 -03:00
Jakob de Maeyer cf9d848f0f Fixed deprecated settings API in cmdline test 2015-04-12 14:18:47 +02:00
Nikolaos-Digenis Karagiannis bf301b7328 encode invalid xpath with unicode_escape under PY2 
The exception quotes an xpath string
which may be unicode.
 2015-04-10 22:23:40 +03:00
Julia Medina 24a07fd892 Verify SPIDER_MANAGER_CLASS interface in CrawlerRunner 2015-04-10 13:43:33 -03:00
nyov 6d48c19acd dissolve `scrapy.contrib_exp` 2015-04-07 21:20:48 +00:00
Pablo Hoffman 4b11501eb5 Merge pull request #963 from tpeng/fix-xmliter-lxml 
[MRG+1] support namespace prefix in xmliter_lxml
 2015-04-07 18:13:01 -03:00
Daniel Graña 9706119336 Merge pull request #1128 from Curita/move-spider-settings-population-to-crawler 
[MRG+1] Move spider settings population from CrawlerRunner to Crawler.__init__
 2015-04-04 18:08:05 -03:00
Julia Medina 6f9265b44b Move spider settings population to Crawler.__init__ 2015-04-01 23:29:10 -03:00
nyov dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
Daniel Graña ec4251afd7 Merge pull request #1089 from drack3800/master 
[MRG+1] Add test for webclient with POST method and no body given
 2015-03-31 17:30:08 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
nyov 84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
drack3800 1b6d5a011a Added webclient test for checking Content-Length header in response for POST request with no given body 2015-03-21 04:02:51 +03:00
Mateusz Golewski f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski 2742b4d8c2 Add tests to extract_first() 2015-03-18 21:11:08 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov 645366a0fd Merge pull request #1070 from berkerpeksag/items-equal 
assertItemsEqual was renamed to assertCountEqual in Python 3.
 2015-03-08 22:54:57 +05:00
Berker Peksag c86e1bebec assertItemsEqual was renamed to assertCountEqual in Python 3. 2015-03-08 15:18:49 +02:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Julia Medina d68615a5af Test the parse command locally instead of against an external url 2015-01-19 10:28:25 -03:00
tpeng 82d138e87e support namespace prefix in xmliter_lxml 2014-12-15 16:17:06 +01:00
Artur Gaspar 403fc686b8 tests for CSS support in link extractors 2014-12-11 18:20:30 -02:00
immerrr 82b187f283 S3DownloadHandler: fix auth for requests with quoted paths/query params 2014-12-11 18:14:36 +03:00
Lev Berman fdb6bb07c0 #977 - test dropping requests 2014-11-28 10:53:33 +03:00
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
Pablo Hoffman c31fb87335 Merge pull request #954 from kalessin/int-download-timeout 
Force to read DOWNLOAD_TIMEOUT as int (for example to pass using environment variable)
 2014-11-26 17:14:18 -02:00
Pablo Hoffman dedea72774 Merge pull request #946 from tpeng/limit-response-size 
avoid download large response
 2014-11-25 17:57:55 -02:00
tpeng cd19382754 attemp to fix travis fails 2014-11-25 14:20:25 +01:00
Daniel Graña 8d8e1b2c0c mitmproxy 0.10.1 needs netlib 0.10.1 too 2014-11-21 12:15:02 -02:00
Daniel Graña 314db3db8b pin mitmproxy 0.10.1 as >0.11 does not work with tests 2014-11-21 10:54:43 -02:00
Martin Olveyra 7910fa0172 Force to read DOWNLOAD_TIMEOUT as int (for example to pass using 
environment variable)
 2014-11-21 01:09:32 -02:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov 49645d4bf9 TST small cleanup of a cookie test 2014-09-21 05:31:34 +06:00
Mikhail Korobov c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
andrewshir e583c030db Test for local domains (without dots) added 2014-09-14 14:24:16 +06:00
Mikael Åhlén 22da1783bd added a test-case for wrong quotechar 2014-09-13 03:47:40 +02:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña 5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña ec93c0fdcc Add the tests changes for previous commit 2014-09-10 12:05:18 -03:00
Daniel Graña 99971dc8a8 Do not pop the crawler from the managed list 2014-09-09 20:59:07 +00:00
Julia Medina c2592b39fd Test verifying that CrawlerRunner populates spider class settings 2014-09-01 21:56:57 -03:00
Julia Medina 9ef3972cfb Per-spider settings tests 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
yakxxx e4689556f0 SgmlLinkExtractor - fix for parsing <area> tag with Unicode present 2014-08-28 18:55:58 +02:00
nramirezuy 2a540206a7 fix xmliter namespace on selected node 2014-08-19 15:44:49 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 70f2010db1 Change error type when updating frozen settings 2014-08-14 11:59:25 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina 3ae971468f Add Settings.copy, freeze and frozencopy method 2014-08-12 14:02:55 -03:00
Julia Medina a995727117 Connect spider_closed signal after a crawler is bound to a Spider 2014-08-12 14:02:55 -03:00
Julia Medina eb0253e530 Update from_crawler method as well as set_crawler on CrawlSpider 2014-08-11 11:24:01 -03:00
Julia Medina 84fa004793 Add from_crawler class method to base Spider 2014-08-11 11:23:57 -03:00
Pablo Hoffman 69a665eb32 Merge pull request #832 from ivannotes/master 
Bugfix for leaking Proxy-Authorization header to targeted host
 2014-08-04 15:20:51 -03:00
Daniel Graña 511a26978f Merge pull request #817 from nramirezuy/fix-startproject-project-name 
[MRG] Fix startproject project name
 2014-08-04 11:55:35 -03:00
Mikhail Korobov e62bbf0766 PY3 top-level shortcuts work 2014-08-02 00:24:32 +06:00
Daniel Graña ee21eaa6f3 Merge pull request #835 from scrapy/modernize-asserts 
[MRG] modernize some of the asserts
 2014-08-01 15:19:36 -03:00
Mikhail Korobov 1f59a69a97 PY3 port scrapy.link 2014-08-02 00:16:01 +06:00
Mikhail Korobov 628a8d8a15 TST modernize some of the asserts 2014-08-01 23:44:23 +06:00
Nuno Maximiano 08224c92f4 add project name validation 2014-08-01 14:36:30 -03:00
Mikhail Korobov e2c1226838 PY3 port scrapy.contrib.feedexport 2014-08-01 23:28:28 +06:00
Mikhail Korobov 51bd9f7d64 TST enable doctests for Python 3 2014-08-01 23:12:52 +06:00
Mikhail Korobov 444052a2f9 PY3 port scrapy.utils.misc.arg_to_iter; fix scrapy.utils.misc.md5sum doctest 2014-08-01 22:56:03 +06:00
Mikhail Korobov 83432184c8 PY3 enable more tests 2014-08-01 16:53:18 +06:00
Mikhail Korobov 669b62c741 PY3 port scrapy.utils.spider 2014-08-01 16:43:39 +06:00
Mikhail Korobov 560c32d330 PY3 fix test_utils_sitemap. 
Sitemap class is only initialized with bytes by Scrapy;
broken tests was only a testing issue.
 2014-08-01 16:17:40 +06:00
Mikhail Korobov 330743411e PY3 enable more tests 2014-08-01 16:10:34 +06:00
Mikhail Korobov 7183a87c35 Merge pull request #828 from dangra/string-headers 
normalize header values to bytes (and PY3)
 2014-08-01 15:51:57 +06:00
Daniel Graña ccae49813a PY3: port scrapy/utils/httpobj.py 2014-08-01 02:38:29 -03:00
Daniel Graña c564334d00 PY3: port scrapy/utils/gz.py 2014-08-01 02:25:33 -03:00
Daniel Graña 910d0d36c1 PY3: port scrapy/utils/conf.py 2014-08-01 02:20:34 -03:00
Daniel Graña 521303e27f fix ftp test cases 2014-08-01 02:10:57 -03:00
Daniel Graña f42b44b50e normalize header values to bytes 2014-08-01 00:02:50 -03:00
ivannotes 484a0159d0 Add test case for tunneling proxy 2014-08-01 09:43:17 +08:00
Daniel Graña 7cd271eb8e remove more references to runtests.sh 2014-07-31 17:30:45 -03:00
Daniel Graña 70ee66ba0c Collect the list of packages not working on python3 2014-07-31 14:06:34 -03:00
Daniel Graña caab9f29f2 move tests requirements under tests dir 2014-07-30 19:45:08 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00