RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,595 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

736 Commits

Author SHA1 Message Date
Daniel Graña 9fef98330c Backport security fix #3415 2018-12-26 12:57:00 -03:00
Vostretsov Nikita 45d2325ef7 blacklist twisted version with regression in constraints file 2018-07-10 02:33:39 +05:00
Lucy Wang 1eb146d0df fix tests on jessie 2018-07-10 00:53:46 +05:00
Lucy Wang a69b8529b9 revert wrong changes 2018-07-10 00:53:33 +05:00
Lucy Wang 90170af462 add a test case 2018-07-10 00:53:27 +05:00
Mikhail Korobov 117e4657a3 TST fix tests to account for changes in w3lib 1.19 2018-07-09 23:28:12 +05:00
Konstantin Lopuhin 041308afe7 Fix get_func_args test for pypy3 
These built-in functions are exposed as methods in PyPy3.
For scrapy this does not matter as:
1) they do not work for CPython at all
2) get_func_args is checked for presense of an argument in scrapy,
   extra "self" does not matter.
But it still makes sense to leave these tests so that we know we
shouldn't use get_func_args for built-in functions/methods.
 2017-12-25 14:48:10 +03:00
Konstantin Lopuhin a21b800419
Merge pull request #3011 from Jane222/master 
[MRG+1] Issues a warning when user puts a URL into allowed_domains (#2250)
 2017-12-08 15:45:16 +03:00
Daniel Graña 3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command 
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
 2017-11-29 16:26:48 -03:00
Jana Cavojska 454d5e5733 checking for subclass of URLWarning instead of checking error message text when URL in allowed_domains 2017-11-26 20:07:04 +01:00
Jana Cavojska 8ec3b476b0 triggering a warning when user puts URL in allowed_domains now covered by test 2017-11-26 16:36:15 +01:00
IAlwaysBeCoding 846fd83512 removed commented out code, wrapped line to pep-8 and removed backlashes 2017-11-11 18:30:01 -05:00
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Daniel Graña 5fac2d7b90 Merge pull request #2923 from rhoboro/fixes-685 
[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage.
 2017-10-02 13:52:20 -03:00
Lucas Moauro 59c3f6f095 Fix typos in tests 2017-10-01 12:24:56 -03:00
Paul Tremberth e733f51d4b Fix test 2017-09-25 12:49:27 +02:00
rhoboro e5d4364b2a Add tests for GCS Storage 2017-09-13 16:24:16 +09:00
Mikhail Korobov 1ba77f0837 Merge pull request #2869 from cclauss/patch-3 
# noqa to close #2836
 2017-08-21 19:15:30 +05:00
Mikhail Korobov 885289f497 Merge pull request #2854 from jenya/2853-sitemap-follow-alternate-fix 
[MRG+1] Follow alternate link for all types of sitemaps #2853
 2017-08-21 19:15:03 +05:00
Daniel Graña d5f3543d77 Merge pull request #2865 from kirankoduru/2831-explicit-msg-for-scrapy-parse-callback 
[MRG+1] Explicit message for scrapy parse callback
 2017-08-09 07:56:57 -03:00
Chomba Ng'ang'a fd27cde24d Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Kiran Koduru 12409a0cf6 Fix broken encoding on text for py 3 2017-08-05 15:40:38 -04:00
kirankoduru 7adab61a7a Added test for NotImplemented Spider.parse method 2017-08-05 15:40:38 -04:00
cclauss 0a69a32b5a Force Travis CI to test again 2017-08-04 14:35:43 +02:00
cclauss c016a4309d # noqa to close #2836 
Marks #2836 as will not fix.
 2017-08-04 01:44:23 +02:00
Paul Tremberth 0cb3085f84 Add test for alternate links 2017-08-03 16:55:09 +02:00
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Daniel Graña 2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña 45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Paul Tremberth 1f08d9a648 Add test for DNS cache disabling 2017-07-04 23:10:19 +02:00
Konstantin Lopuhin 7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin 5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin 6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00
Pengyu CHEN b33e0d5a54 Added: Now supporting <link> tags in Response.follow 2017-06-14 17:09:42 +08:00
Valdir Stumm Junior 4b6f68b9ee make reqser tests create Request with proper callback/errback 2017-06-05 17:26:52 -03:00
Valdir Stumm Junior 60727dedf6 verify if Request callback is callable 2017-05-31 15:00:38 -03:00
Daniel Graña 25f609e2a3 Merge pull request #2675 from simongartz/png-p-to-jpg-conversion-fix 
[MRG+1] Fixes conversion of transparent PNG with palette images to jpg #2452
 2017-05-18 16:34:02 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Harrison Gregg ffef828a8d Add test for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Eugenio Lacuesta 63b8caf5de Feed exports: rewrite indentation test without .strip() 2017-05-09 11:58:53 -03:00
Daniel Graña 362d6f2d20 Merge pull request #2622 from rolando-contrib/download-maxsize-abort 
[MRG+1] Abort connection earlier and avoid to buffer data when max size limit is reached
 2017-05-04 11:44:51 -03:00
Paul Tremberth 6d14e392f1 Remove old test certificate+key 2017-04-27 23:35:01 +02:00
Paul Tremberth e6ab8bc9a5 Change "localhost" test server certificate 2017-04-27 23:22:25 +02:00
Rolando Espinoza 2b34c6edff Abort connection earlier and avoid to buffer data 
A symptom of this issue was having the log message "Received (X) bytes
larger than download max size (Y)" several times printed, with increased
X values.
 2017-04-27 13:28:16 -03:00