RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,470 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

6470 Commits

Author SHA1 Message Date
Lucy Wang acec4260d2 catch CertificateError in tls verification 2018-06-21 11:07:00 -03:00
Mikhail Korobov 65ac0b06df Merge pull request #2894 from redapple/log-custom-overriden-settings 
Move logging of overriden settings to Crawler init
 2017-08-29 19:20:46 +05:00
Pablo Hoffman a429d78019 update scrapinghub.com urls to use https 2017-08-24 16:03:36 -03:00
Paul Tremberth 7a35a1ad4a Remove trailing bracket from components versions log 2017-08-23 17:54:14 +02:00
Paul Tremberth 1968a8ec02 Move logging of overriden settings to Crawler init 2017-08-23 15:08:10 +02:00
Mikhail Korobov 1ba77f0837 Merge pull request #2869 from cclauss/patch-3 
# noqa to close #2836
 2017-08-21 19:15:30 +05:00
Mikhail Korobov 885289f497 Merge pull request #2854 from jenya/2853-sitemap-follow-alternate-fix 
[MRG+1] Follow alternate link for all types of sitemaps #2853
 2017-08-21 19:15:03 +05:00
Mikhail Korobov 984c0c19ee Merge pull request #2884 from iamminji/patch-1 
fix typo
 2017-08-16 16:36:00 +05:00
kim minji 1dcea6a9d4 fix typo 2017-08-16 18:07:52 +09:00
Daniel Graña d5f3543d77 Merge pull request #2865 from kirankoduru/2831-explicit-msg-for-scrapy-parse-callback 
[MRG+1] Explicit message for scrapy parse callback
 2017-08-09 07:56:57 -03:00
Chomba Ng'ang'a fd27cde24d Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Kiran Koduru 2960c9b568 Use self.__class__.__name__ instead of showing generic Spider class name 2017-08-05 16:29:41 -04:00
Kiran Koduru 12409a0cf6 Fix broken encoding on text for py 3 2017-08-05 15:40:38 -04:00
kirankoduru 7adab61a7a Added test for NotImplemented Spider.parse method 2017-08-05 15:40:38 -04:00
kirankoduru be71f98e92 Explicit message for scrapy parse callback 
The scrapy parse method raises a NotImplementedError when not defined,
but for new comers it can be hard to debug what might be going wrong.

Adding an explicit message for NotImplementedError will help new users.
 2017-08-05 15:40:38 -04:00
Paul Tremberth a68a8f8fdc Merge remote-tracking branch 'origin/1.4' 2017-08-05 11:45:50 +02:00
cclauss 0a69a32b5a Force Travis CI to test again 2017-08-04 14:35:43 +02:00
cclauss c016a4309d # noqa to close #2836 
Marks #2836 as will not fix.
 2017-08-04 01:44:23 +02:00
Paul Tremberth 0cb3085f84 Add test for alternate links 2017-08-03 16:55:09 +02:00
Eugene Vorobev 01ac883893 Follow alternate link for all types of sitemaps 2017-08-03 16:55:09 +02:00
david watson 71d5b7d75a fix typo (#2867) 2017-08-01 14:49:22 -03:00
Andrei Petre 6e6b5cc29f Use getfullargspec under the scenes for py3 to stop DeprecationWarning (#2864) 
Use getfullargspec under the scenes for py3 to stop DeprecationWarning. 

Closes #2862
 2017-08-01 11:14:43 -03:00
Daniel Graña 19382c5675 Merge pull request #2755 from redapple/downloader-mdw-template 
[MRG+1] Add template for a downloader middleware
 2017-07-31 11:37:26 -03:00
simik-ru a65fec050a Small fix in description of startproject arguments 2017-07-31 13:03:49 +02:00
Paul Tremberth aaaa4da7a4 Add template for a downloader middleware 2017-07-28 13:48:23 +02:00
Mikhail Korobov 9d34b2fe16 Merge pull request #2857 from redapple/scrapy-components-logs-startup 
Log versions information at startup
 2017-07-28 00:02:06 +05:00
Paul Tremberth bf7ef3e4c3 Move methods to a new scrapy.utils.versions 2017-07-27 20:08:30 +02:00
Paul Tremberth 219c8aa0b6 Log versions information at startup 2017-07-27 17:30:30 +02:00
Daniel Graña 5d9bac789d Merge pull request #2849 from cclauss/patch-2 
[MRG+1] xrange() --> range() for Python 3
 2017-07-26 15:59:37 -03:00
Daniel Graña bbc56e6960 Merge pull request #2852 from starrify/new-http-codes-to-retry-2851 
Added: HTTP status code 522/524 to retry.
 2017-07-26 15:46:35 -03:00
Daniel Graña 881a5e3a57 Merge pull request #2847 from redapple/redirect-308 
Handle HTTP 308 Permanent Redirect
 2017-07-26 15:29:23 -03:00
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Pengyu Chen 11a1f970b7
Added: HTTP status code 522/524 to retry. 2017-07-26 16:11:13 +08:00
cclauss 33dfac5018 xrange() --> range() for Python 3 
Either this PR or #2845.
 2017-07-24 22:06:17 +02:00
Mikhail Korobov 17bbd71433 Merge pull request #2812 from elacuesta/inspect_response_populate_spider 
Populate spider variable when using shell.inspect_response
 2017-07-25 00:28:14 +05:00
Daniel Graña b6d036ede9 Merge pull request #2837 from dguo/patch-2 
Fix a typo in the Items documentation
 2017-07-24 15:55:30 -03:00
Daniel Graña 6de106e7a8 Merge pull request #2763 from scrapy/dataloss-typo 
DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs
 2017-07-24 15:48:03 -03:00
Daniel Graña 12971a7b61 Merge pull request #2764 from scrapy/readme-cleanup 
DOC change "releases" section content
 2017-07-24 15:47:39 -03:00
Daniel Graña c01c1ef2ae Merge pull request #2655 from Granitosaurus/ptpython_support 
[MRG+1] add support for embeded ptpython shell
 2017-07-24 15:42:00 -03:00
Daniel Graña 2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña d1e948cef8 Merge pull request #2791 from starrify/doc-DontCloseSpider 
[MRG+1] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
 2017-07-24 15:29:38 -03:00
Daniel Graña 8e0b640d2b Merge pull request #2826 from dguo/patch-1 
[MRG+1] Tweak the CSVFeedSpider documentation
 2017-07-24 15:28:15 -03:00
Daniel Graña 45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Daniel Graña 50b2567d0d Merge pull request #2848 from redapple/tox-jessie-ssl 
[WIP] Jessie toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography
 2017-07-24 14:44:41 -03:00
Paul Tremberth 1a18587d41 Jessi toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography 2017-07-24 19:30:08 +02:00
Valdir Stumm Junior bb0bd691d9 Improve error message when callback is not callable 2017-07-24 11:12:09 -03:00
Danny Guo 26c488970c Fix a typo in the Items documentation 2017-07-18 19:56:51 -05:00