RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,401 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

33 Commits

Author SHA1 Message Date
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Eugenio Lacuesta 1b4385b7e3 Rule.process_request: move deprecation warnings and compiling code, update tests 2019-03-22 19:46:17 -03:00
Eugenio Lacuesta 01ed605d02 PEP8 changes to test_spider.py 2019-03-15 16:54:14 +00:00
Eugenio Lacuesta 22fda61d62 Rule.process_request: tests 2019-03-13 10:54:38 +00:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Mikhail Korobov 885289f497 Merge pull request #2854 from jenya/2853-sitemap-follow-alternate-fix 
[MRG+1] Follow alternate link for all types of sitemaps #2853
 2017-08-21 19:15:03 +05:00
Daniel Graña d5f3543d77 Merge pull request #2865 from kirankoduru/2831-explicit-msg-for-scrapy-parse-callback 
[MRG+1] Explicit message for scrapy parse callback
 2017-08-09 07:56:57 -03:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Kiran Koduru 12409a0cf6 Fix broken encoding on text for py 3 2017-08-05 15:40:38 -04:00
kirankoduru 7adab61a7a Added test for NotImplemented Spider.parse method 2017-08-05 15:40:38 -04:00
Paul Tremberth 0cb3085f84 Add test for alternate links 2017-08-03 16:55:09 +02:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Eugenio Lacuesta 5ff64ad015 handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
orangain 25c56159b8 Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-06 23:54:07 +09:00
Mikhail Korobov d5984bbea9 PY3 port scrapy.spiders 2015-08-28 04:11:30 +05:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Julia Medina 9ef3972cfb Per-spider settings tests 2014-09-01 21:56:57 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Julia Medina a995727117 Connect spider_closed signal after a crawler is bound to a Spider 2014-08-12 14:02:55 -03:00
Julia Medina eb0253e530 Update from_crawler method as well as set_crawler on CrawlSpider 2014-08-11 11:24:01 -03:00
Julia Medina 84fa004793 Add from_crawler class method to base Spider 2014-08-11 11:23:57 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00