16 Commits

Author	SHA1	Message	Date
Paul Tremberth	11cdf58abe	Always decompress Content-Encoding: gzip at HttpCompression stage ... Let SitemapSpider handle decoding of .xml.gz files if necessary	2017-03-07 11:02:46 +01:00
Mikhail Korobov	a1e8a8525d	fix make_requests_from_url deprcation implementation, add tests	2017-02-17 00:18:29 +05:00
Eugenio Lacuesta	5ff64ad015	handle relative sitemap urls in robots.txt	2016-12-01 09:53:40 -03:00
orangain	25c56159b8	Fix SitemapSpider to extract sitemap urls from robots.txt properly ... This will fix #1766.	2016-02-06 23:54:07 +09:00
Mikhail Korobov	d5984bbea9	PY3 port scrapy.spiders	2015-08-28 04:11:30 +05:00
Julia Medina	d3f576a816	Move scrapy/spider.py to scrapy/spiders/__init__.py	2015-05-09 04:20:09 -03:00
Julia Medina	fc346cba4d	Move scrapy/contrib/spiders to scrapy/spiders	2015-04-29 21:27:19 -03:00
Julia Medina	cf064b1437	Move scrapy/contrib/linkextractors to scrapy/linkextractors	2015-04-29 21:24:30 -03:00
Julia Medina	1c8708eb82	Create a logger for every Spider and adapt Spider.log to log through it	2015-04-22 17:24:40 -03:00
Berker Peksag	31e5f164d4	Import unittest.mock if available. ... mock is in the stdlib since Python 3.3.	2015-03-08 15:03:04 +02:00
Julia Medina	9ef3972cfb	Per-spider settings tests	2014-09-01 21:56:57 -03:00
Julia Medina	870438e5f4	Update tests utils, fixing get_crawler and removing docrawl	2014-08-12 14:02:56 -03:00
Julia Medina	a995727117	Connect spider_closed signal after a crawler is bound to a Spider	2014-08-12 14:02:55 -03:00
Julia Medina	eb0253e530	Update from_crawler method as well as set_crawler on CrawlSpider	2014-08-11 11:24:01 -03:00
Julia Medina	84fa004793	Add from_crawler class method to base Spider	2014-08-11 11:23:57 -03:00
Daniel Graña	242c08553f	move tests under root dir	2014-07-30 19:45:08 -03:00