13 Commits

Author	SHA1	Message	Date
Paul Tremberth	6cd35c77da	Pass user-agent as native str when checking URLs against robots.txt	2016-11-15 17:38:32 +01:00
Paul Tremberth	28155dfccc	Parse robots.txt content as native str ... Fixes #2373	2016-11-09 12:20:06 +01:00
Artur Gaspar	dc8701ea42	Add test for already failed deferreds when downloading page in robots.txt middleware.	2016-01-27 12:56:42 -02:00
Artur Gaspar	1f4af9d81e	Test for robots.txt middleware for logger not being called when the request for robots.txt is ignored.	2015-09-02 10:33:53 -03:00
Artur Gaspar	668e5fd257	Test for robots.txt middleware for processing a request for which the robots.txt parser is ready.	2015-09-02 10:23:24 -03:00
Artur Gaspar	a6a629e707	Call actual error logger in robots.txt middleware tests.	2015-09-02 10:23:24 -03:00
Artur Gaspar	9ce9a293a6	Always check robots.txt before making another request in RobotsTxtMiddleware.	2015-09-02 10:23:24 -03:00
nyov	1a1bc2c6b5	PY3 port scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware	2015-07-29 04:21:57 +00:00
Victoria Terenina (torymur)	667864ae94	testing robotstxt error	2015-04-30 16:28:57 +03:00
Julia Medina	d7c444fefb	Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares	2015-04-29 21:24:30 -03:00
Berker Peksag	31e5f164d4	Import unittest.mock if available. ... mock is in the stdlib since Python 3.3.	2015-03-08 15:03:04 +02:00
Mikhail Korobov	36eec8f413	dont_obey_robotstxt meta key; don't process requests to /robots.txt	2014-09-23 00:10:43 +06:00
Daniel Graña	242c08553f	move tests under root dir	2014-07-30 19:45:08 -03:00