HTTP auth middleware: added doc and unittest

docs/faq.rst

@@ -81,3 +81,8 @@ My Scrapy crawler has memory leaks. What can I do?
 
 See :ref:`topics-leaks`.
 
+Can I use Basic HTTP Authentication in my spiders?
+--------------------------------------------------
+
+Yes, see :class:`~scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware`.
+

docs/topics/downloader-middleware.rst

@@ -167,6 +167,32 @@ DebugMiddleware
     does not come enabled by default. Instead, it's meant to be inserted at the
     point of the middleware that you want to inspect.
 
+HttpAuthMiddleware
+------------------
+
+.. module:: scrapy.contrib.downloadermiddleware.httpauth
+   :synopsis: HTTP Auth downloader middleware
+
+.. class:: HttpAuthMiddleware
+
+    This middleware authenticates all requests generated from certain spiders
+    using `Basic access authentication`_ (aka. HTTP auth).
+
+    To enable HTTP authentication from certain spiders set the ``http_user``
+    and ``http_pass`` attributes of those spiders.
+
+    Example::
+
+        class SomeIntranetSiteSpider(CrawlSpider):
+
+            http_user = 'someuser'
+            http_pass = 'somepass'
+            domain_name = 'intranet.example.com'
+
+            # .. rest of the spider code omitted ...
+
+.. _Basic access authentication: http://en.wikipedia.org/wiki/Basic_access_authentication
+
 HttpCacheMiddleware
 -------------------
 

scrapy/tests/test_downloadermiddleware_httpauth.py

@@ -0,0 +1,28 @@
+import unittest
+
+from scrapy.http import Request
+from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware
+from scrapy.spider import BaseSpider
+
+class TestSpider(BaseSpider):
+    http_user = 'foo'
+    http_pass = 'bar'
+
+class HttpAuthMiddlewareTest(unittest.TestCase):
+
+    def setUp(self):
+        self.mw = HttpAuthMiddleware()
+
+    def tearDown(self):
+        del self.mw
+
+    def test_auth(self):
+        self.mw.default_useragent = 'default_useragent'
+        spider = TestSpider()
+        req = Request('http://scrapytest.org/')
+        assert self.mw.process_request(req, spider) is None
+        self.assertEquals(req.headers['Authorization'], 'Basic Zm9vOmJhcg==')
+
+
+if __name__ == '__main__':
+    unittest.main()